Remote client graphics rendering

A server computer hosts one or more application programs that are accessed by a client computer. Higher-level graphics commands describing graphics images are received from the application programs. The server computer determines whether the client computer is able to generate graphics using the higher-level graphics commands or generates graphics using relatively lower-level graphics commands. The server computer sends higher-level or relatively lower-level graphics commands depending on whether the client computer generates graphics using higher-level or relatively lower-level graphics commands.

TECHNICAL FIELD

This invention relates to the transmission of graphics to remote client computers.

BACKGROUND

A server computer may host application programs remotely accessed by client computers. Terminal service platforms, such as Windows® Server 2003 operating system provided by the Microsoft Corporation, is one implementation of such technology, where application programs run on a central server while a user interface is presented on a remote computer. The user interface in implementations such as this can include complex graphics.

The central server computer is referred to as a host computer or terminal server. The remote computer is referred to as a remote terminal or remote client, and communicates with the server through a communications medium such as a network. Bandwidth over such a communications medium is often limited.

Applications programs are typically designed to operate in conjunction with a local display device or monitor. In many operating systems, such as the Windows® Server 2003 operating system, application programs send relatively high-level graphics commands or primitives to operating system components. Such commands might specify or define colors, lines, shapes, and other graphics constructs. The operating system components interpret or convert such high-level commands into lower-level graphics information such as individual pixel values or bitmaps. Such a process of converting from a relatively high-level graphics specification to a relatively lower-level graphics specification will be referred to herein as rendering.

When operating on a terminal server, an application program can utilize the same operating system components to render low-level graphics data from higher-level commands or primitives. In this situation, however, the operating system components are configured to provide the rendered lower-level graphics commands to a remote client, which utilizes this information to control its display device.

When serving several remote computers, graphics data such as that described above can easily create network congestion. Because of this, attempts are made to optimize and/or compress the lower-level graphics data sent to the client. In some cases, however, such optimization and compression degrades the quality or resolution of the display data. Typically this is not a problem with simple media graphics; however, with rich media graphics having significantly more content, optimized and/or compressed graphics data may not adequately represent what was intended by the application program.

An option is to increase network bandwidth to allow uncompromised lower-level graphics data to be sent to clients. However, increasing network bandwidth adds expense and is not always feasible.

Furthermore, due to their relatively large content as compared to simple graphics, rich media graphics require greater server processing resources than simple graphics. As the server CPU requirements (i.e., CPU cycles) for each client session increases, the overall server scalability is reduced, where scalability is defined as the number of high fidelity concurrent connections to client computers the server can support. Thus the server is able to support fewer client computers.

SUMMARY

One or more client computers remotely access one or more application programs resident and executing on a server computer. The server computer determines the capability of client computers to use relatively higher-level graphics commands. Client computers generate graphics using either the relatively higher-level graphics commands or graphics primitives or relatively lower-level legacy graphics commands. The server computer provides either higher-level graphics commands or lower-level graphics commands to client computers, depending on the capacity of the client computers to generate graphics.

DETAILED DESCRIPTION

Client-Server Computer System

FIG. 1shows a server computer-client computer system100. The computer system100includes a server computer105connected to multiple client computers110(1)-110(3) connected via a network115. One or more application programs run on server computer105on behalf of the client computers110. The application programs generate graphics commands that are ultimately used to generate graphics on a display device of a remote client computer110.

The computer system100is representative of many different architecture including direct dialup via modem, enterprise LANs (local area networks), WANs (wide area networks) and the Internet. The network115may be implemented in a number of ways to support such networking contexts, including both wired-based technologies and wireless technologies. Aspects of this invention are not limited to one specific network architecture or network technology.

The server computer105may be implemented as a Windows® Server 2003, Windows® NT server, or any other server platform. A more detailed description of the server computer105is given below with respect toFIG. 6. Server computer105supports or hosts one more application programs accessed by client computers110through a communication protocol such as remote desktop protocol (RDP) as defined by the Microsoft Corporation. Through the communication protocol, server computer105determines if a client computer of client computers110is able to generate graphics using relatively higher-level graphics commands, commands that are received from the applications programs and used to generate graphics locally at the server computer105. Alternatively, if a client computer110is not able to generate graphics using the relatively higher-level graphics commands, relatively lower-level graphics commands which are rendered at the server computer105are sent to the client computer110.

The client computers110represent various kinds of computers or computing devices that may connect to server105over the network115. A more detailed example of the client computers110is given below with respect toFIG. 6. Client computer110(1) is a conventional desktop personal computer (PC), which includes a local operation system, processing unit, and storage system. As one example, the client computer110(1) is a general-purpose PC implemented with a Windows® brand operating system from the Microsoft Corporation.

Client computer110(2) is illustrated as a portable laptop computer, which may be connected to the network via a conventional network card or a modem connection. The client computer110(2) is standalone computer that can be configured with its own operating system, processing unit, and storage system. The client computer110(2) may likewise be configured to run a Windows® brand operating system.

Client computer110(3) is a handheld PC, which possesses less functionality than a general-purpose computer. The client110(3) is preferably equipped with a Windows® brand operating system, such as Windows® CE operating system from the Microsoft Corporation.

A display device120which includes a display and input devices such as a keyboard and mouse connects locally to server105. Display device120particularly provides user input to application programs resident on server computer105. Graphics are generated at display device120by a graphics composition engine resident in server computer105, where the graphics composition engine receives higher-level graphics commands from the resident application programs.

Architectures

The application programs resident on server computer105specify graphics in terms of relatively higher-level graphics commands or graphics primitives. For example, in the Windows® Server 2003 operating system environment, application programs can utilize graphical display interface (GDI) graphics components of the operating system. Using GDI graphics components application programs can provide the relatively higher-level graphics commands which can specify graphics in terms that specify shapes, colors, fonts, and other attributes. The GDI graphics components may also render the relatively higher-level commands to relatively lower-level commands.

FIG. 2shows example architectures at a server computer and a client computer. Server computer105includes an application program200, and a software graphics component205. Software graphics component205may include multiple components, as further described below. In general, software graphics component205is able to receive higher-level graphics commands from application program200. The higher-level graphics commands are used to generate graphics locally and displayed on display device120. Displaying graphics on display device120is particularly performed when “shadowing” is conducted where the same graphics generated and displayed at client computer110are generated and displayed at server computer105.

The server computer105communicates with the client computer110and determines whether the client computer105is able to generate graphics using higher-level graphics commands or lower-level graphics commands. In cases when the client computer105is able to generate graphics using the higher-level graphics commands, the server computer105sends the higher level graphics commands to the client computer110. In cases where the client computer110is not able to generate graphics using the higher-level graphics commands, the server computer105renders lower-level graphics commands such as bitmaps from the higher-level graphics commands, and sends the lower-level graphics commands to the client computer. A graphics component210receives either higher-level graphics commands or lower-level graphics commands, and generates graphics displayed on a display device215at the client computer110.

FIG. 3shows an example of more detailed architecture implementations at a server computer and a client computer. The architectures are particularly designed to provide higher-level graphics commands in the form of formatted graphics instructions to client computers such as client computers110that are capable of generating graphics using such higher-level graphics commands. Furthermore, for particular client computers110that use lower-level graphics commands or legacy graphics data to generate graphics, lower-level graphics commands are rendered from higher-level graphics commands data at the server computer105; received and used by the particular client computers110to generate graphics. In addition higher-level graphics commands are processed at the server computer105to generate and locally display graphics.

One or more application programs300(1)-300(N), such as application program200ofFIG. 2, may be hosted at the server computer105, and accessed by client computer110. Application programs300provide relatively higher-level graphics commands or are rendered to relatively lower-level graphics commands. Examples of higher-level graphics commands include instructions as to drawing a shape or line, providing a color gradient, and a vector (i.e. direction).

Application programs300may include a dedicated thread for graphics compositing or generating, where the dedicated thread is used to monitor a communication protocol (e.g., RDP) used in connecting compositor/UCE (unified composition engine)305(1)-305(N). A dedicated compositor/UCE305is provided for each of application programs300.

Compositor/UCE modules305are visual composition modules that support a media integration layer (MIL) that provides graphic information used by one or more client computer platforms (e.g., Windows® brand operating systems). In specific, compositor/UCE305processes or extracts higher-level graphics commands from an application program300. As described further below, the higher-level graphics commands are subsequently processed and sent to a client computer110, if client computer110is able to use higher-level graphics commands to generate graphics.

When one or more application programs300are running in a desktop or Windows® environment, a display or graphical image is shown that includes graphics from one more application programs300and a desktop or Window® environment. A desktop manager/UCE310provides higher-level graphics commands as to graphics provided by the desktop or Window® environment in which graphics from application programs300are combined.

Compositor/UCE305and desktop manager/UCE310provide higher-level graphics commands to an application program interface (API) exposer315. API exposer315is an interface that allows compositor/UCE305and desktop manager/UCE310to communicate with a display driver loader320.

For legacy client computers which render graphics using lower-level graphics commands processed from higher-level graphics commands at the server, higher-level graphics commands from application programs300are provided to a GDI/GDI+ renderer325. GDI/GDI+ renderer325renders the graphics primitives into lower-level graphics commands sent to display driver loader320.

Display driver loader320loads a remote display driver330and a local display driver335. Higher-level or lower-level graphics commands are sent to the remote display driver330which formats the graphics commands into an appropriate format as defined by the communication protocol used by the server and the client. In particular, remote display driver330may package, perform any caching on, and/or compress the graphics commands.

The local display driver335receives higher-level graphics commands generated into graphics locally at the server computer105. The local display driver335is used when displaying the graphics locally and/or when shadowing. Shadowing or remote assistance is performed when graphic images (i.e., higher-level or lower-level graphics commands) that are sent to the client computer are viewed at a local display device such as display device120ofFIG. 1.

Formatted higher-level or lower-level graphics commands from remote display driver330are sent to a transport driver335for transmission over a network or networks such as network115ofFIG. 1, to the client computer110. Transport driver335may be based on one of several transport protocols such as transmission control protocol/Internet protocol (TCP/IP). Transport driver335may package the formatted graphics primitives or graphics data into data packets.

Graphics commands (i.e., data packets) are received at the client computer110through a remote device interface345. Remote device interface345may be based on the communication protocol between the server computer105and client computer110, and may negotiate or inform the server computer105, through the use of the communication protocol, as to whether higher-level or lower-level graphics commands are to be sent to the client computer110.

Remote device interface345includes a higher-level graphics command processor350or a GDI/GDI+ rendering engine355for lower-level graphics commands, depending on whether the client computer110generates graphics using higher-level or lower-level graphics commands. In the case where the client computer110generates graphics using higher-level graphics commands, higher-level graphics command processor350extracts data packets that include higher-level graphics commands from a received stream of data from the server computer105. When the client computer105uses lower-level graphics commands to generate graphics, the received stream of data from the server includes lower-level graphics commands which are processed by the GDI/GDI+ rendering engine355and passed to a GDI/GDI+ graphics driver360which generates graphics. The generated graphics are shown on a local display device such as display device215ofFIG. 2.

When higher-level graphics commands are used by the client computer110to generate graphics, a client computer connector365receives the packets of data from higher-level graphics command processor350, and parses the packets into higher-level graphics commands. The parsed higher-level graphics commands are sent from the client computer connector365to a compositor/UCE370which generates graphics using the higher-level graphics commands.

FIG. 4shows a process400to connect server computer105to client computer110and provide higher-level or lower-level graphics commands. The process400takes place when client computer110accesses application programs at server computer105and receives higher-level or lower-level graphics commands in order to generate graphics.

The process400is illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer instructions that, when executed by one or more processors, perform the recited operations. The process400is described with reference to server computer105and client computers110ofFIG. 1, where server computer105provides for means to perform particular processes and one or more of client computers110provide means to perform particular processes.

At block405, the client computer110initiates a session with the server computer105in order to access an application program at the server computer105. The application program may be one of multiple application programs hosted by the server computer105and available to one or more client computers110. The session may be initiated by a user at the client computer110. As described above, the application programs provide graphics commands which may be passed on as formatted higher-level graphics commands or lower-level graphics commands used by the client computer110to generate graphics.

At block410, the client computer110connects to the server computer105using a particular communication protocol such as RDP. Extensions (e.g., data fields) may be provided in the communication protocol to allow the server computer105to determine attributes of the client computer110. In specific, such communication protocol extensions may be provided that allow the server computer105to determine if the client computer110generates graphics using higher-level or lower-level graphics commands, and to send either higher-level or lower-level graphics commands to the client computer110.

At block415, the server computer105registers the client computer110. In specific, registration information is provided by the client computer110to the server computer105. The registration information includes graphics generating capability of the client computer110(i.e., generating graphics using higher-level or lower-level graphics commands). Registration information may also include identifying whether the connection is to a remote client computer.

At block420, the server computer105through the registration information (i.e., information stored at block415) is notified as to a particular connection which may be a remote connection to client computer110. In other situations, the connection may be a local connection or reconnection.

At block425, the server computer105gets system metrics or metrics relevant to the server computer and client computer that describe whether a particular session provides higher-level graphics commands and/or lower-level graphics commands to a local display device (e.g., display device120), to the client computer110, or to both the local display device and the client computer110. When higher-level graphics commands and/or lower-level graphics commands are provided both to the local display device and the client computer, shadowing takes place such that graphics generated at the client computer110are generated or monitored locally at the server computer105.

At block430, the server computer105determines the capability of the client computer110to receive and use either higher-level or lower-level graphics commands. Information as to the capability of the client computer110in generating graphics using either higher-level or lower-level graphics commands may be extracted from registration information as described in block415.

At block435, a data packet is sent from the server computer105to the client computer110. The data packet may be part of a data stream and includes either higher-level graphics commands or lower-level graphics commands processed, formatted, and packaged as described inFIG. 3.

At block440, the client computer110receives the data packet. The data packet which contains higher-level or lower-level graphics commands is processed and rendered into graphics at the client computer110as described inFIG. 3

FIG. 5shows a process500to provide higher-level or lower-level graphics commands to a client computer, and higher-level graphics commands used to generate graphics locally when shadowing takes place.

The process500is illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer instructions that, when executed by one or more processors, perform the recited operations. The process500is described with reference to server computer105ofFIG. 1, where server computer105provides for means to perform particular processes.

At block505, the server computer105gets system metrics which are relevant to a particular session between the server computer105and client computer110. The system metrics describe whether the particular session is to provide higher-level graphics commands used to generate graphics to be displayed to a local display device (e.g., display device120); higher-level or lower-level graphics commands to the client computer110; or whether shadowing is to take place. The system metrics may be extracted from registration information as described above in block415ofFIG. 4.

When shadowing takes place a local compositing or generating of graphics from higher-level graphics commands takes place. In certain cases, graphics will only be generated and displayed locally. In other words, there may be sessions in which client computer110is not involved and remote graphics generating is not performed (i.e., following the “NO” branch of block510). At block515local graphics generating takes place. The compositor/UCE305ofFIG. 3provides higher-level graphics commands from application programs, which are used to generate graphics locally.

For sessions where graphics generating at client computer110takes place (i.e., following the “YES” branch of block510), at block520server computer105determines the system metrics of client computer110which includes whether client computer110may be a legacy client computer which uses lower-level graphics commands, or whether client computer110uses higher-level graphics commands to perform graphics generating.

If client computer110is able to receive higher-level graphics commands or a graphics packet with higher-level graphics commands used to generate graphics (i.e., following the “YES” branch of block525), at block530client computer110is sent the higher-level graphics commands which may be in the form of a packet and compositing or generating of graphics is performed by client computer110.

If client computer110is not able to receive (i.e., does not use) higher-level graphics commands, client computer110may be a legacy computer (i.e., following the “NO” branch of block525). At block535, lower-level graphics commands are sent to client computer110to allow for graphics generating at client computer110.

In certain cases, shadowing takes place in which graphics images represented by higher-level or lower-level graphics commands sent to client computer110are also viewed at server computer105through a local display device such as display device120ofFIG. 1. If shadowing takes place (i.e., following the “YES” branch of block540), local graphics generating is performed as to block515. If shadowing does not take place (i.e., following the “NO” branch of block540), at block545only remote graphics generating is performed at client computer110(as to blocks530or535).

Server and Client Computers

FIG. 6shows an example implementation of a computer600which may be a server computer such as server computer105or a client computer such as client computers110. As appreciated by those skilled in the art, particular components described in reference toFIG. 6are not included in particular implementations. For example, client computer110(3) which is a handheld PC is a more simplified implementation compared to a desktop PC such as client computer110(1), and may not need particular components (e.g., certain storage devices). Furthermore, particular implementations may use different components to provide similar functionality as components described in reference toFIG. 6. In addition, as discussed below a server computer implementation may include particular application programs and graphics drivers.

Computer600may be configured with a Windows® brand operating system. The computer600includes a processing unit605, a system memory610, and a system bus615that interconnects various system components, including the system memory610to the processing unit605. The system bus615may be implemented as any one of several bus structures and using any of a variety of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus.

The system memory610includes read only memory (ROM)620and random access memory (RAM)625. A basic input/output system630(BIOS) is stored in ROM620.

The computer600has one or more of the following drives: a hard disk drive630for reading from and writing to a hard disk or hard disk array, a magnetic disk drive635for reading from or writing to a removable magnetic disk640, and an optical disk drive for reading from or writing to a removable optical disk650such as a CD ROM or other optical media. The hard disk drive630, magnetic disk drive635, and optical disk drive645are connected to the system bus615by a hard disk drive interface660, a magnetic disk drive interface665, and an optical drive interface670, respectively. The drives and their associated computer-readable provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for computer600.

Although hard disk630, removable magnetic disk640, and removable optical disk650are described, other types of computer readable media can be used to store data. Other such media include magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like. Additionally, the computer600may be configured to serve data stored on an independent system, such as a RAID (redundant array of independent disks) storage system, particularly when implemented as a terminal server.

A number of program modules may be stored on the hard disk630, magnetic disk640, optical disk650, ROM620, or RAM625. The programs include a server operating system675, one or more application programs680, other program modules682, and program data684.

A user may enter commands and information into the computer600through input devices such as keyboard686and a mouse688. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, and the like. These an other input devices are connected to the processing unit605through a serial port interface690that is couple to the system bus615, but may alternatively be connected by other interfaces, such a parallel port, game port, or a universal serial bus (USB).

A monitor692or other type of display is also connected to the system bus615via an interface, such as a video adapter card694. The computer600has a network interface or adapter696, a modem698or other means for establishing communications over the network115, such as an Internet connection. The modem698may also facilitate connection from another computer. In a server computer implementation, monitor692and input devices such as keyboard686and mouse688may be considered as a display device such as display device120ofFIG. 1.

CONCLUSION

The above-described server sends graphics primitives or graphics data to client computers which render graphics primitives or graphics data to graphics representative of graphics from application programs resident at the server. Although the invention has been described in language specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed invention.