Remote desktop protocol (RDP) tile image classification for improving compression efficiency and quality

Systems, methods and computer readable media are disclosed for improving compression efficiency and quality in a remote session via tile image classification and variable encoding. A server determines a set of codecs that are shared by both the server and a corresponding client. Then, when it receives an image, it determines whether classification of the image is required. Where classification of the image is not required, the server sends the client the image, either uncompressed or compressed with a default codec and default fidelity. Where classification of the image is required, the server classifies the image (e.g. the image comprises either text or photograph), and based on that classification determines a codec with which to encode the image, and a fidelity to use on the encoding. The server performs that encoding with the codec and the fidelity, and then sends this encoded image to the client.

BACKGROUND OF THE INVENTION

Although computers were once isolated and had minimal or little interaction with other computers, today's computers interact with a wide variety of other computers through communications networks, such as Local Area Networks (LANs) and Wide Area Networks (WANs). With the wide-spread growth of the INTERNET™, connectivity between computers is becoming more important and has opened up many new applications and technologies. The growth of large-scale networks, and the wide-spread availability of low-cost personal computers, has fundamentally changed the way that many people work, interact, communicate, and play.

One increasing popular form of networking may generally be referred to as virtual computing systems, which can use protocols such as Remote Desktop Protocol (RDP), Independent Computing Architecture (ICA), and others to share a desktop and other applications with a remote client over a remote session. Such computing systems typically transmit the keyboard presses and mouse clicks or selections from the client to a server, relaying the screen updates back in the other direction over a network connection (e.g., the INTERNET). As such, the user has the experience as if their machine is operating as part of a LAN, when in reality the client device is only sent screenshots of the applications as they appear on the server side.

To reduce the often limited network bandwidth required for a session, the server may compress an image (frequently, the unit of images is a tile, which comprises a subset of a frame) before sending the image across the network, and the client then performs the corresponding decompression operation after receiving the image. These images are generated on the server as the remote session progresses, so the compression must occur at runtime. Additionally, since the latency of a session is a major factor in the user experience, such compression and decompression must be performed quickly relative to the time savings of sending a smaller compressed image as opposed to a larger uncompressed image. Given these requirements, present runtime classification systems primarily use very constrained criteria to decide on how to encode image data, such as those based on basic content type and/or those that take a running average of network capacity. Both of these techniques may be used for a single type of image, such as text, or high-resolution video. These systems have only a single codec in use at any one time, and the time required to switch codecs takes so long that it negatively impacts the responsiveness of the RDP session.

SUMMARY OF THE INVENTION

It would therefore be an improvement to implement a more generic, modular system that allows multiple factors to be included in a compression decision, the decision including not only the quality setting to use for any particular codec, but also choosing between multiple codecs simultaneously based on the particulars of the remote client. It would additionally be an improvement to institute a hybrid approach where tiles are classified through a preliminary step that determines the preferred type of compressor for the content by taking into account considerations such as the image content (e.g. high-detail text or video), network conditions (e.g. the desired bandwidth used), and CPU load.

In an exemplary embodiment, a server determines a set of codecs that are shared by both the server and a client with which it is conducting a remote session. Then, when it receives an image, it determines whether classification of the image is required. Where classification of the image is not required, the server sends the client the image, either uncompressed or compressed with a default codec and default fidelity. Where classification of the image is required, the server classifies the image (e.g. the image comprises either text or photograph), and based on that classification determines a codec with which to encode the image, and a fidelity to use on the encoding. The server performs that encoding with the codec and the fidelity, and then sends this encoded image to the client.

It can be appreciated by one of skill in the art that one or more various aspects of the disclosure may include but are not limited to circuitry and/or programming for effecting the herein-referenced aspects of the present disclosure; the circuitry and/or programming can be virtually any combination of hardware, software, and/or firmware configured to effect the herein-referenced aspects depending upon the design choices of the system designer.

The foregoing is a summary and thus contains, by necessity, simplifications, generalizations and omissions of detail. Those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 1is a block diagram of a general purpose computing device in which the techniques described herein may be employed. The computing system environment120is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the presently disclosed subject matter. Neither should the computing environment120be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment120. In some embodiments the various depicted computing elements may include circuitry configured to instantiate specific aspects of the present disclosure. For example, the term circuitry used in the disclosure can include specialized hardware components configured to perform function(s) by firmware or switches. In other examples embodiments the term circuitry can include a general purpose processing unit, memory, etc., configured by software instructions that embody logic operable to perform function(s). In example embodiments where circuitry includes a combination of hardware and software, an implementer may write source code embodying logic and the source code can be compiled into machine readable code that can be processed by the general purpose processing unit. Since one skilled in the art can appreciate that the state of the art has evolved to a point where there is little difference between hardware, software, or a combination of hardware/software, the selection of hardware versus software to effectuate specific functions is a design choice left to an implementer. More specifically, one of skill in the art can appreciate that a software process can be transformed into an equivalent hardware structure, and a hardware structure can itself be transformed into an equivalent software process. Thus, the selection of a hardware implementation versus a software implementation is one of design choice and left to the implementer.

Computer141typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer141and includes both volatile and nonvolatile media, removable and non-removable media. The system memory122includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)123and random access memory (RAM)160. A basic input/output system124(BIOS), containing the basic routines that help to transfer information between elements within computer141, such as during start-up, is typically stored in ROM123. RAM160typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit159. By way of example, and not limitation,FIG. 1illustrates operating system125, application programs126, other program modules127, and program data128.

The computer141may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,FIG. 1illustrates a hard disk drive138that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive139that reads from or writes to a removable, nonvolatile magnetic disk154, and an optical disk drive140that reads from or writes to a removable, nonvolatile optical disk153such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive138is typically connected to the system bus121through an non-removable memory interface such as interface134, and magnetic disk drive139and optical disk drive140are typically connected to the system bus121by a removable memory interface, such as interface135.

The drives and their associated computer storage media discussed above and illustrated inFIG. 1, provide storage of computer readable instructions, data structures, program modules and other data for the computer141. InFIG. 1, for example, hard disk drive138is illustrated as storing operating system158, application programs157, other program modules156, and program data155. Note that these components can either be the same as or different from operating system125, application programs126, other program modules127, and program data128. Operating system158, application programs157, other program modules156, and program data155are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer141through input devices such as a keyboard151and pointing device152, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit159through a user input interface136that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor142or other type of display device is also connected to the system bus121via an interface, such as a video interface132. In addition to the monitor, computers may also include other peripheral output devices such as speakers144and printer143, which may be connected through a output peripheral interface133.

The computer141may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer146. The remote computer146may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer141, although only a memory storage device147has been illustrated inFIG. 1. The logical connections depicted inFIG. 1include a local area network (LAN)145and a wide area network (WAN)149, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer141is connected to the LAN145through a network interface or adapter137. When used in a WAN networking environment, the computer141typically includes a modem150or other means for establishing communications over the WAN149, such as the Internet. The modem150, which may be internal or external, may be connected to the system bus121via the user input interface136, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer141, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,FIG. 1illustrates remote application programs148as residing on memory device147. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

Referring now toFIG. 2, it generally illustrates an example environment wherein aspects of the present disclosure can be implemented. One skilled in the art can appreciate that the example elements depicted byFIG. 2provide an operational framework for describing the present disclosure. Accordingly, in some embodiments the physical layout of the environment may be different depending on different implementation schemes. Thus the example operational framework is to be treated as illustrative only and in no way limit the scope of the claims. One skilled in the art can also appreciate that the following discussion is introductory and the elements depicted byFIG. 2are described in more detail within the discussion of the operational procedures ofFIG. 3throughFIG. 8.

Generally,FIG. 2depicts a high level overview of a terminal server environment that can be configured to include aspects of the present disclosure. In reference to the figure, a server204is depicted that can include circuitry configured to effectuate a terminal server and for example, three example clients201,202, and203(while three clients are depicted the server204in embodiments can service more or less clients). The example clients201-203can include computer terminals effectuated by hardware configured to direct user input to the server204and display user interface information generated by the server204. In other embodiments, clients201-203can be computers that include similar elements as those of computer20FIG. 1. In these example embodiments, clients201-203can include circuitry configured to effect operating systems and circuitry configured to emulate the functionality of terminals. In these examples one skilled in the art can appreciate that the circuitry configured to effectuate the operating systems can also include the circuitry configured to emulate terminals.

In the depicted example, the server204can be configured to generate one or more sessions for connecting clients201,202, and203such as sessions1through N (where N is an integer greater than 1). Briefly, a session in example embodiments of the present disclosure can generally include an operational environment that is effectuated by a plurality of subsystems, e.g., software code, that are configured to effectuate an execution environment and interact with a kernel218an operating system214. For example, a session can include a shell and a user interface such as a desktop, the subsystems that track mouse movement within the desktop, the subsystems that translate a mouse click on an icon into commands that effectuate an instance of a program, etc. In another example embodiment the session can include an application. In this example while an application is rendered, a desktop environment may still be generated and hidden from the user. The session in this example can include similar subsystems as the session described above. Generally, a session can be generated by the server204on a user by user basis when, for example, the server204receives a connection request over a network connection from a client such as client201. Generally, a connection request can first be handled by the transport logic210that can, for example, be effectuated by circuitry of the server204. The transport logic210can in some embodiments include a network adaptor, firmware, and software that can be configured to listen for connection messages and forward them to the engine212. As illustrated byFIG. 2, when sessions are generated the transport logic210can include protocol stack instances for each session. Generally, each protocol stack instance can be configured to route user interface output to an associated client and route user input received from the associated client to the appropriate session core244.

As depicted byFIG. 2, during the session generation process the engine212can be configured to obtain a license for the session. For example, in one example embodiment the engine212can receive a license from the client201during the session generation process. In other example embodiments the engine212can receive a copy of a license from a license database222. In some embodiments of the present disclosure the license database222can include a relational database management program that can be executed on an operating system of a computer such as computer20ofFIG. 1. In an example embodiment that includes a license database222, it can store one or more licenses that can be checked out when a client attempts to obtain a session from the server204. In another embodiment each license can itself be associated with an account identifier, e.g., a username/password combination, a smartcard identifier, etc., and each license can only be checked out if the correct account identifier is presented. Generally, the number of connections that a server204can generate can be dependent upon the number of licensees the entity that controls the server204has purchased from a service provider. If for example, the entity has purchased one license, then the server204can be configured to only allow one session. In this example if the license is associated with an account identifier, then only a user that presents the correct account identifier can obtain a session.

In example embodiments of the present disclosure each license can be validated by a service provider262before they can be used. For example, the service provider262can in example embodiments act as a certificate authority that aphorizes and activates licenses and servers. In these embodiments the service provider262can ensure that licenses are not stolen, copied, or pirated. The service provider262can also ensure that the license are only used by the server204they are purchased for by storing a copy of the licenses in a database and associating the licenses with server204.

As illustrated byFIG. 2, a configuration manager224in an example embodiment of the present disclosure can include computer readable instructions that when executed instantiate a process that can receive a license during the session creation process and determine a service level for a newly spawned session by interfacing with various subsystems such as session manager216. The session manager216in an embodiment can be configured to initialize and manage each session by for example, generating a session identifier for a session space; adding the session identifier to a table; assigning memory to the session space; and generating system environment variables and instances of subsystem processes in memory assigned to the session space. As illustrated byFIG. 2, in an embodiment the session manager216can instantiate environment subsystems such as a runtime subsystem240that can include a kernel mode part such as the session core244. For example, the environment subsystems in an embodiment can be configured to expose a subset of services to application programs and provide an access point to the kernel218of the operating system214. As illustrated byFIG. 2, in some embodiments the kernel218can include a security subsystem250and a resource manager256. In an example embodiment the security subsystem250can enforce security policies of the server204by, for example, performing run-time object protection. In these embodiments the resource manager256in an embodiment can create and terminate processes and threads in response to requests from the runtime subsystem240. More specifically, in an embodiment the runtime subsystem240can request the execution of threads and the session core244can send requests to the executive of the kernel218to allocate memory for the threads and schedule time for them to be executed.

Continuing with the description ofFIG. 2, in an embodiment the session core244can include a graphics display interface246(GDI) and an input subsystem252. The input subsystem252in an example embodiment can be configured to receive user input from a client201via the protocol stack instance associated with the session and transmit the input to the session core244. The user input can in some embodiments include signals indicative of absolute and/or relative mouse movement commands, mouse coordinates, mouse clicks, keyboard signals, joystick movement signals, etc. User input, for example, a mouse double-click on an icon, can be received by the session core244and the input subsystem252can be configured to determine that an icon is located at the coordinates associated with the double-click. The input subsystem252can then be configured to send a notification to the runtime subsystem240that can execute a process for the application associated with the icon.

In addition to receiving input from a client201, draw commands can be received from applications and/or a desktop and processed by the GDI246. The GDI246in general can include a process that can generate graphical object draw commands. The GDI246in this example embodiment can be configured to pass the commands to the remote display subsystem254that can instantiate a display driver for the session. In an example embodiment the remote display subsystem254can be configured to include virtual display driver(s) that may not be associated with displays physically attacked to the server204, e.g., the server204could be running headless. The virtual display driver in this embodiment can be configured to receive the draw commands and transmit them to the client201via a stack instance associated with the session.

FIG. 3illustrates An embodiment of a client314and server302communicating via a remote session that utilize variable compression based on image classification.

The server302and the client314maintain a remote session over a network310. The RDP client316on the client computing device314issues commands to the RDP server304on the server computing device302, which the server302executes to produce results and sends those results to the client314for display on display device320. As the session proceeds, the server302sends the client314a plurality of image frame308s. The server302divides each image frame308into a plurality of tiles310.

For each tile310of each frame308, the RDP server304determines whether compression is to be used, if so, compresses the tile310.

The first stage of this process comprises a pre-filter that determines if any classification of any kind is required. This decision takes into account the capabilities of both the server302and client314or clients, the shared codecs available between parties to the remote session, the type of network310connection in use and requirements of the particular scenario for which the remote session is being used (e.g. a medical application). For example, on a very high speed connection or when handling medical images, only currently supported lossless codecs would be utilized. The image classification stage that follows has the potential to require a great deal of computing resources, and the use of those resources can be avoided where it is determined at this point that classification is not required.

Different available codecs have different abilities. For instance, one codec may greatly compress an image and preserve image quality for monochromatic text on a monochromatic background, but greatly reduce image quality for a photograph. A codec that preserves photographic image quality may not be able to compress that text as well as the first technique.

The second stage comprises image classification. If some level of bandwidth reduction is required, then the image to be encoded is then classified into a number of subtypes such as text on a monochromatic background, text on gradient backgrounds, simple image segments and dense image segments. The classification system is based on a histogram analysis of the image and may, depending on its content, recursively analysis the image (i.e. the cost of analyzing any particular image is not constant but based on contents).

The third stage comprises fidelity control. This is where a weighted decision function combines the networking characteristics, CPU loading, image characteristics, and codec capabilities to decide which codec and which fidelity options to use. For example if bandwidth is low, CPU availability is high and the image comprises dense graphics, then a lossy codec can be chosen with a lower quality setting, such as sub-sampling. Alternatively if the image comprises high quality text, and bandwidth is available, then a non-lossy codec can be chosen.

The fourth stage comprises codec invocation. The codec is invoked to encode the bitmap image data. If this stage fails, for example, due to lack of available resources to encode the image in a short enough time, then the fidelity control can be used to choose a different codec and/or fidelity.

Where the image has been processed as described above, it is then sent by the RDP server304to the RDP client316of the client314across the communications network310. The client314receives the image, decodes it with the corresponding codec, and displays it on the client display device320.

FIG. 4illustrates exemplary operational procedures for variable compression of an image for remote display from a server302to a client314during a session across a communications network310.

Operation402depicts determining a set of shared codecs. In an embodiment, the set of shared codecs comprises each codec that is in both a set of server codecs and a set of client codecs. In an embodiment, the set of server codecs comprises those image formats that the server may encode an image to. In an embodiment, the set of client codecs comprises those image formats that the client may decode an image from. In an embodiment, the client sends the server a the set of client codecs when the remote session is initialized. In an embodiment, the client sends the server an updated list, either a partial one that notes only additions and/or deletions or a full list, when its decoding capabilities have changed during the session.

Optional operation404depicts determining whether classification of images is required, and performing operations406-412of classifying the image, determining the codec, determining the fidelity, and encoding the image when classification is required. In an embodiment, determining whether classification of images is required based on at least one capability of the server, at least one capability of the client, the set of shared codecs, a scenario of the session, and/or a connection across the communications network. For example, it may be determined that classifying the image is not required where the set of shared codecs comprises only one codec, so that is the only codec that may be used. It may also be determined that classifying the image is not required where the amount of available bandwidth is great enough that uncompressed images may be sent across the network.

Operation406depicts classifying the image to produce a classification. In an embodiment, this comprises classifying the image using a histogram of component frequencies of the image. In an embodiment, this comprises recursively classifying the image to produce the classification. In an embodiment, this comprises classifying the image as text on a monochromatic background, text on a gradient background, a simple image segment (e.g. a line drawing), or a dense image segment (e.g. a photograph).

In an embodiment, classifying the image comprises performing the classification based on a hint. The hint may comprise an application programming interface (API) call from an application that generated the image, a manifest for the application that generated the image, or an administrator policy. Where the application is aware that a remote session exists, it may make an API call that is detected by the server to indicate what the image is. For instance, the API call may indicate that the corresponding image comprises text on a gradient background, so the image does not need to use computing resources to generate a histogram to make the classification.

Operation408depicts determining a codec of the shared codecs based on the classification, and determining a fidelity based on the classification. In an embodiment, the remote session comprises a scenario, and determining the codec and determining the fidelity includes determining the codec based on the scenario, and determining the fidelity based on the scenario. In an embodiment, the scenario comprises a medical scenario where all photographs (likely to be x-rays or other images for which detail is paramount) must be sent in a lossless format.

In an embodiment, determining the codec and/or the fidelity is done so based on a hint. The hint may comprise an application programming interface (API) call from an application that generated the image, a manifest for the application that generated the image, or an administrator policy. Where the application itself does not generate a hint, or where the application does, but it should be overridden, a server administrator policy may provide that hint. For example, in a hospital, it may be determined that enough x-rays are being transferred that all images should be treated as high quality where only lossless encoding is acceptable, and that may be set as the server administrator policy that the RDP server may access in determining how to encode the image.

In an embodiment, a set of codecs comprises metadata for each codec in the set of codecs, and determining the codec comprises determining the codec based on the metadata of the set of shared codecs. In an embodiment, metadata for a codec comprises whether the client can hardware-accelerate decoding the corresponding codec. Some codecs, such as Joint Photographic Experts Group (JPEG), have many different implementations that operate at different speeds. The metadata may comprise the implementation of the codec. The metadata may comprise how well the client can handle the codec in other ways. For instance, where decoding a codec requires significant memory resources and the client has little memory, the metadata may indicate that the client handles this codec poorly.

In an embodiment, determining the fidelity comprises determining the fidelity using a weighted decision function. In an embodiment, the weighted decision function is based on at least one of at least one network characteristic, a CPU load of the server, a CPU load of the client, at least one characteristic of the image, or at least one capability of the codec. For instance, where the network is fast and has ample bandwidth, it may be determined that there is no transfer benefit to a smaller image size that outweighs the loss of quality, so only lossless codecs will be used.

Operation410depicts encoding the image according to the codec and the fidelity.

Optional operation412depicts when encoding the image fails, determining a second codec and a second fidelity based on the classification, and encoding the image with the second codec and the second fidelity. Encoding the image could fail because insufficient computing resources were available with which to encode it during a prescribed time period. In this instance, a second codec and second fidelity could be selected such that this second encoding requires much less computing resources are required, so the image can be encoded more quickly.

Operation414depicts sending the encoded image to the client across the communications network. In an embodiment, this comprises sending the encoded image to the client according to a remote desktop protocol (RDP).

Optional operation416depicts classifying a second image to produce a second classification; determining a second codec of the shared codecs based on the classification, the second codec differing from the first codec; determining a second fidelity based on the classification; encoding the second image according to the second codec and the second fidelity; and sending the second encoded image to the client across the communications network. Multiple codecs may be used, and where two images are classified in different ways, such that it would be beneficial to the remote session that separate codecs are used to encode them (such as the maximum compression ratio for each image may only be achieved by using different codecs), then a second image may be separately classified and encoded with a second codec as compared to a first image.

Conclusion

While the present disclosure has been described in connection with the preferred aspects, as illustrated in the various figures, it is understood that other similar aspects may be used or modifications and additions may be made to the described aspects for performing the same function of the present disclosure without deviating therefrom. Therefore, the present disclosure should not be limited to any single aspect, but rather construed in breadth and scope in accordance with the appended claims. For example, the various procedures described herein may be implemented with hardware or software, or a combination of both. Thus, the methods and apparatus of the disclosed embodiments, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium. When the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus configured for practicing the disclosed embodiments. In addition to the specific implementations explicitly set forth herein, other aspects and implementations will be apparent to those skilled in the art from consideration of the specification disclosed herein. It is intended that the specification and illustrated implementations be considered as examples only.