Filtering obscured data from a remote client display

Present implementations include one or more applications at a sending computer system that generate data intended for display at a receiving computer system over a network connection. A mirror driver at the sending computer system renders shared data in a bitmap, which is sent to the receiving computer system. The mirror driver also renders one or more simple image files in place of unshared data that is not intended for display, and sends the replacement images to the receiving computer system. The receiving computer system, in turn, renders what it receives using its existing resources. In one implementation, a driver interface mediates at the sending computer system between the one or more applications generating data, one or more display drivers, and the mirror driver. Thus, the driver interface can pass data to a normal display driver, and also to the mirror driver, which then filters the data as appropriate.

BACKGROUND

Background and Relevant Art

As computerized systems have increased in popularity, so have the needs to distribute files and processing resources of computer systems in networks both large and small. In general, computer systems and related devices communicate information over a network for a variety of reasons, for example, to exchange personal electronic messages, sell merchandise, provide account information, and so forth. One will appreciate, however, that as computer systems and their related applications have become increasingly more sophisticated, the challenges associated with sharing data and resources on a network have also increased.

Some current ways for distributing resources within an organizational network include centralized computing scenarios, which might involve a centralized network server sharing resources with one or more clients that do not have those resources installed locally. One such protocol that has been used for this type of functionality is the Remote Desktop Protocol (“RDP”). With the Remote Desktop Protocol, a client computer system can access a centralized network server, which hosts resources of interest. The client computer system can also interact (e.g., sending mouse and keyboard events, etc.) with those resources just as though those resources were installed locally.

The network server in turn, processes those interactions, creates corresponding rendering information of the data using its own video driver, and sends both the processed data and created rendering information back to the client. The client computer system then receives the data and rendering information, and uses a client-side video driver to render and display the received data locally. Ideally, this interaction between the client computer system and network server occurs seamlessly as though the client computer system had actually processed the data locally with its own installed resources. Unfortunately, such systems can be subject to network throughput constraints, which, if burdened, might result in a “lag” between what the local client computer system views in terms of interaction and processing. Furthermore, such systems do not typically have a concept of “shared” and “unshared” resources, such that some windows at the server can be viewed by a local client, but not others. Rather, everything that is open at the network server for the client session can be delivered to the client computer system.

Another type of system that is similar in some respects to the centralized sharing model described above involves a broadcasting (or “sending”) client computer system configured to send window data information to other recipient client computer systems on a network. This feature is also sometimes called “desktop sharing.” In this example, the broadcasting computer (e.g., an “instructor” in a learning environment) and the recipient computer systems (e.g., “students”) connect using a commonly-installed application program that allows for sharing of desktop views and locally installed applications at the instructor computer. Similar to the centralized computing system scenario, the client computer systems might be able to interact with the windows displayed at the instructor computer as though the windows were the student computer's own.

In contrast with the client-server model using RDP described above, this system can implement the concepts of shared and unshared window data. For example, the instructor might not want to share all its open windows (e.g., an Internet browser opened to a bank account) with the student, and thus might designate that the window containing the solution is shared, while the window containing sensitive information is unshared. Nevertheless, the instructor's computer system will still encode data from the unshared window, and then send both the shared and unshared window data to the student's computer system. As a result, the student's computer system will render the shared windows correctly, but will be unable to render the unshared window data due to its unique encoding, despite trying to process it. Thus, the student's computer system might produce some default generic lines and squares where the unshared window data would otherwise be displayed.

Similar to the centralized RDP model described above, this instructor/student(i.e., sender/receiver) type of system also has some disadvantages that make it difficult to apply to centralized desktop sharing environments. For example, such a system typically requires both the sending and receiving programs to have the same application programs and corresponding updates installed locally so that shared and unshared concepts are appropriately managed. In addition, there are certain inefficiencies when encoding unshared data, then requiring the receiving computer to attempt to decode the unshared window data, and make a determination on what to display when the rendering is unsuccessful. Furthermore, security concerns can be present when sending the unshared —albeit encoded—data, since there could still be a possibility that an unscrupulous agent might capture this unshared data and successfully decode it

By contrast, it is not a simple matter to just update the protocol, such as RDP, for consideration of the concepts of shared and unshared window data. In particular, updating such a protocol typically involves an administrator needing to take the time to provide and install corresponding extensions at both the server and each client computer system that is sharing resources from the server over the protocol.

Accordingly, there are a number of aspects of current resource sharing protocols that make it difficult to combine the advantages of such protocols in a desktop sharing environment.

BRIEF SUMMARY

Implementations of the present invention solve one or more of the foregoing problems in the art with systems, methods, and computer program products that enhance the efficiency of application display sharing between networked computer systems. In particular, at least one implementation of the present invention involves a server, or sending computer, identifying areas of a display screen that are meant to be shared, as well as those that are meant not to be shared. The server can then substitute the unshared content with one or more simple image files, and transmit the sample image files in place of the unshared content. As such, sensitive information can be obscured without necessarily requiring any significant processing action from the client computer system.

For example, one method of filtering shared and obscured application commands in accordance with one or more implementations of the present invention involves identifying one or more shared drawing commands that are intended to be displayed at least at a remote client. The method further involves identifying one or more obscured drawing commands that are intended to be hidden from the remote client. In addition, the method involves rendering the one or more shared drawing commands in a bitmap, as well as rendering a replacement image of the one or more obscured drawings commands in the bitmap. As such, the replacement image can be sent in place of content that is intended to be obscured.

In addition, a method of implementing a pause feature when sharing a display with a remote client can involve identifying a plurality of drawing commands from one or more applications at a server system. In this case, at least one of the plurality of drawing commands is intended to be displayed at a client computer system. In addition, the method involves receiving pause instructions from any of the one or more applications at the server, as well as a mirror driver rendering a corresponding pause image in a bitmap. As such, the pause image can be sent to the client computer system in place of any of the plurality of drawing commands.

DETAILED DESCRIPTION

Implementations of the present invention extend to systems, methods, and computer program products that enhance the efficiency of application display sharing between networked computer systems. In particular, at least one implementation of the present invention involves a server, or sending computer, identifying areas of a display screen that are meant to be shared, as well as those that are meant not to be shared. The server can then substitute the unshared content with one or more simple image files, and transmit the simple image files in place of the unshared content. As such, sensitive information can be obscured without necessarily requiring any significant processing action from the client computer system.

As will be understood more fully from the specification and claims, this filtering can be done using very simple remote viewing protocols, without necessarily involving extensive processing from either the server side or the client side. Generally, the terms “remote view,” “remote viewing,” or “remote sharing” refer to the general concept of one computer system viewing at least some of the display screen data of another remotely located computer system. The remote viewing functionally described herein can also allow fairly granular display and obscured-display information to be transmitted to a receiving computer system without necessarily requiring any significant network bandwidth. Furthermore, this communication of shared and obscured application areas through remote viewing procedures can be done using existing protocols, such as remote desktop protocol (i.e., “RDP”), without necessarily requiring significant changes to—and related management of—the protocol at either the server or client side.

As shown inFIG. 1A, for example, a sending computer100communicates application data to receiving computer150over a network140. For example, the receiving computer150(e.g., a client computer system) connects with sending computer100(e.g., a server computer system) to share various resources, such as application110, which receiving computer150will display locally at display155. Alternatively, an end user at sending computer100shares a display screen with an end user at receiving computer150, where the sending computer100display screen105includes a view of application110data as well as application115data. In any event, at least a portion of what is viewable at sending computer100display screen105is intended to be viewable at receiving computer150display screen155. In addition,FIG. 1Ashows that application110is shared while application115is obscured. For example, application115might be accessed by another receiving computer system (not shown), or, alternatively, represents data that an end-user at sending computer100does not want to show to receiving computer150, such as a browser opened to a bank account, etc.

When application110,115performs a drawing operation that results in a visible change at display105, driver interface125will pass this data to display driver130. In one implementation, driver interface125is a device interface component, such as a Graphics Device Interface (“GDI”), which takes graphical function calls from an application program, and provides corresponding graphical instructions to the appropriate device driver. Generically, driver interface125can be any computer program code capable of intermediating between an application program and a display driver, and further configured in some cases to provide a filtering function as described more fully below.

For example,FIG. 1Ashows that driver interface125receives shared display data120from shared application110, and receives obscured display data121from obscured application115. Generally, data120and121comprise at least geometric drawing data, which can be used to render an image generated by the corresponding application. Driver interface125then identifies the correct display driver for local display, in this case driver130, and passes data120and121to the identified display driver. Display driver130can then render the data and displays it locally at server display105, as is typically the case. In particular, driver interface125does not need to provide any local filtering information or functionality since the applications are hosted and viewed locally at server100.

FIG. 1Aalso shows that driver interface125communicates with mirror driver135, which is a “remote view” driver, such as an RDP driver, to provide it with drawing commands received from applications110,115. In general, a mirror driver135can be any type of component or module capable of receiving duplicate drawing commands, the original of which were intended for one display device driver, and rendering those drawing commands in a duplicate (or “mirrored”) fashion. As shown, for example, mirror driver135receives drawing commands from driver interface125, just as those commands are also sent to display driver130.

FIG. 1Aalso shows, however, that driver interface125sends shared data120to mirror driver135, just as provided to display driver130, but also sends filtering instructions123to mirror driver135along with obscured data121. In one implementation, driver interface125may also be configured only to send such drawing instructions to mirror driver135if the instructions pertain to a certain region of, for example, a shared display screen. In any event, filtering instructions123can comprise geometric instructions for the unshared application115, along with a notation that data121are not shared, or can include any image file chosen by application115(or by application110, or a different component—not shown). Upon receipt, mirror driver135processes data120and123, and creates corresponding drawings in sender-side bitmap137a.

In this case, however, mirror driver135draws an image in sender-side bitmap137acorresponding to data120(i.e., “Application110”); and, based on the new instructions in message123, prepares an alternative replacement image131in sender-side bitmap137ain place of obscured data121. Sending computer100then prepares these images as graphics commands in accordance with an appropriate protocol (e.g., RDP), and prepares corresponding network packets of these graphics commands. Sending computer100then sends these network packets over network140, ultimately to corresponding receiver-side bitmap137bat receiving computer150. In addition to the raw data corresponding to drawing commands, these network packets can also include instructions accompanying replacement image131, such as whether to tile, stretch, or otherwise display replacement image131in some specific way.

At receiving computer150, a corresponding viewer application (not shown) reads, from receiver-side bitmap137b, the data stream of network packets as data120and replacement image131(and corresponding display instructions), and forwards this data to driver interface127. Depending on the protocol used (e.g., as with RDP), receiving computer150can also cache replacement image131for later use, or replace replacement image131with some other image it would prefer to use instead. Driver interface127, in turn, passes the data to display driver145, which then renders the data for display at client display screen155. As such, client display screen155shows an identical copy of application110, much like at sending computer display screen105. Nevertheless, in contrast with display screen105, display screen155shows the filtering instructions123(or some other chosen replacement image) in place of application115, since application115is obscured.

Thus, sending computer100performs substantially all of the relevant processing for determining obscured and or shared application data areas, and leaves receiving computer system150to simply read and render what it has received. This allows receiving computer system150to simply perform the functions it is already well-equipped to do, such that the receiving computer system does not need to receive additional updates or extensions to its installed resources. Furthermore, this allows implementations of the present invention to be readily compatible with older versions of remote viewing client/server models and/or related protocols.

An additional aspect of the principles described above is that data filtration can be tracked in kernel-mode by the driver interface and/or corresponding drivers, rather than in user-mode via the application(s). As such, the various drivers can act essentially as synchronizing and funneling agents that render and send images based on an immediate understanding of what is or is not to be shared. For example, when an application receives an identification that it is supposed to be shared or unshared, that designation can be immediately associated with the application's outgoing data (e.g.,120,121), and further directed to mirror driver135as well as driver interface125. At the moment mirror driver135receives an indication that application data is not to be shared, mirror driver135can immediately draw or replace content to be obscured with some default image in sender-side bitmap137a. Thus, since synchronization of shared and unshared concepts with received data can be done at mirror driver135, mirror driver135can avoid conditions where it is sending unshared data due to some delay in receiving synchronization information from the application.

FIG. 1Billustrates an alternative way in which the aforementioned technology can be implemented in a remote viewing or sharing environment. In particular,FIG. 1Bshows how a broadcasting computer system, such as an instructor using a sending computer100in a learning environment, can pause a presentation using the simple remote sharing protocol, such as RDP. The pause command can be implemented in such a way that all data is obscured by a replacement image, prior to being sent to any one or more corresponding receiving computer systems, such as receiving computer150.

For example,FIG. 1Bshows that driver interface125receives any shared and obscured data120,121, as inFIG. 1A. In the illustrated case, however, driver interface125also receives pause instructions160, such as from shared application110using a collaboration API. Driver interface125then passes data120and121to display driver130, as before, and also passes data120and121, as well as pause instructions160to mirror driver135via driver interface125. Pause instructions160might be used in cases, for example, where a sending user, such as an instructor, might be changing an application used in a presentation, which is at the foreground of an application he intends to hide. Rather than momentarily expose the hidden application when closing and opening new windows or applications, the instructor would like to cover the receiving computer system's entire display screen with a momentary pause image. Accordingly, the instructor requests the relevant application to send pause instructions160.

Upon receipt, driver interface125passes along any application data being sent from applications110,115, and also passes along pause instructions160to mirror driver135. Instructions160can be based on what the relevant application (e.g.,110and/or115) identifies regarding the borders of what is presently being displayed, and subsequent preparation of instructions to replace at least a portion between those borders with an appropriately chosen pause image. Upon receipt, mirror driver135can replace data120and121with pause image161.

In any event, mirror driver draws an appropriate pause image in sender-side bitmap137a, where pause image161replaces some or all of data120and121. As discussed inFIG. 1A, sending computer100then sends this data in network packet form over network140to receiver-side bitmap137b. Receiving computer150then reads, relays, and renders this data as appropriate, and ultimately displays pause image161at display155. Furthermore, receiving computer150can cache pause image161for later use, and can also substitute pause image161with another appropriately chosen pause image, as desired. However rendered and displayed, receiving computer150is nevertheless unable to display content that sending computer100specifically intended to be hidden or obscured, at least in part since it has not been received from sending computer100in the first instance.

For example, the schematic diagrams ofFIGS. 1A-1Band corresponding text provide a number of components and means by which the concepts of shared and obscured (or unshared or hidden) data can be implemented in a desktop sharing environment. Implementations of the present invention can also be described in terms of a sequence of acts for accomplishing a method. For example,FIGS. 2 and 3illustrates flowcharts of a series of acts in methods of filtering shared and obscured application drawing commands, and for implementing a pause feature in a remote viewing environment, respectively.FIGS. 2 and 3are described below with further reference to the components and modules ofFIGS. 1A-1B.

In particular,FIG. 2shows that a method at a sending computer of filtering shared and obscured application drawing commands comprises an act200of identifying shared drawing commands. Act200includes identifying one or more shared drawing commands intended to be displayed at a remote client. For example, driver interface125receives drawing commands120from shared application110. The drawing commands120are associated with an instruction indicating that the image defined by the drawing commands is shared, or viewable by receiving computer150. Furthermore,FIG. 2also shows that the method comprises an act210of identifying obscured drawing commands. Act210includes identifying one or more obscured drawing commands intended to be hidden from the remote client. For example, mirror driver135(whether via interface125, or directly from application115) receives data121, which represents drawing commands that are intended to not be displayed at receiving computer150.

In addition,FIG. 2shows that the method comprises an act220of rendering the shared drawing commands. Act220includes rendering the one or more shared drawing commands in a bitmap. For example, mirror driver135receives data120directly from application110, or via driver interface125. Since mirror driver135understands that this is shared content, mirror driver135draws this data into sender-side bitmap137a(i.e., the image of application110), and prepares it to be sent out over network140in packet form.

Furthermore, the method comprises an act230of rendering a replacement image. Act230includes rendering a replacement image of the one or more obscured drawings commands in the bitmap, such that the replacement image can be sent in place of content that is intended to be obscured. For example, mirror driver135receives data messages120and121, as well as filtering instructions123, which indicate geometric data regarding obscured boundaries in a display, and that the data contained in messages120and121are not meant for display. Alternatively, filtering instructions123contain an actual replacement image131to be used in place of data120and121. Accordingly, mirror driver135draws replacement image131in sender-side bitmap137a.

FIG. 3illustrates a similar but alternative method in accordance with an implementation of the present invention, which includes filtering shared and/or obscured application drawing information when implementing a pause feature. For example,FIG. 3shows that the method comprises an act300of identifying a plurality of drawing commands from one or more server applications. Act300includes identifying a plurality of drawing commands from one or more applications at a server, at least one of the plurality of drawing commands being intended to be displayed at a client computer system. For example, as shown inFIG. 1B, mirror driver135, such as via driver interface125, receives any of data120and121messages from applications110,115, etc. This information may be received, for example, during a presentation sent from sending computer100to one or more other receiving computer systems over network140.

In addition,FIG. 3shows that the method comprises an act310of receiving pause instructions. Act310includes receiving pause instructions from any of the one or more applications at the server. For example, driver interface125of server100receives pause instructions160, and passes them directly onward to mirror driver135. Alternatively, mirror driver135receives pause instructions160directly from application110or115. Furthermore,FIG. 3shows that the method comprises an act320of rendering a pause image in place of any drawing commands. Act320includes rendering a pause image in a bitmap, such that the pause image is sent to the client computer system in place of any of the plurality of drawing commands. For example,FIG. 1Bshows that driver interface125sends pause image161(or corresponding instructions of the same) to mirror driver135. Mirror driver135then draws the data160in sender-side bitmap137a, before data160are packetized and sent to receiving computer150over network140.

Accordingly, implementations of the present invention provide a number of components and methods for efficiently rendering shared and unshared (or obscured) data corresponding to data sent from shared or unshared applications. In particular, implementations of the present invention provide a number of advantages in the art since much of the processing that needs to occur is generally more simple than otherwise available. Furthermore, the more complicated aspects of the processing only need to occur at the sending computer. As such, implementations of the present invention can be readily adapted to systems that already incorporate remote sharing protocols, such as RDP, and can provide much richer share/unshared display information without necessarily requiring updates or extensions to existing resources and protocols.

When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of computer-readable media.