Accessing window pixel data for application sharing

Screen sharing in which image pixel data operable to display one or more user interface windows associated with at least one shared application is retrieved from a memory. The retrieving excludes image pixel data operable to display at least one other display object currently contained in a user interface of said computer. The retrieved image pixel data is combined into a shared image.

BACKGROUND OF THE INVENTION

The present invention relates generally to collaboration software and systems, and more specifically to sharing screen data among multiple computers during a collaborative on-line session.

BRIEF SUMMARY

A screen sharing technique is disclosed in which image pixel data operable to completely display one or more user interface windows associated with at least one shared application is selectively retrieved by a collaboration application from a memory in a computer. The disclosed retrieving is selective, in that it only retrieves the image pixel data for windows associated with the shared application, and excludes from the retrieval image pixel data operable to display at least one other display object currently contained in a graphical user interface of the computer. The retrieved image pixel data is combined into a shared composite image for distribution from the computer to one or more other computers of other users participating in an on-line collaboration session.

DETAILED DESCRIPTION

FIG. 1is a first block diagram showing an example of the structure of components in an illustrative embodiment. As shown inFIG. 1, a Computer10includes at least one Processor12, a Memory14, a Graphics Controller16, a Display Device18, and other I/O Devices and/or Interfaces20. In the example ofFIG. 1, the Memory14stores a Collaboration Application22, one or more Shared Application(s)24, and one or more Non-Shared Application(s)26. The Memory14also may store an Operating System28, a Frame Buffer30and a Window Pixel Image Store32. The Collaboration Application includes and/or has access to a Shared Composite Image34, one or more Shared Process Identifiers (IDs)36, and one or more Shared Window Identifiers (IDs)38. The Operating System28may include a Window Image Pixel Data Application Programming Interface (API)40. A Current Composite Display Image42is shown within the Frame Buffer30.

The Window Pixel Image Store32stores a number of files or the like, each containing a complete set of pixels for displaying a corresponding user interface window currently present within the graphical user interface of Computer10, a current view of which is displayed on the Display Device18. Accordingly, each pixel file in the Window Pixel Image Store32contains a complete set of image pixel data operable to completely display a corresponding window that is present within the graphical user interface of Computer10regardless of whether the window is currently obscured, either partly or completely, by another window or other user interface component. Any specific type of graphical user interface may be provided by Computer10that includes window display objects, including but not limited to a virtual desktop or the like.

The pixel files stored within the Window Pixel Image Store32include Shared Window Pixel Files44and Non-Shared Window Files46. Each of the Shared Window Pixel Image Files44contains the pixel data for completely displaying a user interface window owned or otherwise associated with at least one of the Shared Applications24. Each of the Non-Shared Window Pixel Image Files46contains the pixel data for completely displaying a user interface window owned or otherwise associated only with one or more of the Non-Shared Applications26.

The Current Composite Display Image42is a composite image used to display the current view of the user interface. Specifically, the Current Composite Display Image42is made up of only the pixels operable to represent the currently visible portions of display objects in the user interface (e.g. windows) through the Display Device18. Accordingly, the Current Composite Display Image42is made up of the pixels used to generate a single screen currently displayed on the Display Device18. The Current Composite Display Image42does not include pixel data operable to display any currently obscured portions of windows that are only partially visible in the current view.

In one embodiment, the Graphics Controller16operates to generate the Current Composite Display Image42, responsive to pixel data including the window pixel data stored in the Window Pixel Image Store32, and further responsive to a current visual arrangement of display elements within the user interface.

The Collaboration Application22operates to generate the Shared Composite Image34such that the Shared Composite Image34contains only the image data for user interface windows associated with one or more applications that are currently being shared through a collaboration session between the user of Computer10and the users of one or more other computers. The Shared Composite Image34is shared with one or more other computer systems during the collaboration session. The Colloboration Application22may, for example, be embodied as any appropriate type of application, such as a native application executing on the Computer10, a browser-based Web application, or some other specific type of application.

Further with reference to the illustrative embodiment ofFIG. 1, each identifier in the Shared Process ID(s)36uniquely identifies an application on Computer10that is currently being shared, i.e. a corresponding one of the Shared Applications24. The Collaboration Application22may use the Shared Process ID(s)36to obtain the Shared Window ID(s)38. For example the Collaboration Application22may obtain the Shared Window ID(s)38by passing the Shared Process ID(s)36to the Operating System28, which then in turn passes back a list of window identifiers uniquely identifying a set of windows including only those windows associated with the application(s) indicated by the Shared Process ID(s)36, e.g. a list of windows owned by the Shared Application(s)24.

The Collaboration Application22uses the Shared Window ID(s)38to retrieve the Shared Window Pixel Images44from the Window Pixel Image Store32. For example, the Window Image Pixel Data API40provided by Operating System28may accept window identifiers and return the corresponding pixel files from the Window Pixel Image Store32. In such an embodiment, the Collaboration Application22operates to provide the Shared Window ID(s)38to the Window Image Pixel Data API40, and accepts the returned Shared Window Pixel Image Files44. The Collaboration Application22may then operate to combine the Shared Window Pixel Image Files44into a single composite image to be shared, shown inFIG. 1by the Shared Composite Image34.

While the Frame Buffer30, Window Pixel Image Store32, and Graphics Controller16are shown as separate blocks inFIG. 1, those skilled in the art will recognize that the disclosed system is not limited to such an arrangement. Accordingly, these logic blocks may be integrated into a single component or device, or otherwise combined or organized as appropriate for a given operational environment.

FIG. 2is a block diagram showing the structure of components in an illustrative operational environment including the illustrative embodiment of claim1. As shown inFIG. 2, the Shared Composite Image34may be transmitted from Computer10to a number of other computer systems of other users participating in a collaboration session with the user of Computer10, e.g. Other Collaborating Computer48, shown for purposes of illustration including Collaborating Computer47, Collaborating Computer49, etc. The Shared Composite Image34may, for example, be a first image or key frame within the collaboration session, transmitted from the Computer10in a compressed file format, such as JPEG (“Joint Photographic Experts Group”), PNG (“Portable Network Graphics”), or any other appropriate format for a given implementation.

While for purposes of concise illustrationFIG. 2shows the Computer10transmitting the Shared Composite Image34to the Other Collaborating Computers48, those skilled in the art will recognize that one or more server systems may be located between the Computer10and the Other Collaborating Computers48. The Computer10, Other Collaborating Computers48and any such servers may be communicably connected using any specific kind of data communication networks and/or protocols.

The Computer10ofFIG. 1and the Other Collaborating Computers48ofFIG. 2may each be embodied as or include any specific type of computer system and/or intelligent electronic device, such as a desktop, laptop, or palmtop computer system, or a personal digital assistant, cell phone, and/or other electronic device. The Display Device18may be any specific type of display device that is capable of displaying a graphical user interface including user interface window display objects, such as a liquid crystal display (LCD), cathode ray tube (CRT), interferometric modulator display (IMOD), light emitting diode (LED), or other specific type of display device.

Those skilled in the art will recognize that the Collaboration Application22may be embodied using software or firmware, such as computer application program code, operating system program code, middleware, and/or wholly or partly using digital hardware components, such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and the like, and/or combinations of hardware and/or software and/or firmware. Those skilled in the art will further recognize that the Processor(s)12in Computer10may include one or more processors of any specific type, and that the Memory14may be embodied as any specific type of program code or data storage, such as volatile or non-volatile memory, operable to store program code executable on such processors and associated data.

FIG. 3is a first flow chart showing steps performed during operation of an illustrative embodiment. The steps ofFIG. 3may, for example, be performed in whole or in part by the Collaboration Application22shown inFIG. 1. At step50, the disclosed system retrieves image pixel data for user interface windows associated with one or more shared applications, while excluding image pixel data associated with non-shared applications. At step52, the disclosed system combines all the image pixel data retrieved at step50into a shared image to be conveyed to the computer systems of other users participating in the collaboration session. With reference to the embodiment ofFIG. 1, the Collaboration Application22may operate to combine the Shared Windows Pixel Files44returned by the Operating System28into the Shared Composite Image34. For example, the Collaboration Application22may combine the Shared Windows Pixel Files44into the Shared Composite Image34such that the shared user interface windows corresponding to the Shared Windows Pixels Files44are represented in the same relative positions they currently have within the graphical user interface displayed to the user of Computer10through the Display Device18. In one embodiment, the steps ofFIG. 3are performed periodically to refresh the contents of Shared Composite Image34to reflect the most current appearance and contents of the shared windows represented by the Shared Windows Pixel Files44.

FIG. 4is a second flow chart showing steps performed during operation of an illustrative embodiment. The steps shown inFIG. 4are an example of steps that may be performed during step50ofFIG. 3. The steps ofFIG. 4may, for example, be performed by the Collaboration Application22ofFIG. 1. At step60, the disclosed system obtains a set of shared window identifiers based on the process identifiers of one or more applications currently being shared in a collaboration session. For example, the Collaboration Application22may include or have access to Shared Process ID(s)36, which includes a unique process identifier for each of the Shared Applications24. In an alternative embodiment, the Collaboration Application22may operate by collecting a complete set of window identifiers for all windows currently within the user interface, and then select out only the identifiers for windows that are owned by the Shared Applications24, responsive to the Shared Process ID(s)36.

At step62, the disclosed system passes the shared window identifiers obtained in step60as inputs to an application programming interface or the like that returns complete image pixel data for the windows corresponding to the window identifiers. For example, the Collaboration Application22may operate at step62to pass the Shared Window ID(s)38to the Operating System28through the Window Image Pixel Data API40. At step64, the disclosed system receives the complete image pixel data for only the windows indicated by the window identifiers passed to the application user interface at step62, without any image pixel image data for any other windows present within the user interface. For example, at step62the Operating System28may return only the Shared Windows Pixel Files44to the Collaboration Application22. By issuing a request at step62for only the pixel image data for windows that are owned by the applications being shared in the current collaboration session (i.e. Shared Applications24), the Collaboration Application22excludes from the retrieval step64any pixel image data for windows owned by any non-shared applications (i.e. Non-Shared Applications26).

FIG. 5is a screen shot showing at least a portion of a user interface as displayed by a first collaborating computer system consistent with an illustrative embodiment. The screen shot ofFIG. 5may, for example, be a portion of the current view of the graphical user interface displayed to the user of Computer10through the Display Device18. As shown inFIG. 5, the current view of the user interface includes Non-Shared Window170, Non-Shared Window271, Non-Shared Window373, Non-Shared Window475, Shared Window172, and Shared Window274. For example, the shared windows Shared Window172and Shared Window274may be two windows owned by an application being shared during a collaboration session, and for example shown in the Shared Applications24. The shared application may be any specific kind of application. The Non-Shared Windows70,71,73and75may be windows owned by a non-shared application contained in the Non-Shared Applications26, or by any other process not associated with the application being shared through the collaboration session. In the example ofFIG. 5, the user of Computer10is sharing the application that owns Shared Window172and Shared Window274with one or more other users through the Collaboration Application22. However, because the application that owns Non-Shared Window70is not shared, the user of Computer10does not intend to share the contents of Non-Shared Window70. For example, the Non-Shared Window70may be a pop-up window containing an instant messaging dialog between the user of Computer10and some other user.

FIG. 6is a screen shot showing at least a portion of a user interface as displayed by a second collaborating computer system consistent with an illustrative embodiment. The screen shot ofFIG. 6may, for example, be a portion of the current view of the graphical user interface displayed to a user of one of the Other Collaborating Computers48shown inFIG. 2. As shown inFIG. 6, a Collaboration Application Viewer80includes Shared Image82generated from the Shared Composite Image34received from Computer10. The Shared Image82includes the Shared Window172and Shared Window274in the same positions that they have within the user interface of Computer10, but without the Non-Shared Window70.

The disclosed system provides various advantages over other approaches to providing screen sharing. For example, in an approach to screen sharing that does not include the disclosed techniques, pixels for the shared image may be obtained from a frame buffer contained within or controlled by a video display device or controller. Such frame buffer pixels only reflect a current view of the user interface. The resulting shared image is exactly what is currently displayed on the sharing computer's screen, i.e. including all visible portions of application windows (for both shared and non-shared applications), desktop elements, backgrounds, etc., that have already been composited into the single composite image stored in the frame buffer and used by the video device to generate the current view. The disclosed system avoids problems arising from such an approach, allowing the sharing user to share only particular display elements, such as only a single window or only the windows owned by a single application.

For example, a user may wish to share only a particular word processing application. When the sharing user receives an incoming instant message, and the messaging window appears on top of the shared word processing window, the user does not wish to share the contents of the messaging window, since it may contain personal or sensitive information. To avoid sharing the contents of the message window, a screen sharing approach without the disclosed techniques may need to implement “occlusion detection”, in which the screen sharing technology is responsible for monitoring the positions of all open windows and applications, for example via a windowing layer in the operating system, and determining if any part of the shared area is occluded by another application or window. When occlusion is detected, the resulting image may need to be “grayed out” in the area where the occlusion has occurred, in order to avoid sharing personal or sensitive information. As a result of such graying out, areas of the shared image appear to the sharing users as a solid gray color where the occlusion has occurred, at least until the sharing user removes the occluding display objects from his local user interface. The disclosed system advantageously avoids such an aesthetically unpleasant and disruptive approach for viewers in the collaborative session.

The disclosed system also avoids potential security concerns associated with the timing of occlusion detection. Due to the decoupled, asynchronous nature of an operating system's windowing layer with regard to the video device, the timing of any necessary occlusion detection is critical, and certain timing conditions may result in a failure to detect the presence of an occluding display object. If occlusion is not detected properly, and graying out is not performed with regard to such non-shared display objects, the unintentional sharing of a window or application may result.

For example, in a system that does not include the disclosed techniques, at the time an instant message is received a window may be generated that occludes a currently shared application window. If at the same time a capture of image pixel data from the frame buffer to be used in the shared image has already begun, the occlusion may not be detected until the pixels captured from the frame buffer are shared, and the instant messaging window inadvertently revealed to others in the sharing session prior to the occlusion being detected.

The disclosed system also avoids problems that may result from the increasingly prevalent use of transparent windows. A system that does not include the disclosed techniques, and that extracts shared pixels from a composite image stored within a frame buffer used to provide the current view of the user interface on a display device, may inadvertently allow any non-shared display objects located below a shared transparent window to be shared.

Accordingly, by selectively retrieving complete pixel image data for shared windows prior to the pixel data being composited into a display image representing only the current view of the user interface, including only visible portions of shared and non-shared display objects, the disclosed system alleviates various problems and increased complexity associated with occlusion detection, as well as disadvantages associated with “graying out” occluded areas in the shared image.