Abstract:
A computer is operated by displaying a window on a display device, where the window is an opaque window overlaid on the entirety of a system desktop that includes graphical representations of system controls and application controls. The window displays user interface elements of an application program executing on the computer, and also includes a portal region displaying an image of the system desktop including counterparts of the system controls and application controls. The computer provides user-controlled operation of the system controls and application controls based on simulated user interaction with the counterparts of the system controls and application controls in the portal region. The application program can effectively coordinate the use of the display by itself and by the system and other applications using the system desktop, relieving the user of this task.

Description:
BACKGROUND 
       [0001]    The invention is related to the field of graphical user interfaces in computers. In one particular application, the invention is related to the field of graphical user interfaces in applications such as online conferencing in which a presenter shares an application window or local desktop with other conference participants. 
         [0002]    Today, in online meetings, a user interface for a presenter in an online conference can be confusing and awkward. The awkwardness and confusion is particularly noticeable in a common use case—when the presenter is sharing their full desktop. The fundamental problem is that the presenter user interface for the conferencing application resides on the same desktop that is being shared with the other participants. Consider the example where the presenter is showing a full-screen spreadsheet presentation. No matter where the presenter positions windows for incoming webcam views, for example, important parts of the presentation become visually blocked on the presenter&#39;s display. Options for the presenter include hiding the webcam windows behind the shared window, or perhaps moving the webcam windows to a second monitor if one is in use. The rest of the conference user interface, such as a control window, is also positioned over the spreadsheet window, so it is awkward for the presenter to interact with the user interface while presenting. As a result it becomes a stressful experience when interaction with the user interface is needed for conference control purposes, such as muting attendees, monitoring chat messages, etc. 
         [0003]    Another significant problem found in current online meeting interface models is that there is strong division between the presenter who is sharing his or her screen and the other participants who are receiving that screen-sharing broadcast. The interface for presenters needs to balance showing incoming webcam views, controls, and other interface elements against the desktop that is being shared. The non-presenter interface needs to show incoming webcam views, controls, and the incoming desktop view shared by the presenter. Currently, switching roles from non-presenter to presenter may create user confusion, because the interface may automatically change to maximize visual space so that the local desktop can be seen. 
       SUMMARY 
       [0004]    The above-discussed issues of awkwardness in a user interface, in connection with a conferencing application or other application, arise in part because of demands for use of display area by relatively independent functions. In the above example, the demands are from the conferencing application displaying webcams and control windows, on the one hand, and the spreadsheet application displaying a spreadsheet window. It is up to the user to coordinate the use of the single display by the applications. 
         [0005]    The present disclosure is directed to improving a user&#39;s experience by rendering the user&#39;s desktop within a region of a window of another application, allowing controls and other visual content of the application to easily be located outside of the rendered desktop. From a user&#39;s point of view, the disclosed technique solves the problem of having controls and incoming content block part of the desktop. It can also solve the problem of having significantly different interface modes for different users, such as those for a presenter and non-presenter as discussed above. In the disclosed technique, the user&#39;s desktop is treated as content being displayed within the application, and a transition into a presenter mode to share the desktop for example would not require changing the interface&#39;s layout. 
         [0006]    Generally, a disclosed method of operating a computer includes displaying a special window on a display device of the computer, the special window being opaque and being overlaid on a system desktop that includes desktop objects (icons, windows etc.) for user interaction with the computer. The special window displays user interface elements of an application program executing on the computer. In one embodiment the application program is a video conferencing program supporting desktop sharing, i.e., the ability for conference participants to see a presenter&#39;s desktop and other application windows on it. In a portal region of the special window, an image of the system desktop is displayed that includes counterparts of the desktop objects. 
         [0007]    The disclosed method may be used in a display-only manner in which the portal region is used only to display the image of the system desktop and its desktop object. In a more general use, the method also enables a user to interact with the portal region in a simulated manner that provides for user activation/control of the desktop objects behind the special window as well as visual feedback of that activation and control. By this technique, the application program can effectively coordinate its own use of the display with the use of the display by the system and other applications using the system desktop, aiding the user with this task. 
         [0008]    In a more particular aspect, the disclosed technique places the opaque special window over the entire desktop, with this window serving as a background for the application (such as a video conferencing application). The contents of the desktop behind this background window are captured and displayed as a reduced-size image of the desktop within the portal region of the window. A presenter can interact with the reduced-size desktop view with keyboard and mouse as usual—keyboard and mouse events are passed through the special window to the desktop and/or other application windows behind it In one disclosed technique, a simulated pointer is used in the portal region and the system pointer is manipulated in a certain way to effect desired operation. In other technique, the system pointer alone is used, and input events such as mouse movements and button presses in the portal region are captured and used to synthesize corresponding events that are provided to the desktop and desktop objects. In either cases, distinct pointer appearances may be used inside and outside the portal to provide a visual indication when the user is interacting with the desktop rather than the application program. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]    The foregoing and other objects, features and advantages will be apparent from the following description of particular embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. 
           [0010]      FIG. 1  is a block diagram of a distributed computer system described in connection with a video conferencing application; 
           [0011]      FIG. 2  is a block diagram of a computer from a hardware perspective; 
           [0012]      FIGS. 3-9  are schematic diagrams of user interface display elements. 
       
    
    
     DETAILED DESCRIPTION 
       [0013]      FIG. 1  shows an example computer system in the form of a video conferencing system including a conference server  10  and a plurality of conference clients  12  interconnected by a network  14 . The conference server  10  and conference clients  12  are typically computerized devices executing specialized conferencing application software and in some cases including specialized hardware as well. The conference server  10  may be a server computer such as a “blade” computer in a data center, while the client computers  12  are more typically personal devices such as personal computers, laptop computers, tablet computers or smartphones for example. The network  14  may be generally any type of data communications network that provides for transmission of conference data among the participants, including for example the Internet. 
         [0014]    The system of  FIG. 1  may be referred to as a “client-server” arrangement. Those skilled in the art will appreciate that the disclosed techniques may be applied to other arrangements, including so-called “peer-to-peer” arrangements and even to single-computer applications. Also, the presently disclosed techniques are more generally applicable beyond video conferencing. 
         [0015]    In operation, the conference clients  12  establish connections and conference sessions with the conference server  10 . Each conference client  12  executes a client conference application that provides a graphical user interface to a local conference participant or “attendee”, as well as providing for transmission of local audio and video input to the conference server  10  and receiving conference audio and video streams or feeds from the conference server  10  for rendering to the local attendee. The conference server  10  performs merging or “mixing” of the audio and video streams from the conference clients  12  to create the conference feeds provided back to the conference clients  12 . Audio is typically mixed into a single output channel distributed to all conference clients  12 , enabling all participants to hear any participant who is speaking. Video streams such as from local cameras (web cams) are individually copied to all participants, enabling each participant to see all the other participants. The system also enables documents or other application data to be shared among the conference clients, where the source of a shared item is referred to as a “presenter”  16 . For such sharing, the contents of a desktop, window or similar user-interface element are sent from the presenter  16  to the conference server  10 , where they are replicated and provided to the other conference clients  12  for local display. 
         [0016]      FIG. 2  shows an example configuration of a physical computer such as a conference server  10  or conference client  12  from a computer hardware perspective. The hardware includes one or more processors  20 , memory  22 , and interface circuitry  24  interconnected by data interconnections  26  such as one or more high-speed data buses. The interface circuitry  24  provides a hardware connection to the network  14  ( FIG. 1 ) and other external devices/connections (EXT DEVs) such as a video display, audio circuitry, and camera. The processor(s)  20  with connected memory  22  may also be referred to as “processing circuitry” herein. There may also be local storage  28  such as a local-attached disk drive or Flash drive. In operation, the memory  22  stores data and instructions of system software (e.g., operating system) and one or more application programs which are executed by the processor(s)  20  to cause the hardware to function in a software-defined manner. Software may be provided as computer program instructions stored on a non-transitory computer-readable medium such as a magnetic or optical disk, Flash memory, etc. Thus the computer hardware executing instructions of a conference application, such as described herein, can be referred to as a conference circuit or conference component, and it will be understood that a collection of such circuits or components can all be realized and interact with each other as one or more sets of computer processing hardware executing different computer programs as generally known in the art. 
         [0017]      FIG. 3  depicts a graphical display  30  of a conference client as generally known in the art. A desktop  32  (also referred to as a “system desktop” herein) is a special display object displayed on an otherwise blank display screen  34  by a desktop operating system, such as the Windows® operating system for example. As generally known, the desktop  32  serves as a launching point for applications and provides certain kinds of system interactions as well (e.g., the “system tray” in Windows). In operation the desktop  32  can be partially or even completely covered by other windows that are interaction points for applications and system-provided functions. In the illustrated example, windows of a conferencing application including a conference control (CNF CTL) window  36  and a camera viewing window (CONF CAM)  38  are displayed. Also displayed is an “other” application window  40  for a separate application executing on the conference client, such as a word processor or spreadsheet application. 
         [0018]    In  FIG. 3  the conference windows  36 ,  38  cover parts of the desktop  32  as well as parts of the other application window  40 . This is illustrated in  FIG. 4 , where an covered area  42  of window  40  is shown by shading. Any covered parts are not immediately viewable or selectable by the user. This may present difficulties especially for the presenter when sharing his/her desktop with the other participants in a conference. The presenter needs to simultaneously view the conference windows  36 ,  38  and interact with the desktop  32  and other application window  40 , and it may become necessary to move items around to reveal any covered items so that they can be viewed and/or selected. For example, imagine that the presenter needs to select an icon on the desktop  32  to open a document, and that the icon is located in an area covered by the camera viewing window  38 . The presenter must first move or minimize the window  38  in order to access the icon. This need for juggling items on the screen can interfere with smooth and efficient operation of a conference. It will be appreciated that this problem can either be avoided or reduced when the presenter has two display screens and can choose to share the contents of one of them while placing the conference windows  36 ,  38  on the other, non-shared screen. However, the problem occurs especially when a presenter has a single display screen entirely covered by the desktop  32 , because the conferencing application places its windows  36 ,  38  on the same desktop  32  that is being shared by the presenter  16  with other participants  12 . 
         [0019]    In the remaining description the term “desktop objects” is used to refer to displayed GUI objects residing on the desktop  32  that a user may interact with. These can include icons, windows, tool bars, ribbons, and other objects as generally known in the art. With respect to  FIGS. 3 and 4 , the other application window  40  is an example of a desktop object. 
         [0020]      FIG. 5  shows a graphical display  50  of a conference client  16  according to an embodiment of the present invention. In this case, the conferencing application is itself involved in displaying desktop objects and other items to the user, and it uses this capability to better arrange the user display to avoid the problem discussed above. In particular, the conference application displays a special window referred to in one embodiment as a conference background window (CONF BKGND)  52  that completely and opaquely covers the desktop  32 . The conference application uses the background window  52  as a background for displaying the conference windows  36 ,  38 . The background window  52  preferably has a “background” nature from a visual perspective. For example, it may have a plain and uniform color and texture across the entire window  52 . Generally its design is such that it is not misinterpreted by the user as the desktop  32  or as another conference window having important content. Its main job is to hide the desktop  32  and the desktop objects. 
         [0021]    The conference background window  52  includes a portal region  54  that is used to display representations of desktop items that are hidden by the conference background window  52 . In one embodiment, the portal region  54  contains an image that is a scaled-down version of the entire desktop  32  and all items on it except for the conference windows  36 ,  38 ,  52 . This arrangement is useful when a presenter “shares” his/her desktop  32  (i.e., makes it visible to other participants) as part of the conference. In the illustrated example, the image in the portal region  54  includes an image of an application window  40 ′ that is a scaled version of the other application window  40  of  FIG. 3 . Essentially, the desktop  32  and desktop objects (such as other application window  40 ) are represented in the portal region  54  as separate content of the conferencing application, while the real desktop  32  and desktop objects are hidden behind the conference background  52 . A presenter  16  can interact with an image of the desktop  32  that is free of any covering by the separate conference windows  36 ,  38 , improving usability and smooth operation. In one embodiment, the portal region  54  may be a separate window with normal window functions including the ability to be moved, resized, minimized etc. In other embodiments it may be desirable to use some other construct, and the portal region may be relatively fixed in size and location. 
         [0022]      FIG. 6  illustrates one aspect of operation of the arrangement of  FIG. 5 . In this simplified example, a user is moving a system pointer  60  (under control of a mouse, for example) from one location to another, specifically from one icon  62  to another icon  64 . However, the user doesn&#39;t see the desktop  32 , but rather sees a scaled version of it in the portal region  54 . The scaled desktop displays corresponding scaled icons  62 ′ and  64 ′, as well as a simulated pointer  60 ′. As input events such as mouse movements, mouse button presses, and key presses occur, they are applied at the point of the system pointer  60 , and the portal region  54  and simulated pointer  60 ′ are used to mimic the activity on the (hidden) desktop  32  to provide visual feedback to the user. In the example of  FIG. 6 , movement of the (invisible) system pointer  60  causes corresponding scaled movement of the simulated pointer  60 ′ in the portal region  54 , providing visual feedback of the mouse movement.  FIG. 7  is another depiction of the situation of  FIG. 6 . The conference background window  52  is omitted for clarity, and the portal region  54  is shown directly on top of the desktop  32  as it would be in operation (as opposed to the schematic offset depiction of  FIG. 6 ). The system pointer  60  is shown in phantom, representing that it is not actually visible to the user, and its position on the desktop  32  corresponds to the position of the simulated pointer  60 ′ in the portal region  54  (which is user-visible). 
         [0023]      FIGS. 8 and 9  illustrate two distinct modes of operation and the transitions between these operating modes. 
         [0024]      FIG. 8  shows a first operating mode. The system pointer  60  is located outside the portal region  54 . The system pointer  60  is visible to the user and its movements are determined directly by input from the system pointing device (e.g., mouse) in the usual manner. The conference background  52  and its windows  36 ,  38  (omitted from  FIG. 8  for clarity) are treated as normal application windows with respect to their interaction with the system pointer  60 , and they receive input events such as mouse button presses, keyboard input, etc. in the normal manner. The simulated pointer  60 ′ is not active or visible. It may or may not actually be present on the display, but if present it is rendered as completely transparent and it has no effect on operation. 
         [0025]      FIG. 9  shows a second operating mode. The simulated pointer  60 ′ is both visible and active within the portal region  54 . The system pointer  60  is present and active but not visible. The conference background  52  and its windows  36 ,  38  do not interact with the system pointer  60  in the usual manner—this is explained more below. Input events are interpreted based on the location of the system pointer  60  with respect to the desktop  52  and any desktop objects thereon, as though the conference background  52  and its windows  36 ,  38  were not present. In a sense, the system pointer  60  is operated “behind” the conference background  52  and its windows  36 ,  38 .  FIG. 9  shows the system pointer  60  at the far right edge of the desktop  32 , but in this operating mode it may be located anywhere (including directly behind the portal region  54 ). 
         [0026]    The system transitions between the two operating modes based on user movement of the mouse or other pointing device/mechanism. When the system is in the first operating mode ( FIG. 8 ), it transitions into the second operating mode ( FIG. 9 ) as the mouse movement causes the system pointer  60  to encounter a boundary of the portal region  54 . The location of this encounter is referred to as the “point of entry”. When the boundary is encountered, the simulated pointer  60 ′ is visibly rendered at the point of entry, and the system pointer  60  is at the same time rendered invisible and moved to the corresponding scaled location on the desktop  32 . That location will generally be adjacent to an edge of the desktop  32  corresponding to the edge of the portal region  54  that was encountered, such as shown in  FIG. 9 . Once this transition has been made, operation in the second operating mode commences, with the system pointer  60  being free to move anywhere on the desktop  32  and the simulated pointer  60 ′ tracking its movements as described above. 
         [0027]    The reverse transition occurs when the system pointer  60  is moved off the desktop  32 , i.e., there is a mouse movement causing the system pointer  60  to encounter a boundary of the desktop  32 . The simulated pointer  60 ′ is then removed or rendered invisible, and the system pointer  60  is at the same time rendered visible and moved to a corresponding location on the conference background window  42 . That location will generally be near an edge of the portal region  54 , such as shown in  FIG. 8 . Once this transition has been made, operation in the first operating mode commences, with the system pointer  60  now being constrained to movement outside the portal region  54 . 
         [0028]    In the second operating mode, the system pointer  60  is hidden from the user by replacing the shape with a completely transparent one. Also, the conference background  52  and its windows  36 ,  38  are configured to operate in a pass-through manner such that input events (keyboard and mouse) go through them and are passed to the desktop  32  and desktop objects behind them. In a Windows® system, such operation can be obtained by use of an extended window style setting WS_EX_TRANSPARENT, as is known in the art. This window style setting for the conference background  52  and its windows  36 ,  38  is toggled as operation transitions between the first and second modes. In other types of systems, other system-specific mechanisms may be used. 
         [0029]    In the above description it is assumed that the boundary of the portal region  54  corresponds exactly to the boundary of the desktop  32 , this being a practical configuration that may be useful in many embodiments. In alternative embodiments, the portal region  54  may show only a portion of the desktop  32 , and in such configurations the transitions between operating modes occur with respect to the boundary of that portion of the desktop  32  rather than with respect to the boundary of the whole desktop  32  as described above. In one particular use, the portion of the desktop  32  may be that portion displayed on one display monitor of a two-monitor display arrangement in which the entirety of the desktop  32  actually extends across both monitors. 
         [0030]    In alternative embodiments another type of mechanism may be used to handle input events and provide visual feedback to a user when interacting with the portal region  54 . For example, rather than using a simulated pointer  60 ′ and relocating the system pointer  60  as described above, in one alternative no simulated pointer is used—the system pointer  60  is always visible and freely locatable anywhere across the display. Operation depends on its location. When the system pointer  60  is outside the portal region  54 , operation is as described above for the first operating mode, i.e., input events are passed to the conferencing windows  36 ,  38  (or other foreground windows) in the usual manner. When the system pointer  60  is inside the portal region  54 , then a mechanism is employed for translating actual input events to corresponding synthesized events that are provided to the desktop  32  and its desktop objects. For example with reference to  FIG. 6 , a button press at the location of icon  62 ′ in the portal region  54  is translated to a synthesized button press at the location of icon  62  on the desktop  32 . The appearance of the system pointer  60  might also be changed when inside the portal region  54  to provide a visual indication of the different operating mode. 
         [0031]    While various embodiments of the invention have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.