Abstract:
A system is provided to automatically control input permission when  multi users and/or software agents interact with a single, shared application in a window-based computing environment, and to indicate to all users which user or software agent currently is providing input or seeking input permission. The system operates by providing each user and agent with a specialized cursor icon that is used for displaying input-permission status and for requesting and relinquishing input-permission. Cursors have three modes with distinct appearances indicating whether the corresponding user or agent has, does not have, or desires input permission. Cursors are replicated on all displays, so that all users can at all times easily see who has or desires input permission. Users interact with the input-permission system by moving and clicking their mouse. Software agents achieve the same effects by sending control requests directly to a software interface provided as part of the invention. In one embodiment, each user and agent also has an assigned Home Window, which is used for identification and to compose and display text messages.

Description:
This is a continuation of application Ser. No. 08/414,282 filed on Mar. 31, 1995 now abandoned. 
    
    
     FIELD OF THE INVENTION 
     This invention relates to the graphical user interface in window-based computing environments and more particularly to controlling input to a window that is being shared between two or more remote users or programs. 
     BACKGROUND OF THE INVENTION 
     It is often convenient for computer users at different locations to be able to view and interact simultaneously with a single window-based application program. For example, when a user calls a software help line, rather than just describing the problem in words, the user can actually demonstrate the problem to a remote helper using a shared window. During the process, the remote helper can also provide input to the same window to help in the diagnosis. 
     The current state of the art in this kind of remote window sharing is typified by Hewlett packard&#39;s SharedX, Hp product Number B2305 L, or the Xy system described in &#34;Xmc and Xy--Scalable Window Sharing and Mobility,&#34; C. Bormann, and G. Hoffman, 8th Annual X Technical Conference, Boston, Mass., January, 1994. 
     These systems are extensions to the X Window System that allow real-time sharing of X-protocol based applications between two or more remote users or displays. In such systems, the shared application window is automatically replicated on all users&#39;displays, so that all users can simultaneously see changes in the state of the application. However, only users who have input permission are allowed to provide input to the application, e.g., by clicking with a mouse or using a keyboard. Any number of users can have input permission at the same time, with input permissions displayed and changed via toggle buttons next to users&#39;names on a separate control window. 
     Two major areas of deficiency in SharedX and similar systems are described below. First, the graphical user interface in current window sharing systems is confusing and awkward. The main problem, acknowledged in Chapter 4 of the SharedX 2.0 User&#39;s Guide, is that when sharing a window with several remote users who have input permission, it is confusing to tell who is inputting at any given time. 
     For example, suppose three users are sharing the same application window, such as a spreadsheet or drawing program. If two of the users both have input permission and one of them pushes a button on the shared window, the third user will see the button move, but will not be able to tell from his display who pressed it. Furthermore, if two users with input permission both start pressing buttons at the same time, they are likely to interfere with each other&#39;s work. 
     One solution to this problem would be to only let one user have input permission at a time. However, that is not the way that current systems are usually used, because changing input permissions requires going to a separate control window, which is inconvenient and interrupts the flow of the collaboration between users. 
     Moreover, when two people are viewing a display in the traditional way, i.e., standing or sitting side by side, they often point to items on the display with their fingers, or use the mouse to point out items. Unfortunately, window sharing systems like SharedX do not replicate mouse pointers on all users&#39;displays. Instead SharedX provides separate telepointer icons which users have to explicitly click and drag around, which is much more awkward than just pointing with the mouse. 
     The second major area of deficiency of current window sharing systems is that, because they were only designed for sharing between human users, they do not provide a well-defined software interface to allow software agents to play the role of remote users. For the present purposes, a software agent is any program that interacts both with human users and some shared application program. In the future, software help lines, for example, may provide an intelligent software agent instead of a human helper to assist users with simple problems. 
     SUMMARY OF INVENTION 
     The solution to the problems with the prior art described above is, first, to provide a software interface whereby software agents can participate in window sharing on the same basis as human users, second, to provide an automatic input-permission system that allows at most one user or software agent to have input permission at a time, and third, to provide clear indication on all displays of who is currently providing input. 
     The automatic input-permission system allows at most one user or software agent to have input permission at a time, but makes it easy for users and agents to obtain and relinquish input permission without interrupting the flow of collaboration. This is achieved by providing each user or software agent with a single cursor icon on the shared window that is used for pointing, for inputting, for displaying input-permission status, and for requesting and relinquishing inputpermission. Each cursor is replicated on all displays. 
     In order to distinguish identity, each cursor has a different basic appearance, such as a different color or shape. For example, in one embodiment of the invention, each cursor is in a different color pointing hand icon. 
     In order to support software agents as well as human users, in one embodiment the automatic input-permission system is divided into two modules: a controller and a translator. The controller is a program that receives control requests, e.g., requests to grant or relinquish input permission, from other programs. Software agents send control requests directly to the controller&#39;s software interface. The job of the translator component is to translate a user&#39;s cursor actions into control requests according to the rules described below. 
     In one embodiment, the automatic input-permission system operates as follows. Each cursor has three modes, also distinguished by appearance. When a cursor is in the Active mode, it means that the corresponding user or software agent has input permission. In the passive mode, the corresponding user or agent does not have input permission. In the attention-getting Signalling mode, the corresponding user or agent desires input permission, i.e., would like whoever currently has input permission to relinquish it. 
     For example, in one embodiment of the invention in which each cursor is a different color pointing hand icon, when the cursor is Active, the hand is a solid color; when it is passive, the hand is only an outline; and when it is Signalling, the outline hand blinks or oscillates in order to get the other users&#39;attention. 
     Thus at all times it is easy for all users to see who has or desires input permission simply by looking at the appearance of the cursors on the shared window. Software agents can find out who has or desires input permission by sending the appropriate control requests to the controller module. 
     Human users move their cursors and provide input simply by moving and clicking their mouse, i.e., the cursor replaces the usual mouse pointer on the shared window. Because cursors are replicated, they can also be used for pointing out items to remote users, making separate telepointers unnecessary. 
     A software agent does not have a display or mouse. As a result, software agents move their cursors and provide input by sending the appropriate control requests directly to the controller module. 
     All cursors start out in passive mode located off the shared window. When a user or software agent moves his cursor onto the shared window, it remains in the passive mode until the user or agent attempts to provide input, e.g., until the user presses a mouse button or key. At this point, the system automatically grants input permission to that user or software agent, i.e., makes the cursor Active, if and only if no one else is Active. If the user or agent&#39;s cursor is not made Active, then it is put into Signalling mode. 
     In one embodiment, a user or software agent relinquishes input permission simply by moving his cursor off the shared window. Moreover, a user can force the Active user or software agent to relinquish input permission by clicking on that user or agent&#39;s cursor. A software agent can force the Active user or agent to relinquish input permission by sending the appropriate control request to the controller module. 
     In one embodiment, each user and software agent is provided with a special window, called a Home Window, which is a place for the user or agent to put his cursor when he does not want to point or provide input. Like cursors, Home Windows are replicated on all displays. Home Windows assist in identification because each Home Window is labelled with the name and possibly the face of the corresponding user or software agent, and is coded with, for example, the same color as the corresponding cursor. 
     Another function of Home Windows is to support communication between users and software agents. Typically, in addition to sharing windows, human users are also in voice communication, e.g., by telephone. Software agents, however, typically cannot speak or hear, and therefore need some other mechanism for communicating with users. Human users and software agents can therefore display text messages in their Home Windows, which since they are replicated on all displays, will be seen by all users. Users can compose messages either by typing the text or by selecting from a menu of predefined messages. Agents can read users&#39;displayed messages by sending the appropriate control requests to the controller module. As an additional convenience, users can force another user or agent to relinquish input permission by clicking on that user or agent&#39;s Home Window. 
     In summary, a system is provided to automatically control input permission when multiple users and/or software agents interact with a single, shared application in a window-based computing environment, and to indicate to all users which user or software agent currently is providing input or seeking input permission. The system operates by providing each user and agent with a specialized cursor icon that is used for displaying input-permission status and for requesting and relinquishing input-permission. Cursors have three modes with distinct appearances indicating whether the corresponding user or agent has, does not have, or desires input permission. Cursors are replicated on all displays, so that all users can at all times easily see who has or desires input permission. Users interact with the input-permission system by moving and clicking their mouse. Software agents achieve the same effects by sending control requests directly to a software interface provided as part of the invention. In one embodiment, each user and agent also has an assigned Home Window, which is used for identification and to compose and display text messages. 
    
    
     BRIEF DESCRIPTION OF DRAWINGS 
     These and other features of the Subject Invention will be better understood in conjunction with the Detailed Description taken in conjunction with the Drawings of which: 
     FIG. 1 is a diagrammatic illustration of a typical prior art system, showing a shared, replicated application window and an input-permission control window, where one user cannot tell from his display which of the other remote users is currently providing input to the shared window; 
     FIG. 2 is a diagrammatic illustration showing the use of the Subject Invention in the same situation as FIG. 1, with the addition of an automatic input-permission system and replicated cursors, so that a user can easily see which remote user has pressed a particular button, and the addition of a software agent participating in the collaboration on the same basis as the human users; 
     FIGS. 3A and 3B are diagrammatic illustrations showing how a user is automatically granted input permission simply by clicking his mouse, given that no other user or agent has input permission; 
     FIG. 4 is a diagrammatic illustration of the windows seen on all users&#39;displays when Home Windows are used; 
     FIG. 5 is a block diagram of the automatic input-permission system of FIG. 2, showing its decomposition in one embodiment into a translator module and a controller module with associated controller state; 
     FIG. 6 is a flowchart showing the top-level logic of the controller module, which handles ten types of control requests; 
     FIG. 7 is the detailed flowchart for the processing of an Activate request by the controller module; 
     FIG. 8A is the detailed flowchart for the processing of an Deactivate request by the controller module; 
     FIG. 8B is the detailed flowchart for the processing of a Signal request by the controller module; 
     FIG. 9 is the detailed flowchart for processing of an Interrupt request by the controller module; 
     FIG. 10 is the detailed flowchart for processing of a Move request by the controller module; 
     FIG. 11 is the detailed flowchart for processing of an Execute request by the controller module; 
     FIG. 12A is the detailed flowchart for processing of a Status request by the controller module; 
     FIG. 12B is the detailed flowchart for processing of a Show request by the controller module; 
     FIG. 13A is the detailed flowchart for processing of a Clear request by the controller module; 
     FIG. 13B is the detailed flowchart for processing of a Read request by the controller module; 
     FIG. 14 is a flowchart showing the top-level logic of the translator module, which handles four types of window events resulting from a user&#39;s cursor actions; 
     FIG. 15A is the detailed flowchart for processing of an EnterWindow event by the translator module; 
     FIG. 15B is the detailed flowchart for processing of a LeaveWindow event by the translator module; 
     FIG. 16 is the detailed flowchart for processing of an Input event by the translator module; and 
     FIG. 17 is the detailed flowchart for processing of a pointerMotion event by the translator module. 
    
    
     DETAILED DESCRIPTION 
     Referring now to FIG. 1, in the prior art a network 10 connects computer nodes 12 such that windows 14, 14&#39;, and 14&#34; are duplicate window interfaces to a single shared application program. As is common, for example in the SharedX product, an additional replicated control window, 40, 40&#39;, and 40&#34;, is provided to indicate and modify which users have input permission. However, when more than one user has input permission, there is no indication of which user is actually providing input at any given moment. This makes it is more difficult to collaborate, since users cannot ascertain who did what. 
     For example, a user 16 looking at window 14 at node 18 is in a quandry as to which of the other users on a network 10 has caused the depression of a button 20 as indicated. As illustrated, a second user 22 has in fact caused the depression of the button 20 by clicking a button 24 of his mouse 26. This user 22 can see which button of the shared window he is pressing by virtue of the location of a pointer 28 on window 14&#39;, whose position is controlled by mouse 26. However, this pointer 28 does not show up on window 14 which user 16 is looking at. 
     At nearly the same time, another user 30 is attempting through his associated mouse 32 and button 34 to actuate a different button 36 as indicated by pointer 38 on window 14&#34;. In such a situation, it quickly becomes hard for all users to figure out what is going on. 
     In an effort to make it easier for users to share windows without confusion, referring now to FIG. 2, network 10 is provided with an automatic input-permission system 50 which both selects which user has input permission and displays that information to all users through icons on the shared window. For instance, assuming that a user 22 has been granted input permission by automatic input-permission system 50, a unique cursor icon, in this case a solid upward and leftward pointing hand 52, is displayed on all windows at the location at which input will occur. Thus at any given time, it is clear to all users which user or software agent has input permission. 
     In the depicted embodiment, the correspondence between a cursor icon and a user or software agent is indicated by the orientation of the hand. In another embodiment, the color of the cursor icon is used to uniquely identify the corresponding user or software agent. 
     In the depicted embodiment, cursors 52, 54, and 56 belong to the human users shown in the lower right, upper right, and lower left of the figure, respectively. Cursor 58 belongs to a software agent 60 depicted in the upper left of the figure. Using a network 10 and the automatic input-permission system 50, a software agent 60 competes for the opportunity to provide input to the shared application in much the same way as the human users. 
     In the depicted embodiment, the cursor of the user or agent, if any, which currently has input permission is rendered as a solid hand, while all the other cursors are rendered in outline only. Notice that cursor 58 is oscillating, which is an attention-getting signal to the other users that the corresponding user or software agent desires input permission. 
     Referring now to FIG. 3A, when no user or software agent has input permission, all of the cursor icons on window 14 appear in outline. Referring now to FIG. 3B, a user 22 has clicked on a button 24 of a mouse 26, the result being that the corresponding cursor icon 52 has changed to being filled in, indicating that the user 22 has received input permission. 
     Referring now to FIG. 4, in one embodiment each user and software agent is assigned a Home Window 64, 66, 68, and 70, where the corresponding cursor icon appears when it is not on the shared window. Home Windows are replicated on all displays. Moreover, a title bar 65, 67, 69, and 71, on each Home Window can be used to display the name of the corresponding user or software agent. 
     Here it can be seen that a user 22 has moved his cursor icon from his Home Window 66 onto the shared window 14 and has been granted input permission. Also, the user with Home Window 64 is indicating his desire to obtain input permission by Signalling with an oscillating cursor. Home Windows also support the display of text messages to support collaboration between users and software agents. One such text message is illustrated at 72. 
     Referring now to FIG. 5, in one embodiment the automatic input-permission system 50 includes a translator module 80 connected to a controller module 82, which in turn reads and updates a controller state data structure 84. As can be seen, a network 10 transmits window events from all users&#39;input devices to the translator module 80, which generates control requests over line 86 for execution by the controller module 82. 
     The controller module 82 receives control requests from either the translator module 80 or directly from software agents 60. The controller module 82 grants and retracts input permission, as well as producing window events transmitted through a network 10 to update the windows on all the users&#39;displays. 
     By way of illustration, suppose a user 22 clicks a button 24 on his mouse 26 at a moment when his cursor is located on top of another user or agent&#39;s cursor. The resulting window event is transmitted through the network 10 to the translator module 80, where it is translated into an Interrupt request, i.e., a request to force the other user to relinquish input permission. The Interrupt request is then sent to the controller module 82, which checks whether the other user or agent in fact does currently have input control, and if it does, appropriately updates the controller state 84 and transmits through the network 10 the appropriate window events to change the other user&#39;s cursor appearance on all displays. 
     The rules by which the controller module grants and retracts input permission are carried by the data structure 84 interacting with the program stored in the controller module 82. As illustrated, for each user or software agent, there are two pieces of information stored within the data structure 84, namely location and mode. Location refers to the location of that user or software agent&#39;s cursor icon, which is either Empty, meaning that the cursor is not on the shared window, a particular x-y location on the shared window or, in an embodiment with Home Windows, on that user or software agent&#39;s Home Window. Mode refers to whether the user or software agent has input permission. In the Active mode, the user or software agent has input permission. In the passive mode, the user or software agent does not have input permission. In the Passive mode, the user or software agent does not have input permission. In the attention-getting Signalling mode, the user or agent does not have input permission and desires input permission. 
     It will be appreciated that a software agent 60 is connected through the network 10 to the automatic input-permission system 50 via line 88 on which control requests are transmitted directly to the controller module 82, in contradistinction to the human users who send window events to the translator module 90. Software agents can determine the result of their requests to the controller module 82 by examining the return values that are transmitted via line 90 from the controller module to the software agent, 
     FIG. 6 is a high-level flowchart for the processing in one embodiment of the controller module 82. Upon receipt of a request, the controller determines which of the ten valid requests types has been received and dispatches to the appropriate processing routine described further in the indicated figures. The ten decision blocks in this figure check in turn for the receipt of an Activate request, a Deactivate request, a Signal request, an Interrupt request for a given user or agent, a Move request to given location, an Execute request of a given input, a Status request, a Show request for given message, a Clear request, or a Read request for given user or agent. 
     FIG. 7 is a detailed flowchart for the processing of an Activate request. A software agent would send this type of request when it wants to provide input to the shared application window. This request type is automatically generated for users by the translator module whenever a user tries to provide input. If the user or software agent that issued the Activate request is located off the shared window or another user or agent is currently Active, then the requestor&#39;s cursor mode is set to Signalling and the request fails; otherwise, the requestor&#39;s cursor mode is set to Active and the request succeeds. 
     FIGS. 8A is a detailed flowchart for the processing of Deactivate requests. A software agent would send this type of request when it was done providing input. This type of request is automatically generated for users by the translator module whenever a user leaves the shared window. The effect of a Deactivate request is simply to set the cursor mode of the requesting user or software agent to passive. 
     FIGS. 8B is a detailed flowchart for the processing of Signal requests. A software agent would send this type of request when it wanted to get the attention of the other users and/or agents, for example, so it could begin providing input. In embodiments with Home Windows, this type of request is automatically generated for users by the translator module whenever a user clicks the mouse on his own Home Window. The effect of a Signal request is to set the cursor mode of the requesting user or software agent to Signalling, which causes the corresponding cursor icon to oscillate or blink on all displays. A user or software agent&#39;s cursor mode is also set to Signalling whenever an Activate request fails, as shown in FIG. 7. 
     FIG. 9 is a detailed flowchart for the processing of an Interrupt request. This type of request would most typically be used by human users to take input permission away from software agents. In fact, in some embodiments, the automatic input-permission system may restrict who can interrupt whom. The translator module generates an Interrupt request, as shown in 16, whenever a human user clicks on top of another user or software agent&#39;s cursor. If that user or software agent&#39;s cursor is Active, the cursor&#39;s mode is set to passive and the request succeeds; otherwise the request fails. In embodiments with Home Windows, the translator module also generates an Interrupt request whenever a human user clicks on a Home Window other than his own. 
     FIG. 10 is a detailed flowchart for the processing of a Move request. Move requests are automatically generated for human users by the translator module in response to their mouse motions, so that their cursor icons appear to behave the same as their usual mouse pointers. Since software agents don&#39;t have mice, they move their cursors by sending Move requests directly to the controller module. If the requesting user or software agent&#39;s cursor is currently on the shared window and the new location is also on the shared window, then the requestor&#39;s cursor location is simply updated to the new location. If the requestor&#39;s cursor is moving from off the shared window to on or on the shared window to off, then the requestor&#39;s cursor mode is first set to passive. In either case, the Move request succeeds. 
     FIG. 11 is a detailed flowchart for the processing of an Execute request. This request type is used by software agents to provide input, such as mouse and keyboard press and and release events, to the shared application window. From the standpoint of the shared application, input from a software agent looks just like input from any of the users. Execute requests are automatically generated for users by the translator module, as shown in FIG. 16. If the requesting user or software agent is Active, then the given input event is sent to the shared window and the request succeeds; otherwise the processing for an Activate request, shown in FIG. 7, is performed first. If the Activate processing fails, then the Execute request fails. 
     FIG. 12A is a detailed flowchart for the processing of a Status request, which is used by software agents to find out about the location and mode of other users&#39;and software agents&#39;cursors because, unlike human users, software agents cannot simply look at their displays. A Status request returns the complete state of the controller in an appropriate data structure that can be interpreted by the requesting software agent. 
     FIG. 12B is a detailed flowchart for the processing of a Show request, which is used in embodiments with Home Windows only. This request is most likely to be used by software agents to communicate with human users. The Show request generates appropriate window events to display the given text message in the requesting user or software agent&#39;s Home Window. Since Home Windows are replicated, this information is displayed to all users. In one embodiment, human users can generate a Show request by interacting with a menu or text editor in their Home Window. 
     FIG. 13A is a detailed flowchart for the processing of a Clear request, which is used in embodiments with Home Windows only. The Clear request generates appropriate window events to erase any messages currently appearing in the requesting user or software agent&#39;s Home Window. 
     FIG. 13A is a detailed flowchart for the processing of a Read request, which is used by software agents in embodiments with Home Windows to read messages from other users or software agents. If the given user or software agent currently has a text message displayed, then that message is returned to the requesting user or software agent; otherwise the request fails. 
     FIG. 14 is a high-level flowchart for the processing in one embodiment of the translator module 80. Upon receipt of a window event from a user&#39;s display, the translator determines which of four relevant window event types has been received and dispatches to the appropriate processing routine described further in the indicated figures. These basic purpose of these processing routines is to send the appropriate requests to the controller module. The four decision blocks in this figure check in turn for the receipt of an EnterWindow event at a given x-y location, a LeaveWindow event, an Input event, or a pointerMotion event at a given x-y location. These are four generic event types in typical window systems. EnterWindow and LeaveWindow are the events generated when the pointer enters and leaves a window, respectively. pointerMotion is the event generated whenever the pointer moves. Input is the event generated when the user presses or releases a mouse button or key. 
     FIG. 15A is a detailed flowchart for the processing of an EnterWindow event. While a user is moving his pointer around in windows other than the shared application window, his cursor location is either Empty, or in embodiments with Home Windows, at Home. Whenever the user&#39;s pointer enters the shared window, however, a Move request is generated, which updates his cursor&#39;s location in the controller state to the given x-y location, and causes his corresponding cursor icon to appear on the shared window. Notice that simply entering the shared window does not translate into a request for input permission. Input permission is not requested until the user actually clicks or presses a key on the shared window. 
     FIG. 15B is a detailed flowchart for the processing of a LeaveWindow event, in which a Deactivate request is generated. In embodiments with Home Windows, a Move request is also generated to move the user&#39;s cursor to his Home Window. Thus regardless of where else on his display his pointer is located, once a user leaves the shared window, from the standpoint of other remote users, his cursor appears on his Home Window. 
     FIG. 16 is a detailed flowchart for the processing of an Input event. Some Input events, such as clicking on another user&#39;s cursor to generate an Interrupt request, are used to interact with the automatic input-permission module. Other Input events, such as keyboard events, are in effect passed through to the application program. In embodiments with Home Windows, if the user&#39;s cursor is on his own or another user or software agent&#39;s Home Window, then either a Signal or an Interrupt request is generated, respectively; otherwise, if the user&#39;s cursor is on top of another cursor, then an Interrupt request is generated with the other cursor&#39;s owner as argument. Finally, if none of the above special cases hold, then an Execute request is generated with the given input as argument. Assuming that that the requesting user or software agent has or can be granted input permission, the Execute request will then pass this input on to the application program. 
     FIG. 17 is a detailed flowchart for the processing of a pointerMotion event, in which a Move request is generated for the given x-y location. While a user&#39;s cursor is located on the shared window, his cursor icon tracks his mouse motions just like the usual mouse pointer. 
     Below is an embodiment of the Subject Invention written in Hp Common Lisp 4.0.3 from Harlequin, Inc. using the Garnet 2.1 graphics package from Carnegie Mellon University. ##SPC1## 
     In summary, the above describes a system in which multiple human users and/or software agents interact with a single, shared application window, such that each user or software agent can easily tell who is providing input at any given moment, and such that input permission is conveniently and automatically granted and relinquished. In the embodiments described above, the operation of the automatic input-permission system is divided between a controller module, which contains the essential rules for granting and relinquishing input permission, and a translator module, which maps users&#39;graphical motions into controller requests. This decomposition makes it possible for software agents to participate in window sharing on the same basis as human users. Furthermore, in embodiments with Home Windows, the correspondence between users or software agents and their cursor icons is made more evident, as well as facilities provided for text communication between users and/or software agents. 
     Having above indicated a preferred embodiment of the present invention, it will occur to those skilled in the art that modifications and alternatives can be practiced within the spirit of the invention. It is accordingly intended to define the scope of the invention only as indicated in the following claims.