Simultaneous gesture and touch control on a display

A method of controlling contents displayed on a display includes processing inputs from different users across multiple application windows of one or more applications displayed on a single display of a user interface, resulting in a multi-user graphical user interface (GUI), the computing environment running an operating system (OS), the computing environment having a multiple input detection system to respond to user inputs from different users simultaneously, the multiple input detection system including a touch screen on the single display that detects touch inputs and a contactless gesture sensor that detects contactless gestures. When a user input is detected by the touch detector, the GUI displays a first input interface to be used with touch. When a user input is detected by the contactless gesture sensor, the GUI displays a second input interface to be used with gestures, the second input interface being different from the first input interface.

DESCRIPTION OF RELATED ART

There are currently different solutions to using gestures to control a display remotely and different solutions to using touch control on a display when in contact with the display. Gesture solutions include sensing gestures using a wearable device and using complex trajectory information. However, using both types of inputs simultaneously has not been realized.

SUMMARY

In a computing environment, a method for processing inputs from different users across multiple application windows of one or more applications displayed on a single display of a user interface, resulting in a multi-user graphical user interface (GUI), the computing environment running an operating system (OS), the computing environment having a multiple input detection system to respond to user inputs from different users at a same time, the multiple input detection system including a touch screen on the single display that detects touch inputs and a contactless gesture sensor that detects contactless gestures, the method including receiving a first user input from the touch screen; simultaneous to receiving the first user input from the touch screen, receiving a different second user input from the contactless gesture sensor; determining, in a primary application, separate from the OS, that the first user input corresponds to a first window; the primary application sending the first user input to a first secondary application that runs in parallel with the primary application, the first secondary application being designed to run with the OS; determining a first user input event from the first user input; displaying a result of the determined first user input event in the first window associated with the first secondary application; determining, in the primary application, that a second user input corresponds to a second window; the primary application sending the second user input to a second secondary application that runs in parallel with the primary application, the second secondary application being designed to run with the OS; determining a second user input event from the second user input; and displaying a result of the determined second user input event on the second window associated with the second secondary application.

In a computing environment, a method of controlling contents displayed on a display, method for processing inputs from different users across multiple application windows of one or more applications displayed on a single display of a user interface, resulting in a multi-user graphical user interface (GUI), the computing environment running an operating system (OS), the computing environment having a multiple input detection system to respond to user inputs from different users at a same time, the multiple input detection system including a touch screen on the single display that detects touch inputs and a contactless gesture sensor that detects contactless gestures, the method including when a user input is detected by the touch screen, the GUI displays a first input interface to be used with touch; and when a user input is detected by the contactless gesture sensor, the GUI displays a second input interface to be used with gestures, the second input interface being different from the first input interface.

The first input interface may be in a periphery of a selected window and the second input interface overlaps a selected window.

The second input interface may include at least two buttons.

An area of the second input interface outside the at least two buttons may have a default operation.

The second input interface may include another input interface outside of the selected window.

The second input interface may include a mode selection.

The first input interface may include a first set of operations and the second input interface includes a second set of operations, different from the first set of operations.

The second set of operations may have fewer operations than the first set of operations.

The second set of operations may be a subset of the first set of operations.

The second set of operations may include an operation not in the first set of operations.

When an open hand gesture is sensed, an icon is displayed on the display, and moving an open hand over the icon may move the icon on the display in accordance with the movement of the open hand and a closed hand over the icon may move a virtual canvas on the display in accordance with the movement of the closed hand.

When a closed hand gesture is sensed, a window overlapped by the closed hand may be selected.

When an open hand gesture is sensed on a window, buttons corresponding to the second input interface may be displayed on the window overlapped by the open hand.

When a closed hand gesture is sensed on a button, the operations of the second input interface that button corresponds to may be executed.

The contactless gesture sensor may include a camera separate from a video-conferencing camera associated with the display.

The OS may be designed to run with a single user GUI.

DETAILED DESCRIPTION

U.S. Pat. No. 9,596,319, incorporated by reference herein in its entirety for all purposes, generally discloses how to handle multiple user inputs that occur simultaneously. As disclosed therein, all inputs are assumed to function in a same fashion. As disclosed therein, a virtual canvas (“Canvas”) is a virtual region that expands to greater than the physical area of a display region of a display, e.g., any number of times the physical area up to infinite. The use of the Canvas allows additional files to be accessible and can be saved, but off the display region. Gestures, such as pan, zoom and pinch gestures can be made to move and resize the scale of the Canvas, allowing the full canvas to be display at once or only a small section thereof. As disclosed therein, all inputs are assumed to function in a same fashion, i.e., allow all of the same operations to be performed, even when an operating system (OS) is designed to operate with a single graphical user interface (GUI). However, this may not be practical when using a remote, e.g., non-contact, contactless, or touchless, gesture as an input.

First,FIG.1will provide a brief overview of a system100including a display110having a touch sensor120, e.g., overlaying an entirety of the display110, that it is sensitive to touch inputs including taps and gestures, a camera130associated with the display110, e.g., for video conferencing, and a gesture sensor140to detect remote gestures, i.e., non contact or touchless inputs. The system also includes a display computer150connected to all components. The display computer may include an operating system (OS), a graphic processing unit (GPU), a primary application (PA), other applications, e.g., a web browser application (WBA), Power Point, designed to operate with a single or multiple user graphical user interface (GUI), and so forth, a touchscreen (TS) driver, and a gesture sensor (GS) driver.

One or more windows W may be open on the display. The window may be a file, e.g., a video file, a word document, and so forth, a web browser, a streaming video of another display, e.g., a mobile display or a remote display, and so forth.

To perform gesture sensing simultaneously while using direct touch inputs, when a user waves their hand or raises an open hand for a predetermined time, a hand cursor160may appear on the display110. The gesture sensor140may sense a limited set of hand gestures, e.g., those shown inFIGS.2A to2C, examples of the effects of which will be described in detail below. For example, the set of hand gestures may include an open hand, shown inFIG.2A, a first or closed hand, shown inFIG.2B, and a one-hand clap, shown inFIG.2C. Other users may still user other inputs to control windows on the display110screen, e.g., using a mouse and/or direct multi-touch.

When the hand cursor is not over a particular window, as shown inFIG.1, an open hand ofFIG.2Awill move the hand cursor around the display110, e.g., such that it may overlap a window as shown inFIG.3. When the hand cursor is not over a particular window, as shown inFIG.1, a first ofFIG.2Bwill move the entirety of the canvas on which the window is placed. For example, when the gesture sensor140senses a first moving to the left, the entirety of the canvas may be shifted to the left, as illustrated inFIG.4.

Once the hand cursor is moved over a window, as shown inFIG.3, a first may be used to select and move the window, as shown inFIG.5, the movement stopping when the first is opened. When an open hand ofFIG.2Aremains on that window, this may then result in a grid of really big buttons, on a window, e.g., still image, x-ray, and so forth, that can be selected pops up, e.g., as shown inFIG.6. Then, when the hand is closed on that menu item, e.g., expand to full screen as shown inFIG.7or take a snapshot of the window as shown inFIG.8, that menu item is performed. when the closed hand is opened and closed again on the expand screen button on the expanded screen inFIG.7, the window may return to its original size.

If a closed hand is on the window, but does not correspond to a button, the window may be moved as discussed above with respect toFIG.4. Additionally, if the closed hand outside a button, but on the window, moves towards or away from the screen, the window may be zoomed in and out, respectively, as shown inFIG.9.

Given the viewability and maneuverability constraints associated with remote gesture input, a number of actions that may be taken using remote gesture input may be a subset of those that can be performed with a mouse or touch inputs. For example, an open hand may move the window around the screen, while a closed hand, i.e., a fist, may select the window or button. A limited number of other gestures may be used to control the window.

Further, other tools may be available on the display outside of the window to interface contactlessly with the window. For example, a lasso, an arrow, and so forth may be selected by a closed first and dragged to the window to select a portion within a window and when let go, e.g., by opening the closed hand, may leave the selected portion within the window highlighted, as may be seen inFIGS.10to12. A one hand clap shown inFIG.2Cmay be used to any actions taken.

Additionally, as shown inFIG.13, one of the buttons may change a mode of the input interface, e.g., such that the hand cursor is changed to a pin cursor, to allow more precise selection within the window.

In contrast, as may be seen inFIG.14, when the window is activated by a touch input, many more options may be available to manipulate the window, e.g., including a pen to annotate, a keyboard pop up, rotate, and so forth. In other words the input interface may be different for touch inputs than contactless inputs.

The above may be of particular interest when used in an environment in which physical interaction between the display and a user may be impractical, e.g., an operating room or other circumstances requiring a degree of sterility or other types of controlled environments. A user in the controlled environment may use gestures remotely to control the display in general, while a user outside the controlled environment may more finely control the display, i.e., has more options to manipulate windows.

FIG.10illustrates a flowchart of actions taken in response to sensed gestures. In operation S10, the present of a hand is determined. When a hand is detected by the gesture sensor140, in operation S20, a hand cursor is displayed on the display110, e.g., near where the hand was sensed.

In operation S30, whether or not the cursor is on a window. If not, operation S40determines if the hand is open. If not, moving the hand will move the cursor (compareFIGS.1and3). If the hand is closed, moving the hand will move the entire canvas (compareFIGS.1and4).

If operation S30determines the cursor is on a window, operation S70shows buttons of for the contactless gesture input interface. Then, when operation S80determines the hand is closed, operation S90executes the selected operation, e.g., either corresponding to the button or the default operations where there is no button.

The methods and processes described herein may be performed by code or instructions to be executed by a computer, processor, manager, or controller. Because the algorithms that form the basis of the methods (or operations of the computer, processor, or controller) are described in detail, the code or instructions for implementing the operations of the method embodiments may transform the computer, processor, or controller into a special-purpose processor for performing the methods described herein.

Also, another embodiment may include a computer-readable medium, e.g., a non-transitory computer-readable medium, for storing the code or instructions described above. The computer-readable medium may be a volatile or non-volatile memory or other storage device, which may be removably or fixedly coupled to the computer, processor, or controller which is to execute the code or instructions for performing the method embodiments described herein.