Depth-based user interface gesture control

Technologies for depth-based gesture control include a computing device having a display and a depth sensor. The computing device is configured to recognize an input gesture performed by a user, determine a depth relative to the display of the input gesture based on data from the depth sensor, assign a depth plane to the input gesture as a function of the depth, and execute a user interface command based on the input gesture and the assigned depth plane. The user interface command may control a virtual object selected by depth plane, including a player character in a game. The computing device may recognize primary and secondary virtual touch planes and execute a secondary user interface command for input gestures on the secondary virtual touch plane, such as magnifying or selecting user interface elements or enabling additional functionality based on the input gesture. Other embodiments are described and claimed.

BACKGROUND

Modern computing devices include ever-increasing processing power combined with improved ability to sense and interact with the surrounding environment. As a consequence of such improved capabilities, many computing devices are adopting new input modalities such as touch computing and gesture-based computing.

User interface gestures may include touch-based input gestures such as tapping, swiping, and otherwise manipulating a touch surface of a computing device. User interface gestures may also include input gestures made without physically touching the computing device, including moving the user's body, limbs, hands, or fingers to command user interface actions. Such movement-based input gestures are sometimes called perceptual or air gestures.

DETAILED DESCRIPTION OF THE DRAWINGS

Referring now toFIG. 1, in the illustrative embodiment, a computing device100includes a display112, a touch screen114, a gesture sensor120, and a depth sensor122. A user of the computing device100may interact with the computing device100by performing various input gestures, which are detected by the gesture sensor120. As discussed in more detail below, the input gestures may be performed by the user at some distance in front of the display112. The computing device100is configured to determine the distance to the input gesture—and thereby to the user—using the depth sensor122and respond to the input gesture based on the determined depth. Depth-based gesture control allows for natural and fluid gesture control, particularly for user interface metaphors involving depth. Additionally, combining depth-based gesture control with other input modalities such as touch control allows rich interaction with traditional interfaces and may allow users to easily manage complicated user interfaces. Depth-based gesture control also may facilitate interactions with multiple users, particularly when combined with user identification by facial recognition, skeletal tracking, or the like.

The computing device100may be embodied as any type of device capable of performing the functions described herein. For example, the computing device100may be embodied as, without limitation, a computer, a smart phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, a workstation, a tabletop computer, a mobile computing device, a cellular telephone, a handset, a messaging device, a vehicle telematics device, a network appliance, a web appliance, a distributed computing system, a multiprocessor system, a processor-based system, a consumer electronic device, a digital television device, and/or any other computing device configured to respond to depth-based gesture commands. As shown inFIG. 1, the illustrative computing device100includes a processor102, an input/output subsystem104, a memory106, and a data storage device108. Of course, the computing device100may include other or additional components, such as those commonly found in a desktop computer (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise from a portion of, another component. For example, the memory106, or portions thereof, may be incorporated in the processor102in some embodiments.

The processor102may be embodied as any type of processor capable of performing the functions described herein. For example, the processor102may be embodied as a single or multi-core processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit. Similarly, the memory106may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory106may store various data and software used during operation of the computing device100such as operating systems, applications, programs, libraries, and drivers. The memory106is communicatively coupled to the processor102via the I/O subsystem104, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor102, the memory106, and other components of the computing device100. For example, the I/O subsystem104may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem104may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor102, the memory106, and other components of the computing device100, on a single integrated circuit chip.

The data storage device108may be embodied as any type of device or devices configured for short-term or long-term storage of data. For example, the data storage device108may be embodied as memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices.

The communication circuit110of the computing device100may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the computing device100and other remote devices. The communication circuit110may be configured to use any one or more communication technologies (e.g., wireless or wired communications) and associated protocols (e.g., Ethernet, Bluetooth®. Wi-Fi®, WiMAX, etc.) to effect such communication. In some embodiments, the communication circuit110may be embodied as a network adapter, including a wireless network adapter.

As discussed above, the computing device100also includes the display112. The display112may be embodied as any type of display capable of displaying digital information, such as a liquid crystal display (LCD), a light emitting diode (LED) display, a plasma display, a cathode ray tube (CRT), or other type of display device. The display112is coupled to a touch screen114. The touch screen114may be embodied as any type of touch screen capable of generating input data in response to being touched by the user of the computing device100. In some embodiments, the touch screen114may be attached to the display112or physically incorporated in the display112. The touch screen114may use any suitable touch screen input technology to detect the user's tactile selection of information displayed on the display112including, but not limited to, resistive touch screen sensors, capacitive touch screen sensors, camera-based touch screen sensors, surface acoustic wave (SAW) touch screen sensors, infrared touch screen sensors, optical imaging touch screen sensors, acoustic touch screen sensors, and/or other type of touch screen sensors. Additionally, the touch screen114may be responsive to multiple simultaneous touch points.

In some embodiments, the computing device100may include a camera116and/or an audio sensor118. The camera116may be embodied as a digital camera or other digital imaging device integrated with the computing device100. The camera116includes an electronic image sensor, such as an active-pixel sensor (APS), e.g., a complementary metal-oxide-semiconductor (CMOS) sensor, or a charge-coupled device (CCD). The audio sensor118may be embodied as any sensor capable of capturing audio signals such as a microphone, a line input jack, an analog-to-digital converter (ADC), or other type of audio sensor.

The computing device100further includes the gesture sensor120, which may be embodied as any type of electronic sensor capable of detecting and recognizing input gestures performed by the user. For example, the gesture sensor120may be embodied as a digital video camera capable of capturing moving images of the user. In the illustrative embodiment, no particular resolution is required; in some embodiments the gesture sensor120may be capable of resolving only gross motions of the user's body and limbs. In other embodiments, the gesture sensor120may be capable of resolving fine detail of the user's face and/or hands. Of course, in other embodiments, other components such as the camera116and/or the depth sensor122may embody, or otherwise be included in, the gesture sensor120.

The computing device100further includes the depth sensor122. The depth sensor122may be embodied as any type of electronic sensor capable of detecting the distance between the computing device100(and/or the display112) and an input gesture performed by the user. For example, the depth sensor122may be embodied as a stereo depth camera, a structured light camera, or a proximity sensor. In some embodiments, the depth sensor122may be embodied as, or otherwise include, other components of the computing device100. For example, the depth sensor122may be embodied as, or otherwise include a capacitive or resistive sensor in the touch screen114configured to measure distance to the user's fingers. Additionally or alternatively, the depth sensor122may be embodied as, or otherwise include, the communication circuit110. In such embodiments, the communication circuit110may be configured to determine the distance to a transmitter manipulated by the user relative to the computing device100. For example, the computing device100may measure signal strength for a short-range communication such as a Bluetooth® or near-field communication, which may be used to determine the distance to the transmitter. A corresponding transmitter may be embedded in a device manipulated by the user, such as a stylus or other pointing device. Additionally or alternatively, the depth sensor122may include the camera116of the computing device100, which may be configured to measure the location of a shadow cast by the user—for example from a projector-type display112—and determine physical distance to the user based on the apparent location of the shadow.

Referring now toFIG. 2, diagram200illustrates yet another embodiment of a depth sensor122. In the illustrative embodiment, the computing device100includes the display112with an integrated touch screen114. The display112defines a surface normal202projecting away from the display112. The depth sensor122includes two visible light cameras122a,122b. Each of the cameras122a,122bmay be aimed independently, as represented by the line-of-sight vectors204,206. To perform depth sensing, the cameras122a,122bare aimed such that the line-of-sight vectors204,206are parallel to the display112, perpendicular to each other, and perpendicular to the surface normal202. In such configuration, the cameras122a,122bare capable of determining the location of objects such as the user's fingers near the display112, including determining the distance from the display112to the object.

Referring now toFIG. 3, in an illustrative embodiment, the computing device100establishes an environment300during operation. The illustrative embodiment300includes an application302, a user interface command module306, a gesture recognition module308, and a depth recognition module310. Additionally, in some embodiments, the environment300may include a user identification module312. The various modules of the environment300may be embodied as hardware, firmware, software, or a combination thereof.

The application302is configured to present a user interface to the user and respond to user interface commands. Such interface may be embodied as a graphical user interface displayed on the display112or other user interface. In addition to gesture control, the application302may respond to other input modalities such as touch input on the touch screen114or input from other peripheral devices (e.g., a keyboard or mouse) attached to the computing device100. The application302may include an application programming interface304allowing the application302to be controlled by, or otherwise interfaced with, other modules of the computing device100.

The user interface command module306is configured to execute commands at the direction of the user. Such commands may control the operation of the application302executing on the computing device100. In some embodiments, the user interface command module306may control the application302through the application programming interface304. Additionally or alternatively, in some embodiments, the user interface command module306may also control an operating system of the computing device100(not shown).

The gesture recognition module308is configured to recognize input gestures performed by the user. Such input gestures may include perceptual or air gestures such as body and limb movement, hand and finger movement, and facial gestures. The input gestures are recognized based on data received from the gesture sensor120. The recognized input gesture is provided to the user interface command module306to control the application302.

The depth recognition module310is configured to receive data from the depth sensor122and determine the depth of the input gestures recognized by the gesture recognition module308. The depth recognition module310also assigns the input gesture to a particular depth plane, as described in detail below. As described above, the resulting input gesture and depth information is provided to the user interface command module306to control the application302.

In some embodiments, the environment300includes the user identification module312, which is configured to identify one or more users of the computing device100and assign input gestures to the appropriate user. By doing so, the computing device100may be used by multiple users simultaneously, or by a single user among other persons not using the computing device100. The user identification module312may identify and distinguish users based on any combination of facial recognition, skeletal tracking, eye tracking, voice recognition, and/or other user identification technology. In some embodiments, those functions may be performed by sub-modules, for example by a facial recognition module314, a skeletal tracking module316, an eye tracking module318, or a voice recognition module320.

Referring now toFIG. 4, in use, the computing device100may execute a method400for depth-based gesture control. The method400begins with block402, in which the computing device100receives input sensor data from the gesture sensor120. As described above, such sensor data may be embodied as a video feed of the user. In other embodiments, such sensor data may be embodied as input data from the touch screen114or from other peripheral input devices of the computing device100.

In block404, the computing device100analyzes the input sensor data to recognize an input gesture made by the user. The types of input gestures recognized may depend on the particular input sensor data being analyzed, a gesture vocabulary of the computing device100, or on particular input requirements of the application302. For example, in some embodiments, the recognized input gesture may be an air gesture, for example, the user moving about the room or waving a limb. In some embodiments the input gesture may be a hand movement similar to a touch gesture, such as tapping, swiping, pointing, pinching, spreading, or otherwise manipulating fingers, but performed above the touch screen114. Body or hand movements not included in the gesture vocabulary may not be recognized as input gestures.

In block406, the computing device100determines whether an input gesture has been recognized. If no input gesture was recognized, the method400loops back to block402to continue receiving input sensor data. If a gesture was recognized, the method400advances to block408.

In block408, in some embodiments the computing device100may assign the input gesture to an identified user of the computing device100. For example, in some embodiments, the computing device100may be used with multiple users, and each recognized gesture is assigned to a particular identified user. In other embodiments, gestures made by a single user of the computing device100may be distinguished from other persons detected by the computing device100but not interacting with the computing device100.

As discussed above, the computing device100may use any one or more identification techniques to identify the user and assign the recognized gesture to the identified user. For example, in block410, the computing device100may perform facial detection and analysis to identify the user and assign the input gesture to the identified user. The computing device100may perform such facial detection using input data received from the camera116. Additionally or alternatively, in block412, the computing device100may perform skeletal tracking to identify the user and assign the input gesture to the identified user. For example, the computing device100may track all hands, limbs, faces, and other features detectable by the computing device100, and assign such features to a corresponding skeletal model for a particular user. Constraints on the skeletal model (e.g., typical ranges of angles for joints in the human body, typical lengths and length ratios of bones in the human body, etc.) allow the computing device100to distinguish between the features of multiple users. Such skeletal tracking may be performed by the camera116, the gesture sensor120, the depth sensor122, or any combination thereof. Additionally or alternatively, in block414, the computing device100may perform an eye tracking technique to identify the user and assign the gesture to the identified user. For example, the computing device100may determine the angle or fixation point of the user's gaze using the camera116or using a specialized eye tracking sensor (not shown). Additionally or alternatively, in block416, the computing device100may perform voice recognition to identify the user and assign the gesture to the identified user. Such voice recognition may be performed, for example, using the audio sensor118.

In some embodiments, the computing device100may show a visual representation of the user's limb, fingers, or other body parts on the display112. Such visual representation provides feedback to the user on how his or her input is being interpreted, what objects on the display112may be impacted by the user's input, and depth may be assigned to the gesture. In some embodiments, such visual representation may take the form of an actual image of the user's limb, or may be a cursor or game element that corresponds to the action that the gesture will make.

In block418, the computing device100determines a depth of the input gesture based on data received from the depth sensor122. As described in detail above, various embodiments of the depth sensor122determine the depth of the input gesture in different ways. For example, a stereo camera allows depth to be determined by comparing image data from two camera sensors. A structured light camera allows depth to be determined by analyzing a reflected light pattern. An orthogonal pair of visible light cameras allows depth to be determined by comparing image data from each of the pair of cameras. Such analysis for depth may be performed by the computing device100or by hardware, firmware, or software included in the depth sensor122.

In block420, the computing device100assigns the input gesture to a depth plane based on the measured depth. For example, as shown inFIG. 5, a diagram500illustrates the computing device100and a plurality of depth planes502. The illustrative computing device100ofFIG. 5includes a display112with an integrated touch screen114. The display112defines a three-dimensional coordinate system, represented by the axes x, y, and z. The z-axis extends perpendicularly away from the display112and therefore is referred to as a surface normal of the display112. Depth planes504,506,508, and510are each parallel to the surface of the display112and located along the z-axis, thereby representing a series of depth planes502, each progressively further from the display112. Because the user generally interacts with the computing device100while viewing the display112straight-on or nearly straight-on, the depth planes502are also progressively closer to the user. The computing device100assigns recognized input gestures to a depth plane based on the measured depth of the gesture. For example, input gesture512may be assigned to depth plane506and input gesture514may be assigned to depth plane510. Input gestures may be assigned to a depth plane by quantizing their measured depth, that is, by rounding their depth to the nearest depth plane. In some embodiments, input gestures that are at a depth greater than a threshold distance from any depth plane may be dropped altogether.

The depth planes502are located closely enough together to allow for convenient gesture interaction; in many embodiments, the depth planes502may be separated by only a few centimeters. Although the depth planes502are illustrated as being equally separated, in some embodiments each of the depth planes502may be any distance from neighboring depth planes. Additionally, the location and arrangement of the depth planes502may be configurable by the user. Further, although illustrated inFIG. 5as having four depth planes502, a fewer or greater number of depth planes502may be used in other embodiments. For example, in some embodiments two depth planes502may be used. In such embodiments, the depth plane closest to the display112, for example depth plane504, may be designated a primary virtual touch plane. In such embodiments, the depth plane furthest from the display112and closest to the user, for example depth plane506, may be designated a secondary virtual touch plane. Further, although depth plane504is illustrated as coinciding with the surface of the display112, in some embodiments the depth plane504may be any distance away from the surface of the display112.

Referring back toFIG. 4, in block422, the computing device100executes a user interface command based on the recognized input gesture and the associated depth plane. For example, in block424, the computing device100may execute a user interface command additionally based on the user assigned to the gesture in block408. For example, the computing device100may allow each user to control a particular application302running on the computing device100, a particular region of the display112, or particular user interface elements. In some embodiments, each user may be designated a particular depth plane, and input gestures by that user outside of the designated depth plane may be rejected by the computing device100.

Additionally, in block426, in some embodiments the computing device100may control a particular user interface element based on the assigned depth plane. That is, in some embodiments, each depth plane may control a particular user interface element and/or a particular group or type of user interface elements. For example, referring again toFIG. 5, input gestures on depth plane504may control user interface elements of the application302, while input gestures on depth plane506may control “overlay” user interface elements such as transient dialogs, dashboard widgets, operating system controls, and the like. In some embodiments, the application302may render user interface elements in a “layered” presentation, such as in drawing applications. Given such an application302, the user may select a particular layer for manipulation based on the depth plane of the user's input gestures.

In other embodiments, such depth based control of user interface elements may be used with an application302that is embodied as a game. For example, in some embodiments the computing device100may control a virtual object selected by a depth plane. Such virtual object may be represented with a user interface element on the display112and may be modeled with physical characteristics. For example, in such a game application302, the user may select environmental objects of the game world for control based on the depth plane of the input gesture. In some embodiments, player characters may be controlled through depth-based input gestures. In a specific example, a submarine hunt game may use depth planes504,506, and508. Input gestures in the highest depth plane508may control depth charges, input gestures in the middle depth plane506may control submarines, and input gestures in the lowest depth plane504may control sea-floor vehicles. In another specific example, for a real-time-strategy game, input gestures at the lowest depth plane504may control ground units such as marines and medics, input gestures at the middle depth plane506may control ranged armor units such as tanks and walkers, and input gestures at the highest depth plane508may control air- and space-based units such as space cruisers. Referring back toFIG. 4, following execution of the user interface command, the method400loops back to block402to continue receiving sensor input data.

In some embodiments, the user interface command executed in response to a particular gesture may depend on the particular depth plane in which the input gesture was performed. That is, the same input gesture (e.g., a swipe or double click gesture) may generate different user interface commands depending on the particular depth plane at which the input gesture was performed. As such, in some embodiments, the computing device100may execute a method600to execute a primary or secondary user interface command based on the depth plane assigned to an input gesture as shown inFIG. 6. The method600begins in block602, in which the computing device100determines whether a user interaction has been received. Such determination involves receiving sensor input from the gesture sensor120, determining whether an input gesture has been recognized, and in some embodiments assigning an identified user to the recognized input gesture. Such functions are described above with respect to blocks402,404,406, and408ofFIG. 4. If a user interaction has not been received, the method600loops back to continue waiting for user interactions in block602. If a user interaction has been received, the method600advances to block604.

In block604, the computing device100determines the assigned gesture plane for the recognized input gesture, based on sensor data received from the depth sensor122. Such assignment is described above with respect to blocks418and420ofFIG. 4. However, in the illustrated method600, the input gesture may be assigned to one of two depth planes: the primary virtual touch plane or the secondary virtual touch plane. Of course, in other embodiments, additional depth planes may be implemented.

In block606, the computing device100determines whether the input gesture is assigned to the primary virtual touch plane. As described above, the primary virtual touch plane is the depth plane closest to the display112, for example the depth plane504ofFIG. 5. Thus, in some embodiments, if assigned to the primary virtual touch plane, the input gesture may have been performed on the touch screen114itself (e.g., a tactile selection by the user sensed by the touch screen114). If the input gesture was assigned to the primary virtual touch plane, the method600branches to block608in which the computing device100executes a primary user interface command. The primary user interface command may correspond to a primary input modality of the computing device100, for example, tapping the touch screen114or clicking a primary button of a pointing device of the computing device100. Such primary user interface command, when executed, may result in the computing device100executing an application302, opening a file, activating a user interface element, or any other function that may be activated using a typical input modality of a computing device. After executing the primary user interface command in block608, the method600loops back to block602to continue waiting for user interactions.

Referring back to block606, if the input gesture is not assigned to the primary virtual touch plane, the method600advances to block610. In block610, the computing device100determines whether the input gesture is assigned to the secondary virtual touch plane. As described above, the secondary virtual touch plane is a depth plane in front of the display112and closer to the user than the primary virtual touch plane, for example the depth plane506ofFIG. 5. If the input gesture is not assigned to the secondary virtual touch plane, the method600loops back to block602without executing any user interface command to continue waiting for user interactions. In such embodiments, the input gesture may have been performed too far away from the touch screen114, for example. If the gesture is assigned to the secondary virtual touch plane, the method600advances to block612.

In block612, the computing device100executes a secondary user interface command. The secondary user interface command may correspond to a secondary input modality of the computing device100, for example, long-pressing the touch screen114, pointing a cursor using a pointing device of the computing device100, or clicking a secondary button of a pointing device of the computing device100. The secondary user interface command is different from the primary user interface command, and in some embodiments may only be activated through interaction with the secondary virtual touch plane. Numerous such secondary user interface commands are possible. Additionally, in embodiments including additional depth planes, tertiary, quaternary, quinary, etc. interface commands may be assigned to the same input gesture performed on the corresponding depth planes.

The secondary user interface command may be embodied as any type of user interface command different from the primary interface command. For example, in block614, the computing device100may magnify one or more user interface elements on the display112. Such user interface elements include icons, buttons, images, labels, and similar components typical of a graphical user interface. For example, the computing device100may magnify a region of the display112underneath the user's hand and display the magnified region on a different part of the display112, to allow interaction with user interface elements that would otherwise be obscured by the user's hand.

In some embodiments, in block616, the computing device100may select one or more user interface elements. The selected user interface elements may be highlighted or otherwise visually distinguished on the display112. For example, the user may choose from or “scroll” through a group of menu icons (e.g., for moving, copying, or deleting) by gesturing at the icons on the secondary virtual touch plane and select the desired icon from the menu via an input gesture on the primary plane and/or the touch screen114. Additionally, in some embodiments, in block618, the computing device100may activate a secondary interaction mode for one or more user interface elements. The secondary interaction mode may provide commands and allow for manipulation different from the ordinary interaction mode of the user interface elements. Such commands may be accessible through further interactions on the secondary virtual touch plane or on the primary virtual touch plane. User interface elements with an activated secondary interaction mode may be visually distinguished on the display112, for example by rendering the user interface elements as “lifted” or otherwise located in front of other user interface elements. For example, the primary interaction mode, accessible via input gestures performed on the primary plane, may allow icons to be moved, rearranged, or reordered inside a current window on the display112. In such example, the secondary interaction mode, accessible via input gestures performed on the secondary plane, may allow the icons to be moved to a different window on the display112or to another device (not illustrated).

In some embodiments, in block620, the computing device100may present one or more contextual commands to the user based on the input gesture. For example, the computing device100may display contextual commands on the display112near the location of the input gesture. For example, in a drawing application302(seeFIG. 3), the computing device100may display a tool box near the location of the gesture, allowing the user to change the drawing mode or select options without moving the user's hand long distances. In other embodiments, the computing device100may present a group of commands based on a user interface element associated with the input gesture. For example, the computing device100may present a contextual menu based on an icon on the display112underneath the user's hand, similar to a secondary click operation of a conventional mouse.

In some embodiments, in block622, the computing device100may assign a value to a user interface command based on the measured depth of the input gesture. Such assigned value may alter the operation of the user interface command. Returning to the example of the drawing application302, the assigned value may correspond to “pressure,” allowing the user to control the weight of lines drawn by adjusting the distance between the display112and the input gesture.

Additionally, in some embodiments, in block624, the computing device100may execute a user interface command based on a representation of the input gesture in space. Such representation may include the three-dimensional position and velocity vectors of the input gesture. In such embodiments, the gesture is recognized in a touch volume, that is, a three-dimensional region of space that accepts user interactions. The representation of the input gesture in space may correspond to user interface commands to manipulate representations of virtual objects on the display112. For example, the user interface may model the reactions of physical controls such as levers, sliders, rotating dials, thumbwheels, and the like. The reaction to the user interface command depends on the input gesture and the modeled physical attributes of the user interface elements.

After execution of block612, the method600loops back to block602to continue waiting for additional user interactions. In this way, a user of the computing device100may interact with the computing device100to perform different user interface commands using the same input gesture performed at different depth planes.

EXAMPLES

Example 1 includes a computing device for depth-based gesture control, the computing device comprising a display to define a surface normal; a depth sensor to generate depth sensor data indicative of a depth relative to the display of an input gesture performed by a user of the computing device in front of the display; a gesture recognition module to recognize the input gesture; a depth recognition module to receive the depth sensor data from the depth sensor; determine the depth of the input gesture as a function of the depth sensor data; and assign a depth plane to the input gesture as a function of the depth of the input gesture, wherein each depth plane is positioned parallel to the display and intersects the surface normal; and a user command module to execute a user interface command based on the input gesture and the assigned depth plane.

Example 2 includes the subject matter of Example 1, and wherein to assign the depth plane further comprises to assign a depth plane of a plurality of depth planes as a function of the depth of the input gesture relative to the display.

Example 3 includes the subject matter of any of Examples 1 and 2, and wherein the depth sensor comprises a stereo depth camera.

Example 4 includes the subject matter of any of Examples 1-3, and wherein the depth sensor comprises a structured light camera.

Example 5 includes the subject matter of any of Examples 1-4, and wherein the depth sensor comprises a plurality of cameras, wherein each camera of the plurality of cameras is aimed perpendicular to the surface normal of the display and perpendicular to another camera of the plurality of cameras.

Example 6 includes the subject matter of any of Examples 1-5, and further wherein the depth sensor comprises a camera; to receive the depth sensor data comprises to receive image data from the camera; and to determine the depth of the input gesture comprises to determine the depth of the input gesture as a function of a position of a shadow cast by the user captured in the received image data.

Example 7 includes the subject matter of any of Examples 1-6, and wherein the depth sensor comprises a proximity sensor.

Example 8 includes the subject matter of any of Examples 1-7, and wherein the depth sensor comprises a radio receiver; the depth recognition module is further to receive, using the radio receiver, a signal transmitted by a transmitter manipulated by the user; and to receive the depth sensor data comprises to receive signal strength data associated with the received signal from the radio receiver.

Example 9 includes the subject matter of any of Examples 1-8, and wherein to execute the user interface command based on the assigned depth plane comprises to select a virtual object as a function of the assigned depth plane; and to control the virtual object based on the input gesture.

Example 10 includes the subject matter of any of Examples 1-9, and wherein to select the virtual object comprises to select a player character as a function of the assigned depth plane.

Example 11 includes the subject matter of any of Examples 1-10, and wherein the user command module is further to configure, prior to recognition of the input gesture, the user interface command to be executed based on the input gesture and the assigned depth plan.

Example 12 includes the subject matter of any of Examples 1-11, and wherein to execute the user interface command comprises to determine whether the assigned depth plane comprises a secondary virtual touch plane of the computing device; and execute a secondary user interface command in response to a determination that the assigned depth plane comprises the secondary virtual touch plane.

Example 13 includes the subject matter of any of Examples 1-12, and wherein the secondary user interface command is accessible only through the secondary virtual touch plane.

Example 14 includes the subject matter of any of Examples 1-13, and wherein to execute the secondary user interface command comprises to magnify a user interface element displayed on the display.

Example 15 includes the subject matter of any of Examples 1-14, and wherein to execute the secondary user interface command comprises to select a user interface element displayed on the display.

Example 16 includes the subject matter of any of Examples 1-15, and wherein to execute the secondary user interface command comprises to activate a secondary interaction mode for the user interface element, wherein a primary interaction mode for the user interface element is accessible via a touch screen of the display.

Example 17 includes the subject matter of any of Examples 1-16, and wherein to execute the secondary user interface command comprises to display a contextual command menu on the display.

Example 18 includes the subject matter of any of Examples 1-17, and wherein to execute the secondary user interface command comprises to assign a value to the secondary user interface command as a function of the depth of the input gesture.

Example 19 includes the subject matter of any of Examples 1-18, and wherein to execute the secondary user interface command comprises to assign a three-dimensional position and velocity to the secondary user interface command as a function of the input gesture.

Example 20 includes the subject matter of any of Examples 1-19, and further including a user recognition module to identify the user of the computing device; and assign the input gesture to the identified user; wherein to execute the user interface command further comprises to execute the user interface command as a function of the user assigned to the input gesture.

Example 21 includes the subject matter of any of Examples 1-20, and wherein the user recognition module further comprises a facial recognition module to identify the user by facial recognition.

Example 22 includes the subject matter of any of Examples 1-21, and wherein the user recognition module further comprises a skeletal tracking module to track a skeletal model of the identified user, the skeletal model having a limb; and to assign the input gesture to the identified user comprises to assign the input gesture to the limb of the skeletal model of the identified user.

Example 23 includes the subject matter of any of Examples 1-22, and wherein the user recognition module further comprises an eye tracking module to identify the user by eye tracking.

Example 24 includes the subject matter of any of Examples 1-23, and wherein the user recognition module further comprises a voice recognition module to identify the user by voice recognition.

Example 25 includes the subject matter of any of Examples 1-24, and wherein the user recognition module is further to configure the depth plane to be assigned to the input gesture assigned to the user.

Example 26 includes a method for depth-based gesture control, the method comprising recognizing, on a computing device, an input gesture performed by a user of the computing device in front of a display of the computing device; receiving, on the computing device, depth sensor data indicative of a depth relative to the display of the input gesture from a depth sensor of the computing device; determining, on the computing device, the depth of the input gesture as a function of the depth sensor data; assigning, on the computing device, a depth plane to the input gesture as a function of the depth of the input gesture, wherein each depth plane is parallel to the display and intersects a surface normal of the display; and executing, on the computing device, a user interface command based on the input gesture and the assigned depth plane.

Example 27 includes the subject matter of Example 26, and wherein assigning the depth plane further comprises assigning a depth plane of a plurality of depth planes as a function of the depth of the input gesture relative to the display.

Example 28 includes the subject matter of any of Examples 26 and 27, and wherein receiving the depth sensor data comprises receiving depth sensor data from a stereo depth camera of the computing device.

Example 29 includes the subject matter of any of Examples 26-28, and wherein receiving the depth sensor data comprises receiving depth sensor data from a structured light camera of the computing device.

Example 30 includes the subject matter of any of Examples 26-29, and wherein receiving the depth sensor data comprises receiving depth sensor data from a plurality of cameras of the computing device, wherein each camera of the plurality of cameras is aimed perpendicular to the surface normal of the display and perpendicular to another camera of the plurality of cameras.

Example 31 includes the subject matter of any of Examples 26-30, and wherein receiving the depth sensor data comprises receiving image data from a camera of the computing device; and determining the depth of the input gesture comprises determining the depth of the input gesture as a function of a position of a shadow cast by the user captured in the received image data.

Example 32 includes the subject matter of any of Examples 26-31, and wherein receiving the depth sensor data comprises receiving depth sensor data from a proximity sensor of the computing device.

Example 33 includes the subject matter of any of Examples 26-32, and further including receiving, on the computing device, using a radio receiver of the computing device, a signal transmitted by a transmitter manipulated by the user; wherein receiving the depth sensor data comprises receiving signal strength data associated with the received signal from the radio receiver.

Example 34 includes the subject matter of any of Examples 26-33, and wherein executing the user interface command based on the assigned depth plane comprises selecting a virtual object as a function of the assigned depth plane; and controlling the virtual object based on the input gesture.

Example 35 includes the subject matter of any of Examples 26-34, and wherein selecting the virtual object comprises selecting a player character as a function of the assigned depth plane.

Example 36 includes the subject matter of any of Examples 26-35, and further including configuring, on the computing device, prior to recognizing the input gesture, the user interface command to be executed based on the input gesture and the assigned depth plan.

Example 37 includes the subject matter of any of Examples 26-36, and wherein executing the user interface command comprises determining whether the assigned depth plane comprises a secondary virtual touch plane of the computing device; and executing a secondary user interface command in response to determining the assigned depth plane comprises the secondary virtual touch plane.

Example 38 includes the subject matter of any of Examples 26-37, and further including allowing access to the secondary user interface command only through the secondary virtual touch plane.

Example 39 includes the subject matter of any of Examples 26-38, and wherein executing the secondary user interface command comprises magnifying a user interface element displayed on the display of the computing device.

Example 40 includes the subject matter of any of Examples 26-39, and wherein executing the secondary user interface command comprises selecting a user interface element displayed on the display of the computing device.

Example 41 includes the subject matter of any of Examples 26-40, and wherein executing the secondary user interface command comprises activating a secondary interaction mode for the user interface element, wherein a primary interaction mode for the user interface element is accessible via a touch screen of the display.

Example 42 includes the subject matter of any of Examples 26-41, and wherein executing the secondary user interface command comprises displaying a contextual command menu.

Example 43 includes the subject matter of any of Examples 26-42, and wherein executing the secondary user interface command comprises assigning a value to the secondary user interface command as a function of the depth of the input gesture.

Example 44 includes the subject matter of any of Examples 26-43, and wherein executing the secondary user interface command comprises assigning a three-dimensional position and velocity to the secondary user interface command as a function of the input gesture.

Example 45 includes the subject matter of any of Examples 26-44, and further including identifying, on the computing device, the user of the computing device; and assigning, on the computing device, the input gesture to the identified user; wherein executing the user interface command further comprises executing the user interface command as a function of the user assigned to the input gesture.

Example 46 includes the subject matter of any of Examples 26-45, and wherein identifying the user comprises identifying the user by facial recognition.

Example 47 includes the subject matter of any of Examples 26-46, and wherein assigning the input gesture to the identified user comprises tracking a skeletal model of the identified user, the skeletal model having a limb; and assigning the input gesture to the identified user comprises assigning the gesture to the limb of the skeletal model of the identified user.

Example 48 includes the subject matter of any of Examples 26-47, and wherein identifying the user comprises identifying the user by eye tracking.

Example 49 includes the subject matter of any of Examples 26-48, and wherein identifying the user comprises identifying the user by voice recognition.

Example 50 includes the subject matter of any of Examples 26-49, and further including configuring, on the computing device, the depth plane to be assigned to the input gesture assigned to the user.

Example 52 includes one or more machine-readable storage media comprising a plurality of instructions stored thereon that in response to being executed result in a computing device performing the method of any of Examples 26-50.

Example 53 includes a computing device for depth-based gesture control, the computing device comprising means for recognizing, on a computing device, an input gesture performed by a user of the computing device in front of a display of the computing device; means for receiving, on the computing device, depth sensor data indicative of a depth relative to the display of the input gesture from a depth sensor of the computing device; means for determining, on the computing device, the depth of the input gesture as a function of the depth sensor data; means for assigning, on the computing device, a depth plane to the input gesture as a function of the depth of the input gesture, wherein each depth plane is parallel to the display and intersects a surface normal of the display; and means for executing, on the computing device, a user interface command based on the input gesture and the assigned depth plane.

Example 54 includes the subject matter of Example 53, and wherein the means for assigning the depth plane further comprises means for assigning a depth plane of a plurality of depth planes as a function of the depth of the input gesture relative to the display.

Example 55 includes the subject matter of any of Examples 53 and 54, and wherein the means for receiving the depth sensor data comprises means for receiving depth sensor data from a stereo depth camera of the computing device.

Example 56 includes the subject matter of any of Examples 53-55, and wherein the means for receiving the depth sensor data comprises means for receiving depth sensor data from a structured light camera of the computing device.

Example 57 includes the subject matter of any of Examples 53-56, and wherein the means for receiving the depth sensor data comprises means for receiving depth sensor data from a plurality of cameras of the computing device, wherein each camera of the plurality of cameras is aimed perpendicular to the surface normal of the display and perpendicular to another camera of the plurality of cameras.

Example 58 includes the subject matter of any of Examples 53-57, and wherein the means for receiving the depth sensor data comprises means for receiving image data from a camera of the computing device; and the means for determining the depth of the input gesture comprises means for determining the depth of the input gesture as a function of a position of a shadow cast by the user captured in the received image data.

Example 59 includes the subject matter of any of Examples 53-58, and wherein the means for receiving the depth sensor data comprises means for receiving depth sensor data from a proximity sensor of the computing device.

Example 60 includes the subject matter of any of Examples 53-59, and further including means for receiving, on the computing device, using a radio receiver of the computing device, a signal transmitted by a transmitter manipulated by the user; wherein the means for receiving the depth sensor data comprises means for receiving signal strength data associated with the received signal from the radio receiver.

Example 61 includes the subject matter of any of Examples 53-60, and wherein the means for executing the user interface command based on the assigned depth plane comprises means for selecting a virtual object as a function of the assigned depth plane; and means for controlling the virtual object based on the input gesture.

Example 62 includes the subject matter of any of Examples 53-61, and wherein the means for selecting the virtual object comprises means for selecting a player character as a function of the assigned depth plane.

Example 63 includes the subject matter of any of Examples 53-62, and further including means for configuring, on the computing device, prior to recognizing the input gesture, the user interface command to be executed based on the input gesture and the assigned depth plan.

Example 64 includes the subject matter of any of Examples 53-63, and wherein the means for executing the user interface command comprises means for determining whether the assigned depth plane comprises a secondary virtual touch plane of the computing device; and means for executing a secondary user interface command in response to determining the assigned depth plane comprises the secondary virtual touch plane.

Example 65 includes the subject matter of any of Examples 53-64, and further including means for allowing access to the secondary user interface command only through the secondary virtual touch plane.

Example 66 includes the subject matter of any of Examples 53-65, and wherein the means for executing the secondary user interface command comprises means for magnifying a user interface element displayed on the display of the computing device.

Example 67 includes the subject matter of any of Examples 53-66, and wherein the means for executing the secondary user interface command comprises means for selecting a user interface element displayed on the display of the computing device.

Example 68 includes the subject matter of any of Examples 53-67, and wherein the means for executing the secondary user interface command comprises means for activating a secondary interaction mode for the user interface element, wherein a primary interaction mode for the user interface element is accessible via a touch screen of the display.

Example 69 includes the subject matter of any of Examples 53-68, and wherein the means for executing the secondary user interface command comprises means for displaying a contextual command menu.

Example 70 includes the subject matter of any of Examples 53-69, and wherein the means for executing the secondary user interface command comprises means for assigning a value to the secondary user interface command as a function of the depth of the input gesture.

Example 71 includes the subject matter of any of Examples 53-70, and wherein the means for executing the secondary user interface command comprises means for assigning a three-dimensional position and velocity to the secondary user interface command as a function of the input gesture.

Example 72 includes the subject matter of any of Examples 53-71, and further including means for identifying, on the computing device, the user of the computing device; and means for assigning, on the computing device, the input gesture to the identified user; wherein the means for executing the user interface command further comprises means for executing the user interface command as a function of the user assigned to the input gesture.

Example 73 includes the subject matter of any of Examples 53-72, and wherein the means for identifying the user comprises means for identifying the user by facial recognition.

Example 74 includes the subject matter of any of Examples 53-73, and wherein the means for assigning the input gesture to the identified user comprises means for tracking a skeletal model of the identified user, the skeletal model having a limb; and the means for assigning the input gesture to the identified user comprises means for assigning the gesture to the limb of the skeletal model of the identified user.

Example 75 includes the subject matter of any of Examples 53-74, and wherein the means for identifying the user comprises means for identifying the user by eye tracking.

Example 76 includes the subject matter of any of Examples 53-75, and wherein the means for identifying the user comprises means for identifying the user by voice recognition.

Example 77 includes the subject matter of any of Examples 53-76, and further including means for configuring, on the computing device, the depth plane to be assigned to the input gesture assigned to the user.