Three dimensional user interface session control

A method, including receiving, by a computer executing a non-tactile three dimensional (3D) user interface, a set of multiple 3D coordinates representing a gesture by a hand positioned within a field of view of a sensing device coupled to the computer, the gesture including a first motion in a first direction along a selected axis in space, followed by a second motion in a second direction, opposite to the first direction, along the selected axis. Upon detecting completion of the gesture, the non-tactile 3D user interface is transitioned from a first state to a second state.

FIELD OF THE INVENTION

This invention relates generally to user interfaces for computerized systems, and specifically to user interfaces that are based on three-dimensional sensing.

BACKGROUND OF THE INVENTION

Many different types of user interface devices and methods are currently available. Common tactile interface devices include the computer keyboard, mouse and joystick. Touch screens detect the presence and location of a touch by a finger or other object within the display area. Infrared remote controls are widely used, and “wearable” hardware devices have been developed, as well, for purposes of remote control.

Computer interfaces based on three-dimensional (3D) sensing of parts of the user's body have also been proposed. For example, PCT International Publication WO 03/071410, whose disclosure is incorporated herein by reference, describes a gesture recognition system using depth-perceptive sensors. A 3D sensor provides position information, which is used to identify gestures created by a body part of interest. The gestures are recognized based on a shape of a body part and its position and orientation over an interval. The gesture is classified for determining an input into a related electronic device.

As another example, U.S. Pat. No. 7,348,963, whose disclosure is incorporated herein by reference, describes an interactive video display system, in which a display screen displays a visual image, and a camera captures 3D information regarding an object in an interactive area located in front of the display screen. A computer system directs the display screen to change the visual image in response to changes in the object.

Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.

SUMMARY OF THE INVENTION

There is provided, in accordance with an embodiment of the present invention a method, including receiving, by a computer executing a non-tactile three dimensional (3D) user interface, a set of multiple 3D coordinates representing a gesture by a hand positioned within a field of view of a sensing device coupled to the computer, the gesture including a first motion in a first direction along a selected axis in space, followed by a second motion in a second direction, opposite to the first direction, along the selected axis, and transitioning the non-tactile 3D user interface from a first state to a second state upon detecting completion of the gesture.

There is also provided, in accordance with an embodiment of the present invention a method, including receiving, by a computer executing a non-tactile three dimensional (3D) user interface, a set of multiple 3D coordinates representing a gesture by a hand positioned within a field of view of a sensing device coupled to the computer, the gesture including a rising motion along a vertical axis in space, and transitioning the non-tactile 3D user interface from a locked state to an unlocked state upon detecting completion of the gesture.

There is additionally provided, in accordance with an embodiment of the present invention a method, including associating, in a computer executing a non-tactile three dimensional (3D) user interface, multiple regions, including at least first and second regions, within a field of view of a sensing device coupled to the computer with respective states of the non-tactile 3D user interface, including at least first and second states associated respectively with the first and second regions, receiving a set of multiple 3D coordinates representing a hand movement from the first region to the second region, and responsively to the movement, transitioning the non-tactile 3D user interface from the first state to the second state.

There is further provided, in accordance with an embodiment of the present invention an apparatus, including a three dimensional (3D) optical sensor having a field of view and coupled to a computer executing a non-tactile three dimensional (3D) user interface, and an illumination element that when illuminated, is configured to be visible to a user when the user is positioned within the field of view.

There is additionally provided, in accordance with an embodiment of the present invention an apparatus, including a sensing device, and a computer executing a non-tactile three dimensional (3D) user interface and configured to receive, from the sensing device, a set of multiple 3D coordinates representing a gesture by a hand positioned within a field of view of the sensing device, the gesture including a first motion in a first direction along a selected axis in space, followed by a second motion in a second direction, opposite to the first direction, along the selected axis, and to transition the non-tactile 3D user interface from a first state to a second state upon detecting completion of the gesture.

There is also provided, in accordance with an embodiment of the present invention an apparatus, including a sensing device, and a computer executing a non-tactile three dimensional (3D) user interface and configured to receive, from the sensing device, a set of multiple 3D coordinates representing a gesture by a hand positioned within a field of view of the sensing device, the gesture including a rising motion along a vertical axis in space, and to transition the non-tactile 3D user interface from a locked state to an unlocked state upon detecting completion of the gesture.

There is alternatively provided, in accordance with an embodiment of the present invention an apparatus, including a sensing device, and a computer executing a non-tactile three dimensional (3D) user interface and configured to associate multiple regions, including at least first and second regions, within a field of view of the sensing device with respective states of the non-tactile 3D user interface, including at least first and second states associated respectively with the first and second regions, to receive a set of multiple 3D coordinates representing a hand movement from the first region to the second region, and responsively to the movement, to transition the non-tactile 3D user interface from the first state to the second state.

There is also provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile user interface, cause the computer to receive, from a sensing device, a set of multiple 3D coordinates representing a gesture by a hand positioned within a field of view of the sensing device, the gesture including a first motion in a first direction along a selected axis in space, followed by a second motion in a second direction, opposite to the first direction, along the selected axis, and to transition the non-tactile 3D user interface from a first state to a second state upon detecting completion of the gesture.

There is additionally provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile user interface, cause the computer to receive, from a sensing device, a set of multiple 3D coordinates representing a gesture by a hand positioned within a field of view of the sensing device, the gesture including a rising motion along a vertical axis in space, and to transition the non-tactile 3D user interface from a locked state to an unlocked state upon detecting completion of the gesture.

There is further provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile user interface, cause the computer to associate multiple regions, including at least first and second regions, within a field of view of a sensing device with respective states of the non-tactile 3D user interface, including at least first and second states associated respectively with the first and second regions, to receiving a set of multiple 3D coordinates representing a hand movement from the first region to the second region, and responsively to the movement, to transition the non-tactile 3D user interface from the first state to the second state.

DETAILED DESCRIPTION OF EMBODIMENTS

When using physical tactile input devices such as buttons, rollers or touch screens, a user typically engages and disengages control of a user interface by touching and/or manipulating the physical device. Embodiments of the present invention describe gestures for engaging and disengaging control of a user interface based on three-dimensional (3D) sensing (referred to herein as a non-tactile 3D user interface), by a 3D sensor, of motion or change of position of one or more body parts, typically a hand, of the user. Gestures described herein include focus gestures and unlock gestures. A focus gesture enables the user to engage (i.e., take control of) an inactive non-tactile 3D user interface. An unlock gesture enables the user to engage a locked non-tactile 3D user interface, as pressing a specific sequence of keys unlocks a locked cellular phone. In some embodiments, the non-tactile 3D user interface conveys visual feedback to the user performing the focus and the unlock gestures.

Embodiments of the present invention also describe methods for conveying visual feedback to the user, when the user's hand disengages from the non-tactile 3D user interface. The visual feedback typically alerts the user in an unobtrusive manner, thereby enhancing the user's experience.

As described supra, a 3D sensor captures 3D information regarding an object, typically a body part such as a hand, in an interactive area located in front of a display screen. Since the 3D sensor typically has a fixed field of view, a computer can track and accept inputs from the user when the body part is positioned within the field of view. Embodiments of the present invention describe methods and systems for conveying visual feedback to the user when the body part is within the field of view, outside the field of view, and when the user is at the periphery of the field of view.

System Description

FIG. 1is a schematic, pictorial illustration of a non-tactile 3D user interface20(also referred to herein as the user interface) for operation by a user22of a computer26, in accordance with an embodiment of the present invention. The non-tactile 3D user interface is based on a 3D sensing device24coupled to the computer, which captures 3D scene information of a scene that includes the body (or at least a body part, such as one or more of hands30) of the user. Device24or a separate camera (not shown in the figures) may also capture video images of the scene. The information captured by device24is processed by computer26, which drives a display28accordingly.

Computer26, executing 3D user interface20, processes data generated by device24in order to reconstruct a 3D map of user22. The term “3D map” refers to a set of 3D coordinates measured, by way of example, with reference to a generally horizontal X-axis32in space, a generally vertical Y-axis34in space and a depth Z-axis36in space, based on device24. The 3D coordinates represent the surface of a given object, in this case the user's body. In one embodiment, device24projects a pattern of spots onto the object and captures an image of the projected pattern. Computer26then computes the 3D coordinates of points on the surface of the user's body by triangulation, based on transverse shifts of the spots in the pattern. Methods and devices for this sort of triangulation-based 3D mapping using a projected pattern are described, for example, in PCT International Publications WO 2007/043036, WO 2007/105205 and WO 2008/120217, whose disclosures are incorporated herein by reference. Alternatively, interface20may use other methods of 3D mapping, using single or multiple cameras or other types of sensors, as are known in the art.

Computer26typically comprises a general-purpose computer processor, which is programmed in software to carry out the functions described hereinbelow. The software may be downloaded to the processor in electronic form, over a network, for example, or it may alternatively be provided on non-transitory tangible media, such as optical, magnetic, or electronic memory media. Alternatively or additionally, some or all of the functions of the image processor may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although computer26is shown inFIG. 1, by way of example, as a separate unit from sensing device24, some or all of the processing functions of the computer may be performed by suitable dedicated circuitry within the housing of the sensing device or otherwise associated with the sensing device.

As another alternative, these processing functions may be carried out by a suitable processor that is integrated with display28(in a television set, for example) or with any other suitable sort of computerized device, such as a game console or media player. The sensing functions of device24may likewise be integrated into the computer or other computerized apparatus that is to be controlled by the sensor output.

Focus Gestures

In the embodiments described herein, user interface20comprises the following individual states:Unlocked/Locked. While locked, user interface20typically ignores all gestures except for an unlock gesture (described hereinbelow) that transitions the non-tactile 3D user interface to an unlocked state. When unlocked, gestures, such as those from hand30, can interact with user interface20.Tracked/Not-tracked. In embodiments of the present invention, tracking refers to user interface20focusing on a specific body part of an individual in order for the user to interact with the non-tactile 3D user interface. When user interface20is in the tracked state, the non-tactile 3D user interface can track and interpret gestures from the specific body part, e.g., hand30. While in the not-tracked state, the non-tactile 3D user interface is not focusing on any specific individual or body part.Active/Inactive. User interface20is active when the user interface is unlocked, engaged with and tracking user22and able to accept gestures from the user. When user22is disengaged from user interface20, the non-tactile 3D user interface is inactive.

In embodiments of the present invention, the state of user interface20typically comprises a combination of the states described supra. The states of user interface20may include:Tracked, Unlocked and Active.Tracked, Unlocked and Inactive.Not-Tracked, Unlocked and Inactive.Not-Tracked, Locked and Inactive.Tracked, Locked and Inactive.
A state diagram detailing the transitions between the states of 3D user interface20is shown inFIG. 7, described hereinbelow.

To engage 3D user interface20while positioned in a field of view of sensing device24, user22may perform a focus gesture. A well-designed focus gesture typically strikes a balance between ease of use and a low instance of false positives (i.e., a physical gesture that the computer incorrectly identifies as a focus gesture). On the one hand, a simple focus gesture (for example, pointing an index finger) may be easy to learn, but may be prone to generating excessive false positives. On the other hand, a complex focus gesture may generate few false positives, but may also be difficult for the user to learn. Typically, a well designed focus gesture has a false positive rate of less than 2%.

A focus gesture comprising multiple physical motions can be broken down into a series of steps performed in a specific sequence. In some embodiments, computer26conveys feedback to user22during and/or upon completion of each of the steps. The focus gesture steps should typically be distinct enough so as not to interfere with the operation of user interface20(i.e., by generating false positives). For example, if user interface20is configured to show movies from a movie library stored on the computer, the focus gesture steps should be sufficiently different from the gestures used to control the movie library (e.g., gestures that select and control playback of a movie).

A focus gesture, used to engage user interface20, may include a “push” gesture or a “wave” gesture. As described in detail hereinbelow, the focus gesture may comprise user22performing, with hand30, a first motion in a first direction along a selected axis (in space), followed a second motion in a second direction, opposite to the first direction, along the selected axis. In some embodiments, computer26conveys visual feedback to user22as the user performs and/or completes each step of the focus gesture. The feedback can help train user22to perform the focus gesture correctly.

FIG. 2is a schematic pictorial illustration of user22performing a push gesture, in accordance with an embodiment of the present invention. The push gesture comprises user22performing a combination of the following:A first motion comprising pushing hand30forward (i.e., towards display28) at a minimum focus gesture speed and for at least a focus gesture distance along Z-axis36.A second motion comprising pulling hand30back (i.e., towards user22) at a minimum focus gesture speed and for at least a focus gesture distance along Z-axis36.

For example, the minimum focus gesture speed and the focus gesture distance may comprise 10 centimeters per second, and 10 centimeters, respectively. The forward and backward motions of the push gesture are indicated by arrows40. As user22moves hand30along Z-axis36, computer26receives, from sensing device24, a set of multiple 3D coordinates representing the forward and backward motion of the hand (i.e., the push gesture). Upon detecting completion of the push gesture, computer26can transition user interface20from a first state (e.g., not tracked) to a second state (e.g., tracked).

FIG. 3is a schematic pictorial illustration of user22performing a wave gesture, in accordance with an embodiment of the present invention. The wave gesture comprises user22performing a combination of the following:A first gesture comprising moving hand30in a swiping motion from a first side to a second side (i.e., either left-to-right right-to-left) at the minimum focus gesture speed, and for at least the focus gesture distance along X-axis32.A second gesture comprising moving hand30in a swiping motion from the second side to the first side at the minimum focus gesture speed, and for at least the focus gesture distance along X-axis32.

The side-to-side swiping motions of the wave gesture are indicated by arrows50. As user22moves hand30along X-axis32, computer26receives, from sensing device24, a set of multiple 3D coordinates representing the side-to-side motion of the hand (i.e., the wave gesture). Upon detecting completion of the wave gesture, computer26can transition user interface20from a first state (e.g., not tracked) to a second state (e.g., tracked).

FIG. 4is a schematic pictorial illustration of computer (i.e., via display28) conveying visual feedback to user22, as the user performs a focus gesture, in accordance with an embodiment of the present invention. In some embodiments, computer26may control a visual feedback device60coupled to display28and computer26, such as a light emitting diode (LED) that may change color as user22performs the focus gesture.

The visual feedback may comprise a first visual feedback prior to the first gesture of the focus gesture, a second visual feedback subsequent to the first gesture, and a third visual feedback subsequent to the second gesture of the focus gesture. For example, prior to performing the focus gesture, user interface20can illuminate LED60in a first color, e.g., red. After user22performs the first gesture of the focus gesture (e.g., by pushing hand30towards sensing device24to initiate the push gesture or by swiping the hand from a first side to a second side to initiate the wave gesture), computer26can illuminate LED60in a second color, e.g., orange. Finally, after user22completes the second gesture of the focus gesture (e.g., by pulling hand30back from sensing device24to complete the push gesture or by swiping the hand back from the second side to the first side to complete the wave gesture), the computer can illuminate LED60in a third color, e.g., green, and engage user22with user interface20.

In an additional embodiment, visual feedback device60may comprise a single color LED that blinks (i.e., illuminates and darkens) as user22performs a focus gesture. During periods between focus gestures, the single LED may be either constantly illuminated or darkened. In an alternative embodiment, visual feedback device60may comprise multiple LEDs that convey visual feedback to user22before, during and after performing the focus gesture (e.g., separate red, yellow and green LEDs as in a traffic light.

In a further embodiment, visual feedback device60may comprise a vertical or a circular array of LEDs. When user interface20is inactive, computer26darkens the LEDs. As user performs the focus gesture, computer26can illuminate an additional LED with each individual gesture (e.g., the side-to-side swipe of hand30for the wave gesture or the forward and backward motion of hand30for the push gesture). After user22completes the focus gesture, computer26can illuminate all the LEDs.

In still yet another embodiment, visual feedback device60may comprise a horizontal array of LEDs. When user interface20is disengaged, computer26can illuminate a single LED in the horizontal array. As user22performs the focus gesture, computer26can toggle the LEDs in the horizontal array to mimic the motion of hand30.

Additionally or alternatively, computer26may alter a feedback item presented on display28while user22performs the focus gesture. For example, the feedback item may comprise a status icon62that either changes its appearance or displays an animation (e.g., a triangular shape within the icon that alters shape) during the focus gesture.

In alternative embodiments, the feedback item may comprise a circle64on display28, and computer26can change the size of the feedback item depending on the location of hand30during the focus gesture. For example, as user22moves hand30closer to sensing device24to initiate a push gesture, computer26may increase the diameter of circle64, or vice versa. Visual feedback conveyed by computer26may also include an indication as to the speed of the gesture (i.e. whether user22is moving hand30at an appropriate speed or not), and/or an indication when the hand has moved a sufficient distance to complete one of the focus gesture steps.

In further embodiments, the feedback may comprise a text message presented on display28. For example, after user22performs the first gesture of the push gesture (i.e., moving hand30forward), computer28can present a text message such as “Pull hand back to gain control”.

Unlock Gesture

In embodiments of the invention, states of 3D user interface20may include the locked and the unlocked states. The user interface may transition to the locked state either automatically after a defined period of inactivity, or after user22explicitly performs a lock gesture. While in the locked state, user22is disengaged from user interface20. In some embodiments, user22performs the focus gesture followed by an unlock gesture, thereby unlocking and engaging user interface20.

Alternatively, user interface20may implement a spatial aware gesture lock, where the state of the user interface may be unlocked for a specific region including user22, but locked for other regions in proximity to the specific region (and therefore locked for any individuals in the other regions).

Examples of unlock gestures include an “up” gesture (e.g., raising hand30a specified distance), a sequence of two sequential wave gestures, and a sequence of two sequential push gestures, as described in detail hereinbelow.

FIG. 5is a schematic pictorial illustration of user22performing an up gesture, in accordance with an embodiment of the present invention. The up gesture comprises user22raising hand30vertically, at a minimum unlock gesture speed, and for at least an unlock gesture distance along Y-axis34, as indicated by arrow70. For example, the minimum unlock gesture speed and the unlock gesture distance (i.e., for an unlock gesture such as the up gesture) map comprise four centimeters per second, and 20 centimeters, respectively.

As user22elevates hand30along Y-axis34, computer26receives, from sensing device24, a set of multiple 3D coordinates representing the rising motion of the hand (i.e., the up gesture). Upon detecting completion of the up gesture, computer26can transition user interface20from a locked state to an unlocked state.

While locked, the state of user interface20is typically not-tracked, locked and inactive. To unlock user interface20, user22typically first performs a focus gesture, which transitions user interface20to the tracked, locked and inactive state. Upon detecting the focus gesture, computer26may convey feedback (either on display28or on device60) prompting user22to elevate hand30to unlock the user interface (i.e., to perform the unlock gesture). Performing the unlock gesture engages the user interface, and transitions user interface20to the tracked, unlocked and active state.

As described supra, user22can unlock user interface20by performing two focus gestures sequentially. After detecting the first focus gesture, computer26transitions user interface20from the not-tracked, locked and inactive state to the tracked, locked and inactive state, and after detecting the second focus gesture, the computer transitions the non-tactile 3D user interface to the tracked, unlocked and active state. Thus, for example, unlocking user interface20may comprise user22performing either two wave gestures, two push gestures, or a combination of the two.

Computer26may also convey a first visual feedback to the user performing the unlock gesture, and a second visual feedback subsequent to the user performing the unlock gesture. For example, visual feedback device60may comprise a red LED that illuminates when user interface20is the locked state, and a green LED that illuminates when the user interface is in the unlocked state. In an alternative embodiment, visual feedback device60may comprise a multi-colored LED that changes color upon computer26transitioning user interface20to either the locked or the unlocked state.

In an additional embodiment, computer26may convey visual feedback via a feedback item presented on display28. For example, the feedback item may comprise an icon34that is configured to show either a closed padlock or a closed eye when user interface20is in the locked state, and either an open padlock or an open eye when the user interface is the unlocked state.

Dropping Sessions

As hand30interacts with 3D user interface20, the position of the hand may influence the state of the non-tactile 3D user interface. For example, if user22drops hand30to the user's lap, then the user may disengage from the non-tactile 3D user interface, with computer26transitioning user interface20from the tracked, active and unlocked state to the not-tracked, inactive and unlocked state. Upon detecting user20performing a focus gesture, computer26can transition user interface20back to the tracked, active and unlocked state, and reengages the user interface.

FIGS. 6A,6B,6C, and6D are schematic pictorial illustrations of user interface20responding to vertical movement of hand30, in accordance with an embodiment of the present invention. In the example shown inFIG. 6A, hand30can move between an active region80, a pre-drop region84and dropped region86, where each of the regions is associated with a state of 3D user interface20. As shown in the figure, active region80is associated with a tracked and active state, pre-drop region84is associated with a tracked and inactive state, and dropped region86is associated with a not tracked and inactive state.

In operation, computer26defines multiple regions comprising at least a first region and a second region within a field of view of sensing device24, and associates each of the defined regions with a state of user interface20. As user22moves hand30from the first region (e.g., region80) to the second region (e.g., region82), computer26receives a set of multiple 3D coordinates representing the hand moving from the first region to the second region. Upon detecting hand30moving from the first region to the second region, computer26responsively transitions 3D user interface20from the state associated with the first region to the state associated with the second region.

While hand30is within active region80, user interface20may respond to gestures performed by the hand, as the state of the 3D user interface is tracked, active and unlocked. In some embodiments, computer26may convey visual feedback to user22indicating a current state of 3D user interface20. For example, while positioned within region80, hand30may interact with user interface20via a softbar82, as shown inFIG. 6B. Softbar82may also be referred to a horizontal bar user interface. While hand30interacts with softbar82, computer26may position the softbar at a fixed location on display28. A non-tactile 3D user interface incorporating softbar82is described, for example, in U.S. patent application Ser. No. 13/161,508, filed Jun. 16, 2011, whose disclosure is incorporated herein by reference.

If user22lowers hand30from region80to pre-drop region84, computer26transitions the state of user interface20to the tracked, inactive and unlocked state. While hand30is in region84, the hand is disengaged from user interface20(i.e., the non-tactile 3D user interface may ignore gestures from the hand), but the non-tactile 3D user interface is still tracking the hand.

In some embodiments, while hand30is within region84, computer26moves the vertical position of softbar82in synchronization with the hand, as indicated by arrows88inFIG. 6C. The vertical movement of softbar82conveys a “gentle” feedback to user22indicating a potential disengagement from user interface20, should the user move hand30down to dropped region86. If user22lowers hand30into region86, computer26may not present softbar82(as shown inFIG. 6D), and the computer transitions the 3D user interface to the not-tracked, inactive and unlocked state.

To reengage user interface20while hand30is within region84, user22can elevate the hand back to region80, and computer26transitions the non-tactile 3D user interface back to the tracked, active and unlocked state. However, since the state of user interface20is not-tracked, inactive and unlocked while hand30is within region86, the user may be required to perform a focus gesture in order to reengage the 3D user interface.

In some embodiments, active region80comprises a static region whose mid-point has a vertical coordinate where user22performed the focus gesture, thereby engaging user interface20. In alternative embodiments, computer26may adjust boundaries of the regions responsively to recent movements of hand30. For example, computer26may employ temporal filtering (or another similar algorithm) to update the mid-point, by periodically averaging the vertical coordinates of hand30when the hand performed recent gestures. By updating the mid-point, computer may also update the upper and lower boundaries of active region80. Computer26can also use temporal filtering to assist in defining a horizontal (i.e., a side-to-side) active zone (not shown).

In some instances, hand30may engage user interface20, but user22may be physically unable to lower the hand to pre-drop region84. For example, user22may be sitting on a couch with hand30resting on an armrest. In response, computer26may “compress” regions80,84and86, thereby repositioning pre-drop region84to an appropriate (i.e., a reachable) level. Alternatively, computer26may present feedback, prompting user22to elevate hand30in order to engage the non-tactile 3D user interface. For example, computer26may only present the top half of softbar82at the bottom of display28, thereby prompting the user to elevate hand30to a higher vertical position (at which point the softbar may be displayed in its entirety).

FIG. 7is a state diagram90that schematically illustrates the states and the transitions of user interface20, in accordance with embodiments of the present invention. When user22positions hand30in active region80and interacts with user interface20, computer26sets the state the non-tactile 3D user interface to a tracked, unlocked and active state92. Upon user22lowering hand30to pre-drop region84, computer26disengages the hand from user interface20, and the computer transitions the non-tactile 3D user interface from state92to a tracked, unlocked and inactive state94. While in state94, computer26still tracks hand30, but may not accept any commands from the hand. Computer26transitions user interface20back to state92, responsively to detecting that user22elevates hand30back to active region80.

Computer26transitions user interface20from state96back to state92responsively to detecting user22performing a focus gesture as described supra. Upon transitioning to state96, computer26activates a second time-out timer. If computer does not detect a focus gesture within a second specified period (e.g., ten seconds), then the computer transitions user interface20from state96to a not-tracked, locked and inactive state98.

Computer26transitions user interface20from state98to state92(i.e., unlocking and reengaging the user interface) upon detecting user22performing a focus gesture, followed by an unlock gesture. Upon detecting user22performing the focus gesture, computer26transitions user interface20from state98to a tracked, locked and inactive state100. When computer26transitions user interface20to state100, the computer activates a third timeout timer. If computer26detects user22either moving hand30from active region80(the hand is within region80when performing the focus gesture) or not performing a focus gesture within a third specified period, then the computer transition user interface20from state100back to state98. Finally, if user22performs an unlock gesture within the second specified period of time, then computer26transitions user interface20from state100to state92.

Field of View

FIG. 8is a schematic, pictorial illustration showing 3D sensing device24configured to convey visual feedback to user indicating the user's position relative to a field of view110of the 3D sensing device, in accordance with an embodiment of the present invention. Field of view110defines the volume of space that sensing device24can “see”. 3D sensing device24comprises a 3D optical sensor111and an illumination element112that is configured to convey visual feedback to user22indicating when the user is located within field of view110.

In the example shown inFIG. 8, illumination element112comprises a single light emitting diode (LED) positioned in proximity to an apex of a conical shaft114, and where the LED is in proximity to optical sensor111. Typically, when user22is within field of view110, the user can see the LED.

Field of view110comprises a central field of view116bounded by peripheral fields of view118and120. In some embodiments, user22sees the entire illumination element (e.g., a circle) when the user is within central field of view116. As user22moves to periphery fields of view118or120, the user may only see part of the illumination element (e.g., a semicircle). In other words, if user22can see any part of the illumination element, then optical sensor111can see the user.

In some embodiments, conical shaft114may include a customized slit (not shown), thereby enabling 3D sensing device24to present the illumination emanating from the illumination element as a specific shape (e.g., a company logo). In alternative embodiments, illumination element112may comprise multiple (e.g., three) LEDs positioned on 3D sensing device24, where each of the multiple LEDs has a different field of view. When user22sees all the LEDs, the user is within field of view110.

In an additional embodiment, illumination element112may be configured to convey visual feedback to user20indicating a current state of 3D user interface20to the user. In some embodiments, illumination element112may comprise multiple LEDs that are configured to present session indications (e.g., the state of user interface20) to different individuals within field of view110. For example, each of the multiple LEDs may comprise mechanical and/or optical elements that restrict each of the LEDs to different fields of view. Embodiments comprising multiple LEDs with different fields of view can also be used to convey feedback to multiple individuals within field of view110.

In further embodiments, computer26may associate each state of user interface20with a specific color, and illumination element112may be configured to illuminate in different colors, based on the current state of the non-tactile 3D user interface. For example, while user interface20is in tracked, unlocked and active state92to user22, computer26can illuminate illumination element112in green. Likewise, while user interface20is in tracked, unlocked and inactive state94to user22, computer26can illuminate illumination element112in yellow, thereby conveying an indication to the user to raise hand30to region80.

In still yet another embodiment, field of view110may comprise multiple regions (no shown), where additional users (not shown) in each region have a different state with user interface20. For example, a first given user22positioned in a first given region can be in the locked state with 3D user interface20, and a second given user20in a second given region can be in the active state with the non-tactile 3D user interface. Additionally, illumination element112can be configured to convey different visual feedback (e.g., different colors) to each of the regions, depending on their state with user interface20. For example, visual feedback conveyed to the first given user a red illumination indicating that the first given user is positioned in a region that is in not tracked, unlocked and inactive state94. Therefore to engage user interface20, the first given user may be required to perform an unlock gesture.