Creating a virtual environment for touchless interaction

This disclosure is directed to a touchless interactive environment. An input device may be configured to capture electronic images corresponding to physical objects detectable within a physical three-dimensional region. A computer system may establish a virtual three-dimensional region mapped to the physical three-dimensional region, with the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the plurality of electronic images. The computer system may select a virtual object from the plurality of virtual objects as one or more commanding objects, with the one or more commanding objects indicating a command of a graphical user interface to be performed based on a position of the one or more commanding objects. The computer system may then perform the command of the graphical user interface based on the position of the one or more commanding objects.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to a system, method, and computer-readable medium for creating a virtual environment for touchless interaction. In particular, the disclosed subject matter relates to capturing images of an object in a physical three-dimensional region and translating the position of the object in the physical three-dimensional region into a command understandable by a graphical user interface (GUI).

BACKGROUND

A computer system may have an input device, such as a mouse, to receive input and also a display to output information. The mouse may be used on a flat, two-dimensional surface, and the movements performed on it may be replicated to the display in the form of a cursor moving on a two-dimensional plane. In effect, the two-dimensional plane mimics the surface that the mouse is on and the cursor mimics the current position of the mouse on the surface. The visual feedback of the cursor on the display may include an image, such as the image of an arrow. The action of manipulating the mouse, such as pressing a button or scrolling a wheel, may cause the mouse to provide input commands to the computer system in correlation with the current cursor position. The computer system may display which commands are acceptable by way of a GUI displayed on the display.

While a mouse is one type of tactile input device, other tactile input devices may also be used to provide input to the computer system, such as a touchpad, trackball, pointing stick, or touchscreen. Each of these tactile input devices require some kind of physical interaction, e.g., that the user physically touch the input device. However, there are scenarios where physical interaction with the input device may be difficult or impossible, such as an industrial environment, where workers use protective equipment that may hinder the physical interaction with a computer system or, in a healthcare facility, where medical doctors and assistants cannot touch unsterilized objects.

DETAILED DESCRIPTION

Example methods and systems are directed to creating a virtual environment, in accordance with an example embodiment for use with a touchless input device. Several commercially available touchless input devices are available on the market, such as the Microsoft® Kinect® camera and the Leap Motion® device. The Microsoft® Kinect® camera is an input device that includes a red-green-blue (RGB) camera, a depth sensor, and a multi-array microphone, which allows facial recognition and voice recognition to be performed by a connected system. The Leap Motion® device is a peripheral device that generates a three-dimensional pattern of dots of infrared (IR) light, which it uses to determine the locations of objects in space. Other examples of touchless input devices include webcams, motion sensors, microphones, and other such touchless input devices.

However, devices like the Microsoft® Kinect® camera and the Leap Motion® device are typically used in gesture recognition. More specifically, when the touchless input devices are configured to capture objects (e.g., hands, fingers, faces, etc.), the position of the objects captured over time may be used to identify one or more gestures, where each gesture is associated with an action. For instance, a first gesture, e.g., the shake of a hand to say goodbye, may be interpreted as closing a current process in execution. The touchless interaction through gestures is widespread to the point where different software development kits (SDK) for input devices provide a number of gestures natively.

However, there are several technical problems to gesture-based interaction. First, the recognition process is not scalable. In other words, as the number of acceptable gestures (e.g., recognized gestures) increase, the process of discerning between gestures becomes more error prone (one gesture may be interpreted as another gesture). In addition, a regular GUI cannot be used as a visual feedback for acceptable gesture commands, because the user is required to know the gesture beforehand or the regular GUI has to be specifically programmed to recognize a given gesture.

In one embodiment, this disclosure provides for a method that may include capturing, with an input device, a plurality of electronic images corresponding to a plurality of physical objects detected within a physical three-dimensional region and establishing, with one or more processors, a virtual three-dimensional region mapped to the physical three-dimensional region, with the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the plurality of electronic images. The method may also include selecting, with the one or more processors, a virtual object from the plurality of virtual objects as one or more commanding objects, with the one or more commanding objects indicating a command of a GUI to be performed based on a position of the one or more commanding objects, and performing the command of the GUI based on the position of the one or more commanding objects.

In another embodiment, this disclosure provides a system that may include an input device configured to capture a plurality of electronic images corresponding to a plurality of physical objects detectable within a physical three-dimensional region; a non-transitory, computer-readable medium storing computer-executable instruction; and one or more processors in communication with the input device and the non-transitory, computer-readable medium that, having executed the computer-executable instructions, are configured to establish a virtual three-dimensional region mapped to the physical three-dimensional region, with the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the plurality of electronic images. The one or more processors may also be configured to select a virtual object from the plurality of virtual objects as one or more commanding objects, with the one or more commanding objects indicating a command of a GUI to be performed based on a position of the one or more commanding objects, and perform the command of the GUI based on the position of the one or more commanding objects.

This disclosure further provides for a non-transitory, computer-readable medium. In one embodiment, the non-transitory, computer-readable medium may include computer-executable instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method. The method may include receiving a plurality of electronic images captured by an input device, with the plurality of electronic images corresponding to a plurality of physical objects detected within a physical three-dimensional region; establishing a virtual three-dimensional region mapped to the physical three-dimensional region, with the virtual three-dimensional region defining a space where a plurality of virtual objects are instantiated based on the plurality of electronic images; selecting a virtual object from the plurality of virtual objects as one or more commanding objects, with the one or more commanding objects indicating a command of a GUI to be performed based on a position of the one or more commanding objects; and performing the command of the GUI based on the position of the one or more commanding objects.

In an example embodiment, a computer system provides a visual feedback for the user to aid the user in identifying where a recognized object is located relative to a GUI. The visual feedback may be presented in a manner familiar to the user, such as by displaying a graphic in two-dimensional space representative of the three-dimensional object, which makes the touchless interaction a natural extension to familiar tactile interactions. In an example embodiment, the systems and methods may not rely on gestures, which may allow an increasing the range of possible input commands and facilitate the reuse of any already-established GUIs.

FIG. 1is a block diagram of a system100for creating a touchless interaction environment, in accordance with an example embodiment. The system100may include a touchless input device104, a computer system106communicatively coupled to the touchless input device104, and a display108communicatively coupled to the computer system106. A user102may use the touchless input device104to interact with the computer system106.

The touchless input device104may be any one or combination of touchless input devices. As examples, the touchless input device104may be a webcam, microphone, motion capture device, or other such touchless input device. In one embodiment, the touchless input device104may be a Leap Motion® device. In another embodiment, the touchless input device104may be a Microsoft® Kinect® camera. The touchless input device104may be configured to detect one or more objects in a physical, three-dimensional region and provide data to the computer system106relating to the detected objects. As examples, the data may include positional data (e.g., one or more three-dimensional coordinates where an object was detected), image data (e.g., one or more electronic images representing one or more pictures of the detected objects), motion data (e.g., velocity, acceleration, orientation, and so forth of one or more detected objects), or combinations of such data.

The computer system106may be configured to translate the data obtained by the touchless input device104. In particular, the computer system106may translate the data from the touchless input device104into positional data relative to an image displayed by the display108. For example, the display108may display an image having a known height and width (e.g., a GUI having a known resolution), and the computer system106may be configured to translate the three-dimensional position of an object detected by the touchless input device104into a two-dimensional position within the image displayed by the display108. The computer system106may further instantiate a virtual object to represent the physical object, and may display the virtual object with the translated two-dimensional position. As the translation of the three-dimensional position to the two-dimensional position may result in a loss of a positional component of the data relating to the three-dimensional position of the physical object (e.g., a distance of the physical object from a known origin point), the computer system106may modify a characteristic of the virtual object, such as its size, color, height, width, radius, or other such characteristic to denote this missing dimensional component. For example, and in one embodiment, objects having a larger distance component (e.g., are further away from a known origin point) in the physical, three-dimensional region may appear as smaller virtual objects on the display108, whereas objects having a smaller distance component (e.g., are closer to a known origin point) may appear as larger virtual objects on the display108. In this embodiment, and as discussed below with reference toFIG. 6andFIG. 7, the size of the virtual object may indicate whether a virtual object is intended to provide a command understood by a GUI.

Although shown as a single box inFIG. 1, the computer system106may be an individual system or a cluster of systems, and may be configured to perform activities related to receiving positional data and translating such positional data, such as storing positional data, processing the positional data according to scripts and software applications, controlling one or more applications via an operating system, receiving commands for the operating system via a GUI, and other such activities. The system106may include one or more non-transitory, computer-readable storage devices112, such as a hard drive, optical drive, magnetic tape drive, or other such non-transitory, computer-readable media, and may further include one or more processors110.

The one or more processors110may be any type of commercially available processors, such as processors available from the Intel Corporation, Advanced Micro Devices, Texas Instruments, or other such processors. Furthermore, the one or more processors110may be of any combination of processors, such as processors arranged to perform distributed computing via the computer system106.

FIG. 2is a block diagram illustrating a touchless input device104, in accordance with an example embodiment, interacting with a computer system106. The touchless input device104may be communicatively coupled to the computer system106and communicate via one or more communication interfaces. The computer system106may communicate with the display108via one or more display interfaces212. In one embodiment, the touchless input device104may include one or more image sensor(s)202, one or more light source(s)204, one or more processors206, an electronic data storage device120, and a communication interface208.

The one or more image sensor(s)202may be configured to detect one or more objects within a three-dimensional region. In one embodiment, the one or more image sensor(s)202may be monochromatic infrared cameras. In another, the one or more image sensor(s)202may be a depth sensor implemented as an infrared laser projector combined with an active-pixel sensor, such as a CMOS sensor. In yet a further embodiment, the one or more image sensor(s)202may be an RGB camera. Combinations and variations of the foregoing sensors are also possible. The one or more image sensor(s)202may provide positional information214that includes, but is not limited to, two-dimensional coordinates of one or more detected objects, three-dimensional coordinates of one or more detected objects, infrared images of detected objects, colorized images of detected objects, and other such information. Furthermore, the touchless input device104may be configured to interpret the obtained positional information214from the image sensor(s)202so as to determine relative locations of objects in the received positional data. Alternatively or additionally, the touchless input device104may send the received positional information214to the computer system106, which may then determine the locations of objects detected by the one or more image sensor(s)202.

The touchless input device104may also include one or more light source(s)204for illuminating a three-dimensional region in which one or more objects may be detected. In one embodiment, the one or more light source(s)204include infrared light emitting diodes (LEDs). In this embodiment, each infrared LED may emit a single beam of light, which may be reflected by an object in the three-dimensional region. The reflection of the beam of light may then be detected by one or more of the image sensor(s)202. Alternatively or additionally, the one or more light source(s)204may include full spectrum (e.g., white light) LEDs, ultraviolet LEDs, incandescent or compact fluorescent lights, or any other type of light source. In this manner, the light source(s)204illuminate a three-dimensional region where an object may be detected.

Referring toFIG. 5, there is shown an example three-dimensional region504where the touchless input device104ofFIG. 1may detect one or more objects506a-506e. In one embodiment, the touchless input device104may be configured to establish the three-dimensional region504using one or more of the imaging sensor(s)202and light source(s)204, where objects506a-506einside the three-dimensional region504are captured by the imaging sensor(s)202and objects outside the three-dimensional region202are not. Alternatively or additionally, one or more of the imaging sensor(s)202may capture a single object (e.g., a user's hand) and then the touchless input device104and/or computer system106may computationally segment the user's hand into the objects506a-506e.

The three-dimensional region504may be established as a viewing frustum502for one or more of the imaging sensor(s)202. In one embodiment, the shape of the viewing frustum502may be a rectangular pyramid, where a top portion of the viewing frustum502is sliced by a plane extending through the viewing frustum502and parallel to a base plane of the viewing frustum502. The base of this rectangular pyramid may be the maximum distance from the imaging sensor(s)202to detect one or more objects, and the top may be the minimum distance. The viewing frustum502may delimit where the user may interact with the computer system106using his or her hands, fingers, pointing devices, and other such objects. The mathematical formula that defines the viewing frustum502of the touchless input device104may depend on a distance to an origin point and, as one example, a point in the center of a region of interaction. As discussed below with reference toFIG. 6, the viewing frustum502combined with a virtual frustum (e.g., virtual frustum604) may create a virtual environment for visual feedback.

Referring back toFIG. 2, the touchless input device104may also include a communication interface208for sending positional information214to the computer system106. The communication interface208may include any combination of wired and/or wireless interfaces. For example, the communication interface208may include a Universal Serial Bus (USB) interface. Alternatively, or additionally, the communication interface208may include an Ethernet interface. In yet another embodiment, the communication interface208may include a wireless interface for communicating wirelessly with the computer system106using one or more wireless communication protocols, such as 802.11b/g/n, Bluetooth®, infrared (e.g., the Infrared Data Association set of protocols), or any other wireless communication protocol.

To receive the positional information214from the touchless input device104, the computer system106may also include a communication interface210. The communication interface210may include any combination of wired and/or wireless interfaces. For example, the communication interface210may include USB. Alternatively, or additionally, the communication interface210may include an Ethernet interface. In yet another embodiment, the communication interface210may include a wireless interface for communicating wirelessly with the touchless input device104using one or more wireless communication protocols, such as 802.11b/g/n, Bluetooth®, infrared (e.g., the Infrared Data Association set of protocols), or any other wireless communication protocol.

The computer system106may store the positional information214received from the touchless input device104in one or more electronic data storages112via one or more processors110. The computer system106may then transform the received positional information214into two-dimensional position coordinates that correspond to image pixels of an image displayed by the display108. Furthermore, the computer system106may instantiate one or more virtual objects corresponding to the objects detected by the touchless input device104. The virtual objects may then be displayed in the image displayed by the display108at the two-dimensional position coordinates determined by the computer system106. Although two-dimensional position coordinates are discussed above, the computer system106may determine any type of positional coordinates (e.g., three-dimensional position coordinates) based on the positional information214received from the touchless input device104. As discussed below, the computer system106may be configured with one or more transformation matrices to transform the positional information214from one coordinate system (e.g., the coordinate system used by the touchless input device104) to any other type of coordinate system (e.g., the coordinate system of the image displayed by the display108).

To display the virtual objects on the display108, the computer system106may be configured with a display interface212for communicating with the display108. The display interface212may include one or more wired and/or wireless display interfaces. For example, the display interface212may include a High-Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI) interface, a DisplayPort interface, Intel® WiDi, and other such display interfaces.

FIG. 3illustrates a computer system106, in accordance with an example embodiment. In one embodiment, the electronic data storage112may be configured with one or more applications302and various types of data304. The one or more applications302may include a GUI306, a transformation engine308, a command translation engine310, and an object instantiation engine312. The various types of data may include physical coordinates of detected objects314, one or more transformation matrices316, virtual coordinates of virtual objects318, and one or more instantiated virtual objects320based on the detected objects. The computer system106may invoke one or more of the applications302to transform the positional information of detected objects based on the data304into virtual objects and virtual positions, and to display such virtual objects on the display108via the display interface212.

The GUI306may be configured to accept commands for controlling the computer system106. As discussed previously, a GUI306may accept commands via a tactile input device (e.g., a mouse, a keyboard, etc.), and modifications to the GUI306may be required to accept input via a touchless input device. Modifications may include translating gestures into commands understandable by the GUI306. However, as also previously discussed, such gestures may be difficult or time consuming to learn, and it would be advantageous for a user to be able to interact with the GUI306without having to learn such gestures. Thus, in one embodiment, the GUI306is unmodified in the sense that a user is not required to learn gestures to provide commands to the computer system106.

FIG. 4illustrates a GUI402, in accordance with an example embodiment. The GUI402may include various graphical controls404-414for providing commands to the computer system106. As examples, and without limitation, these controls may include one or more graphical controls404, one or more selectable check boxes412, one or more selectable option boxes414, one or more selectable drop-down menus410, one or more selectable and controllable scroll bars406, and various selectable and controllable horizontal and/or vertical sliders408. Each of the graphical controls404-414may provide a different command to the GUI402and may be displayed individually, concurrently, or in combinations thereof.

To accept commands from the touchless input device104that control the GUI402, the computer system106may establish a virtual environment in which virtual objects are instantiated based on physical objects (e.g., objects506a-506e) detected by the touchless input device104.FIG. 6illustrates an example instantiation of virtual objects608a-608ebased on the user interacting with the touchless input device104. In one embodiment, the computer system106may replicate the three-dimensional region504of the touchless input device104as a virtual three-dimensional region602.

In one embodiment, objects appearing in the physical, three-dimensional region504are instantiated as virtual objects608a-608einside the virtual, three-dimensional region602. The virtual objects608a-608emay be instantiated with any shape, such as rectangles, spheres, triangles, or other such shapes. Furthermore, each object detected by the touchless input device104may be instantiated as a unique virtual object, such that each virtual object has different characteristics (e.g., shape, color, size, orientation, etc.). Alternatively, or in addition, the virtual objects608a-608emay each represent a portion of a larger object detected within the physical three-dimensional region504. For example, where a user's hand and/or finger is detected within the physical, three-dimensional region504, a virtual object may be instantiated representing a portion of the user's hand or finger, such as the tip of the finger. In this embodiment, the fingertip may be represented as a sphere displayed on the display108.

The computer system106may instantiate the virtual objects608a-608eby establishing a virtual camera606that creates a second viewing frustum604which mirrors (or is substantially similar to) the viewing frustum502of the one or more image sensor(s)202of the touchless input device104. In one embodiment, the computer system106may emulate the virtual camera606, which may be responsible for several tasks: 1) to define the boundaries of the virtual world where the virtual objects exist; and 2) to define a perspective projection to be performed on the virtual objects608a-608eonto a projection plane. The projection plane may be established at several locations on the second viewing frustum604, such as the bottom or the top of the frustum pyramid. Alternatively, or in addition, the projection plane may be defined to align with the displayed GUI402. In other words, the projection plane may have the same, or substantially the same, height and width as the image of the GUI402.

The computer system106may establish the virtual camera606at one or more locations along the second viewing frustum604, such as in front of or behind the projection plane. As shown inFIG. 6, the computer system106has established the virtual camera606behind the projection plane. Moreover, the virtual camera606and the image sensor(s)202of the touchless input device104may be located at different positions and, further still, may have different orientations. As the virtual camera606and the image sensors(s)202may have different positions, orientations, and viewing frustum dimensions, the computer system106may employ one or more transformation matrices316that map the first viewing frustum502to the second viewing frustum604and one or more transformation matrices316that map the second viewing frustum604to the first viewing frustum502. The technical effect of mapping the viewing frustums502,604is that the computer system106can translate three-dimensional coordinates of detected objects in the physical, three-dimensional region504to virtual, three-dimensional coordinates for virtual objects instantiated in the virtual, three-dimensional region602. To accomplish this mapping, the computer system106may be configured with one or more transformation engines308to apply the one or more transformation matrices316to the positional information of the detected objects. In addition, the second viewing frustum604may be calibrated at will for a better user experience.

Below is an example of converting physical coordinates of detected objects to virtual coordinates of virtual objects. In this example, the touchless input device104may be the Leap Motion® device and the display108may be the screen of a laptop or other computing device having known dimensions (e.g., height and width). The viewing frustum (e.g., the viewing frustum502) created by the Leap Motion® device may have a coordinate system with an origin established seven centimeters (7 cm.) away from it, a y-axis pointing away from the Leap Motion® device, and an x-axis and a z-axis spanning the device plane. It should be understood that the foregoing arrangement of axes and origin point may be specific to the Leap Motion® device and that other touchless input devices, such as the Microsoft® Kinect® camera, may establish a different arrangement of origin point and axes. However, such variations and arrangements are contemplated as falling within the breadth of this disclosure.

The display108may also have a virtual viewing frustum (e.g., viewing frustum604). The virtual viewing frustum defined for the display108may have a coordinate system with an origin point located approximately at the same distance as the origin point for the coordinate system of the touchless input device104. However, other orientations or arrangements of axes may differ. For example, the z-axis of the coordinate system for the virtual viewing frustum may point away from the display108, with the x-axis and y-axes spanning the height and width, respectively, of the display108.

The computer system106may be configured with a first transformation matrix, M2←1, that is used by one or more transformation engines308, to convert positions from a coordinate system of a first frustum (e.g., frustum1) to a coordinate system of a second frustum (e.g., frustum2). In one embodiment, the transformation matrix M2←1may be defined as:

In one embodiment, the touchless input device104may provide electronic images of the three-dimensional region504as positional information, and the computer system106may leverage one or more image recognition algorithms to determine the location of one or more objects within the provided electronic images. Alternatively, or additionally, the touchless input device104may capture the electronic images of the three-dimensional region504, and provide physical coordinates of detected objects within the electronic images to the computer system106.

The computer system106may further process the physical coordinates of the detected objects. In particular, the computer system106may normalize the physical coordinates. In one example, the physical coordinates are provided in a predetermined scale (e.g., millimeters, centimeters, etc.) and the computer system106normalizes these measurements to another scale (e.g., 0-1, 0-10, etc.), where the minimum value of the scale (e.g., 0) is at one side of the region of interaction and the maximum value of the scale (e.g., 1) is at the opposite side of the region of interaction. The computer system106may then transform the normalized coordinates by applying the transformation matrix M2←1. The result of this application may be one or more coordinates within the virtual three-dimensional region602where one or more virtual objects are to be instantiated.

Having obtained the virtual coordinates for the virtual, three-dimensional region602, the computer system106may further transform the virtual coordinates into coordinates mapped to the dimensions of the display108. In particular, the computer system106may be configured with a second transformation matrix, Mw×h, which one or more transformation engines308may apply to convert the virtual coordinates to the display coordinates, where the display108has a first set of pixels in width (w) and a second set of pixels in height (h). In one embodiment, the transformation matrix Mw×hmay be defined as:

These sequences of transformations allow the computer system106to display physical objects detected in the three-dimensional region504as virtual objects instantiated a two-dimensional plane (namely, the plane of the image where the GUI402is displayed). Other types or combinations of projections may also be implemented.

After obtaining coordinates where the virtual objects are to be instantiated, the computer system106may then display the instantiated virtual objects with characteristics relative to their locations in the physical world. In one embodiment, the computer system106may display the virtual objects as images appearing in a projection image overlaid on the image of the GUI402. Where no object is detected, the projection image may be transparent; otherwise, the projection image may contain the shape and the color of the virtual object. This projection image may be combined with the image of the GUI402to generate the visual feedback of the user interacting with the GUI402.

FIG. 7illustrates an example composite image702where a projection image704having the instantiated virtual objects608a-608eofFIG. 6is combined with an image of the GUI402. Virtual objects608a-608emay be instantiated as spheres, which may then be projected onto the projection image overlaid on the GUI402. The result of projecting the virtual objects608a-608einto the projection image704is that the virtual objects608a-608emay appear as circles displayed on a transparent image combined with the GUI402(e.g., as the composite image702).

As discussed previously, a characteristic of the virtual objects608a-608emay be modified to indicate a distance component which may be lost in the projection of the virtual objects from the three-dimensional realm to the two-dimensional. In particular, a size of the virtual objects608a-608emay be modified to reflect distance. However, other characteristics may also be modified, such as color, shape, orientation, or other characteristic. Furthermore, the modifiable characteristic may be selectable and/or configurable by the user.

Where the size of the virtual object is used to indicate distance, virtual objects close to the virtual camera may appear larger while virtual objects further from the virtual camera may appear smaller. This effect helps the user to discern which physical objects are being used to provide commands to the GUI (e.g., is a commanding object). For instance, if one finger is over a particular GUI component, such as a button, that component may be highlighted to indicate the possibility of activating it. If the same finger moves closer to the component, it may be activated.

In one embodiment, the computer system106may be calibrated and/or configured by the user with varying distances to discern between whether a GUI component is being selected and whether that GUI component is being activated. For example, the computer system106may be configured with a command translation engine310to determine whether the user has intended a command to be performed given the location of a physical object and whether that physical object has moved closer or farther from the given location. In one embodiment, the command translation engine310may be configured with a threshold command distance, which may be a distance from the virtual camera that indicates whether an object is intended to provide a command. Thus, objects in or past the threshold command distance may be considered commanding objects. Furthermore, in some instances, the command translation engine310may be configured such that moving a physical object closer to the virtual camera may activate the GUI component. In other instances, the command translation engine310may be configured such that moving the physical object further from the virtual camera may activate the GUI component. In either instance, the user may configure the command translation engine310and/or the computer system106to indicate whether movements closer or farther from the virtual camera are intended to provide a command. Moreover, as the movements are intuitive, the user does not need to learn any new or extra gestures to provide commands to the GUI402.

FIGS. 8A-8Cillustrate a method802, in accordance with an example embodiment, of creating a touchless interaction environment. The method802may be implemented on the touchless input device104and, accordingly, is merely described by way of reference thereto. In an example embodiment, initially, the touchless input device104may be configured to establish a three-dimensional region504based on a viewing frustum502where one or more objects are to be detected (Operation804). In addition, various distances, orientations, positions, and other such characteristics may be established for determining whether an object detected by the touchless input device104is a commanding object and whether that commanding object has provided a command to a GUI (e.g., GUI402) (Operation806). Moreover, a virtual, viewing frustum (e.g., frustum604) may be established based on the viewing frustum502of the touchless input device104to create a virtual, three-dimensional region where one or more virtual objects are to be instantiated.

The touchless input device104may then capture one or more electronic images of the three-dimensional region504(Operation808). The electronic images may be stored by the touchless input device104as positional information214. In one embodiment, the touchless input device104may determine whether objects are present in the captured electronic images (Operation810) and, if so, determine corresponding three-dimensional coordinates for the detected objects (Operation812). Additionally, or alternatively, the touchless input device104may communicate the electronic images to the computer system106, which may then perform object detection and coordinate determination on the electronic images.

The computer system106may store the three-dimensional coordinates of the detected objects314as physical coordinates. Thereafter, the computer system106may apply a first transformation matrix via one or more transformation engines308to obtain virtual, three-dimensional coordinates corresponding to the physical, three-dimensional coordinates of the detected objects (Operation814). Accordingly, the computer system106may then instantiate one or more virtual objects based on the detected objects, where the one or more virtual objects are located at the determined virtual, three-dimensional coordinates (Operation816). Alternatively, or in addition, the computer system106may instantiate one or more virtual GUI objects corresponding to GUI components, such that the GUI object provide additional feedback (e.g., visual feedback, audio feedback, etc.) to the user. For example, the computer system106may instantiate virtual GUI objects as rectangular boxes around the standard GUI components, such that interactions with the virtual GUI objects cause a corresponding change in these objects (e.g., the virtual GUI objects change color).

The computer system106may then proceed to displaying the virtual objects. In particular, the computer system106may apply another transformation matrix to the virtual, three-dimensional coordinates to obtain two-dimensional coordinates corresponding to the dimensions of a display (Operation818). As one or more positional components may be lost in the transformation (e.g., a distance component), the computer system106may modify one or more characteristics of the virtual objects to indicate relative differences in the missing one or more positional components (Operation820). For example, the computer system106may modify the size of the virtual objects to indicate the distance of each of the virtual objects from a virtual camera.

The computer system106may then display the modified virtual objects by overlaying or superimposing a projection image atop an image of the GUI, where the projection image includes images of the modified virtual objects (Operation822). The projection image and the image of the GUI may be displayed as a composite image on the display.

Thereafter, the computer system106may then determine whether any one of the virtual objects is a commanding object (Operation824). In one embodiment, the computer system106may perform this determination by comparing the distance of a virtual object to the virtual camera with a commanding object distance threshold. Where more than one virtual object has crossed the commanding object distance threshold, the virtual object with the least (e.g., closest) or greatest (e.g., furthest) distance may be selected as the commanding virtual object (Operation826).

The computer system106may then compare the two-dimensional location of the commanding virtual object with the location of one or more GUI components (Operation828). Where the commanding virtual object is at the same (or about the same) location as a GUI component, the computer system106may modify the appearance of the graphical user interface, such as by changing its color, hue, brightness, font, or other such characteristic (Operation830). Should the computer system106then determine that the commanding virtual object has moved closer to (or farther away from) the GUI component (Operation832), the computer system106may determine that the user has intended that the computer system106perform the command associated with the GUI component (Operation834).

FIG. 9is a block diagram illustrating components of a machine900, in accordance with an example embodiment, configured to read instructions from a machine-readable medium and perform any one or more of the disclosed methodologies. Specifically,FIG. 9shows a diagrammatic representation of the machine900in the example form of a computer system and within which instructions924(e.g., software) for causing the machine900to perform any one or more of the methodologies discussed herein may be executed. In alternative examples, the machine900operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine900may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine900may be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions924, sequentially or otherwise, that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions924to perform any one or more of the methodologies discussed herein.

The machine900includes a processor902(e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), or any suitable combination thereof), a main memory904, and a static memory906, which are configured to communicate with each other via a bus908. The machine900may further include a video display910(e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The machine900may also include an alphanumeric input device912(e.g., a keyboard), a cursor control device914(e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit916, a signal generation device918(e.g., a speaker), and a network interface device920.

The storage unit916includes a machine-readable medium922on which is stored the instructions924(e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions924may also reside, completely or at least partially, within the main memory904, within the processor902(e.g., within the processor's cache memory), or both, during execution thereof by the machine900. Accordingly, the main memory904and the processor902may be considered as machine-readable media. The instructions924may be transmitted or received over a network926via the network interface device920.

Although the foregoing discussion has assumed that the same virtual object is the commanding object throughout the various operations, it is contemplated that the commanding object may change at any point depending on the locations of the objects detected by the touchless input device104. Thus, a first virtual object being designated as the commanding object at Block826may not necessarily be the commanding object by Block832. Thus, the determination of a commanding object is fluid and is subject to change according to the movements of the objects detected by the touchless input device104.

In this manner, the disclosed systems and methods provide a solution whereby a user may interact with a GUI without having to learn any additional gestures or movements. In effect, the user interacts with the GUI intuitively, and the disclosed systems and methods enable a user to quickly configure a touchless input device to work with a computer system and its existing GUI. Thus, the disclosed systems and methods provide a technical advantage over other input devices because they do not require the user to engage a tactile input device nor does it require modifications to the GUI to understand gestures or other complex motions.