Patent Description:
An effortless and subtle way to indicate a user's context is to take advantage of gaze tracking data to infer a user's current reference frame. Several problems arise with this approach though, as eye tracking and additional commands are asynchronous (i.e., the eye gaze is usually preceding manual inputs and may have moved on to new targets upon finishing recognition of the manual input). In addition, due to technological constraints of the tracking system as well as physiological constraints of the human visual system, the computed gaze signal may be jittery and show offsets compared to the actual eye gaze. This increases the problem of reliably referring to small and closely positioned targets. Thus, an overall problem arises about how such multimodal inputs can be appropriately combined.

<CIT> discloses a method for presenting a mixed reality environment via a head-mounted display device.

<CIT> discloses an augmented-reality device which includes a gaze detector, a camera, and a communication interface.

<CIT> discloses a method for processing an input in a mobile terminal that recognizes inputs such as a visual input and/or voice input. When detecting that one eye is closed for a threshold time or longer, a controller selects an object in a selection region corresponding to a position viewed by the user. Next, when the user moves the other open eye, the object moves to a position corresponding to the eye movement.

<CIT> discloses a method of processing user input in a three-dimensional coordinate system, comprising receiving a user input of an origin reset for the three-dimensional coordinate system, responsive to receiving the user input of the origin reset, resetting an origin of the three-dimensional coordinate system, receiving three-dimensional user input in the three-dimensional coordinate system, while the origin remains set, measuring at least one three-dimensional displacement of the three-dimensional user input relative to the origin, and causing movement of a user interface element displayed in a user interface, the movement based on the at least one measured three-dimensional displacement.

<CIT> discloses a system and method for building and experiencing three-dimensional virtual objects from within a virtual environment in which they will be viewed upon completion.

<CIT> discloses a method to identify a targeted object based on eye tracking and gesture recognition.

In one aspect, the present invention provides a method according to claim <NUM>. In another aspect, the present invention provides a system according to claim <NUM>. Certain more specific aspects of the invention are set out in the dependent claims.

Described is a method for improving user interaction with a virtual environment includes measuring a first position of the user's gaze relative to a virtual element, selecting the virtual element in the virtual environment at an origin when the user's gaze overlaps the virtual element, measuring a second position of a user's gaze relative to the virtual element, presenting a visual placeholder at a second position of the user's gaze when the second position of the user's gaze is beyond a threshold distance from the origin, and moving the visual placeholder relative to a destination using a secondary input device.

Also described is a method for improving user interaction with a virtual environment includes measuring a first position of a user's gaze relative to a virtual element, selecting the virtual element in the virtual environment at an origin when the user's gaze overlaps the virtual element, receiving an input from a secondary input device, changing at least one dimension of the virtual element in response to the input from the secondary input device, and finalizing the at least one dimension upon deselecting the virtual element.

Also described is a system for presenting visual information to a user includes a head-mounted display. The head-mounted display includes a display device, a gaze-tracking device, a processor, and a hardware storage device in data communication. The hardware storage device has instructions thereon that, when executed by the processor, cause the processor to measure a first position of a user's gaze relative to a virtual element, select the virtual element in the virtual environment at an origin when the user's gaze overlaps the virtual element, measure a second position of a user's gaze relative to the virtual element, present a visual placeholder at a destination based on the position of the user's gaze when the second position of the user's gaze is beyond a threshold distance from the origin, and move the visual placeholder relative to the second position of the user's gaze with a secondary input device.

Features of the present disclosure will become more fully apparent from the following description and appended claims or may be learned by the practice of the disclosure as set forth hereinafter.

In order to describe the manner in which the above-recited and other features of the disclosure can be obtained, a more particular description will be rendered by reference to specific examples which are illustrated in the appended drawings. For better understanding, the like elements have been designated by like reference numbers throughout the various accompanying figures. While some of the drawings may be schematic or exaggerated representations of concepts, at least some of the drawings may be drawn to scale. Understanding that the drawings depict some examples, the examples will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:.

This disclosure generally relates to devices, systems, and methods for visual user interaction with virtual environments. More specifically, the present disclosure relates to improving interaction with virtual elements using gaze-based selection and manipulation. Visual information may be provided to a user by a near-eye display. A near-eye display may be any display that is positioned near a user's eye, either to supplement a user's view of their surroundings, such as augmented or mixed reality devices, or to replace the user's view of their surroundings, such as virtual reality devices. An augmented reality or mixed reality device may be a head-mounted display (HMD) that presents visual information to a user overlaid on the user's view of their surroundings. For example, the visual information from the HMD may be combined with ambient or environment light to overlay visual information, such as text or images, on a user's surroundings.

The user's field of view may be at least partially encompassed by a waveguide through which the user views their surroundings. The waveguide may direct display light from a display device to the user's field of view. The waveguide may guide the display light before out-coupling the light. Upon out-coupling the light, the waveguide may combine the visual information of the display light with ambient light from the user's surroundings to deliver the visual information to the user. Overlaying the visual information from the HMD on the user's surroundings may require precise generation and positioning of the visual information relative to the user's eyes.

Visual information including virtual elements may be positioned in the user's field of view on the waveguide or other near-eye display. A gaze-tracking device of the HMD may image at least a portion of the user's eye (such as the pupil, the iris, the sclera) and identify a direction or location of the user's gaze. The direction or location of the user's gaze may then be extrapolated to a position on the near-eye display. A selection cursor may be associated with the gaze location to allow the user to highlight or select a virtual element by looking at the virtual element. In other implementations, a gaze-tracking device may include a gyroscope, an accelerometer, a plurality of sensors to triangulate position, or other devices that allow for the measurement of the orientation and/or position of the HMD relative to the virtual environment. For example, the user's "gaze" may be a ray cast from the HMD forward from the HMD to approximate the user's gaze by approximating the user's head position and orientation as their gaze direction. In some examples, such a head-tracking "gaze" may be simpler than an eye-tracking gaze, as the user remains free to glance around in their field of view without inadvertently moving a gaze position cursor. In other examples, an eye-tracking gaze may be more intuitive as a user will naturally look at whatever object is of interest to the user during interactions with the virtual environment.

In some implementations, gaze-informed movement and manipulation of virtual objects, either by eye-tracking gaze or by head-tracking gaze, may allow for rapid movement of virtual objects within a virtual or shared environment. In other implementations, gaze-informed movement and manipulation of virtual object may be employed in combination with manual, voice, peripheral, or other inputs to provide different scales of movement and manipulation for both speed and precision.

<FIG> is a perspective view of a user <NUM> wearing an HMD <NUM>. The HMD <NUM> may have a housing <NUM> that contains one or more processors, storage devices, power supplies, audio devices, display devices, cameras, communication devices, or combinations thereof, that receive, collect, store, process, or calculate information that is provided to the user. For example, a display device <NUM> may be positioned optically adjacent a waveguide(s) or other near eye display <NUM> to provide visual information to the near eye display <NUM>, which may, in turn, be presented in the user's field of view by the near eye display <NUM>.

The HMD <NUM> may have a near eye display <NUM> positioned near the user <NUM> to direct visual information to the user <NUM>. The HMD <NUM> may include a single near eye display <NUM>, a separate near eye display <NUM> for each of the user's eyes (i.e., two near eye displays <NUM>), or more than two near eye displays <NUM> to provide visual information over a larger field of view.

The HMD <NUM> may include one or more cameras <NUM> that may image the user's physical environment. For example, the camera(s) <NUM> may be visible light camera(s) <NUM> that may image the surrounding environment. A processor may perform image recognition routines on the visible light image to detect and recognize elements in the surrounding environment. In other examples, the camera(s) <NUM> may be depth sensing camera(s) that may create a depth image of the surrounding environment. For example, the camera <NUM> may be a time-of-flight camera, a structured light camera, stereo cameras, or other cameras that may use visible, infrared, ultraviolet, or other wavelengths of light to collect three-dimensional information about the surrounding environment. In at least one example, the camera(s) <NUM> may be gesture recognition cameras that allow the HMD <NUM> to recognize and interpret hand gestures performed by the user <NUM> in front of the HMD <NUM>.

The HMD <NUM> may further include a gaze-tracking device <NUM> positioned in the HMD <NUM> to track a direction of the user's gaze. The gaze-tracking device <NUM> may include a camera or a plurality of cameras to image the user's eyes. In other words, the gaze-tracking device <NUM> may image the user's pupil, iris, sclera, other portions of the user's eye, or combinations thereof to calculate the direction the user is looking. In some implementations, the gaze-tracking device <NUM> may measure and/or calculate the x- and y-components of the user's gaze. In other implementations, the gaze-tracking device <NUM> may include a gyroscope, an accelerometer, a plurality of sensors to triangulate position, or other devices that allow for the measurement of the orientation and/or position of the HMD relative to the virtual environment. For example, the user's "gaze" may be a ray cast from the HMD forward from the HMD to approximate the user's gaze by approximating the user's head position and orientation as their gaze direction.

<FIG> is a schematic representation of the HMD <NUM>. The display device <NUM> in communication with the near eye display <NUM> may be in data communication with a processor <NUM>. Similarly, the camera <NUM> and gaze-tracking device <NUM> may be in data communication with the processor <NUM>. The processor <NUM> may further be in data communication with a storage device <NUM>. The storage device <NUM> may be a hardware storage device, such as a platen-based storage device, a solid-state storage device, or other non-transitory or long-term storage device. The storage device <NUM> having instructions stored thereon to perform one or more methods or portions of a method described herein.

<FIG> is a flowchart illustrating a method of moving a virtual element within a virtual environment. The method includes measuring a first position of a user's gaze location at <NUM>. The first position may be measured by using a gaze-tracking device described herein to measure the position of either the user's gaze based on eye-tracking or the user's gaze based on heading tracking.

The first position may include x- and y-coordinates, such as when interacting with a virtual desktop with an HMD. The first position may include x-, y-, and z-coordinates relative to the user, such as when interacting with virtual elements in three-dimensional space. In some implementations, the three-dimensional space may be a virtual environment generated by the HMD or other computing device in communication with the HMD. In other implementations, the three-dimensional space may be a shared environment. For example, a mixed reality HMD may present virtual elements in combination with a surrounding physical environment of the user. In such implementations, the HMD may measure the surrounding physical environment of the user using, for example, the cameras on the HMD or other sensors to impart information of the surrounding physical environment into a virtual environment to create a shared environment. The HMD may then use the shared environment to position a virtual element in a virtual environment to physical element of the surrounding physical environment.

In some implementations in a three-dimensional space, the first position may be measured by casting a ray from the HMD in the direction of a user's gaze detected by the gaze-tracking device, and first position may be the location where the ray interacts with a surface of the virtual environment or the shared environment. For example, the ray may interact with an upper surface of a virtual element, and the first position may be measured as having coordinated on the upper surface of the virtual element. In other examples, the ray may interact with a physical object, such as a surface of a table in front of a user. The first position may be measured as having coordinated on the surface of the physical table.

In other implementations in a three-dimensional space, the first position may be measured using the focal distance of the detected gaze of each of the user's eyes. For example, the detected eye position and related gaze direction of each eye will be different based on the distance of the object at which the user is looking. The different gaze locations of each eye may allow the calculation of a focal depth of the user. Measuring the first location using focal depth of the user's gaze may allow for the first position to be located in space, and not on a (virtual or physical) surface.

The method further includes selecting a virtual element at the first position at <NUM>. The virtual element is selected upon receiving a selection command from an input device. The selection command may then cause the HMD to select the virtual element at the first position. In some implementations, the input device may be a gesture recognition device, such as the camera(s), in data communication with the processor of the HMD. In other implementations, the input device may be a voice recognition device, such as a microphone, in data communication with the processor of the HMD. In yet other implementations, the input device may be a motion controller, such as a six degree-of-freedom (6DOF) controller, in data communication with the processor of the HMD. In yet other implementations, the input device may be an input of a touch-sensing device, trackpad, mouse, keyboard or other conventional human interface device (HID) of a computer.

The selection command and the intended gaze location at the time of the selection command are asynchronous. For example, a user may look at a virtual element that the user intends to select. The user may begin providing a selection input (e.g., a gesture input, a mouse click, a voice input) to provide the selection command and look away to a second position, moving the gaze location, before the selection input completes and the selection command is received. This may be common among even experienced users, as conventional interactions with computer systems allow for independent movement of the user's eyes and a selection cursor.

The selection command selects a virtual element the gaze location was positioned on prior to moving away. For example, measuring the first position of the gaze location may include measuring a series of first locations of the gaze location such that the system may "look back" to where the gaze location was prior to receive the selection command. In some implementations, measuring the first position of the gaze location may include measuring and retaining in memory at least <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more gaze locations. In other implementations, measuring the first position of the gaze location may include measuring and retaining in memory all positions of the gaze location for a buffer time, such as <NUM> milliseconds (ms), <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more time. For example, upon receiving a selection command, the system may select the virtual element the user was looking at <NUM> prior to receiving the selection command. In other examples, upon receiving a selection command, the system may iterate through the gaze locations measured during the buffer time until a gaze location coincides with a virtual element.

The method further includes measuring a second position of the user's gaze at <NUM>. The second position of the user's gaze may be measured similar to the first position. Upon measuring the second position of a user's gaze, the method includes presenting a visual placeholder at a location coinciding with the measured second position of the user's gaze. Various examples of the visual placeholder will be described in relation to <FIG> and <FIG>.

After presenting the visual placeholder at the second location, the method further includes moving the visual placeholder using a secondary input device at <NUM>. In some implementations, the secondary input device may be a gesture recognition device, such as the camera(s), in data communication with the processor of the HMD. In other implementations, the secondary input device may be a voice recognition device, such as a microphone, in data communication with the processor of the HMD. In yet other implementations, the secondary input device may be a motion controller, such as a six degree-of-freedom (6DOF) controller, in data communication with the processor of the HMD. In yet other implementations, the secondary input device may be an input of a touch-sensing device, trackpad, mouse, keyboard or other conventional human interface device (HID) of a computer.

The method may optionally include deselecting the virtual object with a deselection command from the input device and moving the virtual object to the destination location of the visual placeholder upon deselection.

<FIG> is a schematic representation of a virtual environment <NUM> containing a virtual element <NUM> that may be presented to a user on a display, such as the waveguide of the HMD <NUM> of <FIG>, a display of a MR system, a display of a VR system, a display of an AR system, or other near-eye display in communication with a processor and/or a gaze-tracking device. While the present disclosure described interaction with the virtual environment <NUM> through an HMD, it should be understood that in other implementations, the virtual environment <NUM> may be presented on another display, such as a laptop, tablet, desktop, large format, or other display in communication with a processor and/or a gaze-tracking device. For example, the methods and systems described herein may be equally applicable to a user interacting with a large format display on a wall of a conference room. The user may move and/or manipulate virtual elements using a gaze-tracking device and other input devices in communication with the large format display. In other examples a laptop may have a front facing camera that may function as a gaze-tracking device to allow gaze-based movement and/or manipulation of virtual elements by a user interacting with the laptop.

The virtual element <NUM> may be any element of the virtual environment that is selectable in the available software. For example, the virtual element <NUM> may be a window of an application presented in the virtual environment. In other examples, the virtual element <NUM> may be a computer assisted design (CAD) model, a menu, a scroll bar, or another movable element of the virtual environment <NUM> or user interface. In yet other examples, the virtual element <NUM> may be a virtual representation of a physical element of the physical environment around a user.

The user interacts with the virtual element <NUM> and/or the virtual environment <NUM> by positioning their gaze at the virtual element <NUM> or other portion of the virtual environment <NUM>. The gaze-tracking system in data communication with the processor of the HMD measures a gaze location <NUM> at a first location on the virtual element <NUM>.

Referring now to <FIG>, the virtual element <NUM> is selected while the gaze location <NUM> is positioned on the virtual element <NUM> by providing a selection command <NUM> with an input device <NUM>. In the depicted implementation, the selection command <NUM> may be a downward "click" of a user's finger and the input device <NUM> may be gesture recognition device that recognizes the click of the user's finger or other hand gesture. In <FIG>, the gesture recognition device is schematically represented outside of the virtual environment <NUM> by the user's hand and gesture itself. In some implementations, the gesture recognition device may be the camera described in relation to <FIG>. In other implementations, the gesture recognition device may be camera not connected to the housing of the HMD. In yet other implementations, the gesture recognition device may be a glove or other wearable electronic device that senses the movement and position of the hand and parts of the hand.

<FIG> shows the input device <NUM> continuing to hold the selection command by keeping the finger of the hand in the downward position until the virtual element <NUM> is deselected by ceasing the selection command. In other examples, the selection command may toggle the selection of the virtual element <NUM>, and the selection command may be input once with the virtual element <NUM> remaining selected until a subsequent deselection command is received.

The gaze location <NUM> moves away from an origin (i.e., a geometric or volumetric center of the virtual element <NUM>) past a threshold <NUM> to prompt the presentation of a visual placeholder <NUM>. The threshold <NUM> may be an angular displacement relative to the user's viewpoint. For example, a rotation of the gaze location <NUM> relative to the user of more than <NUM>° may prompt the presentation of a visual placeholder <NUM>. In other examples, a rotation of the gaze location <NUM> relative to the user of more than <NUM>° may prompt the presentation of a visual placeholder <NUM>. In yet other examples, a rotation of the gaze location <NUM> relative to the user of more than <NUM>° may prompt the presentation of a visual placeholder <NUM>. In other implementations, the threshold <NUM> may be a translational distance relative the three-dimensional space of the virtual environment <NUM>. For example, translating the virtual element <NUM> meter backward on a tabletop relative to a first position <NUM> may require less angular displacement of the gaze location <NUM> (i.e., movement of the user's eyes or head) than translating the virtual image <NUM> meter to the right of the first position <NUM>. However, the nominal distance moved may be the same.

Therefore, in some instances it may be more efficient for the threshold <NUM> to be relative to the translational distance of the gaze location <NUM> in the virtual environment <NUM>. In some embodiments, the threshold <NUM> may be in a range having an upper value, a lower value, or upper and lower values including any of <NUM> centimeter (cm), <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or any values therebetween. For example, the threshold <NUM> may be greater than <NUM>. In other examples, the threshold <NUM> may be less than <NUM>. In yet other examples, the threshold <NUM> may be between <NUM> and <NUM>. In further examples, the threshold <NUM> may be between <NUM> and <NUM>.

In other implementations, the threshold <NUM> may be relative to the size of the virtual element <NUM> selected. For example, the threshold <NUM> may be a nominal distance that is <NUM> % of the largest dimension of the virtual element <NUM>. In other words, the gaze location <NUM> may exceed the threshold <NUM> when the gaze location <NUM> is "outside" of the virtual element <NUM>. In other examples, the threshold <NUM> may be a nominal distance that is <NUM> % of the largest dimension of the virtual element <NUM>. In yet other examples, threshold <NUM> may be a nominal distance that is <NUM> % of the largest dimension of the virtual element <NUM>.

Once the gaze location is beyond the threshold <NUM>, a visual placeholder <NUM> is presented to the user in the virtual environment <NUM> centered on the gaze location <NUM>. The user may move the gaze location <NUM> to a desired destination, such as the second location <NUM> of the gaze location <NUM> and the visual placeholder follows the gaze location <NUM>. By following the gaze location <NUM>, the visual placeholder <NUM> may allow a user to rapidly and intuitively place the visual placeholder <NUM> at or near a desired destination.

The visual placeholder <NUM> may be a standard placeholder irrespective of the virtual element <NUM>. For example, the visual placeholder <NUM> may be a sphere. In other examples, the visual placeholder <NUM> may be a cube. In yet other examples, the visual placeholder <NUM> may be any three-dimensional solid. In further examples, the visual placeholder <NUM> may be any two-dimensional image. In at least one example, the visual placeholder <NUM> may be an "X" presented on a surface of another virtual or physical element. In other implementations, the visual placeholder <NUM> may be a copy of the virtual element <NUM>. For example, the visual placeholder <NUM> may be a "ghost" or partially transparent copy of the virtual element <NUM>. In yet other implementations, the visual placeholder <NUM> may be a duplicate of the virtual element <NUM>, retaining any shape, size, opacity, textures, or other properties.

<FIG> illustrates fine adjustment of the position of the visual placeholder <NUM> in the virtual environment <NUM>. In some implementations, gaze-tracking may be imprecise, such that gaze-based movement and positioning of a virtual element <NUM> is performed coarsely and rapidly with gaze-tracking while precision placement is performed subsequent to the gaze-based movement. The precision placement is performed by moving the visual placeholder <NUM> with a secondary input device. The secondary input device is a different input device from the input device <NUM> used to select the virtual element <NUM>.

As described herein, the secondary input device may be a gesture recognition device in data communication with the processor of the HMD. In other implementations, the secondary input device may be a voice recognition device, such as a microphone, in data communication with the processor of the HMD. In yet other implementations, the secondary input device may be a motion controller, such as a 6DOF controller, in data communication with the processor of the HMD. In yet other implementations, the secondary input device may be an input of a touch-sensing device, trackpad, mouse, keyboard or other conventional human interface device (HID) of a computer.

<FIG> shows a movement input <NUM> from the secondary input device (which, in the depicted example, but not according to the present invention, is the same as the input device <NUM>). The movement input <NUM> causes the visual placeholder to decouple from the gaze location <NUM> and move relative to the gaze location <NUM> and the second location <NUM>. For example, a user may want to move a candlestick from a first end of a table in the virtual environment to a second end of the table. The user may select the virtual element (the candlestick) with a downward click of the user's finger (recognized by a gesture recognition device) and then look at the second end of the table to present a visual placeholder at a far end of a table. The visual placeholder may follow the gaze location until the use begins moving their hand relative to the gesture recognition device, at which point the visual placeholder may decouple from the gaze location, allowing the user to look away from the visual placeholder without disturbing the position of the visual placeholder. For example, the user may look back to the candlestick still at the first end of the table and verify the placement of the candlestick relative to other elements on the table and then look back to the second end of the table with the visual placeholder to precisely place the visual placeholder relative to similar elements at the second end of the table.

<FIG> schematically illustrates moving the virtual element <NUM> within the virtual environment <NUM> from first position <NUM> to the destination <NUM>. A deselection command <NUM>, such as an "unclick" gesture from the input device <NUM> may deselect the virtual element <NUM>. Upon deselecting the virtual element <NUM>, the virtual element may move to and replace the visual placeholder at the destination <NUM>. It should be understood that the destination <NUM> may be different from the second location <NUM> and/or the gaze location <NUM>.

A similar method of gaze-informed interaction with a virtual element may include resizing or scaling of a virtual element. <FIG> depict an example of resizing a virtual element using a gaze location to select the virtual element. <FIG> is a flowchart of a method of manipulating a virtual element. The method may include measuring a first position of a user's gaze location at <NUM>. The first position may be measured by using a gaze-tracking device described herein to measure the position of either the user's gaze based on eye-tracking or the user's gaze based on heading tracking. Measuring the first position of the user's gaze location may be similar to measuring the first position in the method described in relation to <FIG>.

The method may further include selecting a virtual element at <NUM>. In some implementations, selecting the virtual element may be similar to selecting the virtual element in the method as described in relation <FIG>. In other implementations, selecting the virtual element may include provided at least two selection commands simultaneously. Providing at least two selection commands may provide two input devices, the relative motion or relative inputs of which may provide the manipulation input to manipulate the virtual element. For example, the manipulation input may be provided by a multi-touch touch-sensing device. In such implementations, the selection inputs may be at least two inputs simultaneously on the multi-touch touch-sensing device. Moving the two inputs relative to one another, such as moving the inputs together or moving the inputs apart, may resize the virtual element. In other examples, moving the two inputs in an arc around a mutual centerpoint may rotate the virtual element.

The method further includes receiving the manipulation input at <NUM> and presenting a manipulated visual placeholder at <NUM>. In some implementations, the manipulation input may be received from a gesture recognition device, such as the camera(s), in data communication with the processor of the HMD. In other implementations, the manipulation input may be received from a voice recognition device, such as a microphone, in data communication with the processor of the HMD. In yet other implementations, the manipulation input may be received from a motion controller, such as a six degree-of-freedom (6DOF) controller, in data communication with the processor of the HMD. In yet other implementations, the manipulation input may be received from a touch-sensing device, trackpad, mouse, keyboard or other conventional human interface device (HID) of a computer.

In some implementations, the visual placeholder may be a copy of the virtual element, such as a ghost of the virtual element. In other implementations, the visual placeholder may be a sphere, a cube, a pyramid, or other three-dimensional solid to reflect the dimensional changes during resizing and/or rotating. In some examples, a three-dimensional solid may be rotationally symmetrical and object axes may be included in the visual placeholder to assist the user in identifying the manipulations applied to the visual placeholder before deselecting and confirming any changes to the virtual element.

<FIG> is a schematic representation of a virtual environment <NUM> with a virtual element <NUM> and a gaze location <NUM> of a user's gaze. The virtual element <NUM> may be selected by a selection command <NUM> from an input device. The input device may be a plurality of input devices <NUM>-<NUM>, <NUM>-<NUM> that may allow for a greater range of commands in the three-dimensional space. For example, a first input device <NUM>-<NUM> may be a user's left hand and a second input device <NUM>-<NUM> may be a user's right hand. While a "click" of one hand provides a selection command to allow subsequent movement of the visual placeholder using the user's hand as described in relation <FIG>, a "click" of two hands simultaneously may provide a selection command and tracking of both hands to allow subsequent movement and/or manipulation of the visual placeholder or virtual element <NUM>. In other examples, the manipulation command may be received from a single input device, such as from a 6DOF peripheral having a scroll wheel, thumb stick, or other directional or rotary input mechanism.

<FIG> illustrates an example of a visual placeholder <NUM> being manipulated by a manipulation command <NUM> from the first input device <NUM>-<NUM> and the second input device <NUM>-<NUM> while remaining at or near a gaze location <NUM>. The selected virtual element <NUM> may remain stationary and/or static while a visual placeholder <NUM> is manipulated relative to the virtual element <NUM> for comparison. For example, scaling or rotating the virtual element <NUM> may be represented by visualizing the relative scaling or rotating of the visual placeholder <NUM> relative to the "original" virtual element <NUM>. In other implementations, the virtual element <NUM> may be scaled or rotated dynamically and/or in real time, such that a user may visualize the effects of the manipulation as the manipulation input <NUM> is received by the system.

As described herein, the visual placeholder <NUM> may be a standard placeholder irrespective of the virtual element <NUM>. For example, the visual placeholder <NUM> may be a sphere. In other examples, the visual placeholder <NUM> may be a cube. In yet other examples, the visual placeholder <NUM> may be any three-dimensional solid. In other implementations, the visual placeholder <NUM> may be a copy of the virtual element <NUM>. For example, the visual placeholder <NUM> may be a "ghost" or partially transparent copy of the virtual element <NUM>. In yet other implementations, the visual placeholder <NUM> may be a duplicate of the virtual element <NUM>, retaining any shape, size, opacity, textures, or other properties. In some examples, a three-dimensional solid may be rotationally symmetrical and object axes may be included in the visual placeholder to assist the user in identifying the manipulations applied to the visual placeholder before deselecting and confirming any changes to the virtual element.

<FIG> illustrates the virtual environment <NUM> with the virtual element <NUM> of <FIG> and <FIG> resized according to the manipulation input <NUM> of <FIG>. A deselection command <NUM> may be received from the input device(s) <NUM>-<NUM>, <NUM>-<NUM> to deselect the virtual element <NUM>. The virtual element <NUM> may then be resized, rotated, or otherwise manipulated as represented by the visual placeholder upon the deselection command <NUM> being received.

While movement and resizing of virtual elements <NUM> are described individually, it should be understood that a user may move a virtual element <NUM> by moving the gaze location <NUM> beyond a threshold, as described in relation to <FIG>, and simultaneously and/or subsequently resize, rotate, or otherwise manipulate the virtual element <NUM> and/or visual placeholder <NUM>, as described in relation to <FIG>. For example, a user may provide a selection command <NUM> via hand gestures with two hands, as described in relation to <FIG>. The user may then move the gaze location <NUM> beyond a threshold to prompt the presentation of a visual placeholder <NUM> at a second location, as described in relation to <FIG>. The user may subsequently move the visual placeholder by moving both hands relative to the gesture recognition device to provide a movement command <NUM>, as described in relation to <FIG>, and may move both hands relative to one another to provide a manipulation command <NUM> as described in relation to <FIG>. Upon deselecting the virtual element <NUM>, such as described in relation to <FIG>, the virtual element <NUM> may move to a destination <NUM>, as described in relation to <FIG>, and may be resized, rotated, or otherwise manipulated as represented by the visual placeholder upon the deselection command <NUM> being received, as described in relation to <FIG>.

Further, it should be understood that input device and secondary input devices may be used in any combination. For example, a selection command may be provided by a HID, such as a mouse, keyboard, trackpad, trackball, or other convention HID, while a movement command may be provided by a touch-sensing device, such as a multi-touch device, and a manipulation command may be provided by a voice recognition device (e.g., recognizing a user speaking "shrink" or "rotate <NUM>°"). In other examples, a selection command may be provided by a voice recognition device (e.g., recognizing a user speaking "select"), while a movement command may be provided by a 6DOF peripheral. In yet other examples, a selection command may be provided by a motion controller (e.g., via a button on the motion controller), and a manipulation command may be provided by a multi-touch touch-sensing device. In further examples, a gesture recognition device may provide a selection command, a movement command, and a manipulation command (e.g., tracking the rotation of a user's hand); while a voice recognition device may provide a second manipulation command (e.g., recognizing a user speaking "scale up <NUM> %").

The articles "a," "an," and "the" are intended to mean that there are one or more of the elements in the preceding descriptions. For example, any element described in relation to an embodiment herein may be combinable with any element of any other embodiment described herein. Numbers, percentages, ratios, or other values stated herein are intended to include that value, and also other values that are "about" or "approximately" the stated value, as would be appreciated by one of ordinary skill in the art. A stated value should therefore be interpreted broadly enough to encompass values that are at least close enough to the stated value to perform a desired function or achieve a desired result. The stated values include at least the variation to be expected in a suitable manufacturing or production process, and may include values that are within <NUM> %, within <NUM> %, within <NUM> %, or within <NUM> % of a stated value.

Claim 1:
A method for improving user interaction with a virtual environment (<NUM>), the method comprising:
measuring (<NUM>) a first position of a user's gaze relative to a virtual element (<NUM>);
selecting (<NUM>) the virtual element (<NUM>) in the virtual environment (<NUM>) upon receiving a selection command from an input device, and thereby identifying an origin, the origin being a geometric or volumetric centre of the virtual element (<NUM>);
measuring (<NUM>) a second position (<NUM>) of a user's gaze relative to the virtual element (<NUM>);
presenting a visual placeholder (<NUM>) at the second position (<NUM>) of the user's gaze when the second position (<NUM>) of the user's gaze is beyond a threshold (<NUM>), the threshold (<NUM>) being an angular displacement or translational distance from the origin; and
moving (<NUM>) the visual placeholder (<NUM>) relative to the second position (<NUM>) to a destination (<NUM>) using a movement input (<NUM>) from a secondary input device, the secondary input device being a different input device from the input device used to select the virtual element (<NUM>);
wherein the movement input (<NUM>) causes the visual placeholder (<NUM>) to decouple from the location of the user's gaze and to move relative to the location of the user's gaze and the second position (<NUM>);
wherein the selection command and the measurement of the first position (<NUM>) are asynchronous; and
wherein the selecting comprises selecting the virtual element (<NUM>) on which the first position of the user's gaze was located prior to the user's gaze moving away from the virtual element (<NUM>).