Patent Description:
An effortless and subtle way to indicate a user's context is to take advantage of gaze tracking data to infer a user's current reference frame. Several problems arise with this approach though, as eye tracking and additional commands are asynchronous (i.e., the eye gaze usually precedes manual inputs and may have moved on to new targets upon finishing recognition of the manual input). In addition, due to technological constraints of the tracking system as well as physiological constraints of the human visual system, the computed gaze signal may be jittery and show offsets compared to the actual eye gaze. This increases the problem of reliably referring to small and closely positioned targets. Thus, an overall problem arises about how such multimodal inputs can be appropriately combined. <CIT> describes a method for controlling a combined eye and gesture tracking system, wherein the method comprises providing a graphical user interface to a user including a plurality of graphical items displayed on the graphical user interface, displaying a graphical pointer on the graphical user interface, detecting a user eye gaze associated with the eyes of the user, detecting a displacement user gesture associated with a body part of the user, and controlling a rate of movement of the graphical pointer on the graphical user interface based on the displacement user gesture and the user eye gaze. <CIT> describes an eye-gaze position detecting unit of an input device detecting an eye-gaze position on a touch panel. A display control unit of the input device compares a current cursor position and the eye-gaze position. The display control unit of the input device sets the cursor position until a distance between the current cursor position and the eye-gaze position reaches a threshold or more. <CIT> describes systems and methods for discerning the intent of a device wearer primarily based on movements of the eyes. The system may be included within unobtrusive headwear that performs eye tracking and controls screen display. The system may also utilize remote eye tracking camera(s), remote displays and/or other ancillary inputs. Screen layout is optimized to facilitate the formation and reliable detection of rapid eye signals. The detection of eye signals is based on tracking physiological movements of the eye that are under voluntary control by the device wearer. <CIT> describes a method of controlling the movement of a cursor which includes selecting a direction of intended movement of the cursor. Thereafter a jump mode is executed in which the cursor is caused to jump in at least one step towards a predetermined target. Subsequently, a drift mode is executed in which the cursor is caused to move substantially continuously in at least one further direction towards the predetermined target. Finally, when the predetermined target has been reached, a control option is executed in dependence upon the nature of the predetermined target.

Features and advantages of the disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. Features of the present disclosure will become more fully apparent from the following description and appended claims or may be learned by the practice of the disclosure as set forth hereinafter.

In order to describe the manner in which the above-recited and other features of the disclosure can be obtained, a more particular description will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. For better understanding, the like elements have been designated by like reference numbers throughout the various accompanying figures. While some of the drawings may be schematic or exaggerated representations of concepts, at least some of the drawings may be drawn to scale. Understanding that the drawings depict some example embodiments, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:.

This disclosure generally relates to devices, systems, and methods for visual user interaction with virtual environments. More specifically, the present disclosure relates to improving interaction with virtual elements using gaze-based guidance of a targeting ray used for selection and manipulation. In some embodiments, visual information may be provided to a user by a near-eye display. A near-eye display may be any display that is positioned near a user's eye, either to supplement a user's view of their surroundings, such as augmented or mixed reality (MR) devices, or to replace the user's view of their surroundings, such as virtual reality (VR) devices. An augmented reality (AR) or MR device is a head-mounted display (HMD) that presents visual information to a user overlaid on the user's view of their surroundings. For example, the visual information from the HMD may be combined with ambient or environment light to overlay visual information, such as text or images, on a user's surroundings.

In some embodiments, the user's field of view may be at least partially encompassed by a waveguide through which the user views their surroundings. The waveguide may direct display light from a display device to the user's field of view. The waveguide may guide the display light before out-coupling the light. Upon out-coupling the light, the waveguide may combine the visual information of the display light with ambient light from the user's surroundings to deliver the visual information to the user. Overlaying the visual information from the HMD on the user's surroundings may require precise generation and positioning of the visual information relative to the user's eyes.

Visual information including virtual elements may be positioned in the user's field of view on the waveguide or other near-eye display. A gaze-tracking device of the HMD may image at least a portion of the user's eye (such as the pupil, the iris, the sclera) and identify a direction or location of the user's gaze. The direction or location of the user's gaze may then be extrapolated to a position on the near-eye display. A selection cursor may be associated with the gaze location to allow the user to highlight or select a virtual element by looking at the virtual element. In other embodiments, a gaze-tracking device may include a gyroscope, an accelerometer, a plurality of sensors to triangulate position, or other devices that allow for the measurement of the orientation and/or position of the HMD relative to the virtual environment. For example, the user's "gaze" may be a ray cast from the HMD forward from the HMD to approximate the user's gaze by approximating the user's head position and orientation as their gaze direction. In some examples, such a head-tracking "gaze" may be simpler than an eye-tracking gaze, as the user remains free to glance around in their field of view without inadvertently moving a gaze position cursor. In other examples, an eye-tracking gaze may be more intuitive as a user will naturally look at whatever object is of interest to the user during interactions with the virtual environment.

In some embodiments, gaze-informed movement and manipulation of virtual objects, either by eye-tracking gaze or by head-tracking gaze, may allow for rapid movement of virtual objects within a virtual or shared environment. In other embodiments, gaze-informed movement and manipulation of a virtual object may be employed in combination with manual, voice, peripheral, or other inputs to provide different scales of movement and manipulation for both speed and precision.

<FIG> is a perspective view of a user <NUM> wearing a HMD <NUM>. In some embodiments, the HMD <NUM> may have a housing <NUM> that contains one or more processors, storage devices, power supplies, audio devices, display devices, cameras, communication devices, or combinations thereof, that receive, collect, store, process, or calculate information that is provided to the user. For example, a display device <NUM> may be positioned optically adjacent a waveguide(s) or other near-eye display <NUM> to provide visual information to the near-eye display <NUM>, which may, in turn, be presented in the user's field of view by the near-eye display <NUM>.

In some embodiments, the HMD <NUM> may have a near-eye display <NUM> positioned near the user <NUM> to direct visual information to the user <NUM>. The HMD <NUM> may include a single near-eye display <NUM>, a separate near-eye display <NUM> for each of the user's eyes (i.e., two near-eye displays <NUM>), or more than two near-eye displays <NUM> to provide visual information over a larger field of view.

In some embodiments, the HMD <NUM> may include one or more cameras <NUM> that may image the user's physical environment. For example, the camera(s) <NUM> may be visible light camera(s) <NUM> that may image the surrounding environment. A processor may perform image recognition routines on the visible light image to detect and recognize elements in the surrounding environment. In other examples, the camera(s) <NUM> may be depth sensing camera(s) that may create a depth image of the surrounding environment. For example, the camera <NUM> may be a time-of-flight camera, a structured light camera, stereo cameras, or other cameras that may use visible, infrared, ultraviolet, or other wavelengths of light to collect three-dimensional information about the surrounding environment. In at least one example, the camera(s) <NUM> may be gesture recognition cameras that allow the HMD <NUM> to recognize and interpret hand gestures performed by the user <NUM> in front of the HMD <NUM>.

The HMD <NUM> further includes a gaze-tracking device <NUM> positioned in the HMD <NUM> to track a direction of the user's gaze. The gaze-tracking device <NUM> may include a camera or a plurality of cameras to image the user's eyes. In other words, the gaze-tracking device <NUM> may image the user's pupil, iris, sclera, other portions of the user's eye, or combinations thereof to calculate the direction the user is looking. In some embodiments, the gaze-tracking device <NUM> may measure and/or calculate the x- and y-components of the user's gaze. In other embodiments, the gaze-tracking device <NUM> may include a gyroscope, an accelerometer, a plurality of sensors to triangulate position, or other devices that allow for the measurement of the orientation and/or position of the HMD relative to the virtual environment. For example, the user's "gaze" may be a ray cast from the HMD forward from the HMD to approximate the user's gaze by approximating the user's head position and orientation as their gaze direction.

<FIG> is a schematic representation of the HMD <NUM>. The display device <NUM> in communication with the near-eye display <NUM> may be in data communication with a processor <NUM>. Similarly, the camera <NUM> and gaze-tracking device <NUM> may be in data communication with the processor <NUM>. The processor <NUM> may further be in data communication with a storage device <NUM>. The storage device <NUM> may be a hardware storage device, such as a platen-based storage device, a solid-state storage device, or other non-transitory or long-term storage device. The storage device <NUM> has instructions stored thereon to perform one or more methods or portions of a method described herein. In some embodiments, an input device <NUM> may be in data communication with the processor <NUM>. The input device <NUM> may be any device capable of providing user inputs including location and/or movement inputs to the system. For example, any device capable of positioning a cursor at a location may be an input device <NUM> for the present description. In some examples, an input device <NUM> may be a motion controller, such as a six-degree-of-freedom (6DOF) controller or a three-degree-of-freedom (3DOF) controller; a gesture recognition device configured to recognize user hand locations and/or gesture inputs; a touch-sensing device, such as a touchscreen device, a trackpad, or other touch-sensing device; a mouse; a trackball; or other device capable of positioning a cursor at a location in the virtual environment presented by the HMD <NUM>.

<FIG> is a flowchart illustrating a method <NUM> of improving user interaction with a virtual environment, according to at least one embodiment of the present disclosure. The method <NUM> includes presenting a virtual environment to a user on a display at <NUM>. The virtual environment may be combined with a physical environment to create a shared environment, such as when the virtual environment is displayed on a waveguide or other transparent near-eye display. In other examples, the virtual environment may be presented on an opaque near-eye display, replacing the user's view of the physical environment with the presented virtual environment.

The method <NUM> includes measuring a gaze location of the user's gaze relative to the virtual environment at <NUM>. The gaze location may be measured by using a gaze-tracking device described herein to measure the position of either the user's gaze based on eye tracking or the user's gaze based on head tracking. In some embodiments, a gaze location may include jitter. The jitter may be mitigated and/or the movement of the gaze location may be smoothed by calculating the gaze location as a gaze cloud or by using smoothing functions.

The gaze cloud may include a quantity of detected gaze positions that are collected in sequence by the gaze-tracking device. The gaze cloud may be calculated by including a plurality of detected gaze positions. For example, the gaze cloud may include a quantity of detected gaze positions in a range having an upper value, a lower value, or upper and lower values including any of <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or any values therebetween. For example, the gaze cloud may include at least <NUM> detected gaze positions. In other examples, the gaze cloud may include less than <NUM> detected gaze positions. In yet other examples, the gaze cloud may include between <NUM> and <NUM> detected gaze positions. In further examples, the gaze cloud may include between <NUM> and <NUM> detected gaze positions. In yet other examples, the gaze cloud may include between <NUM> and <NUM> detected gaze positions. In at least one example, the gaze cloud may be calculated using about <NUM> detected gaze positions.

In some embodiments, the gaze location may include x- and y-coordinates relative to the user's perspective in the virtual environment, such as when interacting with a virtual desktop with a HMD. In other embodiments, the gaze location may include x-, y-, and z-coordinates relative to the user, such as when interacting with virtual elements in three-dimensional space. In some embodiments, the three-dimensional space may be a virtual environment generated by the HMD or other computing device in communication with the HMD. In other embodiments, the three-dimensional space may be a shared environment. For example, a mixed reality HMD may present virtual elements in combination with a surrounding physical environment of the user. In such embodiments, the HMD may measure the surrounding physical environment of the user using, for example, the cameras on the HMD or other sensors to impart information of the surrounding physical environment into a virtual environment to create a shared environment. The HMD may then use the shared environment to position a virtual element in a virtual environment relative to a physical element of the surrounding physical environment.

In some embodiments in a three-dimensional space, the gaze location may be measured by casting a ray from the HMD in the direction of a user's gaze detected by the gaze-tracking device, and the gaze location may be the location where the ray interacts with a surface of the virtual environment or the shared environment. For example, the ray may interact with an upper surface of a virtual element, and the gaze location may be measured as having coordinates on the upper surface of the virtual element. In other examples, the ray may interact with a physical object, such as a surface of a table in front of a user. The first position may be measured as having coordinates on the surface of the physical table.

In other embodiments in a three-dimensional space, the gaze location may be measured using the focal distance of the detected gaze of each of the user's eyes. For example, the detected eye position and related gaze direction of each eye will be different based on the distance of the object at which the user is looking. The different gaze locations of each eye may allow the calculation of a focal depth of the user. Measuring the gaze location using focal depth of the user's gaze may allow for the gaze location to be located in space, and not on a (virtual or physical) surface.

The method <NUM> further includes casting an input ray from an input device at <NUM>. In some embodiments, the input device may be a gesture recognition device, such as the camera(s), in data communication with the processor of the HMD. In other embodiments, the input device may be a voice recognition device, such as a microphone, in data communication with the processor of the HMD. In yet other embodiments, the input device may be a motion controller, such as a six degree-of-freedom (6DOF) controller, in data communication with the processor of the HMD. In yet other embodiments, the input device may be an input of a touch-sensing device, trackpad, mouse, keyboard or other conventional human interface device (HID) of a computer.

The input ray is cast from the input device in the virtual environment. For example, the input ray of a 6DOF motion controller originates at the position of the 6DOF motion controller in the virtual environment measured by the camera(s) of the HMD. In other examples, the input ray of a gesture recognition device originates at the position of the user's hand in the virtual environment measured by the camera(s) of the HMD. In yet other examples, the input ray of a touch-sensing device positioned on a desk or other surface may originate at the position of the touch-sensing device on the desk in the virtual environment measured by the camera(s) of the HMD.

The method <NUM> further includes measuring an input ray location at a distal point of the input ray at <NUM> distal from the input device, itself. For example, the input ray may originate at the input device and the input ray location may be a point along the input ray, outward from the input device in the virtual environment.

In some embodiments, the input ray location includes x- and y-coordinates relative to the user's perspective in the virtual environment, such as when interacting with a virtual desktop with a HMD. In other embodiments, the input ray location includes x-, y-, and z-coordinates relative to the user, such as when interacting with virtual elements in three-dimensional space. In some embodiments, the three-dimensional space is a virtual environment generated by the HMD or other computing device in communication with the HMD. In other embodiments, the three-dimensional space is a shared environment.

In some embodiments, the input ray location may be a point at which the input ray intersects with a point, line, or surface in the virtual environment. For example, the input ray may be cast from the input device like a laser, and the input ray location may be the point illuminated by the laser. In other examples, the input ray location may be a point at a set distance along the input ray. For example, the input ray may be cast from the input device like a rod or pointer stick, and the input ray location may be the terminal distal end of the rod. In at least one embodiment, the input ray location may be a point of the input ray closest to the gaze location. For example, the gaze location may have x-, y-, and z-coordinates relative to the user in the virtual environment and the input ray may project from the input device through the virtual environment. The input ray location may be the point (having x-, y-, and z- coordinates relative to the user in the virtual environment) that is closest to the gaze location in the virtual environment.

The method <NUM> further includes snapping a presented ray location to the gaze location when the input ray location is within a snap threshold distance of the input ray location at <NUM>. In some embodiments, a visualization of the input ray may be displayed to the user in the virtual environment. The presented ray may coincide with the input ray until the ray "snaps" to the gaze location. The presented ray may display a curve, bend, corner, or other deviation from the input ray, such that the presented ray is visualized to the user as projecting from the input device to the gaze location. The input ray and input ray location may be measured by the HMD or other computing device, while the presented ray is displayed to the user. The presented ray location is the location of the presented ray intersecting the gaze location.

At least some interactions with the virtual environment via the input device, such as selecting a virtual element at the gaze location, may be considered to be directed at the presented ray location irrespective of the input ray location. For example, the input ray may be positioned in the virtual environment to the side of a virtual element, while the gaze location is on the virtual element. When the input ray location is with the snap threshold distance, the presented ray location may snap to the gaze location on the virtual element, enabling interaction with the virtual element even if the input ray does not coincide with the virtual element.

In some embodiments, the method <NUM> may further include selecting a virtual element at the presented ray location when snapped to the gaze location. The virtual element may be selected upon receiving a selection command from an input device.

In some embodiments, the selection command and the intended gaze location at the time of the selection command may be asynchronous. For example, a user may look at a virtual element that the user intends to select. The user may begin providing a selection input (e.g., a gesture input, a mouse click, a voice input) to provide the selection command and look away to a second position, moving the gaze location, before the selection input completes and the selection command is received. This may be common among even experienced users, as conventional interactions with computer systems allow for independent movement of the user's eyes and a selection cursor.

In such embodiments, the selection command may select a virtual element the gaze location was positioned on prior to moving away. For example, measuring the first position of the gaze location may include measuring a series of first locations of the gaze location such that the system may "look back" to where the gaze location was prior to receiving the selection command. In some embodiments, measuring the first position of the gaze location may include measuring and retaining in memory at least <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more gaze locations. In other embodiments, measuring the first position of the gaze location may include measuring and retaining in memory all positions of the gaze location for a buffer time, such as <NUM> milliseconds (ms), <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more time. For example, upon receiving a selection command, the system may select the virtual element the user was looking at <NUM> prior to receiving the selection command. In other examples, upon receiving a selection command, the system may iterate through the gaze locations measured during the buffer time until a gaze location coincides with a virtual element.

<FIG> is a schematic representation of a virtual environment <NUM> containing a virtual element <NUM> that may be presented to a user on a display, such as the waveguide of the HMD <NUM> of <FIG>, a display of a MR system, a display of a VR system, a display of an AR system, or other near-eye display in communication with a processor and/or a gaze-tracking device. While the present disclosure described interaction with the virtual environment <NUM> through a HMD, it should be understood that in other embodiments, the virtual environment <NUM> may be presented on another display, such as a laptop, tablet, desktop, large format, or other display in communication with a processor and/or a gaze-tracking device. For example, the methods and systems described herein may be equally applicable to a user interacting with a large format display on a wall of a conference room. The user may move and/or manipulate virtual elements using a gaze-tracking device and other input devices in communication with the large format display. In other examples a laptop may have a front facing camera that may function as a gaze-tracking device to allow gaze-based movement and/or manipulation of virtual elements by a user interacting with the laptop.

In some embodiments, the virtual element <NUM> is any element of the virtual environment that is selectable in the available software. For example, the virtual element <NUM> may be a window of an application presented in the virtual environment. In other examples, the virtual element <NUM> may be a computer-assisted design (CAD) model, a menu, a scroll bar, or another movable element of the virtual environment <NUM> or user interface. In yet other examples, the virtual element <NUM> may be a virtual representation of a physical element of the physical environment around a user.

In at least one example, the user may interact with the virtual element <NUM> and/or the virtual environment <NUM> by positioning their gaze at the virtual element <NUM> or another portion of the virtual environment <NUM>. The gaze-tracking system <NUM>, schematically represented in <FIG> by a user's eyes, in data communication with the processor of the HMD may measure a gaze location <NUM> at the virtual element <NUM>.

The input device <NUM>, schematically represented in <FIG> by a user's hand, may project the input ray <NUM> in the virtual environment. The input ray location <NUM> is a point along the input ray <NUM>. As the input ray location <NUM> and gaze location <NUM> move closer relative to one another, either by translation or rotation of the input device <NUM>, the gaze location <NUM> approaches a snap region <NUM> around the input ray location <NUM> that is based on the snap threshold distance <NUM>.

Before the gaze location <NUM> is within the snap region <NUM>, the input ray <NUM> projected from the input device <NUM> in the virtual environment <NUM> and the presented ray <NUM> that is displayed to the user in the virtual environment <NUM> may coincide with one another.

Referring now to <FIG>, movement of the input ray <NUM> in the virtual environment <NUM> causes the snap region <NUM> and gaze location <NUM> to coincide. When the gaze location <NUM> is in the snap region <NUM>, the presented ray <NUM> may deviate from the input ray location <NUM>. In some embodiments, the snap region <NUM> and gaze location <NUM> may both be overlapping a virtual element <NUM> without the input ray location <NUM> overlapping the virtual element <NUM>.

<FIG> illustrates the presented ray <NUM> and the presented ray location <NUM> snapped to the gaze location <NUM>. The presented ray <NUM> and the presented ray location <NUM> may be displayed to the user on the HMD or other display, while the input ray <NUM> and input ray location <NUM> may remain projected in the virtual environment <NUM> without being displayed. In other words, the input ray <NUM> and the input ray location <NUM> may be projected in a straight line from the input device <NUM>, while the user sees the presented ray <NUM> deviating from where the input ray <NUM> is located, such that the presented ray <NUM> appears to connect the snapped gaze location <NUM> and presented ray location <NUM> to the input device <NUM>. In doing so, the user is provided visual feedback that the input device <NUM> is targeting the gaze location <NUM>, even though the input device <NUM> may be casting an input ray <NUM> that is slightly off the gaze location <NUM>. In such an embodiment, the targeting ray of the input device <NUM> is the presented ray <NUM>, and the input device <NUM> may allow interaction with the virtual element <NUM> or other object at the gaze location <NUM> when the gaze location <NUM> and snap region <NUM> overlap, irrespective of the position of the input ray location <NUM>. The presented ray <NUM> may remain snapped to and may track the gaze location <NUM> as the gaze location <NUM> moves within the snap region <NUM> (or within the unsnap region, as will be described in more detail).

In some embodiments, the snap region <NUM> is defined by the snap threshold distance <NUM>. In some embodiments, the snap threshold distance <NUM> is an angular displacement relative to the user's viewpoint. For example, a snap threshold distance <NUM> may be less than <NUM>° of angular movement relative to the user's perspective. In other examples, a snap threshold distance <NUM> may be less than <NUM>° of angular movement relative to the user's perspective. In yet other examples, a snap threshold distance <NUM> may be less than <NUM>° of angular movement relative to the user's perspective. In at least one example, it may be critical that the snap threshold distance <NUM> may be less than the foveal vision of the user such that the input device does not target objects in the user's peripheral vision. In other embodiments, the snap threshold distance <NUM> may be a translational distance relative the three-dimensional space of the virtual environment <NUM>. For example, translating the input ray location <NUM> one meter backward on a tabletop relative to a gaze location <NUM> may require less angular displacement relative to the gaze location <NUM> than translating the input ray location <NUM> one meter to the right of the gaze location <NUM>. However, the nominal distance moved may be the same.

Therefore, in some instances it may be more efficient for the snap threshold distance <NUM> to be relative to the translational distance of the gaze location <NUM> in the virtual environment <NUM>. In some embodiments, the snap threshold distance <NUM> may be in a range having an upper value, a lower value, or upper and lower values including any of <NUM> centimeter (cm), <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or any values therebetween. For example, the snap threshold distance <NUM> may be greater than <NUM>. In other examples, the snap threshold distance <NUM> may be less than <NUM>. In yet other examples, the snap threshold distance <NUM> may be between <NUM> and <NUM>. In further examples, the snap threshold distance <NUM> may be between <NUM> and <NUM>.

Targeting the gaze location <NUM> for interaction with the input device <NUM> may allow more precise targeting by snapping the targeting ray to the gaze location <NUM> instead of the input ray location <NUM> when the gaze location <NUM> is within the snap region <NUM>. <FIG> illustrates an example of the user looking at a different object or location within the virtual environment <NUM>, moving the gaze location <NUM> outside of the snap region <NUM>. The presented ray <NUM> may unsnap back to coincide with the input ray <NUM> after the gaze location <NUM> (and therefore, the user's attention) has left the snap region <NUM>.

However, due to jitter in the gaze tracking (i.e., the gaze location <NUM>), jitter in the input ray casting (i.e., the input ray location <NUM>), or both, the presented ray <NUM> may snap and unsnap rapidly when the gaze location <NUM> is near the edge of the snap region <NUM>. To mitigate this effect, some embodiments may have a snap region <NUM> and an unsnap region <NUM> that is different from the snap region <NUM>, as illustrated in <FIG>. For example, the presented ray <NUM> may snap to the gaze location <NUM> when the gaze location is within the snap region <NUM>, as described herein. The presented ray <NUM> may then remain snapped to the gaze location <NUM> until the gaze location <NUM> moves outside of the unsnap region <NUM>. In some examples, the unsnap threshold distance <NUM> may be greater than the snap threshold distance <NUM>. In some embodiments, the unsnap threshold distance <NUM> may be at least <NUM>% greater than the snap threshold distance <NUM>. In some embodiments, the unsnap threshold distance <NUM> may be at least <NUM>% greater than the snap threshold distance <NUM>. In some embodiments, the unsnap threshold distance <NUM> may be at least <NUM>% greater than the snap threshold distance <NUM>. In some embodiments, the unsnap threshold distance <NUM> may be at least <NUM>% greater than the snap threshold distance <NUM>.

The user may look away from the intended target of an interaction before "clicking" or otherwise providing the selection command. In some embodiments, the system may use previous gaze locations for the interaction, as described herein; however in some examples, the movement of the gaze location <NUM> may unsnap the targeting ray (e.g., the presented ray <NUM>) from the gaze location <NUM>, decoupling the selection command from the gaze location <NUM>. In other embodiments, after the user's gaze saccades away from the original gaze location <NUM>, the presented ray location <NUM> may return toward the input ray location <NUM> after a delay and/or over an unsnap duration. This delay and/or dampening of the presented ray location <NUM> moving to the input ray location <NUM> may slow the straightening of the presented ray <NUM>, allowing the selection command to be received and applied to the intended presented ray location <NUM>.

In some embodiments, the delay between the user's gaze saccading away from the gaze location <NUM> and the presented ray location <NUM> moving back to the input ray location <NUM> may be in a range having an upper value, a lower value, or upper and lower values including any of <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or any values therebetween. For example, the delay may be greater than <NUM>. In other examples, the delay may be less than <NUM>. In yet other examples, the delay may be between <NUM> and <NUM>. In further examples, the delay may be between <NUM> and <NUM>. In at least one example, the delay may be about <NUM>.

In other embodiments, the unsnapping (i.e., straightening) of the presented ray <NUM> may be dampened such that the presented ray <NUM> straightens over an unsnap duration having an upper value, a lower value, or upper and lower values including any of <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or any values therebetween. For example, the unsnap duration may be greater than <NUM>. In other examples, the unsnap duration may be less than <NUM>. In yet other examples, the unsnap duration may be between <NUM> and <NUM>. In further examples, the unsnap duration may be between <NUM> and <NUM>. In at least one example, the unsnap duration may be about <NUM>.

While embodiments of methods of improving user interaction with a virtual environment have been described herein in relation to snapping a targeting ray to a gaze location, selection precision of virtual elements may be increased by using the gaze location and snap region of the input ray to snap the targeting ray to the virtual element instead of the gaze location. For example, a user's attention may be naturally drawn to high-contrast areas or borders of virtual elements. While a natural behavior, this behavior can limit the effectiveness of gaze-based targeting. <FIG> illustrates an example of a gaze location <NUM> at an edge of a virtual element <NUM>. The input ray location <NUM> is positioned on the virtual element <NUM>, and the snap region <NUM> includes over half of the virtual element <NUM>. However, the gaze location <NUM> is positioned at the edge of the virtual element <NUM> and snapping the targeting ray to the gaze location <NUM> may pull the targeting ray off of the virtual element <NUM>, interfering with selection of or other interaction with the virtual element <NUM>.

In some embodiments, the targeting ray may snap to an origin of the virtual element <NUM> when the gaze location <NUM> is within the snap region <NUM> and either of the input ray location <NUM> or the gaze location <NUM> is within the edges of the virtual element <NUM>. <FIG> illustrates an example of a gaze location <NUM> positioned within the edges of the virtual element <NUM>, but near the edge, and the input ray location <NUM> is within the edges of the virtual element <NUM>, but near the edge. The presented ray <NUM> is snapped to an origin <NUM> of the virtual element <NUM> that is positioned away from the edges of the virtual element <NUM> that may be in or out of the snap region <NUM>. Snapping the targeting ray (i.e., the presented ray <NUM>) to the origin <NUM> may ensure a selection of or other interaction with the virtual element <NUM> is received in relation to the virtual element <NUM>.

For example, the origin <NUM> may be the geometric center (center of the height and center of the width) of the virtual element <NUM>, as shown in <FIG>. In other examples, the origin may be the volumetric center of the virtual element. In a specific example, the origin of a virtual model of the Eiffel Tower may be positioned closer to the bottom when the origin is a volumetric origin than when the origin is a geometric origin. In yet other examples, the origin may be a "higher certainty area", such as a visible area, a high-contrast area, or other area of the virtual element as selected by the user or by the system.

Snapping the targeting ray to the origin may intuitively indicate to a user to look at or near the origin of the virtual element during interaction with the virtual element, further reinforcing the targeting of the virtual element by drawing the gaze location toward the origin. For example, snapping the targeting ray at the origin of a button may provide a user with an intuitive instruction to look at the end of the targeting ray to positively select the button. Additionally, snapping the end of the targeting ray may leverage the same involuntary response described earlier to unconsciously guide the user's gaze toward the origin of the virtual element by presenting a high-contrast or moving visual element at the origin. For example, the sudden appearance of the snapped targeting ray at the origin of a button may provoke an involuntarily response from the user to look at the end of the snapped targeting ray.

As described herein, the snapping behavior of the targeting ray and the unsnapping behavior of the targeting ray may be asymmetric. For example, the snap region and the unsnap region around the input ray location may be different, as described in relation to <FIG>. This can limit intended unsnapping of the targeting ray due to jitter in the eye-tracking and/or ray casting, as well as mitigate effects from a leave-before-clicking behavior by a user. Additionally, different snap and unsnap regions, a delay, or dampening (e.g., an unsnap duration of the unsnapping and straightening of the targeting ray) may mitigate unintended movement of the input ray and/or input device.

<FIG> illustrates an embodiment of a input device <NUM> that is a 6DOF motion controller. The position and orientation of the input device <NUM> is projected in the virtual environment <NUM> as the input ray location <NUM>. The presented ray <NUM> is snapped to an origin <NUM> of the virtual element <NUM>. When a user provides a selection command, such as by pressing a button, trigger, thumb stick, pad, or other interaction element <NUM> of the input device <NUM>, the input device <NUM> may move, producing an associated movement of the input ray location <NUM> and the unsnap region <NUM>. In the illustrated example, the unsnap region <NUM> has moved such that the gaze location <NUM> is outside of the unsnap region <NUM>. This may result in the presented ray <NUM> unsnapping from the origin <NUM> (or, in other examples, from the gaze location <NUM>). As the input ray location <NUM> is now off of the virtual element <NUM>, the selection command from the input device <NUM> would fail to select the virtual element <NUM>. In at least one embodiment described herein, a delay or dampening of the unsnapping of the targeting ray may allow the virtual element <NUM> to be correctly selected by the selection command, despite the movement of the input device <NUM>.

<FIG> illustrate examples of snap regions changing size and/or shape to change the snapping behavior of the targeting ray. <FIG> is an example of snap region <NUM> that may be sized relative to the dimensions of the virtual elements <NUM> in the virtual environment <NUM>. The snap threshold distance <NUM> may be based, at least partially, on the width <NUM> and/or height <NUM> of a selectable virtual element <NUM>.

This may be particularly helpful in a crowded virtual environment <NUM> with a plurality of virtual elements <NUM>. In some examples, the snap threshold distance <NUM> may be changed based on the smaller of the width <NUM> and height <NUM> of the virtual element <NUM>, such that the targeting ray does not snap to the gaze location <NUM> undesirably or continue to follow the gaze location <NUM> as the user looks at other virtual elements in the virtual environment <NUM> while holding the input device <NUM> stationary. As the user may look at each of the virtual elements <NUM> successively, such as when making a selection from a menu, the targeting ray may otherwise remain snapped to the gaze location <NUM>, distracting the user and interfering with the user's interaction with the virtual environment <NUM>. In this way, the snapping behavior of the targeting ray may balance convenience and distraction to the user.

<FIG> is an example of a user moving the input device <NUM> toward the virtual element <NUM> in the virtual environment <NUM> to reduce the snap threshold distance <NUM> around the input ray location <NUM>. For example, it may be intuitive for some users to reach toward an object while looking at the object to identify the object more precisely. The present example leverages that natural behavior to reduce the snap threshold distance <NUM> so that the targeting ray snaps to the gaze location <NUM> only when the targeting ray is closer to the gaze location <NUM>.

In some embodiments, the change in the snap threshold distance <NUM> may be linear to the change in the distance from the input device <NUM> to the virtual element <NUM> in the virtual environment <NUM>. For example, moving the input device <NUM> in the virtual environment <NUM><NUM>% of the distance to the virtual element <NUM> may reduce the snap threshold distance <NUM> by <NUM>%. In such embodiments, if a user were to reach out with their hand as the input device <NUM> and "touch" the virtual element <NUM>, the snap threshold distance <NUM> may approach zero, as the input device <NUM> would be in contact with the virtual element <NUM> and the gaze location <NUM> may be irrelevant in that situation.

In other embodiments, the change in the snap threshold distance <NUM> may be non-linear to the change in the distance from the input device <NUM> to the virtual element <NUM> in the virtual environment <NUM>. For example, moving the input device <NUM> in the virtual environment <NUM><NUM>% of the distance to the virtual element <NUM> may reduce the snap threshold distance <NUM> by <NUM>%. In yet other embodiments, the change in the snap threshold distance <NUM> may be non-linear to the change in the distance from the input device <NUM> to the virtual element <NUM> in the virtual environment <NUM>. For example, moving the input device <NUM> in the virtual environment <NUM><NUM>% of the distance to the virtual element <NUM> may reduce the snap threshold distance <NUM> by <NUM>% and moving the input device <NUM> the remaining <NUM>% of the distance to the virtual element <NUM> may reduce the snap threshold distance <NUM> by the remaining <NUM>% such that if a user were to reach out with their hand (the user's hand acting as the input device <NUM> in conjunction with a gesture recognition device) and "touch" the virtual element <NUM>, the snap threshold distance <NUM> approaches zero.

In at least one embodiment, a method of interacting with a virtual environment according to the present disclosure may allow for users to select or interact with virtual elements using an input device while assisting the targeting with the user's gaze. Because users typically look at an object or virtual element before interacting with the object, the method is natural and intuitive while increasing accuracy and providing positive feedback. As users become more familiar with a UI or environment, the users' interactions may become faster and exacerbate leave-before-click behaviors. In at least some embodiments according to the present disclosure, methods described herein contemplate one or more features to mitigate or eliminate errors associates with leave-before-click behaviors. Numbers, percentages, ratios, or other values stated herein are intended to include that value, and also other values that are "about" or "approximately" the stated value, as would be appreciated by one of ordinary skill in the art encompassed by embodiments of the present disclosure. A stated value should therefore be interpreted broadly enough to encompass values that are at least close enough to the stated value to perform a desired function or achieve a desired result. The stated values include at least the variation to be expected in a suitable manufacturing or production process, and may include values that are within <NUM>%, within <NUM>%, within <NUM>%, or within <NUM>% of a stated value.

The described embodiments are to be considered as illustrative and not restrictive. The scope of the disclosure is, therefore, indicated by the appended claims rather than by the foregoing description.

Claim 1:
A method for improving user interaction with a virtual environment, the method comprising:
presenting (<NUM>) the virtual environment (<NUM>) to a user on a head-mounted display (<NUM>);
measuring (<NUM>) a gaze location (<NUM>) of a user's gaze relative to the virtual environment (<NUM>) using a gaze-tracking device (<NUM>) of the head-mounted display (<NUM>);
generating (<NUM>) an input ray (<NUM>) from an input device (<NUM>);
measuring (<NUM>) an input ray location (<NUM>) at a distal point of the input ray (<NUM>);
and characterized by:
displaying a presented ray (<NUM>), coinciding with the input ray (<NUM>), such that a presented ray location (<NUM>) coincides with the input ray location (<NUM>);
snapping (<NUM>) the presented ray location (<NUM>) from the input ray location (<NUM>) to the gaze location (<NUM>) when the gaze location (<NUM>) is within a snap threshold distance (<NUM>) of the input ray location (<NUM>).