Patent ID: 12236634

DESCRIPTION OF IMPLEMENTATIONS

In various circumstances, an electronic device determines a gaze vector associated with an eye of a user. For example, the electronic device determines the gaze vector by tracking an eye within image data of the eye, such as via computer vision. Tracking the eye includes tracking a movement of the eye. The movement of the eye may be concurrent with a movement of the electronic device. As one example, an eye initially focuses on an object of interest. While the electronic device moves and in order to maintain the eye focus, the eye moves to account for the movement of the electronic device. However, tracking the eye within the image data is computationally expensive, and therefore the eye tracking may substantially lag the actual eye position of the user. Moreover, the lag may be exacerbated based on the motion of the electronic device.

By contrast, various implementations disclosed herein include methods, electronic devices, and systems for supplementing eye tracking based on device motion information. To that end, an electronic device includes an image sensor that captures image data of an eye of a user of the electronic device. While the electronic device is in a first position, the electronic device determines a gaze vector based on the image data, such as via computer vision. For example, the electronic device determines the gaze vector by identifying a first subset of pixels of a first image of the image data, wherein the first subset of pixels corresponds to the eye. As noted above, tracking the eye within the image data is computationally expensive (e.g., introduces latency). Accordingly, the electronic device assesses positional sensor data from a positional sensor in order to update the gaze vector. The positional sensor data indicates a positional change of the electronic device from the first position to a second position. For example, in some implementations, updating the gaze vector includes repositioning the gaze vector (e.g., changing the angle of the gaze vector). As another example, in some implementations, updating the gaze vector includes changing a targeting tolerance (e.g., for targeting an object) associated with the gaze vector, without repositioning the gaze vector. Assessing the positional sensor data is generally less computationally expensive than tracking the eye within the image data. Accordingly, in order to account for an eye movement concurrent with the positional change, assessing the positional sensor data enables the electronic device to update the gaze vector faster than by assessing the eye movement within the image data.

Reference will now be made in detail to implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described implementations. However, it will be apparent to one of ordinary skill in the art that the various described implementations may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the implementations.

It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described implementations. The first contact and the second contact are both contacts, but they are not the same contact, unless the context clearly indicates otherwise.

The terminology used in the description of the various described implementations herein is for the purpose of describing particular implementations only and is not intended to be limiting. As used in the description of the various described implementations and the appended claims, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes”, “including”, “comprises”, and/or “comprising”, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting”, depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event]”, depending on the context.

A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mountable system may be configured to accept an external opaque display (e.g., a smartphone). The head mountable system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mountable system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person's eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

FIG.1is a block diagram of an example of a portable multifunction device100(sometimes also referred to herein as the “electronic device100” for the sake of brevity) in accordance with some implementations. The electronic device100includes memory102(which optionally includes one or more computer readable storage mediums), a memory controller122, one or more processing units (CPUs)120, a peripherals interface118, an input/output (I/O) subsystem106, a speaker111, a display system112, an inertial measurement unit (IMU)130, image sensor(s)143(e.g., camera), contact intensity sensor(s)165, audio sensor(s)113(e.g., microphone), eye tracking sensor(s)164(e.g., included within a head-mountable device (HMD)), an extremity tracking sensor150, and other input or control device(s)116. In some implementations, the electronic device100corresponds to one of a mobile phone, tablet, laptop, wearable computing device, head-mountable device (HMD), head-mountable enclosure (e.g., the electronic device100slides into or otherwise attaches to a head-mountable enclosure), or the like. In some implementations, the head-mountable enclosure is shaped to form a receptacle for receiving the electronic device100with a display.

In some implementations, the peripherals interface118, the one or more processing units120, and the memory controller122are, optionally, implemented on a single chip, such as a chip103. In some other implementations, they are, optionally, implemented on separate chips.

The I/O subsystem106couples input/output peripherals on the electronic device100, such as the display system112and the other input or control devices116, with the peripherals interface118. The I/O subsystem106optionally includes a display controller156, an image sensor controller158, an intensity sensor controller159, an audio controller157, an eye tracking controller160, one or more input controllers152for other input or control devices, an IMU controller132, an extremity tracking controller180, and a privacy subsystem170. The one or more input controllers152receive/send electrical signals from/to the other input or control devices116. The other input or control devices116optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate implementations, the one or more input controllers152are, optionally, coupled with any (or none) of the following: a keyboard, infrared port, Universal Serial Bus (USB) port, stylus, paired input device, and/or a pointer device such as a mouse. The one or more buttons optionally include an up/down button for volume control of the speaker111and/or audio sensor(s)113. The one or more buttons optionally include a push button. In some implementations, the other input or control devices116includes a positional system (e.g., GPS) that obtains information concerning the location and/or orientation of the electronic device100relative to a particular object. In some implementations, the other input or control devices116include a depth sensor and/or a time of flight sensor that obtains depth information characterizing a particular object.

The display system112provides an input interface and an output interface between the electronic device100and a user. The display controller156receives and/or sends electrical signals from/to the display system112. The display system112displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some implementations, some or all of the visual output corresponds to user interface objects. As used herein, the term “affordance” refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to inputs directed toward the graphical user interface object). Examples of user-interactive graphical user interface objects include, without limitation, a button, slider, icon, selectable menu item, switch, hyperlink, or other user interface control.

The display system112may have a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on haptic and/or tactile contact. The display system112and the display controller156(along with any associated modules and/or sets of instructions in the memory102) detect contact (and any movement or breaking of the contact) on the display system112and converts the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed on the display system112. In an example implementation, a point of contact between the display system112and the user corresponds to a finger of the user or a paired input device.

In some implementations, the display system112corresponds to a display integrated in a head-mountable device (HMD), such as AR glasses. For example, the display system112includes a stereo display (e.g., stereo pair display) that provides (e.g., mimics) stereoscopic vision for eyes of a user wearing the HMD.

The display system112optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other implementations. The display system112and the display controller156optionally detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the display system112.

The user optionally makes contact with the display system112using any suitable object or appendage, such as a stylus, a paired input device, a finger, and so forth. In some implementations, the user interface is designed to work with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen. In some implementations, the electronic device100translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user.

The speaker111and the audio sensor(s)113provide an audio interface between a user and the electronic device100. Audio circuitry receives audio data from the peripherals interface118, converts the audio data to an electrical signal, and transmits the electrical signal to the speaker111. The speaker111converts the electrical signal to human-audible sound waves. Audio circuitry also receives electrical signals converted by the audio sensors113(e.g., a microphone) from sound waves. Audio circuitry converts the electrical signal to audio data and transmits the audio data to the peripherals interface118for processing. Audio data is, optionally, retrieved from and/or transmitted to the memory102and/or RF circuitry by the peripherals interface118. In some implementations, audio circuitry also includes a headset jack. The headset jack provides an interface between audio circuitry and removable audio input/output peripherals, such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone).

The inertial measurement unit (IMU)130includes accelerometers, gyroscopes, and/or magnetometers in order measure various forces, angular rates, and/or magnetic field information with respect to the electronic device100. Accordingly, according to various implementations, the IMU130detects one or more positional change inputs of the electronic device100, such as the electronic device100being shaken, rotated, moved in a particular direction, and/or the like.

The image sensor(s)143capture still images and/or video. In some implementations, an image sensor143is located on the back of the electronic device100, opposite a touch screen on the front of the electronic device100, so that the touch screen is enabled for use as a viewfinder for still and/or video image acquisition. In some implementations, the image sensor(s) are integrated within an HMD.

The contact intensity sensors165detect intensity of contacts on the electronic device100(e.g., a touch input on a touch-sensitive surface of the electronic device100). The contact intensity sensors165are coupled with the intensity sensor controller159in the I/O subsystem106. The contact intensity sensor(s)165optionally include one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). The contact intensity sensor(s)165receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the physical environment. In some implementations, at least one contact intensity sensor165is collocated with, or proximate to, a touch-sensitive surface of the electronic device100. In some implementations, at least one contact intensity sensor165is located on the side of the electronic device100.

The eye tracking sensor(s)164detect eye gaze of a user of the electronic device100and generate eye tracking data indicative of the eye gaze of the user. In various implementations, the eye tracking data includes data indicative of a fixation point (e.g., point of regard) of the user on a display panel, such as a display panel within a head-mountable device (HMD), a head-mountable enclosure, or within a heads-up display. In some implementations, the eye tracking sensor(s)164include an image sensor that captures image data (e.g., a series of images) of an eye of a user. In some implementations, the eye tracking controller160tracks the eye within the image data, such as via computer vision.

The extremity tracking sensor150obtains extremity tracking data indicative of a position of an extremity of a user. For example, in some implementations, the extremity tracking sensor150corresponds to a hand tracking sensor that obtains hand tracking data indicative of a position of a hand or a finger of a user within a particular object. In some implementations, the extremity tracking sensor150utilizes computer vision techniques to estimate the pose of the extremity based on camera images.

In various implementations, the electronic device100includes a privacy subsystem170that includes one or more privacy setting filters associated with user information, such as user information included in extremity tracking data, eye gaze data, and/or body position data associated with a user. In some implementations, the privacy subsystem170selectively prevents and/or limits the electronic device100or portions thereof from obtaining and/or transmitting the user information. To this end, the privacy subsystem170receives user preferences and/or selections from the user in response to prompting the user for the same. In some implementations, the privacy subsystem170prevents the electronic device100from obtaining and/or transmitting the user information unless and until the privacy subsystem170obtains informed consent from the user. In some implementations, the privacy subsystem170anonymizes (e.g., scrambles or obscures) certain types of user information. For example, the privacy subsystem170receives user inputs designating which types of user information the privacy subsystem170anonymizes. As another example, the privacy subsystem170anonymizes certain types of user information likely to include sensitive and/or identifying information, independent of user designation (e.g., automatically).

FIGS.2A-2Iare examples of updating a gaze vector based on positional sensor data in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein.

As illustrated inFIG.2A, a user50wears a head-mountable device (HMD)210that operates according to an operating environment200. The operating environment200includes a first physical wall202, a second physical wall204, a physical painting205hanging on the first physical wall202, and a physical credenza206. The operating environment200further includes a computer-generated lamp208(generated by the HMD210). The computer-generated lamp208is anchored (e.g., world-locked) to the surface of the physical credenza206. In some implementations, the operating environment200corresponds to an XR environment previously described. Although the examples described with reference toFIGS.2A-2I,3A-3B, and4include the HMD210, one of ordinary skill in the art will appreciate that some implementations include another type of electronic device, such as a mobile device (e.g., a smartphone, tablet, etc.) including an image sensor and a positional sensor.

In some implementations, the HMD210includes an display212(e.g., a built-in display) that displays a representation of the operating environment200. In some implementations, the HMD210includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., a smartphone). For example, in some implementations, a smartphone slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) a representation of the operating environment200. The display212includes a viewable region214that includes the first physical wall202, the second physical wall204, the physical painting205, and the physical credenza206. Accordingly, the HMD210displays, on the display212, respective representations of the first physical wall202, the second physical wall204, the physical painting205, and the physical credenza206, in addition to displaying the computer-generated lamp208.

As further illustrated inFIG.2A, while the HMD210is in a first position, the HMD210is associated with a first head forward vector216. The first head forward vector216is normal (e.g., orthogonal) to the HMD210, and terminates at a point on the first physical wall202. The first head forward vector216is a function of the current position (e.g., location and orientation) of the HMD210. Accordingly, the first head forward vector216characterizes the first position of the HMD210.

In various implementations, the HMD210includes a positional sensor that generates positional sensor data. Moreover, in some implementations, the HMD210determines a head forward vector (e.g., the first head forward vector216) based at least in part on the positional sensor data. For example, with reference toFIG.4, the HMD210includes a positional sensor450, such as an inertial measurement unit (IMU). Moreover, the HMD210may include a motion system432that determines a head forward vector based on the positional sensor data from the positional sensor450. Moreover, the motion system432may determine a movement type of the HMD210based on the positional sensor data, such as a rotational movement. The HMD210may also determine its location and orientation using image data from one or more image sensors (e.g., a second image sensor402illustrated inFIG.4) alone or in combination with the positional sensor data from the positional sensor450using, for example, a visual odometry (VO) or visual inertial odometry (VIO) technique.

Referring toFIG.2B, the HMD210determines a first gaze vector220associated with an eye10of the user50. The first gaze vector220intersects with the first head forward vector216at a first angle θ1. Moreover, the first gaze vector220is associated with a first targeting region (e.g., a point of regard (POR) or gaze region) that intersects with the upper left portion of the computer-generated lamp208. The first targeting region is indicated by a first reticle222, which may or may not be displayed on the display212.

With reference toFIG.4, in order to determine a gaze vector418(e.g., the first gaze vector220), the HMD210may include an image driven eye tracker410. The image driven eye tracker410includes a light source412, a first image sensor414, and a controller416.

The light source412emits light onto the eye10that reflects as a light pattern (e.g., a circle of glints) that can be detected by the first image sensor414. Accordingly, the first image sensor414captures image data of the eye10. Based on the light pattern, the controller416can determine an eye tracking characteristic of the eye10. For example, the controller416determines the gaze vector418based on the image data. As one example, the controller416identifies a first subset of pixels of a first image of the image data, wherein the first subset of pixels corresponds to the eye10.

The image data from the first image sensor414may include one or more images of the eye10. For example, with reference toFIG.3A, the image data includes a first image300. The first image300includes a first representation302of the eye10. The first representation302includes a pupil306surrounded by an iris310, both covered by a cornea308. The first representation302also includes a sclera304(also known as the white of the eye10).

In some implementations, the controller416determines the gaze vector418based on an offset vector associated with the first image300. For example, as illustrated inFIG.3A, a reference line312(illustrated for purely explanatory purposes) runs along the center of the first representation302of the eye10. Moreover, the controller416determines a first eye position line314(illustrated for purely explanatory purposes). The first eye position line314runs along the center of the iris310. For example, the first eye position line314intersects with the pupil306. Moreover, the controller416determines a first offset vector316that relates the reference line312to the first eye position line314. Namely, the first offset vector316points rightwards and has a magnitude corresponding to a distance between the reference line312and the first eye position line314. Accordingly, with reference toFIG.2B, based on the first offset vector316the controller416determines that the first gaze vector220points leftwards (opposite of the direction of the first offset vector316) relative to the HMD210, at the first angle θ1that is based on the magnitude of the first offset vector316.

With reference toFIG.4, in some implementations, the HMD210includes an object selector420that selects an object represented in XR environment data408, based on the gaze vector418. The object may be computer-generated or physical. To that end, in some implementations, the HMD210includes a second image sensor402that captures environment image data404of a physical environment400. Moreover, the HMD210may include an XR environment generator406that generates the XR environment data408based on the environment image data404and computer-generated content405(e.g., the computer-generated lamp208). For example, with reference toFIG.2B, the XR environment generator406composites the computer-generated lamp208with environment image data representing the first physical wall202, the second physical wall204, the physical painting205, and the physical credenza206. Referring back toFIG.4, the object selector420selects the computer-generated lamp208based on the first gaze vector220intersecting with the computer-generated lamp208. In some implementations, the object selector420includes a focus system422, which determines a level of user focus with respect to an object. For example, the focus system422determines how long the eye10focuses on the object, and the object selector420selects the object when the length of the focus exceeds a threshold.

Referring back toFIG.2B, in some implementations, the first gaze vector220is associated with a first targeting tolerance. The HMD210may use the first targeting tolerance to determine whether to target (e.g., select) a particular object of the operating environment200. For example, the first targeting tolerance defines a first targeting region, which is indicated by the first reticle222. In some implementations, a targeting region can be represented by a gaze cone positioned such that the apex of the gaze cone is located at an eye of a user and the gaze vector runs through the center of the gaze cone. The size of the targeting region at a particular location may be a function of a distance between a user and an object, or a distance between an electronic device (e.g., the HMD210) and the object. For example, for a given distance between the user or electronic device and the object, the gaze cone has a corresponding radius that corresponds to the first targeting region. In these implementations, the targeting tolerance can be represented by the angle formed between the axis and surface of the gaze cone (e.g., larger tolerances correspond to larger angles, and smaller tolerances correspond to smaller angles). Continuing with this example, the HMD210targets (e.g., selects) the computer-generated lamp208because the first targeting region includes the computer-generated lamp208.

As illustrated inFIG.2C, the user50initiates a first rightwards rotation of the HMD210from the first position to a second position, as indicated by a first rotation line230. The second position is characterized by a second head forward vector232, as illustrated inFIG.2D. The second head forward vector232intersects with the physical painting205. The HMD210receives positional sensor data indicative of the first rightwards rotation.

During the first rightwards rotation, the eye10maintains focus on the computer-generated lamp208. For example, as illustrated inFIG.3B, based on the first rightwards rotation of the HMD210, the image sensor captures a second image320. The second image320includes a second representation322of the eye10. The position of the center of the iris310in the second image320is indicated by a second eye position line324(illustrated for purely explanatory purposes). The second eye position line324of the second image320is rightwards of the first eye position line314of the first image300, because, in order to maintain focus on the computer-generated lamp208, the eye10moves to counter the first rightwards rotation of the HMD210. As illustrated inFIG.3B, a second offset vector326relates the reference line312to the second eye position line324. The second offset vector326points rightwards and has a magnitude corresponding to a distance between the reference line312and the second eye position line324.

As described above, tracking (e.g., via computer vision) the movement of the eye10via the image driven eye tracker410is processor intensive and thus relatively slow. Accordingly, a substantial lag may exist between the generation of the positional sensor data (indicative of the first rightwards rotation), and the controller410determining the second eye position line324. Before the image driven eye tracker410can determine the second eye position line324, the image driven eye tracker410continues to identify the eye10within the first image300. Accordingly, as illustrated inFIG.2D, the image driven eye tracker410determines a first unsupplemented gaze vector236based on the position of the eye in the first image300(the first eye position line314). Accordingly, as compared with the first gaze vector220, the first unsupplemented gaze vector236is repositioned based on the first rightwards rotation of the HMD210. The first unsupplemented gaze vector236, therefore, is offset from the second head forward vector232according to the first angle θ1. The first unsupplemented gaze vector236is associated with a second targeting region that intersects with a portion of the first physical wall202. The second targeting region is indicated by a second reticle234. Moreover, the first unsupplemented gaze vector236is offset from the first head forward vector216according to a second angle θ2.

In order to supplement the relatively slow tracking of the movement of the eye within the image data, the HMD210assesses positional sensor data to update the unsupplemented first gaze vector236. For example, with reference toFIG.4, the HMD210may include a gaze vector updater430that updates the gaze vector418(e.g., the first unsupplemented gaze vector236) based on the positional sensor data from the positional sensor450, in order to determine an updated gaze vector440. The positional sensor data is indicative of the positional change from the first position to the second position (e.g., the first rightwards rotation). To that end, in some implementations, the gaze vector updater430includes a motion system432that determines a supplemental positional value based on the positional sensor data. The supplemental positional value may include an angular change, positional displacement, movement speed, etc. For example, with reference toFIG.2D, the motion system432determines an angular change of the HMD210resulting from the first rightwards rotation, wherein the angular change corresponds to the sum of the first angle θ1and the second angle θ2. To that end, the motion system432may detect the first head forward vector216and the second head forward vector232based on the positional sensor data, and determine the angular change based on a difference between the first head forward vector216and the second head forward vector232.

In some implementations, updating a gaze vector includes repositioning the gaze vector based on the supplemental positional value. To that end, in some implementations, the gaze vector updater430includes a repositioning system434. For example, with reference toFIG.2E, the repositioning system434repositions the first unsupplemented gaze vector236based on the angular change, in order to determine a repositioned gaze vector240. The repositioned gaze vector240is associated with a third targeting region that intersects with the upper left portion of the computer-generated lamp208, as indicated by a third reticle238. Thus, the repositioned gaze vector240reflects the actual eye gaze of the user50, which is maintained on the computer-generated lamp208during the first rightwards rotation of the HMD210. The HMD210, therefore, updates a gaze vector based on the positional sensor data, before determining the corresponding movement of an eye within image data (e.g., via computer vision). Illustration of the first head forward vector216is omitted fromFIGS.2E-2Ifor the sake of clarity. In some implementations, a known or estimated latency associated with the image driven eye tracker410may be determined and used by the repositioning system434. For example, the repositioning system434may reposition the first unsupplemented gaze vector236using a supplemental positional value determined by the motion system432based on positional sensor data accumulated over a period of time corresponding to the known or estimated latency. In this way, a determined gaze vector418may be repositioned based on movement that has occurred since a previously determined gaze vector.

In some implementations, in addition to or instead of repositioning a gaze vector based on positional sensor data, updating the gaze vector includes changing a targeting threshold associated with the gaze vector based on the positional sensor data. To that end, in some implementations, the gaze vector updater430includes a tolerance adjuster436. For example, as illustrated inFIG.2F, while the HMD210is in the second position (associated with the second head forward vector232), the HMD210determines a second gaze vector242based on image data of the eye10. The second gaze vector242is associated with a fourth targeting region that intersects with a portion of the physical painting205, wherein the fourth targeting region is indicated by a fourth reticle244. In some implementations, the HMD210selects the physical painting205based on the intersection.

As illustrated inFIG.2G, the user50initiates a second rightwards rotation of the HMD210from the second position to a third position, as indicated by a second rotation line245. During the second rightwards rotation, the eye10remains focused on the physical painting205. Moreover, during the second rightwards rotation, the HMD210receives positional sensor data indicative of the second rightwards rotation.

As illustrated inFIG.2H, the second rightwards rotation changes the HMD210from the second head forward vector232to a third head forward vector246. Illustration of the second head forward vector232is omitted fromFIGS.2H and2Ifor the sake of clarity.

As described above with reference toFIG.2D, a lag exists between the generation of the positional sensor data (indicative of the second rightwards rotation) and determination of a corresponding movement of the eye10within image data. Based on the lag, the HMD210determines a second unsupplemented gaze vector248, as illustrated inFIG.2H. The second unsupplemented gaze vector248is repositioned (relative to the second gaze vector242) based on the second rightwards rotation. The second unsupplemented gaze vector248is associated with a fifth targeting region that includes a portion of the first physical wall202, but does not include the physical painting205. The fifth targeting region is indicated by a fifth reticle250. Accordingly, the HMD210may cease to select the physical painting205based on the fifth targeting region, even though the eye10maintains focus on the physical painting205.

Accordingly, in some implementations, based on detecting the second rightwards rotation, the tolerance adjuster436increases a targeting tolerance associated with second unsupplemented gaze vector248in order to maintain targeting (e.g., selection) of the physical painting205. In some implementations, the HMD210maintains the position of the unsupplemented gaze vector248, while increasing the targeting tolerance. For example, as illustrated inFIG.2I, the HMD210increases the targeting tolerance associated with second unsupplemented gaze vector248from the fifth targeting region to a sixth targeting region. The sixth targeting region is indicated by a sixth reticle252. In contrast to the fifth targeting region, the sixth targeting region includes the physical painting205. Accordingly, the HMD210is able to maintain targeting (e.g., selecting) of the physical painting205with greater confidence (e.g., more reliably).

In some implementations, the amount of the targeting tolerance increase (e.g., increase of the size of the targeting region) is based on the amount of an angular or lateral change associated with a detected positional change of the HMD210. For example, with reference toFIGS.2G-2IandFIG.4, the motion system432determines an angular change based on the difference between the third head forward vector232and the fourth head forward vector246. Continuing with this example, the amount of increase associated with changing from the fifth targeting region to a sixth targeting region is proportional or directly related to the angular change. In some implementations, the targeting tolerance is based on an amount of angular, lateral, or a combination of angular and lateral change associated with a detected positional change of the HMD210over a preceding window of time. For example, the targeting tolerance may be at a first value during a time in which HMD210remains relatively stationary. The targeting tolerance may temporarily increase to a second, larger value during a time in which HMD210undergoes quick rotational or lateral movement. The targeting tolerance may then lower to the first value after the HMD210returns to a relatively stationary state.

FIG.5is an example of a flow diagram of a method500of updating a gaze vector based on positional sensor data in accordance with some implementations. In various implementations, the method500or portions thereof are performed by an electronic device including image sensor(s) and a positional sensor, such as the HMD210described with reference toFIGS.2A-2I,3, and4. In some implementations, the method500is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method500is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in method500are, optionally, combined and/or the order of some operations is, optionally, changed.

As represented by block502, the method500includes capturing image data of an eye using an image sensor of an electronic device performing the method500. The image data may include a plurality of images of the eye, such as the first image300and the second image320respectively illustrated inFIGS.3A and3B. As one example, the image sensor corresponds to the first image sensor414inFIG.4, wherein the first image sensor414captures image data of the eye10based on light emitted from the light source412onto the eye10.

As represented by block504, the method500includes, while the electronic device is in a first position, determining a gaze vector based on the image data. For example, with reference toFIGS.2B and3A, the HMD210captures a first image300of the eye10, and determines the first gaze vector220based on an offset between the reference line312(running along the center of the eye10) and the first eye position line314(running along the center of the iris310) within the first image300.

Determining the gaze vector may include identifying a position of an object that intersects with the gaze vector. For example, with reference toFIG.2B, the HMD210identifies the position of the computer-generated lamp208with which the gaze vector220intersects (e.g., the first reticle222).

In some implementations, determining the gaze vector includes performing computer vision with respect to the image data, such as instance segmentation or semantic segmentation. For example, as represented by block506, determining the gaze vector includes identifying a first subset of pixels of the image data corresponding to the eye. Determining the gaze vector based on the image data may be associated with a first resource utilization level of the electronic device, such as a first number of processor cycles associated with the corresponding computer vision operation. In some implementations, the method500includes displaying a spatial selection indicator at a first position on the display corresponding to the gaze vector. The spatial selection indicator provides feedback to a user regarding the current gaze of the user.

As represented by block508, in some implementations, the gaze vector is associated with a first targeting tolerance. For example, the first targeting tolerance characterizes the ease with which the gaze vector can target (e.g., select) a particular object. In some implementations, the first targeting tolerance defines a first targeting region associated with the gaze vector. For example, with reference toFIG.2B, the gaze vector220is associated with the first targeting region, which is indicated by the first reticle222. Continuing with this example, in some implementations, the HMD210selects the computer-generated lamp208because the first targeting region includes at least a portion of the computer-generated lamp208.

In some implementations, the first targeting region characterizes a gaze cone, which may be used to determine whether or not to target an object. For example, the vertex of the gaze cone is positioned at an eye of a user, such that the gaze vector runs through the center of the gaze cone. In these implementations, the first targeting tolerance can be represented by the angle formed between the axis and surface of the gaze cone (e.g., larger tolerances correspond to larger angles, and smaller tolerances correspond to smaller angles). The radius of the gaze cone may be a function of a distance between an electronic device and an object. For example, for a given distance between the electronic device and the object, the gaze cone has a corresponding radius that corresponds to the first targeting region. As the distance between the electronic device and the object increases, the radius (and thus the size of the targeting region) increases. As the distance between the electronic device and the object decreases, the radius (and thus the size of the targeting region) decreases.

As represented by block510, the method500includes detecting, based on positional sensor data from a positional sensor, a positional change of the electronic device from the first position to a second position. For example, with reference toFIGS.2C and2D, the HMD210detects the first rightwards rotation of the HMD210, wherein the first rightwards rotation corresponds to an angular change from the first head forward vector216to the second head forward vector232. As another example, with reference toFIGS.2G and2H, the HMD210detects the second rightwards rotation of the HMD210, wherein the second rightwards rotation corresponds to an angular change from the second head forward vector232to the third head forward vector246. In some implementations and with reference toFIG.4, the positional sensor data from the positional sensor450indicates a plurality of positional values, wherein each of the plurality of positional values indicates the current position (e.g., orientation) of the HMD210. Continuing with this example, the motion system432detects a positional change of the HMD210by assessing the plurality of positional values. The positional sensor data may include IMU data, magnetic sensor data (e.g., from a magnetic sensor that measures weak magnetic fields), etc. Examples of the positional change include a rotation, translational movement, shake movement, etc.

As represented by block512, in some implementations, the method500includes determining that the positional change exceeds a motion threshold. For example and with reference toFIG.4, the motion system432determines, based on positional sensor data from the positional sensor450, that the positional change exceeds the motion threshold. Determining the positional change exceeds the motion threshold may include determining more than a threshold amount of angular change of the electronic device.

As represented by block514, the method500includes, in response to detecting the positional change, updating the gaze vector based on the positional sensor data. In some implementations, updating the gaze vector based on the positional sensor data may include using positional sensor data accumulated over a period of time corresponding to a known or estimated latency associated with capturing the image data of the eye of the user (e.g., block502) and processing the image data of the eye of the user (e.g., block504). Updating the gaze vector may be associated with a second resource utilization level of the electronic device, such as a second number of processor cycles associated with processing the positional sensor data. The second resource utilization level may be less than the first resource utilization level, which is associated with determining the gaze vector based on the image data (block504). For example, the method500includes performing computer vision on the image data to determine the gaze vector, which is more computationally expensive than updating the gaze vector based on the positional sensor data. In some implementations, updating the gaze vector is in response to determining that the positional change exceeds the motion threshold (block512). In some implementations, updating the gaze vector occurs before detecting a movement of the eye from the first subset of pixels to a second subset of pixels of the image data. In other words, updating the gaze vector may be independent of the image data of the eye.

As represented by block516, in some implementations, updating the gaze vector includes repositioning the gaze vector based on a positional supplement value. Examples of the positional supplement value includes an angular change, as represented by block518. For example, with reference toFIG.2D, the angular change is the sum of the first angle θ1and the second angle θ2. As another example, before updating the gaze vector, the gaze vector is characterized by an initial angle, and repositioning the gaze vector includes changing the initial angle based on the angular change. As another example, with reference toFIGS.2C-2E, the HMD210repositions the gaze vector220to the repositioned gaze vector238based on the angular change, which corresponds to the angular change resulting from the first rightwards rotation of the HMD210. Continuing with this example, the HMD210detects the first head forward vector216(the first position of the HMD210) and the second head forward vector232(the second position of the HMD210), and assesses the difference thereof in order to determine the angular change. In some implementations, the method500includes, in further response to detecting the positional change, moving the spatial selection indicator to a second position on the display corresponding to the repositioned gaze vector.

In some implementations, repositioning the gaze vector is in further response to determining that the gaze vector satisfies a focus criterion with respect to an object. For example and with reference toFIG.4, the focus system422determines that the gaze vector418intersects with a particular object within the XR environment data408, and further determines that the intersection satisfies the focus criterion. To that end, the focus system422identifies a position of the object that intersects with the gaze vector. The focus criterion may be satisfied when the gaze vector intersects with the object for at least a threshold amount of time. The focus criterion may be satisfied when, after initially determining that the gaze vector intersects with the object, detecting less than a threshold amount of gaze drift. In some implementations, repositioning the gaze vector is further based on the position of the object. For example and with reference toFIG.2B, the HMD210determines, based on the image data of the eye, that the gaze vector220satisfies the focus criterion with respect to the computer-generated lamp208. Continuing with this example and with reference toFIG.2E, the HMD210repositions the gaze vector220such that the repositioned gaze vector240also intersects with the computer-generated lamp208.

As represented by block520, in some implementations, updating the gaze vector includes updating the gaze vector from the first targeting tolerance to a second targeting tolerance. The second targeting tolerance is greater than the first targeting tolerance. To that end, the method500may include determining the second targeting tolerance based on the positional sensor data. The second targeting tolerance may account for anticipated lag associated with computer vision tracking of the eye within image data. For example, in some implementations, the method500includes maintaining the second targeting tolerance until completion of computer vision tracking of a movement of the eye within the image data, at which the method500may include restoring the first targeting tolerance. Accordingly, in some implementations, after updating the gaze vector to the second targeting tolerance, the method500includes continuing to track the eye of the user within the image data. For example, in some implementations, the method500includes detecting a movement of the eye from the first subset of pixels to a second subset of pixels of the image data.

In some implementations, while the gaze vector is associated with the second targeting tolerance, the method500includes determining that the gaze vector intersects with an object based on the image data. As one example and with reference toFIG.2I, after the HMD210has increased the targeting tolerance to the sixth targeting region (indicated by the sixth reticle252), the HMD210identifies the position of the eye within the image data in order to determine that the second unsupplemented gaze vector248intersects with the physical painting205.

As represented by block522, the second targeting tolerance may define a second targeting region associated with the gaze vector, wherein the second targeting region is larger than the first targeting region. For example, the second targeting region has a larger area or volume, as compared with the first targeting region. In some implementations, the size difference between the first and second targeting regions is based on (e.g., proportional to or directly related to) the amount or rate of device movement. As one example and with reference toFIGS.2H and2I, the HMD210increases from the fifth targeting region (indicated by the fifth reticle250) to the sixth targeting region (indicated by the sixth reticle252).

As represented by block524, in some implementations, increasing the targeting region includes modifying a gaze cone. In some implementations, the method500includes targeting (e.g., selecting) an object with which the gaze cone intersects. The gaze cone may be positioned at an eye of a user, wherein the angle of the gaze cone is such that the gaze vector runs through the center of the gaze cone. Moreover, the radius of the gaze cone may vary based on a distance to the object with which the gaze vector intersects. For example, the radius of the gaze cone is larger nearer to the eye of the user (farther from the object). Modifying the gaze cone may include increasing the volume to increase the targeting region. Additionally or alternatively, modifying the gaze cone may include changing the angle of gaze cone (relative to eye of user).

Updating the gaze vector may include repositioning the gaze vector and updating (e.g., increasing) the targeting tolerance associated with the gaze vector.

The present disclosure describes various features, no single one of which is solely responsible for the benefits described herein. It will be understood that various features described herein may be combined, modified, or omitted, as would be apparent to one of ordinary skill. Other combinations and sub-combinations than those specifically described herein will be apparent to one of ordinary skill, and are intended to form a part of this disclosure. Various methods are described herein in connection with various flowchart steps and/or phases. It will be understood that in many cases, certain steps and/or phases may be combined together such that multiple steps and/or phases shown in the flowcharts can be performed as a single step and/or phase. Also, certain steps and/or phases can be broken into additional sub-components to be performed separately. In some instances, the order of the steps and/or phases can be rearranged and certain steps and/or phases may be omitted entirely. Also, the methods described herein are to be understood to be open-ended, such that additional steps and/or phases to those shown and described herein can also be performed.

Some or all of the methods and tasks described herein may be performed and fully automated by a computer system. The computer system may, in some cases, include multiple distinct computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate over a network to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non-transitory computer-readable storage medium or device. The various functions disclosed herein may be implemented in such program instructions, although some or all of the disclosed functions may alternatively be implemented in application-specific circuitry (e.g., ASICs or FPGAs or GP-GPUs) of the computer system. Where the computer system includes multiple computing devices, these devices may be co-located or not co-located. The results of the disclosed methods and tasks may be persistently stored by transforming physical storage devices, such as solid-state memory chips and/or magnetic disks, into a different state.

Various processes defined herein consider the option of obtaining and utilizing a user's personal information. For example, such personal information may be utilized in order to provide an improved privacy screen on an electronic device. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent. As described herein, the user should have knowledge of and control over the use of their personal information.

Personal information will be utilized by appropriate parties only for legitimate and reasonable purposes. Those parties utilizing such information will adhere to privacy policies and practices that are at least in accordance with appropriate laws and regulations. In addition, such policies are to be well-established, user-accessible, and recognized as in compliance with or above governmental/industry standards. Moreover, these parties will not distribute, sell, or otherwise share such information outside of any reasonable and legitimate purposes.

Users may, however, limit the degree to which such parties may access or otherwise obtain personal information. For instance, settings or other preferences may be adjusted such that users can decide whether their personal information can be accessed by various entities. Furthermore, while some features defined herein are described in the context of using personal information, various aspects of these features can be implemented without the need to use such information. As an example, if user preferences, account names, and/or location history are gathered, this information can be obscured or otherwise generalized such that the information does not identify the respective user.

The disclosure is not intended to be limited to the implementations shown herein. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. The teachings of the invention provided herein can be applied to other methods and systems, and are not limited to the methods and systems described above, and elements and acts of the various implementations described above can be combined to provide further implementations. Accordingly, the novel methods and systems described herein may be implemented in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the disclosure. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the disclosure.