FRAME TRACKING SYSTEM FOR A HEAD-MOUNTED DEVICE

An apparatus, system, and method for an eye tracking system for a head-mounted device include a light source, an image sensor, and a frame tracking system. The light source and image sensor may be configured to determine absolute and relative eye orientations. The frame tracking system may be configured to determine the displacement of a head-mounted device frame relative to the face of the user. The frame tracking system may provide displacement data for the displacement of the head-mounted device frame to the eye tracking system to enable the eye tracking system to compensate for frame slippage, frame jostling, or other frame displacement (relative to the face of the user). The frame tracking system may include one or more position sensors disposed on the head-mounted device frame and logic configured to determine displacement data using information from the one or more position sensors.

TECHNICAL FIELD

This disclosure relates generally to head-mounted devices, and in particular to eye tracking systems in head-mounted devices.

BACKGROUND INFORMATION

Eye tracking technology enables head-mounted devices to interact with users based on the users' eye movement or eye orientation. The accuracy of eye tracking systems can be limited by noise introduced into the eye tracking system.

DETAILED DESCRIPTION

In some implementations of the disclosure, the term “near-eye” may be defined as including an element that is configured to be placed within 50 mm of an eye of a user while a near-eye device is being utilized. Therefore, a “near-eye optical element” or a “near-eye system” would include one or more elements configured to be placed within 50 mm of the eye of the user.

In aspects of this disclosure, visible light may be defined as having a wavelength range of approximately 380 nm-700 nm. Non-visible light may be defined as light having wavelengths that are outside the visible light range, such as ultraviolet light and infrared light. Infrared light having a wavelength range of approximately 700 nm-1 mm includes near-infrared light. In aspects of this disclosure, near-infrared light may be defined as having a wavelength range of approximately 700 nm-1.6 μm.

In aspects of this disclosure, the term “transparent” may be defined as having greater than 90% transmission of light. In some aspects, the term “transparent” may be defined as a material having greater than 90% transmission of visible light.

As used herein a frame refers to a head-worn device that carries at least one lens assembly. A frame could refer to devices having spectacle form factor, devices that are helmet based, devices having a virtual reality (VR) form-factor, or devices having an augmented reality (AR) form factor, for example.

Eye-tracking in AR/VR devices can be performed using camera-based technologies, which is known as video oculography. Eye tracking operations may include capturing an image of the eye (e.g., for every frame), identifying relevant regions of interest in the images, and estimating the size and location of the pupil center and cornea center. Estimating the size and location of the pupil center and cornea center may include calibration operations that determine the curvature of the cornea. Eye tracking operations may use the size and location of the pupil center and cornea to determine a gaze vector or a gaze of the user's eye, which may be referred to as the absolute gaze.

Some approaches to eye tracking have drawbacks. Some of the drawbacks may include: 1) high power consumption and latency related to capturing/recording every image frame for processing; 2) a lot of computations are consumed for every frame that drive up power consumption; 3) temporal sampling rate can be limited by the frame rate limitations of the camera; and 4) relative changes in eye-position from a starting point are challenging to estimate if the entire pipeline of eye tracking operations are executed.

Even with advances in eye tracking systems, frame slippage, frame movement, or other frame displacement may impact the accuracy of the eye tracking system. Embodiments of a frame tracking system are disclosed herein that may be used to determine head-mounted device frame displacement and to provide the displacement data to the eye tracking system, which may disambiguate eye movement from frame movement.

The frame tracking system may include one or more position sensors and processing logic, in accordance with aspects of the disclosure. The one or more position sensors may be coupled to various locations on a head-mounted device frame (e.g., near a user's temple) and configured to detect displacement of the head-mounted device frame relative to the head of a user. The one or more position sensors may include a low-power and high-speed sensor for tracking the movement of the frame with respect to the head. The one or more positions sensors may include relative position sensors, absolute position sensors, references sensors, proximity sensors, inertial measurement units (IMUs), laser speckle interferometry with optic flow, and/or optic flow without laser, according to various implementations. The processing logic may be coupled to the one or more position sensors to receive displacement data, and the processing logic may be configured to determine a quantity of displacement of the head-mounted device frame relative to the head of the user. Various aspects of embodiments of the frame tracking system and eye tracking system are further detailed below.

The apparatus, system, and method for frame tracking that are described in this disclosure may enable improvements in eye tracking technologies, for example, to support operations of a head-mounted device. These and other embodiments are described in more detail in connection withFIGS.1A-5.

FIGS.1A,1B, and1Cillustrate an example of a head mounted device100that is configured to use frame tracking to improve eye tracking accuracy, in accordance with aspects of the disclosure. Head-mounted device100includes a frame tracking system102and an eye tracking system104that are coupled to a frame106(inclusive of an arm108), according to an embodiment. By using frame tracking system102to determine the displacement of frame106relative to a user's face110, eye tracking system104can identify and compensate for noise in eye tracking system104, according to an embodiment. Because eye tracking system104may be configured to use changes in eye orientation to perform eye tracking, displacement of frame106relative to user's face110may be inadvertently interpreted as a change in orientation of an eye112, even if eye112happened to be fixed. By ignoring or compensating for displacement of frame106relative to user's face110, eye tracking system104may improve the accuracy of eye tracking, according to an embodiment. A head-mounted device, such as head-mounted device100, is one type of smart device. In some contexts, head-mounted device100is also a head-mounted display (HMD) that is configured to provide artificial reality. Artificial reality is a form of reality that has been adjusted in some manner before presentation to the user, which may include, e.g., virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and/or derivative thereof.

Frame tracking system102includes a position sensor114, a reference sensor116, and a position sensor118, for determining a position or displacement of frame106, according to an embodiment. Position sensor114may be configured to determine a quantity of displacement experienced by arm108along the y-axis or z-axis, and the displacement of arm108may correspond with a displacement of frame106, according to an embodiment. Position sensor114may use a light source and an image sensor to determine position or displacement along at least two axes. Position sensor114may include a gyroscope and/or an accelerometer to determine an absolute displacement of arm108or head-mounted device100. Reference sensor116may include a gyroscope and/or an accelerometer to determine an absolute displacement of a user's head120. A difference between the displacement measured by position sensor114and reference sensor116may be indicative of displacement of frame106relative to user's face110, according to an embodiment. If the displacement measured by position sensor114and reference sensor116are similar, the measurements may be indicative of unified movement of frame106and user's head (or face)120, according to an embodiment. Position sensor118may be implemented as a proximity sensor that detects position or displacement of frame106relative to a bridge of the user's nose124. When implemented as a proximity sensor, position sensor118may provide displacement data for frame106along a z-axis, according to an embodiment. Position sensor118may also be configured to provide displacement data along the x, y, and z axes by including, for example, an optical flow sensor and/or an IMU. Displacement data from position sensor114, reference sensor116, and position sensor118may be individually used or may be combined and provided to eye tracking system104or controller140to enable eye tracking system104to account for displacement of frame106relative to user's face110, according to an embodiment. For illustration purposes, position sensor114, reference sensor116, and position sensor118are shown positioned in particular locations on arm108and frame106, but it is to be understood that these sensors could perform their function at a number of various locations on arm108and frame106. Additionally, frame tracking system102may include multiple position sensors and proximity sensors positioned at various locations on frame106(inclusive of arm108). Frame tracking system102may also include a controller or control logic (e.g., controller140, processing logic142).

Eye tracking system104may include at least one absolute eye orientation sensor126, at least one relative eye orientation sensor128, and a number of light sources130, according to an embodiment. Components of eye tracking system104may be located on a bottom of frame106(e.g., near the cheek bond), along a side of frame106, or on arm108(e.g., near the side of eye112), according to various embodiments. Absolute eye orientation sensor126may be implemented as an image sensor configured to capture an image of at least a portion of eye112and determine an orientation of eye112based on the image. The image may be a relatively high-resolution image, which may consume more resources (time, battery, processing power) than a low-resolution image capture. Absolute eye orientation sensor126may be configured to be operated less frequently (e.g., twice a second) than relative eye orientation sensor128. Absolute eye orientation sensor126may be implemented as a high resolution, high accuracy, low precision, and low frame rate sensor configured to image the eye and determine the absolution position of the eye and the gaze at intermittent time intervals.

Relative eye orientation sensor128may be implemented as a single sensor or an array of simple sensors configured to capture sparse signals from eye112, according to an embodiment. Sparse signals includes signals resulting from merging input from various sensor sources. Sparse signals may be read and transmitted frequently while consuming less resources (e.g., power, bandwidth, processing, time) than data from absolute eye orientation sensor126, according to an embodiment. Sparse signals may be used to infer the change in orientation of eye112from one instant in time to the next instant in time, without generating a traditional 2D image. These sensors might be optical or could also be non-optical sensors. Relative eye orientation sensor128may be implemented as a photodetector, an ultrasonic sensor, capacitive sensor, electrooculography (EOG), or some other optical or non-optical sensors. The sparse signals may be used to infer a change in orientation of eye112using models (e.g., predictive models, machine learning models, etc.). Since the output of relative eye orientation sensor128is sparse and frequently updated, absolute eye orientation sensor126may be periodically used to compensate for any drift (e.g., gradually accumulated error) that may occur while using relative eye orientation sensor128to track eye112. Relative eye orientation sensor128may be configured to capture sparse signals more frequently (e.g., 4000 times a second) than absolute eye orientation sensor126. Relative eye orientation sensor128may be implemented as a low-weight, small-sized, low-power, high-precision, low-accuracy, and low-cost sensor that is configured to measure the relative change in position of the eye over time. In other words, relative eye orientation sensor128may be implemented as a SWAP-C (size, weight, power, cost) optimized sensor. A compute platform, e.g., controller140, is configured to receive data from the two different types of sensors (i.e., absolute and relative orientation sensors) and merge these data streams together to provide a high precision, high accuracy signal about where the eye is oriented at all times, at a higher temporal resolution than can be achieved (e.g., reasonably implemented in a head-mounted device) by an absolute eye orientation sensor alone, according to an embodiment.

Absolute eye orientation sensor126and relative eye orientation sensor128may operate on specular reflections and diffuse scattering of light that are provided by light sources130. Light sources130may emit light in the non-visible light spectrum (e.g., infrared). For example, light sources130are configured to emit infrared light, for example, having a wavelength in the range of 750 nm to 1500 nm, according to an embodiment. Light sources130may be implemented as light emitting diodes (LEDs), vertical-cavity surface-emitting lasers (VCSELs), edge-emitting laser (EEL), micro light emitting diode (micro-LED), an edge emitting LED, a superluminescent diode (SLED), or another type of light source. In one embodiment, light emitted from light sources130is infrared light centered around 850 nm.

Referring toFIG.1C, head-mounted device100may be a type of device that is typically worn on the head of a user to provide artificial reality content to the user. Artificial reality is a form of reality that has been adjusted in some manner before presentation to the user, which may include, e.g., virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and/or derivative thereof.

Head-mounted device100may have multiple arms and multiple position sensors. Arm108may include arm108A and arm108B. Position sensor114may include position sensor114A coupled to arm108A and may include sensor114B coupled to arm108B. Additional position sensors may be disposed on various portions of frame106and arm108A/108B.

Head-mounted device100may include a lens assembly132(individually, lens assembly132A and132B). Lens assembly132is mounted to, inserted into, or otherwise carried by frame106. Lens assembly132may include a prescription optical layer matched to a particular user of head-mounted device100or may be a non-prescription lens. Lens assembly132may include a waveguide134(individually, waveguide134A and134B) and a projector136(individually, projector136A and136B) configured to display information to a user during operation of head-mounted device100. Waveguide134may be included in one of a number of optical layers of lens assembly132, or waveguide134may be integrated into, for example, a single optical layer that defines lens assembly132. Projector136may be positioned at least partially in or on frame106and may be optically coupled to waveguide134. Lens assembly132may appear transparent to a user to facilitate augmented reality or mixed reality and to enable a user to view scene light from the environment around her while also receiving image light directed to her eye(s). Consequently, lens assembly132may be considered (or include) an optical combiner. Lens assembly132may include two or more optical layers. In some embodiments, display light from one or more integrated waveguide displays is directed into one or both eyes of the wearer of head-mounted device100. The illustrated head-mounted device100is configured to be worn on or about a head of a wearer of head-mounted device100.

Head-mounted device100may include one or more outward facing cameras138that may be positioned on frame106or on arm108A/108B. Outward facing cameras138may be configured to image surroundings of head-mounted device100, and head-mounted device100may be configured to use the images to customize user interface options for a user.

Head-mounted device100includes a controller140communicatively coupled to the various electronics carried by head-mounted device100, according to an embodiment. Controller140may be configured to operate frame tracking system102and eye tracking system104. Controller140may include processing logic142and one or more memories144to analyze image data received from one or more of absolute eye orientation sensor126, relative eye orientation sensor128, position sensor114A/114B, reference sensor116, position sensor118, and cameras138, to determine an orientation of one or more of a user's eyes, to perform one or more frame tracking operations, to perform one or more eye tracking operations, and/or to display or provide user interface elements in lens assembly132, according to an embodiment. Controller140may include a wired and/or wireless data interface for sending and receiving data and graphic processors and may use one or more memories144for storing data and computer-executable instructions. Controller140and/or processing logic142may include circuitry, logic, instructions stored in a machine-readable storage medium, ASIC circuitry, FPGA circuitry, and/or one or more processors. In one embodiment, head-mounted device100may be configured to receive wired power. In one embodiment, head-mounted device100is configured to be powered by one or more batteries. In one embodiment, head-mounted device100may be configured to receive wired data including video data via a wired communication channel. In one embodiment, head-mounted device100is configured to receive wireless data including video data via a wireless communication channel.

FIG.2illustrates an example diagram of a tracking system environment200, in accordance with aspects of the disclosure. Tracking system environment200includes a frame tracking system202and an eye tracking system204, according to an embodiment. Frame tracking system202may be configured to determine a displacement of a head-mounted device frame and provide displacement data to eye tracking system204. Eye tracking system204may be configured to use the displacement data with other eye tracking information to determine an eye orientation of a user of a head-mounted device. Frame tracking system202is an example implementation of frame tracking system102(shown inFIG.1A), and eye tracking system204is an example implementation of eye tracking system104(shown inFIG.1B), according to an embodiment.

Frame tracking system202may be configured to use one or more of a variety of sensors to determine displacement data for a head-mounted device frame, according to an embodiment. For example, frame tracking system202may interact with or may include a position sensor206, a reference sensor208, and a proximity sensor210, to determine position or displacement of the head-mounted device frame. Position sensor206may include a light source212, an image sensor214, a gyroscope216, and an accelerometer218. Position sensor206may use the various components to generate position data220, which may be absolute or relative position data. Reference sensor208may include a speaker222(e.g., an earbud), a communication sensor224(e.g., Bluetooth, WiFi, etc.), a gyroscope226, and an accelerometer228, according to an embodiment. Reference sensor208may use the various components to generate reference data230, which may be position or displacement data associated with a particular part of the user's body (e.g., ear, neck, etc.), according to an embodiment. Proximity sensor210may include a light source232and an image sensor234, which may be used to generate distance data236, according to an embodiment. Distance data236may represent a distance between the head-mounted device frame and a particular portion of a user's face (e.g., the bridge of a nose, a forehead, a cheekbone, etc.).

In operation, frame tracking system202may include or may progress through a number of operation blocks to acquire and utilize position data220, reference data230, and/or distance data236, in accordance with aspects of the disclosure. The operation blocks may be performed in parallel or in an order other than the described order.

At operation block238, frame tracking system202may request position data220, reference data230, and/or distance data236. Requesting the various data may include establishing communication channels that are wired or wireless to communicate with position sensor206, reference sensor208, and/or proximity sensor210. Operation block238may proceed to operation block240, according to an embodiment.

At operation block240, frame tracking system202may receive position data220, reference data230, and/or distance data236, according to an embodiment. Operation block240may proceed to operation block242, according to an embodiment.

At operation block242, frame tracking system202may use position data220, reference data230, and/or distance data236to determine displacement data244. Displacement data244represents a displacement of a head-mounted device frame in one, two, or three axes (e.g., x, y, z-axes), according to an embodiment. Operation block242may proceed to operation block243, according to an embodiment.

At operation block243, frame tracking system202may provide displacement data244to eye tracking system204. Frame tracking system202may provide displacement data244to eye tracking system204to support and improve accuracy in eye tracking functionality in a head-mounted device, for example.

Frame tracking system202may also include memory246and logic248. Memory246may be used to store computer readable instructions that are associated with operations of frame tracking system202, and logic248may be configured to execute the instructions to support operation of frame tracking system202, according to an embodiment.

Eye tracking system204may include a light source250(e.g., LED, VCSEL, laser, etc.) and an eye orientation sensor252to support eye tracking functionality, according to an embodiment. Light source250may be configured to illuminate an eyebox region with, for example, non-visible light. Light source250may include a number of light sources disposed in various locations on or around a head-mounted device frame. Eye orientation sensor252may be configured to provide sensor data254, which may be representative of ultrasonic data, photodetector data, capacitive data, and/or an image of reflections from light source250from an eyebox region (e.g., an eye of a user) of a head-mounted device. As used herein an eyebox region is generally the area and volume where a user's eyes may be positioned while wearing a head-mounted device.

Eye tracking system204may be configured to perform a number of operations to determine an eye orientation based on displacement data244and other eye tracking information.

At operation block256, eye tracking system204may illuminate an eyebox region. Illuminating the eyebox region may include providing a pattern, a particular frequency, or other control signals to light source250to cause light source250to illuminate the eyebox region. Operation block256proceeds to operation block258, according to an embodiment.

At operation block258, eye tracking system204may receive reflections from the eyebox region, according to an embodiment. The reflections may be received by eye orientation sensor252, a photodetector, or some non-optical sensor. Eye orientation sensor252may represent more than one sensor and may represent, for example, an absolute eye orientation sensor and a relative eye orientation sensor. Reflections from the eyebox region may also include ultrasonic reflections, or reflections that may be detected by a capacitive sensor. Operation block258proceeds to operation block260, according to an embodiment.

At operation block260, eye tracking system204may capture sensor data from the eyebox region. Capturing sensor data may include receiving sensor data254from eye orientation sensor252, according to an embodiment. Eye orientation sensor252may represent an image sensor used as an absolute eye orientation sensor and may include one or more relative eye orientation sensors (e.g., photodetector, ultrasonic sensor, capacitive sensor, or other sparse signal sensor). Operation block260may proceed to operation block262, according to an embodiment.

At operation block262, eye tracking system204may determine facial expressions from sensor data254, according to an embodiment. Facial expressions may be determined by, for example, a machine learning model or other pattern recognition techniques applied to one or more images of the eyebox region or image of the facial features surrounding the eyebox region. Eye tracking system204and/or eye orientation sensor252may include one or more cameras that are positioned on the head-mounted device and configured to capture portions of the user's face in order to determine facial expressions. Because facial expressions can displace the head-mounted device frame (while an eye orientation is relatively fixed), facial expressions can be considered noise in an eye tracking algorithm or operation. Facial expressions can include raised eyebrows, a scrunched nose, squinted eyes, a wink, a contraction of face muscles (e.g., as done in a sneeze), a smile, or similar expressions. Facial expressions may be used by eye tracking system204to identify and reduce noise from an eye orientation signal, to improve the accuracy of eye tracking system204, according to an embodiment. Operation block262may proceed to operation block264, according to an embodiment.

At operation block264, eye tracking system204may determine an eye orientation based on displacement data244, facial expressions, and/or sensor data254, according to an embodiment. Eye tracking system204may be configured to subtract displacement data244from eye orientation measurements from sensor data254. Eye tracking system204may be configured to reduce or ignore eye orientation movement or measurements based on whether one or more facial expressions were detected.

Eye tracking system204may include memory266and logic268that are configured to store and execute various instructions to perform operations of eye tracking system204. Memory266may be at least partially shared with memory246, and logic268may be at least partially shared with logic248, according to an embodiment. Frame tracking system202may be a subsystem of eye tracking system204, or frame tracking system202may be configured to operate independent of eye tracking system204, according to various embodiments.

FIGS.3A,3B,3C, and3Dillustrate example diagrams of sensor configurations that may be used to determine head-mounted device frame displacement, in accordance with aspects of the disclosure. One or more of the sensor configurations may be disposed at a single location or at several locations around a head-mounted device frame to determine displacement data for the head-mounted device frame at one or multiple locations. Various features of the position sensors may be combined with one or more other position sensors.

FIG.3Aillustrates a sensor configuration300that may be used to determine a displacement distance for a head-mounted device frame302, in accordance with aspects of the disclosure. Head-mounted device frame302is representative of a portion of a head-mounted device frame (inclusive of head-mounted device arms, a helmet, or other cranial attachment implement). Sensor configuration300may include a position sensor304that is coupled to or partially integrated into head-mounted device frame302. Position sensor304may be configured to determine a displacement distance for head-mounted device frame302based on optic flow of a pattern. Optic flow of a pattern may refer to the change of position of a pattern within an image, with respect to time. Position sensor304may include a light source306configured to illuminate a patch of skin308. Light source306may be an LED, a VCSEL, a micro-LED, an edge-emitting LED, a SLED, or another type of light source. Light source306may illuminate patch of skin308with a light pattern, and the light pattern may include fringes patterns, dots, grids, lines, concentric circles, or other shapes. Position sensor304may include an image sensor310configured to generate an image based on reflections from patch of skin308. The image may include a pattern312of hair follicles314or of the emitted light pattern. Position sensor304may determine a displacement of head-mounted device frame302based on how much pattern312shifts in any particular direction. In one embodiment, position sensor304is used to measure a depth map change as a function of time. In one embodiment, a simultaneous localization and mapping (SLAM) sensor may be used to calculate the movement of head-mounted device frame302. In one embodiment, position sensor304may be configured as a time-of-flight (ToF) sensor that is configured to measure, for example, absolute distance to a user's face or head.

FIG.3Billustrates a sensor configuration320that may be used to determine a displacement distance for a head-mounted device frame322, in accordance with aspects of the disclosure. Sensor configuration320may include a position sensor324that is coupled to or partially integrated into head-mounted device frame322. Position sensor324may be configured to determine a displacement distance for head-mounted device frame322based on laser speckle interferometry. Position sensor324may include a light source326configured to illuminate a patch of skin328. Light source326may include a laser. The laser may be a long-coherence-length laser configured to provide light having electromagnetic wave propagation that are in phase in space and time. Position sensor324may condition the laser light using a beam-shaping optics, such as a lens, a grating, or a prism configured to change the far field light distribution from the laser source. Position sensor324may include an image sensor330configured to generate an image based on reflections from patch of skin334. Image sensor330may be a single pixel sensor or may include a two-dimensional array of pixels. The image may include a speckle pattern332caused by constructive and destructive interference of reflections of the laser light. Position sensor324may determine displacement of head-mounted device frame322based on how much speckle pattern332shifts in any particular direction when subsequently captured images are compared. An advantage of laser speckle interferometry is that the technique may enable high-resolution tracking of surfaces, as compared to other types of light sources. Position sensor324may be implemented as a ToF sensor, as a light detection and ranging (LIDAR) sensor, as a frequency modulated continuous wave (FMCW) LIDAR sensor, or as a, optical coherence tomography (OCT) sensor, according to various embodiments.

FIG.3Cillustrates a sensor configuration340that may be used to determine a displacement distance for head-mounted device frame342, in accordance with aspects of the disclosure. Sensor configuration340may include a position sensor344that is coupled to or partially integrated into head-mounted device frame342. Position sensor344may be configured to determine a displacement distance for head-mounted device frame342based on relative distance displacement between position sensor344and a reference sensor346. Position sensor344may include at least one gyroscope348and at least one accelerometer350. Gyroscope348and accelerometer350may operate together as an inertial measurement unit (IMU) to generate and provide displacement data352. Reference sensor346may include at least one gyroscope354, at least one accelerometer356, and a communications sensor358(e.g., Bluetooth). Gyroscope354and accelerometer356may operate together as an IMU to generate and provide reference (displacement) data360. Reference data360may be provided using communications sensor358. Sensor configuration340may include a controller362that is configured to receive displacement data352and reference data360. Controller362may be configured to use the difference between displacement data352and reference data360to determine a relative displacement of head-mounted device frame342with respect to a user's face or head. Reference sensor346may be implemented as an earpiece (e.g., an earbud) that a user may use to receive or provide audio information.

FIG.3Dillustrates a sensor configuration380that may be used to determine a displacement distance for head-mounted device frame342, in accordance with aspects of the disclosure. Sensor configuration380may include a reference sensor364. Reference sensor364may be implemented as a neckpiece (e.g., a collar or necklace) that a user may wear ornamentally, for example, on or around the user's neck. Reference sensor364may include at least one gyroscope366, at least one accelerometer368, and a communications sensor370(e.g., Bluetooth). Gyroscope366and accelerometer368may operate together as an IMU to generate and provide reference (displacement) data372. Reference data372may be provided using communications sensor370. Sensor configuration380may include a controller374that is configured to receive displacement data352and reference data372. Controller374may be configured to use the difference between displacement data352and reference data372to determine a relative displacement of head-mounted device frame342with respect to a user's face or head.

FIG.4illustrates an example timing diagram400that shows when various sensors may be read, in accordance with aspects of the disclosure. Timing diagram400may have a number of waveforms that correspond with operating or reading data from various sensors. Timing diagram400may include, for example, a waveform402, a waveform404, and a waveform406. Waveform402may correspond with an absolute eye orientation sensor read (or operation). Waveform404may correspond with a relative eye orientation sensor read. Waveform406may correspond with a relative frame position sensor read. As shown, an absolute eye orientation sensor may be read at different intervals (e.g., less frequently) than a relative eye orientation sensor. An absolute eye orientation sensor may also be read at different intervals (e.g., less frequently) than a relative frame orientation sensor. Each of the various types of position sensors (e.g., eye, frame, position, reference, and/or proximity sensors) may be read at different intervals, or some of the various types of position sensors may be read at similar intervals (e.g., the relative eye orientation sensor and the relative frame position sensor), according to various embodiments.

FIG.5illustrates a flow diagram of process500for eye tracking for a head-mounted device, in accordance with aspects of the disclosure. The order in which some or all of the process or operation blocks appear in process500should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the operation blocks may be executed in a variety of orders not illustrated, or even in parallel.

At operation block502, process500may determine, using one or more position sensors, a displacement of a head-mounted device frame relative to a portion of a user's head, according to an embodiment. Operation block502may proceed to operation block504, according to an embodiment.

At operation block504, process500may determine, using one or more eye orientation sensors, an orientation of an eye of the user with respect to the head-mounted device frame, according to an embodiment. Operation block504may proceed to operation block506, according to an embodiment.

At operation block506, process500may adjust the determined orientation of the eye of the user based on the displacement of a head-mounted device frame, according to an embodiment. Process500may include adjusting the determined orientation of the user's eye based on identified facial expressions, which may also introduce noise into eye orientation determinations. The head-mounted device frame displacement distance may be determined based on one or more of a frame position sensor, a frame reference sensor (e.g., an earbud, a collar, a headband, or another wearable sensor), and a proximity sensor. Accounting and compensating for displacement or movement of the head-mounted device frame may be used to improve the accuracy of eye tracking systems.

The term “processing logic” (e.g., controller140, processing logic142) in this disclosure may include one or more processors, microprocessors, multi-core processors, Application-specific integrated circuits (ASIC), and/or Field Programmable Gate Arrays (FPGAs) to execute operations disclosed herein. In some embodiments, memories (not illustrated) are integrated into the processing logic to store instructions to execute operations and/or store data. Processing logic may also include analog or digital circuitry to perform the operations in accordance with embodiments of the disclosure.

A “memory” or “memories” (e.g., memory144) described in this disclosure may include one or more volatile or non-volatile memory architectures. The “memory” or “memories” may be removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Example memory technologies may include RAM, ROM, EEPROM, flash memory, CD-ROM, digital versatile disks (DVD), high-definition multimedia/data storage disks, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device.

A network may include any network or network system such as, but not limited to, the following: a peer-to-peer network; a Local Area Network (LAN); a Wide Area Network (WAN); a public network, such as the Internet; a private network; a cellular network; a wireless network; a wired network; a wireless and wired combination network; and a satellite network.

Communication channels may include or be routed through one or more wired or wireless communication utilizing IEEE 802.11 protocols, short-range wireless protocols, SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), USB (Universal Serial Port), CAN (Controller Area Network), cellular data protocols (e.g. 3G, 4G, LTE, 5G), optical communication networks, Internet Service Providers (ISPs), a peer-to-peer network, a Local Area Network (LAN), a Wide Area Network (WAN), a public network (e.g. “the Internet”), a private network, a satellite network, or otherwise.

A computing device may include a desktop computer, a laptop computer, a tablet, a phablet, a smartphone, a feature phone, a server computer, or otherwise. A server computer may be located remotely in a data center or be stored locally.