Eye data and operation of head mounted device

A method of operating a head mounted device includes capturing eye data with one or more sensors of the head mounted device. The one or more sensors are configured to sense an eyebox region. Operations of the head mounted device are adjusted in response to the eye data.

TECHNICAL FIELD

This disclosure relates generally to optics, and in particular to head mounted devices.

BACKGROUND INFORMATION

A head mounted device is a wearable electronic device, typically worn on the head of a user. Head mounted devices may include one or more electronic components for use in a variety of applications, such as gaming, aviation, engineering, medicine, entertainment, activity tracking, and so on. Head mounted devices may include display to present virtual images to a wearer of the head mounted device. When a head mounted device includes a display, it may be referred to as a head mounted display. Head mounted devices may have user inputs so that a user can control one or more operations of the head mounted device.

DETAILED DESCRIPTION

In some implementations of the disclosure, the term “near-eye” may be defined as including an element that is configured to be placed within 50 mm of an eye of a user while a near-eye device is being utilized. Therefore, a “near-eye optical element” or a “near-eye system” would include one or more elements configured to be placed within 50 mm of the eye of the user.

In aspects of this disclosure, visible light may be defined as having a wavelength range of approximately 380 nm – 700 nm. Non-visible light may be defined as light having wavelengths that are outside the visible light range, such as ultraviolet light and infrared light. Infrared light having a wavelength range of approximately 700 nm – 1 mm includes near-infrared light. In aspects of this disclosure, near-infrared light may be defined as having a wavelength range of approximately 700 nm - 1.6 µm.

In aspects of this disclosure, the term “transparent” may be defined as having greater than 90% transmission of light. In some aspects, the term “transparent” may be defined as a material having greater than 90% transmission of visible light.

Implementations of devices, systems, and methods of operating a head mounted device in response to eye data are disclosed herein. Eye data of a user of a head mounted device may be captured by sensors of the head mounted device. The sensors may include image sensors, photodiodes, micro-electro-mechanical systems (MEMS) mirrors, ultrasound, or LIDAR units, for example. The eye data may include one or more images of the eye, a position of the eye, a measurement of the eye (e.g. pupil size), and/or a measurement of the eye over time (e.g. speed of pupil dilation). The eye data may also include a movement of eyebrows, movement of eyelids, and/or facial micro gestures. Other examples of eye data will be disclosed in more detail below.

In an implementation of the disclosure, a transparency of a lens of a head mounted device is adjusted in response to eye data from an eyebox region of the head mounted device. By way of example, eye data may indicate that a user is squinting and a transparency of a lens of the head mounted device may be adjusted (e.g. darkened or lightened) in response to the eye data so that user is more comfortable in viewing scene light from the outside world. If the head mounted device includes a display (e.g. augmented reality head mounted display), darkening the transparency of the lens in the head mounted device may also allow the user to view virtual images generated by the display. The transparency of the lens may also be adjusted in response to an ambient light reading by a photodetector of the head mounted device. In some implementations, the transparency of the lens may be adjusted according to previously selected lens transparency settings selected by the user in similar environmental conditions (e.g. similar previous ambient light measurements). In some implementations, the transparency of the lens may be adjusted according to a predetermined lens transparency value. The predetermined lens transparency value may be derived from crowd-sourced or aggregate eye data corresponding to a particular ambient light measurement value, for example.

In an implementation of the disclosure, a display brightness of a head mounted display (e.g. augmented reality head mounted device) is adjusted in response to eye data. By way of example, eye data may indicate that a user is squinting and a display brightness of the head mounted device may be adjusted (e.g. dimmed or brightened) in response to the eye data so that user is more comfortable in viewing the display. The brightness of the display may also be adjusted in response to an ambient light reading by a photodetector of the head mounted device. In some implementations, the brightness of the display may be adjusted according to a previously selected display brightness value that was previously selected by the user in similar environmental conditions (e.g. similar previous ambient light measurements). In some implementations, the brightness of the display may be adjusted according to a predetermined display brightness value. The predetermined display brightness value may be derived from crowd-sourced or aggregate eye data corresponding to a particular ambient light measurement value, for example.

In an implementation of the disclosure, a volume of an audio output of a head mounted display (e.g. augmented reality head mounted device) is adjusted in response to eye data. By way of example, eye data may indicate that a user is uncomfortable with a particular audio output level and/or a change in an audio output level. Therefore, a volume of an audio output may be adjusted (e.g. volume up or volume down) in response to the eye data so that user is more comfortable with the audio level. The volume of the audio output may also be adjusted in response to an ambient noise measurement reading by a microphone of the head mounted device. A user comfort with a particular audio output level may depend on the noise level in a user environment. For example, the user may be comfortable with a higher audio output level in a high-noise environment (e.g. an airplane) whereas the user may be uncomfortable with the same higher audio output level in a low-noise environment (e.g. a library). These and other embodiments are described in more detail in connection withFIGS.1-7.

FIG.1illustrates an example head mounted device100, in accordance with aspects of the present disclosure. The illustrated example of head mounted device100is shown as including a frame102, temple arms104A and104B, and near-eye optical elements110A and110B. Cameras108A and108B are shown as coupled to temple arms104A and104B, respectively. Cameras108A and108B may be configured to image an eyebox region to image the eye of the user to capture eye data of the user. Cameras108A and108B may image the eyebox region directly or indirectly. For example, optical elements110A and/or110B may have an optical combiner that is configured to redirect light from the eyebox to the cameras108A and/or108B. In some implementations, near-infrared light sources (e.g. LEDs or vertical-cavity side emitting lasers) illuminate the eyebox region with near-infrared illumination light and cameras108A and/or108B are configured to capture infrared images. Cameras108A and/or108B may include complementary metal-oxide semiconductor (CMOS) image sensor. A near-infrared filter that receives a narrow-band near-infrared wavelength may be placed over the image sensor so it is sensitive to the narrow-band near-infrared wavelength while rejecting visible light and wavelengths outside the narrow-band. The near-infrared light sources (not illustrated) may emit the narrow-band wavelength that is passed by the near-infrared filters.

In addition to image sensors, various other sensors of head mounted device100may be configured to capture eye data. Ultrasound or LIDAR chips may be configured in frame102to detect a position of an eye of the user by detecting the position of the cornea of the eye, for example. Discrete photodiodes included in frame102or optical elements110A and/or110B may also be used to detect a position of the eye of the user. Discrete photodiodes may be used to detect “glints” of light reflecting off of the eye, for example. Eye data generated by various sensors may not necessarily be considered “images” of the eye.

FIG.1also illustrates an exploded view of an example of near-eye optical element110A. Near-eye optical element110A is shown as including an optically transparent layer120A, an illumination layer130A, a display layer140A, and a transparency modulator layer150A. Display layer140A may include a waveguide148A that is configured to direct virtual images included in visible image light141to an eye of a user of head mounted device100that is in an eyebox region of head mounted device100. In some implementations, at least a portion of the electronic display of display layer140A is included in the frame102of head mounted device100. The electronic display may include an LCD, an organic light emitting diode (OLED) display, micro-LED display, pico-projector, or liquid crystal on silicon (LCOS) display for generating the image light141.

When head mounted device100includes a display, it may be considered a head mounted display. Head mounted device100may be considered an augmented reality (AR) head mounted display. WhileFIG.1illustrates a head mounted device100configured for augmented reality (AR) or mixed reality (MR) contexts, the disclosed embodiments may also be used in other implementations of a head mounted display such as virtual reality head mounted displays. Additionally, some implementations of the disclosure may be used in a head mounted device that do not include a display.

Illumination layer130A is shown as including a plurality of in-field illuminators126. In-field illuminators126are described as “in-field” because they are in a field of view (FOV) of a user of the head mounted device100. In-field illuminators126may be in a same FOV that a user views a display of the head mounted device100, in an embodiment. In-field illuminators126may be in a same FOV that a user views an external environment of the head mounted device100via scene light191propagating through near-eye optical elements110. Scene light191is from the external environment of head mounted device100. While in-field illuminators126may introduce minor occlusions into the near-eye optical element110A, the in-field illuminators126, as well as their corresponding electrical routing may be so small as to be unnoticeable or insignificant to a wearer of head mounted device100. In some implementations, illuminators126are not in-field. Rather, illuminators126could be out-of-field in some implementations.

As shown inFIG.1, frame102is coupled to temple arms104A and104B for securing the head mounted device100to the head of a user. Example head mounted device100may also include supporting hardware incorporated into the frame102and/or temple arms104A and104B. The hardware of head mounted device100may include any of processing logic, wired and/or wireless data interface for sending and receiving data, graphic processors, and one or more memories for storing data and computer-executable instructions. In one example, head mounted device100may be configured to receive wired power and/or may be configured to be powered by one or more batteries. In addition, head mounted device100may be configured to receive wired and/or wireless data including video data.

FIG.1illustrates near-eye optical elements110A and110B that are configured to be mounted to the frame102. In some examples, near-eye optical elements110A and110B may appear transparent or semi-transparent to the user to facilitate augmented reality or mixed reality such that the user can view visible scene light from the environment while also receiving image light141directed to their eye(s) by way of display layer140A. In further examples, some or all of near-eye optical elements110A and110B may be incorporated into a virtual reality headset where the transparent nature of the near-eye optical elements110A and110B allows the user to view an electronic display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or micro-LED display, etc.) incorporated in the virtual reality headset.

As shown inFIG.1, illumination layer130A includes a plurality of in-field illuminators126. Each in-field illuminator126may be disposed on a transparent substrate and may be configured to emit light to an eyebox region on an eyeward side109of the near-eye optical element110A. In some aspects of the disclosure, the in-field illuminators126are configured to emit near infrared light (e.g. 750 nm - 1.6 µm). Each in-field illuminator126may be a micro light emitting diode (micro-LED), an edge emitting LED, a vertical cavity surface emitting laser (VCSEL) diode, or a Superluminescent diode (SLED).

Optically transparent layer120A is shown as being disposed between the illumination layer130A and the eyeward side109of the near-eye optical element110A. The optically transparent layer120A may receive the infrared illumination light emitted by the illumination layer130A and pass the infrared illumination light to illuminate the eye of the user. As mentioned above, the optically transparent layer120A may also be transparent to visible light, such as scene light191received from the environment and/or image light141received from the display layer140A. In some examples, the optically transparent layer120A has a curvature for focusing light (e.g., display light and/or scene light) to the eye of the user. Thus, the optically transparent layer120A, in some examples, may be referred to as a lens. In some aspects, the optically transparent layer120A has a thickness and/or curvature that corresponds to the specifications of a user. In other words, the optically transparent layer120A may be a prescription lens. However, in other examples, the optically transparent layer120A may be a non-prescription lens.

FIGS.2A and2Billustrate an eye203in an eyebox region201, in accordance with implementations of the disclosure.FIG.2Aillustrates eye203that is open andFIG.2Billustrates eye203shut. Eye data of eye203may include a position of eye203, a measurement of the eye203(e.g. pupil size), and/or a measurement of the eye over time (e.g. speed of pupil dilation). The eye data may also include a movement and/or shape of eyebrows262, movement and/or shape of eyelid261, and/or facial micro gestures associated with skin lines for example. Eye203may be wide open, shut, or any variation in between. In some contexts, eye203may be squinting.

InFIG.2A, most of the iris205and large portions of the sclera207are visible. When eye203is squinting, less of iris205and sclera207will be visible. InFIG.2B, neither iris205nor sclera207are visible. InFIG.2A, eyebrow262is arched. InFIG.2B, eyebrow262is flattened and closer to eye203than inFIG.2A. Similarly, smile line264is flattened inFIG.2Bcompared to the more arched smile line264inFIG.2A.FIG.2Balso illustrates an increased number of lines in corner region263compared to the lines in corner region263. The shape and or number of lines in corner region263may correspond to micro gestures, squinting, cringing, eye strain, and/or user discomfort, for example. Therefore, detecting the size, shape, or quantity of various eye features in eyebox region201provides eye data that can be indicative of a user reaction or adaptation to a particular environmental context.

FIG.2Cillustrates eye203having a pupil266with a diameter of dimension291.FIG.2Dillustrates eye203having a pupil266with a diameter of dimension292that is larger than dimension291. In some implementations of the disclosure, eye data may include the size (e.g. diameter) of pupil266. In some implementations of the disclosure, eye data may include the size (e.g. diameter) of pupil266over a particular time period. Thus, when the size of pupil266is captured over a plurality of time periods, the speed of pupil dilation may be determined.

FIGS.2E and2Fillustrate another example measurement of eye203that may be included in eye data, in accordance with implementations of the disclosure. For example, the dimension293of iris205between top eyelid261and bottom eye lid265is larger inFIG.2Ewhen compared to the dimension294of iris205when eye203is squinting inFIG.2F.

FIGS.3A-3Cillustrates different positions of eye203at different times, in accordance with implementations of the disclosure.FIG.3Aillustrates eye203in a centered position381at a time t1. Centered position381may be associated with eye203looking straight forward at an object in the far field (e.g. focus distance of infinity).FIG.3Billustrates eye203in a right-of-center position382at a time t2.FIG.3Cillustrates eye203in a slightly-left-of-center position383at a time t3. The position of eye203may be determined by tracking the position of pupil266, iris205, tracking the cornea (not specifically illustrated), and/or other suitable eye-tracking techniques. Thus, eye data may include a position of eye203and/or positions of eye203over time. The speed of a position change of eye203may be included in eye data. For example, if eye203goes from position381to position382very quickly (e.g. within 200 ms), this movement may be considered a saccade. Smaller movements of eye203in short time periods may be considered micro-saccades. The number of saccades or micro-saccades in a particular time period may be counted using image processing techniques or other suitable pupil position techniques. The number of saccades or micro-saccades in a particular time period may be included in eye data, in various implementations of the disclosure. The position changes of eye203may be considered gaze flickering where eye203changes position often but does not change position rapidly enough to be considered a saccade or micro-saccade. Gaze flickering may be a sign of discomfort due to ambient light brightness or display brightness.

FIG.4illustrates a top view of a portion of an example head mounted device400, in accordance with implementations of the disclosure. Head mounted device400may include an optical element410that includes a transparency modulator layer450, a display layer440, and an illumination layer430. Additional optical layers (not specifically illustrated) may also be included in example optical element410. For example, a focusing lens layer may optionally be included in optical element410to focus scene light456and/or virtual images included in image light441generated by display layer440. Transparency modulator layer450modulates the intensity of incoming scene light456so that the scene light459that propagates to eyebox region201may have a reduced intensity when compared to the intensity of incoming scene light456.

Display layer440presents virtual images in image light441to an eyebox region201for viewing by an eye203. Processing logic470is configured to drive virtual images onto display layer440to present image light441to eyebox region201. Processing logic470is also configured to adjust a brightness of display layer440. In some implementations, adjusting a display brightness of display layer440includes adjusting the intensity of one or more light sources of display layer440. All or a portion of display layer440may be transparent or semi-transparent to allow scene light456from an external environment to become incident on eye203so that a user can view their external environment in addition to viewing virtual images presented in image light441.

Transparency modulator layer450may be configured to change its transparency to modulate the intensity of scene light456that propagates to the eye203of a user. Processing logic470may be configured to drive an analog or digital signal onto transparency modulator layer450in order to modulate the transparency of transparency modulator layer450. In an example implementation, transparency modulator layer450includes liquid crystals where the alignment of the liquid crystals is adjusted in response to a drive signal from processing logic470to modulate the transparency of transparency modulator layer450. Other suitable technologies that allow for electronically controlled dimming of transparency modulator450may be included in transparency modulator450.

Illumination layer430includes light sources426configured to illuminate an eyebox region201with infrared illumination light427. Illumination layer430may include a transparent refractive material that functions as a substrate for light sources426. Infrared illumination light427may be near-infrared illumination light. Camera477is configured to image (directly) eye203, in the illustrated example ofFIG.4. In other implementations, camera447may (indirectly) image eye203by receiving reflected infrared illumination light from an optical combiner layer (not illustrated) included in optical element410. The optical combiner layer may be configured to receive reflected infrared illumination light (the infrared illumination light427reflected from eyebox region201) and redirect the reflected infrared illumination light to camera447. In this implementation, camera447would be oriented to receive the reflected infrared illumination light from the optical combiner layer of optical element410.

Camera447may include a complementary metal-oxide semiconductor (CMOS) image sensor, in some implementations. An infrared filter that receives a narrow-band infrared wavelength may be placed over the image sensor so it is sensitive to the narrow-band infrared wavelength while rejecting visible light and wavelengths outside the narrow-band. Infrared light sources (e.g. light sources426) such as infrared LEDs or infrared VCSELS that emit the narrow-band wavelength may be oriented to illuminate eye203with the narrow-band infrared wavelength. Camera447may capture eye-tracking images of eyebox region201. Eyebox region201may include eye203as well as surrounding features in an ocular area such as eyebrows, eyelids, eye lines, etc. Processing logic470may initiate one or more image captures with camera477and camera477may provide eye-tracking images479to processing logic470. Processing logic470may perform image processing to determine the size and/or position of various features of the eyebox region201. For example, processing logic470may be configured to determine size and/or position of the features described in association withFIGS.2A-3C. For example, processing logic470may perform image processing to determine a pupil position or pupil size of pupil266. Light sources426and camera477are merely an example eye-tracking configuration and other suitable eye-tracking systems and techniques may also be used to capture eye data, in implementations of the disclosure. In an implementation, a MEMS mirror-based RGB laser system is used for capturing eye data.

In the illustrated implementation ofFIG.4, a memory475is included in processing logic470. In other implementations, memory475may be external to processing logic470. In some implementations, memory475is located remotely from processing logic470. In implementations, virtual image(s) are provided to processing logic470for presentation in image light441. In some implementations, virtual images are stored in memory475. Processing logic470may be configured to receive virtual images from a local memory or the virtual images may be wirelessly transmitted to the head mounted device400and received by a wireless interface (not illustrated) of the head mounted device.

FIG.4illustrates that processing logic470is communicatively coupled to ambient light sensor423. Processing logic470may be communicatively coupled to a plurality of ambient light sensors, in some implementations. Ambient light sensor423may include one or more photodetectors (e.g. photodiodes). Ambient light sensor423may include more than one photodetector with corresponding filters so that ambient light sensor423can measure the color as well as the intensity of scene light456. Ambient light sensor423may include a red-green-blue (RGB)/infrared/monochrome camera sensor to generate high certainty measurements about the state of the ambient light environment. In some implementations, a world-facing image sensor of head mounted device400that is oriented to receive scene light456may function as an ambient light sensor. Ambient light sensor423is configured to generate an ambient light measurement429. In the illustrated implementation, processing logic470is configured to receive ambient light measurement429from ambient light sensor423. Processing logic470may also be communicatively coupled to ambient light sensor423to initiate the ambient light measurement.

Processing logic470is communicatively coupled to microphone488of head mounted device400, in the example implementation ofFIG.4. Processing logic470may be communicatively coupled to a plurality of microphones, in some implementations. Processing logic470may be configured to initiate an ambient noise measurement with microphone488. Processing logic470is configured to receive ambient noise measurement486generated by microphone488, inFIG.4. Ambient noise measurement486may be an analog or digital output of microphone488that is representative of the noise-level of the external environment of the head mounted device. Processing logic470is also configured to adjust a volume481of an audio output483of head mounted device400. The audio output483may be an actual physical speaker or a wired or wireless output that drives auxiliary headphones.

In operation, transparency modulator layer450may be driven to various transparency values by processing logic470in response to various eye data and ambient light measurements429. By way of example, a pupil diameter of an eye may indicate that scene light456is brighter than the user prefers. Other measurements of an ocular region (e.g. dimension of eyelids, sclera, number of lines in corner region263, etc.) of the user may indicate the user is squinting and that scene light456may be brighter than the user prefers. Thus, a transparency of transparency modulator layer450may be driven to a transparency that makes the user more comfortable with the intensity of scene light459that propagates through transparency modulator layer450. The transparency of transparency modulator layer450may be modulated to various levels between 10% transparent and 90% transparent, in response to the eye data and the ambient light measurement, for example.

FIG.5Aillustrates an example method of adjusting a transparency of a lens of a head mounted device, in accordance with implementations of the disclosure. The order in which some or all of the process blocks appear in process500should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel. In some implementations, processing logic470executes all or a portion of process500.

In process block505, eye data is captured with one or more sensors of a head mounted device. The one or more sensors are configured to sense the eyebox region. The sensors may include one or more photodiodes, image sensors, or any other suitable sensor to capture eye data. As described previously, the eye data may include images of an eyebox region for example. The eye data may include positions of various features of an ocular area of a user in the eyebox region.

In process block510, an ambient light measurement is initiated with a photodetector (e.g. ambient light sensor423) of the head mounted device. The ambient light measurement (e.g. ambient light measurement429) may be initiated during a same time period as the eye data is captured.

In process block515, a transparency of a lens of the head mounted device is adjusted in response to the eye data and the ambient light measurement. The “lens” of a head mounted device may be an optical element (e.g. optical element110or410) that the user views the world through. In other words, scene light (e.g. scene light456) from an external environment of the head mounted device may propagate through the lens prior to becoming incident on an eye203. In the illustration ofFIG.4, a transparency of transparency modulator layer450may be adjusted in response to eye data and ambient light measurement429.

In some implementations, the eye data includes at least one of a pupil size of an eye, speed of pupil dilation of the eye, a gaze direction of the eye, or eye-movement data (e.g. number of saccades in a given time period).

Eye data may include one or more images (e.g. image(s)479) of eye203or the ocular region of a user that occupies eyebox region201, for example. In an implementation, process500further includes performing image processing on the one or more images of the eye to determine a heart rate of a user of the head mounted device. The heart rate of the user may be determined by pupil size over a time period, for example. Adjusting the transparency of the lens may be based at least in part on the heart rate of the user.

In an implementation, process500further includes performing image processing on the one or more images of the eye to determine at least one of movement of eyebrows, movement of eyelids, or facial muscle micro gestures. Adjusting the transparency of the lens may be based at least in part on the movement of eyebrows, movement of eyelids, and/or facial muscle micro gestures.

In an implementation of process500, adjusting the transparency of the lens of the head mounted device in response to the eye data and the ambient light measurement includes associating the eye data and the ambient light measurement to previous eye data paired with a previous ambient light measurement and adjusting the transparency of the lens to a previously selected lens transparency selected by the user. The previously selected lens transparency corresponds to the previous eye data paired with the previous ambient light measurement. The previous eye data and the previous ambient light measurement were captured by the head mounted device during a same time period. In this way, previous transparency selections of the user can be driven onto transparency modulator layer450to drive the transparency of optical element410to be personalized to previous selections of the user.

To illustrate,FIG.5Billustrates example previous eye data paired with previous ambient light measurements in a memory575, in accordance with implementations of the disclosure. Memory575may be included in memory475, for example.

InFIG.5B, previous eye data is paired with previous ambient light measurements and previously selected lens transparency values correspond to the previous ambient light measurements paired with the previous eye data. Therefore, memory575includes the transparency values a user selected in various ambient light environments where certain eye data was also present. For example, first previous eye data512is paired with first previous ambient light measurement513. First previously selected lens transparency value511corresponds with the pairing of data512and513. A user of a head mounted device may have previously selected first previously selected lens transparency value511when (or immediately after) the first previous ambient light measurement513and first previous eye data512were collected by a head mounted device.

Second previous eye data522is paired with second previous ambient light measurement523. Second previously selected lens transparency value521corresponds with the pairing of data522and523. A user of a head mounted device may have previously selected second previously selected lens transparency value521when (or immediately after) the second previous ambient light measurement523and second previous eye data522were collected by a head mounted device.

Memory575may include integer n number of previously selected lens transparency values corresponding to previous eye data paired with previous ambient light measurements. Thus, given an ambient light measurement (e.g.429) and eye data, a personalized transparency value (previously selected by the user during similar ambient light conditions matched to similar eye data) can be driven onto transparency modulator layer450to adjust the intensity of scene light459.

In an implementation of process500, adjusting the transparency of the lens of the head mounted device in response to the eye data and the ambient light measurement includes adjusting the transparency of the lens of the head mounted device to a predetermined lens transparency value associated with aggregate eye data corresponding to the ambient light measurement. In this way, transparency selections determined in testing, crowd-sourced data or averaged from user preferences can be driven onto transparency modulator layer450to drive the transparency of optical element410to predetermined transparency values that were comfortable under similar ambient light conditions and eye data.

To illustrate,FIG.5Cillustrates example aggregate eye data paired with ambient light measurements in a memory576, in accordance with implementations of the disclosure. Memory576may be included in memory475, for example.

InFIG.5C, aggregate eye data is paired with previous ambient light measurements and predetermined lens transparency values correspond to the previous ambient light measurements paired with the aggregate eye data. Therefore, memory576may include predetermined lens transparency values selected by the average user in various ambient light environments where certain eye data was also present. For example, first aggregate eye data517is paired with first previous ambient light measurement518. First predetermined lens transparency value516corresponds with the pairing of data517and518. An average user setting of a head mounted device may have first predetermined lens transparency value516when (or immediately after) the first previous ambient light measurement518and first aggregate eye data517were collected by a head mounted device.

Second aggregate eye data527is paired with second previous ambient light measurement528. First predetermined lens transparency value526corresponds with the pairing of data527and528. An average user setting of a head mounted device may have second predetermined lens transparency value526when (or immediately after) the second previous ambient light measurement528and second aggregate eye data527were collected by a head mounted device.

Memory576may include integer n number of predetermined lens transparency values corresponding to aggregate eye data paired with previous ambient light measurements. Thus, given an ambient light measurement (e.g.429) and aggregate eye data for that particular ambient light measurement, a predetermined lens transparency value known to be suitable for the ambient light measurement and eye data can be driven onto transparency modulator layer450to adjust the intensity of scene light459.

FIG.6Aillustrates an example method of adjusting a display brightness of head mounted device, in accordance with implementations of the disclosure. The order in which some or all of the process blocks appear in process600should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel. In some implementations, processing logic470executes all or a portion of process600.

In process block605, eye data is captured with one or more sensors of a head mounted device. The one or more sensors are configured to sense the eyebox region. The sensors may include one or more photodiodes, image sensors, or any other suitable sensor to capture eye data. As described previously, the eye data may include images of an eyebox region, for example. The eye data may include positions of various features of an ocular area of a user in the eyebox region.

In process block610, an ambient light measurement is initiated with a photodetector (e.g. ambient light sensor423) of the head mounted device. The ambient light measurement (e.g. ambient light measurement429) may be initiated during a same time period as the eye data is captured.

In process block615, a brightness of a display of the head mounted device is adjusted in response to the eye data and the ambient light measurement. In the illustration ofFIG.4, a brightness of display layer440may be adjusted in response to eye data and ambient light measurement429.

In some implementations, the eye data includes at least one of a pupil size of an eye, speed of pupil dilation of the eye, a gaze direction of the eye, or eye-movement data (e.g. number of saccades in a given time period).

Eye data may include one or more images (e.g. image(s)479) of eye203or the ocular region of a user that occupies eyebox region201, for example. In an implementation, process600further includes performing image processing on the one or more images of the eye to determine a heart rate of a user of the head mounted device. The heart rate of the user may be determined by pupil size over a time period, for example. Adjusting the brightness of the display may be based at least in part on the heart rate of the user.

In an implementation, process600further includes performing image processing on the one or more images of the eye to determine at least one of movement of eyebrows, movement of eyelids, or facial muscle micro gestures. Adjusting the brightness of the display may be based at least in part on the movement of eyebrows, movement of eyelids, and/or facial muscle micro gestures.

In an implementation of process600, adjusting the brightness of the display in response to the eye data and the ambient light measurement includes associating the eye data and the ambient light measurement to previous eye data paired with a previous ambient light measurement and adjusting the brightness of the display to a previously selected display brightness selected by the user. The previously selected display brightness corresponds to the previous eye data paired with the previous ambient light measurement and the previous eye data and the previous ambient light measurement were captured by the head mounted device during a same time period. In this way, previous display brightness selections of the user can be driven onto display layer440to drive the brightness of the display to be personalized to previous selections of the user.

To illustrate,FIG.6Billustrates example previous eye data paired with previous ambient light measurements in a memory675, in accordance with implementations of the disclosure. Memory675may be included in memory475, for example.

InFIG.6B, previous eye data is paired with previous ambient light measurements and previously selected display brightness values correspond to the previous ambient light measurements paired with the previous eye data. Therefore, memory675includes the brightness values a user selected in various ambient light environments where certain eye data was also present. For example, first previous eye data612is paired with first previous ambient light measurement613. First previously selected display brightness value611corresponds with the pairing of data612and613. A user of a head mounted device may have previously selected first previously selected display brightness value611when (or immediately after) the first previous ambient light measurement613and first previous eye data612were collected by a head mounted device.

Second previous eye data622is paired with second previous ambient light measurement623. Second previously selected display brightness value621corresponds with the pairing of data622and623. A user of a head mounted device may have previously selected second previously selected display brightness value621when (or immediately after) the second previous ambient light measurement623and second previous eye data622were collected by a head mounted device.

Memory675may include integer n number of previously selected display brightness values corresponding to previous eye data paired with previous ambient light measurements. Thus, given an ambient light measurement (e.g.429) and eye data, a personalized display brightness value (previously selected by the user during similar ambient light conditions matched to similar eye data) can be driven onto display layer440to adjust a brightness of image light441.

In an implementation of process600, adjusting the brightness of the display of the head mounted device in response to the eye data and the ambient light measurement includes adjusting the brightness of the display to a predetermined display brightness associated with aggregate eye data corresponding to the ambient light measurement. In this way, display brightness selections determined in testing, crowd-sourced data, or averaged from user preferences can be driven onto display layer440to drive display layer440to brightness levels that are comfortable under similar ambient light conditions and eye data.

To illustrate,FIG.6Cillustrates example aggregate eye data paired with ambient light measurements in a memory676, in accordance with implementations of the disclosure. Memory676may be included in memory475, for example.

InFIG.6C, aggregate eye data is paired with previous ambient light measurements and predetermined display brightness values correspond to the previous ambient light measurements paired with the aggregate eye data. Therefore, memory676may include predetermined display brightness values selected by the average user in various ambient light environments where certain eye data was also present. For example, first aggregate eye data617is paired with first previous ambient light measurement618. First predetermined display brightness value616corresponds with the pairing of data617and618. An average user setting of a head mounted device may have first predetermined display brightness value616when (or immediately after) the first previous ambient light measurement618and first aggregate eye data617were collected by a head mounted device.

Second aggregate eye data627is paired with second previous ambient light measurement628. First predetermined display brightness value626corresponds with the pairing of data627and628. An average user setting of a head mounted device may have second predetermined display brightness value626when (or immediately after) the second previous ambient light measurement628and second aggregate eye data627were collected by a head mounted device.

Memory676may include integer n number of predetermined display brightness values corresponding to aggregate eye data paired with previous ambient light measurements. Thus, given an ambient light measurement (e.g.429) and aggregate eye data for that particular ambient light measurement, a predetermined display brightness value known to be suitable for the ambient light measurement and eye data can be driven onto display layer440to adjust a brightness of image light441.

FIG.7illustrates an example method of adjusting a volume of an audio output of a head mounted device in response to eye data, in accordance with implementations of the disclosure. The order in which some or all of the process blocks appear in process700should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel. In some implementations, processing logic470executes all or a portion of process700.

In process block705, eye data is captured with one or more sensors of a head mounted device. The one or more sensors are configured to sense the eyebox region. The sensors may include one or more photodiodes, image sensors, or any other suitable sensor to capture eye data. As described previously, the eye data may include images of an eyebox region, for example. The eye data may include positions of various features of an ocular area of a user in the eyebox region. Eye data may include at least one of a pupil size of an eye, speed of pupil dilation of the eye, a gaze direction of the eye, or eye-movement data, for example. The eye data may include a number of saccades in a time period.

In process block710, a volume of an audio output of the head mounted device is adjusted in response to the eye data. Adjusting the volume of the audio output may include adjusting a sound magnitude of speakers of the head mounted device where the speakers are oriented to provide sound to ears of a user of the head mounted device.

In an implementation, process700further includes initiating an ambient noise measurement with a microphone of the head mounted device. In this implementation, adjusting the volume of the audio output of the head mounted device is in response to the eye data and to the ambient noise measurement.

The term “processing logic” (e.g.470) in this disclosure may include one or more processors, microprocessors, multi-core processors, Application-specific integrated circuits (ASIC), and/or Field Programmable Gate Arrays (FPGAs) to execute operations disclosed herein. In some embodiments, memories (not illustrated) are integrated into the processing logic to store instructions to execute operations and/or store data. Processing logic may also include analog or digital circuitry to perform the operations in accordance with embodiments of the disclosure.

A “memory” or “memories” described in this disclosure may include one or more volatile or non-volatile memory architectures. The “memory” or “memories” may be removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Example memory technologies may include RAM, ROM, EEPROM, flash memory, CD-ROM, digital versatile disks (DVD), high-definition multimedia/data storage disks, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device.

Network may include any network or network system such as, but not limited to, the following: a peer-to-peer network; a Local Area Network (LAN); a Wide Area Network (WAN); a public network, such as the Internet; a private network; a cellular network; a wireless network; a wired network; a wireless and wired combination network; and a satellite network.

Communication channels may include or be routed through one or more wired or wireless communication utilizing IEEE 802.11 protocols, BlueTooth, SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), USB (Universal Serial Port), CAN (Controller Area Network), cellular data protocols (e.g. 3G, 4G, LTE, 5G), optical communication networks, Internet Service Providers (ISPs), a peer-to-peer network, a Local Area Network (LAN), a Wide Area Network (WAN), a public network (e.g. “the Internet”), a private network, a satellite network, or otherwise.

A computing device may include a desktop computer, a laptop computer, a tablet, a phablet, a smartphone, a feature phone, a server computer, or otherwise. A server computer may be located remotely in a data center or be stored locally.