Patent Description:
Near-eye light field displays project images directly into a user's eye, encompassing both near-eye displays (NEDs) and electronic viewfinders. Conventional near-eye displays (NEDs) generally have a display element that generates image light that passes through one or more lenses before reaching the user's eyes. Additionally, NEDs in augmented reality systems are typically required to be compact and light weight, and to provide large exit pupil with a wide field-of-vision for ease of use. However, designing a conventional NED with scanners providing high brightness and uniform illumination intensity can result in a low outcoupling efficiency of the image light received by the user's eyes.

<CIT>, an ocular projection display device and method based on pupil position of an eye of a user.

The aspects of the invention are defined as set out in the claims.

The following drawings and associated description are present for illustration purposes only and shall aide the skilled reader in understanding the claimed invention, the scope of which is set out only in the appended claims.

The figures depict various embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

A near-eye-display (NED) includes an eye tracking system and a waveguide display. The eye tracking system tracks and moves eyebox locations based on a movement of the user's eyes. The waveguide display includes a source assembly, an output waveguide and a controller. The output waveguide includes a coupling grating that receives an image light
from the source assembly, and a decoupling grating that outputs the expanded image light to one or more eyebox locations.

The decoupling grating is a 2D array of spatially switchable liquid crystal (LC) gratings. The 2D array of spatially switchable LC gratings include an active subset of LC pixels outcoupling an expanded image light from the output waveguide to a location of the user's eye tracked by the eye tracking system. The decoupling grating dynamically adjusts where the image light exits by controlling the LC gratings corresponding to the eyebox location. In this manner, the NED is able to output the expanded image light in an eyebox that in tandem with the movements of the eye. The controller generates switching instructions to the 2D array of LC gratings to dynamically activate the LCs based on the tracked eye movements.

<FIG> is a diagram of a near-eye-display (NED) <NUM> (also referred to as a head-mounted display), in accordance with an embodiment. The NED <NUM> presents media to a user. Examples of media presented by the NED <NUM> include one or more images, video, audio, or some combination thereof. In some embodiments, audio is presented via an external device (e.g., speakers and/or headphones) that receives audio information from the NED <NUM>, a console (not shown), or both, and presents audio data based on the audio information. The NED <NUM> is generally configured to operate as a virtual reality (VR) NED. However, in some embodiments, the NED <NUM> may be modified to also operate as an augmented reality (AR) NED, a mixed reality (MR) NED, or some combination thereof. For example, in some embodiments, the NED <NUM> may augment views of a physical, real-world environment with computer-generated elements (e.g., images, video, sound, etc.).

The NED <NUM> shown in <FIG> includes a frame <NUM> and a display <NUM>. The frame <NUM> includes one or more optical elements which together display media to users. The display <NUM> is configured for users to see the content presented by the NED <NUM>. As discussed below in conjunction with <FIG>, the display <NUM> includes at least one source assembly to generate an image light to present media to an eye of the user. The source assembly includes, e.g., a source, an optics system, or some combination thereof.

<FIG> is a cross section <NUM> of the NED <NUM> illustrated in <FIG>, in accordance with an embodiment. The cross section <NUM> includes at least one waveguide assembly <NUM>, an exit pupil <NUM>, and an eye tracking system <NUM>. The exit pupil <NUM> is a location where the eye <NUM> is positioned when the user wears the NED <NUM>. In some embodiments, the frame <NUM> may represent a frame of eye-wear glasses. For purposes of illustration, <FIG> shows the cross section <NUM> associated with a single eye <NUM> and a single waveguide assembly <NUM>, but in alternative embodiments not shown, another waveguide assembly which is separate from the waveguide assembly <NUM> shown in <FIG>, provides image light to another eye <NUM> of the user.

The waveguide assembly <NUM>, as illustrated below in <FIG>, is configured to direct the image light to the eye <NUM> through the exit pupil <NUM>. The waveguide assembly <NUM> may be composed of one or more materials (e.g., plastic, glass, etc.) with one or more refractive indices that effectively minimize the weight and widen a field of view (hereinafter abbreviated as 'FOV') of the NED <NUM>. In alternate configurations, the NED <NUM> includes one or more optical elements between the waveguide assembly <NUM> and the eye <NUM>. The optical elements may act to, e.g., correct aberrations in image light emitted from the waveguide assembly <NUM>, magnify image light emitted from the waveguide assembly <NUM>, some other optical adjustment of image light emitted from the waveguide assembly <NUM>, or some combination thereof. The example for optical elements may include an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, or any other suitable optical element that affects image light.

In some embodiments, the waveguide assembly <NUM> may include a source assembly to generate an image light to present media to user's eyes. The source assembly includes, e.g., a source, an optics system, or some combination thereof.

The eye tracking system <NUM> tracks a position and movement of a user's eye at one or more eyebox locations of the NED <NUM>. The eyebox location is a region that outputs an image light corresponding to the media presented through the NED <NUM>. In one example, the eyebox location includes a length of <NUM> and a width of <NUM>. A camera or other optical sensor inside the NED <NUM> controlled by the eye tracking system <NUM> captures images of one or both eyes of the user, and the eye tracking system <NUM> uses the captured images to determine eye tracking information. Eye tracking information is an information associated with a position and/or movement of a user's eye at one or more eyebox locations. For example, the eye tracking information may include, e.g., an interpupillary distance, an interocular distance, a three-dimensional (3D) position of each eye relative to the NED <NUM> for distortion adjustment purposes, including a magnitude of torsion and rotation, and gaze directions for each eye. The eye tracking system <NUM> tracks different types of eye movements including, but not restricted to, a saccadic eye movement (e. g rapid and conjugate movements), a pursuit movement (e.g. a slow-tracking), a compensatory eye movement (e.g. smooth movements compensating for active or passion motions), a vergence eye movement (e.g. two eye moving in opposite directions), a miniature eye movement (e.g. a steady and fixed view of a target), an optokinetic nystagmus (e.g. a sawtooth pattern), or some combination thereof.

<FIG> illustrates an isometric view of a waveguide display <NUM>, in accordance with an embodiment. In some embodiments, the waveguide display <NUM> is a component (e.g., display assembly <NUM>) of the NED <NUM>. In alternate embodiments, the waveguide display <NUM> is part of some other NED, or other system that directs display image light to a particular location.

The waveguide display <NUM> includes at least a source assembly <NUM>, an output waveguide <NUM>, and a controller <NUM>. For purposes of illustration, <FIG> shows the waveguide display <NUM> associated with a single eye <NUM>, but in some embodiments, another waveguide display separate (or partially separate) from the waveguide display <NUM>, provides image light to another eye of the user. In a partially separate system, one or more components may be shared between waveguide displays for each eye.

The source assembly <NUM> generates image light. The source assembly <NUM> includes a source <NUM> and an optics system <NUM>. The source <NUM> is an optical source that generates an image light, as described in detail below with regard to <FIG>. The optics system <NUM> is a set of optical components (e.g. lens, mirrors, etc.) that direct the image light received from the source <NUM>. The source assembly <NUM> generates and outputs an image light <NUM> to a coupling element <NUM> of the output waveguide <NUM>.

The output waveguide <NUM> is an optical waveguide that outputs image light to one or more eyebox locations associated with the eye <NUM> of a user. The output waveguide <NUM> receives the image light <NUM> at one or more coupling elements <NUM>, and guides the received input image light to one or more dynamic decoupling elements <NUM>. In some embodiments, the coupling element <NUM> couples the image light <NUM> from the source assembly <NUM> into the output waveguide <NUM>. The coupling element <NUM> may be, e.g., a diffraction grating, a holographic grating, some other element that couples the image light <NUM> into the output waveguide <NUM>, or some combination thereof. For example, in embodiments where the coupling element <NUM> is diffraction grating, the pitch of the diffraction grating is chosen such that total internal reflection occurs, and the image light <NUM> propagates internally toward the decoupling element <NUM>. For example, the pitch of the diffraction grating may be in the range of <NUM> to <NUM>.

The dynamic decoupling element <NUM> dynamically decouples the total internally reflected image light from the output waveguide <NUM>. The dynamic decoupling element <NUM> includes a two-dimensional array of liquid crystal gratings and control electrodes as described below in conjunction with <FIG>. In some configurations, the dynamic decoupling element <NUM> is transparent in a visible band of light. The dynamic decoupling element <NUM> dynamically decouples the total internally reflected image light based on a switching of an active subset of LC pixels in the two-dimensional array of liquid crystal gratings located at different portions for a threshold value of switching time in response to the positions and/or movements of the user's eyes tracked by the eye tracking system <NUM>. The dynamic decoupling element <NUM> may be, e.g., a diffraction grating, a holographic grating, some other element that decouples image light out of the output waveguide <NUM>, or some combination thereof.

The dynamic decoupling element <NUM> includes an off-element, an on-element and one or more control electrodes, as described in detail below in conjunction with <FIG>. Each of the off-element and the on-element is a two-dimensional array of liquid crystal (LC) gratings. In some configurations, the on-element has a length of <NUM> along the X dimension and a width of <NUM> along the Y dimension to cover an eye pupil with a diameter of at most <NUM>. The on-element is made of LC pixels controlled by a pixelated electrode with size as small as <NUM> microns. In one example, the on-element has a length of <NUM> microns and receives an electrical bias from the control electrode. Each LC pixel in the dynamic decoupling element <NUM> is coupled to the control electrode that sets an activation state of the LC pixel.

In embodiments where the dynamic decoupling element <NUM> is a diffraction grating, the pitch of the diffraction grating is chosen to cause incident image light to exit the output waveguide <NUM>. For example, the pitch of the diffraction grating may be in the range of <NUM> to <NUM>. The pitch of the diffraction grating is chosen such that the image light <NUM> from the plurality of optical sources undergoes a total internal reflection inside the output waveguide <NUM> without leakage through higher order diffraction (e.g. second reflected order). An orientation and position of an image light <NUM> exiting from the output waveguide <NUM> is controlled by changing an orientation and position of the image light <NUM> entering the coupling element <NUM>. In some embodiments, the direction of the image light <NUM> exiting from the output waveguide <NUM> is same as the direction of the image light <NUM>. In one example, the position of the image light <NUM> exiting from the output waveguide <NUM> is controlled by the location of the plurality of optical sources of the source assembly <NUM>, the location of the coupling element <NUM>, the location of the dynamic decoupling element <NUM>, and a switching of liquid crystal gratings of the dynamic decoupling element <NUM>.

In alternate embodiments, the coupling element <NUM> includes at least one of one-dimensional array and two-dimensional array of liquid crystal gratings and control electrodes (not shown here). The coupling element <NUM> receives a right circularly polarized image light <NUM> from the source assembly <NUM> and diffracts a left circularly polarized image light of first order to the dynamic decoupling element <NUM>.

The output waveguide <NUM> may be composed of one or more materials that facilitate total internal reflection of the image light <NUM>. The output waveguide <NUM> may be composed of e.g., silicon, plastic, glass, or polymers, or some combination thereof. The output waveguide <NUM> has a relatively small form factor. For example, the output waveguide <NUM> may be approximately <NUM> wide along X-dimension, <NUM> long along Y-dimension and <NUM>-<NUM> thick along Z-dimension.

The controller <NUM> controls the scanning operations of the source assembly <NUM>. The controller <NUM> determines scanning instructions for the source assembly <NUM> based at least on the one or more display instructions. Display instructions are instructions to render one or more images. In some embodiments, display instructions may simply be an image file (e.g., bitmap). The display instructions may be received from, e.g., a console of a system (e.g., as described below in conjunction with <FIG>). Scanning instructions are instructions used by the source assembly <NUM> to generate the image light <NUM>. The scanning instructions may include, e.g., a type of a source of image light (e.g. monochromatic, polychromatic), an identifier for a particular source assembly, a scanning rate, an orientation of the source, one or more illumination parameters (described below with reference to <FIG>), or some combination thereof.

The controller <NUM> takes content for display, and divides the content into discrete sections. The controller <NUM> instructs the source <NUM> to sequentially present the discrete sections using individual source elements corresponding to a respective row in an image ultimately displayed to the user. The controller <NUM> instructs the optics system <NUM> to scan the presented discrete sections to different areas of the coupling element <NUM> of the output waveguide <NUM>. Accordingly, at the exit pupil of the output waveguide <NUM>, each discrete portion is presented in a different location. While each discrete section is presented at different times, the presentation and scanning of the discrete sections occurs fast enough such that a user's eye integrates the different sections into a single image or series of images.

The controller <NUM> determines switching instructions for the control electrode in the dynamic decoupling element <NUM> based on the eye tracking information received from the eye tracking system <NUM>. The controller <NUM> uses the received eye tracking information to predict the position and/or movement of user's eyes and then generates switching instructions based on this prediction. The switching instructions may include, e.g. a switching time, an electrical bias, a spatial location of one or more pixels of the control electrode, or some combination thereof. The switching time is a time difference between the first instance when the control electrode receives zero electrical bias and the second instance when the control electrode receives a non-zero electrical bias. In one example, the switching instruction includes a switching time of <NUM> and the address (e.g. x-y co-ordinates) of the pixels of the control electrode. The controller <NUM> determines an eyebox location based on the locations of the eyes tracked by the eye tracking system <NUM>. The controller <NUM> determines a subset of LC pixels that emit light and falling within the eyebox location. The controller <NUM> generates the switching instructions causing the dynamic decoupling element <NUM> to activate the subset of LC pixels to form an active subset of LC pixels. The controller <NUM> generates the switching instructions such that the LC pixels that are not in the active subset are in an inactive subset of LC pixels that do not out-couple light from the dynamic decoupling element <NUM>. The controller <NUM> generates the switching instructions such that each LC pixel that is coupled to the control electrode is set to an activation state. For example, the activation state of a LC pixel refers to a preferred spatial orientation of a LC molecule in an arbitrary direction in a three-dimensional space. The controller <NUM> includes a combination of hardware, software, and/or firmware not shown here so as not to obscure other aspects of the disclosure.

<FIG> illustrates a cross section <NUM> of the waveguide display <NUM> of <FIG>, in accordance with an embodiment. The cross section <NUM> includes the source assembly <NUM> and the output waveguide <NUM>. The source assembly <NUM> includes a source <NUM> and an optics system <NUM>. The source <NUM> is an embodiment of the source <NUM> of <FIG>. The optics system <NUM> is an embodiment of the optics system <NUM> of <FIG>. The output waveguide <NUM> includes the coupling element <NUM> and a dynamic decoupling element <NUM>.

The source assembly <NUM> generates light in accordance with scanning instructions from the controller <NUM>. The source assembly <NUM> includes a source <NUM>, and an optics system <NUM>. The source <NUM> is a source of light that generates a spatially coherent or a partially spatially coherent image light. The source <NUM> may be, e.g., a superluminous LED, a laser diode, a vertical cavity surface emitting laser (VCSEL), a light emitting diode, a tunable laser, or some other light source that emits coherent or partially coherent light. The source <NUM> emits light in a visible band (e.g., from about <NUM> to <NUM>), and it may emit light that is continuous or pulsed. In some embodiments, the source <NUM> may be a superluminous LED (SLED) array of densely packed ridge waveguides with a wide emission spectrum. The source <NUM> emits light in accordance with one or more illumination parameters received from the controller <NUM>. An illumination parameter is an instruction used by the source <NUM> to generate light. An illumination parameter may include, e.g., source wavelength, pulse rate, pulse amplitude, beam type (continuous or pulsed), other parameter(s) that affect the emitted light, or some combination thereof.

The optics system <NUM> includes one or more optical components that condition the light from the source <NUM>. Conditioning light from the source <NUM> may include, e.g., expanding, collimating, adjusting orientation in accordance with instructions from the controller <NUM>, some other adjustment of the light, or some combination thereof. The one or more optical components may include, e.g., lenses, mirrors, apertures, gratings, or some combination thereof. Light emitted from the optics system <NUM> (and also the source assembly <NUM>) is the image light <NUM>. The optics system <NUM> outputs the image light <NUM> at a particular orientation (in accordance with the scanning instructions) toward the output waveguide <NUM>.

The dynamic decoupling element <NUM> is an embodiment of the dynamic decoupling element <NUM> of <FIG>. In this example, the dynamic decoupling element <NUM> includes an off-element <NUM>, an on-element <NUM> and one or more control electrodes <NUM>. The off-element <NUM> are a plurality of liquid crystal (LC) gratings in a state that prevents light from outcoupling from the output waveguide <NUM>. And the on-element <NUM> is a plurality of LC gratings in a state that outcouple light to the output waveguide <NUM>. A LC grating includes liquid crystal molecules from a group including, but not restricted to, a thermotropic LC, a lyotropic LC, a metallotropic LC, or some combination thereof. The thermotropic LC shows a phase transition into the liquid-crystal phase with a change in temperature. The lyotropic LC shows phase transitions based on both temperature and concentration of the LC molecules in water. The metallotropic LC includes both organic and inorganic molecules but their liquid-crystal transition depends on temperature, concentration, and the inorganic-organic composition ratio.

The control electrode <NUM> is an electrical component that applies an electrical bias to the dynamic decoupling element <NUM>. In the example of <FIG>, the control electrode <NUM> is a transparent conducting electrode that controls the off-element <NUM> and the on-element <NUM> based on the switching instructions from the controller <NUM>. The control electrode <NUM> is composed of one or more pixels of transparent conducting materials such as Indium Tin Oxide (ITO), transparent conductive oxides (TCO), graphene, etc. Each pixel of the control electrode <NUM> has a diameter in the order of few microns that in combination covers an eye pupil of size in the order of few millimeters. In one example, the control electrode <NUM> includes pixels with a diameter of three microns corresponding to an eye pupil with a diameter of three millimeters. In some embodiments, the dynamic decoupling element <NUM> is a two-dimensional array of LC gratings with a thickness ranging from submicron to a few microns and a birefringence ranging from <NUM> to <NUM>. In some configurations, the dynamic decoupling element <NUM> receives a right circularly polarized image light directly from the source assembly <NUM> (not shown here) and dynamically decouples a right circularly polarized image light of zeroth order of diffraction and a left circularly polarized image light of first order of diffraction In some configurations, the dynamic decoupling element <NUM> has a diffraction efficiency of sin<NUM>(δ/<NUM>) for a first diffraction order and cos<NUM>(δ/<NUM>) for a zeroth diffraction order, where δ is the retardance magnitude in radians provided by the LC gratings for an image light with a wavelength λ. The retardance magnitude δ for a layer of LC gratings with LC directors oriented in one direction is (2π/λ,)Δn×t where t is the thickness of LC gratings and Δn is the birefringence of the LC gratings with a certain orientation of the LC director. In some configurations, along the on-element <NUM>, all LC directors are oriented in the X-Y plane and thus provides the same retardance magnitude, but a periodically varying retardance orientation. If δ is π radians, diffraction efficiency of the zeroth order and the first order are <NUM>% and <NUM>% respectively. The off-element <NUM> includes a first set of liquid crystals whose directors are oriented out of the plane of the surface of the output waveguide <NUM>. In the example of <FIG>, the off-element <NUM> includes liquid crystals oriented along the Z direction. The on-element <NUM> includes a second set of liquid crystals oriented along the surface of the output waveguide <NUM>. In the example of <FIG>, the on-element <NUM> is oriented along the X-Y plane. In some configurations, the dynamic decoupling element <NUM> includes a two-dimensional array of LCs with a spatial arrangement forming a linear waveplate with a periodically varying LC director. The periodicity in variation of the LC molecules varies the optical phase of the light passing through this LC layer periodically, similar to a diffraction grating, and diffracts light based on the light's polarization. In this embodiment, an alignment layer would orient the LC directors within the array of LCs in the x-y plane. In the example of <FIG>, the dynamic decoupling element <NUM> includes LCs with a periodically varying direction of preferred orientation of LC molecules along the X-Y plane. The period of varying direction of the preferred orientation of the LC gratings controls the diffraction angle, following a grating equation.

The controller <NUM> determines switching instructions for the control electrode <NUM> in the output waveguide <NUM>, as described above in conjunction with <FIG>. The controller <NUM> determines switching instructions to efficiently decouple the image light <NUM> to the eye <NUM> with very high brightness and uniform illumination only to portions of the output waveguide <NUM> where the user's eye is currently looking at.

<FIG> is a first portion <NUM> of the output waveguide <NUM>, in accordance with an embodiment. The first portion <NUM> illustrates the output waveguide <NUM> emitting an image light <NUM> to a first eyebox location that is occupied by an eye of the user. The output waveguide <NUM> emits the image light <NUM> based on the scanning instructions and switching instructions from the controller <NUM>.

The image light <NUM> is an embodiment of the image light <NUM> of <FIG>. The image light <NUM> is a portion of the media presented to a user's eye <NUM> wearing the NED <NUM>. The output waveguide <NUM> emits the image light <NUM> based on a switching of the two-dimensional array of LC gratings for <NUM> to <NUM> milliseconds. In alternate configurations, as shown in <FIG>, the output waveguide <NUM> emits the image light <NUM> from a different portion of the output waveguide <NUM> corresponding to a different eyebox location based on an eye movement tracked by the eye tracking system <NUM>. The controller <NUM> receives the tracked eye movement from the eye tracking system <NUM> and instructs the output waveguide <NUM> to activate some of the LC gratings to match with the current position of the eye <NUM>.

<FIG> is a second portion <NUM> of the output waveguide <NUM>, in accordance with an embodiment. The second portion <NUM> illustrates the output waveguide <NUM> emitting an image light <NUM> to a second eyebox location that includes the eye <NUM>. The output waveguide <NUM> emits the image light <NUM> based on the scanning instructions and switching instructions from the controller <NUM>. The image light <NUM> is an embodiment of the image light <NUM> of <FIG>. The output waveguide <NUM> emits the image light <NUM> from a second portion of the output waveguide <NUM> corresponding to a second eyebox location based on the eye movement tracked by the eye tracking system <NUM>. In the example of <FIG>, the image light <NUM> corresponds to the second portion of the output waveguide <NUM> which is relatively smaller in size when compared to the image light <NUM> of <FIG> and also from a different location on the output waveguide <NUM>.

<FIG> is a block diagram of a system <NUM> including a NED <NUM>, according to an embodiment. The system <NUM> shown by <FIG> comprises the NED <NUM>, an imaging device <NUM>, and an input/output interface <NUM> that are each coupled to the console <NUM>. While <FIG> shows an example system <NUM> including one NED <NUM>, one imaging device <NUM>, and one input/output interface <NUM>, in other embodiments, any number of these components may be included in the system <NUM>. For example, there may be multiple NEDs <NUM> each having an associated input/output interface <NUM> and being monitored by one or more imaging devices <NUM>, with each NED <NUM>, imaging devices <NUM>, and the input/output interface <NUM>, communicating with the console <NUM>. In alternative configurations, different and/or additional components may be included in the system <NUM>. Similarly, functionality of one or more of the components can be distributed among the components in a different manner than is described here. For example, some or all of the functionality of the console <NUM> may be contained within the NED <NUM>. Additionally, in some embodiments, the system <NUM> may be modified to include other system environments, such as an AR system environment.

The NED <NUM> inside the system <NUM> is an embodiment of the NED <NUM> of <FIG>. The NED <NUM> is a near-eye display that presents media to a user. Examples of media presented by the NED <NUM> include one or more images, video, audio, or some combination thereof. In some embodiments, audio is presented via an external device (e.g., speakers and/or headphones) that receives audio information from the NED <NUM>, the console <NUM>, or both, and presents audio data based on the audio information. In some embodiments, the NED <NUM> may also act as an AR eye-wear glass. In these embodiments, the NED <NUM> augments views of a physical, real-world environment with computer-generated elements (e.g., images, video, sound, etc.).

The waveguide assembly <NUM>, as illustrated above in conjunction with <FIG>, is configured to direct an image light to user's eyes wearing the NED <NUM>. The waveguide assembly <NUM> may be composed of one or more materials (e.g., plastic, glass, etc.) with one or more refractive indices that effectively minimize the weight. The waveguide assembly <NUM> may include a source assembly to generate an image light to present media to user's eyes and an output waveguide with a coupling element and a dynamic decoupling element. The dynamic decoupling element decouples the total internally reflected image light from the output waveguide. The dynamic decoupling element includes a two-dimensional array of liquid crystal gratings and control electrodes as described above in conjunction with <FIG>. In some examples, the dynamic decoupling element includes an off-element, an on-element and one or more control electrodes, as described above in conjunction with <FIG>. Each of the off-element and the on-element is a two-dimensional array of liquid crystal (LC) gratings. The source assembly includes, e.g., a source, an optics system, or some combination thereof.

The eye tracking system <NUM> tracks a position and/or movement of a user's eye at one or more eyebox locations of the NED <NUM>. A camera or other optical sensor inside the NED <NUM> captures information of user's eyes, and the eye tracking system <NUM> uses the captured information to determine interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye relative to the NED <NUM> (e.g., for distortion adjustment purposes), including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye. The eye tracking system <NUM> tracks different types of eye movements including, but not restricted to, a saccadic eye movement (e. g rapid and conjugate movements), a pursuit movement (e.g. a slow-tracking), a compensatory eye movement (e.g. smooth movements compensating for active or passion motions), a vergence eye movement (e.g. two eye moving in opposite directions), a miniature eye movement (e.g. a steady and fixed view of a target), an optokinetic nystagmus (e.g. a sawtooth pattern), or some combination thereof.

The IMU <NUM> is an electronic device that generates fast calibration data indicating an estimated position of the NED <NUM> relative to an initial position of the NED <NUM> based on measurement signals received from one or more of the position sensors <NUM>. A position sensor <NUM> generates one or more measurement signals in response to motion of the NED <NUM>. Examples of position sensors <NUM> include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of the IMU <NUM>, or some combination thereof. The position sensors <NUM> may be located external to the IMU <NUM>, internal to the IMU <NUM>, or some combination thereof. In the embodiment shown by <FIG>, the position sensors <NUM> are located within the IMU <NUM>, and neither the IMU <NUM> nor the position sensors <NUM> are visible to the user (e.g., located beneath an outer surface of the NED <NUM>).

Based on the one or more measurement signals generated by the one or more position sensors <NUM>, the IMU <NUM> generates fast calibration data indicating an estimated position of the NED <NUM> relative to an initial position of the NED <NUM>. For example, the position sensors <NUM> include multiple accelerometers to measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes to measure rotational motion (e.g., pitch, yaw, roll). In some embodiments, the IMU <NUM> rapidly samples the measurement signals from various position sensors <NUM> and calculates the estimated position of the NED <NUM> from the sampled data. For example, the IMU <NUM> integrates the measurement signals received from one or more accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point (not shown) on the NED <NUM>. The reference point is a point that may be used to describe the position of the NED <NUM>. While the reference point may generally be defined as a point in space; however, in practice, the reference point is defined as a point within the NED <NUM> (e.g., the reference point representing a center of the IMU <NUM>).

The imaging device <NUM> generates slow calibration data in accordance with calibration parameters received from the console <NUM>. The imaging device <NUM> may include one or more cameras, one or more video cameras, one or more filters (e.g., used to increase signal to noise ratio), or any combination thereof. The imaging device <NUM> is configured to detect image light emitted or reflected in the FOV of the imaging device <NUM>. In embodiments where the NED <NUM> include passive elements (e.g., a retroreflector), the imaging device <NUM> may retro-reflect the image light towards the image light source in the imaging device <NUM>. Slow calibration data is communicated from the imaging device <NUM> to the console <NUM>, and the imaging device <NUM> receives one or more calibration parameters from the console <NUM> to adjust one or more imaging parameters (e.g., focal length, focus, frame rate, ISO, sensor temperature, shutter speed, aperture, etc.).

The input/output interface <NUM> is a device that allows a user to send action requests to the console <NUM>. An action request is a request to perform a particular action. For example, an action request may be to start or end an application or to perform a particular action within the application. The input/output interface <NUM> may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving action requests and communicating the received action requests to the console <NUM>. An action request received by the input/output interface <NUM> is communicated to the console <NUM>, which performs an action corresponding to the action request. In some embodiments, the input/output interface <NUM> may provide haptic feedback to the user in accordance with instructions received from the console <NUM>. For example, haptic feedback is provided when an action request is received, or the console <NUM> communicates instructions to the input/output interface <NUM> causing the input/output interface <NUM> to generate haptic feedback when the console <NUM> performs an action.

The console <NUM> provides media to the NED <NUM> for presentation to the user in accordance with information received from one or more of: the imaging device <NUM>, the NED <NUM>, and the input/output interface <NUM>. In the example shown in <FIG>, the console <NUM> includes an application store <NUM>, a tracking module <NUM>, and an engine <NUM>. Some embodiments of the console <NUM> have different modules than those described in conjunction with <FIG>. Similarly, the functions further described below may be distributed among components of the console <NUM> in a different manner than is described here.

The application store <NUM> stores one or more applications for execution by the console <NUM>. An application is a group of instructions, that when executed by a processor, generates content for presentation to the user. Content generated by an application may be in response to inputs received from the user via movement of the NED <NUM> or the input/output interface <NUM>. Examples of applications include: gaming applications, conferencing applications, video playback application, or other suitable applications.

The tracking module <NUM> calibrates the system <NUM> using one or more calibration parameters and may adjust one or more calibration parameters to reduce error in determination of the position of the NED <NUM>. For example, the tracking module <NUM> adjusts the focus of the imaging device <NUM> to obtain a more accurate position on the NED <NUM>. Moreover, calibration performed by the tracking module <NUM> also accounts for information received from the IMU <NUM>. Additionally, if tracking of the NED <NUM> is lost, the tracking module <NUM> re-calibrates some or the entire system <NUM>.

The tracking module <NUM> tracks movements of the NED <NUM> using slow calibration information from the imaging device <NUM>. The tracking module <NUM> also determines positions of a reference point of the NED <NUM> using position information from the fast calibration information. Additionally, in some embodiments, the tracking module <NUM> may use portions of the fast calibration information, the slow calibration information, or some combination thereof, to predict a future location of the NED <NUM>. The tracking module <NUM> provides the estimated or predicted future position of the NED <NUM> to the engine <NUM>.

The engine <NUM> executes applications within the system <NUM> and receives position information, acceleration information, velocity information, predicted future positions, or some combination thereof of the NED <NUM> from the tracking module <NUM>. In some embodiments, the information received by the engine <NUM> may be used for producing a signal (e.g., switching instructions) to the display assembly <NUM> that determines the type of content presented to the user. For example, if the received information indicates that the user has looked to the left, the engine <NUM> generates content for the NED <NUM> that mirrors the user's movement in a virtual environment by determining the appropriate switching time of the array of liquid crystal gratings of the output waveguide in the display assembly <NUM>. For example, the engine <NUM> may produce a display instruction that would cause the display assembly <NUM> to generate content that would move along with the eye position tracked. Additionally, the engine <NUM> performs an action within an application executing on the console <NUM> in response to an action request received from the input/output interface <NUM> and provides feedback to the user that the action was performed. The provided feedback may be visual or audible feedback via the NED <NUM> or haptic feedback via the input/output interface <NUM>.

The foregoing description of the embodiments of the disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments of the disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

Embodiments of the disclosure may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Claim 1:
A near-eye-display (NED) system (<NUM>) comprising:
an eye tracking system (<NUM>) configured to:
track a location of an eye (<NUM>) of a user of the NED (<NUM>);
determine an eyebox location based in part on the tracked location of the user's eye (<NUM>); and
move the eyebox to one or more eyebox locations based on a movement of the user's eye (<NUM>);
a light source (<NUM>) configured to emit an image light (<NUM>);
an output waveguide (<NUM>) comprising:
an input grating (<NUM>) configured to receive the image light (<NUM>) emitted from the light source (<NUM>) and expand the received image light (<NUM>) in at least one dimension to transmit an expanded image light, and a dynamic output grating (<NUM>) comprising a plurality of liquid crystal (LC) pixels that are arranged in a <NUM>-dimensional array, the plurality of pixels including an active subset of LC pixels (<NUM>) that are configured to out-couple the expanded image light from the output waveguide (<NUM>) to one or more eyebox locations tracked by the eye tracking system (<NUM>) in accordance with switching instructions; and
a controller (<NUM>) configured to generate the switching instructions and provide the switching instructions to the dynamic output grating (<NUM>).