Patent Description:
The present disclosure generally relates to enhancing images from electronic displays, and specifically to a multiplanar display having two or more display planes located at different optical distances that can be dynamically adjusted based on a location within a scene presented by a head mounted display.

A head mounted display can be used to simulate artificial or virtual environments. For example, stereoscopic images can be displayed on an electronic display inside the headset to simulate the illusion of depth and head tracking sensors can be used to estimate what portion of the artificial environment is being viewed by the user. Such a simulation, however, can cause visual fatigue and nausea resulting from an inability of existing headsets to correctly render or otherwise compensate for vergence and accommodation conflicts.

<CIT> describes apparatuses, methods and systems that provide three dimensional gradient and dynamic light fields for display in 3D technologies, in particular 3D augmented reality (AR) devices, by coupling visual accommodation and visual convergence to the same plane at any depth of an object of interest in real time.

According to an aspect of the invention, there is provided a head mounted display according to claim <NUM>.

Further features according to embodiments of the invention are defined in the dependent claims.

A multiplanar head mounted display (FDVID) may include two or more virtual display planes for each eye located at optical distances that can be dynamically adjusted based on a location within a scene presented by the FDVID that the user views. According to the invention, a scene is presented on two or more electronic display elements (e.g., screens) of the FDVID. A focal length of an optics block that directs image light from the electronic display elements towards the eyes of a user is adjusted using a varifocal system (e.g., an element that mechanically changes a distance between a lens system in the optics block and the electronic display element, an element that changes shape of one or more lenses in the lens system in the optics block, etc.) based on a location or object within the scene where the user is looking.

A head mounted display (HMD) includes a first electronic display located at first optical distance from an optics block of the HMD and a second electronic display (for each eye of the user) located at a second optical distance from the optics block. Each electronic display is configured to display a different virtual plane of the virtual scene to the user wearing the HMD. Objects in the virtual scene are mapped to scene geometry or depth information provided by a content creator and each electronic display is mapped to a different subset of the objects based on their scene geometry. According to the invention, a first electronic display displaying a first virtual display plane located at a first optical distance displays objects within the virtual scene that are mapped to scene geometry at or near a first virtual display plane at the first optical distance. Accordingly, each subsequent electronic display displays different objects based on the location of the virtual display plane and the scene geometry. Accordingly, the virtual scene content from each of the virtual display planes is combined for viewing by the viewing user. Thus, at least one of the electronic displays is at least semi-transparent to allow image light of the virtual scene to be combined for viewing by the user wearing the HMD.

A head mounted display (HMD) not falling under the scope of the appended claims includes a single electronic display (for each eye of the user) that sequentially displays each of the virtual display planes. As above, objects in the virtual scene are mapped to scene geometry of the virtual scene and each of the two or more virtual planes display a different subset of the objects based on the scene geometry, but sequentially instead of simultaneously as described previously.

Moreover, the HMD automatically adjusts its focus based on a location within a virtual scene presented by the virtual reality headset that the user views. The virtual scene is presented on the one or more electronic displays and a focal length of an optics block that directs image light from the electronic displays to the eyes of the user is adjusted using a varifocal element (e.g., an element that mechanically changes a distance between a lens system in the optics block and the electronic display element, an element that changes shape of one or more lenses in the lens system in the optics block, etc.) based on a location or object within the virtual scene where the user is looking. According the the invention, the HMD tracks a user's eyes to approximate gaze lines and determines a gaze point including a vergence depth as an estimated point of intersection of the gaze lines. The gaze point identifying an object or plane of focus for a particular frame of the virtual scene presented to the user by the HMD.

Embodiments according to the invention are in particular disclosed in the attached claims directed to a HMD, wherein any feature mentioned in one claim category, e.g. HMD, can be claimed in another claim category, e.g. headset, system, method, storage medium, or computer program product, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof is disclosed and can be claimed regardless of the dependencies chosen as defined in the appended claims.

In an embodiment according to the invention, a HMD may comprise:
one or more optical combining elements, wherein the first electronic display is positioned perpendicular to the second electronic display and the one or more optical combining elements combine the first virtual plane of the virtual scene with the second virtual plane of the virtual scene for viewing by the user wearing the HMD.

At least one of the first electronic display and the second electronic display may be at least semi-transparent to allow image light of the virtual scene to be combined for viewing by the user wearing the HMD.

The varifocal actuation block may be configured to change the focal length of the optics block by changing a distance between the optics block and the first electronic display and the second electronic display.

The varifocal actuation block may be configured to change the focal length of the optics block by changing a shape or optical path length of a lens included in the optics block.

Changing the shape or optical path length of the lens of the optics block may include using at least one selected from a group consisting of: a shape-changing polymer lens, a liquid lens and electrowetting, an Alvarez-Lohmann lens, a deformable membrane mirror, a liquid crystal (electroactive) lens, a phase-only spatial light modulator (SLM), and any combination thereof.

The figures depict embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits touted, of the disclosure described herein.

A multiplanar head mounted display (HMD) includes two or more artificial display planes for each eye located at optical distances that can be dynamically adjusted based on a location within a scene presented by the HMD that the user views. According to the invention, a scene is presented on two or more electronic display elements (e.g., screens) of the HMD. A focal length of an optics block that directs image light from the electronic display elements towards the eyes of a user is adjusted using a varifocal system (e.g., an element that mechanically changes a distance between a lens system in the optics block and the electronic display element, an element that changes shape of one or more lenses in the lens system in the optics block, etc.) based on a location or object within the scene where the user is looking.

<FIG> shows example multiplanar display system <NUM> that comprises four separate electronic displays 102a, 102b, 102c, and 102d (collectively referred to as "electronic displays <NUM>") located at different distances relative to optics block <NUM> and exit pupil <NUM> of multiplanar display system <NUM>. Each electronic display <NUM>, in one embodiment, displays a different depth component or focal plane of a scene (e.g., a virtual reality scene) that together present the user with a complete image of the scene. The different electronic displays <NUM> may have dissimilar resolutions, refresh rates, number of color channels, brightness capabilities, or some combination thereof.

Electronic displays <NUM> produce multiple image planes. For example, each image of an artificial scene may include z-values corresponding to distance values for each pixel currently displayed. While one pixel is illuminated, the visually aligned pixels of the other electronic displays <NUM> remain transparent.

Optics block <NUM> presents content from one or more of electronic displays <NUM> at specific focal planes based in part on their position relative to the optics block. Thus, the position of each electronic display <NUM> relative to optics block <NUM> has a corresponding focal plane <NUM>. A varifocal system can change the location of focal plane <NUM>, such as by changing position <NUM> of optics block <NUM> relative to electronic displays <NUM> or other property, based on a location within the scene corresponding to where the user is looking. Instead of mechanically moving optics block <NUM> to change the location of focal plane <NUM>, as shown in <FIG>, properties of optics block <NUM> affecting the focal length could be varied to change to location of focal plane <NUM> or the positions of one or more of electronic displays <NUM> could change relative to each other and/or exit pupil <NUM>.

To determine the location or object within the scene where the user is looking, multiplanar display system <NUM> of an HMD includes an eye tracking system configured to detect vergence and/or accommodation and other viewing parameters (e.g., position, orientation, and pupil size). Thus, multiplanar display system <NUM> tracks a user's eyes and approximates a gaze point and vergence depth based on the detected vergence and other viewing parameters and the varifocal system adjusts the power of the optics block to provide accommodation for the eyes of the user at the vergence depth.

Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

<FIG> shows example multiplanar display system <NUM> that, in one embodiment not falling under the scope of the appended claims, comprises a single electronic display <NUM> located a distance from optics block <NUM>, an eye tracking system configured to detect vergence and/or accommodation and other viewing parameters (e.g., position, orientation, and pupil size), and a varifocal system adjusts the focus of optics block <NUM> to a location corresponding to the detected vergence. When illuminated, light emitted from single electronic display <NUM> is presented at a particular focal depth as an artificial display plane. The location of the artificial focal plane is based in part on a focal setting of the optics block <NUM> that is set by the varifocal system. Electronic display <NUM> may display multiple artificial display planes by illuminating (e.g., emitting image light) electronic display <NUM> multiple times while changing the fixed focus of optics block <NUM> using the varifocal system. Thus, electronic display <NUM>, in one embodiment, is a high-refresh-rate display (e.g., <NUM> times the number of display planes). The varifocal element may then oscillate in a fixed focus pattern (e.g., sinusoidal oscillation from <NUM> to <NUM> diopters) and electronic display <NUM> refresh can be timed to create artificial display planes at arbitrary optical depths. The optical depth of the artificial display planes are then adjusted based on vergence, position, orientation, pupil size, and other measurements received from the eye tracking system.

At least one artificial display plane is rendered at a plane close to an estimated accommodation plane of the viewer based on a viewing location of the user within the artificial scene. The images of the artificial scene are, thus, displayed with each artificial display plane as a layer to maximize a visual quality metric using a multiplane decomposition algorithm. Further, the artificial planes at which objects in the artificial scene are displayed can be based on scene geometry, depth information provided by a content creator, or on information from the eye tracking system.

<FIG> shows example multiplanar display system <NUM> that, in one embodiment, comprises two electronic displays 102a and 102b, optical combining element <NUM>, such as a beam splitter, and an eye tracking system and a varifocal system, as similarly described above. In this example, electronic displays 102a and 102b are positioned perpendicular relative to each other and optical combining element <NUM> is orientated at an angle relative to both electronic displays 102a and 102b to combine images of the two displays. The two electronic displays 102a and 102b can produce multiple artificial display planes within a scene and the optical depth of each of electronic displays 102a and 102b can be adjusted by the varifocal system. For example, electronic display 102a may display portions of the scene image located in an artificial plane closer to the user relative to portions of the scene image displayed by electronic display 102b.

<FIG> shows example multiplanar display system <NUM> that, in one embodiment, comprises two electronic displays 102a and 102b and an eye tracking system and a varifocal system, as similarly described above. The two electronic displays 102a and 102b are, in this example, positioned in optical series at different distances relative to optics block <NUM> and, thus, may be semi-transparent (e.g., a stack of TOLEDs or LCDs). Additionally, electronic displays 102a and 102b may not only emit light, but may also absorb it (e.g., LCDs).

Electronic displays 102a and 102b, in one embodiment, may each display a set of different artificial planes of the scene that together present the user with a complete image of the scene. The different set of artificial planes may also correspond to artificial planes that do not need to be rendered in the highest resolution and, thus, electronic displays 102a and 102b may each have dissimilar resolutions, refresh rates, number of color channels, brightness capabilities, and so forth. Accordingly, multiple artificial display planes or layers of the scene are produced by two electronic displays 102a and 102b and the optical depth of each display is adjusted, as above, using the varifocal system to keep eye <NUM> of the user in accommodation as the user's vergence depth, as detected by the eye tracking system.

<FIG> shows multi-planar varifocal system <NUM> in which a head-mounted display (HMD) <NUM> operates. Varifocal system <NUM> may be for use as a virtual reality (VR) system, an augmented reality (AR) system, a mixed reality (MR) system, or some combination thereof. In this example, varifocal system <NUM> includes HMD <NUM>, imaging device <NUM>, and I/O interface <NUM>, which are each coupled to console <NUM>. While <FIG> shows a single HMD <NUM>, a single imaging device <NUM>, and a single I/O interface <NUM>, in other embodiments, any number of these components may be included in the system. For example, there may be multiple HMDs <NUM> each having an associated I/O interface <NUM> and being monitored by one or more imaging devices <NUM>, with each HMD <NUM>, I/O interface <NUM>, and imaging devices <NUM> communicating with the console <NUM>. In alternative configurations, different and/or additional components may also be included in the system environment.

HMD <NUM> presents content to a user. Example content includes images, video, audio, or some combination thereof. Audio content may be presented via a separate device (e.g., speakers and/or headphones) external to HMD <NUM> that receives audio information from HMD <NUM>, console <NUM>, or both. HMD <NUM> includes electronic display(s) <NUM>, optics block <NUM>, varifocal actuation block <NUM>, focus prediction module <NUM>, eye tracking module <NUM>, vergence processing module <NUM>, one or more locators <NUM>, internal measurement unit (IMU) <NUM>, head tracking sensors <NUM>, and scene rendering module <NUM>.

Optics block <NUM> directs light from electronic display(s) <NUM> to an exit pupil for viewing by a user using one or more optical elements, such as apertures, Fresnel lenses, convex lenses, concave lenses, filters, and so forth, and may include combinations of different optical elements. In some embodiments, one or more optical elements in optics block <NUM> may have one or more coatings, such as anti-reflective coatings. Magnification of the image light by optics block <NUM> allows electronic display(s) <NUM> to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification of the image light may increase a field of view of the displayed content. For example, the field of view of the displayed content is such that the displayed content is presented using almost all (e.g., <NUM> degrees diagonal), and in some cases all, of the user's field of view.

Optics block <NUM> may be designed to correct one or more optical errors. Examples of optical errors include: barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and so forth. In some embodiments, content provided to electronic display(s) <NUM> for display is pre-distorted, and optics block <NUM> corrects the distortion when it receives image light from electronic display(s) <NUM> generated based on the content.

Varifocal actuation block <NUM> includes a varifocal element that causes optics block <NUM> to vary the focal length (or optical power) of HMD <NUM> in order to keep a user's eyes in a zone of comfort as vergence and accommodation change. In one embodiment, varifocal actuation block <NUM> physically changes the distance between electronic display(s) <NUM> and optical block <NUM> by moving electronic display(s) <NUM> or optical block <NUM> (or both). Alternatively, varifocal actuation block <NUM> changes the focal length of optics block <NUM> by adjusting one or more properties of one or more lenses. Example properties of a lens adjusted by the varifocal actuation block include: an optical path length, an index of refraction of a lens medium, a shape of a lens, and so forth. For example, varifocal actuation block <NUM> changes the focal length of the one or more lenses using shape-changing polymer lenses, electrowetting methods with liquid lenses, Alvarez-Lohmann lenses, deformable membrane mirrors, liquid crystal (electroactive) lenses, or phase-only spatial light modulators (SLMs), or any other suitable component. Additionally, moving or translating two or more lenses relative to each other may also be used to change the focal length of HMD <NUM>. Thus, varifocal actuation block <NUM> may include actuators or motors that move one or more of electronic display(s) <NUM> and/or optical block <NUM> on a track to change the distance between them or may include actuators and other components or mechanisms for changing the properties of one or more lenses included in optics block <NUM>. Varifocal actuation block <NUM> may be separate from or integrated into optics block <NUM> in various embodiments.

Each state of optics block <NUM> corresponds to a focal length of HMD <NUM> or to a combination of the focal length and eye position relative to optics block <NUM> (as discussed further below). In operation, optics block <NUM> may move in a range of ~<NUM> with a positional accuracy of ∼<NUM> for a granularity of around <NUM> focal lengths, corresponding to <NUM> states of optics block <NUM>. Any number of states could be provided; however, a limited number of states accommodate the sensitivity of the human eye, allowing some embodiments to include fewer focal lengths. For example, a first state could correspond to a focal length of a theoretical infinity meters (<NUM> diopter), a second state could correspond to a focal length of <NUM> meters (<NUM> diopter), a third state could correspond to a focal length of <NUM> meters (<NUM> diopter), a fourth state could correspond to a focal length of <NUM> meters (<NUM> diopter), a fifth state could correspond to a focal length of <NUM> meters (<NUM> diopter), and a sixth state could correspond to a focal length of <NUM> meters (<NUM> diopter). Varifocal actuation block <NUM>, thus, sets and changes the state of optics block <NUM> to achieve a desired focal length.

Focus prediction module <NUM> is an encoder including logic that tracks the state of optics block <NUM> to predict to one or more future states or locations of optics block <NUM>. For example, focus prediction module <NUM> accumulates historical information corresponding to previous states of optics block <NUM> and predicts a future state of optics block <NUM> based on the previous states. Because rendering of a virtual scene by HMD <NUM> is adjusted based on the state of optics block <NUM>, the predicted state allows scene rendering module <NUM>, further described below, to determine an adjustment to apply to the virtual scene for a particular frame. Accordingly, focus prediction module <NUM> communicates information describing a predicted state of optics block <NUM> for a frame to scene rendering module <NUM>. Adjustments for the different states of optics block <NUM> performed by scene rendering module <NUM> are further described below.

Eye tracking module <NUM> tracks an eye position and eye movement of a user of HMD <NUM>. In one embodiment, a camera or other optical sensor inside HMD <NUM> captures image information of a user's eyes, and eye tracking module <NUM> uses the captured information to determine interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye relative to HMD <NUM> (e.g., for distortion adjustment purposes), including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye. In one example, infrared light is emitted within HMD <NUM> and reflected from each eye. The reflected light is received or detected by the camera and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of a user can be used by eye tracking module <NUM>. Accordingly, eye tracking module <NUM> may track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes of a user to estimate a gaze point (i.e., a 3D location or position in the virtual scene where the user is looking). For example, eye tracking module <NUM> integrates information from past measurements, measurements identifying a position of a user's head, and 3D information describing a scene presented by electronic display element <NUM>. Thus, information for the position and orientation of the user's eyes is used to determine the gaze point in a virtual scene presented by HMD <NUM> where the user is looking.

Further, 3D location of a pupil relative to optics block <NUM> changes as the eye moves to look in different directions. The varying 3D location of the pupil relative to optics block <NUM> as viewing direction changes contributes to distortion perceived by the user as "pupil swim. " Accordingly, measuring distortion at different 3D eye positions relative to optics block <NUM> and generating distortion corrections for different positions and distances allows mitigation of distortion caused by "pupil swim" by tracking the 3D position of a user's eyes and applying a distortion correction corresponding to the 3D position of each of the user's eye at a given point in time. Thus, knowing the 3D position of each of a user's eyes allows for the mitigation of distortion caused by changes in the distance between the pupil of the eye and optics block <NUM> by applying a distortion correction for each 3D eye position. Methods for reducing pupil swim are further described in <CIT> which is hereby incorporated by reference in its entirety.

In addition to contributing to distortion perceived by the user as "pupil swim," the varying 3D location of the pupil can cause the user to perceive parallax in a multiplanar display system. For example, the varying 3D location of the pupil within the eyebox relative to electronic displays displaying images at different depths relative to the eyebox can cause problems in how the final image (or composite image) appears to the user. Accordingly, by tracking the 3D location of each of a user's eyes, a parallax correction can be rendered into one or more frames of the content displayed by the multiple electronic displays based on a current location of the user's 3D eye position to account for parallax.

Vergence processing module <NUM> determines a vergence depth of a user's gaze based on the gaze point or an estimated intersection of the gaze lines determined by eye tracking module <NUM>. Vergence is the simultaneous movement or rotation of both eyes in opposite directions to maintain single binocular vision, which is naturally and automatically performed by the human eye. Thus, a location where a user's eyes are verged is where the user is looking and is also typically the location where the user's eyes are focused. For example, vergence processing module <NUM> triangulates the gaze lines to estimate a distance or depth from the user associated with intersection of the gaze lines. The depth associated with intersection of the gaze lines can then be used as an approximation for the accommodation distance, which identifies a distance from the user where the user's eyes are directed. Thus, the vergence distance allows determination of a location where the user's eyes should be focused and a depth from the user's eyes at which the eyes are focused, thereby, providing information, such as an object or plane of focus, for rendering adjustments to the virtual scene.

In some embodiments, rather than provide accommodation for the eye at a determined vergence depth, accommodation may be directly determined by a wavefront sensor, such as a Shack-Hartmann wavefront sensor; hence, a state of optics block <NUM> may be a function of the vergence or accommodation depth and the 3D position of each eye, so optics block <NUM> brings objects in a scene presented by electronic display element <NUM> into focus for a user viewing the scene. Further, vergence and accommodation information may be combined to focus optics block <NUM> and to render synthetic depth of field blur.

Locators <NUM> are objects located in specific positions on HMD <NUM> relative to one another and relative to a specific reference point on HMD <NUM>. Locator <NUM> may be a light emitting diode (LED), a corner cube reflector, a reflective marker, a type of light source that contrasts with an environment in which HMD <NUM> operates, or some combination thereof. Active locators <NUM> (i.e., an LED or other type of light emitting device) may emit light in the visible band (~<NUM> to <NUM>), in the infrared (IR) band (~<NUM> to <NUM>), in the ultraviolet band (<NUM> to <NUM>), some other portion of the electromagnetic spectrum, or some combination thereof.

Locators <NUM> can be located beneath an outer surface of HMD <NUM>, which is transparent to the wavelengths of light emitted or reflected by locators <NUM> or is thin enough not to substantially attenuate the wavelengths of light emitted or reflected by locators <NUM>. Further, the outer surface or other portions of HMD <NUM> can be opaque in the visible band of wavelengths of light. Thus, locators <NUM> may emit light in the IR band while under an outer surface of HMD <NUM> that is transparent in the IR band but opaque in the visible band.

IMU <NUM> is an electronic device that generates fast calibration data based on measurement signals received from one or more of head tracking sensors <NUM>, which generate one or more measurement signals in response to motion of HMD <NUM>. Examples of head tracking sensors <NUM> include accelerometers, gyroscopes, magnetometers, other sensors suitable for detecting motion, correcting error associated with IMU <NUM>, or some combination thereof. Head tracking sensors <NUM> may be located external to IMU <NUM>, internal to IMU <NUM>, or some combination thereof.

Based on the measurement signals from head tracking sensors <NUM>, IMU <NUM> generates fast calibration data indicating an estimated position of HMD <NUM> relative to an initial position of HMD <NUM>. For example, head tracking sensors <NUM> include multiple accelerometers to measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes to measure rotational motion (e.g., pitch, yaw, and roll). IMU <NUM> can, for example, rapidly sample the measurement signals and calculate the estimated position of HMD <NUM> from the sampled data. For example, IMU <NUM> integrates measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on HMD <NUM>. The reference point is a point that may be used to describe the position of HMD <NUM>. While the reference point may generally be defined as a point in space, in various embodiments, reference point is defined as a point within HMD <NUM> (e.g., a center of the IMU <NUM>). Alternatively, IMU <NUM> provides the sampled measurement signals to console <NUM>, which determines the fast calibration data.

IMU <NUM> can additionally receive one or more calibration parameters from console <NUM>. As further discussed below, the one or more calibration parameters are used to maintain tracking of HMD <NUM>. Based on a received calibration parameter, IMU <NUM> may adjust one or more IMU parameters (e.g., sample rate). In some embodiments, certain calibration parameters cause IMU <NUM> to update an initial position of the reference point to correspond to a next calibrated position of the reference point. Updating the initial position of the reference point as the next calibrated position of the reference point helps reduce accumulated error associated with determining the estimated position. The accumulated error, also referred to as drift error, causes the estimated position of the reference point to "drift" away from the actual position of the reference point over time.

Scene render module <NUM> receives content for the virtual scene from engine <NUM> and provides the content for display on electronic display(s) <NUM>. Additionally, scene render module <NUM> can adjust the content based on information from focus prediction module <NUM>, vergence processing module <NUM>, IMU <NUM>, and head tracking sensors <NUM>. For example, upon receiving the content from engine <NUM>, scene render module <NUM> adjusts the content based on the predicted state (i.e., eye position and focal length) of optics block <NUM> received from focus prediction module <NUM> by adding a correction or pre-distortion into rendering of the virtual scene to compensate or correct for the distortion caused by the predicted state of optics block <NUM>. Scene render module <NUM> may also add depth of field blur based on the user's gaze, vergence depth (or accommodation depth) received from vergence processing module <NUM>, or measured properties of the user's eye (e.g., 3D position of the eye, etc.). Additionally, scene render module <NUM> determines a portion of the content to be displayed on electronic display(s) <NUM> based on one or more of tracking module <NUM>, head tracking sensors <NUM>, or IMU <NUM>, as described further below.

Imaging device <NUM> generates slow calibration data in accordance with calibration parameters received from console <NUM>. Slow calibration data includes one or more images showing observed positions of locators <NUM> that are detectable by imaging device <NUM>. Imaging device <NUM> may include one or more cameras, one or more video cameras, other devices capable of capturing images including one or more locators <NUM>, or some combination thereof. Additionally, imaging device <NUM> may include one or more filters (e.g., for increasing signal to noise ratio). Imaging device <NUM> is configured to detect light emitted or reflected from locators <NUM> in a field of view of imaging device <NUM>. In embodiments where locators <NUM> include passive elements (e.g., a retroreflector), imaging device <NUM> may include a light source that illuminates some or all of locators <NUM>, which retro-reflect the light towards the light source in imaging device <NUM>. Slow calibration data is communicated from imaging device <NUM> to console <NUM>, and imaging device <NUM> receives one or more calibration parameters from console <NUM> to adjust one or more imaging parameters (e.g., focal length, focus, frame rate, ISO, sensor temperature, shutter speed, aperture, etc.).

I/O interface <NUM> is a device that allows a user to send action requests to console <NUM>. An action request is a request to perform a particular action. For example, an action request may be to start or end an application or to perform a particular action within the application. I/O interface <NUM> may include one or more input devices. Example input devices include a keyboard, a mouse, a game controller, or any other suitable device for receiving action requests and communicating the received action requests to console <NUM>. An action request received by I/O interface <NUM> is communicated to console <NUM>, which performs an action corresponding to the action request. In some embodiments, I/O interface <NUM> may provide haptic feedback to the user in accordance with instructions received from console <NUM>. For example, haptic feedback is provided by the I/O interface <NUM> when an action request is received, or console <NUM> communicates instructions to I/O interface <NUM> causing I/O interface <NUM> to generate haptic feedback when console <NUM> performs an action.

Console <NUM> provides content to HMD <NUM> for presentation to the user in accordance with information received from imaging device <NUM>, HMD <NUM>, or I/O interface <NUM>. In the example shown in <FIG>, console <NUM> includes application store <NUM>, tracking module <NUM>, and virtual reality (VR) engine <NUM>. Some embodiments of console <NUM> have different or additional modules than those described in conjunction with <FIG>. Similarly, the functions further described below may be distributed among components of console <NUM> in a different manner than is described here.

Application store <NUM> stores one or more applications for execution by console <NUM>. An application is a group of instructions, that when executed by a processor, generates content for presentation to the user. Content generated by an application may be in response to inputs received from the user via movement of HMD <NUM> or interface device <NUM>. Examples of applications include gaming applications, conferencing applications, video playback application, or other suitable applications.

Tracking module <NUM> calibrates the system using one or more calibration parameters and may adjust one or more calibration parameters to reduce error in determining position of HMD <NUM>. For example, tracking module <NUM> adjusts the focus of imaging device <NUM> to obtain a more accurate position for observed locators <NUM> on HMD <NUM>. Moreover, calibration performed by tracking module <NUM> also accounts for information received from IMU <NUM>. Additionally, if tracking of HMD <NUM> is lost (e.g., imaging device <NUM> loses line of sight of at least a threshold number of locators <NUM>), tracking module <NUM> re-calibrates some or all of the system components.

Additionally, tracking module <NUM> tracks the movement of HMD <NUM> using slow calibration information from imaging device <NUM> and determines positions of a reference point on HMD <NUM> using observed locators from the slow calibration information and a model of HMD <NUM>. Tracking module <NUM> also determines positions of the reference point on HMD <NUM> using position information from the fast calibration information from IMU <NUM> on HMD <NUM>. Additionally, tracking module <NUM> may use portions of the fast calibration information, the slow calibration information, or some combination thereof, to predict a future location of HMD <NUM>, which is provided to engine <NUM>.

Engine <NUM> executes applications within the system and receives position information, acceleration information, velocity information, predicted future positions, or some combination thereof for HMD <NUM> from tracking module <NUM>. Based on the received information, engine <NUM> determines content to provide to HMD <NUM> for presentation to the user, such as a virtual scene. For example, if the received information indicates that the user has looked to the left, engine <NUM> generates content for HMD <NUM> that mirrors or tracks the user's movement in a virtual environment. Additionally, engine <NUM> performs an action within an application executing on console <NUM> in response to an action request received from the I/O interface <NUM> and provides feedback to the user that the action was performed. The provided feedback may be visual or audible feedback via HMD <NUM> or haptic feedback via I/O interface <NUM>.

As discussed above, multi-planar varifocal system <NUM> may dynamically vary its focus to bring images presented to a user wearing HMD <NUM> into focus, which keeps the user's eyes in a zone of comfort as vergence and accommodation change.

Accordingly, a position, an orientation, and/or a movement of HMD <NUM> are determined by a combination of locators <NUM>, IMU <NUM>, head tracking sensors <NUM>, imagining device <NUM>, and tracking module <NUM>, as described above in conjunction with <FIG>. Portions of a virtual scene presented by HMD <NUM> are mapped to various positions and orientations of HMD <NUM>. Thus, a portion of the virtual scene currently viewed by a user is determined based on the position, orientation, and movement of HMD <NUM>. After determining the portion of the virtual scene being viewed by the user, the system may then determine a location or an object within the determined portion at which the user is looking to adjust focus for that location or object accordingly.

To determine the location or object within the determined portion of the virtual scene at which the user is looking, HMD <NUM> tracks the position and location of the user's eyes. Thus, HMD <NUM> determines an eye position for each eye of the user. For example, HMD <NUM> tracks at least a subset of the 3D position, roll, pitch, and yaw of each eye and uses these quantities to estimate a 3D gaze point of each eye. Further, information from past eye positions, information describing a position of the user's head, and information describing a scene presented to the user may also be used to estimate the 3D gaze point of an eye in various embodiments. For example, referring to <FIG>, HMD <NUM> includes camera <NUM> for tracking the position of each eye <NUM>. In this example, camera <NUM> captures images of the user's eyes and eye tracking module <NUM> determines an output for each eye <NUM> and gaze lines <NUM> corresponding to the gaze point or location where the user is looking based on the captured images.

<FIG>, further show vergence depth (dv) <NUM> of the gaze point for the user is determined based on an estimated intersection of gaze lines <NUM>. As shown in <FIG>, gaze lines <NUM> converge or intersect at dv <NUM>, where object <NUM> is located. Because virtual distances within the virtual scene are known to the system, the vergence depth <NUM> can be filtered or verified to determine a more accurate vergence depth for the virtual scene. For example, vergence depth <NUM> is an approximation of the intersection of gaze lines <NUM>, which are themselves an approximation based on the position of a user's eyes <NUM>. Gaze lines <NUM> do not always appear to accurately intersect. Thus, virtual distances within the virtual scene can be compared to the vergence depth for the portion of the virtual scene to generate a vergence depth. Determining a more accurate vergence depth or gaze point enables the virtual scene to more accurately determine a user's object or plane of focus, allowing scene rendering module <NUM> to add depth of field blur to proper depths and/or objects in the virtual scene or otherwise modify to virtual scene to appear more realistic.

Accordingly, a state of optics block <NUM> is determined for a frame of the virtual scene based on states of optics block <NUM> during presentation of previous frames of the virtual scene. For example, focus prediction module <NUM> tracks the state of optics block <NUM> for various frames of the virtual scene to predict to future a state of optics block <NUM> for subsequent frames of the virtual scene. The predicted state of optics block <NUM> (e.g., a predicted location of optics block <NUM>) allows the scene rendering module <NUM> to determine an adjustment to apply to a frame of the virtual scene so distortion caused by the predicted state of optics block <NUM> corrects or cancels the applied adjustment rather than distorting the frame. Thus, based on the state of optics block <NUM>, a distortion correction is determined for application to a frame of the virtual scene to correct optical error introduced by the state of optics block <NUM>.

<FIG> shows an example of how the human eye experiences vergence and accommodation in the real world. Vergence is the simultaneous movement or rotation of both eyes in opposite directions to obtain or maintain single binocular vision and is connected to accommodation of the eye. Under normal conditions, changing the focus of the eyes to look at an object at a different distance automatically causes vergence and accommodation. In the example of <FIG>, the user is looking at real object 700A (i.e., the user's eyes are verged on real object 700A and gaze lines from the user's eyes intersect at real object 700A. As real object 700A is moved closer to the user, as indicated by the arrow in <FIG>, each eye <NUM> rotates inward to stay verged on real object 700A. As real object 700A gets closer, eye <NUM> must "accommodate" for the closer distance by reducing the power or focal length of eye <NUM> by changing its shape. Thus, under normal conditions in the real world, the vergence depth (dv) equals the focal length (df).

However, <FIG> shows an example conflict between vergence and accommodation that can occur with some three-dimensional displays. In this example, a user is looking at virtual object 700B displayed on 3D electronic screen <NUM>; however, the user's eyes are verged on and gaze lines from the user's eyes intersect at virtual object 700B, which is a greater distance from the user's eyes than 3D electronic screen <NUM>. As virtual object 700B is rendered on 3D electronic display <NUM> to appear closer to the user, each eye <NUM> again rotates inward to stay verged on virtual object 700B, but the power or focal length of each eye is not reduced; hence, the user's eyes do not accommodate as in <FIG>. Thus, instead of reducing power or focal length to accommodate for the closer vergence depth, eye <NUM> maintains accommodation at a distance associated with 3D electronic display <NUM>. Thus, the vergence depth (dv) often does not equal the focal length (df) for the human eye for objects displayed on 3D electronic displays. This discrepancy between vergence depth and focal length is referred to as "vergence-accommodation conflict. " A user experiencing only vergence or accommodation and not both will eventually experience some degree of fatigue and nausea, which is undesirable desirable for virtual reality system creators. Changes in vergence for a 3D electronic screen may be accommodated by a headset dynamically adjusting the power of an optics block based on the vergence depth (or predicted vergence depth).

Accordingly, the focal length (or power) of optics block <NUM> is adjusted for the presented frame of the virtual scene to provide accommodation for the generated vergence depth. <FIG> show an example process for adjusting the focal length of optics block <NUM> by varying the distance between electronic display(s) <NUM> and optics block <NUM> using varifocal element <NUM>. In the example of <FIG>, varifocal actuation block <NUM> includes varifocal element <NUM>, such as an actuator or motor and track <NUM>, but may also include other components enabling optics block <NUM>, electronic display(s) <NUM> (e.g., 802a, 802b), or both to move along track <NUM> to dynamically adjust the optical power of optics block <NUM>.

<FIG> shows an example of HMD <NUM> providing focus adjustment for frame n of a virtual scene. In this example, virtual scene includes object <NUM> displayed on electronic display 802a at which the gaze of user <NUM> is directed (i.e., verged). A virtual image of object <NUM> is located a virtual distance di, behind electronic display 802a, from exit pupil <NUM>. In the example of <FIG>, optics block <NUM> is in position pi, which provides accommodation for distance di to enable comfortable viewing of object <NUM>.

<FIG> shows HMD <NUM> providing focus adjustment for a subsequent frame n+<NUM> of the virtual scene. In this example, user <NUM> may have repositioned its eyes as object <NUM> quickly moved toward user <NUM> in the virtual scene. As a result, the virtual image of object <NUM> is located relatively close to or on electronic display 802b. In response to the location of object <NUM> closer, eyes of user <NUM> rotate inward to verge on object <NUM>, causing vergence processing module <NUM> to determine a new vergence depth for frame n+<NUM> and to provide the new vergence depth to varifocal actuation block <NUM>. Based on the new vergence depth, varifocal element <NUM> moves optics block <NUM> from position pi to new position pf to accommodate user <NUM> at the new vergence depth df for the closer object <NUM>.

In one example, each state of optics block <NUM> corresponds to a combination of focal length and eye position, provides accommodation for a range of vergence depths, and is associated with a specific position of optics block <NUM>. Accordingly, vergence depths may be mapped to positions of optics block <NUM> and stored in a lookup table. Thus, when a vergence depth is received from vergence processing module <NUM>, varifocal actuation block <NUM> automatically moves optics block <NUM> to a position corresponding to the received vergence depth based on the lookup table.

In many instances, virtual reality systems aim to present users with a virtual environment that closely simulates a real world environment or provides users with content causing the users to get lost in the illusion created by the virtual reality systems. To provide users with a realistic or captivating virtual environment, a virtual reality system implements multiple systems and methods discussed herein to operate together at efficiencies that are imperceptible to a user. For example, transition delays are particularly costly to user experience with virtual reality systems. If a user is waiting for the virtual scene presented by a headset to catch up to what the user's brain is already expecting, the illusion is broken and/or the user may get nauseous. However, processing speeds and commercially available actuators are currently faster than the coordination of the human eye to change the shape of its lens and the human brain to register what the new shape of the lens is focused on, allowing the disclosed systems and methods to provide users with high-quality virtual environments.

Referring back to <FIG> to provide accommodation for a new vergence depth while also leaving time to perform additional calculations without users perceiving a delay, a speed at which varifocal element <NUM> moves optics block <NUM> is limited by a rate at which the human eye performs accommodation. For example, assuming human eye accommodation has a <NUM> diopter/sec peak velocity, a <NUM> diopter/sec<NUM> peak acceleration, and changing the distance between electronic display(s) <NUM> and optics block <NUM> moves a virtual image about <NUM> diopters/mm, varifocal element <NUM> operates with a minimum velocity of <NUM>/<NUM> = <NUM>/sec and a minimum acceleration of <NUM>/<NUM> = <NUM>/sec<NUM> acceleration to prevent a user from perceiving the repositioning of optics block <NUM> relative to electronic display(s) <NUM>. There are commercially available actuators satisfying the preceding values.

Further, depth of field blur can be determined for the scene based on the user's viewing location. To determine depth of field blur, a point within the scene presented to the user by the HMD <NUM> where the user's gaze is directed is determined, and optics block <NUM> is configured to a state in which the point within the scene where the user's gaze is directed is brought into focus for the user. Depth of field blur is then determined relative to the point within the scene where the user's gaze is directed. In one example, the depth within the scene geometry (e.g., distances within the virtual scene) corresponding to the vergence depth is determined as the plane of focus for the frame of the virtual scene. Accordingly, objects or features of the virtual environment with distances within the virtual scene greater or less than a distance of the plane of focus from the user's eyes may be rendered with synthetic blur. In another example, the depth of field blur is determined based on an object in the scene on which the user's gaze is focused at the depth corresponding to the vergence depth (i.e., a "focal object"). Thus, the focal object, rather than the plane of focus, provides a reference point to identify other objects in the scene that are rendered with depth of field blur, even if the other objects have a similar depth in the scene as the focal object.

The blur may be progressive, with as a level of blur applied to objects or features based on a distance of the objects or features from the plane of focus (or object of focus), or a generally uniform level of blur may be applied to objects or features in the virtual scene. Depth of field blur is a natural consequence of binocular vision, so including depth of field blur in the virtual scene furthers to the illusion of the virtual scene by providing the user with an expected depth cue, which may enhance the user experience with the virtual scene. Further, the blur may be based at least in part on measured properties of the user's eye. For example, wavefront aberrations of the user's eye could be measured by a wavefront aberrometer, with depth of field blur based at least in part on the measured wavefront aberrations. Example wavefront aberrations of the user's eye may include higher-order aberrations not typically corrected by eye glasses, contact lenses, or refractive surgery. Accounting for properties of the user's eye when determining the depth of field blur may improve user comfort when viewing the scene.

The frame of the virtual scene corresponding to the portion of the virtual scene being viewed by the user is displayed on electronic display(s) <NUM> with a distortion correction to correct optical error caused by the determined state of optics block <NUM> and with depth of field blur based on the vergence depth. Further, varifocal actuation block <NUM> has changed the focus of optics block <NUM> to provide focus and accommodation to the location in the portion of the virtual scene where the user's eyes are verged.

The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Claim 1:
A head mounted display (HMD) (<NUM>, <NUM>) comprising:
at least one processor (<NUM>);
a multiplanar display comprising:
a first electronic display (102a, 602a) configured to display a first virtual plane within a virtual scene to a user wearing the HMD;
a second electronic display (102b, 602b) configured to display a second virtual plane of the virtual scene to the user wearing the HMD, the first virtual plane being located at a different optical depth to the second virtual plane,
wherein objects in the virtual scene are mapped to scene geometry or depth information provided by a content creator, and
wherein the first electronic display is mapped to a first subset of said objects based on their scene geometry and the second electronic display is mapped to a different second subset of said objects based on their scene geometry;
an optics block (<NUM>) configured to direct light from the first electronic display and the second electronic display to an exit pupil of the HMD;
an eye tracking system (<NUM>) including an image capturing element (<NUM>), the eye tracking system configured to determine an eye position of each eye of the user and gaze lines for each eye of the user;
memory including instructions that, when executed by the at least one processor, cause the at least one processor to:
determine a vergence depth for the user based on an estimated intersection of the gaze lines for each eye of the user; and
a varifocal actuation block (<NUM>) configured to change a focal length of the optics block based at least in part on the vergence depth for each eye of the user corresponding to a viewing location of the user within the virtual scene.