Lens Distance Test for Head-Mounted Display Devices

Systems and methods are disclosed to enable performance of a lens distance test in head-mounted displays (HMDs) to determine the distance between a user's eye and a lens of the HMD (e.g. the display lens in a virtual or augmented reality device). In embodiments, the HMD is configured to determine a current pose of the eye based on a series of captured eye images. The pose information is used to determine the distance from the apex of the cornea to a closest point on the lens. If the determined distance is too small or too large, an alert or notification is generated instructing to adjust the HMD or change the light seal to achieve better distancing, in order to reduce the risk of eye injury and/or improve user experience. In embodiments, the lens distance test may be repeated during a user session to reevaluate and/or monitor the lens distance.

BACKGROUND

Virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene. When the user views images through a virtual reality system, the user may thus feel as if they are moving within the scenes from a first-person point of view. Similarly, mixed reality (MR) or augmented reality (AR) systems combine computer generated information (referred to as virtual content) with real world images or a real world view to augment, or add content to, a user's view of the world. The simulated environments of VR and/or the mixed environments of MR may thus be utilized to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the viewer's environment, interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the Internet, or the like.

SUMMARY

Various embodiments of methods and apparatus for performing a lens distance test in head-mounted displays (HMDs), such as AR or VR headsets. The distance test is performed to determine the distance between a user's eye and a lens of the HMD (e.g. the display lens of the HMD or a clip-on lens). In some embodiments, the HMD is configured to determine a current pose of the eye based on a series of captured eye images. The pose may be determined using a gaze tracking system implemented by the HMD, and is used by the HMD to determine the distance from the apex of the cornea to a closest point on the lens. If the determined distance is too small or too large, an alert or notification is generated instructing the user to adjust the position of the HMD or change the light seal to achieve better distancing, so as to reduce the risk of eye injury and/or improve the user experience. In some embodiments, the distance test may be repeated during a user session to reevaluate and/or monitor the lens distance.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus are described for performance of a lens distance test in head-mounted display (HMD) devices. HMDs may include devices such as headsets, helmets, goggles, or glasses, etc., that are designed to be worn by a user and include a display mechanism (e.g., left and right near-eye display panels) for displaying visual content to the user. In some embodiments, the display mechanism may include displays for both eyes of the user to provide 3D visual views to the user. In some embodiments, the HMD may be a virtual reality (VR) or augmented reality (AR) device. For AR applications, the HMD may include or be coupled to one or more external video cameras that capture video of the user's environment for display. The HMD may include a controller component that renders frames for display to the left and right displays. Alternatively, the controller component may be implemented by an external device that is coupled to the HMD via a wired or wireless connection.

In some embodiments, the HMD may include left and right optical lenses (e.g. display lenses) located between the display and the user's eyes. The distance between the user's eye (e.g. the cornea surface) and the display lens can have significant impacts on the user experience of the device. For example, on VR and AR headsets, the best user experience is typically achieved when the user's eyes are within an optimal range of distances from the lens. Moreover, a lens distance that is too small can raise safety concerns, such as potential injuries to the cornea when the user experiences a fall. To avoid such problems, embodiments of an HMD described herein are configured to perform an automatic lens distance test, and notify the user when the measured distance is too close or too far. In some embodiments, the notification may be generated as an alert that is visually displayed to the user, and recommends the user to adjust the HMD positioning or use a different light seal that better fits the user's facial structure. In some embodiments, the lens distance test may be performed during an initialization process of the HMD that occurs when the user first puts on the HMD. In some embodiments, the lens distance test may be repeated during a user session of the HMD, for example, when the HMD is detected to have moved relative to the user's eyes, or according to a periodic retesting schedule.

In some embodiments, the lens distance test may be performed using a gaze tracking system included in the HMD for detecting the gaze direction of the user's eyes. In some embodiments, the gaze tracking system may include at least one eye tracking camera (e.g., infrared (IR) or near-IR (NIR) cameras) positioned at each side of the user's face, and illumination sources (e.g., IR or NIR light sources such as an array or ring of LEDs) that emit light (e.g., IR or NIR light) towards the user's eyes. The eye tracking cameras may be pointed towards the user's eyes to receive reflected IR or NIR light from the light sources directly from the eyes, or alternatively may be pointed towards “hot” mirrors located between the user's eyes and the display panels that reflect IR or NIR light from the eyes to the eye tracking cameras while allowing visible light to pass. The gaze tracking system may determine a current pose of the user's eye, which may indicate the apex location of the cornea. In some embodiments, the distance from the apex location to a nearest point on the lens is used as the lens distance. If the lens distance is too short of too long (e.g. outside of a pre-specified threshold or range), the HMD will generate an alert to the user.

FIG.1illustrates an HMD device that implements a lens distance test to determine the distance between the eye of a user and a lens of the HMD, according to some embodiments.

As shown, the figure depicts an HMD device100worn by a user102. The HMD100may include, but is not limited to, a display110(e.g., a left and right display panel), two display lenses120, and a gaze tracking system that includes at least one eye tracking camera140(e.g., infrared (IR) or near-IR (NIR) cameras) positioned at each side of the user's face, and an illumination source130(e.g., IR or NIR light sources such as an array or ring of NIR light-emitting diodes (LEDs)) that emit light (e.g., IR or NIR light) towards the user's eyes104. Depending on the eye tracking cameras140may be pointed towards mirrors located between the user's eyes104and the display110that reflect IR or NIR light from the eyes104while allowing visible light to pass, or alternatively pointed towards the user's eyes104to receive reflected IR or NIR light from the eyes104as shown in the figure.

As shown, the HMD100may include a light seal150that encloses light generated by the display110so that visual content generated by the display appears brighter to the user120. The light seal150may also be fitted to the user102, for example, adjusted to a particular shape to fit the user's facial structure or place the display110at a particular distance170from the user's eyes. In some cases, an HMD with a light seal150adapted for one user may not properly fit the facial structure of another user, so that the other user cannot achieve proper eye-to-lens distancing with the light seal.

In some embodiments, the HMD100may include a controller160that may be configured to render AR or VR content (e.g., left and right frames for left and right display panels) and provide the frames to the display110. In some embodiments, the controller160may be integrated in the HMD. In some embodiments, the controller160may be a computer device with its own processors and memory. In some embodiments, at least some functionality of the controller160may be implemented by a device external to the HMD and coupled to the HMD by a wired or wireless connection. The user looks through the display lenses120onto the display110(e.g., on to left and right display panels through left and right display lenses120).

In some embodiments, the controller160implements gaze tracking using eye tracking camera(s)140for various purposes. The controller160may estimate the user's point of gaze on the display110based on the gaze tracking input obtained from the eye tracking cameras140using glints or reflections from the eye produced by the light source(s)130. The point of gaze estimated from the gaze tracking input may be used to determine the direction in which the user is currently looking.

As shown, in some embodiments, the light source(s)130may be arranged a circle around each of the display lenses120. However, in other embodiments, more or fewer light sources130may be used, and other arrangements and locations of light sources130may be used. In some embodiments, the eye tracking cameras140may be pointed towards mirrors located between the user's eyes104and the display110to reflect IR or NIR light from the eyes104while allowing visible light to pass. In other embodiments, the light sources130may be pointed towards the user's eyes104to receive reflected IR or NIR light from the eyes104, as shown.

As shown, in the controller160of the HMD100may implement a lens distance test180. As discussed, this lens distance test180may be performed at various times during the operation of the HMD to determine the distance170between the user eye104and the display lens120, and notify the user if the distance is outside a threshold or tolerance range. In some embodiments, these thresholds or tolerance ranges may be configurable via a configuration interface of the HMD100. In some embodiments, the HMD may be configured with multiple distance thresholds. For example, exceeding a first threshold may only cause the HMD to generate a warning the lens is too close or too far, and exceeding a second threshold may prevent the HMD from operating altogether.

In some embodiments, the lens distance test180may be performed using the gaze tracking functionality of the HMD. For example, the gaze tracking system of the HMD may be configured to continuously determine the current pose of the eye104based on an eye model. In some embodiments, the eye model may implement a function that takes the eye images captured by the camera140(or the glint readings) and translates the data to a particular spatial configuration of the eye (e.g. the current positions of the cornea center, optical axis, pupil, etc.). In some embodiments, the lens distance test uses such an eye model to quickly determine the current pose of the eye, including the location of the cornea apex, which is the point of the cornea that protrudes furthest from the eyeball. In some embodiments, the lens distance test uses this apex location to determine the distance170from the eye to the lens.

FIG.2Aillustrates steps in a lens distance test performed by the HMD, according to some embodiments.

The top portion ofFIG.2Ashows certain details about the user eye104and lens120ofFIG.1. As shown, the user eye104includes the pupil210and the apex of the cornea212. The point on the lens120that is closest to the apex212is point214. In some embodiments, the distance determined by the lens distance test180is the nearest distance220between the apex212and the nearest point214on the lens.

The bottom portion ofFIG.2Ashows steps performed by a particular embodiment of the lens distance test180. As shown, the process begins with the eye images230captured by the camera(s)140. In some embodiments, these eye images230are reduced to a set of readings corresponding to the observed reflection or glint values generated by light sources130.

The images or readings are fed into an eye model232, which determines a current pose234of the eye. In some cases, the eye model232may be a user-specific model that was previously generated for a user, for example, during an enrollment process of the user. The HMD may determine that a current wearer of the device fits the eye model (e.g. based on a match of current eye images or a biometric authentication of the user), and elect to use the eye model to perform the lens distance test. In some embodiments, use of the eye model of a known user enhances the accuracy of the test. In other cases, the eye model232may be a general eye model that is not specific to any user. Such a general eye model may be based on an average of the eye features of many users, and can be used to perform the lens distance test with less accuracy. Use of the general eye model is useful in situations when the HMD is being used by someone other than the normal user of the device, which is typically when lens distancing issues can arise. Once the eye pose234is determined, the test determines the apex location212of the cornea of the eye. In some embodiments, the apex location212may be directly obtained from the eye model232.

As shown, the test process next uses a lens model236to determine the nearest point214on the lens to the apex location. In some embodiments, the lens model236may represent the spatial shape of the lens120in a reference coordinate system, and the apex location212is also expressed in the same coordinate system. Depending on the embodiment, the coordinate system may be centered on the lens120, the camera(s)140, or some other point on the HMD. In some embodiments, the display lens120is fixed to the HMD, and the lens model236is determined by a factory calibration process of the HMD.

As shown, the test process next determines the nearest distance220using the apex location212and the nearest point on the lens214. In some embodiments, this distance220is used as the lens distance170. If the distance is too short or too long (e.g. shorter or longer than a distance threshold), the HMD will generate an alert238. In some embodiments, the alert may be generated visually via the display on the HMD. In some embodiments, the alert will indicate that the lens distance is too close or too far, and instruct or recommend the user to either adjust the position of the HMD or use a different light seal. In some embodiments, if the user is not recognized as a known user of the HMD, the warning may ask the user to authenticate (so that a more accurate eye model can be used to retry the lens distance test), or enroll as a new user.

FIG.2Billustrates a lens distance test that determines a distance from a user's eye to a clip-on lens added to the HMD, according to some embodiments.

In some embodiments, when the clip-on lens240is added to the HMD, the HMD will perform a field calibration process to precisely determine the position of the clip-on lens relative to the HMD. This location data may be stored (e.g. as part of the lens model236), so that it can be used to perform lens distance test180. When a clip-on lens is present, the lens distance test will measure the nearest distance250from the eye104to the clip-on lens, instead of the display lens120. In some embodiments, the distance thresholds for generating the alert may also be changed when a corrective lens is installed, so that the user is encouraged to properly distance the eyes in accordance with the characteristics of the corrective lens. In some embodiments, the lens distance test180may be performed as part of the field calibration process when a new clip-on lens is installed.

FIG.3is a flowchart illustrating the performance of a lens distance test, according to some embodiments. The process shown inFIG.3may be performed by an embodiment of the HMD100ofFIG.1.

The process begins at operation310, where image(s) of an eye of the user is captured using camera(s) (e.g. camera(s)140) on the HMD. In some embodiments, the cameras may be part of a gaze tracking system implemented by the HMD, and the images may indicate reflections or glints produced by LED light sources positioned around the eye. In some embodiments, the image(s) may be taken at a single point in time and constitute one captured frame of the eye.

At operation320, a determination is made whether a user-specific eye model associated with the user is available. In some embodiments, the eye model may be a function that takes the captured images or readings derived from the images to a particular pose of the eye. A user-specific eye model may be created during an enrollment process of the user. Such a user-specific eye model may be preferable to a general eye model because it can determine the eye pose more accurately. In some embodiments, the HMD may recognize the particular user by analyzing the captured images, certain features of the eye, or as a result of an authentication of the user.

If the user-specific eye model is available, the process proceeds to operation330, where the user-specific eye model is used to determine the pose of the eye, based on the captured images or readings derived from the images. If the user-specific eye model is not available (e.g. because the user is not recognized as a known user that was previously enrolled), the process proceeds to operation340, where a general eye model is used to determine the pose of the eye. A general eye model is not specific to the user, and so the pose determined using the general model may not be as accurate. In some embodiments, the general eye model may be based on an average of features of many users.

Once the eye pose is determined, at operation350, an apex location (e.g. apex location212) corresponding to the cornea apex of the eye is determined using the eye pose. The eye pose may fully represent the 3D shape of the eye in space, including the apex of the cornea. In some embodiments, the apex location may be directly output by the eye model. The apex location may be specified in reference coordinate system, which may be centered around the display lens, the camera(s) or other fixed point with respect to the HMD.

At operation360, a nearest point on the lens in question (e.g. display lens120or a clip-on lens240) to the apex location is determined. In some embodiments, the lens's spatial position and shape may be modeled in a lens model in the same coordinate system of the apex location, and the lens model can be used to determine the nearest point. At operation370, a nearest distance is determined between the apex location and the nearest point on the lens.

At operation380, one or more check(s) are made to determine whether the nearest distance is greater or less than one or more distance threshold(s). The distance threshold(s) may specify optimal or preferred distance range(s) from the eye to the lens. In some embodiments, the thresholds may be configurable via a configuration interface of the HMD.

If a distance problem is discovered (e.g. if the nearest distance is outside an acceptable distance range), at operation390, an alert is output indicating that the lens distance is too close or too far. In some embodiments, the alert may recommend that the user adjust the positioning of the HMD or use a different light seal that is better fit to the user's face. In some embodiments, the HMD may recommend that the user authenticate so that a more accurate user-specific eye model can be used, or enroll as a new user so that a user-specific eye model can be configured. In some embodiments, the HMD may prevent the user from initiating a user session using the HMD if the lens distance test fails. In some embodiments, the alert may be generated visually to the user via the display lens.

As shown, in some embodiments, the lens distance test may be repeated even if the lens distance is determined to be acceptable. For example, the test may be performed for multiple frames to measure the lens distance in multiple eye poses. In some embodiments, if any eye pose fails the test, the alert will be generated. In some embodiments, the alert may be generated based on an average of the measured distances for the different poses, or based on a number of poses that fails the test.

FIG.4is a flowchart illustrating a process of repeating the lens distance test using a user session of the HMD, according to some embodiments. The process shown inFIG.3may be performed by an embodiment of the HMD100ofFIG.1.

As shown, in some embodiments, one or more runs of the lens distance test (e.g. lens distance test180) may be performed420during an initialization process410for a user session. This initialization process may occur when a user first puts on (or turns on) the HMD device, when the HMD will configure various settings for the user setting. In some embodiments, the initialization process410may occur when the HMD device is switched from one user to another user. As discussed, the lens distance test may be performed multiple times in different poses. In some embodiments, the HMD may perform the lens distance test on the two eyes independently.

Additionally, in some embodiments, the lens distance test may be repeated450during the user session430, for example, as the user is using an AR or VR application executed by the HMD. The retesting may be triggered by one or more events or conditions440configured for the HMD. For example, in some embodiments, a retest of the lens distance may be triggered in response to detection of a fall or a significant movement of the user's eyes relative to the HMD. In some embodiments, the lens distance may be retested according to a set schedule (e.g. once a minute), so that the distance can be continuously monitored.

FIG.5is a block diagram illustrating various components of an example VR/AR system that implements the lens distance test, according to some embodiments. In some embodiments, a VR/AR system may include an HMD2000such as a headset, helmet, goggles, or glasses. HMD2000may implement any of various types of virtual reality projector technologies. For example, the HMD2000may include a VR projection system that includes a projector2020that displays frames including left and right images on screens or displays2022A and2022B that are viewed by a user through eye lenses2220A and2220B. The VR projection system may, for example, be a DLP (digital light processing), LCD (liquid crystal display), or LCoS (liquid crystal on silicon) technology projection system. To create a three-dimensional (3D) effect in a 3D virtual view, objects at different depths or distances in the two images may be shifted left or right as a function of the triangulation of distance, with nearer objects shifted more than more distant objects. Note that other types of projection systems may be used in some embodiments.

In some embodiments, HMD2000may include a controller2030that implements functionality of the VR/AR system and that generates frames (each frame including a left and right image) that are displayed by the projector2020. In some embodiments, HMD2000may also include a memory2032that stores software (code2034) of the VR/AR system that is executable by the controller2030, as well as data2038that may be used by the VR/AR system when executing on the controller2030. For example, in some embodiments, the code2034may include code to execute the lens distance test180, and the data2038may include the captured eye images230and the determined nearest distance220.

In some embodiments, HMD2000may also include one or more interfaces (e.g., a Bluetooth technology interface, USB interface, etc.) that communicate with an external device2100via a wired or wireless connection. In some embodiments, at least a part of the functionality described for the controller2030may be implemented by the external device2100. External device2100may be or may include any type of computing system or computing device, such as a desktop computer, notebook or laptop computer, pad or tablet device, smartphone, hand-held computing device, game controller, game system, and so on.

In various embodiments, controller2030may be a uniprocessor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number). Controller2030may include central processing units (CPUs) that implement any suitable instruction set architecture, and may execute instructions defined in that instruction set architecture. For example, in various embodiments controller2030may include general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of the processors may commonly, but not necessarily, implement the same ISA. Controller2030may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out of order, in order, speculative, non-speculative, etc., or combinations thereof. Controller2030may include circuitry to implement microcoding techniques. Controller2030may include one or more processing cores that each execute instructions. Controller2030may include one or more levels of caches, which may employ any size and any configuration (set associative, direct mapped, etc.). In some embodiments, controller2030may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuitry. Generally, a GPU may render objects to be displayed into a frame buffer (e.g., one that includes pixel data for an entire frame). A GPU may include one or more graphics processors that may execute graphics software to perform a part or all of the graphics operation, or hardware acceleration of certain graphics operations. In some embodiments, controller2030may include one or more other components for processing and rendering video and/or images, for example image signal processors (ISPs), coder/decoders (codecs), etc.

Memory2032may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such as mDDR3, etc., or low power versions of the SDRAMs such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled onto a circuit board to form memory modules such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc. Alternatively, the devices may be mounted with an integrated circuit implementing system in a chip-on-chip configuration, a package-on-package configuration, or a multi-chip module configuration.

In some embodiments, the HMD2000may include one or more cameras2050that capture video of the user's environment for AR applications. In some embodiments, the HMD2000may render and display frames to provide an augmented or mixed reality (AR) view for the user at least in part according to camera2050inputs. The AR view may include renderings of the user's environment, including renderings of real objects in the user's environment, based on video captured by one or more video cameras2050that capture high-quality, high-resolution video of the user's environment for display. In some embodiments, the cameras2050may be equipped with autofocus mechanisms. While not shown, in some embodiments, the HMD2000may also include one or more sensors that collect information about the user's environment and actions (depth information, lighting information, user motions and gestures, etc.). The cameras2050and sensors may provide the information to the controller2030of the VR/AR system.

As shown, HMD2000may be positioned on the user's head such that the displays2022A and2022B and eye lenses2220A and2220B are disposed in front of the user's eyes2292A and2292B. IR or NIR light sources2230A and2230B (e.g., IR or NIR LEDs) may be positioned in the HMD2000(e.g., around the eye lenses2220A and2220B, or elsewhere in the HMD2000) to illuminate the user's eyes2292A and2292B with IR or NIR light. Eye tracking cameras2240A and2240B (e.g., IR or NIR cameras, for example 400×400 pixel count cameras) are located at each side of the user's face, for example at or near the user's cheek bones. Note that the location of eye tracking cameras2240A and2240B is given by way of example, and is not intended to be limiting. In some embodiments, there may be a single eye tracking camera2240located on each side of the user's face. In some embodiments there may be two or more eye tracking cameras2240on each side of the user's face. For example, in some embodiments, a wide-angle camera2240and a narrower-angle camera2240may be used on each side of the user's face. A portion of IR or NIR light emitted by light sources2230A and2230B reflects off the user's eyes2292A and2292B either directly to respective eye tracking cameras2240A and2240B or via mirrors2250A and2250B located between the user's eyes2292and the displays2022, and is captured by the eye tracking cameras2240A and2240B to image the user's eyes2292A and2292B. Gaze tracking information captured by the cameras2240A and2240B may be provided to the controller2030. The controller2030may analyze the gaze tracking information (e.g., images of the user's eyes2292A and2292B) to determine gaze direction, eye position and movement, pupil dilation, or other characteristics of the eyes2292A and2292B.

The gaze tracking information obtained and analyzed by the controller2030may be used by the controller in performing various VR or AR system functions. For example, the point of gaze on the displays2022A and2022B may be estimated from images captured by the eye tracking cameras2240A and2240B using the glint-assisted methods. The estimated point of gaze may, for example, be used to render virtual content differently based on the determined direction of the user's gaze.

Embodiments of the HMD2000as illustrated herein may also be used in virtual reality (VR) applications to provide VR views to the user. In these embodiments, the controller2030of the HMD2000may render or obtain virtual reality (VR) frames that include virtual content, and the rendered frames may be provided to the projector2020of the HMD2000for display to displays2022A and2022B. In some embodiments, for VR applications, the controller2030may obtain distance information for virtual content to be displayed on the display panels2022, and may use this distance information to direct the eye lenses2220to adjust focus according to the distance of virtual content that the user is currently looking at according to the gaze tracking information.