Patent Description:
<CIT> discloses a display device which includes a two-dimensional array of tiles. Each tile includes a two-dimensional array of pixels and a lens, of a two-dimensional array of lenses. The display device also includes one or more processors coupled with the two-dimensional array of tiles and configured to: obtain a transformed image for projecting a non-transformed image on a retina of an eye of a user; activate a first subset of the two-dimensional array of tiles for projecting a first portion of the transformed image on the retina of the eye of the user with a first resolution; and activate a second subset of the two-dimensional array of tiles for projecting a second portion of the transformed image on theretina of the eye of the user with a second resolution.

<CIT> discloses an auto-focus head mounted display which dynamically generates aberration adjusted images based on measured accommodation of a user's eyes.

In one aspect, the present invention provides a head-mounted display device according to claim <NUM>. In another aspect, the present invention provides a method according to claim <NUM>. Certain more specific aspects are set out in the dependent claims.

Examples are disclosed that relate to reducing binocular rivalry in a near-eye display. One aspect of the invention provides a head-mounted display device as defined in claim <NUM>, comprising a near-eye display system configured to output a first-eye image to a first eyebox and a second-eye image to a second eyebox. The head-mounted display device further comprises a logic device and a storage device comprising instructions executable by the logic device to receive an input of a three-dimensional (3D) location of a pupil of a first eye and a 3D location of a pupil of a second eye relative to the near-eye display system, based upon the 3D location of the pupil of the first eye and the 3D location of the pupil of the second eye determine a location at which the pupil of the first eye begins to exit the first eyebox, and attenuate a luminance of the second-eye image at a location in the second-eye image based upon the location at which the pupil of the first eye begins to exit the first eyebox.

Near-eye displays (such as head-mounted displays) having relatively larger fields of view may offer more immersive augmented and/or virtual reality experiences compared to those with relatively smaller fields of view. In a binocular near-eye display, a left image viewable by a left eye is displayed within a left eyebox and a right image viewable by a right eye is displayed within a right eyebox. The size of an eyebox is influenced by an interpupillary distance (IPD) range of a target group of users and by a desired field of view. For a wide-angle near-eye display, a large eyebox for each eye may be required in order for all users of a population of users to see the entire displayed region at all times. This may involve the use of large and costly optics. To help avoid such optics, a head-mounted display system may be designed such that a significant percentage (e.g. <NUM>-<NUM>%) of IPDs will fall fully within the eyebox across the full field of view. In such a design, most people will not experience vignetting. However, people with IPDs that are either very large or very small compared to the population as a whole may experience some vignetting in a left-eye or right-eye image when gazing at virtual imagery displayed near or at an edge of the field of view, or when looking well past the edge of the field of view (e.g. to look at a real-world object). Where the vignetting is in the image for one eye and not the other eye, or otherwise dissimilar between the images for each eye, the visual experience may lead to a possibly distracting binocular rivalry effect.

<FIG> shows an example use scenario <NUM> in which a viewer <NUM> wearing a near-eye display device <NUM> is viewing a displayed virtual object <NUM> in an augmented reality experience. The virtual object <NUM> is displayed in a location in a far-left region of the display field of view. Referring to <FIG>, due to the locations of the user's pupils relative to the corresponding eyeboxes, a left-eye image <NUM> of the virtual object <NUM> is fully visible in the left eye field of view <NUM>, but a left side of right-eye image <NUM> of the virtual object <NUM> is partially cut off. This may occur when the right eye pupil crosses fully beyond an edge of the right eyebox. In instances where the pupil only partially leaves the eyebox, the affected side of the image will be visible, but appear dimmer. In either case, the resulting inconsistency between what is perceived by the left eye and the right eye may be distracting or uncomfortable to view. It will be understood that vignetting may occur on any edge of either a left or right eye image, depending upon the location of each pupil relative to its corresponding eyebox.

Accordingly, examples are disclosed herein that relate to attenuating a luminance of a portion of an image for a first eye based on a location of a pupil of a second eye with respect to an edge of the eyebox for the second eye. Attenuating a luminance of an image in this manner may help the appearance of the image for the first eye to more closely match the image perceived by the second eye and thereby mitigate any binocular rivalry that may otherwise arise from the differences between the images.

A location at which a pupil will begin to cross an edge of an eyebox is determined based upon data regarding a three-dimensional location of the pupil relative to the near-eye display system. In some examples, a three-dimensional location of the pupil relative to a near-eye display system may be determined by calibration using an eye-tracking system, e.g. at a beginning of a use session. Such a calibration may be updated periodically during a use session, for example, to correct for any positional shifting of the near-eye display device on a user's head during use.

Any suitable method may be used by an eye tracking system to determine a three-dimensional location of a pupil of an eye. For example, an eye-tracking system may direct light from a plurality of eye-tracking light sources toward the eye, acquire image data of the eye, detect a pupil image and images of reflections of the eye-tracking light sources from the user's eye in the image data, and then fit the observed locations of the pupil relative to the eye tracking light source reflections to a three-dimensional eye model to determine the three-dimensional pupil location. In other examples, a three-dimensional location of a pupil may be determined via an estimation based on a user-provided IPD and assumed eye relief values. In yet other examples, non-image-based eye trackers may also be used, such as an eye tracker that detects extraocular muscle signals.

The determined three-dimensional location of each eye may be compared to an eyebox model to determine the location at which the pupil will start to exit the eyebox. <FIG> shows an example eyebox model <NUM> illustrating an eyebox <NUM> as a diamond-shaped area in space. It will be understood that the eyebox also comprises a third dimension in a direction normal to the page. In this model, an exit surface of the waveguide is represented by <NUM>. Lines A and A' represent a region in space between which one edge of a display field of view is viewable, and lines B and B' represent a region between which the other edge of the display field of view is viewable.

<FIG> also shows several example pupil locations within the model. First, pupil <NUM> (represented by a horizontal bar indicating an example pupil diameter) is fully within the eyebox <NUM>, and thus would experience no vignetting. Pupil <NUM> is positioned such that it experiences no vignetting at the edge of the field of view represented by lines A-A', but does experience vignetting at the edge of the field of view represented by lines B-B'. Because pupil <NUM> crosses line B, the edge of the field of view represented by lines B-B' would appear dimmer than other portions of the image. Pupil <NUM> is positioned such that it experiences vignetting at the edge of the field of view represented by lines A-A', but no vignetting at the edge of the field of view represented by lines B-B'. Because pupil <NUM> crosses line A, the edge of the field of view represented by lines A-A' would appear dimmer. Pupil <NUM> is positioned such that it receives no light from the edge of the field of view represented by lines B-B'. As such, the viewed image would appear to fade and disappear before reaching the edge of the field of view represented by lines B-B'. Knowledge of the three-dimensional location of a pupil thus allows locations at which the pupil of the eye would leave the eyebox to be determined by comparing the three-dimensional location of the pupil to the eyebox model. Based upon this determination, a corresponding portion of an image displayed to the other eye is attenuated to reduce apparent differences between the two images.

As shown in <FIG>, as a pupil crosses out of an eyebox, less and less light from that edge enters the pupil. Thus, the eye perceives a roll-off in intensity or luminance of the displayed image at that edge. The resulting roll-off in luminance may be modeled mathematically using the eyebox model <NUM> to calculate a roll-off profile for the impacted edge of the field of view. The determined roll-off profile then may be used to attenuate luminance in the edge of the image for the un-impacted eye. The steepness of the roll-off is a function of the pupil diameter, where a smaller diameter pupil perceives a steeper roll-off. In some examples, a fixed roll-off profile may be used based, for example, on a determined or estimated average pupil diameter across a population. In other examples, various methods may be used to determine or estimate a pupil diameter in real-time for dynamically determining a roll-off profile.

As mentioned above, in some examples, a near-eye display system utilizes eye tracking to determine in real time when a pupil is exiting the eyebox, and dynamically adjusts an image displayed to the other eye in response. This is referred to herein as dynamic adjustment.

Dynamic adjustment may provide real-time, frame-by-frame matching of left and right eye images, and thus may result in a more accurate image adjustment than static methods. Further, as mentioned above, in some examples, a pupil diameter may be tracked in addition to a pupil location to help more accurately determine when the pupil begins to leave the eyebox. In yet other examples, eye motion prediction may be used to help inform eye tracking.

In some examples, image adjustment may be performed by reducing the luminance in the corresponding region of the image for one eye to fully match the reduction in luminance perceived by the other eye. In other examples, the luminance may be adjusted to a lesser extent than that perceived by the other eye, but to a sufficient extent to mitigate a sensation of binocular rivalry (e.g. as determined based on data from a large sample of users, or to a level set as a user preference).

<FIG> schematically illustrates an example of a left-eye image <NUM> and a right-eye image <NUM>, where each image <NUM> and <NUM> represents a full field of view. In this example, it is determined that a user's eye will be approximately halfway past the edge of the eyebox at a location illustrated by dotted line <NUM>. Accordingly, a corresponding region <NUM> of the left-eye image is attenuated in luminance.

As the right pupil exits the eyebox in this example, the right-eye image would be perceived as having a roll-off in luminance. Thus, the luminance of the left-eye image is adjusted to roll off toward the left edge of the image in a similar manner. In some examples, the roll-off profile of the attenuated image may be centered such that the <NUM>% luminance point intersects line <NUM>, so that both eyes would perceive an approximately <NUM>% luminance of left and right images at this point, while the roll-off profile may be centered differently in other examples. In the depicted example, the intensity is reduced to zero toward the left side of the left-eye image. This may occur, for example, where the pupil of the other eye has fully exited the corresponding side of the eyebox. In other examples, as mentioned above, the luminance may be attenuated to a level that is greater than <NUM>%, but sufficiently reduced as to avoid the perception of distracting binocular rivalry.

As mentioned above, the steepness of the roll-off in luminance seen by an eye exiting an eyebox is dependent upon a pupil size. As such, in some examples, a near-eye display system may determine a pupil size (e.g. pupil diameter) and pupil center location, and determine a profile of the roll-off profile to be applied to the fully viewable image to more closely match an appearance of the images. <FIG> shows an example of a steeper roll-off profile <NUM> calculated for a relatively smaller diameter pupil, and <FIG> shows an example of a more gradual roll-off profile <NUM> for a larger pupil diameter. Such an adjustment may be based upon pupil images captured in eye tracking data, based upon a detected ambient light intensity, and/or on any other suitable factors (e.g. average pupil response data, which may be based on an age of a user).

<FIG> shows a flow diagram illustrating a method <NUM> of attenuating a luminance in a portion of a first eye image to more closely match the perception of a second eye image, in accordance with a reference example. Method <NUM> may be performed on any suitable near-eye display system, including but not limited to those described above.

User characteristics <NUM> (e.g. IPD), eye location data from the eye location system <NUM> (e.g. three-dimensional pupil location data), and a mathematical eyebox model <NUM> are used as inputs to determine at <NUM> whether the image for either eye is degraded by the pupil partially or fully exiting the eyebox. The eye location system <NUM> may provide either static or dynamic three-dimensional pupil location information, as described above. If it is determined that neither eye experiences a degraded image based on both pupils being fully within their respective eyeboxes at the determined pupil locations, then both the left eye and right eye display images are transferred to the display without modification, at <NUM>. When it is determined that the left eye image is degraded, then at <NUM> the right eye is matched to the left image by attenuating a luminance of a portion of the right eye image to have a more similar appearance to the left eye image. When it is determined that the right eye image is degraded, then at <NUM> the left eye image is matched to the right eye image by attenuating a luminance of a portion of the left eye image.

Any suitable method may be used to attenuate a luminance of a portion of an image. In some examples, a software shader or other software algorithm may be used to apply a luminance attenuation prior to rendering. In other examples, hardware used to display the image may be controlled, for example, by controlling a display panel at the hardware level as a last-stage adjustment to the displayed image.

<FIG> shows a flow diagram depicting a method <NUM> for adjusting a displayed image to reduce binocular rivalry on a near-eye display system, in accordance with an embodiment of the present invention. Method <NUM> includes, at <NUM>, receiving an input of a three-dimensional location of a pupil of a first eye and a three-dimensional location of a pupil of a second eye relative to the near-eye display system. The input of pupil location data may be received from any suitable data source. For example, the input may be received from an eye-tracking system, at <NUM>, which may determine the three-dimensional location of each pupil during calibration <NUM> (e.g. performed at the beginning of a use session and possibly periodically during a use session), or may continuously track pupil locations <NUM>. In other examples, a three-dimensional location of a pupil may be determined based on a user input of IPD, at <NUM>, and assumed eye relief values.

In some examples, the system may optionally detect or track the pupil size of the pupil of the first eye and the pupil size of the pupil of the second eye, at <NUM>. The pupil size may vary for a same user in the light versus the dark and/or due to other factors, and the detected pupil size may be used to compute a roll-off profile when applying a determined luminance reduction.

Method <NUM> further includes, at <NUM>, based upon the three-dimensional location of the pupil of the first eye and the three-dimensional location of the pupil of the second eye, determining a location at which the pupil of the first eye begins to exit the first eyebox. In some examples, this location may be computed based upon the three-dimensional pupil location information, user characteristics, and a mathematical eyebox model, as indicated at <NUM>. In other examples, a pre-computed look-up table may be used to determine this location, as indicated at <NUM>.

Method <NUM> further includes, at <NUM> attenuating a luminance of a portion of the second-eye image at a location in the second-eye image based upon the location at which the pupil of the first eye begins to exit the first eyebox, thus attenuating luminance of a portion of the second-eye image in a region corresponding to a region of the first-eye image that is reduced in luminance from a perspective of the pupil of the first eye. As described above, the luminance of image may be attenuated according to a roll-off profile, as indicated at <NUM>, which may be based on an average pupil size across a population of intended users, or which may be determined dynamically (e.g. via an eye-tracking system). Further, as pupil size may be affected by ambient brightness, in some examples the roll-off profile may be based on environment brightness and/or display brightness, e.g. as detected by a sensed ambient light level. Where the roll-off is based on a sensed ambient light level, a user's age may be used as an additional input to determine a likely pupil response of that user. The luminance adjustment may be performed via software prior to rendering, or may be performed by controlling the hardware used to display the second-eye image, at <NUM> (e.g. by controlling a display panel at the hardware level as a last-stage adjustment).

<FIG> shows an example near-eye display device in the form of a head-mounted display device <NUM> that is configured to perform the binocular rivalry mitigation method of the invention. The head-mounted display device <NUM> includes a frame <NUM> in the form of a band wearable around a head of user that supports see-through display componentry positioned nearby the user's eyes. As mentioned above, the head-mounted display device <NUM> may utilize augmented reality technologies to enable simultaneous viewing of virtual display imagery and a real-world background. As such, the display device <NUM> may generate virtual images via see-through display <NUM>, which includes separate right and left eye displays 904R and <NUM>, and which may be wholly or partially transparent. The see-through display <NUM> may take any suitable form, such as a waveguide or prism configured to receive a generated image and direct the image towards a wearer's eye. The see-through display <NUM> may include a backlight and a microdisplay, such as liquid-crystal display (LCD) or liquid crystal on silicon (LCOS) display, in combination with one or more light-emitting diodes (LEDs), laser diodes, and/or other light sources. In other examples, the see-through display <NUM> may utilize quantum-dot display technologies, active-matrix organic LED (OLED) technology, a scanning laser display, and/or any other suitable display technologies. The see-through display <NUM> further may utilize pupil replication, as mentioned above, to expand an exit pupil of a display system. It will be understood that while shown in <FIG> as a flat display surface with left and right eye displays, the see-through display <NUM> may be a single display, may be curved, or may take any other suitable form.

The head-mounted display device <NUM> further includes an additional see-through optical component <NUM>, shown in <FIG> in the form of a see-through veil positioned between the see-through display <NUM> and the background environment as viewed by a wearer. A controller <NUM> is operatively coupled to the see-through optical display <NUM> and to other display componentry. The controller <NUM> includes one or more logic devices and one or more computer memory devices storing instructions executable by the logic device(s) to enact functionalities of the display device, and more specifically to enact the method of the invention. The display device <NUM> may further include a front-facing two-dimensional image camera <NUM> (e.g. a visible light camera and/or infrared camera), a front-facing depth camera <NUM>, and/or an eye tracking system <NUM>. The eye tracking system <NUM> may include a plurality of eye-tracking light sources (e.g. infrared light sources) directed toward a region of space intended to be occupied by a user's eye, and one or more image sensors (e.g. visible light cameras, depth cameras, infrared cameras) configured to acquire image data of a user's eyes to detect images of a pupil and of reflections of light from the eye-tracking light sources from a user's eye. The display device <NUM> may further include an ambient light sensor <NUM> that detects ambient brightness and/or display brightness. In other examples, ambient light levels may be determined from an outward-facing image sensor. The display device <NUM> may further include other components that are not shown, including but not limited to speakers, microphones, accelerometers, gyroscopes, magnetometers, temperature sensors, touch sensors, biometric sensors, other image sensors, energy-storage components (e.g. battery), a communication facility, a GPS receiver, etc..

<FIG> schematically shows a non-limiting embodiment of a computing system <NUM> that is configured to enact the method of the invention.

Computing system <NUM> includes a logic subsystem <NUM> and a storage subsystem <NUM>.

Logic subsystem <NUM> includes one or more physical devices configured to execute instructions.

The logic subsystem <NUM> may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic subsystem <NUM> may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic subsystem <NUM> may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Aspects of the logic subsystem <NUM> may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.

Storage subsystem <NUM> includes one or more physical devices configured to hold instructions executable by the logic machine to implement the method of the invention. When such methods and processes are implemented, the state of storage subsystem <NUM> may be transformed-e.g., to hold different data.

Storage subsystem <NUM> may include removable and/or built-in devices. Storage subsystem <NUM> may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage subsystem <NUM> may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.

It will be appreciated that storage subsystem <NUM> includes one or more physical devices.

When included, display subsystem <NUM> may be used to present a visual representation of data held by storage subsystem <NUM>. Such display devices may be combined with logic subsystem <NUM> and/or storage subsystem <NUM> in a shared enclosure, or such display devices may be peripheral display devices.

Another example provides a head-mounted display device, comprising a near-eye display system configured to output a first-eye image to a first eyebox and a second-eye image to a second eyebox, a logic device, and a storage device comprising instructions executable by the logic device to receive an input of a three-dimensional (3D) location of a pupil of a first eye and a 3D location of a pupil of a second eye relative to the near-eye display system, based upon the 3D location of the pupil of the first eye and the 3D location of the pupil of the second eye, determine a location at which the pupil of the first eye begins to exit the first eyebox, and attenuate a luminance of the second-eye image at a location in the second-eye image based upon the location at which the pupil of the first eye begins to exit the first eyebox. The head-mounted display device may additionally or alternatively include an eye tracking system, wherein the instructions are executable to detect the 3D location of the pupil of the first eye and the 3D location of the pupil of the second eye via the eye tracking system. The eye tracking system may additionally or alternatively be configured to detect a pupil size of the pupil of the first eye and a pupil size of the pupil of the second eye. The instructions may additionally or alternatively be executable to receive an input of an interpupillary distance between the pupil of the first eye and the pupil of the second eye. The instructions may additionally or alternatively be executable to attenuate the luminance of the second-eye image using a roll-off profile. The instructions may additionally or alternatively be executable to determine the roll-off profile based on a detected diameter of the pupil of the first eye. The instructions may additionally or alternatively be executable to determine the location at which the pupil of the first eye begins to exit the first eyebox using a lookup table. The instructions may additionally or alternatively be executable to attenuate the luminance of the second-eye image by adjusting luminance via software prior to rendering the second-eye image. The instructions may additionally or alternatively be executable to attenuate the luminance of the second-eye image by controlling hardware used to display the second-eye image.

Another example provides a head-mounted display device, comprising a near-eye display system configured to output a first-eye image to a first eyebox and a second-eye image to a second eyebox, an eye tracking system, a logic device, and a storage device comprising instructions executable by the logic device to, via the eye tracking system, determine a 3D location of a pupil of a first eye and a 3D location of a pupil of a second eye relative to the near-eye display system, based upon the 3D location of the pupil of the first eye and the 3D location of the pupil of the second eye, determine that the pupil of the first eye is exiting the first eyebox, based on determining that the pupil of the first eye is exiting the first eyebox, determine a region of the first-eye image that is at least partially reduced in luminance from a perspective of the pupil of the first eye, and attenuate a luminance of the second-eye image in a corresponding region of the second-eye image based on determining the region of the first-eye image. The instructions may additionally or alternatively be executable to attenuate the luminance of the second-eye utilizing a roll-off profile. The instructions may additionally or alternatively be executable to determine the roll-off profile based on a detected diameter of the pupil of the first eye. The instructions may additionally or alternatively be executable to attenuate the luminance of the second-eye image by adjusting luminance via software prior to rendering the second-eye image. The instructions may additionally or alternatively be executable to attenuate the luminance of the second-eye image by controlling hardware used to display the second-eye image.

Claim 1:
A head-mounted display device (<NUM>), comprising:
a near-eye display system (<NUM>) configured to output a first-eye image (<NUM>) to a first eyebox and a second-eye image (<NUM>) to a second eyebox;
a logic device (<NUM>); and
a storage device comprising instructions executable by the logic device (<NUM>) to cause the logic device (<NUM>) to:
receive (<NUM>) an input of a three-dimensional, 3D, location of a pupil of a first eye and a 3D location of a pupil of a second eye relative to the near-eye display system (<NUM>),
and characterized in that the storage device further comprises instructions executable by the logic device (<NUM>) to cause the logic device to:
based upon the 3D location of the pupil of the first eye and the 3D location of the pupil of the second eye, determine (<NUM>) a location at which the pupil of the first eye begins to exit the first eyebox, and
attenuate (<NUM>) a luminance of the second-eye image (<NUM>) at a location in the second-eye image (<NUM>) based upon the location at which the pupil of the first eye begins to exit the first eyebox.