Patent Description:
Head-mounted displays, HMDs, are used in a variety of scenarios for displaying images to a user. These are commonly in extended reality (XR) applications. XR applications include virtual reality (VR) where a person is completely immersed in a virtual environment provided by the HMD, augmented reality (AR) where a person has their real-world environment supplemented with a computer-generated overlay and mixed reality (MR) where the computer-generated overlay of AR is combined in a way that the computer-generated objects interact with the real world as seen by the user.

<CIT> discloses a parameter calibration method, applicable to a stereoscopic display device including a camera. The parameter calibration method includes the following steps: displaying a virtual object for calibration; obtaining, by using the camera, a two-eye visual angle by which the virtual object for calibration is watched; calibrating a human eye parameter according to the two-eye visual angle, where the human eye parameter includes at least one of an interpupillary distance parameter and an eye relief parameter; and setting the stereoscopic display device according to the calibrated human eye parameter, such that the stereoscopic display device displays according to the calibrated human eye parameter.

<CIT> discloses a method for viewing a virtual environment through of an optical system. The method includes determining a focus of the optical system configured to view the virtual environment and reconfiguring the optical system for viewing the virtual environment in response to the determining of the focus of the optical system. The method may further include modifying a rendering of the virtual environment in response to the reconfiguring of the optical system.

The standard structure of an HMD includes a frame or strap that enables the HMD to be worn by the user, a housing that incorporates the electronics of the HMD and into which the user looks, and one or more displays that display images to the eyes of the user. As the displays are necessarily displayed very closely to the eye of the user, a lens is interposed between each eye and the display or displays, which allows the eye of the user to focus on a virtual image of the display at a distance that allows the eye to focus.

The arrangement of the lens and display means that, depending on the distance between the eye of the user and the lens, the view through the lens, in particular the amount of the display that is visible to the user, may vary. The distance between the eye and the lens is known as the eye relief, and its impact on the field of view of the user is depicted in <FIG>. More specifically, for the matters discussed herein, the eye relief is between the entrance pupil of the eye and the lens. However, it may also be considered to be between the lens surface and the cornea surface. For the purposes of discussion, it is necessary to be consistent with the boundaries between which the measurement is taken.

As can be seen from <FIG>, the angle α<NUM> from which the eye can receive light through the lens <NUM>, <NUM> is much greater in the left-hand image <NUM>, with eye relief dER1, than the angle α<NUM> in the right-hand image <NUM>, with eye relief dER2. The difference in field of view through the lens <NUM>, <NUM> will show itself as a decrease in viewable angle of a display positioned at the other end of the lens from the eye.

An example difference in field of view (FOV) of a display viewed through each lens is shown in <FIG>. The change in eye relief means that the view of a display <NUM>, <NUM> through a lens with a fixed power will change. Being closer to the lens <NUM>, as shown in the left-hand image, gives a greater FOV <NUM> than the FOV <NUM> shown in relation to an eye further from the lens <NUM>. The area of the display visible at each eye relief is therefore changeable depending on the eye relief. The FOV <NUM>, <NUM> of each arrangement is shown in <FIG>, face-on to the viewer, highlighting the difference in visible area. In the depicted embodiment, the FOV is shown as circular, as the lenses in the assembly are circular. Other shapes of lens may also be used, and these will result in differing FOVs dependent on their shape.

Without being able to adjust eye relief in an HMD, the different profiles of user's heads will result in differing eye relief for each user. A typical distribution may be as follows (sums to <NUM>%, due to rounding):.

Of course, some HMDs therefore incorporate the ability to adjust ER for each user. However, whilst this might mitigate against the most extreme ERs, a large range of ER will still remain across the population, and it may change during use.

It is an objective of the present disclosure to use eye relief measurements to improve the performance of head-mounted displays, such as by improvement of image processing and/or image generation.

According to a first aspect of the invention, there is provided a head-mounted display system according to claim <NUM>.

By measuring and processing the eye relief signal, the display can be modified such that is optimised for use with the measured eye relief, reducing the workload on image processing equipment being used in conjunction with the head-mounted display system.

Eye relief in this context is used generally as a term for a measured distance between a point on the HMD and a point on the user. Commonly, the point on the HMD will be a part of the lens through which the user looks, and more specifically a point on the top surface of the lens known in the art as the lens vertex. Similarly, the point on the user may be the entrance pupil of the eye of the user. Of course, the eye relief signal may not measure these positions directly but may provide a signal indicative of these positions by measuring other parts of the user and HMD and applying an offset. For example, the eye relief signal may be a measure of the distance between the eye relief sensor and the cornea of the eye of the user, as this is the part of the eye closest to the lens vertex. One or more offsets may then be applied in order to take into account the standard or mean distance between the cornea and the entrance pupil and a known distance between the eye relief sensor and the lens or display of the HMD. The entrance pupil of the eye gives the greatest benefit since it defines the actual field of view that the user sees. However, taking another measure that is close to the entrance pupil will also give a benefit and may be simpler to determine.

The output of the display is modified in a portion of the display. The remainder of the display may be left unmodified.

The portion of the display may be a peripheral portion of the display. In this way, the extremities of the display, where the user cannot view the display, may be modified, whilst providing less or no modification to the more central portions of the display. The peripheral portion may be around the entire periphery of the display or may be around a portion of the periphery of the display.

The size of the portion of the display may be dependent on the eye relief signal. The portion of the display being modified - for example the part no longer being rendered - may grow as the eye relief signal is indicative of an increased eye relief. Conversely, if the eye relief signal is indicative of a decreased eye relief, the portion of the display being modified may fall.

The output of the display may be modified by varying a total area rendered on the display. By decreasing the total area rendered on the display, the processing load of the HMD or of other systems in communication with the HMD may be decreased.

The total area rendered on the display may be decreased when the eye relief signal is indicative of an increased eye relief.

Alternatively, the total area rendered may be simplified. For example, an area of the display that is not required may be rendered as a block of colour, such as black, which requires minimal processing.

The output of the display may be modified by varying a rendering quality of the display.

"Rendering quality" may include one or more of resolution, shading, and level of detail (LOD).

The sensor system may comprise an eye tracking system for tracking a gaze direction of the user, the eye tracking system outputting an eye tracking signal to the processor.

The at least one processor may be configured to further modify the output of the display in response to the eye tracking signal. The at least one processor may utilise the eye tracking signal in order to better determine a visible portion of the display or in order to provide further modification such as foveated rendering.

The at least one processor may be further configured to process together the eye tracking signal and the eye relief signal to produce a signal indicative of an area of the display visible to the user.

The eye relief signal may be updated during continued use of the head-mounted display system and the display may be modified accordingly.

Alternatively, the eye relief signal may be generated during an initial period of use of the head-mounted display system, such as during a calibration procedure, and may be static during continued use of the head-mounted display system.

The eye relief signal may include a component indicative of a Z-distance from a gaze origin. The Z-distance is the distance from each eye of the user towards the respective lens of the head-mounted display system in a direction perpendicular to a plane extending from a centre of the lens to an edge of the lens.

The eye relief signal may include a component indicative of a Z-distance from an entrance pupil of the eye. The Z-distance is the direction from each eye of the user towards the respective lens of the HMD.

The eye relief signal may include at least one predetermined offset. The predetermined offset may take into account that the visible area of the display is dependent on a length from a specific part of the eye to the lens of the head-mounted display system. If the distance being measured by the sensor system is not between these specific parts then the use of one or more offsets can allow for this measurement to be corrected for use in further processing.

The predetermined offset may include a term that modifies the eye relief signal in view of an input strength of prescription for eye correction devices, such as eyeglasses or contact lenses. Prescription eyewear can modify the effective eye relief, i.e. they move the effective 3D placement of the entrance pupil so it is closer or further away than it otherwise would be, and therefore taking into account this fact can be used to make the present system more effective.

According to a second aspect of the invention, there is provided a method of improving performance of a head-mounted display system according to claim <NUM>.

Modifying the output of the display comprises modifying a portion of the display. The remainder of the display may be left unmodified.

The method may further comprise tracking a gaze direction of the user, using an eye tracking system, and outputting an eye tracking signal from the eye tracking system to the processor.

The method may further comprise modifying the output of the display in response to the eye tracking signal. The at least one processor may utilise the eye tracking signal in order to better determine a visible portion of the display or in order to provide further modification such as foveated rendering.

The method may further comprise processing together the eye tracking signal and the eye relief signal to produce a signal indicative of an area of the display visible to the user.

The method may further comprise updating the eye relief signal during continued use of the HMD and modifying the display accordingly.

The method may further comprise generating the eye relief signal during an initial period of use of the HMD, such as during a calibration procedure, and maintaining the eye relief signal during continued use of the HMD.

The eye relief signal may include a component indicative of a Z-distance from a gaze origin. The Z-distance is the distance in a direction perpendicular to a plane extending from a centre of the lens to an edge of the lens.

The eye relief signal may include at least one predetermined offset. The predetermined offset may take into account that the visible area of the display is dependent on a length from a specific part of the eye to the lens of the HMD. If the distance being measured by the sensor system is not between these specific parts then the use of one or more offsets can allow for this measurement to be corrected for use in further processing.

According to a third aspect of the invention, there is provided a computer program according to claim <NUM>.

The method may further comprise any additional or optional features as mentioned in relation to the second aspect.

Specific embodiments will now be described in detail with reference to the accompanying drawings, in which:.

A head-mounted display system <NUM> (referred to as an "HMD") is depicted in <FIG>. The HMD <NUM> comprises a housing <NUM> within which are housed two displays <NUM>, <NUM>. Two lenses <NUM>, <NUM> are provided through which a user can view the displays <NUM>, <NUM>. Each eye <NUM>, <NUM> of the user can see one display <NUM>, <NUM> through the respective lens <NUM>, <NUM>, i.e. the right eye <NUM> of the user can only see the right display <NUM> and the left eye <NUM> of the user can only see the left display <NUM>. Of course, rather than providing two displays <NUM>, <NUM>, an HMD may include only a single display that is viewable by both eyes. In such a case, the display will include regions of the display that are visible to only one of the eyes, in order that a stereoscopic view is provided to the eyes. The HMD <NUM> will also include a strap or mount for enabling the HMD <NUM> to be wearable by a user. However, this is omitted from the depiction, for clarity.

An eye tracking system <NUM> is also provided. In the depicted embodiment, the eye tracking system <NUM> comprises two eye tracking cameras <NUM>, <NUM>, each one directed to track the movement of one of the eyes <NUM>, <NUM> of the user. The eye tracking system <NUM> is depicted in schematic form in <FIG>. The eye tracking system <NUM> of the present embodiment utilises pupil centre corneal reflection (PCCR) to track the movement of the eyes <NUM>, <NUM> and therefore each camera <NUM>, <NUM> has associated illuminators <NUM>, <NUM> that shine at the eyes <NUM>, <NUM> of the user in order to generate glints, i.e. reflections that can be used by the eye tracking system <NUM> to detect a gaze direction of the user. Other methods of eye tracking are also possible in place of PCCR, the options for which will be known to the skilled person in the context of the present application.

Signals output from the eye tracking system <NUM> are received by a processor, which in the depicted embodiment is a CPU <NUM>. In turn, these signals can be processed by the CPU <NUM> in order to control an output of each of the two displays <NUM>, <NUM>. Rendering of the two displays <NUM>, <NUM> is performed by a graphics processing unit (GPU) <NUM> that is in communication with the CPU <NUM>, either via a wired or a wireless connection. The GPU <NUM> of the present embodiment is located in an external computer to which the HMD <NUM> is attached.

Whilst the described embodiment refers to a CPU internal to the HMD and a GPU external to the HMD, this is just one of a plurality of possible arrangements. The processing may be carried out in a single processor within the HMD or external to the HMD. Alternatively, more than one processor may be used, and all of the processors may be located within the HMD, all of the processors may be located external to the HMD, or some processors may be located within the HMD and other processors may be located external to the HMD. Processing may also be executed in one or more external servers, for example in a cloud computing system.

Where cloud computing is used, the present invention may have additional benefits due to the fact that the lowering of the rendered area may allow the system to operate at a higher framerate, for example, than equivalent systems that render the entire area of the display independent of eye relief.

The method of operation of the HMD <NUM> is described with reference to the preceding Figures and the flow chart of <FIG>.

The CPU receives S102 the eye tracking signals from the eye tracking system <NUM>, including an eye relief signal indicative of the eye relief of the user. The eye relief is a measure of the distance between an eye of the user and the lens <NUM>, <NUM> of the HMD <NUM>. There may be a single eye relief signal indicative of an eye relief of both eyes <NUM>, <NUM>, for example an average, maximum, or minimum eye relief of both eyes together, or a separate eye relief signal may be received for each eye <NUM>, <NUM>.

The eye relief signal can be calculated by the eye tracking system <NUM>, for example by using position of glints caused by illuminators <NUM>, <NUM> in the eye tracking system <NUM> to determine the distance from each illuminator <NUM>, <NUM> to the cornea. As the position of each illuminator <NUM>, <NUM> relative to the respective camera <NUM>, <NUM> is known, as is the position of the lens <NUM>, <NUM>, geometry can be used to determine the distance between the cornea and the lens <NUM>, <NUM>. The distance between the lens <NUM>, <NUM> and the cornea, commonly referred to as the Z-distance, is a normal output signal of an eye tracking system <NUM>.

If the eye relief signal is required to be between the entrance pupil of the eye and the lens <NUM>, <NUM>, a predetermined offset can be applied that is equal to an average distance between the cornea and the entrance pupil. This predetermined offset may be <NUM> to <NUM>.

The determination of eye relief may also take into account the gaze angle of the user, in order to generate a more accurate signal. As the eye rotates, the centre of the cornea will rotate relative to the centre of rotation of the eye, and therefore the distance between the cornea and the lens <NUM>, <NUM> will change. The direction of gaze may therefore be used to modify the eye relief signal.

The eye tracking signal is used by the CPU <NUM> to determine S104 what parts of the display <NUM>, <NUM> are visible to the user. The visible parts of the display <NUM>, <NUM> are calculated by the CPU <NUM> by reference to a look-up table that correlates a known visible portion of the display <NUM>, <NUM> with the eye relief indicated by the eye tracking system <NUM>. This is possible as the portion of the display <NUM>, <NUM> visible will be the same with any user when the eye relief is a certain distance. The look-up table may therefore be predetermined and stored in memory. The values stored in the look-up table are initially calculated in knowledge of the dimensions, characteristics, and relative positions and/or sizes of the components of the HMD <NUM>, and in particular the displays <NUM>, <NUM> and lenses <NUM>, <NUM>, e.g. the diameter of the lens <NUM>, <NUM>.

Alternatively, the visible parts of the display <NUM>, <NUM> may be determined through calculations performed by the CPU <NUM> during use of the system. For example, simple geometric calculations may calculate the visibility of the display in knowledge of the eye relief of the user and the geometry of the HMD <NUM>.

In some arrangements, the shape of the lens <NUM>, <NUM> may also affect the visible display area. For example, some lenses include cut-out portions in order to fit around a nose of the user. In such a case, the or each processor may consider the shape of the lens when determining which areas of the display to render at each eye relief.

Once a visible portion of the display <NUM>, <NUM> has been determined by the processor, the CPU <NUM> will then determine the image to be rendered and this information can be transmitted S106 to the GPU <NUM>. The GPU <NUM>, in turn, will only render S108 the portion of the image for display <NUM>, <NUM> that will be visible to the user of the HMD <NUM>. In this way, the total rendering workload of the GPU <NUM> can be lowered to only that which the user can see, which can improve latency, rendering quality, framerate, and power consumption of the HMD <NUM>, for example. Once the GPU <NUM> has rendered the image, each image is fed S110 to the displays of the HMD.

Where the eye relief is determined separately for each eye, the CPU <NUM> and GPU <NUM> will determine and render, respectively, different areas of the display <NUM>, <NUM> depending on the specific eye relief of each eye <NUM>, <NUM>.

<FIG> is a depiction of the difference that the operation of the present HMD <NUM> has on the total area rendered on each display <NUM>, <NUM>. One display <NUM> is shown in <FIG>, although it will be apparent that both displays <NUM>, <NUM> will be affected when the described method is in operation.

A total area of the display <NUM> is shown as a rectangle. The concentric circles denote areas <NUM>, <NUM>, <NUM> within the rectangle representing the different portions of the display <NUM> that are rendered depending on the eye relief determined by the eye tracking system <NUM>. The visible area is approximated as a circle due to the effect of the lens <NUM>, which is circular in the present embodiment. It will be apparent to the skilled person that the actual rendered area on the display <NUM> may not be circular, as there may be distortion - e.g. pin cushion distortion - introduced to the image by the lens <NUM> and therefore the image rendered on the display <NUM> must include an inverse of this distortion in order for the rendered image to be displayed correctly to the user.

An outermost area <NUM> is that visible to the user when the eye relief is at a minimum, in this case <NUM>. The visible area <NUM> is therefore a maximum and incorporates a large proportion of the total area of the display <NUM>. It will be apparent that the areas of the display <NUM> that are not rendered, i.e. all of the area of the display that is outside of the outermost area <NUM>, are not visible to the user of the HMD <NUM> and therefore no image need be displayed on these portions. Where each pixel is illuminated without the need for a backlight, such as in an OLED display, the fact that a proportion of the pixels will not be emitting any light will result in a lower overall power consumption of the display <NUM>.

The visible area of the display <NUM> in the example of <FIG> may change with movements of the user's eye, due to changes in viewpoint caused by eye rotation, but it is mainly dependent upon the eye relief. As such, there will exist non-active portions of the display that cannot be viewed by the user and will have no effect on the user's enjoyment of the HMD <NUM>.

The two further circles or areas <NUM>, <NUM> depict the visible area of two other eye reliefs of potential users. For example, the innermost area <NUM> depicts the visible area at a <NUM> eye relief whilst the intermediate area <NUM> depicts the visible area at a <NUM> eye relief. It will be clear to the skilled person that the visible area of the display <NUM> is smaller the larger the eye relief of the user. As such, a greater proportion of the display <NUM> is not used for displaying an image.

It will be apparent that the eye relief of the user is not limited to three different values and that therefore the system may vary the area rendered in smaller increments, such as every millimetre or every two millimetres. Alternatively, the area rendered may be continuously variable based on a measurement of eye relief that is only limited by the resolution of the eye relief signal generated by the eye tracking system <NUM>. In such a case, in order that a look-up table need not be generated for every possible variation of eye relief, the rendered area may be calculated by the processor or processors during operation of the HMD <NUM>, using an algorithm.

Changing eye relief can have a significant impact on the viewable area of the display <NUM>, as is shown in the table below. For comparison purposes, the values in the table are based on an assumed <NUM> distance between the lens and the display, and a <NUM> x <NUM> display size.

As can be seen, an increase in eye relief of <NUM> can result in a reduction in viewable area of <NUM>% for the user. The savings in processing and battery, amongst other things, can therefore be very great if the system takes into account the physical limitations of the user.

<FIG> shows the effect of another method of operation of the system on what is viewable on the display. In this method, the CPU <NUM> also receives a gaze tracking signal from the eye tracking system <NUM>. The CPU <NUM> can therefore instruct the GPU <NUM> to carry out foveated rendering on the image to be displayed, i.e. to vary the resolution of the displayed image based on the position of the gaze of the user.

In <FIG>, the user has an eye relief of <NUM>, and therefore the total area visible to the user is depicted by the large circle <NUM>. However, as the gaze of the user is also being tracked by the eye tracking system <NUM>, it is possible to determine that the actual visible region - i.e. the region that the user can see at any one time - is actually smaller than the total area that is potentially visible. The area to render can therefore be shrunk by the same amount.

The processor therefore determines the actual visible area <NUM>, taking into account the gaze direction of the user, and instructs the GPU to render the area that is visible. As before, the CPU <NUM> may refer to a look-up table that includes not only eye relief but also gaze direction in order to determine the visible area <NUM> of the display <NUM>, or the CPU <NUM> may use an algorithm to determine the visible area <NUM>.

In addition to determining the visible area <NUM>, the CPU <NUM> also determines an area of the display that the gaze is focussed on and therefore can utilise the fact that the human eye can only see clearly in a relatively small area of vision, which is provided by the part of the retina called the fovea. Because of this phenomenon, it is not necessary to render the entire visible area in full resolution. Instead, the parts of the image directly around the gaze point, termed the foveal region <NUM>, can be rendered in full resolution and the parts of the visible region outside of this may be rendered at a lower resolution. This is shown in <FIG>. By limiting the resolution of the area outside of the foveal region <NUM>, the total amount of processing required from the GPU <NUM> can be further reduced, without any detrimental effect on the user experience.

As the gaze of the user moves, the foveal region <NUM> will also move. The CPU <NUM> will therefore need to continually update the area requiring rendering, taking into account the current gaze direction of the user. Whilst the processing of the GPU <NUM> is therefore further reduced, the CPU <NUM> may have a greater workload when providing foveated rendering than when only tracking the eye relief of the user.

As discussed in relation to <FIG>, the movement of the eyes to the sides can have a further limiting effect to the viewable area of the display. Again assuming a <NUM> lens to display distance and a <NUM> x <NUM> display, the following table shows how rotation of the eyes to sides has a significant effect on the viewable area.

In these cases, the side the user is looking towards has a lower field of view in the periphery, whilst the side the user is looking away from will have a larger field of view in the periphery. As the peripheral vision of a person is not as detailed as the foveal view, the foveated rendering can take this into account when determining how much of the display to render. When foveated rendering is not used, such as in the embodiment described below, it is possible to simply omit to render the parts of the display in the direction the user is not directly looking, as the peripheral vision of the user is incapable of properly distinguishing the image anyway.

In a further embodiment, the CPU <NUM> utilises the eye tracking signal and the eye relief signal as in the previous embodiment, but the system does not perform foveated rendering. As can be seen in <FIG>, the total visible area is again depicted as a circle, and is dependent only on the eye relief of the user. A gaze point is also shown. As the gaze point is slightly to the right of centre, it will be apparent that the user will no longer be able to see the area of the display on the far left, i.e. whilst it is possible for that area of the screen to be seen by the user through the lens, at the current time, the portion on the far left is not visible due to the gaze direction of the user. As such, the area of the total visible area that is not visible need not be rendered by the GPU.

Movement of the gaze point towards the right, however, does not cause additional rendering on the right side of the total visible area, as this will remain non-visible due to the lens in conjunction with the eye relief of the user. In this way, the total rendered area can be lowered by taking into account the gaze direction of the user but without implementing foveated rendering. This means that the processor will need to continuously receive the gaze tracking signal and feed this to the GPU to instruct rendering of the correct area, but does not require as precise eye tracking data as when implementing foveated rendering.

In embodiments where an eye tracking signal is not required, the sensor system may utilise a sensor other than an eye tracking sensor to detect the eye relief of the user. Any other sensor suitable for detecting a distance between a part of the HMD and the eye of the user can be used. Specific, but nonlimiting, examples of such sensors include time-of-flight sensors, ultrasonic sensors, and capacitive sensors. It is also possible to use manual adjustment by the user of an eye relief setting, which manually controls how far away the lens is from the user's eye. In this case, the sensor system will be adapted to determine an adjustment of the manual eye relief setting using any suitable form of sensor. Of course, where manual adjustment is used, an assumption will have to be made about the likely actual eye relief of the user based on the manually set distance. A predetermined offset may be used for this purpose, which is based on the eye relief of an average user at that manual distance setting.

In some embodiments, it may be advantageous to include a degree of user feedback when determining how much of the display to render. This could ensure that there is a degree of flexibility to take into account any tolerances of the system and in particular the eye relief signal. For example, the at least one processor may be configured to perform a calibration where rendered area of the display is varied over time and the user indicates when they can - or cannot - see an edge of the rendered image. Thus, the system can learn or refine the rendered area on the display in response to user feedback.

Claim 1:
A head-mounted display system (<NUM>), comprising:
a display (<NUM>, <NUM>) for displaying an image to an eye (<NUM>, <NUM>) of a user;
a lens (<NUM>, <NUM>), interposed between the eye (<NUM>, <NUM>) of the user and the display (<NUM>, <NUM>), through which the user views the display;
a sensor system (<NUM>, <NUM>) configured to detect an eye relief and to output an eye relief signal indicative of the eye relief; and
at least one processor (<NUM>) configured to receive the eye relief signal and to modify an output of a portion of the display (<NUM>, <NUM>) in response to the eye relief signal, wherein the size of the portion of the display is dependent on the eye relief signal.