Patent Description:
A user of a VR headset that implements the rolling band display driving technique may, when looking at the top or the bottom of the display(s), notice unwanted visual artifacts, such as "judder," with respect to moving or animating objects in those regions. Judder causes the user to perceive a "double ghosting effect" where a moving object - such as a virtual representation of the user's hand moving at the bottom of the display - appears to bounce between two locations (or separate from itself) frame-to-frame. The cause of this unwanted visual artifact near the top and bottom of the display(s) is partly due to the target render time that is provided to the application for rendering a given frame, and partly due to a re-projection step that applies rotational corrections to the application-rendered frame to present an image that is aligned with the most up-to-date estimate of the user's head pose.

Consider a graphics-based application, such as a video game, which outputs rendered frames that are used to present images on the display panel(s) of a VR headset. In order to render the correct scene based on the user's head pose, the application receives, as input, a predicted pose of the user's head. However, with a rasterization-based rendering approach, the application can only accept a single pose (based on a single target render time) as input for rendering the frame in question. This means that a single target render time must be chosen for the original pose prediction of the user's head. One option, in a VR system that uses the rolling band display driving technique, is to choose, as the target render time for every frame, the time when the middle row of pixels will be illuminated. Although this approach is expected to minimize the total amount of re-proj ection adjustments for the entire frame, it nevertheless fails to mitigate the unwanted judder artifacts that are noticed by the user near the top and the bottom of the display(s) because the user is not always looking at the middle of the display panel(s). In fact, the user's eyes tend to wander about the display panel(s), often gazing at the top or the bottom of the display(s) as the VR headset is used. When the user's eyes stray from the middle of the display to the top or the bottom of the display, for example, the re-projection adjustments that are applied to the pixel values near the top or the bottom of the display(s) can cause noticeable judder artifacts with respect to moving or animating objects in those regions of the display.

Provided herein are technical solutions to improve and enhance these and other systems.

<CIT> discloses a head-mounted display (HMD) device that determines a predicted pose associated with a future position and orientation of the HMD, renders a current frame based on the predicted pose, determines a set of motion vectors based on the current frame and a previous frame, generates an updated image based on the set of motion vectors and the current frame, and displays the updated image on the HMD.

<CIT> discloses a virtual reality apparatus comprising: a compute cluster comprising global illumination circuitry and/or logic to perform global illumination operations on graphics data in response to execution of a virtual reality application and to responsively generate a stream of samples; a filtering/compression module to perform filtering and/or compression operations on the stream of samples to generate filtered/compressed samples; a network interface to communicatively couple the compute cluster to a network, the filtered/compressed samples to be streamed over the network; a render node to receive the filtered/compressed samples streamed over the network, the render node comprising: decompression circuitry/logic to decompress the filtered/compressed samples to generate decompressed samples; a sample buffer to store the decompressed samples; and sample insertion circuitry/logic to asynchronously insert samples into a light field rendered by a light field rendering circuit/logic.

"<NPL> discusses various challenges and trade-offs that should be considered when implementing an asynchronous time warp on consumer virtual reality hardware.

<CIT> discloses a method which includes determining a position of the wearable display device based on a motion sensor; rendering, by a graphics processing unit, an image based on the determined position; determining a first updated position of the wearable display device based on the motion sensor; warping, by a warp engine, a first portion of the rendered image based on the first updated position; and displaying the warped first portion of the rendered image on a display of the wearable display device.

The detailed description is described with reference to the accompanying drawings.

Described herein are, among other things, techniques and systems, including a head-mounted display (HMD) system, for dynamically targeting a render time for a given frame based on eye tracking and the implied scan out latency for individual subsets of pixels. Using this approach, little-to-no re-projection adjustments are applied to the pixel values at the location of the display(s) where the user is looking during image presentation, which mitigates unwanted, re-projection-based visual artifacts in a "region of interest" where the user is looking. In general, when large-scale re-projection adjustments are applied to an application-rendered frame, the resulting re-projected frame will exhibit unwanted visual artifacts, such as judder, which manifests as a moving or animating object that splits apart into two or more objects, frame-to-frame. By contrast, when small-scale re-projection adjustments are applied to an application-rendered frame, the resulting re-projected frame will mitigate these unwanted visual artifacts caused by re-projection. Thus, with the goal of making as accurate of a prediction as possible to render the frame correctly, and to make as few re-projection adjustments as possible during re-projection, the techniques and systems described herein improve the image quality in the region of interest where the user is looking by virtue of minimizing the re-projection adjustments to the pixel values in the region of interest. This improves the display quality of a HMD that implements a rolling band display driving technique.

In an illustrative example, a HMD may be worn by a user for purposes of immersing the user in a virtual reality (VR) environment or an augmented reality (AR) environment. One or more display panels of the HMD present images based on frames that are output by an application (e.g., a video game), and these images are viewed by a user through the optics that are included in the HMD, making the user perceive the images as if the user was immersed in a VR or AR environment. The HMD may utilize a rolling band technique to drive a display panel(s) of the HMD by illuminating individual subsets of the pixels in sequence over an illumination time period. Thus, the illumination propagates (or "rolls") across the display panel(s) during the presentation of an image for a given frame.

The HMD system, including the HMD, may also utilize a technique called "re-projection" to compensate for slight inaccuracies in an original pose prediction of the HMD and/or to compensate for the application failing to make frame rate, which has the same effect as an original pose prediction that is slightly inaccurate. For example, a re-projected frame can be generated using pixel data from an application-rendered frame by transforming (e.g., through rotation and re-projection calculations) the application-rendered frame in a way that accounts for an updated prediction of the pose of the HMD.

Described herein are techniques and systems for mitigating unwanted visual artifacts caused by re-projection in a region of interest on the display(s) where a user is looking, thereby improving the display performance by making these unwanted visual artifacts, if present, unnoticeable to the viewing user because they will not manifest in the region of interest. In an example process, before providing pose data as input to an application for rendering the next frame, logic of the HMD system (e.g., a compositor) may predict a location on the display panel where a user (who is wearing the HMD) will be looking during the illumination time period for the frame. This prediction may be based at least in part on eye tracking data generated by an eye tracking system of the HMD system. In some embodiments, the logic of the HMD system is configured to predict a gaze point where the user will be looking during the illumination time period based on the eye tracking data, and to determine a location on the display panel that corresponds to the predicted gaze point. The logic of the HMD system is also configured to determine a time, within the illumination time period, at which an individual subset of the pixels - that corresponds to the predicted location on the display panel - will be illuminated. Using head tracking data generated by a head tracking system of the HMD system, the logic can then predict a pose that the HMD will be in at the determined time (which was dynamically targeted based on the eye tracking data indicating where the user will be looking during the illumination time period). Pose data indicative of the predicted pose of the HMD can then be sent to the application for rendering a frame, and logic of the HMD system, upon receiving the frame from the application, can apply re-projection adjustments to the pixel data for the application-rendered frame to obtain modified pixel data associated with a re-projected frame. This modified pixel data can be output to a frame buffer for presenting an image on the display panel(s), whereby the individual subsets of the pixels are illuminated in sequence over the illumination time period to present the image.

By dynamically targeting the render time for an application-rendered frame based on eye tracking and the implied scan out latency for individual subsets of pixels, the scene (or picture) can be rendered for a time at which light (photons) from the region of interest on the display panel(s) will actually reach the user's eye. This means that the scene can be rendered correctly in the region of interest, and, hence, merely small-scale re-projection adjustments, if any, are applied to the pixel data (values) in the region of interest where the user is looking during image presentation. In other words, if, by tracking the user's eyes, it can be determined that the user is likely to look at subset of the pixels within the top portion of the display panel(s) during the rolling illumination time period, logic (e.g., the compositor) of the HMD system can tell the application to render the scene in accordance with the user's head pose that is predicted for the time when that subset of the pixels will illuminate during the rolling illumination time period. If, for a subsequent frame, the user is predicted to look at a subset of the pixels within the bottom portion of the display panel(s) during the rolling illumination time period, the logic can tell the application to render the scene in accordance with the user's head pose that is predicted for the time when that subset of the pixels will illuminate during the rolling illumination time period. Thus, instead of always rendering the given frame for a time when the middle row of pixels will illuminate, the techniques described herein are directed to dynamically selecting a render time for the next frame based on where on the display panel(s) the user is predicted to look during image presentation. Said another way, for each frame, the logic of the HMD system is configured to tell the application which point in time to choose for the "picture of the world" that is to be presented on the display panel(s), and this point in time is chosen based on a prediction of where the user will be looking when that picture of the world is presented.

Also disclosed herein are systems, for example, systems including a display system (e.g., a HMD system), configured to implement the techniques and processes disclosed herein, as well as non-transitory computer-readable media storing computer-executable instructions to implement the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed, by way of example, in the context of video game applications, and specifically VR gaming applications, it is to be appreciated that the techniques and systems described herein may provide benefits with other applications, including, without limitation, non-VR applications (e.g., AR applications), and/or non-gaming applications, such as industrial machine applications, defense applications, robotics applications, and the like.

<FIG> is a diagram illustrating an example technique for dynamically targeting a render time for a given frame based on eye tracking. <FIG> depicts a head-mounted display (HMD) <NUM> worn by a user <NUM>. The HMD <NUM> in the example of <FIG> may include a single display panel <NUM> or multiple display panels <NUM>, such as a left display panel and a right display panel of a stereo pair of display panels. The one or more display panels <NUM> of the HMD <NUM> may be used to present a series of image frames (herein referred to as "frames") that are viewable by a user <NUM> wearing the HMD <NUM>. It is to be appreciated that the HMD <NUM> may include any number of display panels <NUM> (e.g., more than two display panels, a pair of display panels, or a single display panel). Hence, the terms "display panel," as used in the singular herein, may refer to either display panel <NUM> of a pair of display panels of a two-panel HMD <NUM>, or it may refer to a single display panel <NUM> of a HMD <NUM> with any number of display panels (e.g., a single-panel HMD <NUM> or a multi-panel HMD <NUM>). In a two-panel HMD <NUM>, a stereo frame buffer may render, for instance, <NUM> x <NUM> pixels on both display panels of the HMD <NUM> (e.g., <NUM> x <NUM> pixels per display panel).

The display panel(s) <NUM> of the HMD <NUM> may utilize any suitable type of display technology, such as an emissive display that utilizes light emitting elements (e.g., light emitting diodes (LEDs)) to emit light during presentation of frames on the display panel(s) <NUM>. As an example, display panels <NUM> of the HMD <NUM> may comprise liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, inorganic light emitting diode (ILED) displays, or any other suitable type of display technology for HMD applications.

The display panel(s) <NUM> of the HMD <NUM> may operate at any suitable refresh rate, such as a <NUM> Herz (Hz) refresh rate. The "refresh rate" of a display is the number of times per second the display can redraw the screen. The number of frames displayed per second may be limited by the refresh rate of the display. Thus, a series of frames may be processed (e.g., rendered) and displayed as images on the display such that a single frame of the series of frames is displayed with every screen refresh. That is, in order to present a series of images on the display panel(s) <NUM>, the display panel(s) <NUM> may transition from frame-to-frame, in the series of frames, at the refresh rate of the display, illuminating the pixels at every screen refresh. In some embodiments, the frame rate can be throttled, and phantom frames (based on re-projection) can be inserted between application-rendered frames.

The display system of the HMD <NUM> may also implement a rolling band type of display driving scheme. That is, the light emitting elements of the display panel(s) <NUM> may be individually-addressable such that individual subsets of the light emitting elements, and, hence, individual subsets of the pixels, can be illuminated independently and sequentially in a rolling band of illumination during an illumination time period. In some embodiments, both the array of pixels and the light emitting elements on the display panel(s) <NUM> are arranged in rows and columns, but not necessarily with a one-pixel per one-light emitting element correspondence. In this configuration, individual rows and/or individual columns of light emitting elements may be addressed in sequence, and/or individual groups of contiguous rows and/or individual groups of contiguous columns of light emitting elements may be addressed in sequence. As a consequence of addressing the light emitting elements in this "rolling" manner, the subsets of pixels that correspond to these individually-addressable subsets of light emitting elements may be "illuminated" independently.

As used herein, "illuminating a pixel" means illuminating the light emitting element that corresponds to that pixel. For example, a LCD illuminates a light emitting element of a backlight to illuminate the corresponding pixel(s) of the display. Furthermore, as used herein, a "subset of pixels" may comprise an individual pixel or multiple pixels (e.g., a group of pixels). In some embodiments, a subset of pixels includes a row of pixels, a column of pixels, a group of contiguous rows of pixels, or a group of contiguous columns of pixels. Thus, in an aspect of the techniques and systems described herein, subsets of pixels can be scanned out and illuminated in sequence (sequentially), such as by scanning out and illuminating each row of pixels in sequence, starting with a first row of the pixels (e.g., a top row of pixels) and ending with a last row of the pixels (e.g., a bottom row of pixels). However, any suitable pattern of illumination can be employed using the techniques and systems described herein (e.g., a snake-like pattern of illumination, column-by-column illumination, multiple rows/columns of pixels at a time in sequence, etc.).

In order to drive the display panel(s) <NUM> in this "rolling" manner, the HMD <NUM> may include, among other things, a display controller(s), display driver circuitry, and similar electronics for driving the display panel(s) <NUM>. Display driver circuitry may be coupled to an array of light emitting elements of the display panel(s) <NUM> via conductive paths, such as metal traces, on a flexible printed circuit. In an example, a display controller(s) may be communicatively coupled to the display driver circuitry and configured to provide signals, information, and/or data to the display driver circuitry. The signals, information, and/or data received by the display driver circuitry may cause the display driver circuitry to illuminate the light emitting elements in a particular way. That is, the display controller(s) may determine which light emitting element(s) is/are to be illuminated, when the element(s) is/are to illuminate, and the level of light output that is to be emitted by the light emitting element(s), and may communicate the appropriate signals, information, and/or data to the display driver circuitry in order to accomplish that objective.

Pixel data for a given frame can be output to a frame buffer for presenting the frame as an image on the display panel <NUM>. Pixel data for each frame may, in some embodiments, include a two-dimensional array of per-pixel values (e.g., color values). In some embodiments, the pixel data further includes additional data or metadata, such as depth values. In some embodiments, pixel data may include data for each pixel that is represented by a single set of color and alpha values (e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values for one or more alpha channels). This pixel data can be output to a frame buffer (e.g., a stereo frame buffer) to present on image(s) on the display panel(s) <NUM> of the HMD <NUM> with a desired visual effect.

The HMD <NUM> may represent a VR headset for use in VR systems, such as for use with a VR gaming system. However, the HMD <NUM> may additionally, or alternatively, be implemented as an AR headset for use in AR applications. In AR, a user <NUM> sees virtual objects overlaid on a real-world environment, whereas, in VR, the user <NUM> does not see a real-world environment, but is fully immersed in a virtual environment, as perceived via the display panel(s) <NUM> and the optics (e.g., lenses) of the HMD <NUM>. Examples described herein pertain primarily to a VR-based HMD <NUM>, but it is to be appreciated that the HMD <NUM> is not limited to implementation in VR applications.

In general, a graphics-based application <NUM> (e.g., a video game) executing on a computing device - such as the HMD <NUM> itself, or a computing device (e.g., a personal computer (PC), game console, etc.) associated with, and coupled to, the HMD <NUM> as part of a HMD system - may be configured to output a series of frames <NUM>(<NUM>), <NUM>(<NUM>), <NUM>(<NUM>), and so on (collectively <NUM>). The series of frames <NUM> are ultimately presented as images on the display panel(s) <NUM> of the HMD <NUM>. The example of <FIG> depicts three example frames <NUM>(<NUM>) (or frame "F"), <NUM>(<NUM>) (or frame "F+<NUM>"), and <NUM>(<NUM>) (or frame "F+<NUM>") with respect to a rendering timeline <NUM> to illustrate how the frames <NUM> can be rendered in series. Here, the application <NUM> renders frame F first, then frame F+<NUM>, and then frame F+<NUM>, in sequence, from left to right on the rendering timeline <NUM>. The rendering timeline <NUM> also shows the rendering workloads <NUM> of a compositor <NUM> of the HMD <NUM> (or HMD system) towards the end of each rendering interval for each frame <NUM>. An individual rendering workload <NUM> of the compositor <NUM> for a given frame <NUM> may represent adjustments that are applied to the pixel data output by the application <NUM> before rendering a final image on the HMD <NUM>. Such adjustments may include, without limitation, adjustments for chromatic distortion, panel masking, re-projection, and the like, which are applied to the frame <NUM> output by the application <NUM> before rendering a final image on the HMD <NUM>. Accordingly, the frames <NUM> that are shown in <FIG> are meant to represent "actual" frames in the sense that they are output from the application <NUM>, which may represent a video game application, or any other type of graphics-based application. The application <NUM> may be executed in a graphics pipeline that outputs pixel data, and the compositor <NUM> is configured to modify that pixel data, and to output the modified pixel data to a frame buffer (e.g., a stereo frame buffer).

During runtime, an eye tracking system of the HMD <NUM> (or HMD system) may generate eye tracking data about a gaze point where the user <NUM> is looking at any given moment in time. The eye tracking system may include a camera or other optical sensor inside the HMD <NUM> to capture image data of a user's eyes <NUM>, and the captured image data can be used to determine motion vectors, interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye <NUM> relative to HMD <NUM>, including a magnitude of torsion and rotation (e.g., roll, pitch, and yaw) and gaze directions for each eye <NUM>. In one example, infrared light is emitted within the HMD <NUM> and reflected from each eye <NUM>. The reflected light is received or detected by a camera of the HMD <NUM> and analyzed to extract eye rotation from changes in the infrared light reflected by each eye <NUM>. In some embodiments, the eye tracking system may integrate information from past measurements, measurements identifying a position of a user's <NUM> head, and 3D information describing a scene presented on the display panel(s) <NUM>. Thus, information for the position and orientation of the user's <NUM> eyes <NUM> can be used to determine the gaze point in a virtual scene presented by HMD <NUM> where the user <NUM> is looking. Many methods for tracking the eyes <NUM> of the user <NUM> can be employed by the eye tracking system of the HMD <NUM> (or HMD system), and these are merely examples of eye tracking techniques that can be employed.

In any case, the eye tracking data generated by the eye tracking system of the HMD <NUM> (or HMD system) can be used by the compositor <NUM> to predict a gaze point where the user <NUM> will be looking during an illumination time period for the upcoming frame <NUM>, and this gaze point can be correlated to a location on the display panel(s) <NUM>, which can be defined at any suitable level of granularity (e.g., an individual subset (e.g., row) of the pixels, an individual group of contiguous subsets (e.g., rows) of the pixels, other regions of the display panel(s) <NUM>, etc.). In some embodiments, the most recent eye tracking data (e.g., the gaze point) can be used as a proxy for predicting where the user <NUM> will be looking during the illumination time period for an upcoming frame <NUM>. With today's eye tracking technology, it is difficult to accurately predict where the user's eyes will be at a future time because of the ballistic motions that can be exhibited by the eyes during rapid eye movement. This is one reason for using the most recent gaze point estimation as a proxy for predicting a future gaze point. However, as eye tracking improves over time, the prediction of where the user <NUM> will be looking at a future time may become more accurate, and, hence, a prediction of a future gaze point may be determined as a function of past eye tracking data. For instance, a future gaze point can be predicted based on motion vector estimations of the user's eyes <NUM>, and/or based on additional data.

<FIG> also shows a "scan out + illumination" timeline <NUM> to illustrate the illumination time periods corresponding to each frame <NUM> that is rendered in the rendering timeline <NUM>. For example, after frame F is rendered by the application <NUM>, and after the compositor <NUM> performs its rendering workload <NUM>(<NUM>) to modify the pixel data for frame F, an image corresponding to frame F is presented on the display panel(s) <NUM> during the illumination time period <NUM>(<NUM>) for frame F. During this illumination time period <NUM>(<NUM>), subsets of pixel values (pixel data) for frame F are scanned out to the display panel(s) <NUM> via a display port (e.g., a high-definition multimedia interface (HDMI)), in sequence, and the light emitting elements of the display panel(s) <NUM> are illuminated in sequence to cause individual subsets of the pixels to illuminate in sequence. <FIG> illustrates an example where the individual subsets of pixels correspond to individual rows of pixels. For example, the display panel(s) <NUM> may have multiple rows of pixels, starting with a first row ("Row <NUM>"), which may be the top row, and ending with a last row ("Row N"), which may be the bottom row. The illumination time period <NUM> may also be described in terms of the times ti to tN shown in <FIG>. For example, the first subset of pixels (e.g., "Row <NUM>") may illuminate at time, ti, and the last subset of pixels (e.g., "Row N") may illuminate at time, trr, while the intermediate subsets of pixels may illuminate at respective times between these outer bounds of the illumination time period <NUM>. As mentioned, for a <NUM> refresh rate, the illumination time period <NUM> may be roughly <NUM> in duration.

In the example of <FIG>, in order to render frame F, the compositor <NUM> may predict the gaze point where the user <NUM> will be looking during the illumination time period <NUM>(<NUM>) for frame F. In the example of <FIG>, the predicted gaze point for frame F is a gaze point that corresponds to an individual subset (or a group of contiguous subsets) of the pixels within a top portion of the display panel(s) <NUM>. This may be described as the compositor <NUM> predicting a location of the display panel(s) <NUM> where the user <NUM> will be looking during the illumination time period <NUM>(<NUM>), and, in this case, the predicted location for frame F is a location that corresponds to an individual subset (or a group of contiguous subsets) of the pixels within the top portion of the display panel(s) <NUM>. It is to be appreciated that a "location" of the display panel(s) <NUM> can be defined in any suitable manner and/or at any suitable level of granularity suitable for describing where, on the display panel(s) <NUM> the user <NUM> may be looking. The example of <FIG> illustrates an example where the location of the display panel(s) <NUM> can be determined based on the rows of pixels of the display panel(s). Depending on the accuracy of the eye tracking system of the HMD <NUM> (or HMD system), and the resolution at which the individual subsets of pixels can be addressed, the prediction of where the user <NUM> will be looking can be made at any suitable level of granularity, such as by estimating that the user <NUM> is looking at a given subset(s) of pixels (e.g., an individual row(s) of pixels, an individual column(s) of pixels, etc.), or individual groups of contiguous subsets (e.g., groups of contiguous rows, groups of contiguous columns, etc.) of pixels.

Once the compositor <NUM> determines where the user <NUM> will be looking when the image corresponding to frame F is presented, the compositor <NUM> may be further configured to determine a time, within the illumination time period <NUM>(<NUM>) for frame F, at which an individual subset of the pixels that corresponds to the location on the display panel(s) <NUM> (where the user <NUM> is predicted to be looking) will be illuminated. For example, if the user <NUM> is predicted to be looking at an individual subset (or a group of contiguous subsets) of the pixels within the top portion of the display panel(s) <NUM> during the illumination time period <NUM>(<NUM>) for frame F, the compositor <NUM> may determine a time at which that subset of the pixels (within the top portion of the display panel(s) <NUM>) will be illuminated. In an example, the compositor <NUM> may determine that the first row of pixels (e.g., "Row <NUM>") - which is within the top portion of the display panel(s) <NUM> - will illuminate at time ti. In this example, the compositor <NUM> may determine a target render time (within the illumination time period <NUM>(<NUM>) for frame F) as target render time = t<NUM>, because the user <NUM> is predicted to be looking at a location corresponding to the first row of pixels within the top portion of the display panel(s) <NUM> during the illumination time period <NUM>(<NUM>) for frame F, based on the eye tracking data. As another example, if the user <NUM> is predicted to look at the <NUM>th row of pixels, for example, the compositor <NUM> may determine, as the target render time, an illumination time of the <NUM>th row of pixels.

In addition to the eye tracking system, a head tracking system of the HMD <NUM> (or HMD system) may generate head tracking data about the pose of the HMD <NUM>, and the compositor <NUM> is configured to predict a pose that the HMD <NUM> will be in at the determined time (the targeted render time). Thus, for frame F, the compositor <NUM> may predict a pose that the HMD <NUM> will be in at the time, ti, which was determined to be the time when the subset of pixels (e.g., "Row <NUM>") within the top portion of the display panel(s) <NUM> will be illuminated during the illumination time period <NUM>(<NUM>). The compositor <NUM> may send pose data indicative of the predicted pose to the application <NUM> for rendering frame F. Providing the pose data to the application <NUM> in advance allows the application <NUM> to output pixel data for rendering imagery on the HMD <NUM> in a way that is correct for the user's <NUM> predicted head pose at the future, targeted render time, ti. This means that the application <NUM> renders a scene that is appropriate for the user's predicted head pose at time, ti, when light from the portion of the display panel(s) <NUM> where the user <NUM> is predicted to be looking reaches the user's eye(s) <NUM>.

The graphics logic of the HMD <NUM> (or HMD system) may be asynchronous, or synchronous. In an asynchronous system, the compositor <NUM> runs separate (on a separate, asynchronous thread) from the application <NUM> on a graphics processing unit (GPU) of the HMD <NUM> (or HMD system). For instance, the application <NUM> may call a function to receive pose data from the compositor <NUM>, and the compositor <NUM> may provide the application <NUM> with the requested pose data (predicted to the target render time, ti, for frame F) so that the application <NUM> can render the frame <NUM> (e.g., frame F) according to the pose data, which corresponds to a virtual camera pose used to render the scene. Assuming the application <NUM> finishes rendering the frame <NUM> before the compositor's workload <NUM> starts, the compositor <NUM> is configured to take the frame <NUM> (e.g., left and right image frames) from the application <NUM> and distort the frame(s) <NUM> into the back buffer(s) onto the display panel(s) <NUM> of the HMD <NUM>. For example, the compositor <NUM> may perform, without limitation, chromatic distortion, panel masking, re-projection, and the like, before rendering a final image on the HMD <NUM> based on the frame <NUM> output by the application <NUM>.

The compositor <NUM> may execute in a high-priority context mode, which means that when it is time for the compositor <NUM> to start its workload <NUM>, the GPU driver allows the compositor <NUM> to preempt the application <NUM> (e.g., by interrupting the application <NUM> from rendering, if it is still rendering a frame <NUM>, and/or preventing the application <NUM> from starting to render a next frame <NUM>). The compositor <NUM> may allocate a time slot (up to <NUM> on average) at the end of each rendering interval to do its work <NUM>, regardless of what is happening with the application <NUM>. Thus, at every rendering interval, the compositor <NUM> renders "something" in the sense that the compositor <NUM> obtains the best frame <NUM> it can obtain from the application <NUM> (e.g., either a fresh/new frame <NUM>, or a previously-rendered frame <NUM>), and the compositor <NUM> uses the pixel data associated with that frame <NUM> to put pixel data in the frame buffer for output on the display panel(s) <NUM> of the HMD <NUM>. The compositor's <NUM> ability to output different pixel data for each screen refresh, no matter what, is the mechanism that keeps everything "live" and keeps the images rendered on the HMD <NUM> from hitching badly when the application <NUM> is not making frame rate.

By dynamically targeting the render time as a time (e.g., ti for frame F) at which a subset of pixels nearest to where the user will be looking will illuminate, the image corresponding to frame F can be presented during the illumination time period <NUM> with merely small-scale re-projection adjustments, if any, at a location on the display panel <NUM> where the user <NUM> is looking during the illumination time period <NUM>. This means that the user <NUM> will not see unwanted visual artifacts caused by re-projection with respect to moving or animating objects in the region of interest where the user is looking during the presentation of the image.

After frame F is rendered, the dynamic render time targeting may iterate for each frame in the series of frames <NUM>, as they are rendered. For example, for frame F+<NUM>, the eye tracking data may indicate that the user <NUM> will be looking at an individual subset (or a group of contiguous subsets) of the pixels within the middle portion of the display panel(s) <NUM> during the illumination time period <NUM>(<NUM>) for frame F+<NUM>, which causes the compositor <NUM> to dynamically target, as the render time for frame F+<NUM>, a time at which that subset (or group of subsets) of pixels in the middle portion of the display panel(s) <NUM> will illuminate. For example, the compositor <NUM> may determine that the middle row of pixels will illuminate at time, <MAT>, and, thus, may predict a pose that the HMD <NUM> will be in at time, <MAT>. The compositor <NUM> may send pose data indicative of this predicted pose to the application <NUM> for rendering frame F+<NUM>, and, in response, the compositor <NUM> may receive pixel data for frame F+<NUM>. During the compositor's <NUM> rendering workload <NUM>(<NUM>) for frame F+<NUM>, the compositor <NUM> may apply re-projection adjustments to the pixel data for frame F+<NUM>, among other adjustments that are applied during the rendering workload <NUM>(<NUM>), to obtain modified pixel data associated with a re-projected frame for frame F+<NUM>, and the compositor <NUM> may output the modified pixel data to a frame buffer so that a corresponding image can be presented on the display panel(s) <NUM> during the illumination time period <NUM>(<NUM>) for frame F+<NUM>.

Likewise, for frame F+<NUM>, the eye tracking data may indicate that the user <NUM> will be looking at an individual subset (or a group of contiguous subsets) of the pixels within the bottom portion of the display panel(s) <NUM> during the illumination time period <NUM>(<NUM>) for frame F+<NUM>, which causes the compositor <NUM> to dynamically target, as the render time for frame F+<NUM>, a time at which that subset (or group of subsets) of pixels in the bottom portion of the display panel(s) <NUM> will illuminate. For example, the compositor <NUM> may determine that the Nth row of pixels will illuminate at time, trr, and, thus, may predict a pose that the HMD <NUM> will be in at time, trr. The compositor <NUM> may send pose data indicative of this predicted pose to the application <NUM> for rendering frame F+<NUM>, and, in response, the compositor <NUM> may receive pixel data from the application <NUM> for frame F+<NUM>. During the compositor's <NUM> rendering workload <NUM>(<NUM>) for frame F+<NUM>, the compositor <NUM> may apply re-projection adjustments to the pixel data for frame F+<NUM>, among other adjustments, to obtain modified pixel data associated with a re-projected frame for frame F+<NUM>, and the compositor <NUM> may output the modified pixel data to a frame buffer so that a corresponding image can be presented on the display panel(s) <NUM> during the illumination time period <NUM>(<NUM>) for frame F+<NUM>.

In some embodiments, logic of the HMD <NUM> (or HMD system) may be configured to address different locations of the display panel(s) <NUM> at different resolutions. In this manner, since the eye tracking data can be used to predict a region of interest on the display panel(s) <NUM> where the user <NUM> will be looking during image presentation, the HMD <NUM> (or HMD system) can leverage this dynamic addressing mechanism to dynamically address the location of the display panel(s) <NUM> where the user <NUM> is looking at a first (high) resolution, and to dynamically address a remainder of the display panel(s) <NUM> at a second (lower) resolution that is less than the first resolution. This may provide improved display quality in the region of interest, as well as improved display performance in terms of reducing latency for scanning out pixel data because the area outside of the region of interest may have less pixel data to scan out.

<FIG> is a diagram illustrating how the degree of re-projection adjustments varies across the display for different subsets of pixels when using an approach to dynamically target the render time for a given frame <NUM> based on eye tracking, in accordance with embodiments disclosed herein. <FIG> shows a display panel <NUM> of the HMD <NUM>, both of which were introduced in <FIG>. In some embodiments, this display panel <NUM> may represent a left display panel or a right display panel of a stereo pair of display panels of the HMD <NUM>. The display panel <NUM> is shown as including a M × N array of pixels. In this case, the pixels are arranged in M columns and N rows, where M is any suitable number, N is any suitable number, and M and N may or may not be equal. As described herein, because the display panel <NUM> is driven using a rolling band display driving technique, the rolling illumination during the presentation of an image for a given frame <NUM> occurs by illuminating individual subsets of pixels in sequence over the illumination time period <NUM>, which was introduced in <FIG>. In the example of <FIG>, the individual subsets of pixels correspond to individual rows of pixels (e.g., Rows <NUM>-N), and the rows are illuminated in sequence from a first row ("Row <NUM>") to a last row ("Row N") over the illumination time period <NUM> for a given frame <NUM>. In the illustrated example, this means that the rolling illumination occurs from top-to-bottom, but it is to be appreciated that this is merely an example sequence in which subsets of pixels can be illuminated.

Consider an example where the targeted render time <NUM> is predicted to be the time, t<NUM>, within the illumination time period <NUM>, which may be described as a time period from time, ti, to time, trr. This example targeted render time <NUM> may be determined, for example, if the eye tracking data indicates that the user <NUM> will be looking at an individual subset (or a group of contiguous subsets) of the pixels within a top portion of the display panel <NUM> during the illumination time period <NUM> for the given frame; e.g., the third row of pixels ("Row <NUM>"). Accordingly, the compositor <NUM> is configured to predict a pose that the HMD <NUM> will be in at the targeted render time <NUM> (e.g., time, t<NUM>), and the compositor <NUM> can send pose data indicative of this predicted pose to the application <NUM> for rendering the given frame <NUM>. After receiving pixel data for the application-rendered frame <NUM>, the compositor <NUM>, during its rendering workload <NUM> for the given frame <NUM>, can apply re-projection adjustments to the pixel data.

To determine the re-projection adjustments that are to be applied to the received pixel data, the compositor <NUM>, after receiving the pixel data, can predict - based at least in part on updated head tracking data generated by the head tracking system of the HMD <NUM> (or HMD system) - a plurality of updated poses of the HMD <NUM> for a plurality of times, within the illumination time period <NUM>, at which the individual subsets of the pixels will be illuminated. For example, upon receiving, from the application <NUM>, the pixel data associated with the given frame <NUM>, the compositor <NUM> can predict a first updated pose that the HMD <NUM> will be in at time, ti, for the first row of pixels ("Row <NUM>"), a second updated pose that the HMD <NUM> will be in at time, t<NUM>, for the second row of pixels ("Row <NUM>"), a third updated pose that the HMD <NUM> will be in at time, t<NUM>, for the third row of pixels ("Row <NUM>"),. , and an Nth updated pose that the HMD <NUM> will be in at time, tN, for the Nth row of pixels ("Row N"). With the plurality of updated pose predictions in hand, the compositor <NUM> can calculate (or otherwise determine) a plurality of adjustments <NUM>(<NUM>)-(N) based at least in part on comparisons between the original pose prediction and individual ones of the updated poses. For example, the plurality of adjustments <NUM>(<NUM>)-(N) may include a first adjustment <NUM>(<NUM>) for a first individual subset of the pixels (e.g., the first row of pixels, "Row <NUM>"), a second adjustment <NUM>(<NUM>) for a second individual subset of the pixels (e.g., the second row of pixels, "Row <NUM>"), a third adjustment <NUM>(<NUM>) for a third individual subset of the pixels (e.g., the third row of pixels, "Row <NUM>"),. , and an Nth adjustment <NUM>(N) for an Nth individual subset of the pixels (e.g., the Nth row of pixels, "Row N"). In the example of <FIG>, an individual adjustment (e.g., the first adjustment <NUM>(<NUM>)) may be calculated (or otherwise determined) as a difference calculation based on the original pose prediction that was provided as input to the application <NUM> and the updated pose prediction for the corresponding pixel subset (e.g., the updated pose prediction for the time, ti, when the first row of pixels ("Row <NUM>") is going to illuminate).

The graph <NUM> shown in <FIG> illustrates how the re-projection adjustments are small-scale adjustments for the subsets of pixels near the location of the display panel <NUM> where the user <NUM> is looking. In the example of <FIG>, these pixels correspond to the pixels near the third row of pixels ("Row <NUM>"). For instance, there may be little-to-no re-projection adjustments made to the pixel values for the third row of pixels ("Row <NUM>"), and merely small-scale re-projection adjustments for pixel values of adjacent rows of pixels. This is because, in the example of <FIG>, the original pose prediction was made for the target render time, t<NUM>, which is a time when the third row of pixels ("Row <NUM>") will illuminate during the illumination time period <NUM> (i.e., during image presentation), and the updated pose prediction for the same time, t<NUM>, associated with the third row of pixels ("Row <NUM>") is likely to be the same as the original pose prediction, or, if different, the difference is expected to be smaller than the differences between the original pose prediction and others of the updated pose predictions. Accordingly, the re-projection adjustments applied to the pixel values for the third row of pixels ("Row <NUM>") are small-scale re-projection adjustments, if any are applied at all. The re-projection adjustments gradually increase in amount for rows that are farther from the third row of pixels ("Row <NUM>"). Thus, any re-projection adjustments for individual subsets (rows) of pixels that are adjacent to the third row of pixels ("Row <NUM>") will be less than the re-projection adjustments for those subsets (rows) of pixels that are far from (i.e., not adjacent to) the third row of pixels ("Row <NUM>"). This is why the re-projection adjustment applied to the pixel values for the Nth row of pixels ("Row N") is much larger than the re-projection adjustment applied to the pixel values for, say, the second row of pixels ("Row <NUM>"). However, because the user <NUM> is looking at an individual subset (or a group of contiguous subsets) of the pixels within the top portion of the display panel <NUM> (e.g., the third row of pixels ("Row <NUM>")) during image presentation, any unwanted visual artifacts caused by re-projection adjustments at the bottom of the display panel <NUM> go unnoticed. Meanwhile, the region of interest where the user <NUM> is looking will be free from these types of unwanted visual artifacts caused by re-projection.

The processes described herein are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes.

<FIG> illustrates a flow diagram of an example process <NUM> for dynamically targeting a render time for a given frame based on eye tracking, in accordance with embodiments disclosed herein. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, a frame <NUM>, of a series of frames, may be rendered for presenting a corresponding image on a display panel <NUM> of a HMD <NUM>. As mentioned, if the HMD <NUM> includes a pair of display panels, this frame <NUM> may be rendered for presenting a first image on the left display panel <NUM> and a second image on the right display panel <NUM>, or vice versa. As shown by the various sub-blocks within <NUM>, there may be various sub-operations that are performed by the logic of the HMD <NUM> (or HMD system) for purposes of rendering the frame <NUM> at block <NUM>.

At sub-block <NUM>, logic of a HMD <NUM> (or HMD system) (e.g., logic including the compositor <NUM>) may predict a location on the display panel <NUM> where a user <NUM> wearing the HMD <NUM> will be looking during an illumination time period <NUM> for the frame <NUM>. Here, the illumination time period <NUM> is a duration over which individual subsets of the pixels on the display panel <NUM> are illuminated in sequence to present an image on the display panel <NUM>. As shown by sub-block <NUM>, the prediction of the location on the display panel <NUM> where the user <NUM> will be looking may be based at least in part on eye tracking data generated by the eye tracking system of the HMD <NUM> (or HMD system). For instance, at sub-block <NUM>, the logic (e.g., compositor <NUM>) may predict, based at least in part on eye tracking data generated by the eye tracking system, a gaze point where the user <NUM> will be looking during the illumination time period <NUM> for the frame <NUM>, and this gaze point may correspond to a particular location on the display panel <NUM>. As such, the logic (e.g., compositor <NUM>) may be configured to determine a location on the display panel <NUM> that corresponds to the predicted gaze point (e.g., by accessing association data between possible gaze points and various locations on the display panel <NUM>, and/or calculating the location based on the eye tracking data and other parameters). In an illustrative example, the eye tracking data (e.g., eye orientation data) may indicate an eye orientation of the user wearing the HMD <NUM>, such as in the form of a number of degrees from a midline of a field of view (FOV) parameter of the HMD <NUM>. For example, based on a FOV parameter (e.g., <NUM> degree) of the HMD <NUM>, and possibly based on distortion parameters of the HMD optics), the eye orientation data may be used to estimate a subset (e.g., a row) or a group of contiguous subsets (e.g., a group of contiguous rows) of pixels that are closest or nearest to where the user will be looking during the illumination time period of the upcoming frame. For instance, if the eye orientation data indicates that the user will be looking +<NUM> degrees from a midline of a <NUM> degree FOV, this may be calculated degree offset may be correlated, through mathematical calculation, to a row (or group of contiguous rows) of the display panel <NUM> (e.g., the <NUM>th row on a <NUM> row display panel). Thus, the prediction of the location of the display panel <NUM> where the user will be looking may be based on the FOV parameter of the HMD <NUM>, in combination with the eye tracking data.

At sub-block <NUM>, the logic (e.g., compositor <NUM>) may dynamically target a render time <NUM> for the frame <NUM> based on the prediction of where, on the display panel <NUM>, the user <NUM> will be looking. That is, the logic may determine a time, within the illumination time period <NUM> for the frame <NUM>, at which an individual subset of the pixels that corresponds to (e.g., are within) the location on the display panel <NUM> will be illuminated. In the pictorial diagrams on the right of <FIG>, an example is shown where the logic predicts, at sub-block <NUM>, that the user <NUM> will be looking at a bottom portion of the display panel <NUM>, and, at sub-block <NUM>, the logic determines a targeted render time <NUM> within the illumination time period <NUM> (e.g., from time, ti, to time, tN) that corresponds to a time when a subset (e.g., row) of pixels in the bottom portion of the display panel <NUM> will illuminate.

At sub-block <NUM>, the logic (e.g., compositor <NUM>) may predict a pose that the HMD <NUM> will be in at the time that was determined at sub-block <NUM> (i.e., the targeted render time <NUM>). The original prediction of the pose of the HMD <NUM> at sub-block <NUM> may be based at least in part on head tracking data generated by the head tracking system of the HMD <NUM> (or HMD system).

At sub-block <NUM>, the logic (e.g., compositor <NUM>) may send, to an application <NUM> for purposes of rendering the frame <NUM>, pose data indicative of the pose predicted at sub-block <NUM>.

At sub-block <NUM>, the logic (e.g., compositor <NUM>) may receive, from the application <NUM>, pixel data for the frame <NUM>. The pixel data may include pixel values, as described herein, for individual pixels in the array of pixels of the display panel <NUM>.

At sub-block <NUM>, the logic (e.g., compositor <NUM>) may apply re-projection adjustments to the pixel data for the frame <NUM> to obtain modified pixel data associated with a re-projected frame. In the diagram of <FIG>, this may occur during the compositor's <NUM> rendering workload <NUM> after receiving the pixel data for the frame <NUM> from the application <NUM>.

At sub-block <NUM>, the logic (e.g., compositor <NUM>) may output the modified pixel data to a frame buffer(s). Again, for a HMD <NUM> with a pair of display panels <NUM>, this pixel data may correspond to a frame that represents a pair of images to be displayed on the pair of display panels <NUM>.

At <NUM>, logic of the HMD <NUM> (or HMD system) (e.g., a display controller, display driver circuitry, etc.) may cause an image to be presented on the display panel <NUM> based on the frame <NUM> that was rendered at block <NUM> (e.g., based at least in part on the modified pixel data output to the frame buffer at sub-block <NUM>). As shown by the sub-blocks <NUM> and <NUM>, the presentation of the image at block <NUM> may include one or more sub-operations.

At sub-block <NUM>, for example, the modified pixel data may be scanned out in sequence for individual subsets of pixels on the display panel <NUM>. For example, the pixel values for individual rows of pixels may be sequentially scanned out to the display panel(s) <NUM> via a display port (e.g., a HDMI), starting with a first subset (e.g., row) of pixels and ending with a last subset (e.g., row) of pixels.

At sub-block <NUM>, the individual subsets of pixels may be illuminated in the sequence over the illumination time period <NUM> for the frame <NUM> in order to present an image on the display panel(s) <NUM>. This may be accomplished by driving the light emitting elements of the display panel(s) <NUM> using a rolling band display driving technique, where individual subsets of light emitting elements of the display panel(s) <NUM> are illuminated in sequence to cause corresponding subsets of pixels to illuminate in sequence during image presentation.

<FIG> illustrates a flow diagram of an example sub-process <NUM> for applying re-projection adjustments (e.g., a sub-process of block <NUM> of the process <NUM>), in accordance with embodiments disclosed herein. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM> - after the compositor <NUM> has received, from the application <NUM>, pixel data for the frame <NUM> - logic of the HMD <NUM> (or HMD system) (e.g., logic including the compositor <NUM>) may predict, based at least in part on updated (second) head tracking data generated by the head tracking system of the HMD <NUM> (or HMD system), a plurality of updated poses of the HMD <NUM> for a plurality of times, within the illumination time period <NUM> for the frame <NUM>, at which the individual subsets of the pixels will be illuminated. In the example where the individual subsets of the pixels are individual rows of pixels, the plurality of updated poses that are predicted at block <NUM> may include a first updated pose of the HMD <NUM> for a time at which the first row of pixels ("Row <NUM>") will be illuminated, a second updated pose of the HMD <NUM> for a time at which the second row of pixels ("Row <NUM>") will be illuminated, and so on, for N rows of pixels, N being any suitable number.

At <NUM>, the logic (e.g., compositor <NUM>) may calculate (or otherwise determine) a plurality of adjustments (e.g., re-projection adjustments) based at least in part on comparisons between the originally-predicted pose and individual ones of the updated poses. That is, individual ones of the updated poses predicted at block <NUM> may be compared to the original pose that was predicted at block <NUM> of the process <NUM>, and which was sent to the application <NUM> as input for rendering the frame <NUM>. The comparison between poses at block <NUM> may determine a transform that can be applied to the originally-predicted pose to obtain the individual updated pose in question. For example, one of the adjustments determined at block <NUM> may, for example, correspond to a transform that can be applied to the originally-predicted pose to obtain the updated pose of the HMD <NUM> for the first row of pixels. Additional transforms can be determined in a similar manner for other subsets of pixels.

At <NUM>, the logic (e.g., compositor <NUM>) may modify the pixel data for the frame <NUM> (which was received from the application <NUM>), based at least in part on the plurality of adjustments calculated at block <NUM>, to obtain the modified pixel data associated with the re-projected frame. Referring briefly again to <FIG>, the graph <NUM> illustrates an example of how the adjustments calculated at block <NUM> can vary across the display panel(s) <NUM> due to the rolling band display driving technique that illuminates the individual subsets of pixels at different times, in sequence, and due to the fact that the user's <NUM> head may be constantly moving in space. The adjustments for subsets of pixels that are adjacent to the subset of pixels associated with the targeted render time <NUM> are small-scale adjustments, while other adjustments for subsets of pixels that are not adjacent to the subset of pixels associated with the targeted render time <NUM> are larger-scale adjustments; just how large these larger-scale adjustments are, as compared to the small-scale adjustments of adjacent pixel subsets, depends on the amount (or rate) of head rotation that is occurring while the frame <NUM> is rendered and a corresponding image is presented. In any case, a first adjustment, of the plurality of adjustments, for a first individual subset of the pixels that is adjacent to the subset of pixels associated with the targeted render time <NUM> is likely to be less than a second adjustment, of the plurality of adjustments, for a second individual subset of the pixels that is not adjacent to the subset of pixels associated with the target render time <NUM>, as depicted in the graph <NUM> of <FIG>. This means that the image presented on the display panel(s) <NUM> is free from unwanted artifacts (e.g., judder) caused by re-projection at the location on the display panel(s) <NUM> where the user <NUM> is looking.

<FIG> illustrates example components of a HMD <NUM> (or a HMD system that includes the HMD <NUM>), such as a VR headset, according to the embodiments disclosed herein may be embedded. The HMD <NUM> may be the same as, or similar to, the HMD <NUM> referenced in the previous figures, and, therefore, the components of the HMD <NUM> shown in <FIG> may be implemented in the HMD <NUM>. The HMD <NUM> may be implemented as a standalone device that is to be worn by a user <NUM> (e.g., on a head of the user <NUM>). In some embodiments, the HMD <NUM> may be head-mountable, such as by allowing a user <NUM> to secure the HMD <NUM> on his/her head using a securing mechanism (e.g., an adjustable band) that is sized to fit around a head of a user <NUM>. In some embodiments, the HMD <NUM> comprises a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display(s). As such, the terms "wearable device", "wearable electronic device", "VR headset", "AR headset", and "head-mounted display (HMD)" may be used interchangeably herein to refer to the device <NUM> of <FIG>. However, it is to be appreciated that these types of devices are merely example of a HMD <NUM>, and it is to be appreciated that the HMD <NUM> may be implemented in a variety of other form factors. It is also to be appreciated that some or all of the components shown in <FIG> may be implemented on the HMD <NUM>. Accordingly, in some embodiments, a subset of the components shown in <FIG> may be implemented on a computing device that is part of the HMD system, but is separate from the HMD <NUM> itself, such as a PC, a game console, or any other suitable computing device.

In the illustrated implementation, the HMD <NUM> includes one or more processors <NUM> and memory <NUM> (e.g., computer-readable media <NUM>). In some implementations, the processors(s) <NUM> may include a central processing unit (CPU), a graphics processing unit (GPU)(s) <NUM>, both CPU and GPU <NUM>, a microprocessor, a digital signal processor or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) <NUM> may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems.

The memory <NUM> may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. The memory <NUM> may be implemented as computer-readable storage media ("CRSM"), which may be any available physical media accessible by the processor(s) <NUM> to execute instructions stored on the memory <NUM>. In one basic implementation, CRSM may include random access memory ("RAM") and Flash memory. In other implementations, CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s) <NUM>.

In general, the HMD <NUM> may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein. The computer-readable media <NUM> is shown as including various modules, such as instruction, datastores, and so forth, which may be configured to execute on the processor(s) <NUM> for carrying out the techniques, functionality, and/or operations described herein. A few example functional modules are shown as stored in the computer-readable media <NUM> and executable on the processor(s) <NUM>, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC), and/or other logic.

An operating system module <NUM> may be configured to manage hardware within and coupled to the HMD <NUM> for the benefit of other modules. In addition, in some instances the HMD <NUM> may include one or more applications <NUM> stored in the memory <NUM> or otherwise accessible to the HMD <NUM>. In this implementation, the application(s) <NUM> includes a gaming application <NUM>. However, the HMD <NUM> may include any number or type of applications and is not limited to the specific example shown here. The gaming application <NUM> may be configured to initiate gameplay of a video-based, interactive game (e.g., a VR game) that is playable by the user <NUM>, and to output frames <NUM> to be rendered on the display panels of the HMD <NUM>. A compositor <NUM>, in combination with other logic of the HMD <NUM>, may be configured to perform the techniques described herein to dynamically target a render time for an application (e.g., the application <NUM>) based on eye tracking.

Generally, the HMD <NUM> has input devices <NUM> and output devices <NUM>. The input devices <NUM> may include control buttons. In some implementations, one or more microphones may function as input devices <NUM> to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement unit (IMU)) may function as input devices <NUM> to receive gestural input, such as a hand and/or head motion of the user <NUM>. In some embodiments, additional input devices <NUM> may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, and the like. In other embodiments, the HMD <NUM> may omit a keyboard, keypad, or other similar forms of mechanical input. Instead, the HMD <NUM> may be implemented relatively simplistic forms of input device <NUM>, a network interface (wireless or wire-based), power, and processing/memory capabilities. For example, a limited set of one or more input components may be employed (e.g., a dedicated button to initiate a configuration, power on/off, etc.) so that the HMD <NUM> can thereafter be used. In one implementation, the input device(s) <NUM> may include control mechanisms, such as basic volume control button(s) for increasing/decreasing volume, as well as power and reset buttons.

The output devices <NUM> may include a display(s) <NUM>, which may include one or multiple display panels <NUM> (e.g., a stereo pair of display panels <NUM>), as described herein. The output devices <NUM> may further include, without limitation, a light element (e.g., LED), a vibrator to create haptic sensations, a speaker(s) (e.g., headphones), and/or the like. There may also be a simple light element (e.g., LED) to indicate a state such as, for example, when power is on.

The HMD <NUM> may further include a wireless unit <NUM> coupled to an antenna <NUM> to facilitate a wireless connection to a network. The wireless unit <NUM> may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that the HMD <NUM> may further include physical ports to facilitate a wired connection to a network, a connected peripheral device (including a PC, game console, etc.), or a plug-in network device that communicates with other wireless networks, and which may be part of the HMD system.

The HMD <NUM> may further include optical subsystem <NUM> that directs light from the electronic display(s) <NUM> to a user's eye(s) using one or more optical elements. The optical subsystem <NUM> may include various types and combinations of different optical elements, including, without limitations, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and so forth. In some embodiments, one or more optical elements in optical subsystem <NUM> may have one or more coatings, such as anti-reflective coatings. Magnification of the image light by optical subsystem <NUM> allows electronic display(s) <NUM> to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification of the image light may increase a field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., <NUM>-<NUM> degrees diagonal), and in some cases all, of the user's FOV. AR applications may have a narrower FOV (e.g., about <NUM> degrees FOV). Optical subsystem <NUM> may be designed to correct one or more optical errors, such as, without limitation, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and so forth. In some embodiments, content provided to electronic display(s) <NUM> for display is pre-distorted, and optical subsystem <NUM> corrects the distortion when it receives image light from electronic display(s) <NUM> generated based on the content.

The HMD <NUM> may further include one or more sensors <NUM>, such as sensors used to generate motion, position, and orientation data. These sensors <NUM> may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors. The sensors <NUM> may also include sub-portions of sensors, such as a series of active or passive markers that may be viewed externally by a camera or color sensor in order to generate motion, position, and orientation data. For example, a VR headset may include, on its exterior, multiple markers, such as reflectors or lights (e.g., infrared or visible light) that, when viewed by an external camera or illuminated by a light (e.g., infrared or visible light), may provide one or more points of reference for interpretation by software in order to generate motion, position, and orientation data. The HMD <NUM> may include light sensors that are sensitive to light (e.g., infrared or visible light) that is projected or broadcast by base stations in the environment of the HMD <NUM>.

In an example, the sensor(s) <NUM> may include an inertial measurement unit (IMU) <NUM>. IMU <NUM> may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and/or other sensors suitable for detecting motion, correcting error associated with IMU <NUM>, or some combination thereof. Based on the measurement signals such motion-based sensors, such as the IMU <NUM>, may generate calibration data indicating an estimated position of HMD <NUM> relative to an initial position of HMD <NUM>. For example, multiple accelerometers may measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). IMU <NUM> can, for example, rapidly sample the measurement signals and calculate the estimated position of HMD <NUM> from the sampled data. For example, IMU <NUM> may integrate measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on HMD <NUM>. The reference point is a point that may be used to describe the position of the HMD <NUM>. While the reference point may generally be defined as a point in space, in various embodiments, reference point is defined as a point within HMD <NUM> (e.g., a center of the IMU <NUM>). Alternatively, IMU <NUM> provides the sampled measurement signals to an external console (or other computing device), which determines the calibration data.

The sensors <NUM> may operate at relatively high frequencies in order to provide sensor data at a high rate. For example, sensor data may be generated at a rate of <NUM> (or <NUM> sensor reading every <NUM> millisecond). In this way, one thousand readings are taken per second. When sensors generate this much data at this rate (or at a greater rate), the data set used for predicting motion is quite large, even over relatively short time periods on the order of the tens of milliseconds.

As mentioned, in some embodiments, the sensors <NUM> may include light sensors that are sensitive to light emitted by base stations in the environment of the HMD <NUM> for purposes of tracking position and/or orientation, pose, etc., of the HMD <NUM> in 3D space. The calculation of position and/or orientation may be based on timing characteristics of light pulses and the presence or absence of light detected by the sensors <NUM>.

The HMD <NUM> may further include an eye tracking system <NUM>, as described elsewhere herein as the system that generates eye tracking data. The eye tracking system <NUM> may include, without limitation, a camera or other optical sensor inside HMD <NUM> to capture image data (or information) of a user's eyes <NUM>, and the eye tracking system <NUM> may use the captured data/information to determine motion vectors, interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye <NUM> relative to HMD <NUM>, including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye <NUM>. In one example, infrared light is emitted within HMD <NUM> and reflected from each eye <NUM>. The reflected light is received or detected by a camera of the eye tracking system <NUM> and analyzed to extract eye rotation from changes in the infrared light reflected by each eye <NUM>. Many methods for tracking the eyes <NUM> of a user <NUM> can be used by eye tracking system <NUM>. Accordingly, eye tracking system <NUM> may track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes <NUM> of a user <NUM> to estimate a gaze point (i.e., a 3D location or position in the virtual scene where the user is looking), which may map to a location(s) on the display panel(s) <NUM> for predicting where the user <NUM> will be looking in terms of an individual subset (e.g., a row) or a group of contiguous subsets (e.g., a group of contiguous rows) of the pixels of the display panel(s) <NUM>. For example, eye tracking system <NUM> may integrate information from past measurements, measurements identifying a position of a user's <NUM> head, and 3D information describing a scene presented by electronic display(s) <NUM>. Thus, information for the position and orientation of the user's <NUM> eyes <NUM> is used to determine the gaze point in a virtual scene presented by HMD <NUM> where the user <NUM> is looking, and to map that gaze point to a location(s) on the display panel(s) <NUM> of the HMD <NUM>. As mentioned earlier, a most recent estimate/prediction of a gaze point(s) or a location(s) on the display panel(s) <NUM> where the user <NUM> is looking may be used as a proxy for predicting where the user <NUM> will be looking during a forthcoming illumination time period <NUM> for a given frame <NUM>. However, other techniques for predicting where the user <NUM> will be looking in the future may be utilized, such as utilizing motion vectors to predict a trajectory of eye movement, and the like.

The HMD <NUM> may further include a head tracking system <NUM>. The head tracking system <NUM> may leverage one or more of the sensor <NUM> to track head motion, including head rotation, of the user <NUM>, as described above. For example, the head tracking system <NUM> can track up to six degrees of freedom of the HMD <NUM> (i.e., 3D position, roll, pitch, and yaw). These calculations can be made at every frame <NUM> of a series of frames <NUM> so that the application <NUM> can determine how to render a scene in the next frame <NUM> in accordance with the head position and orientation. In some embodiments, the head tracking system <NUM> and/or the compositor <NUM> using the head tracking system <NUM> is configured to predict a future pose (position and/or orientation) of the HMD <NUM> based on current and/or past data, and/or based on the known/implied scan out latency of the individual subsets of pixels in a display system that implements a rolling band display driving technique, as described herein. This is because the application is asked to render a frame <NUM> before the user <NUM> actually sees the light (and, hence, the image) on the display(s) <NUM>. Accordingly, a next frame <NUM> can be rendered based on this future prediction of head position and/or orientation that was made at an earlier point in time, which may be roughly within the range of <NUM>-<NUM> milliseconds (ms) prior to rendering the frame <NUM>, depending on the target render time <NUM> determined for the given frame <NUM>. Rotation data provided by the head tracking system <NUM> can be used to determine both direction of HMD <NUM> rotation, and amount of HMD <NUM> rotation in any suitable unit of measurement. For example, rotational direction may be simplified and output in terms of positive or negative horizontal and positive or negative vertical directions, which correspond to left, right, up, and down. Amount of rotation may be in terms of degrees, radians, etc. Angular velocity may be calculated to determine a rate of rotation of the HMD <NUM>.

Claim 1:
A head-mounted display, HMD, system comprising:
a HMD (<NUM>; <NUM>) including a display panel (<NUM>) having an array of pixels;
an eye tracking system (<NUM>);
a head tracking system (<NUM>);
a processor (<NUM>); and
memory (<NUM>) storing computer-executable instructions that, when executed by the processor, cause the HMD system to:
predict, based at least in part on eye tracking data generated by the eye tracking system, a location on the display panel where a user (<NUM>) wearing the HMD will be looking during an illumination time period, wherein individual subsets of the pixels are illuminated in sequence over the illumination time period to present an image on the display panel;
determine a time, within the illumination time period, at which an individual subset of the pixels that corresponds to the location on the display panel will be illuminated;
predict, based at least in part on head tracking data generated by the head tracking system, a pose that the HMD will be in at the time;
send pose data indicative of the pose to an application for rendering a frame (<NUM>) corresponding to the image;
receive, from the application, pixel data for the frame;
apply re-projection adjustments to the pixel data for the frame to obtain modified pixel data associated with a re-projected frame; and
present the image on the display panel based at least in part on the modified pixel data, wherein the individual subsets of the pixels are illuminated in the sequence over the illumination time period during presentation of the image on the display panel.