Patent Description:
On the opposite end the spectrum, all-in-one (or standalone) VR headsets perform the entirety of the graphics processing operations to display imagery, without the aid of a separate machine. While standalone VR headsets provide a user with greater mobility because they do not have to be tethered to a host computer, manufacturing an all-in-one VR headset that is both comfortable and capable of rendering high-quality graphics can be challenging. For example, standalone VR headsets that are tasked with performing computationally-intensive, high-power-consuming graphics-processing operations to render high-quality graphics tend to get hot very quickly, and they also tend to be cumbersome and/or heavy, making them uncomfortable to wear for long periods of time. To alleviate these drawbacks, some standalone VR headsets trade quality for comfort by using lower-quality graphics processing components that render graphics at lower resolution, lower dynamic range, and/or with a limited set of only basic textures, which makes the graphics processing operations onboard the headset less computationally-intensive, allowing for a lighter-weight headset that does not get too hot and is therefore more comfortable to wear. However, users who wish to experience high quality graphics in VR are left dissatisfied with today's standalone VR headsets, which are unable to provide both quality and comfort.

Provided herein are technical solutions to improve and enhance these and other systems.

<CIT> discloses an apparatus comprising: a compute cluster comprising global illumination circuitry and/or logic to perform global illumination operations on graphics data in response to execution of a virtual reality application and to responsively generate a stream of samples; a filtering/compression module to perform filtering and/or compression operations on the stream of samples to generate filtered/compressed samples; a network interface to communicatively couple the compute cluster to a network, the filtered/compressed samples to be streamed over the network; a render node to receive the filtered/compressed samples streamed over the network, the render node comprising: decompression circuitry/logic to decompress the filtered/compressed samples to generate decompressed samples; a sample buffer to store the decompressed samples; and sample insertion circuitry/logic to asynchronously insert samples into a light field rendered by a light field rendering circuit/logic.

<CIT> discloses a system for reducing bandwidth and/or reducing power consumed by a display that comprises a display having a background plane and a region of interest plane that is identified by a gaze tracker.

"<NPL>, discusses virtual hand representation when interacting in virtual environments.

The detailed description is described with reference to the accompanying drawings.

A head-mounted display (HMD) may be worn by a user for purposes of immersing the user in a virtual reality (VR) environment or an augmented reality (AR) environment. One or more display panels of the HMD present images based on data generated by an application (e.g., a video game). The application executes on a host computer that is communicatively coupled to the HMD, and the application generates pixel data for individual frames of a series of frames. The pixel data is sent to the HMD to present images that are viewed by a user through the optics included in the HMD, making the user perceive the images as if the user was immersed in a VR or AR environment.

Described herein are, among other things, techniques and systems for splitting a rendering workload for an individual frame between the HMD and the host computer such that the host computer performs a first portion of the rendering workload and the HMD performs a second portion of the rendering workload. For a given frame, the HMD is configured to send head tracking data to the host computer, and the host computer is configured to use the head tracking data to generate the pixel data for the frame and extra data in addition to the pixel data. The extra data can include, without limitation, pose data, depth data, motion vector data, parallax occlusion data, and/or extra pixel data. For example, the host computer may use the head tracking data to generate pose data indicative of a predicted pose that the HMD will be in at a time at which light emitting elements of the display panel(s) of the HMD will illuminate for the frame. The host computer may additionally, or alternatively, instruct the application to generate depth data and/or extra pixel data based at least in part on the pose data. The host computer may also generate motion vector data based at least in part on the head tracking data and/or movement within the scene being rendered. Some or all of this extra data may be sent from the host computer to the HMD, and the HMD may use at least some of the extra data it receives for purposes of modifying the pixel data, such as by applying re-projection adjustments to the pixel data. "Re-projection" is a technique used to compensate for slight inaccuracies in an original pose prediction of the HMD and/or to compensate for the application failing to make frame rate, which has the same effect as an original pose prediction that is slightly inaccurate. For example, a re-projected frame can be generated using pixel data from an application-rendered frame by transforming (e.g., through rotation and re-projection calculations) the application-rendered frame in a way that accounts for an updated prediction of the pose of the HMD. Accordingly, the modified pixel data obtained from applying the re-projection adjustments (and possibly other adjustments) may be used to present an image(s) on the display panel(s) of the HMD for the given frame, and this process may iterate for a series of frames.

In some embodiments, the extra data - besides the pixel data - that is generated, sent, and/or utilized for rendering frames may vary frame-to-frame. For example, the host computer may dynamically determine, for individual frames, the type and/or extent of extra data that is to be generated as part of the first portion of the rendering workload and/or the type and/or extent of the extra data that is to be sent to the HMD. Meanwhile, the HMD may dynamically determine, for individual frames, the type and/or extent of the extra data received from the host computer to utilize as part of the second portion of the rendering workload.

Splitting the rendering workload for a given frame between a host computer and a HMD allows for implementing a system where the host computer and the HMD can be wirelessly connected to each other; something that is currently impracticable with today's high-latency wireless communication protocols and HMDs that are fully-reliant on a host computer. Splitting the rendering workload, in turn, allows for providing a high-quality VR or AR experience on a HMD that is also comfortable to wear for long periods of time because the high-computing capacity of the host computer can still be leveraged in the system disclosed herein. Furthermore, the HMD disclosed herein can be, and can remain, physically untethered from the host computer, providing a user with greater mobility, as compared to a tethered HMD, in that the user is better able to walk around a space while wearing the HMD, without concern for accidentally unplugging the HMD or the like. Given user demand for high-fidelity, high-resolution VR graphics, a wireless VR system that adheres to these demands will tend to be subjected to higher latencies in data transfer over a wireless communication link due to the greater amount of data that is transferred wirelessly. This means that a pose prediction of the HMD used by the application to render a given frame is made farther in advance in the system disclosed herein, as compared to the pose prediction for a conventional physically-tethered HMD that can avail itself to the higher data transfer rate of a wired connection. A pose prediction that is made farther in advance of the illumination time for the frame means there is more error in the pose prediction, as compared to the later-in-time pose prediction for a physically-tethered HMD, which, in turn, means that the HMD disclosed herein is tasked with performing computationally-intensive graphics processing operations in order to modify the pixel data received from the host computer (e.g., to correct for errors in the pixel data received from the host computer) so that a suitable image(s) is displayed on the HMD. In general, the HMD, armed with extra data received from the host computer, is in a better position to account for a relatively-lower data transfer rate over the wireless communication link between the host computer and the HMD in order to modify the received pixel data in a way that improves the quality of the resulting image(s) presented on the display panel(s) of the HMD. In addition, the split rendering techniques and systems described herein can allow for a different rendering frequency (or frame rate) on each of the host computer and the HMD.

Accordingly, the disclosed HMD is configured to perform a portion of the rendering workload for a given frame, which allows data to be transferred wirelessly between the host computer and the HMD notwithstanding the relatively higher latency of the wireless connection, as compared to the relatively low-latency wired connection of today's HMDs. The HMD can compensate for the higher latency of the wireless communication link using graphics-processing logic onboard the HMD that is used to correct for errors in the data generated by the host computer. In addition, this onboard graphics-processing logic allows the HMD to be used as a standalone device, perhaps in limited use scenarios. For example, the HMD disclosed herein can be used in standalone mode to play video games that render more basic graphics in their imagery, thereby requiring less computationally-intensive graphics processing operations to render frames. As another example, the HMD disclosed herein can be used in standalone mode to playback movies and/or video clips on the HMD, all without relying on the host computer. When a user of the HMD disclosed herein wishes to play a video game with richer graphics, however, the user may operate the HMD in connected mode to leverage the additional graphics processing capacity of the host computer by connecting the HMD thereto, either over a wired or wireless communication link. A wired communication link may still be utilized by users who wish to play video games with richer graphics for long periods of time by leveraging the additional power capacity of the host computer (e.g., so the HMD does not run out of battery power). As compared to today's all-in-one systems, for example, a user can benefit from a high-fidelity graphics experience that is provided by a connected host computer along with the increased mobility that is enabled by virtue of an available wireless connection between the host computer and the HMD.

Also disclosed herein are non-transitory computer-readable media storing computer-executable instructions to implement the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed, by way of example, in the context of video game applications, and specifically VR gaming applications, it is to be appreciated that the techniques and systems described herein may provide benefits with other applications, including, without limitation, non-VR applications (e.g., AR applications), and/or non-gaming applications, such as industrial machine applications, defense applications, robotics applications, and the like.

<FIG> is a diagram illustrating an example technique for splitting a rendering workload <NUM> for a frame between a head-mounted display (HMD) and a host computer. <FIG> depicts a head-mounted display (HMD) <NUM> worn by a user <NUM>, as well as a host computer(s) <NUM>. <FIG> depicts example implementations of a host computer <NUM> in the form of a laptop <NUM>(<NUM>) carried in a backpack, for example, or a personal computer (PC) <NUM>(N), which may be situated in the user's <NUM> household, for example. It is to be appreciated, however, that these exemplary types of host computers <NUM> are non-limiting to the present disclosure. For example, the host computer <NUM> can be implemented as any type and/or any number of computing devices, including, without limitation, a PC, a laptop computer, a desktop computer, a portable digital assistant (PDA), a mobile phone, tablet computer, a set-top box, a game console, a server computer, a wearable computer (e.g., a smart watch, etc.), or any other electronic device that can transmit/receive data. The host computer <NUM> may be collocated in the same environment as the HMD <NUM>, such as a household of the user <NUM> wearing the HMD <NUM>. Alternatively, the host computer <NUM> may be remotely located with respect to the HMD <NUM>, such as a host computer <NUM> in the form of a server computer that is located in a remote geographical location with respect to the geographical location of the HMD <NUM>. In a remote host computer <NUM> implementation, the host computer <NUM> may be communicatively coupled to the HMD <NUM> via a wide-area network, such as the Internet. In a local host computer <NUM> implementation, the host computer <NUM> may be collocated in an environment (e.g., a household) with the HMD <NUM>, whereby the host computer <NUM> and the HMD <NUM> may be communicatively coupled together either directly or over a local area network (LAN) via intermediary network devices.

As shown in <FIG>, for a given frame, the host computer <NUM> is configured to perform a first partial rendering workload <NUM>(<NUM>) (e.g., a first portion of the rendering workload <NUM> for a given frame), and the HMD <NUM> is configured to perform a second partial rendering workload <NUM>(<NUM>) (e.g., a second portion of the rendering workload <NUM> for the given frame). In this manner, the HMD <NUM> and the host computer <NUM> are communicatively coupled together and are configured to work together in a collaborative fashion to render a given frame by generating pixel data that is ultimately used to present a corresponding image(s) on a display panel(s) <NUM> of the HMD <NUM>.

The HMD <NUM> in the example of <FIG> may include a single display panel <NUM> or multiple display panels <NUM>, such as a left display panel and a right display panel of a stereo pair of display panels. The one or more display panels <NUM> of the HMD <NUM> may be used to present a series of image frames (herein referred to as "frames") that are viewable by the user <NUM> wearing the HMD <NUM>. It is to be appreciated that the HMD <NUM> may include any number of display panels <NUM> (e.g., more than two display panels, a pair of display panels, or a single display panel). Hence, the terminology "display panel," as used in the singular herein, may refer to either display panel <NUM> of a pair of display panels of a two-panel HMD <NUM>, or it may refer to a single display panel <NUM> of a HMD <NUM> with any number of display panels (e.g., a single-panel HMD <NUM> or a multi-panel HMD <NUM>). In a two-panel HMD <NUM>, a stereo frame buffer may render, for instance, <NUM> x <NUM> pixels on both display panels of the HMD <NUM> (e.g., <NUM> x <NUM> pixels per display panel).

The display panel(s) <NUM> of the HMD <NUM> may utilize any suitable type of display technology, such as an emissive display that utilizes light emitting elements (e.g., light emitting diodes (LEDs)) to emit light during presentation of frames on the display panel(s) <NUM>. As an example, display panel(s) <NUM> of the HMD <NUM> may comprise liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, inorganic light emitting diode (ILED) displays, or any other suitable type of display technology for HMD applications.

The display panel(s) <NUM> of the HMD <NUM> may operate at any suitable refresh rate, such as a <NUM> Hertz (Hz) refresh rate, which can be a fixed refresh rate or a variable refresh rate that dynamically varies over a range of refresh rates. The "refresh rate" of a display is the number of times per second the display redraws the screen. The number of frames displayed per second may be limited by the refresh rate of the display, if using a fixed refresh rate. Thus, a series of frames may be processed (e.g., rendered) and displayed as images on the display such that a single frame of the series of frames is displayed with every screen refresh. That is, in order to present a series of images on the display panel(s) <NUM>, the display panel(s) <NUM> may transition from frame-to-frame, in the series of frames, at the refresh rate of the display, illuminating the pixels at every screen refresh. In some embodiments, the frame rate can be throttled and/or the application can fail to hit the target frame rate, and phantom frames (based on re-projection) can be inserted between application-rendered frames.

The display system of the HMD <NUM> may implement any suitable type of display driving scheme, such as a global flashing type of display driving scheme, a rolling band type of display driving scheme, or any other suitable type of display driving scheme. In a global flashing type of display driving scheme, the array of light emitting elements of the display illuminate simultaneously at every screen refresh, thereby flashing globally at the refresh rate. In a rolling band type of display driving scheme, individual subsets of the light emitting elements of the display can be illuminated independently and sequentially in a rolling band of illumination during an illumination time period. These types of display driving schemes may be enabled by the light emitting elements being individually addressable. If the array of pixels and the array of light emitting elements on the display panel(s) <NUM> are arranged in rows and columns (but not necessarily with a one-pixel per one-light emitting element correspondence), individual rows and/or individual columns of light emitting elements may be addressed in sequence, and/or individual groups of contiguous rows and/or individual groups of contiguous columns of light emitting elements may be addressed in sequence for a rolling band type of display driving scheme.

In general, as used herein, "illuminating a pixel" means illuminating the light emitting element that corresponds to that pixel. For example, a LCD illuminates a light emitting element of a backlight to illuminate the corresponding pixel(s) of the display. Furthermore, as used herein, a "subset of pixels" may comprise an individual pixel or multiple pixels (e.g., a group of pixels). In order to drive the display panel(s) <NUM>, the HMD <NUM> may include, among other things, a display controller(s), display driver circuitry, and similar electronics for driving the display panel(s) <NUM>. Display driver circuitry may be coupled to the array of light emitting elements of the display panel(s) <NUM> via conductive paths, such as metal traces, on a flexible printed circuit. In an example, a display controller(s) may be communicatively coupled to the display driver circuitry and configured to provide signals, information, and/or data to the display driver circuitry. The signals, information, and/or data received by the display driver circuitry may cause the display driver circuitry to illuminate the light emitting elements in a particular way. That is, the display controller(s) may determine which light emitting element(s) is/are to be illuminated, when the element(s) is/are to illuminate, and the level of light output that is to be emitted by the light emitting element(s), and may communicate the appropriate signals, information, and/or data to the display driver circuitry in order to accomplish that objective.

In the illustrated implementation, the HMD <NUM> includes one or more processors <NUM> and memory <NUM> (e.g., computer-readable media <NUM>). In some implementations, the processors(s) <NUM> may include a central processing unit (CPU)(s), a graphics processing unit (GPU)(s) <NUM>, both CPU(s) and GPU(s) <NUM>, a microprocessor, a digital signal processor or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) <NUM> may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems.

The memory <NUM> may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. The memory <NUM> may be implemented as computer-readable storage media ("CRSM"), which may be any available physical media accessible by the processor(s) <NUM> to execute instructions stored on the memory <NUM>. In one basic implementation, CRSM may include random access memory ("RAM") and Flash memory. In other implementations, CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s) <NUM>.

In general, the HMD <NUM> may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein. The computer-readable media <NUM> can include various modules, such as instruction, datastores, and so forth, which may be configured to execute on the processor(s) <NUM> for carrying out the techniques, functionality, and/or operations described herein. An example functional module in the form of a compositor <NUM> is shown as being stored in the computer-readable media <NUM> and executable on the processor(s) <NUM>, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC), and/or other logic. Furthermore, additional or different functional modules may be stored in the computer-readable media <NUM> and executable on the processor(s) <NUM>. The compositor <NUM> is configured to modify pixel data received from the host computer <NUM> as part of the second partial rendering workload <NUM>(<NUM>), and to output the modified pixel data to a frame buffer (e.g., a stereo frame buffer) so that a corresponding image(s) can be presented on the display panel(s) <NUM> of the HMD <NUM>.

The HMD <NUM> may further include a head tracking system <NUM> and a communications interface(s) <NUM>. The head tracking system <NUM> may leverage one or more sensors (e.g., infrared (IR) light sensors mounted on the HMD <NUM>) and one or more tracking beacon(s) (e.g., IR light emitters collocated in the environment with the HMD <NUM>) to track head motion or movement, including head rotation, of the user <NUM>. This example head tracking system <NUM> is non-limiting, and other types of head tracking systems <NUM> (e.g., camera-based, inertial measurement unit (IMU)-based, etc.) can be utilized. The head tracking system <NUM> is configured to generate head tracking data <NUM>, which can be sent, via the communications interface(s) <NUM> to the host computer <NUM> during runtime, as frames are being rendered.

The communications interface(s) <NUM> of the HMD <NUM> may include wired and/or wireless components (e.g., chips, ports, etc.) to facilitate wired and/or wireless data transmission/reception to/from the host computer <NUM>, either directly or via one or more intermediate devices, such as a wireless access point (WAP). For example, the communications interface(s) <NUM> may include a wireless unit coupled to an antenna to facilitate a wireless connection with the host computer <NUM> and/or another device(s). Such a wireless unit may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. The communications interface(s) <NUM> may further include one or more physical ports to facilitate a wired connection with the host computer <NUM> and/or another device(s) (e.g., a plug-in network device that communicates with other wireless networks).

In the illustrated implementation, the host computer <NUM> includes one or more processors <NUM> and memory <NUM> (e.g., computer-readable media <NUM>). In some implementations, the processors(s) <NUM> may include a CPU(s), a GPU(s) <NUM>, both CPU(s) and GPU(s) <NUM>, a microprocessor, a digital signal processor or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include FPGAs, ASICs, ASSPs, SOCs, CPLDs, etc. Additionally, each of the processor(s) <NUM> may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems.

The memory <NUM> may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. The memory <NUM> may be implemented as CRSM, which may be any available physical media accessible by the processor(s) <NUM> to execute instructions stored on the memory <NUM>. In one basic implementation, CRSM may include RAM and Flash memory. In other implementations, CRSM may include, but is not limited to, ROM, EEPROM, or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s) <NUM>.

In general, the host computer <NUM> may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein. The computer-readable media <NUM> can include various modules, such as instruction, datastores, and so forth, which may be configured to execute on the processor(s) <NUM> for carrying out the techniques, functionality, and/or operations described herein. Example functional modules in the form of applications <NUM>, such as a video game <NUM>(<NUM>), and a render component <NUM> are shown as being stored in the computer-readable media <NUM> and executable on the processor(s) <NUM>. In some embodiments, the functionality of the render component <NUM> may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC), and/or other logic. Furthermore, additional or different functional modules may be stored in the computer-readable media <NUM> and executable on the processor(s) <NUM>.

The host computer <NUM> may further include a communications interface(s) <NUM>, which may include wired and/or wireless components (e.g., chips, ports, etc.) to facilitate wired and/or wireless data transmission/reception to/from the HMD <NUM>, either directly or via one or more intermediate devices, such as a WAP. For example, the communications interface(s) <NUM> may include a wireless unit coupled to an antenna to facilitate a wireless connection with the HMD <NUM> and/or another device(s). Such a wireless unit may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, RF, and so on. The communications interface(s) <NUM> may further include one or more physical ports to facilitate a wired connection with the HMD <NUM> and/or another device(s) (e.g., a plug-in network device that communicates with other wireless networks).

It is to be appreciated that the HMD <NUM> may represent a VR headset for use in VR systems, such as for use with a VR gaming system, in which case the video game <NUM>(<NUM>) may represent a VR video game <NUM>(<NUM>). However, the HMD <NUM> may additionally, or alternatively, be implemented as an AR headset for use in AR applications, or a headset that is usable for VR and/or AR applications that are not game-related (e.g., industrial applications). In AR, a user <NUM> sees virtual objects overlaid on a real-world environment, whereas, in VR, the user <NUM> does not typically see a real-world environment, but is fully immersed in a virtual environment, as perceived via the display panel(s) <NUM> and the optics (e.g., lenses) of the HMD <NUM>. It is to be appreciated that, in some VR systems, pass-through imagery of the real-world environment of the user <NUM> may be displayed in conjunction with virtual imagery to create an augmented VR environment in a VR system, whereby the VR environment is augmented with real-world imagery (e.g., overlaid on a virtual world). Examples described herein pertain primarily to a VR-based HMD <NUM>, but it is to be appreciated that the HMD <NUM> is not limited to implementation in VR applications.

In general, the application(s) <NUM> executing on the host computer <NUM> can be a graphics-based application(s) <NUM> (e.g., a video game <NUM>(<NUM>)). An application <NUM> is configured to generate pixel data for a series of frames, and the pixel data is ultimately used to present corresponding images on the display panel(s) <NUM> of the HMD <NUM>. During runtime, for a given frame, the render component <NUM> may determine a predicted "illumination time" for the frame. This predicted "illumination time" for the frame represents a time at which light emitting elements of the display panel(s) <NUM> of the HMD <NUM> will illuminate for the frame. This prediction can account for, among other things, the inherent latency of a wireless communication link between the host computer <NUM> and the HMD <NUM>, as well as a predicted render time and/or a known scan-out time of the pixels from the frame buffer(s). In other words, the prediction may be different for a wireless communication link than it is for a wired communication link. For instance, the render component <NUM> may, for a wired communication link, predict an illumination time that is a first amount of time in the future (e.g., about <NUM> milliseconds in the future), whereas the render component <NUM> may, for a wireless communication link, predict an illumination time that is a second, greater amount of time in the future (e.g., about <NUM> milliseconds in the future), due to the inherent differences in latency when transferring data over a wired connection verses a wireless connection.

The host computer <NUM> may also receive, from the HMD <NUM>, the head tracking data <NUM> (e.g., first head tracking data <NUM>) generated by the head tracking system <NUM> of the HMD <NUM>. This head tracking data <NUM> may be generated and/or sent at any suitable frequency, such as a frequency corresponding to the target frame rate and/or the refresh rate of the HMD <NUM>, or a different (e.g., faster) frequency, such as <NUM> (or <NUM> sensor reading every <NUM> millisecond). The render component <NUM> is configured to determine a predicted pose that the HMD <NUM> will be in at the predicted illumination time based at least in part on the head tracking data <NUM>. The render component <NUM> may then provide pose data indicative of the predicted pose to the executing application <NUM> for rendering the frame (e.g., generating pixel data for the frame) based on the predicted pose, and the render component <NUM> may obtain, from the application <NUM>, pixel data <NUM> associated with the frame. This pixel data <NUM> may correspond to an array of pixels of the display panel(s) <NUM> of the HMD <NUM>. For example, the pixel data <NUM> output by the application <NUM> based on the pose data may include a two-dimensional array of per-pixel values (e.g., color values) for the array of pixels on the display panel(s) <NUM> of the HMD <NUM>. In an illustrative example, a stereo pair of display panels <NUM> may include an array of <NUM> x <NUM> pixels on both display panels of the HMD <NUM> (e.g., <NUM> x <NUM> pixels per display panel). In this illustrative example, the pixel data <NUM> may include <NUM> x <NUM> pixel values (or <NUM>,<NUM>,<NUM> pixel values). In some embodiments, the pixel data <NUM> may include data for each pixel that is represented by a single set of color and alpha values (e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values for one or more alpha channels).

Logic of the host computer <NUM> may also generate extra data <NUM> besides (or, in addition to) the pixel data <NUM>, and at least some of this extra data <NUM> may be sent to the HMD <NUM> to aid the HMD <NUM> in the second partial rendering workload <NUM>(<NUM>). For example, the extra data <NUM> can be packaged with the pixel data <NUM> and sent to the HMD <NUM>, and at least some of the extra data <NUM> can be used by logic of the HMD <NUM> to modify the pixel data <NUM> for purposes of presenting an image(s) corresponding to the frame on the display panel(s) <NUM> of the HMD <NUM>. The extra data <NUM> can include, without limitation, the pose data generated by the render component <NUM>, depth data, motion vector data, parallax occlusion data, and/or extra pixel data. For example, in providing the pose data to the executing application <NUM> for rendering the frame, the render component <NUM> may further instruct the application <NUM> to generate depth data (e.g., Z-buffer data) for the frame and/or extra pixel data (sometimes referred to herein as "out-of-bounds pixel data" or "additional pixel data"), and, in response, the render component <NUM> may obtain, from the application <NUM>, the depth data and/or the extra pixel data associated with the frame. Additionally, or alternatively, the render component <NUM> may generate motion vector data based at least in part on the head tracking data <NUM> received from the HMD <NUM>. For example, motion vector data can be generated based on a comparison of head tracking data generated at two different points in time (e.g., a comparison of head tracking data separated by a few milliseconds). Logic of the HMD <NUM> (e.g., the compositor <NUM>) can utilize some or all of the extra data <NUM> for purposes of modifying the pixel data <NUM> to correct for errors in the pose prediction made ahead of time by the render component <NUM>, which accounted for the inherent latency of the wireless connection between the host computer <NUM> and the HMD <NUM>. For example, the compositor <NUM> may apply re-projection adjustments based at least in part on the extra data <NUM> received from the host computer <NUM>. Other adjustments made by the compositor <NUM> as part of the second partial rendering workload <NUM>(<NUM>) may include, without limitation, adjustments for geometric distortion, chromatic aberration, re-projection, and the like. Ways in which the extra data <NUM> can be utilized as part of the second partial rendering workload <NUM>(<NUM>) are described in more detail below with reference to the following figures.

<FIG> is a diagram illustrating two example timelines <NUM>(<NUM>) and <NUM>(<NUM>) showing respective rendering workloads for individual frames, the respective rendering workloads being split between a host computer <NUM> and a HMD <NUM>, in accordance with embodiments disclosed herein. The example of <FIG> depicts three example frames - frame "F", frame "F+<NUM>", and frame "F+<NUM>" - with respect to the first timeline <NUM>(<NUM>) associated with the host computer <NUM>. This first timeline <NUM>(<NUM>) illustrates how the frames can be rendered in series by an executing application <NUM> on the host computer <NUM> using a GPU(s) <NUM> of the host computer <NUM>. Here, the application <NUM> renders frame F as part of first partial rendering workload <NUM>(<NUM>)(a), then frame F+<NUM> as part of a second partial rendering workload <NUM>(<NUM>)(b), and then frame F+<NUM> as part of a third partial rendering workload <NUM>(<NUM>)(c), in sequence, from left to right on the first timeline <NUM>(<NUM>). The ellipses on the first timeline <NUM>(<NUM>) indicate that this may continue for any number of frames as the application <NUM> continues to execute. The first timeline <NUM>(<NUM>) also implies, by the vertical lines oriented orthogonally to the horizontal timeline <NUM>(<NUM>), that the application <NUM> is targeting a target frame rate (e.g., a frame rate of <NUM> where the vertical lines would be separated by about <NUM> milliseconds). In the example of <FIG>, the application <NUM> executing on the host computer <NUM> happens to be hitting the target frame rate over the series of three example frames, but this may not always be the case, as the application <NUM> may, in some instances (e.g., for scenes with a high number of moving objects or complex textures), take longer than the allotted time to render a given frame <NUM>. This scenario is sometimes referred to as the application <NUM> failing to hit the target frame rate.

The second timeline <NUM>(<NUM>) in <FIG>, which is associated with the HMD <NUM>, shows the partial rendering workloads <NUM>(<NUM>)(a), <NUM>(<NUM>)(b), and <NUM>(<NUM>)(c) of the compositor <NUM> of the HMD <NUM> for the individual frames. An individual rendering workload <NUM>(<NUM>) of the HMD's <NUM> compositor <NUM> for a given frame may represent adjustments that are applied to the pixel data <NUM> generated by the application <NUM> executing on the host computer <NUM> before a final image(s) is presented on the display panel(s) <NUM> of the HMD <NUM>. Such adjustments may include, without limitation, adjustments for geometric distortion, chromatic aberration, re-projection, and the like, which are applied to the pixel data <NUM> received from the host computer <NUM> before rendering a final image(s) on the HMD <NUM>. At least some of these adjustments may utilize the extra data <NUM> received from the host computer <NUM>, such as the pose data, depth data, extra pixel data, parallax occlusion data, and/or motion vector data, as described herein. Accordingly, the frames that are shown in <FIG> are meant to represent "actual" frames in the sense that they are output from the application <NUM>, which may represent a video game application <NUM>(<NUM>), or any other type of graphics-based application. By contrast, if the application <NUM> failed to hit the target frame rate for a given frame, or if the frame rate was throttled to a lower rate than the refresh rate of the display panel(s) <NUM> of the HMD <NUM>, the compositor <NUM> of the HMD <NUM> may use the previously-received pixel data <NUM> for a preceding frame to generate a "phantom" frame (e.g., using re-projection) based on the pose prediction of the preceding frame and an updated pose prediction made by the HMD <NUM>. In any case, the result of the partial rendering workloads <NUM>(<NUM>) is the generation of modified pixel data that may be output to a frame buffer (e.g., a stereo frame buffer). This distinction between an "actual" frame and a "phantom" frame is not meant to imply that an actual frame is not adjusted on the HMD <NUM>, and, in this sense, the frames generated on the HMD side are all effectively synthesized (i.e., not the same as the original frames output by the application <NUM> executing on the host computer <NUM>).

The second timeline <NUM>(<NUM>) of <FIG> also shows a scan-out time <NUM>(a), <NUM>(b), and <NUM>(c) for each frame, as well as an illumination time <NUM>(a), <NUM>(b), and <NUM>(c) for each frame. During the scan-out time <NUM> for a given frame, subsets of pixel values (of the modified pixel data) are scanned out to the display panel(s) <NUM> via a display port (e.g., a high-definition multimedia interface (HDMI)), and during the illumination time <NUM> for the given frame, the light emitting elements of the display panel(s) <NUM> are illuminated to cause the pixels of the display panel(s) <NUM> to illuminate. <FIG> illustrates an example of a global flashing type of display driving scheme, which may be used with LCD panels to simultaneously emit light from the light emitting elements of the display panel(s) <NUM> at the refresh rate of the HMD <NUM>. In an illustrative example, if the HMD <NUM> is operating at a <NUM> refresh rate, the illumination time <NUM> for each frame may be separated by roughly <NUM> milliseconds.

It is to be appreciated that, although <FIG> depicts that the respective rendering cycles of the host computer <NUM> and the HMD <NUM> appear to be synchronized (which they can be), the techniques and systems described herein do not require synchronization of frames between the two devices. In general, the compositor <NUM> of the HMD <NUM> may start its rendering workload <NUM>(<NUM>) for a given frame as soon as the data (e.g., the pixel data <NUM> and the extra data <NUM>) is received from the host computer <NUM>, and/or as soon as the HMD <NUM> determines that the application <NUM> of the host computer <NUM> may have missed a frame or that packets may have been dropped in transit, etc. Due to varying conditions of the wireless communications link, the processing loads on the respective devices, and/or other factors, the respective rendering cycles of the host computer <NUM> and the HMD <NUM> may at times be out-of-synch/unsynchronized relative to each other. Accordingly, while the host computer <NUM> and the HMD <NUM> are configured to work together in a collaborative fashion by splitting the rendering workload for a given frame into partial workloads performed on the respective devices, one can appreciated that the devices may operate independently of one another to perform their respective portions of the workload.

The processes described herein are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes.

<FIG> illustrates a flow diagram of an example process <NUM> for rendering a frame by splitting the rendering workload <NUM> for the frame between a HMD <NUM> and a host computer <NUM>, in accordance with embodiments disclosed herein. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, a HMD <NUM> may send, to a host computer <NUM> that is communicatively coupled to the HMD <NUM>, first head tracking data <NUM> generated by a head tracking system <NUM> of the HMD <NUM>. The way in which the host computer <NUM> and the HMD <NUM> are communicatively coupled can vary by implementation. For an implementation where the host computer <NUM> is wirelessly coupled to the HMD <NUM>, the first head tracking data <NUM> may be sent wirelessly from the HMD <NUM> to the host computer <NUM> (e.g., using WiFi, Bluetooth, or any suitable wireless communication protocol, including proprietary protocols). For an implementation where the host computer <NUM> is coupled to the HMD <NUM> via a wired connection (e.g., a data cable), the first head tracking data <NUM> may be sent from the HMD <NUM> to the host computer <NUM> over a wired connection, such as a data cable. Furthermore, if the host computer <NUM> is located at a remote geographical location with respect to the HMD <NUM>, the first head tracking data <NUM> may be sent at block <NUM> from the HMD <NUM> to the host computer <NUM> over a wide-area network, such as the Internet.

At <NUM>, the host computer <NUM> may receive, from the HMD <NUM>, the first head tracking data <NUM>. As mentioned with respect to block <NUM>, the first head tracking data <NUM> may be received in various ways, depending on the implementation, such as wirelessly, over a wired connection, via a wide-area network, etc. At this point in time when the host computer <NUM> receives the first head tracking data <NUM>, the host computer <NUM> may be executing an application <NUM> thereon, such as a video game <NUM>(<NUM>), that is tasked with rendering a first frame of a series of frames for purposes of creating visual video game content to be displayed on the HMD <NUM>.

At <NUM>, logic of the host computer <NUM> (e.g., the render component <NUM>) may determine a predicted illumination time representing a time at which light emitting elements of the display panel(s) <NUM> of the HMD <NUM> will illuminate for the first frame. That is, the logic of the host computer <NUM> may determine a time when the photons associated with the imagery that is presented for the first frame will actually reach the user's <NUM> eye(s). This predicted illumination time is a time in the future (e.g., about <NUM> milliseconds in the future) because the render component <NUM> has to account for the time it takes for the application <NUM> to generate the pixel data <NUM>, the time it takes the pixel data <NUM> to be transmitted from the host computer <NUM> to the HMD <NUM>, and the time it takes for the pixel data <NUM> to be modified and scanned out on the HMD <NUM> before a corresponding image is ultimately presented on the display panel(s) <NUM> of the HMD <NUM>.

At <NUM>, the logic of the host computer <NUM> (e.g., the render component <NUM>) may determine, based at least in part on the first head tracking data <NUM> received at block <NUM>, a predicted pose that the HMD <NUM> will be in at the predicted illumination time that was determined at block <NUM>. For example, the head tracking system <NUM> of the HMD <NUM> may be configured to track up to six degrees of freedom of the HMD <NUM> (e.g., 3D position, roll, pitch, and yaw), which can be sent as head tracking data <NUM> to the host computer <NUM> to determine a predicted pose of the HMD <NUM> (e.g., accounting for predicted head movement resulting in a future pose of the HMD <NUM>).

At <NUM>, the logic of the host computer <NUM> (e.g., the render component <NUM>) may provide, to the application <NUM> executing on the host computer <NUM> for purposes of rendering the first frame, pose data indicative of the predicted pose, which was determined at block <NUM>. For instance, the application <NUM> may call a function to receive pose data from the render component <NUM>, and the render component <NUM> may provide the application <NUM> with the requested pose data (predicted to the target illumination time for the first frame, and predicted based at least in part on the head tracking data <NUM> received from the HMD <NUM>) so that the application <NUM> can render the first frame according to the pose data, which corresponds to a virtual camera pose used to render the scene. In some embodiments, the render component <NUM> may instruct the application <NUM> to generate not only the pixel data <NUM> for the frame, but extra data, such as depth data and/or extra pixel data.

At <NUM>, the logic of the host computer <NUM> (e.g., the render component <NUM>) may obtaining (or receive), from the application <NUM>, data (e.g., pixel data <NUM>) associated with the first frame. The pixel data <NUM> may include pixel values, as described herein, for individual pixels in the array of pixels of the display panel(s) <NUM> of the HMD <NUM>. As mentioned, in embodiments where the render component <NUM> requested the application <NUM> to generate extra data <NUM> in addition to the pixel data <NUM>, the render component <NUM>, at block <NUM> may obtain (or receive) extra data <NUM>, such as depth data (e.g., Z-buffer data) and/or extra pixel data that includes extra pixel values outside of the boundary of the array of pixels of the display panel(s) <NUM>. For example, if the display panel(s) <NUM> have an array of <NUM> x <NUM> pixels, the pixel data <NUM> may correspond to the pixel values in the <NUM> x <NUM> array of pixels, while extra pixel data may correspond to pixels that are outside of the boundary of the <NUM> x <NUM> array. Accordingly, the pixel data <NUM> and the extra pixel data may constitute a larger number of pixel values, such as a larger array of, say, <NUM> x <NUM> pixels, as an example.

At <NUM>, the logic of the host computer <NUM> (e.g., the render component <NUM>) may compress and/or serialize data that is to be sent to the HMD <NUM> for purposes of rendering imagery associated with the first frame. This data may include the pixel data <NUM> and any extra data <NUM> that was generated, whether the extra data <NUM> was generated by the application <NUM>, by the render component <NUM>, and/or any other component of the host computer <NUM>. The compression at block <NUM> may be optimized for a pre-distorted image(s), as opposed to a post-distorted image(s). For example, in systems where the entirety of the rendering workload is performed on a host computer, any compression of the pixel data that is sent to a headset may utilize a compression algorithm that mitigates stereo compression artifacts. In the process <NUM>, because the pixel data <NUM> generated by the application <NUM> is "pre-distorted," meaning that adjustments - such as re-projection adjustments, geometric distortion adjustments, chromatic aberration adjustments, etc. - are to be applied after compression, the compression algorithm utilized at block <NUM> may not need to account for stereo compression artifact. In some embodiments, the pixel data <NUM> output by the application <NUM> may be classified into foreground pixels and background pixels (and potentially intermediate layers of pixels), and different compression algorithms or schemes can be used for the different pixel layers. That is, foreground pixels can be compressed differently (e.g., using a different compression algorithm) than the background pixels are compressed.

At <NUM>, the host computer <NUM> may send, to the HMD <NUM>, data that includes the pixel data <NUM> and extra data <NUM>. The extra data <NUM> sent at block <NUM> can include, without limitation, the pose data <NUM>(<NUM>) generated based on the first head tracking data <NUM> and used by the application <NUM> to generate the pixel data <NUM> for the first frame, depth data <NUM>(<NUM>) generated by the application <NUM>, extra pixel data <NUM>(<NUM>) generated by the application <NUM>, motion vector data <NUM>(<NUM>) generated by the render component <NUM> based on head tracking data <NUM> and/or based on movements of virtual objects within the scene being rendered, parallax occlusion data <NUM>(<NUM>), and/or cube map data (e.g., for rapid, large-scale head movements so that the HMD <NUM> has other options besides presenting dark pixels where it does not have any data). Again, the pixel data <NUM> and the extra data <NUM> may be sent to the HMD <NUM> in various ways, depending on the implementation, such as wirelessly, over a wired connection, via a wide-area network, etc..

At <NUM>, the HMD <NUM> may receive, from the host computer <NUM>, the pixel data <NUM> associated with the first frame and the extra data <NUM> in addition to the pixel data <NUM>. Again, the pixel data <NUM> and the extra data <NUM> may be received from the host computer <NUM> in various ways, depending on the implementation, such as wirelessly, over a wired connection, via a wide-area network, etc. Furthermore, when compressed, serialized data is received by from the host computer <NUM> at block <NUM>, the data may be deserialized and decompressed at block <NUM>.

At <NUM>, logic of the HMD <NUM> (e.g., the compositor <NUM>) may determine, based at least in part on second head tracking data <NUM> generated by the head tracking system <NUM> of the HMD <NUM>, an updated pose that the HMD <NUM> will be in at the illumination time for the first frame, which represents a time at which the light emitting elements of the display panel(s) <NUM> of the HMD <NUM> will illuminate for the first frame. Because the determination at block <NUM> is closer in time to the illumination time for the first frame, the pose prediction at block <NUM> is more accurate (e.g., has less error) than the pose prediction that was determined at block <NUM>, which was further ahead the illumination time. In some embodiments, the determination of the updated pose of the HMD <NUM> at block <NUM> may be based at least in part on motion vector data <NUM>(<NUM>) received from the host computer <NUM>, or generated locally on the HMD <NUM>. For example, the motion vector data <NUM>(<NUM>) generated from head tracking data <NUM> may be indicative of predicted head movement of the user <NUM>, and the compositor <NUM> of the HMD <NUM> may use the motion vector data <NUM>(<NUM>) to, among other things, make the updated pose prediction of the HMD <NUM> with improved accuracy.

At <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may modify the pixel data <NUM> received from the host computer <NUM> to obtain modified pixel data. As shown by the sub-blocks of block <NUM>, this modification of the pixel data <NUM> may include various sub-operations.

At sub-block <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may apply re-projection adjustments to the pixel data <NUM> to obtain modified pixel data associated with the first frame. The re-projection adjustments applied at sub-block <NUM> may be based at least in part on the extra data <NUM> received from the host computer <NUM>. For example, a comparison between the original predicted pose determined at block <NUM> (and exhibited in the pose data <NUM>(<NUM>)) and the updated pose determined at block <NUM> may reveal a delta (or difference) between the compared poses, and the re-projection adjustments may include rotational calculations to compensate for this delta (e.g., by shifting and/or rotating the pixel data <NUM> one way or another, depending on the delta between the two pose determinations).

In some embodiments, the extra data <NUM> received from the host computer <NUM> includes depth data <NUM>(<NUM>) associated with the first frame, and the re-projection adjustments applied at sub-block <NUM> may be based at least in part on the depth data <NUM>(<NUM>). For example, the depth data <NUM>(<NUM>), such as from a depth buffer (or Z-buffer), may be indicative of occluded objects in the scene. Accordingly, the depth data <NUM>(<NUM>) can be used to, among other things, adjust for the parallax of objects in the scene (e.g., a ship that is far away in world space may not move as much with head movement as a close-up object will move with the same head movement). Knowing the depth of pixels that correspond to virtual objects in the scene is helpful to know how to adjust for such parallax during re-projection on the HMD <NUM> at sub-block <NUM>. Other ways of using depth data <NUM>(<NUM>) on the HMD-side are described in more detail below. In some embodiments, the extra data <NUM> received from the host computer <NUM> includes extra pixel data <NUM>(<NUM>) associated with the first frame, and the re-projection adjustments applied at sub-block <NUM> may be based at least in part on the extra pixel data <NUM>(<NUM>). For example, in cases where rapid/large-scale head movement is occurring, at least some of the pixel values of the pixel data <NUM> may be replaced with the extra pixel values of the extra pixel data <NUM>(<NUM>) to render a portion of the scene that corresponds to the user's <NUM> current head movement, as determined by the HMD <NUM>. In some embodiments, the extra data <NUM> received from the host computer <NUM> includes motion vector data <NUM>(<NUM>) associated with the first frame, and the re-projection adjustments applied at sub-block <NUM> may be based at least in part on the motion vector data <NUM>(<NUM>).

At sub-block <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may apply geometric distortion adjustments to the pixel data <NUM> to obtain the modified pixel data associated with the first frame at block <NUM>. Geometric distortion adjustments may compensate for the distortion of the near-to-eye optical subsystem (e.g., lenses and other optics) of the HMD <NUM>. For similar reasons, at sub-block <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may apply chromatic aberration adjustments to the pixel data <NUM> to obtain the modified pixel data associated with the first frame at block <NUM>.

At sub-block <NUM>, the pixel data <NUM> may be modified by overlaying one or more virtual hands on the scene represented by the pixel data <NUM> to obtain the modified pixel data associated with the first frame at block <NUM>. For example, the HMD <NUM> may be communicatively coupled to one or more handheld controllers whose movement and position in 3D space is tracked, much like the HMD <NUM>, and the HMD <NUM> may receive hand tracking data from the handheld controller(s) (e.g., over a direct wireless data transmission from the handheld controller(s) to the HMD <NUM>). The hand tracking data can be processed by the logic of the HMD <NUM> (e.g., the compositor <NUM>) to render a virtual hand(s) on the scene, which are overlaid on the content of the scene.

At <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may output the modified pixel data to a frame buffer(s). Again, for a HMD <NUM> with a pair of display panels <NUM>, this modified pixel data may correspond to a frame that represents a pair of images to be displayed on the pair of display panels <NUM>, and may be output to a stereo frame buffer accordingly.

At <NUM>, logic of the HMD <NUM> may cause a first image(s) to be presented on the display panel(s) <NUM> of the HMD <NUM> based on the modified pixel data output to the frame buffer at block <NUM>. This may involve scanning out the modified pixel data to the display panel(s) <NUM> of the HMD <NUM> and illuminating the light emitting elements of the display panel(s) <NUM> to illuminate the pixels on the display panel(s) <NUM>.

<FIG> illustrates a flow diagram of an example process <NUM> for applying re-projection adjustments on a HMD <NUM> based on motion vector data <NUM>(<NUM>) generated by a host computer <NUM>, in accordance with embodiments disclosed herein. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, the HMD <NUM> may send head tracking data <NUM> to the host computer <NUM>. This head tracking data <NUM> may be generated and/or sent at any suitable frequency, such as a frequency corresponding to the target frame rate of the application <NUM> and/or the refresh rate of the HMD <NUM>, or a different (e.g., faster) frequency, and the head tracking data <NUM> may indicate head movement of the user <NUM> who is wearing the HMD <NUM>. As such, the host computer <NUM>, at block <NUM>, may receive multiple instances of head tracking data <NUM> over time, such as first head tracking data <NUM> generated at time, t<NUM>, second head tracking data <NUM> generated at time, t<NUM>, and so on. Furthermore, the host computer <NUM> may maintain a history of head tracking data <NUM> that it received over time so that it has multiple instances of head tracking data <NUM> available at any given time. The host computer <NUM> can discard head tracking data <NUM> that is older than a predefined age to conserve memory resources, while retaining at least some past head tracking data <NUM>.

At <NUM>, logic of the host computer <NUM> (e.g., the render component <NUM>) may generate motion vector data <NUM>(<NUM>) based at least in part on head tracking data <NUM> received from the HMD <NUM>. For example, the host computer <NUM> may receive first head tracking data <NUM>, and may have received second head tracking data <NUM> prior to receiving the first head tracking data <NUM>, and a comparison can be made between these sets of head tracking data <NUM> to generate motion vector data <NUM>(<NUM>). In some embodiments, motion vector data <NUM>(<NUM>) can additionally, or alternatively, be generated from pixel data associated with multiple frames rendered by the application. For example, a pair of frames previously rendered by the application <NUM> may be provided as input to the GPU(s) <NUM> of the host computer <NUM>, and a decoder of the GPU(s) <NUM> may generate motion vector data <NUM>(<NUM>) based at least in part on the input frames, such as by looking for similarities (e.g., similar color values, similar luminance values, etc.) in the pair of frames and mapping motion vectors to the corresponding positions of similarities in the pair of frames. In some embodiments, the logic of the host computer <NUM> may be configured to filter out motion relating to moving virtual objects in the scene so that it is left with motion due to head movement and not motion of virtual objects in the scene.

At <NUM>, the HMD <NUM> may receive the motion vector data <NUM>(<NUM>) from the host computer <NUM>. For example, this motion vector data <NUM>(<NUM>) may be sent as extra data <NUM> along with (e.g., packaged with) the pixel data <NUM> for a given frame whose imagery is to be presented on the HMD <NUM>. Thus, at block <NUM>, or at a slightly different (e.g., earlier or later) time, the HMD <NUM> may receive pixel data <NUM> for a given frame.

At <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may apply re-projection adjustments to the pixel data <NUM> based at least in part on the motion vector data <NUM>(<NUM>) received at block <NUM> to obtain modified pixel data <NUM> associated with a given frame. The motion vector data <NUM>(<NUM>) may be considered in addition to pose data indicative of an original predicted pose and an updated pose determination of the HMD <NUM>, as described herein. As such, the motion vector data <NUM>(<NUM>) may be augmentative to the pose data <NUM>(<NUM>) for purposes of predicting a pose of the HMD <NUM> and thereby applying re-projection adjustments (e.g., rotational calculations to shift and/or rotate the pixel data <NUM> one way or another). Alternatively, the motion vector data <NUM>(<NUM>) may be applied directly to the pixel data <NUM>, such as by converting the motion vector data <NUM>(<NUM>) into a motion vector field that corresponds to the screen space of the scene that is to be rendered, and shifting the pixel values based on the magnitude and direction of the motion vectors that correspond to those pixel values.

<FIG> illustrates a flow diagram of an example process <NUM> for applying re-projection adjustments based on extra pixel data <NUM>(<NUM>) generated by an application <NUM> executing on a host computer <NUM>, in accordance with embodiments disclosed herein. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, the HMD <NUM> may send head tracking data <NUM> to the host computer <NUM>. This head tracking data <NUM> may be generated and/or sent at any suitable frequency, such as a frequency corresponding to the target frame rate of the application <NUM> and/or the refresh rate of the HMD <NUM>, or a different (e.g., faster) frequency, and the head tracking data <NUM> may indicate head movement of the user <NUM> who is wearing the HMD <NUM>.

At <NUM>, logic of the host computer <NUM> (e.g., the render component <NUM>) may instruct an application <NUM> executing on the host computer <NUM>, such as a video game <NUM>(<NUM>), to generate extra pixel data <NUM>(<NUM>) in addition to the pixel data <NUM> that corresponds to the array of pixels on the display panel(s) <NUM> of the HMD <NUM>. For example, if the display panel(s) <NUM> have an array of <NUM> x <NUM> pixels, the pixel data <NUM> may correspond to the pixel values in the <NUM> x <NUM> array of pixels, while extra pixel data <NUM>(<NUM>) may correspond to pixels that are outside of the boundary of the <NUM> x <NUM> array. Accordingly, the pixel data <NUM> and the extra pixel data <NUM>(<NUM>) being requested at block <NUM> may constitute a larger number of pixel values, such as a larger array of, say, <NUM> x <NUM> pixels, as an example. The determination to instruct the application <NUM> to generate extra pixel data <NUM>(<NUM>) at block <NUM> may be a dynamic determination for the given frame, as illustrated by the sub-blocks of block <NUM>.

At sub-block <NUM>, logic of the host computer <NUM> (e.g., the render component <NUM>) may determine to generate (e.g., by instructing the application <NUM> to generate) extra pixel data <NUM>(<NUM>) based at least in part on the head tracking data <NUM> received at block <NUM> indicating an amount of movement of the HMD <NUM> that is greater than a threshold amount of movement. For example, two instances of head tracking data <NUM> may be compared to determine an amount of movement of the HMD <NUM> over a period of time (e.g., a period of a few milliseconds), and if that amount of movement is greater than a threshold amount of movement, the render component <NUM> may determine to generate extra pixel data <NUM>(<NUM>) for the given frame. If, on the other hand, the amount of movement of the HMD <NUM> is less than or equal to the threshold amount of movement, the render component <NUM> may determine to refrain from generating extra pixel data <NUM>(<NUM>) for the given frame. This is shown pictorially to the right of sub-block <NUM> in <FIG>, where the render component <NUM> is configured to instruct the application <NUM> to render only the pixel data <NUM> (and to not render any extra pixel data <NUM>(<NUM>) if the amount of HMD movement is at or below a threshold amount of movement, and to instruct the application <NUM> to render the pixel data <NUM> and extra pixel data <NUM>(<NUM>) if the amount of HMD movement is above the threshold amount of movement. This allows for conserving resources by determining that extra pixel data <NUM>(<NUM>) is unlikely to be used if there is currently little-to-no head movement, and to reserve the generation of extra pixel data <NUM>(<NUM>) for instances where there is currently a large amount and/or rapid head movement.

At sub-block <NUM>, logic of the host computer <NUM> (e.g., the render component <NUM>) may determine to render (e.g., by instructing the application <NUM> to render) a number of the extra pixel values in the extra pixel data <NUM>(<NUM>) based at least in part on an amount of movement of the HMD <NUM> indicated by the head tracking data <NUM> received at block <NUM>. This is shown pictorially to the right of sub-block <NUM> in <FIG>, where the render component <NUM> is configured to instruct the application <NUM> to render a first number of extra pixel values in the extra pixel data <NUM>(<NUM>) if the amount of HMD movement is a first, lesser amount of movement, and to instruct the application <NUM> to render a second, greater number of extra pixel values in the extra pixel data <NUM>(<NUM>) if the amount of HMD movement is a second, greater amount of movement. For example, for less head movement, the render component <NUM> may instruct the application <NUM> to render a total of <NUM> x <NUM> pixels, and for more head movement, the render component <NUM> may instruct the application <NUM> to render a total of <NUM> x <NUM> pixels (i.e., a greater amount of extra pixel values for a fixed number of "on-screen" pixel values of the pixel data <NUM>). In other words, the buffer of extra pixels may expand or shrink based on the degree of head movement. This, again, is a technique to conserve resources on the host computer <NUM> by generating a sufficient amount of extra pixel data <NUM>(<NUM>), but not a superfluous amount of extra pixel data <NUM>(<NUM>).

At sub-block <NUM>, logic of the host computer <NUM> (e.g., the render component <NUM>) may determine to generate (e.g., by instructing the application <NUM> to generate) particular extra pixel data <NUM>(<NUM>) based at least in part on at least one of the motion vector data <NUM>(<NUM>) or predictive data known to and/or generated by the application <NUM>. For example, if the motion vector data <NUM>(<NUM>) is indicative of head movement in an upward and leftward direction with reference to a view looking at a front side of the display panel(s) <NUM> of the HMD <NUM>, the render component <NUM> may determine to instruct the application <NUM> to generate extra pixel data <NUM>(<NUM>) above the topmost pixels in the pixel data <NUM> and to the left of the leftmost pixels of the pixel data <NUM>, and to refrain from generating extra pixel data <NUM>(<NUM>) to the right or below the rightmost and bottommost pixels, respectively, of the pixel data <NUM>, which conserves resources of the host computer <NUM>. In other words, if it is unlikely that the user <NUM> will move his/her head to the right and/or down, then extra pixels to the right of the rightmost pixels in the scene and below the bottommost pixels in the scene do not have to be rendered due to the low likelihood of the user <NUM> needing to see those pixels. Similarly, if the motion vector data <NUM>(<NUM>) is indicative of head movement in a downward and rightward direction with reference to a view looking at a front side of the display panel(s) <NUM> of the HMD <NUM>, the render component <NUM> may determine to instruct the application <NUM> to generate extra pixel data <NUM>(<NUM>) below the bottommost pixels in the pixel data <NUM> and to the right of the rightmost pixels of the pixel data <NUM>, and to refrain from generating extra pixel data <NUM>(<NUM>) to the left or above the leftmost and topmost pixels, respectively, of the pixel data <NUM>, which conserves resources of the host computer <NUM>. This is shown pictorially to the right of sub-block <NUM> in <FIG>.

In some embodiments, at block <NUM>, the determination of which "out-of-bounds" pixels to render may be determined from predictive data obtained from the application <NUM>. For example, if, based on information obtained from the execution application <NUM>, the render component <NUM> knows (or predicts with high likelihood) that an explosion is about to occur very soon on the left side of the screen, the render component <NUM> can predict, with high likelihood, future head movement in the leftward direction (i.e., towards the explosion). Accordingly, for the given frame, the render component <NUM> may instruct the application <NUM>, at sub-block <NUM>, to render "out-of-bounds" pixels beyond the leftmost pixels in the scene in anticipation of the leftward head movement so that, if the user <NUM> moves his/her head, as predicted, to the left in reaction to seeing the explosion, the out-of-bounds pixels may utilized for presenting the image for that frame.

At sub-block <NUM>, logic of the host computer <NUM> (e.g., the render component <NUM>) may determine to generate (e.g., by instructing the application <NUM> to generate) the pixel data <NUM> at a first resolution and to generate the extra pixel data <NUM>(<NUM>) at a second resolution lower than the first resolution. This is yet another technique to conserve resources where it is anticipated that the extra pixel data <NUM>(<NUM>) will be used to create a re-projected frame if there is large-scale or rapid head movement, but otherwise may not be used, and because the extra pixel data <NUM>(<NUM>) is used during large-scale or rapid head movement, the lower resolution is unlikely to be noticed by the user <NUM> when the scene in front of the user's <NUM> eyes is moving. In other words, the "out-of-bounds" pixels in the extra pixel data <NUM>(<NUM>) are expected to be used if there is unexpected head movement, but otherwise discarded, which means that head movement is likely to occur during presentation of the image based on extra pixel data <NUM>(<NUM>), and the user can often tolerate lower-resolution images during head movement because their eyes are busy tracking the scene, and the fine details in a scene often go unnoticed during head movement.

At <NUM>, after the render component <NUM> obtains the extra pixel data <NUM>(<NUM>) from the application <NUM>, the HMD <NUM> may receive the extra pixel data <NUM>(<NUM>) from the host computer <NUM>. For example, this extra pixel data <NUM>(<NUM>) may be sent as extra data <NUM> along with (e.g., packaged with) the pixel data <NUM> for a given frame whose imagery is to be presented on the HMD <NUM>. Thus, at block <NUM>, or at a slightly different (e.g., earlier or later) time, the HMD <NUM> may receive pixel data <NUM> for a given frame.

At <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may apply re-projection adjustments to the pixel data <NUM> based at least in part on the extra pixel data <NUM>(<NUM>) received at block <NUM> to obtain modified pixel data <NUM> associated with a given frame. The extra pixel data <NUM>(<NUM>) may include extra pixel values outside of a boundary of the array of pixels of the display panel(s) <NUM> of the HMD <NUM>, and may be used in re-projection by replacing at least some of the pixel values included in the pixel data <NUM> with at least some of the extra pixel values included in the extra pixel data <NUM>(<NUM>). For example, if re-projection adjustments shift the pixel data <NUM> leftward, the extra pixel data <NUM>(<NUM>) to the right of the rightmost pixels of the pixel data <NUM> may be output in the modified pixel data <NUM>. These "out-of-bounds" pixel values (e.g., pixel values that are beyond the left, right, top, and/or bottom edge of the pixels corresponding to the display panel(s) <NUM>), allow for content to be displayed in a final image instead of displaying dark pixels. Although this means that more pixels are rendered than the number of pixels that will actually be displayed in a final image, this redundancy measure compensates for latency in a wireless communication link between the HMD <NUM> and the host computer <NUM>, for example. That said, the resource conserving techniques described with reference to the process <NUM> may improve the performance of the computing device regardless of the redundant measures that are taken in account of the latency in data transfer.

<FIG> illustrates a flow diagram of an example process <NUM> for applying re-projection adjustments based on depth data <NUM>(<NUM>) generated by an application <NUM> executing on a host computer <NUM>, in accordance with embodiments disclosed herein. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, the render component <NUM> may instruct the application <NUM> to generate, and the render component <NUM> may obtain from the application <NUM>, depth data <NUM>(<NUM>) associated with a given frame. The depth data <NUM>(<NUM>) (or, Z-buffer data), may be indicative of occluded objects in a scene, indicating their depth relative to a location of the user in world space.

At <NUM>, the HMD <NUM> may receive the depth data <NUM>(<NUM>) from the host computer <NUM>. For example, this depth data <NUM>(<NUM>) may be sent as extra data <NUM> along with (e.g., packaged with) the pixel data <NUM> for a given frame whose imagery is to be presented on the HMD <NUM>. Thus, at block <NUM>, or at a slightly different (e.g., earlier or later) time, the HMD <NUM> may receive pixel data <NUM> for a given frame.

At <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may apply re-projection adjustments to the pixel data <NUM> based at least in part on the depth data <NUM>(<NUM>) received at block <NUM> to obtain modified pixel data <NUM> associated with a given frame. As shown by the sub-blocks of block <NUM>, the modification of the pixel data <NUM> based on the depth data <NUM>(<NUM>) may involve sub-operations.

At sub-block <NUM>, the logic of the HMD <NUM> (e.g., the compositor <NUM>) may classify, based at least in part on the depth data <NUM>(<NUM>), a first subset of the pixel values included in the pixel data <NUM> as foreground pixels <NUM> and a second subset of the pixel values included in the pixel data <NUM> as background pixels <NUM>. At block <NUM>, one or more intermediate layers may be used to classify pixels, such as a mid-layer between the foreground and the background. Thus, the pixels included in the pixel data <NUM> can be classified into multiple layers at any suitable granularity.

At sub-block <NUM>, the applying of the re-projection adjustments at block <NUM> may include modifying the first subset of the pixel values classified as the foreground pixels <NUM>, and refraining from modifying the second subset of the pixel values classified as the background pixels <NUM>. In other words, the compositor <NUM> of the HMD <NUM> may apply different adjustments, or no adjustments, for a given pixel depending on the layer in which that pixel is classified (e.g., foreground vs. background). This is yet another a resource conservation technique to avoid wasting computing resources on applying re-projection adjustments to background pixels <NUM>, seeing as how slight inaccuracies in the background content will go unnoticed by the user <NUM>. This technique described in the process <NUM> may be enabled by defining different update rates at which re-projection adjustments are applied for different layers of classified pixels, as determined from the depth data <NUM>(<NUM>). For example, background pixels corresponding to object in the background of a scene may be updated (for re-projection) less frequently than foreground pixels corresponding to objects in the foreground of the scene. For example, logic of the HMD <NUM> (e.g., the compositor <NUM>) may perform re-projection adjustments on background pixels <NUM> at <NUM>, while performing re-projection adjustments on foreground pixels <NUM> at a higher frequency, such as <NUM>. For a frame rate of <NUM>, this would mean that foreground pixels <NUM> are modified for re-projection adjustments at every frame, while background pixels <NUM> are modified for re-projection adjustments every third frame in a series of frames, which means that two sequential frames would include no re-projection adjustments for background pixels to conserve resources. This is at least partly based on the notion of the re-projection adjustments for background pixels <NUM> are expected to be smaller-scale adjustments than the re-projection adjustments for foreground pixels <NUM> given the same error in the original pose prediction of the HMD <NUM>.

<FIG> illustrates a flow diagram of an example process <NUM> for an HMD <NUM> to receive hand tracking data directly from a handheld controller, and overlaying a virtual hand(s) on an application-rendered scene using the hand tracking data, in accordance with embodiments disclosed herein. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, the HMD <NUM> may receive, from at least one handheld controller <NUM> that is communicatively coupled to the HMD <NUM>, hand tracking data <NUM>. The handheld controller(s) <NUM> may represent video game controllers, such as a handheld VR controller whose movement and/or position in 3D space is tracked much like the movement and/or position of the HMD <NUM> can be tracked. Instead of the handheld controller(s) <NUM> sending hand tracking data <NUM> to the host computer <NUM> for overlaying virtual hands on the scene at the host computer <NUM>, a latency reducing measure can be taken by having the handheld controller(s) <NUM> send hand tracking data <NUM> indicative of movement and/or position of the handheld controller(s) <NUM> directly to the HMD <NUM>, without sending the hand tracking data <NUM> to the host computer <NUM> before the hand tracking data <NUM> is received by the HMD <NUM>. The hand tracking data <NUM> may include data generated by tracking sensors mounted on the handheld controller(s) <NUM> and possibly proximity sensor data generated by proximity sensors of the handheld controller <NUM> (e.g., capacitive sensors) to indicate finger positioning/distance relative to the handheld controller <NUM>.

At <NUM>, logic of the HMD <NUM> (e.g., the compositor <NUM>) may modify, based at least in part on the hand tracking data <NUM>, the pixel data <NUM> for a given frame to include one or more virtual hands <NUM> overlaid on a scene <NUM> represented by the pixel data <NUM> to obtain the modified pixel data associated with the given frame. For example, at sub-block <NUM>, the compositor <NUM> of the HMD <NUM> may overlay one or more virtual hands on <NUM> on the scene rendered by the application <NUM> by replacing pixel values generated by the application <NUM> with pixel values generated by the compositor <NUM> to present the virtual hand(s) <NUM>. The HMD <NUM> can therefore render the virtual hand(s) <NUM> on the scene <NUM> based on the hand tracking data <NUM> the HMD <NUM> receives directly from the handheld controller(s) <NUM>. This may allow for minimizing re-projection adjustments for the rendering of the virtual hand(s) <NUM> because the pose prediction for the handheld controller(s) <NUM> does not have to be made as far in advance if data does not have to transmitted to the host computer <NUM> and then from the host computer <NUM> to the HMD <NUM>.

<FIG> illustrate two alternative setups of a system that splits a rendering workload <NUM> for a frame between a HMD <NUM> and a host computer <NUM>, in accordance with embodiments disclosed herein. Briefly referring to <FIG>, an example implementation is where the host computer <NUM> is collocated in an environment with the HMD <NUM> worn by the user <NUM>. For example, the host computer <NUM> may be located in the user's <NUM> house while the user <NUM> is using the HMD <NUM> in the house, regardless of whether the host computer <NUM> is located in the same room or a different room as the HMD <NUM>. Alternatively, the host computer <NUM> in the form of a mobile computing device (e.g., a tablet or laptop) may be carried in a backpack on the back of the user <NUM>, thereby allowing for greater mobility. For example, the user <NUM> could be located in a public park while using such a system.

<FIG> shows an alternative implementation where the host computer <NUM> represents one or more server computers located at a geographically remote location with respect to the HMD <NUM>. In this case, the HMD <NUM> may be communicatively coupled to the host computer(s) <NUM> via an access point (AP) <NUM>, such as a wireless AP (WAP), a base station, etc. In an illustrative example, data is exchanged (e.g., streamed) between the host computer <NUM> and the HMD <NUM> via the AP <NUM>, such as by streaming data over the Internet.

<FIG> shows yet another alternative implementation where the host computer <NUM> is communicatively coupled to the HMD <NUM> via an intermediate computing device <NUM>, such as a laptop or a tablet computer. A difference between <FIG> is that the AP <NUM> in <FIG> may simply act as a data routing device that does not perform rendering, while the intermediate computing device <NUM> of <FIG> may perform a portion of the rendering workload <NUM>. That is, instead of bifurcating the rendering workload <NUM> between the host computer <NUM> and the HMD <NUM>, the rendering workload <NUM> can be partitioned between more than two devices, such as three device: the host computer <NUM>, the intermediate computing device <NUM>, and the HMD <NUM>. In the scenario of <FIG>, the host computer <NUM> may generate pixel data <NUM>, as described herein, the intermediate computing device <NUM> may perform a first set of rendering operations to modify the pixel data <NUM>, and the HMD <NUM> may perform a final set of rendering operations to modify the modified pixel data.

<FIG> illustrates example components of a wearable device, such as a HMD <NUM> (e.g., a VR headset), and a host computer <NUM>, in which the techniques disclosed herein can be implemented, according to the embodiments disclosed herein. The HMD <NUM> may be implemented as a connected device that is communicatively coupled to the host computer <NUM> during operation, and/or as a standalone device. In either mode of operations, the HMD <NUM> is to be worn by a user <NUM> (e.g., on a head of the user <NUM>). In some embodiments, the HMD <NUM> may be head-mountable, such as by allowing a user <NUM> to secure the HMD <NUM> on his/her head using a securing mechanism (e.g., an adjustable band) that is sized to fit around a head of a user <NUM>. In some embodiments, the HMD <NUM> comprises a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display(s). As such, the terms "wearable device", "wearable electronic device", "VR headset", "AR headset", and "head-mounted display (HMD)" may be used interchangeably herein to refer to the device <NUM> of <FIG>. However, it is to be appreciated that these types of devices are merely example of a HMD <NUM>, and it is to be appreciated that the HMD <NUM> may be implemented in a variety of other form factors. It is also to be appreciated that some or all of the components shown in <FIG> may be implemented on the HMD <NUM>. Accordingly, in some embodiments, a subset of the components shown as being implemented in the HMD <NUM> may be implemented on the host computer <NUM> or another computing device that is separate from the HMD <NUM>.

In the illustrated implementation, the HMD <NUM> includes the aforementioned processor(s) <NUM>, which may include one or more GPUs <NUM>, as well as the memory <NUM> storing the compositor <NUM> that is executable by the processor(s) <NUM>, the display panel(s) <NUM>, the head tracking system <NUM>, and the communications interface(s) <NUM>.

Additional functional modules are shown as being stored in the computer-readable media <NUM> and executable on the processor(s) <NUM>, although the same functionality may alternatively be implemented in hardware, firmware, or as a SOC, and/or other logic. For example, an operating system module <NUM> may be configured to manage hardware within and coupled to the HMD <NUM> for the benefit of other modules. In addition, in some instances the HMD <NUM> may include one or more applications <NUM> stored in the memory <NUM> or otherwise accessible to the HMD <NUM>. For example, the application(s) <NUM> may include, without limitation, a video game application (e.g., a basic video game with graphics that are less-computationally-intensive to process), a video playback application (e.g., an application that accesses a video content library stored on the HMD <NUM> and/or in the cloud), etc. The HMD <NUM> may include any number or type of applications <NUM> and is not limited to the specific examples described herein.

Generally, the HMD <NUM> has input devices <NUM> and output devices <NUM>. The input devices <NUM> may include control buttons. In some implementations, one or more microphones may function as input devices <NUM> to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement unit (IMU)) may function as input devices <NUM> to receive gestural input, such as a hand and/or head motion of the user <NUM>. In some embodiments, additional input devices <NUM> may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, and the like. In other embodiments, the HMD <NUM> may omit a keyboard, keypad, or other similar forms of mechanical input. Instead, the HMD <NUM> may be implemented relatively simplistic forms of input device <NUM>, a network interface (wireless or wire-based), power, and processing/memory capabilities. For example, a limited set of one or more input components may be employed (e.g., a dedicated button to initiate a configuration, power on/off, etc.) so that the HMD <NUM> can thereafter be used. In one implementation, the input device(s) <NUM> may include control mechanisms, such as basic volume control button(s) for increasing/decreasing volume, as well as power and reset buttons.

The output devices <NUM> may include a display panel(s) <NUM>, which may include one or multiple display panels <NUM> (e.g., a stereo pair of display panels <NUM>), as described herein. The output devices <NUM> may further include, without limitation, a light element (e.g., LED), a vibrator to create haptic sensations, a speaker(s) (e.g., headphones), and/or the like. There may also be a simple light element (e.g., LED) to indicate a state such as, for example, when power is on.

The HMD <NUM> may further include a communications interface(s) <NUM> including, without limitation, a wireless unit <NUM> coupled to an antenna <NUM> to facilitate a wireless connection to a network and/or to a second device, such as the host computer <NUM>. The wireless unit <NUM> may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that the HMD <NUM> may further include physical ports to facilitate a wired connection to a network and/or a second device, such as the host computer <NUM>.

The HMD <NUM> may further include optical subsystem <NUM> that directs light from the display panel(s) <NUM> to a user's eye(s) using one or more optical elements. The optical subsystem <NUM> may include various types and combinations of different optical elements, including, without limitations, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and so forth. In some embodiments, one or more optical elements in optical subsystem <NUM> may have one or more coatings, such as anti-reflective coatings. Magnification of the image light by optical subsystem <NUM> allows display panel(s) <NUM> to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification of the image light may increase a field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., <NUM>-<NUM> degrees diagonal), and in some cases all, of the user's FOV. AR applications may have a narrower FOV (e.g., about <NUM> degrees FOV). Optical subsystem <NUM> may be designed to correct one or more optical errors, such as, without limitation, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and so forth. In some embodiments, content provided to display panel(s) <NUM> for display is pre-distorted (e.g., by the applied geometric distortion adjustments and/or chromatic aberration adjustments described herein), and optical subsystem <NUM> corrects the distortion when it receives image light from display panel(s) <NUM> generated based on the content.

The HMD <NUM> may further include one or more sensors <NUM>, such as sensors used to generate motion, position, and orientation data. These sensors <NUM> may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors. The sensors <NUM> may also include sub-portions of sensors, such as a series of active or passive markers that may be viewed externally by a camera or color sensor in order to generate motion, position, and orientation data. For example, a VR headset may include, on its exterior, multiple markers, such as reflectors or lights (e.g., infrared or visible light) that, when viewed by an external camera or illuminated by a light (e.g., infrared or visible light), may provide one or more points of reference for interpretation by software in order to generate motion, position, and orientation data. The HMD <NUM> may include light sensors that are sensitive to light (e.g., infrared or visible light) that is projected or broadcast by base stations in the environment of the HMD <NUM>.

In an example, the sensor(s) <NUM> may include an inertial measurement unit (IMU) <NUM>. IMU <NUM> may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and/or other sensors suitable for detecting motion, correcting error associated with IMU <NUM>, or some combination thereof. Based on the measurement signals such motion-based sensors, such as the IMU <NUM>, may generate calibration data indicating an estimated position of HMD <NUM> relative to an initial position of HMD <NUM>. For example, multiple accelerometers may measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). IMU <NUM> can, for example, rapidly sample the measurement signals and calculate the estimated position of HMD <NUM> from the sampled data. For example, IMU <NUM> may integrate measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on HMD <NUM>. The reference point is a point that may be used to describe the position of the HMD <NUM>. While the reference point may generally be defined as a point in space, in various embodiments, reference point is defined as a point within HMD <NUM> (e.g., a center of the IMU <NUM>). Alternatively, IMU <NUM> provides the sampled measurement signals to an external console (or other computing device), which determines the calibration data.

The sensors <NUM> may operate at relatively high frequencies in order to provide sensor data at a high rate. For example, sensor data may be generated at a rate of <NUM> (or <NUM> sensor reading every <NUM> millisecond). In this way, one thousand readings are taken per second. When sensors generate this much data at this rate (or at a greater rate), the data set used for predicting motion is quite large, even over relatively short time periods on the order of the tens of milliseconds.

As mentioned, in some embodiments, the sensors <NUM> may include light sensors that are sensitive to light emitted by base stations in the environment of the HMD <NUM> for purposes of tracking position and/or orientation, pose, etc., of the HMD <NUM> in 3D space. The calculation of position and/or orientation may be based on timing characteristics of light pulses and the presence or absence of light detected by the sensors <NUM>.

The HMD <NUM> may further include an eye tracking system <NUM> that generates eye tracking data. The eye tracking system <NUM> may include, without limitation, a camera or other optical sensor inside HMD <NUM> to capture image data (or information) of a user's eyes, and the eye tracking system <NUM> may use the captured data/information to determine motion vectors, interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye relative to HMD <NUM>, including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye. In one example, infrared light is emitted within HMD <NUM> and reflected from each eye. The reflected light is received or detected by a camera of the eye tracking system <NUM> and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of a user <NUM> can be used by eye tracking system <NUM>. Accordingly, eye tracking system <NUM> may track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes of a user <NUM> to estimate a gaze point (i.e., a 3D location or position in the virtual scene where the user is looking), which may map to a location(s) on the display panel(s) <NUM> for predicting where the user <NUM> will be looking in terms of an individual subset (e.g., a row) or a group of contiguous subsets (e.g., a group of contiguous rows) of the pixels of the display panel(s) <NUM>. For example, eye tracking system <NUM> may integrate information from past measurements, measurements identifying a position of a user's <NUM> head, and 3D information describing a scene presented by display panel(s) <NUM>. Thus, information for the position and orientation of the user's <NUM> eyes is used to determine the gaze point in a virtual scene presented by HMD <NUM> where the user <NUM> is looking, and to map that gaze point to a location(s) on the display panel(s) <NUM> of the HMD <NUM>.

The HMD <NUM> may further include the aforementioned head tracking system <NUM>. The head tracking system <NUM> may leverage one or more of the sensor <NUM> to track head motion, including head rotation, of the user <NUM>, as described above. For example, the head tracking system <NUM> can track up to six degrees of freedom of the HMD <NUM> (i.e., 3D position, roll, pitch, and yaw). These calculations can be made at every frame of a series of frames so that the application <NUM> can determine how to render a scene in the next frame in accordance with the head position and orientation. In some embodiments, the head tracking system <NUM> is configured to generate head tracking data <NUM> that is usable to predict a future pose (position and/or orientation) of the HMD <NUM> based on current and/or past data, and/or based on the known/implied scan out latency of the individual subsets of pixels in a display system. This is because the application <NUM> is asked to render a frame before the user <NUM> actually sees the light (and, hence, the image) on the display panel(s) <NUM>. Accordingly, a next frame can be rendered based on this future prediction of head position and/or orientation that was made at an earlier point in time. Rotation data provided by the head tracking system <NUM> can be used to determine both direction of HMD <NUM> rotation, and amount of HMD <NUM> rotation in any suitable unit of measurement. For example, rotational direction may be simplified and output in terms of positive or negative horizontal and positive or negative vertical directions, which correspond to left, right, up, and down. Amount of rotation may be in terms of degrees, radians, etc. Angular velocity may be calculated to determine a rate of rotation of the HMD <NUM>.

In the illustrated implementation, the host computer includes the aforementioned processor(s) <NUM>, which may include one or more GPUs <NUM>, as well as the memory <NUM> storing the application(s) <NUM> and render component <NUM> that are executable by the processor(s) <NUM>, and the communications interface(s) <NUM>.

The memory <NUM> may further include an operating system <NUM> configured to manage hardware within and coupled to the host computer <NUM> for the benefit of other modules. The host computer <NUM> may also have a video game client <NUM> installed in the memory <NUM>. The video game client <NUM> may represent an executable client application that is configured to launch and execute programs, such as video games (or video game programs). In other words, the video game client <NUM> may include gaming software that is usable to play video games on the system that includes the HMD <NUM> and the host computer <NUM>. With the video game client <NUM> installed, the host computer <NUM> may then have the ability to receive (e.g., download, stream, etc.) video games from a remote system over the computer network (e.g., the Internet), and to execute the video games via the video game client <NUM>. Any type of content-distribution model can be utilized for this purpose, such as a direct purchase model where video games are individually purchasable for download and execution on a host computer <NUM>, a subscription-based model, a content-distribution model where video games are rented or leased for a period of time, and so on. Accordingly, the host computer <NUM> may include one or more video games, such as the video game <NUM>(<NUM>), within a video game library <NUM>. These video games may be retrieved and executed by loading the video game client <NUM>. In an example, a user <NUM> may choose to play one of multiple video games they have purchased and downloaded to the video game library <NUM> by loading the video game client <NUM> and selecting a video game <NUM>(<NUM>) to start execution of the video game <NUM>(<NUM>). The video game client <NUM> may allow users to login to a video game service using credentials (e.g., a user account, password, etc.).

The host computer <NUM> may further include a communications interface(s) <NUM> including, without limitation, a wireless unit <NUM> coupled to an antenna <NUM> to facilitate a wireless connection to a network and/or to a second device, such as the HMD <NUM>. The wireless unit <NUM> may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that the host computer <NUM> may further include physical ports to facilitate a wired connection to a network and/or a second device, such as the HMD <NUM>.

Claim 1:
A head-mounted display, HMD, (<NUM>) comprising:
one or more display panels (<NUM>) having an array of light emitting elements;
a head tracking system (<NUM>);
a processor (<NUM>); and
memory (<NUM>) storing computer-executable instructions that, when executed by the processor, cause the HMD to:
send, to a host computer (<NUM>) that is communicatively coupled to the HMD, first head tracking data (<NUM>) generated by the head tracking system;
receive, from the host computer, and based at least in part on the first head tracking data:
pixel data (<NUM>) associated with a first frame rendered at a frame rate by an application executing on the host computer; and
extra data (<NUM>) in addition to the pixel data, the extra data including:
pose data (<NUM>(<NUM>)) indicative of a predicted pose of the HMD that was used by the application to generate the pixel data; and
depth data (<NUM>(<NUM>)) associated with the first frame;
determine, based at least in part on second head tracking data (<NUM>) generated by the head tracking system, an updated pose that the HMD will be in at a time at which the light emitting elements will illuminate for the first frame;
classify, based at least in part on the depth data, the pixel data as foreground pixels (<NUM>) and background pixels (<NUM>);
apply, based at least in part on the depth data and a comparison between the predicted pose and the updated pose, re-projection adjustments to the pixel data to obtain modified pixel data associated with the first frame,
wherein applying the re-projection adjustments to the pixel data based at least in part on the comparison between the predicted pose and the updated pose comprises compensating for a difference between the predicted pose and the updated pose by shifting and/or rotating the pixel data, and
wherein applying the re-projection adjustments to the pixel data based at least in part on the depth data comprises:
modifying the foreground pixels based at least in part on a first rate at which the re-projection adjustments are to be applied to the foreground pixels; and
modifying the background pixels based at least in part on a second rate at which the re-projection adjustments are to be applied to the background pixels, the second rate being less than the first rate and less than the frame rate; and
present a first image on the one or more display panels based at least in part on the modified pixel data.