Patent Description:
Head-mounted displays (HMDs) can be worn by users for purposes of immersing the users in a virtual reality (VR) environment (e.g., a VR game) or an augmented reality (AR) environment. A HMD operates by displaying images on one or more display panels based on frames that are output by an application (e.g., a video game). These images are viewed by a user through the optics (e.g., lenses) that are included in the HMD, making the user perceive the images as if the user was immersed in a VR or AR environment.

Some HMDs employ a single display panel, while others employ a pair of display panels. One reason for using a single display panel is that it is cheaper to implement (from a cost of materials and/or manufacturing standpoint) than a pair of display panels. However, using two display panels provides several advantages, such as increased display bandwidth, the ability to utilize more of the available pixels, and allowing users to adjust the distance between the two display panels, among other advantages.

The HMDs on the market today all have one thing in common: they have one or more display panels in an upright panel orientation. For example, HMDs with two display panels orient the display panels directly in front of the user's face in a coplanar, side-by-side arrangement, with their respective side edges parallel to each other. This type of upright panel orientation suffers from several drawbacks. For example, panel artifacts, such as stripes or bands running the length of the panels, often manifest on the upright-oriented panels along with the displayed images. These panel artifacts are visually distracting to the viewing user. In addition, the field of view (FOV) available to the user is limited to, or by, the width of the display panels, which are typically rectangular in shape. In addition, rudimentary filters (e.g., box filters) that are used to resample upright-oriented images often introduce sampling artifacts (e.g., jagged lines) along the verticals and horizontals of the displayed images. These sampling artifacts are also quite visually distracting to the viewing user, especially when coupled with the above-mentioned panel artifacts.

Provided herein are technical solutions to improve and enhance these and other systems.

<CIT> discloses a head mounted display device that comprises image display elements which may be rhombus-shaped.

"Leap Motion Project North Star - Mechanical Guide" from https://leapmotion. io/ProjectNorthStar/mechanical. html is a mechanical guide for constructing an augmented reality headset having two rotated displays.

<CIT> discloses a helmet-mounted display for pilot training and/or gaming using counter-rotated displays, wherein the virtual images may be electronically de-rotated, such that they appear properly oriented.

The detailed description is described with reference to the accompanying drawings.

Described herein are, among other things, a head-mounted display (HMD) system that includes a HMD having a housing and a pair of display panels mounted within the housing, the display panels being counterrotated in orientation. The pair of display panels include a first display panel (e.g., a left display panel configured to be viewed by a left eye of a user wearing the HMD) and a second display (e.g., a right display panel configured to be viewed by a right eye of the user wearing the HMD). The first display panel is oriented in a first orientation by the first display panel being rotated, relative to an upright panel orientation, in a first (e.g., clockwise) direction about a first axis that is orthogonal to a frontal plane of the first display panel, while the second display panel is oriented in a second orientation by the second display panel being rotated, relative to the upright panel orientation, in second (e.g., counterclockwise) direction opposite the first direction about a second axis that is orthogonal to a frontal plane of the second display panel. In a configuration where the display panels are coplanar and adjacent to each other, a side edge of the first display panel and a side edge of the second display panel are not parallel (e.g., the respective side edges of the pair of display panels form an angle between <NUM> and <NUM> degrees, exclusive).

Also described herein are techniques for providing camera pose data with counterrotated camera orientations to an executing application (e.g., a video game application), and resampling the frames received from the application, with or without rotational adjustments, depending on whether the display panels of the HMD are upright-oriented or counterrotated in orientation. For example, computer-readable instructions (e.g., a compositor) of the HMD system that are executed for resampling frames that are output by an application (e.g., a video game application) may send, to the application, a first rotated camera orientation of a first virtual camera (e.g., a left virtual camera) that is rotated in a first (e.g., counterclockwise) direction, and a second rotated camera orientation of a second virtual camera (e.g., a right virtual camera) that is rotated in a second (e.g., clockwise) direction opposite the first direction. The compositor may then receive a frame output by the application in accordance with the counterrotated camera orientations. This counterrotated camera orientation approach can be used with upright-oriented display panels of the HMD, in which case, the compositor resamples the frame by rotating first and second subsets of the pixel data of the frame in opposite directions to that of the rotated camera orientations to display a pair of upright-oriented images on the upright-oriented display panels. This means that the disclosed techniques for providing counterrotated camera orientations to the executing application may be performed independently of counterrotating the display panels of the HMD.

In embodiments of the claimed invention, a combined approach is implemented in a HMD system that includes a HMD with a pair of display panels counterrotated in orientation, whereby the compositor provides camera pose data with counterrotated camera orientations to an executing application (e.g., a video game application), and the compositor resamples the frame received from the application without making rotational adjustments based on the camera pose data or based on the orientations of the display panels. In this manner, upright-oriented images are displayed on the counterrotated display panels without having to perform rotational adjustments (based on the camera pose data or the orientations of the display panels) in the resampling step. This combined approach provides several benefits, many related to improving the display performance of the HMD. For example, counterrotating the display panels of the HMD mitigates panel artifacts, such as stripes or bands running the length of the panel in the verticals and horizontals of the panel, by camouflaging the panel artifacts. That is, the panel artifacts that manifest on the individual display panels still exist, but because the display panels are rotated, they are no longer visible to both eyes of the user at the same locations on the display. As a consequence, the panel artifacts are made inconspicuous to the viewing user so that they are no longer visually distracting. Counterrotated display panels also offer a wider field of view (FOV) to the viewing user, as compared to upright-oriented display panels, and they also allow for decreasing the depth dimension of the HMD by enabling the manufacturer of the HMD (and/or the user) to move the display panels closer to the user's face, which can provide a more ergonomic HMD that is smaller, lighter in weight, and more comfortable to wear. Moreover, in embodiments where the compositor of the HMD is programmed to send camera pose data with counterrotated camera orientations to an executing application, sampling artifacts (e.g., jagged lines) are also mitigated. This is because the filters (e.g., box filters) that are used to resample counterrotated images no longer cause the sampling artifacts that they would otherwise cause if upright-oriented images were to be resampled using the same filters, thereby providing smoother-looking lines in the displayed images.

It is to be appreciated that this disclosure describes systems (e.g., a HMD system) configured to implement the techniques and processes disclosed herein. In some embodiments, the HMD includes a pair of display panels that are counterrotated in orientation. In other examples, which do not fall within the scope of the claims, the HMD may include upright-oriented display panels, in which case counterrotated camera orientations are provided to an executing application by the compositor of the HMD in the process of rendering frames. Also disclosed herein are non-transitory computer-readable media storing computer-executable instructions to implement the techniques and processes disclosed herein. Although many of the examples below are described in the context of video game applications, and specifically VR gaming applications, it is to be appreciated that the techniques and systems described herein may provide benefits with other applications, including, without limitation, non-VR applications (e.g., AR applications), and/or non-gaming applications, such as industrial machine applications, defense applications, robotics applications, and the like.

<FIG> is a diagram illustrating an example head-mounted display (HMD) <NUM>, while being worn by a user <NUM>. The HMD <NUM> in the example of <FIG> is shown as including a pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>) that are counterrotated in orientation. Each a display panel <NUM>, a lens tube <NUM>, and a lens <NUM> make up a lens-and-display assembly. As such, the HMD <NUM> may include a pair of lens-and-display assemblies including a first lens-and-display assembly comprised of a first display <NUM>(<NUM>), a first lens tube <NUM>(<NUM>), and a first lens <NUM>(<NUM>), and a second lens-and-display assembly comprised of a second display <NUM>(<NUM>), a second lens tube <NUM>(<NUM>), and a second lens <NUM>(<NUM>). The first lens-and-display assembly may correspond to the user's <NUM> left eye, while the second lens-and-display assembly may correspond to the user's <NUM> right eye. Each lens-and-display assembly is aligned on its own primary optical axis, labeled as the Z-axes in <FIG>. Each Z-axis (associated with each lens-and-display assembly) is orthogonal to a frontal plane of the display panel <NUM>. Reference planes, such as the aforementioned frontal plane, are discussed in more detail below with reference to <FIG>.

The pair of lens-and-display assemblies shown in <FIG> may be mounted within a housing <NUM> of the HMD <NUM>. An outer shell of the housing <NUM> is shown in <FIG>, but it is to be appreciated that the housing <NUM> may include various components and/or structures that are used to mount various electronic components and printed circuit boards within the housing <NUM>. The pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>) (collectively <NUM>) and the pair of lenses <NUM>(<NUM>) and <NUM>(<NUM>) (collectively <NUM>) may be mounted to opposite ends of the lens tubes <NUM>(<NUM>) and <NUM>(<NUM>) (collectively <NUM>), respectively, using any suitable attachment mechanism (e.g., screws, adhesive, latches, pins, etc.). Furthermore, the lens-and-display assemblies may be mounted to a midframe within the housing <NUM> that abuts the back surface of the display panels <NUM>. This midframe can house other components of the HMD, such as a motherboard with various sensors, processors, etc. The housing <NUM> may further include a shrouded (or hooded) midframe that attaches to the lens-and-display assemblies with apertures for the lenses <NUM>. This shrouded midframe may be configured to block ambient light from entering the space between the user's eyes and the lenses <NUM>(<NUM>) and <NUM>(<NUM>) (collectively <NUM>) when the user <NUM> is wearing the HMD <NUM>.

In general, the HMD <NUM> may be worn by the user <NUM> for purposes of immersing the user in a virtual reality (VR) environment (e.g., a VR game) and/or an augmented reality (AR) environment. Accordingly, the HMD <NUM> may, in some examples, represent a VR headset for use in VR systems, such as for use with a VR gaming system. However, the HMD <NUM> may additionally, or alternatively, be implemented as an AR headset for use in AR applications. In AR, a user <NUM> sees virtual objects overlaid on a real-world environment, whereas, in VR, the user <NUM> does not see a real-world environment, but is fully immersed in a virtual environment, as perceived via the display panels <NUM> and the optics (e.g., lenses <NUM>) of the HMD <NUM>. Examples described herein pertain primarily to a VR-based HMD <NUM>, but it is to be appreciated that the HMD <NUM> is not limited to implementations in VR applications.

The pair of display panels <NUM> are configured to display images based on a series of image frames (herein referred to as "frames") that are output by an application (e.g., a video game). These images are viewed by the user <NUM> through the lenses <NUM>, which may include various types of optics to provide a near-to-eye display. In this manner, the user <NUM> perceives the images as if the user <NUM> was immersed in a VR or AR environment. A compositor of the HMD <NUM> may utilize various techniques to resample the frames that are output by the application, such as performing chromatic distortion, re-projection, and so on. In some examples, which do not fall within the scope of the claims, the compositor may resample the frames by counterrotating subsets of pixel data of the individual frames (in clockwise and counterclockwise directions, respectively) to ensure that images are displayed in an upright-orientation on the display panels <NUM>, as will be described in more detail below. Regardless of whether the resampling step includes rotational adjustments based on the camera pose data and/or the orientations of the display panels, the modified pixel data of the resampled frames may be output to a frame buffer for displaying images on the display panels <NUM>. For instance, the frame buffer used by the HMD <NUM> may be a stereo frame buffer that renders a pair of images on the pair of display panels. For example, pixel data for <NUM> x <NUM> pixels can be output to the frame buffer for display on the pair of display panels <NUM> of the HMD <NUM> (e.g., <NUM> x <NUM> pixels per display panel <NUM>).

The display panels <NUM> may utilize any suitable type of display technology, such as an emissive display that utilizes light emitting elements to emit light during presentation of frames on the display panel(s) <NUM> of the HMD <NUM>. As an example, the display panels <NUM> of the HMD <NUM> may comprise liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, inorganic light emitting diode (ILED) displays, or any other suitable type of display that utilizes a suitable display technology used for HMD <NUM> applications. In some embodiments, the display system of the HMD <NUM> may be a low-persistence display system, meaning that the light emitting elements emit light for a small fraction of a frame time (e.g., roughly <NUM> millisecond (ms) out of a frame time of <NUM>) for each frame that is used to render an image on the display panels.

<FIG> is a diagram illustrating a front view and a side view of a display panel <NUM> in an upright panel orientation <NUM>. The panel <NUM> in <FIG> may represent either one of the display panels <NUM>(<NUM>) or <NUM>(<NUM>), but with the upright panel orientation <NUM>. <FIG> also shows three reference planes - a frontal plane, a midsagittal plane, and a transverse plane - which are referenced herein to describe how each display panel <NUM> can be rotated and/or canted into a particular orientation within the housing <NUM> of the HMD <NUM>.

As shown in <FIG>, the frontal plane of the display panel <NUM> is parallel to a front and back surface of the display panel <NUM>. The front surface of the display panel <NUM> is the surface of the display panel <NUM> that a user <NUM> looks at during image presentation. The frontal plane can bisect the display panel <NUM> into a front half and a back half, the front half having the front (viewing) surface of the display panel <NUM>. Furthermore, an axis that is orthogonal to the frontal plane of the display panel <NUM> is labeled as the Z-axis in <FIG>. This Z-axis can, in some configurations, correspond to a primary optical axis along which light travels from the display panel <NUM> to the user's <NUM> eye. Although positive and negative directions can be defined in any manner, the examples herein are described with reference to a positive Z-direction that is pointing from the display panel <NUM> towards the user <NUM> wearing the HMD <NUM>. Accordingly, the negative Z-direction is considered to be in a direction pointing from the user <NUM> wearing the HMD <NUM> towards the display panel <NUM>.

The midsagittal plane bisects the display panel <NUM> in the vertical direction to create a left half and a right half. An axis that is orthogonal to a midsagittal plane of the display panel <NUM> is labeled as the X-axis in <FIG>. Again, although positive and negative directions can be defined in any manner, the examples herein are described with reference to a positive X-direction that is pointing from the left half of the display panel <NUM> towards the right half of the display panel <NUM> (from the perspective of the user <NUM> looking at the front surface of the display panel <NUM>). Accordingly, the negative X-direction is considered to be in a direction pointing from the right half of the display panel <NUM> towards the left half of the display panel <NUM> (again, from the perspective of the user <NUM> looking at the front surface of the display panel <NUM>).

The transverse plane bisects the display panel <NUM> in the horizontal direction to create a top half and a bottom half. An axis that is orthogonal to a transverse plane of the display panel <NUM> is labeled as the Y-axis in <FIG>. Again, although positive and negative directions can be defined in any manner, the examples herein are described with reference to a positive Y-direction that is pointing from the bottom half of the display panel <NUM> towards the top half of the display panel <NUM>. Accordingly, the negative Y-direction is considered to be in a direction pointing from the top half of the display panel <NUM> towards the bottom half of the display panel <NUM>.

<FIG> is a diagram illustrating a front view <NUM> of a pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>) that are counterrotated in orientation, as well as a top view <NUM> of the pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>) counterrotated in orientation, and an additional top view <NUM> of the pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>) after canting the display panels <NUM>(<NUM>) and <NUM>(<NUM>) in opposite directions.

The front view <NUM> represents a view of the front surfaces of the display panels <NUM>(<NUM>) and <NUM>(<NUM>), the front surfaces being the surfaces on which images are presented via an array of pixels on each display panel <NUM>. Thus, the first display panel <NUM>(<NUM>) may represent a left display panel corresponding to the user's <NUM> left eye, and the second display panel <NUM>(<NUM>) may represent a right display panel corresponding to the user's <NUM> right eye. As shown in <FIG>, the first display panel <NUM>(<NUM>) is oriented in a first orientation by the first display panel <NUM>(<NUM>) being rotated, relative to the upright panel orientation <NUM>, in a first direction (e.g., a clockwise direction) about a first axis (e.g., the Z-axis) that is orthogonal to a frontal plane of the first display panel <NUM>(<NUM>). Meanwhile, the second display panel <NUM>(<NUM>) is oriented in a second orientation by the second display panel <NUM>(<NUM>) being rotated, relative to the upright panel orientation <NUM>, in a second direction (e.g., a counterclockwise direction) opposite the first direction about a second axis (e.g., the Z-axis) that is orthogonal to a frontal plane of the second display panel <NUM>(<NUM>). Thus, the pair of display panels <NUM> are counterrotated in orientation because they are each rotated about their respective Z-axes in opposite directions relative to each other.

In some embodiments, the second display panel <NUM>(<NUM>) is adjacent to the first display panel <NUM>(<NUM>), and vice versa, and the two display panels <NUM>(<NUM>) and <NUM>(<NUM>) are coplanar. The spacing between the display panels <NUM> is configurable and may be based on heuristics associated with interpupillary distances (IPDs) of users. In some embodiments, the innermost points (e.g., adjacent top corners) of the two display panels <NUM> may be touching (i.e., in contact), or they may be separated by a small gap, such as a distance of about <NUM>-<NUM> millimeters (mm). In some embodiments, the display panels <NUM>(<NUM>) and <NUM>(<NUM>) may be disparately located with respect to each other. For instance, instead of positioning the display panels <NUM>(<NUM>) and <NUM>(<NUM>) directly in front of the user's <NUM> eyes, the display panels <NUM> may be located elsewhere in the HMD (e.g., near the user's <NUM> temples, directly in front of the user's <NUM> forehead, etc., and optical components, such as waveguides, mirrors, etc., may be used to transmit the images output on the display panels <NUM> to the user's <NUM> eyes, via the lenses <NUM>. Even in these embodiments where the display panels <NUM>(<NUM>) and <NUM>(<NUM>) are not adjacent to each other in the HMD <NUM>, the display panels <NUM>(<NUM>) and <NUM>(<NUM>) may still be counterrotated by rotating the first display panel <NUM>(<NUM>) in a first direction about an axis orthogonal to its frontal plane, and by rotating the second display panel <NUM>(<NUM>) in a second direction opposite the first direction about an axis orthogonal to its frontal plane.

The position and the orientation of the display panels <NUM> within the housing <NUM> of the HMD may be fixed (i.e., not adjustable), which may allow for calibrating the HMD <NUM> at the manufacturer so that a calibrated HMD <NUM> is provided "out-of-the-box. " However, the housing <NUM> of the HMD <NUM> may include one or more mechanisms to allow a user to adjust the position and/or the orientation of each display panel <NUM> individually, or relative to each other, within the housing <NUM>. For example, a threaded rod may connect the pair of lens-and-display assemblies shown in <FIG> together, and the HMD <NUM> may include a knob, or a similar adjustment mechanism, that a user <NUM> can manipulate to rotate the threaded rod, causing the distance between the pair of display panels <NUM> to increase or decrease, depending on which direction the user <NUM> rotates the knob. Similar mechanisms may be used to adjust the vertical positions of the display panels <NUM> (e.g., moving the display panels <NUM> up or down along the Y-axis), or to adjust the amount of rotation (e.g., about the Z-axis), and/or to adjust the amount by which the display panels <NUM> are canted (e.g., about the X-axis and/or the Y-axis). While this may provide added flexibility to the user <NUM> to adjust the position and/or orientation of the display panels <NUM>, it may create some added difficulties in maintaining the HMD <NUM> in a calibrated state.

The front view <NUM> of <FIG> also shows a configuration where the display panels <NUM> are adjacent to each other and coplanar such that a side edge <NUM>(<NUM>) of the first display panel <NUM>(<NUM>) and a side edge <NUM>(<NUM>) of the second display panel <NUM>(<NUM>) are not parallel. This can be contrasted with a pair of display panels <NUM> that are oriented in the upright orientation <NUM> because the side edges <NUM>(<NUM>) and <NUM>(<NUM>) would be parallel if the pair of display panels <NUM> were each oriented in the upright orientation <NUM>. With counterrotated display panels <NUM>, an angle within the range of <NUM> and <NUM> degrees (exclusive) is formed between the side edges <NUM>(<NUM>) and <NUM>(<NUM>) of the pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>).

The front view <NUM> of <FIG> also shows that the first display panel <NUM>(<NUM>) and the second display panel <NUM>(<NUM>) are each rotated by a substantially equal amount of rotation, θ. Rotating each panel by different amounts of rotation may cause additional problems that degrade the viewing experience of the user. As such, each panel <NUM> is to be rotated by a substantially equal amount of rotation, θ. A "substantially equal amount of rotation," as used herein, means within two degrees. Thus, if the first display panel <NUM>(<NUM>) is rotated <NUM> degrees, and if the second display panel <NUM>(<NUM>) is rotated <NUM> degrees, they are considered, herein, to be rotated by a substantially equal amount of rotation.

The amount of rotation, θ, by which each panel <NUM> is rotated may be within a range of <NUM> to <NUM> degrees, relative to the upright panel orientation <NUM>. Amounts of rotation outside of this range may lessen the extent to which the benefits of a counterrotated panel orientation are perceived by the user <NUM>. In other words, the visually-distracting defects that manifest in an axis-aligned manner (e.g., along the verticals and horizontals of the display panel) may remain noticeable to the user <NUM> if the display panels are rotated by an insignificant amount (e.g., less than <NUM> degrees) or too much (e.g., between <NUM> and <NUM> degrees). In some embodiments, for a pair of display panels <NUM> that are counterrotated in orientation, the amount of rotation, θ, that each display panel <NUM> is rotated may be within a range of <NUM> to <NUM> degrees, within a range of <NUM> to <NUM> degrees, or within a range of <NUM> to <NUM> degrees, relative to the upright panel orientation <NUM>. In some embodiments, the display panels <NUM> may be counterrotated, relative to the upright panel orientation <NUM>, by an amount of rotation, θ, that is equal to <NUM> degrees. A <NUM>-degree rotation may provide the widest FOV <NUM>, as shown in the front view <NUM> of <FIG>, as well as the greatest mitigation of panel artifacts, depending on the pixel layout/pattern of the display panels <NUM>. A common pixel pattern is a pattern where the pixels are arranged in vertical columns and horizontal rows on the display panel. With this type of pixel pattern, a <NUM>-degree rotation may provide the greatest mitigation of panel artifacts. However, other pixel patterns are considered, which do not have the pixels arranged in vertical columns and horizontal rows. For these other types of pixel patterns, different amounts of rotation, θ, may optimally mitigate the panel artifacts. Furthermore, an amount of rotation less between <NUM> and <NUM> degrees may be optimal for following the contour of the user's <NUM> nose <NUM>, which may allow for optimized ergonomic benefits (e.g., moving the display panels <NUM> as close as possible to the user's <NUM> face.

The various benefits that are realized from orienting the display panels <NUM> in a counterrotated panel orientation are now described. One benefit is an increased (e.g., wider) FOV <NUM>. This increased FOV <NUM> can be provided to the viewing user "for free" because pixels that are not utilized in existing HMDs <NUM> with display panels oriented in the upright panel orientation <NUM> can be utilized when the display panels <NUM> when they are counterrotated in orientation, as shown in <FIG>. For example, pixels located in the corners of the display panels may not be utilized when the display panels are orientated in an upright panel orientation <NUM> due to the reverse pincushion distortion caused by lenses, and due to the HMD "cutting off" the corners and not utilizing the pixels in the corners to provide the user with a more aesthetically-pleasing profile of the images. The front view <NUM> of <FIG> illustrates how the FOV <NUM> may be increased to a maximum in the horizontal dimension for rectangular-shaped display panels <NUM>, when those display panels <NUM> are rotated by an amount of rotation, θ, that is equal to <NUM> degrees, relative to the upright panel orientation <NUM>. An increased FOV <NUM> in the horizontal dimension (i.e., the positive and negative X-direction with reference to the upright panel orientation <NUM>) may actually provide a more realistic feeling to a viewing user <NUM> who, in the real world, is able to see with fairly wide-ranging FOV using peripheral vision.

Another of the benefits realized by orienting the display panels <NUM> in a counterrotated panel orientation is the mitigation of panel artifacts (e.g., stripes or bands running the length of the panel) by camouflaging them so that they are no longer visually distracting. To explain, display panels often include defects called panel artifacts, and these panel artifacts often occur in axis-aligned manners (i.e., correlated to horizontals or verticals on the display panel). At least one type of panel artifact - bright and dark stripes or bands vertically running the length of the display - is caused by a display driving scheme called "inversion," which drives the display, frame-by-frame, using an alternating positive and negative voltage waveform. Although the positive voltage and the negative voltage are meant to be applied with an equal amount of voltage, they are typically applied with different voltages in practice. This difference in voltage between the positive and negative cycles of the alternating voltage waveform causes flickering output per column of pixels, every frame, as images are displayed on the display panel. This is visually perceived by a viewing user as vertical stripes (e.g., bright and dark stripes, interleaved), which are unrelated to the content that is being displayed in the images. These stripes are visually distracting to the viewing user. In HMDs, this visual distraction is even more conspicuous due to the unique manner in which the users head and eyes are counterrotating while using the HMD. Moreover, with upright-oriented display panels, both eyes see the bright and dark stripes in the same locations on the displays across frames, causing a visual distraction for the user.

Another example panel artifact is caused by the vertical alignment of red, green, and blue pixels, which manifests with upright-oriented display panels as colored bands of reds, greens, and blues running the length of the upright oriented display panels, which are seen by both eyes at the same locations. This can become noticeable when the user <NUM> is moving his/her head at a precise rate of rotation.

By orienting the display panels <NUM> in the counterrotated panel orientation, the above-mentioned panel artifacts are mitigated. <FIG> shows a first group of pixels <NUM>(<NUM>) on the first display panel <NUM>(<NUM>) and a second group of pixels <NUM>(<NUM>) on the second display panel <NUM>(<NUM>), as oriented when the panels <NUM> are in the counterrotated panel orientation. This counterrotated panel orientation causes the above-mentioned panel defects to be seen differently in each eye, which actually camouflages the panel defects, causing them to go unnoticed by the viewing user <NUM>. This is because the groups of pixels <NUM>(<NUM>) and <NUM>(<NUM>) are no longer aligned in each eye due to their counterrotated orientation, which means that each eye sees a different defect in terms of their respective locations within the scene that is imaged on the counterrotated display panels <NUM>, and the user <NUM> no longer perceives the panel defects in the scene as a consequence. It is recognized that a purely random distribution of red, green, and blue pixels may be optimal for an optimized mitigation of panel defects, but a purely random distribution may be impracticable and/or cost prohibitive. The counterrotated panel orientation is sufficient to cause the pixels to be misaligned, thereby preventing stereo fusion where both eyes see the same pixel defects, located at the same locations on both panels <NUM>, at the same time. Even if some defects align themselves in both of the counterrotated display panels <NUM> across sequential frames, these defects manifest at the per-pixel level, and they go unnoticed by the user <NUM> wearing the HMD <NUM> due to the counterrotated panel orientation. Accordingly, the counterrotated panel orientation shown in <FIG> causes the aforementioned panel defects to be misaligned in terms of the vertical and horizontal direction of the user's <NUM> FOV, and the user <NUM> does not see the same panel defects in both eyes as a consequence, which means that the user's <NUM> vision no longer "snags" on the panel defects, and they remain inconspicuous.

Another benefit realized by orienting the display panels <NUM> in a counterrotated panel orientation is improved ergonomics. This can be seen in the top view <NUM> of <FIG> where the users <NUM> nose <NUM> is able to pass through the frontal plane of each of the coplanar display panels <NUM>. With upright-oriented display panels, there is often insufficient space provided between the display panels, which means that the display panels cannot be brought closer to the user's face past the point where they conflict with the user's nose <NUM>. By orienting the display panels <NUM> in a counterrotated orientation, sufficient space for the user's <NUM> nose <NUM> is provided between the display panels <NUM>, and the display panels <NUM> can thereafter be moved closer to the user's <NUM> face. This means that the depth of the HMD <NUM> can be further reduced, as compared to existing HMDs with upright-orientated display panels directly in front of the user's <NUM> eyes. A smaller, lighter, and closer fitting HMD <NUM> improves the ergonomics of the HMD <NUM> by making the HMD <NUM> more comfortable to wear for longer periods of time, and/or usable by young users (e.g., children) who may be more sensitive to heavier/bulkier HMDs <NUM>.

The additional top view <NUM> of <FIG> shows yet another orientation of the display panels <NUM>(<NUM>) and <NUM>(<NUM>) that can be provided by canting each display panel <NUM> about an axis (e.g., the Y-axis) that is orthogonal to the transverse plane of the upright panel orientation <NUM>. For example, the first display panel <NUM>(<NUM>) can be canted in a first direction (e.g., a counterclockwise direction) about the orthogonal axis to the transverse plane, and the second display panel <NUM>(<NUM>) can be canted in a second direction (e.g., a clockwise direction) opposite the first direction about the orthogonal axis to the transverse plane. In addition, or alternatively, the display panels <NUM> may be canted about a second axis (e.g., the X-axis) that is orthogonal to the midsagittal plane of the upright panel orientation <NUM>.

The processes described herein are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes.

<FIG> is a flow diagram of an example process <NUM> for sending camera pose data with counterrotated camera orientations to an executing application, resampling a frame output by the application without rotational adjustments based on the camera pose data or the orientations of the display panels, and outputting pixel data for the resampled frame to a frame buffer for display on counterrotated display panels. For discussion purposes, the process <NUM> is described with reference to the previous figures.

<FIG> shows an application <NUM>, which may represent a video game application, or any other type of graphics-based application. The application <NUM> may execute on a computing device - such as the HMD <NUM> itself, or a computing device (e.g., a personal computer (PC), game console, etc.) associated with, and coupled to, the HMD <NUM> as part of a HMD system. As such, the application <NUM> may be configured to output a series of frames that are resampled by a compositor <NUM> of the HMD <NUM> (or HMD system), and ultimately displayed as images on the display panel(s) <NUM> of the HMD <NUM>. Before rendering frames, the application <NUM> may receive various types of data from the compositor <NUM>. At least one type of data that the compositor <NUM> sends to the application <NUM> for use by the application <NUM> in rendering frames is camera pose data <NUM>, which includes a set of poses, including position and orientation, of a pair of virtual cameras. In order to provide rotated and/or canted camera orientations to the application <NUM>, the application <NUM> may, in some embodiments, inform the compositor <NUM> of its capabilities (e.g., indicating that it is capable of receiving more complex camera orientations other than upright camera orientations that are not canted). The graphics logic of the HMD <NUM> may be asynchronous, or synchronous. In an asynchronous system, the compositor <NUM> runs separate (on a separate, asynchronous thread) from the application <NUM> on a graphics processing unit(s) (GPU(s)) of the HMD <NUM> (or HMD system).

At <NUM>, the compositor <NUM> of the HMD <NUM> (or HMD system) sends camera pose data <NUM> to an executing application <NUM> that is configured to output a frame of pixel data. A first subset of this pixel data is associated with the first display panel <NUM>(<NUM>), a first image, a first virtual camera, and/or a first eye of the user wearing the HMD <NUM>, while a second subset of this pixel data is associated with the second display panel <NUM>(<NUM>), a second image, a second virtual camera, and/or a second eye of the user wearing the HMD <NUM>. The camera pose data <NUM> informs the application <NUM> regarding how to render a next frame, in a series of frames, in accordance with the camera pose data <NUM>. The operation(s) at block <NUM> may be performed as part of an application programming interface (API) call).

As shown by sub-block <NUM>, the camera pose data <NUM> includes a first rotated camera orientation associated with a first image for the frame that is to be rendered on the first display panel <NUM>(<NUM>), and this first rotated camera orientation specifies (or indicates) that a first virtual camera, of the pair of virtual cameras, is rotated, relative to an upright camera orientation, in a first direction (e.g., a counterclockwise direction). The amount and direction of rotation specified in the first rotated camera orientation is based on the amount and direction of rotation of the first display panel <NUM>(<NUM>) in terms of its rotated panel orientation. That is, if the first display panel <NUM>(<NUM>) is rotated clockwise by <NUM> degrees, relative to the upright panel orientation <NUM>, the first rotated camera orientation may indicate that a first virtual camera is rotated counterclockwise by <NUM> degrees, relative to the upright camera orientation. The first rotated camera orientation may further specify (or indicate) that the first virtual camera is canted, if the first display panel <NUM>(<NUM>) happens to be canted in orientation, as described herein.

As shown by sub-block <NUM>, the camera pose data <NUM> further includes a second rotated camera orientation associated with a second image for the frame that is to be rendered on the second display panel <NUM>(<NUM>), and this second rotated camera orientation specifies (or indicates) that a second virtual camera, of the pair of virtual cameras, is rotated, relative to the upright camera orientation, in a second direction (e.g., a clockwise direction) opposite the first direction. The amount and direction of rotation specified in the second rotated camera orientation is based on the amount and direction of rotation of the second display panel <NUM>(<NUM>) in terms of its rotated panel orientation. That is, if the second display panel <NUM>(<NUM>) is rotated counterclockwise by <NUM> degrees, relative to the upright panel orientation <NUM>, the second rotated camera orientation may indicate that a second virtual camera is rotated clockwise by <NUM> degrees, relative to the upright camera orientation. The second rotated camera orientation may further specify (or indicate) that the second virtual camera is canted, if the second display panel <NUM>(<NUM>) happens to be canted in orientation, as described herein.

At <NUM>, the compositor <NUM> receives, from the application <NUM>, a frame that includes a first image <NUM>(<NUM>) (e.g., a left image) and a second image <NUM>(<NUM>) (e.g., a right image). It is to be appreciated that these images <NUM>(<NUM>) and <NUM>(<NUM>) (collectively <NUM>) may be received, from the application <NUM> and by the compositor <NUM>, as pixel data. Pixel data for each image <NUM> may, in some embodiments, include a two-dimensional array of per-pixel values (e.g., color values). In some embodiments, the pixel data further includes additional data or metadata, such as depth values. In some embodiments, pixel data may include data for each pixel that is represented by a single set of color and alpha values (e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values for one or more alpha channels. As shown in <FIG>, the pair of images <NUM> have been rendered by the application <NUM> in accordance with the camera pose data <NUM>, which included the counterrotated camera orientations. As such, the scenes in the pair of images <NUM> are counterrotated by a corresponding amount of rotation, based on the counterrotated camera orientations. Said another way, the application <NUM> natively renders the counterrotated camera orientations of the virtual cameras.

At <NUM>, the compositor <NUM> may resample the frame without any rotational adjustments based on the camera pose data or the orientations of the display panels. Resampling may involve applying one or more digital transformations to the pixel data of the frame output by the application <NUM> in order to modify/tweak/correct the application-rendered frame before it is output to the frame buffer for presentation on the display panels <NUM>. For example, the compositor <NUM> may resample the frame to obtain a resampled frame by modifying the pixel data of the frame <NUM>(<NUM>) to obtain modified pixel data of the resampled frame. Notably, this modification of the frame's pixel data at block <NUM> does not involve rotational adjustments of the pixel data based on the camera pose data <NUM> or the orientations of the display panels <NUM>. For instance, the resampling operation(s) at block <NUM> may include, without limitation, a correction for lens distortion (e.g., chromatic distortion), applying re-projection transforms to modify the pixel data in accordance with a predicted head pose of the user <NUM>, and/or other mathematical operations that modify the pixel data to create a resampled frame that is improved, or otherwise corrected, from a visual perspective. Although re-projection may involve rotational transforms that effectively rotate the frame about a reference axis, these rotational transforms are based on the predicted head pose of the user <NUM> in re-projection, and are not based on the camera pose data <NUM> pertaining to the virtual cameras or the rotation, relative to upright, of the display panels <NUM>. By not performing rotational adjustments based on the camera pose data <NUM> or the orientations of the display panels <NUM> at block <NUM>, the rendering pipeline is made more efficient, as compared to other embodiments described herein that involve rotational adjustments (in the clockwise/counterclockwise directions) in the resampling step.

At <NUM>, the compositor <NUM> outputs the modified pixel data associated with the resampled frame to a frame buffer so that the resampled frame can be output as a pair of images on the counterrotated display panels <NUM>(<NUM>) and <NUM>(<NUM>) of the HMD <NUM>.

At <NUM>, a pair of images based on the resampled frame may be displayed on the counterrotated display panels <NUM>(<NUM>) and <NUM>(<NUM>) of the HMD <NUM>. As shown in <FIG>, the images presented on the display panels <NUM> are in an upright orientation despite the counterrotated orientation of the display panels <NUM>. This is due to the counterrotated camera orientations that are provided in the camera pose data <NUM> to the application <NUM>. It is to be appreciated that the process <NUM> involves rendering a single frame, and that the process <NUM> may iterate to render multiple sequential frames during execution of the application <NUM>.

An example additional benefit provided from using counterrotated camera orientations, as is the case in the process <NUM>, is the mitigation of sampling artifacts (e.g., jagged lines) in the displayed images. To explain, the application <NUM> may, in the process of rendering a frame including images <NUM>(<NUM>) and <NUM>(<NUM>), perform various sampling steps itself, such as super-sampling, antialiasing (e.g., resolving the multisample antialiasing (MSAA) buffer from <NUM> samples (4x4) per pixel to <NUM> sample per pixel), etc. These sampling steps in the application <NUM> introduce sampling artifacts that tend to align themselves on the verticals and horizontals of the frame. In addition, because the compositor <NUM> typically has somewhere between <NUM> to <NUM> to resample the frame received from the application <NUM>, fast filters (e.g., box filters, linear filters, etc.) are used to speed up the resampling operations. These types of filters often have axis-aligned properties that cause sampling artifacts to manifest in in resampled frame that are also axis aligned (in the verticals and horizontals of the frame). Because vertically-aligned objects and horizontally-aligned objects are typically ubiquitous in virtual reality environments, the resulting images, when upright camera orientations are used for the virtual cameras, often include many sampling artifacts along the verticals and horizontals of the image, such as jagged horizontal lines and jagged vertical lines that look like "stair steps" in the displayed image. This can be quite visually distracting to the viewing user, especially when these jagged lines come in-and-out of view. By contrast, lines that run diagonally across the image often look smooth, with no discernable jagged appearance. By specifying counterrotated camera orientations in the camera pose data <NUM> that is provided to the application <NUM> for rendering a frame, as described in the process <NUM>, for example, these sampling artifacts are mitigated because the they end up at non-vertical and non-horizontal angles in the final images, making the objects in the images more aesthetically-pleasing to look at.

<FIG> is a diagram illustrating a view <NUM> of displayed images, as seen from the perspective of a viewing user <NUM> through the lenses <NUM> of the HMD <NUM>, in accordance with embodiments disclosed herein. The user <NUM> reaps the benefits described herein, many of which relate to improved display performance (e.g., mitigated panel artifacts, mitigated sampling artifacts, increased FOV, etc.), without being able to tell that the display panels <NUM> are not in an upright panel orientation <NUM>.

<FIG> is a flow diagram of an example process <NUM> that does not fall within the scope of the claims. The process <NUM> is for sending camera pose data with upright camera orientations to an executing application, resampling a frame output by the application with rotational adjustments in the clockwise/counterclockwise directions, and outputting pixel data for the resampled frame to a frame buffer for display on counterrotated display panels. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, the compositor <NUM> of the HMD <NUM> (or HMD system) may send camera pose data <NUM> to an executing application <NUM> that is configured to output a frame of pixel data. A first subset of this pixel data is associated with the first display panel <NUM>(<NUM>), a first image, a first virtual camera, and/or a first eye of the user wearing the HMD <NUM>, while a second subset of this pixel data is associated with the second display panel <NUM>(<NUM>), a second image, a second virtual camera, and/or a second eye of the user wearing the HMD <NUM>. The camera pose data <NUM> informs the application <NUM> regarding how to render a next frame, in a series of frames, in accordance with the camera pose data <NUM>. The operation(s) at block <NUM> may be performed as part of an API call. In some examples, the compositor <NUM> may further request the application <NUM> to render additional pixels to those that it would typically ask the application <NUM> to render if the display panels were upright-oriented. This is because the images of the frame will be counterrotated in the resampling operation(s) at block <NUM>, below, and the additional pixels can be used to fill in the gaps that result from rotating the images in orientation during resampling.

As shown by sub-block <NUM>, the camera pose data <NUM> may include a first upright camera orientation associated with a first image for the frame that is to be rendered on the first display panel <NUM>(<NUM>), and this first upright camera orientation specifies (or indicates) that a first virtual camera, of the pair of virtual cameras, is in an upright camera orientation. The first upright camera orientation may further specify (or indicate) that the first virtual camera is canted, if the first display panel <NUM>(<NUM>) happens to be canted in orientation, as described herein.

As shown by sub-block <NUM>, the camera pose data <NUM> may further include a second upright camera orientation associated with a second image for the frame that is to be rendered on the second display panel <NUM>(<NUM>), and this second upright camera orientation specifies (or indicates) that a second virtual camera, of the pair of virtual cameras, is in the upright camera orientation. The second upright camera orientation may further specify (or indicate) that the second virtual camera is canted, if the second display panel <NUM>(<NUM>) happens to be canted in orientation, as described herein.

At <NUM>, the compositor <NUM> may receive, from the application <NUM>, a frame that includes a first image <NUM>(<NUM>) (e.g., a left image) and a second image <NUM>(<NUM>) (e.g., a right image). It is to be appreciated that these images <NUM>(<NUM>) and <NUM>(<NUM>) (collectively <NUM>) may be received, from the application <NUM> and by the compositor <NUM>, as pixel data. Moreover, as shown in <FIG>, the pair of images <NUM> have been rendered by the application <NUM> in accordance with the camera pose data <NUM>, which included upright camera orientations. As such, the scenes in the pair of images <NUM> are upright, and are not rotated, as was the case in the process <NUM>.

At <NUM>, the compositor <NUM> may resample the frame by counterrotating the images <NUM> of the frame. For example, the compositor <NUM> may resample the frame by rotating a first subset of pixel data of the frame associated with the first virtual camera in a first direction (e.g., a counterclockwise direction) to obtain a first subset of modified pixel data of a resampled frame that corresponds to a first rotated image <NUM>(<NUM>), and by rotating a second subset of the pixel data of the frame associated with the second virtual camera in a second direction (e.g., a clockwise direction) opposite the first direction to obtain a second subset of the modified pixel data of the resampled frame that corresponds to a second rotated image <NUM>(<NUM>). These rotational adjustments may utilize an X-Y mapping of the pixels, and may perform mathematical calculations to move each pixel from a starting coordinate (X1,Y1) to a target coordinate (X2,Y2). The resampling operation(s) at block <NUM> may include additional digital transformations such as, without limitation, a correction for lens distortion (e.g., chromatic distortion), applying re-projection transforms to modify the pixel data in accordance with a predicted head pose of the user <NUM>, etc..

At <NUM>, the compositor <NUM> may output the modified pixel data associated with the resampled frame to a frame buffer so that the resampled frame can be output as a pair of images on the counterrotated display panels <NUM>(<NUM>) and <NUM>(<NUM>) of the HMD <NUM>.

At <NUM>, the pair of rotated images <NUM> of the resampled frame may be displayed on the counterrotated display panels <NUM>(<NUM>) and <NUM>(<NUM>) of the HMD <NUM>. As shown in <FIG>, the images presented on the display panels <NUM> are in an upright orientation despite the counterrotated orientation of the display panels <NUM>. This is due to the rotational adjustments in the resampling operation(s) at block <NUM> that compensate for the counterrotated camera orientations that are provided in the camera pose data <NUM> to the application <NUM>. It is to be appreciated that the process <NUM> involves rendering a single frame, and that the process <NUM> may iterate to render multiple sequential frames during execution of the application <NUM>. Although the sampling artifacts described herein may manifest themselves in the images displayed in the example of <FIG>, the counterrotated orientation of the display panels <NUM> provides the remaining benefits described herein, such as mitigation of panel artifacts, increased FOV, and improved ergonomics.

<FIG> is a diagram illustrating a front view <NUM> of a pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>) in upright panel orientations <NUM>, as well as a top view <NUM> of the pair of the display panels <NUM>(<NUM>) and <NUM>(<NUM>) in the upright panel orientations <NUM>, and an additional top view <NUM> of the pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>) after canting the display panels. The orientations of the display panels <NUM>(<NUM>) and <NUM>(<NUM>) in <FIG> is presented in the context of the example described in <FIG>, below. That is to say, the display panels <NUM> are not counterrotated in this example, which does not fall within the scope of the claims, and are, instead oriented in the upright panel orientation <NUM>. In some examples, the display panels <NUM> may be canted, such as by canting each display panel <NUM> about an axis (e.g., the Y-axis) that is orthogonal to the transverse plane of the upright panel orientation <NUM>. For example, the first display panel <NUM>(<NUM>) can be canted in a first direction (e.g., a counterclockwise direction) about the orthogonal axis to the transverse plane, and the second display panel <NUM>(<NUM>) can be canted in a second direction (e.g., a clockwise direction) opposite the first direction about the orthogonal axis to the transverse plane. In addition, or alternatively, the display panels <NUM> may be canted about a second axis (e.g., the X-axis) that is orthogonal to the midsagittal plane of the upright panel orientation <NUM>.

<FIG> is a flow diagram of an example process <NUM> that does not fall within the scope of the claims. The process <NUM> is for sending camera pose data with counterrotated camera orientations to an executing application, resampling a frame output by the application with rotational adjustments in the clockwise/counterclockwise directions, and outputting pixel data for the resampled frame to a frame buffer for display on upright-oriented display panels, such as the display panels shown in <FIG>. For discussion purposes, the process <NUM> is described with reference to the previous figures.

At <NUM>, the compositor <NUM> of the HMD <NUM> (or HMD system) may send camera pose data <NUM> to an executing application <NUM> that is configured to output a frame of pixel data. A first subset of this pixel data is associated with the first display panel <NUM>(<NUM>), a first image, a first virtual camera, and/or a first eye of the user wearing the HMD <NUM>, while a second subset of this pixel data is associated with the second display panel <NUM>(<NUM>), a second image, a second virtual camera, and/or a second eye of the user wearing the HMD <NUM>. The camera pose data <NUM> informs the application <NUM> regarding how to render a next frame, in a series of frames, in accordance with the camera pose data <NUM>. The operation(s) at block <NUM> may be performed as part of an API call. In some examples, the compositor <NUM> may further request the application <NUM> to render additional pixels to those that it would typically ask the application <NUM> to render if the compositor <NUM> were to send upright camera orientations in the camera pose data <NUM>. This is because the images of the frame will be counterrotated in the resampling operation(s) at block <NUM>, below, in order to counteract the counterrotated camera orientations in the camera pose data <NUM>, and the additional pixels can be used to fill in the gaps that result from rotating the images in orientation during resampling.

As shown by sub-block <NUM>, the camera pose data <NUM> may include a first rotated camera orientation associated with a first image for the frame that is to be rendered on the first display panel <NUM>(<NUM>), and this first rotated camera orientation specifies (or indicates) that a first virtual camera, of the pair of virtual cameras, is rotated, relative to an upright camera orientation, in a first direction (e.g., a counterclockwise direction). The amount of rotation specified in the first rotated camera orientation may be within a range of <NUM> to <NUM> degrees, within a range of <NUM> to <NUM> degrees, within a range of <NUM> to <NUM> degrees, or within a range of <NUM> to <NUM> degrees, relative to the upright camera orientation. A <NUM>-degree rotation may provide the greatest mitigation of sampling artifacts. The first rotated camera orientation may further specify (or indicate) that the first virtual camera is canted, if the first display panel <NUM>(<NUM>) happens to be canted in orientation, as described herein.

As shown by sub-block <NUM>, the camera pose data <NUM> may further include a second rotated camera orientation associated with a second image for the frame that is to be rendered on the second display panel <NUM>(<NUM>), and this second rotated camera orientation specifies (or indicates) that a second virtual camera, of the pair of virtual cameras, is rotated, relative to the upright camera orientation, in a second direction (e.g., a clockwise direction) opposite the first direction. The amount of rotation specified in the second rotated camera orientation may be substantially equal to the amount of rotation specified in the first rotated camera orientation, and may be within the ranges of rotation specified with reference to sub-block <NUM>. The second rotated camera orientation may further specify (or indicate) that the second virtual camera is canted, if the second display panel <NUM>(<NUM>) happens to be canted in orientation, as described herein.

At <NUM>, the compositor <NUM> may receive, from the application <NUM>, a frame that includes a first image <NUM>(<NUM>) (e.g., a left image) and a second image <NUM>(<NUM>) (e.g., a right image). It is to be appreciated that these images <NUM>(<NUM>) and <NUM>(<NUM>) (collectively <NUM>) may be received, from the application <NUM> and by the compositor <NUM>, as pixel data. Moreover, as shown in <FIG>, the pair of images <NUM> have been rendered by the application <NUM> in accordance with the camera pose data <NUM>, which included the counterrotated camera orientations. As such, the scenes in the pair of images <NUM> are counterrotated by a corresponding amount of rotation, based on the counterrotated camera orientations. Said another way, the application <NUM> natively renders the counterrotated camera orientations of the virtual cameras.

At <NUM>, the compositor <NUM> may resample the frames by counterrotating the images <NUM> of the frame. For example, the compositor <NUM> may resample the frame by rotating a first subset of pixel data of the frame associated with the first virtual camera in a first direction (e.g., a clockwise direction) to obtain a first subset of modified pixel data of a resampled frame that corresponds to a first rotated image <NUM>(<NUM>), and by rotating a second subset of the pixel data of the frame associated with the second virtual camera in a second direction (e.g., a counterclockwise direction) opposite the first direction to obtain a second subset of the modified pixel data of the resampled frame that corresponds to a second rotated image <NUM>(<NUM>). The amount of rotation that the second subset of the pixel data is rotated may be substantially equal to the amount of rotation that the first subset of the pixel data is rotated, and may be within the ranges of rotation specified with reference to sub-block <NUM>. The resampling operation(s) at block <NUM> may include additional digital transformations such as, without limitation, a correction for lens distortion (e.g., chromatic distortion), applying re-projection transforms to modify the pixel data in accordance with a predicted head pose of the user <NUM>, etc..

At <NUM>, the compositor <NUM> may output the modified pixel data associated with the resampled frame to a frame buffer so that the resampled frame can be output as a pair of images on the upright-oriented display panels <NUM>(<NUM>) and <NUM>(<NUM>) of the HMD <NUM>, such as those shown in <FIG>.

At <NUM>, the pair of rotated images <NUM> may be displayed on the upright-oriented display panels <NUM>(<NUM>) and <NUM>(<NUM>) of the HMD <NUM>. As shown in <FIG>, the images presented on the display panels <NUM> are in an upright orientation. This is due to the rotational adjustments in the resampling operation(s) at block <NUM> that compensate for the counterrotated camera orientations that are provided in the camera pose data <NUM> to the application <NUM>. It is to be appreciated that the process <NUM> involves rendering a single frame, and that the process <NUM> may iterate to render multiple sequential frames during execution of the application <NUM>. The process <NUM>, despite its use in a HMD <NUM> having upright-oriented display panels <NUM>, mitigates the sampling artifacts (e.g., jagged lines) described herein to improve the display performance of the HMD <NUM>.

<FIG> illustrates example components of a HMD <NUM> (or a HMD system that includes the HMD <NUM>), such as a VR headset, according to the embodiments and examples disclosed herein may be embedded. The HMD <NUM> may be implemented as a standalone device that is to be worn by a user <NUM> (e.g., on a head of the user <NUM>). In some embodiments, the HMD <NUM> may be head-mountable, such as by allowing a user <NUM> to secure the HMD <NUM> on his/her head using a securing mechanism (e.g., an adjustable band) that is sized to fit around a head of a user <NUM>. In some embodiments, the HMD <NUM> comprises a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display(s). As such, the terms "wearable device", "wearable electronic device", "VR headset", "AR headset", and "head-mounted display (HMD)" may be used interchangeably herein to refer to the device <NUM> of <FIG>. However, it is to be appreciated that these types of devices are merely example of a HMD <NUM>, and it is to be appreciated that the HMD <NUM> may be implemented in a variety of other form factors. It is also to be appreciated that some or all of the components shown in <FIG> may be implemented on the HMD <NUM>. Accordingly, in some embodiments, a subset of the components shown in <FIG> may be implemented on a computing device that is part of the HMD system, but is separate from the HMD <NUM> itself, such as a PC, a game console, or any other suitable computing device.

In the illustrated implementation, the HMD <NUM> includes one or more processors <NUM> and memory <NUM> (e.g., computer-readable media <NUM>). In some implementations, the processors(s) <NUM> may include a central processing unit (CPU)(s), a graphics processing unit (GPU)(s), both CPU(s) and GPU(s), a microprocessor(s), a digital signal processor(s) or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) <NUM> may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems.

The memory <NUM> may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. The memory <NUM> may be implemented as computer-readable storage media ("CRSM"), which may be any available physical media accessible by the processor(s) <NUM> to execute instructions stored on the memory <NUM>. In one basic implementation, CRSM may include random access memory ("RAM") and Flash memory. In other implementations, CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s) <NUM>.

In general, the HMD <NUM> may include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein. The computer-readable media <NUM> is shown as including various modules, such as instruction, datastores, and so forth, which may be configured to execute on the processor(s) <NUM> for carrying out the techniques, functionality, and/or operations described herein. A few example functional modules are shown as stored in the computer-readable media <NUM> and executable on the processor(s) <NUM>, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC), and/or other logic.

An operating system module <NUM> may be configured to manage hardware within and coupled to the HMD <NUM> for the benefit of other modules. In addition, in some instances the HMD <NUM> may include one or more applications <NUM> stored in the memory <NUM> or otherwise accessible to the HMD <NUM>. In this implementation, the application(s) <NUM> includes a video game application(s) <NUM>. However, the HMD <NUM> may include any number or type of applications and is not limited to the specific example shown here. The video game application(s) <NUM> may be configured to initiate gameplay of a video-based, interactive game (e.g., a VR game) that is playable by the user <NUM>, and to output frames to be rendered on the display panels <NUM> of the HMD <NUM>. A compositor <NUM>, in combination with other logic of the HMD <NUM>, may be configured to perform the techniques described herein to provide camera pose data <NUM>/<NUM>/<NUM> to the application <NUM>, resample frames, and output pixel data for the resampled frames to a frame buffer.

Generally, the HMD <NUM> has input devices <NUM> and output devices <NUM>. The input devices <NUM> may include control buttons. In some implementations, one or more microphones may function as input devices <NUM> to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement unit (IMU)) may function as input devices <NUM> to receive gestural input, such as a hand and/or head motion of the user <NUM>. In some embodiments, additional input devices <NUM> may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, and the like. In other embodiments, the HMD <NUM> may omit a keyboard, keypad, or other similar forms of mechanical input. Instead, the HMD <NUM> may be implemented relatively simplistic forms of input device <NUM>, a network interface (wireless or wire-based), power, and processing/memory capabilities. For example, a limited set of one or more input components may be employed (e.g., a dedicated button to initiate a configuration, power on/off, etc.) so that the HMD <NUM> can thereafter be used. In one implementation, the input device(s) <NUM> may include control mechanisms, such as basic volume control button(s) for increasing/decreasing volume, as well as power and reset buttons.

The output devices <NUM> may include the pair of display panels <NUM>(<NUM>) and <NUM>(<NUM>). The output devices <NUM> may further include, without limitation, a light element (e.g., LED), a vibrator to create haptic sensations, a speaker(s) (e.g., headphones), and/or the like. There may also be a simple light element (e.g., LED) to indicate a state such as, for example, when power is on.

The HMD <NUM> may further include a wireless unit <NUM> coupled to an antenna <NUM> to facilitate a wireless connection to a network. The wireless unit <NUM> may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that the HMD <NUM> may further include physical ports to facilitate a wired connection to a network, a connected peripheral device (including a PC, game console, etc.), or a plug-in network device that communicates with other wireless networks, and which may be part of the HMD system.

The HMD <NUM> may further include optical subsystem <NUM> that directs light from the display panels <NUM> to a user's eyes using one or more optical elements. The optical subsystem <NUM> may include various types and combinations of different optical elements, including, without limitations, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and so forth. Each of the lenses <NUM>(<NUM>) and <NUM>(<NUM>) may include some or all of these optical elements. In some embodiments, one or more optical elements in optical subsystem <NUM> may have one or more coatings, such as anti-reflective coatings. Magnification of the image light by optical subsystem <NUM> allows display panels <NUM> to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification of the image light may increase a field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., <NUM>-<NUM> degrees diagonal), and in some cases all, of the user's FOV. Coupled with the counterrotated orientation of the display panels <NUM>(<NUM>) and <NUM>(<NUM>), this FOV may be increased in the horizontal even further. AR applications may have a narrower FOV (e.g., about <NUM> degrees FOV). Optical subsystem <NUM> may be designed to correct one or more optical errors, such as, without limitation, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and so forth. In some embodiments, content provided to the display panels <NUM> for display is pre-distorted, and optical subsystem <NUM> corrects the distortion when it receives image light from display panels <NUM> generated based on the content. In some embodiments, the compositor <NUM> is configured to resample frames in the rendering pipeline to correct for distortion, as described above.

The HMD <NUM> may further include one or more sensors <NUM>, such as sensors used to generate motion, position, and orientation data. These sensors <NUM> may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors. The sensors <NUM> may also include sub-portions of sensors, such as a series of active or passive markers that may be viewed externally by a camera or color sensor in order to generate motion, position, and orientation data. For example, a VR headset may include, on its exterior, multiple markers, such as reflectors or lights (e.g., infrared or visible light) that, when viewed by an external camera or illuminated by a light (e.g., infrared or visible light), may provide one or more points of reference for interpretation by software in order to generate motion, position, and orientation data. The HMD <NUM> may include light sensors that are sensitive to light (e.g., infrared or visible light) that is projected or broadcast by base stations in the environment of the HMD <NUM> in order to generate motion, position, and orientation data.

In an example, the sensor(s) <NUM> may include an inertial measurement unit (IMU) <NUM>. IMU <NUM> may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and/or other sensors suitable for detecting motion, correcting error associated with IMU <NUM>, or some combination thereof. Based on the measurement signals such motion-based sensors, such as the IMU <NUM>, may generate calibration data indicating an estimated position of HMD <NUM> relative to an initial position of HMD <NUM>. For example, multiple accelerometers may measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). IMU <NUM> can, for example, rapidly sample the measurement signals and calculate the estimated position of HMD <NUM> from the sampled data. For example, IMU <NUM> may integrate measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on HMD <NUM>. The reference point is a point that may be used to describe the position of the HMD <NUM>. While the reference point may generally be defined as a point in space, in various embodiments, reference point is defined as a point within HMD <NUM> (e.g., a center of the IMU <NUM>). Alternatively, IMU <NUM> provides the sampled measurement signals to an external console (or other computing device), which determines the calibration data.

The sensors <NUM> may operate at relatively high frequencies in order to provide sensor data at a high rate. For example, sensor data may be generated at a rate of <NUM> (or <NUM> sensor reading every <NUM> millisecond). In this way, one thousand readings are taken per second. When sensors generate this much data at this rate (or at a greater rate), the data set used for predicting motion is quite large, even over relatively short time periods on the order of the tens of milli seconds.

As mentioned, in some embodiments, the sensors <NUM> may include light sensors that are sensitive to light emitted by base stations in the environment of the HMD <NUM> for purposes of tracking position and/or orientation, pose, etc., of the HMD <NUM> in 3D space. The calculation of position and/or orientation may be based on timing characteristics of light pulses and the presence or absence of light detected by the sensors <NUM>.

The HMD <NUM> may further include an eye tracking module <NUM>. A camera or other optical sensor inside HMD <NUM> may capture image information of a user's eyes, and eye tracking module <NUM> may use the captured information to determine interpupillary distance, interocular distance, a three-dimensional (3D) position of each eye relative to HMD <NUM> (e.g., for distortion adjustment purposes), including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye. In one example, infrared light is emitted within HMD <NUM> and reflected from each eye. The reflected light is received or detected by a camera of the HMD <NUM> and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of a user <NUM> can be used by eye tracking module <NUM>. Accordingly, eye tracking module <NUM> may track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes of a user <NUM> to estimate a gaze point (i.e., a 3D location or position in the virtual scene where the user is looking). For example, eye tracking module <NUM> may integrate information from past measurements, measurements identifying a position of a user's <NUM> head, and 3D information describing a scene presented by display panels <NUM>. Thus, information for the position and orientation of the user's <NUM> eyes is used to determine the gaze point in a virtual scene presented by HMD <NUM> where the user <NUM> is looking.

The HMD <NUM> may further include a head tracking module <NUM>. The head tracking module <NUM> may leverage one or more of the sensor <NUM> to track head motion, including head rotation, of the user <NUM>, as described above. For example, the head tracking module <NUM> can track up to six degrees of freedom of the HMD <NUM> (i.e., 3D position, roll, pitch, and yaw). These calculations can be made at every frame (or frame pair) of a series of frames (or frame pairs) so that the application <NUM> can determine how to render a scene in the next frame in accordance with the head position and orientation. In some embodiments, the head tracking module <NUM>, and/or the compositor <NUM> using the head tracking module <NUM>, is configured to predict a future position and/or orientation of the HMD <NUM> based on current and/or past data. This is because the application <NUM> is asked to render a frame (or frame pair) before the user <NUM> actually sees the light (and, hence, the image) on the display panels <NUM>. Accordingly, a next frame (or frame pair) can be rendered based on this future prediction of head position and/or orientation that was made at an earlier point in time, such as roughly <NUM>-<NUM> milliseconds (ms) prior to rendering the frame (or frame pair). Rotation data provided by the head tracking module <NUM> can be used to determine both direction of HMD <NUM> rotation, and amount of HMD <NUM> rotation in any suitable unit of measurement. For example, rotational direction may be simplified and output in terms of positive or negative horizontal and positive or negative vertical directions, which correspond to left, right, up, and down. Amount of rotation may be in terms of degrees, radians, etc. Angular velocity may be calculated to determine a rate of rotation of the HMD <NUM>.

Unless otherwise indicated, all numbers expressing quantities used in the specification and claims are to be understood as being modified in all instances by the term "about. " Accordingly, unless indicated to the contrary, the numerical parameters set forth in the specification and attached claims are approximations that may vary depending upon the desired properties sought to be obtained by the present disclosure. At the very least, and not as an attempt to limit the application of the doctrine of equivalents to the scope of the claims, each numerical parameter should at least be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. When further clarity is required, the term "about" has the meaning reasonably ascribed to it by a person skilled in the art when used in conjunction with a stated numerical value or range, i.e. denoting somewhat more or somewhat less than the stated value or range, to within a range of ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; ±<NUM>% of the stated value; or ±<NUM>% of the stated value.

Claim 1:
A head-mounted display, HMD, system comprising:
a HMD (<NUM>) including:
a housing (<NUM>);
a pair of display panels (<NUM>) mounted within the housing, the pair of display panels including a first display panel and a second display panel, wherein:
the first display panel (<NUM>(<NUM>)) is oriented in a first orientation by the first display panel being rotated, relative to an upright panel orientation, in a clockwise direction about a first axis that is orthogonal to a frontal plane of the first display panel; and
the second display panel (<NUM>(<NUM>)) is oriented in a second orientation by the second display panel being rotated, relative to the upright panel orientation, in a counterclockwise direction about a second axis that is orthogonal to a frontal plane of the second display panel;
the HMD system further comprising:
one or more processors (<NUM>, <NUM>); and
computer-executable instructions stored in memory (<NUM>) of the HMD system and executable by the one or more processors to:
send (<NUM>) camera pose data (<NUM>) relating to a pair of virtual cameras, to an application (<NUM>) that is configured to output a frame, wherein the camera pose data (<NUM>) informs the application (<NUM>) regarding how to render a next frame, in a series of frames, in accordance with the camera pose data (<NUM>), the camera pose data including:
a first rotated camera orientation specifying that a first virtual camera, of the pair of virtual cameras, is rotated, relative to an upright camera orientation, in the counterclockwise direction; and
a second rotated camera orientation specifying that a second virtual camera, of the pair of virtual cameras, is rotated, relative to the upright camera orientation, in the clockwise direction;
receive (<NUM>) the frame from the application;
resample (<NUM>) the frame, without any rotational adjustments to the frame based on the camera pose data or the orientations of the display panels, by modifying pixel data of the frame to obtain modified pixel data of a resampled frame; and
output (<NUM>) the modified pixel data to a frame buffer.