Patent Description:
<CIT> discloses a system for varying effective resolution by screen location in graphics processing by approximating projection of vertices onto curved viewport. <CIT> discloses a graphics pipeline method and apparatus. <NPL> discloses pre-warping for head mounted displays. <CIT> discloses a head mounted display, particularly for motor vehicles.

According to the invention, there is provided a method for generating images at a head mounted display according to claim <NUM>, an apparatus for generating images for a head mounted display according to claim <NUM> and a head mounted display according to claim <NUM>.

Significant processing power is wasted in rendering uniformly spaced pixels in window space because the high pixel resolution needed to generate good quality images at the center of the screen dictates the resolution used to render all the pixels in the window space. However, the image quality at the periphery is not required to be as high as the quality at the center of the screen because the subtended angle of view maps to more than one pixel. For example, higher resolution rendering is necessary to generate images in a high-acuity region that surrounds the central gaze direction (or the center of the lens of an HMD display device) but is not necessary to generate images of sufficient quality in lower-acuity regions in the visual periphery or in the peripheral regions of a lens. The high-acuity region typically includes a portion of the field-of-view that is within some angular distance of the central gaze direction. The angular distance from the central gaze direction is also referred to as the eccentricity. The lower-acuity regions include portions of the field-of-view that are at larger eccentricities. For example, the high-acuity region can include a portion of the field-of-view that is within an eccentricity of <NUM>-<NUM>°, which corresponds to a portion of the field-of-view that projects to a retinal region in the human eye called the fovea.

The power and computing resources that are consumed to generate images for display by an HMD are reduced without compromising perceived image resolution by rendering pixels in window space with a non-uniform pixel spacing. In some embodiments, the non-uniform pixel spacing corresponds to an approximately constant pixel density per subtended angle across a user's field-of-view. For example, the pixel density per subtended angle is relatively high in a fovea region and relatively low on the visual periphery. A scan converter samples the pixels in window space through a first distortion function that maps the non-uniform pixels to uniformly spaced pixels in raster space. The scan converter renders display pixels based on values of the uniformly spaced pixels in raster space and the display pixels are passed through a second distortion function prior to display to the user.

Some embodiments of the first distortion function include a vertical distortion function that maps a first dimension of the non-uniform pixels to the uniformly spaced pixels and a horizontal distortion function that maps a second dimension of the non-uniform pixels to the uniformly spaced pixels. The first and second dimensions are orthogonal to each other. Some embodiments of the vertical and horizontal distortion functions are defined by corresponding sets of displacements from a linear function that maps uniformly spaced pixels in window space to uniformly spaced pixels in raster space. For example, the set of displacements for the vertical and horizontal distortion functions defines a set of points that are joined by a smooth curve that represents the first distortion function. Some embodiments of the vertical and horizontal distortion functions are modified to increase the resolution of pixels in a portion of the window space that corresponds to a location of a fovea region.

<FIG> is a block diagram of a graphics pipeline <NUM> that implements a DX12 application programming interface (API) according to some embodiments. The graphics pipeline <NUM> is capable of processing high-order geometry primitives to generate rasterized images of three-dimensional (<NUM>-D) scenes at a predetermined resolution. The graphics pipeline <NUM> has access to storage resources <NUM> such as a hierarchy of one or more memories or caches that are used to implement buffers and store vertex data, texture data, and the like. Some embodiments of shaders in the graphics pipeline <NUM> implement single-instruction-multiple-data (SIMD) processing so that multiple vertices are processed concurrently. The graphics pipeline <NUM> therefore implements the concept of unified shader model so that the shaders included in the graphics pipeline <NUM> have the same execution platform on the shared SIMD compute units. The shaders are therefore implemented using a common set of resources including one or more processors. The common set of resources is referred to herein as the unified shader pool <NUM>.

In a typical DX12 rendering pipeline, an input assembler <NUM> is configured to access information from the storage resources <NUM> that is used to define objects that represent portions of a model of a scene. A vertex shader <NUM>, which, in some embodiments, is implemented in software, logically receives a single vertex of a primitive as input from the input assembler <NUM> and outputs a single vertex. A hull shader <NUM> operates on input high-order patches or control points that are used to define the input patches. The hull shader <NUM> outputs tessellation factors and other patch data.

Primitives generated by the hull shader <NUM> can optionally be provided to a tessellator <NUM>. The tessellator <NUM> receives objects (such as patches) from the hull shader <NUM> and generates information identifying primitives corresponding to the input object, e.g., by tessellating the input objects based on tessellation factors provided to the tessellator <NUM> by the hull shader <NUM>. Tessellation subdivides input higher-order primitives such as patches into a set of lower-order output primitives that represent finer levels of detail, e.g., as indicated by tessellation factors that specify the granularity of the primitives produced by the tessellation process.

A domain shader <NUM> inputs a domain location and (optionally) other patch data. The domain shader <NUM> operates on the provided information and generates a single vertex for output based on the input domain location and other information. A geometry shader <NUM> receives an input primitive and outputs up to four primitives that are generated by the geometry shader <NUM> based on the input primitive.

The primitives are then mapped from a view window that contains the scene to a grid of pixels that represent the image that will be displayed to a user, e.g., using a window-to-viewport transformation. The term "window space" refers to the pixels that are generated by the graphics pipeline <NUM> up to this point in the processing. As discussed herein, the pixels in window space that are generated by the graphics pipeline <NUM> are non-uniformly distributed and have a non-uniform spacing between the pixels. In some embodiments, the graphics pipeline <NUM> renders the pixels in window space such that a pixel density per subtended area is constant across a user's field of view. For example, the pixels in window space are rendered with a relatively high pixel density in a fovea region of a user wearing an HMD that implements the graphics pipeline <NUM>. The pixels in window space are rendered with a relatively low pixel density in the visual periphery.

A distortion function <NUM> is used to transform the non-uniformly spaced pixels in window space to uniformly spaced pixels in a raster space used by a scan converter <NUM>. Some embodiments of the distortion function <NUM> implement a vertical distortion function that maps a first dimension of the non-uniformly spaced pixels in window space to the uniformly spaced pixels in raster space and a horizontal distortion function that maps a second dimension of the non-uniformly spaced pixels in window space to the uniformly spaced pixels in raster space. The vertical and horizontal distortion functions are defined based on sets of displacements from a linear distortion function that maps uniformly spaced pixels in window space to uniformly spaced pixels in the raster space. the scan converter <NUM> samples pixels in window space through the distortion function <NUM> to generate uniformly spaced pixels in raster space.

A pixel shader <NUM> receives a pixel flow from the scan converter <NUM> and outputs zero or another pixel flow in response to the input pixel flow. An output merger block <NUM> performs blend, depth, stencil, or other operations on pixels received from the pixel shader <NUM>.

<FIG> illustrates a display system <NUM> that includes an electronic device <NUM> configured to provide immersive VR or AR functionality according to some embodiments. The electronic device <NUM> is used to display images using values of pixels that are output from some embodiments of the graphics pipeline <NUM> shown in <FIG>. A back plan view of an example implementation of the electronic device <NUM> in an HMD form factor in accordance with at least one embodiment of the present disclosure is shown in <FIG>. In other embodiments, the electronic device <NUM> is implemented in other form factors, such as a form factor for glasses, a smart phone form factor, tablet form factor, a medical imaging device form factor, a standalone computer, a system-on-a-chip (SOC), and the like, which implement configurations analogous to those illustrated. As illustrated by the back plan view, the electronic device <NUM> can include a face gasket <NUM> mounted on a surface <NUM> for securing the electronic device <NUM> to the face of the user (along with the use of straps or a harness).

The electronic device <NUM> includes a display <NUM> that is used to generate images such as VR images or AR images that are provided to the user. The display <NUM> is divided into two substantially identical portions, a right portion to provide images to the right eye of the user and a left portion to provide images to the left eye of the user. In other embodiments, the display <NUM> is implemented as two different displays, one dedicated to each eye. The visual acuity of a user wearing the electronic device <NUM> depends on a distance from the user's center of gaze, e.g., the eccentricity. For example, the field-of-view for the user's left eye is subdivided into a high-acuity region <NUM> that surrounds a central gaze direction <NUM>. The field-of-view for the user's left eye also includes a low-acuity region <NUM> in the visual periphery. Similarly, the field-of-view for the user's right eye is subdivided into a high-acuity region <NUM> that surrounds a central gaze direction <NUM> and a low-acuity region <NUM> in the visual periphery. The central gaze directions <NUM>, <NUM> are set equal to the center of a current field-of-view or they are determined on the basis of eye tracking measurements that detect the central gaze direction of the user's eyes.

As discussed herein, a distortion function such as the distortion function <NUM> shown in <FIG> is used to map non-uniformly rendered pixels in window space to uniformly rendered pixels in raster space. A graphics pipeline that implements the distortion function is therefore able to render pixels in window space at relatively high resolution within the high-acuity regions <NUM>, <NUM>, e.g., by rendering the pixels at a resolution that is equal to the native resolution supported by the display. The graphics pipeline renders the pixels in window space within the low-acuity regions <NUM>, <NUM> at lower resolutions, thereby reducing the power and computing resources needed to render the pixels in window space. The rendered pixels in window space are subsequently sampled by a scan converter through the distortion function to determine values of uniformly spaced pixels in raster space.

<FIG> illustrates a display system <NUM> that includes an electronic device <NUM> configured to provide AR or VR functionality to a user wearing the electronic device <NUM> via a display according to some embodiments. The electronic device <NUM> is used to implement some embodiments of the electronic device <NUM> shown in <FIG>. The electronic device <NUM> is shown in <FIG> as being mounted on a head <NUM> of a user. As illustrated, the electronic device <NUM> includes a housing <NUM> that includes a display <NUM> that generates an image for presentation to the user. The display <NUM> is implemented using some embodiments of the display <NUM> shown in <FIG>. In the illustrated embodiment, the display <NUM> is formed of a left display <NUM> and a right display <NUM> that are used to display stereoscopic images to corresponding left eye and right eye. However, in other embodiments, the display <NUM> is a single monolithic display <NUM> that generates separate stereoscopic images for display to the left and right eyes. The electronic device <NUM> also includes eyepiece lenses <NUM> and <NUM> disposed in corresponding apertures or other openings in a user-facing surface <NUM> of the housing <NUM>. The display <NUM> is disposed distal to the eyepiece lenses <NUM> and <NUM> within the housing <NUM>. The eyepiece lens <NUM> is aligned with the left eye display <NUM> and the eyepiece lens <NUM> is aligned with the right eye display <NUM>.

In a stereoscopic display mode, imagery is displayed by the left eye display <NUM> and viewed by the user's left eye via the eyepiece lens <NUM>. Imagery is concurrently displayed by the right eye display <NUM> and viewed by the user's right eye via the eyepiece lens <NUM>. The imagery viewed by the left and right eyes is configured to create a stereoscopic view for the user. Some embodiments of the displays <NUM>, <NUM>, <NUM> are fabricated to include a bezel (not shown in <FIG>) that encompasses outer edges of the displays <NUM>, <NUM>, <NUM>. In that case, the lenses <NUM>, <NUM> or other optical devices are used to combine the images produced by the displays <NUM>, <NUM>, <NUM> so that bezels around the displays <NUM>, <NUM>, <NUM> are not seen by the user. Instead, lenses <NUM>, <NUM> merge the images to appear continuous across boundaries between the displays <NUM>, <NUM>, <NUM>.

Some or all of the electronic components that control and support the operation of the display <NUM> and other components of the electronic device <NUM> are implemented within the housing <NUM>. Some embodiments of the electronic device <NUM> include a processing unit such as a processor <NUM> and a memory <NUM> (or other hardware, firmware, or software) that can be used to implement some or all of a graphics pipeline such as the graphics pipeline <NUM> shown in <FIG>. In some embodiments the workload associated with acquiring actual or virtual images and rendering these images for display on the display <NUM> can be shared with external processing units that are implemented outside of the electronic device <NUM>. Some embodiments of the electronic device <NUM> include an eye tracker <NUM> to track movement of the user's eyes and determine a center of gaze for each eye in real-time. The electronic device <NUM> also includes one or more motion sensors <NUM>. Examples of motion sensors <NUM> include accelerometers, gyroscopic orientation detectors, or other devices capable of detecting motion of the electronic device <NUM>.

<FIG> illustrates a mapping <NUM> of non-uniformly spaced pixels in a window space <NUM> to uniformly spaced pixels in a raster space <NUM> according to some embodiments. The mapping <NUM> is implemented in some embodiments of the graphics pipeline <NUM> shown in <FIG>, the electronic device <NUM> shown in <FIG>, and the electronic device <NUM> shown in <FIG>. The pixels in the window space <NUM> are generated by a geometry portion of the graphics pipeline and a scan converter (or rasterizer) determines values of the pixels in the raster space <NUM> by sampling the pixels in the window space <NUM> through a distortion function <NUM>, which is used to implement some embodiments of the distortion function <NUM> shown in <FIG>.

The non-uniformly spaced pixels are distributed throughout the window space <NUM> according to the distortion function <NUM>. In the illustrated embodiment, the distortion function <NUM> is implemented using a vertical distortion function and a horizontal distortion function. Horizontal boundaries of the pixels in the window space <NUM> are separated by decreasing distances moving from the top of the window space <NUM> to the middle of the window space <NUM>, and are then separated by increasing distances moving from the middle of the window space <NUM> to the bottom of the window space <NUM>. Vertical boundaries of the pixels in the window space <NUM> are separated by decreasing distances moving from the left side of the window space <NUM> to the middle of the window space <NUM> and are then separated by increasing distances moving from the middle of the window space <NUM> to the right side of the window space <NUM>. Thus, the highest resolution pixels are near the center of the window space <NUM> and the lowest resolution pixels are near the edges of the window space <NUM>.

The non-uniformly spaced pixels in the window space <NUM> are mapped to the uniformly spaced pixels in the raster space <NUM> by the distortion function <NUM>. For example, the pixel <NUM> in the raster space <NUM> is mapped to the pixel <NUM> in the window space <NUM>. The scan converter therefore performs relatively dense sampling of the rendered pixel <NUM> to determine values of the pixel <NUM>. For another example, the pixel <NUM> in the raster space <NUM> is mapped to the pixel <NUM> in the window space <NUM>. The scan converter therefore performs relatively sparse sampling of the rendered pixel <NUM> to determine values of the pixel <NUM>.

The distortion function <NUM> is therefore able to adjust the sampling density of rasterization in a programmable way, which is used to reduce unnecessary pixel shading for applications such as VR and AR implemented in an HMD. Some embodiments of the distortion function <NUM> are implemented as a state-controlled 2D spatial distortion transformation that is disposed between a post-viewport transformed geometry window space (the window space <NUM>) and the scan converter view of the geometry being rendered, e.g., in the raster space <NUM>. In the illustrated embodiment, horizontal and vertical distortion curves compensate for static characteristics of the optics of the HMD and dynamic characteristics such as the changing gaze of the user that is wearing the HMD. The distortion function <NUM> also supports a dynamic trade-off of image quality versus performance on a frame-by-frame basis. The trade-off is performed by changing the size of a viewport window, a window associated with the distortion function <NUM>, and a post-processing sample window without changing the distortion function <NUM>. Rendered geometry can also be flexibly mapped to portions of a selected render target using the distortion function <NUM>.

<FIG> is a block diagram illustrating a process <NUM> of rendering of display pixels for display to an eye <NUM> of the user wearing an HMD according to some embodiments. The process <NUM> is implemented in some embodiments of the graphics pipeline <NUM> shown in <FIG>, the electronic device <NUM> shown in <FIG>, and the electronic device <NUM> shown in <FIG>.

A geometry pipeline provides rendered pixels in window space <NUM>. The field-of-view of the window space <NUM> is determined by a projection matrix and optical characteristics of the HMD. As discussed herein, the pixels in the window space <NUM> are rendered non-uniformly in accordance with a distortion function <NUM> such as the distortion function <NUM> shown in <FIG> and the distortion function <NUM> shown in <FIG>. Some embodiments of the distortion function <NUM> are implemented in hardware, firmware, software, or a combination thereof. The distortion function <NUM> is configurable based on register values, as discussed below. The pixels in the window space <NUM> are distributed to establish near constant pixel density per subtended angle across the field-of-view of the user wearing the HMD.

A scan converter samples the non-uniform pixels in the window space <NUM> through the distortion function <NUM> to generate values of uniformly spaced pixels in the raster space <NUM>. In some embodiments, postprocessing is performed in the raster space <NUM> by distortion-aware shaders and other filters. The values of the pixels in the raster space <NUM> are then used to generate values of display pixels that are provided to a display buffer <NUM>. In some embodiments, the rendered or reconstructed image is sampled from the raster space <NUM> with a modified mesh to perform chromatic adjustments or other filtering.

Display pixels stored in the display buffer <NUM> are provided to a display <NUM> implemented in the HMD. The display <NUM> uses the values of the display pixels to generate images that are presented to the eye <NUM> using one of a corresponding pair of lenses <NUM>, <NUM>. Implementing the distortion function <NUM> to support rendering of non-uniformly spaced pixels in the window space <NUM> therefore reduces the resources consumed in the geometry pipeline to render pixels in the visual periphery at an excessively high resolution, which enables higher quality rendering at lower resolutions. The scan converter performs nearly <NUM>:<NUM> sampling of the pixels in the window space <NUM> to render the pixels in the raster space <NUM>. A dynamic trade-off between quality and performance is achieved by changing the resolution of the window space <NUM> and the raster space <NUM>.

A geometric transformation between a 2D post viewport window space (such as the window space <NUM>) and a 2D raster view (such as the raster space <NUM>) uniquely and bi-directionally maps any X,Y point in the raster space to a distorted x,y the window space. An example of the mapping is given by: <MAT> where ws refers to the window space, rs refers to the raster space, and F() and F'() define the spatial transformation between the two 2D images. Some embodiments represent the distortion function as a vertical distortion function and a horizontal distortion function. For example, the distortion in the horizontal and vertical components is represented as: <MAT> <MAT> State tables and logic are used to construct Fx, F'x, Fy, F'y for sampling the distortion maps used for each primitive.

The state parameters for a horizontal and vertical directions of each eye and each pipe are listed in the table below. The state defining the distortion curves is stored as configuration state. The enabling state and the state for cooperating with binning is referred to as "render state.

In some embodiments, a control method for defining the distortion function includes defining a window, e.g., using the upper left and lower right coordinates of the window in the window space <NUM>. Values of the state parameters for the horizontal and vertical directions are accessed from a set of lookup tables (LUTs). For example, the system can implement one LUT for each combination of two graphics pipelines, two eyes, and two directions (horizontal and vertical) for a total of eight LUTs. As discussed in detail below, LUT defines an eight region curve with an alpha controlled RCP curve between points used to generate the actual horizontal and vertical distortion curve. Register state values are used to control the distortion process during primitive rasterization.

<FIG> is a set <NUM> of plots that represent a horizontal distortion function <NUM> and a vertical distortion function <NUM> that are used to implement a distortion function according to some embodiments. The horizontal distortion function <NUM> and the vertical distortion function <NUM> are implemented in some embodiments of the distortion function <NUM> shown in <FIG>, the electronic device <NUM> shown in <FIG>, the electronic device <NUM> shown in <FIG>, the distortion function <NUM> shown in <FIG>, and the distortion function <NUM> shown in <FIG>. The horizontal distortion function <NUM> and the vertical distortion function <NUM> are implemented in hardware that is configurable on the basis of state information stored in one or more registers.

For each eye, state data defines the horizontal distortion function <NUM> and the vertical distortion function <NUM> to map coordinates X, Y in a raster space to coordinates x,y in a window space. In some embodiments, the optical center in the vertical direction and the horizontal direction is determined based on the HMD manufacturer specification or application programming interface (API). The distortion function is typically a 2D function and can be queried from the HMD driver via an API. The optical center corresponds to the location where the distance between two pixels sampled in the render target is the smallest. Once the center is found, at the vertical center, a curve representing the horizontal distortion function <NUM> is extracted, and at the horizontal center, a curve representing the vertical distortion function <NUM> is extracted. Once a distortion curve has been established, the system determines the parameters of an eight segment smooth piecewise function that is used to represent the horizontal distortion function <NUM> and the vertical distortion function <NUM> in hardware.

The horizontal distortion function <NUM> and the vertical distortion function <NUM> are used to distort an image or undistort an image, as indicated by the arrows <NUM>, <NUM>, <NUM>, <NUM>.

<FIG> is an illustration <NUM> of a distortion curve <NUM> that represents a horizontal distortion function or a vertical distortion function according to some embodiments. The distortion curve <NUM> implemented in some embodiments of the distortion function <NUM> shown in <FIG>, the electronic device <NUM> shown in <FIG>, the electronic device <NUM> shown in <FIG>, the distortion function <NUM> shown in <FIG>, and the distortion function <NUM> shown in <FIG>. The distortion curve <NUM> is implemented in hardware that is configurable on the basis of state information stored in one or more registers. The distortion curve <NUM> illustrates an example of a representation of a distortion curve that is used to map coordinates from window space to raster space. However, other embodiments of distortion curves are represented using other parameterizations or functional representations of the window-to-raster space mapping.

The distortion curve <NUM> represents a mapping of one coordinate from window space to raster space. The window space coordinate is represented on the vertical axis and the raster space coordinate is represented on the horizontal axis. If the distortion curve <NUM> represents the mapping of an X coordinate, then the distortion curve <NUM> represents a horizontal distortion function. If the distortion curve represents the mapping of a Y coordinate, then the distortion curve <NUM> represents a vertical distortion function. In the interest of clarity, the following discussion assumes that the distortion curve <NUM> represents a horizontal distortion function that maps the X coordinate from the window space to the raster space. If the distortion curve <NUM> is mapped to position points <NUM> (only one indicated by a reference numeral in the interest of clarity) on a diagonal line <NUM>, there is no distortion and an x of window space is equal to an X in raster space.

In the illustrated embodiment, positioning of points <NUM> (only one indicated by a reference numeral in the interest of clarity) on the distortion curve <NUM> are controlled by state data: YPosition, YPositionShift, XPosition, XPositionShift. The position values that represent the locations of the points <NUM> are stored as horizontal and vertical deltas from seven equally spaced points <NUM> on the diagonal <NUM> of the distortion table after a common PositionShift has been applied. However, as discussed above, some embodiments of the distortion curve <NUM> are represented by different numbers of points, which are not necessarily equally spaced. Furthermore, continuous functions or piecewise continuous functions such as splines, polynomials, or other parametric curves are used to identify the locations of the points <NUM> in some embodiments.

The PositionShift values are determined by finding the exponent of the next power of two value that contain the largest position difference from the diagonal. The position values increase both horizontally and vertically, as the function used for the distortion is monotonically increasing. For example, the coordinates of the point <NUM> are determined by corresponding values of XPosition and YPosition. The location of the point <NUM> on the distortion curve <NUM> is then determined relative to the point <NUM> by the values YPositionShift and XPositionShift, which are indicated by the arrows <NUM>, <NUM>, respectively. The alpha values are then used to modify the curvature of the curve between the nearest X,Y positions so the curve passes thru the desired location of the point <NUM>. Once the locations of the points <NUM> are determined, a set of piecewise curves (using alpha values) is selected to create a smooth distortion curve <NUM>.

<FIG> illustrates selection of a piecewise curve <NUM> that is used to form a smooth distortion curve <NUM> according to some embodiments. The distortion curve <NUM> represents some embodiments of the distortion curve <NUM> shown in <FIG>. In the illustrated embodiment, a set of points that determines the distortion curve <NUM> have been configured, e.g., on the basis of positions of points on a diagonal line and corresponding offsets. The plot <NUM> illustrates a set of candidate piecewise curves that are available to be selected as the piecewise curve <NUM> that is used to form the smooth distortion curve <NUM> in the section identified by the bounding box <NUM>. In the illustrated embodiment, the piecewise curve <NUM> has been selected from the set of candidate piecewise curves.

Shapes of the candidate piecewise curves are determined by alpha values that define how the curve travels from the bottom left corner of the bounding box <NUM> to the top right corner of the bounding box <NUM>. The selection of a position value placement uses a search space to find the best set of piecewise curves that create a smooth distortion curve <NUM> to encapsulate the actual optics curve and the diagonal to prevent under-sampling. In one embodiment, points are placed with equal horizontal separation and then potential vertical positions and alpha values are searched, followed by a shift in horizontal spacing and adjustment to other values to find the best curve match. The alpha value that represents the piecewise curve <NUM> is selected using a sampling method. For example, the distortion curve <NUM> is sampled at a midpoint of the segment and the alpha value is determined based on based on the x coordinate value (in window space) at the sampled point of the distortion curve <NUM>. For another example, multiple points on the distortion curve <NUM> are sampled and the alpha value is determined based on an average of the x coordinate values. For yet another example, multiple points on the distortion curve <NUM> are sampled and the alpha value with the smallest squared error sum relative to the sampled x coordinate values is selected.

<FIG> is a set <NUM> of plots that illustrates a fovea modification of a distortion curve <NUM> according to some embodiments. The graph <NUM> illustrates the distortion curve <NUM> and a fovea curve <NUM> that represents a central gaze direction <NUM> and a falloff in acuity with increasing eccentricity from the central gaze direction <NUM>. The graph <NUM> illustrates a modified distortion curve <NUM> that increases resolution at an X, Y location corresponding to the central gaze direction <NUM> and within a corresponding bounding region. The modified distortion curve <NUM> is modulated with the fovea curve <NUM> to heighten the resolution with the prescribed falloff. In some embodiments, fovea modification of the distortion curve <NUM> does not increase the cost of pixel shading because the fovea modification increases spacing in the visual periphery by a corresponding amount to decrease packing inside the fovea region. In some cases, additional pixel rending is added for the fovea regions as well.

In some embodiments, state information indicating the boundaries of the window, e.g., coordinates of the bottom right and upper left of the window, are changed to adjust the number of pixels that are rendered. In that case, the width and height used by the viewport transform width/height, as well as state information indicating an offset, is changed by a percentage corresponding to a percentage change in the size of the window. Full screen viewport and mirror or window rendering in the viewport use the same percentage. The triangle/quad mesh or compute shader that samples buffers for each chromatic sample uses the same UV mesh since the coordinates are in the <NUM> to <NUM> range and only need to be scaled by the texture size.

<FIG> illustrates binning of tiles that represent portions of a primitive <NUM> in a raster space <NUM> of uniformly spaced pixels and a window space <NUM> of non-uniformly spaced pixels according to some embodiments. Binning is performed in some embodiments of the graphics pipeline <NUM> shown in <FIG>. In the illustrated embodiment, a bin <NUM> is subdivided into multiple smaller bins. A minimum bin size for implementing distortion functions as disclosed herein is <NUM>×<NUM> in some cases because this size corresponds to the size of a rasterization tile in a scan converter such as the scan converter <NUM> shown in <FIG>. However, the sizes, including the minimum bin size, are matters of design choice. Some embodiments of the graphics pipeline include a binner (such as a primitive batch binner or a draw stream binning rasterizer) that produces larger or smaller bins, e.g., down to <NUM>×<NUM>, for depending on the target binning characteristics or other characteristics of the graphics processing system.

Binning in the uniform raster space <NUM> uses intersections of the primitive <NUM> with the uniform pixels <NUM> (only one indicated by a reference numeral in the interest of clarity) in the bin <NUM> to identify the sub-bins <NUM> that are touched by the primitive <NUM>.

In the non-uniform window space <NUM>, an additional unit (referred to hereinafter as a flexible bin walker, FBW) is used to split larger bins and do queries for distorted bin boundaries at a destination resolution determined by a distortion function. For example, the bin <NUM> is subdivided on the basis of the distorted pixels <NUM> by identifying the sub-bins <NUM> that are touched by the primitive <NUM>. The FBW is therefore able to produce distorted geometry at <NUM>×<NUM> resolution when the bin size is configured to be larger. In some embodiments, resolution can also be modified by combining multiple bins into a single distorted bin, which allows <NUM>×<NUM> warp granularity with <NUM>×<NUM> bin resolution. Some embodiments of the FBW are controlled via two signed <NUM>-bit registers, separately for X and Y. Positive values indicate that the bin <NUM> is to be split further for warping and negative values indicate that the bin <NUM> is to be combined for larger warp bins. A maximum split count of three indicates that the bin <NUM> is split to eight distorted bins. In practice this means that <NUM>×<NUM> bin size is the largest bin size that is able to be split to <NUM>×<NUM> warp bin size. As discussed above, the particular bin sizes discussed herein are illustrative and some embodiments implement larger or smaller bin sizes depending on characteristics of the graphics processing system.

When the FBW subdivides the larger bin <NUM>, an output is only created for each sub-bin <NUM>, <NUM> that intersects with the primitive <NUM>. No clock cycles are consumed to maintain untouched sub-bins. Each sub-primitive output is appropriately distorted according to the distortion function by the scan converter. Some embodiments of the binner are limited to a maximum of 256x256 bin sizes in order to support unique output distortion on the basis of the distortion function for each 32x32 region. If the binner uses larger bin sizes, the effects of the distortion function are coarser grain. The binner, the FBW, and the scan converter operate in a uniform raster space, but inclusion of the distortion function provides a per-primitive/sub-primitive distorted window space view of the geometry as illustrated by the nonuniform window space <NUM> shown in <FIG>. For 256x256 bin sizes or smaller, the distortion function is applied on the geometry per 32x32 raster space region. The programmable eight segment LUT with alpha described above is used in both the binner and the FBW.

<FIG> illustrates distortion-aware routing <NUM> of primitives based on a comparison of a primitive bounding box to a distorted raster view according to the invention. A primitive assembler such as the input assembler <NUM> shown in <FIG> uses a bounding box <NUM> to route a primitive <NUM> to the appropriate shader engines and scan converter for rasterization. However, the distortion function changes the mapping of the primitive <NUM> to the uniformly spaced pixels in raster space. For example, in the undistorted raster space <NUM>, the bounding box <NUM> is entirely within a single pixel that is routed to corresponding shader engines and a scan converter. However, in the distorted view <NUM> of the raster space, the bounding box <NUM> overlaps with four of the non-uniformly spaced pixels. Consequently, the primitive assembler makes different routing decisions to route the primitive <NUM> to the appropriate shader engines and scan converters.

The primitive assembler uses an inverse of the distortion function to perform routing of the primitive <NUM>. For example, the inverse distort function is applied to four values (minx, miny, maxx, maxy) that define the boundaries of the bounding box <NUM> to create the raster view of the primitive. In some embodiments, the following code fragment is used to generate the rasterizer/scan converter (SC) view of the bounding box. //get rasterizer view of bounding box to determine SC(s) primitive routing
def undistortBBox(x0,y0,x1,y1,x2,y2) :
minx = self. undistort_coord(min(x0,x1,x2))
miny = self. undistort_coord(min(y0,y1,y2))
maxx = self. undistort_coord(max(x0,x1,x2))
maxy = self. undistort_coord(max(y0,y1,y2))
return (minx, miny, maxx, maxy).

Some embodiments of a binner/vertical rasterizer determine vertical bin rows using the same unwarping of the bounding box <NUM> as the primitive assembler uses to select shader engines.

<FIG> is a block diagram of a portion <NUM> of a graphics pipeline that illustrates binning of primitives. This embodiment is not falling under the scope of the appended claims and is to be considered merely as an example suitable for understanding the invention. The portion <NUM> of the graphics pipeline is implemented in some embodiments of the graphics pipeline <NUM> shown in <FIG>. In the illustrated embodiment, a primitive <NUM> is provided to a geometry shader <NUM> that performs rendering of non-uniformly spaced pixels in window space, as discussed herein. The geometry shader <NUM> provides the rendered window space pixels to a distortion function <NUM>, which is implemented in hardware and is configurable on the basis of values of registers.

A binner <NUM> receives a batch of primitives such as triangles from the geometry shader <NUM> via the distortion function <NUM>. Some embodiments of the binner <NUM> operate in two stages: vertical and horizontal. The vertical stage answers the question: "which triangles of the current batch touch a bin row Y". The horizontal stage answers the question: "which triangles of the bin row Y touch a bin X". The binner <NUM> answers these questions for each bin that potentially is touched by any triangle of a batch. For each bin, the binner <NUM> sends each triangle that touches the bin toward the scan converter <NUM>. The binner <NUM> sets a scissor rectangle around the current bin so the scan converter <NUM> only rasterizes within the current bin. In some cases, the triangle vertex positions are fixed point <NUM>-bit XY coordinates. Thus, the binner <NUM> presents a different view of geometry to the scan converter <NUM> based on the distortion function <NUM>, such that each bin processed by the scan converter <NUM> is able to process a unique resolution of the window to generate display pixels that are provided to a display device such as an HMD <NUM>.

Some embodiments of the portion <NUM> of the geometry pipeline also include a flexible bin walker <NUM> that operates in the manner discussed above. The flexible bin walker <NUM> is an optional element that is not necessarily implemented in the portion <NUM> of the geometry pipeline, as indicated by the dashed lines.

Some embodiments of the binner <NUM> perform vertical and horizontal rasterization in accordance with vertical and horizontal distortion functions such as the vertical and horizontal distortion functions <NUM>, <NUM> shown in <FIG>. The vertical rasterizer in the binner <NUM> calculates and stores vertical extents of each triangle. The resolution of this information is in bin rows. Without multiresolution rendering, calculation of the vertical extents is very cheap as it only requires picking a correct number of MSB bits. With multiresolution rendering, the vertical rasterizer calculates the same undistorted bounding box as the primitive assembler does for shader engine selection. In some cases, the vertical rasterizer undistorts the vertical extents of the bounding box, but if the horizontal part of the bounding box is also calculated, the vertical rasterizer is used to eliminate false positive bin rows for narrow and tall triangles. The vertical rasterizer is preferred for this task because with multi-shader-engine configurations each bin column has a certain pattern that has bin rows not belonging to the current binner <NUM>. Thus, the vertical rasterizer is modified to add a calculation of the undistorted bounding box. The following pseudocode is an example technique for determining the bins that are intercepted by a triangle. Determine the first and last spans rows the triangle intercepts //get rasterizer view of bounding box top and bottom <MAT> <MAT> // Get integer first and last bin row the triangle touches <MAT> <MAT>.

The horizontal rasterizer receives the current bin row Y from the vertical rasterizer together with all the triangles touching the current bin row Y on successive clock cycles. In response to the horizontal rasterizer receiving each triangle, the horizontal rasterizer calculates left most and right most intersection points the triangle touches between the bin row top and bottom boundaries. To enable multiresolution rendering, the horizontal rasterizer it is also configured to distort (on the basis of the distortion function) the bin row top and bottom Y coordinates to be used to solve horizontal intersection points and undistort (on the basis of the distortion function) the calculated horizontal intersection points to determine which bins the horizontal span covers. The following pseudocode is an example of the processing performed by the horizontal rasterizer. //Horizontal processing
<IMG>.

The binner <NUM> outputs triangles to scan converter <NUM> together with a scissor rectangle set to current bin boundaries. With multiresolution rendering, the amount of original viewport seen "through" each bin is different. This is achieved by calculating a unique scale and offset for each bin to apply to the triangle vertices before being sent to the scan converter <NUM>. In the case of larger bins, the FBW <NUM> is programmed to adjust the output bin size within its ability to subdivide or aggregate the bins. In some embodiments, scale and offset are calculated by warping the scissor rectangle coordinates to warped space and calculating the size and offset difference between the original and warped scissor rectangles. The scale and offset modify the vertex XY positions. This is implemented by moving a gradient set up calculation from a primitive assembler to the scan converter <NUM>. Otherwise, the output of the binner <NUM> would have to scale also the barycentric and Z gradients, which would require extra arithmetic. The following pseudocode illustrates example operation of the binner <NUM> and the FBW <NUM>. //output and FBW - call distortTriangleToBin to create each output def distortTriangleToBin(self, binX0, binY0, binX1, binY1, x0, y0, x1, y1, x2, y2) : # distort the bin rectangle to distorted sample space wx0 = self. xdistort_coord. get(binX0) wy0 = self. ydistort_coord. get(binY0) wx1 = self. xdistort_coord. get(binX1) wy1 = self. ydistort_coord. get(binY1) # Calculate X & Y scale factors from the size difference # between non-distorted and distorted bin rectangle <MAT> <MAT> # This is used to transform triangle so that # distorted bin rectangle corner is at the origin preXTrans = -wx0 preYTrans = -wy0 # This is used to transform triangle back so that # distorted triangle is relative to the bin postXTrans = binX0 postYTrans = binY0 # Transform the triangle <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> return (tx0, ty0, tx1, ty1, tx2, ty2).

In summary, a primitive assembler is configured to undistort triangle bounding box to determine shader engine that is to get the primitive. A vertical rasterizer in the binner <NUM> undistorts a bounding box for the primitive and the horizontal rasterizer distorts the top and bottom boundaries of a bin row. The horizontal rasterizer also undistorts the horizontal span endpoints. Output from the binner <NUM>, and in some cases the FBW <NUM>, is used to calculate a scale and offset for the current bin and apply the scale and offset to the triangle vertex positions. A set up function is then be moved to a point in the graphics pipeline following the binner <NUM>. This allows the set up unit to calculate the triangle slopes normally using the new and modified triangle vertex positions output by the binner <NUM> or the FBW <NUM>.

<FIG> is a flow diagram of a method <NUM> for rendering sampling non-uniform pixels in a window space through a distortion function according to some embodiments. The method <NUM> is implemented in some embodiments of the graphics pipeline <NUM> shown in <FIG>, the electronic device <NUM> shown in <FIG>, and the electronic device <NUM> shown in <FIG>.

At block <NUM>, the geometry pipeline renders pixels in window space with a non-uniform pixel spacing. At block <NUM>, the non-uniformly spaced pixels in window space are sampled through a distortion function. At block <NUM>, a scan converter renders display pixels based on uniformly spaced pixels in raster space. The uniformly spaced pixels are determined by sampling the non-uniformly spaced pixels through the distortion function. At block <NUM>, an image is generated for display to a user based on the display pixels.

In some embodiments, the apparatus and techniques described above are implemented in a system comprising one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the graphics pipeline described above with reference to <FIG>. Electronic design automation (EDA) and computer aided design (CAD) software tools may be used in the design and fabrication of these IC devices. These design tools typically are represented as one or more software programs. The one or more software programs comprise code executable by a computer system to manipulate the computer system to operate on code representative of circuitry of one or more IC devices so as to perform at least a portion of a process to design or adapt a manufacturing system to fabricate the circuitry. This code includes instructions, data, or a combination of instructions and data. The software instructions representing a design tool or fabrication tool typically are stored in a computer readable storage medium accessible to the computing system. Likewise, the code representative of one or more phases of the design or fabrication of an IC device may be stored in and accessed from the same computer readable storage medium or a different computer readable storage medium.

A computer readable storage medium may include any non-transitory storage medium, or combination of non-transitory storage media, accessible by a computer system during use to provide instructions and/or data to the computer system.

The software includes the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above.

Claim 1:
A method for generating images at a head mounted display, comprising:
rendering, in a graphics pipeline (<NUM>), pixels in window space (<NUM>, <NUM>) with a non-uniform pixel spacing, wherein the non-uniformly spaced pixels in the window space are mapped to uniformly spaced pixels in raster space (<NUM>, <NUM>) through a distortion function (<NUM>, <NUM>, <NUM>);
comparing a bounding box (<NUM>) defined by dimensions of a primitive (<NUM>) comprising a polygon or patch to boundaries of the non-uniformly spaced pixels in window space;
using an inverse of the distortion function to perform routing of the primitive (<NUM>) to an appropriate shader engine and scan converter (<NUM>) for rasterization based on the comparison; and
sampling, with the scan converter (<NUM>) to which the primitive has been routed, the pixels in window space (<NUM>, <NUM>) through the distortion function (<NUM>, <NUM>) to generate values of the uniformly spaced pixels in raster space; and
generating an image for display to a user using values of display pixels that are based on the values of the uniformly spaced pixels in raster space.