Patent Description:
Remote rendering of imagery may be employed by some augmented/mixed virtual reality computing systems. In a remote rendering scenario, a remote computing system may render an image and send, via a network connection, the rendered virtual image to an augmented/mixed virtual reality computing system for display. For example, remote rendering may be employed in a scenario where the augmented/mixed virtual reality computing system has limited power and/or computer processing capabilities, such as with mobile computing devices. In this case, the lack of power and/or computer processing capabilities of the mobile computing device may be compensated for by the capabilities of the remote computing system.

In a remote rendering scenario, latency between when an image is rendered and when the image is displayed may result in potentially significant pose error. To avoid user discomfort caused by such pose error, the augmented/mixed virtual reality computing system may locally modify the rendered image produced for an erroneous predicted pose to more closely align with the actual pose of the augmented/mixed virtual reality computing system when the image is displayed. Such local modification may be referred to as "late-stage reprojection.

Existing reprojection techniques generally cannot produce high-quality reprojected images with the power / performance constraints of mobile computing devices, due to the high processing cost of applying per-pixel corrections. Therefore, existing techniques usually use a form of down-sampling of depth values in a depth buffer to reduce computational complexity, such that reprojection can be performed by a mobile computing device having limited power / performance capabilities. However, such down-sampling results in a reduction in quality of the reprojected images.

Accordingly, the present disclosure is directed to an approach for performing late-stage reprojection on rendered imagery in a resource-efficient manner while avoiding down-sampling of depth values in the depth buffer and otherwise maintaining a high level of quality. In particular, the reprojection approach may include determining a geometric complexity of each of a plurality of different tiles that collectively make up an image. Such geometric complexity may be estimated based on a planar deviation error value that penalizes a tile in proportion to an extent to which a geometry of the tile deviates from a plane. A tessellated mesh is produced based on the planar deviation error values calculated for the plurality of tiles. The tessellated mesh may have a variable density that corresponds to the localized geometric complexity of the different tiles, such that tiles having greater geometric complexity have more primitives (e.g., triangles, vertices) and tiles having lesser geometric complexity have fewer primitives. The tessellated mesh may be rendered based on an updated pose to output a reprojected image. Such a reprojection approach may enable a computing device having limited power / performance capabilities to produce high quality reprojected imagery.

<FIG> shows aspects of different example augmented/mixed virtual reality computing devices (100A and 100B) that may display late-stage reproj ected imagery as part of an augmented/mixed virtual reality experience. Device 100A is a smartphone. Device 100B is a head-mounted, near-eye display device. Such computing devices may have limited power/performance capabilities that may such computing device from producing high-quality reprojected images using existing reprojection techniques. On the other hand, the example late-stage reproj ection approaches disclosed herein may be used by these and other computing devices to produce high quality reprojected imagery.

<FIG> shows an example computing system <NUM> in simplified form. For example, the computing system <NUM> may embody the computing devices 100A and 100B described above and illustrated in <FIG>, or any other suitable computing system configured to perform the example late-stage reprojection approaches described herein. The computing system <NUM> is communicatively coupled to a remote image processing computing system <NUM> via a computer network <NUM>, such as the Internet. The remote image processing computing system <NUM> is configured to perform remote image rendering for the computing system <NUM>. As discussed above, in some examples, remote rendering may be employed in order to compensate for limited power / processing capabilities of the computing system <NUM>.

The computing system <NUM> includes a pose tracking subsystem <NUM>, a graphics processing subsystem <NUM>, an eye gaze tracking subsystem <NUM>, and a display <NUM>. The pose tracking subsystem <NUM> is configured to determine a pose of the computing system <NUM> in physical space. In some examples, the pose tracking subsystem <NUM> may include one or more optical sensors, such as a color or monochrome camera, an infrared camera, and/or a camera sensitive to various other ranges of electromagnetic radiation, such as Ultra-Violet light, near infrared light, or other suitable frequencies. In some examples, the pose tracking subsystem <NUM> may include a pair of stereo cameras. The pose tracking subsystem <NUM> configured to determine a pose of the computing system <NUM> based on images acquired by the one or more optical sensors. In some examples, the pose tracking subsystem <NUM> configured to determine a pose of the computing system <NUM> using depth-based techniques, such as time-of-flight or structured light imaging techniques.

In some examples, the pose tracking subsystem <NUM> may include an inertial measurement unit (IMU), which may include accelerometers, gyroscopes, and/or a compass that can detect, for example, a <NUM>-degree-of-freedom (6DOF) position and orientation of the computing system <NUM>. In some examples, the pose tracking subsystem <NUM> may be configured to further refine the 6DOF output of the IMU using visual tracking to search for movement of identified visual features in a series of images captured by the optical sensor(s) to generate an estimate of the relative movement of the computing system <NUM> based upon the movement of these visual features within successive image frames captured by the optical sensor(s) over time.

The pose tracking subsystem <NUM> may be configured to track the pose of the computing system <NUM> in any suitable manner using any suitable type of sensors. The determined pose of the computing system <NUM> may take any suitable form. In some examples, the pose may include a 6DOF position of the computing system <NUM> in a world coordinate system. In other examples, the pose may include a relative position of the computing system <NUM> in relation to other feature identified in image(s) acquired by the pose tracking subsystem <NUM>.

In some implementations, the remote image processing computing system <NUM> optionally may perform at least some pose tracking functionality on behalf of the computing system <NUM>.

The computing system <NUM> is configured to send the pose determined by the pose tracking subsystem <NUM> to the remote image processing computing system <NUM> via the computer network <NUM>. The remote image processing computing system <NUM> is configured to render an image based on the pose of the computing system <NUM>. In some examples, the remote image processing computing system <NUM> may render the image based on the pose and additional prior pose(s) received from the computing system <NUM> that may inform an estimate of a change in pose of the computing system <NUM> over time. In some examples, the remote image processing computing system <NUM> may render a series of images based on the determined pose. In some examples, the remote image processing computing system <NUM> may render one or more image based on a predicted pose of the computing system <NUM> that is derived from the determined pose of the computing system <NUM>.

The remote image processing computing system <NUM> is configured to send, via the computer network <NUM>, the rendered image to the computing system <NUM> for display. In some examples, the remote image processing computing system <NUM> may be implemented as a cloud computing service that renders high-quality interactive content in the cloud and streams the rendered imagery to the computing system <NUM>.

As discussed above, since the remote image processing computing system <NUM> and the computing system <NUM> are connected via the computer network <NUM>, latency between when an image is rendered by the remote image processing computing system <NUM> and when the image is displayed by the computing system <NUM> may result in potentially significant pose error. As such, the graphics processing subsystem <NUM> is configured to perform late-stage reprojection operations to modify the rendered image locally on the computing system <NUM> to more closely align with the actual pose of the computing system <NUM> when the image is displayed via the display <NUM>.

In some implementations, the computing system <NUM> optionally may include the eye gaze tracking subsystem <NUM>. The eye gaze tracking subsystem <NUM> may include an eye tracking camera configured to detect a direction of gaze or location of focus of one or more eyes of a user. The eye tracking camera may be configured to determine a user's gaze in any suitable manner. For example, eye gaze tracking subsystem <NUM> may include one or more glint sources, such as infrared light sources, configured to cause a glint of light to reflect from one or more eyes of the user. The eye tracking camera may be configured to capture an image of one or more eyes of the user including the glint. Changes in the glints from the user's eyes as determined from image data gathered via the eye tracking camera may be used to determine a direction of eye gaze. Further, a location at which gaze lines projected from the user's eyes intersect the display <NUM> may be used to determine an object at which the user is gazing (either a virtual object presented on the display or a real-world object viewed through the display in examples where the display is at least partially see-through).

In some examples, the eye gaze tracking subsystem <NUM> may be configured to determine a region of interest in an image presented via the display <NUM> based on eye gaze information. For example, the region of interest may be determined based on a gaze location or a dwell location of the user. The region of interest may be any suitable size within the image. For example, the region may be a single pixel or group of pixels. The eye gaze tracking subsystem <NUM> may be configured to send the determined region of interest (and/or the eye gaze information) to the graphics processing subsystem <NUM> to inform the image reprojection process performed by the graphics processing subsystem <NUM>.

In some implementations, eye gaze tracking subsystem <NUM> may be omitted from the computing system <NUM>, such that the computing system <NUM> does not have eye gaze tracking functionality.

<FIG> shows different example modules of the graphics processing subsystem <NUM> shown in <FIG> that may be configured to perform different operations during the reprojection process to produce a reprojected image. The graphics processing subsystem <NUM> is configured to receive an input image <NUM> produced based on a predicted pose of the computing system <NUM>. For example, the input image <NUM> may be remotely rendered by the remote image processing computing system <NUM> shown in <FIG>. The graphics processing subsystem <NUM> includes a depth buffer <NUM>, a planar deviation error analysis module <NUM>, an error distribution module <NUM>, a tessellation module <NUM>, a reprojection module <NUM> and an infilling module <NUM>. Note that the graphics processing subsystem <NUM> may include additional modules (not shown) that perform other operations related to image reprojection.

The depth buffer <NUM> is a data structure representing depth in the image. The depth buffer <NUM> may be configured to store a 2D-array that includes a depth value for each screen pixel of the input image <NUM>. In some examples, the depth buffer <NUM> may have the same dimensions as a screen/frame buffer that stores pixel data for the image. The depth buffer <NUM> may be configured to store depth values in any suitable form. In some implementations, the depth buffer <NUM> may be a perspectively projected depth buffer comprising depth values (e.g., Z/W) that are non-linearly mapped between near and far planes that define a depth range in the image. In some examples, the depth buffer <NUM> may have a traditional convention in which the depth values increase in magnitude from the near plane to the far plane. In other examples, the depth buffer <NUM> may have a reverse convention in which the depth values increase in magnitude from the far plane to the near plane. In other implementations, the depth buffer <NUM> may comprise depth values that represent linear distance from the camera plane.

The planar deviation error analysis module <NUM> is configured to divide the depth buffer <NUM> of depth values corresponding to the input image <NUM> into a plurality of tiles <NUM>. Each of the tiles may have a same fixed size. The planar deviation error analysis module <NUM> may be configured to divide the depth buffer <NUM> into any suitable number of tiles. As one example, the planar deviation error analysis module <NUM> may be configured to divide the depth buffer <NUM> into a plurality of 16x16-pixel tiles, arranged in a regular grid. In other examples, the tile size may be selected from anywhere between 8x8 pixels and 64x64 pixels. In some examples, the tile size can be selected flexibly depending on the type of content included in the reprojected image. For example, a smaller tile size may allow for a better fit to highly complex geometry, at the cost of a larger number of tiles, whereas a larger tile size may allow for covering more area cheaply for geometry of lower complexity.

The planar deviation error analysis module <NUM> divides the depth buffer <NUM> into the plurality of tiles <NUM>, such that the planar deviation error analysis module <NUM> can perform local geometric complexity analysis on each of the tiles. In particular, the planar deviation error analysis module <NUM> is configured, for each tile of the plurality of tiles <NUM>, to calculate a planar deviation error value <NUM> that estimates a geometric complexity of the tile and penalizes the tile in proportion to an extent to which a geometry of the tile deviates from a plane. Such planar deviation error analysis may leverage the fact that flat or approximately flat (i.e., planar) content is common in augmented/mixed virtual reality content. For example, such content may include user interfaces projected onto planar surfaces. Thus, such planar deviation error analysis allows for detecting a large number of the depth buffer tiles as flat or substantially flat, and correspondingly determining that such tiles have little or no planar deviation error (e.g., a complete flat or planar tile may have a planar deviation error of zero).

As discussed above, in some implementations, the graphics processing subsystem <NUM> may receive eye gaze information (e.g., a region of interest in the input image) from the eye gaze tracking subsystem <NUM> shown in <FIG>. The planar deviation error analysis module <NUM> optionally may be configured to scale the planar deviation error values of the plurality of tiles based on a distance of the tile from the region of interest. For example, the planar deviation error values for tiles closer to the region of interest may be scaled up and the planar deviation error values for tiles further from the region of interest may be scaled down. Such scaling may be performed to leverage the fact that a human's peripheral vision has lower perceptive resolution relative to a central vision region. Thus, tiles that are located in peripheral regions of the user's field of view may be rendered with lower tessellation levels without sacrificing perceived visual quality.

The error distribution module <NUM> is configured to assign a budget or allotment of primitives (e.g., polygons, triangles, vertices) to each depth buffer tile based on the planar deviation error value <NUM> determined for the tile by the planar deviation error analysis module <NUM>.

The tessellation module <NUM> is configured to produce a tessellated mesh <NUM> of the plurality of tiles <NUM> based on the allotment of primitives that is derived from the planar deviation error values <NUM> calculated for the plurality of tiles <NUM>. In particular, each tile includes a primitive count approximately proportional to the planar deviation error value for the tile. The primitive count may be approximately proportional, because the primitive count may be rounded to a nearest integer value. By forming the tessellated mesh in this manner, computing resources may be allocated to a greater extent to tiles having greater geometric complexity and computing resources may be allocated to a lesser extent to tiles having lesser geometric complexity. The tessellation module <NUM> may include any suitable tessellation hardware unit configured to generate a tessellated mesh <NUM> of primitives (e.g., polygons, triangles, vertices) representing the depth values in the depth buffer <NUM>.

The reprojection module <NUM> is configured to render the tessellated mesh <NUM> based on an updated pose of the computing system <NUM> to produce a reprojected image <NUM>. For example, the reprojection module <NUM> may receive the updated pose from the pose tracking subsystem <NUM> shown in <FIG>. In some examples, the reprojection module <NUM> may be configured to render the tessellated mesh <NUM> using rasterization to produce the reprojected image <NUM>. In other examples, the reprojection module <NUM> may be configured to render the tessellated mesh <NUM> using raytracing to produce the reprojected image <NUM>. The reprojection module <NUM> may be configured to sample the color input texture for each pixel and map the sampled color input textures to the new screen coordinates of the pixels in the reprojected image <NUM>.

In some examples, when reprojecting an image from one pose to another pose, certain areas of the virtual world may become uncovered that were previously occluded by in the previous pose. As there is no color information available for these previously occluded regions in the input image <NUM>, the infilling module <NUM> may be configured to reconstruct plausible color information to fill in such regions by performing inpainting or infilling. Accordingly, the infilling module <NUM> is configured to infill one or more areas in the reprojected image that were previously occluded based on the predicted pose and then uncovered based on the updated pose to produce an infilled reprojected image.

The graphics processing subsystem <NUM> may output the reprojected image <NUM>. For example, the graphics processing subsystem <NUM> may output the reprojected image <NUM> to the display <NUM> for presentation to a user.

Note that although the resource-efficient reprojection approaches have been discussed in the context of scenarios that involve remote rendering, the resource-efficient reprojection approaches may be broadly applicable to other scenarios. For example, in some implementations, initial image rendering and image reprojection may be performed by the same computing system.

<FIG> is a flowchart of an example method <NUM> for performing late-stage image reprojection in a resource-efficient manner. For example, the method <NUM> may be performed by the graphics processing subsystem <NUM> shown in <FIG> and <FIG>.

<FIG> shows an example input image <NUM> that may be remotely rendered based on a predicted pose of the computing system. The input image <NUM> includes a first user interface panel <NUM> having a first depth and a second user interface panel <NUM> having a second depth that is greater than the first depth. In this case, the second panel <NUM> is partially occluded by the first panel <NUM>. Each of the panels <NUM> and <NUM> are flat or planar. Further, the first panel <NUM> includes a plurality of menus <NUM> that protrude from the first panel <NUM>, and each of the menus have planar surfaces.

Returning to <FIG>, at <NUM>, a depth buffer of depth values corresponding to an image produced based on a predicted pose is divided into a plurality of tiles. For example, the depth buffer may be divided into a plurality of tiles by the planar deviation error analysis module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

<FIG> shows an example visualization <NUM> of the depth buffer corresponding to the input image <NUM> shown in <FIG>. The visualization <NUM> only includes depth data and does not include any pixel data of the image <NUM>. As such, the first panel including the plurality of menus and the second panel are represented as flat surfaces corresponding to the associated depth values. Further, the depth buffer is shown is divided into a plurality of tiles <NUM>. Each of the tile may be analyzed to determine local geometric complexity.

Returning to <FIG>, in some implementations, at <NUM>, a region of interest optionally may be determined based on eye gaze information. For example, the eye gaze information may be provided by the eye gaze tracking subsystem <NUM> shown in <FIG>.

At <NUM>, for each tile of the plurality of tiles, a planar deviation error value is calculated. The planar deviation error value may estimate a geometric complexity of the tile and penalize the tile in proportion to an extent to which a geometry of the tile deviates from a plane. A higher geometric complexity implies that different pixels in the tile require different amounts of positional correction, and thus a proportionally higher amount of processing resources / effort (e.g., additional reprojection operations) is to be allocated for reprojection of the tile. On the other hand, a lower geometric complexity implies that a proportionally lower amount of processing resources / effort is to be allocated for reprojection of the tile. In one example, the planar deviation error value may be specified as a root mean square error between sample values in the depth buffer tile and a least-square plane fitted to the same values. For a square tile of size T with coordinates centered around (x, y) = (<NUM>, <NUM>), this error can be computed as: <MAT> According to this error metric, depth buffer regions that are covered by geometry which is perfectly flat in the virtual scene has a planar deviation error of zero. This is ensured by using perspectively projected depth values (e.g., Z/W), rather than a depth buffer that represents linear distance from the camera plane. With perspectively projected depth values, planar regions in the virtual world may be rasterized as depth values which depend linearly on their screen space coordinates, allowing a perfect planar fit for each depth buffer tile. Such a perfect-fit property is not provided by other complexity metrics, such as using the range (min-max) or standard deviation of the samples to calculate error.

Additionally, the above described planar deviation error metric may be flexibly applied to different types of images. In particular, the error metric is invariant to the projection matrix convention that was used to produce the image. In other words, the error metric provides the same result for both a conventional and reverse depth buffer, where the values of near and far plane are swapped. For example, the planar deviation error values may be calculated for the plurality of tiles by the planar deviation error analysis module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

In some implementations, at <NUM>, the planar deviation error value of each of the tiles optionally may be scaled based on a distance of the tile from the region of interest determined based on the user's eye gaze. For example, the error estimate may be reduced in the peripheral regions of the user's visual field, and correspondingly lower tessellation levels can be applied without significantly sacrificing perceived visual quality of the reprojected image. For example, the planar deviation error values may be scaled based on the eye gaze information by the planar deviation error analysis module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

<FIG> shows an example visualization <NUM> of planar deviation error values produced from the depth buffer shown in <FIG>. Note that the edges <NUM> between objects and different objects (e.g., between the two user interface panels, between the menus and the user interface panels) and edges <NUM> between objects and the background are detected as having high error. On the other hand, the surfaces of the user interface panels and menus have zero error. According to this planar deviation error metric, allocation of the reprojection performance budget may be biased toward those tiles having geometric complexity where detailed per-pixel reprojection is required (e.g., tiles including edges) to achieve high quality. Correspondingly, less computing resources may be allocated toward tiles completely covered by flat surfaces.

Returning to <FIG>, at <NUM>, a total primitive budget is determined for allocation across the plurality of tiles. In some examples, the total primitive budget can be a predetermined value, ensuring deterministic performance cost independent of the input image. In other examples, the total primitive budget may be scaled based on a distance between the predicted pose used to initially render the image and the actual pose of the computing system when the image is displayed. Such scaling of the primitive budget may reduce performance costs for images that require little overall correction during reprojection. For example, the total primitive budget may be determined by the error distribution module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

At <NUM>, for each tile of the plurality of tiles, a primitive count is assigned to the tile in proportion to the planar deviation error for the tile, such that a total sum of primitives assigned to the plurality of tiles matches the total primitive budget. The primitives may take any suitable form. For example, the primitives may include polygons (e.g., triangles) or vertices. In some examples, the type of primitive that is used may depend on the type of tessellation hardware unit employed by the computing system.

In one example where the primitives are triangles, each tile may have a minimum count of two triangles (corresponding to one quadrilateral per tile), and a maximum count of 2xTxT triangles for a tile size of T (corresponding to one quadrilateral per pixel). Additionally, the triangle count per tile must be naturally constrained to be integer values. In some implementations, the triangle distribution may include one or multiple passes. In each pass, each tile may be assigned a triangle count in proportion to the error value of the tile, and then the triangle count may be clamped and rounded to the nearest integer. This process may be repeated until the total sum of triangles assigned to all tiles matches the total triangle budget. For example, the primitive count may be assigned to the plurality of tiles by the error distribution module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

At <NUM>, a tessellated mesh of the plurality of tiles is produced based on the planar deviation error values calculated for the plurality of tiles. More particularly, the tessellated mesh may be produced based on the primitive counts of the plurality of tiles that are derived from the planar deviation error values. The tessellated mesh may have a variable density of different numbers of primitive corresponding to different tiles. In this way, computing resources are allocated to a greater extent to tiles having greater geometric complexity and computing resources are allocated to a lesser extent to tiles having lesser geometric complexity. The tessellated mesh is effectively a sparse representation of the densely sampled depth buffer and represents the geometry that is to be reprojected in an efficient manner. For example, the tessellated mesh may be produced by the tessellation module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

<FIG> shows an example visualization <NUM> of a tessellated mesh <NUM> produced based on the planar deviation error values of the visualization <NUM> shown in <FIG>. The tessellated mesh <NUM> includes a mesh of triangles that represent the surfaces of the first panel including the plurality of menus and the second panel of the input image <NUM> shown in <FIG>. In this example, the regions of the tessellated mesh <NUM> that include the edges of the panels and the menus may include a higher triangle count than other tiles that include flat surfaces.

Returning to <FIG>, at <NUM>, an updated pose of the computing system is received. For example, the updated pose may be received from the pose tracking subsystem <NUM> of the computing system <NUM> shown in <FIG>. At <NUM>, the tessellated mesh is rendered based on the updated pose to produce a reprojected image. For example, the tessellated mesh may be rendered to produce the reprojected image by the reprojection module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

At <NUM>, one or more areas in the reprojected image that were previously occluded based on the predicted pose and then uncovered based on the updated pose are identified. In one example, each area is identified based on one or more primitives in the area being distorted by more than a threshold. In the triangle tessellated mesh example, the mesh does not have any holes, and instead triangles whose vertices lie on different sides of a discontinuity in the original depth buffer become heavily distorted as their vertices are projected to far-apart positions based on the updated pose. Such distortion and corresponding holes can be detected, in one example, from the Jacobian matrix of the reprojection that maps from a pair of source texel coordinates (u,v) to a pair of target pixel coordinates (x,y): <MAT> From the Jacobian matrix the singular values σ<NUM> and σ<NUM> can be computed as follows: <MAT> <MAT> <MAT> The ratio <MAT> represents the anisotropy of the reprojection, or in other words, the amount of distortion that each triangle experiences. By thresholding this distortion, for example limiting it to <NUM>:<NUM>, the triangles can be classified as stemming from continuous and discontinuous regions of the source depth buffer. Correspondingly, areas of the reprojected image which are covered by highly anisotropic triangles can be identified as needing to be infilled, whereas areas of the reprojected image covered by non-distorted triangles do not need any post-processing infilling. For example, the one or more areas may be identified by the infilling module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

<FIG> shows an example visualization <NUM> including an infill mask <NUM> for a reprojected image <NUM> produced from rendering the tessellated mesh <NUM> shown in <FIG> based on an updated pose of the computing system. In the illustrated example, the panels and menus are shifted rightward based on a shift leftward of the pose of the computing system in the reprojected image <NUM> (shown in dotted lines). Such a shift causes previously occluded areas to be revealed. The infill mask <NUM> (shown in solid lines) identifies these areas that were previously occluded based on the predicted pose and then uncovered based on the updated pose. In one example, the infill mask <NUM> may identify areas of the tessellated mesh including triangles that are distorted with an anisotropy greater than <NUM>:<NUM>.

Returning to <FIG>, at <NUM>, the one or more areas identified in the reprojected image may be infilled. In some implementations, each area of the one or more areas may be infilled with a color of a primitive in the area having a greatest depth value. Such an infilling technique may be performed to avoid bleeding of foreground color information into the background of the reprojected image as rendering artifacts may be perceived as less severe if they occur in the background, versus occurring in the foreground of an image. In the example of the triangle mesh, for triangles which exceed the distortion threshold, instead of interpolating the source color sample, a constant color taken from one of the three triangle vertices which is at the furthest distance from the camera may be applied instead in the infilling process. In some cases, such an infilling approach may produce artifacts in the form of perceivable lines. Accordingly, in some examples, infilling may include blurring the one or more areas that are infilled in the reprojected image. In one example, blurring may be performed by applying a Gaussian blur with an elliptical footprint of varying width, such that a blur kernel parameter σ increases linearly with the distance from the background pixels towards the foreground pixels. For example, the one or more areas may be infilled by the infilling module <NUM> of the graphics processing subsystem <NUM> shown in <FIG>.

Note that in some examples, a reproj ected image may not have any areas that were previously occluded. In such examples, infilling operations may be bypassed.

At <NUM>, the infilled reprojected image may be output. For example, the infilled reprojected image may be output to the display <NUM> of the computing system <NUM> for presentation to a user.

<FIG> shows an example image <NUM> that is reprojected using the above described method. The image <NUM> is produced for the actual position of the computing system at the time of display. In particular, the pose of the computing system is shifted leftward of the initial predicted position, and thus the reprojection is shifted rightward relative to the input image <NUM> shown in <FIG>. The reprojected image <NUM> may have minimal artifacts and avoids issues that would otherwise result from down-sampling of the depth buffer.

The above described method may be performed to carry out late-stage reprojection on rendered imagery in a processing resource-efficient manner while avoiding down-sampling of depth values in the depth buffer. Such a method may enable a computing device having limited power / performance capabilities to produce high quality reprojected imagery by dynamically distributing reprojection resources to geometrically complex regions of an image.

In some implementations, the methods and processes described herein may be tied to a computing system of one or more computing devices.

<FIG> schematically shows a non-limiting implementation of a computing system <NUM> that can enact one or more of the methods and processes described above. Computing system <NUM> may embody the computer devices 100A-100B described above and illustrated in <FIG> as well as the computing system <NUM> and the remote image processing computing system <NUM> described above and illustrated in <FIG>. Computing system <NUM> may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices, and wearable computing devices such as smart wristwatches and head mounted augmented/mixed virtual reality devices.

Computing system <NUM> includes a logic processor <NUM>, volatile memory <NUM>, and a non-volatile storage device <NUM>.

The term "module" may be used to describe an aspect of computing system <NUM> typically implemented in software by a processor to perform a particular function using portions of volatile memory, which function involves transformative processing that specially configures the processor to perform the function. Thus, a module may be instantiated via logic processor <NUM> executing instructions held by non-volatile storage device <NUM>, using portions of volatile memory <NUM>. It will be understood that different modules may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term "module" may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc..

In an example, an image reprojection method comprises dividing a depth buffer of depth values corresponding to an image produced based on a predicted pose into a plurality of tiles, for each tile of the plurality of tiles, calculating a planar deviation error value that estimates a geometric complexity of the tile and penalizes the tile in proportion to an extent to which a geometry of the tile deviates from a plane, producing a tessellated mesh of the plurality of tiles based on the planar deviation error values calculated for the plurality of tiles, receiving an updated pose, and rendering the tessellated mesh based on the updated pose to produce a reprojected image. In this example and/or other examples, the planar deviation error value may be calculated as a root mean square error between depth values in the tile and a least-square plane fitted to the depth values in the tile. In this example and/or other examples, the depth buffer may be a perspectively projected depth buffer comprising depth values that are non-linearly mapped between near and far planes. In this example and/or other examples, the method may further comprises determining a region of interest in the image based on eye gaze information; and the planar deviation error value may be scaled based on a distance of the tile from the region of interest. In this example and/or other examples,.

producing a tessellated mesh may include allocating computing resources to such production based on the planar deviation errors, such that computing resources are allocated to a greater extent to tiles having greater geometric complexity and computing resources are allocated to a lesser extent to tiles having lesser geometric complexity. In this example and/or other examples, the method may further comprises determining a total primitive budget for the depth buffer, and for each tile of the plurality of tiles, assigning a primitive count to the tile in proportion to the planar deviation error for the tile, such that a total sum of primitives assigned to the plurality of tiles matches the total primitive budget, and the tessellated mesh may be produced based on the primitive counts for the plurality of tiles. In this example and/or other examples, the total primitive budget may be a predetermined value. In this example and/or other examples, the total primitive budget may be scaled based on a distance between the predicted pose and the updated pose. In this example and/or other examples, the method may further comprise identifying one or more areas in the reprojected image that were previously occluded based on the predicted pose and then uncovered based on the updated pose, and infilling the one or more areas in the reprojected image to produce an infilled reprojected image. In this example and/or other examples, the one or more areas may be identified based on primitives in the one or more areas being distorted by more than a threshold. In this example and/or other examples, each area of the one or more areas may be infilled with a color of a primitive in the area having a greatest depth value.

In another example, a computing system comprises a logic subsystem, a storage subsystem holding instructions executable by the logic subsystem to divide a depth buffer of depth values corresponding to an image produced based on a predicted pose into a plurality of tiles, for each tile of the plurality of tiles, calculate a planar deviation error value that estimates a geometric complexity of the tile and penalizes the tile in proportion to an extent to which a geometry of the tile deviates from a plane, produce a tessellated mesh of the plurality of tiles based on the planar deviation error values calculated for the plurality of tiles, receive an updated pose, and render the tessellated mesh based on the updated pose to produce a reprojected image. In this example and/or other examples, the planar deviation error value may be calculated as a root mean square error between depth values in the tile and a least-square plane fitted to the depth values in the tile. In this example and/or other examples, the depth buffer may be a perspectively projected depth buffer comprising depth values that are non-linearly mapped between near and far planes. In this example and/or other examples, the storage subsystem may hold instructions executable by the logic subsystem to determine a total primitive budget for the depth buffer, and for each tile of the plurality of tiles, assign a primitive count to the tile in proportion to the planar deviation error for the tile, such that a total sum of primitives assigned to the plurality of tiles matches the total primitive budget, and the tessellated mesh may be produced based on the primitive counts for the plurality of tiles. In this example and/or other examples, the storage subsystem may hold instructions executable by the logic subsystem to identify one or more areas in the reprojected image that were previously occluded based on the predicted pose and then uncovered based on the updated pose, and infill the one or more areas in the reprojected image to produce an infilled reprojected image. In this example and/or other examples, the one or more areas may be identified based on primitives in the one or more areas being distorted by more than a threshold. In this example and/or other examples, each area of the one or more areas may be infilled with a color of a primitive in the area having a greatest depth value. In this example and/or other examples, the storage subsystem may hold instructions executable by the logic subsystem to determine a region of interest in the image based on eye gaze information, and the planar deviation error value may be scaled based on a distance of the tile from the region of interest.

In an example, an image reprojection method comprises dividing a depth buffer of depth values corresponding to an image produced based on a predicted pose into a plurality of tiles, for each tile of the plurality of tiles, calculating a planar deviation error value that estimates a geometric complexity of the tile and penalizes the tile in proportion to an extent to which a geometry of the tile deviates from a plane, producing a tessellated mesh of the plurality of tiles based on the planar deviation error values calculated for the plurality of tiles, receiving an updated pose, rendering the tessellated mesh based on the updated pose to produce a reprojected image, and infilling one or more areas in the reprojected image that were previously occluded based on the predicted pose and then uncovered based on the updated pose to produce an infilled reprojected image.

Claim 1:
A computer-implemented image reprojection method (<NUM>) comprising at a logic subsystem:
dividing (<NUM>) a depth buffer of depth values corresponding to an image produced based on a predicted pose into a plurality of tiles;
for each tile of the plurality of tiles, calculating (<NUM>) a planar deviation error value that estimates a geometric complexity of the tile and penalizes the tile in proportion to an extent to which a geometry of the tile deviates from a plane;
producing (<NUM>) a tessellated mesh of the plurality of tiles based on the planar deviation error values calculated for the plurality of tiles;
receiving (<NUM>) an updated pose;
rendering (<NUM>) the tessellated mesh based on the updated pose to produce a reprojected image
identifying one or more areas in the reprojected image that were previously occluded based on the predicted pose and then uncovered based on the updated pose, where the one or more areas are identified based on primitives in the one or more areas being distorted by more than a threshold; and
infilling the one or more areas in the reprojected image to produce an infilled reprojected image.