View rendering from multiple server-side renderings

A first user input is received when a client program executed by a client computing device is in a first state. The first user input is sent to a server computing device to render a view of a virtual scene. A state change from the first state in the client program due to a second user input or a program event is identified. One or more gaps in a server-rendered current view due to the state change are determined. A rendering of the one or more gaps is selected from among the server-rendered current view, a server-rendered predicted view and one or more prior-rendered views. A current view is rendered using a simplified model of the virtual scene by rendering the one or more gaps from the selected rendering. The current rendered view is visually presented via a display of the client computing device.

BACKGROUND

A client computing device may offload image rendering operations to a server computing device. In particular, the server computing device may render images and send the rendered images to the client computing device via a wide area network. For example, a client computing device may offload rendering operations to a server computing device, because the client computing device may have hardware limitations that restrict the image rendering capabilities of the client computing device.

SUMMARY

Systems and methods for view rendering from multiple server-side renderings are disclosed. According to one aspect, a first user input is received when a client program executed by the client computing device is in a first state. The first user input is sent to a server computing device to render a view of a virtual scene. A server-rendered current view of the virtual scene that is based on the first input is received from the server computing device. A server-rendered predicted view of the virtual scene that is based on the first input is received from the server computing device. A simplified model of geometry of the virtual scene is received from the server computing device. One or more prior-rendered views of the virtual scene are retrieved from memory of the client computing device. A state change from the first state in the client program due to a second user input or a program event is identified. One or more gaps in the server-rendered current view due to the state change are determined. A rendering of the one or more gaps is selected from among the server-rendered current view, the server-rendered predicted view, and the one or more prior-rendered views. A current view is rendered from the simplified model by rendering the one or more gaps from the selected rendering. The current view is visually presented via a display of the client computing device.

DETAILED DESCRIPTION

As discussed above, dedicated server computing devices render and stream content (e.g., rendered images) over a wide area network to various client computing devices in what may be referred to as a “cloud-based streaming platform” for server-side rendered content Such cloud-based streaming platforms allow service providers to centralize rendering operations of graphical virtual scenes (e.g., video games, virtual reality, augmented reality) at server computing devices of a service provider. Server-side rendering offers several advantages. For example, high quality images rendered by powerful server GPUs may be streamed to client computing devices. Such, images may be of a higher quality (e.g., resolution) than images rendered by the less powerful processors of client computing devices. In another example, centralization of image rendering to the server computing devices increases debugging efficiency, software updates, hardware updates, and content additions.

One issue of a cloud-based streaming platform is that wide-area networks can be subject to periods of high latency due to network congestion, packet loss, etc., and/or continually high latency caused by a large distance between a client computing device and a server computing device. Such periods of high latency may cause pauses and delays of streaming content. In one approach to compensate for periods of high latency, the client computing device may perform a post-render image warping operation in which a server-rendered view of a virtual scene is modified to account for intermediate changes in perspective (e.g., translation and/or rotation) that occur between a time at which the server-rendered view of a virtual scene was rendered at the server computing device and a time at which a rendered view of the virtual scene is visually presented at the client computing device. For example, post-render image warping operations may use depth information (e.g., included in the server-rendered view) to perform three dimensional warping to account for the change in perspective.

However, post-render image warping operations that merely use depth information do have some drawbacks. For example, such post-render image warping operations may reveal pixels in the client-rendered view that were occluded in the server-rendered view on which the post-render image warping operations were performed. Because the newly revealed pixels were occluded in the server-rendered view, there is no depth information for the newly revealed pixels. As such, the newly revealed pixels cannot be suitably warped. Instead, gaps corresponding to the newly revealed pixels are created in the client-rendered view. In one approach, gaps are filled in the client-rendered view by an in-painting process that approximates the values of the pixels in the gaps. However, such an in-painting process results in undesirable visual artifacts that lower the quality of the rendered image that is visually presented by the client computing device. In another example, the depth information of a single server-rendered reference image that is used for post-render image warping does not account for whether neighboring pixels are connected. As such, if two neighboring pixels have different depths, the above described post-render image warping operations cannot determine whether the neighboring pixels are connected via the same surface or whether the neighboring pixels are part of separate adjacent surfaces.

The present description relates to an approach for rendering an image using post-render image warping operations that are informed by multiple different server-rendered views of a virtual scene. More particularly, the present description relates to an approach for rendering gaps formed as a result of performing post-render image warping operations on a server-rendered view. The gaps may be rendered using pixel information gleaned from different server-rendered views in which the gaps are visible. In particular, the different server-rendered views may have different perspectives of the virtual scene from which pixels corresponding to the gaps are visible. In other words, the different server-rendered views may include information about the gaps that are not included in the server-rendered view used as a reference for the post-render image warping operations.

Such an approach may leverage the greater processing power of a server computing device to render a high-quality image. Further such an approach, may compensate for latency between the server computing device and the client computing device by post-render warping the server-rendered image at the client computing device. Further still, such an approach may accurately render gaps formed as a result of the post-render image warping operations, such that undesirable visual artifacts are reduced or eliminated from the client-rendered view of the virtual scene.

FIG. 1shows an example computing system100including a client device102configured to communicate with a server computing device104over a network106. The network106may be a wide area network such as the Internet, or another suitable computer network. The client computing device102may be any suitable type of computing device. Non-limiting examples include, but are not limited to, a mobile computing device such as a smartphone, tablet, laptop, or head-mounted augmented reality computing device, a personal computing device, and a game console. The client computing device102includes a processor108, volatile memory110, non-volatile memory112, a display114, an input subsystem116and a communication subsystem118. The non-volatile memory112holds a client program120executable by the processor108to perform a variety of client-side functions, as described below. Likewise, the server computing device104includes a processor122, volatile memory124, and non-volatile memory126. The non-volatile memory holds a server program128executable by the processor122to perform a variety of server-side functions, as described below.

The input subsystem116of the client computing device102receives user input130from a user input device132. The input subsystem116stores and organizes the user input130, and forwards that user input130over the network106to the server computing device104. The user input device132may be any suitable type of user input device. Non-limiting examples of the user input device132include, but are not limited to, a touchscreen, a keyboard, a mouse, and sensors (e.g., gyroscope, accelerometers, depth camera, and/or RGB camera).

In one particular example, the client program120and/or the server program128are configured as a video game that includes a virtual scene that occupies a two- or three-dimensional game space. In one example, the user input130is navigational input interpreted by the client program120and/or the server program128to navigate a playable character (or other controllable entity) through the virtual scene generated by the client program120and/or the server program128. In another example, the user input130includes one or more impulse inputs that are non-navigational inputs. Non-limiting examples of impulse inputs include, but are not limited to interacting with an object, activating an ability, and adjusting a game state (e.g., adjusting a view, visually presenting a menu). In the context of a first-person perspective game, for example, an impulse input may cause a playable character to wield a weapon, attack with the weapon, turn invisible, light a torch, etc. These inputs do not themselves cause a change in navigation of the playable character within the virtual scene.

The server computing device104receives the user input130over the network106. The server program128determines a current program state134of the client program120based on the user input130. For example, the server program120may track the current program state134of the client program120and update the program state134based on the user input130. In some implementations, the client computing device102may send the current program state134to the server computing device102along with the user input130.

The server program128may include a prediction module136configured to calculate prediction information related to the client program120. For example, the prediction information may include future user input (e.g., navigation input, impulse input), program events, and other state changes of the client program120based on the current program state134and the user input130. For example, if a user input stream specifies that the user input has included a forward navigation input for the past few image frames, then the prediction module136may predict that the view of the virtual scene (e.g., the first-person perspective of the playable character) will progress forward in the same manner for the next few image frames in the future. Furthermore, the prediction information may be based on a position of the view/perspective in the virtual scene. For example, if the view/perspective is at the edge of a path, then the prediction module136may predict that the view/perspective will turn to stay within the path.

The prediction module136may employ any suitable prediction techniques. Non-limiting examples of prediction techniques that may be employed by the prediction module136include, but are not limited to, a neural network time-series prediction model, a linear and polynomial regression model, and a Markov model. The prediction information produced by the prediction module136may be used by a server rendering module138of the server program128to render a server-rendered predicted view142of the virtual scene. For example, the server-rendered predicted view142of the virtual scene may be used to render gaps formed in a post-render image warped view rendered by the client computing device102.

The server rendering module138may be configured to render views of the virtual scene that may be used for visual presentation by the client computing device102. In particular, the server rendering module138may be configured to render a server-rendered current view140of the virtual scene and a server-rendered predicted view142of the virtual scene. The server-rendered current view140may be rendered based on the user input130and the current state134of the client program120. In one example, the server-rendered current view140provides a reference view from which a majority of a corresponding client-rendered current view152is rendered by the client computing device102. The server-rendered predicted view142may be rendered based on prediction information provided by the prediction module136. The server-rendered predicted view142represents a predicted future perspective of the virtual scene based on predicted future user input, state changes, and/or program events of the client program120. In other words, the server-rendered current view140is a representation of a current state of the virtual scene and the server-rendered predicted view142is a representation of a future state of the virtual scene.

The server rendering module138renders the current view140and the predicted view142with corresponding depth information. In particular, each rendered view contains depth values representing distances in the virtual world (e.g., game space) from a point corresponding to a perspective of the view to each pixel (or group of pixels) in each rendered surface of the virtual scene. Further, each rendered view contains color data associated with each pixel (or group of pixels).

The server-rendered current view140and the server-rendered predicted view142may be sent from the server computing device104to the client computing device102in accordance with any suitable transmission rate. In one example, a new server-rendered current view140may be sent to the client computing device102every image frame time slot for visual presentation by the client computing device102. Further, a new server-rendered predicted view42may sent to the client computing device102every fifth image frame time slot. In other words, the client computing device102may receive five server-rendered current views for every one server-rendered predicted view. For example, the server-rendered predicted view may be produced and sent on a limited basis in order to reduce usage of processing resources of the server computing device104and reduce usage of network bandwidth. In some implementations, the transmission rate of the server-rendered predicted view142may be different than every five image frame time slots. In some implementations, the transmission rate of the server-rendered predicted view142may be varied based on hardware resource availability and/or network bandwidth.

The server program128includes a model generation module144configured to generate a simplified model146of geometry of the virtual scene. The simplified model146may be used by the client computing device102as a proxy for a full model148of the virtual scene. In particular, the client computing device102may use the simplified model to render a current view of the virtual scene by piecing together (e.g., post-render image warping) image fragments from multiple server-rendered views of the virtual scene.

The simplified model146of geometry of the virtual scene may be derived from the full model148that defines the virtual scene. The full model148of geometry may be a representation of an entirety of the virtual scene. For example, if the virtual scene is a level or virtual world of a video game, then the full model148of geometry may include every vertex (or other geometric representation) that defines the level or virtual world of the game. The full model148of geometry may be maintained by the model generation module144. In some implementations, a copy of the full model148also may be maintained by the client computing device102. The model generation module144may be configured to select a subset of geometry of the full model148of geometry of the virtual scene for inclusion in the simplified model146based on the state of the client program120and the user input130. In one example, the model generation module144may perform computations to determine portions of the model that could possibly be rendered in a future view within a threshold number of frames of the current view, and geometry of the identified portions may be included in the simplified model146. In some implementations, the simplified model144may include geometry that is not included in the full model148, but is derived from geometry in the full model148. For example, the simplified model146may include a geometric element that is formed from collapsing multiple geometric elements of the full model148.

The model generation module144may generate the simplified model146in any suitable manner. In some implementations, geometry of the virtual scene may be simplified or approximated in the simplified model146. For example, various complex geometries in the virtual scene may be approximated by planes in the simplified model146. In some implementations, the model generation module144may employ an edge-collapse algorithm to simplify the geometry of the virtual scene in the simplified model146. In some implementations, the simplified model146may have a same volume as the full model148, but less geometric features. The simplified model146may be sent from the server computing device104to the client computing device102via the network106. For example, the simplified model146may be sent every image frame time slot, every fifth image frame time slot, or another interval.

The client computing device102includes a client rendering module150configured to render the client-rendered current view152from the simplified model146using image fragments and associated information from one or more of the server-rendered current view140, the server-rendered predicted view142, and one or more prior-rendered views154. All of the listed views may be referred to herein as candidate views that may potentially contribute image fragments to the client-rendered current view152. In particular, the client rendering module150determines texture coordinates of server-rendered view fragments from the different server-rendered views based on the perspective of the view and a projection matrix of the view. In other words, the client rendering module150may be configured to warp a server rendered view to texture the simplified model146based on a current state of the client program120. The current state of the client program120may differ from a state of the client program120that was used to render the server-rendered views. For example, a user input or program event may occur after rendering of the server-rendered current view140that updates the perspective of the client-rendered current view152at the client computing device102.

As discussed above, post-render image warping operations may generate gaps. As such, the client program120may use information selected from different server-rendered views to render the gaps in the client-rendered current view152. In particular, the server-rendered predicted view142and one or more prior sever-rendered views154may provide different perspectives of the virtual scene relative to the server-rendered current view140. These differing perspectives may reveal pixels that are not visible in the server-rendered current view140and may be used to render the gaps.

The client computing device102includes a prior-rendered view buffer156in volatile memory110. The buffer156includes a plurality of prior-rendered views of the virtual scene. By maintaining the prior-rendered views in the buffer156, the prior-rendered views may be quickly retrieved from local memory of the client computing device102to be used for gap filling purposes. In one example, the prior-rendered view buffer156is sized to store thirty prior-rendered views. The buffer156may be sized to store any suitable number of prior-rendered views. Depending on the processing resources of the client computing device102more than one prior-rendered view may be selected for use in rendering the client-rendered current view152.

In some implementations, the client program120may determine which prior-rendered views are selected for storage in the buffer156based on a contribution of each view toward filling gaps in other client-rendered views. For example, whenever a new client-rendered view is generated, the view may be stored in the buffer156, and the prior-rendered view that was used the least to fill gaps may be removed from the buffer156to make room for the new client-rendered view.

The client computing device includes a quality determination module158configured to, for each gap, select a candidate view as a rendering to render the gap in the client-rendered current view152. In one example, the quality determination module158determines if a gap is visible in any of the candidate views by obtaining a coarse representation of depth values of the simplified model146. For example, the quality determination module158may employ MIP mapping to extract a coarse level of the depth-/z-values of the texture. The quality determination module158projects the course depth data into the client-rendered current view152. The quality determination module158checks that a depth of a rendered image fragment (e.g., one or more pixels) of a candidate view is within a designated deviation of a depth of the surface defined by the simplified model146. Fragments that pass the depth test are determined to be visible. Fragments that do not pass the depth test are discarded as not being visible.

If the gap is visible in only one candidate view, then the quality determination module158selects the candidate view as the rendering of the gap in the client-rendered current view152. If the gap is visible in more than one candidate view, then the quality determination module158assigns a quality score to each candidate view, and selects a candidate view having a highest quality score as the rendering of the gap in the client-rendered current view152.

The quality score may be determined in any suitable manner. In one example, the quality score is derived from one or more of an angle of a perspective of the candidate view relative to the gap and a distance between a surface corresponding to the gap and the perspective of the candidate view. In particular, a candidate view having an angle straight-on or closer to perpendicular may be assigned a higher quality score. On the other hand, a candidate view having a shallower angle may be assigned a lower quality score. Further, a candidate view having a shorter distance may be assigned a higher quality score. On the other hand, a candidate view having a greater distance may be assigned a lower quality score.

Furthermore, if the gap is not visible in any of the candidate views, then the client rendering module150renders the gap using a smoothing or blurring algorithm. The client rendering module150may use any suitable type of smoothing or blurring algorithm to render the gap.

The display114may be configured to display the client-rendered current view152as part of execution of the client program120. By rendering the client-rendered current view152in the manner described above, a thin client computing device may visually present high-quality images having little or no visual artifacts due to gaps. Moreover, the high-quality images may be visually presented with little or no lag due to high latency between the client computing device102and the server computing device104.

The computing system100described above enables various methods for predicting and rendering content. Accordingly, some such methods are now described, by way of example, with continued reference to above configurations. Such methods, and others fully within the scope of the present disclosure, may be enabled via other configurations as well.

FIGS. 2 and 3show an example method200for predicting and rendering content, such as a view of a virtual scene. In one example, the method200is performed by client program120of the client computing device102ofFIG. 1.FIGS. 4-8show various scenarios that may occur in the course of performing the method200, and will be referenced throughout discussion of the method200.

At202, the method200includes receiving a first user input when the client program120executed by the client computing device102is in a first state. For example, the first state may include a particular perspective of a virtual scene of the client program.

At204, the method200includes sending the first user input to the server computing device104to render a view of the virtual scene.

At206, the method200includes receiving from the server computing device104the server-rendered current view140of the virtual scene that is based on the first input.

At208, the method200includes receiving from the server computing device104the server-rendered predicted view142of the virtual scene that is based on the first input.

At210, the method200includes receiving from the server computing device the simplified model146of geometry of the virtual scene.

At212, the method200includes retrieving one or more prior-rendered views154of the virtual scene from local memory of the client computing device102, which may be volatile memory110or other type of memory. In one example, the one or more prior-rendered views154are retrieved from the prior-rendered view buffer156.

At214, the method200includes identifying a state change from the first state in the client program120due to a second user input or a program event. For example, the state change may cause the perspective of the current view of the virtual scene to change from the state of the client program on which the server-rendered current view140was based. Such a state change causes the server-rendered current view140to no longer accurately represent the current view of the virtual scene. Moreover, the state change is the reason for performing post-render image warping operations at the client computing device102to accurately represent the current view in accordance with the state change.

FIG. 4shows an example timing diagram of operations and communications performed by the client computing device102and the server computing device104. At time T0, the client computing device receives the first user input and the client program is in the first state. At time T1, the client computing device102sends the first user input to the server computing device104to render the server-rendered current view140of the virtual scene. At time T2, the client computing device102receives the server-rendered current view140from the server computing device104. If the state change of the client program120occurs at any time in between time T1and time T2, then the client computing device102has to render the client-rendered current view152including accommodating for gaps as a result of post-render image warping. If the state change occurs prior to time T1, then the client computing device102merely sends the updated state information to the server computing device104to render the view of the virtual scene. If the state change occurs after time T2, then the client computing device102can merely visually present the server-rendered current view140because the state of the client program has not changed.

Continuing withFIG. 2, at216, the method200includes determining one or more gaps in the server-rendered current view due to the state change. For example, the gaps may correspond to pixels that are not visible in the server-rendered current view140, but which are revealed as a result of applying the projection matrix of the current view corresponding to the changed state of the client program120.

Turning toFIG. 3, at218, the method200includes selecting a rendering of the one or more gaps from among the server-rendered current view, the server-rendered predicted view and the one or more prior-rendered views. The server-rendered current view, the server-rendered predicted view and the one or more prior-rendered views may be considered candidate views from which image fragments may be selected to render the one or more gaps in the client-rendered current view152. In one example, selection of the rendering is performed by the quality determination module158of the client computing device102ofFIG. 1.

In some implementations, at220, selecting a rendering of the one or more gaps from the candidate views optionally may include selecting a candidate view in which the one or more gaps are visible as the rendering. In one example, if a gap is visible in only one candidate view, then the candidate view is selected as the rendering.

FIG. 5shows an example scenario where a gap is visible in exactly one candidate view. A simplified model500of geometry of the virtual scene may be viewed from a perspective of a prior-rendered view502, a perspective of a server-rendered current view504, and a perspective of a server-rendered predicted view506. When the simplified model500is viewed from the perspective of the server-rendered current view504, an occluded portion508(indicated by dotted lines) is not visible. When the server-rendered current view is transformed (e.g., post-render image warped) to the client-rendered current view to account for the state change, the occluded portion508becomes a gap, because the server-rendered current view does not include any information (e.g., depth, color) for those pixels. The occluded portion508is not visible from the perspective of the prior-rendered view502. However, the occluded portion508is visible from the perspective of the server-rendered predicted view506. As such, the server-rendered predicted view506may be selected to render the gap508, because the server-rendered predicted view is the only candidate view having pixel information that can be used to render the gap.

Continuing withFIG. 3, in some implementations, at222, selecting a rendering of the one or more gaps from the candidate views optionally may include, if the one or more gaps are visible in more than one candidate view, assigning a quality score to each candidate view. Further, at224selecting a rendering of the one or more gaps from the candidate view optionally may include selecting a candidate view having a highest quality score as the rendering. For example, the quality score may be derived from one or more of an angle of a perspective of the candidate view relative to the one or more gaps and a distance between a surface corresponding to the one or more gaps and the perspective of the candidate view.

FIG. 6shows an example scenario where a candidate view is selected to render a gap based on the candidate view being a closest view in which the gap is visible. A simplified model600of geometry of the virtual scene may be viewed from a perspective of a prior-rendered view602, a perspective of a server-rendered current view604, and a perspective of a server-rendered predicted view606. When the simplified model600is viewed from the perspective of the server-rendered current view604, an occluded portion608(indicated by dotted lines) is not visible. When the server-rendered current view is transformed (e.g., post-render image warped) to the client-rendered current view to account for the state change, the occluded portion608becomes a gap, because the server-rendered current view does not include any information (e.g., depth, color) for those pixels. In this case, the occluded portion608is visible from the perspective of the prior-rendered view602and the perspective of the server-rendered predicted view606. Because the occluded portion608is visible in more than one candidate view, a quality score is assigned to each of the candidate views. In this example, a quality score of the prior-rendered view602is higher than a quality score of the server-rendered predicted view606, because a distance between the perspective of the prior-rendered view602and the occluded portion608is shorter than a distance between the perspective of the server-rendered predicted view606and the occluded portion608. Because the prior-rendered view602has the higher quality score, the prior-rendered view602is selected to render the gap in the client-rendered current view.

FIG. 7shows an example scenario where a candidate view is selected to render a gap based on the candidate view having an angle that is closest to perpendicular relative to the gap. A simplified model700of geometry of the virtual scene may be viewed from a perspective of a prior-rendered view602, a perspective of a server-rendered current view704, and a perspective of a server-rendered predicted view706. When the simplified model700is viewed from the perspective of the server-rendered current view704, an occluded portion708(indicated by dotted lines) is not visible. When the server-rendered current view is transformed (e.g., post-render image warped) to the client-rendered current view to account for the state change, the occluded portion708becomes a gap, because the server-rendered current view does not include any information (e.g., depth, color) for those pixels. In this case, the occluded portion708is visible from the perspective of the prior-rendered view702and the perspective of the server-rendered predicted view706. Because the occluded portion708is visible in more than one candidate view, a quality score is assigned to each of the candidate views. In this example, a quality score of the prior-rendered view702is higher than a quality score of the server-rendered predicted view706, because an angle of the perspective of the prior-rendered view702relative to a surface of the occluded portion708is closer to perpendicular than an angle of the perspective of the server-rendered predicted view606relative to a surface of the occluded portion708. In other words, the server-rendered predicted view706may be assigned a lower quality score because the server-rendered predicted view706has a shallow angle. Because the prior-rendered view702has the higher quality score, the prior-rendered view702is selected to render the gap in the client-rendered current view. In this example, the angle is weighted more heavily in the quality score metric than the distance. As such, although the server-rendered predicted view706is closer to the occluded portion708than the prior-rendered view702, the shallow angle of the server-rendered predicted view706make the server-rendered predicted view706less suitable than the prior-rendered view702for rendering the occluded portion708.

Returning toFIG. 3, at226, the method200includes rendering from the simplified model a current view by rendering the one or more gaps from the selected rendering. In one example, rendering of the client-rendered current view is performed by the client rendering module150of the client computing device102ofFIG. 1.

In some implementations, multiple gaps may be determined in the server-rendered current view, and not all of those gaps may be visible in the initially selected rendering. Accordingly, in some such implementations, at228, rendering the current view optionally may include, if a subset of gaps are not visible in the selected rendering, select a different candidate view to render the subset of gaps. The different candidate view may be selected in the same manner as described above. In particular, if the subset of gaps are visible in only one remaining candidate view, then the candidate view is selected as the different rendering. If the subset of gaps are visible in more than one remaining candidate view, then a quality score is assigned to each remaining candidate view, and a candidate view having a highest quality score is selected to render the subset of gaps. Further, at230, rendering the current view optionally may include rendering the subset of gaps from the different selected candidate view. In some implementations, at232, rendering the current view optionally may include rendering gaps that are not visible in any of the candidate views using a smoothing or blurring algorithm.

FIG. 8shows an example scenario where multiple gaps are rendered by different candidate views. A simplified model800of geometry of the virtual scene may be viewed from a perspective of a prior-rendered view802, a perspective of a server-rendered current view804, and a perspective of a server-rendered predicted view806. When the simplified model800is viewed from the perspective of the server-rendered current view804, a first occluded portion808and a second occluded portion810(both indicated by dotted lines) are not visible. When the server-rendered current view is transformed (e.g., post-render image warped) to the client-rendered current view to account for the state change, the first occluded portion808and the second occluded portion become gaps, because the server-rendered current view does not include any information (e.g., depth, color) for those pixels. In this case, the first occluded portion808is visible from the perspective of the prior-rendered view802. The first occluded portion808is not visible from the perspective of any other candidate views. As such, the prior-rendered view802is selected to render the first occluded portion808. However, the second occluded portion810is not visible from the perspective of the prior-rendered view802. As such, the remaining candidate views are evaluated in terms of the second occluded portion810. In this case, the second occluded portion is visible from the perspective of the server-rendered predicted view806. The second occluded portion810is not visible from the perspective of any other candidate views. As such, the server-rendered predicted view806is selected to render the second occluded portion810.

Returning toFIG. 3, at234, the method200includes visually presenting the rendered view via a display of the client computing device. In one example, the client-rendered current view152is visually presented via the display114of the client computing device102ofFIG. 1.

Returning toFIG. 1, in some implementations, the server rendering module138may be configured to render the server-rendered predicted view142in a particular manner that is likely to include gaps from the server-rendered current view140. In some implementations, a perspective of the server-rendered predicted view140is placed at a predicted most likely future position given previous movements. IN one example, the previous position is determined using an Extended Kalman filter that predicts a full view matrix based on a constant velocity assumption and a history of real view matrices. In other implementations, instead of using a Kalman Filter for determining the perspective of the server-rendered predicted view, a constant offset to the perspective of the server-rendered current view may be used. The constant offset may be determined by performing an offline optimization process in which a full geometry, a view matrix and an expected latency are input to the optimization process. Further, the optimization process outputs an offset matrix that gives an optimal offset between the perspective of the current view and the predicted view, while reducing gaps/disocclusions and increasing quality scores of server-rendered predicted views.

FIG. 9shows an example perspective of a server-rendered current view902relative to a perspective of a server-rendered predicted view906. The perspective of the server-rendered current view902produces a field of view904. The perspective of the server-rendered predicted view906produces a field of view908. The field of view908of the server-rendered predicted view906is wider than the field of view904of the server-rendered current view902.906. Furthermore, the perspective of the server-rendered predicted view906is higher than the perspective of the server-rendered current view902. In other words, a distance between a ground surface in the virtual scene and the perspective of the predicted view906is greater than a distance between the ground surface in the virtual scene and the perspective of the current view902. The wider field of view and the higher perspective of the server rendered predicted view may increase a likelihood of gaps being visible in the sever-rendered predicted view.

The above described systems and methods offer the potential advantage of enabling a client computing device to post-render image warp high-quality server-rendered images to alleviate lag due to high latency conditions between a server computing device and the client computing device. More particularly, the resulting client-rendered images may have little or no visual artifacts due to in-painting of gaps. Accordingly, client computing device having lower processing power (e.g., mobile computing devices) may visually present high-quality rendered imagery in a timely manner.

The client computing device102and the server computing device104illustrated inFIG. 1and described herein may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices.

Each such computing device includes a processor, volatile memory, and non-volatile memory, as well as a display, input device, and communication system configured to enable the computing device to communicate with other devices via a computer network.

The processor of each computing device is configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

The processor of each device is typically configured to execute software instructions that are stored in non-volatile memory using portions of volatile memory. Additionally or alternatively, the processor may include one or more hardware or firmware processors configured to execute hardware or firmware instructions. Processors used by the devices described herein may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the processor optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the processor may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.

Non-volatile memory is configured to hold software instructions even when power is cut to the device, and may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), solid state memory (e.g., EPROM, EEPROM, FLASH memory, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Volatile memory is configured to hold software instructions and data temporarily during execution of programs by the processor, and typically such data is lost when power is cut to the device. Examples of volatile memory that may be used include RAM, DRAM, etc.

Aspects of processor, non-volatile memory, and volatile memory may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.

The terms “module” and “program” may be used to describe an aspect of the client computing device102and the server computing device104implemented to perform a particular function. In some cases, a module or program may be instantiated via a processor executing instructions stored in non-volatile memory using portions of volatile memory at execution time. It will be understood that different modules and/or programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module” and “program” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

Each computing device may include an associated display, which may be used to present a visual representation of data computed and output by the processor. This visual representation may take the form of a graphical user interface (GUI). Such display devices may be combined with processor, volatile memory, and non-volatile memory in a shared enclosure, or such display devices may be peripheral display devices. Touch screens may be utilized that function both as a display and as an input device.

Each computing device may include a user input device such as a keyboard, mouse, touch pad, touch screen, microphone or game controller.

Each computing device may include a communication subsystem configured to communicatively couple the computing device with one or more other computing devices. The communication subsystem may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone or data network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem may allow the computing device to send and/or receive messages to and/or from other devices via a network such as the Internet.

Additional aspects of the present disclosure are described below. In one aspect, a client computing device comprises a processor configured to: receive a first user input when the client program is in a first state, send the first user input to a server computing device to render a view of a virtual scene, receive from the server computing device a server-rendered current view of the virtual scene that is based on the first input, receive from the server computing device a server-rendered predicted view of the virtual scene that is based on the first input, receive from the server computing device a simplified model of geometry of the virtual scene, retrieve one or more prior-rendered views of the virtual scene from memory of the client computing device, identify a state change from the first state in the client program due to a second user input or a program event, determine one or more gaps in the server-rendered current view due to the state change, select a rendering of the one or more gaps from among the server-rendered current view, the server-rendered predicted view and the one or more prior-rendered views, and render from the simplified model a current view by rendering the one or more gaps from the rendering, and a display configured to visually present the current view. In this aspect, the server-rendered predicted view and the one or more prior-rendered views may be candidate views, and selecting the rendering of the one or more gaps may include selecting a candidate view in which the one or more gaps are visible as the highest quality rendering. In this aspect, selecting the rendering of the one or more gaps may include if the one or more gaps are visible in more than one candidate view, assigning a quality score to each candidate view, and selecting a candidate view having a highest quality score as the rendering. In this aspect, the quality score may be derived from one or more of an angle of a perspective of the candidate view relative to the one or more gaps and a distance between a surface corresponding to the one or more gaps and the perspective of the candidate view. In this aspect, the one or more gaps may include a plurality of gaps, and rendering the current view may include if a subset of gaps of the plurality of gaps are not visible in the rendering, selecting a different rendering that includes the subset of gaps, and rendering the subset of gaps from the different rendering. In this aspect, rendering the current view may include rendering gaps that are not visible in any of the candidate views using a smoothing or blurring algorithm. In this aspect, the simplified model of geometry of the virtual scene may include a subset of geometry of the virtual scene that is selected for inclusion in the simplified model based the first state of the program and the first user input. In this aspect, the server-rendered current view and the server-rendered predicted view may include a plurality of pixels each having color data and depth data corresponding to a surface in the virtual scene. In this aspect, the server-rendered predicted view may have a field of view that is wider than a field of view of the server-rendered current view. In this aspect, the server-rendered predicted view may have a perspective of the virtual scene that is higher than a perspective of the server-rendered current view.

According to a second aspect, a method for predicting and rendering content, executable on a client computing device comprises receiving a first user input when a client program executed by the client computing device is in a first state, sending the first user input to a server computing device to render a view of a virtual scene, receiving from the server computing device a server-rendered current view of the virtual scene that is based on the first input, receiving from the server computing device a server-rendered predicted view of the virtual scene that is based on the first input, receiving from the server computing device a simplified model of geometry of the virtual scene, retrieving one or more prior-rendered views of the virtual scene from memory of the client computing device, identifying a state change from the first state in the client program due to a second user input or a program event, determining one or more gaps in the server-rendered current view due to the state change, selecting a rendering of the one or more gaps from among the server-rendered current view, the server-rendered predicted view and the one or more prior-rendered views, rendering from the simplified model a current view by rendering the one or more gaps from the rendering, and visually presenting the rendered view via a display of the client computing device. In this aspect, the server-rendered current view, the server-rendered predicted view and the one or more prior-rendered views may be candidate views, and selecting the rendering of the one or more gaps may include selecting a candidate view in which the one or more gaps are visible as the rendering. In this aspect, selecting the rendering of the one or more gaps may include if the one or more gaps are visible in more than one candidate view, assigning a quality score to each candidate view, and selecting a candidate view having a highest quality score as the rendering. In this aspect, the quality score may be derived from one or more of an angle of a perspective of the candidate view relative to the one or more gaps and a distance between a surface corresponding to the one or more gaps and the perspective of the candidate view. In this aspect, the one or more gaps may include a plurality of gaps, and rendering the current view may include if a subset of gaps of the plurality of gaps are not visible in the rendering, selecting a different rendering that includes the subset of gaps, and rendering the subset of gaps from the different rendering. In this aspect, rendering the current view may include rendering gaps that are not visible in any of the candidate views using a smoothing or blurring algorithm. In this aspect, the simplified model of geometry of the virtual scene may include a subset of geometry of the virtual scene that is selected for inclusion in the simplified model based on the first state of the program, and the first user input.

According to a third aspect, a client computing device comprises a processor configured to: receive a first user input when the client program is in a first state, send the first user input to a server computing device to render a view of a virtual scene, receive from the server computing device a server-rendered current view of the virtual scene that is based on the first input, receive from the server computing device a server-rendered predicted view of the virtual scene that is based on the first input, receive from the server computing device a simplified model of geometry of the virtual scene, retrieve one or more prior-rendered views of the virtual scene from memory of the client computing device, identify a state change from the first state in the client program due to a second user input or a program event, determine one or more gaps in the server-rendered current view due to the state change, identify one or more candidate views in which the one or more gaps are visible from among the server-rendered current view, the server-rendered predicted view and the one or more prior-rendered views, if only one candidate view is identified, select the candidate view as a rendering of the one or more gaps, if more than one candidate view is identified, 1) assign a quality score to each candidate view, the quality score derived from one or more of an angle of a perspective of the candidate view relative to the one or more gaps and a distance between a surface corresponding to the one or more gaps and the perspective of the candidate view, and 2) select a candidate view having a highest quality score as the rendering of the one or more gaps, and render from the simplified model a current view by rendering the one or more gaps from the rendering, and a display configured to visually present the current view. In this aspect, the one or more gaps may include a plurality of gaps, and rendering the current view may include if a subset of gaps of the plurality of gaps is not visible in the rendering, selecting a different rendering that includes the subset of gaps, rendering the subset of gaps from the different rendering, and rendering gaps that are not visible in any of the candidate views using a smoothing or blurring algorithm. In this aspect, the simplified model of geometry of the virtual scene may include a subset of geometry of the virtual scene that is selected for inclusion in the simplified model based on the first state of the program and the first user input.