System and method for hybrid format spatial data distribution and rendering

Systems and methods are described for providing spatial content using a hybrid format. In some embodiments, a client device receives, from a server, surface light field representations of a plurality of scene elements in a 3D scene, including a first scene element. The client device provides to the server an indication of a dynamic behavior of a second scene element different from the first scene element. Further, in response to the indication, the client device receives from the server information defining the first scene element in a 3D asset format. The client device then renders at least the first scene element in the 3D asset format.

BACKGROUND

Immersive displays have created a demand for realistic content in which a viewer can interact and navigate freely. Typically, only real-time three-dimensional (3D) rendering may enable such viewer experience.

SUMMARY

According to some embodiments a method, performed by a client device, includes: receiving, from a server, surface light field representations of a plurality of scene elements in a 3D scene, including a first scene element; providing to the server an indication of a dynamic behavior of a second scene element different from the first scene element; in response to the indication, receiving from the server information defining the first scene element in a 3D asset format; and rendering at least the first scene element in the 3D asset format.

In some embodiments, the method further includes: after providing to the server the indication of the dynamic behavior of the second scene element, receiving, from the server, a second indication of at least the first scene element being visually affected by the dynamic behavior of the second scene element. Further, in some embodiments, the method includes: receiving, from the server, scene description meta-data for the 3D scene; based on the scene description meta-data, requesting from the server the second scene element in the 3D asset format to be rendered locally at the client device; and receiving, from the server, information defining the second scene element in the 3D asset format.

Additionally, in some embodiments, the method further includes: in response to receiving the second indication of at least the first scene element being visually affected by the dynamic behavior of the second scene element, requesting, form the server, the first scene element in the 3D asset format locally at the client device. The scene description meta-data includes, in some embodiments, scene logic corresponding to assets around a current user location and timeline information for future events, and the method further includes using the indication of the dynamic behavior of the second scene element in combination with the scene logic to determine which one or more scene elements, the one or more scene elements including the first scene element, to request from the server in the 3D asset format. In some embodiments, providing to the server the indication of the dynamic behavior of the second element includes processing at least a user input and scene logic included in the scene description meta-data to determine the second element that has the dynamic behavior as a result of the processing.

Further, in some embodiments, the method includes: observing one or more performance metrics; and limiting an amount of interactive behavior in the 3D scene when a local rendering performance at the client device falls below a threshold. In this regard, in some embodiments, the one or more performance metrics include at least one of processing load or rendering frame rate. In some embodiments, limiting the amount of interactive behavior in the 3D scene includes limiting at least one of a number of interactive events or a number of scene elements having the dynamic behavior.

Yet further, in some embodiments, the method further includes: in addition to receiving, form the server, the information defining the first scene element in the 3D asset format, further receiving, from the server, one or more updated surface light field representations, wherein rendering at least the first scene element in the 3D asset format includes rendering a combination of the first scene element in the 3D asset format and the one or more updated surface light field representations.

According to some embodiments, a method performed by a server, includes sending, to a client device, surface light field representations of a plurality of scene elements in a 3D scene, including a first scene element; receiving, from the client device, an indication of a dynamic behavior of a second scene element different from the first scene element; determining that the first scene element is visually affected by the dynamic behavior of the second scene element; and in response to a determination that the first scene element is visually affected by the dynamic behavior of the second scene element, sending, to the client device, information defining the first scene element in a 3D asset format.

In some embodiments, the method further includes: after receiving from the client device the indication of the dynamic behavior of the second scene element, sending, to the client device, a second indication of at least the first scene element being visually affected by the dynamic behavior of the second scene element. Further, in some embodiments, the method includes: after receiving, from the client device, the indication of the dynamic behavior of the second scene element, updating one or more surface light field representations; and in addition to sending, to the client device, the information defining the first scene element in the 3D asset format, further sending, to the client to the client device, the one or more updated surface light field representations to be rendered locally at the client device in combination with the first scene element in the 3D asset format.

According to some embodiments, an apparatus includes a processor configured to perform at least: receiving, from a server, surface light field representations of a plurality of scene elements in a 3D scene, including a first scene element; providing to the server an indication of a dynamic behavior of a second scene element different from the first scene element; in response to the indication, receiving from the server information defining the first scene element in a 3D asset format; and rendering at least the first scene element in the 3D asset format.

In some embodiments, the processor is further configured to perform: after providing to the server the indication of the dynamic behavior of the second scene element, receiving, from the server, a second indication of at least the first scene element being visually affected by the dynamic behavior of the second scene element. Also, in some embodiments, the processor is further configured to perform: receiving, from the server, scene description meta-data for the 3D scene; based on the scene description meta-data, requesting, from the server, the second scene element in the 3D asset format to be rendered locally at the client device; and receiving, from the server, information defining the second scene element in the 3D asset format.

EXAMPLE NETWORKS FOR IMPLEMENTATION OF THE EMBODIMENTS

DETAILED DESCRIPTION

Overview

As noted above, immersive displays have created a demand for realistic content in which a viewer can interact and navigate freely. Further, at the moment, typically only real-time rendered 3D (spatial) content enables full unlimited six degrees of freedom (6DoF) motion and interaction within such content. However, one issue with real-time 3D rendering is that realistic image quality can generally be achieved only with relatively simple scenes. For instance, the real-time 3D rendering may produce realistic visuals only with the simplest of scenes.

Off-line rendering can produce highly realistic images using advanced rendering approaches such as ray tracing. For example, off-line rendering may accurately simulate light transport between scene objects with different materials. One solution for enabling similar high-quality visualization for real-time applications is to use off-line rendering to produce light fields which enable interactive viewpoint manipulation, such as within a light field area. Typical light fields consist of only a single flat two-dimensional (2D) window from which light is captured, generally enabling only limited changes to the viewpoint. Furthermore, purely image-based approaches, in which a light field is represented as an array of RGB images approximating a four-dimensional (4D) light field, require data amounts that often prohibit a practical use. Also, an additional limitation of light fields is that they typically can only represent elements with pre-determined dynamic behaviors, thus limiting an applicability for interactive experiences.

One approach to reduce a sampling density required from a light field is to add 3D geometry information to accompany image information. The geometry information may be depth information enabling more accurate image warping between viewpoints included with the light field images, or the geometry information may be in the form of an explicit 3D mesh representing or approximating scene objects.

Another approach involves surface light fields. In general, surface light fields are a mix between pure image-based light fields and a traditional real-time 3D computer graphics. By providing 3D geometry to assist in image data re-mapping, surface light fields can result in, e.g., significant data optimization, while still providing image quality improvements similar to the light fields. Further, surface light fields generally operate by having accurate approximation of scene objects as 3D meshes to which multiple textures are mapped. Textures mapped to the geometry are light fields that contain surface appearance as a function of surface location and viewpoint direction. This type of surface light field reduces the amount of data, since much sparser image samples suffice as compared with purely image-based light fields. Surface light fields may also produce sharper rendering results than light fields alone. Furthermore, with a type of surface light fields that provide optimized proxy geometry approximating the original geometry, data amount reduction may be significant, enabling efficient realistic rendering of, e.g., very complex scenes on even low-end devices having mobile System on a Chip (SoC) processors.

Despite advantages in enabling practical light field rendering on mobile VR HMDs, current surface light field solutions typically provide limited support for interactive scene element behaviors, wider area navigation, and adaptability to varying client capabilities. As an example, freedom for 6DoF navigation can be normally supported only in a limited area. Similarly, only fully static scenes are normally supported.

Some embodiments described herein enable high image quality with a low-scene complexity and rendering cost similar to surface light fields while providing support for interactive dynamic content.

In general, in some embodiments, with light fields, the need for overlapping visual data from multiple viewpoints may be simplified by providing a 3D geometry information about a scene in addition to image data. In some embodiments, surface light fields can provide geometry, either a full original geometry or geometry proxies approximating the original geometry, to which light fields are mapped or projected, e.g., similar to normal texture mapping used with contemporary real-time 3D graphics. Typically, an exception to the texture mapping may be that instead of having a single texture that defines RGB colors along the geometry surface, surface light fields provide color information as a four-dimensional (4D) function, where in addition to a location on the surface, also a viewing angle determines color values.

Surface light fields normally require pre-processing, typically restricting their applicability to only static scenes without an interactive behavior. However, according to some embodiments, static scenes optimized and represented as surface light fields may be combined with real-time rendered content elements to provide dynamic and interactive elements. In some embodiments, full global light transport is reproduced for such hybrid scenes in real-time rendering.

In general, surface light fields can simplify client-side rendering of, e.g., virtual reality (VR) content streams. However, surface light fields typically allow for only limited interactivity. Various embodiments of the present disclosure improve interactivity (interactive behavior) by providing some scene elements in a full 3D asset format.

Further, various embodiments of the present disclosure enable content delivery in a hybrid format, where, for example, on per object basis, the content may be selectively provided in an optimized surface light field format or in a full 3D asset format. According to some embodiments, a content server estimates a format required by a viewing client, thus reducing data needed to be downloaded by the viewing client to, e.g., the minimum amount while still enabling interactive high-quality immersive rendering, even on low-end client devices.

In accordance with some embodiments, various solutions of the present disclosure enable a viewing client and a content server to collaborate on providing optimized distribution and balance between using surface light fields and real-time 3D assets in order to create highly realistic interactive immersive experiences even on viewing client devices with sub-optimal processing performance. More specifically, in some embodiments, the content server may provide a scene pre-processed into an optimized surface light field representation instead of full 3D assets, and when the viewing client executes an experience associated with the scene, the viewing client may selectively download any full 3D asset(s) for scene element(s) that the client wants to feature as having interactive dynamic behavior(s) and that hence could benefit from local rendering. Since, in such embodiments, the viewing client does not have the full scene in an original 3D format, the client will not normally be able to estimate locally what other scene elements are visually impacted by the interactive dynamic behavior(s) in the scene.

In some embodiments, an immersive 3D scene may be streamed to a client device from a server. Some scene elements may be streamed in a surface light field format, and other scene elements mat be streamed in a full 3D (asset) format (e.g. a 3D mesh plus texture). The client device may report interactivity (interactive behavior) to the server, and the server may, in turn, determine which scene element(s) to send in which particular format. For example, the server may determine that an interaction with a first object (e.g., an object A) that has a dynamic behavior requires sending a second object (e.g., an object B) in a full 3D asset format because the reflection of the object A interacts with the object B, by for example, being visible in the object B.

To illustrate,FIG.2shows an example of a static scene, according to some embodiments. The scene illustrated inFIG.2represents an example of a static scene which can be, e.g., fully delivered using a surface light field format. By way of example, as shown inFIG.2, the static scene may include such static (scene) elements as an element (a couch)200, an element (a wall mirror)202, an element (a floor)204, an element206(a wall), and an element208(a hanging light)FIG.3then illustrates an example of an addition of a dynamic interactive element to the example scene ofFIG.2, according to some embodiments.

Referring toFIG.3, when a client adds a dynamic interactive (scene) element300to the scene shown inFIG.2, in this case a robot with a smile300, in some embodiments, the client first requests a robot asset from a content server in an original full 3D format, so that the client locally renders the asset in real-time. Such client-side rendering may enable real-time interactive behaviors without network latency that would be caused by server-side rendering. However, since the client does not have the full scene in an original 3D format, the client may not able to determine visual impact(s) that the addition of the dynamic interactive element300rendered locally on the client may cause to the rest of the scene.

The example inFIG.3illustrates examples of visual impacts on the rest of scene elements, e.g., the elements200-208, that may be caused by a locally rendered dynamic behavior, in this case the smiling robot300, in order to maintain a visual integrity of the scene. In this example case, as shown inFIG.3, visual impacts include a shadow302cast by the robot300onto the floor204and a reflection304of the robot300in the mirror202on the wall206.

In some embodiments, to correctly handle those visual impacts, the content server determines area(s) of the scene where original surface light fields representing the scene (and provided by the server to the client, as explained above) have become invalid due to the addition of the dynamic interactive element(s)300and signals that information back to the client.FIG.4illustrates an example of invalid surface light field areas of the static scene ofFIG.2after the addition of the dynamic interactive element300ofFIG.3, according to some embodiments.

More specifically, inFIG.4, elements400and402denoted by bolded and dashed lines (e.g., an area containing a shadow of the robot (400) and an area enclosing the wall mirror (402)) are examples of the areas originally represented by surface lights fields that have become invalid in the static scene after the addition of the smiling robot element300. As shown in the example ofFIG.4and noted above, those impacted static-scene elements include a portion of the floor204onto which the shadow302of the robot300is cast and the wall mirror202that now includes the reflection304of the robot300.

In some embodiments, the client may acquire full 3D format assets corresponding to the scene elements (in this example case, the wall mirror202and the floor204) that are visually impacted (by the addition of the interactive element300) from the content server and render those assets locally, thus being able to produce or render the visual impacts correctly. Further, in some embodiments, the content server itself may also assist in rendering of the visual impacts or may maintain multiple versions of pre-produced surface light fields (or, more specifically, surface light field representations of elements) which can be subsequently switched between during run-time based on state(s) of dynamic element(s).

According to some embodiments, the (viewing) client may signal local dynamic behaviors back to the content server, which in turn may estimate which parts of the static scene are impacted by the dynamic behavior(s) based on the original full 3D scene information. The server may then signal element(s) impacted back to the client, so that the client can download the impacted scene elements as full 3D assets and render them locally in order to have the whole scene reflect dynamic behaviors correctly.

To illustrate, in the context of the examples shown inFIGS.3-4, the client may signal an indication of a dynamic behavior of the robot300(the dynamic interactive element), while in turn, the server may estimate which parts of the static scene, as shown inFIG.2, are impacted by the dynamic behavior(s) of the robot300based on the original full 3D scene information. In this example, the server may determine that the portion of the floor204onto which the shadow302of the robot300is cast and the wall mirror202(that now includes the reflection304of the robot300) are impacted. Subsequently, the server may then signal information defining the impacted element(s) back to the client (in this case, the wall mirror202and the floor204) defined as full 3D assets. Accordingly, the client can render those elements defined in the 3D asset format locally in order to have the whole scene reflect dynamic behaviors correctly.

With a benefit of the above approach, the client does not need to download the full scene in a 3D format, but instead may need, e.g., only a minimal amount of data in the original 3D format and a minimal amount of real-time rendering while still being able to provide interactive dynamic behaviors. Furthermore, in some selected embodiments, the client may observe performance metrics and, based on the observed metrics, restrict possibilities for user interaction accordingly to avoid a need for local real-time 3D rendering that would exceed 3D rendering performance of a viewing device.

Further, in accordance with various embodiments of the present disclosure, a client device (including, e.g., a viewing client as described above) may receive, from a server (e.g., a content sever), surface light field representations a plurality of scene elements in a 3D scene, including a first scene element. The client device may subsequently provide to the server an indication of a dynamic behavior of a second scene element different from the first scene element. For example, during a viewing experience, the client device may add a dynamic interactive scene element to the scene, while the first scene element remains static. In response to the indication, the client device may receive from the server information defining the first scene element in a 3D asset format. Accordingly, the client device may render at least the first scene element in the 3D asset format. Accordingly, in some embodiments, a viewing client on the client device (e.g., a virtual reality (VR) client) may receive 3D scene data in a surface light field format (which, e.g., may be easier to render), but then the server may switch to providing assets in a full 3D format where needed to render scenes dynamically.

In some embodiments, a viewing client (e.g., client) reports interactivity to a server, and the client receives VR scene elements from the server as either 3D assets or surface light fields.

In some embodiments, a server (e.g., a VR server) receives interactivity reports from a client and provides VR scene elements as 3D assets when required by interactivity or as surface light fields otherwise. Further, in some embodiments, the server updates some of the surface light fields in view of interactivity. Yet further, in some embodiments, the server uses predefined surface light fields to allow for interactivity.

In some embodiments, scene elements represent different 3D objects (or different portions of a 3D object) within a scene. Scene elements may be nodes of a scene graph describing the scene.

In some embodiments, surface light field data for a scene element, such as a 3D object, includes (i) mesh data describing a surface geometry and (ii) light field texture data. For each of a plurality of points on the surface, the light field texture data indicates a color (e.g. a radiance, an RGB value, or an RGBA value) for each of a plurality of directions from the respective point. In some embodiments, data for a scene element in a non-light-field 3D format includes (i) mesh data describing a surface geometry and (ii) non-directional texture data. The non-directional texture data indicates a color (e.g. an RGB value or RGBA value) for the respective point. Data for a scene element in a non-light-field 3D format may further include information on reflectance and/or reflectivity of portions of the scene element or portions thereof (e.g. a bidirectional reflectance distribution function (BRDF)).

In some cases, the mesh data used in a surface light field representation of a scene element may be the same as the mesh data used in a non-light-field representation of the element. In other cases, the mesh data may be different. For example, the mesh data used in the surface light field representation may be simpler (e.g. may have fewer polygons and/or vertices) than the mesh data used in the non-light-field 3D format.

In some embodiments, a client device operates to determine, for a scene element within a 3D scene, whether the scene element is dynamic (e.g. is changing in position or appearance). In response to a determination that the scene element is dynamic, the client device retrieves the scene element from a content server in a non-light-field format. In response to a determination that the scene element is not dynamic, the client device retrieves the scene element from a content server in a surface-light-field format. The same determination may be made for a plurality of scene elements. The retrieved scene elements may be rendered by the client device. In some embodiments, the client device reports dynamic behavior of at least a first scene element (e.g. a change in position) to the content server, and the content server identifies to the client any additional scene elements that have become dynamic because of the dynamic behavior of the first scene element (e.g. scene elements that change in position or orientation due to motion of the first scene element, or scene elements with shadowing or lighting that is changed due to dynamic behavior of the first scene element). In some embodiments, a determination of whether to retrieve a scene element in a surface-light-field format or non-light-field format may be based at least in part on processing resources available to the client device; the number of scene elements retrieved in non-light-field format may be limited when client processing resources are limited.

Brief Summary of Example Processes and Entities

In some embodiments, an example process described above may involve (1) pre-processing stage executed by a content server, (2) a run-time stage executed by a content server, (3) and an execution stage performed by a viewing client.

More specifically, as explained above, surface light fields typically use pre-processing for producing static scene representation. In some embodiments, example pre-processing executed by a content server may involve, without limitation, a series of steps including the following: (1) receiving a full 3D scene; (2) producing a surface light field representation of the scene; and (3) storing the full 3D scene, surface light field representation, and a corresponding scene meta-data.

In some embodiments, an example run-time process executed by the content server may involve, without limitation, a series of steps including the following: (1) waiting for content requests from a viewing client; (2) providing the scene meta-data, the surface light fields and assets in an original full 3D format based on the client request; (3) receiving signaling from the viewing client indicating (i) which area(s) in the scene experience dynamic behavior(s, (ii) a current client viewpoint; (4) estimating area(s) visually impacted by the dynamic behavior(s) signaled by the viewing client; and (5) signaling 3D model(s) of content in the impacted areas to the viewing client.

In some embodiments, example process executed by the viewing client may involve, without limitation, a series of steps including the following: (1) requesting content from the content server; (2) receiving the scene meta-data from the content server; (3) downloading the scene in surface light field representation; (4) downloading locally rendered/interactive assets in a full 3D format; (5) processing user input and execute scene logic; (6) signaling a current viewpoint and dynamic behavior(s) of the scene to the content server; (7) receiving content server estimation of the visually impacted scene element(s); (8) downloading visually impacted element(s) in the full 3D format; (9) rendering the surface light field representation and the full 3D asset(s) as a combination; (10) observing performance metrics; (11) if needed, limiting element(s) that can have dynamic behavior; and (12) if end of session is not requested, returning to the step (4) of the client-side process.

Various processing stages are described in more detail below.

Advantageously, various embodiments of the present disclosure, may enable high-quality immersive interactive rendering on devices with, e.g., sub-optimal performance capabilities. Some benefits of embodiments disclosed herein include an ability to support dynamic content with, for example, minimal data and processing required by a client device and an ability for the client device to dynamically adjust user interactions so that local processing required by dynamic behavior(s) can be achieved with local computing performance.

Example System Arrangement and Detailed Operation

FIG.5is a block diagram of an example of a system arrangement500in which example embodiments of the present disclosure may be carried out.

As shown inFIG.5, the example system arrangement500includes a content server502(e.g., a virtual reality (VR) server) that may serve content requested by a client device504. The server502may include a first processor506and a first non-transitory computer-readable memory508containing a plurality of instructions510that are executable by the processor to carry out embodiments of various methods disclosed herein. For example, the executable instructions510may define program logic for pre-processing518that may be executed by the first processor506and may define logic for run-time processing520that may be executed by the first processor506(as noted above). Although not explicitly shown, the first processor506and the first memory508may be interconnected via a bus or a similar mechanism. As shown, the content server502may also contain or be coupled to a database512holding, e.g., scene(s) in a full 3D format (denoted as a “full 3D scene”). Further, the content server may also contain or be coupled to a database514holding, e.g., surface light filed representations of elements in the 3D scene (denoted as “surface light fields”). In some embodiments, surface light field data is produced by the content server502.

The content server502and the client device may504be coupled to each other for communication via any suitable wired and/or wireless network(s)534, such as the Internet and/or other networks. The client device504may include a viewing client516that, in some embodiments, may be a stand-alone application or may be integrated with another application run by the client device504. As shown, the viewing client516may include one or more modules522(e.g., program logic in the form of processor-executable instructions) for providing scene management, input processing, and rendering. The client device504may also include a second processor528and a second non-transitory computer-readable memory530containing a plurality of instructions532that are executable by the client device504to carry out embodiments of various methods disclosed herein. Although not explicitly shown, the second processor528and the second memory530may be interconnected via a bus or a similar mechanism. As further shown, the client device504may also contain a surface light field cache532(e.g., for storing surface light field element representation(s), e.g., received from the server502) or a 3D asset cache (e.g., for storing information defining element(s) in a 3D asset format), e.g., received from the server502. Although not explicitly shown, coupled to the viewing client may be any suitable tracking and input means for receiving user inputs, navigation, etc. and a display that displays content rendered by the viewing client to a user.

Note that various entities shown inFIG.5may be coupled to each other via any suitable wired and/or wireless links and/or other intermediate elements now shown.

FIG.6illustrates an example message sequence diagram600, in accordance with some embodiments. The example message exchange shown inFIG.6takes place between a server602and a client604.

As shown inFIG.6, initially, at606, the server602may produce surface light fields for 3D scene elements (or elements in a 3D scene). On the other hand, at608, the client604may initially collect sensor and configuration data. Subsequently, at610, the client604may request content, from the server, that corresponds to the 3D scene.

In response, at612, the server602may first provide scene description meta-data to the client604. Based on the received scene description meta-data, at614, the client604may determine, for instance, on a per-element basis, element formats to be requested from the server602. As shown, the client604may then, at616, request from the server element(s) to be represented by surface light fields and element(s) corresponding to assets in a full 3D format.

At618, after a receipt of the requested scene elements, the client604may process sensor data and user input at620. Further, at622, the client604may execute scene logic (contained in the scene description meta-data) and update a viewpoint. Next, at624, the client604may signal dynamic behavior(s) of scene element(s), together with the updated viewpoint, back to the server602. In response, at626, the server may evaluate one or more elements impacted by dynamic behavior(s) and, at628, signal those impacted element(s) back to the client604. In turn, at630, the client604may request from the server602the impacted element(s) in a full 3D asset format, e.g., may send a request at632to request asset(s) corresponding to those impacted elements(s) in a full 3D format, as shown inFIG.6

Furthermore, in response to the client request, the server302may provide the requested asset(s) in the full 3D format to the client at634. Subsequently, at636, the client may render a combination of surface light fields and 3D assets. The client may also observe performance metric(s) during content execution, and limit interaction if so indicated by the metric(s).

FIG.7illustrates an example message sequence diagram700in more detail, in accordance with some embodiments. In some embodiments, a sequence of communication between entities shown inFIG.7may occur in a typical use session. Accordingly, as shown by way of example inFIG.7, the typical use session may involve interactions among a user702, a viewing client704, a content server706, and a content provider708.

Among various functions illustrated inFIG.7, in some embodiments, at the beginning of a session, a viewing client704downloads scene meta-data describing, for example, scene elements, surface light field geometry and asset correspondence, and scene logic. Using the scene meta-data the viewing client704may then optimize download and client-side processing. In some embodiments, the client optimizes the download by choosing to download in an original full 3D format only those assets corresponding to scene elements that the viewing client wants to feature as having an interactive dynamic behavior. Further, in some embodiments, when executing a 3D scene experience, the viewing client signals interactive dynamic behavior(s) back to the content server. In turn, the content server, having full 3D scene information may then determine which area(s) of the full scene are impacted by the interactive behavior(s) and signal that information back to the viewing client. Accordingly, in some embodiments, the viewing client does not need to have the full 3D scene while being aware of what area(s) are impacted by the interactive behavior(s), e.g., know the area(s) of the full scene that display visual changes due to the interactive element(s).

More specifically, referring toFIG.7, during a content pre-processing stage710, at712, the content provider708may provide an original 3D scene and a navigation area (both in any suitable format) to the content server706. The 3D scene may include multiple scene elements. In turn, at714, the content server714may produce surface light fields for a selected viewing area, such as surface light field representations of scene elements within the viewing area. This may complete the content pre-processing stage710.

During the following content distribution stage716, a message exchange may take place between the user702, the viewing client704, and the content server706. In particular, at718, the user702may submit a content request to the viewing client704via any suitable interaction with the viewing client704. At720, the viewing client704may pass the content request to the content server706. Further, at722, the viewing client704may collect sensor and configuration data. In return, at724, the content server may provide scene description meta-data to the viewing client704. As noted above, using the scene meta-data the viewing client704may then optimize download and client-side processing. For instance, in the example ofFIG.7, at726, the viewing client may select an initial viewpoint and scene element(s) to be rendered locally. Accordingly, at728, the viewing client704may request, from the content server706, surface light fields (surface light field representations) for some selected element(s) (e.g., static element(s)) and full 3D assets for other element(s) (e.g., only those assets corresponding to scene element(s) that the viewing client704wants to feature/render locally as having an interactive dynamic behavior). At730, the content server may serve the requested surface light fields and full 3D assets to the viewing client704.

Subsequently, during a run-time (loop) stage732, at734, the user702may provide a user input to the viewing client704. At736, the viewing client704may process user input and scene logic, and update a viewpoint accordingly. At738, when executing a 3D scene experience for instance, the viewing client704may signal an indication of (interactive) dynamic behavior(s) (e.g., an indication of a dynamic behavior of one or more scene elements and a viewpoint back to the content server706. In response, at740, the content server706, having, e.g., full 3D scene information may then evaluate which area(s) of the full scene are impacted by the interactive behavior(s)/dynamic elements(s), and signal that information back to the viewing client704, at742, by providing an indication of assets impacted by the dynamic element(s). In response, at744, the viewing client704may send a request for certain 3D assets to be rendered locally. Accordingly, as noted above, the viewing client704does not need to have the full 3D scene while being aware of what area(s) are impacted by the interactive behavior(s), e.g., knowing the area(s) of the full scene that display visual changes due to the interactive element(s).

Then, at746, the content server706may provide the requested 3D assets. Also, at748, the content server may update surface light fields and, at750, send the updated surface light fields to the viewing client704. As a result, at752, the viewing client704may render a combination of the surface lights fields and 3D assets available at the viewing client704. Additionally, in some embodiments, the viewing client, at754, may observe QoE (Quality of Experience) metrics and, and756, restrict dynamic behavior of the scene according to (or based on) the observed QoE metrics. Note that the run-time (loop) stage may732may be “looped” (iteratively repeated) throughout a 3D-scene viewing experience of the user702.

Various processing steps of embodiments disclosed hereinabove are described in more detail below.

Content Server Pre-Processing

FIG.8is a flow chart800illustrating an example pre-processing of content executed by a content server, according to some embodiments. As a general matter, in the pre-processing stage, in some embodiments, the content server may produce surface light fields for assets of an original full 3D scene.

Referring toFIG.8, the process starts at step802, when a full 3D scene is uploaded to the content server for distribution. Uploaded information may contain, for example, all scene assets, a scene graph describing the scene structure and scene logic describing interactive behavior(s) of the scene. In some embodiments, surface light fields may support only a limited viewing volume, so a content provider uploading the full 3D scene to the content server may also determine a viewing area for which a surface light field representation is to be created. The content server may store, at step806, data corresponding to the full 3D scene for subsequent processing.

At step804, the optimized scene geometry representation suitable for surface light field representation of the 3D scene may be produced using Seurat or any other suitable tool. Namely, at804, the content server renders RGB-D images (that, at step808, may be stored at the content server for further processing) from random locations inside the viewing area (e.g., for which a surface light field representation is to be created, as noted above). In general, Seurat uses several RGB-D images from random locations within a selected viewing volume from which it produces optimized geometry representation of a given scene. Such optimized representation can feature view-dependent shading effect by some form of surface light fields.

At step810, the content server produces surface lights fields (that, at step812, may be stored at the content server for further processing) from the RGB-D images for each viewing area. In some embodiments, using a suitable light field creation tool, surface light field representation is only produced for static element(s) and element(s) with pre-determined dynamic behavior(s), scene element(s) that are intended to feature client-controlled dynamic behaviors are omitted or recorded as part of the surface light field in their scene start-up state. In addition to creating a surface light field representation, in some embodiments, the content server also records a correspondence between original 3D assets and resulting surface light field geometry. The correspondence information may be stored along with scene meta-data and, in some embodiments, used by a client in order to remove surface light field element(s) being rendered using full 3D asset(s).

As shown inFIG.8, at step814, based on the full 3D scene data, the content server compiles various information, including scene meta-data describing the scene, (scene) logic, and assets, e.g., associated with scene elements. The scene-meta data held at the content server (step816) may include a link to the surface light field data, scene graph describing hierarchical relationships between scene elements (e.g., assets with links to the actual asset data), scene logic (e.g., behavior of assets in terms of relationship with other assets), timeline, user input, a correspondence between the surface light field geometry and assets, and/or the like. Subsequently, the example pre-processing of content executed by the content server may terminate at step818.

Example Content Server Run-Time Processing

Once the content server has performed pre-processing of the 3D scene, the content server may start run-time processing during which it distributes content for viewing clients (or content distribution stage, as illustrated in the example ofFIG.7).FIG.9is a flow chart900illustrating an example run-time processing executed by a content server, in accordance with some embodiments.

In general, when a viewing client starts a new session, the client may first request and download scene meta-data from the content server. Based on the scene meta-data, the viewing client may then request and download scene 3D assets and surface light fields, as needed. More specifically, in some embodiments, when the viewing client is executing the session, the client may process scene logic locally based on local context and user input. As described above, the scene logic may be provided to the viewing client by the content server as a part of the scene meta-data. As the client executes the scene logic, interactive element(s) of the scene may be accordingly updated locally on the client side. In some embodiments, the client signals the interactive dynamic behavior(s) of the locally-processed scene elements back to the content server. Based on the signaled information, the content server that has full 3D scene data may be able to estimate (e.g., resolve) scene area(s) that are visually impacted (affected) by the interactive dynamic behavior(s) indicated by the client. Further, in some embodiments, in addition to the interactive dynamic behavior(s), the viewing client signals a current viewpoint to the content server.

In order to determine the visually impacted area(s), in some embodiments, the content server may perform local rendering or low sample rate scene visibility/light transport simulation. An indication (e.g., an estimation) of the impacted areas may be then signaled back to the viewing client by the content server. In some embodiments, the indication includes list of impacted assets, but in other embodiments, the indication may be in a different form. Once the viewing client receives the indication of the impacted areas, the client may download assets corresponding to those areas in a full 3D format to be rendered locally.

More specifically, referring toFIG.9, the process may start at step902when one or more viewing clients, e.g., (i) request a new session from a content server, (ii) request for scene 3D assets and/or surface light fields, as needed, from the content server, or (iii) signal other type of information to the content server including an indication of a dynamic behavior of scene element(s). At step904, the content server waits for request(s) or signaling from viewing client(s). For purpose of illustration only, the example ofFIG.9will assume that the content server receives a request or signaling from a single viewing client. However, in practice, the content server may handle multiple client requests and/or signaling, and process each client request or signaling according to the example process shown inFIG.9.

At step906, the content server determines if communication received from the viewing client is a client request or if the client is signaling scene dynamic behavior(s). The client request may be, for example, one of two types of requests: (1) a request associated with a start of a new session (e.g., an initial request for scene meta-data for a 3D scene) or (2) a request for scene 3D asset(s) and/or surface light fields, as needed.

In one example, the request for scene 3D asset(s) and/or surface light fields may be based on the scene meta-data that has been already sent to the viewing client by the content server (such as based on an earlier client request for a new session). In another example, the viewing client may generate such request during client execution of scene logic, where interactive (dynamic) element(s) of the scene may be requested/downloaded from the content server and rendered locally on the client side.

If, at step906, the content server determines that the client communication is a client request, the process moves to step908at which the content server determines a type of the received client request. If, at step908, the request is determined to be a request associated with a start of a new session by the viewing client, the process moves to step916, at which scene meta-data is sent to the viewing client by the content server. The scene meta-data may be stored at or otherwise made available to the content server (step924). Afterwards, the process may return to step904. As described earlier, based on the scene meta-data, the client may then request and download surface light fields and scene 3D assets, as needed, to be rendered locally by the viewing client.

If, at step908, the request is determined to be a request for scene 3D asset(s) and/or surface light fields, the process moves to step910at which the content server sends (to the viewing client) data according to such request (e.g., the content server sends surface light field representation(s) and/or 3D assets for requested scene element(s)). Although not explicitly shown inFIG.9, the content server may store (or otherwise have an access to) data of a scene in a full 3D format and data representative of surface light fields. Then, at step912, the content server may determine whether an end of content serving processing has been requested (e.g., explicitly by the client or otherwise), and if so, terminate the process at step914. Otherwise, the process may return back to step904.

Referring back to step906, as noted above, in this step, the content server may determine that, instead of the client request, the content server has received (from the viewing client) a signal indicating scene dynamic behavior(s). For instance, as described in connection withFIG.7, the viewing client may process user input and the scene logic, and update a viewpoint accordingly. Afterwards, when executing a 3D scene experience, the viewing client may signal an indication of (interactive) dynamic behavior(s) (e.g., an indication of a dynamic behavior of one or more scene elements) and the viewpoint back to the content server.

If that is the case, the process moves to step918at which the content server determines scene area(s) affected by the dynamic behavior(s) indicated by the viewing client. In this regard, as described in connection withFIG.7, the content server, having, e.g., full 3D scene information may evaluate which area(s) of the full scene are impacted by the interactive behavior(s)/dynamic elements(s), and, at step920, signal that information back to the viewing client by, e.g., providing an indication of assets impacted by the dynamic behavior(s). As further noted above, in order to determine the visually impacted area(s), in some embodiments, the content server may perform local rendering or low sample rate scene visibility/light transport simulation. Further, the indication (e.g., an estimation) of the impacted area(s) may be in the form of a list of impacted assets or in some other form.

In response, at step922, the content server receives (from the viewing client) a request for one or more scene elements in a 3D asset format to be rendered locally by the viewing client. For example, a dynamic behavior of a particular element may impact one or more other different elements (e.g., static elements). Accordingly, the client may request 3D assets corresponding to those different element(s) that have been affected by the dynamic behavior of that particular element.

The process may then return to step908. Then, at step910, the content server would send (to the viewing client) data according to the 3D asset request the content server. In this particular case, the content server would, e.g., provide the viewing client with data corresponding to information defining one or more scene elements (that were affected by the scene dynamic behavior(s)) in a 3D asset format. Although not explicitly shown inFIG.9, in addition to the requested element(s) in a 3D asset format, in some embodiments (as described earlier), the content server may also update surface light fields. Accordingly, at step910, the content server may also send the updated surface light fields to the viewing client so that the client may, e.g., download and render locally a combination of the of the surface lights fields and the requested 3D assets.

The process may then proceed to step912at which the content server determines whether the end of content serving processing has been requested. Subsequently, the process either loops back to step904or terminates at step914.

Example Viewing Client Processing

FIG.10is a flow chart1000illustrating an example processing executed by a viewing client, in accordance with some embodiments. In some embodiments, at step1002, the process executed by the viewing client starts when a user, for example, launches on a client device an application implementing the viewing client. For instance, when the user starts the application, he or she may also define content to be viewed. In some embodiments, content includes a link to scene meta-data residing on a content server. The link to the scene meta-data may be a Uniform Resource Locator (URL) identifying the content server and a specific file stored at the server. The viewing client application may be launched either by an explicit command provided by the user (e.g., via a suitable user input) or automatically by an operating system of the client device based on a request identifying content type and an application associated with that specific content type. In this regard, in some embodiments, rather than being a stand-alone application, the viewing client maybe integrated with a web browser, may be integrated with a social media client, and/or or may be a part of the operating system.

In some embodiments, when the viewing client application is launched, the application launch may also initialize sensor data collection. Further, in some embodiments, when the viewing client has initialized sensor data collection, the viewing client may select an initial viewpoint and select surface light field and full 3D asset(s) to be downloaded initially using the scene meta-data. Accordingly, at step1004, the viewing client requests content from a content server, and at step1006initializes tracking of user input. The user input may include tracking of the user head that controls the viewpoint within the 3D content when HMD is being used and/or tracking of other input devices/user motions that the user can use to control the virtual experience created with the 3D content.

At step1008, the viewing client receives scene meta-data from the content server. At step1010, the viewing client may store (e.g., locally at the client) the received scene meta-data for execution. Based on the scene meta-data, at steps1012and1014, the viewing client may request and download surface lights fields and 3D assets to be rendered locally. At step1010, the viewing client selects a (initial) viewpoint. Based on the scene meta-data, at steps1012and1014, the viewing client may request and download surface lights fields and 3D assets to be rendered locally. The surface light fields and 3D assets may be locally stored, at steps1016and1018, such as in respective caches at a client device. Note that, although steps1012and1014are shown as separate steps, those steps may be combined in to a single step (e.g., the viewing client downloading both types of assets at the same time), executed concurrently, etc.

Once the viewing client completes a download of the surface light fields and the 3D assets, the viewing client may begin to execute a virtual experience by processing the sensor data and user input and updating the scene based on the processed input data and scene logic described in the scene meta-data. As the client executes the scene logic, some scene elements may feature dynamic behavior as a result of the executed scene logic and the user input. In some embodiments, the viewing client signals an indication of all elements currently having dynamic behavior to the content server. Namely, at step1020, the viewing client may process user input and scene logic (included in the scene meta-data), and, at step1022, update the viewpoint and signal a current viewpoint and an indication of dynamic element(s) to the content server. The viewing client may also signal to the content server a current pose (e.g., a transformation in terms of scene graph coordinates) of those elements, together, e.g., with the current viewpoint used for displaying the scene to the user.

Based on the client signaling of the interactive dynamic behaviors, the content server determines (e.g., estimates) scene elements visually impacted by those behaviors and signals a suitable indication of the visually impacted elements back to the viewing client. Accordingly, in some embodiments, at step1024(update), the viewing client may receive the indication, which by way of example, is a list indicating assets for one or more elements visually impacted by the dynamic element(s). Accordingly, at step1026, using the list of the impacted assets received from the content server and also, for example, inspecting the scene meta-data (especially, e.g., scene logic of assets around a current user location and timeline information for near future events), the client may decide which scene element(s) to request in a full 3D (asset) format in order to be able to perform rendering locally. This way, the client may be able to visualize impact of interactive dynamic behaviors in real-time. When the client decides which elements to request in a 3D format, at step1028, the client may then download corresponding 3D assets (or the assets corresponding to the visually impacted elements) from the content server. As shown inFIG.10, the viewing client may locally store the downloaded corresponding 3D assets in, e.g., a local cache (step1018).

In some embodiments, the viewing client may continuously update and render the scene. According to illustrative embodiments, in rendering of the scene, as shown inFIG.10, at step1030, the viewing client combines surface light field representation(s) and asset(s) downloaded in their full 3D format to be rendered locally using, e.g., normal 3D rendering. In this regard, as depicted inFIG.10, the viewing client may pull the surface lights fields and 3D assets from, e.g., local storage (see steps1016and1018). Further, in some embodiments, during rendering of the combination, geometries of surface light field assets and 3D assets may be uploaded to a graphics processing unit GPU as normal geometry in a unified coordinate space when the viewing client has first removed the geometry representing assets in full 3D format from the surface light field geometry. This functionality may be performed by the viewing client using the geometry correspondence described, for example, as a part of the scene meta-data. Once the up-to-date geometry of both surface light fields and individual assets in full 3D format are present in the graphics processing unit (GPU), the GPU may perform rendering normally, while using, for instance, surface light field specific shaders on surface light field geometry that takes into an account texture processing of the current viewpoint.

Furthermore, during the execution of the (virtual) experience, in some embodiments, at step1032, the viewing client may also observe one or more performance metrics, such as, e.g., a processing load and a rendering frame rate, in an effort to create a balance between dynamic interactive events in the scene and processing resources. As an example, in some embodiments, if a processing performance drops (e.g., the processing resources are insufficient to process amount of dynamic interactive events present in the experience), at step1034, the viewing client limits dynamic elements according to the observed performance metrics. By way of example, the viewing client may limit a number of interactive events and, e.g., a number of assets featuring interactive dynamic behavior in order to maintain processing within client device performance limits, and thus maintain quality of experience.

In some embodiments, continuous execution of the experience is carried out by the viewing client until an end of processing is requested by either the user or is indicated by the scene logic in the scene meta-data. More particularly, at step1036, the viewing client checks whether an end of processing has been requested, and if not, the process returns to step1020. Otherwise, the process terminates at step1038

FIG.11is a flow chart illustrating an example method1100, in accordance with some embodiments. In illustrative embodiments, the method is performed by a client device (including, e.g., a viewing client as described above). At step1102, the client device receives, from a server, surface light field representations of a plurality of scene elements in a 3D scene, including a first scene element. At step1104, client device provides to the server an indication of a dynamic behavior of a second scene element different from the first scene element. At step1104, in response to the indication, the client device receives from the server information defining the first scene element in a 3D asset format. Finally, at step1106, the client device renders at least the first scene element in the 3D asset format.

FIG.12is a flow chart illustrating an example method1200, in accordance with some embodiments. In illustrative embodiments, the method is performed by a server (e.g., a content server as described above). At step1202, the server sends, to a client device, surface light field representations of a plurality of scene elements in a 3D scene, including a first scene element. At step1204, the server receives, from the client device, an indication of a dynamic behavior of a second scene element different from the first scene element. At step1206, the server determines that the first scene element is visually affected by the dynamic behavior of the second scene element. Finally, at step1108, the server sends, to the client device, information defining the first scene element in a 3D asset format.

Some Variations of Example Embodiments

Example variations involving a server-side processing, in accordance with some embodiments, will now be described.

Various embodiments described herein primarily use client-based pull model, where, e.g., all content updates and downloads are determined by a viewing client. In alternative embodiments, at least a portion of processing may be shifted onto a server side.

To illustrate, in some embodiments, the server may process part of content rendering. In one such variation, the content server, when determining (e.g., estimating) a visual impact of interactive dynamic behavior of one or more scene elements, may also determine (e.g., estimate) if some of the visual impact(s) can be rendered on the server side. For example, in some embodiments, the content server may update a certain area of a surface light field texture that is specific to the viewing client or the content server may pre-produce several versions of the surface light fields in an anticipation of certain impacts from the dynamic behavior(s). Subsequently, during a run time, the content server can swap between multiple surface light field versions depending on the state of the dynamic behavior(s). As a result, in some cases, the viewing client may not operate to download full 3D assets from the content server but instead just receive from the server a minimal texture update for the existing surface light field.

In some embodiments, the content server may extend surface light fields from static elements to also pre-defined dynamic elements. In this regard, instead of providing a single surface light field, the content server may provide streamed surface light field geometry and textures containing, e.g., pre-defined dynamic deformations and changes. In this example variation, the server may provide a more complex content distribution to support streaming nature of a portion of the content and, for example, may also provide some signaling to the client when updates to the surface light field data are available.

In some embodiments, a navigation area supported by the surface light field may be extended by having or producing not just a single viewing volume, but instead, producing several interleaved navigation areas on the server side so that the viewing client can swap from one viewing area to another viewing area, such as to enable, for instance, a wider area 6DoF navigation.

Various example embodiments of the present disclosure have been described herein above. Further, some example concepts in various methods described herein may include, without limitation, the following concepts listed below.

In some embodiments, surface light fields used to represent static scenes are combined with real-time rendering of interactive elements.

In some embodiments, content is delivered in a hybrid format. In this regard, scene elements may be available at a content server both as optimized surface light fields and full 3D assets. In some embodiments, a client is able to select complexity based on bandwidth and rendering power. In some embodiments, the client is signaled rendering complexity in addition to an object size and/or bitrate.

In some embodiments, the client signals dynamic scene behavior(s) to the content server, which then determines (e.g., estimates) which part(s) of the scene are impacted by the dynamic behavior(s). In this regard, the content server may update a media presentation description (MPD) or other manifest file. In some embodiments, client-side logic may be used with a scene graph (e.g., provided as a part of scene meta-data) to determine part(s) of scene and a level of detail to be requested from the content server.

In some embodiments, the content server determines correct content format for scene elements for the client, such as surface light fields for static elements and 3D assets for elements impacted by the dynamic behaviors.

In some embodiments, the content server signals an indication of the impacted elements back to the client. The client may download requested assets in a full 3D asset format and render the impacted elements locally.

In some embodiments, the client does not need to download a full scene in a full 3D asset format. Instead, in some embodiments, the client downloads, e.g., only a minimal number of elements in the full 3D asset format and, e.g., minimal real-time 3D rendering.

In some embodiments, the client adapts content to local capabilities of a viewing device by observing performance metric(s) and responsively limiting interactivity (interactive dynamic behavior) such as, e.g., to avoid local real-time 3D rendering so as not to exceed 3D rendering performance capabilities of the viewing device.

Additional (e.g., related) embodiments have been described hereinabove.

According to some embodiments, a method, performed by a client device, includes: requesting content from a server, the content including a three-dimensional (3D) scene having a plurality of scene elements; downloading, from the server, surface light field representations of selected scene elements; providing to the server an indication of a dynamic behavior of one or more scene elements; receiving, from the server, an indication of at least one scene element that is visually affected by the dynamic behavior of the one or more scene elements; downloading, from the server, the at least one visually affected scene element in a 3D format; rendering a combination of (i) the surface light field representations of the selected scene elements and (ii) the at least one visually affected scene in the 3D format; and displaying a result of the rendering to a user.

In some embodiments, the method further includes: prior to downloading the surface light field representation of the selected elements in the 3D scene, receiving from the server a scene description meta-data. The method may further include downloading, from the server, one or more 3D assets to be rendered locally at the client device.

In some embodiments, providing to the server the indication of the dynamic behavior of the one or more scene elements includes processing at least a user input and scene logic included in the scene description meta-data to determine the one or more scene elements that have the dynamic behavior as a result of the processing. Further, the client device may process sensor data collected by the client device.

In some embodiments, providing to the server the indication of the dynamic behavior of the one or more scene elements includes providing the indication together with a current viewpoint used for displaying the 3D scene to the user. In this regard, in the above methods, the surface light field representations of the selected scene elements are produced by the server. Further, in some embodiments, downloading, from the server, the at least one visually affected scene element in the 3D format includes the client determining to download the at least one visually affected scene element in the 3D format based on scene logic in the scene description meta-data. Here, the scene logic may include scene logic corresponding to assets around a current user location and timeline information for future.

In some embodiments, the above-methods further include observing one or more performance metrics; and limiting an amount of interactive behavior in the 3D scene when a local rendering performance at the client device falls below a threshold. The one or more performance metrics may include at least one of processing load or rendering frame rate. Further, limiting the amount of interactive behavior in the 3D scene includes limiting at least one of a number of interactive events or a number of scene elements having the dynamic behavior.

In some embodiments, the method may be executed repeatedly, and further include receiving, from the server, one or more updated surface light field representations.

In some embodiments, rendering the combination includes combining the surface light field representations of the selected scene elements and the at least one visually affected scene element in the 3D format together. In some embodiments, combining includes providing a combined geometry in a unified coordinate space to a graphics processing unit (GPU) for rendering.

According to some embodiments, another method, performed by a client device, includes: requesting spatial content from a server; receiving, from the server, a scene description meta-data describing the spatial content; based on the scene description content meta-data, selecting content elements to be rendered locally at the client; requesting, from the server, the selected elements in a surface light field or 3D asset format, wherein at least a portion of the selected elements is requested in the surface light field format; receiving, from the server, the requested elements; processing user input and scene logic contained in the scene description meta-data to determine one or more scene elements featuring dynamic interactive behavior; providing a current viewpoint and an indication of the dynamic behavior of the one or more scene elements to the server; receiving server estimation of at least one scene element impacted by the dynamic behavior of the one or more scene elements; downloading, from the server, the at least one impacted scene element in a 3D asset format; rendering and displaying a combination of surface light field elements and one or more scene elements corresponding to locally rendered 3D assets; and observing at least one performance metric and responsively limiting an amount of interactive behaviors if a local rendering performance falls below a threshold.

According to some embodiments, a client device includes a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform any of the methods performed by the client device and disclosed hereinabove.

According to some embodiments, a method, performed by a server, includes: receiving a content request from a client device, the content including a three-dimensional (3D) scene having a plurality of scene elements; providing, to the client device, surface light field representations of selected scene elements; receiving, from the client device, an indication of a dynamic behavior of one or more scene elements; providing, to the client device, an indication of at least one scene element that is visually impacted by the dynamic behavior of the one or more scene elements; receiving, from the client device, a download request for the at least one visually impacted scene element in a 3D format; and providing, to the client device, the at least one visually impacted scene element in the 3D format.

According to some embodiments, a server includes a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform any of the methods performed by the server and disclosed hereinabove.

According to some embodiments, a system includes: a content server configured to deliver content corresponding to a three-dimensional (3D) scene, wherein the content is delivered as surface light fields and 3D assets; and a client device configured to (i) receive the content, (ii) signal to the content server one or more static portions of the 3D scene affected by a dynamic behavior of one or more scene elements, and (iii) receive from the content server 3D assets corresponding to the one or more portions of the 3D scene affected by the dynamic behavior of the one or more scene elements.

According to some embodiments, disclosed herein is a method that includes combining surface light fields used to represent static scene elements with real-time rendering of interactive elements at a client device.

According to some embodiments, another method is disclosed that includes delivering content to a client device in a hybrid format, wherein the hybrid format includes surface light fields and three-dimensional (3D) assets, where a first portion of the 3D assets is delivered to the client device upon content request and a second portion of the 3D assets is delivered to the client device in response to the first portion causing a dynamic behavior affecting one or more static elements.

Further, according to some embodiments, a system is disclosed that includes a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform any of the methods disclosed hereinabove.

Note that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.