Patent Description:
The MPEG-DASH protocol addresses dynamic variation in streaming media distribution bandwidth by focusing on video content. Some previous systems of adaptive spatial content streaming focus on a single spatial content type, such as 3D data in a polygon mesh format. Some systems adjust to bandwidth limitations and to computing performance at the client.

Contrary to video content, where streamed data is always essentially a sequence of image frames, spatial data may have much more variability in how the content is organized and intended to be used for producing the images finally at the client side sent to the display. Different content formats have different characteristics and variation in content quality, memory consumption, and freedom of navigation permitted. Furthermore, some spatial content formats may in some cases require a large amount of content assets to be downloaded before the content rendering may begin. A known technology is disclosed in <CIT>.

An method and apparatus in accordance with the claimed invention includes features as recited in the appended claims.

The entities, connections, arrangements, and the like that are depicted in-and described in connection with-the various figures are presented by way of example and not by way of limitation. As such, any and all statements or other indications as to what a particular figure "depicts," what a particular element or entity in a particular figure "is" or "has," and any and all similar statements-that may in isolation and out of context be read as absolute and therefore limiting-may only properly be read as being constructively preceded by a clause such as "In at least one embodiment,. " For brevity and clarity of presentation, this implied leading clause is not repeated ad nauseum in the detailed description.

A wireless transmit/receive unit (WTRU) may be used, e.g., as a content server, a viewing client, a head mounted display (HMD), a virtual reality (VR) display device, a mixed reality (MR) display device, and/or an augmented reality (AR) display device in some embodiments described herein.

As shown in <FIG>, the communications system <NUM> may include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN <NUM>/<NUM>, a CN <NUM>/, a public switched telephone network (PSTN) <NUM>, the Internet <NUM>, and other networks <NUM>, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements.

Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN <NUM>, the Internet <NUM>, and/or the other networks <NUM>.

For example, the base station 114a in the RAN <NUM>/<NUM> and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface <NUM> using wideband CDMA (WCDMA).

The RAN <NUM>/<NUM> may be in communication with the CN <NUM>, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. Although not shown in <FIG>, it will be appreciated that the RAN <NUM>/<NUM> and/or the CN <NUM> may be in direct or indirect communication with other RANs that employ the same RAT as the RAN <NUM>/<NUM> or a different RAT. For example, in addition to being connected to the RAN <NUM>/<NUM>, which may be utilizing a NR radio technology, the CN <NUM> may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA <NUM>, WiMAX, E-UTRA, or WiFi radio technology.

In view of <FIG>, and the corresponding description of <FIG>, one or more, or all, of the functions described herein with regard to one or more of: WTRU 102a-d, Base Station 114a-b, and/or any other device(s) described herein, may be performed by one or more emulation devices (not shown).

Communication of spatial data may increase demand for content streaming bandwidth and for the ability to dynamically adapt to the changing of resources available. For 2D video content, some systems adjusted just the resolution and compression rate across the whole image area depending on the available bandwidth. Some embodiments disclosed herein may balance between bandwidth consumption and quality of experience (QoE) metrics. For example, if using spatial data, reducing the content navigation area instead of reducing the resolution may result in a better QoE depending on viewing conditions.

The complexity of requirements are increasing. MPEG-DASH addresses dynamic variation in the streaming media distribution bandwidth by focusing on video content. With spatial media, a dynamic adaptive streaming process may use a multitude of spatial content formats and additional contextual conditions. These conditions may include variation from session to session and variations within a session, such as type and number of display devices, number of users, and environment layout. Systems and methods disclosed herein in accordance with some embodiments may account for these conditions by balancing bandwidth and quality of experience (QoE) parameters.

With spatial data, content may be distributed using a larger selection of content formats. Different content formats may have different characteristics and variations in content quality, memory consumption, and freedom of navigation permitted.

Some adaptive spatial content streaming devices focus on a single spatial content type, namely 3D data in polygon mesh format. See the following three articles: <NPL>), <NPL>), and <NPL>). These articles expand content adjustment schema at the client side from just adjusting to bandwidth limitations to also adjusting to computing performance at the client side. Zampoglou investigates applicability of the MPEG-DASH standard to transmit 3D data with multiple levels of detail (LoD) together with associated metadata. Lavoué and Evans both propose a progressive algorithm for 3D graphics data suitable for adaptive LoD streaming.

Expanding adaptive spatial data streaming by considering multiple spatial data formats is understood to not yet be much explored. Spatial data, such as light fields, may enable free content navigation while providing higher visual quality than 3D polygon mesh data. Light fields may be formatted as an array of images that may be used together to enable viewpoint adjustment within a limited viewing volume. For adaptive streaming, if only limited content distribution bandwidth is available, a better QoE may be achieved for the end user by limiting both the resolution and the motion parallax (the number of distinct views).

For some embodiments, dynamically adaptive streaming of spatial data may balance quality of experience (QoE) and available resources. As the number of available data formats increases, a larger selection of parameters, such as, e.g., light field resolution, area for which motion parallax is supported, and spatial data format, may be used. Information about available spatial data formats and suggested use may be communicated from a content server to a viewing client. The viewing client may adapt such spatial content to meet session conditions. Dynamic streaming of spatial data may use a content server streaming spatial content with various formats and quality settings, allowing a viewing client to dynamically adapt the content streaming to the viewing conditions within limits of the available bandwidth, client performance, and per session conditions for some embodiments. In addition to several quality and format streams, the server provides metadata about the available streams to the viewing client. A viewing client may select streams to be used based on information about, e.g., the content received as metadata from the server, the contextual information the viewing client has about the viewing conditions, available bandwidth, and processing resources for some embodiments.

Systems and methods disclosed herein in accordance with some embodiments may use a content server that communicates to a viewing client the available content streams for levels of freedom for content navigation. A viewing client may use such levels of freedom of navigation in addition to levels of detail (LoD) as an adjustment parameter. Based on the freedom of content navigation schemas, the client may adjust the content complexity and the amount of data communicated. For some embodiments, freedom of content navigation uses levels of degrees of freedom (DoF) to classify content streams and assets. In some embodiments, levels used in the DoF schemas indicating various levels of freedom of content navigation are, e. g, 0DoF, 3DoF, 3DoF+, and 6DoF. For example, degrees of freedom representations may comprise 0DoF, 3DoF, 3DoF+, and 6DoF representations of content.

Based on the content, the content server compiles DoF schema and LoD versions according to the different spatial content formats and quality versions in a manifest file, such as, e.g., a media presentation description (MPD) file or a set of one or more files (such as an XML document) that include metadata that may be used for configuring a device. In some embodiments, at the beginning of a streaming session, the viewing client loads the MPD. Based on the MPD, current conditions, and current capabilities, the viewing client may select a version of the data to be downloaded. Content segment format and resolution may be adapted to meet data transmission parameter and quality metric thresholds for available resources. For some embodiments, representation of content may be selected based in part on client capabilities and/or range of motion of a client. In some embodiments, capabilities of a client device may include, e.g., one or more capabilities such as, display characteristics, such as, e.g., resolution, display size, pixel size, number of dimensions supported, degrees of freedom supported (e.g., 0DoF, 3DoF, 3DoF+, and 6DoF), levels of detail supported, bandwidth supported, processing power, processing performance, start-up delay, latency delay, image quality, and spatial content types supported. A start-up delay may include a latency delay waiting for a full geometry to be available at the client device prior to starting 3D rendering of an object, such as a 3D polygon mesh. It will be understood that "capabilities of a client device" will in general refer to, e.g., one or more (e.g., relevant) capabilities of a client device with respect to, e.g., context, such as content representation, not, e.g., in general to every literal "capability" of a client device, regardless of or divorced from context or relevance.

For some embodiments, the content server may execute a process that includes: receiving spatial data; generating (which may include producing and organizing) LoD versions of the spatial data; generating (which may include producing and organizing) DoF versions of the spatial data; generating (which may include producing) an MPD for a scene; waiting for content requests from viewing clients; sending the MPD to the client; and transferring data elements to the client based on client content requests (which may be HTTP requests for some embodiments).

For some embodiments, the viewing client may execute a process that includes: requesting specific content for a scene from the content server; collecting information on session specific viewing conditions; receiving the MPD for the scene from the content server; selecting an initial viewpoint of the scene; requesting an initial set of segments of the scene data using application specific initial requirements (which may include initial levels for the LoD and DoF); displaying the current set of content segments; processing scene logic and user feedback input, updating the viewpoint of the scene accordingly; determining (which may include observing and/or measuring) QoE metrics (network and processing performance and session conditions); requesting an updated set of content segments matching LoD and DoF levels adapted to the QoE metrics; and repeating the process by returning to displaying the updated content until a session termination is indicated or signaled. The initial segment request may use the lowest requirements (e.g., 0DoF with the lowest bandwidth requirement closest to the selected viewpoint) or higher requirements if the viewing client determines that a higher capacity is available.

Systems and methods disclosed herein in accordance with some embodiments may enable progressive and adaptive distribution of spatial data to client devices with large variation in the capabilities and display characteristics of these client devices. Such systems and methods in accordance with some embodiments may also take into account, e.g., transmission bandwidth and client device processing performance. Web-based distribution of spatial scenes with multiple spatial content types and minimal latency and start-up delays may be enabled for systems and methods disclosed herein in accordance with some embodiments.

<FIG> is a system diagram illustrating an example set of interfaces for a viewing client according to some embodiments. For some embodiments, a viewing client <NUM> may interface with a display <NUM> and one or more sensors <NUM>. A viewing client <NUM> may include local cache memory <NUM>. One or more displays <NUM> and one or more sensors <NUM> may be located locally for some embodiments. For other embodiments, one or more displays <NUM> and one or more sensors <NUM> may be located externally. A viewing client <NUM> may interface via a network, e.g., a cloud network, to a content server <NUM>. Media presentation description (MPD) files <NUM> and levels of detail (LoD) versions of spatial data <NUM> may be stored on the content server <NUM>. For some embodiments, one or more degrees of freedom (DoF) representations of spatial data may be stored on the content server <NUM>. For some embodiments, a system <NUM> may include a viewing client <NUM> interfacing with a display <NUM>, a sensor <NUM>, and a content server <NUM>.

In some embodiments, the content server streams spatial content with multiple formats and quality settings and enables a viewing client to dynamically adapt to the available bandwidth, client performance, and per session conditions. In addition to several quality and format streams, the content server provides metadata about the available streams to the viewing client as a manifest file such as a Media Presentation Description (MPD) file for some embodiments. To enable dynamic adjustment, the content server creates schemas for the content elements that use freedom of content navigation to further adjust to available bandwidth, client performance, and per session conditions in some embodiments. Based on the freedom of content navigation schemas, the client may adjust the content complexity and amount of data transferred.

<FIG> is a message sequencing diagram illustrating example processes for pre-processing content and for streaming content to a viewing client according to some embodiments. For some embodiments, a process <NUM> may include a content pre-processing process <NUM> and a content streaming process <NUM>. The content pre-processing process <NUM> may include, for some embodiments, a content provider <NUM> sending <NUM> spatial data to a content server <NUM>. The content server <NUM> may compile (or generate) <NUM> multiple levels of detail (LoD) and degrees of freedom (DoF) versions of the content as content segments. In some embodiments, the content server <NUM> may produce an MPD that includes one or more of the LoD and DoF versions of the content.

For some embodiments, the content streaming process <NUM> may include a viewing (or viewer) client <NUM> receiving <NUM> a content request from a client or user <NUM>. The viewing client <NUM> may send <NUM> a content request to a content server <NUM>. The viewing client <NUM> may collect <NUM> sensor and configuration data for some embodiments. The content server <NUM> may send <NUM> a media presentation description (MPD) file to the viewing client <NUM>. The example contents of an example MPD in accordance with some embodiments are described in more detail in relation to <FIG> and <FIG>. In some embodiments, an initial viewpoint is selected <NUM> by the viewing client <NUM>. For some embodiments, the viewing client <NUM> may send <NUM> a request to the content server <NUM> for the lowest LoD and DoF segment for the selected viewpoint. For some embodiments, the level of detail (LoD) may be ordered in resolution size so that the lowest LoD is the LoD with the smallest number of total pixels. For some embodiments, the degrees of freedom (DoF) may be ordered by the number of degrees of freedom such that the lowest DoF is the lowest DoF available (for example, in the order of 0DoF, 3DoF, 3DoF+, and 6DoF). The content server <NUM> may send <NUM> the requested segment to the viewing client <NUM>. The contents may be displayed <NUM> by the viewing client <NUM> and seen by the user <NUM>. The user <NUM> may respond <NUM> with a user input. The viewing client <NUM> may process <NUM> the user input and scene logic and update the viewpoint. For some embodiments, the user input may be motion or feedback on the displayed content. The viewing client <NUM> may observe <NUM> QoE metrics and may request <NUM> a LoD and DoF segment according to the QoE metrics. For example, the QoE metrics may indicate that the user experience is below a threshold, and the viewing client <NUM> may request a segment with a higher LoD. The content server <NUM> responds <NUM> with the requested segment, and the viewing client <NUM> renders and displays <NUM> the LoD and DoF representations of the content for the user <NUM>.

For some embodiments, the viewing client may determine QoE metrics, such as, for example, network performance, processing performance, client computing performance, and session conditions. The process of determining the QoE metrics, selecting LoD and DoF representations based on the QoE metric, and requesting LoD and DoF content segments may be an iterative process that may be continually repeated for some embodiments. The LoD and DoF representations may be selected from a set of one or more LoD and DoF representations described in an MPD file. For some embodiments, a viewpoint of a user is determined, and the content is rendered for the determined viewpoint. With some embodiments, the DoF and LoD representations are selected based on the viewpoint of the user. A viewpoint may be associated with particular DoF and LoD schema. For example, a viewpoint may be associated with 3DoF and 0DoF DoF schema. The DoF scheme may be updated to select one of the available DoF schema associated with the viewpoint. The LoD scheme may be updated to select one of the available LoD for the selected DoF. For example, 3DoF may be selected as an update to the DoF scheme, and a medium level LoD with a resolution of <NUM> x <NUM> may be selected. Some embodiments may limit the viewpoint of the user to a viewing area that may be indicated in the MPD file. In some embodiments, the viewpoint of the user may be limited to a combination of the viewing area and a navigation area that may be indicated in the MPD file. For some embodiments, selecting a level of detail representation from one or more level of detail representations for the selected degrees of freedom representation based on a viewpoint of a user, such that the selected degrees of freedom representation may include the one or more level of detail representations. For some embodiments, a process may include limiting the viewpoint of the user to a viewing area for the user, wherein the manifest file may include the viewing area for the user.

<FIG> is a message sequencing diagram illustrating an example process for a viewing client requesting content based on QoE metrics according to some embodiments. The server <NUM> (which may be a content server) may determine (e.g., compile or generate) LoD and DoF versions of the content. The server <NUM> may generate <NUM> the MPD file, which may indicate the LoD and DoF versions compiled. In a client pull model, the viewing client <NUM> requests <NUM> content from the server. The viewing client <NUM> may collect <NUM> sensor and configuration data about viewing conditions. The viewing client <NUM> may collect system information by using available sensors and by monitoring network communication and processing performance parameters. The server <NUM> sends <NUM> the MPD file to the viewing client <NUM>. The viewing client <NUM> selects <NUM> an initial viewpoint and representation (e.g., DoF and LoD). The viewing client <NUM> requests <NUM> initial LoD and DoF segments, and the content server <NUM> responds <NUM> with spatial data for the requested segments. The viewing client <NUM> renders <NUM> and displays the requested segments. The viewing client <NUM> may observe <NUM> QoE metrics and may select LoD and DoF levels for additional segments based on the MPD file provided by the content server. The QoE metrics may be determined based on the dynamically changing viewing conditions. For some embodiments, the LoD and DoF levels for additional segments may be selected to adaptively balance the QoE metrics and available resources. The selected LoD and DoF segments may be requested <NUM> by the viewing client <NUM>, and the content server <NUM> may respond <NUM> with the requested segments. The viewing client <NUM> may select among the DoF and LoD options based on user motion and bandwidth constraints for some embodiments.

<FIG> is a data structure diagram illustrating an example MPEG-DASH Media Presentation Description (MPD) according to some embodiments. <FIG> shows a structure <NUM> of an MPEG-DASH media presentation description (MPD) file <NUM>. This file format may be used for the MPD transmitted by the content server to the viewing client. For some embodiments, the MPD file <NUM> may be sent to start initialization of a streaming session. The MPD file <NUM> may include one or more periods <NUM>, <NUM>. The period <NUM>, <NUM> may include a start time and duration for content. The period <NUM>, <NUM> may include one or more adaptation sets <NUM>, <NUM>. The adaptation set <NUM>, <NUM> contains a media stream. The adaptation set <NUM>, <NUM> may include one or more representations <NUM>, <NUM>. Representations <NUM>, <NUM> may include one or more encodings of content, such as 720p and 1080p encodings. Representations <NUM>, <NUM> may include one or more segments <NUM>, <NUM>. The segment <NUM>, <NUM> is media content data that may be used by a media player (or viewing client) to display the content. The segment <NUM>, <NUM> may include one or more sub-segments <NUM>, <NUM> that represent sub-representations <NUM>, <NUM> with a representation field <NUM>, <NUM>. Sub-representations <NUM>, <NUM> contain information that apply to a particular media stream.

<FIG> is a data structure diagram illustrating an example Media Presentation Description (MPD) with example Degrees of Freedom (DoF) and Levels of Detail (LoD) schemes according to some embodiments. The format <NUM> of the MPD file <NUM> shown in <FIG> may be used for adaptive spatial data streaming within the MPEG-DASH protocol structure for MPD files <NUM>. For some embodiments, the top hierarchical entity is a period <NUM>, <NUM> with each period including the information of a single consistent virtual scene composed of spatial data. A single scene, for example, may be a single virtual environment in which an interactive and/or pre-defined virtual experience takes place. The virtual experience may include several scenes, and each scene may include one or more period blocks, similar to a movie that has several scenes. Each period <NUM>, <NUM> may include a scene graph <NUM> and one or more DoF blocks <NUM>, <NUM>, <NUM>, each containing a description of an available viewport-associated DoF scheme available for the scene. DoF scheme elements <NUM>, <NUM>, <NUM> (as well as LoD data structures <NUM>, <NUM>, <NUM> for a DoF schema) and the scene graph <NUM> are described in more detail in relation to <FIG> and <FIG>, respectively.

Relating <FIG> and <FIG> together, DoF divisions <NUM>, <NUM>, <NUM> may correspond to MPEG-DASH adaptation sets, and LoD divisions <NUM>, <NUM>, <NUM> under a given DoF may correspond to MPEG-DASH representations and segments. For some embodiments, media blocks <NUM>, <NUM> may correspond to MPEG-DASH representations, and time steps <NUM>, <NUM>, <NUM> may correspond to sub-representations. For some embodiments, each LoD block <NUM>, <NUM>, <NUM> may include a URL <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> for a corresponding time step <NUM>, <NUM>, <NUM>. For some embodiments, a period <NUM>, <NUM> may include DoF blocks (or DoF representations) <NUM>, <NUM>, <NUM> for 3DoF, 3DoF+, and 6DoF representations.

In some embodiments, a viewing client adaptively manages tradeoffs between degrees of freedom (DoF) and levels of detail (LoD) based on device capabilities and available bandwidth. Other tradeoffs that may be managed include angular density and angular range, in addition to spatial and temporal resolutions. In some embodiments, spatial data may be formatted, for example, as a light field, a point cloud, or a mesh. A light field may be a function that maps light rays to points in space. A point cloud may be a set of points that indicate surfaces of a 3D object. A mesh may be a set of surfaces, polygons, faces, edges, and vertices that describe a 3D object. For example, at a given bandwidth, a viewing client with motion tracking may select a 6DoF representation with coarse angular density, and a viewing client with a light field display may select a 3DoF+ representation to display fine motion parallax.

Table <NUM> shows an example illustrating three DoF schemes (6DoF, 3DoF+, and <NUM>) and three content types (light field, point cloud, and video). For the example shown in Table <NUM>, the AdaptationSet id field indicates the DoF scheme, and the contentType field indicates the content type. Within an adaptation set, the content type is fixed. For example, the content type may be "light field" for each representation within an adaptation set, but the spatial and angular resolutions may differ for each representation. Table <NUM> does not show details of MPD syntax.

For some embodiments, DoF schemas indicate levels of freedom of navigation that are supported for a given viewpoint. In addition, in some embodiments, the schemas may indicate requirements to support a particular DoF schema. For a given viewpoint, multiple schemas may be indicated, and the viewing client may use schemas to adapt freedom of navigation during a viewing session to the available resources. For some embodiments, the viewing client executes a process that uses quality metrics and a rules set for DoF adaptation. For some embodiments, DoF schemas do not describe rules by which the viewing client may switch between DoF schemas. The viewing client may implement the logic for DoF adaptation that depends on the viewing client use. <FIG> describes an exemplary processing executed by the viewing client.

Requirements for a given DoF schema may include a network bandwidth threshold used to stream the content (such as to meet a QoE threshold) as well as amount of data transmission used by the initial content download. With some formats of spatial data, for example a 3D polygon mesh, the full geometry may need to be available at the client side upon starting the 3D rendering. Upon receiving the full mesh at the client, the mesh may be reused for different temporal steps. The appearance of a full mesh (which may have been previously received) may be modified between temporal steps with additional control data in another format, such as, for example, skeleton pose data that may be used for a skeleton animation rig embedded with the original full mesh. Some embodiments divide transmission bandwidth requirements between the initial download and the streaming bandwidth.

<FIG> is a schematic perspective view illustrating an example virtual scene environment according to some embodiments. <FIG> shows an exemplary scene <NUM>. The example shown indicates five viewpoints <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and associated available DoF schemas, DoF viewing areas, and DoF navigation areas. For 0DoF, the viewpoint is shown as a point. Viewpoints <NUM> and <NUM> (<NUM>, <NUM>) are associated with available 0DoF schemas. The associated viewing area <NUM>, <NUM> for 0DoF is a two-dimensional shape, such as a rectangle. Examples of 0DoF viewing areas <NUM>, <NUM> are shown for viewpoints <NUM> and <NUM> (<NUM>, <NUM>). For 3DoF, shown in viewpoint <NUM> (<NUM>), there are three degrees of freedom: yaw, roll, and pitch. These three degrees of freedom indicate orientation (or viewpoint) of a user. For 3DoF, the location of the user's viewpoint <NUM> is fixed, and the viewing area <NUM> is a three-dimensional shape, such as a frustum. For 3DoF+, the location of the user's viewpoint may change. For 3DoF+, the navigation area is a three-dimensional shape, such as a cube. Viewpoint <NUM> (<NUM>) is associated with an available 3DoF+ schema. The viewing area <NUM> for 3DoF+ is shown as a three-dimensional shape, such as a frustum. The user may move within the navigation area to adjust the viewpoint. The viewing area indicates the area for which spatial data is available for a user moving within the navigation area. For 6DoF, there are six degrees of freedom: yaw, roll, pitch, up/down, left/right, and forward/backward. Yaw, roll, and pitch indicate orientation (or viewpoint) of the user. Up/down, left/right, and forward/backward indicate position of the user within a scene. Viewpoint <NUM> (<NUM>) is associated with an available 6DoF schema. For some embodiments, spatial information and interactive behavior rules may be indicated relative to a scene root node <NUM>, such as the one indicated in <FIG>. For the example shown in <FIG>, some object information may be relative to one or more objects. For example, spatial information for object <NUM> (<NUM>) may be relative to spatial information for object <NUM> (<NUM>). Likewise, spatial information for object <NUM> (<NUM>) may be relative to spatial information for object <NUM> (<NUM>) and thereby relative to spatial information for object <NUM> (<NUM>). Object <NUM> (<NUM>) may have spatial information that is relative to spatial information for object <NUM> (<NUM>). Spatial information for some objects, such as object <NUM> (<NUM>), may be independent of other objects. <FIG>'s scene graph, which is described below, also has examples of objects that are described relative to other objects.

The five viewpoints <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and associated DoF schemas shown in <FIG> are described in more detail below. In the example MPD structure shown in <FIG>, a period may include one or more DoF schemas and include some of the details shown below. The values for, e.g., download size, required network capacity, and resolution, are example values for purposes of the example viewpoints and associated schemas.

For some embodiments, within each DoF schema, the streams of the scene content are described as multiple media elements. Each media element may contain spatial data in some spatial data format. Spatial data contained in the media may be described as temporal segments, or in case of static content, a single temporal step. Also, combinations of static content and temporal segments may be used, for example, a polygon mesh, animated with a skeleton animation rig. Within each media element for each temporal step, one or more LoD versions of the media may be listed under the media block. For each LoD version of the data, streaming bandwidth requirements may be indicated as well as if the data is progressive (such that higher LoD levels build on top of lower LoD levels). In some embodiments, for higher LoD used with progressive data, the lower LoD data needs to be received in addition to the higher LoD data.

<FIG> and <FIG> are a scene graph illustrating an example data structure for a virtual scene environment according to some embodiments. A scene graph <NUM> is a description of structure and behavior of a scene. For some embodiments, the description may include a hierarchical structure of spatial relationships between scene elements and logic rules indicating interactive behavior of scene elements. A scene graph <NUM> may contain information related to, for example, scene audio and physics relationships of objects. For adaptive streaming, a scene graph <NUM> may contain information about available viewpoints and associated DoF schemas. For some embodiments, each viewpoint described in the scene graph is a point or area within the scene for which viewpoints (which may be one or more viewpoints from 0DoF to 6DoF) are available. Viewpoints may be described as a combination of available DoF, points or areas and supported viewing directions. Viewpoint information may be individual elements under a period and linked with the associated DoF schemas included in the MPD.

<FIG> and <FIG> show a hierarchical structure scene graph. Viewpoint <NUM> (<NUM>) is associated with a 6DoF schema with links to assets, such as object spatial relationships, object behavioral rules, and other viewpoints. <FIG> and <FIG> show transformations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> from the scene's root node <NUM> to objects <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and viewpoints <NUM>, <NUM>, <NUM>, <NUM> within the scene. Viewpoint <NUM> (<NUM>) is associated with a 3D0F+ schema and has links to example light field data sources. Viewpoint <NUM> (<NUM>) is associated with a 0DoF schema and has links to example 2D video data sources. Viewpoint <NUM> (<NUM>) is associated with a 0DoF schema and has links to example 2D video data sources. Viewpoint <NUM> (<NUM>) is associated with a 3DoF schema and has links to example <NUM>-degree video data sources. Similar to <FIG>, the scene graph of <FIG> and <FIG> indicates 3D navigation area <NUM> and viewing areas <NUM> for 3DoF+ schemas and 2D viewing areas <NUM>, <NUM>, <NUM> for 0DoF. <FIG> and <FIG> also show <NUM> example objects <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. Spatial data is indicated for each object <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, such as geometry <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, textures <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, shading <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and animation data <NUM>. Some object data (e.g., <NUM>) is indicated with a transformation (e.g., <NUM>) relative to another object (e.g., <NUM>), such as three objects stacked on top of one another. The scene graph <NUM> of <FIG> and <FIG> indicates links to the associated data sources for each viewpoint and DoF schema.

DoF may be used as a variable that may be used to control the tradeoffs between bandwidth, complexity, and QoE. The scene graph structure (an example of which is shown in <FIG> and <FIG>) may be extended with information indicating viewpoints and associated DoF schema (or levels). Some scene graphs of synthetic 3D scenes include real-time 3D graphic data for image production rendering and information about alternative visual information sources for visualization with lower DoF schema, such as video sources for pre-rendered views.

<FIG> is a flowchart illustrating an example process for handling viewing client requests according to some embodiments. For some embodiments of a process <NUM>, a content server stores the spatial data <NUM> to be distributed along with the MPDs <NUM> for the data. At run-time, the content server may distribute data based on client request type <NUM> in a client pull model, as illustrated in <FIG>. For a new data streaming session, the viewing client may request an MPD, and the content server may send <NUM> an MPD. Based on the MPD, the viewing client requests spatial data segments according to the MPD and QoE metrics measured by the viewing client (such as available resources and session conditions), and the content server may send <NUM> a data segment. The example process <NUM> may determine <NUM> if an end of processing is requested and continue by waiting <NUM> for a request from a viewing client, e.g., until an end of processing <NUM> is requested.

In some embodiments, the server may produce some of the DoF and LoD versions automatically. For example, given 0DoF data, the content server may produce various LoD versions from the video file enabling 0DoF viewing. Also, for some embodiments, with higher DoF versions, the content server may produce lower DoF versions automatically. For example, if spatial content is fully synthetic 6DoF content, the server may automatically produce lower DoF versions based on user indicated viewpoints.

For embodiments of a server process, a data segment request may indicate the selected degrees of freedom representation (or schema). The selected degrees of freedom may be selected from an ordered set of available degrees of freedom, which may be indicated in the manifest file (such as an MPD). The data segment request also may indicate an LoD that is selected from a set of available LoDs indicated in the manifest file (e.g., the MPD). The DoF schema of the data segment sent to the viewing client may match the DoF schema indicated in the data segment request.

<FIG> is a flowchart illustrating an example process for adjusting content requests based on QoE metrics according to some embodiments. <FIG> illustrates an example process <NUM> executed by the viewing client. For some embodiments, the process <NUM> starts with a user, e.g., launching an application on the viewing client and initiating <NUM> a request for content from the content server. Content may be indicated as a link to the MPD residing on the content server. The link to the MPD may be a uniform resource location (URL) identifying the content server and specific content. For some embodiments, the viewing client application is launched by an explicit command from the user or automatically by the operating system based on identifying content type request and application associated with the specific content type. For some embodiments, a viewing client may be a stand-alone application, an application integrated with a web browser, a social media client, or a part of the operating system. If a viewing client application is launched, sensor data collection may be initialized <NUM> and configuration data may be collected. For some embodiments, sensor data collection may include collecting information about the viewing conditions that the viewing client may use to adapt content streaming. For example, in some embodiments, sensors may collect data in order to, e.g., identify the quantity and locations of users and display devices, in which the locations may be relative to the viewing client or relative to a scene root node location for some embodiments.

If the viewing client has initialized sensor and configuration data collection, a process, e.g., a run-time process, may be performed continually throughout the content streaming session. In the run-time process, the viewing client receives <NUM> the MPD from the content server. For some embodiments, based on the MPD, collected viewing conditions information, application default settings, and user preferences, the application selects <NUM> an initial viewpoint to the spatial data from the MPD and requests <NUM> data segments from the content server using initial requirements for DoF schemas and LoD levels. For some embodiments, the initial request may use the lowest requirements, e.g., 0DoF with the lowest bandwidth requirement closest to the selected viewpoint. If the viewing client application determines that higher capacity is available, a DoF schema and LoD level with higher requirements may be used.

The viewing client receives and displays <NUM> the requested content. User input may be collected <NUM>, and scene logic may be processed <NUM>. The viewpoint of the user may be updated <NUM>, and QoE metrics may be collected <NUM>. The DoF and LoD may adapted for the user's current viewpoint based on the QoE metrics and adaptation rules, for some embodiments. In some embodiments, the MPEG-DASH adaptation set (of which, DoF is an example) and the MPEG-DASH representation (of which, LoD is an example) may be adapted <NUM> for the user's current viewpoint based on the QoE metrics and adaptation rules. Examples of QoE metrics include encoding parameters, resolution, sample rate, content update rate, delay, and jitter. DoF and LoD may be updated based on one or more of these QoE metrics examples for some embodiments. For example, DoF and LoD may be adjusted if the amount of jitter in displayed content exceeds a threshold. The next set of segments may be requested <NUM> for the adjusted DoF and LoD. The process may determine <NUM> if the end of processing is requested. If an end of processing is requested, the process ends <NUM>. Otherwise, the process repeats with receiving and displaying of content.

For some embodiments, the viewing client's process may include determining available processing power for processing the selected degrees of freedom schema (or representation) and selecting a level of detail representation based on the available processing power. For some embodiments, the selected degrees of freedom representation comprises the selected level of detail representation. The LoD selected is available for the selected DoF. For some embodiments, the available processing power may include local rendering power and view interpolation power. For some embodiments, a DoF and a LoD may be selected such that local rendering power is capable of rendering content segments for the selected DoF and LoD. For example, a DoF scheme of 3DoF and a LoD scheme supporting a resolution of <NUM> x <NUM> may be selected if the local rendering power is capable of displaying <NUM> x <NUM> with support for three degrees of freedom for the orientation of the viewer. For some embodiments, the viewing client's process may include tracking a range of motion of the client, and responsive to detecting a reduction in the range of motion of the client, selecting an updated DoF schema (or representation). The updated DoF schema may be selected from a ordered set of available DoF schemas. The updated DoF schema may have less degrees of freedom than the previously selected DoF schema for some embodiments. For some embodiments, the viewing client's process may include detecting a change in the range of motion of the client and responsive to detecting the change in the range of motion of the client, selecting a representation from one or more degrees of freedom representations.

<FIG> is a graph illustrating example relationships of Degrees of Freedom (DoF) and Levels of Detail (LoD) for multiple adaptation parameters according to some embodiments. The example graph <NUM> of <FIG> shows Levels of Detail (LoD) resolution/vertices <NUM> vs. Degrees of Freedom (DoF) <NUM>. QoE metrics, for some embodiments, include data the viewing client collects in order to adapt content streaming to processing and computation performance limitations. Network performance may be measured, for example, by measuring latency between segment request and display. For example, a latency requirement may be set to be below the target frame rate of the rendering in order to not cause content to lag behind due to network bandwidth. Client computing performance may be measured, for example, using rendering frame rate as a QoE metric. Rendering falling below a given threshold may indicate that the content exceeds the complexity the client device is able to handle. This situation may be corrected by reducing the LoD of the content, thereby reducing the rendering complexity. For some embodiments, clients in 2D ("0DoF") space may use adaptation of DoF/LoD based on bandwidth, client motion freedom, and processing power (rendering view interpolation) factors. DASH may typically vary only LoD (resolution) in response to bandwidth constraints. Some embodiments may impose client device capability limits for high DoF or LoD selections.

<FIG> shows three traces <NUM>, <NUM>, <NUM> for DoF/LoD adaptation based on adaptation processes that emphasize different parameters. For example, the bandwidth adaptation trace <NUM> indicates a matching of DoF and LoD schemas for a process that emphasizes the LoD schema as the primary parameter for adapting bitrate. The DoF may be maintained high (3DoF+) as LoD adjusts bandwidth until the bandwidth is insufficient to support 3DoF+ at the lowest LoD. The processing constraint trace <NUM> indicates a matching of DoF and LoD schemas for a process that emphasizes the DoF schema as the primary parameter. For the processing constraint trace <NUM>, the LoD may be maintained at high while the DoF varies from 3DoF to 6DoF. For some embodiments, the LoD is reduced only if the DoF is at a minimum (3DoF) The motion adaptation trace <NUM> indicates a matching of DoF and LoD schemas for a process that emphasizes motion of the user. Both the DoF and LoD vary with the relative value of DoF/LoD depending upon user motion (e.g., for little motion, low DoF but high LoD may be used and for high motion, high DoF but low LoD may be used). For a 6DoF schema, the user is able to move to a 3D position within a scene with a yaw-pitch-roll viewing orientation. With such a large area over which the user may move, the LoD schema may be set low to maintain a bandwidth limit. As the user changes to a lower DoF schema (which may correspond to a limitation on user motion), the LoD schema may be successively increased (e.g., "medium" LoD corresponding to 3DoF+, and "high" LoD corresponding to 3DoF) and the amount of content data may still remain below the bandwidth limit.

For some embodiments, the viewing client may implement an adaptation control logic process that applies to a particular environment and application. For some embodiments, the control logic may adapt the LoD to the available bandwidth and processing performance for a DoF that matches the display capabilities of the viewing client. For some embodiments, the best QoE may be achieved with an adaptation logic process that mixes both LoD and DoF representations levels simultaneously. Mixed adaptation may be used because the highest DoF representation may not provide the best visual quality and a lower DoF with higher image quality may be sufficient to support viewpoint motion of the specific session. For some embodiments, depending on viewpoint motion, a higher DoF may be preferred by a user during a session over visual quality to support a level of content navigation desired by the user (viewer). If the DoF is dynamically changed during a session due to changes in available resources or viewpoint motion, the LoD may be re-adjusted for each change of DoF. Exemplary pseudo code for an adaptation logic process implementing mixed adaptation is shown in Table <NUM>. Setting of the lowest available DoF and LoD may be based on bandwidth and/or processing power requirements for some embodiments. For example, the lowest DoF may be the lowest number of degrees of freedom available, and the lowest LoD may be the lowest total number of pixels for a resolution for the selected DoF.

In addition to the control parameters described in the pseudo code example in Table <NUM>, the control logic may balance between DoFs and LoDs using some weighting in order to balance more finely between, for example, DoF and perceivable resolution so that, in some cases, the freedom of navigation may be decreased in order to achieve a higher perceivable resolution. This process could be used, for example, to drop from 3DoF to 0DoF if the final 3DoF rendering causes the perceivable resolution to be significantly lower than what 0DoF is able to provide. Another control element not described in the pseudo code example of Table <NUM> is user preferences. In some embodiments, user preferences may affect an adaptation process, with the process, e.g., configured to incorporate, e.g., specific user preferences. For example, a user may prefer 0DoF content over 3DoF content, and this preference may be incorporated into, e.g., adaptation process logic. User preferences may be determined from users directly, or inferred or assumed based on, e.g., prior user streaming activity or viewing behavior.

With a 2D display, a default process for the viewing client may be to select a viewpoint based on the user preferences and scene logic described in the scene graph for available 0DoF viewpoints. The process may adapt the 0DoF LoD during a session to the available network bandwidth. If the viewing client uses a process to enable a user to interactively navigate content, the viewing client may enable navigation by switching to a higher DoF schema.

For spatial display with multiple viewers, such as a multi-view tabletop display, the spatial content may be adjusted to the number and location of multiple users in order to achieve best QoE for all viewers. In this case, the viewing client may monitor the location of the users, and based on the locations of users, select multiple viewpoints for the content's scene graph. Depending on user preferences and the locations of users, viewpoints may use data streamed with different DoF schemas.

Depending on user preferences and particular use case, the viewing client (which may be a head mounted display (HMD), for example) may use 3DoF+ content over full 6DoF content because of the better image quality enabled by the 3DoF+ data even if full 6DoF schema is available. For some embodiments, if free content navigation is enabled by the viewing client, the viewing client may switch between 6DoF and 3DoF+ schemas as the user navigates the content based on availability of 3DoF+ data for a particular viewpoint. For some embodiments, a 6DoF version of a synthetic 3D scene may be a 3D polygonal mesh representation that the user is able to navigate and for selected viewpoints, pre-rendered light fields may be available to enable higher image quality with a limited navigation area.

<FIG> is a flowchart illustrating an example process for a viewing client streaming content from a content server according to some embodiments. Some embodiments of a content streaming process <NUM> may include receiving <NUM>, at a client device, a manifest file describing an ordered plurality of degrees of freedom representations of content. The manifest file may not be ordered for some embodiments. In some embodiments, the content streaming process <NUM> may further include estimating <NUM>, at the client device, bandwidth available for streaming content to the client device. For some embodiments, the content streaming process <NUM> may further include selecting <NUM>, at the client device, a first degrees of freedom representation from the ordered plurality of degrees of freedom representations. Some embodiments of the content streaming process <NUM> may further include detecting <NUM>, at the client device, a change in the bandwidth available for streaming the content. In some embodiments, the content streaming process <NUM> may further include responsive to detecting the change in the bandwidth available, selecting <NUM>, at the client device, a second degrees of freedom representation from the ordered plurality of degrees of freedom representations. Some embodiments of the content streaming process <NUM> may further include requesting <NUM>, from a streaming server, the second degrees of freedom representation. Some embodiments of an apparatus may include a processor and a non-transitory computer-readable medium storing instructions that are operative, when executed by the processor, to perform the content streaming process described above.

For some embodiments, if a content streaming process estimates a reduction in available bandwidth, an updated DoF schema may be selected that decreases the degrees of freedom (such as a switch from a 6DoF schema to a 3DoF+ schema). For some embodiments, if a content streaming process estimates an increase in available bandwidth, an updated DoF schema may be selected that increases the degrees of freedom (such as a switch from a 3DoF+ schema to a 6DoF schema). For some embodiments, a content streaming process may include retrieving a content representation and rendering the representation.

<FIG> is a flowchart illustrating an example process for a content server streaming content to a viewing client according to some embodiments. Some embodiments of a content server process <NUM> may include receiving <NUM>, at a content server, a request for a manifest file describing an ordered plurality of degrees of freedom representations of content. In some embodiments, the content server process <NUM> may include generating <NUM> the manifest file for the content. With some embodiments, the content server process <NUM> may include sending <NUM>, to a client device, the manifest file. For some embodiments, the content server process <NUM> may include receiving <NUM>, from the client device, a request for a data segment of the content. In some embodiments, the content server process <NUM> may include sending <NUM>, to the client device, the data segment of the content, wherein at least one of the ordered plurality of degrees of freedom representations comprises at least two level of detail representations of the content. Some embodiments of an apparatus may include a processor and a non-transitory computer-readable medium storing instructions that are operative, when executed by the processor, to perform the streaming server process described above.

Streaming media may need to adjust to requirements that are generally becoming more complex. MPEG-Dash addresses dynamic variation in the streaming media distribution bandwidth with focus on video content. With spatial media, similar dynamic adaptive streaming may be used but with a model that takes into an account multitude of spatial content formats as well as an even wider gamut of contextual conditions. Some content formats may require, for example, only minimal amount of initial download, but instead consume more bandwidth during the whole streaming session. Some devices use larger chunks of data at some parts of the experience, and users may desire a balance among initial wait-up time, streaming bandwidth, and image quality.

Many current adaptive spatial content streaming devices focus on a single spatial content type, namely 3D data in polygon mesh format, as understood according to the articles <NPL>) ("Lavoué"); <NPL>) ("Evans"); and <NPL>) ("Zampoglou"). These academic efforts are understood to expand content adjustment schema at the client side by adjusting to bandwidth limitations and adjusting to computing performance. In Zampoglou, applicability of MPEG-Dash standard to transmit 3D data with multiple levels of detail (LoD) together with associated metadata is understood to be investigated. Both Lavoué and Evans are understood to propose a progressive compression algorithm for 3D graphics data suitable for adaptive LoD streaming.

<FIG> is a process diagram illustrating an example communication of video content according to some embodiments. As shown in the process <NUM> of <FIG>, an HTTP server <NUM> with video content may have quality levels that vary over time. A network (the Internet) <NUM> with a variable bandwidth availability also varies over time. A user with a tablet <NUM> (or other wireless device) downloads content onto the device. The user's demand for content also varies over time.

Spatial data may increase demand for content streaming bandwidth and the ability to be able to dynamically adapt to the changing resources available. With spatial data, unlike 2D video content, balancing between bandwidth consumption and QoE may be more than just adjusting resolution / compression rate across the whole image area depending on the available bandwidth. With spatial data, for example, switching between different content formats during streaming instead of just changing level of detail within single format may result in a better QoE, but this depends on the viewing conditions. Some formats, e.g., require different amounts of data to be pre-downloaded before rendering and display is enabled. One example is a model that is animated by streaming commands. In some embodiments, the model must be downloaded before the small animation command stream may be used.

For some embodiments, viewing clients may be informed of available spatial data formats and associated data download specifications. In addition to streaming manifest communication, a client may handle adaptation in order to achieve an optimal QoE for some embodiments. Some embodiments may balance QoE, taking into account, for example, required initial downloads and anticipated streaming specifications to ensure smooth playback. Some embodiments may include expanding adaptive spatial data streaming to balance between initial download, streaming bandwidth, and image quality by dynamically adjusting between different spatial data formats. Adaptive streaming prepares content at different bitrates, allowing a client to adapt to different bandwidth. The streaming rate of the stream is communicated in an MPD for some embodiments. In some example embodiments, a potential challenge regarding how to handle fixed-size data needs and burst data needs is addressed.

<FIG> is a system diagram illustrating an example set of interfaces for a content server-viewing client network according to some embodiments. For some embodiments of a system <NUM>, a content server <NUM> streaming spatial content data <NUM> with various formats and quality settings may allow a viewing client <NUM> to dynamically adapt the content streaming to the viewing conditions within limits of the available bandwidth, client performance, and per session conditions. For some embodiments, the content server <NUM> may store media presentation descriptions <NUM> which may relate to one or more sets of spatial data. In addition to several quality and format streams, the content server <NUM> may provide metadata about the available streams to the viewing client <NUM>. A viewing client <NUM> may select the streams to be used based on information about the content received as streaming manifest metadata from the server <NUM>, the contextual information the viewing client has about the viewing conditions, available bandwidth, and available processing resources. For some embodiments, a viewing client <NUM> may include a local cache, which may be used to store content streams of media presentation descriptions <NUM>. The viewing client <NUM> may send content streams to a display <NUM>. The viewing client <NUM> may receive sensor data from a sensor <NUM>.

For some embodiments, an adaptive media manifest is expanded with specification of the initial download specification for the content streams. Similar to the MPEG-Dash media presentation description (MPD), metadata about the content streams may be composed in a structured document extended with the initial download specifications defined for each content stream version. For some embodiments, at the beginning of a streaming session, the viewing client may download an MPD from the content server. Based on, e.g., the MPD, current conditions, and local client/display capabilities, the viewing client may select versions of the content data to be downloaded and adapt data transmission and quality by selecting content segments in a format and resolution that is most appropriate and complies with the available resources. This functionality may enable the viewing client to control the wait-up time a user waits before the execution of the experience may be launched. Furthermore, during the session, the client may inspect available bandwidth, and may download concurrently with the real-time streaming, content elements that are part of the initial download used by another type of spatial data.

For some embodiments, progressive and adaptive distribution of spatial data to client devices may be enabled with large variation in capabilities and display characteristics of client devices while also adapting to the transmission bandwidth and client device processing performance. For some embodiments, web-based distribution of spatial scenes with multiple spatial content types with controllable latency and start-up delay may be enabled.

<FIG> is a message sequencing diagram illustrating an example process for communication and processing of a typical use session according to some embodiments. <FIG> illustrates an example communication sequence <NUM> in an example use session with content pre-processing <NUM>, content distribution <NUM>, and a run-time loop <NUM> in accordance with some embodiments. For some embodiments, spatial data may be provided <NUM> by the content provider <NUM> to the content server <NUM>, from which the viewer client (or, e.g., "viewing client") <NUM> may select versions of the content to be downloaded. The content server <NUM> may compile <NUM> various versions of streamed data as segments and identify download specifications for one or more (or, in some example cases, all) data elements for the MPD.

A user <NUM> may send <NUM> a content request to the viewer client <NUM>, and the viewer client <NUM> may send <NUM> a content request to the content server. The viewing client <NUM> may collect <NUM> sensor information about the viewing conditions by collecting system configuration information, by collecting available sensor data, and by observing network communication and processing performance. The viewer client <NUM> may collect <NUM> sensor and configuration data. The content server <NUM> may send <NUM> an MPD to the viewer client <NUM>, and the viewer client <NUM> may select <NUM> an initial viewpoint. The viewer client <NUM> may select <NUM> spatial data elements to be requested. The viewer client <NUM> may send <NUM> a request for initial content data to the content server, and the content server <NUM> may send <NUM> the requested content elements to the viewer client <NUM>. The viewer server <NUM> may wait <NUM> for the initial downloads to be completed. The viewer client <NUM> may send <NUM> a request for streamed content data to the content server <NUM>, and the content server <NUM> may send <NUM> the requested content elements to the viewer client <NUM>. The content may be displayed <NUM> to the user <NUM>, and the user <NUM> may send <NUM> user input to the viewer client <NUM>. The viewer client <NUM> may process <NUM> the user input and scene information and update the viewpoint. The viewer client <NUM> also may observe <NUM> QoE metrics. Based on the QoE metrics observed and/or inferred from the collected dynamically changing viewing conditions, the viewing client may request specific versions of the spatial data media segments based on the Media presentation description (MPD) provided by the content server, adaptively balancing start-up delays, QoE and available resources.

For some embodiments, a QoE metric for a selected content representation (such as a selected spatial data element) may be determined to be less than a threshold, and a second content representation may be selected from one or more content representations. For some embodiments, selecting the second content element representation may include determining that a QoE metric corresponding to the second content element representation exceeds a minimum threshold. For some embodiments, a QoE metric for a selected content element representation may be determined, and a second content element representation may be selected from the plurality of content element representations based on the determined QoE metric. For some embodiments, selecting the second content element representation includes determining that the QoE metric corresponding to the second content element representation exceeds a minimum threshold. For some embodiments, a process may include determining a quality of experience (QoE) metric for the selected representation is less than a threshold; and responsive to determining the QoE metric for the selected representation is less than the threshold, selecting a still further representation from the one or more degrees of freedom representations.

<FIG> is a message sequencing diagram illustrating an example process for streaming and displaying content data according to some embodiments. For some embodiments, an example process <NUM> may include a server <NUM> generating <NUM> an MPD with initial download specifications (e.g., requirements). For some embodiments, the example process <NUM> may further include a client <NUM> sending <NUM> a content request to the server <NUM>. For some embodiments, the example process <NUM> may further include the server <NUM> sending <NUM> a Media Presentation Description (MPD) to the client <NUM>. For some embodiments, the example process <NUM> may further include the client <NUM> estimating <NUM> available bandwidth and start-up latency. For some embodiments, the example process <NUM> may further include the client <NUM> selecting <NUM> an appropriate media representation. The appropriate media representation may be selected to reduce start-up latency based on the estimated available bandwidth, which is discussed in more detail later. For some embodiments, the example process <NUM> may further include the client <NUM> requesting <NUM> initial download data. For some embodiments, the example process <NUM> may further include the server <NUM> transmitting <NUM> the requested data and the client <NUM> receiving <NUM> the initial download data. For some embodiments, the example process <NUM> may further include the client <NUM> requesting <NUM> streamed spatial data. For some embodiments, the example process <NUM> may further include the server <NUM> transmitting <NUM> the requested streamed segments and the client <NUM> receiving <NUM> the streamed spatial data. For some embodiments, the example process <NUM> may further include the client <NUM> observing <NUM> quality of experience (QoE) metrics. For some embodiments, the example process <NUM> may further include the client <NUM> displaying <NUM> the content.

For some embodiments, a full spatial data scene view may include initial download data and a stream segment. For some embodiments, selecting a content element representation may include: determining a respective start-up delay for one or more of the plurality of content elements; determining a minimum start-up delay of the determined respective start-up delays; and selecting the content element representation corresponding to the minimum start-up delay, wherein the timeline information includes information regarding the respective start-up delay for one or more of the one or more of the plurality of content elements.

For some embodiments, a viewing client process may include retrieving a stream segment for a content element representation; and displaying the stream segment of the content element representation. For some embodiments, a viewing client may display received initial download data and received stream segment(s). For some embodiments, selecting a content element representation may include: determining a respective latency time associated with the initial download specification for one or more of the plurality of content element representations; and selecting one of the plurality of content element representations, wherein the latency time of the selected content element representation is less than a threshold. For some embodiments, a viewing client may determine a respective latency time for each of a plurality of content element representations, such that selecting the content element representation uses the determined respective latency times.

<FIG> is a flowchart illustrating an example process for producing an example Media Presentation Description (MPD) according to some embodiments. <FIG> also illustrates an example content pre-processing process <NUM> executed by the content server. For some embodiments, in the pre-processing phase, the content server may produce metadata description(s) of the available content, e.g., in the form of a media presentation description file (MPD). The MPD according to the example may provide an overview of the scene, relationships of the scene elements in the form of a scene graph, a timeline associated with the scene elements, one or more different versions of the media assets available, and associated specifications. The content server may store the spatial data <NUM> (such as the scene graph, the timeline, and the media assets) in memory, which may be, for example, local memory location(s) of the server. The content server may produce <NUM> one or more versions of the media assets. For example, the versions may differ regarding encoding bitrate, display resolution, and total media asset size. As part of the pre-processing, the content server may produce various versions of the existing scene elements that enable streaming adaptation by the client. The content server may produce <NUM>, e.g., specifications for the asset versions, such as minimum network bandwidth used to support, minimum network latency used to support, minimum display resolution size used to support, and minimum display refresh rate used to support. The content server may produce <NUM> the MPD and store the MPD in memory. For some embodiments, once the content server has produced an MPD file <NUM> with different asset versions and metadata describing available streams, the content server starts run-time processing such that the content server distributes content to the viewing clients.

For some embodiments, selecting a content element representation may be based on, e.g., representation size, the estimated bandwidth, and playback duration until the content element is displayed. For some embodiments, a manifest file may include timeline information regarding one or more of the plurality of content elements, and a content element representation may be selected based on the timeline information.

<FIG> is a data structure diagram illustrating an example MPEG-DASH Media Presentation Description (MPD) according to some embodiments. For some embodiments, the general structure <NUM> of the MPEG-Dash MPD illustrated in <FIG> may be used as the file format used for transmitting the overall media descriptions. The viewing client may download the MPD as part of a streaming session initialization. The MPD file <NUM> may include one or more periods <NUM>, <NUM>. The period <NUM>, <NUM> may include a start time and duration for content. The period <NUM>, <NUM> may include one or more adaptation sets <NUM>, <NUM>. The adaptation set <NUM>, <NUM> contains a media stream. The adaptation set <NUM>, <NUM> may include one or more representations <NUM>, <NUM>. Representations <NUM>, <NUM> may include one or more encodings of content, such as 720p and 1080p encodings. Representations <NUM>, <NUM> may include one or more segments <NUM>, <NUM>. The segment <NUM>, <NUM> is media content data that may be used by a media player (or viewing client) to display the content. The segment <NUM>, <NUM> may include one or more sub-segments <NUM>, <NUM> that represent sub-representations <NUM>, <NUM> with a representation field <NUM>, <NUM>. Sub-representations <NUM>, <NUM> contain information that apply to a particular media stream.

Table <NUM> shows an example MPD that corresponds with the fields shown in <FIG>. For the example shown in Table <NUM>, the AdaptationSet id field indicates the content scheme, and the Representation id field indicates an identifier that indicates a display size resolution or level of detail (LOD). The bandwidth field may indicate a minimum bandwidth that a network has available for streaming the particular version of the content. The width and height fields indicate the respective width and height display sizes of the content. The filesize field indicates the memory size of the content.

<FIG> is a timing diagram illustrating an example timeline of video and objects according to some embodiments. For some embodiments, timeline information <NUM> is a list of assets and, e.g., the respective temporal presence of these assets in the scene during a user experience. This timeline information <NUM> allows the viewing client, in accordance with some embodiments, to keep track of which assets are used at which time of the user experience and to determine when to begin downloading and streaming of new assets. Timeline information may be stored as part of the scene graph and may be, for example, attached as part of the per scene graph node information. For some embodiments, client capabilities may be tracked. A change in client capabilities may be detected, and responsive to detecting the change in client capabilities, may select a representation from one or more degrees of freedom representations.

The MPD may include details of initial downloads, e.g., as required by different content elements in different formats. Different level of detail (LoD) representations correspond to different file sizes. Also, timeline information may be included in the MPD, enabling a client to initiate content downloads in time. Based on QoE preferences, the client may switch between content representations to balance between initial downloads and, e.g., required streaming bandwidth. For some embodiments, the client may balance between initial start-up delay (e.g., latency) and image quality (e.g., resolution). Such a process may enable web-based distribution of spatial scenes with multiple spatial content types balanced with controllable latency and start-up delay.

The example timeline shown in <FIG> shows a timeline <NUM> of the overall video for a scene. Two example objects <NUM>, <NUM> related to the scene (labeled as Object <NUM> (<NUM>) and Object <NUM>(<NUM>)) are shown with the relative timings of these example objects in relation to each other and the overall video content <NUM>.

<FIG> is a data structure diagram illustrating an example Media Presentation Description (MPD) with example Degrees of Freedom (DoF) and Levels of Detail (LoD) schemes according to some embodiments. <FIG> illustrates how the MPD data enabling streaming and initial download balancing may be organized with the general MPEG-DASH MPD structure <NUM>. For some embodiments, the top hierarchical entity in the MPD file <NUM> is a period <NUM>, <NUM>. Each period <NUM>, <NUM> provides the information of a single consistent virtual scene composited of spatial data. A single scene, for example, may be a single virtual environment in which an interactive and/or pre-defined virtual experience takes place. The virtual experience may include several scenes, and each scene may include one or more period blocks, similar to a movie that has several scenes. According to the example, each period <NUM>, <NUM> may include a scene graph <NUM> and one or more DoF blocks <NUM>, <NUM>, <NUM>, each containing a description of available assets and associated data formats, versions, and specifications.

Relating <FIG> and <FIG> together, DoF divisions <NUM>, <NUM>, <NUM> may correspond to MPEG-DASH adaptation sets, and LoD divisions <NUM>, <NUM>, <NUM> under a given DoF <NUM>, <NUM>, <NUM> may correspond to MPEG-DASH representations and segments. For some embodiments, media blocks <NUM>, <NUM> may correspond to MPEG-DASH representations, and time steps <NUM>, <NUM>, <NUM> may correspond to sub-representations. For some embodiments, a time step <NUM>, <NUM>, <NUM> may contain a URL <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> for a corresponding LoD block <NUM>, <NUM>, <NUM>. For some embodiments, a period <NUM>, <NUM> may include DoF blocks (or DoF representations) <NUM>, <NUM>, <NUM> for 3DoF, 3DoF+, and 6DoF representations.

A scene graph is the description of the structure and behavior of the scene. The description may be formed as a hierarchical description of spatial relations between scene elements, as well as logic indicating interactive behavior of the scene elements. In addition, a scene graph may contain information, for example, related with scene audio and physics. For adaptive streaming, the scene graph may contain information about timeline of presence of assets, available viewpoints, and associated asset versions. The client may use timeline information to estimate when to begin the initial downloading of assets (if applicable) in order to have the assets available without waiting when the assets are used. Viewpoint information may indicate the location and the type of navigation area from which the scene may be viewed or inspected. The viewpoint information may be linked with asset versions if the assets are available in different formats. Such a structure may allow different initial download, freedom of navigation, or viewpoints to be stored.

For some embodiments, this MPD structure provides to the client, e.g., both timeline information and per asset initial download specifications. Clients may use local criteria to select a version of an asset that provides a high (or the best in some embodiments) QoE and enables more accurate per-buffering of spatial content in multiple formats, which may avoid interruptions during a user experience.

<FIG> is a flowchart illustrating an example content server run-time process according to some embodiments. The content server may store the spatial data <NUM> to be distributed along with the MPDs <NUM> for the data. In an example run-time process <NUM>, the content server may distribute data based on client request types <NUM> in a strict client pull model, as illustrated in <FIG>. If a content request is received from a viewing client, the content server may determine the request type <NUM>. If the request type <NUM> is for a new session, the content server may retrieve the MPD from memory and send <NUM> the MPD to the viewing client. If the request type <NUM> is for a data segment or initial download, the content server may retrieve the spatial data from memory and send <NUM> the data segment or initial download assets, respectively. The viewing client may request spatial data segments according to the MPD and QoE metrics measured by the viewing client, which may include the available resources and session conditions. The process may determine if an end of processing was requested <NUM>. If an end of processing request was not received, the process <NUM> may wait <NUM> for a content request from a viewing client. The process <NUM> may continue until an end of processing request <NUM> is received.

For some embodiments, an example process executed by the content server may include: receiving the spatial data. The spatial data may be pre-processed and organized into different versions. The content server may analyze initial download times, e.g., that may be required by each content version. An MPD of the scene may be produced. The content server may wait for content requests from viewing clients. Upon receive a content request, the content server may send the MPD to the client. The content server may transfer data elements to the client based on client HTTP requests, such as the content transfer process described above in relation to <FIG>.

<FIG> are flowcharts illustrating an example viewing client process according to some embodiments. For some embodiments, a user launches an application implementing the viewing client. The user starts the application and may indicate the content to be viewed. For some embodiments, the content is a link to the MPD residing on the content server. The link to the MPD may be a URL identifying the content server and specific content. A viewing client application may be launched, e.g., either by an explicit command by the user or automatically by the operating system based on identifying a content type request and application associated with the specific content type. For some embodiments, the viewing client is a stand-alone application. For some embodiments, the viewing client may be integrated with a web browser or a social media client, or the viewing client may be part of the operating system. For some embodiments of a client process <NUM>, content may be requested <NUM> from a content server. If a viewing client application is launched, sensor data may be initialized, and sensors may be configured for data collection. For some embodiments, sensor data collection collects information about the viewing conditions that the viewing client may use to adapt content streaming. For some embodiments, sensor data may be collected from a sensor and analyzed by the viewing client. For some embodiments, the sensor data may be, for example, RGB video data from a camera or RGBD data from a depth camera. For some embodiments, the number and locations of the users and display devices may be identified.

If the viewing client has initialized <NUM> sensor and configuration data collection, the viewing client may begin the run-time operation, which may be performed continually throughout the content streaming session. In the run-time processing, the viewing client receives <NUM> the MPD from the content server. For some embodiments, based on the MPD, collected viewing conditions information, application default settings, and user preferences, the application selects <NUM> the initial viewpoint to the spatial data from the MPD and requests data segments according to the timeline information, loading assets that are used first. According to an illustrative example, the client may, e.g., balance between wait-up time caused by using asset formats that use an initial download and bandwidth that is consumed continually with asset formats such as light field video which may be streamed. For some embodiments, balancing is based on per client local criteria.

During run-time, the viewing client may continually observe QoE metrics and timeline information in order to be able to swap between asset formats to achieve better QoE and to estimate when to start downloading of assets. For some embodiments, an estimate of when to start downloading an asset may be based on when the asset may be used by a user experience. For some embodiments, an estimate of when to start downloading may determine an estimate of when an asset may be fully downloaded under current network conditions. For some embodiments, such pre-buffering <NUM> by the client may estimate how much excess download bandwidth is currently available and given that excess bandwidth, how long initial download of each asset may take. For some embodiments, content elements to be requested may be selected <NUM> based on a timeline, and initial content data may be requested <NUM>.

For some embodiments, a process executed by a viewing client may include requesting specific content from the content server. The viewing client may collect session-specific viewing condition information. The viewing client may receive the MPD from the content server. The viewing client may select <NUM> content streams to be used based on, e.g., application specific initial specifications. The viewing client may request <NUM> initial downloads for the selected scene data streams and may request the first segments of the real-time streamed scene data. The viewing client may display <NUM> the content. The viewing client may observe <NUM> QoE metrics (such as network performance (which may include consumption of available bandwidth), processing performance (which may include computing load reported by the operating system), client computing performance, and session conditions) and may select <NUM> the content stream to be requested based on the QoE metrics. The viewing client may request the next spatial data segments, and, e.g., if required, begin downloading <NUM> initial data along with real-time streaming. The viewing client may pause streaming to wait <NUM> for completion of the initial downloads. The viewing client may repeat the requesting <NUM> and processing <NUM> of streams until a session termination <NUM> is received.

For some embodiments, QoE metrics are data the viewing client collects in order to adapt content streaming to the bandwidth and computation performance limitations. It will be understood that details for how to implement adaptation of content streaming may vary from client to client, and the scenarios described herein and below are examples. Network performance may be measured, for example, by measuring latency between requesting a segment and displaying the segment. For some embodiments, the viewing client may make adjustments so that the latency is below a target frame rate of the rendering in order to not cause content to lag behind due to the network bandwidth. Client computing performance may be a QoE metric that uses rendering frame rate. Rendering falling below a given threshold may indicate that the content exceeds the complexity for which the client device may handle. This situation which may be corrected, for example, by reducing the LoD of the content or by switching to a content format that uses less rendering computation, reducing the rendering complexity.

For some embodiments, spatial content may be requested from a server. For some embodiments, timeline information regarding one or more of a plurality of content elements may be received, wherein selecting the content element representation may be based on representation size, the estimated bandwidth, and playback duration until the content element is displayed. For some embodiments, selecting a content element representation may include: determining a respective minimum bandwidth for each of the plurality of content element representations; and selecting the content element representation from the plurality of content element representations associated with a highest level of detail available such that the expected bandwidth exceeds the respective minimum bandwidth. For some embodiments, selecting a selected representation may include determining a respective minimum bandwidth for each of the one or more degrees of freedom representations and selecting the selected representation from the one or more degrees of freedom representations associated with a highest level of detail available such that the respective minimum bandwidth is less than the tracked bandwidth available. For some embodiments, selecting the selected representation may include: determining a respective start-up delay for one or more of a plurality of content elements; determining a minimum start-up delay of the determined respective start-up delays; and selecting the degrees of freedom representation corresponding to the minimum start-up delay.

Exemplary pseudocode for some embodiments of example adaptation control logic is shown in Table <NUM>. In some embodiments, a viewing client may implement adaptation control logic using other logic and pseudocode (e.g., other than the non-limiting illustrative example provided as follows) that is adapted to a specific application and use case.

One example of another additional control element not described in the pseudo code explanatory non-limiting example of Table <NUM> is user preferences. In some embodiments, user preferences may impact adaptation. For example, a user preference may indicate a preference for full 3D content but allow free 6DoF navigation at all times. This preference may be implemented in adaptation control logic. For some embodiments, adaptation logic may indicate that assets that, e.g., require initial download are to be used instead of streamed versions.

<FIG> is a flowchart illustrating an example process according to some embodiments. For some embodiments, an example process <NUM> may include receiving <NUM> a manifest file describing a plurality of content element representations of portions of a spatial scene with associated initial download and streaming specifications for a corresponding plurality of content elements. For some embodiments, the example process <NUM> may further include determining <NUM> estimated bandwidth available for streaming and download latency. For some embodiments, the example process <NUM> may further include responsive to estimated download latency, selecting <NUM> a selected content element representation from the plurality of content element representations. For some embodiments, the example process <NUM> may further include retrieving <NUM> initial download data of the selected content element representation. For some embodiments, the example process <NUM> may further include retrieving <NUM> a stream segment of the selected content element representation. For some embodiments, the example process <NUM> may further include displaying <NUM> the received initial download data and the stream segment.

Some embodiments of the example process may further include requesting spatial content from a server. Some embodiments of the example process may further include displaying the received initial download data and the stream segment including a full spatial data scene view. For some embodiments of the example process, retrieving initial download data of the selected content element representation may include: requesting initial download data of the selected content element representation; and receiving the initial download data. For some embodiments of the example process, retrieving a stream segment of the selected content element representation may include: requesting a stream segment of the selected content element representation; and receiving the stream segment of the selected content element representation. For some embodiments, an apparatus may include a processor and a non-transitory computer-readable medium storing instructions that are operative, when executed by the processor, to perform any of the example processes.

For some embodiments, an example process may include requesting spatial content from a server. For some embodiments, retrieving initial download data of the selected content element representation may include: requesting initial download data of the selected content element representation; and receiving the initial download data. For some embodiments, retrieving a stream segment of the selected content element representation may include requesting a stream segment of the selected content element representation.

For some embodiments, a viewing client may receive a manifest file that includes: (<NUM>) a plurality of content element representations of portions of a spatial scene with associated initial download and streaming specifications for a corresponding plurality of content elements, and (<NUM>) timeline information regarding one or more of the plurality of content elements. For some embodiments, a viewing client may perform a process further including: determining an estimated bandwidth available for streaming content; selecting a content element representation from the plurality of content element representations based on at least one of the estimated bandwidth, initial download and streaming specifications, and the timeline information; retrieving initial download data of the selected content element representation; and retrieving a stream segment of the selected content element representation.

For some embodiments, a viewing client may perform a process that includes: determining a respective estimated download latency of a plurality of content element representations; selecting a content element representation from the plurality of content element representations based on the respective estimated download latency; and retrieving a stream segment of the selected content element representation. For some embodiments, the process may include rendering the representation. For some embodiments, selecting a degrees of freedom representation from one or more degrees of freedom representation may be responsive to an estimated download latency.

For some embodiments, an apparatus may include a processor; and a non-transitory computer-readable medium storing instructions that are operative, when executed by the processor, to perform an example process described above.

<FIG> is a flowchart illustrating another example process according to some embodiments. For some embodiments, an example process <NUM> may include receiving <NUM> a manifest file for streaming content, the manifest file identifying one or more degrees of freedom representations of content. For some embodiments, the example process <NUM> may further include tracking <NUM> bandwidth available. For some embodiments, the example process <NUM> may further include selecting <NUM> a selected representation from the one or more degrees of freedom representations based on the bandwidth available. For some embodiments, the example process <NUM> may further include retrieving <NUM> the selected representation. For some embodiments, the example process <NUM> may further include rendering <NUM> the selected representation. For some embodiments, an apparatus may include a processor and a non-transitory computer-readable medium storing instructions that are operative, when executed by the processor, to cause the apparatus to perform the example process <NUM> or any of the methods described above.

Claim 1:
A method comprising:
receiving, at a client device, a manifest file describing a plurality of degrees of freedom representations of adaptive spatial content with different associated degrees of freedom, DoF, and the manifest file further describing, for each representation, (i) degrees of freedom associated with the representation from among a plurality of available degrees of freedom, and (ii) a bitrate associated with the representation, wherein the plurality of representations include two or more of 0DoF, 3DoF, 3DoF+, and 6DoF representations of the adaptive spatial content;
estimating bandwidth available for streaming content to the client device;
selecting, at the client device, a first degrees of freedom representation from the plurality of degrees of freedom representations based on a first estimate of the available bandwidth;
requesting, from a server, the first representation;
rendering, at the client device, the first representation of the content using the degrees of freedom associated with the first representation of the content;
selecting, at the client device, a second representation from the plurality of degrees of freedom representations based on at least one of a second estimate of the available bandwidth and available processing resources of the client device;
requesting, from a server, the second degrees of freedom representation;
switching from rendering the first representation of the content to rendering the second representation of the content using the degrees of freedom associated with the second representation of the content, wherein the first representation is associated with a different degrees of freedom than the second representation.