Patent Description:
Field: The disclosed embodiments relate to real-time or pseudo-real-time video transfer between a source and a destination where packets are transmitted.

Background: One of the most important approaches to create virtual reality (VR) experiences is that of reusing existing digital video capabilities instead of relying solely on computer three-dimensional (3D) graphics. Reusing existing digital video is generally referred to as VR video (e.g. <NUM> degree and <NUM> degree video) and can be applied both for artificial environments and real world video capture alike. The use of VR video has the advantage of re-utilizing existing capabilities in user devices already massively adopted such as smartphones, personal computers and VR head glasses.

A main differentiator of the VR video approach compared to traditional 3D graphics methods is that even using a <NUM> Ultra-High Definition (UHD) television is not sufficient to provide the right level of quality to produce a satisfactory VR immersion for users. Also, with the higher frame rates needed to provide a realistic experience for most content involving dynamic scenes, like sports, concerts, action movies or similar type of
content, <NUM> frames per second becomes a minimum necessary frame rate. Adding to these requirements, the stereo video format can even duplicate the amount of information needed. As a consequence, resolutions above <NUM> and bitrates, even well above <NUM> Mbps, may be needed to transmit high quality signals, which may be unmanageable for massive deployments over existing networks and for user devices and players. These high bandwidth and resolution requirements can prove to be a difficult barrier to overcome for users, depending on factors such as income level, geography and maturity of internet connectivity, thus the addressable population might be severely narrowed.

In the context of VR video and its high-resolution requirements, the existing transcoding methods may also result in the need of a very high computational power, and the time to prepare the streams increases exponentially. Another significant consequence is that the transmission of live events with high resolution and reliability is difficult with current technology. <NPL>'', describes ClassX which is an interactive online lecture viewing system developed at Stanford University. Unlike existing solutions that restrict the user to watch only a pre-defined view, ClassX allows interactive pan/tilt/zoom while watching the video. The interactive video streaming paradigm avoids sending the entire field-of-view in the recorded high resolution, thus reducing the required data rate. To alleviate the navigation burden on the part of the online viewer, ClassX offers automatic tracking of the lecturer. ClassX also employs slide recognition technology, which allows automatic synchronization of digital presentation slides with those appearing in the lecture video. Mavlankar et al. presents a design overview of the ClassX system and the evaluation results of a <NUM>-month pilot deployment at Stanford University. The results demonstrate that our system is a low-cost, efficient and pragmatic solution to interactive online lecture viewing. <NPL>" describes that video streaming with virtual pan/tilt/zoom functionality allows the viewer to watch arbitrary regions of a high-spatial-resolution scene. In our proposed system, the user controls his region-of-interest (ROI) interactively during the streaming session. The relevant portion of the scene is rendered on his screen immediately. An additional thumbnail overview aids his navigation. We design a peer-to-peer (P2P) multicast live video streaming system to provide the control of interactive region-of-interest (IROI) to large populations of viewers while exploiting the overlap of ROIs for efficient and scalable delivery. Our P2P overlay is altered on-the-fly in a distributed manner with the changing ROIs of the peers. The main challenges for such a system are posed by the stringent latency constraint, the churn in the ROIs of peers and the limited bandwidth at the server hosting the IROI video session. Experimental results with a network simulator indicate that the delivered quality is close to that of an alternative traditional unicast client-server delivery mechanism yet requiring less uplink capacity at the server. <CIT> describes content delivery and playback methods and apparatus. The methods and apparatus are well suited for delivery and playback of content corresponding to a <NUM> degree environment and can be used to support streaming and/or real time delivery of 3D content corresponding to an event, e.g., while the event is ongoing or after the event is over. Portions of the environment are captured by cameras located at different positions. The content captured from different locations is encoded and made available for delivery. A playback device selects the content to be received based on a user's head position. <CIT> describes a process for increasing the Quality of Experience for users that watch on their terminals (<NUM>) a high definition video stream (<NUM>, I, V) captured by at least one video capturing device (<NUM>) and provided by a server (<NUM>) to which said users are connected through their terminals (<NUM>) in a network, said process providing for: collecting, for each user of a sample of the whole audience of said video stream, at least information about the position of the gaze of said user on said video stream; aggregating all of said collected information and analysing said aggregated information to identify the main regions of interest (R1, R2, R3, R4) for said video stream according to the number of users' gazes positioned on said regions of interest; selecting at least a region of interest (R1, R2, R3) of said video stream to be displayed on some terminals (<NUM>) of said users.

What is needed are solutions capable of improving the quality of the <NUM> degree video experience for users and the speed at which high quality VR video content is created, including creation of VR video content in real-time. Additionally, solutions are needed to improve the transmission of VR video.

The present invention is defined by the features disclosed in the independent claims. Additional embodiments are defined in the dependent claims. The present disclosure describes a solution capable of improving the quality of the <NUM> video experience and the speed to prepare high quality VR video content in real-time. The solution solves the current problems in a novel way that remains compatible with existing hardware, video codecs (coder-decoder) and streaming protocols. An additional benefit of the disclosed solutions is that there are no additional requirements
imposed on devices. This solution includes server and client components that enable streaming over the internet at optimized bitrates with high quality, and is applicable for live content and for file based content.

The first part of the description covers an ingest method of the video content and a mechanism to segment the incoming content in fragments and store them in a repository. Such segmentation allows the synchronization of live video content when processing in parallel processes in subsequent steps in the workflow.

According to an aspect of the description, the method covers the processing of a VR video into a set of videos that allocate a larger part of the resolution to a specific region of the input image. Each of these specific regions is called a viewport.

Another aspect of the disclosure describes a process by which an input "recipe" file is used to orchestrate the generation of adequate viewports and time periods in a flexible way.

Still another aspect of the disclosure describes a secondary mechanism to reassign pixels within a viewport to maximize resolution around the center of the viewport.

Yet another aspect of the disclosure describes a mechanism for video frame-stamping. which is done in order to provide the client-side player with a way to determine the exact characteristics of each optimized video frame displayed to the user.

Another aspect of the disclosure describes the mechanism to map desired values into a bit stream for adequate frame stamping.

Another aspect of the disclosure describes a parity-based data quality assurance mechanism for the frame stamps.

Yet another aspect of the disclosure describes the flexibility frame stamping provides with regards to viewport implementation and switching, allowing for a rich mix of geometrical transformation, projection, and video resolution approaches.

Still another aspect of the disclosure describes the utilization of standard codecs (such as H. <NUM> or H. <NUM> (or MPEG-<NUM> Part <NUM>, Advanced Video Coding (MPEG-<NUM> AVC) a block-oriented motion-compensation-based video compression standard) and package the viewports for average bitrate (ABR) transmission following standard mechanisms (such as MPEG-DASH or HLS) to encode and package the generated videos. The disclosure also covers a secondary mechanism to improve encoding efficiency (without modifying codec implementation) in the case of stereoscopic video by removing unnecessary image complexity in parts of the stereoscopic image.

Another aspect of the disclosure describes a method for the generation of additional signaling metadata to identify the viewport conformation.

Yet another aspect of the disclosure describes the client side of the method followed to decode and present the VR video content.

Still another aspect of the disclosure presents the mechanism used by the client to decode the metadata produced in order to receive the information about viewport configurations, and use the viewport configuration information to fetch the most relevant optimized viewport.

Another aspect of the disclosure presents the mechanism used by the client to use the decoded metadata to present the incoming video frame appropriately, according to the specific viewport configuration.

Yet another aspect of the disclosure presents the utilization of client headset mechanisms to determine the position of the head of the user, used to select the adequate viewport stream.

Still another aspect of the disclosure further describes the client-side utilization of predictive models to estimate the future head position, which is in turn used to further optimize the request of appropriate viewports on the client side.

Another aspect of the disclosure further describes the capability to request more than one viewport in parallel to allow for faster viewport change capability, and the mechanism to detect the moment when the viewport change can be displayed to the user seamlessly.

Yet another aspect of the disclosure describes the mechanism by which the client decodes the frame stamp to acquire information about the frame about to be displayed, and the mechanism by which adjustments in the projected viewport are performed at the time of displaying the frame to the user.

Still another aspect of the disclosure further describes the utilization of the parity-based data quality mechanism to ensure that the frame stamp is being read properly and to act as backward compatibility enabler.

Another aspect of the disclosure further describes the parallel processing mechanism by which the set of viewports can be generated and packaged in real-time.

Yet another aspect of the disclosure describes the data collection mechanism by which information about the video playback and the user's position is transmitted back to an analytics backend, as well as the relevant data elements collected to know about user experience and behavior.

Still another aspect of the disclosure further depicts the backend mechanisms to aggregate and treat data to train machine learning models that help in client-side predictions.

These and other aspects of the disclosure and related embodiments will become apparent in view of the detailed disclosure of the embodiments that follow below.

An aspect of the disclosure is directed to methods comprising: receiving a video input having at least an <NUM> resolution; processing the received video input into two or more viewport segments that at least allocate more pixels in a first region and fewer pixels in a second region wherein processing the received video input into two or more viewport segments is performed in parallel; generating a first signaling information wherein the first signaling information is external metadata; and generating a second signaling information wherein the second signaling information is embedded metadata. Additionally, the methods can include one or more of the steps of playing the processed video, embedding the first signaling information and the second signaling information into one or more video frames, processing the received video input real-time, and generating adaptive bitrate renditions. Additionally, the step of generating adaptive bitrate renditions is configurable to further comprise a frame treatment process to optimize transmission of a stereoscopic video. The methods can also include the additional steps of communicating with a gaze position monitor to fetch an appropriate viewport, and parsing an embedded frame metadata at a client side for playback. Additionally, in some configurations, the step of calculating a predicted head position for a user and adjusting a playback request can be included in the response to the predicted head position. The steps of fetching a model state, training the model state, and saving the model state can also be included.

Another aspect of the disclosure is directed to servers comprising: a memory; a controller configured to receive a video input having at least an <NUM> resolution; process the video input into two or more viewport segments that allocate more pixels to a first region, resulting in less pixels allocated to a second region, wherein the two or more viewport segments are created in parallel, generate signaling information, both as external metadata and as embedded metadata in the video frames, and deliver the processed video input to a standard streaming origin folder for device streaming. The servers can be streaming servers. Additionally, the controller can be further configurable to at least one or more of segment the input video as a first process, set-up the processing tasks from a segmented source, detect pending processing tasks and process only those, which allows for several such servers to work efficiently in parallel, and generate an adaptive bitrate rendition, with additional optional frame treatment to further optimize transmission of stereoscopic videos.

Still another aspect of the disclosure is directed to methods comprising: receiving a video input with a resolution of at least <NUM> resolution having two or more video frames into a system; processing the received video input into two or more viewport segments that at least allocate more pixels in a first region and fewer pixels in a second region wherein processing the received video input into two or more viewport segments is performed in parallel; generating a first signaling information as external metadata and a second signaling information as embedded metadata in the two or more video frames; and delivering a processed video input from the system to a client device. The methods are further configurable to comprise one or more of the steps of adding embedded metadata into the video frames and generating additional signaling information of viewports, and generating adaptive bitrate renditions, with additional optional frame treatment to further optimize transmission of stereoscopic videos.

Yet another aspect of the disclosure is directed to methods comprising: receiving a video input having at least an <NUM> resolution; processing the received video input; generating a first signaling information wherein the first signaling information is external metadata; generating a second signaling information wherein the second signaling information is embedded metadata; and embedding the first signaling information and the second signaling information into one or more video frames. The methods are further configurable to comprise one or more of playing the processed video, processing the received video input into two or more viewport segments that at least allocate more pixels in a first region and fewer pixels in a second region wherein processing the received video input into two or more viewport segments is performed in parallel, processing the received video input real-time, and generating adaptive bitrate renditions. Additionally, the step of generating adaptive bitrate renditions is further configurable to comprise a frame treatment process to optimize transmission of a stereoscopic video in some configurations. Additionally, the steps of communicating with a gaze position monitor to fetch an appropriate viewport, and parsing an embedded frame metadata at a client side for playback can also be included in some configurations. The step calculating a predicted head position for a user and adjust a playback request in response to the predicted head position can also be provided. In some configurations, the steps also include fetching a model state, training the model state, and saving the model state.

Another aspect of the disclosure is directed to servers comprising: a memory; a controller configured to receive a video input having at least an <NUM> resolution; process the video input, segment the input video as a first process from a segmented source, generate signaling information, both as external metadata and as embedded metadata in the video frames, and deliver the processed video input to a standard streaming origin folder for device streaming. The server can be a streaming server, or any other suitable server configuration. Additionally, the controller is further configurable to process the video input into two or more viewport segments that allocate more pixels to a first region, resulting in less pixels allocated to a second region, wherein the two or more viewport segments are created in parallel. In some configurations, the controller is further configured to detect pending processing tasks and process only those, which allows for several such servers to work efficiently in parallel. In still other configurations, the controller is further configured to generate an adaptive bitrate rendition, with additional optional frame treatment to further optimize transmission of stereoscopic videos.

Yet another aspect of the disclosure is directed to methods comprising: receiving a video input with a resolution of at least <NUM> resolution having two or more video frames into a system; processing the received video input; generating a first signaling information wherein the first signaling information is external metadata; generating a second signaling information wherein the second signaling information is embedded metadata; embedding the first signaling information and the second signaling information into one or more video frames, and delivering a processed video input from the system to a client device. The methods can further comprise the step of adding embedded metadata into the video frames and generating additional signaling information of viewports and/or generate adaptive bitrate renditions, with additional optional frame treatment to further optimize transmission of stereoscopic videos.

Still another aspect of the disclosure is directed to methods comprising: receiving a video input having at least an <NUM> resolution; processing the received video input into two or more viewport segments; generating a first signaling information wherein the first signaling information is external metadata; generating a second signaling information wherein the second signaling information is embedded metadata; and communicating with a gaze position monitor to fetch an appropriate viewport. The methods can also comprise the step of playing the processed video. In at least some configurations, the step of embedding the first signaling information and the second signaling information into one or more video frames. Additionally, the received video input can be processed real-time. Some configurations of the method also include the step of generating adaptive bitrate renditions. Generating adaptive bitrate renditions can also further comprise a frame treatment process to optimize transmission of a stereoscopic video. In at least some configurations, the steps include one or more of parsing an embedded frame metadata at a client side for playback, calculating a predicted head position for a user and adjust a playback request in response to the predicted head position, fetching a model state, training the model state, and saving the model state.

Another aspect of the disclosure is directed to servers comprising: a memory; a controller configured to receive a video input having at least an <NUM> resolution; process the video input, generate signaling information, both as external metadata and as embedded metadata in the video frames, deliver the processed video input to a standard streaming origin folder for device streaming; and communicate with a gaze position monitor to fetch an appropriate viewport. The controller can be further configured to segment the input video as a first process, set-up the processing tasks from a segmented source, detect pending processing tasks and process only those, which allows for several such servers to work efficiently in parallel, and/or generate an adaptive bitrate rendition, with additional optional frame treatment to further optimize transmission of stereoscopic videos.

Yet another aspect of the disclosure is directed to methods comprising: receiving a video input with a resolution of at least <NUM> resolution having two or more video frames into a system; processing the received video; generating a first signaling information as external metadata and a second signaling information as embedded metadata in the two or more video frames; and communicating with a gaze position monitor to fetch an appropriate viewport. The methods are configurable to further comprise the step of adding embedded metadata into the video frames and generating additional signaling information of viewports. The step of generating adaptive bitrate renditions, with additional optional frame treatment can be further configured to optimize transmission of stereoscopic videos.

Another aspect of the disclosure is directed to methods comprising: receiving a video input having at least an <NUM> resolution; processing the received video input having two or more video frames wherein each video frame has a first half and a second half; increasing a bitrate in the first half of a first video frame and reducing a bitrate in the second half of the first video frame; and reducing an overall encoded bitrate for the video input. The methods can additional comprise one or more of playing the processed video, embedding a first signaling information and a second signaling information into one or more video frames, processing the received video input real-time, and generating adaptive bitrate renditions. Additionally, in some configurations, step of generating adaptive bitrate renditions can further comprise a frame treatment process to optimize transmission of a stereoscopic video. Additionally, the method can comprise the steps of communicating with a gaze position monitor to fetch an appropriate viewport, and parsing an embedded frame metadata at a client side for playback. The method can also include the step of calculating a predicted head position for a user and adjust a playback request in response to the predicted head position. In at least some configurations, the method also includes fetching a model state, training the model state, and saving the model state.

Another aspect of the disclosure is directed to servers comprising: a memory; a controller configured to receive a video input having at least an <NUM> resolution; increase a bitrate in the first half of a first video frame and reduce a bitrate in the second half of the first video frame; and reduce an overall encoded bitrate for the video input. Servers can be streaming servers. Additionally, the controller can further be configured to at least one of segment the input video as a first process, set-up the processing tasks from a segmented source, detect pending processing tasks and process only those, which allows for several such servers to work efficiently in parallel, and generate an adaptive bitrate rendition, with additional optional frame treatment to further optimize transmission of stereoscopic videos.

Still another aspect of the disclosure is directed to methods comprising: receiving a video input with a resolution of at least <NUM> resolution having two or more video frames into a system; processing the received video input into two or more viewport segments that at least allocate more pixels in a first region and fewer pixels in a second region wherein processing the received video input into two or more viewport segments is performed in parallel; generating a first signaling information as external metadata and a second signaling information as embedded metadata in the two or more video frames; and delivering a processed video input from the system to a client device. Additionally, the method can comprise the step of adding embedded metadata into the video frames and generating additional signaling information of viewports, and/or generating adaptive bitrate renditions, with additional optional frame treatment to further optimize transmission of stereoscopic videos.

The invention is set forth with particularity in the appended claims.

As illustrated in <FIG>, a VR video processing flow <NUM> is illustrated which is an entry point of the disclosed methods and apparatuses. On the backend of the VR video processing flow <NUM> is a high quality VR video input (e.g., <NUM> and <NUM> degree videos being the most common videos); an equi rectangular projection is a commonly utilized format for this purpose, though other geometries are also possible (e.g. cube maps, fisheye lens output). The VR video processing flow <NUM> starts <NUM> with an incoming video segment input <NUM>. According to the invention the incoming VR video segment input <NUM> is processed and segmented in order to perform the step of generating and stamping viewports according to the recipe <NUM>, which will be further processed to apply the transformation according to the specified geometry and re-encoded to general several bitrates for generating ABR renditions <NUM>. Once the ABR renditions <NUM> are generated, the package viewport is set <NUM>. Finally, additional signaling information is generated <NUM> to allow the client side to parse the viewport information appropriately. At that point the VR video processing flow <NUM> ends <NUM>.

All the processing steps (occurring during steps <NUM>-<NUM> shown in <FIG>) need parallel computation power due to the high resolution requirements, frame rate and stereoscopy of the content. This additional processing power is needed to address potential problems including, for example: desynchronization of frames, which can result in desynchronization of the audio and video tracks, or even between viewports. In order to ensure synchronization, the proposed embodiment proposes a method to ensure the ingestion of the video content in time segments (or simply segments) as the <NUM> video segment input <NUM> is provided. The video segments use a closed group of pictures (GOP) and are stored in files in a shared repository, with a file name coded as a function of the fragment number, as illustrated in <FIG>. Further processing will thus be able to be applied to individual segment files in parallel without losing the synchronization of the entire video by using the file name as the reference of the sequence.

All of the parallel processing needed for the viewport generation and encoding will be managed automatically by the system by adjusting the number of processes to the generated segmented files. <FIG> illustrate the considerations taken for on-demand and live scenarios: In <FIG> the on-demand case is depicted as requiring less parallel computing for the generation of views and for the transcoding of the views, whereas <FIG> depicts that, in order to attain a real-time processing speed, the systems and apparatuses are designed to take "n" parallel ViewGen computing nodes for a single viewport.

The disclosed methods ensure that a single process is assigned to a single file at a time, by assigning the correct single file name. When a file is being processed, the file name is modified to a new coded name, so that a new process is only established for the files that have not been renamed and thus already assigned to a process. By modifying the file name to a new coded name, the system can run several processes simultaneously and in parallel. For example, different video files can be processed in parallel while the methods are run on lighter computing nodes.

As will be appreciated by those skilled in the art, the parallel processes are applicable in contexts beyond VR video content. For example, parallel processes can be applied to any video content type where parallel processing of the content is necessary or useful and synchronization of the parallel jobs can become a concern.

As it becomes evident, the amount of possible combinations (codecs, resolutions, geometrical transformations and other adjustments that can be performed to the VR video stream) is quite significant. For this reason, it becomes necessary during the step of generating and stamping viewports according to the recipe <NUM> to use a configuration file containing a set of steps describing the transformations that should take place for a given source video. As will be appreciated by those skilled in the art, the recipe can provide for quite a bit of flexibility during actual operation while still achieving the desired results. This file is called the processing recipe and is illustrated in, for example, <FIG>, illustrate how the recipe is configurable to contain different descriptive blocks of viewports detailing the geometric and video encoding characteristics of the block viewports, as well as the period of time in which the block viewports could be utilized for display. In addition to this file, a full parsing and execution process can be provided to adequately perform the step of generating and stamping viewports according to the recipe <NUM>.

<FIG> illustrates a high-level flow for viewport generation <NUM>. The high-level flow for the viewport generation <NUM> process starts <NUM> by determining whether a transform map exists <NUM>. If a transform map exists (YES) then a transformation map is loaded <NUM>. After loading the transformation map <NUM>, pixels are translated by the system from input to output following a transformation rule <NUM>. Additionally, once the transformation map is loaded <NUM> by the system, data can be retrieved from or provided to a transformation map cache <NUM> for storage. If the transformation map does not exist (NO), the system determines whether all target pixels are mapped <NUM>. If all target pixels are not mapped (NO), then the output video pixel is mapped by the system to the 3D coordinates and the system determines that the video pixels should represent according to, for example, output geometry type <NUM>. As an optional step, once the output video pixel is mapped, the 3D coordinates are adjusted according to pre-warp strength factors to allocate more granularity to selected angles of the target geometry of 3D coordinates according to pre-warping strength <NUM>. As will be appreciated by those skilled in the art, multiple pre-warp methods can also be used during this optional step without departing from the scope of the disclosure.

Once the 3D coordinates are adjusted by the system, the 3D coordinates are mapped by the system to the pixels where the pixels should be represented in the input video according to an input geometry type <NUM>. Once the 3D coordinates are mapped by the system, a transformation rule is created from one or more source pixels to one or more target pixels <NUM>. At this point, the target pixels are mapped (YES), and the transformation rule set (or transformation map) is stored <NUM>. Data can be retrieved from or provided to a transformation map cache <NUM> from the stored transformation rule set. Once the transformation rule set is stored <NUM> by the system, pixels from input to output can be translated following the transformation rule <NUM>, and then the process proceeds to the stamp transformed viewport <NUM> step in <FIG>.

<FIG> describes in further detail the inner workings of the step of generating and stamping viewports <NUM> by the system according to the step of generating and stamping viewports of the recipe <NUM> (<FIG>), which are controlled by a dedicated orchestration compute node. After starting <NUM>, during a first step, the recipe file is parsed <NUM> to determine the amount of time periods that should be processed. Once the recipe is parsed <NUM>, the system determines if there are unprocessed periods <NUM>. If there are no unprocessed periods (NO), then an ABR manifest is generated by the system of the entire set of periods <NUM>, followed by generating additional signally for the entire set of periods <NUM>, and ending the process <NUM>.

If there are unprocessed periods (YES), then the system determines whether unprocessed viewports are in the period <NUM>. If there are no unprocessed viewports in the period (NO), then the process returns to the step of determining if there are unprocessed periods <NUM>. If there are unprocessed viewports in the period (YES), then the system determines if there are unprocessed representations in the viewport <NUM>. If there are no unprocessed representations in the viewport (NO), then the process returns to the step of determining whether unprocessed viewports are in the period <NUM>. If there are unprocessed representations in the viewport (YES), then the system generates geometrical transformations of a type and size specified for the viewport and profile at the step of generating geometrical transformation <NUM>, and stamp transformed viewport <NUM>. The step of generating geometrical transformation <NUM> and stamping transformed viewports <NUM>, can involve the actual generation of a geometrical transformation. The step of generating geometrical transformation <NUM> is summarized in <FIG> (steps <NUM>-<NUM>) and the process of stamping transformed viewports <NUM> is summarized in <FIG> (steps <NUM>-<NUM>). As will be appreciated by those skilled in the art, these two processes can be performed either in sequential mode (for example, performed by the same computing node called ViewGen in <FIG>), or in a parallel mode (for example, by separate nodes coordinated by the orchestration node in <FIG>). The processing in parallel would typically require the nodes generating the views to receive and report on individual processing jobs to the orchestrator, particularly regarding the completion of the processes. Once all periods, viewports and representations are finished, it can proceed to the step of generating ABR renditions <NUM> as explained in <FIG>.

The geometrical transformation process is outlined in <FIG>: The input video format must have a known geometrical equivalence. The most common case of geometrical transformation is occurs in an equirectangular video, where the mapping of the equirectangular video by the system to a spherical video geometry is known. See, for example, <FIG> which illustrates a mapping process into an equirectangular format <NUM> from left <NUM>, front <NUM>, right <NUM>, and back <NUM>. Other projections, such as cube maps and <NUM> degree projections, are also possible. Given a target geometry, the mathematical transformation from the target video input to a 3D space is achieved during the step of mapping the output video pixel to the 3D coordinates they should represent <NUM>, and the translation from that 3D space to a new plane projection for the input geometry type <NUM>.

<FIG> illustrates an example of cube mapping, which transforms an input of the equirectangular format <NUM>, shown in <FIG>, into a cube map <NUM> which has, for example, <NUM> squares including right <NUM>, left <NUM>, top <NUM>, bottom <NUM>, front <NUM> and back <NUM>.

Since videos are discrete matrices and the 3D equivalence are real numbers, it is possible that the equivalence is not exact or that artifacts may occur due to discretization of the 3D space. For this reason, a submapping is done for each target pixel to determine if the effect of slight variations in the target 2D coordinates results in different source video 2D coordinates. If so, a weighted averaging approach can be captured as part the process of creating a transformation rule from the source pixels to the target pixels <NUM>.

Additionally, it should be noted that a slight "padding" can be added by the system to the faces. Padding is performed to prevent artifacts when displaying the video in a 3D scenario and effectively duplicates some of the 3D coordinates around the edges of the faces. Padding is a preemptive step that can be used to ensure that the virtual space remains stable and continuous in the event of user head movements at the render time in the client devices. As will be appreciated by those skilled in the art, it is a well-known practice in <NUM>-D graphics in general when using meshes that surround the user. The exact strength of padding is a parameter known and shared between backend and client player as metadata as part of the signaling information <NUM>.

The allocation of greater pixels to areas of interest can be achieved anytime during the processes outlined in <FIG> by, for example, loading the transformation map <NUM> into the system and translating the pixels from input to output following the transformation rule <NUM>. Any intervening steps, by, for example, having different face mappings, can also be performed as part of the transformation rule. For example, one of the faces can occupy a larger 2D space but be meant to represent a smaller 3D area, thus being able to capture further detail on that area. Another way to achieve this effect is moving the "origin point" of the transformation function, from the default position at the geometrical center of the 3D space to an "off-centered" origin, which achieves the effect of a smooth and gradual change in pixel allocation per mapped face in the "off-centered" axis.

<FIG> illustrates two possible implementations of the disclosed approach. In a first configuration, the cube mapping is a cube map <NUM>, as shown and described in <FIG>. Slight variations can then be applied by the system to the cube mapping technique to, for example, allocate more 2D surface to one of the faces of the cube. In a first cube map variation <NUM>, the right <NUM>, left <NUM>, and top <NUM> are positioned adjacent one another from left to right and have approximately the same size. Back <NUM>, and bottom <NUM> are positioned top to bottom below the top <NUM>, also having a similar size to the right <NUM>, left <NUM> and top <NUM> which run from left to right along the top row. Front <NUM> is positioned to the left of the back <NUM> and bottom <NUM>, and below the right <NUM> and left <NUM> and has a size about 4x any of the remaining squares. A corner of the front <NUM> touches a corner of the top <NUM>, but no side of the front <NUM> touches any size of the top <NUM>. The second cube map variation <NUM> has a right <NUM>, left, <NUM>, top <NUM>, bottom <NUM> and back <NUM> positioned top to bottom along one side (the right side as shown), with the front <NUM> positioned adjacent the right <NUM>, left, <NUM>, top <NUM>, bottom <NUM> and back <NUM>. One side of the front <NUM> touches one side of each of the right <NUM>, left, <NUM>, top <NUM>, bottom <NUM> and back <NUM>. A third cube map variation <NUM> is illustrated with a right <NUM> and a left <NUM> positioned adjacent one another from left to right along a top edge, and a top <NUM>, back <NUM>, and bottom <NUM> along a side edge perpendicular to the top edge. A front <NUM> is positioned below the right <NUM> and a portion of the left <NUM> on one side and adjacent one side of the top <NUM>, back <NUM> and bottom <NUM> on a second side perpendicular to the first side. A fourth cube map variation <NUM> is illustrated with a right <NUM> and a left <NUM> from left to right along a first side of a front <NUM>. The top <NUM>, bottom <NUM> and back <NUM> are positioned in order from top to bottom adjacent one side of the left <NUM>. A side of the right <NUM>, left <NUM> and back <NUM> are adjacent one side of the front <NUM>. The remaining three sides of the front <NUM> do not touch any other sides of the right <NUM>, left <NUM>, top <NUM>, bottom <NUM>, or back <NUM>.

The optimization at one subset of the <NUM>-D space is done at the cost of having less 2D pixel space to represent a larger 3D area, thus resulting in a loss of quality in that part of the space. Therefore, an individual viewport optimized with this technique will provide a better resolution only within certain boundaries and not in the remaining space. This is why several optimized viewports are needed and why the method is designed to generate, signal and reproduce multiple viewports placed at different head positions and rotations. <FIG> illustrates a possible viewport layout <NUM> with <NUM> view points, all in the same perspective, with a pitch <NUM> about a first axis and a yaw <NUM> about a second axis. The pitch <NUM> includes, for example, points <NUM>, <NUM>, <NUM>, and <NUM>, while the yaw <NUM> includes points <NUM>, <NUM>, <NUM>, <NUM> and <NUM>.

Pixel mapping can be a heavy computational task, and once the pixel mapping is calculated it remains valid as long as source and target videos retain the same geometrical meaning. For this reason, a caching mechanism is added to the method and performed by the system. As discussed above in <FIG>, the transformation map cache <NUM> is checked by the system before attempting to compute the pixel mapping and loaded if successful in the step of loading the transformation map <NUM>. Once the pixel mapping is computed, the pixel mapping is stored by the system in the cache during step of storing transformation rule set <NUM> for use in future transformation tasks. The transformation map cache <NUM> will be determined to be existing if the same geometrical and video characteristics are found in both the input videos and the output videos, including, for example:.

Depending on the exact nature of the geometrical transformation applied by the system, a pixel density warp phenomenon might occur on the client side as a result of placing a flat face on a 3D environment.

<FIG> illustrates what the density warp phenomenon would do to a cube map face <NUM> once it is projected in a 3D environment <NUM>: essentially, the pixels at the center of the view will occupy a larger angular space due to the fact that they are placed closer to the user virtual position, therefore providing lower resolution precisely in the section where greater resolution should be allocated. In order to offset the density warp, within each section of the geometrical transform, an inverse pixel allocation warp is added in the step of adjusting the 3D coordinates according to pre-warping strength <NUM> shown in <FIG>. Adding the inverse pixel allocation warp effectively allocates more pixels by the system to the center of the transformation face which provides the greatest resolution. As illustrated in <FIG>, as a first pixel density map <NUM> is transformed into a second pixel density map <NUM>, the pixel allocation warp changes.

Greater granularity towards the exterior of each image frame will be provided at the cost of reducing the pixel density outside of the area, but with the desired effect of providing the maximum overall resolution at playback time on the client-side. Unlike similar approaches such as equiangular projections, the desired effect is not one of uniformity in pixel allocation, but one of providing the greatest pixel density in the center areas where the user attention is most likely to occur as illustrated in <FIG>. The first pixel density map <NUM> illustrates greater pixel density in the center, with noticeably decreased pixel density at the exterior corners. The second pixel density map <NUM> illustrates greater pixel density in the center and decreased pixel density at the exterior corners, where the decreased pixel density at the corners is a higher exterior corner pixel density than illustrated in the first pixel density map <NUM>.

After the frame with the desired geometrical transformation has been generated and warped, it can proceed to stamping. This procedure becomes necessary to allow the entire apparatus to remain agnostic to compression mechanisms and codecs, since it does not rely on proprietary metadata streams, tiling nor other codec-specific techniques. The stamping layout has been designed following these criteria:.

The subsystem to generate signaling stamps of the process for stamping <NUM> is described in <FIG>. This subsystem relies on the premise that there is a set of bits that need to be stamped in the frame, but makes no pre-defined assumptions with regards to the exact meaning of the bits. Consequently, there must be previously generated handshake mapping of the encoded bits (and later decoded in the player side), with meaningful values, and a mechanism to generate a bit stream out of these values before entering the stamping phase, such as the ones illustrated in <FIG>.

Once the stamping process <NUM> starts <NUM>, a black strip of the same width as the symbols is inserted <NUM> in the external border of the video to mark an area where stamps will be placed. Note that stamp will overlap with the frame, overwriting a small part of it, but the resolution of the frame remains the same, and the padding at the moment of generating the frame should be calibrated to avoid that the stripe overwrites meaningful frame information, done as part of creating a transformation rule from source pixels to target pixels <NUM> in <FIG>. Next, the first <NUM> symbols are placed at the middle of each of the edges of the picture <NUM>, stamping e.g. a white symbol for bit "<NUM>" (as illustrated in <FIG>) or a black symbol for bit "<NUM>" (as illustrated in <FIG>). The spaces in the <NUM> corners can be used for the <NUM> parity bits (achieved in the step of stamping the parity value in the corners <NUM> as binary shown in <FIG>). The ordering of magnitude for each of the sides must be set, e.g.: <NUM>-top, <NUM>-left, <NUM>-right, <NUM>-bottom, where <NUM> is the least significant bit. The pre-set understanding of the direction of growth makes it possible to derive meaning from the stamps. The center of each symbol must be aligned with the center of the side of the <NUM>-length frame <NUM> and after the first pass of the algorithm, the frame stamps will have been placed as shown in <FIG>. The center of each symbol aligns with the center of the side of the frame to allow correct stamp readings. The corners of the <NUM>-length frame <NUM> clockwise from top-left are: P0, P1, P3, and P2. A determination of whether more layers of depth are required <NUM> (NO), then symbols are placed in the middle of the remaining subdivisions of each side <NUM>; the same operation is repeated for each of the corners, placing symbols at the middle of the remaining subdivisions of each side and following the same order of relevance for the sides. If a determination of whether more layers of depth are required (YES), then the system obtains a parity check value <NUM> by counting the number of "<NUM>" bits stamped and applies a mod <NUM> operation. The parity value is then stamped in the corners <NUM> as a binary and the process ends <NUM>.

<FIG> illustrate frame layouts of bits that the disclosed algorithm produces with lengths of <NUM>-length frame <NUM> and the <NUM>-length frame <NUM> respectively. In <FIG>, the corners of the <NUM>-length frame <NUM> and the <NUM>-length frame <NUM> clockwise from top-left are: P0, P1, P3, and P2. The total number of symbols placed with a given number of layers (n) is defined by a geometric series: <MAT>
after which the process of obtaining the parity check value <NUM> is reached. Once all the bits have been placed in the stamps, a "parity" operation is performed by the system. Once the system obtains a parity check value <NUM> by counting the number of "<NUM>" bits stamped, the parity check value can be used on the client side for data integrity purposes. The parity check value can have a value between <NUM> and <NUM> which can be calculated by doing the modulo <NUM> operation on the number of bits set to "<NUM>" in the stamp. <MAT> The parity check value can be stamped in the frames in the corners <NUM> as a binary that were reserved previously. The ordering of those bits goes from left to right and then from top to bottom, where P0 (top-left corner) is the least significant bit and P3 (bottom-right corner) is the most significant bit, as illustrated in <FIG>. Once this step is finished, the frame is completely stamped and ready to proceed to packaging.

Once the viewports are generated and stamped in the step of generating and stamping viewports according to the recipe <NUM>, the content is encoded using a standard codec, such as H. <NUM> or H. <NUM>, when the ABR renditions are generated <NUM> and stored in new file segments that follow the same structure and timing as the ones provided by the segmenter in the ingest as shown in <FIG>.

<FIG> illustrates a VR video processing flow with parallel scalability structure for live video streaming. A video source <NUM> is provided which is in communication with a segmenter <NUM>. The segmenter <NUM> provides data to a segmenter database <NUM>. Data from the segmenter database <NUM> is processed through a plurality of views, from ViewGen xn<NUM> <NUM>, ViewGen xn<NUM> <NUM>, ViewGen xni <NUM>, and ViewGen xnv <NUM>, where n<NUM> is the number of view generators per viewport, and v is the number of viewports to generate. Resulting data from the ViewGen is compiled in a view generated database <NUM>. The data from the view generated database <NUM> is processed via a plurality of transcoders, transcoder xr<NUM> <NUM>, transcoder xr<NUM> <NUM>, transcoder xri <NUM>, and transcoder xrv <NUM>. The data resulting from the transcoders is put into a transcoder database <NUM> and then processed via a packager <NUM> and then an origin database <NUM>.

It is also possible to encode the content in several bitrates to produce ABR video, again using known standard transcoding techniques. All resulting files are then stored by the system in a new directory, following a name coding to preserve the synchronization. Such file structure will be used when the package viewport is set <NUM> to form the packets to be served by a commercially available packager, such as MPEG-DASH or HTTP live streaming (HLS streams). Such commercially available packager is an adaptive bitrate streaming technique that breaks the content into a sequence of small HTTP-based file segments. The commercially available packager will also provide the adequate manifest of the segments, that will include the stream collection of all viewports and bitrates, as in a standard case. It should be noted that, although it is possible to generate ABR renditions using the standard transcoding techniques, the embodiment here described expand this capability to allow to add different geometries in the same view direction, thus generating a subset of ABR renditions that are not simply composed of lower resolutions and bitrates, but also of different geometrical transformations and therefore different Field Of View (FOV) optimizations.

Since existing video encoding algorithms are generally more efficient when presented with low frequency pictures (e.g. areas with the same color encoded), a further optional improvement can be applied at the encoding stage for the case of stereoscopic videos by calculating a delta of each eye and encoding the result of such operation, instead of the original frames. In such encoding, one possible logic for calculating the encoded values would be:
Let L and R be the original values of color to be shown to the user in a scale from <NUM> to MAX, with EncodedL and EncodedR the values to be finally encoded in the video: <MAT> <MAT> A sample encoding output using this logic is shown in <FIG>. The encoding uses the inherent redundancy of stereo video to its advantage: increasing the bitrate slightly in one half of the image <NUM> and reducing the bitrate dramatically on the other half of the image <NUM>, thus reducing overall encoded bitrate of the entire video for a constant level of quality.

After the package viewport is set <NUM> as shown in <FIG>, according to the invention, the ABR manifest is configurable to contain all viewports in the same list. Additional signaling may be needed by a client player in order to perform a mapping of which videos in the ABR manifest belong to which viewports in the 3D space. In order to keep the ABR manifest fully standard-compliant, the ABR manifest is not altered or augmented. Instead a complementary signaling file called a viewport map generated as illustrated in the example of the additional signaling information generated in JSON format shown in <FIG>. The information about the mapping is parsed from the workflow recipe metadata outlined in the step of generating and stamping viewports according to the recipe <NUM> (<FIG>) and contains the center of the optimized view in horizontal (yaw) and vertical (pitch) angles. There are a couple of additional aspects to take into consideration when building this signaling file.

In addition to the spatial mapping described above, the viewport map file also provides information about multiple periods during playback. As a default, the spatial mapping applies for the duration of the video, but it is also possible for the spatial mapping to define signaling sections for specific periods of time, delimited by the start and end time. <FIG> provides an example of the output format of this signal in JSON format, where the default case is signaled by not having any specific timeline as part of its description, and a specific period will be preceded by its lower and upper time bounds. Optionally the complementary signaling can also include a description of different points of view, which client-side is used to restrict video alternatives to include only those relevant to that perspective. Although some streaming protocols such as MPEG DASH also provide their own period implementations, this approach allows the capability to be detached from protocol implementation and makes the overall apparatus agnostic to the streaming protocol.

The next part of the description corresponds to the client-side reproduction of the VR video. <FIG> depicts the main functional components of the client-side application <NUM> (user interface <NUM>, VABR player <NUM>, analytics reporter <NUM>, local web server <NUM>, signaling decoder <NUM>, viewport renderer <NUM> and pixel reallocation <NUM>), which is designed to run on existing home and mobile devices.

<FIG> covers the playback flow of the VR video <NUM> given the disclosed method for generating viewports and signaling them. The internet <NUM> is in communication with a client app <NUM>. The client app <NUM> has a network interface <NUM>, a user interface <NUM>, a VABR player <NUM> (which can include signaling decoder <NUM> and gaze position monitor <NUM>), a device video decoder <NUM>, and a viewport rendered <NUM> (which can include a stamp reader <NUM>, and a 3D scene adjuster <NUM>). Each component of the client app <NUM> is in communication with and/or provides data between one or more sub-components of the client app <NUM>.

Prior to actual video playback, a user engages with the user interface <NUM> and selects a video to play. The address of the selected video content is then sent to the Virtual Reality-Adaptive Bitrate player (referred as VABR Player <NUM> in <FIG>) during the UI-VABR communication process <NUM>, which in turn triggers the preparation of the 3D scene by the viewport rendered <NUM> where the VR video will be displayed and initiates a remote request of the VR video signaling metadata. The remote request takes place between the network interface and VABR player and <NUM> and between the network interface <NUM> and internet <NUM> during a first I-NI data transfer process <NUM>.

When the remote server responds with the signaling information between the internet <NUM> and network interface <NUM> during the second I-NI data transfer process <NUM>, the signaling information is immediately sent from the network interface <NUM> to VABR player <NUM> during a second NI-VABR data transfer process <NUM> and parsed by the VABR player <NUM> at the signaling decoder <NUM>. The signaling decoder <NUM> reads signaling metadata to set up relevant viewports in the signaled positions and sends data from the signaling decoder <NUM> and gaze position monitor <NUM> during a first SD-GPM data transfer process <NUM> to initialize gaze position monitor subcomponent. Once the gaze position monitor <NUM> and signaling decoder <NUM> are initialized, the VABR player <NUM> is ready to request the first segment indicated by the signaling metadata. The signaling decoder <NUM> and the gaze position monitor <NUM> are configurable to communicate back and forth by sending data in a first direction and a second direction between the components as shown by first SD-GPM data transfer process <NUM> and second SD-GPM data transfer process <NUM>.

<FIG> also covers the flow of VR video playback once the preparations stated previously are finished: The gaze position monitor continuously calculates the current and predicted head position using on one hand headset's interfaces and performing a prediction of the future head position, using a regression model that takes into consideration first and second derivatives of the head position and a set of offsetting variables to favor movements towards the equator of the field of view. This prediction is combined with the viewport configuration information previously set (at the gaze position monitor <NUM>) to compute the viewport most relevant for the user, defined as the one with the shortest angular distance to the predicted head position in the currently active point of view. This information is returned to the VABR player <NUM> and used by the VABR player <NUM> together with network conditions as parsed from the network interface (e.g. bandwidth, latency) and playback buffer conditions to determine which viewport to request next and at which bitrate. The request is then made from the VABR player <NUM> to the network interface <NUM> during a first NI-VABR data transfer process <NUM> and then forwarded by the network interface <NUM> to the internet <NUM> during the first I-NI data transfer process <NUM>. The response for this request is received from the remote server between the network interface <NUM> and internet <NUM> during the first I-NI data transfer process <NUM> and the network interface <NUM> and VABR player <NUM> during the second NI-VABR data transfer process <NUM>. It is important to note that, as long as the viewport is not changing, the VABR player <NUM> will request the segments in sequence and store them in a sequential buffer. In the case of viewport change, a second chain of buffering and decoding is created by the VABR player <NUM> in order to be able to handle two viewport decoding processes in parallel. Thus, if the result of the VABR player <NUM> is a new viewport, a request between the VABR player <NUM> and network interface <NUM> during the first NI-VABR data transfer process <NUM> will not follow the sequence of the existing viewport and instead attempt to load segments close to the playback position. The response from the request between the VABR player <NUM> and network interface <NUM> during the first NI-VABR data transfer process <NUM>, once it is received from the internet <NUM> by the network interface <NUM> during a second I-NI data transfer process <NUM> will be sent to from the network interface <NUM> to VABR player <NUM> during the second NI-VABR data transfer process <NUM> as a new buffer and decode chain. When data is transmitted to VABR player <NUM> from device video encoder <NUM> during a first VE-VABR communication process <NUM>, the VABR player <NUM> sends the received frame to the device video encoder <NUM>. Note that in the case of a viewport change, there might optionally be more than one decode chain as a result of the multiple request-response chains. When data is transmitted by the system to VABR player <NUM> from device video encoder <NUM> during first VE-VABR communication process <NUM>, the device video encoder <NUM> returns the decoded frame to the VABR player <NUM> and the VABR player <NUM> communicated with the viewport renderer <NUM> during VABR-VR data transfer process <NUM> to evaluate which frame to send for rendering. The decision of which frame to send for rendering is a mere pass-through of the frame when a single decode chain exists, but in the case of viewport change, the VABR player <NUM> will send the frame corresponding to the new viewport only once it has reached sync with the pre-existing viewport. Otherwise the VABR player will send the frame from the old viewport. The decoded frame that has been determined to be the most relevant one for the user is sent to the 3D engine, where a stamp reader <NUM> re-assembles the frame characteristics, e.g. the center of the viewport in angular coordinates, the center of the perspective point as Cartesian coordinates, an identifier for the exact geometrical transformation used in the viewport, thee version of stamp method, etc. The information is encoded as a bit stream as explained above, and the exact meaning of the bit stream is subject to a bit-to-decimal transformation such as the one illustrated in <FIG>.

The stamp reader component has the information about the exact mapping of decimals to bits according to the version read from the first significant bits of the stamp and proceeds with decoding the entire bit stream following that shared definition. Once these parameters are decoded in the 3D scene adjuster <NUM>, the scene adjuster subcomponent within the 3D engine takes the information by the stamp reader <NUM> and makes any necessary adjustments to the playback 3D models to adjust exactly to the metadata stamped (e.g. rotate the scene, move the center of perspective, prepare the exact 3D geometry that matches the one the frame was encoded for, recompose the original frames with the inverse operations as those done during stereo video encoding etc.). Finally, once the adjustments to the 3D scene are done, the 3D engine is ready to display the frame to the user and the data is transmitted from the viewport renderer to user interface <NUM> during VR-UI data transfer process <NUM>. As it becomes clear, this playback method allows to mix different methods of optimization on a frame-by-frame basis, accommodating for dynamic switching scenarios while also guaranteeing that the projection is performed appropriately so that the end user experience remains unaffected.

In the event of a viewport change, two approaches can be taken. The first approach is to sequentially place the video belonging to the new view port in the playback queue as part the transfer of data from the network interface to VABR player <NUM>, substituting the streaming chunks in the video buffer progressively as they become available. The second approach achieves an even faster viewport change can be achieved by having two device video decoders will be running in parallel, providing frames corresponding to different viewports when data is transferred from device video encoder <NUM> to VABR player <NUM> during first VE-VABR communication process <NUM>. These are two views of the same content that needs to be swapped when a viewport change is detected. This swap needs to be effectively done seamlessly to the user between the originating and the destination viewports. Only client devices with support for parallel video decoding can benefit from the second approach described here.

After the last frame of the originating viewport from the viewport rendered <NUM> is presented to the user interface <NUM> during data transfer <NUM>, the first frame of the destination viewport needs to be available. Those two consecutive frames of different viewports must correspond to consecutive frames in the original equirectangular video. Both streams of video frames obtained out of the device video decoder <NUM> during a second VE-VABR data transfer <NUM> and corresponding to the originating and destination viewports are continuously available to the stamp reader <NUM>. Stamp reader <NUM> is able to continuously read the stamping information that includes enough data to sequence the video frames, and to determine the exact original matching frame in both viewports. Thus, during the data transfer <NUM> between the stamp reader <NUM> and the 3D scene adjuster <NUM>, both video frame streams are adjusted so that the sequence number of the originating viewport is (n-<NUM>), being (n) the sequence number of the first frame available on the destination viewport.

The apparatus is also configurable to support playback of unstamped videos. Supporting playback of unstamped videos does not impose the requirement that the content go through the stamping process and allows backward compatibility with previously existing content in equirectangular format. The viewport rendered <NUM> is configurable to render the data without stamping, where parity bits are used to validate that the signaling metadata decoded is validated by making the same calculation of bit counting and comparing it to the read bits. If the parity test is passed, then the 3D scene adjuster <NUM> acts as described above. If the parity test is not passed, the frame is considered equirectangular by default and the 3D scene adjuster <NUM> takes the default behavior of projecting the frame as an equirectangular one.

The gaze position monitor <NUM> is optionally configurable to perform a measurement and prediction of the future head position. <FIG> expands on the gaze position monitor <NUM> to describe a method of data collection and treatment that results in the head position prediction. Once the selection of content is performed at the user interface <NUM> and the manifest information arrives from the network interface <NUM> to the VABR player <NUM> via data transfer <NUM>, information about a playback state is collected by the VABR player <NUM>. The collected video playback information <NUM> includes elements about the video itself and the collected visualization information <NUM> which includes elements about VR-specific visualization aspects. An illustration of the data elements collected are illustrated in <FIG>.

<FIG> illustrates a block diagram of data elements <NUM> collected from the client. The client <NUM> includes head position (x,y,z) <NUM>, head orientation (yaw, pitch, roll) <NUM>, eye orientation (yaw, pitch, roll) <NUM>, controller position (xyz) <NUM>, controller orientation (yaw, pitch, roll) <NUM>, content URL <NUM>, bitrate <NUM>, viewport displayed <NUM>, timestamp <NUM>, and content timestamp <NUM>.

<FIG> also illustrates a flow diagram of the treatment of the client data before it can be used for predictive modeling <NUM>: The client <NUM> starts <NUM> (from the data transfer <NUM> between network interface <NUM> and VABR player <NUM> in <FIG>). After starting, video playback information is collected <NUM>, followed by collecting visualization information <NUM>. From there the process proceeds to the backend data layer <NUM> and aggregates client information <NUM>. Once client information is aggregated, a filter and data cleanup process <NUM> is performed, after which metadata can optionally be enriched <NUM>. The result of either the filter clean-up or enrichment steps is provided to an intermediate data store <NUM>. Additionally, the result can be provided to external data layer <NUM> and its metadata database <NUM>. The backend data layer <NUM> can also include a model data store <NUM> which is in communication with a model acquisition state <NUM> process on the client <NUM> and a fetch model state <NUM> on a backend machine learning layer <NUM>.

The model acquisition state <NUM> provides information to a gaze position monitor <NUM> which calculates a predicted head position <NUM> and then adjusts a playback request <NUM>. Once the playback request is adjusted, the process can end or return to the gaze position monitor <NUM> in the VABR player <NUM> shown in <FIG>. Additionally, a backend video original layer includes an origin folder <NUM> which provides information to origin folder process <NUM> on the client <NUM>. The origin folder <NUM> received video workflow <NUM> which includes information from the processes illustrated in <FIG>, <FIG>, <FIG> and <FIG>.

The backend machine learning layer <NUM> receives information from intermediate data storage <NUM> and then prepares data via a feature engineer <NUM>. The output from the feature engineer <NUM> is the provided to a process for selecting training, test and cross-validation sets <NUM>. Once the training, test and cross-validation sets are selected by the system, the process determines if there is a pre-existing model state <NUM> exists. If there is a pre-existing model state (YES), then a model state is fetched <NUM>. The model state can be fetched from the model data store <NUM> in the backend data layer <NUM>. Then the output provided to train a model <NUM>. If there is no pre-existing model state (NO), then the process proceeds directly to train a model <NUM>. Once the model train process is completed the model state is saved <NUM> and the output is provided to the model data store <NUM>.

When the client information is aggregated <NUM>, the raw data from individual clients is aggregated by the backend data layer <NUM>, and a filtering and data cleanup process <NUM> is applied to it, in order to make information about timestamps uniform, prevent fake data elements from arriving, tagging the data elements with additional information about their origin and processing and merging multiple individual client requests into larger data sets more suitable for data storage. During the video workflow <NUM> (referring to <FIG>, <FIG>, <FIG> and <FIG>), a third-party content metadata provider can be contacted to expand on information about the content being watched by the user. This step is optional as models can work without this information. This processed data is then sent by the system to a data store for future usage in model training.

The backend machine learning layer also depicted in <FIG> can be set to run at regular intervals. Furthermore, the backend machine learning layer can also be configured to take the information produced by the backend data layer to perform model training. For that, the data is prepared via several feature engineering techniques in the feature engineer <NUM>. The data preparation in the feature engineer <NUM> can vary depending on the type of raw data elements and can include discretization, normalization range bucketing, convolution, and compression, among others.

<FIG> illustrate an exemplar of feature engineering steps that would be used to transform the stored data into elements that are ready to be processed through the machine learning model. With the data transformation completed, it is possible to use the transformed data in a machine learning training and validation pipeline:.

<FIG> illustrates a schema of a data set that would enter the validation pipeline as illustrated in <FIG>. The data set is split into training, test and cross-validation sets during the select training, test and cross-validation sets <NUM> in <FIG>. In the event that there is a prior state of the model trained by the system already, the model is fetched by the system to initialize the models during the fetch model state <NUM>. The data set is then used by the system to train the model to train the model <NUM>. The resulting model state is either added or updated in the corresponding data store to save the model state <NUM>.

The model weights and parameters generated during the trained model <NUM> process and stored during the save model state <NUM> are meant to be utilized by one or more clients <NUM>, as the model weights and parameters which are processed and stored contain the weights that will be employed to calibrate the model to predict the future head position. The model state relevant to that playback session is retrieved from the remote data store at the acquire model stage <NUM>. The current head position of the user is then obtained by the system from the headset and added as an input for the estimation model to produce an estimation of what the future head position will be in a given timeframe when the predicted head position <NUM> is calculated (the length of that prediction being a configurable parameter of the gaze position monitor <NUM>). That head position prediction is used in combination with parsed signaling metadata which the signaling decoder <NUM> (<FIG>) uses to calculate the viewport with the closest angular distance to the predicted head position. Once the viewpoint calculation is completed, a playback process continues at the gaze position monitor <NUM> and then proceeds as described above.

The model training pipeline in the backend learning layer <NUM> is configurable to accommodate various levels of granularity, which are used depending on the number of training samples available:.

These models are fetched in separate pipelines as part of the fetch model state <NUM>, trained as part of the training model <NUM>, and independent model states are stored as part of the save model state <NUM>, but only one of the saved models is given to the client <NUM> as part of the model acquisition state <NUM>. The basic logic used by the system to determine which model to provide is: choose the highest qualifying model level: where a model level is considered as qualifying if the number of training samples used to train it is above a specified minimum sample threshold.

The systems and methods according to aspects of the disclosed subject matter may utilize a variety of computer and computing systems, communications devices, networks and/or digital/logic devices for operation. Each may, in turn, be configurable to utilize a suitable computing device that can be manufactured with, loaded with and/or fetch from some storage device, and then execute, instructions that cause the computing device to perform a method according to aspects of the disclosed subject matter. Certain parts of this disclosure may be activated or deactivated according to the computing capabilities of each specific device.

A computing device can include without limitation a mobile user device such as a mobile phone, a smartphone and a cellular phone, a personal digital assistant ("PDA"), a tablet, a laptop, a dedicated Virtual Reality Head-Mounted Display (HMD), and the like. In at least some configurations, a user can execute a browser application over a network, such as the Internet, to view and interact with digital content, such as screen displays. A display includes, for example, an interface that allows a visual presentation of data from a computing device. Access could be over or partially over other forms of computing and/or communications networks. A user may access a web browser, e.g., to provide access to applications and data and other content located on a website or a webpage of a website.

A suitable computing device may include a processor to perform logic and other computing operations, e.g., a stand-alone computer processing unit ("CPU"), or hard wired logic as in a microcontroller, or a combination of both, and may execute instructions according to its operating system and the instructions to perform the steps of the method, or elements of the process. The user's computing device may be part of a network of computing devices and the methods of the disclosed subject matter may be performed by different computing devices associated with the network, perhaps in different physical locations, cooperating or otherwise interacting to perform a disclosed method. For example, a user's portable computing device may run an app alone or in conjunction with a remote computing device, such as a server on the Internet. For purposes of the present application, the term "computing device" includes any and all of the above discussed logic circuitry, communications devices and digital processing capabilities or combinations of these.

Certain embodiments of the disclosed subject matter may be described for illustrative purposes as steps of a method that may be executed on a computing device executing software, and illustrated, by way of example only, as a block diagram of a process flow. Such may also be considered as a software flow chart. Such block diagrams and like operational illustrations of a method performed or the operation of a computing device and any combination of blocks in a block diagram, can illustrate, as examples, software program code/instructions that can be provided to the computing device or at least abbreviated statements of the functionalities and operations performed by the computing device in executing the instructions. Some possible alternate implementation may involve the function, functionalities and operations noted in the blocks of a block diagram occurring out of the order noted in the block diagram, including occurring simultaneously or nearly so, or in another order or not occurring at all. Aspects of the disclosed subject matter may be implemented in parallel or seriatim in hardware, firmware, software or any combination(s) of these, co-located or remotely located, at least in part, from each other, e.g., in arrays or networks of computing devices, over interconnected networks, including the Internet, and the like.

Claim 1:
A method comprising:
receiving (<NUM>), into a system, a 3D virtual reality, VR, video input having at least an <NUM> resolution; segmenting (<NUM>) the received video input into video time segments;
processing (<NUM>), according to a recipe, the video time segments in real-time into two or more viewports, wherein processing the video time segments into two or more viewports is performed in parallel and comprises re-encoding to several bitrates for generating ABR renditions (<NUM>) for each viewport,
wherein the recipe is a configuration file containing different descriptive blocks of viewports detailing the geometric and video encoding characteristics of the viewports of a block, as well as the period of time in which the viewports of a block are utilized for display;
generating (<NUM>) an adaptive bitrate rendition, ABR, manifest, wherein the ABR manifest is configured to contain all viewports in a same list;
generating (<NUM>) a viewport map file containing a mapping of which video segments in the ABR manifest belong to which viewports in 3D space;
wherein the viewport map file furthermore contains information about spatial location of the viewports;
generating metadata (<NUM>) containing the information about the spatial location of the viewports, and furthermore containing information about how the viewports are configured in 3D space, and
wherein the metadata (<NUM>) is added into one or more video frames; and
delivering processed video input from the system to a client device (<NUM>).