Synchronization and presentation of multiple 3D content streams

Systems, methods, and computer-readable media are disclosed for synchronization and presentation of multiple 3D content streams. Example methods may include determining a first content stream of 3D content to send to a user device, where movement of the user device causes presentation of different portions of the 3D content at the user device, and determining a first position of the user device. Some methods may include causing presentation of a first portion of the first content stream at the user device, where the first portion corresponds to the first position, determining a second content stream of 3D content, where movement of the user device causes presentation of different portions of the 3D content at the user device, and causing presentation of a second portion of the second content stream at the user device, where the second portion corresponds to the first position of the user device.

BACKGROUND

Certain digital content, such as movies, television shows, live content, and other video content may be streamed using electronic devices. In some instances, live content, such as sporting events and concerts, may also be streamed. However, such streaming events may not be immersive or may provide a user experience that is vastly different than experiencing the live event in person. Moreover, interaction with certain types of 3D content may present technical challenges. Accordingly, synchronization and presentation of multiple 3D content streams may be desired.

DETAILED DESCRIPTION

Overview

Digital content may include video content, such as movies, television shows, streaming shows (e.g., made for Amazon Prime, Netflix, etc.), and other video content. Users may stream content using one or more devices. For example, content can be streamed using virtual reality headsets or other equipment, smartphones, tablets, laptops, augmented reality hardware, and other types of computer systems. Three-dimensional (3D) content streams may include audio and video that is captured using one or more 3D cameras. For example, a 3D camera system may include multiple cameras (and optionally other hardware, such as microphones, etc.) arranged in different orientations that allow for capturing of content about 360 degrees of the 3D camera system. In some instances, the content captured by a 3D camera system can be streamed to users for consumption on any suitable device. The content streamed may include the video content, audio content, and may optionally include metadata related to a specific camera from which the content is captured, so as to allow the user to change their perspective based on manipulation of the device on which the content stream is being consumed. For example, a user wearing virtual reality hardware may physically move the hardware to change the perspective and/or camera from which the content stream is presented, thereby providing an immersive experience for the user. In addition, synchronization of presentation of content may be important for the user, such that switching between content streams from different cameras of a 3D camera system does not change a point in time from which the content begins streaming (e.g., the timing is aligned so that the content stream does not move forward or backwards relative to a previous stream, etc.). Moreover, synchronization of audio content during movement of a device and corresponding changes to content streams may be important to provide a high quality user experience.

Such issues may be more important during presentation of content streams from live events. For example, 3D content streams at locations such as concerts, sporting events, and other live events may present additional challenges, as users consuming the content streams may desire different points of view. In one example, a user may desire to consume a 3D content stream from a first 3D camera system at a location, such as onstage with a performer, and then switch to a 3D content stream from a second 3D camera system at another location, such as in a crowd of concertgoers. Such changes between content streams may present synchronization issues that can be amplified due to various factors, such as network connectivity. However, synchronization may be important for a user experience, and may enhance an immersive quality of the content streaming experience. Further, when switching between more than one content stream, the user experience may be diminished if there is a large discrepancy in content quality. For instance, if one content stream is significantly worse than the others, the user may not want to view the content stream anymore. Such discrepancies can occur due to network issues, content processing issues, and the like.

Embodiments of the disclosure provide synchronized presentation of content, such as audio and video content, across multiple 3D camera systems, so as to provide an immersive and improved user viewing experience that allows users to not only feel present as part of a livestreamed event, but also allows users to manipulate views and camera positions seamlessly. Some embodiments include capturing of 3D video content using a plurality of 3D camera systems that can be positioned at different physical locations within a live event venue, such that the event can be livestreamed and enjoyed virtually by those not physically present. Users may select from different viewing experiences based on the different positioning of the 3D camera systems, such as from onstage experiences, backstage experiences, sideline experiences, front row experiences, penalty box experiences, and so forth. The 3D nature of the content streams may provide an immersive experience, such that a user viewing an onstage stream can not only watch a performer from a few feet away, but can also turn and look out at the crowd to see the perspective of the performer. Some embodiments may be configured to modify content streams to include graphical representations of users viewing the content stream, such as graphical avatars, to allow users to digitally attend live events together. Any suitable device may be used to consume such 3D content streams, including accelerometer-enabled hand-held devices, gaming consoles, smart television devices, voice controlled devices (e.g., Echo, Echo Show, etc.), and other devices. Embodiments may therefore allow for users to switch viewing positions seamlessly without a noticeable break in playback or jarring audio change, and may also support group viewing with friends and family with everyone watching the feed in a synchronized fashion.

Referring toFIGS.1A-1B, an example use case100for synchronization and presentation of multiple 3D video streams or content streams is depicted in accordance with one or more example embodiments of the disclosure. In the example ofFIG.1A, 3D content (“3D content” as used herein may include video content, audio content, metadata, and/or other data) may be streamed at a device, such as a smartphone. The 3D content may be captured from a number of 3D camera systems, such as a first 3D camera system120, a second 3D camera system130, a third 3D camera system140, and so forth. Some embodiments may include two or more 3D camera systems. The 3D camera systems may be positioned at various locations at an event venue110, so as to stream the live event. For example, the third 3D camera system140may be disposed onstage at the live event, which may be a concert. The first 3D camera system120may be disposed at a left side of the stage in the example ofFIG.1A, and the second 3D camera system130may be disposed at a right side of the stage in the example ofFIG.1A. Other embodiments may have different positioning and may include additional or fewer camera systems. The individual 3D camera systems may include multiple cameras. For example, inFIG.1A, the 3D camera systems may include six different cameras arranged at different orientations, as represented by the hexagonal shape of the 3D camera systems inFIG.1A. Other embodiments may utilize a greater or lesser number of cameras for each 3D camera system. For example, some embodiments may use eight cameras arranged at different orientations. The cameras may be used to stitch together a 3D video of the environment surrounding the respective 3D camera system (e.g., 360 degree coverage about the 3D camera system, etc.). The different cameras of the different 3D camera systems may provide portions of the 3D view.

The user may view different portions of the individual 3D camera system streams based on manipulation or movement of the device. The user may also switch between the 3D camera systems to consume different 3D content streams and to enjoy a fully immersive experience. For example, at a first point in time122, the user may be consuming a portion of the 3D content stream from the first 3D camera system120. The portion being consumed may be captured by the camera having a field of view represented by dashed lines inFIG.1A. Accordingly, the user may be viewing a band onstage from a left side of the stage during the first point in time122.

At a second point in time142, the user may switch to consume a portion of the 3D content stream from the third 3D camera system140. The portion being consumed may be captured by the camera having a field of view represented by dashed lines inFIG.1A, which may be an onstage view. Accordingly, the user may be viewing a band onstage from an onstage viewpoint during the second point in time142.

At a third point in time132, the user may switch to consume a portion of the 3D content stream from the second 3D camera system130. The portion being consumed may be captured by the camera having a field of view represented by dashed lines inFIG.1A, which may be a view of the band onstage from a right side of the stage. Accordingly, the user may be viewing a band onstage from a right side of the stage viewpoint during the third point in time142.

The user may physically manipulate the device, or may otherwise provide input commands, to change the views being streamed to the different viewpoints of the cameras for individual 3D camera systems. For example, the user may spin 180 degrees while streaming content from the third 3D camera system140to view a crowd that is in attendance from the band's perspective. As the user rotates, the content stream presented may change from a first camera of the third 3D camera system140, to another camera, to another camera, and so forth, thereby allowing the user to feel immersed in the live performance and/or ambiance.

To synchronize the 3D content streams across the 3D camera systems, one or more remote servers, such as a streaming server, may perform operations in an example process flow150. The streaming server may include at least one memory that stores computer-executable instructions and at least one processor configured to access the at least one memory and execute the computer-executable instructions to perform various actions or operations, such as one or more of the operations in the process flow150ofFIG.1.

At a first block152, the streaming server may cause presentation of a first content stream from a first 3D camera. For example, the streaming server may cause presentation of a content stream at a user device, where the first content stream is captured using the first 3D camera system120. In some embodiments, the content stream generated by the 3D camera systems may be singular and include data from all of the cameras at the 3D camera system, whereas in other embodiments, the 3D camera systems may generate multiple content streams, each corresponding to the individual cameras used to capture the 3D content at a particular 3D camera system.

At a second block154, the streaming server may present a portion of the first content stream corresponding to movement of a streaming device, such as the smartphone in the example ofFIG.1A. In an embodiment, the content stream from a 3D camera system may include data for the content captured by multiple cameras of the same 3D camera system. The streaming server may determine or select a view, which may be represented as a portion of a content stream, to present at the streaming device based at least in part on a position of the streaming device. This may allow for a user to move within a virtual 3D space by physical movement of the device, as the view or portion of the content stream presented may change in correspondence with the movement of the user device. For example, a user moving their head while wearing streaming goggles or a streaming headset may seamlessly change content stream portions based at least in part on the amount of physical movement of the device. The user may therefore feel like they are present at the event.

At a third block156, the streaming server may determine a point in virtual 3D space that corresponds to a streaming device position. For example, the streaming server may be configured to track a position of the streaming device in a 3D virtual space, such that if the user switches to a content stream from a different 3D camera system, a point of view or a subject of the camera system can be maintained, even if the camera providing the view is different. In the example ofFIG.1A, when the user switches between the first 3D camera system120and the second 3D camera system130, the streaming server may use a default positioning of the content stream oriented towards the stage, provided that the user was viewing the stage using the first 3D camera system120before switching to the content stream from the second 3D camera system130. The user may therefore enjoy a seamless experience without having to continually physically move a streaming device to orient to a desired viewpoint when changing content streams.

At a fourth block158, the streaming server may cause presentation of a second content stream from a second 3D camera system using the determined point for initial presentation. Accordingly, the user may seamlessly switch between views from different 3D camera systems while maintaining a position in a three-dimensional space. Physical manipulation of a streaming device may be used to change the position of the user in the 3D virtual space, and may remain consistent across switches in content streams by providing the same initial view that the user was consuming immediately prior to switching content streams.

InFIG.1B, a top view160of a 3D camera system, such as any of those discussed with respect toFIG.1A, is depicted. The 3D camera system may include a number of cameras used to capture video content of 360 degrees about the 3D camera system. In the example ofFIG.1B, the 3D camera system may include six cameras, where each of the cameras has a different field of view with respect to the 3D camera system. For example, a first camera may have a first field of view, labeled View 1 inFIG.1B, a second camera may have a second field of view labeled View 2, a third camera may have a third field of view labeled View 3, a fourth camera may have a fourth field of view labeled View 4, a fifth camera may have a fifth field of view labeled View 5, a sixth camera may have a sixth field of view labeled View 6, and so forth. Different embodiments may have different number and/or orientations of cameras. Although illustrated as discrete field of views for ease of illustration, there may be at least some overlap between the fields of view of adjacent cameras. Such overlap may allow for seamless transitions between viewpoints from the different cameras, such as if a user rotates their head while wearing a virtual reality headset.

To view content stream portions corresponding to the different fields of view, a user may physically move a streaming device, such as a headset, smartphone, or other streaming device. In the example ofFIG.1B, the user may move a device as shown in movements170. At a first position172, the position of the user device in a 3D space may correspond to View 1, or a content stream from the first camera of the 3D camera system. At a second position174, the user may rotate the streaming device about 135 degrees, and the content stream corresponding to View 3 from the third camera may be presented. The user may rotate the streaming device another approximately 100 degrees, and the content stream corresponding to View 5 from the fifth camera may be presented. As the streaming device is rotated, intervening content streams may be presented during motion of the device. For instance, the content stream from View 2 of the second camera may be presented between the first position172and the second position174, etc.

When stitched together, the content stream from the 3D camera system may create a video in the form180, where a different view (or blend between adjacent views), is presented corresponding to position data associated with a streaming device (where applicable). For static devices, a user may manipulate the presented content stream via one or more inputs instead of physical movement, such as cursors, touch input, etc. The content stream from the 3D camera system may therefore provide a 360 degree view about the 3D camera system, and different portions of the content stream can be consumed based on physical positioning and/or other inputs at a streaming device.

Accordingly, a computer system, such as one or more streaming servers, may receive a first request from a user device (e.g., a streaming device, etc.) for a first live video stream from a first three-dimensional camera that is positioned at a first location at a venue, where the first live video stream includes 360 degree video of the first location. The streaming server may send the first live video stream to the user device, where content presented at the user device corresponds to a physical positioning of the user device, and where movement of the user device causes different portions of the 360 degree video captured by the first three-dimensional camera to be presented at the user device.

In some embodiments, the streaming server may determine a first physical position of the user device, and may cause presentation of a first portion of the 360 degree video captured by the first three-dimensional camera that corresponds to the first physical position. In other embodiments, the user device may determine a first physical position of the user device, and may present a first portion of the 360 degree video captured by the first three-dimensional camera that corresponds to the first physical position.

The streaming server may receive a second request from the user device for a second live video stream from a second three-dimensional camera that is positioned at a second location at the venue, where the second live video stream includes 360 degree video of the second location. Accordingly, the user may desire to switch from one camera system to another.

The streaming server may send the second live content stream to the user device, where content presented at the user device corresponds to a physical positioning of the user device, and where movement of the user device causes different portions of the 360 degree video captured by the second three-dimensional camera to be presented at the user device. The streaming server or the user device may determine a second physical position of the user device, and cause presentation of (or may present in the case of the user device) a second portion of the 360 degree video captured by the second three-dimensional camera that corresponds to the second physical position.

Example embodiments of the disclosure provide a number of technical features or technical effects. For example, in accordance with example embodiments of the disclosure, certain embodiments of the disclosure may automatically synchronize presentation of content streams from 3D camera systems across multiple 3D camera systems. Some embodiments may determine points of focus to maintain consistency across changes in camera systems by selecting an initial view corresponding to a most recent point of focus at a prior camera system. As a result of improved functionality, bandwidth utilization may be optimized, and latency and errors in streamed data may be reduced. The above examples of technical features and/or technical effects of example embodiments of the disclosure are merely illustrative and not exhaustive.

One or more illustrative embodiments of the disclosure have been described above. The above-described embodiments are merely illustrative of the scope of this disclosure and are not intended to be limiting in any way. Accordingly, variations, modifications, and equivalents of embodiments disclosed herein are also within the scope of this disclosure. The above-described embodiments and additional and/or alternative embodiments of the disclosure will be described in detail hereinafter through reference to the accompanying drawings.

Illustrative Process and Use Cases

FIG.2depicts an example process flow200for synchronization and presentation of multiple 3D content streams in accordance with one or more example embodiments of the disclosure. While example embodiments of the disclosure may be described in the context of live events, it should be appreciated that the disclosure is more broadly applicable to any type of streamable video content using multiple 3D camera systems. Some or all of the blocks of the process flows in this disclosure may be performed in a distributed manner across any number of devices. The operations of the process flow200may be optional and may be performed in a different order.

In one example embodiment, the process flow200may be executed by one or more remote servers, such as a streaming server, that is configured to stream content from 3D camera systems to one or more streaming devices, such as user devices. In other embodiments, the process flow200may be executed locally at a device, where the device may make certain determinations and send requests for certain content streams and/or portions of content streams to a streaming server.

At block210of the process flow200, computer-executable instructions stored on a memory of a device, such as a remote server or a user device, may be executed to determine a first content stream of 3D content to send to a user device, wherein movement of the user device causes presentation of different portions of the 3D content at the user device. For example, a content streaming engine and/or content streaming module(s) executed at a server or a device may determine a first content stream of 3D content to send to a user device, wherein movement of the user device causes presentation of different portions of the 3D content at the user device. To determine the first content stream of 3D content, a user may make a selection as to a particular 3D camera system and/or a particular view of the 3D camera system, and the streaming server may receive an indication of the selection. The streaming server may determine a first content stream, which may be a portion of a single content stream from a 3D camera system, or may be an individual content stream from a camera of the 3D camera system, for presentation at the user device. Accordingly, data from one or more cameras at the 3D camera system may be used to provide a content stream, or the first content stream, to the user device for presentation. In the event that the position of the device corresponds to a view between two adjacent cameras, the content stream presented at the device may be formed using data from both cameras (e.g., the frames may be stitched together, etc.). In some embodiments, the 3D content may be a content stream that includes 360 degree video captured by the 3D camera system. The streaming device used to consume the 3D content stream may be configured to be moved so as to present portions of the 360 degree video during movement of the user device.

At block220of the process flow200, computer-executable instructions stored on a memory of a device, such as a remote server or a user device, may be executed to determine a first position of the user device. For example, the content streaming engine executed at a device or a server may determine a first position of the user device. The position of the user device may be determined by the streaming server based at least in part on data received from the user device, such as accelerometer data, which may indicate movements in a datum or 3D virtual space of the user device. In another embodiment, the user device may receive a content stream that includes data from some or all of the cameras at a 3D camera system, and the device may modify the portion of the content presented at the device based at least in part on accelerometer data representing motion of the device.

At block230of the process flow200, computer-executable instructions stored on a memory of a device, such as a remote server or a user device, may be executed to cause presentation of a first portion of the first content stream at the user device, wherein the first portion corresponds to the first position. For example, the content streaming engine executed at a device or server may cause presentation of a first portion of the first content stream at the user device, wherein the first portion corresponds to the first position. The server may determine a view and/or portion of content stream corresponding to the position of the device, and may send the portion to the user device for presentation. In another embodiment, the user device may determine the portion of the content stream to present, and may present the portion of the content stream or content stream at the device.

At block240of the process flow200, computer-executable instructions stored on a memory of a device, such as a remote server or a user device, may be executed to determine a second content stream of 3D content, wherein movement of the user device causes presentation of different portions of the 3D content at the user device. For example, the content streaming engine executed at a device or a server may determine a second content stream of 3D content, wherein movement of the user device causes presentation of different portions of the 3D content at the user device. For example, a user may decide to switch content streams from a first 3D camera system to a second 3D camera system. The streaming server may therefore determine a second content stream of 3D content from the second 3D camera system. Presentation of different portions of the second content stream may be modified via manipulation of a streaming device on which the second content stream is presented. For example, similar to the first 3D camera system, movement of the user device causes presentation of different portions of the 3D content at the user device.

At block250of the process flow200, computer-executable instructions stored on a memory of a device, such as a remote server or a user device, may be executed to cause presentation of a second portion of the second content stream at the user device, wherein the second portion corresponds to the first position of the user device. For example, the content streaming engine executed at a device or a server may cause presentation of a second portion of the second content stream at the user device, wherein the second portion corresponds to the first position of the user device. As a result, when the user switches from a first content stream from a first 3D camera system to a second content stream from a second 3D camera system, the initially presented portion of the content stream of the second 3D camera system may correspond to the first position of the user device. For example, if the user is viewing a portion of a content stream oriented towards an onstage performer from a left side of the stage, and the user switches to a 3D camera system on a right side of the stage, the initially presented portion of the second content stream may also be oriented towards the onstage performer. In this manner, the user may not have to continually adjust physical positioning in order to have a consistent viewing experience across different 3D camera systems. The first content stream or first content stream may be captured by a first three-dimensional camera at a first location of a venue, and the second content stream may be captured by a second three-dimensional camera at a second location of the venue. Movement of the user device in a three-dimensional space during viewing of the first content stream can be reflected in an initial viewing position of the second content stream.

FIG.3is a schematic illustration of an example data flow300for synchronization and presentation of multiple 3D content streams in accordance with one or more example embodiments of the disclosure. Different embodiments may include different, additional, or fewer inputs or outputs than those illustrated in the example ofFIG.3. The data flow300may be performed at one or more remote servers, such as a streaming server, configured to deliver content streams to a user device, or may optionally be performed at a user device locally.

InFIG.3, a content streaming engine310and/or one or more content streaming module(s) may be configured to detect or determine one or more features or presentation aspects associated with streamed 3D content. The content streaming engine310may be stored at and/or executed by a user device or one or more remote servers. The content streaming engine310may include one or more modules or algorithms, and may be configured to output content presentation data370, which may include content stream data and/or identifiers of portions of content for presentation at a device, 3D content stream data, metadata, and/or other content presentation related data. The content streaming engine310and an optional augmented reality engine380may be used in conjunction with each other to generate augmented reality modifications to content streams.

For example, the content streaming engine310may include one or more content processing module(s)320, one or more view consistency module(s)330, and/or one or more streaming device data module(s)340. Additional or fewer, or different, modules may be included. The content processing module(s)320may be configured to process and/or analyze streamed content, including audio content and video content. For example, the content processing module(s)320may be configured to communicate with 3D camera systems, and to provide content streams of video and audio captured at the 3D camera systems to one or more streaming devices. The content processing module(s)320may be configured to determine portions of content streams that correspond to different views and/or orientations, as well as portions of content streams that correspond to streaming device positioning.

The view consistency module(s)330may be configured to maintain consistency of views across a number of different 3D camera systems and corresponding content streams. For example, the view consistency module(s)330may be configured to track states and/or function as a state machine to determine a current state of view of a content stream that is presented from a first 3D camera system, and to determine a corresponding view for a second content stream from a second 3D camera system. The view consistency module(s)330may use device positioning data to determine a state of a device and/or positioning in a 3D virtual space. In some embodiments, the view consistency module(s)330may be configured to receive device position data from the streaming device data module(s)340.

The streaming device data module(s)340may be configured to determine various data associated with a particular streaming device, such as present network conditions, device positioning data, presently viewed portions of content streams, and the like. In some embodiments, the streaming device data module(s)340may be configured to ingest external data, such as streaming device position data360that may be received as feedback from one or more streaming devices390at which content streams are being presented. The streaming device data module(s)340may be configured to analyze and/or assess local network conditions. The streaming device data module(s)340may be configured to determine available bandwidth at a given point in time, as well as bandwidth consumed by individual content streams and/or bandwidth allocated to individual content streams. In some embodiments, the streaming device data module(s)340may be configured to determine an incremental amount by which a bitrate for a particular content stream can be increased, considering other factors such as bandwidth allocated to other content streams, bandwidth available to the device, and/or other factors.

The content streaming engine310may receive one or more inputs for which content presentation data370is to be generated. For example, the content streaming engine310may receive streaming data350from individual 3D camera systems. The streaming data350may include one or more content streams from individual 3D camera systems, and may include additional data, such as metadata, camera identifiers, and/or other data. The content streaming engine310may optionally receive resolution data and bitrate data associated with individual content streams.

The content streaming engine310may process the respective data associated with the content streams. For example, the streaming data350may be processed using one or more of the content processing module(s)320, the view consistency module(s)330, and/or the streaming device data module(s)340.

Using one or more algorithms or modules, the content streaming engine310may generate the content presentation data370based at least in part on the streaming data350. The content presentation data370may be used to render particular video content at a display of a user device. The content presentation data370may be in the form of a content stream, and/or may include markers or pointers identifying portions of content streams that are to be presented. As updated data is received by the content streaming engine310, updated content presentation data370may be output.

The content presentation data370may optionally be input at an augmented reality engine380. The augmented reality engine380may be configured to generate graphical content overlays and/or other augmented reality content for presentation with content streams. For example, the augmented reality engine380may receive watch party data382that may indicate a group of users that are watching a content stream together and/or at the same time. The watch party data382may include graphical avatar data for the users in the watch party, where the graphical avatar data can be used to generate graphical representations of the users in an ambient environment of a 3D camera system, so as to provide a virtual hangout functionality, as discussed in more detail with respect toFIG.5. The augmented reality engine380may be configured to generate overlays at predetermined positions and/or at random positions that show the graphical avatars as part of a content stream.

The augmented reality engine380may ingest the content presentation data370and may cause delivery of a modified content stream and/or instructions to a rendered at one or more streaming devices390. For example, content streams may be delivered for presentation at a first streaming device, a second streaming device, a third streaming device, and so forth. Feedback from the respective devices, such as positioning feedback, may be sent as a signal back to the streaming server and used to determine portions of a content stream that are to be presented at the streaming device.

FIG.4is a schematic illustration400of example synchronization and presentation of multiple 3D content streams in accordance with one or more example embodiments of the disclosure. InFIG.4, content streams are represented by individual frames. The content may be any suitable livestream content. Other embodiments may have a different camera system arrangement than that illustrated inFIG.4. The operations described with respect toFIG.4may be performed in any suitable order across any number of devices, and may include operations that are performed at least partially concurrently.

In the example ofFIG.4, a set of camera systems may be positioned at different locations at a venue, so as to livestream an event, such as a concert. A first camera system410may be positioned to the left of a stage, a second camera system440may be positioned in front of the stage, and a third camera system470may be positioned to the right of the stage. The camera systems may be 3D camera systems in that the camera systems may be configured to provide 360 degree views from their respective positions.

Users viewing the event virtually may stream content streams from any of the camera systems in any of the available views. In some instances, users may switch between the different content streams from the different camera systems. In such instances, consistency between the initial view at the new camera system and the immediate prior view from the previous camera system may provide an improved user experience, as the user may not have to readjust positioning manually. For example, if the user is streaming a first view420from the first camera system410of the stage, and switches to the second camera system440, the initial view presented from the second camera system440may be a second view450that includes the stage. In this manner, the user may view the same focus of the content stream (e.g., where the focus is the onstage act, etc.) across the different camera systems. Similarly, if the user then switches to the third camera system470, the initial view presented from the third camera system470may be a third view480of the stage. Accordingly, although the particular views are different in that different camera orientations from the individual camera systems are used to provide the views, the user may experience a consistent view across changes in camera systems, and may not have to manually adjust positioning when switching between the different camera systems.

In some embodiments, a streaming server may determine a position of a user device used to stream content during presentation of a first content stream from a first camera system, and may determine a portion of a second content stream from a different camera system that corresponds to the first position. The streaming server may cause presentation of a portion of the second content stream from the different camera system that corresponds to the first position.

In some embodiments, anchor points may be used in a virtual 3D space to maintain a streaming device position, such that the user can manipulate positioning consistently across a number of camera systems without having to turn full circles, etc. Accordingly, in some embodiments, a user device and/or a streaming server may associate a first position of a user device with a first portion of a first content stream from a first camera system and a second portion of a second content stream from a second camera system as an anchor point, such that rotation from the user device results in different changes to viewpoints at the different camera systems. In another example, a user device and/or streaming server may determine a first physical position of the user device at a time a request to change camera systems is sent by the user device, and may initially present, or cause presentation of, a portion of a 360 degree video captured by the second three-dimensional camera that corresponds to the physical position when the second live content stream is initially received by the user device. As a result, a first orientation of a subject in the first portion of the first content stream relative to a first camera may be different than a second orientation of the subject in the third portion of the second content stream relative to a second camera. This can be seen in the example ofFIG.4, where the stage is at different positions relative to the different camera systems.

In some embodiments, a focal point of a content stream may be used to provide consistent viewing experiences across different camera systems. For example, inFIG.4, a first focal point430of the first view420may be determined to be a lead singer. The focal point may be determined based at least in part on computer vision processing of the content stream, user input, and/or other signals. Accordingly, if the user switches to the second camera system440, the initially presented view may be of the lead singer, such that the first focal point430is maintained across both the first camera system410and the second camera system440. If the user again switches to the third camera system470, the first focal point430may again be initially presented. In one example, a streaming server may determine a subject or focal point430of a first portion of a first content stream, and may determine a second portion of a second content stream that includes the subject or focal point430, and may initially cause the user device to present the second portion when switching from the first content stream to the second content stream of the respective camera systems. However, if the user rotates 180 degrees, while at the second camera system440, such that a crowd in in a fourth view460, the crowd may be the focal point if the user switches to the third camera system470, and a view of the crowd may be initially presented.

In some embodiments, a user device and/or streaming server may determine a first portion of a 360 degree video captured by the first three-dimensional camera410at a time a first request is sent by the user device, and may determine a subject of the first portion. The user device and/or streaming server may determine a second portion of the 360 degree video captured by the second three-dimensional camera440that corresponds to the first portion, where the subject is present in the second portion, and may initially present the second portion of the 360 degree video captured by the second three-dimensional camera, and may associate a physical position of the user device in a three-dimensional ambient environment of the user device with the respective portions of the content streams from the different camera systems.

FIG.5is a schematic illustration of an example use case500of synchronization and presentation of multiple 3D content streams with graphical avatars in accordance with one or more example embodiments of the disclosure. InFIG.5, content streams are represented by individual frames. The content may be any suitable livestream content. Other embodiments may have a different camera system arrangement than that illustrated inFIG.4. The operations of the process flow inFIG.5may be performed in any suitable order across any number of devices, and may include operations that are performed at least partially concurrently.

InFIG.5, users may view content streams together virtually as part of a watch party. For example, users may join a watch party together and may view the same content stream on different devices. In the example of a concert, such as that ofFIG.4, the users may be viewing the crowd510. To provide an immersive viewing experience, one or more remote servers may be configured to cause presentation of a graphical avatar at one or more user devices during presentation of a content stream. For example, a watch party may include User A and User B. User A may be associated with a first graphical avatar520, and User B may be associated with a second graphical avatar530. The first graphical avatar520and the second graphical avatar530may be presented as digital overlays with the crowd to provide an augmented reality experience. The first graphical avatar520and the second graphical avatar530may be presented at predetermined locations, or may be dynamically positioned based on the view presented in a content stream.

In some embodiments, the streaming server may be configured to execute a process flow540to present graphical avatars. At a first block550, the streaming server may determine a current camera view for a watch party, such as a view of the crowd at a concert. At a second block560, the streaming server may determine digital representations associated with users in the watch party, such as the first graphical avatar520and the second graphical avatar530. At a third block570, the streaming server may determine respective locations for the digital representations. The locations may be predetermined, such as particular seat numbers or marked locations, or may determined dynamically using machine learning and/or computer vision processing. At a fourth block580, the streaming server may generate a content stream that includes the camera data and the digital representations, such as the frame510presented in the example ofFIG.5.

FIG.6is a schematic illustration of an example use case600of two-way immersive live streaming in accordance with one or more example embodiments of the disclosure. InFIG.6, content streams are represented by individual frames. The content may be a live event. The illustration ofFIG.6is provided for illustrative purposes only.

In some embodiments, a user identifiers of users that are consuming a livestream of content610may be presented to performers or other individuals that are in the vicinity of a 3D camera system. The performers or other individuals may therefore engage with the users that are virtually present, such as by saying the user identifier or name of the user. For example, the users that are consuming a livestream of content610may be users currently viewing a content streaming from a first 3D camera system at a basketball game. At a first instance620, a basketball player may see the users that are consuming the livestream, and may shout out a username, such as by saying “User3this one's for you” during a slam dunk. Accordingly, the users viewing the livestream may feel immersed and engaged in the viewing experience. Users may switch to another 3D camera system that is courtside, as shown at a second point in time630, that may allow for up close interaction with performers during pregame workouts or other times. The streaming server and/or another computer system may therefore cause presentation of an identifier associated with a user device (and/or a user account using a user device) at a display positioned at a physical location of a first content stream while the user device is receiving the first content stream or the second content stream, and the performer or another user may engage with those in virtual attendance directly.

Illustrative Device Architecture

FIG.7is a schematic block diagram of an illustrative remote server700in accordance with one or more example embodiments of the disclosure. The remote server700may include any suitable computing device capable of receiving and/or sending data including, but not limited to, a mobile device such as a smartphone, tablet, e-reader, wearable device, or the like; a desktop computer; a laptop computer; a content streaming device; a set-top box; or the like. The remote server700may correspond to an illustrative device configuration for the remote server and/or streaming devices ofFIGS.1-6.

The remote server700may be configured to communicate via one or more networks with one or more servers, search engines, user devices, or the like. In some embodiments, a single remote server or single group of remote servers may be configured to perform more than one type of content streaming-related determinations and/or machine learning functionality.

In an illustrative configuration, the remote server700may include one or more processors (processor(s))702, one or more memory devices704(generically referred to herein as memory704), one or more input/output (I/O) interface(s)706, one or more network interface(s)708, one or more sensors or sensor interface(s)710, one or more transceivers712, one or more optional speakers714, one or more optional microphones716, and data storage720. The remote server700may further include one or more buses718that functionally couple various components of the remote server700. The remote server700may further include one or more antenna(s)734that may include, without limitation, a cellular antenna for transmitting or receiving signals to/from a cellular network infrastructure, an antenna for transmitting or receiving Wi-Fi signals to/from an access point (AP), a Global Navigation Satellite System (GNSS) antenna for receiving GNSS signals from a GNSS satellite, a Bluetooth antenna for transmitting or receiving Bluetooth signals, a Near Field Communication (NFC) antenna for transmitting or receiving NFC signals, and so forth. These various components will be described in more detail hereinafter.

The data storage720may include removable storage and/or non-removable storage including, but not limited to, magnetic storage, optical disk storage, and/or tape storage. The data storage720may provide non-volatile storage of computer-executable instructions and other data. The memory704and the data storage720, removable and/or non-removable, are examples of computer-readable storage media (CRSM) as that term is used herein.

The data storage720may store computer-executable code, instructions, or the like that may be loadable into the memory704and executable by the processor(s)702to cause the processor(s)702to perform or initiate various operations. The data storage720may additionally store data that may be copied to memory704for use by the processor(s)702during the execution of the computer-executable instructions. Moreover, output data generated as a result of execution of the computer-executable instructions by the processor(s)702may be stored initially in memory704, and may ultimately be copied to data storage720for non-volatile storage.

More specifically, the data storage720may store one or more operating systems (O/S)722; one or more database management systems (DBMS)724; and one or more program module(s), applications, engines, computer-executable code, scripts, or the like such as, for example, one or more optional machine learning module(s)726, one or more communication module(s)728, one or more content streaming engine/module(s)730, and/or one or more augmented reality engine/module(s)732. Some or all of these module(s) may be sub-module(s). Any of the components depicted as being stored in data storage720may include any combination of software, firmware, and/or hardware. The software and/or firmware may include computer-executable code, instructions, or the like that may be loaded into the memory704for execution by one or more of the processor(s)702. Any of the components depicted as being stored in data storage720may support functionality described in reference to correspondingly named components earlier in this disclosure.

The data storage720may further store various types of data utilized by components of the remote server700. Any data stored in the data storage720may be loaded into the memory704for use by the processor(s)702in executing computer-executable code. In addition, any data depicted as being stored in the data storage720may potentially be stored in one or more datastore(s) and may be accessed via the DBMS724and loaded in the memory704for use by the processor(s)702in executing computer-executable code. The datastore(s) may include, but are not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed datastores in which data is stored on more than one node of a computer network, peer-to-peer network datastores, or the like. InFIG.7, the datastore(s) may include, for example, user preference information, active streaming data, available bandwidth data, historical network performance information, and other information.

The processor(s)702may be configured to access the memory704and execute computer-executable instructions loaded therein. For example, the processor(s)702may be configured to execute computer-executable instructions of the various program module(s), applications, engines, or the like of the remote server700to cause or facilitate various operations to be performed in accordance with one or more embodiments of the disclosure. The processor(s)702may include any suitable processing unit capable of accepting data as input, processing the input data in accordance with stored computer-executable instructions, and generating output data. The processor(s)702may include any type of suitable processing unit including, but not limited to, a central processing unit, a microprocessor, a Reduced Instruction Set Computer (RISC) microprocessor, a Complex Instruction Set Computer (CISC) microprocessor, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a System-on-a-Chip (SoC), a digital signal processor (DSP), and so forth. Further, the processor(s)702may have any suitable microarchitecture design that includes any number of constituent components such as, for example, registers, multiplexers, arithmetic logic units, cache controllers for controlling read/write operations to cache memory, branch predictors, or the like. The microarchitecture design of the processor(s)702may be capable of supporting any of a variety of instruction sets.

Referring now to functionality supported by the various program module(s) depicted inFIG.7, the optional machine learning module(s)726may include computer-executable instructions, code, or the like that responsive to execution by one or more of the processor(s)702may perform functions including, but not limited to, determining approval of requests to improve bitrate, determining rejection of requests to improve bitrate, determining current streaming performance values, determining bandwidth usage, determining or detecting actions and/events, generating one or more machine learning models or algorithms, determining or classifying objects or actions, determining frames of content, and the like.

The communication module(s)728may include computer-executable instructions, code, or the like that responsive to execution by one or more of the processor(s)702may perform functions including, but not limited to, communicating with one or more devices, for example, via wired or wireless communication, communicating with remote servers, communicating with remote datastores, sending or receiving notifications or commands/directives, communicating with cache memory data, communicating with user devices, and the like.

The content streaming engine/module(s)730may include computer-executable instructions, code, or the like that responsive to execution by one or more of the processor(s)702may perform functions including, but not limited to, analyzing digital content, detecting servers and/or communicating with egress systems, determining streaming content quality values, determining streaming content, determining or analyzing audio files, identifying certain portions of content, extracting segments of content, generating video files, generating 3D content streams, and the like.

The augmented reality engine/module(s)732may include computer-executable instructions, code, or the like that responsive to execution by one or more of the processor(s)702may perform functions including, but not limited to, determining video files, generating digital representations and/or graphical content overlays, determining three dimensional positioning and/or datum positioning, determining user device movements, determining content screen resolution, and the like.

Referring now to other illustrative components depicted as being stored in the data storage720, the O/S722may be loaded from the data storage720into the memory704and may provide an interface between other application software executing on the remote server700and hardware resources of the remote server700. More specifically, the O/S722may include a set of computer-executable instructions for managing hardware resources of the remote server700and for providing common services to other application programs (e.g., managing memory allocation among various application programs). In certain example embodiments, the O/S722may control execution of the other program module(s) to for content rendering. The O/S722may include any operating system now known or which may be developed in the future including, but not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

The DBMS724may be loaded into the memory704and may support functionality for accessing, retrieving, storing, and/or manipulating data stored in the memory704and/or data stored in the data storage720. The DBMS724may use any of a variety of database models (e.g., relational model, object model, etc.) and may support any of a variety of query languages. The DBMS724may access data represented in one or more data schemas and stored in any suitable data repository including, but not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed datastores in which data is stored on more than one node of a computer network, peer-to-peer network datastores, or the like. In those example embodiments in which the remote server700is a mobile device, the DBMS724may be any suitable light-weight DBMS optimized for performance on a mobile device.

Referring now to other illustrative components of the remote server700, the input/output (I/O) interface(s)706may facilitate the receipt of input information by the remote server700from one or more I/O devices as well as the output of information from the remote server700to the one or more I/O devices. The I/O devices may include any of a variety of components such as a display or display screen having a touch surface or touchscreen; an audio output device for producing sound, such as a speaker; an audio capture device, such as a microphone; an image and/or video capture device, such as a camera; a haptic unit; and so forth. Any of these components may be integrated into the remote server700or may be separate. The I/O devices may further include, for example, any number of peripheral devices such as data storage devices, printing devices, and so forth.

The remote server700may further include one or more network interface(s)708via which the remote server700may communicate with any of a variety of other systems, platforms, networks, devices, and so forth. The network interface(s)708may enable communication, for example, with one or more wireless routers, one or more host servers, one or more web servers, and the like via one or more of networks.

The antenna(s)734may include any suitable type of antenna depending, for example, on the communications protocols used to transmit or receive signals via the antenna(s)734. Non-limiting examples of suitable antennas may include directional antennas, non-directional antennas, dipole antennas, folded dipole antennas, patch antennas, multiple-input multiple-output (MIMO) antennas, or the like. The antenna(s)734may be communicatively coupled to one or more transceivers712or radio components to which or from which signals may be transmitted or received.

The antenna(s)734may additionally, or alternatively, include a Wi-Fi antenna configured to transmit or receive signals in accordance with established standards and protocols, such as the IEEE 802.11 family of standards, including via 2.4 GHz channels (e.g., 802.11b, 802.11g, 802.11n), 5 GHz channels (e.g., 802.11n, 802.11ac), or 60 GHz channels (e.g., 802.11ad). In alternative example embodiments, the antenna(s)734may be configured to transmit or receive radio frequency signals within any suitable frequency range forming part of the unlicensed portion of the radio spectrum.

The transceiver(s)712may include any suitable radio component(s) for—in cooperation with the antenna(s)734—transmitting or receiving radio frequency (RF) signals in the bandwidth and/or channels corresponding to the communications protocols utilized by the remote server700to communicate with other devices. The transceiver(s)712may include hardware, software, and/or firmware for modulating, transmitting, or receiving—potentially in cooperation with any of antenna(s)734— communications signals according to any of the communications protocols discussed above including, but not limited to, one or more Wi-Fi and/or Wi-Fi direct protocols, as standardized by the IEEE 802.11 standards, one or more non-Wi-Fi protocols, or one or more cellular communications protocols or standards. The transceiver(s)712may further include hardware, firmware, or software for receiving GNSS signals. The transceiver(s)712may include any known receiver and baseband suitable for communicating via the communications protocols utilized by the remote server700. The transceiver(s)712may further include a low noise amplifier (LNA), additional signal amplifiers, an analog-to-digital (A/D) converter, one or more buffers, a digital baseband, or the like.

The optional speaker(s)714may be any device configured to generate audible sound. The optional microphone(s)716may be any device configured to receive analog sound input or voice data.