Facilitating panoramic video streaming with brain-computer interactions

Aspects of the subject disclosure may include, for example, obtaining one or more signals, the one or more signals being based upon brain activity of a viewer while the viewer is viewing media content; predicting, based upon the one or more signals, a first predicted desired viewport of the viewer; obtaining head movement data associated with the media content; predicting, based upon the head movement data, a second predicted desired viewport of the viewer; comparing the first predicted desired viewport to the second predicted desired viewport, resulting in a comparison; and determining, based upon the comparison, to use the first predicted desired viewport to facilitate obtaining a first subsequent portion of the media content or to use the second predicted desired viewport to facilitate obtaining a second subsequent portion of the media content. Other embodiments are disclosed.

FIELD OF THE DISCLOSURE

The subject disclosure relates to facilitating panoramic video streaming with brain-computer interactions.

BACKGROUND

Panoramic or immersive video (each of which can include, for example, 360-degree video), is a critical component in the Virtual Reality (“VR”) ecosystem. Such 360-degree video (sometimes referred to herein as 360° video) is becoming increasingly popular on commercial video content platforms. In a typical 360-degree video system, a user wearing a VR headset can freely change his or her viewing direction. Technically, the user is situated in the center of a virtual sphere, and the panoramic content downloaded from video server(s) is projected onto the sphere (e.g., using equirectangular projection). The user's viewport (visible area) is typically determined by his or her viewing direction (in latitude/longitude) and the Field of View (“FoV”) of the VR headset in real-time. The FoV defines the extent of the observable area, which is usually a fixed parameter of a VR headset. As shown inFIG. 2A, a user wearing a VR headset201can adjust his or her orientation by changing the pitch, yaw, and roll, which correspond to rotating along the X, Y, and Z axes, respectively (see, also, in this FIG. example viewport203).

Maintaining good Quality of Experience (“QoE”) for 360° videos over bandwidth-limited links on commodity mobile devices remains challenging. First, 360° videos are large: under the same perceived quality, 360° videos have around 5× larger sizes than conventional videos. Second, 360° videos are complex: sophisticated projection and content representation schemes may incur high overhead. Third, 360° videos are still under-explored: there is a lack of real-world experimental studies of key aspects such as rate adaptation, QoE metrics, and cross-layer interactions (e.g., with TCP and web protocols such as HTTP/2).

Certain existing work on 360° video streaming can be divided into two categories: monolithic streaming and tile-based streaming. A simply monolithic streaming delivers uniformly encoded panoramic views and is widely used by most commercial 360° video content providers. The drawback is that, at any given time, typically only 15-20% of the content that is being downloaded is actually in the FoV of the end user, which is a waste of bandwidth resources. For more advanced schemes that perform viewport adaptation, a 360° video has multiple versions each having a different scene region, sometimes called a Quality Emphasized Region (“QER”), with a high encoding rate. A video player picks the correct version based on the viewer's head orientation. This scheme is sometimes referred to as versioning-based 360° video streaming. One practical issue of this versioning-based 360° video streaming scheme is that it incurs significant overhead at the server side (e.g., the FACEBOOK OCULUS 360 mechanism is believed to require servers to maintain up to 88 versions of the same video).

For the tiling scheme, a 360° video is spatially segmented into tiles. Delivered are mainly tiles overlapping with predicted FoVs for viewport-adaptive video streaming. To increase the robustness, a video player can also fetch the remaining tiles at lower qualities. Each 360° video chunk is pre-segmented into multiple smaller chunks, which are called tiles. The easiest way to generate the tiles is to evenly divide a chunk containing projected raw frames into m×n rectangles each corresponding to a tile. Suppose, for example, that the projected visible area is ⊖. In this example, the client (e.g., video player) only sends requests for the tiles that overlap with ⊖.

Referring now toFIG. 2B, an example is shown where m=6 and n=4, and ⊖ is the shaded oval region210. An original video chunk is segmented into tiles. A tile (see, e.g., the tile in the upper right-hand corner) has the same duration and number of frames as the chunk it belongs to, but occupies only a small spatial portion. Each tile can be independently downloaded and decoded. A tile can also refer to a small spatial portion only in a frame. In that sense, a tile-based video chunk can be independently fetched and decoded. In this example, the client will only request the six tiles overlapping with ⊖ (that is, where 1≤x≤3, 1≤y≤2). Note that due to projection, despite the viewer's FoV being fixed, the size of ⊖ and thus the number of requested tiles may vary. Compared to FoV-agnostic approaches, tiling offers significant bandwidth savings. Also note that the tiling scheme can be applied to not only videos using Equirectangular projection, but also those with Cube Map projection.

Certain proposals have previously been made to improve the accuracy of viewport prediction by leveraging data fusion of multiple sources, such as head movement, video content analysis and user profile. Popular 360° videos from commercial content providers and video-sharing websites attract a large number of viewers (e.g., more than 4 million views of the video represented byFIG. 2B). Also, it is known that users' viewing behaviors are often affected by the video content. It is believed that this is also true for 360° videos: at certain scenes, viewers are more likely to look at certain spots or directions, and thus the FoV can be predicted based on the video content. Consider an example of a mountain climbing video. When viewers are “climbing” towards the peak, they may look upward most of the time to figure out how long it will take to reach the peak.

Based on the above observation, there have been proposals to use crowdsourced viewing statistics by instrumenting the 360° video players to record the frequency of a given FoV, which can be easily be collected by video servers. With the crowdsources data, a heat map can be generated showing the most frequently viewed content in a 360° video. In the literature, viewing statistics have been leveraged to estimate the video abandonment rate and to automatically rate video contents. In the context of 360° videos, for each chunk, the server can also record download frequencies of its tiles, and provide client video players with such statistics as a heat map of content access pattern through metadata exchange. A tile's download frequency is defined in this context as the number of video sessions that fetch this tile divided by the total number of sessions accessing this video.

Besides the heat map based approach, certain proposals have previously been made to employ object-feature detection for certain types of videos. For example, for soccer and tennis videos, these objects could be the soccer and tennis balls, key soccer players and referee. When watching these sport videos, most likely the viewers will follow the movement of the soccer and tennis balls. Even without using the heat map, it can be predicted that the tiles containing the ball will very likely overlap with the FoV and these tiles can be identified via object-feature detection of video frames.

Moreover, certain existing work has demonstrated that it is possible to model the video viewing behavior of users by leveraging stochastic models such as a Markovian model. The model can be constructed using actions from a user when viewing a 360° video, including pause, stop, jump, forward and rewind. This type of user profile complements the head movement based FoV prediction. Even if a user does not change the view direction, the FoV may change dramatically if a forward/rewind action is issued by the viewer. The stochastic models of video viewing behavior can help improve the accuracy of FoV prediction. The future FoV prediction can also leverage the personal interest of a user. For example, if it is known from a profile that a user does not like thrilling scenes, very likely he/she will skip this type of content when watching a 360° video. Thus, the probability of predicting a FoV from these scenes will be low.

Reference will now be made to certain aspects of conventional Brain-Computer Interfaces for VR. According to Wikipedia: A brain-computer interface (BCI), sometimes called a neural-control interface (NCI), mind-machine interface (MMI), direct neural interface (DNI), or brain-machine interface (BMI), is a direct communication pathway between an enhanced or wired brain and an external device. Certain existing BCI mechanisms can be divided into three categories: invasive BCIs that are directly implanted into the grey matter of the brain; partially invasive BCI devices which are implanted inside the skull with the rest outside the brain and thus the grey matter; and non-invasive BCIs. The most widely used non-invasive BCIs leverage electroencephalography (“EEG”), mainly due to its portability, ease of use, fine temporal resolution and low set-up cost. However, it is somewhat susceptible to noise. Other technologies that have been used successfully for non-invasive BCIs include magnetoencephalography (“MEG”) and functional Magnetic Resonance Imaging (“fMRI”).

A number of prototypes have been proposed to enable users to navigate in virtual environments [see, e.g., Doron Friedman, Robert Leeb, Christoph Guger, Anthony Steed, Gert Pfurtscheller and Mel Slater. Navigating Virtual Reality by Thought: What Is It Like?Presence, Vol. 16, No. 1, pages 100-110, 2007] and manipulate virtual objects [see, e.g., Anatole Lécuyer, Fabien Lotte, Richard B. Reilly, Robert Leeb, Michitaka Hirose and Mel Slater. Brain-Computer Interfaces, Virtual Reality, and Videogames.Computer, Vol. 41, No. 10, pages 66-72, 2008] solely by BCIs, for example, by analyzing cerebral activity which is recorded on the scalp via EEG electrodes. In terms of conventional use in facilitating panoramic video viewing experiences, BCIs can be decomposed into several elementary tasks, such as moving left/right and up/down, in order to change the viewport when viewing 360° videos. It has actually been shown by Pfurtscheller and Neuper [see, e.g., Gert Pfurtscheller and Christa Neuper. Motor Imagery and Direct Brain-Computer Communication. Proceedings of IEEE 82(7), pages 1123-1134] that one can identify from EEG signals several mental processes, for example, the imagination of various predefined movements. One can then transform such thought-related EEG signals into a control signal, which can in turn be associated with a few simple computer commands, such as cursor movement.

Referring now toFIG. 2C, a diagram is depicted that shows how certain conventional BCIs can facilitate various VR applications (including navigation in the context of panoramic 360° video streaming). Essentially, there is a closed loop with four steps. First, the EEG device222collects the “thoughts” from a viewer through the BCI device224attached on the head of the viewer. Second, the EEG analyzer226processes the signals and transforms them into navigation commands, such as (for example) moving toward left. Third, the EEG analyzer226sends the instructions to the VR display device228(which could be, for example, combined with the BCI device224into a single unit). Finally, the VR device228changes the viewport based on the instruction from the EEG analyzer226.

DETAILED DESCRIPTION

The subject disclosure describes, among other things, illustrative embodiments for facilitating panoramic video streaming with brain-computer interactions. Other embodiments are described in the subject disclosure.

Various embodiments facilitate and optimize the delivery of 360° videos by leveraging brain-computer interactions (e.g., to improve the accuracy of head movement based viewport prediction). As described herein, certain conventional viewport prediction techniques have enabled viewport adaptive streaming of 360° videos (which delivers only predicted viewport of a frame at high quality and the remainder of the frame with a low quality). Such certain conventional work mainly utilizes viewport movement traces (e.g., by collecting the head movement traces when viewing 360° videos with a head-mounted display) for the prediction. Although this approach is lightweight and can achieve a reasonable accuracy in general, it is typically limited by the inherent “randomness” of the movement trajectory. Further, viewport changes are naturally controlled by the brain of the viewer and certain conventional Brain-Computer Interfaces (“BCIs”) have been extensively investigated to facilitate various aspects of human-computer interactions. On one hand, it has been demonstrated that BCI could potentially provide more intuitive and suitable interactions for VR applications. On the other hand, the research community widely accepts that VR could be a promising medium for efficiently improving BCI systems. Various embodiments described herein combine the predictions of future viewport from head movement traces and viewport moving direction derived from the analysis of brain signals. If the two predictions are consistent, then (in one embodiment) the fine-granularity prediction from head movement traces will be used to actively and adaptively prefetch video content in the predicted viewport in advance. Otherwise (that is, if the two predictions are not consistent), the viewport video prefetching will (in one embodiment) be guided (e.g., largely guided) by the prediction of viewport moving direction from brain signals (which, it is believed, should be more accurate than the head movement based viewport prediction).

As described herein, an essential component in certain viewport-adaptive 360° video players is to predict users' future viewports (e.g., via head movement prediction), which is important for both tiling-based and versioning-based viewport-adaptive 360° video streaming. Provided herein is a discussion directed to a systematic investigation that was made of real users' head movements, as well as how to efficiently perform viewport prediction on mobile devices. Using a dataset consisting of 4420-minute 360° video playback time, studied were a wide spectrum of machine learning (“M”L) algorithms for viewport prediction. Also, designed were lightweight but robust viewport prediction methods by strategically leveraging off-the-shelf ML algorithms.

With reference now to viewport prediction accuracy, it is noted that ideally, if a viewer's future FoV during a 360° video session is known beforehand, the optimal sequence of tiles that minimizes the bandwidth consumption can be generated. By leveraging head movement traces, for example, a sliding window of 1 second from T−1 to T can be used to predict future head position (and thus the FoV) at T+δ for each dimension of yaw, pitch, and roll. Evaluated were the prediction accuracy of various Machine Learning algorithms for three prediction windows, 0.2, 0.5 and 1 s (seeFIGS. 2D, 2E, 2F).

Still referring toFIGS. 2D, 2E, 2F, the training data is historical head movement traces collected during the user study mentioned above with more than 130 participants. Used were four Machine Learning algorithms—3 classical models and 1 neural network model. The classical models are Linear Regression, Ridge Regression and Support Vector Regression (with rbf kernel). The neural network model is Multi-Layer Perceptron. Also used was a simple heuristic, called Static, which assumes that the viewport does not change from T to T+δ. For the 4×6 segmentation scheme, the viewport prediction is accurate if the tile set determined by the predicted viewport is exactly the same as the ground truth. The key take-away fromFIGS. 2D, 2E, 2Fis that the viewport prediction accuracy depends heavily on the prediction window. The longer this window is, the lower the prediction accuracy. However, smaller prediction windows lead to a strict requirement on the end-to-end latency.

As described herein in connection with various embodiments, if it can be determined in advance how a viewer is going to change the viewport (for example, how the viewer is going to move his or her head) when watching a 360° video then this information can be utilized to improve the accuracy of viewport prediction. In one example, the future head movement can be predicted by analyzing brain signals/waves of the viewer.

Referring now toFIG. 2G, a diagram (according to an embodiment) of workflow that leverages BCIs for improving the accuracy of viewport prediction for 360° video streaming is shown. In this FIG. the workflow includes step232, which is collecting and processing the brain signals of a viewer when he or she watches a 360° video. By analyzing these signals (see step234), it can be determined roughly to which direction the viewer wants to change his or her viewport. The workflow also includes step236, which is collecting viewport movement trajectory, for example, from head movement traces collected by motion sensors. At step238one or more machine learning technologies are applied to predict one or more future viewports based on the collected movement traces. At this point, there are now two sources of prediction for future viewports (one source based on BCI and another source based on head movement data). At step240the two viewport predictions from the two sources are compared (and it is determined whether the two viewport predictions are consistent with each other).

Still referring toFIG. 2G, if the moving direction analyzed from the BCI signals aligns with the future viewport predicted using machine learning algorithms (e.g., linear regression), then the workflow prefetches video content in the machine learning predicted (consistent) viewport which usually can provide fine-granularity information (see step244as a result of “YES” from step240). Otherwise (see step242as a result of “NO” from step240), BCI-based prediction will be used to guide the content prefetch (which, in theory, should be more accurate than machine learning based prediction). During the user study described herein, it was found that some viewers will suddenly and dramatically change the viewing direction, for example, when attracted by a loud sound from the left while moving their heads toward the right. In this case, viewport prediction by applying machine learning algorithms on head movement traces will become inaccurate, because the predicted trend is no longer valid any more. On the other hand, the BCI-based solution of various embodiments described herein can offer more accurate prediction by analyzing what the viewer's “thoughts” are. Of note, various embodiments described herein can be agnostic to the underlying technology to support BCIs (which can be, for example, EEG, MEG, and/or fMRI). In another example, an underlying technology to support BCIs can be any technology that enables the understanding of the mapping between brain signals and the predefined navigation commands.

Referring now toFIG. 2H, various steps of a method2000according to an embodiment are shown. As seen in thisFIG. 2H, step2002comprises obtaining, by a system including a processor, one or more signals, the one or more signals being based upon brain activity of a viewer while the viewer is viewing media content. Next, step2004comprises predicting by the system, based upon the one or more signals, a first predicted desired viewport of the viewer. Next, step2006comprises obtaining, by the system, head movement data associated with the media content. Next, step2008comprises predicting by the system, based upon the head movement data, a second predicted desired viewport of the viewer. Next, step2010comprises comparing, by the system, the first predicted desired viewport to the second predicted desired viewport, resulting in a comparison. Next, step2012comprises determining by the system, based upon the comparison, to use the first predicted desired viewport to facilitate obtaining a first subsequent portion of the media content or to use the second predicted desired viewport to facilitate obtaining a second subsequent portion of the media content.

Referring now toFIG. 2I, various steps of a method3000according to an embodiment are shown. As seen in thisFIG. 2I, step3002comprises obtaining from one or more sensors brain activity data that is based upon brain activity of a viewer, the brain activity data being associated with viewing by the viewer of media content. Next, step3004comprises predicting a first predicted desired viewport of the viewer, the first predicted desired viewport being predicted based upon the brain activity data. Next, step3006comprises obtaining head movement data associated with the media content. Next, step3008comprises predicting a second predicted desired viewport of the viewer, the second predicted desired viewport being based upon the head movement data. Next, step3010comprises determining, based upon a comparison of the first predicted desired viewport to the second predicted desired viewport, whether to use the first predicted desired viewport to facilitate obtaining a first subsequent portion of the media content or to use the second predicted desired viewport to facilitate obtaining a second subsequent portion of the media content.

Referring now toFIG. 2J, various steps of a method4000according to an embodiment are shown. As seen in thisFIG. 2J, step4002comprises receiving from a device a request for a portion of media content, the request indicating a desired viewport, the request having been made by the device in accordance with a determination by the device to use as the desired viewport one of a first predicted desired viewport or a second predicted desired viewport, the determination having been based upon a comparison between the first predicted desired viewport and the second predicted desired viewport, the first predicted desired viewport having been predicted by the device based upon brain activity of a viewer engaged in watching an earlier portion of the media content, and the second predicted desired viewport having been predicted by the device based upon head movement data associated with the earlier portion of the media content. Next, step4004comprises sending, to the device, the portion of media content that had been requested.

Referring now toFIG. 2K, depicted is a block diagram illustrating an example, non-limiting embodiment of a system250in accordance with various aspects described herein. As seen in this FIG. server(s)252are in bi-directional communication with media player254via the Internet256. The server(s)252store content (e.g., 360° video content) that is streamed to the media player. The server(s)252also store a database of historic head movement data associated with the stored video content. The media player254obtains (from BCI258) BCI data (e.g., real-time BCI data) associated with a viewer who is using the media player254to view a video (the BCI258can be separate from the media player254or integrated with/into the media player254). The media player254requests from the server(s)252appropriate portions of the video (e.g., appropriate tiles). The appropriate portions can be determined by the media player254using determination techniques as described herein.

In another example, the media player254sends to the server(s)252the BCI data, the server(s)252determine (based upon the BCI data and/or the historic head movement data) the appropriate portions of the video (e.g., appropriate tiles) to send back to the media player254, and the server(s)252send back to the media player254the determined appropriate portions of the video (e.g., appropriate tiles). The appropriate portions can be determined by the servers(s)252using determination techniques as described herein.

As described herein, a key basis of certain previous proposals related to viewport prediction of 360° videos is the historical head movement trajectory and the consideration of other factors that may indirectly affect the head movement, for example, video content and user profile. However, some of these schemes are typically not very accurate, especially for large prediction windows, due to the inherent randomness of human head movement when watching 360° videos. Further, how the viewport changes can actually be considered as directly determined by what content the viewer wants to consume and thus is controlled by the viewer's brain. Therefore, various embodiments described herein target an improvement of the viewport prediction accuracy (for example, in the context of panoramic video streaming) by leveraging brain-computer interactions. In one specific example, viewport prediction accuracy is improved for large prediction windows.

As described herein, in a manner differently from certain existing head-mounted displays that control the viewport mainly based on motion data from sensors, when using a BCI according to an embodiment a viewer does not need to move his or her head in order to change the content they are seeing for 360° video streaming. The viewer can navigate via his or her “thoughts”.

As described herein, a BCI can be used to navigate the viewing. In an embodiment, even if a BCI is not used to actually control the navigation, a BCI can be used to leverage the BCI signals to facilitate the viewport prediction.

As described herein, a comparison can be made between a BCI-based prediction (based on brain activity) and a traditional viewport prediction (based on head movement trajectory). If they are consistent, the fine-granularity prediction from head movement traces can be used to actively and adaptively prefetch video content in the predicted viewport in advance. Otherwise, the viewport video prefetching can be largely guided by the prediction of viewport moving direction from brain signals (which should be more accurate than the head movement based viewport prediction).

In one example, the determination that the viewport predicted by the head movement data should be used can be made based upon the viewport predicted by the head movement data corresponding to a set of tiles that match on a one-to-one basis with a set of tiles that correspond to the viewport predicted by the brain-computer interface data (wherein, in this example, if there is not a one-to-one match in tiles the viewport predicted by the brain-computer interface data would be used instead).

In another example, the determination that the viewport predicted by the head movement data should be used can be made based upon the viewport predicted by the head movement data corresponding to a set of tiles that match at least on a percent basis (the percent of this example being less than 100 percent and greater than 0 percent) with a set of tiles that correspond to the viewport predicted by the brain-computer interface data (wherein, in this example, if there is not a match above a certain threshold percentage basis then the tiles of the viewport predicted by the brain-computer interface data would be used instead).

As described herein, improvements can be provided to viewport prediction accuracy and streaming efficiency for 360° videos via brain-computer interactions. Various embodiments bring one or more of the following four key benefits. First, congestion can be alleviated in the cellular core network by delivering less data for 360° videos through more accurate predication and more efficient content caching. Second, the cellular data usage of mobile users can be optimized and the stall time of video playback can be reduced, thus improving the quality of user experience. Third, energy consumption on mobile devices can be decreased, by avoiding transmitting unnecessary data when delivering 360° videos. Finally, various embodiments are lightweight and enable true spatial immersion by delivering 4K+ quality videos over a network infrastructure with limited bandwidth.

In another example, various embodiments can be implemented in the context of any type of panoramic or immersive video (e.g. 360° video, 180° video).

Referring now toFIG. 3, a block diagram300is shown illustrating an example, non-limiting embodiment of a virtualized communication network in accordance with various aspects described herein. In particular a virtualized communication network is presented that can be used to implement some or all of the subsystems and functions of communication network100, the subsystems and functions of system220, the functions of workflow230, method2000, method300and method400presented inFIGS. 1, 2C, 2G, 2H, 2I AND 2J. For example, virtualized communication network300can facilitate in whole or in part panoramic video streaming (such as in the context of viewport prediction/selection as described herein).

Turning now toFIG. 4, there is illustrated a block diagram of a computing environment in accordance with various aspects described herein. In order to provide additional context for various embodiments of the embodiments described herein,FIG. 4and the following discussion are intended to provide a brief, general description of a suitable computing environment400in which the various embodiments of the subject disclosure can be implemented. In particular, computing environment400can be used in the implementation of network elements150,152,154,156, access terminal112, base station or access point122, switching device132, media terminal142, and/or VNEs330,332,334, etc. Each of these devices can be implemented via computer-executable instructions that can run on one or more computers, and/or in combination with other program modules and/or as a combination of hardware and software. For example, computing environment400can facilitate in whole or in part panoramic video streaming (such as in the context of viewport prediction/selection as described herein).

Turning now toFIG. 6, an illustrative embodiment of a communication device600is shown. The communication device600can serve as an illustrative embodiment of devices such as data terminals114, mobile devices124, vehicle126, display devices144or other client devices for communication via either communications network125. For example, computing device600can facilitate in whole or in part panoramic video streaming (such as in the context of viewport prediction/selection as described herein).

Some of the embodiments described herein can also employ artificial intelligence (AI) to facilitate automating one or more features described herein. The embodiments (e.g., in connection with automatically predicting and/or selecting one or more viewports for use in performing panoramic video streaming) can employ various AI-based schemes for carrying out various embodiments thereof. Moreover, the classifier can be employed to determine a ranking or priority of each cell site of the acquired network. A classifier is a function that maps an input attribute vector, x=(x1, x2, x3, x4, . . . , xn), to a confidence that the input belongs to a class, that is, f(x)=confidence (class). Such classification can employ a probabilistic and/or statistical-based analysis (e.g., factoring into the analysis utilities and costs) to determine or infer an action that a user desires to be automatically performed. A support vector machine (SVM) is an example of a classifier that can be employed. The SVM operates by finding a hypersurface in the space of possible inputs, which the hypersurface attempts to split the triggering criteria from the non-triggering events. Intuitively, this makes the classification correct for testing data that is near, but not identical to training data. Other directed and undirected model classification approaches comprise, e.g., naïve Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, and probabilistic classification models providing different patterns of independence can be employed. Classification as used herein also is inclusive of statistical regression that is utilized to develop models of priority.