# A review on machine learning based user-centric multimedia streaming techniques

Monalisa Ghosh, Chetna Singhal

<figure><img src="image_1.png" /></figure>

## To cite this version:

Monalisa Ghosh, Chetna Singhal. A review on machine learning based user-centric multimedia streaming techniques. Computer Communications, 2025, 231, pp.108011. 10.1016/j.comcom.2024.108011. hal-05449003

HAL Id: hal-05449003

https://inria.hal.science/hal-05449003v1

Submitted on 8 Jan 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# A review on Machine Learning based User-Centric Multimedia
Streaming Techniques

Monalisa Ghosh¹ and Chetna Singhal†

¹Indian Institute of Technology, Kharagpur, INDIA

†INRIA centre at University of Rennes, France

## Abstract

The multimedia content and streaming are a major means of information exchange in the modern era and there is an increasing demand for such services. This coupled with the advancement of future wireless networks B5G/6G and the proliferation of intelligent handheld mobile devices, has facilitated the availability of multimedia content to heterogeneous mobile users. Apart from the conventional video, the 360° videos have gained significant attention and are quickly emerging as the popular multimedia format for virtual reality experiences. All formats of videos (conventional and 360°) undergo processing, compression, and transmission across dynamic wireless channels with restricted bandwidth to facilitate the streaming services. This causes video impairments, leading to quality degradation and poses challenges for the content providers in delivering good Quality-of-Experience (QoE) to the viewers. The QoE is a prominent subjective measure of quality, which has become a crucial component in assessing multimedia services and operations. So, there has been a growing preference for QoE-aware multimedia services over heterogeneous networks with a need to address design issues like how to evaluate and quantify end-to-end QoE. Efficient multimedia streaming techniques can improve the service quality while dealing with dynamic network and end-user challenges. A paradigm shift in user-centric multimedia services is envisioned with a focus on Machine Learning (ML) based QoE modeling and streaming strategies. This survey paper presents a comprehensive overview of the overall and continuous, time varying QoE modeling for the purpose of QoE management in multimedia services. It also examines the recent research on intelligent and adaptive multimedia streaming strategies, with a special emphasis on ML based techniques for video (conventional and 360°) streaming. This paper discusses the overall and continuous QoE modeling to optimize the end-user viewing experience, efficient video streaming with a focus on user-centric strategies, associated datasets for modeling and streaming, along with existing shortcoming and open challenges.

**Keywords:** Intelligent adaptive streaming, User-centric Multimedia Service, Machine learning models, Mean Opinion Score, Quality of Experience, Video Quality Assessment.

## 1. Introduction

The popularity of video data and video streaming services has increased exponentially in the recent years. Numerous multimedia applications that include video teleconferencing, video streaming
and video-on-demand will dominate the next-generation wireless networks due to the widespread use of internet. Besides the use of conventional video format, 360° videos are being used for the augmented reality (AR) and virtual reality (VR) applications that are gaining immense popularity. The emergence of the Metaverse has led to the proliferation of AR and VR in the field of entertainment. In such applications the 360° videos are the preferred multimedia format that provide immersive viewing experience to the end-users.

Video delivery has emerged as a major component in the bulk of the present day's internet traffic (more than 70%) and is expected to grow even further. According to Ericsson reports [1], globally the average data consumption per smart phone is expected to exceed 21 GB per month by the end of 2024. Particularly, video devices will raise the volume of existing traffic, accounting for 74% of total traffic by the end of 2024. On a global scale, the average monthly consumption of video streaming will amount to 16.3 GB. The UltraHigh Definition (UHD), or 4K, and UHD-2 (8K) video streaming has increased the impact of video devices on traffic because the bit-rate of 4K video, which is 15 to 18 Mbps, and the bit-rate of 8K, which is 20 to 26 Mbps, is more than twice/thrice that of High Definition (HD) and nine times that of Standard Definition (SD) videos. Sixty-six percent (66%) of all new flat-panel TV sets are predicted to be UHD. As per the latest Ericsson Mobility Report [1], it is estimated that 5G will account for nearly 75% of mobile data traffic by 2029. Furthermore, the number of 5G mobile subscriptions is expected to reach a total of 5.6 billion in 2029. Mobile video transmission is undoubtedly a crucial service provided by the 5G mobile networks, and will continue to be so in the upcoming beyond 5G (B5G) era. Service providers are under immense pressure to enhance Quality of Service (QoS) due to the exponential growth in wireless data utilization (primarily multimedia). Increasing video traffic over wireless networks increases the demand for superior multimedia content delivery.

There is an increasing demand for multimedia applications. Video content streaming, exchange and sharing of video-based information are becoming more and more popular among a large number of subscribers. This has compelled service providers to deliver video content of superior quality. Thus, there is an increasing need to ensure that users enjoy a higher Quality of Experience (QoE). In our day-to-day lives, we rely heavily on video content sharing through video calling apps and social media content uploads (in Facebook, Instagram). The majority of popular online services stream video to heterogeneous devices over bandwidth constrained communication networks.

The limited bandwidth and unreliable channel conditions pose further challenge. In order to meet the requirements of the users, high data-rate transmission is required. To facilitate the transmission of HD videos at an increased data rate, it is imperative to employ a system that possesses the capability to adapt its configuration based on the prevailing network conditions. A feasible solution is the presence of feedback mechanism between the server and the client, which enables the regular updating of the channel state matrix and other user-centric feedback factors at the server. Recently, DASH [2, 3]—Dynamic Adaptive Streaming over HTTP— has become the industry standard for online video streaming. For DASH systems, a variety of rate adaption techniques are suggested to provide video quality in accordance with the network throughput.
In AR/VR applications, the seamless streaming of 360° video content there is a need for greater bandwidth and reduced latency compared to existing methods of 2D video delivery. Meeting the bandwidth needs becomes increasingly challenging when streaming the same content to several VR clients. A conventional setup for viewing 360° video involves a user engaging with the scene via a Head-Mounted Display (HMD) device such as the Oculus Rift, Samsung Gear VR, HTC VIVE, Google Cardboard, and Daydream. The 360° video adaptation can be done in a manner similar to conventional videos, as in DASH. A number of research solutions that aid in facilitating an immersive visual experience of 360° video include viewport-dependent/independent and tile-based adaptive streaming. Viewport independent solutions often lead to wastage of network resources as entire content is streamed to the users irrespective of their viewport position.

## 1.1. Survey novelty and contributions

This article provides a state-of-the-art survey on the existing solutions for predicting video quality and the methods and strategies that can be used to improve the streaming experience. The major contributions of this survey are as follows:

1. To provide a review of different existing and emerging video quality assessment models that forecast the video QoE over short duration.

2. To provide an overview of continuous, time varying video QoE estimation models that are particularly useful in streaming sessions.

3. Survey on efficient user-centric multimedia streaming techniques that have been developed so far for DASH-based streaming, and 360° immersive streaming applications.

4. A thorough analysis of the most recent Machine Learning (ML)-based QoE prediction models and adaptive streaming techniques.

5. To provide a list of open source datasets that are accessible publicly for QoE modeling and adaptive streaming of 2D and 360° videos.

6. The summary of findings, list of open challenges and future scope are highlighted.

This review provides an analysis of the recent and developing techniques for user-centric multimedia streaming and a thorough examination of the latest ML-based models for predicting QoE. The structure of the survey is as follows: Section 2 starts with a discussion on QoE definition, assessment methodologies, need for QoE modeling and correlation existing between Video Quality Assessment (VQA) measures and subjective video quality. In Section 3, we briefly introduce the QoE driven, intelligent and adaptive multimedia streaming. In Section 4, we present evaluation metrics and key considerations for ML-based multimedia streaming. In Section 5, we discuss the various state-of-the-art models for overall and continuous, time-varying QoE prediction. Section 6 presents an elaborate review of the intelligent and adaptive streaming techniques for 2D and 360° videos. Section 7 contains several video quality databases and performance metrics that can serve as a helpful tool for future researchers in designing and validating the models. These include the subjective datasets, network traces, 360° viewport
traces, head movement, and eye tracking datasets as well as different performance metrics to evaluate
the effectiveness of the QoE prediction models. In Section 8, we identify the issues/ challenges that
need to be addressed in the future research. We provide a concise summary of our observations and
conclude the paper in Section 9.

## 2. Need for video QoE Modeling in user-centric multimedia streaming

### 2.1. Video QoE: Definition and assessment methodologies

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Factors influencing QoE</figcaption></figure>

According to the European Union (EU) Qualinet Community [4], QoE is defined as “*the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his/her expectations with respect to the utility and/enjoyment of the application or service in the light of the user’s personality and current state*”. As per International Telecommunications Union (ITU), “*QoE is defined as the overall quality of an application or a service as perceived subjectively by the end user*” [5]. Several factors influencing QoE include- System Influence Factors (IFs), Human IFs, Context IFs, and Content IFs. Fig. 1 shows different factors influencing QoE. System IFs are largely concerned with technical elements of quality. Human IFs include, but are not limited to, a user’s physical and mental condition, state of mind, memory, and focus, as well as their needs from the service, recency effects, prior use of the application, and more. Factors like location, end-user surrounding, period of the day, merely casual surfing, service intake period (off-hours, peak hours), are considered as context IFs. The content IFs focus on the content’s individual traits that include content-related details about the service or application under examination.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Categorization of VQA methodologies</figcaption></figure>
<table><caption>Table 1: Video quality and corresponding opinion scores of users.</caption><thead><tr><th>Video quality</th><th>Opinion score</th></tr></thead><tbody><tr><td>Excellent</td><td>5</td></tr><tr><td>Good</td><td>4</td></tr><tr><td>Fair</td><td>3</td></tr><tr><td>Poor</td><td>2</td></tr><tr><td>Bad</td><td>1</td></tr></tbody></table>

<table><caption>Table 2: Features: Definitions and Equations for VQA/IQA metrics.</caption><thead><tr><th>VQA</th><th>Definition</th><th>Equations</th><th>Variables</th></tr></thead><tbody><tr><td>FR</td><td>Peak Signal to Noise Ratio (PSNR)</td><td>PSNR = 20 log<sub>10</sub>(255/√MSE)(1)<br>MSE = <sup>N<sub>1</sub></sup>/<sub>Σ<sub>1</sub></sub> <sup>N<sub>2</sub></sup>/<sub>Σ<sub>2</sub></sub> [X(i,j) - Y(i,j)]<sup>2</sup>/(N<sub>1</sub>·N<sub>2</sub>)(2)</td><td>X(i,j) is the reference frame of original video, Y(i,j) is the reference frame of distorted video; N<sub>1</sub>·N<sub>2</sub> is video resolution</td></tr><tr><td>FR</td><td>Non-Content-based Perceptual PSNR (NC-PSNR) [6]</td><td>NC-PSNR = 10 log <sup>I<sup>2</sup><sub>max</sub></sup>/<sub>Σ<sub>i,j</sub> (I<sub>x</sub>(i,j)-I<sub>y</sub>(i,j))<sup>2</sup>·&#773;(i,j)</sub> (3)<br>w(i,j) = max<sub>(i,j)∈V<sub>i,j</sub></sub> w(i,j) (4)</td><td>v(i,j) = u(-360(<sup>i-1</sup>/<sub>N<sub>1</sub>-1</sub> - 1/2), -180(<sup>j-1</sup>/<sub>N<sub>2</sub>-1</sub> - 1/2)) is pixel alignment likelihood in viewing direction. I<sub>x</sub>(i,j), I<sub>y</sub>(i,j) are pixel intensities in original and distorted video, resp. w&#773;(i,j) is normalized non-content-based weight map, w(i,j). V is viewport viewing directions.</td></tr><tr><td>FR</td><td>Weighted Crater Parabolic Projection PSNR (WCSNR) [7]</td><td>WCPSNR = 10log<sub>10</sub> <sup>255<sup>2</sup></sup>/<sub>WMSE</sub> (5)<br>WMSE = <sup>Σ<sub>j=0</sub><sup>N<sub>2</sub>-1</sup> Σ<sub>i=0</sub><sup>N<sub>1</sub>-1</sup> ((a<sub>i</sub>,b<sub>i</sub>)<sub>x</sub> - (a<sub>i</sub>,b<sub>i</sub>)<sub>y</sub>)<sup>2</sup>J<sub>j</sub></sup>/<sub>Σ<sub>j=0</sub><sup>N<sub>2</sub>-1</sup> Σ<sub>i=0</sub><sup>N<sub>1</sub>-1</sup> J<sub>j</sub></sub> (6)</td><td>(a<sub>i</sub>,b<sub>j</sub>)<sub>x</sub>, (a<sub>i</sub>,b<sub>j</sub>)<sub>y</sub> denote nearest-neighbor pixel intensities at sampling points in the projected frame following crater parabolic transformation of original and distorted frames, resp. J is Jacobian determinant for the transformation between cartesian and spherical coordinates.</td></tr><tr><td>FR</td><td>Structural Similarity Index Measurement (SSIM) [8]</td><td>SSIM = <sup>(2μ<sub>X</sub>μ<sub>Y</sub> + C<sub>1</sub>)(2σ<sub>X</sub>Y + C<sub>2</sub>)</sup>/<sub>(μ<sub>X</sub><sup>2</sup> + μ<sub>Y</sub><sup>2</sup> + C<sub>1</sub>)(σ<sub>X</sub><sup>2</sup> + σ<sub>Y</sub><sup>2</sup> + C<sub>2</sub>)</sub> (7)</td><td>C<sub>1</sub>, C<sub>2</sub> are constants; μ<sub>X</sub>, μ<sub>Y</sub> and σ<sub>X</sub>, σ<sub>Y</sub> are mean and standard deviation of the original and distorted frame luminance intensities, resp.</td></tr><tr><td>FR</td><td>Multi-Scale SSIM (MS-SSIM)</td><td>MS-SSIM = <sup>[l<sub>U</sub>(X,Y)]<sup>α<sub>U</sub></sup> ∏<sub>j=1</sub><sup>U</sup> [c<sub>j</sub>(X,Y)]<sup>β<sub>j</sub></sup> [s<sub>j</sub>(X,Y)]<sup>γ<sub>j</sub></sup> (8)</td><td>α<sub>U</sub>, β<sub>u</sub>, and γ<sub>u</sub> are relative significance parameters of luminance l<sub>U</sub>(X,Y), contrast c(X,Y) and structure s(X,Y) components</td></tr><tr><td>FR</td><td>Weighted-to-Spherically Uniform SSIM (WS-SSIM) [9]</td><td>WS-SSIM = <sup>SSIM × w<sub>ck</sub></sup>/<sub>N</sub> <sup>w<sub>ck</sub></sup>/<sub>Σ<sub>k=1</sub><sup>N</sup></sub> (9)</td><td>w(i,j) = cos((j - N<sub>1</sub>/2 + 1/2) × π/2), (i,j) is current pixel position on projected plane. w<sub>ck</sub> is weight of center point of k<sup>th</sup> sliding window, N is number of slides.</td></tr><tr><td>FR</td><td>M-Singular Value Decomposition (M-SVD) [10]</td><td>M-SVD = <sup>u×v</sup>/<sub>Σ<sub>i=1</sub><sup>u</sup></sub> |D<sub>i</sub> - D<sub>mi</sub>|/(v.v) (10)<br>D<sub>i</sub> = √<sup>κ<sub>b</sub></sup>/<sub>Σ<sub>i=1</sub><sup>b</sup></sub> (x<sub>i</sub> - y<sub>i</sub>)<sup>2</sup> (11)</td><td>v = N<sub>1</sub>·N<sub>2</sub>/κ<sub>b</sub>; κ<sub>b</sub> denotes the block size. D<sub>mi</sub> is the mid-point of sorted D<sub>i</sub>; x<sub>i</sub> and y<sub>i</sub> denotes the original and distorted block, resp.</td></tr><tr><td>RR</td><td>Space-Time Generalized Entropic Differences (ST-GREED) [11]</td><td>GREED = f(SG, TG<sub>i</sub>) (12)<br>SG = <sup>1</sup>/<sub>p</sub> <sup>T</sup>/<sub>Σ<sub>i=1</sub><sup>p</sup></sub> (<sup>1</sup>/<sub>p</sub> <sup>P</sup>/<sub>Σ<sub>i=1</sub><sup>p</sup></sub> |φ<sup>Y</sup><sub>pt</sub> - φ<sup>X</sup><sub>pt</sub>|) (13)<br>TG<sub>i</sub> = <sup>1</sup>/<sub>p</sub> <sup>T</sup>/<sub>Σ<sub>i=1</sub><sup>p</sup></sub> (<sup>1</sup>/<sub>p</sub> <sup>P</sup>/<sub>Σ<sub>i=1</sub><sup>p</sup></sub> (1 + |ε<sup>Y</sup><sub>lpt</sub> - ε<sup>X</sup><sub>lpt</sub>|) <sup>ε<sup>X</sup><sub>lpt</sub>+1</sup>/<sub>ε<sup>X</sup><sub>lpt</sub>+1 - 1</sub>) (14)</td><td>f(.) is mapping function; φ<sup>Y</sup><sub>pt</sub> and φ<sup>X</sup><sub>pt</sub> are modified entropies of distorted and original frames, resp.; p is frame patches; l is sub-band; t- istime instant; ε<sup>X</sup><sub>lpt</sub>, ε<sup>X</sup><sub>lpt</sub>, and ε<sup>Y</sup><sub>lpt</sub> are scaled entropies of original, distorted, and pseudo-reference frames, resp.</td></tr><tr><td>RR</td><td>Spatial Efficient Entropic Differencing (SpEED) [12]</td><td>SpEED = <sup>1</sup>/<sub>B<sub>k</sub></sub> <sup>B<sub>k</sub></sup>/<sub>Σ<sub>b=1</sub><sup>B<sub>k</sub></sup></sub> |h<sub>bkX</sub> - h<sub>bkY</sub>| (15)</td><td>h<sub>bkX</sub> and h<sub>bkY</sub> are locally weighted entropies of original and distorted luminance frames, resp. in block b and scale k. B<sub>k</sub> is scalar quantity per block, based on original and distorted frames.</td></tr><tr><td>NR</td><td>Visual Perception Natural Image Quality Evaluator (VP-NIQUE) [13]</td><td>Q<sub>VP-NIQUE</sub> = <sup>1</sup>/<sub>N</sub> <sup>N</sup>/<sub>Σ<sub>i=1</sub><sup>N</sup></sub> q<sub>i</sub> - <sup>1</sup>/<sub>K</sub> <sup>K</sup>/<sub>Σ<sub>i=1</sub><sup>K</sup></sub> c<sub>i</sub> (16)<br>q<sub>i</sub> = √(<sup>F<sub>i</sub> - μ<sub>X</sub></sup>/<sub>Σ<sub>X</sub> + Σ<sub>Y</sub></sub>)<sup>-1</sup>(F<sub>i</sub> - μ<sub>Y</sub>) (17)</td><td>q<sub>i</sub> represents quality of i<sup>th</sup> patch; F<sub>i</sub> is feature vectors of distorted image; N is length of vector q<sub>i</sub>; μ<sub>X</sub>, μ<sub>Y</sub> and Σ<sub>X</sub>, Σ<sub>Y</sub> are mean vectors and covariance matrices of Multivariate Gaussian model of original and distorted images, resp. c<sub>i</sub> is object recognition confidence; K objects are there in distorted image.</td></tr></tbody></table>

The Mean Opinion Score (MOS) or Difference MOS (DMOS) is a well-known measure to evaluate the video QoE. The strategies employed for assessing video quality encompass both subjective and objective approaches, as shown in Fig. 2. A well-defined method for evaluating QoE encompasses the collection of video quality evaluations through the direct involvement of human participants, usually referred to
as Subjective Video Quality Assessment. The method entails undertaking subjective tests by asking human participants about their perspective on the perceived quality. Subjective quality evaluation is conducted according to Recommendation of ITU-R BT.500-11 (2000). Assessors require knowledge of assessment methodologies, grading scales, impairment types etc. Table 1 displays a variety of scores and the associated quality. Collecting subjective assessments from numerous non-expert assessors is really a difficult process. Also, a controlled environment needs to be established to facilitate the optimal viewing of videos. Hence, it is essential to improve the performance of multimedia applications to meet reasonable standards of human perception.

Alternately, objective Image/Video Quality Assessment (IQA/VQA) metrics can be generated by calculating the artifacts and content features that have an impact on the overall video quality, referred to as Objective Video Quality Assessment. Peak-Signal-to-Noise-Ratio (PSNR), Visual Signal-to-Noise-Ratio (VSNR), Singular Value Decomposition (SVD), Structural Similarity Index (SSIM), Multi-Scale SSIM (MS-SSIM), Motion-based Video Integrity Evaluator (MOVIE), Spatio-Temporal RR Entropic Differencing (STRRED), Temporal MOVIE (TMOVIE), and Spatial MOVIE (SMOVIE) constitute some of the objective quality metrics [14]. These metrics are calculated on the luminance component at each frame with reference to source video in a similar set. The mean value across all frames yields the objective metrics associated with the video sequence. The objective VQA metrics specifically formulated for 360° videos are NCP-PSNR [6], WCPPPSNR [7], spherical-SSIM [15], and WS-SSIM [9]. Table 2 lists the key VQA/IQA metrics, spanning Full Reference (FR), Reduced Reference (RR), and No Reference (NR) types. The VQA metrics are categorized into FR, RR, and NR depending on the information availed from the source videos. FR (e.g., [8], [16]) necessitates complete information; RR (e.g., [11]), relies on partial information; and NR (e.g., [17, 18, 19, 13]) operates without any dependence on the available source video data.

## 2.2. Need for video QoE modeling

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Video transmission over wireless access network</figcaption></figure>

As videos pass through multiple processing stages before reaching the end-users, the effect of most of them is to degrade the video quality. Videos can have significant distortions at several stages, such as during processing (image acquisition), compression (encoding), or transmission. At most of the stages, video quality deteriorates. Fig. 3 presents the flow diagram of video transmission process which shows that the videos are subject to several processing stages before and in the course of delivery to viewers
that results in potential loss of video quality. Recent developments in video coding and compression
enable customers to access high-quality video services. The content providers use several compression
techniques to offer affordable services. Video encoding causes distortions due to compression. The
application of block-based coding and motion compensation techniques by most video coding and com-
pression standards subjects the decoded videos to one or more compression artifacts. The compression
standards, e.g., MPEG 2, MPEG 4, H.263, H.264/Advanced Video Coding (AVC), H.265/ High Effi-
ciency Video Coding (HEVC), and VP9 exhibit blocking, ringing, and blurring artifacts [20]. Spatial
artifacts that frequently result from encoding comprise false contouring, mosaic patterns, and contrast
distortion. In addition, videos often experience significant temporal artifacts, primarily resulting from
transmission across communication networks. Video transmission via wireless or IP networks leads to
packet loss or corrupted frames that suffer from temporal artifacts [21], like jitter, additive noise, and
motion compensation mismatch, which affect individual frames.

This is even more challenging incase of 360° videos. There are several issues associated with efficient
encoding and processing of 360° videos [22, 23]. It is necessary to project the 360° videos onto a flat
surface because most filters and coding tools are based on 2D images. At every stage of the 360° video
processing pipeline, distortion may be introduced, starting with the acquisition of images. Omnidirec-
tional images and videos are typically stitched from numerous cameras [24], which can incorporate a
variety of peripheral problems. Typical issues include loss of information, misaligned edges, temporal
coherence, ghosting, camera jittering, dominant foreground objects traveling across views [25], and dif-
ferences in exposure that are most pronounced at the poles. Temporal discontinuities can also occur
in the videos, like objects appearing and vanishing. Most cameras fail to capture these, while some of
them are subsequently reconstructed in the post-processing stage.

HEVC allows independently decodable tiling of 360° videos with less overhead, allowing adaptive
streaming according to the user's Field Of View (FoV).

After acquisition of 360° video frame, it must be transformed into a planar representation meant for
encoding and storage. The Equi-Rectangular Projection (ERP), is the most prevalent projection for
360° video. It divides/cuts the visible sphere into several sets of rectangles all sharing the same solid
angle. The ERP is used in studies in [26], [27], [28]. This projection is inefficient due to distortion at
the poles, since more pixels are used to encode the poles compared to the equator. As viewers mostly
pay attention in the vicinity of the equator, the poles tend to fall beyond the FoV. The Cubic Mapping
Projection (CMP) [29] is another type of projection in which a cube is built around the sphere with
rays radiating outward from the center. The projection mapping is the result of each ray intersecting
with only one point on both solids' surfaces. Compared to the ERP, the CMP results in less geometric
distortion at the poles than at the edges/corners of a face. The CMP reproduces a sphere better towards
the middle of each face, making this intuitive. In [30], a hybrid version consisting of equi-rectangular
CMP is used to improve the coding efficiency. The projection uses a mapping function and accomplishes
a higher level of uniform sampling, thus minimizing the presence of boundary artifacts.

Few other projections include- Polar square projection [32] (similar to barrel projection, but maps
<table><caption>Table 3: Correlation between objective metrics and DMOS for LIVE [31] dataset.</caption><thead><tr><th>Metrics</th><th>PSNR</th><th>SSIM</th><th>MS-SSIM</th><th>VSNR</th><th>M-SVD</th><th>SMOVIE</th><th>TMOVIE</th><th>MOVIE</th></tr></thead><tbody><tr><td>SROCC</td><td>0.3527</td><td>0.5119</td><td>0.6749</td><td>0.6022</td><td>0.3906</td><td>0.7607</td><td>0.7462</td><td>0.7217</td></tr></tbody></table>

the poles to squares), Octagonal [33] projection (can reduce oversampling areas and assemble points into an octagon that can be rearranged into a rectangle prior to encoding), Rotated Sphere Projection (RSP) [34] (unfolds the sphere under two rotation angles and sutures it in the form of a baseball, improving coding efficiency), and offset projection [35] (higher number of pixels are used to encode areas near the estimated gaze direction, whereas areas at greater angles from it are compressed more tightly; thus, saving bandwidth).

On generating the planar representation of the 360° video, tiling (dividing into tiles) is done, which is a step in the encoding process that can have a considerable effect on the streaming performance. For effective compression of 360° videos, the projection and tiling strategy substantially affect the intensity of geometric distortion. It is possible to independently encode the Quality Emphasized Region (QER) of a video at a higher quality compared to the rest parts [36]. With Motion Constrained Tiles (MCT), tiled video segments can be efficiently encoded and decoded independently [22, 37]. The work in [23] predicts the FoV tiles and viewport with high accuracy by using Long Short Term Memory (LSTM). It is necessary to develop methods that leverage ensemble learning [38] to enhance viewport prediction accuracy and assign high resolution to tiles where a user's viewpoint might appear. Determining the right tile size is necessary for conserving bandwidth. The videos can be encoded depending on the popular viewing areas, as in Macrotile [39]. It is critical to identify such larger viewing areas and adjust the macrotiles as per random head movements. This can maximize the quality and reduce the data to be downloaded, resulting in bandwidth and energy savings. Still, the user's immersive experience will be impacted by any tiles that might be missed while streaming. Thus, video quality needs to be monitored, assessed, and improved.

## 2.3. VQA/ IQA correlation with subjective video quality

Computing the correlation coefficient between the objective metrics and subjective video quality assessment scores indicates/helps assess the relationship between objective and subjective ratings (MOS/DMOS). The correlation coefficient indicates the dependency of DMOS on a given objective score. Correlation coefficient gives the statistical relationship between PSNR and DMOS, SSIM and DMOS, MSSIM and DMOS, Spatial MOVIE and DMOS etc. The computed correlation coefficient values are depicted in Table 3 for the LIVE [31] dataset. Fig. 4(a) shows the correlation (SROCC) existing between video quality of frames at current instant and the past instants for videos v1 to v6 shown in Fig. 4(b) from [48, 49, 50]. The correlation gradually decreases for a past frames is further away in the timeline. This correlation is useful for building the continuous, time varying QoE models [51].

The video QoE modeling and user-centric multimedia streaming with special emphasis on ML are less considered in many previous research works [40]–[47]. Table 4 provides a comparative summary
<table><caption>Table 4: Summary of existing surveys on video QoE modeling and streaming.</caption><thead><tr><th>S. No.</th><th>Reference</th><th>Major contributions</th><th>Difference</th><th>QoE modeling</th><th>ML based QoE models</th><th>Adaptive streaming</th><th>ML based streaming</th><th>bit-rate prediction</th></tr></thead><tbody><tr><td>1</td><td>[40]</td><td>Survey on QoE assessment and management of video transmission</td><td>Detailed discussion of QoE models extended to 360° videos</td><td>Yes</td><td>No</td><td>Brief</td><td>No</td><td>No</td></tr><tr><td>2</td><td>[41]</td><td>A complete review of subjective and objective audio-visual quality assessment methods, different factors influencing quality and its degradation, datasets for quality evaluation</td><td>Elaborate discussions on continuous, time varying models with emphasis on ML based techniques</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>3</td><td>[42]</td><td>Survey on video quality metrics (subjective and objective), QoE modelling and methodologies</td><td>Extension to ML-based 2D and 360° video quality modeling</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>4</td><td>[43]</td><td>Overview of video coding standards and challenges in HAS, bit-rate adaptation schemes and state-of-the-art related researches</td><td>Discussion on the state-of-the-art and challenges in 360° video streaming with focus on user-centric feedbacks and ML techniques</td><td>Yes</td><td>No</td><td>Yes</td><td>No</td><td>Yes</td></tr><tr><td>5</td><td>[44]</td><td>Exhaustive review on subjective and objective video quality evaluation methods, QoE modelling for HAS, their influence factors and associated challenges</td><td>Extension to 360° video streaming with emphasis on ML-based user-centric techniques</td><td>Yes</td><td>Brief</td><td>Yes</td><td>No</td><td>Brief</td></tr><tr><td>6</td><td>[45]</td><td>Review of subjective and objective QoE evaluation methodologies, certain features of QoE assessment for video streaming, gaming extended reality</td><td>Discussion about video (normal and 360° videos) bit-rate adaptation techniques</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td><td>No</td></tr><tr><td>7</td><td>[46]</td><td>Survey on continuous time varying QoE assessment models</td><td>Detailed discussion on ML-based continuous QoE prediction modeling and streaming</td><td>Yes</td><td>Brief</td><td>No</td><td>No</td><td>No</td></tr><tr><td>8</td><td>[47]</td><td>Briefly reviews the quality assessment models for adaptive video streaming that includes subjective studies and objective models along with their performance analysis</td><td>Extension to 2D and 360° ML-based user-centric video streaming</td><td>Yes</td><td>Brief</td><td>No</td><td>No</td><td>No</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 4. (a) Correlation between QoE of current and past frame indices for the given video samples (b) Video samples (from left to right) (i) v1 (ii) v2 (iii) v3 (iv) v4 (v) v5 (vi) v6</figcaption></figure>

of our article with respect to existing surveys in literature. In this survey we holistically cover the ML-based QoE modeling, adaptive and ML-based streaming, as well as bit-rate prediction methods along with the prevalent datasets used in this domain for the conventional and 360° video applications.
### 3. User-Centric Multimedia Streaming

This section discusses adaptive streaming of multimedia content, wherein the client has complete control over the streaming session and can possibly adapt the multimedia stream to its context, such as network conditions, device capabilities, perceptual quality, etc. The adaptive multimedia streaming solutions employ an explicit adaptation loop/logic where clients perform different measurements and push the information towards the server using sophisticated schemes/algorithms.

#### 3.1. QoE driven multimedia streaming

In recent times, there has been a notable worldwide increase in the use of multimedia streaming applications. The prevailing major participants in the present global market for these kinds of applications are the Akamai Technologies, Netflix, Apple Inc, Amazon Web Services, and Hulu. The rapid evolution of communication networks and widespread usage of smartphones and smart portable devices, with enhanced processing capabilities, facilitate seamless access to multimedia content.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Adaptive video streaming framework</figcaption></figure>

DASH is an over-the-top wireless streaming technology that is prominent and effectively enables the adaptability of content delivery in response to changing network conditions. Fig. 5 depicts the visual representation of the streaming concept. The fundamental concept behind DASH is to break up the multimedia content into small chunks. To have various interpretations, multiple sets of parameters (i.e., bit-rate, quantization parameter (QP), frameate, resolution) are utilised to encode each chunk. The DASH server hosts the video segments (of certain duration, generally 2, 4, or 10 seconds), where each segment is encoded to have various representation of quality levels. The obtained representations are logged in the Media Presentation Description (MPD) file that gives an index for the listed media segments at the server.

DASH effectively manages diverse network conditions through the dynamic adjustment of video parameters. The client-side of DASH is responsible for monitoring various network parameters such as network conditions and buffer size. Based on this information, the client-side determines the appropriate representation of the media chunk to be played next. DASH depends on HTTP and Transmission Control Protocol (TCP) that ensures reliable delivery of data to intended destination. The acknowledgement between the sender and receiver occurs when the receiver notifies the sender on the successful
reception of packets, as well as any instances of lost or erroneous packets, prompting the sender to
re-transmit them.

Transmission protocols play an important role on QoE performance in video streaming applications. Both TCP and User Datagram Protocol (UDP) are widely utilized for live video streaming. Nevertheless, numerous live streaming industries are adopting UDP for high-motion videos that typically possess more intricate temporal information. The end-to-end delays associated with UDP are far lower than those of TCP, which is essential for events like live sports. UDP doesn't use handshakes, delivery assurances, or duplicate protections to make sure data is correct. Instead, it relies on fundamental mechanisms and checksums to maintain data integrity. The likelihood of certain video distortions, including frame loss and flicker, is increased when UDP is employed. Reliable Multidestination Data Transport Protocol (RMDT) [52], an UDP based-transport protocol has proved to be more optimal for certain point-to-multipoint streaming scenarios. RMDT is developed to mitigate common IP network impairments, including packet losses, latency, and jitter, to guarantee reliable transmission. RMDT may serve as a viable alternative to TCP and UDP in future high-bandwidth applications.

The latest version of dash.js [53], v 5.0, is a free, open source MPEG-DASH player that functions as a reference client and is useful for academic purposes. The other commercial players in the market include Apple's AVPlayer, Shaka Player, and HLS.js. The dash.js provides a number of functionalities pertaining to adaptive media streaming. This encompasses the playback of both dynamic and static content via DASH and Smooth Streaming formats. The period transitions operate differently in dash.js, permitting codec alterations, whereas, due to single presentation in Shaka Player, the codecs may remain unchanged. The dash.js v 5.0 supports BOLA [54], buffer [55] and throughput based ABR algorithms.

The codecs govern the multimedia data representation format in adaptive video streaming applications. There has been an ongoing effort towards developing next-generation coding solutions. The AVC/H.264 is a popular codec that uses a 16x16 macroblock configuration for frame encoding. HEVC/H.265 is a more recent codec that reduces video bitrate by around 50% relative to AVC/H.264 while maintaining similar subjective quality. The enhanced coding efficiency of HEVC facilitate the 4K video streaming with superior fidelity, high dynamic range (HDR), and wide chromatic gamut (WCG). HEVC/H.265 provides a tiling feature for optimized video streaming, utilizing a 64x64 Coding Tree Unit (CTU) structure to encode each tile. Thus, attaining a superior compression ratio compared to AVC/H.264. The VVC/H.266 is the latest codec that has a coding efficiency of ~50% higher for similar subjective video quality compared to HEVC, particularly for HD and UHD video resolutions and ~75% higher than AVC [56]. The new features in the latest VVC were developed to carry out adaptive streaming with resolution variations, 360° immersive video and ultralow-latency streaming.

Video quality measurement holds significant importance for video service providers as it pertains to enhancing service delivery and ensuring satisfactory performance under typical network impairments. According to a study conducted by Conviva [57], network operators have experienced significant financial losses as a result of inferior streaming quality. Hence, network providers strive to enhance streaming quality and optimize the whole viewing experience. The network-related video impairment examination
is of the utmost importance due to its detrimental impact on QoE. Different methods for evaluating video quality contribute to the monitoring of quality, thereby ensuring the fulfilment of QoS standards and enhancing the overall performance of the system.

The success of a video streaming service or multimedia application is reliant upon the level of contentment experienced by its end-users [58]. Eventually, it is humans that benefit from the utilisation of these services. The perceptual expectations of end consumers are consistently increasing with the hope of superior quality. QoE depends on many things, such as the analysis of the observer, the type of service, the variety of user tools, and the length of the video. QoE is the end-user's overall impression or appreciation of a service. In [59], a survey investigated QoE to be the most desirable user choice when it comes to video delivery, surpassing other factors such as the type of content, ease-of-use, mobility, timing, and sharing. QoE indicates the extent of a viewer's comprehensive perspective and contentment with various aspects, including the video content, communication networks, service quality, and environmental factors. Therefore, enhancing QoE and satisfying the requirements for superior video quality has emerged as a key goal for content providers, academicians, and video streaming companies.

Due to persistently bad network conditions, video data in the buffer depletes, owing to late arrival of video packets, causing playback to stall, popularly known as rebuffering. The frequent appearance of rebuffering events introduces impairments that have a detrimental effect on the QoE. The frequency and length of stalling, the amplitude and frequency of quality transitions, as well as their temporal positions, are some of the aspects that are taken into consideration for impairment measurements. The estimation of QoE is of utmost importance in order to optimize the delivered quality. Furthermore, compression as well as video bit-rate implemented by DASH, results in degradation of customer's QoE.

The adoption of the rate adaptation approach results in a persistent variation in perceptual quality over time, often referred to as time-varying quality [60]. The resultant QoE is influenced by a combination of rate adaptation and rebuffering events. Hence, the video quality encountered by end users throughout a streaming session is characterised by a dynamic variation over time. The continuous assessment of QoE in a streaming session poses a significant challenge due to the presence of non-linear relationships among several elements (for instance, video quality, stalling and frequent bit-rate adaptation) that influence QoE. In addition to spatial distortions, the quality of streaming video exhibits intricate temporal correlations. Continuous monitoring of QoE throughout a streaming session is crucial for effectively managing shared resources among users and maximizing the perceived video quality. Moreover, it has the potential to mitigate the loss of quality by effectively adjusting the video bit-rate at the recipient's end.

## 3.2. QoE-aware 360° video streaming for VR/AR

The 360° video transmission provides viewers with an immersive experience and is a fundamental component of numerous applications, including Metaverse. The 360° video content offers AR/VR application users an immersive experience through the usage of HMD. Users have the ability to rotate their heads in any direction and maintain a seamless, uninterrupted view of the surrounding with these
<figure><img src="image_8.png" /><figcaption>Fig. 6. (a) Head navigation direction parameters (b) User's viewport (c) Equirectangular projection.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. (a) Number of users (b) Bit-rate (c) PSNR of the 360° video tiles.</figcaption></figure>

360° videos. Fig. 6 displays the parameters indicating the direction of head navigation for the users of 360° videos. Fig. 6(b) and 6(c) depict the user's viewport and the equirectangular projection of 360° video content, respectively. In equirectangular projection, the sphere is spread out on a flat surface like a cylinder on a 2D sheet.

There is a growing concern over the safety and visual comfort while viewing VR/AR content. A few studies [61], [62] have reported a range of symptoms including headaches, trouble concentrating, and dizziness, that attribute to VR sickness, also termed as cybersickness. Contributing factors to cybersickness include optical flow, VR fidelity, user interaction (e.g., navigation methods, controllability), age, gender, VR experience, FoV, latency, and HMD types. Chattha *et al.* [63] empirically evaluated motion sickness in VR.

We study the user head orientations while viewing a 360-degree video using a head mounted stream-
ing device for a set of 55 independent viewings, based on the dataset [64, 65, 23]. Fig. 7(a) shows the
number of users watching the specific tile numbers based on their FoV during the viewing experiment.
Certain tiles (57-64) are the least viewed, while others (45-54) are viewed by most of the users. Fig 7(b)
and Fig. 7(c) show the bit-rate and PSNR (respectively) of individual tiles of the 360° video for low
(QP=15) and high (QP=35) QP values. A low QP tile has a high bit-rate and a high PSNR than a
high QP one. However, these values are not the same for all the tiles of the video at the same QP value,
motivating FoV based bit-rate adaptation for efficient 360° video streaming.

At higher resolutions, these videos require an exceptionally high bit-rate to deliver an immersive
experience [23]. Excessive bandwidth is wasted while delivering portions out-of-viewport that the end-
user never watches. It is difficult to accurately predict the specific content that would interest the
viewer in the future playback. This is due to the limited exposure of users to a few 360° videos in the
<figure><img src="image_10.png" /><figcaption>Fig. 8. Outline of a ML based QoE predictor</figcaption></figure>

past and their fluctuating psychological states and emotional conditions during each playback. Users might adhere to entirely different content depending on moods and mental state. Therefore, it is crucial to develop effective strategies for transmitting 360° videos over resource-limited wireless networks while optimizing the viewing experience. Using AR and VR representations, the 360° videos aid in simulating an immersive experience of the real world.

The most important challenge for successful multimedia streaming is in ensuring the perceptual satisfaction/ pleasure of the end-users. Multiple users may demand different immersive experiences since user equipments' ability for FoV prediction might vary depending on various FoV prediction methods. This necessitates the development of novel approaches for synchronous and asynchronous frame structures comprising a basic tier and an enhancement tier for video transmission in multiuser cellular networks, wherein the basic tier is utilized to boost resilience against FoV prediction failures [66]. The users tend to quit the streaming session if the viewing experience falls below a certain threshold level. Accurate QoE estimation/ prediction can help in adapting the content transmission to enhance the viewing experience of the end users. So, a lot of effort is directed towards accessing the 360° video quality and improving the immersive experience. The overall video quality is assessed for different segments of the video with short durations ranging from 5-10 seconds. The assessment of video streaming QoE in a continuous manner (on a per-frame basis) is essential for the purpose of regulating the degradation in video quality throughout the entire streaming session. The use of optimized ML models helps in accurately evaluating the impact of several impairments on the viewers perceptual QoE using different input features (VQA objective metrics and impairment factors). Fig. 8 shows the general design of a ML-based QoE predictor.

### 3.3. Intelligent and adaptive multimedia streaming

The DASH server delivers videos at different bit-rates depending on user needs [67]. The display devices are capable of adapting various levels of quality based on the bandwidth that is available. Consequently, this can result in the occurrence of compression or scaling artifacts. For instance, video content that has been encoded at lower resolution may be upscaled to a significantly higher resolution on the viewing device. If the video bit-rate is lower than the available bandwidth, it not only leads to smooth playing but also results in the inefficient use of resources that could otherwise be allocated
towards improving the video quality.

The standardization of DASH does not mention the execution of the adaptation strategy. For example, how a client can adaptively select the video quality according to the present network statistics and other factors. In order to switch between multiple streams, a controlling mechanism can be configured either at the server/ client side that dynamically predicts and subsequently requests an optimal video segment representation. This request is made considering several factors like network conditions, buffer occupancy, requester's device features etc.

Recent studies [54, 68, 69, 38, 39] report several controlling mechanism, also called Adaptive Bit-Rate (ABR) schemes/ algorithms for DASH. The ABR schemes primarily aim to maximize end user's QoE by adjusting the video bit-rate/ tiles to fluctuating network conditions and other factors. Choosing the appropriate bit-rate might be difficult due to (i) fluctuations in network throughput (ii) varying video QoE requirements, such as minimizing rebuffers, reducing startup delays, or achieving high bit-rate (iii) challenge of maintaining consistent playback state (play or pause) for multiple clients located in different geographical positions while delivering video content simultaneously (iv) The consequences of bit-rate decisions, such as selecting a high bit-rate that may drain the buffer to a critical level and cause rebuffering (v) confined availability of bit-rate options, which creates a conflict between prioritizing higher video quality and the chance of rebuffering.

The ABR algorithms do not properly represent the client's instantaneous visual perception of quality, nor do they quantify the measure of QoE objectively. Strategies like reducing the number of rebuffering and bit-rate switches may appear logical as they help reduce viewer frustration and ultimately enhance user involvement in the video session. However, they fail to comprehensively record or ensure the optimal (immediate or overall) visual QoE. Most of the adaptation schemes depend on fixed rules that govern the bit-rate decisions, which needs a lot of parameter tuning and may not generalize well to large network scenarios. The efficacy of Reinforcement Learning (RL) techniques [70, 71, 72] demonstrates their viability as a solution in scenarios where prior assumptions about the operational environment are not required.

A 360° video is a bounding sphere that includes all of the surrounding content. A higher bit-rate is needed for smooth streaming at high resolution (≥4K) and high frame rate (≥60 fps) to provide immersive content. The amount of bandwidth needed is many times greater than normal videos of same quality.

It is possible to greatly lower the bit-rate needed for 360° video only by ABR streaming the parts of video which lie in the user's FoV [73].

This includes breaking down the 360° video into smaller tiled video segments, streaming the segments that lie in FoV at higher resolution/ quality and the remaining at lower quality.

Such VR content, even when encapsulated with the most recent codecs, necessitates a significantly higher bit-rate for transmission at UHD or higher resolutions. Despite the growth in network capacity, it is still not adequate to support streaming higher content and fulfill the growing demands of higher video quality [66]. Given the widespread accessibility of display devices that incorporate cutting-edge
<figure><img src="image_11.png" /><figcaption>Fig. 9. Generalized framework for user-centric multimedia streaming</figcaption></figure>

technological improvements and the significant network bandwidth requirements of streaming users, the primary obstacle in content delivery lies in the development of network-aware solutions aimed at enhancing the overall viewing experience. Multiple service providers might employ varying encoding choices for the same bit-rate [74, 75]. However, there may be fluctuations in network conditions as a result of factors such as congestion caused by heavy usage during peak hours, the mobility of users, and the extent of network coverage.

Streaming multimedia content is challenging due to unreliable network conditions and the heterogeneity of end-user devices. Ensuring optimal performance is imperative for streaming applications, since it is crucial to maintain a seamless playback experience with minimal buffering and frequent fluctuations in quality. Devising intelligent schemes can select proper multimedia content based on user-centric feedback. Adaptive schemes/ algorithms help to select appropriate bit-streams based on several user-centric feedbacks, such as buffer occupancy, playback rate, instantaneous throughput of the network, and experienced video quality. The use of Deep Reinforcement Learning (DRL) in bit-rate adaptation schemes facilitates the selection of appropriate bit-rates. For streaming the 360° videos, that are of higher resolution, a substantially higher bit-rate is required, even when employing the most recent codecs for encoding. Devising ML based intelligent and adaptive strategies can help reduce the data rate requirement for streaming higher resolution videos. Fig. 9 shows a generalized framework for user-centric multimedia streaming.

## 4. Evaluation Metrics and Key Considerations

Here, we discuss the evaluation metrics and key considerations for the ML-based user-centric multimedia streaming.
### 4.1. Objective function and k fold cross validation

For video quality estimation, the objective/ loss function is MSE distance between the predicted ($\hat{\mathbf{y}}'$) and subjective (ground truth) ($\mathbf{y}'$) video quality scores, given as

$$ \mathcal{L} = \|\hat{\mathbf{y}}' - \mathbf{y}'\|_2 \qquad (18) $$

When training the QoE framework in a classification problem, cross-entropy is the commonly used loss function, given as

$$ \mathcal{L}_c = - \sum_{i=1}^{n} y'_i \log \hat{y}'_i \qquad (19) $$

where, $y'_i$ is the $i^{th}$ ground truth, $\hat{y}'_i$ is the $i^{th}$ predicted quality and $n$ represents the batch size. To reduce the bias caused by random sampling of training and holdout data samples when comparing the predicted accuracy of multiple ML-based QoE models, studies commonly employ k-fold cross-validation. In cross-validation approach, the complete dataset $D$ is initially shuffled. The dataset $D$ is randomly divided into $k$ mutually exclusive subsets $D_1, D_2, \dots, D_k$. Among a collection of $k$ subsets, a combination of ($k-1$) subsets is chosen for training the regression model, while the remaining subset is chosen as the test set. Therefore, a specific fold is created, comprising a training set and a test set. In this way, $k$ folds can be generated, denoted as $F_l, \forall l = 1, 2, \dots, k$. During the process of creating folds, it should be ensured that each specific subset is chosen only once as the test set. The cross-validation estimate of the overall performance criteria is determined by taking the average of the $k$ individual performance metrics given as,

$$ \mathcal{T} = \frac{1}{k} \sum_{i=1}^{k} \mathcal{P}_i \qquad (20) $$

where, $\mathcal{P}$ represents the performance measure for each fold.

### 4.2. Performance Metrics

This subsection discusses the popularly used performance metrics for assessing the performance of various QoE models.

i) **Root Mean Square Error (RMSE):** The computation involves taking the square root of the average of all squared errors, which are calculated by comparing each element of the predicted QoE and the actual video QoE of $n$ test videos. $RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y'_i - \hat{y}'_i)^2}$ where, $y'_i$ represents the actual score of $i^{th}$ test video and $\hat{y}'_i$ represents the predicted score of that video.

ii) **Mean Relative Absolute Accuracy (MRAA):** The Mean Relative Absolute Error (MRAE) is calculated by taking the average of all relative absolute errors.

$$ \text{MRAE} = \frac{1}{n} \sum_{i=1}^{n} \left| \frac{y'_{i} - \hat{y}'_{i}}{y'_{i}} \right| \cdot \text{MRAA} \text{ is calculated as, } \text{MRAA} = (1 - \text{MRAE}) \times 100\% $$

iii) **Root Mean Relative Square Accuracy (RMRSA):** Root Mean Relative Square Error (RMRSE) is calculated by taking the square root value of the average of all relative squared errors between each element of predicted and actual video QoE score. $RMRSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (\frac{y'_{i} - \hat{y}'_{i}}{y'_{i}})^2}$. Thus, RMRSA is calculated as $RMRSA = (1 - RMRSE) \times 100\%$.
iv) *Spearman Rank Order Correlation Coefficient (SROCC)*: It is a statistical index that gives a measure of monotonicity. $SROCC = \frac{\sum_{i=1}^{t}(r_i - \bar{y}_r)(s_i - \bar{y}_s)}{\sqrt{\sum_{i=1}^{t}(r_i - \bar{y}_r)^2}\sqrt{\sum_{i=1}^{t}(s_i - \bar{y}_s)^2}}$ where, $r_i$ and $s_i$ are the rank of $y'_i$ and $\bar{y}'_i$ in actual and predicted score vector, respectively. $\bar{y}_r$ and $\bar{y}_s$ represents mean of $r_i$ and $s_i$, respectively.

v) *Pearson Linear Correlation Coefficient (PLCC)*: PLCC gives an accuracy measurement. It is calculated as follows, $PLCC = \frac{\sum_{i=1}^{n}(y'_i - \bar{y})(\bar{y}'_i - \bar{y}')}{\sqrt{\sum_{i=1}^{n}(y'_i - \bar{y})^2}\sqrt{\sum_{i=1}^{n}(\bar{y}'_i - \bar{y}')^2}}$ where, $\bar{y}$ and $\bar{y}'$ are mean of actual and predicted score, respectively of test videos.

vi) *Perceptually Weighted Rank Correlation (PWRC)*: PWRC [76] measures the rank accuracy of VQA/IQA metrics, the impact of perceptual significance modifications, and the level of subjective opinion ambiguity by developing non-uniform weighting and adaptive activation methods. The rank accuracy is evaluated using a rank correlation measure that takes into account the confidence level. This measure calculates the area under the curve by collecting PWRC (Partial Weighted Rank Correlation) values within a certain threshold range [$T_{min}, T_{max}$], given as: $\int_{T_{min}}^{T_{max}} S(y, \hat{y}, T) dT$, where, $y$ and $\hat{y}$ represent the actual and predicted score, respectively. $S(y, \hat{y}, T)$ is the overall sorting accuracy indicator. $T$ represents the sensory threshold.

vii) *Outlier Ratio (OR)*: It is used to measure the consistency. OR is calculated as ratio of number of outlier-points to number of test videos. $OR = \frac{\text{Total no. of outliers}}{n}$.

# 5. QoE modeling and Evaluation for efficient multimedia streaming

Significant studies have been undertaken in understanding and evaluating the impact of several impairments on the viewer's perceptual QoE. The QoE predictors can be modeled to forecast two aspects: (i) the overall video QoE for short-duration segments, and (ii) the continuous, time-varying QoE on a per-frame basis during the streaming session.

## 5.1. Overall video QoE prediction

Assessment of video quality can be done using both subjective and objective methodologies. Recent studies have examined the subjective evaluation of video quality in [49], [77], [78], [79], [80], [81], while [82] examines the significant difficulties associated with this assessment. Subjective assessment methods reflect that tremendous effort has been put in designing the databases for VQA and obtaining the subjective scores. These methods are infact time-consuming and cumbersome. Considering the difficulties and challenges connected with subjective assessment, many approaches in [83, 84, 11] resorted to objective assessment methods. Although subjective assessment tasks are difficult, yet their goal is to help in the development of automatic objective VQA models.

The subjective assessment methods provide valuable insights into optimizing adaptive streaming performance and user experience. They generate valuable data for studying the impact of various streaming dimensions (including buffering events of various durations, start-up delays, and quality transitions) on end-user experience. These data facilitate the assessment of video quality and QoE models, enabling an analysis of their merits and weaknesses. The subjective studies capture the critical
<figure><img src="image_12.png" /><figcaption>Fig. 10. (a), (b) Variation of different input features vs frame index for the distorted video sample Mask from [48] (c) Variation of subjective MOS vs frame index and indication of re-buffering events (d) Snapshot of the video sample Mask</figcaption></figure>

elements of practical systems by integrating real network measurements and/or client ABR schemes. The start-up phase poses the greatest challenge for all ABR schemes, as they have not yet established the video buffer. Thus making them susceptible to network fluctuations that can significantly diminish QoE. The subjective data indicates that viewers can perceive these differences during the start-up phase. These data illustrate the significance of subjective assessment methods, particularly during the start-up phase, for improving streaming performance. Moreover, continuous scores obtained from subjective assessment methods can be used to train QoE predictor models that can guide ABR schemes.

Objective assessment methods, instead rely on objective VQA metrics, such as, PSNR, Video Multi-method Assessment Fusion (VMAF), FUNQUE [16], ChipQA [19], FOVQA [17], ST-GREED [11], VP-NIQE [13] etc. for predicting the video quality. VQA/IQA gives information of video distortions like compression, packet loss etc. Fig. 10(a) and 10(b) shows the variation of different input features (PSNR, VMAF, SSIM, MS-SSIM) vs frame number for the video sample shown in Fig. 10(d). Fig. 10(c) displays the variation of subjective MOS with respect to the frame index and the occurrence of rebuffering events for the corresponding video sample.

With the increasing popularity of VR, there is a growing effort towards developing objective measures that are specifically designed for VR. Adhuran *et al.* [7] presented a weighted crater parabolic projection-based PSNR that enhances the existing metric by incorporating rectilinear functions to support VVC for VR-VQA. In [9], SSIM is exemplified by weighted multiplication in several regions, ensuring that the spherical distortion perceived by the viewer has a linear correspondence with the distortion plane. Croci *et al.* [85] partitioned the 360° video into several patches using Voronoi diagram
and subsequently applied the 2D methods on these patches to minimize distortion. The traditional
objective metrics are not relevant for the VR videos as they do not consider the geometric distortion
resulting from the spherical image projection.

In [44], Barman et al. covers the HTTP Adaptive Streaming (HAS) QoE models, influencing factors that impact QoE modelling, and related challenges.

In [86], Lievens et al. presented a MOS predictor in accordance with the user assessment using a parametric equation defined by taking the quality switching, frameate, and rebuffering events into consideration. However, the model's performance is not reported/ verified using subjective assessment. The work in [87] presents a methodological framework for assessing subjective QoE considering several factors like video features, initial latency, segment duration, switching technique, stalls, and QoS indicators. Experimental analysis revealed that objective measurements can be correlated with the most critical subjective parameters of user experience. Yamagishi et al. in [88] used separately audio and video quality estimation module that outcomes quality scores per second, which was later integrated in audiovisual integration module into each second audio-visual coding quality. In [89], the authors designed a QoE model taking encoded video quality, which is computed per segment by considering average QP, and quality variation into account.

Goring et al. [90] devised hybrid video quality evaluation models that use client-accessible metadata in addition to pixel-based NR to FR models. The hybrid models continue to use the basic information of the distorted videos, like the video codec, resolution, bit-rate, and frameate. Robitza et al. [91] used the ITU-T P.1203 model to measure the streaming quality considering factors such as loading time, stalling, and consumer involvement. For P.1203, Model 0 was utilized, which necessitates the inclusion of codec, bit-rate, frameate, and resolution data. It has limited applications because of the use of specific codecs and basic inputs. The authors in [92] initially derived the development of the key dimensions of video quality to formulate the entire quality using a new method, known as direct scaling. Their work solely forecasts the overall video quality, rather than focusing on the streaming video quality. The most prevalent models employ various measurement criteria, including resolution, QP [93], video bit-rate [94], and frame rate along with stalling events, to forecast the quality of a video. These models depend on statistical variables such as bit-rate, resolution, QP, and frame rate that are insufficient in capturing the distorted video features and addressing the overall influence on QoE.

In [95], Tran et al. examines the impact of QoE deterioration due to factors like starting quality, quality alterations, initial delay, encoding, and rebuffering. Robitza et al. [96] introduced a parametric candidate model, known as P.NATS, for ITU-T P.1203 which adopts a modular approach, combining audio and video quality scores to obtain the final MOS. It obtains coding relevant data per-frame and incorporates changes in quality over time, as well as stalling. Yet these models do not take into account VQA/IQA measurements as inputs which are perceptually correlated with the quality. The ITU-P.1204.3 [97] is the quality assessment standard for 4K/UHD videos. The developed bit-stream model [98], based on ITU-P.1204.3 forecasts quality ratings by combining Random Forest regression with parametric model. The RF is used to estimate the residual MOS that the parametric part fails to
predict. The RF uses additional features (average motion/frame, horizontal motion, and frame sizes),
apart from the parametric features. The idea behind this is that the parametric component is not able
to adequately encompass the spatial and temporal complexity of the video sequences. It can be noted
that in contrast to the conventional FR ([99]) and NR models ([100]) that process decoded frames,
which consist of pixel values; the bit-stream based models ([95, 96, 98]) directly analyze information
obtained from the encoded bit-stream. This reduces extra computational load of decoding the videos
before evaluating their quality.

The work in [80], examined the effectiveness of various VQA metrics on distortions like scaling
and compression while viewing on multiple display devices. The SpEED [12] demonstrated superior
performance in a particular distortion category, while ST-GREED [11] performed better in another
kind of distortion. Various metrics exhibit varying performance for a specific distortion category when
viewed on several display devices. These experimental results revealed that the effectiveness of each VQA
metric was constrained in one of the distortion category. This highlights the constraints of individual
VQA/IQA measures in considering additional parameters such device display size, video resolution,
viewing distance, and other factors that impact video quality.

Quality assessment using machine learning techniques has gained a lot of interest in the recent years.
Support Vector Machine (SVM) is used in VMAF (0.3.1) [99] to forecast subjective quality based on
three elementary input features: motion information, detail loss metric, and VIF. Further improvements
to VMAF has been made in [101] through the use of two SVM models by integrating new features and
then merging the results from each of them. The additional features encompass the widely-used VQA
metrics (PSNR, SSIM) and are computed independently on the luma and chroma channels, as well as
at four distinct resolution scales (2D DWT decomposition). Utilizing such a combination allows for
training on multiple databases. In [102], two NR ML methods— Support Vector Regression (SVR)
and neural networks— are used to estimate the game video streaming quality. It is constrained by the
use of fundamental feature representations, such as bit-rate, temporal information (TI), and resolution.
In [103], Ghosh et al. used SVM to estimate the overall QoE. Multi-dimensional QoE estimation for
mobile video was achieved using ML models such as SVM, linear regression, bagging tree, and others
that were tweaked using the Weka-ML Software tool [104]. Yet, this study was limited by the use of
fixed model parameters to achieve the desired performance and was evaluated on a single database. The
fundamental idea in these works [101]-[104] is that any basic measure/elementary metric may possess its
own advantages and disadvantages in relation to the features of the source content, distortion level, and
kind of artifact. The authors integrate basic measures through ML models, that assigns appropriate
weights to each basic metric. This enables the final metric to retain all the advantages of the component
metrics and produce a final score that is more accurate.

Lekharu *et al.* deployed a Deep Neural Network (DNN)-based model to select the appropriate bit-rate, resulting in total QoE maximization for the user. Tao *et al.* [105] utilises a Deep Learning (DL) method to estimate the QoE of mobile videos. This is achieved by creating a substantial dataset consisting of more than 80,000 data points, each including 89 network parameters. Nevertheless, gathering
all 89 network parameters is a labor-intensive procedure. In DEMI [106], Zadtootaghaj et al. introduces a Convolutional Neural Network (CNN) (i.e., DenseNet 121) trained using an objective measure that allows it to identify video artifacts. The model is subsequently fine-tuned based on the dataset, taking into account the ratings for blockiness and blurriness. Then, a Random Forest algorithm is applied to pool frame-level estimations and TI of the videos in order to evaluate the overall video quality. For the evaluation of holographic AR devices, [107] developed a QoE model that employs a fuzzy inference system (FIS) to accurately assess the device's performance in a quantitative manner. FIS utilizes fuzzy logic to map the given inputs (i.e., quality of content and hardware, knowledge of environment, user interaction) to an output. The system employs membership functions to precisely specify the degree of fuzziness in a fuzzy collection and calculates output values by applying fuzzy rules. In DeepQoE [108], Zhang et al. used a comprehensive framework that utilizes DL techniques (namely, word embedding) to extract generalized features, merge them, and feed them into a neural network for representation learning, with the aim of predicting video quality. DeepQoE demonstrates superior performance when applied to a massive dataset exclusively focused on compression artifacts. The drawback of DL techniques is that they require a large amount of training data, typically in the scale of thousands or millions. However, the subjective QoE scores collected in complex environments have significantly lower dimensionality and limited public accessibility.

In case of user-generated video content uploads, where the reference video is unavailable, different Blind VQA (BVQA) models have been developed. Compared to conventional feature-based blind VQA models, DL-based BVQA models have attracted interest recently. The Patch-VQ [109] extracts the spatial and temporal features using PaQ-2-PiQ and ResNet 3D network, then spatio-temporal pooling is applied in region-of-interest to model the relationship between local-to-global space-time quality. Ultimately, PatchVQ combines quality scores in a spatio-temporal manner to generate the final predictions. In [110], the authors used transfer learning for BVQA, combining pre-training on both real IQA databases (to acquire frame-level feature extraction) and large-scale action recognition sets (to acquire motion perception of videos). However, they did not consider the viewing conditions of videos in different environments, which can impact the predicted quality. In [111], Guan et al. developed a visual attention module that acquired perceptual quality scores at the frame level. Primarily, the proposed method integrates attention information with spatial-temporal content aspects in a cyclic manner. Next, utilizing a memory attention module which prioritizes quality, the video-level attention-guided characteristics are extracted by changing the dimensions and attention of frame-level depiction. To generate the best prediction results, studies in [81, 112] have gained the benefit of using both shallow (statistical) and DL features. It was observed that the DL-based blind VQA models performed significantly better than feature-based models in case of large datasets.

The papers [113, 114, 115, 116] used techniques of fusing multiple features to combine several objective quality measures. Azevedo et al. [113] used various objective measures taken from viewports to forecast the quality of omnidirectional videos. The video samples in [114] are first categorized according to their content, taking into account spatial and temporal information values. Multiple FR
<figure><img src="image_13.png" /><figcaption>**Fig. 11.** Scatter plot depicting the correlation between actual and predicted video quality scores for the most effective MO-QoE model for different test ((a) LIVE [31] (b) VQEG HD3 [117] (c) Waterloo [118]) datasets. Source[14]</figcaption></figure>

VQA measures are combined within the groups and given as input to the SVM to estimate the video quality. Their research focused exclusively on compression distortions found in the MCL-V dataset, with input limitations restricted to FR. The ATLAS [115] framework utilizes SVM to forecast video quality based on a limited set of buffering, QoE, and memory-related features. The fusion methods in [113, 114, 115] lack any clear objective and are instead merged in a random manner. In addition, metrics that have nearly identical characteristics, such as PSNRhvs and PSNRhvs-M, are not excluded, even if they have only minor differences in performance. This leads to an increase in the total number of inputs and unnecessarily complicates the model.

In MO-QOE [14], the authors developed a framework that predicts the video QoE using Multi-Feature Fusion based Optimized Learning Models (OLM). The OLMs are the optimized neural network models designed using different combinations of dataset pre-processing techniques, optimization algorithms (Adam/ Batch gradient) and neural network topologies (ANN/FNN). The MO-QOE framework uses a feedback mechanism to determine which features are to be selected in the fusion process using the MFF Algorithm. Adaptive moment estimation and Batch gradient descent techniques are employed to iteratively update the weight and bias parameters of the learning models. The developed MFF algorithm incorporates the feature fusion approach, thus, reducing the model complexity by minimizing the number of inputs. The experimental findings indicate that by employing the MFF-OLM algorithm, adequate performance can be attained with the careful selection of a limited number of features, typically ranging from four to six. Scatter plots of VQA model predictions effectively illustrate model correlations. Fig. 11 displays the MO-QoE [14] model's predictions using five-fold cross validation on different test datasets. It is shown to have a more compact distribution, which aligns with its superior correlation with MOS. The RMSE values obtained for different QoE models when trained on LIVE NFLX II+LIVE Netflix and tested on LFOVIA are shown in Fig. 12(a). The RMSE values when trained on LIVE NFLX II and tested on the LIVE Netflix dataset are given in Fig. 12(b). The MO-QoE framework performs the best with both the test datasets, obtaining the least RMSE values of 6.853 and 0.117, attributed to its use of optimized models to predict the video QoE.

The work in [119] introduced a hybridized framework consisting of neural network and fuzzy logic.
<figure><img src="image_14.png" /><figcaption>**Fig. 12.** RMSE of different QoE models (LSTM [60], DEMI [106], DeSVQ [51], DeepQoE [108], MO-QoE [14]) when tested on (a) LFOVIA (b) LIVE Netflix dataset</figcaption></figure>

Initially, the neuro-fuzzy framework extracts frame-level features from artifacts and video content, then pools these features using a multi-stage temporal pooling approach. After pooling, these features are applied to a neuro-fuzzy model to predict the quality. Next, the encoding dimensions (e.g., spatial resolution, frame rate, and QP) are adapted, and the acquired features, along with the frame-level features, are applied to the CoActive Neuro-Fuzzy Inference System (CANFIS), followed by Adaptive Neuro-Fuzzy Inference System (ANFIS). The work in [116] discusses integration of objective metrics at the block level. Their scheme is constrained by a content-driven approach. The paper [120] combines many FR IQA measurements using a greedy strategy that utilises both linear and non-linear regression approaches. The best possible global solution may not necessarily result from such a greedy approach. Additionally, the database under test simply addresses compression artifacts.

## 5.2. Continuous, Time-varying QoE prediction

With the advancement of multimedia streaming applications, significant attention is focused on enhancing the seamless and dynamic watching experience. Multiple studies in [121, 122, 123, 124, 60] have conducted research on continuous Time Varying Subjective Quality (TVSQ) evaluation and designing prediction models for continuous time video quality. Crucial elements that affect the quality of watching videos in a streaming scenario are the spatial and temporal characteristics of the reference videos, length and frequency of rebuffering incidents, and the initial delays while starting the video. The studies conducted in [125, 126, 127] examine several factors that influence the user experience during a streaming session. They conclude that when the video quality does not meet the acceptable standards, users are more likely to terminate the session. Wu et al. calculates the average watching percentage in [125] utilizing publicly available information, like the video's content, channel, and context, without taking user feedback. Lebreton et al. forecasts the proportion of users viewing a certain video with reference to time in [127]. They examined and found the causes of user disengagement, which include stalls, poor coding quality, and a number of quality-related issues.

Callet et al. in [121] extract several features from MPEG-2 video frames and applied them to CNN to evaluate the continuous video quality. However, their evaluation was limited only to distortions generated by coding artifacts, while distortions due to transmission losses were not examined. In [122],
Chao *et al.* made use of the Hammerstein Weiner (HW) model to get the spatio-temporal features from videos by finding out the Short Time Subjective Quality (STSQ) of video segments. STSQ gives an estimate of the perceptual quality of viewers calculated by using VQA metric for every segment of the video. The calculated STSQ were put into a dynamic model which accounts for the hysteresis effects of human conduct and forecasts the TVSQ. Their work is limited in that the TVSQ is only assessed in the context of rate adaptation, without considering rebuffering effects. In [128], the authors proposed a kind of Recurrent Neural Network (RNN) using VQA metrics, forecasted video quality ratings, re-buffering, and memory relevant inputs. These inputs are obtained from video at regular intervals and given as input to a nonlinear single hidden layer neural network.

The study in [123] conducts a subjective assessment of streaming videos, as well as QoE prediction using SVR during playback with exponential modeling during rebuffering state. In [129], Ghadiyaram et al. determined the time-varying QoE using a multi-stage (where a learner's estimate from one stage is fed into another HW model at the next stage) and multi-learner (where the estimate from each HW model is used to train a different learner) method that examines association between stalling occurrences. The model successfully captures the interactions between stalling occurrences and subsequently analyzes the spatial and temporal content of the video. Bampis et al. devised a continuous QoE prediction model, known as NARX [124], by formulating it as a time series forecasting problem using a non-linear autoregressive model with external outputs. NARX's autoregressive memory enables it to consider recent events. Such models are very useful in understanding the human visual system, which generates the hysteresis effect, in which previous events have a significant influence on the QoE at the present moment. The inclusion of external variables in NARX enables it to represent long-term memory effects, the impact of rebuffering on perceived quality, and current as well as historical video quality. An issue that may occur when employing autoregressive models for real-time QoE prediction is the potential propagation or amplification of prediction errors when the estimated outputs are given back into the prediction engine.

Eswara et al., used a network of LSTM [60] to record the temporal dependencies for forecasting the continuous, time varying QoE of streaming videos. The LSTM network receives inputs via assessing the STSQ using VQA metrics. Also, it incorporates rebuffering-related inputs at a reasonable time step, which is measured per second. The unique configuration of the gating mechanisms and memory cell in LSTMs allows them to successfully capture the temporal intricate dependencies in QoE modeling. Also, it helps to overcome the limitations of ordinary RNNs, like the issue of vanishing gradients. The unidirectional LSTM architecture just takes forward dependencies into account. There is a chance of overlooking valuable information. So, the work in [130] employed a Bidirectional LSTM model to process the inputs derived from VQA metrics, memory-relevant data, and rebuffering. The Bidirectional LSTM considers both forward and backward dependencies, thus improving the prediction accuracy.

Eswara et al. [60] exclusively employed the LSTM network, whereas Duc et al. [131] solely utilized the Temporal CNN (TCNN), employing a nearly same set of inputs as [128]. Duc et al. [131] used the TCNN to overcome the computational intricacies of LSTM networks. Because of the sequential
processing feature in its architecture, LSTM has a high computational complexity. This gives rise to a question of how well it would function on devices with low computing power. TCN has the feature of parallel computing, which offers benefits in terms of modeling and computation. Also, TCN utilizes dilated causal convolutions to effectively capture the temporal dependencies. Due to the sequential nature of LSTM's recurrent structure, the model was unable to properly harness the parallel computing capabilities, resulting in an increase in computational cost. In [132], the authors examined DASH video application in a practical emulation environment that relies on actual 5G traces (in both static and mobility conditions) to evaluate the QoE of three ABR algorithms (i.e., hybrid, buffer, and rate-based). Then they proposed a supervised ML classifier to forecast user contentment by taking into account network characteristics, including Round Trip Time (RTT), throughput, and the quantity of packets.

In M-3R [133], the framework is designed using LSTM networks, which are able to accurately simulate the temporal dynamics of streaming video quality under 3R settings. The "3R"- settings combine the perceptual contributions of numerous FR, RR, and NR VQA metrics. The 3R input features (per frame basis) are given to the LSTM networks because they can correlate highly with perceptual quality. A frame based approach is utilised in M-3R learning framework, as their analysis revealed that QoE is more sensitive to the correlation existing between the frames. In DeSVQ [51], an integrated framework is designed comprising of CNN and LSTM networks. It uses a two-stage feature processing approach, with the first stage processing high level spatio-temporal features and the second stage processing low-level features examined by the VQA metrics. In stage-I, in order to extract the high level spatio-temporal features from the distorted videos, CNN is quite effective. In stage II, the non-linearities and temporal dependencies associated with QoE changes are captured by the LSTM networks. A linear layer combines the output from both stages and feeds it to the decision trees. This is how the framework maps the video features to continuous quality scores. Yang et al. proposed a Light-weight QoE model, LiteDC [134] by integrating temporal Dilated Convolution network with a tailored pruning strategy for multi-device video streaming. Dilated convolutions fundamentally include the incorporation of "holes" or intervals between the elements of the convolutional kernel, hence expanding the kernel's engagement with the input data while preserving a streamlined model architecture. This allows the network to effectively capture long-range dependencies without a corresponding increase in computational complexity, as in LSTM networks.

The existence of such ML-based continuous time varying QoE evaluation models assist in taking decisions on the adaptation of video streams. During adaptive HTTP streaming, a decision is made regarding which chunk should be selected for the next delivery. Such kind of modeling incorporated with hysteresis effect and other non-linearities, aid in selecting the segment with the best quality. Fig. 13(a) displays the outcome of the continuous video quality prediction model DeSVQ [51] on a test sample 13(b) taken from the LIVE Netflix dataset. It can be seen that the predicted QoE very closely follows the actual QoE.

Table 5 provides a review of ML-based continuous, time-varying as well as over the entire video QoE modeling. The (↑) and (↓) indicates the increase and decrease in value respectively.
<figure><img src="image_15.png" /><figcaption>**Fig. 13.** (a) Continuous video QoE prediction result on test video sample from LIVE Netflix [49] shown in (b). CI represents the confidence interval.</figcaption></figure>

# 6. Intelligent and Adaptive Video Streaming Techniques

This section provides a review of the broad research efforts directed towards the design of bit-rate adaptation schemes accounting for perceptual optimization of streaming video.

## 6.1. *Video streaming schemes/strategies*

A survey undertaken by Seufert et. al in [148] categorises the streaming video QoE influencing factors into perceptual (i.e., time varying video quality, stalling frequency and its duration, initial delay) and technical (i.e., video content and encoding parameters, segment size and duration, adaptation logic/algorithm, hardware/software employed in video streaming system) that directly or indirectly affect the QoE. Impact of such factors and their techniques of measurement can be found in [149]. These factors are challenging when considered for obtaining a trade-off between the conflicting objectives (e.g., maximizing bit-rate vs. minimizing rebuffering).

The paucity of concrete quantitative methods for measuring streaming video QoE itself is an extensive area of research. There exists not an exclusive set of QoE measurement metrics that can comprehensively measure QoE. Works in [123], [80] assess QoE using FR objective metrics that do not capture temporal distortions well [49]. RR (e.g., Video-RRED) and NR (e.g., VP-NIQUE [13]) metrics are not effective in evaluating perceptual video quality subject to streaming relevant distortions, like rate adaptations, rebuffering or mixed interplay of both. These methods are followed by DASH adaptation schemes in [148], [150]. In D-DASH [151], Gadaleta et al. selected SSIM (that needs complete info of uncompressed segment) to access the instantaneous video quality which was pre-computed on the server side and added in the MPD. However, this increases the computational load on the server side. In [152], Bampis et. al designed a database that has subjective QoE scores collected in adaptive streaming scenario. Here, VMAF was used for quality computations that was saved in a chunk map and presented for bit-rate adaptation on client side. It is not viable to compute the FR and RR metrics on the client side as they require complete/ partial info of the reference segment. This will deviate from the realistic assumption of the absence of reference information on the client side. Although NR metrics can be used, the consequences of inaccurate estimates is a matter of concern.
<table><caption>Table 5: Summary of the reviewed ML-based QoE models. Cont.: Continuous</caption><thead><tr><th>Category</th><th>ML Technique</th><th>Influence Factors</th><th>Accuracy</th><th>Datasets used</th><th>Reference paper</th><th>Prediction</th></tr></thead><tbody><tr><td rowspan="10">QoE modeling</td><td>SVM</td><td>STRRED, stalling(length, duration, frequency), last bit-rate drop time, impairment duration</td><td>lowest prediction uncertainty than existing</td><td>LIVE Netflix[49], Waterloo[118]</td><td>ATLAS [115]</td><td>Overall</td></tr><tr><td>SVM, bagging tree, linear regression</td><td>duration, frequency &amp; starting time of stalling, playback delay</td><td>reduces prediction errors between 25% and 50% than existing</td><td>LIVE avaasi Mobile[135]</td><td>[104]</td><td>Overall</td></tr><tr><td>CNN (DenseNet 121) + Random Forest</td><td>VMAF, PSNR, blockiness &amp; blurr ratings</td><td>same as existing models (on gaming data)</td><td>KUGVD[136]; LIVE Netflix, LIVE NFLX II[48]</td><td>DEMI [106]</td><td>Overall</td></tr><tr><td>3D CNN</td><td>video, text, categorical info (resolution), continuous values</td><td>35.71-44.82% (on small data);90.94% (on large dataset)</td><td>WHU-MVQoE2016 [137], LIVE Netflix, LIVE NFLX II</td><td>DeepQoE [108]</td><td>Overall</td></tr><tr><td>Fuzzy-logic</td><td>content quality, hardware quality, environment understanding and user interaction.</td><td>3.89-5.791 (RMSE)</td><td>Young Conker, RoboRaid</td><td>[107]</td><td>Overall</td></tr><tr><td>Optimized Learning Models (ANN, FNN)</td><td>VQA metrics+ impairment factors (multi-feature fusion strategy)</td><td>≈2 to 3× than existing</td><td>LIVE[31], Waterloo, VQEG HD3[117], VQEG HD4[117], LIVE Netflix, LIVE NFLX II</td><td>MO-QoE [14]</td><td>Overall</td></tr><tr><td>convLSTM, ResNet 50</td><td>video (spatial features, spatial-temporal attention-guided features)</td><td>0.894 (SROCC), 0.902 (LCC) (on CVD2014); 0.836 (SROCC), 0.834 (LCC) (on LIVE-Qualcomm)</td><td>VD2014, LIVE-Qualcomm, KoNViD-1 k, LIVE-VQC, Youtube-UGC</td><td>[111]</td><td>Overall</td></tr><tr><td>Neuro-fuzzy (AN-FIS, CANFIS)</td><td>Artifacts (blockiness, blurr, noise)+ content characteristics</td><td>0.94-0.97 (LCC), 0.26-5.2 (RMSE)</td><td>LIVE[31], LIVE MVQ[138], CSIQ[139], IVP[140]</td><td>[119]</td><td>Overall</td></tr><tr><td>Non-linear autoregressive models</td><td>VQA metrics, playback status, time since last video impairments</td><td>Outperforms HW model in RMSE and outage rate</td><td>LIVE Netflix</td><td>NARX [124]</td><td>Cont.</td></tr><tr><td>RNN &amp; feed-forward multi-layer neural network</td><td>Estimated video quality, rebuffering, recency</td><td>performance improvement by 5-10% in OR</td><td>LIVE-avaasi Mobile, LIVE Netflix, LIVE HTTP video streaming [122]</td><td>Bampis et al. [128]</td><td>Cont.</td></tr><tr><td>Linear Regression</td><td>current short time &amp; previous time varying video quality</td><td>39-41%↓ rmse</td><td>LIVE HTTP video streaming</td><td>Eswara et al. [141]</td><td>Cont.</td></tr><tr><td>SVM</td><td>Current &amp; previous time slot's video quality, rebuffering frequency, &amp; duration</td><td>34.1%↑ relative LCC gain than eTVSQ [142]</td><td>LFOVIA[123], LIVE HTTP video streaming</td><td>Eswara et al. [123]</td><td>Cont.</td></tr><tr><td>LSTM</td><td>short time video quality, playback indicator, duration since last rebuffering</td><td>4-29.1%↑ relative LCC than [123, 122, 143]</td><td>LFOVIA, LIVE Netflix, Mobile stall II[50], LIVE HTTP video streaming</td><td>Eswara et al. [60]</td><td>Cont.</td></tr><tr><td>Temporal convolutional network</td><td>STRRED, playback indicator, #rebuffering, time since last impairment</td><td>1.15-38.7% ↑ relative LCC than [60, 124, 143, 144]</td><td>LFOVIA, Mobile stall II, LIVE Netflix</td><td>Duc et al. [131]</td><td>Cont.</td></tr><tr><td>Bidirectional LSTM</td><td>short time video quality, playback indicator, duration since last rebuffering, number of rebuffering</td><td>11.4-43.9% ↑ relative LCC than [60, 143, 124]</td><td>LIVE Netflix</td><td>[130]</td><td>Cont.</td></tr><tr><td>Decision Tree, Multi-linear and Random Forest Regression</td><td>RTT, throughput, packet count per video segment</td><td>87.6% (static) and 79% (mobility scenarios) accuracy by Random Forest</td><td>[145], [146]</td><td>Mustafa et al. [132]</td><td>Cont.</td></tr><tr><td>LSTM</td><td>VQA metrics (FR+RR+NR)</td><td>2.5-20% relative PLCC gain</td><td>LIVE Netflix, LIVE NFLX II, Mobile stall II</td><td>M-3R [133]</td><td>Cont.</td></tr><tr><td>LSTM+CNN+XGBoost</td><td>distorted video frames+ VQA metrics</td><td>4.8-20% relative PLCC gain</td><td>LIVE Netflix, LIVE NFLX II, Mobile stall II</td><td>DeSVQ [51]</td><td>Cont.</td></tr><tr><td>Temporal dilated convolution network</td><td>VQA metrics, time since last rebuffering, playback indicator, rebuffering duration from beginning of playback, VQA switch</td><td>6.4%↑ in prediction accuracy than baseline</td><td>LFOVIA [123], MCQoE [147]</td><td>LiteDC [134]</td><td>Cont.</td></tr></tbody></table>
Online video streaming sessions need consistent track of end user's QoE under dynamic network conditions; so that the perceived video quality does not fall below an acceptable level. In-depth reviews of the parametric, bit-stream, and hybrid models are analyzed and discussed in [44]. Parametric and bit-stream models exclusively reflect the QoE as a function of influencing elements, such as initial loading delay, switching amplitude, stalling frequency and duration, QP, and frame rate. These models use various mathematical functions, such as linear, exponential delay, curve fitting, and ordinal logistic, to describe the relationship between QoE and these parameters. Taha et al. [153] developed a streaming framework based on adaptive quantization. They established a relation between the QP in H.264 and H.265 codecs and the QoS in 5G wireless networks. They simulated packet loss characteristic of the network to assess the influence of QP on the delivered video quality utilizing both objective (PSNR and SSIM) and subjective (DMOS) quality metrics. The framework can automatically estimate the QoE by identifying the packet losses. Thus finding the optimal QP value to improve end-user QoE. The DASH framework in PENSIEVE [154], Model Predictive Control (MPC) [155], and Buffer Occupancy based Lyapunov Algorithm (BOLA) [54] employ several QoE measures, where QoE is defined as a linear and log function of segment bit-rate and rebuffering time. QoE metrics defined in this manner (i.e., as simple representations of influencing factors) are not capable of addressing the distortions and long-lasting dependencies inherent in time-varying QoE.

A detailed survey of different rate adaptation algorithms for DASH is presented in [43]. Bit-rate adaptation schemes are categorized as client-based, network-aided, server-based, and hybrid adaptation, according to the specific system entity where the logic is implemented. In [156], Ozfatura et al. proposed an optimal network-supported multi-user DASH video streaming. Client based adaptations chose suitable bit-rate by adapting to one or more parameters such as bandwidth variations, buffer size, etc. PiStream [157], an adaptation scheme allows DASH clients in LTE network to determine the accessible bandwidth by virtue of a resource control component. In [158], Xiao et al. devised a DASH to Mobile (DASH2M) streaming strategy using HTTP/2 server push along with stream completion properties for adjusting bandwidth so as to reduce client's battery usage while improving QoE. Huang et al. [55] put forward Buffer Based (BB) rate adaptation algorithm where the client picks bit-rate of the next segment depending on buffer occupancy to avoid excessive rebuffering. Jiang et al. [159] devised a Rate Based (RB) adaptation scheme where the efficiency can be improved in HTTP adaptive video streaming by choosing the maximum possible set of bit-rates in order to optimize the end user experience. BOLA [54], an online control logic frames adaptation of bit-rate as a utility maximization problem that includes two main elements of QoE, i.e., video's average bit-rate and rebuffering duration.

In [160], Yadav et al. introduced a method called QUETRA (QUEuing Theory approach to DASH Rate Adaptation) which models the client as an M/D/1/K queue. This model allows for the computation of the desired buffer level based on parameters such as bit-rate option, buffer space, and network throughput. Nevertheless, their assessment was restricted to only four network profiles, consisting of two profiles from the DASH Industry Forum [161] and others from HSDPA [162] dataset. Yin et al. devised a MPC [155] technique which utilizes a throughput estimator to forecast the anticipated throughput for
the subsequent five chunks. They developed fastMPC, a method that combines throughput and buffer capacity predictions to generate decisions that approach performance levels near to that of MPC. Such control theoretic techniques have constraints on development time, which must be significantly shorter due to the need for a complete redesign when altering model parameters. This approach is similar to dynamic programming, and the effectiveness of the adaption logic may be compromised if the throughput estimator is not correct, leading to suboptimal decisions. Furthermore, the MPC method does not possess a comprehensive assessment of its performance in real-world tests. The bandwidth-reliant adaptations in [157, 158] often have restrictions in obtaining improved QoE due to the lack of accurate methods for predicting bandwidth, which leads to frequent buffer underflow. Buffer-based adaption methods in [54, 55] often have instability issues when there are prolonged variations in bandwidth.

A number of interesting studies in [163], and [164] have utilized the Markov Decision Process (MDP) to develop bit-rate adaptation techniques. These techniques employ dynamic programming to determine the optimal approach for adaptation. The study in [163] used MDP to simulate perceived content-aware bit-rate adjustment for adaptive streaming issues. Next, they formulated a segmented value iteration rate adaption approach to address this issue, which disaggregates an MPD session into several sub-sessions and applies the value iteration method to determine the ideal solution for each interval. In mDASH [164], Zhou et al. introduced a method that incorporates a bit-rate adaption algorithm based on MDP optimization that takes into account buffer measurements, bit-rate stability, and bandwidth scenarios as state variables. They suggested a pseudo-greedy heuristic approach that suffers from computational burden. The complexity of such models is prohibitively great to be solved in real-time.

In order to address the primary challenges of dynamic programming, namely the computational burden and the need for prior knowledge of network conditions and video content, various studies have employed RL techniques. The RL agent acquires experience through its interactions with the environment and constructs an ideal policy. PENSIEVE [154] is a learning system that utilizes data collected by DASH clients, including throughput traces and buffer occupancy, from previous segments to make observations. PENSIEVE employs an actor-critic approach to learn its policy, utilizing the fundamental gradient ascent method. The paper [165] utilizes an Online RL (ORL) technique, similar to PENSIEVE, which is based on the fundamental policy gradient method. These methods have limitations, specifically, they are not efficient in terms of sample usage as they require collecting new samples for practically every policy update. Comyco [166] utilizes imitation learning instead of previous RL-based approaches to train the neural network. The reason is that the near-optimal policy may be accurately and immediately approximated from the present state in the ABR scenarios. Also, the gathered expert policies can facilitate rapid learning of the neural network. Comyco's objective is to prioritize the selection of bit-rates that offer superior perceptual video quality, instead of high video bit-rates. For mobile edge computing-supported short video applications, [167] uses a DRL strategy to optimize video quality benefit while minimizing bearer costs as well as latency penalties. The method uses a policy gradient approach to pick video quality levels without the need for explicit computation of the actual video quality. Simply linking higher resolution to improved video quality does not provide
<table><caption>Table 6: Overview of comparison between the state-of-the-art ABR schemes</caption><thead><tr><th>Scheme</th><th>Buffer adaptation</th><th>Bandwidth estimation</th><th>Network</th><th>QoE Optimization</th><th>QoE modeling methodology</th></tr></thead><tbody><tr><td>BB [55]</td><td>✓</td><td>X</td><td>Fixed network</td><td>X</td><td>-</td></tr><tr><td>RB [159]</td><td>X</td><td>✓</td><td>Fixed network</td><td>X</td><td>-</td></tr><tr><td>MPC [155]</td><td>✓</td><td>✓</td><td>Simulated</td><td>✓</td><td>Linear equation</td></tr><tr><td>PENSIEVE [154]</td><td>X</td><td>X</td><td>Simulated, WiFi</td><td>✓</td><td>Linear, Log equation</td></tr><tr><td>BOLA [54]</td><td>✓</td><td>X</td><td>Simulated</td><td>✓</td><td>Average playback interruption</td></tr><tr><td>D-DASH [151]</td><td>✓</td><td>X</td><td>Simulated</td><td>✓</td><td>Objective metric (SSIM)</td></tr><tr><td>ORL [165]</td><td>X</td><td>X</td><td>Simulated</td><td>✓</td><td>Linear equation</td></tr><tr><td>SAC-ABR [68]</td><td>X</td><td>X</td><td>Simulated</td><td>✓</td><td>Linear, Log equation</td></tr><tr><td>ABRaider [69]</td><td>✓</td><td>✓</td><td>Simulated, Experimental (WiFi,4G)</td><td>✓</td><td>Linear, Log equation</td></tr></tbody></table>

an accurate representation of the experienced video quality as QoE is affected by a number of factors.

In SAC-ABR [68], Naresh et al. proposed an off-policy method for ABR streaming which utilizes Soft Actor-Critic (SAC) driven DRL. The objective of SAC-ABR is to optimize the entropy while simultaneously improving the expected benefits, leading to a more balanced approach between exploration and exploitation. Guo et al. introduced a DRL approach in their paper [168] to address the issue of streaming in small-scale wireless networks by incorporating a buffer-aware strategy. To determine the productive video streaming period when neither an underflow nor an overflow of playback occurs, they created a reward function. Their proposal involves framing the issue of allocating bandwidth and managing buffers (using MDP) simultaneously in order to optimize the duration for which each user can efficiently stream videos. Simply mere extension of high-quality video streaming duration may not necessarily yield an improvement in perceptual quality. Furthermore, the system is restricted to employing only one method, specifically buffer-aware, without considering additional observations.

In ABRaider [69], Choi et al. introduced a multiphase RL approach that incorporates both online and offline phases to enhance adaptive video streaming. ABRaider integrates multiple ABR algorithms and generates suitable policies for different settings during the offline phase, with a focus on the specific users' environments during the online phase. Nevertheless, the methodology is complex and confusing due to the recent development of multiple ABR algorithms, leaving uncertainty over the necessary combination of these algorithms. In addition, every algorithm possesses its own unique set of deficiencies, which are further compounded during the course of their execution and might negatively impact the overall efficiency. The work in [169] devised a multimedia system comprising three fundamental components: end users, network, and servers. Initially, they presented a DL model that includes data pre-processing, representation learning, and QoS/QoE prediction. Subsequently, they introduced a mechanism for regulating the bit-rate using RL. In D-DASH [151], Gadaleta et al. introduced a Deep Q-Learning model that combines DL and RL techniques to enhance the QoE of DASH video streaming. They proposed and assessed multiple learning frameworks that integrate feed-forward and RNNs using sophisticated techniques. Nevertheless, these networks were incapable of managing jobs with continuous action spaces, as they usually kept and iterated the value functions of state-action pairs as a lookup table. Integrating the advantages of fuzzy logic with sophisticated DRL techniques, Yaqoob et al. proposed FReD-ViQ [170], an adaptive streaming solution that uses Fuzzy RL to provide
<table><caption>Table 7: Summary of ML-based user-centric adaptive streaming techniques</caption><thead><tr><th>Category</th><th>ML Technique</th><th>Influence Factors</th><th>Accuracy</th><th>Datasets used</th><th>Reference paper</th></tr></thead><tbody><tr><td rowspan="6">Adaptive streaming</td><td>Imitation learning (Neural Network)</td><td>past network, video, and playback features</td><td>7.5-16.79% average QoE improvement</td><td>HSDPA[162], FCC[171], Oboe[172]</td><td>comyco [166]</td></tr><tr><td>Reinforcement Learning (policy gradient)</td><td>network throughput, chunk download time, sizes of subsequent chunks, number of chunks, chunk bit-rate</td><td>12-15% QoE improvement</td><td>FCC, 3G/HSDPA</td><td>Pensieve [154]</td></tr><tr><td>policy gradient (online RL)</td><td>throughput, download time, next segment size, buffer, #remaining segments, previously requested segment size</td><td>2.5-28% ↑ QoE than existing</td><td>FCC, 3G/HSDPA, Belgium[173]</td><td>ORL [165]</td></tr><tr><td>Soft Actor-Critic (SAC) driven DRL (offpolicy)</td><td>throughput, download time, next segment size, buffer, #remaining segments, previously requested segment size</td><td>27.42% ↑ QoE than existing</td><td>FCC, Oboe, [174]</td><td>SAC-ABR [68]</td></tr><tr><td>Deep Q-Learning</td><td>previous downloaded segment quality, buffer, quality-rate of next segment, channel capacity</td><td>1.2-1.74 %↑ in SSIM (image quality)</td><td>Belgium</td><td>D-DASH [151]</td></tr><tr><td>DRL (3D ConvNet)</td><td>bandwidth, buffer occupancy</td><td>9.3-12.9%↑ video quality than [55, 159, 155]</td><td>FCC, 3G/HSDPA</td><td>[169]</td></tr><tr><td rowspan="2">multiphase RL (online and offline)</td><td rowspan="2">aggregation of ABR algorithms (BOLA, MPC, Festive etc)</td><td rowspan="2">19.9% (VoD), 42.2% (live) QoE improvement</td><td rowspan="2">FCC,UCC[175], UCC5G[146], Belgium, HS-DPA, Oboe</td><td rowspan="2">ABraider [69]</td></tr><tr><td></td></tr><tr><td>Fuzzy logic + Dueling Double Deep Q-Network</td><td>Buffer occupancy, bandwidth, past bit-rates, download time, segment count and segment sizes</td><td>QoE gain of 23.1%, 23.97%, 33.42% than Linear, Log, and HD QoE models.</td><td>HSDPA [162]</td><td>FReD-ViQ [170]</td></tr></tbody></table>

excellent, personalized user experiences. Initially the fuzzy-logic models the network changes managing the vast dimensionality of the state space, which frequently obstructs learning-based algorithms. Then the double Deep Q-Network is enhanced by integrating a Dueling structure, adaptive noise injection and a sampling approach employing prioritized experience replay. FReD-ViQ strikes a balance between exploration and exploitation, allowing for quick response to dynamic environments. It yields enhanced QoE performance, assessed through the instantaneous perceived quality of every segment, quality fluctuations amongst video segments, and the rebuffering occurrences.

Table 6 presents a concise overview of the comparison of the recent ABR schemes. Table 7 summarizes the ML-based user-centric adaptive streaming approaches.

## 6.2. Intelligent streaming of 360° videos

For 360° video streaming, on tiling, segments with short duration are generated. The client can request tiles of varying quality levels in particular regions by fetching the relevant video segment files. In order to achieve flexible transmission, adaptive tiling techniques are used that not only improve responsiveness of the streaming system to viewport variations, but also boosts compression efficiency. In [176], based on the analysis that fixation point prediction error distribution is normal, Nguyen et al. predicted user's future viewport. Then, a tile selecting algorithm determined the most recent viewport region repeatedly without latency and picked a set of tiles to transfer encompassing the region. Rossi et al. [177] designed an adaptive streaming system that considers tile-dependent coded content and proposed an adaptation logic that can optimize the download rate of each tile on the client side based on user's navigational history. The optimization of tile-rate was formulated as an integer linear programming problem.

Effective tiling methods need to balance trade-off between tile size selection and coding efficiency.
First, tile size should be small, so that entire region of 360° views is covered without consuming addi-
tional pixels. On the contrary, tile size should not be too small such that the advantage of consuming
lesser pixels is over-weighed by reduced coding efficiency. The second trade-off is between employing
more tiles, which takes up more storage space but improves streaming efficiency against using fewer tiles,
which takes up less storage space but limits different tiling options during streaming. ClusTile [178]
approach picks a group of tiles that was encoded and saved on the server by resolving an integer linear
programming that was designed to approximately capture these trade-offs. However, the selected set of
tiles it generates may overlap and lacks a straightforward method for choosing tiles from this set which
reduces network bandwidth while streaming. In [179], Zou et al. suggested a rate adaptation technique
for users contending for server-side transmission capacity. The spherical viewport is mapped to planar
projection based on the viewport forecast, and the visibility probability of every tile is calculated for
each user. To reduce the experienced video distortion, the server chooses transmission tile-rate for each
user.

Ghosh et al. [180] encoded the visible tiles in higher resolution and the rest in lower resolution
depending on viewport data and fluctuating network constraints. They demonstrated that streaming
at different quality levels for visible and non-visible regions can improve the efficacy of the QoE metric
by ~20%. Xie et al. [181] deployed a probabilistic strategy to prefetch tiles inorder to reduce viewpoint
estimation error and developed a viewpoint adaptive system. Pano [182] suggests a tiling method with
tiles of different sizes to establish a good balance between perceptual video quality and efficiency with
which it can be encoded. Dividing the projected video into tiles, Pano aggregates smaller tiles into
bigger ones for encoding, guaranteeing that regions with comparable content are encoded together,
while those with substantially varied compressibility are encoded differently.

In [183], Shafi et al. decided the tiling strategy and resolution for streaming 360° video using multi-
tile arrangements. Within the adaptation interval, the size of the tiles were determined based on the
vast circular distance between the original and forecasted viewports. Bit-rates are subsequently assigned
non-uniformly to the viewport and non-viewport areas. Graf et al. [184] compared the efficiency of
various tiling patterns to that of conventional monolithic streaming. They analyzed that a 6 × 4 tiling
pattern (compared to 8×5, 5×3, 3×2, and 1×1) would offer a beneficial compromise between coding
efficiency and bandwidth usage for various categories of content. Additionally, they demonstrated that
they could achieve a considerable bandwidth reduction using a basic streaming method to stream a
given viewport’s data.

The upsampling models, specifically for super-resolution, have employed diverse networks such as
DNN [185], Deep CNN [186], Generative Adversarial Networks (GAN), and autoencoders. UNets [187],
ResNets [188], and DenseNets [186] have been deployed in this field of work. Each of these models
includes convolutional layers with several channels that can feed forward the residual information after
some layers. A super-resolution embedded ABR method, SR-ABR [185] incorporates content-induced
performance variation of super-resolution DNNs into bit-rate decision making processes. SR-ABR
employs DRL to determine future bit-rates, catering diverse network conditions. To effectively leverage
the content-induced performance variation of super-resolution-DNNs, the authors initially characterized
this variability across diverse video content and subsequently employed a 2D convolution kernel to
extract the features of performance variation of super-resolution-DNNs for a brief future video chunk
as one of their inputs. Super-resolution is an advantageous method to enhance the QoE for video
streaming.

In recent years, DRL has been used in bit-rate adaptation algorithms [151], [154]. The DRL agent
can acquire the bit-rate adaption policy through a series of exploration steps, with punishment and
rewards determined by the actions chosen. In [189], ML was employed for video bit-rate adaption.
Unlike the independent data samples in DL, the RL agents constitute strongly correlated states in
the sequential phases and do not need hand-labelled data. RL can be employed to determine the
ideal strategy for selecting the quality [190]. However, it is essential for the network to adhere to the
Markovian Property. PENSIEVE [154] employs a RL technique for dynamic streaming, although it is
specifically designed for conventional videos and is not well-suited for streaming tile-based 360° videos.
In [191], the HotDASH enhances video quality by pre-loading a segment with high temporal priority
for streaming. PARSEC [192] adopts a different methodology that decreases the amount of bandwidth
needed to transmit by utilizing a client-side computation approach with DNN model. The prediction
of FoV for 360° adaptive video streaming does not include the quality preference for tiled streaming in
[192, 193, 194].

In [195], Fu et al. proposed 360SRL, a Sequential RL (SRL) based ABR scheme for 360° videos.
Initially, the decision space of 360SRL was transformed from exponential to linear by implementing
a sequential ABR decision framework. Then, 360SRL generates ABR selections exclusively based on
prior QoE performance, not precise bandwidth estimates. A similar approach using DRL is adopted in
DRL360 [196], [197], [198], and [199]. The DRL360 model contributes to system performance improve-
ment by jointly maximizing several QoE criteria over a diverse collection of dynamic features. From the
observations acquired by client side video players, the DRL360 model adaptively distributes rates for
the tiles of next video frames. In [197], Kan et al. developed an Asynchronous Advantage Actor-Critic
(A3C) algorithm for adapting the spatial and temporal rates using a constant tiling method in which
the viewport estimation error was not considered. The authors in [198] proposed RAPT360, a DRL
approach for rate adaptation combined with adaptive prediction and tiling strategy for streaming 360°
video. They designed a buffer occupancy-based viewport recognition approach to identify a user's view-
port region, which could encompass the real viewport with any probable confidence level. They devised
a viewport-aware adaptive tiling method to make better use of bandwidth. Jiang et al. proposed RLVA
[199], RL-based Viewport-Adaptive 360° video streaming which improves the model performance in
viewport estimation, prefetch scheduling, and rate adjustment. They proposed a tile fetch scheduling
algorithm to update tiles based on the most recent estimation results, reducing the negative impact of
estimation error even further.

In [200], the authors propose QBAM, a QoE-fairness bit-rate allocation algorithm to overcome the
challenges arising out of difference in preference of individual users in choosing their bit-rates according
to their viewports. Firstly, a clustering technique is used to examine preference of viewers. Then a DRL
is used to train bit-rate allocation algorithm. Manfredi et al. [201] presents LSTM-based approach for
predicting viewports in immersive video systems. Through comprehensive evaluations, the LSTM-based
viewport prediction system demonstrates superior performance in accurately anticipating user patterns,
contributing to an improved overall immersive experience. It predicts user's point of interest by using
temporal dependencies within user interactions, making the response more responsive and immersive.
In [202], Islam et al. creates ML models that can be used by network operators to regulate real-time
QoE of virtual reality video sessions. In their proposed solution, packet level information is used as
data input for training the ML model to estimate target QoE. Through experimental test results in 4G
and 5G environment, the study establishes that the trained model provides output of reasonable accu-
racy and successfully predicts QoE for HTTPS and QUIC. The study in [203] presents content-driven
viewport predictor framework that is integrated with personalized federated learning methods and data
fusion techniques. First, a saliency detection model is framed, which relies on a Spherical Convolutional
Neural Network (SPCNN) for extracting salient regions from 360° video frames. Then an algorithm is
presented for head movement prediction which incorporates and increases the accuracy using person-
alized federated learning. Finally, the viewport prediction framework is designed, whereby the fusion
approaches are used to generate a fused feature map out of the saliency map and head orientation map.
To tackle the issues of faulty network estimations and inherent saliency bias that restricts enhancements
in QoE, Wang et al. developed a resilient saliency-driven quality adaptation framework for 360° video
streaming, RoSal360 [204]. They designed the model for predicting the transmission duration of video
tiles using a decoupled self-attention architecture and a DNN that considers the tile size. Also, they
developed an online correction approach that is driven by RL to effectively compensate for the incor-
rect quality allocations caused by saliency bias. Through comprehensive prototype trials conducted on
actual wireless networks, RoSal360 enhances video quality as well as minimizes rebuffering.

FBRA360 [205] presents Fuzzy-Based bit-rate Adaptation Scheme to deal with viewport prediction and bit-rate selection demanding situations in tile-based streaming. It uses a fuzzy based controller to control weighted mix of viewport from each user's history trajectory and multi-user attention distribution along with the buffer occupancy. In UVPFL [206], the viewport is predicted using Federated Learning based on user-profiles. Federated Learning is used to forecast the user's viewport by analysing their head movement patterns across various video categories. It utilises both the observed client head movements and the past viewport data to make predictions. The obtained viewports are then compared with either a similar user viewport or a historical viewport. The predicted viewport is then merged and updated if overlapping area exceeds a certain value. MOSAIC [207] presents an end-to-end video streaming system that predicts viewport and delivers tiles encoded at suitable bit-rates to optimize end user video quality within the network capacity. To forecast viewports, a CNN model integrated with LSTM is trained using motion, saliency maps, and client's head tracking data that learns both spatial and temporal features individually. Further, 3DCNN is used to capture the spatio-temporal features. DeepVR [208] provides a tile-based adaptive immersive video streaming system with DRL-agent that
can effectively adjust its bit-rate allocation strategy according to the changing environment. The FoV in the following seconds is predicted using attentive LSTM networks. DeepVR obtains a better QoE score through the incorporated DRL algorithm Rainbow, outperforming current panoramic video streaming systems. NOVA [73] is an effective Neural-Optimized Viewport Adaptive streaming system designed to enhance the QoE of users. In NOVA, initially a foveated rendering super-resolution method executes super-resolution operations at the tile level to convert low into high-resolution video tiles at the edge server. Then a meta-learning-based multi-agent RL algorithm is proposed to effectively learn video tile selection and super-resolution enhancement decisions to optimize the long-term user experience under prevailing network constraints. Comprehensive experimental findings indicate that NOVA significantly enhances average QoE while utilizing less bandwidth than existing methods.

In [209], a 360° live video ingest system, Vaser takes into account viewport information for neural enhancement and tile uploading. It considers viewport data to cut down on upload bandwidth requirements without compromising user experiences. It proposes an improved patch selection method to increase the super resolution model training efficiency and taps into DRL for setting upload stream bit-rate and super resolution model update frequency. The study in [210] introduces a two-stage approach to forecast the quality of virtual reality videos streamed over mobile networks. Initially, they predict the video playout performance metrics employing regression trees using network QoS indicators and video structure as input. Subsequently the obtained playout metrics are used to model and estimate the experienced video quality. The work in [211] proposes a subjective and objective framework for virtual reality QoE evaluation that can meet the requirement of real applications. The subjective evaluation component utilises four dimensions, i.e., the following scores: quality, immersion, non-spinning sensation and global score. The objective component employs an enhanced neural network, which was created by incorporating the characteristics of psychology and cognitive neurology.

The study in [212] proposes a spherical-CNN model designed for accurately predicting the long-term viewport in 360° videos. The network employs feature extraction to condense spatial-temporal 360° information. The work in [213] attempts to address the stringent criteria for video processing time as the delay caused by viewport prediction adds to live streaming latency. Also, it tries to avoid user video trails and user traces, as these data might not be available to construct the prediction model. So, they change the workflow of CNN (AlexNet) model and training/testing process to achieve this. Peng et al. [214] offers a DNN model based on spherical convolution to learn spatial aspects of 360° videos. The approach encodes distortion invariance into the CNN architecture. MAIVS [37] presents an adaptive UHD 360° immersive video streaming solution that leverages machine learning techniques. The solution aims to mitigate the data-rate demand associated with streaming high-resolution (UHD) videos. The process involves spatially dividing the 360° videos into MCT. After being downscaled, these tiles are then encoded using HEVC. A DNN is trained to enhance the resolution (at the client) of the encoded tiles by upscaling it. The encoded tiles, along with the model parameters, are packaged into mp4 containers at various quality levels. The DASH technique is employed to stream video tiles and model parameters progressively. The Deep Q-Network (DQN) is trained using feedback (like, video
quality parameter, buffer and network conditions) to carefully select the bit-rate quality segments.

The survey in [215] explores how to best deliver 360° videos on mobile devices, with a focus on edge caching and multicasting. A variety of caching namely tile based caching, collaborative caching, proactive and projection based caching can bring in optimization in streaming. Reinforcement based dynamic caching using ML is the most sophisticated of all. Furthermore, the research delves into the possible advantages of multicasting, which include lowering network latency and enhancing scalability while effectively delivering 360° videos to numerous users at once. In [216], Prabavathy et al. presents a real-time shot boundary detection system for live 360° virtual reality streaming using DL. The system automatically identifies transitions between shots, enhancing the seamless viewing experience in live VR environments. Its real-time capabilities ensure smooth adaptation to content that changes based on data and preferences, intensify the overall viewing experience. An innovative approach in [217] transforms the virtual reality experience by fusing 360° video streaming with omnidirectional olfaction technology. The immersive quality of 360° videos is improved through the seamless integration of scent delivery with both visual and audio stimuli. The system aims to enhance overall sensory engagement and capture viewers. Scents are discharged in the same direction as relevant objects appear.

The study in [237] presents an adaptive streaming method that combines two FoV prediction techniques to provide more accurate dynamic viewing area recognition, allowing for interactive tile choices. It uses priority-based bit-rate adaptation approach that enhances end-user QoE. [226] presents a content-based viewport predictor framework, integrated with personalized federated learning (PFL) methods and data fusion techniques. A saliency detection model is presented, which relies on a spherical convolutional neural network (SPCNN) for extracting salient regions from 360° video. An algorithm is presented for head movement prediction which incorporates and increases the prediction accuracy using PFL. Then, the viewport prediction framework is discussed, whereby the fusion approaches are used to generate a fused feature map out of the saliency map and head orientation map. Finally, the proposed framework is evaluated in terms of accuracy and precision metrics against existing viewport prediction algorithms.

Table 8 summarizes the ML-based 360° adaptive video streaming techniques along with IFs, major observations and used datasets.

# 7. Video Quality Datasets and Performance metrics

## 7.1. *Subjective video quality datasets*

This subsection discusses the commonly used video quality databases such as LIVE [31], VQEG HD3 [117], VQEG HD4 [117], LIVE Netflix [238], LIVE NFLX II [48], Waterloo [118], CSIQ [139], LIVE HTTP video streaming [122], LIVE Avaasi [135], and LFOVIA [123]. The video sequences in the datasets contain various natural scenes, movie clips, sports, animation, music, advertisements, documentaries etc. The databases cover a diverse set of videos with a broad range of Spatial Information (SI) and TI values. Table 9 lists the frequently used video quality databases with various distortions, video characteristics, and display device.
<table><caption>Table 8: Summary of ML-based 360° adaptive video streaming</caption><thead><tr><th>Category</th><th>ML Technique</th><th>Influence Factors</th><th>Datasets used</th><th>Major observation/ application</th><th>Accuracy</th><th>Reference paper</th></tr></thead><tbody><tr><td rowspan="10">360° video streaming</td><td>Deep Q-Learning</td><td>network throughput, buffer status, segment PSNR</td><td>[218]</td><td>mitigate data-rate demand for streaming UHD 360° videos</td><td>8.52%,48.18% lowered bit-rate segments than PARSEC[192] & [154]</td><td>[37]</td></tr><tr><td>multi-agent RL</td><td>user viewpoint trajectory & preference for video quality, buffer time, quality switching</td><td>3G/HSDPA, [219]</td><td>obtaining QoE equitable bit-rate allocation for 360° video transmission</td><td>quality difference between users is 6-7</td><td>[200]</td></tr><tr><td>LSTM</td><td>time samples, viewport trajectory, playout buffer</td><td>Avtrack [220],[221, 222, 223, 224]</td><td>addresses critical challenges of viewport-adaptive streaming, showcasing efficacy of LSTM</td><td>prediction accuracy ~60%</td><td>[201]</td></tr><tr><td>Tree-based methods, DNN & K-NN</td><td>Throughput, #packet interarrival time, bit-rate of streaming, stall time, startup delay, quality switch</td><td>[225]</td><td>predict QoE using encrypted network level QoS data</td><td>RMSE of 0.09</td><td>[202]</td></tr><tr><td>spherical CNN+ Federated Learning</td><td>video frames, head movement measures</td><td>[203]</td><td>content-driven viewport prediction based on saliency & head orientation maps</td><td>4.35%, 5.88%, 7.46%↑ than existing</td><td>[226]</td></tr><tr><td>fuzzy logic controller</td><td>Buffer occupancy, throughput deviation, user behaviour bias</td><td>SalientVR [227], [228]</td><td>Establishes bidirectional dependency between bit-rate selection & viewport prediction</td><td>video quality 7.5%↑ than existing</td><td>[205]</td></tr><tr><td>Sphere U-Net, LSTM</td><td>spatial features, user's trajectory, saliency map</td><td>[203]</td><td>saliency network embedded with FoV prediction framework for better prediction</td><td>33-71% relative LCC↑ in saliency detection</td><td>[214]</td></tr><tr><td>Federated Learning</td><td>Tiles of current frame, saliency data</td><td>[229]</td><td>Improves viewport prediction in absence of historical data</td><td>90% (viewport prediction), 1.12-64.9%↑ than existing</td><td>[206]</td></tr><tr><td>Deep Learning</td><td>viewport info, throughput, ratio of received frames, SR gain</td><td>[230, 231],[232], FCC</td><td>reduces 360° live video upload bandwidth needs, without affecting quality</td><td>bandwidth need 40-55.6%↓, utility 1.15-3.61× ↑</td><td>[209]</td></tr><tr><td>DRL</td><td>Throughput, remaining chunks & tiles, rebuffering time, next tile size & viewing probability, bit-rate</td><td>[162],FCC, [173],[229]</td><td>optimizes streaming in viewport prediction, prefetch scheduling, & rate adaption</td><td>4.8-66.8% ↑ in QoE than existing</td><td>[199]</td></tr><tr><td>CNN (ResNet 101)+LSTM & 3DCNN</td><td>saliency, motion map, head tracking trace</td><td>[229], 4G/LTE [173]</td><td>combines neural network-based viewport prediction & tile-rate allocation, optimizing video quality</td><td>47-191% ↑ video quality than existing</td><td>[207]</td></tr><tr><td>Decision Trees</td><td>delay, packet loss, throughput, startup delay, quality, quality switches count, video stalls</td><td>[219], [173]</td><td>predicts perceived quality using network QoS indicators & video playout performance metrics</td><td>predicts perceived quality with 4% ↓ prediction error</td><td>[210]</td></tr><tr><td>Improved Neural Network (INN)</td><td>bandwidth, latency, packet loss, quality score; immersion, non-spinning & global score</td><td>own [211]</td><td>predicts quality using no-reference two-stage improved neural network</td><td>8.9%↑ in PLCC than traditional NN</td><td>[211]</td></tr><tr><td>spherical CNN</td><td>user's viewing history, head motion info</td><td>-</td><td>long-term viewport prediction using spherical-CNN</td><td>40-46% viewport prediction accuracy</td><td>[212]</td></tr><tr><td>CNN (AlexNet)</td><td>Tiles, head movement traces</td><td>[219]</td><td>viewport prediction accuracy with low bandwidth & low timing overhead</td><td>70-88% prediction accuracy</td><td>[213]</td></tr><tr><td>attentive LSTM, DQN</td><td>viewpoint trajectory, past throughput, chunk size, buffer, bit-rate, remaining chunks, download time</td><td>[233], [162], [175], [234]</td><td>Tile-based adaptive streaming with DRL policy adjustment</td><td>16%↑ in QoE than existing</td><td>[208]</td></tr><tr><td>multi-agent RL</td><td>downlink throughputs, tile dimensions for prior tile downloads, download durations, viewport trajectories</td><td>[173], [235], [196]</td><td>optimized tile-selection and edge-assisted super resolution enhancement for maximizing QoE and adapting to network changes</td><td>enhances user-perceived QoE up to 27%</td><td>[73]</td></tr><tr><td>DNN+RL</td><td>tile throughput, tile size, saliency-based behavior prediction & viewport prediction accuracy, buffer occupancy,time before playback for tile,tile's TTP result,TTP confidence</td><td>[236], [230]</td><td>saliency-based video quality adaptation framework that incorporates tile-size-aware transmission time estimation DNN model and a saliency-aware quality distribution for mobile video streaming</td><td>enhances QoE (31.45% MOS gain on average)</td><td>[204]</td></tr></tbody></table>

The damaged video sequences for every source video in the LIVE dataset [31] are acquired using the following techniques: i) There are 4 test videos that use H.264 compression. The compression
<table><caption>Table 9: Subjective Databases Overview. SV: Source Videos, DV: Distorted Videos, TE: Transmission error, cont.: continuous</caption><thead><tr><th>Database</th><th>Resolution</th><th>#SV</th><th>#DV</th><th>Duration (sec)</th><th>Scores range</th><th>Distortion Type</th><th>Display Device</th><th>Rating type</th></tr></thead><tbody><tr><td>LIVE [31]</td><td>768×432</td><td>10</td><td>150</td><td>10</td><td>[0,100]</td><td>H.264, MPEG 2, TE</td><td>CRT Monitor</td><td>overall</td></tr><tr><td>VQEG HD3 [117]</td><td>1080p</td><td>13</td><td>155</td><td>10</td><td>[0,5]</td><td>H.264, MPEG 2, TE</td><td>Samsung TV</td><td>overall</td></tr><tr><td>VQEG HD4</td><td>1080i</td><td>13</td><td>155</td><td>10</td><td>[0,5]</td><td>H.264, MPEG 2, TE</td><td>LG TV</td><td>overall</td></tr><tr><td>LIVE Netflix [49]</td><td>1080p</td><td>7</td><td>112</td><td>>60</td><td>[0.38,4.97]</td><td>H.264,scaling, freezing</td><td>Mobile</td><td>cont.</td></tr><tr><td>LIVE NFLX II [48]</td><td>1080p</td><td>15</td><td>420</td><td>≥25</td><td>[0.19,4.9]</td><td>H.264,scaling, freezing</td><td>Computer monitor</td><td>cont.</td></tr><tr><td>Waterloo [118]</td><td>1080p</td><td>20</td><td>180</td><td>≈ 15</td><td>[0,100]</td><td>compression,TE</td><td>LCD Monitor</td><td>overall</td></tr><tr><td>CSIQ [139]</td><td>832×480</td><td>12</td><td>216</td><td>10</td><td>[0,100]</td><td>compression, TE</td><td>LCD Monitor</td><td>overall</td></tr><tr><td>LFOVIA [123]</td><td>2K, 4K</td><td>18</td><td>36</td><td>120</td><td>[0,100]</td><td>encoding,freezing</td><td>CES Android app</td><td>overall+cont.</td></tr><tr><td>Mobile stall II [50]</td><td>1280×720, 1280×640</td><td>24</td><td>174</td><td>29-134</td><td>[19.12, 75.82]</td><td>freezing</td><td>Mobile</td><td>overall+cont.</td></tr><tr><td>LIVE Avaasi Mobile [135]</td><td>1280×720, 640×360</td><td>24</td><td>180</td><td>29-134</td><td>[0,3]</td><td>stalling+startup delays</td><td>Apple iPhone</td><td>overall</td></tr><tr><td>LIVE HTTP video streaming [122]</td><td>720p</td><td>3</td><td>15</td><td>300</td><td>[30,70]</td><td>quality switches</td><td>TV</td><td>overall+cont.</td></tr><tr><td>MCQoE [147]</td><td>360p-2160p</td><td>7</td><td>14</td><td>60</td><td>[0,100]</td><td>H.264, TE, freezing</td><td>TV,PC,HD-phone</td><td>overall+cont.</td></tr></tbody></table>

rates for these videos range from 200 Kbps to 5 Mbps. Additionally, there are test videos that use 'MPEG-2' compression with rates ranging from 700 Kbps to 4 Mbps. ii) There are three IP test videos (using RTP) created by intentionally dropping packets in a particular error pattern specified by the Video Coding Experts Group (VCEG) and have loss rates of 3%, 5%, 10%, and 20% from the H.264 compressed videos. iii) There are four test videos created by transmitting H.264 compressed videos over error-prone wireless networks. The errors in these networks are simulated using bit error patterns resulting in packet error rates ranging from 0.5% to 10%.

The videos in VQEG HD3 [117] are subjected to compression artifacts caused by differences in framerate, bit-rate, and codec type, as well as transmission artifacts. Transmission error in IP networks (via User Datagram Protocol (UDP)) encompass packet losses that occur randomly, bit errors, and variations in packet delay. The packet loss rates (PLR) are 0.015%, 0.024%, 0.035%, and 0.3%. The VQEG HD4 dataset [117] includes similar impairments such as compression and transmission artifacts, as well as varying levels of packet losses. However, VQEG HD4 had varying viewing conditions, including differences in monitor size, dot pitch, calibration process, refresh rate, and bit depth, detailed in [117].

In [49], the LIVE Netflix dataset consists of streaming videos that contain Netflix content and video sequences. The distortions in these videos are caused by H.264 compression, stalling, and a mix of both. They created eight playout patterns, with one of these patterns assuming sufficient bandwidth to allow the client to play out content at a consistent rate (assuming no network impairments). Rest of the patterns replicate severe impairments by taking into account the depletion of the buffer. It considers
both impairment free as well as impaired network conditions.

In LIVE NFLX II [48], the streaming scenario was modeled by four main modules: encoding, video quality, network transmission, and client. The encoding module creates by determining the encoding parameters, like resolution and QP, using an optimization framework. The video quality module assessed the quality of the video. The network transmission module accounted for the effects of varying network conditions. The client module requested the next video chunk to be played using four ABR algorithms. In order to quantify the effects of varying network circumstances, a selection of seven network traces was made from the HSDPA dataset, which contains authentic 3G traces collected during various travel journeys throughout Norway. The traces encompass network behaviors that range from low to high bandwidth conditions, which can lead to rapid fluctuations in bit-rate or quality and buffering. ABR algorithms effectively measure the influence of adjustments on the client side QoE, a factor that is absent in other datasets.

The Waterloo dataset [118] includes streaming videos' quality, where the videos are encoded using the H.264 codec and have three various bit-rates (i.e., 500, 1500, and 3000 Kbps) to achieve varying quality levels. Stalling events, each lasting five seconds, were added at the beginning or middle of these encoded sequences. In CSIQ dataset [139] there are a total of eighteen categories that have been distorted, and within these categories, there are six different kinds of distortions. The distortion kinds include four video compression methods (H.264, Motion JPEG, HEVC, and wavelet compression) and two transmission-based methods (e.g., packet loss in a wireless network and additive white Gaussian noise). It encompasses a variety of frame rates, including 24, 25, 30, 50, and 60 frames per second (fps).

The LFOVIA dataset [123] comprises nine source videos, each in Full HD and Ultra HD resolution. The collection of 36 videos exhibits distortions that result from rate adaption and rebuffering. The rebuffering frequency ranges from 0.5 to 5 occurrences per minute. Bit-rate adjustments are made both upward and downward, wherein the video rate is altered to a higher or lower level. The subjective scores obtained from LIVE and CSIQ was in the form of DMOS, whereas rest of the databases were in terms of MOS. The subjective scores in various databases are in different ranges.

The Mobile stall II [50] comprises a collection of 174 streaming videos on mobile devices, featuring 26 distinct stalling patterns. The patterns include duration, position, and recurrence rate of stall occurrences. It contains videos with quality varying over time during the playback. It has both per-frame continuous as well as overall QoE scores.

The LIVE-avaasi [135] is a mobile video dataset that simulates network impairment distortions which includes stalling events and start-up delays. It replicates realistic stalling patterns by altering multiple QoE-influencing factors, including the location, frequency, and duration of the stalls as well as the kind of video content on end users' QoE. It has 24 source videos with 180 distorted videos having 26 distinct stalling patterns and 4830 opinions gathered from 54 individuals.

The LIVE HTTP [122] dataset facilitates the creation of TVSQ prediction models for HTTP-based video streaming. To mimic the quality changes observed in HTTP-based streaming, the STSQs of the videos were created to fluctuate randomly across time scales spanning multiple seconds. It has 15
<table><caption>Table 10: Overview of databases for different network conditions</caption><thead><tr><td>Dataset</td><td>Description</td><td>Network</td><td>Download url</td></tr></thead><tbody><tr><td>FCC [171]</td><td>Raw data collected from fixed ISPs in US. It includes over 1 million throughput measures with average throughput at 5-second intervals.</td><td>Broadband</td><td>[171]</td></tr><tr><td>HSDPA [162]</td><td>HDSPA gathers 30-minute uninterrupted video streaming throughput data from mobile devices throughout several Norwegian routes utilising various modes of transportation (e.g., vehicle, bus, train)</td><td>3G</td><td>[162]</td></tr><tr><td>UCC [175]</td><td>It includes 4G traces of client-side cellular KPIs from two Irish mobile operators, covering various mobility patterns. It contains 135 traces with an average time of 15 minutes and observable throughput from 0 to 173 Mbit/s at one sample per second.</td><td>LTE</td><td>[239]</td></tr><tr><td>UCC5G [146]</td><td>It contains 5G traces gathered from Irish mobile operator, that includes static and mobility patterns for video streaming and file download. It comprises of client-side cellular KPIs consisting of channel, context, cell-related metrics and throughput information</td><td>5G/ LTE</td><td>[240]</td></tr><tr><td>Belgium [173]</td><td>This report includes 40 throughput traces for 4G networks in Ghent, Belgium, collected using 6 transportation modes over 5 hours of monitoring. The bandwidth ranged from 0 to 111 Mbps, averaging 30.3 Mbps ± 16.7 Mbps</td><td>4G/ LTE</td><td>[241]</td></tr><tr><td>Oboe trace [172]</td><td>The throughput traces contain chunk sizes and their download times for on-demand video sessions. Traces include desktop sessions with wired and mobile device sessions with WiFi or cellular connections. Around 5K traces were collected from wired PCs and 4K traces from WiFi or 3G/4G mobile devices.</td><td>WiFi/3G/ LTE</td><td>[242]</td></tr><tr><td>E2E dataset of video streaming [243]</td><td>It presents a variety of comprehensive metrics, referred as Key Quality Indicators (e.g., number, frequency, duration of stalls) for evaluating the end-to-end performance of video streaming and cloud gaming across various network technologies</td><td>LTE/5G/ Ethernet/ WiFi</td><td>[244]</td></tr></tbody></table>

videos, each 5 minutes long, and viewed by 25 participants.

The MCQoE [147] is a Multi-device Continuous QoE dataset which is publicly available. It has over 76,000 continuous and 1260 overall scores collected from 60 participants. It incorporates artificial glitches, like rebuffering events and fluctuations in quality, to replicate the video streaming experience. The ratings are collected across several devices, i.e., 75-inch UHD TV, 24-inch QHD PC, and 6.1-inch HD smartphone.

Table 10 summarizes the publicly available databases that cater to various network conditions.

## 7.2. HMD, Eye-tracking and Viewport traces Datasets

In [218], the dataset contains head movement of users recorded from 59 people while watching 360° videos each of duration 70 sec. The chosen videos cover a broad range of 360° information, thus varying user engagement and navigation patterns. The user's head position log files contain the timestamp, frame ID, and unit quaternion values that are used to determine the user's head position. The dataset in [219] has data records from 48 people that viewed eighteen 360° videos across 5 categories. The way viewers watch the videos, their head movement during each session, where they focus, and what they can recall are all recorded following every session. The log files comprises of the timestamp, playback time, unit quaternion values and position of the HMD device. It contains demographic profiles and records of user behavior.

Avtrack [220] presents visual assessment of 48 participants who viewed 20 distinct and interesting 360° videos. As the subjects watched the contents, their head movements were captured. It gives the participants' exploration activity by providing the angular ranges they covered and an analysis of the specific regions where they spent most of their time. The gathered data can also be displayed as head-saliency maps. The available data includes HMD's current tracking location, current head rotation, video playback time along with timestamp. In [221], the dataset provides an eye-tracking recordings
<table><caption>Table 11: Overview of databases for 360° videos containing records of head movements, eye tracking and viewport traces. FR/SF: framerate/ sampling frequency</caption><thead><tr><th>Dataset</th><th>Description</th><th>Subjects</th><th>No. of videos</th><th>Resolution</th><th>FR/SF</th><th>duration (sec.)</th><th>Watching device</th><th>Download url</th></tr></thead><tbody><tr><td>[218]</td><td>head movement of users</td><td>59 (48 males, 11 females)</td><td>5</td><td>3840×2048</td><td>25, 30, 60 fps</td><td>70</td><td>Razer OSVR HDK2 HMD</td><td>[245]</td></tr><tr><td>[219]</td><td>head movement of users, records of user behavior</td><td>48 (24 males, 24 females)</td><td>18</td><td>2560x1440</td><td>25, 29, 30 fps</td><td>146-506</td><td>HTC Vive headset</td><td>[246]</td></tr><tr><td>Avtrack [220]</td><td>HMD's current tracking location, current head rotation, video playback time along with timestamp.</td><td>48 (25 female, 23 male)</td><td>20</td><td>4K</td><td>-</td><td>30</td><td>HTC Vive HMD</td><td>[247]</td></tr><tr><td>[221]</td><td>eye-tracking recordings i) Gaze point's x and y coordinates ii) head direction coordinates and its tilt.</td><td>13</td><td>14</td><td>upto 4K</td><td>120 Hz</td><td>60</td><td>FOVE VR headset+ eye tracker</td><td>[248]</td></tr><tr><td>[222]</td><td>viewport traces and their heatmap, response by the subjects to questions</td><td>60 (17 female, 43 male)</td><td>28</td><td>upto 3840x2160</td><td>29, 30 fps</td><td>60</td><td>Oculus Go HMD</td><td>[249]</td></tr><tr><td>[223]</td><td>navigation trajectories using several viewing platforms</td><td>94 (29 females, 65 males)</td><td>15</td><td>2560×1440</td><td>24, 25 30 fps</td><td>20</td><td>Oculus Rift HMD, Alienware15 Laptop, Apple iPad Pro10.5 tablet</td><td>[250]</td></tr><tr><td>[203]</td><td>ground truth (original) gaze points of the observers</td><td>27</td><td>104</td><td>upto 4K</td><td>-</td><td>20-60</td><td>HTC Vive HMD+ 7invensun a-Glass eye tracker</td><td>[251]</td></tr><tr><td>[227]</td><td>gaze-annotated dataset that would help in attention behaviour analysis.</td><td>30 (14 female, 16 male)</td><td>20</td><td>4K</td><td>30-60 fps</td><td>120-660</td><td>HTC Vive Pro Eye VR headset</td><td>[236]</td></tr><tr><td>[230]</td><td>eye tracking dataset</td><td>45(20 females, 25 males)</td><td>208</td><td>4K</td><td>25 fps</td><td>20-60</td><td>HTC Vive HMD+ 7invensun a-Glass eye tracker</td><td>[252]</td></tr><tr><td>[253]</td><td>more than 3700 viewport traces</td><td>-</td><td>88</td><td>upto 4K</td><td>10 Hz</td><td>30-655</td><td>HMD</td><td>[254]</td></tr></tbody></table>

for both real-world 360° videos and a synthetic video clip. The recordings have a comparatively high frequency of 120 Hz, making it easier to infer eye movements. Details include i) Gaze point's x and y coordinates (in equi-rectangular coordinate) throughout the 360° video surface ii) The head direction coordinates and its tilt.

The dataset in [222] comprises viewport traces collected from 60 people who watched the 360° videos. Furthermore, it offers feedback of the viewers on their experience following the viewing of each video. The log files include timestamp, the quaternion components representing rotation of the viewport, and cartesian coordinates of the vector pointing towards the center of the user's viewport. The dataset also includes response given by the participants to the questions, heatmap of viewport traces, and pitch and yaw angle histograms for every video. In [223], it offers navigation trajectories obtained for heterogeneous 360° videos using several viewing platforms. The subjective experiments were conducted at Trinity College Dublin and University College London. It examines the behavior of users across different content and viewing devices. Initially, analysis is conducted using metrics like angular velocity and viewport center distribution that emphasizes the extent to which the user's navigation is influenced by display device. Next, the similarity among users is examined based on the viewport that is displayed over a period of time.

In [203], the dataset comprises of the ground truth (original) gaze points of the observers, collected from more than 20 subjects for 104 video clips. The 360° source videos are collected from Sports-360 dataset (contents like basketball, skateboarding, parkour, BMX, and dance) that have duration
of 20 to 60 sec. The HMD has an integrated eye tracker that records the participants' eye fixation positions while they watched the videos. The dataset in [227] presents a gaze-annotated long 360° video dataset that would help in attention behaviour analysis. The dataset is further processed as follows for more practical applications: i) The missing values (produced by eye closure) are handled using moving average interpolation ii) data is transformed from millisecond level into frame level iii) the head directions' quaternion coordinates are changed to 2D plane coordinates.

Xu et al. [230] presents an extensive eye tracking dataset for dynamic VR scenes. There are 208 HD 360° videos (resolution-4K, 25 fps) of duration 20-60 sec. in the collection, and at least 31 participants as viewers. The dataset analysis reveals that contents of the image and history of the scan path influence gaze prediction. It includes a wide range of video content, including , including indoor and outdoor scenes, music, sports etc. A combined dataset in [253] consists of 88 videos of duration ≥ 30 sec. and includes more than 3700 viewport traces, which contain a total of more than 142 hours of watching records. Various formats of distinct datasets are merged such as head orientation, trace duration, and data sampling rates. Yaw and Pitch angles were expressed in radians and roll angle changes were ignored. To address the disparities in sampling rates, the complete dataset was re-sampled at 10 Hz.

Table 11 lists the various publicly available datasets for 360° videos containing head movement directions, viewport traces, and eye tracking records.

8. Open Challenges

User-centric multimedia streaming systems have played a pivotal role in enabling convenient access to multimedia content, while improving the service quality. Although several issues pertaining to it have been addressed, still there are some issues that need to be considered for future research.

1) *Subjective Quality Assessment:* The developed subjective databases used as ground truth to predict the video quality ratings have design limitations. Conventional subjective video quality databases are usually created by initially choosing a limited number of excellent sources, known as “reference”, and subsequently artificially altering them using simulated methods. These databases have little variation in their content characteristics and distortions, which makes them unable to accurately replicate complex impairments in real-world user created videos. All visual media share fundamental principles of perception. However, to comprehend artifacts and distortions specific to a given domain, it is necessary to create subjective databases specifically designed for each domain. Constructing extensive, diversified, and impartial subjective video datasets is still a crucial area of study for researchers. Additionally, because of the small size, the publicly available subjective databases devoted to video quality are unable to fully utilize the potential of DL techniques.

Immersion is a relatively new QoE attribute. Currently, immersion does not have a standardized
quantitative quality measurement index. Various IFs and the intricate interplay between them could
be contributing factors. The available subjective assessments primarily address visual quality, presence
perception, physical comfort, and multisensory integration. Subjective assessment challenges of 360°
videos include need to establish test procedures for subjects with different viewing device character-
istics, lack of unified rating scale, limited well-defined objective metrics that can take into account
the diverse quality of omnidirectional videos, difficulty in assessing quality degradation by taking into
account eye movements across the viewport, different content characteristics, and impact of network
impairments on quality ratings. Furthermore, the significant factors to consider are cybersickness and
spatial presence, which arise from watching 360° videos using VR headsets. These effects are exclusive
to VR experiences and do not manifest when users watch normal videos on flat screens.

2) *Objective Quality Assessment:* Traditional objective VQA metrics are not suitable for evaluating the 360° videos as the quality is significantly affected by network conditions, end-user viewing behavior, and current 2D planar projection methods that do not provide a uniform sample density at each pixel position. The quality metrics for 360° videos involve remapping based on the relevant projection format. The experimental analysis demonstrates that the present objective metrics for 360° videos (e.g., [9], [7]), when compared to the ground-truth quality exhibit less correlation with the quality perceived subjectively. Hence, there is a strong requirement for an optimal quality metric that can demonstrate a higher level of correlation with the ground truth, specifically tailored for 360° content.

3) *Ensuring Visual Comfort:* The majority of the studies focus on utilizing an adaptive architecture determined by the viewport and distribution of quality throughout the entire image. Nevertheless, present research studies rarely prioritize the aspect of visual comfort. The user's enjoyment with 360° video content is more affected by disturbance while using a headset compared to when using a traditional display. It is vital to lighten the weight of the head mounted device, making it more comfortable to wear. Additionally, decreasing cybersickness also plays a significant part in enhancing visual comfort. A number of viewers have claimed to experience fatigue, motion sickness, and nausea while using VR. This has become one of the factors contributing to users' reluctance to embrace VR technology. The user experience must be prioritized while evaluating VR video streams. Research indicates that type of content (e.g., fast-moving, resolution), high usability, and presence ratings can increase the likelihood of experiencing cybersickness. It is therefore imperative to investigate ways to lower the prevalence of cybersickness. Watching VR using HMDs adds factors like display configurations and rendering delays, which greatly affect QoE. Existing QoE models and conventional objective metrics, such as PSNR, fail to capture such factors. Therefore, it is crucial to create accurate mathematical models that are specifically designed for user characteristics in order to optimize individualized QoE frameworks. This area necessitates additional investigation in future VR studies.

4) *User interaction in AR:* When users interact with the real-world environment via AR technology, certain objective quality factors are taken into account. Some of these factors include the rate at which tasks are completed, the accuracy of interactions, and the time it takes for users to respond. User interaction (e.g., gesture, speech, and movement commands) is critical to the user's overall QoE. Nevertheless, there isn't a reliable and standard metric to gauge how the users felt about the interaction. During the interaction, there is no way to get their feedback. Only [107] used interaction in the objective QoE forecast. Unfortunately, this problem has received minimal attention. Particularly in AR
scenarios, there remain unresolved issues, such as the precise alignment of virtual objects with actual objects and the need to minimize both the rate of interaction errors and any delays that may occur.

5) *QoE modeling and Complexity Analysis:* There is a lack of a comprehensive QoE model that accurately defines the effects of each influence factor in an adaptive streaming system. The discussed QoE predictor models can be extended to include additional complex metrics and transmission/ streaming parameters as input features, but this can cause the complexity to increase. In comparison to the training set, the domain of potential inputs to the ML-based QoE models is much larger. It is still very difficult and mostly unsolved to find accurate and efficient VQA models that can make estimations for streaming services almost in real time.

Additionally, the study of perceived quality in VR using different senses, like olfactory, tactile, and taste, is still extensively unexplored. Broadly, these sensations and emotions are difficult to quantify. There is a lack of a complete theoretical framework/model that considers and evaluates how people feel about these emotions. Precise evaluation of QoE is a necessary requirement for adaptive streaming solutions. It is a crucial element in maximizing the efficiency of 360° video streaming service. So, there is a strong need for more systematic study in order to develop appropriate models and metrics for assessing the QoE in 360° videos that are widely accepted.

Most of the studies lack discussion on the complexity of QoE model. Although demonstrating exceptional performance, the contemporary DL-based QoE models, necessitate substantial floating-point computations. Therefore, they are not practical or efficient to be widely used in their present state. Implementing complex QoE models can result in decreased application performance due to higher power consumption and greater use of computing resources. Further, these models can be used to provide QoE feedback based solutions to achieve optimized resource allocation and efficient bandwidth utilization. The end user is solely a consumer of the video service and lacks the ability to exert any influence, although there might be a possibility of including direct feedback in the future. However, client-side monitoring always has risk of privacy invasion. Although a number of ML models have been used for QoE evaluation and ML-based streaming techniques have been developed, the selection of the most appropriate ML model for a specific application is still a topic of ongoing research.

6) *Storage, transmission and rendering of VR content:* VR provides a more realistic viewing experience compared to the usual way of watching images and videos on phones, TVs, and computers with flat screens. The viewer is free to observe it in any direction due to the image's ability to occupy the full viewing space. At any instant, they can gaze at only a small part of the image. Their visual perception is determined by their visual attention, spatial arrangement of the image content, and the object that garners their attention. Unrestricted access to high-resolution, immersive VR entails a substantial amount of data. This presents difficulties in terms of storage, transmission, and rendering of images, potentially impacting the quality of the watching experience. The VR applications require bigger file sizes, a variety of storage formats, and immersive viewing conditions. This creates substantial challenges in obtaining, compressing, transmitting, and presenting high-quality VR video.

7) *Open source tools and Applications:* Various open source tools such as Avtrack360 [220] and Open-
Track [255] are capable of capturing the head traces of users viewing 360° videos, ALTRUIST [256] can conduct subjective QoE assessment tests, Unity3D-based app [257] is available for 360° VR quality measurement. There is still a need for a variety of tools and applications that can capture head traces, conduct VR subjective tests, and measure the quality of VR content. These should be made openly accessible to the public. The tools and apps must be user-friendly, and their accuracy must be ascertained.

8) *Viewport Prediction*: There is a requirement to address additional concern associated with delivering immersive 360° video content, such as ensuring correct prediction of the viewer's viewport over an extended period of time. Moreover, there is a perpetual query regarding the accuracy of viewport prediction. If video tiles are requested based on an incorrect estimate, the viewer's actual viewport may be obscured by black tiles for which no content has been requested. The viewer's movement exhibits significant volatility while watching certain portions of the video, which puts pressure on the training of ML models. The majority of viewport prediction approaches prioritize examining saliency patterns in addition to location information. In order to accurately forecast areas of attention and comprehend the correlation between the user's watching preference and saliency maps, it is necessary to enhance and properly train saliency models using extensive datasets, particularly those obtained from various camera rotations. However, since various motions may compromise the reliability of predictions, it is also necessary to examine the motion maps. There is a need to explore deep attention-based architectures and the involved computational complexity to improve the integration of multiple modalities (e.g., history of past viewings, video content) that vary in time and space. The viewport prediction models involve additional computational complexity, and most of the studies lack analysis on the computational complexity.

9) *Intelligent and adaptive multimedia streaming solutions:* In a highly dynamic environment, the throughput estimation might not be very accurate. The proposed intelligent and adaptive video streaming algorithms do not consider errors caused by these incorrect estimates. All the heterogeneous aspects of the end-users like device characteristics (e.g., display resolution, battery constraints) can also be considered for more efficient user-centric streaming solutions. For efficient video streaming, the timely delivery of videos is of utmost importance, as it is significantly impacted by various factors such as dynamic network conditions and restricted transmission resources. Poor network conditions result in increased round-trip latency, which impacts the perceived quality [258]. Perceptual QoE driven resource allocation strategies can effectively meet the demands of video streaming users without compromising the viewing experience. There is often a lag between buffering and playback in the events of poor network condition. The effect on QoE due to the rebuffering events with the mentioned lag needs to be evaluated to study the extent of variation in lag between buffering and playback. Furthermore, adaptation strategies can be developed in the streaming session considering the lag factor. The different heterogeneity aspects of the end users (e.g., display resolution, battery constraints) should be considered for developing energy-efficient streaming solutions for diverse multimedia content and applications. Despite the emergence of several hybrid ML frameworks (such as integration of fuzzy logic with RL
[170]) for intelligent and adaptive streaming, most studies [170, 204, 207] lack performance evaluations on runtime analysis and GPU utilization.

10) *Low latency multimedia streaming:* Multimedia streaming services, particularly 360° and VR/AR videos, necessitate minimal response latency, encompassing network, buffering, edge processing, request overhead, interaction, and feedback delay, all of which impact the video quality. With variation in user head movements, the brain anticipates immediate updates in auditory and visual content at different viewing angles. Consequently, ultra-low endpoint processing latency, minimal network latency, rapid edge computing are essential to achieve this degree of responsiveness to viewer head movements. In viewport based adaptive streaming, ultra-low network latency must be guaranteed to immediately provide the viewport due to the viewer's constant interaction via end-user devices. The caching and multicasting techniques mostly focus on optimizing network-level parameters (latency) [215], instead of optimizing the experienced perceptual quality. To mitigate latency in multi-user VR video streaming, it is essential to develop QoE-aware DL-assisted multicast framework. The next 5G and 6G networks are supposed to meet the substantial need for immersion, low latency, high capacity, highly reliable transmission, and real-time requirements in VR/AR application services [259].

11) *Efficient multimedia streaming solutions:* To enhance the streaming efficiency, it is viable to optimize the extent of viewport coverage and surrounding regions by dynamically adjusting them according to the user's head motions and prediction errors. In order to reduce the data rate requirement, upscaling the resolution of 360° video segments on the client side from the downsampled ones result in loss of information that can impact the quality. The discussion persists on how to achieve a trade-off between minimizing information loss while simultaneously reducing the data rate. In tile-based streaming, tiles with varying quality based on user preferences are selected to maintain balance between quality and required bandwidth. Several factors influence this balance and further efforts are required to enhance QoE and minimize bandwidth usage. Enhanced design insights must be taken into account for the dynamic selection of tiles, with optimal allocation of bandwidth per tile within the framework of prioritized 360° video delivery. Investigating the impact of variable size tiling on streaming performance and its impact on viewing experience with associated costs is another important issue. Priority based high resolution 360° tiles, can be transmitted along the optimal available path in the event of multi-path transmission to enhance perceptual experience and transmission flexibility. With a receiver-centric design, numerous users are anticipated to view several parts of the identical content. It is vital to enhance the resolution without increasing latency of the system. In order to achieve low latency, it is necessary to develop efficient compression techniques for 360° data units, which is still an open challenge. New encoding techniques have to be developed to achieve better compression efficiency and have faster representation switches, hence providing reduced latency and computational expense.

## 9. Conclusion

This paper has reviewed the solutions pertinent to user-centric multimedia streaming, especially in modeling accurate video QoE predictors (overall as well as continuous and time varying), efficient
(intelligent and adaptive) multimedia streaming using user-centric feedback, and intelligent 360° video
streaming to reduce the data rate requirement while enhancing the immersive experience. It includes
information on how to prepare, process, and transmit the video content to end-user display devices,
such as VR headsets, mobiles, smartphones, and monitors. Typically, the end users desire an acceptable
QoE. Additionally, they seek an application that is user-friendly, without requiring manual adjustment
prior to or during service usage. Other factors related to end users, such as the energy consumption of
viewing devices and the amount of client bandwidth used, are also of relevance. We reviewed state-of-
the-art ML-based QoE prediction models for video streaming and extended reality. In the context of
ML applications, there has been a recent trend towards solutions utilizing LSTM for QoE evaluation
and DRL for streaming frameworks. This tendency has been driven by the abundance of available data
and the increased processing capability of compute devices. The QoE evaluation has witnessed a rise in
the utilization of ML techniques due to their improved accuracy in predicting QoE. This also facilitates
real-time, accurate, and flexible QoE management frameworks.

We have also discussed the various user-centric adaptive streaming techniques that consider inputs
such as buffer level, throughput conditions, and chunk bit-rate. These schemes use ML approaches
including the viewport and tile-based adaptation for 360° videos. We have provided an overview of
each scheme by outlining the issues they intend to address, their objectives, research findings, key
elements, and major observation/ application. The goal of these adaptive techniques is to ensure
the highest possible perceptual quality while monitoring the resulting QoE at the client side that
enables immediate feedback in order to enhance the QoE of a specific user. Researchers studying
adaptive streaming may find our comparison useful since it provides a generic, consistent framework for
explicitly evaluating, comparing, and testing the effectiveness of various bit-rate adaptation techniques.
Specifically, the requirements of 360° videos are relatively different in terms of processing parameters
(such as resolution, framerate, bit-rate, tile quality) as well as network transmission parameters (such as
end-to-end latency, network capacity, low response delay) than conventional videos. Also from the users'
perspective, the viewing behaviour and dynamic interaction with the scenes contribute to a distinctly
unique immersive experience.

Many research issues related to viewport prediction methods, tiling strategy, QoE evaluation, and
the influence of additional limitations on 360° video streaming to multiple VR clients is discussed. The
datasets are summarized that have records of throughput traces, video quality scores, head-movement,
and eye-tracking logs which can be of further help to the researchers. We have listed several open chal-
lenges pertaining to the design of QoE predictor models and efficient multimedia streaming techniques.
The limitations with respect to the existing methods are summarized and a perspective is offered on
possible solution approaches to address these research challenges, with an insight into possible direc-
tions where research may be extended. As a whole the review reflects on how the recent technologies
have enhanced performance and overcome many problems with the aid of sophisticated and intelligent
adaptive strategies. However, there still exists many open challenges in this domain.
References

[1] Ericsson mobility report, [Online] https://www.ericsson.com/49ed78/assets/local/reports-papers/mobility-report/documents/2024/ericsson-mobility-report-june-2024.pdf (July, 2024).

[2] F. E. Subhan, A. Yaqoob, C. H. Muntean, G.-M. Muntean, EDQD: An edge-driven multi-agent DRL solution for improving joint QoE in DASH-based rich media content delivery, in: International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), IEEE, 2024, pp. 1–7.

[3] A. A. Simiscuka, M. A. Togou, M. Zorrilla, G.-M. Muntean, 360-ADAPT: An open-RAN-based adaptive scheme for quality enhancement of opera 360° content distribution, IEEE Transactions on Green Communications and Networking 8 (3) (2024) 924–938.

[4] K. Brunnström, S. A. Beker, K. De Moor, A. Dooms, S. Egger, M.-N. Garcia, T. Hossfeld, S. Jumisko-Pyykkö, C. Keimel, M.-C. Larabi, et al., Qualinet white paper on definitions of Quality of Experience (2013) 1–20.

[5] Document ITU-T G.1080, Quality of Experience requirements for IPTV services, Dec. 2008.

[6] M. Xu, C. Li, Z. Chen, Z. Wang, Z. Guan, Assessing visual quality of omnidirectional videos, IEEE Transactions on Circuits and Systems for Video Technology 29 (12) (2019) 3516–3530.

[7] J. Adhuran, C. Galkandage, G. Kulupana, A. Fernando, A novel spherical video quality metric for 360° video coding, in: Proc. IEEE International Conference on Quality of Multimedia Experience (QoMEX), 2022, pp. 1–4.

[8] A. H. Baker, A. Pinard, D. M. Hammerling, On a structural similarity index approach for floating-point data, IEEE Transactions on Visualization and Computer Graphics 30 (9) (2024) 6261–6274.

[9] Y. Zhou, M. Yu, H. Ma, H. Shao, G. Jiang, Weighted-to-spherically-uniform SSIM objective quality evaluation for panoramic video, in: Proc. IEEE International Conference on Signal Processing (ICSP), 2018, pp. 54–57.

[10] V. Kalantzis, P. A. Traganitis, Rayleigh-Ritz based updates of the multilinear singular value decomposition, in: Proc. IEEE Asilomar Conference on Signals, Systems, and Computers, 2023, pp. 1059–1063.

[11] P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, A. C. Bovik, ST-GREED: Space-time generalized entropic differences for frame rate dependent video quality prediction, IEEE Transactions on Image Processing 30 (2021) 7446–7457.

[12] C. G. Bampis, P. Gupta, R. Soundararajan, A. C. Bovik, SpEED-QA: Spatial efficient entropic differencing for image and video quality, IEEE signal processing letters 24 (9) (2017) 1333–1337.

[13] L. Wu, X. Zhang, H. Chen, D. Wang, J. Deng, VP-NIQE: An opinion-unaware visual perception natural image quality evaluator, Neurocomputing 463 (2021) 17–28.

[14] M. Ghosh, C. Singhal, MO-QoE: Video QoE using multi-feature fusion based optimized learning models, Signal Processing: Image Communication 107 (2022) 116766.

[15] S. Chen, Y. Zhang, Y. Li, Z. Chen, Z. Wang, Spherical structural similarity index for objective omnidirectional video quality assessment, in: Proc. IEEE International Conference on Multimedia and Expo (ICME), 2018, pp. 1–6.

[16] A. K. Venkataramanan, C. Stejerean, I. Katsavounidis, A. C. Bovik, One transform to compute them all: Efficient fusion-based full-reference video quality assessment, IEEE Transactions on Image Processing 33 (2024) 509–524.

[17] Y. Jin, A. Patney, R. Webb, A. C. Bovik, FOVQA: Blind foveated video quality assessment, IEEE Transactions on Image Processing 31 (2022) 4571–4584.
[18] Q. Zheng, Z. Tu, X. Zeng, A. C. Bovik, Y. Fan, A completely blind video quality evaluator, IEEE Signal Processing Letters 29 (2022) 2228-2232.

[19] J. P. Ebenezer, Z. Shang, Y. Wu, H. Wei, S. Sethuraman, A. C. Bovik, ChipQA: No-reference video quality prediction via space-time chips, IEEE Transactions on Image Processing 30 (2021) 8059-8074.

[20] N. Jiang, W. Chen, J. Lin, T. Zhao, C.-W. Lin, Video compression artifacts removal with spatial-temporal attention-guided enhancement, IEEE Transactions on Multimedia 26 (2024) 5657-5669.

[21] L. Lin, S. Yu, L. Zhou, W. Chen, T. Zhao, Z. Wang, PEA265: Perceptual assessment of video compression artifacts, IEEE Transactions on Circuits and Systems for Video Technology 30 (11) (2020) 3898-3910.

[22] C. Singhal, J. Chakareski, EVB360: efficient 360-degree video broadcast in next-generation cellular networks, in: Proc. IEEE Intern. Symp. on Broadband Multimedia Systems and Broadcasting, 2022, pp. 1-6.

[23] C. Singhal, I2MB: Intelligent immersive multimedia broadcast in next-generation cellular networks, IEEE Access 10 (2022) 98882-98895.

[24] C. Ascencio, Estimation of the homography matrix to image stitching, Applications of hybrid metaheuristic algorithms for image processing (2020) 205-230.

[25] J. Bosch, K. Istenič, N. Gracias, R. Garcia, P. Ridao, Omnidirectional multicamera video stitching using depth maps, IEEE Journal of Oceanic Engineering 45 (4) (2020) 1337-1352.

[26] S.-H. Tsang, Y.-L. Chan, 360-degree intra coding mode for equirectangular projection format videos, in: Proc. IEEE International Symposium on Circuits and Systems (ISCAS), 2020, pp. 1-5.

[27] I. Storch, B. Zatt, L. Agostini, G. Correa, L. A. da Silva Cruz, D. Palomino, Spatially adaptive intra mode pre-selection for ERP 360 video coding, in: Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 2178-2182.

[28] Y. Wang, Z. Chen, S. Liu, Equirectangular projection oriented intra prediction for 360-degree video coding, in: Proc. IEEE International Conference on Visual Communications and Image Processing (VCIP), 2020, pp. 483-486.

[29] Y. Fan, Y. Jin, Z. Meng, X. Zeng, Pixels and panoramas: An enhanced cubic mapping scheme for video\/image-based virtual-reality scenes, IEEE Consumer Electronics Magazine 8 (2) (2019) 44-49.

[30] J.-L. Lin, Y.-H. Lee, C.-H. Shih, S.-Y. Lin, H.-C. Lin, S.-K. Chang, P. Wang, L. Liu, C.-C. Ju, Efficient projection and coding tools for 360 video, IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9 (1) (Mar., 2019) 84-97.

[31] LIVE-laboratory for Image and Video quality Engineering, an image quality assessment database, in: [Online] Available: http://live.ece.utexas.edu/research/quality/subjective.htm.

[32] Y. Wang, R. Wang, Z. Wang, K. Fan, Y. Deng, S. Syu, Polar square projection for panoramic video, in: Proc. IEEE Visual Communications and Image Processing (VCIP), 2017, pp. 1-4.

[33] W. Chengjia, Z. Haiwu, S. Xiwu, Octagonal mapping scheme for panoramic video encoding, IEEE Transactions on Circuits and Systems for Video Technology 28 (9) (2018) 2402-2406.

[34] A. Abbas, D. Newman, Ahg8: rotated sphere projection for 360 video, JVET-F0036, Hobart, AU 31 (2017).

[35] C. Zhou, Z. Li, J. Osgood, Y. Liu, On the effectiveness of offset projections for 360-degree video streaming, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14 (3s) (2018) 1-24.
[36] H. Hu, Z. Xu, X. Zhang, Z. Guo, Optimal viewport-adaptive 360-degree video streaming against random head movement, in: Proc. IEEE International Conference on Communications (ICC), 2019, pp. 1–6.

[37] C. Singhal, M. Ghosh, H. Mittal, P. Dewangan, MAIVS: Machine learning based adaptive UHD 360-degree immersive video streaming, in: Proc. IEEE International Conference on Communications Workshops (ICC Workshops), 2023, pp. 1082–1087.

[38] Y. Zhang, Y. Guan, K. Bian, Y. Liu, H. Tuo, L. Song, X. Li, EPASS360: QoE-aware 360-degree video streaming over mobile devices, IEEE Transactions on Mobile Computing 20 (7) (2020) 2338–2353.

[39] X. Chen, T. Tan, G. Cao, Macrotile: Toward QoE-aware and energy-efficient 360-degree video streaming, IEEE Transactions on Mobile Computing 23 (2) (2024) 1112–1126.

[40] T. Zhao, Q. Liu, C. W. Chen, QoE in video transmission: A user experience-driven strategy, IEEE Communications Surveys & Tutorials 19 (1) (2016) 285–302.

[41] Z. Akhtar, T. H. Falk, Audio-visual multimedia quality assessment: A comprehensive survey, IEEE Access 5 (2017) 21090–21117.

[42] M. F. M. Hossain, M. Sarkar, S. H. Ahmed, Quality of Experience for video streaming: A contemporary survey, in: 13th International Wireless Communications and Mobile Computing Conference (IWCMC), 2017, pp. 80–84.

[43] A. Bentaleb, B. Taani, A. C. Begen, C. Timmerer, R. Zimmermann, A survey on bitrate adaptation schemes for streaming media over HTTP, IEEE Communications Surveys and Tutorials 21 (1) (2018) 562–585.

[44] N. Barman, M. G. Martini, QoE modeling for HTTP adaptive video streaming–A survey and open challenges, IEEE Access 7 (2019) 30831–30859.

[45] G. Kougioumtzidis, V. Poulkov, Z. D. Zaharis, P. I. Lazaridis, A survey on multimedia services QoE assessment and machine learning-based prediction, IEEE Access 10 (2022) 19507–19538.

[46] C. T. Hewage, A. Ahmad, T. Mallikarachchi, N. Barman, M. G. Martini, Measuring, modeling and integrating time-varying video quality in end-to-end multimedia service delivery: A review and open challenges, IEEE Access 10 (2022) 60267–60293.

[47] W. Zhou, X. Min, H. Li, Q. Jiang, A brief survey on adaptive video streaming quality assessment, Journal of Visual Communication and Image Representation 86 (2022) 103526.

[48] LIVE-Laboratory for Image and Video quality Engineering, an image quality assessment database, in: Available: http://live.ece.utexas.edu/research/LIVE NFLX II/live nfx plus.html.

[49] C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron, A. C. Bovik, Study of temporal effects on subjective video Quality of Experience, IEEE Transactions on Image Processing 26 (11) (2017) 5217–5231. doi:10.1109/TIP.2017.2729891.

[50] D. Ghadiyaram, J. Pan, A. C. Bovik, A subjective and objective study of stalling events in mobile streaming videos, IEEE Transactions on Circuits and Systems for Video Technology 29 (1) (2017) 183–197.

[51] M. Ghosh, D. C. Singhal, R. Wayal, DeSVQ: Deep learning based streaming video QoE estimation, in: Proc. ACM International Conference on Distributed Computing and Networking, 2022, pp. 19–25.

[52] K. Karpov, D. Kachan, E. Siemens, M. Popova, A comparative study of reliable multidestination data transport protocol with TCP and UDP in a point-to-multipoint streaming environment, in: 2023 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), IEEE, 2023, pp. 366–368.
[53] D. Silhavy, S. Pham, S. Arbanowski, S. Steglich, B. Harrer, Latest advances in the development of the open-source player dash.js, in: Proceedings of the 1st Mile-High Video Conference, 2022, pp. 32–38.

[54] K. Spiteri, R. Urgaonkar, R. K. Sitaraman, BOLA: Near-optimal bitrate adaptation for online videos, IEEE/ACM Transactions on Networking 28 (4) (2020) 1698–1711.

[55] T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, M. Watson, A buffer-based approach to rate adaptation: Evidence from a large video streaming service, in: Proc. ACM conference on SIGCOMM, 2014, pp. 187–198.

[56] B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, J.-R. Ohm, Overview of the versatile video coding (VVC) standard and its applications, IEEE Transactions on Circuits and Systems for Video Technology 31 (10) (2021) 3736–3764.

[57] Conviva Inc., Viewer experience report, http://www conviva.com/conviva-customer-survey-reports/ott-beyond-entertainment-csr/, online.

[58] J. Chen, Y. Deng, J. Jia, M. Dohler, A. Nallanathan, Cross-layer QoE optimization for D2D communication in CR-enabled heterogeneous cellular networks, IEEE Transactions on Cognitive Communications and Networking 4 (4) (2018) 719–734.

[59] Cisco IBSG Youth Focus Group, Cisco IBSG youth sur, http://www.cisco.com/c/dam/en/_us/about/ac79/docs/ppt/Video\_Disruption\_SP\_Strategies\_IBSG.pdf, online.

[60] N. Eswara, S. Ashique, A. Panchbhai, S. Chakraborty, H. P. Sethuram, K. Kuchi, A. Kumar, S. S. Channappayya, Streaming video QoE modeling and prediction: A long short-term memory approach, IEEE Transactions on Circuits and Systems for Video Technology 30 (3) (2020) 661–673.

[61] N. Tian, P. Lopes, R. Boulic, A review of cybersickness in head-mounted displays: raising attention to individual susceptibility, Virtual Reality 26 (4) (2022) 1409–1441.

[62] P. Kourtesis, R. Amir, J. Linnell, F. Argelaguet, S. E. MacPherson, Cybersickness, cognition, & motor skills: The effects of music, gender, and gaming experience, IEEE Transactions on Visualization and Computer Graphics 29 (5) (2023) 2326–2336.

[63] U. A. Chattha, U. I. Janjua, F. Anwar, T. M. Madni, M. F. Cheema, S. I. Janjua, Motion sickness in virtual reality: An empirical evaluation, IEEE Access 8 (2020) 130486–130499.

[64] A. Dharmasiri, C. Kattadige, V. Zhang, K. Thilakarathna, Viewport-aware dynamic 360° video segment categorization, in: Proc. ACM Wkshp. NOSSDAV, 2021, p. 114–121.

[65] X. Corbillon, A. Devlic, G. Simon, J. Chakareski, Viewport-adaptive navigable 360-degree video delivery, IEEE, Paris, France, 2017.

[66] M. Hu, L. Wang, B. Tan, S. Jin, Two-tier 360-degree video delivery control in multiuser immersive communications systems, IEEE Transactions on Vehicular Technology 72 (3) (2023) 4119–4123.

[67] S. Kim, H. Oh, C. Kim, eff-HAS: Achieve higher efficiency in data and energy usage on dynamic adaptive streaming, Journal of Communications and Networks 20 (3) (2018) 325–342.

[68] M. Naresh, N. Gireesh, P. Saxena, M. Gupta, SAC-ABR: Soft actor-critic based deep reinforcement learning for adaptive bitrate streaming, in: Proc. IEEE International Conference on COMmunication Systems & NETworkS (COMSNETS), 2022, pp. 353–361.

[69] W. Choi, J. Chen, J. Yoon, Abraider: Multiphase reinforcement learning for environment-adaptive video streaming, IEEE Access 10 (2022) 53108–53123.
[70] T. T. Nguyen, N. D. Nguyen, S. Nahavandi, Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications, IEEE Transactions on Cybernetics 50 (9) (2020) 3826-3839.

[71] Z. Yan, Y. Xu, Data-driven load frequency control for stochastic power systems: A deep reinforcement learning method with continuous action search, IEEE Transactions on Power Systems 34 (2) (2018) 1653-1656.

[72] S. A. A. Rizvi, Z. Lin, Reinforcement learning-based linear quadratic regulation of continuous-time systems using dynamic output feedback, IEEE Transactions on Cybernetics 50 (11) (2019) 4670-4679.

[73] B. Hou, S. Yang, F. Li, L. Zhu, X. Chen, Y. Wang, X. Fu, NOVA: Neural-optimized viewport adaptive 360-degree video streaming at the edge, IEEE Transactions on Services Computing (2024) 1-15.

[74] C. Singhal, C. F. Chiasserini, C. Casetti, Efficient multimedia broadcast for heterogeneous users in cellular networks, in: Proc. International Wireless Communications and Mobile Computing Conference (IWCMC), 2016, pp. 315-320.

[75] C. Singhal, C. F. Chiasserini, C. E. Casetti, EMB: Efficient multimedia broadcast in multi-tier mobile networks, IEEE Transactions on Vehicular Technology 68 (11) (2019) 11186-11199.

[76] Q. Wu, H. Li, F. Meng, K. N. Ngan, A perceptually weighted rank correlation indicator for objective image quality assessment, IEEE Transactions on Image Processing 27 (5) (2018) 2499-2513.

[77] X. Yu, Z. Tu, Z. Ying, A. C. Bovik, N. Birkbeck, Y. Wang, B. Adsumilli, Subjective quality assessment of user-generated content gaming videos, in: Proc. IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 74-83.

[78] Z. Shang, J. P. Ebenezer, A. K. Venkataramanan, Y. Wu, H. Wei, S. Sethuraman, A. C. Bovik, A study of subjective and objective quality assessment of HDR videos, IEEE Transactions on Image Processing 33 (2023) 42-57.

[79] A. Saha, Y.-C. Chen, C. Davis, B. Qiu, X. Wang, R. Gowda, I. Katsavounidis, A. C. Bovik, Study of subjective and objective quality assessment of mobile cloud gaming videos, IEEE Transactions on Image Processing 32 (2023) 3295-3310.

[80] J. P. Ebenezer, Z. Shang, Y. Chen, Y. Wu, H. Wei, S. Sethuraman, A. C. Bovik, HDR or SDR? A subjective and objective study of scaled and compressed videos, IEEE Transactions on Image Processing 33 (2024) 3606-3619.

[81] X. Yu, Z. Ying, N. Birkbeck, Y. Wang, B. Adsumilli, A. C. Bovik, Subjective and objective analysis of streamed gaming videos, IEEE Transactions on Games 16 (2) (2024) 445-458.

[82] A. K. Venkataramanan, Z. Shang, J. P. Ebenezer, M. Chen, Z. Tu, A. C. Bovik, Quality assessment in media and entertainment: Challenges and trends, Computer Vision: Challenges, Trends, and Opportunities (2024) 239.

[83] S. Saini, A. Saha, A. C. Bovik, HIDRO-VQA: High dynamic range oracle for video quality assessment, in: Proc. IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 469-479.

[84] A. K. Venkataramanan, C. Stejerean, I. Katsavounidis, A. C. Bovik, A FUNQUE approach to the quality assessment of compressed HDR videos, in: Picture Coding Symposium (PCS), IEEE, 2024, pp. 1-5.

[85] S. Croci, C. Ozcinar, E. Zerman, J. Cabrera, A. Smolic, Voronoi-based objective quality metrics for omnidirectional video, in: Proc. IEEE International Conference on Quality of Multimedia Experience (QoMEX), 2019, pp. 1-6.

[86] J. Lievens, A. Munteanu, D. De Vleeschauwer, W. Van Leekwijck, Perceptual video quality assessment in HTTP adaptive streaming, in: Proc. IEEE International Conference on Consumer Electronics (ICCE), 2015, pp. 72-73.

[87] M. Taha, A. Ali, J. Lloret, P. R. Gondim, A. Canovas, An automated model for the assessment of QoE of adaptive video streaming over wireless networks, Multimedia Tools and Applications 80 (17) (2021) 26833-26854.
[88] K. Yamagishi, T. Hayashi, Parametric quality-estimation model for adaptive-bitrate-streaming services, IEEE Transactions on Multimedia 19 (7) (2017) 1545-1557.

[89] H. T. Tran, T. Vu, N. P. Ngoc, T. C. Thang, A novel quality model for HTTP adaptive streaming, in: Proc. IEEE International Conference on Communications and Electronics (ICCE), 2016, pp. 423-428.

[90] S. Göring, R. R. R. Rao, B. Feiten, A. Raake, Modular framework and instances of pixel-based video quality models for UHD-1/4K, IEEE Access 9 (2021) 31842-31864.

[91] W. Robitza, A. M. Dethof, S. Göring, A. Raake, A. Beyer, T. Polzehl, Are you still watching? Streaming video quality and engagement assessment in the crowd, in: Proc. IEEE International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1-6.

[92] F. Schiffner, S. Moller, Direct scaling and quality prediction for perceptual video quality dimensions, in: Proc. IEEE International Conference on Quality of Multimedia Experience (QoMEX), 2018, pp. 1-3.

[93] C. Singhal, S. De, R. Trestian, G.-M. Muntean, eWU-TV: User-centric energy-efficient digital TV broadcast over Wi-Fi networks, IEEE Transactions on Broadcasting 61 (1) (2015) 39-55.

[94] V. P. M. Kumar, S. Mahapatra, Quality of Experience driven rate adaptation for adaptive HTTP streaming, IEEE Transactions on Broadcasting 64 (2) (2018) 602-620.

[95] H. T. T. Tran, N. P. Ngoc, A. T. Pham, T. C. Thang, A multi-factor QoE model for adaptive streaming over mobile networks, in: Proc. IEEE Globecom Workshops (GC Wkshps), 2016, pp. 1-6.

[96] W. Robitza, M.-N. Garcia, A. Raake, A modular HTTP adaptive streaming QoE model — candidate for ITU-T P.1203 ("P.NATS"), in: Proc. IEEE International Conference on Quality of Multimedia Experience (QoMEX), 2017, pp. 1-6.

[97] I. Recommendation, 1204: Video quality assessment of streaming services over reliable transport for resolutions up to 4K (2023).

[98] R. R. R. Rao, S. Göring, P. List, W. Robitza, B. Feiten, U. Wüstenhagen, A. Raake, Bitstream-based model standard for 4K/UHD: ITU-T P.1204.3—model details, evaluation, analysis and open source implementation, in: Proc. IEEE International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1-6.

[99] Z. Li, A. Aaron, A. K. Moorthy, A. Manohara, Toward a practical perceptual video quality metric, in: Available: http://techblog.netflix.com/2016/06/toward-practical-perceptual-video.html.

[100] P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, A. C. Bovik, ONVIQT: Contrastive video quality estimator, IEEE Transactions on Image Processing 32 (2023) 5138-5152.

[101] F. Zhang, A. Katsenou, C. Bampis, L. Krasula, Z. Li, D. Bull, Enhancing VMAF through new feature integration and model combination, in: 2021 Picture Coding Symposium (PCS), IEEE, 2021, pp. 1-5.

[102] N. Barman, E. Jammeh, S. A. Ghorashi, M. G. Martini, No-reference video quality estimation based on machine learning for passive gaming video streaming applications, IEEE Access 7 (2019) 74511-74527.

[103] M. Ghosh, C. Singhal, Machine learning-based subjective quality estimation for video streaming over wireless networks, in: Next-Generation Wireless Networks Meet Advanced Machine Learning Applications, IGI Global, 2019, pp. 235-254.

[104] P. Casas, S. Wassermann, Improving QoE prediction in mobile video through machine learning, in: Proc. IEEE International Conference on the Network of the Future (NOF), 2017, pp. 1-7.
[105] X. Tao, Y. Duan, M. Xu, Z. Meng, J. Lu, Learning QoE of mobile video transmission with deep neural network: A data-driven approach, IEEE Journal on Selected Areas in Communications 37 (6) (2019) 1337-1348.

[106] S. Zadtootaghaj, N. Barman, R. R. R. Rao, S. Göring, M. G. Martini, A. Raake, S. Möller, DEMI: Deep video quality estimation model using perceptual video quality dimensions, in: Proc. IEEE International Workshop on Multimedia Signal Processing (MMSP), 2020, pp. 1-6.

[107] L. Zhang, H. Dong, A. El Saddik, Towards a QoE model to evaluate holographic augmented reality devices, IEEE MultiMedia 26 (2) (2018) 21-32.

[108] H. Zhang, L. Dong, G. Gao, H. Hu, Y. Wen, K. Guan, DeepQoE: A multimodal learning framework for video quality of experience (QoE) prediction, IEEE Transactions on Multimedia 22 (12) (2020) 3210-3223.

[109] Z. Ying, M. Mandal, D. Ghadiyaram, A. Bovik, Patch-VQ: 'Patching up' the video quality problem, in: Proc. IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14019-14029.

[110] B. Li, W. Zhang, M. Tian, G. Zhai, X. Wang, Blindly assess quality of in-the-wild videos via quality-aware pre-training and motion perception, IEEE Transactions on Circuits and Systems for Video Technology 32 (9) (2022) 5944-5958.

[111] X. Guan, F. Li, Y. Zhang, P. C. Cosman, End-to-end blind video quality assessment based on visual and memory attention modeling, IEEE Transactions on Multimedia 25 (2022) 5206-5221.

[112] Z. Tu, X. Yu, Y. Wang, N. Birkbeck, B. Adsumilli, A. C. Bovik, RAPIQUE: Rapid and accurate video quality prediction of user generated content, IEEE Open Journal of Signal Processing 2 (2021) 425-440.

[113] R. G. de A. Azevedo, N. Birkbeck, I. Janatra, B. Adsumilli, P. Frossard, A viewport-driven multi-metric fusion approach for 360-degree video quality assessment, in: Proc. IEEE International Conference on Multimedia and Expo (ICME), 2020, pp. 1-6.

[114] J. Y. Lin, T.-J. Liu, E. C.-H. Wu, C.-C. J. Kuo, A fusion-based video quality assessment (FVQA) index, in: Proc. IEEE Signal and Information Processing Association Annual Summit and Conference (APSIPA), Asia-Pacific, 2014, pp. 1-5.

[115] C. G. Bampis, A. C. Bovik, Feature-based prediction of streaming video QoE: Distortions, stalling and memory, Signal Processing: Image Communication 68 (2018) 218-228.

[116] M. A. Papadopoulos, A. V. Katsenou, D. Agrafiotis, D. R. Bull, A multi-metric approach for block-level video quality assessment, Signal Processing: Image Communication 78 (2019) 152-158.

[117] Video Quality Experts Group (VQEG). VQEG HDTV Phase I, in: [Online]. Available: https://www.its.bldrdoc.gov/vqeg/projects/hdtv/hdtv.aspx, Aug. 15, 2017.

[118] Z. Duanmu, K. Zeng, K. Ma, A. Rehman, Z. Wang, A Quality-of-Experience index for streaming video, IEEE Journal of Selected Topics in Signal Processing 11 (1) (2017) 154-166.

[119] V. Phani, M. Ghosh, S. Mahapatra, No-reference video quality assessment from artifacts and content characteristics: A neuro-fuzzy framework for video quality evaluation, Multimedia Tools and Applications 83 (16) (2024) 48049-48074.

[120] A. Choudhury, S. Daly, Combining quality metrics using machine learning for improved and robust HDR image quality assessment, Electronic Imaging 2019 (10) (2019) 307-1.

[121] P. Le Callet, C. Viard-Gaudin, D. Barba, A convolutional neural network approach for objective video quality assessment, IEEE Transactions on Neural Networks 17 (5) (2006) 1316-1327.