# VQM4HAS: A Real-Time Quality Metric for HEVC Videos in HTTP Adaptive Streaming

Hadi Amirpour, Jingwen Zhu, Wei Zhou, Patrick Le Callet, Christian Timmerer

► To cite this version:

Hadi Amirpour, Jingwen Zhu, Wei Zhou, Patrick Le Callet, Christian Timmerer. VQM4HAS: A Real-Time Quality Metric for HEVC Videos in HTTP Adaptive Streaming. IEEE Transactions on Multimedia, 2025, 27, pp.9619 - 9631. 10.1109/tmm.2025.3613110. hal-05520399

HAL Id: hal-05520399

https://hal.science/hal-05520399v1

Submitted on 20 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# VQM4HAS: A Real-Time Quality Metric for HEVC Videos in HTTP Adaptive Streaming

Hadi Amirpour<sup>©</sup>, Member, IEEE, Jingwen Zhu, Wei Zhou<sup>©</sup>, Senior Member, IEEE, Patrick Le Callet<sup>©</sup>, Fellow, IEEE, and Christian Timmerer<sup>©</sup>, Senior Member, IEEE

**Abstract**—In HTTP Adaptive Streaming (HAS), a video is encoded at various bitrate-resolution pairs, collectively known as the bitrate ladder, allowing users to select the most suitable representation based on their network conditions. Optimizing this set of pairs to enhance the Quality of Experience (QoE) requires accurately measuring the quality of these representations. VMAF and ITU-T's P.1204.3 are highly reliable metrics for assessing the quality of representations in HAS. However, in practice, using these metrics for optimization is often impractical for live streaming applications due to their high computational costs and the large number of bitrate-resolution pairs in the bitrate ladder that need to be evaluated. To address their high complexity, our paper introduces a new method called VQM4HAS, which extracts low-complexity features, including (i) video complexity features, (ii) frame-level encoding statistics logged during the encoding process, and (iii) lightweight video quality metrics. These extracted features are then fed into a regression model to predict VMAF or P.1204.3. The VQM4HAS model is designed to operate on a per bitrate-resolution pair, per-resolution, and cross-representation basis, optimizing quality predictions across different scenarios. Our experimental results demonstrate that VQM4HAS achieves a high correlation with VMAF and P.1204.3, with Pearson correlation coefficients (PCC) ranging from 0.95 to 0.96 for VMAF and 0.97 to 0.99 for P.1204.3, depending on the resolution. Despite achieving a high correlation with VMAF and P.1204.3, VQM4HAS exhibits significantly less complexity than both metrics, with 98% and 99% less complexity for VMAF and P.1204.3, respectively, making it suitable for live streaming scenarios. We also conduct a feature importance analysis to further reduce the complexity of the proposed method. Furthermore, we evaluate the effectiveness of our method by using it to predict subjective quality scores. The results show that VQM4HAS achieves a higher correlation with subjective scores at various resolutions despite its minimal complexity.

**Index Terms**—Video quality, HTTP Adaptive Streaming, P1204.3, VMAF, QoE, bitstream.

Received 10 November 2024; revised 11 February 2025; accepted 16 March 2025. Date of publication 22 September 2025; date of current version 17 December 2025. This work was supported in part by the Austrian Federal Ministry for Digital and Economic Affairs, in part by the National Foundation for Research, Technology and Development, and in part by the Christian Doppler Research Association, Christian Doppler Laboratory ATHENA. The associate editor coordinating the review of this article and approving it for publication was Ms. Vania Vieira Estrela. (Corresponding author: Hadi Amirpour.)

Hadi Amirpour and Christian Timmerer are with the Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, 9020 Klagenfurt, Austria (e-mail: hadi.amirpour@aua.at; christian.timmerer@aua.at).

Jingwen Zhu and Patrick Le Callet are with the Ecole Centrale Nantes, CNRS, LS2N, Nantes Université, 44300 Nantes, France (e-mail: jingweng.zhu@univ-nantes.fr; patrick.legallet@univ-nantes.fr).

Wei Zhou is with the School of Computer Science and Informatics, Cardiff University, CF24 4AG Cardiff, U.K. (e-mail: zhouw26@cardiff.ac.uk).

The source code is available at https://github.com/cd-athena/VQM4HAS.

Digital Object Identifier 10.1109/TMM.2025.3613110

## I. INTRODUCTION

VIDEO streaming is on the rise, benefiting from the widespread availability of high-speed Internet and the proliferation of mobile devices [1]. It has become integral to our daily routines, serving various purposes in entertainment, education, business, and more. Noteworthy examples include streaming platforms such as Netflix, Hulu, and Amazon Prime Video, allowing us to enjoy our preferred content at our convenience. Online learning platforms such as Coursera and Udemy enable students to learn autonomously and from anywhere globally, facilitated by video streaming technology. Additionally, this technology enables real-time communication between people, regardless of their location, leading to considerable savings in time and cost, eliminating the need for travel.

Video streaming services commonly depend on HTTP-based Adaptive Streaming (HAS) [2] as their primary technology for delivering content to viewers. HAS partitions each video content into smaller segments, enabling clients to receive and play the video promptly without waiting for the entire video to download. Additionally, each video segment is offered at various bitrates, referred to as representations, enabling clients to adapt dynamically depending on the available network bandwidth and the capabilities of the device, ensuring a seamless viewing experience [3].

Adaptive bitrate (ABR) algorithms [4] are typically employed on the client side to facilitate this dynamic adaptation. These algorithms select the most appropriate bitrate for each requested video segment based on real-time factors like network conditions and buffer occupancy. The ABR mechanism is crucial in optimizing the viewer's QoE by choosing the highest bitrate that the network can sustain without causing buffering interruptions. Consequently, the QoE is shaped by the interaction between network bandwidth, device capabilities, and the ABR algorithm's bitrate selection. Several factors, such as initial delay, stall or buffering events, video quality, and quality switch events, influence QoE [3], [5], [6], [7], [8]. For example, frequent stalls disrupt the viewing experience; video quality may suffer due to compression artifacts or upscaling; and quality switches occur when the video shifts between different resolutions across segments.

Among those factors, measuring the perceived video quality is crucial in evaluating QoE within video streaming services. The quality of the video is typically impacted by two main factors: (i) compression artifacts and (ii) upscaling artifacts. Compression artifacts arise from the video compression
process, which is essential to reduce video size and bitrate to enable efficient network transmission. On the other hand, upscaling artifacts occur when adjusting the video resolution to match the display resolution if they are not inherently the same. Video encoders are used in the video compression process to encode video data and eliminate redundant information, leading to a loss of certain details and visual quality. The most accurate approach for evaluating video quality involves subjective testing, utilizing human scores. This method is widely adopted in video quality assessment. The subjective test entails presenting video content, typically both the original and compressed versions [9], to a group of participants. These participants are then requested to assess the video quality based on their perception, usually employing the Mean Opinion Score (MOS) [10], [11], [12]. MOS is a widely utilized method for gauging the perceived quality of audio and video signals. Although subjective testing yields valuable insights into users' perceived video quality, conducting such tests can be time-consuming and expensive. This is due to the necessity of involving a large group of participants and utilizing specialized equipment [3], [13].

Objective metrics can be employed to assess video quality as a cost-effective substitute for subjective testing. These metrics evaluate the visual quality of compressed videos using mathematical models. Using objective quality metrics aids in recognizing compression artifacts, and they are frequently automated, demanding less time and resources compared to subjective testing. Numerous objective metrics are available for the evaluation of video quality. Some of the most commonly used ones include: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [14], Multi-Scale Structural Similarity Index (MS-SSIM) [15], Video Quality Metric (VQM) [16], Visual Information Fidelity (VIF) [17], Feature SIMilarity index (FSIM) [18], Visual Signal-to-Noise Ratio (VSNR) [19], MOTion-based Video Integrity Evaluation (MOVIE) [20], Video-BLIINDS [21], Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [22], and Spatio-Temporal Reduced Reference Entropic Differences (ST-RRED) [23]. However, objective metrics may not always accurately predict the perceived video quality or MOS values.

Among objective quality metrics, VMAF [24] has gained significant attention in recent years due to its high correlation with subjective testing results [25], [26], [27], [28]. It was first developed as a fusion of different video quality metrics using support vector machines (SVM), resulting in the “Fusion-based Video Quality Assessment (FVQA) Index [29].” This index was shown to outperform existing video quality metrics on a subjective video quality database [29]. It was further developed in collaboration with Netflix, using different subjective video datasets, including a Netflix-owned dataset. The metric was subsequently renamed to “Video Multimethod Assessment Fusion (VMAF)” and was made available under a permissive open-source license.<fnref n="1" /> The metric has been updated over the years to support custom models, including an adaptation for cellular phone screen viewing and prediction of the quality of up to 4 K resolution content.<fnref n="2" />

The VMAF algorithm and model utilize several elementary metrics fused by SVM regression. These metrics include:

* Visual Information Fidelity (VIF) [17]: The VIF image quality metric measures quality based on the loss of information fidelity. In its original form, it combines four scales to measure loss of fidelity. VMAF uses a modified version of VIF that includes loss of fidelity in each scale as an elementary metric.

* Detail Loss Metric (DLM) [30]: DLM (called ADM in VMAF) is an image quality metric that separately measures the loss of details affecting content visibility and the redundant impairment that distracts viewer attention.

* Motion: This metric measures the temporal difference between frames by averaging the absolute pixel difference for the luminance (Y) component.

While VMAF is highly accurate in predicting human video quality perception, it has been criticized for its slow processing speed [31]. This can increase computation costs for content providers and make it impractical for real-time applications. This is particularly important for streaming applications, where the quality metric needs to be evaluated **multiple times** for the same video across different representations. For example, the HLS ladder [32] recommends encoding each video into 12 different representations, while MC-DASH [33] suggests using 15 representations for HEVC videos. Fig. 1 shows the VMAF computation process for *n* representations of a bitrate ladder in a streaming application. The video is encoded at multiple bitrates, resulting in several representations (encodings or bitrate-resolution pairs). Each representation is then **upscaled** to the resolution of the original video, and VMAF is calculated between each upscaled representation and the original video.

ITU-T P.1204.3 [34] is an alternative to VMAF as a bitstream-based quality metric that uses three different types of degradations for prediction: (i) quantization degradation, (ii) upscaling degradation, and (iii) temporal degradation. To use this model, features must be extracted from the encoded bitstream, which involves reading data from the bitstream. This process results in a complex model, particularly for high bitrate and high-resolution bitstreams.

**Problem Statement and Motivation:** While VMAF and similar metrics provide accurate assessments of video quality, they come with significant computational overhead. In practical streaming applications—where each video is encoded into multiple representations (e.g., up to 12 in HLS [32] or 15 in MC-DASH [33])— the need to calculate these metrics for every representation leads to high processing costs. This overhead is particularly problematic for live streaming and other real-time applications, where fast and frequent quality evaluations are essential to maintain optimal QoE. In many cases, the computational complexity forces service providers to forgo quality metric calculations, potentially degrading the viewing experience.

**Contributions:** In this paper, we propose a novel video quality metric named VQM4HAS to reduce the computation demand for video quality calculation in HAS using VMAF or P.1204.3. The proposed VQM4HAS metric significantly reduces the computational cost of VMAF and P.1204.3 while maintaining a high correlation with them. This is achieved by using a combination of low-complexity features, such as (i) video complexity

<footnote n="1">¹https://github.com/Netflix/vmaf/blob/master/LICENSE</footnote>

<footnote n="2">²https://github.com/Netflix/vmaf/tree/master/model</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Assessing streaming quality by calculating VMAF scores for different bitrate levels across a bitrate ladder. Different versions of the same video are encoded, and each version is subsequently upscaled to calculate VMAF with respect to the original video.</figcaption></figure>

features, (ii) frame-level encoding statistics logged during encoding, and (iii) basic video quality metrics logged during encoding, for making predictions. We evaluate the metric on a per-representation basis, per-resolution basis, and finally across representations.

Building upon our previous work [35], which introduced the foundation for leveraging low-complexity features for video quality estimation (Items 1 and 3 below), this paper extends and generalizes the approach to improve efficiency, applicability, and accuracy. The main contributions of this paper are summarized as follows:

* We propose a quality metric that leverages low-complexity features—namely, video complexity metrics, frame-level encoding statistics, and basic video quality metrics logged during encoding—to accurately predict VMAF. Our approach employs either a linear or random forest regressor and eliminates the need to upscale the video.

* We extend our work by evaluating the effectiveness of these features in predicting not only VMAF but also the well-known quality metric P.1204.3, demonstrating the broader applicability of our method across different video quality assessment models.

* We integrate per-representation models by employing a single regressor for each resolution, thereby reducing the total number of regressors while maintaining high accuracy.

* We investigate the effect of cross-representation feature extraction by analyzing the impact of using features from one representation to predict the quality of another.

* We conduct an extensive analysis of feature importance for each regressor to identify the most critical features. This enables a balanced trade-off between computational complexity and accuracy, ensuring efficiency across different scenarios.

* We evaluate the accuracy of the proposed method in predicting subjective test outcomes, demonstrating promising results.

This paper is organized as follows: Section II provides details of the proposed VQM4HAS metric. Section III presents the evaluation setup and results. Finally, Section IV concludes the paper.

## II. A VIDEO QUALITY METRIC FOR HAS

In this section, we present our proposed video quality metric, VQM4HAS, which leverages low complexity features extracted from both the original video and its encoded versions for quality prediction. These features are classified into three main categories: (i) video complexity features (cf. II-A), (ii) encoding statistic features (cf. II-B), and (iii) basic quality metrics (cf. II-C) which are used to construct VQM4HAS.

Fig. 2 shows a high-level overview of the VQM4HAS architecture. The video complexity features are extracted from both original ($C_{org}$) and encoded video ($C_{rec}$) (collectively referred to as $C$). The encoding statistics ($S$) are logged during the encoding process for each representation with minimal overhead. Additionally, low-complexity video quality metrics ($Q$) are computed between the original video and its encoded versions during the encoding.

These features ($\{C, S, Q\}$) are then fed into a machine learning model, which predicts video quality scores. Since video quality in HAS is typically evaluated at the segment level rather than individual frames, VQM4HAS predicts segment-level quality by averaging the features and target label values across frames. Various machine learning approaches are applied and evaluated to predict video quality scores based on the extracted features.

To address the typical use case of live video streaming, where a fixed set of bitrate-resolution pairs is employed, we conduct the following evaluations: (1) we first train a separate model for each representation in a fixed bitrate ladder and assess our method's ability to predict the corresponding video quality metrics. (2) Next, we evaluate VQM4HAS's performance in predicting the quality of video segments that share the same resolution but
<figure><img src="image_3.png" /><figcaption>Fig. 2. Overview of VQM4HAS architecture. During the encoding of each representation, encoding statistics (S), lightweight quality metrics (Q), and complexity metrics of the compressed video ($C_{rec}$) are logged. Additionally, complexity metrics of the original video ($C_{org}$) are extracted from the source video. These features are then fed into a regressor to predict the quality score.</figcaption></figure>

differ in bitrate, extending its applicability to a broader range of scenarios. (3) Lastly, we investigate whether features extracted from a single representation can effectively predict the quality across all representations.

We also evaluate the importance of features to determine which features contribute most significantly to the prediction of video quality. This step helps to avoid unnecessary computational overhead by identifying less important features that may not contribute significantly to the model's overall accuracy.

While the primary goal of the VQM4HAS approach is to serve as a replacement for existing quality metrics like VMAF and P.1204.3, we also assess its ability to predict subjective scores compared to these metrics. To achieve this, we fine-tune the model using a training set based on subjective test data and evaluate its performance on a corresponding test set. In the following sections, we provide a detailed explanation of the feature extraction process.

## A. Video Complexity Features

The perceived quality of video frames at a fixed bitrate is affected by various factors, including the spatial and temporal complexities of the frames. These features have a significant impact on video compressibility [36] and are commonly utilized in video quality metrics [16]. The spatial complexity of frames refers to the amount of detail and variation in the video frames, such as textures, edges, and colors. Temporal complexity, on the other hand, refers to the amount of motion and changes between frames, such as camera movements, object movements, and scene changes. Videos with high spatial and temporal complexities, compared to videos with lower complexities, typically require higher bitrates to maintain the same perceived video quality. For instance, video frames with low temporal complexity can be predicted efficiently from the previously encoded frames, leading to less residual data. Therefore, videos with lower temporal complexity tend to have less information

loss at a fixed bitrate, which leads to higher perceived quality. Thus, we compute video complexity features for both the original and encoded videos, denoted as $C_{org}$ and $C_{rec}$, respectively.

In this paper, we use a DCT-based energy function ($E_{DCT}$) [37], [38] to quantify the spatial complexity of video frames. The energy function serves as a tool for mapping the texture of a block of pixels from a multi-dimensional frequency space to a one-dimensional energy space. It assigns higher costs to higher DCT frequencies. The DC value, which is color-dependent and does not contribute to the texture, is treated separately. In fact, the energy function is used to determine the likelihood of pixels in a block belonging to a single region, and as such, the average color is not considered. The function is defined as follows:

$$E_{DCT} = \sum_{i=1}^{w} \sum_{j=1}^{h} e^{\left[ \frac{(i \cdot j)}{w \cdot h} \right]^2 - 1} |DCT(i-1, j-1)| \quad (1)$$

where $E_{DCT}$ represents the energy of the block, with $w$ and $h$ denoting its width and height, respectively. The $(i, j)^{th}$ DCT component is given by $DCT(i, j)$ when $i + j$ is greater than 2, and is 0 otherwise. The energy function per frame $E$ is calculated by averaging over entire blocks [38]. Blocks of highly textured pixels produce high energy values, containing many high-frequency DCT values. Conversely, blocks of homogeneously colored pixels with few high-frequency DCT values produce low energy values.

For the temporal complexity, we use the temporal function of the DCT-based energy function ($h$) [38], which is expressed as follows:

$$h = \frac{1}{C} \sum_{c=1}^{C-1} \frac{1}{w^2} SAD(E_{DCT}(t, c), E_{DCT}(t-1, c)) \quad (2)$$

here, $c$ denotes the block index, $C$ represents the total number of blocks, $w$ signifies the width of the blocks, and $t$ denotes the
<table><caption>TABLE I
DESCRIPTION OF ENCODING STATISTICS WHEN -CSV-LOG-LEVEL IS GREATER THAN OR EQUAL TO 1</caption><thead><tr><th>Parameter</th><th>Description</th><th>Used for evaluation</th><th>Name</th></tr></thead><tbody><tr><td>Encode Order</td><td>The frame order in which the encoder encodes.</td><td>No</td><td></td></tr><tr><td>Type</td><td>Slice type of the frame.</td><td>No</td><td></td></tr><tr><td>POC</td><td>Picture Order Count - The display order of the frames.</td><td>No</td><td></td></tr><tr><td>QP</td><td>Quantization Parameter decided for the frame.</td><td>Yes</td><td>qp</td></tr><tr><td>Bits</td><td>Number of bits consumed by the frame.</td><td>Yes</td><td>bit</td></tr><tr><td>Scenecut</td><td>1 if the frame is a scenecut, 0 otherwise.</td><td>No</td><td></td></tr><tr><td>RateFactor</td><td>Applicable only when CRF is enabled.</td><td>No</td><td></td></tr><tr><td>BufferFill</td><td>Bits available for the next frame. Includes bits carried over from the current frame.</td><td>No</td><td></td></tr><tr><td>BufferFillFinal</td><td>Buffer bits available after removing the frame out of CPB.</td><td>No</td><td></td></tr><tr><td>UnclippedBufferFillFinal</td><td>Unclipped buffer bits available after removing the frame out of CPB only used for csv logging purposes.</td><td>No</td><td></td></tr><tr><td>Latency</td><td>Latency in terms of number of frames between when the frame was given in and when the frame is given out.</td><td>No</td><td></td></tr><tr><td>Ref lists</td><td>POC of references in lists 0 and 1 for the frame.</td><td>No</td><td></td></tr></tbody></table>

<table><caption>TABLE II
DESCRIPTION OF ENCODING STATISTICS WHEN -CSV-LOG-LEVEL IS GREATER THAN OR EQUAL TO 2</caption><thead><tr><th>Parameter</th><th>Description</th><th>Used for evaluation</th><th>Name</th></tr></thead><tbody><tr><td colspan="4" style="text-align: center;">Analysis statistics</td></tr><tr><td>CU Statistics</td><td>Percentage of CU modes.</td><td>No</td><td></td></tr><tr><td>Distortion</td><td>Average luma and chroma distortion.</td><td>Yes</td><td>luma_dist, chroma_dist</td></tr><tr><td>Psy Energy</td><td>Average psy energy calculated as the sum of absolute difference between source and recon energy.</td><td>Yes</td><td>psy_energy</td></tr><tr><td>Residual Energy</td><td>Average residual energy.</td><td>Yes</td><td>res_energy</td></tr><tr><td>Luma/Chroma Values</td><td>minimum, maximum and average luma and chroma values of source for each frame.</td><td>Yes</td><td>luma_avg, cr_avg, cb_avg</td></tr><tr><td>PU Statistics</td><td>Percentage of PU modes at each depth.</td><td>No</td><td></td></tr><tr><td colspan="4" style="text-align: center;">Performance statistics</td></tr><tr><td>DecideWait ms</td><td>number of milliseconds the frame encoder had to wait.</td><td>No</td><td></td></tr><tr><td>RowWait ms</td><td>number of milliseconds since the frame encoder received a frame to encode before its first row of CTUs is allowed to begin compression.</td><td>No</td><td></td></tr><tr><td>Wall time ms</td><td>number of milliseconds between the first CTU being ready to be compressed and the entire frame being compressed and the output NALs being completed.</td><td>No</td><td></td></tr><tr><td>Ref Wait Wall ms</td><td>number of milliseconds between the first reference row being available and the last reference row becoming available.</td><td>No</td><td></td></tr><tr><td>Total CTU time ms</td><td>the total time (measured in milliseconds) spent by worker threads compressing and filtering CTUs for this frame.</td><td>Yes</td><td>cpu_time</td></tr><tr><td>Stall Time ms</td><td>the number of milliseconds of the reported wall time that were spent with zero worker threads, aka all compression was completely stalled.</td><td>No</td><td></td></tr><tr><td>Total frame time</td><td>Total time spent to encode the frame.</td><td>Yes</td><td>frame_time</td></tr><tr><td>Avg WPP</td><td>the average number of worker threads working on this frame, at any given time.</td><td>No</td><td></td></tr><tr><td>Row Blocks</td><td>the number of times a worker thread had to abandon the row of CTUs it was encoding.</td><td>No</td><td></td></tr></tbody></table>

frame index. For representations, these features ($E_{rec}$ and $h_{rec}$) can be obtained by logging them during the encoding process, since the DCT is calculated during encoding. Alternatively, the VCA open-source software<fnref n="3" /> can be utilized to calculate these features. For the original video, the VCA software is utilized to calculate video complexity features ($E_{org}$ and $h_{org}$). In summary,

$$C_{rec} = \{E_{rec}, h_{rec}\}, C_{org} = \{E_{org}, h_{org}\}.$$

## B. Encoding Statistics Features

Encoding statistics and information are valuable because they provide insights into the compressibility of the video. While some of this data is inherently embedded in the bitstream, additional details may need to be logged during encoding when necessary. In FFmpeg, this information, denoted as *S* in this paper, can be logged into a CSV file by adding `-csv <filename>` to the command line. The level of detail in the CSV log files can be adjusted using `-csv-log-level <integer>` as follows:

* 0 - summary (default)

* 1 - frame level logging

* 2 - frame level logging with performance statistics

Table I summarizes encoding statistics when the `-csv-log-level` is ≥1, while Table II summarizes encoding statistics when the `-csv-log-level` is ≥2.

From Table I, two parameters, *QP* and *Bits*, are selected as features for evaluation. The *QP* (Quantization Parameter) is used in video coding to control the quantization process, which directly influences both bitrate and video quality. In constant bitrate (CBR) rate control, the *QP* is dynamically adjusted for each block to achieve the target bitrate. Lower *QP* values result in higher video quality but increased bitrate, while higher *QP* values reduce the bitrate at the cost of video quality. The average *QP* per frame is recorded as a feature during the encoding process.

*Bits* denotes the number of bits required to encode each frame is another feature that can be logged during encoding. This feature can provide information on the complexity of each frame, which can be useful for predicting video quality.

From Table II, six parameters are selected for evaluation. *Distortion* denotes the average distortion of video frames after compression and is typically reported as two separate values: one for the luma (brightness) component and one for the chroma (color) component of the frames. *Psy Energy* refers to the amount of energy in a video frame that is not perceived by the human visual system. It is calculated as the sum of the absolute difference (SAD) between the source and reconstructed energy of a frame. *Residual Energy* is a measure of the energy of the difference between the original image and the reconstructed image. It is calculated as the sum of squared error (SSE) between the original and reconstructed images before quantization. *Luma/Chroma Values* represent the brightness (luma) and color (chroma) information of a video frame. For our evaluation, we focused solely

<footnote n="3">³https://github.com/cd-athena/VCA</footnote>
on their average values, disregarding their minimum and maximum values. We also used the average time for compressing and filtering Coding Tree Units (CTUs) of each frame, referred to as Total CTU time, as well as the total time to encode the frame, referred to as Total frame time, for evaluation. In summary,

$$S = \{qp, bit, luma_dist, chroma_dist, \\ \quad psy_energy, res_energy, luma_avg, \\ \quad cr_avg, cb_avg, cpu_time, frame_time\}.$$

### C. Lightweight Video Quality Metrics

Video codecs often provide two key quality metrics, namely PSNR and SSIM, which are essential for evaluating the visual quality of encoded video. In FFmpeg, these metrics can be enabled by appending the options `-psnr` and `-ssim` to the command line. A significant advantage of this approach is that these metrics, referred to as $Q$ in this paper, are computed during the encoding process, without incurring any substantial additional time overhead. Unlike external tools that require re-reading and processing video frames post-encoding, these metrics are calculated on-the-fly by the encoder, eliminating the need for separate computation. This enhances the overall efficiency of the workflow, making it particularly suitable for real-time video applications.

PSNR is a widely used, low-cost metric for measuring the quality of reconstructed video frames relative to the original frames. However, it has been shown to have limited accuracy in predicting human perception of video quality. Despite this limitation, PSNR is effective when comparing two encoded versions of the same video [39]. Therefore, we include PSNR values for both the Y (luma) and YUV (luma and chroma) components, denoted as $y\_psnr$ and $yuv\_psnr$, as features in our proposed video quality metric. While not always reflective of perceived quality, PSNR can serve as an indicator of video quality degradation and provide complementary information alongside other features. It is calculated as follows:

$$PSNR = 10 \log_{10} \frac{MAX_I^2}{MSE} \qquad (3)$$

where $MAX_I$ is the maximum possible pixel value (255 for 8-bit images), and $MSE$ is the mean squared error between the original and reconstructed frames.

SSIM, denoted as $ssim$, is another commonly used full-reference quality assessment method. It measures the structural similarity between the Y components of the reference image and that of the distorted image. The formula for SSIM is as follows:

$$SSIM(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)} \qquad (4)$$

where $x$ and $y$ are the reference and distorted images, respectively. $\mu_x$ and $\mu_y$ are the mean values of $x$ and $y$, $\sigma_x$ and $\sigma_y$ are the standard deviations of $x$ and $y$, and $\sigma_{xy}$ is the cross-covariance between $x$ and $y$. $c_1$ and $c_2$ are constants used to stabilize the

### Algorithm 1: VQM4HAS: A Real-Time Quality Metric for HEVC Videos in HTTP Adaptive Streaming.

**Require:** A video to be encoded at a bitrate-resolution pair

**Ensure:** Predicted video quality score (VQM4HAS)

1. Extract video complexity features for the original video during encoding or using VCA [38]:

$$C_{org} = \{E_{org}, h_{org}\}$$

2. Extract video complexity features for the encoded video during encoding or using VCA [38]:

$$C_{rec} = \{E_{rec}, h_{rec}\}$$

3. Extract encoding statistics during encoding:

$$S = \{qp, bit, luma_dist, chroma_dist, \\ \quad psy_energy, res_energy, luma_avg, \\ \quad cr_avg, cb_avg, cpu_time, frame_time\}$$

4. Extract lightweight video quality metrics during encoding:

$$Q = \{y\_psnr, yuv\_psnr, ssim\}$$

5. Combine extracted features:

$$F = \{C_{org}, C_{rec}, S, Q\}$$

6. Predict video quality score VQM4HAS using the trained model.

division with a weak denominator. In summary,

$$Q = \{y\_psnr, yuv\_psnr, ssim\}.$$

Algorithm 1 summarizes the process of computing VQM4HAS.

## III. EXPERIMENTAL RESULTS

In this section, we present the experimental results obtained by evaluating VQM4HAS on the Inter4K dataset [40]. It contains 1000 high-quality 4 K resolution video clips with a frame rate of 60 frames per second (fps). The videos encompass a variety of content types. We use FFmpeg's libx265, based on the open-source x265 v3.4 encoder, to compress the videos, following Apple's recommendations [32]. This process produces 12 representations of the HLS bitrate ladder, as listed in Table III. The spatial ($E_{org}$) and temporal ($h_{org}$) complexity features are extracted from the original videos using the open-source VCA software [38]. The features from the encoded representation are extracted by logging them into a csv file by using -csv-log-level option in x265. The x265 encoder was modified to log the spatial ($E_{rec}$) and temporal ($h_{rec}$) complexity features during encoding. To compute VMAF scores, the encoded representations are first decoded and then upscaled to 4 K resolution using bicubic interpolation, as it is assumed that the videos will be viewed on a 4 K display. The VMAF score is then
<table><caption>TABLE III
HLS BITRATE LADDER</caption><thead><tr><th></th><th>Represnetation ID</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th></tr></thead><tbody><tr><th>Bitrate ladder</th><th>Bitrate (kbps)</th><td>145</td><td>300</td><td>600</td><td>900</td><td>1600</td><td>2400</td><td>3400</td><td>4500</td><td>5800</td><td>8100</td><td>11600</td><td>16800</td></tr><tr><th></th><th>Resolution</th><td>360p</td><td>432p</td><td>540p</td><td>540p</td><td>540p</td><td>720p</td><td>720p</td><td>1080p</td><td>1080p</td><td>1440p</td><td>2160p</td><td>2160p</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Fig. 3. (a) VMAF and (b) P.1204.3 scores for all representations of all videos encoded using the HLS ladder after upscaling to a resolution of 4 K.</figcaption></figure>

calculated between the original video and the upscaled representation using the VMAF model vmaf_4k_v0.6.1.<fnref n="4" /> To compute P.1204.3 scores, the ITU-T P.1204.3 reference implementation<fnref n="5" /> is used.

The box plot in Fig. 3 illustrates the distribution of VMAF and P.1204.3 scores for all video representations in the dataset. It is evident that there is significant variation in both VMAF and P.1204.3 scores across different videos at all bitrates. For instance, the VMAF scores for videos encoded at 600 kbps range from 18 to 60, indicating a substantial variability. Hence, it is crucial to predict these scores using low-complexity features. The VMAF and P.1204.3 values are used as separate target labels for prediction. We first evaluate the effectiveness of VQM4HAS in predicting VMAF and P.1204.3 scores for each representation. In the following experiments, 80% of videos are selected for training the model, while the remaining 20% are used for testing.

### A. Performance Analysis

To predict the per-segment VMAF and P.1204.3 scores for each representation in the bitrate ladder, we perform temporal pooling and calculate the average of the per-frame features for each segment. We then use both linear regression and random forest models to predict the scores. Tables IV and V summarize the Pearson correlation coefficient (PCC) [41] scores obtained when using VQM4HAS to predict per-segment VMAF and P.1204.3 scores, respectively. PCC measures the linear relationship between predicted and actual values, ranging from -1 (perfect negative correlation) to 1 (perfect positive correlation). It is observed that the PCC of VMAF is slightly lower only for the lowest bitrate (0.87), whereas with increasing bitrate, the PCC can reach as high as 0.96. The PCC results for per-segment P.1204.3 prediction are slightly better than those for VMAF prediction, ranging from 0.96 to 0.99.

To determine the contribution of each feature towards the prediction of VMAF and P.1204.3 scores, we extract the feature importance from the random forest model of each representation using SHAP (SHapley Additive exPlanations) [42]. We plot these feature importances in Figs. 4 and 5 for VMAF and P.1204.3, respectively. The analysis of the feature importance plots shows that in the lowest bitrate, the PSNR of the Y channel (y_psnr) has the highest importance in predicting the VMAF scores, followed by the temporal complexity of the original video ($h_{org}$), $QP$. The SHAP summary plot for all the videos in the test set for the lowest bitrate representation is presented in Fig. 4(b). In the midrange bitrates, it is indicated that luma distortion (luma_dist), SSIM, and Psy Energy play a dominant role in predicting the VMAF scores. With an increase in bitrate, $QP$, and SSIM emerge as the most significant feature in predicting the VMAF scores. The SHAP summary plot for all the videos in the test set for the highest bitrate representation is presented in Fig. 4(c). It is worth noting that for all representations, bit, chroma_dist, luma_avg, cr_avg, and frame_time have minimal influence on the model output.

Fig. 5 illustrates the feature importance for the per-segment prediction of P.1204.3 scores. The analysis indicates that, for predicting P.1204.3 scores, $QP$ is the only feature that has a significant impact, while other features have minimal influence. This is because $QP$ is explicitly extracted from the bitstream for P.1204.3 prediction, while other features are not used in the prediction of P.1204.3. In addition, we have calculated the correlation between VMAF and P.1204.3, and the results are presented in Table VI. Although there is no significant correlation between these two metrics, it is worth noting that VQM4HAS has the capability to accurately predict both of them with a significant level of correlation.

<footnote n="4">⁴https://github.com/Netflix/vmaf/blob/master/resource/doc/models.md</footnote>

<footnote n="5">⁵https://github.com/Telecommunication-Telemedia-Assessment/bitstream_mode3_p1204_3</footnote>
<table><caption>TABLE IV
PCC FOR VQM4HAS WHEN PREDICTING PER-SEGMENT VMAF SCORES</caption><thead><tr><th></th><th>Representation ID</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th></tr></thead><tbody><tr><th rowspan="2">Model</th><th>Linear</th><td>0.83</td><td>0.86</td><td>0.90</td><td>0.91</td><td>0.93</td><td>0.95</td><td>0.95</td><td>0.95</td><td>0.95</td><td>0.94</td><td>0.92</td><td>0.90</td></tr><tr><th>Random Forest</th><td>0.87</td><td>0.90</td><td>0.94</td><td>0.94</td><td>0.95</td><td>0.95</td><td>0.95</td><td>0.96</td><td>0.95</td><td>0.95</td><td>0.95</td><td>0.94</td></tr></tbody></table>

<table><caption>TABLE V
PCC FOR VQM4HAS WHEN PREDICTING PER-SEGMENT P.1204.3 SCORES</caption><thead><tr><th></th><th>Representation ID</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th></tr></thead><tbody><tr><th rowspan="2">Model</th><th>Linear</th><td>0.98</td><td>0.98</td><td>0.97</td><td>0.96</td><td>0.95</td><td>0.96</td><td>0.95</td><td>0.96</td><td>0.96</td><td>0.94</td><td>0.96</td><td>0.94</td></tr><tr><th>Random Forest</th><td>0.98</td><td>0.98</td><td>0.99</td><td>0.98</td><td>0.97</td><td>0.97</td><td>0.96</td><td>0.97</td><td>0.98</td><td>0.96</td><td>0.98</td><td>0.97</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 4. (a) The feature importance of the features used to predict VMAF scores for all representations. SHAP summary plot for (b) the lowest and (c) highest bitrate representations.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 5. (a) The feature importance of the features used to predict P.1204.3 scores for all representations. SHAP summary plot for (b) the lowest and (c) highest bitrate representations.</figcaption></figure>

<table><caption>TABLE VI
PCC BETWEEN VMAF AND P.1204.3</caption><thead><tr><th>Representation ID</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th></tr></thead><tbody><tr><th>PCC</th><td>0.50</td><td>0.62</td><td>0.72</td><td>0.69</td><td>0.70</td><td>0.81</td><td>0.80</td><td>0.80</td><td>0.78</td><td>0.76</td><td>0.76</td><td>0.69</td></tr></tbody></table>

In live video streaming, a fixed bitrate ladder is typically used, which makes it feasible to use a regression model for each representation. However, if a new bitrate-resolution pair is added to the ladder, a new regression model will be required. Considering that infinite bitrates are possible for encoding, we evaluate

VQM4HAS per resolution, as only a limited number of standard resolutions are used for encoding. To achieve this, we train a random forest model on the training set of the representations that share the same resolution. For instance, we train a random forest model on the training videos with bitrates of 600 kbps,
<table><caption>TABLE VII
PCC FOR VQM4HAS WHEN PREDICTING PER-RESOLUTION VMAF AND P.1204.3 SCORES</caption><thead><tr><th></th><th>Resolution</th><th>540p</th><th>720p</th><th>1080p</th><th>2160p</th></tr></thead><tbody><tr><th rowspan="2">VQM4HAS</th><th>VMAF prediction</th><td>0.95</td><td>0.96</td><td>0.96</td><td>0.96</td></tr><tr><th>P.1204.3 prediction</th><td>0.99</td><td>0.97</td><td>0.98</td><td>0.98</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Fig. 6. The feature importance of the features used to predict VMAF scores for (a) 540p, (b) 720p, (c) 1080p, and (d) 2160p resolutions.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 7. The scatter plot between the actual and predicted (a) VMAF and (b) P.1204.3 scores on the test set at a resolution of 540p.</figcaption></figure>

900 kbps, and 1600 kbps, all having a resolution of 540p. This model is used to predict the VMAF and P.1204.3 scores of the test set for the same representations. The PCC results for the resolutions that have more than one bitrate in the HLS ladder (Table III) are summarized in Table VII. The SHAP summary plot for all resolutions is shown in Fig. 6, which is consistent with the per-representation feature importance analysis. Note that similar to per-representation analysis, the *QP* feature is the dominant feature in predicting P.1204.3 scores per resolution. Fig. 7 illustrates the scatter plot between the actual and predicted VMAF and P.1204.3 scores on the test set at a resolution of 540p. It can be observed that *VQM4HAS* accurately predicts VMAF and P.1204.3 scores per resolution with high accuracy. It is important to note that the model remains resolution-dependent because basic video quality metrics are computed based on the resolution of the target representation.

Finally, we evaluate whether encoding statistics from a single representation can be effectively leveraged to predict the quality metrics of other representations. Specifically, we aim

to predict the VMAF and ITU-T P.1204.3 scores for all representations using features extracted from just one representation, thus eliminating the need for feature extraction across all representations. Fig. 8 visualizes the correlations between using one representation to predict others for the test videos. Notably, mid-range representations (representations 4-9) demonstrate strong overall performance in predicting the quality metrics of other representations, suggesting they are the most robust for this predictive task.

## B. Time-Complexity Analysis

VMAF and P.1204.3 have gained popularity due to their strong correlation with human perception, but their main concern is the time complexity required to calculate them. The computation of these metrics is resource-intensive and time-consuming, which makes them unsuitable for real-time video quality assessment, especially for applications such as live streaming. In this section, we compare the time complexity of these two metrics
<figure><img src="image_9.png" /><figcaption>Fig. 8. Correlations between using a single representation to predict the (a) VMAF and (b) P.1204.3 scores of other representations.</figcaption></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 9. (a) The time complexity of different methods when computing 12 representations of a 5-second video encoded with the HLS ladder parameters. The complexity breakdown across representations for (b) VQM4HAS, (c) VMAF, and (d) P.1204.3.</figcaption></figure>

with the proposed VQM4HAS. To this end, we calculate all the metrics on an Amazon EC2 c5.4xlarge instance, assuming that the videos are being watched on a 4 K display. Therefore, for the VMAF calculation, the representations are upscaled to 4 K, which is also the resolution of the original video. This means that the time complexity of the VMAF calculation is independent of the resolution of the representation, while P.1204.3⁶ and VQM4HAS are dependent on the bitrate and resolution of the representation. Fig. 9 compares the time complexity of VQM4HAS, VMAF, and P.1204.3 when used to compute all 12 representations of a 5-second video encoded with the HLS ladder parameters. Additionally, the figure shows the time complexity of all three metrics for each representation. The results indicate that the VQM4HAS exhibits significantly lower time complexity in comparison to VMAF and P.1204.3, making it suitable even for live-streaming applications. The maximum delay added by the

<table><caption>TABLE VIII
PCC BETWEEN MOS SCORES AND OBJECTIVE METRICS ON WATERLOO
DATASET [43]</caption><thead><tr><th></th><th>Resolution</th><th>540p</th><th>1080p</th><th>2160p</th></tr></thead><tbody><tr><th rowspan="5" scope="rowgroup">Method</th><th>PSNR</th><td>0.32</td><td>0.56</td><td>0.51</td></tr><tr><th>SSIM</th><td>0.46</td><td>0.68</td><td>0.66</td></tr><tr><th>VMAF</th><td>0.73</td><td>0.89</td><td>0.83</td></tr><tr><th>P.1204.3</th><td>0.94</td><td>0.94</td><td>0.90</td></tr><tr><th>VQM4HAS</th><td>0.96</td><td>0.93</td><td>0.93</td></tr></tbody></table>

computation of VQM4HAS is related to Representation 12 in the bitrate ladder, which has a delay of 2.5 ms *per frame*. This delay is much lower than that of real-time computation (33.3 ms or 30 fps).

### C. Comparison With MOS

While the primary objective of this paper is to propose a fast video quality metric for HAS applications as a potential replacement for VMAF and P.1204.3, we also evaluate our method's performance against subjective scores to assess its effectiveness compared to VMAF and P.1204.3. The strong correlation of VQM4HAS with VMAF and P.1204.3 suggests that it has the potential to produce results with similar correlation to subjective evaluations, as achieved by these metrics. However, comparison with subjective results is further investigated with an additional dataset to validate the effectiveness of VQM4HAS in real-world scenarios. To accomplish this, we opted to use the Waterloo subjective test [43] results for our evaluation, which contain three resolutions and four distortion levels per resolution. To conduct the evaluation, we partitioned the videos into training and test sets for each resolution. We train VQM4HAS on a training set and then evaluate PSNR, SSIM, VMAF, P.1204.3, and VQM4HAS on the test set by computing their correlation with MOS values. The results are summarized in Table VIII. It is observed that VQM4HAS is capable of predicting the subjective scores with high accuracy across different resolution-distortion scenarios, demonstrating its robustness. Moreover, these results

<footnote n="6">⁶https://github.com/Telecommunication-Telemedia-Assessment/bitstream_mode3_p1204_3</footnote>
indicate that VQM4HAS not only aligns well with existing
objective metrics but also maintains consistency with human vi-
sual perception, making it a promising candidate for widespread
adoption in adaptive streaming applications.

D. Discussion

VQM4HAS has demonstrated its efficiency in predicting video
quality metrics and can serve as a strong replacement for cur-
rent metrics that are highly correlated with human vision, such
as VMAF and P.1204.3. It can be seamlessly integrated into
per-title encoding methods [44], where knowing the quality of
representations is crucial. For example, it is well established
that encoding with a VMAF score higher than 94 falls within
the perceptually lossless range [45]. This means that increas-
ing the quality beyond this threshold does not lead to noticeable
improvements in QoE but results in a higher bitrate require-
ment. Consequently, this increases the risk of playback stalls
and significant QoE degradation, especially under bandwidth
fluctuations. With our method, we can quickly recognize such
encodings and avoid streaming them, ensuring efficient bitrate
allocation without compromising perceptual quality. Addition-
ally, similar to JND-based approaches [46], VQM4HAS can help
eliminate representations with similar quality, thereby reducing
streaming costs. Moreover, since VMAF is commonly used to
select the best resolution at the same bitrate in per-title encoding,
VQM4HAS can significantly streamline the resolution selection
process [47], [48]. In this approach, the video is encoded at
different resolutions, upscaled, and their VMAF scores are cal-
culated to determine the resolution with the highest VMAF at a
given bitrate. Our method accelerates this process by eliminat-
ing the need for explicit VMAF computation, enabling faster and
more efficient resolution selection. Additionally, VQM4HAS can
be embedded into the bitstream to guide adaptive bitrate (ABR)
algorithms [4], allowing them to select more efficient represen-
tations and enhance the overall Quality of Experience (QoE).
By incorporating VQM4HAS's quality predictions directly into
the streaming manifest or bitstream metadata, ABR algorithms
can make more informed decisions beyond traditional bitrate
and buffer-based heuristics. This enables the selection of rep-
resentations that provide the best perceptual quality while min-
imizing unnecessary bitrate increases. As a result, VQM4HAS
helps prevent inefficient quality jumps, optimizes bandwidth us-
age, and ensures a smoother playback experience, particularly
in fluctuating network conditions. However, one key drawback
of VQM4HAS is its dependency on the video codec. For each
codec, a new regression model must be trained, limiting its direct
application across different encoding formats without further ad-
justments.

IV. CONCLUSION

This paper introduced a new video quality metric called
VQM4HAS, which is capable of accurately predicting VMAF,
P.1204.3, and subjective scores with high correlation while
maintaining significantly lower computational complexity. In or-
der to accurately model perceived video quality, low complexity
features, including (i) video complexity features, (ii) bitstream

features, and (iii) basic video quality metrics, are extracted and
fed into a regression model. All of these features are logged dur-
ing the encoding of representations, incurring minimal costs,
and only the video complexity features are extracted from the
original video. Linear and random forest regression models were
evaluated for the study. Per-resolution evaluations demonstrated
that VQM4HAS is capable of predicting VMAF with a Pearson
correlation coefficient (PCC) ranging from 0.95 to 0.96, while
also predicting P.1204.3 with a PCC ranging from 0.97 to 0.99.
The prediction of VMAF is largely influenced by a few key
features depending on the resolution/representation, while for
the P.1204.3 model, the dominant feature is QP. VQM4HAS
exhibits significantly lower time complexity than VMAF and
P.1204.3, with a reduction of up to 98% and 99%, respectively,
when evaluating all 12 representations of the HLS ladder. This
makes it highly suitable for use in live-streaming applications.
Finally, the performance of VQM4HAS in predicting subjective
scores was evaluated, demonstrating comparable performance
to P.1204.3 and VMAF.

ACKNOWLEDGMENT

Christian Doppler Laboratory ATHENA: https://athena.itec.aau.at/.

REFERENCES

[1] Cisco, "Cisco visual networking index: Forecast and trends, 2018-2023," *White Paper*, Mar. 2020. [Online]. Available: https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.pdf

[2] I. Sodagar, "The MPEG-DASH standard for multimedia streaming over the internet," *IEEE Multimedia*, vol. 18, pp. 62–67, 2011.

[3] M. Seufert et al., "A survey on quality of experience of HTTP adaptive streaming," *IEEE Commun. Surveys Tuts.*, vol. 17, no. 1, pp. 469–492, First Quarter 2015.

[4] A. Bentaleb, B. Taani, A. C. Begen, C. Timmerer, and R. Zimmermann, "A survey on bitrate adaptation schemes for streaming media over HTTP," *IEEE Commun. Surveys Tuts.*, vol. 21, no. 1, pp. 562–585, First Quarter 2019.

[5] B. Taraghi, M. Nguyen, H. Amirpour, and C. Timmerer, "Intense: In-depth studies on stall events and quality switches and their impact on the quality of experience in HTTP adaptive streaming," *IEEE Access*, vol. 9, pp. 118087–118098, 2021.

[6] H. Zhang et al., "DeepQoE: A multimodal learning framework for video quality of experience (QoE) prediction," *IEEE Trans. Multimedia*, vol. 22, pp. 3210–3223, 2020.

[7] C.-C. Wu, K.-T. Chen, Y.-C. Chang, and C.-L. Lei, "Crowdsourcing multimedia QoE evaluation: A trusted framework," *IEEE Trans. Multimedia*, vol. 15, pp. 1121–1137, 2013.

[8] D. S. Hands, "A basic multimedia quality model," *IEEE Trans. Multimedia*, vol. 6, pp. 806–816, 2004.

[9] "Methods for the Subjective Assessment of Video Quality, Audio Quality and Audiovisual Quality of Internet Video and Distribution Quality Television in any Environment," ITU-R P.913, International Telecommunications Union, Geneva, Switzerland, Jul. 2021.

[10] "Mean Opinion Score (MOS) Terminology," Rec. ITU-T P.800.1, International Telecommunications Union, Geneva, Switzerland, 2003.

[11] "Methodologies for the Subjective Assessment of the Quality of Television Images," ITU-R BT.500-14, International Telecommunications Union, Geneva, Switzerland, Oct. 2019.

[12] "Subjective Video Quality Assessment Methods for Multimedia Applications", ITU-R P.910, International Telecommunications Union, Geneva, Switzerland, Jul. 2022.

[13] N. Barman and M. G. Martini, "QoE modeling for HTTP adaptive video streaming-A survey and open challenges," *IEEE Access*, vol. 7, pp. 30831–30859, 2019.
[14] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” *IEEE Trans. Image Process.*, vol. 13, no. 4, pp. 600–612, Apr. 2004.

[15] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in *Proc. 37th Asilomar Conf. Signals, Syst. Comput.*, Pacific Grove, CA, USA, 2003, pp. 1398–1402.

[16] M. H. Pinson and S. Wolf, “A new standardized method for objectively measuring video quality,” *IEEE Trans. Broadcast.*, vol. 50, no. 3, pp. 312–322, Sep. 2004.

[17] H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” *IEEE Trans. Image Process.*, vol. 15, no. 2, pp. 430–444, Feb. 2006.

[18] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity index for image quality assessment,” *IEEE Trans. Image Process.*, vol. 20, no. 8, pp. 2378–2386, Aug. 2011.

[19] D. M. Chandler and S. S. Hemami, “VSNR: A wavelet-based visual signal-to-noise ratio for natural images,” *IEEE Trans. Image Process.*, vol. 16, no. 9, pp. 2284–2298, Sep. 2007.

[20] K. Seshadrinathan and A. C. Bovik, “Motion tuned spatio-temporal quality assessment of natural videos,” *IEEE Trans. Image Process.*, vol. 19, no. 2, pp. 335–350, Feb. 2010.

[21] M. A. Saad, A. C. Bovik, and C. Charrier, “Blind prediction of natural video quality,” *IEEE Trans. Image Process.*, vol. 23, no. 3, pp. 1352–1365, Mar. 2014.

[22] A. Mittal, A. K. Moorthy, and A. C. Bovik, “Blind/Referenceless image spatial quality evaluator,” in *Proc. Conf. Rec. 45th Asilomar Conf. Signals, Syst. Comput.*, Pacific Grove, CA, USA, 2011, pp. 723–727.

[23] R. Soundararajan and A. C. Bovik, “Video quality assessment by reduced reference spatio-temporal entropic differencing,” *IEEE Trans. Circuits Syst. Video Technol.*, vol. 23, no. 4, pp. 684–694, Apr. 2013.

[24] Z. Li et al., VMAF: The Journey Continues, Netflix Technol. Blog: Los Gatos, CA, USA, 2018, vol. 25.

[25] R. Rassool, “VMAF reproducibility: Validating a perceptual practical video quality metric,” in *Proc. IEEE Int. Symp. Broadband Multimedia Syst. Broadcast.*, Cagliari, Italy, 2017, pp. 1–2.

[26] N. Barman, S. Schmidt, S. Zadtootaghaj, M. G. Martini, and S. Möller, “An evaluation of video quality assessment metrics for passive gaming video streaming,” in *Proc. 23rd Packet Video Workshop*, Amsterdam Netherlands, 2018, pp. 7–12.

[27] C. Lee et al., “Comparison of objective quality models for adaptive bit-streaming services,” in *Proc. 8th Int. Conf. Inf., Intell., Syst. Appl.*, Larnaca, 2017, pp. 1–4.

[28] Video Processing AI, “Video quality metrics benchmarks,” 2024. Accessed: Nov. 10, 2024. [Online]. Available: https://videoprocessing.ai/benchmarks/video-quality-metrics_both.html

[29] J. Yuchieh et al., “A fusion-based video quality assessment (fvqa) index,” in *Proc. Signal Inf. Process. Assoc. Annu. Summit Conf.*, Asia-Pacific, Chiang Mai, Thailand, 2014, pp. 1–5.

[30] S. Li, F. Zhang, L. Ma, and K. Ngi Ngan, “Image quality assessment by separately evaluating detail losses and additive impairments,” *IEEE Trans. Multimedia*, vol. 13, no. 5, pp. 935–949, Oct. 2011.

[31] A. K. Venkataramanan, C. Stejerean, and A. C. Bovik, “FUNQUE: Fusion of unified quality evaluators,” in *Proc. IEEE Int. Conf. Image Process.*, 2022, pp. 2147–2151.

[32] Apple, “HTTP live streaming (HLS) authoring specification for apple devices| apple developer documentation,” 2015. [Online]. Available: https://developer.apple.com/documentation/http-live-streaming/hls-authoring-specification-for-apple-devices

[33] B. Taraghi, H. Amirpour, and C. Timmerer, “Multi-codec ultra high definition 8K MPEG-DASH dataset,” in *Proc. 13th ACM Multimedia Syst. Conf.*, Athlone Ireland, 2022, pp. 216–220.

[34] R. R. R. Rao et al., “Bitstream-based model standard for 4K/UHD: ITU-T P.1204.3 — Model details, evaluation, analysis and open source implementation,” in *Proc. 12th Int. Conf. Qual. Multimedia Experience*, Athlone, Ireland, 2020, pp. 1–6.

[35] H. Amirpour, J. Zhu, P. Le Callet, and C. Timmerer, “A real-time video quality metric for HTTP adaptive streaming,” in *Proc. Int. Conf. Acoust., Speech, Signal Process.*, 2024, pp. 3810–3814.

[36] W. Robitza, R. R. Ramachandra, S. R. Göring, and A. Raake, “Impact of spatial and temporal information on video quality and compressibility,” in *Proc. 13th Int. Conf. Qual. Multimedia Experience*, 2021, pp. 65–68.

[37] M. King, Z. Tauber, and Z.-N. Li, “A new energy function for segmentation and compression,” in *Proc. IEEE Int. Conf. Multimedia Expo*, Beijing, China, 2007, pp. 1647–1650.

[38] V. V. Menon, C. Feldmann, H. Amirpour, M. Ghanbari, and C. Timmerer, “VCA: Video complexity analyzer,” in *Proc. 13th ACM Multimedia Syst. Conf.*, 2022, pp. 259–264.

[39] Q. Huynh-Thu and M. Ghanbari, “Scope of validity of PSNR in image/video quality assessment,” *Electron. Lett.*, vol. 44, no. 13, pp. 800–801, 2008.

[40] A. Stergiou and R. Poppe, “AdaPool: Exponential adaptive pooling for information-retaining downsampling,” *IEEE Trans. Image Process.*, vol. 32, pp. 251–266, 2023.

[41] K. Pearson, “Note on regression and inheritance in the case of two parents,” *Proc. Roy. Soc. London Ser. I*, vol. 58, pp. 240–242, Jan. 1895.

[42] S. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in *Proc. Adv. Neural Inf. Process. Syst.*, 2017, pp. 4768–4777.

[43] Z. Duanmu et al., “The Waterloo streaming quality-of-experience database-IV,” *IEEE Dataport*, Sep. 24, 2020, doi: 10.21227/j15a-8r35.

[44] V. V. Menon, H. Amirpour, M. Ghanbari, and C. Timmerer, “Perceptually-aware per-title encoding for adaptive video streaming,” in *Proc. Int. Conf. Multimedia Expo.* 2022, pp. 1–6.

[45] H. Amirpour, R. Schatz, and C. Timmerer, “Between two and six? Towards correct estimation of JND step sizes for VMAF-based bitrate laddering,” in *Proc. 14th Int. Conf. Qual. Multimedia Experience*, 2022, pp. 1–4.

[46] J. Zhu, H. Amirpour, R. Schatz, P. L. Callet, and C. Timmerer, “Beyond curves and thresholds - introducing uncertainty estimation to satisfied user ratios for compressed video,” in *Proc. Picture Coding Symp.*, 2024, pp. 1–5.

[47] A. V. Katsenou, J. Sole, and D. R. Bull, “Efficient bitrate ladder construction for content-optimized adaptive video streaming,” *IEEE Open J. Signal Process.*, vol. 2, pp. 496–511, 2021.

[48] V. V. Menon, H. Amirpour, M. Ghanbari, and C. Timmerer, “OPTE: Online per-title encoding for live video streaming,” in *Proc. Int. Conf. Acoust., Speech, Signal Process.*, 2022, pp. 1865–1869.

<figure><img src="image_12.png" /></figure>

**Hadi Amirpour** (Member, IEEE) is currently an Assistant Professor with the Institute of Information Technology, University of Klagenfurt, Klagenfurt, Austria. His research interests include video streaming, image and video compression, quality of experience, emerging 3D imaging technologies, and medical image analysis. He has contributed to standardization committees such as JPEG Pleno and MPEG and has been co-chairing Qualinet TF7 since 2021. He is an Associate Editor for IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY.

He was the recipient of the Best Paper Awards from PCS 2024 and NAB 2025. He has also played an active role in organizing conferences, special sessions, workshops, and tutorials at leading international events. Further information at https://hadiamirpour.github.io

<figure><img src="image_13.png" /></figure>

**Jingwen Zhu** received the B.S. and master's degrees from the Ocean University of China, Qingdao, China, the second master's degree from the Nantes University of France, and the Ph.D. degree from Nantes University, in 2024. Her research interests include human vision perception and multimedia.

<figure><img src="image_14.png" /></figure>

**Wei Zhou** (Senior Member, IEEE) received the Ph.D. degree jointly from the University of Science and Technology of China, Hefei, China, in 2021, and University of Waterloo, Waterloo, ON, Canada. Wei Zhou was a Visiting Scholar with the National Institute of Informatics, Chiyoda City, Japan, Research Assistant with Intel, Research Intern with Microsoft Research and Alibaba Cloud, and Postdoctoral Fellow with the University of Waterloo. Wei Zhou is currently an Assistant Professor with Cardiff University, Cardiff, U.K. Wei Zhou's research interests include multimedia computing, perceptual image processing, and computational vision.

Wei Zhou is also an Associate Editor for IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, *Pattern Recognition*, and *Neurocomputing*.
<figure><img src="image_15.png" /></figure>

**Patrick Le Callet** (Fellow, IEEE) is currently a Full Professor of EE and CS with Nantes Université, Nantes, France. He is the coauthor of more than 350 publications and communications and Co-Inventor of 16 international patents in his research interests, which include twin transition addressing quality of experience assessment for sustainable visual communication, quality of life measurement for better healthcare or inclusive technologies. He is also a Senior Member of the Institut Universitaire de France (IUF), France. He is with the Steering Board of the

CNRS LS2N lab (450 researchers). He is leading a multi-disciplinary team to conduct inter disciplinary research dealing with the application of human perception in media processing and cognitive computing, including all form of artificial intelligence. He is/was an Associate Editor or Guest Editor of several journals, such as *IEEE Signal Processing Magazine*, *IEEE TRANSACTIONS ON IMAGE PROCESSING*, *IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING*, *IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY*, *SPRINGER EURASIP Journal on Image and Video Processing*, and SPIE JEI. He has been in IEEE IVMSP-TC since 2015 and IEEE MMSP-TC since 2015 and Chair of EURASIP TAC (Technical Area Team) on Visual Image Processing. He is chairing activities in Standard (VQEG and IEEE-SA). He was the co-recipient of Emmy Award in 2020 for his work on development of Perceptual metrics for video encoding optimization.

<figure><img src="image_16.png" /></figure>

**Christian Timmerer** (Senior Member, IEEE) is currently a Full Professor of computer science with the Institute of Information Technology and Director with Christian Doppler Laboratory ATHENA (https://athena.itec.aau.at/). In 2013, he cofounded Bitmovin (http://www.bitmovin.com/) to provide professional services around MPEG-DASH, where he is the Chief Innovation Officer (CIO) — Head of Research and Standardization. He has coauthored seven patents and more than 300 articles in his research interests which include immersive multimedia communica-

tion, streaming, adaptation, and quality of experience. He was the General Chair of WIAMIS 2008, QoMEX 2013, MMSys 2016, and PV 2018 and has partici- pated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, COST IC1003 QUALINET, ICoSOLE, and SPIRIT. He also participated in ISO/MPEG work for several years, notably in the area of MPEG-21, MPEG-M, MPEG-V, and MPEG-DASH, where he was the Standard Editor. Further information at http://timmerer.com.