# Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding

Vignesh Menon, Jingwen Zhu, Prajit Rajendran, Samira Afzal, Klaus

Schoeffmann, Patrick Le Callet, Christian Timmerer

► To cite this version:

Vignesh Menon, Jingwen Zhu, Prajit Rajendran, Samira Afzal, Klaus Schoeffmann, et al. Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding. MHV '24: Mile-High Video Conference, Mar 2024, Denver CO USA, France. pp.61-67, 10.1145/3638036.3640807. hal-05114285

HAL Id: hal-05114285

https://hal.science/hal-05114285v1

Submitted on 21 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding

Vignesh V Menon

vignesh.menon@hhi.fraunhofer.de

Video Communication and Applications Dept

Fraunhofer HHI

Berlin, Germany

Samira Afzal

samira.afzal@au.at

Institute of Information Technology

Alpen-Adria-Universität

Klagenfurt, Austria

Jingwen Zhu

jingwen.zhu@etu.univ-nantes.fr

Nantes Université

École Centrale Nantes, CNRS, LS2N, UMR 6004

Nantes, France

Klaus Schoeffmann

klaus.schoeffmann@au.at

Institute of Information Technology

Alpen-Adria-Universität

Klagenfurt, Austria

Prajit T Rajendran

prajit.thazhurazhikath@cea.fr

CEA, List, F-91120 Palaiseau

Université Paris-Saclay

Paris, France

Patrick Le Callet

patrick.lecallet@univ-nantes.fr

Nantes Université

École Centrale Nantes, IUF, CNRS, LS2N, UMR

6004

Nantes, France

Christian Timmerer

christian.timmerer@au.at

Institute of Information Technology

Alpen-Adria-Universität

Klagenfurt, Austria

## ABSTRACT

In HTTP adaptive live streaming applications, video segments are encoded at a fixed set of bitrate-resolution pairs known as *bitrate ladder*. Live encoders use the fastest available encoding configuration, referred to as *preset*, to ensure the minimum possible latency in video encoding. However, an optimized preset and optimized number of CPU threads for each encoding instance may result in (i) increased quality and (ii) efficient CPU utilization while encoding. For *low latency* live encoders, the encoding speed is expected to be more than or equal to the video frame rate. To this light, this paper introduces a *Just Noticeable Difference (JND)-Aware Low latency Encoding Scheme (JALE)*, which uses random forest-based models to jointly determine the optimized encoder preset and thread count for each representation, based on video complexity features, the target encoding speed, the total number of available CPU threads, and the target encoder. Experimental results show that, on average, JALE yield a quality improvement of 1.32 dB PSNR and 5.38 VMAF points with the same bitrate, compared to the fastest preset encoding of the HTTP Live Streaming (HLS) bitrate ladder using x265 HEVC open-source encoder with eight CPU threads used for each representation. These enhancements are achieved while maintaining the desired encoding speed. Furthermore, on average, JALE results in an overall storage reduction of 72.70%, a reduction in the total number of CPU threads used by 63.83%, and a 37.87% reduction in the overall encoding time, considering a JND of six VMAF points.

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution International 4.0
License.

MHV '24, February 11–14, 2024, Denver, CO, USA

© 2024 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-0493-2/24/02.

https://doi.org/10.1145/3638036.3640807

## CCS CONCEPTS

* Information systems → Multimedia streaming.

## KEYWORDS

Live streaming, low latency, encoder preset, CPU threads, HEVC.

## ACM Reference Format:

Vignesh V Menon, Jingwen Zhu, Prajit T Rajendran, Samira Afzal, Klaus Schoeffmann, Patrick Le Callet, and Christian Timmerer. 2024. Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding. In *Mile-High Video Conference (MHV '24), February 11–14, 2024, Denver, CO, USA.*, 7 pages. https://doi.org/10.1145/3638036.3640807

## 1 INTRODUCTION

The Moving Picture Experts Group (MPEG) has developed a standard called *Dynamic Adaptive Streaming over HTTP* (MPEG-DASH) [1] to meet the high demand for streaming high-quality video content over the Internet and overcome the associated challenges in *HTTP Adaptive Streaming* (HAS) [2]. The main idea behind HAS is to divide the video content into *segments* and to encode each segment at various bitrates and resolutions, called *representations*. These representations enable a continuous adaptation of the video delivery to the client's network conditions and device capabilities [3]. The increase in the computational complexity using codecs such as *High Efficiency Video Coding* (HEVC) [4] and *Versatile Video Coding* (VVC) [5], and improvements in video characteristics such as resolution [6], frame rate [7], and bit-depth raises the need to develop a large-scale, highly efficient video encoding environment [8]. This is crucial for DASH-based content provisioning as it requires encoding multiple representations of the same video content in an *encoding server*.

**Motivation:** Traditionally, a fixed bitrate ladder, e.g., HTTP Live Streaming (HLS) bitrate ladder [9], is used in live streaming applications. Furthermore, for every representation, maintaining an
<figure><img src="image_4.png" /><figcaption>**Figure 1:** The encoding speed of each representation in HLS bitrate ladder [9] for the Wood_s000 sequence [13] using ultrafast preset of x265 [14] using 4, 8, and 16 CPU threads for each representation.</figcaption></figure>

encoding speed that is the same as or greater than the video framerate, regardless of the complexity of the video content, is a crucial goal for a low latency live encoder [10]. Although the output video's compression efficiency (in terms of the obtained perceptual quality and bitrate) is an essential metric for the encoder, maintaining the encoding speed takes precedence in live streaming scenarios. This is because a reduction in encoding speed may lead to the unacceptable outcome of dropped frames during transmission, eventually decreasing the quality of experience [10]. The encoding speed depends on video content complexity and parameters such as (i) target resolution, (ii) target bitrate, (iii) number of CPU threads [11], and (iv) encoder configuration [12].

**Optimized resource allocation in encoding servers:** In adaptive streaming, strategically allocating an optimized number of CPU threads for each video encoder instance at a cloud server is crucial. Tailoring CPU thread count to encoding resolution and target bitrate allows for precise resource allocation, enhancing the efficiency of the encoding process. The cloud server can significantly streamline the encoding process by dynamically adjusting CPU thread counts based on resolution and bitrate, accommodating diverse video qualities within an adaptive streaming environment. Figure 1 shows the encoding time measurement of an entire HLS HEVC bitrate ladder encoding of the Wood_s000 sequence [13] using ultrafast preset of x265 [14] with 4, 8, and 16 CPU threads for each of the twelve encoding representations. Since the video (segment) is of 30 fps, the target encoding speed is considered 30 fps [10]. However, specific thread configurations cannot deliver certain representations' desired encoding speed of 30 fps. For example, the 11.6 Mbps and 16.8 Mbps representations do not achieve the target encoding speed (30 fps) using 4 or 8 threads. This emphasizes the impact of CPU utilization and thread configuration on the encoding performance.

**Optimized encoding preset:** Traditional open-source encoders like x264 [15], x265 [14], and VVenC [16] have pre-defined sets of encoding parameters (termed as *presets*), which present a trade-off between the encoding time and compression efficiency [7, 17]. The preset for the fastest encoding (*ultrafast* for x264 and x265) is used as the encoder preset for the entire live content, independent of the video content complexity [10]. Moreover, the streaming service provider arbitrarily chooses the number of CPU threads for each representation, irrespective of the content complexity. Though the

conservative technique of fixing the preset and thread count for each encoding instance may achieve the intended result of a low latency encoding, the resulting encoding is sub-optimal, especially when the type of the content is dynamically changing, which is the typical use case for live streams [18]. Furthermore, when the content becomes easier to encode (i.e., slow-moving videos or videos that have simpler textures are easy to encode as predicting the current frame from a previous frame is simpler, resulting in smaller residuals), the encoder would achieve a higher encoding speed than the target encoding speed. This, in turn, introduces unnecessary CPU idle time as it waits for the video feed. If the encoder preset is configured such that this higher encoding speed can be reduced while still being compatible with the expected live encoding speed, the quality of the encoded content achieved by the encoder can be improved. Subsequently, when the content becomes complex again, the encoder preset needs to be reconfigured to move back to the faster configuration that achieves live encoding speed [7, 19]. By employing efficient storage techniques and removing unnecessary representations, the energy consumption associated with storing and transmitting redundant data can be minimized [20].

**Contributions:** This paper proposes a Just Noticeable Difference (JND)-aware low latency encoding scheme (JALE) that jointly determines the CPU thread count and encoder preset configuration for each bitrate representation dynamically, adaptive to the video content to achieve low latency encoding. Content-aware features, i.e., Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features, are extracted to determine video segments' characteristics, which random forest-based models use to predict optimized thread count and encoder preset for each representation to maintain the target encoding speed. JALE achieves the desired target encoding speed while maximizing compression efficiency and minimizing the total CPU threads used. Furthermore, based on JND, JALE removes perceptual redundancy between representations in the bitrate ladder.

**Paper outline:** The remainder of this paper is organized as follows. Section 2 describes the proposed JALE encoding architecture. In Section 3, the performance of JALE is evaluated, and Section 4 concludes the paper.

## 2 JALE ARCHITECTURE

The architecture of JALE for streaming applications is presented in Figure 2, according to which the number of threads and encoder preset for every segment in each representation of the bitrate ladder is predicted using spatiotemporal features of the input video segment, the target video encoding speed ($s_T$), the set of pre-defined supported thread count per instance ($C$), and the set of pre-defined encoder presets ($\mathcal{P}$). The encoding process is carried out with the predicted encoder preset and the number of threads for each video segment. JALE is classified into three steps:

(1) video complexity feature extraction,

(2) joint thread count and preset prediction,

(3) perceptually-redundant representation elimination.
<figure><img src="image_5.png" /><figcaption>Figure 2: Live encoding using JALE envisioned in this paper.</figcaption></figure>

## 2.1 Video complexity feature extraction

Predictive models can comprehensively understand the content complexity and characteristics by extracting relevant spatiotemporal features, such as motion vectors, texture patterns, and frame-to-frame differences [21]. In this paper, three DCT-energy-based features [22], the average luma texture energy ($E_Y$), the average gradient of the luma texture energy ($h$), and the average luminance ($L_Y$), for each segment are extracted using open-source Video Complexity Analyzer (VCA) [6, 22].

## 2.2 Joint thread count and preset prediction

Selection of the optimized thread count-preset pair for each segment per representation based on the video content complexity is decomposed into two parts:

(1) train models to predict the encoding speed for each thread count-preset pair,

(2) develop a function to obtain the optimized thread count-preset pair for each representation.

The encoding speed of the $t^{th}$ representation of the input video segment ($s_t$) is modeled as a function of the video content complexity features, the target representation (resolution $r_t$ and bitrate $b_t$) [7], the number of threads $n_t$, and the encoder preset $p_t$, as shown in the equation:

$$s_t = f_S(E_Y, h, L_Y, r_t, b_t, n_t, p_t) \quad (1)$$

We use random forest models [23] to predict the encoding speed for each thread count-preset pair. ($P_T \times C_T$) models are trained, where $P_T$ and $C_T$ represent the number of encoding presets and the number of supported thread count per instance, respectively.

The optimized thread count-preset prediction function has a look-up table of ($\hat{n}_t, \hat{p}_t$) pairs. The supported encoder presets are chosen based on the target encoder and the preference of the streaming service provider. For example, presets ranging from ultrafast to veryslow can be chosen for x264 and x265 encoders. The set of possible thread counts ($C$) for every encoder instance is input by the streaming service provider based on the encoding server architecture. The priority of ($\hat{n}_t, \hat{p}_t$) pairs is decided based on the following constraints:

(1) the achieved encoding speed $\hat{s}_t$ of the $t^{th}$ representation must be greater than or equal to the target encoding speed $s_T$, i.e., $\hat{s}_t \ge s_T$.

<figure><img src="image_6.png" /><figcaption>Figure 3: ($\hat{n}, \hat{p}$) look-up table used in the experimental validation of this paper.</figcaption></figure>

(2) total number of CPU threads used for each representation is minimized.

An example look-up for x265 encoder is shown in Figure 3, where $C: \{4, 8, 12, 16, 20, 24\}$, and $P: \{\text{medium}, \text{fast}, \text{faster}, \text{veryfast}, \text{superfast}, \text{ultrafast}\}$. When the look-up table is scanned in the priority order, if the ($\hat{n}_t, \hat{p}_t$) pair yields an encoding speed higher than $f_T$, it is chosen as the optimized thread count-preset pair for the $t^{th}$ representation.

## 2.3 Perceptually-redundant representation elimination

JALE uses the JND-based representation elimination algorithm proposed in our previous work [7]. However, it is described in this paper (cf. Algorithm 1) to make it self-contained. The perceptual quality of the $t^{th}$ representation ($v_t$) is modeled as a function of the video content complexity features, the target representation (resolution $r_t$ and bitrate $b_t$), and the encoder preset $p_t$, as shown in the equation [7, 24]:

$$v_t = f_V(E_Y, h, L_Y, r_t, b_t, p_t) \quad (2)$$

Random forest models [23] are trained to predict the perceptual quality of each representation.

This paper uses VMAF as the perceptual quality metric for each representation, while other quality metrics can be envisioned and are subject to future work. In practice, it is often observed that the VMAF scores of different representations are very similar, which introduces perceptual redundancy in the bitrate ladder. To address this issue, this paper leverages the concept of the JND threshold, which represents the minimum threshold at which the human eye
**Algorithm 1:** JND-based representation elimination.

**Input:**

$q$: number of representations in $\mathcal{R}$

$$ \mathcal{R} = \bigcup_{t=1}^{q} \{(r_t, b_t, \hat{n}_t, \hat{p}_t)\} $$

representations with predicted thread count and preset

$\hat{v}_t$; $1 \le t \le q$: predicted VMAF

$v_T$: maximum VMAF threshold

$v_j$: average target JND

**Output:** $\hat{\mathcal{R}} = (r, b, \hat{n}, \hat{p})$: set of encoding configurations

$$ \hat{\mathcal{R}} \leftarrow \{(r_1, b_1, \hat{n}_1, \hat{p}_1)\} $$

$u \leftarrow 1$

if $\hat{v}_1 \ge v_T$ then

return $\hat{\mathcal{R}}$

$t \leftarrow 2$

while $t \le q$ do

if $\hat{v}_t - \hat{v}_u \ge v_j$ then
$\hat{\mathcal{R}} \leftarrow \hat{\mathcal{R}} \cup \{(r_t, b_t, \hat{n}_t, \hat{p}_t)\}$
$u \leftarrow t$
if $\hat{v}_t \ge v_T$ then
return $\hat{\mathcal{R}}$
$t \leftarrow t+1$

return $\hat{\mathcal{R}}$

can perceive differences in quality [25–27]. The paper aims to eliminate perceptually redundant representations by utilizing the JND threshold. While [28, 29] have explored VMAF-based JND thresholds, their complexity is unsuitable for live-streaming applications. Therefore, this paper adopts a fixed JND threshold [30, 31] denoted as $v_j$, an input from the streaming service provider. If the difference in the predicted VMAF of two representations is smaller than $v_j$, the representation with a higher bitrate will be eliminated. If the predicted VMAF of a representation is larger than $v_T$, ($v_T = 100 - v_j$), i.e., the threshold above which the representation is deemed perceptually lossless, the corresponding representation is eliminated from the bitrate ladder. In this manner, overall storage consumption and encoding energy of representations of an input video segment is reduced.

# 3 EVALUATION

This section introduces the test methodology used in this paper and presents the experimental results.

## 3.1 Test Methodology

We use four hundred sequences (80% of the sequences) from the video complexity dataset [13] as the training dataset and the remaining (20%) as the test dataset. We encode the sequences at 30 fps using x265 v3.5 [14] with multi-threading and x86 SIMD [32] optimizations. The experimental parameters used in this paper are listed in Table 1. We achieve CBR encoding for a target bitrate of $b_t$ (in Mbps) by setting the *bitrate* and *vbv-maxrate* option of x265 as $b_t$, and enabling *strict-cbr* mode. We run all experiments on a dual-processor server with Intel Xeon Gold 5218R (80 cores, frequency at 2.10 GHz). We consider the bitrate ladder from HLS authoring specification for Apple devices [9]. We consider $v_j$ as two [30], four, and six [31] based on current industry practices.

**Algorithm 2:** Dataset generation.

**Inputs:**

$Q$: set of supported resolutions

$\mathcal{B}$: set of target bitrates

$C$: set of thread count for each encoding instance

for each video segment do

Determine $\{E_Y, h, L_Y\}$

for each $r \in Q$ do

for each $b \in \mathcal{B}$ do

for each $c \in C$ do

Encode segment with CBR $b$;

Record $E_Y, h, L_Y, r, b$, achieved bitrate $b'$,

VMAF $v$, and PSNR $s$;

**Prediction models:** To ensure the robustness and generalization of the prediction models, we perform a five-fold cross-validation scheme for video sequences and average the results. The scheme also ensures splitting the test and training segments. We perform the hyperparameter tuning on the random forest prediction models on the ultrafast preset to balance the size and prediction accuracy of the models. The selected hyperparameters [33] are `min Samples Leaf= 1`, `min Samples Split= 2`, `n estimators= 100`, `max depth= 14`.

**Benchmark schemes:** We compare JALE with the following encoding schemes:

(1) Default: ultrafast preset with eight threads for each encoding instance [9].

(2) Bruteforce: optimized thread count-preset pair with and without JND-based representation elimination when the models are fully accurate. This is accomplished by bruteforce encoding using all thread count-preset pairs and selecting the optimized pair [8]. Hence, it is suitable only for video-on-demand applications.

(3) CAPS [34] determines the optimized preset for each representation for a target encoding speed of 30 fps. We evaluate CAPS where $c=4$, 8, and 16, respectively.

**Performance metrics:** We compare JALE with the benchmark schemes using BDR<sub>P</sub> and BDR<sub>V</sub> [35], which refers to the average increase in bitrate of the representations compared to the reference bitrate ladder encoding scheme to maintain the same PSNR and VMAF. A negative BDR suggests a boost in the coding efficiency of the considered encoding scheme compared to the reference bitrate ladder encoding scheme. Furthermore, we calculate BD-PSNR and BD-VMAF, which refer to the average increase in PSNR and VMAF at the same bitrate compared with the reference bitrate ladder encoding scheme. Positive BD-PSNR and BD-VMAF denote an increase in the coding efficiency of the considered encoding scheme compared to the reference bitrate ladder encoding. Relative storage space difference between the considered encoding scheme $b_{opt}$ and the reference encoding scheme $b_{ref}$ to store all bitrate ladder representations is evaluated as:

$$ \Delta S = \frac{\sum b_{opt}}{\sum b_{ref}} - 1. \qquad (3) $$
<table><caption>Table 1: Experimental parameters of JALE used in this paper.</caption><thead><tr><th colspan="2">Parameter</th><th>Symbol</th><th colspan="10">Values</th></tr></thead><tbody><tr><td rowspan="2">Set of representations</td><td>Resolution height [pixels]</td><td>R</td><td>360</td><td>432</td><td>540</td><td>540</td><td>540</td><td>720</td><td>720</td><td>1080</td><td>1080</td><td>1440</td><td>2160</td><td>2160</td></tr><tr><td>Bitrate [Mbps]</td><td></td><td>0.145</td><td>0.300</td><td>0.600</td><td>0.900</td><td>1.600</td><td>2.400</td><td>3.400</td><td>4.500</td><td>5.800</td><td>8.100</td><td>11.600</td><td>16.800</td></tr><tr><td colspan="2">Set of presets [x265]</td><td>P</td><td colspan="10">0 (ultrafast) - 5 (medium)</td></tr><tr><td colspan="2">Set of supported thread counts</td><td>C</td><td colspan="2">4</td><td colspan="2">8</td><td colspan="2">12</td><td colspan="2">16</td><td colspan="2">20</td><td colspan="2">24</td></tr><tr><td colspan="2">Total CPU threads</td><td>N</td><td colspan="10">96</td></tr><tr><td colspan="2">Encoding speed threshold [fps]</td><td>s<sub>T</sub></td><td colspan="10">30</td></tr><tr><td colspan="2">Average target JND</td><td>v<sub>J</sub></td><td colspan="2">2</td><td colspan="2">4</td><td colspan="2">4</td><td colspan="2">6</td><td colspan="2">6</td><td colspan="2">6</td></tr><tr><td colspan="2">Maximum VMAF threshold</td><td>v<sub>T</sub></td><td colspan="2">98</td><td colspan="2">98</td><td colspan="2">96</td><td colspan="2">96</td><td colspan="2">94</td><td colspan="2">94</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>(a) average thread count-preset</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) average encoding speed</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(c) average VMAF</figcaption></figure>

<caption>Figure 4: Results for each representation in JALE. JND-based representation elimination is not considered in these plots.</caption>

We further determine the relative difference in the CPU thread count ($\Delta N$) needed for all bitrate ladder representations of the considered encoding schemes. We measure encoding energy consumption using the Running Average Power Limit (RAPL) interface and the CodeCarbon tool [36].

## 3.2 Experimental Results

**Predictions:** Figure 4a shows the average encoder preset and the number of CPU threads chosen for each representation of the bitrate ladder across the test dataset. On average, the 0.145 Mbps representation chooses medium preset ($p = 5$) and 4 threads, while 11.6 Mbps and 16.8 Mbps representations choose ultrafast preset ($p = 0$) and 24 threads. This is because faster encoding presets and more computational resources are needed to encode high-bitrate representations, such that the encoding speed is above the threshold. Figure 4b shows the average encoding speed for every representation in the bitrate ladder using the default encoding and JALE. It is observed that, on average, the JALE encodings have speeds above the lower bound of $s_T = 30$ fps. It is also observed that default encoding has a high encoding speed for lower bitrate representations. This scenario occurs when the encoding tasks are relatively less demanding regarding CPU resources, allowing the CPU to remain substantially under-utilized while executing the encoding operations during a live feed. Furthermore, it could not achieve the minimum target encoding speed at higher bitrates (11.6 Mbps and 16.8 Mbps). However, JALE controls the encoding speed to be greater than $s_T$ but not significantly higher than the default encoding. This ensures higher CPU utilization when the encodings are carried out concurrently during a live feed.

**Rate-distortion performance:** Figure 4c shows the average VMAF for every representation in the bitrate ladder using the default, CAPS and JALE encodings, while Figure 5 shows the RD curves of the representative video segments of various video complexities with encoding using benchmark schemes, and JALE ($v_J = 6$). It is observed that the VMAF achieved by JALE is higher than or close to CAPS encoding and consistently higher than the default encoding at the same target bitrates. Moreover, perceptually redundant representations are eliminated in JALE. Furthermore, we observe a substantial improvement in quality at lower bitrate representations, owing to the selection of slower presets.

**Bjontegaard delta rates (BDR):** We evaluate the coding efficiency using BDR<sub>P</sub>, BDR<sub>V</sub>, BD-PSNR, and BD-VMAF compared to the default encoding, as shown in Table 2. Bruteforce encoding [8] yields 100% accurate results representing the highest bound of the compression efficiency improvement (in VMAF) compared to the default encoding. Compared to the default encoding, the coding efficiency improvement achieved by JALE is similar to the brute-force encoding. Hence, the prediction models used in JALE are deemed fairly accurate. Using JALE ($v_J=6$), we observe an average bitrate reduction of 25.93%, 25.47% to maintain the same PSNR and VMAF. Furthermore, we observe an average quality improvement of 1.32 dB PSNR and 5.38 VMAF points, respectively, at the same target bitrate.

**Storage consumption:** We evaluate the relative difference in the storage space between the considered encoding schemes and the default encoding scheme to store all bitrate ladder representations. When the JND value increases, the number of representations in the bitrate ladder decreases, causing a decrease in the overall storage
<figure><img src="image_10.png" /><figcaption>Figure 5: Rate-distortion (RD) curves of representative sequences (segments) for Default [9] encoding (blue line), CAPS (c = 8) [34] encoding (red line), compared to JALE ($v_J = 6$).</figcaption></figure>

<table><caption>Table 2: Average results of the encoding schemes compared to the default encoding.</caption><thead><tr><th>Method</th><th>$v_J$</th><th>BDR<sub>P</sub> [%]</th><th>BDR<sub>V</sub> [%]</th><th>BD-PSNR [dB]</th><th>BD-VMAF</th><th>ΔS [%]</th><th>ΔN [%]</th><th>ΔE [%]</th></tr></thead><tbody><tr><td rowspan="3">Bruteforce</td><td>2</td><td>-23.85</td><td>-23.02</td><td>1.21</td><td>4.61</td><td>-51.32</td><td>-43.42</td><td>1688.91</td></tr><tr><td>4</td><td>-24.54</td><td>-24.27</td><td>1.25</td><td>5.26</td><td>-62.13</td><td>-53.08</td><td>1688.91</td></tr><tr><td>6</td><td>-26.30</td><td>-25.77</td><td>1.32</td><td>5.53</td><td>-72.03</td><td>-63.38</td><td>1688.91</td></tr><tr><td>CAPS (c = 4)</td><td>-</td><td>-12.31</td><td>-14.98</td><td>0.82</td><td>2.91</td><td>-0.06</td><td>-50.00</td><td>8.65</td></tr><tr><td>CAPS (c = 8)</td><td>-</td><td>-20.26</td><td>-20.95</td><td>1.06</td><td>4.03</td><td>-0.09</td><td>0</td><td>39.68</td></tr><tr><td>CAPS (c = 16)</td><td>-</td><td>-28.13</td><td>-29.35</td><td>1.62</td><td>6.04</td><td>-0.35</td><td>100.00</td><td>96.84</td></tr><tr><td rowspan="3">JALE</td><td>2</td><td>-23.30</td><td>-22.80</td><td>1.19</td><td>4.47</td><td>-51.34</td><td>-43.13</td><td>-2.69</td></tr><tr><td>4</td><td>-24.41</td><td>-24.08</td><td>1.25</td><td>5.05</td><td>-62.95</td><td>-54.08</td><td>-22.70</td></tr><tr><td>6</td><td>-25.93</td><td>-25.47</td><td>1.32</td><td>5.38</td><td>-72.70</td><td>-63.83</td><td>-37.87</td></tr></tbody></table>

space needed for the representations of an input video segment. JALE yields average storage reduction of up to 72.70 % for various target $v_J$ values.

**Encoding energy consumption:** We conduct a comprehensive evaluation of encoding schemes by analyzing the relative differences in energy consumption during encoding (ΔE<sub>enc</sub>) of the bitrate ladder compared to the default encoding scheme. Predictably, compared to Default, and JALE encoding, CAPS (c = 16) yields the highest encoding energy consumption, owing to the selection of slower presets at all bitrates, as the constraint on total CPU threads is not considered. JALE yields the lowest encoding energy compared to other benchmark methods. Moreover, as $v_J$ increases, the energy consumption is lowered.

## 4 CONCLUSIONS

This paper proposed JALE, a JND-aware low latency encoding scheme for adaptive live streaming applications. JALE jointly predicts the optimized encoder preset and CPU thread count for a given representation for each video segment based on the video content complexity features, target encoding speed, and the total number of available CPU threads. It helps improve quality and CPU utilization

during encoding. Furthermore, the JND-based representation elimination algorithm removes perceptually redundant representations in the bitrate ladder. The performance of JALE is analyzed using the x265 open-source HEVC encoder for the HLS bitrate ladder encoding. It is observed that JALE yields an overall average quality improvement of 0.98 dB PSNR and 4.41 VMAF points at the same bitrate, compared to ultrafast encoding of the reference HLS bitrate ladder using eight CPU threads for each representation. Considering a JND of six VMAF points, storage, thread count, and encoding time reductions of 72.70 %, 63.83 %, and 37.87 %, respectively, are observed.

In the future, JALE will support the addition of encoder presets beyond the pre-defined options, enhancing efficiency in the encoding process. This may ensure better flexibility in selecting encoding parameters such that the achieved encoding speed is the same as the target. Furthermore, storage reduction techniques and representation elimination to enhance the overall energy efficiency of video streaming and improve the sustainability of video streaming systems shall be investigated.

## 5 ACKNOWLEDGMENT

The financial support of the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, Austrian Research Promotion Agency (FFG), grant agreement FO999897846 (GAIA) and the Christian Doppler Research Association is gratefully acknowledged. Christian Doppler Laboratory ATHENA: https://athena.itec.aau.at/.
REFERENCES

[1] Iraj Sodagar, "The MPEG-DASH Standard for Multimedia Streaming Over the Internet," in *IEEE MultiMedia*, vol. 18, no. 4, 2011, pp. 62–67.

[2] Theodoros Karagkious, Cyril Colangelo, Dimitrios Tsilimantos, and Stefan Valentin, "A Comparative Case Study of HTTP Adaptive Streaming Algorithms in Mobile Networks," in *Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video*, ser. NOSSDAV'17, 2017, p. 1–6.

[3] Abdelhak Bentaleb, Bayan Taani, Ali C. Begen, Christian Timmerer, and Roger Zimmermann, "A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP," in *IEEE Communications Surveys Tutorials*, vol. 21, no. 1, 2019, pp. 562–585. [Online]. Available: https://doi.org/10.1109/COMST.2018.2862938

[4] Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand, "Overview of the high efficiency video coding (HEVC) standard," in *IEEE Transactions on circuits and systems for video technology*, vol. 22, no. 12. IEEE, 2012, pp. 1649–1668.

[5] Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J. Sullivan, and Jens-Rainer Ohm, "Overview of the Versatile Video Coding (VVC) Standard and its Applications," in *IEEE Transactions on Circuits and Systems for Video Technology*, vol. 31, no. 10, 2021, pp. 3736–3764.

[6] Vignesh V Menon, Prajit T Rajendran, Christian Feldmann, Klaus Schoeffmann, Mohammad Ghanbari, and Christian Timmerer, "JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming," in *IEEE Transactions on Circuits and Systems for Video Technology*, 2023, pp. 1–1. [Online]. Available: https://doi.org/10.1109/TCSVT.2023.3290725

[7] Vignesh V Menon, Samira Afzal, Prajit T Rajendran, Klaus Schoeffmann, Radu Prodan, and Christian Timmerer, "Content-Adaptive Variable Framerate Encoding Scheme for Green Live Streaming." 2023. [Online]. Available: https://arxiv.org/pdf/2311.08074

[8] Jan De Cock, Zhi Li, Megha Manohara, and Anne Aaron, "Complexity-based consistent-quality encoding in the cloud," in *2016 IEEE International Conference on Image Processing (ICIP)*, 2016, pp. 1484–1488.

[9] Apple Inc., "HLS Authoring Specification for Apple Devices." [Online]. Available: https://developer.apple.com/documentation/http live streaming/hls authoring specification for apple devices

[10] Pradeep Ramachandran, Shushuang Yang, Praveen Tiwari, and Gopi Akisetty, "Content Adaptive Live Encoding with Open Source Codecs," in *Proceedings of the 11th ACM Multimedia Systems Conference*, ser. MMSys '20. New York, NY, USA: Association for Computing Machinery, 2020, p. 345–348. [Online]. Available: https://doi.org/10.1145/3339825.3393580

[11] Wei Xiao, Bin Li, Jizheng Xu, Guangming Shi, and Feng Wu, "HEVC encoding optimization using multicore CPUs and GPUs," in *IEEE Transactions on Circuits and Systems for Video Technology*, vol. 25, no. 11. IEEE, 2015, pp. 1830–1843.

[12] Vignesh V. Menon, "Video Coding Enhancements for HTTP Adaptive Streaming," in *Proceedings of the 30th ACM International Conference on Multimedia*, ser. MM '22. New York, NY, USA: Association for Computing Machinery, 2022, p. 6905–6909. [Online]. Available: https://doi.org/10.1145/3503161.3548753

[13] Hadi Amirpour, Vignesh V Menon, Samira Afzal, Mohammad Ghanbari, and Christian Timmerer, "VCD: Video Complexity Dataset," in *Proceedings of the 13th ACM Multimedia Systems Conference*. New York, NY, USA: Association for Computing Machinery, 2022, p. 234–239. [Online]. Available: https://doi.org/10.1145/3524273.3532892

[14] VideoLAN, "x265." [Online]. Available: https://www.videolan.org/developers/x265.html

[15] VideoLAN, "x264." [Online]. Available: https://www.videolan.org/developers/x264.html

[16] Adam Wieckowski, Jens Brandenburg, Tobias Hinz, Christian Bartnik, Valeri George, Gabriel Hege, Christian Helmrich, Anastasia Henkel, Christian Lehmann, Christian Stoffers, Ivan Zupancic, Benjamin Bross, and Detlev Marpe, "VVenC: An Open And Optimized VVC Encoder Implementation," in *Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW)*, pp. 1–2.

[17] Dieison Silveira, Marcelo Porto, and Sergio Bampi, "Performance and energy consumption analysis of the X265 video encoder," in *2017 25th European Signal Processing Conference (EUSIPCO)*, 2017, pp. 1519–1523.

[18] Qingxiong Huangyuan, Li Song, Zhengyi Luo, Xiangwen Wang, and Yanan Zhao, "Performance evaluation of H.265/MPEG-HEVC encoders for 4K video sequences," in *Signal and Information Processing Association Annual Summit and Conference (APSIPA)*, 2014 Asia-Pacific, 2014, pp. 1–8.

[19] Sergey Zvezdakov, Denis Kondranin, and Dmitriy Vatolin, "Machine-Learning-Based Method for Content-Adaptive Video Encoding," in *2021 Picture Coding Symposium (PCS)*, 2021, pp. 1–5.

[20] Quortex, "Mission: Emission," https://www.quortex.io/wp-content/uploads/2022/05/WhitePaper-Mission-Emission.pdf, 2022, [Accessed May 2023].

[21] Vignesh V Menon, Anastasia Henkel, Prajit T Rajendran, Christian R. Helmrich, Adam Wieckowski, Benjamin Bross, Christian Timmerer, and Detlev Marpe, "All-Intra Rate Control Using Low Complexity Video Features for Versatile Video Coding," in *2023 IEEE International Conference on Image Processing (ICIP)*, 2023, pp. 2760–2764.

[22] Vignesh V Menon, Christian Feldmann, Klaus Schoeffmann, Mohammad Ghanbari, and Christian Timmerer, "Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming," in *Proceedings of the First International Workshop on Green Multimedia Systems*, 2023, p. 16–18. [Online]. Available: https://doi.org/10.1145/3593908.3593942

[23] Leo Breiman, "Random Forests," in *Machine Learning*, vol. 45, 2001. [Online]. Available: https://doi.org/10.1023/A:1010933404324

[24] Vignesh V Menon, Reza Farahani, Prajit T Rajendran, Samira Afzal, Klaus Schoeffmann, and Christian Timmerer, "Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming," in *2023 International Conference on Visual Communications and Image Processing (VCIP)*, 2023.

[25] Joe Yuchieh Lin, Lina Jin, Sudeng Hu, Ioannis Katsavounidis, Zhi Li, Anne Aaron, and C-C Jay Kuo, "Experimental design and analysis of JND test on coded image/video," in *Applications of digital image processing XXXVIII*, vol. 9599. SPIE, 2015, pp. 324–334.

[26] Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchieh Lin, Lina Jin, Longguang Song, Ping Wang, Ioannis Katsavounidis, Anne Aaron, and C-C Jay Kuo, "MCL-JCV: A JND-based H. 264/AVC video quality assessment dataset," in *2016 IEEE international conference on image processing (ICIP)*. IEEE, 2016, pp. 1509–1513.

[27] Haiqiang Wang, Ioannis Katsavounidis, Jiantong Zhou, Jeonghoon Park, Shawmin Lei, Xin Zhou, Man-On Pun, Xin Jin, Ronggang Wang, Xu Wang, Yun Zhang, Jiwu Huang, Sam Kwong, and C.-C. Jay Kuo, "VideoSet: A large-scale compressed video quality dataset based on JND measurement," in *Journal of Visual Communication and Image Representation*, vol. 46, 2017, pp. 292–302. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1047320317300950

[28] Jingwen Zhu, Suiyi Ling, Yoann Baveye, and Patrick Le Callet, "A framework to map vmaf with the probability of just noticeable difference between video encoding recipes," in *2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP)*. IEEE, 2022, pp. 1–5.

[29] Vignesh V Menon, Jingwen Zhu, Prajit T Rajendran, Hadi Amirpour, Patrick Le Callet, and Christian Timmerer, "Just Noticeable Difference-Aware Per-Scene Bitrate-Laddering for Adaptive Video Streaming," in *2023 IEEE International Conference on Multimedia and Expo (ICME)*, 2023, pp. 1673–1678.

[30] Andreas Kah, Christopher Friedrich, Thomas Rusert, Christoph Burgmair, Wolfgang Ruppel, and Matthias Narroschke, "Fundamental relationships between subjective quality, user acceptance, and the VMAF metric for a quality-based bit-rate ladder design for over-the-top video streaming services," in *Applications of Digital Image Processing XLIV*, vol. 11842, International Society for Optics and Photonics. SPIE, 2021, p. 118420Z.

[31] Jan Ozer, "Finding the Just Noticeable Difference with Netflix VMAF," Sep. 2017. [Online]. Available: https://streaminglearningcenter.com/codecs/finding-the-just-noticeable-difference-with-netflix-vmaf.html

[32] Praveen Kumar Tiwari, Vignesh V Menon, Jayashri Murugan, Jayashree Chandrasekaran, Gopi Satykrishna Akisetty, Pradeep Ramachandran, Sravanthi Kota Venkata, Christopher A Bird, and Kevin Cone, "Accelerating x265 with Intel® Advanced Vector Extensions 512," in *White Paper on the Intel Developers Page*, 2018. [Online]. Available: https://www.intel.com/content/dam/develop/external/us/en/documents/mcw-intel-x265-avx512.pdf

[33] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Courneaupeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay, "Scikit-learn: Machine Learning in Python," in *Journal of Machine Learning Research*, vol. 12, 2011, pp. 2825–2830.

[34] Vignesh V Menon, Hadi Amirpour, Prajit T Rajendran, Mohammad Ghanbari, and Christian Timmerer, "Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming," in *2022 Picture Coding Symposium (PCS)*, 2022, pp. 253–257.

[35] HSTP-VID-WPOM, "Working practices using objective metrics for evaluation of video coding efficiency experiments," in *International Telecommunication Union*, 2020. [Online]. Available: http://handle.itu.int/11.1002/pub/8160e8da-en

[36] BCG-GAMMA and MILA, "CodeCarbon." [Online]. Available: https://codecarbon.io/