# HTTP Adaptive Streaming Framework with Online Reinforcement Learning
## 4 Performance Evaluation
### Results  
First, according to the evaluation scenario described in Section 4.2, normalized QoE values and target reward values for the proposed scheme and the existing schemes were measured. Figure 5 shows the normalized QoE values and target reward values measured for each test dataset. The proposed scheme has the highest normalized QoE and reward values compared to the existing schemes. This shows the effectiveness of the method of updating the neural network model according to the network environment of the proposed scheme. In particular, when the proposed scheme and Pensieve are compared using Dataset #1 and Dataset #2 with low average bandwidth values, a more pronounced performance improvement is shown. Through this, we show that the proposed scheme effectively improves QoE in an environment where the performance of the existing schemes has deteriorated. Figure 6 shows how much performance improvement occurs and when it converges with time from the time online learning is started. This experiment was conducted based on Dataset #2, in which the performance improved most clearly by about 21%. The proposed scheme uses the A3C network to update the model. As a result, it can be confirmed that the performance increases linearly according to the time step despite the asynchronous model updating of each agent. Also, through the experimental results, it can be confirmed that the performance improvement converges after about 18 h. This is due to the time it takes for each agent's online learning to complete and the model to update, implying that additional techniques are required to improve the learning rate. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Trace Dataset**} \\ \hline Characteristics & \#1 & \#2 & \#3 & \#4 & \#5 & \#6 & \#7 \\ \hline Mean Throughput (Mbps) & 0.98 & 1.57 & 2.62 & 3.23 & 4.75 & 5.79 & 6.52 \\ \hline Max Throughput (Mbps) & 3.45 & 3.38 & 6.54 & 8.97 & 8.23 & 9.65 & 11.53 \\ \hline Min Throughput (Mbps) & 0.09 & 0.23 & 0.21 & 0.38 & 0.26 & 0.36 & 0.39 \\ \hline Coefficient of Variation & 0.76 & 0.44 & 0.56 & 0.72 & 0.35 & 0.39 & 0.37 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Throughput information of network datasets. Figure 5: Normalized QoE and reward according to the dataset, (**a**) normalized QoE, (**b**) normalized reward. \begin{table}
\begin{tabular}{c c} \hline \hline
**Scheme** & **Characteristics** \\ \hline robustMPC & \
\begin{tabular}{c} 1. Model predictive control-based adaptation \\ 2. Traditional video adaptation scheme \\ \end{tabular} \\ \hline Pensieve &
\begin{tabular}{c} 1. Reinforcement learning (Server-side) \\ 2. QoE model with perceptual quality \\ 3. QoE degradation for specific network \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Analysis of existing video adaptation schemes for performance evaluation. Figure 7 shows the results of experiments in the same environment after increasing the number of threads used by each agent to improve the training speed of each agent. As the number of threads increases, the degree of performance improvement increases with time. In particular, when four threads are used, the degree of performance improvement converges after about 12 h has elapsed. If more threads are used, the performance improvement is predicted to occur faster, but when several agents are training at the same time, the risk of an agent that cannot use multi-threading increases depending on the computing power. In addition, for the model to be updated, the normalized QoE value must fall below the threshold, and the state of the network must also change. As a result, even if more threads are used, performance cannot be improved beyond a certain point unless the network has very high bandwidth fluctuations. Figure 8 shows the results of an experiment using a dataset with a very wide bandwidth fluctuation to check the degree of performance improvement according to an increase in the number of threads in a rapidly changing network environment.