# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
## 5 Policy Gradient for MA-Stream
### Evaluation Results  
The numerical results for the QoE reward function (Equation (16)) is depicted in Figure 6. It is shown that our proposed MAPG-finite algorithm outperforms the static \"Even\" and dynamic \"Adaptive\" strategies by 23.34% and 169.66% (in terms of achieved QoE) with the shared download link of 1500 KB/s. With the 2000 KB/s download link, MAPG-finite still obtains a 15.30% and 32.58% higher QoE reward than the \"Even\" and \"Adaptive\" policies, while the improvement becomes smaller because of a smaller marginal QoE improvement when the stall time is already small under a higher bandwidth. As for the SARSA policy, it is unable to cope with the nonlinear utility function and fails to achieve much improvement over its initial decision policy--\"Even\". Since the QoE reward function is nonlinear to the assigned bandwidth, we also observe that the \"Adaptive\" policy  
Figure 5: The implementation of our testbed, where the blocks on the left and right show the internal architectures of an individual user module and the MAPG-finite engine, respectively. The four state variables \((\triangledown(t),d(t),z(t),c(t))\) are collected/calculated by each user and delivered to the _action execution_ module (via 2, 4, 5, 1 \(\rightarrow\)6). The action is then sent back to each user via 7���  
(allocating bandwidth proportional to desired video bitrate) performs worse than \"Even\" in both cases. These can be further seen from Table 3, which shows the stall time and reward breakdown of different policies. Apparently, the \"Adaptive\" policy achieves a similar stall time for both HD and LD users, whereas the \"Even\" policy sacrifices the performance of HD users, and, in return, significantly reduces the stall time of LD users, leading to a higher overall QoE. More precisely, according to the QoE reward curve shown in Figure 4a, the reward boost for the LD users is much greater than the loss suffered by the HD users, which finally results in the overall QoE improvement in the \"Even\" policy. As a learning-based algorithm, MAPG-finite can achieve a substantially better performance, since it is aware of the current network conditions and system states, in order to optimize the bandwidth distribution between the users. For example, when a user has enough cached chunks for future playback, its bandwidth can be temporarily turned over to the other users, who recently started playing a new video or are suffering from a stall. Thus, all users are able to obtain increased QoE rewards under the MAPG-finite strategy, compared with the baselines. Results for the fairness reward function are shown in Figure 7. We note that, due to the use of the logarithmic fairness function, the improvement appears to be smaller when measured by the fairness reward than by the QoE reward, while the gains should be \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Policy**} & \multirow{2}{*}{**Total Reward**} & \multicolumn{3}{c}{**User Average**} \\ \cline{3-5}  & & **User** & **Stall Ratio** & **Reward** \\ \hline \multirow{2}{*}{**\"Even\"**} & 3877.05 & 1, 2, 3 (HD) & 0.64 & 0.07 \\  & & 4, 5 (LD) & 0.13 & 0.86 \\ \hline \multirow{2}{*}{**“Adaptive\"**} & \multirow{2}{*}{1773.30} & 1, 2, 3 (HD) & 0.51 & 0.20 \\  & & 4, 5 (LD) & 0.59 & 0.14 \\ \hline \multirow{4}{*}{**MAPG-finite**} & \multirow{4}{*}{4781.88} & 1 & 0.34 & 0.52 \\  & & 2 & 0.50 & 0.27 \\ \cline{1-1}  & & 3 & 0.39 & 0.44 \\ \cline{1-1}  & & 4 & 0.07 & 0.91 \\ \cline{1-1}  & & 5 & 0.08 & 0.91 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Reward breakdown for QoE function on 1500 KB/s download link. Figure 6: The QoE reward comparison. With the 1500 KB/s download link, MAPG-finite outperforms “Even” by 23.34%, “Adaptive” by 169.66%, and SARSA by 25.84%. With the 2000 KB/s download link, MAPG-finite outperforms “Even” by 15.30%, “Adaptive” by 32.58%, and SARSA by 15.27%. interpreted in the \"multiplicative\" sense.