# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
## 5 Policy Gradient for MA-Stream
### Evaluation Results
Our proposed MAPG-finite still outperforms the \"Even\" and \"Adaptive\" strategies (in terms of the logarithmic fairness reward) by 4.83% and 6.75% for the 2000 KB/s downloading link, and 8.28% and 2.30% for the 2500 KB/s link. With the 2500 KB/s downlink, Table 4 shows that the dynamic \"Adaptive\" strategy achieves similar performances for all users, leading to a better fairness reward than the \"Even\" policy, which creates more difference between the HD and LD users that have significantly different bitrate requirements. On the other hand, our MAPG-finite strategy reduces the average stall ratio of HD users by 38.46%, with a cost of a 37.50% higher stall ratio for LD users compared to the \"Even\" policy. This way, it is able to reduce the stall ratio deviation of all users from 0.1698 to 0.0712, and improves the fairness reward. Comparing with the \"Adaptive\" policy, MAPG-finite has an approximately 30% higher stall ratio deviation. However, the optimization object, known as the proportional fairness utility [13], is not solely about \"equalizing\" different users' performances. (To illustrate this, we construct a \"Low Dev\" policy in Table 4 that has a close-to-zero stall ratio deviation but a low fairness reward.) The use of a proportional fairness reward function indeed balances two important objectives--efficiency (i.e., assigning more bandwidth to users that can achieve a higher reward per unit bandwidth) and fairness (i.e., balancing different users' performances). MAPG-finite is able to attain the highest reward under the choice of proportional fairness utilities, demonstrating its ability to achieve complex optimization objectives. The evaluation results show that the \"Even\", \"Adaptive\", and SARSA policies fail to perform consistently under different application scenarios and network conditions, whereas our learning-based MAPG-finite policy is able to achieve the highest reward. Figure 8 depicts the average download rate distribution decided by the MAPG-finite policy. The gray bars represent the average video bitrates requested by the users. Figure 7: The fairness reward comparison. With the 2000 KB/s download link, MAPG-finite outperforms “Even” by 4.83%, “Adaptive” by 6.75%, and SARSA by 22.47%. With the 2500 KB/s download link, MAPG-finite outperforms “Even” by 8.28%, “Adaptive” by 2.30%, and SARSA by 30.15%. The white bars represent the average download rate achieved by our MAPG-finite policy (for presentation purposes, the unit of download rates is converted from KB/s to Mbps). It is can be seen that, in order to maximize the fairness reward, MAPG-finite ensures that **(i)** for users with the same HD preference (e.g., users 1, 2, and 3), the same average downloading bandwidth is assigned to obtain a similar stall time for these users, and **(ii)** for users watching videos with lower desired bitrates, less bandwidth is assigned to balance the stall time, since video chunks are consumed at a slower pace. According to Figure 7, the \"Adaptive\" policy achieves a lower fairness reward than \"Even\" under the total download link of 2000 KB/s, which indicates that proportionally adjusting the download bandwidth does not always achieve a better result when fairness is concerned. Through the exploration and training of RL, MAPG-finite is able to self-teach, improve, and finally converge to an optimal policy, making the model-free suitable for bandwidth allocation with complex networks and objectives that often do not have a straightforward mathematical formulation. To further illustrate the agility of our proposed MAPG-finite algorithm, we perform another evaluation on a 1500 KB/s downlink, with an ABR streaming algorithm implemented. Figure 9 depicts the QoE performances for this evaluation. We utilize a basic buffer-based ABR algorithm proposed in [37]. Each time a video chunk is requested, if the last chunk downloaded is already being played, the bitrate is adjusted to 80% of the