# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
###### Abstract
The service provider is interested in optimizing a logarithmic utility of the users' aggregate QoE, i.e., \(u=\ln(\sum_{t}r_{A}(t))+\ln(\sum_{t}r_{B}(t))\) (which corresponds to the notion of maximum proportional fairness [13; 16]), for the two time slots \(t=\{1,2\}\). It is easy to see that, in Case 1, user A received \(r_{A}(1)=6\) in time slot 1 and user B \(r_{B}(1)=4\). If we stream HD to user A and LD to user B in the next time slot, then the total received utility becomes \(\ln(6+6)+\ln(4+4)\approx 4.56\), whereas the opposite assignment achieves a higher utility \(\ln(6+3)+\ln(4+8)\approx 4.68\). However, in Case 2, choosing user A in time slot 2 to receive HD and user B LD gives the highest utility of \(\ln(2+6)+\ln(5+4)\approx 4.28\). Thus, the optimal decision in time slot 2 depends on the reward received in all past time slots (while we have only shown the rewards in time slot 1 for simplicity in this example). In general, the dependence of the utility objective on all past rewards implies a violation of the Markovian property, as the actions should only be affected by the currently observed system state in MDP. This mandates a new family of RL algorithms that are able to cope with nonlinear objective functions, which is the motivation of this paper. Figure 1: An illustrative example showing that the optimal choice of bandwidth assignment must depend on past rewards/states in order to maximize a logarithmic fairness objective. We note that, in video streaming optimization, the action space for learning-based algorithms can still be prohibitively large, since the bandwidth assignment decisions are continuous (or at least fine-grained if we sample the continuous decision space), whereas the action space suitable for RL algorithms should be small as the output layer sizes of neural networks are limited. The action space for bandwidth assignment increases undesirably with both the growing number of users and the increasing amount of network resources in the video streaming system. To overcome this challenge, we propose an adaptive bandwidth adjustment process. It leverages two separate reinforcement learning modules running in parallel, each tasked to select a target user to increase or decrease their current bandwidth by one unit. This effectively reduces the action space of each RL module to exactly the number of users in the system, while both learning modules can be trained in parallel with the same set of data through backpropagation. This technique significantly reduces the action spaces in MAPG-finite and makes the optimization problem tractable. To evaluate the proposed algorithm, we develop a modularized testbed for the event-driven simulation of video streaming with a multi-agent bandwidth assignment. In particular, a bandwidth assigner is developed to observe the agent states, obtain an optimized distribution from the activated action executor, and then adjust the bandwidth of each agent on the fly. We implement this distribution-generating solution along with the model-free multi-agent deep policy gradient algorithm, and compare the performance with static and dynamic baseline strategies, including \"Even\" (which guarantees balanced bandwidths for all users), \"Adaptive\" (which assigns more bandwidth to users consuming higher bitrates), and SARSA (which is a standard single-agent RL-driven policy that fails to consider inter-agent utility optimization). By simulating various network environments, as well as both constant and adaptive bitrate policies, we validate that the proposed MAPG-finite outperforms all other tested algorithms. With Constant Bitrate (CBR) streaming, MAPG-finite is able to improve the achieved QoE by up to 169.66%, and the fairness by up to 8.28% compared with baseline strategies. Further, with the Adaptive Bitrate (ABR) streaming, up to a 41.25% QoE improvement can be obtained. We conclude that the key contributions of this work are:  
* We model the bandwidth assignment problem for optimizing QoE and fairness objectives in multi-user online video streaming.