# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
###### Abstract
Such inter-agent utility optimizations are widely considered in video streaming problems, e.g., to optimize the fairness of network resource allocation and to maximize a non-linear QoE function of individual agent's performance metrics. More precisely, in a dynamic setting, the problem being solved must be modeled as an MDP, where agents take actions based on some policy \(\pi\) and observed system states, causing the system to transition to a new state. A reward \(r_{k}\) is fetched for each agent \(k\). The transition probability to the new state is dependent only on the previous state and the action taken in the previous state. RL algorithms aim to find an optimal policy \(\pi\) to maximize the sum of (discounted) rewards \(\sum_{t=1}^{\infty}\gamma^{t}\sum_{k}r_{k}(t)\) for all users. However, when QoE and fairness objectives are concerned, a nonlinear function \(f\), such as the fairness utility [13] and sigmoid QoE function reported by [14], must be applied to the rewards received by different agents, resulting in the optimization of a new objective \(f(\bar{r}_{1},\bar{r}_{2},\ldots)\), where \(\bar{r}_{k}=\frac{1}{T}\sum_{t=1}^{T}\gamma^{t}r_{k}(t)\) is the average discounted reward for each agent \(k\) in a finite time \(T\). It is easy to see that such nonlinear functions will potentially violate the memoryless rule of MDP which is required for RL since the optimization objective now depends on all past rewards/states. In this paper, we will develop a new family of multi-agent reinforcement learning algorithms to optimize such nonlinear objectives for QoE and fairness maximization in video streaming. We propose _Multi-agent Policy Gradient for Finite Time Horizon_ (MAPG-finite) for the optimization of nonlinear objective functions of cumulative rewards of multiple agents with a finite time horizon. We employ MAPG-finite in online video streaming with the goal of maximizing QoE and fairness objectives by adjusting the download bandwidth distribution. To this end, we quantify the stall time for online video streaming with multiple agents under a shared network link and dynamic video switching by the agents. At the end of the time horizon, a nonlinear function \(f(\cdot)\) of the agents' individual cumulative rewards is calculated. The choice of \(f(\cdot)\) is able to capture different notions of fairness--e.g., the well-known \(\alpha\)-fairness utility [13] that incorporates proportional fairness and max-min fairness as special cases, and the sigmoid-like QoE function reported by [14] that indicates that users with a mediocre waiting time tend to be more sensitive than the rest--and thus balancesthe performance received by different agents in online video streaming. We leverage the RL algorithm proposed in [15] to develop a model-free multi-agent reinforcement learning algorithm to cope with the inter-agent fairness reward for multiple users. In particular, this RL algorithm modifies the traditional policy gradient to find a proper ascending direction for the nonlinear objective function using random sampling. We improve the convergence of the proposed algorithm to at least a local optimal of the target optimization problem. The proposed multi-agent algorithm is model-free and shown to efficiently solve the QoE and fairness optimization in online video streaming. To demonstrate the challenge associated with optimizing nonlinear objective functions, consider the example shown in Figure 1. Two users, A and B, share a download link for video streaming. Due to the bandwidth constraint, the link is only able to stream one high-definition (HD) video and one low-definition (LD) video at a time. In each time slot \(t\), the two users' QoE, denoted by \(r_{A}(t)\) and \(r_{B}(t)\), are measured by a simple policy that multiplies the quality of the _content_ (from one to five stars) and the resolution of the video (1 for LD and 2 for HD).