# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
## 5 Policy Gradient for MA-Stream
### Evaluation Setup
#### 6.1.4 Implementation  
We implement a testbed using Python 3.5. The workflow of the testbed is depicted in Figure 5. First, at the beginning of a new cycle, according to the video switching rates (which result from the known video duration distributions), the _video switch_ module randomly schedules a user who will be the next candidate to change their video. The users then start downloading video chunks continuously, and their _download timers_ record the timestamp when each chunk is successfully downloaded (i.e., \(t^{\prime}_{m+1}\) in Figure 3). Next, based on the download timestamps and the bitrate of the current video, the _playback timer_ schedules the playback and further obtains \(t_{m}\) and \(t^{\prime}_{m}\). With all timestamps confirmed, the  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**User** & **Resolutions** & **Bitrates** & **Probabilities** \\ \hline \multirow{2}{*}{1} & 1080p & 8 Mbps & 0.5 \\  & 720p & 5 Mbps & 0.5 \\ \hline \multirow{2}{*}{2} & 1080p & 8 Mbps & 0.5 \\  & 720p & 5 Mbps & 0.5 \\ \hline \multirow{2}{*}{3} & 1080p & 8 Mbps & 0.5 \\  & 720p & 5 Mbps & 0.5 \\ \hline \multirow{2}{*}{4} & 480p & 2.5 Mbps & 0.5 \\  & 360p & 1 Mbps & 0.5 \\ \hline \multirow{2}{*}{5} & 480p & 2.5 Mbps & 0.5 \\  & 360p & 1 Mbps & 0.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Simulated user preferences.  
Figure 4: The reward functions used to evaluate the performances of different bandwidth assignment algorithms. The QoE function is defined by Equation (16), and the fairness function is defined by Equation (17).  
stall calculator_ is able to calculate the stall time for the user/videos. Such a stall time is transferred to the distribution module for training and evaluation. Finally, the chosen user randomly picks a new video from its _video library_ at the end of the current cycle.  
Recall that, from Section 4, the state variables utilized for MAPG-finite decision making include _video bitrates_, _downloading speeds_, _accumulated stall time_, and _residue video chunks_, which can be reported through output paths 2, 4, 5, and 1, respectively, in Figure 5. The state variables are then collected by the _state listener_ through input path 6. Further, utilizing the _neural network_, a bandwidth distribution--as the action--is decided and sent by the _action sender_ to all of the users via 7. For training and evaluation purpose, a copy of the stall ratio is also sent from 3 to 8. It is processed by the _reward function_\(f(\cdot)\) to calculate either the QoE (Equation (16)) or the fairness (Equation (17)) reward. Finally, the reward is delivered to the _neural network_ for policy backpropagation, and also logged for experiment evaluation. By input path 5 of each user, the users adjust their _download rates_ according to the bandwidth decision from 7. At this point, the testbed completes one workflow cycle and prepares to initiate the next cycle, starting with the _video switch_.  
Using the modularized testbed implemented in this project, we are able to evaluate different policies under various environment configurations, including with different reward functions, network conditions, and user behavior settings. We note that, with some minor logic adjustments, we can even test the streaming performance in a discrete time domain, while this paper focuses on continuous time evaluations.