## 3 System Overview
### Deep Reinforcement Learning Framework  
In this paper, we use the Proximal Policy Optimization Algorithm (PPO) [18] to train the DRL neural network, a policy-based actor-critic algorithm. Each DRL agent must train a policy network \(\pi_{\theta}\) and a value network \(v_{\phi}\) simultaneously. The specific functions of the algorithm are as follows:  
_Status_: observed value \(O_{i}(QoE_{i}^{\gamma,h},g_{i},f_{i},b_{i},T_{i},C_{i},L_{i})\), \(QoE_{i}^{\gamma,h}\) represents the QoE score of historical discount of \(h\) video blocks played by agent \(i\) in the past; \(g_{i}\) represents the set of file sizes of six optional bitrates for the next video block; \(f_{i}\) represents the set of video quality scores of the six selectable bitrates for the next video block; \(b_{i}\) represents the current buffer level of agent \(i\); \(T_{i}\) represents the current bandwidth estimation of agent \(i\); \(C_{i}\) represents the current QoE coefficient of variation of agent \(i\); \(L_{i}\) represents the current QoE level of the client of agent \(i\).  
_Action_: The action taken by the agent in time step \(t_{i}\) is the weight of bandwidth allocation \(w_{i}\in[0.4,0.9]\) of the video user during the next video block download. The \(w_{i}\) is modeled as a set of continuous variables to explore the nearly optimal strategy in the appropriate action space. The value range of \(w_{i}\) is limited to avoid excessive adjustment caused by a too large or too small retreat factor.  
_Reward_: The common goal of all agents is to maximize overall QoE efficiency and ensure QoE fairness among users. Therefore, the reward \(R_{i}\) of each agent is defined as the linear combination of its own historical discount QoE score and QoE fairness index (Equation (1)):  
\[R_{i}=QoE_{i}^{\gamma,h}+\delta f(QoE_{1}^{\gamma,h},QoE_{2}^{\gamma,h},\dots,QoE_{n}^{\gamma,h}) \tag{8}\]  
The loss function of the cumulative discount reward of Policy Network \(\pi_{\theta}\) is:  
\[\bigtriangledown_{\theta}L^{n}(\theta)=E_{o_{t}^{n},a_{t}^{n}}\Big{[}\bigtriangledown_{ \theta}log\Big{\{}min\Big{(}\frac{\pi_{\theta}(a_{t}^{n}|o_{t}^{n})}{\pi_{ \theta^{\prime}}(a_{t}^{n}|o_{t}^{n})}A(o_{t}^{n},a_{t}^{n}),clip\Big{(}\frac {\pi_{\theta}(a_{t}^{n}|o_{t}^{n})}{\pi_{\theta^{\prime}}(a_{t}^{n}|o_{t}^{n })},1-\xi,1+\xi\Big{)}A(o_{t}^{n},a_{t}^{n})\Big{)}\Big{\}}\Big{]} \tag{9}\]  
where \(\pi_{\theta}\) is the old policy parameter before the update, \(A(o_{t}^{n},a_{t}^{n})\) is the advantage function, \(E_{o_{t}^{n},a_{t}^{n}}\) represents the mathematical expectation of the loss function for all observation-action cases, and \(clip\) is the clipping function: if the first term is less than the second term, output \(1-\xi\); if the first term is greater than the third term, output \(1+\xi\). \(\xi\) is a hyperparameter, which is set to 0.2 in this paper.  
The update rule for the policy network \(\pi_{\theta}\) is given by:  
\[\theta^{{}^{\prime}}\leftarrow\theta+\varepsilon\bigtriangledown_{\theta} log\Big{\{}min\Big{(}\frac{\pi_{\theta}(a_{t}^{n}|o_{t}^{n})}{\pi_{\theta^{ \prime}}(a_{t}^{n}|o_{t}^{n})}A(o_{t}^{n},a_{t}^{n}),clip\Big{(}\frac{\pi_{ \theta}(a_{t}^{n}|o_{t}^{n})}{\pi_{\theta^{\prime}}(a_{t}^{n}|o_{t}^{n})},1- \xi,1+\xi\Big{)}A(o_{t}^{n},a_{t}^{n})\Big{)}\Big{\}} \tag{10}\]  
The update rule of the value network \(v_{\phi}\) is:  
\[\phi^{{}^{\prime}}\leftarrow\phi+\varepsilon^{{}^{\prime}}\bigtriangledown_{ \phi}L^{n}(\phi) \tag{11}\]\[L^{n}(\phi)=E_{\omega_{i}^{n}}\left[\bigtriangledown_{\phi}log\Big{(}(V_{\phi}( \omega_{t}^{n})-R_{t}^{n})^{2}\Big{)}\right] \tag{12}\]  
where \(\varepsilon\) and \(\varepsilon^{{}^{\prime}}\) are the learning rates of the policy network and value network, respectively. \(v_{\phi}\) is the value policy parameter, \(R\) is the discount reward. In this paper, the neural networks trained by each agent are homogeneous, and the same hyper-parameters are adopted in the training process. The values of \(\varepsilon\) and \(\varepsilon^{{}^{\prime}}\) are set to 0.005.