# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
## 4 Problem Formulation
### Action and State Transition  
At the beginning of time slot \(l\), in order to find the optimal download speed distribution, multiple decisions are needed to adjust the observed speed distribution. We utilize two decision processes to obtain the optimal distribution \(\tilde{\pi}(l)\) while maintaining the constraint shown in Equation (23). One of the processes is a _decreasing_ process that decides for which user the download speed will be decreased by one unit of rate, and the other process is an _increasing_ process that decides the user that will obtain the released one unit of download speed.  
The download speed distribution is iteratively adjusted to a final distribution by recursively running the _decreasing_ and _increasing_ decision processes. A distribution will not be assigned to the system until the final decision is concluded, and the system will not transit into the next time slot. Assuming, at time slot \(l\), with the observed state \(s(l)=(\vartheta(l),\tilde{d}(l),z(l),\tilde{c}(l))\), actions \(a_{-},a_{+},a_{-}\
eq a_{+}\) are made by the decreasing and increasing processes. The intermediate state \(s(\tau^{\prime})\) can be derived by  
\[s(\tau^{\prime})=(\vartheta(l),d_{1}(l),\cdots,d_{a_{-}}(l)-1, \cdots,\] \[d_{a_{+}}(l)+1,\cdots,d_{K}(l),\tilde{z}(l),\tilde{c}(l)). \tag{25}\]  
Now, this intermediate state \(s(\tau^{\prime})\) is used in the decision making for both processes. New actions will be made to push the distribution towards the final state. Finally, at state \(s(\tau)\), when both the increasing and decreasing processes give the same action \(a_{+}=a_{-}\), the distribution \(\tilde{\pi}\) is obtained as \(\tilde{\pi}(l)=(d_{1}(\tau),d_{2}(\tau),...,d_{K}(\tau))/B\).  
According to \(\pi(l)\), the system distributes the bandwidth to each user for the time slot \(l\). The next time slot \(l+1\) will be triggered when a user switches its playing video. We assume that the new content request for all users follows Poisson arrival processes with arrival rate \(\lambda_{k}\) for user \(k\), so the mean value of slot duration \(T_{l}\) can be derived by \(1/\sum_{k\in K}\lambda_{k}\), and the probability that user \(k\) triggers the state transition is \(\lambda_{k}/\sum_{k\in K}\lambda_{k}\). For time slot \(l+1\), the initial system state \(s(l+1)=(\vartheta(t+1),d(t+1),z(t+1),\tilde{c}(t+1))\) should have video bitrates \(v_{\kappa}(l+1)=v_{\kappa}(l),(\forall k\in K,\kappa\
eq k)\) if CBR is activated as the bitrate policy, and downloading speeds \(d(l+1)=\pi(l)B\) calculated in the previous time slot.  
The accumulated stalls \(z(l)\) can be calculated using Equations (4) and (15). Let \(v_{k}(l)\) be the video played by the user \(k\) in time slot \(l\), and let \(l_{v_{k}(l),0}\) be the time slot where user \(k\) starts playing video \(v_{k}(l)\). Let \(T_{l^{\prime}}\) denote the length of time slot \(l^{\prime}\); we have  
\[z_{k}(l)=\sum_{l^{\prime}=l_{v_{k}(l),0}}^{l-1}T_{s}(l^{\prime},T_{l^{\prime}},k). \tag{26}\]  
The number of remaining chunks \(\tilde{c}(l)\) can easily be tracked during the downloading/playing procedures, and observed whenever the information is needed. Both the downloading and playing processes can be monitored. For the downloading process, let \(c_{d}(l)=h_{d}(l)+\rho_{d}(l)\), where \(h_{d}(l)\in\mathbb{N}\) denotes the chunk of the video being played at the beginning of time slot \(l\) being downloaded, and \(\rho_{d}(l)\in[0,1)\) denotes the ratio or percentage of chunk \(h_{d}(l)\) that has been completed. The similar mechanism holds for the playing process, \(c_{p}(l)=h_{p}(l)+\rho_{p}(l)\). Using both of the processes, the remaining chunks \(c(l)\) can be calculated by \(c(l)=c_{d}(l)-c_{p}(l)\) in any time slot \(l\).