# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
## 3 System Model  
We consider a server with a total bandwidth of \(B\) stream videos to users in set \([K]=\{1,2,\cdots,K\}\), in which, all of the users are consuming videos continuously. We consider a streaming session to be divided into nonidentical logical slots. In each time slot, all of the users will maintain requesting/playing chunks from the same videos. Once any user \(k\in[K]\) starts a request for a new video in the current time slot \(l\), the slot counter increments, thus the new slot \(l+1\) starts for all users in \([K]\), even if the video does not change for users \(k^{\prime}\in[K]/\{k\}\). Using the logical time slot setting described above, at time slot \(l\in[L]=\{1,\cdots,L\}\), a user \(k\in[K]\) consumes downloading rate \(d_{k}(l)\geq 0\) to fetch video \(v_{k}\), which is coded with bitrate \(r_{k}(l)\). The downloading speed is limited by \(\sum_{k\in K}d_{k}(l)=B,\forall l\in[L]\), and may update for all users when the time slot increments in the system. The video server continuously sends video chunks to the user with downloading speed \(d_{k}(l)\), and the user plays the video with bitrate \(r_{k}(l)\), which is defined by the property of the video. For Adaptive Bitrate (ABR) videos, we update the bitrate of the chunk on starting a new slot only. Thus, all of the chunks sent in slot \(l\) are of bitrate \(r_{k}(l)\). For Constant Bitrate (CBR) videos, \(r_{k}(l)\) may remain constant across time slots \(l\in[L]\) if multiple slots span the video \(v_{k}\). A list of the key variables used in this paper can be seen in Table 1.  
In each slot, we reset the clock to zero. We use \(t^{\prime}_{k}(l,m)\) to denote the time when the server starts to send the \(m\)-th chunk of video \(v_{k}\) in the time slot \(l\), \(t_{k}(l,m)\) to denote the time when the user starts to play video chunk \(m\), and \(\bar{t}_{k}(l,m)\) to denote the moment that chunk \(m\) is finished playing. For analysis, we consider that the size of each chunk is normalized to 1 unit.  
With our formulation, there will be two classes of users in a time slot \(l\). The first class is of users who requested a new video and triggered the increment of the time slot to \(l\). Since the user has requested a new video, it can purge the already downloaded chunks for the previous video. Users in this class may observe a new downloading rate \(d_{k}(l)\) and video bit rate \(r_{k}(l)\). The second class of users are those who do not request a new video, but a new streaming rate \(d_{k}(l)\) is assigned to them because some other user \(k^{\prime}\in[K]\) has requested a new video and triggered a slot change. For these users, the video bitrates will remain the same from the previous slot \(l-1\), or be adjusted solely by the ABR streaming policy when CBR or ABR policies are activated, whereas, for the downloading rate \(d_{k}(l)\), it is updated by the bandwidth distribution policy. Note that a resource allocation scheme  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Variable** & **Description** \\ \hline \(K\) & number of clients in the system \\ \(k\) & index for agents, runs from 1 to \(K\) \\ \(B\) & total bandwidth of the system \\ \(l\) & slot index \\ \(L\) & total slots considered \\ \(d_{k}(l)\) & download rate for user \(k\) in slot \(l\) \\ \(r_{k}(l)\) & bitrate of chunk sent to user \(k\) in slot \(l\) \\ \(v_{k}(l)\) & index of video streamed by user \(k\) in slot \(l\) \\ \(m\) & chunk index of the video \(v_{k}(l)\) \\ \(t_{k}(l,m)\) & time at which user \(k\) starts playing chunk \(m\) for video \(v_{k}(l)\) \\ \(t^{\prime}_{k}(l,m)\) & time at which server starts sending chunk \(m\) for video \(v_{k}(l)\) \\ \(\bar{t}_{k}(l,m)\) & time at which user \(k\) finishes playing chunk \(m\) \\  & for video \(v_{k}(l)\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: List of the key variables used in this paper.  
may still allocate bandwidth to the user such that \(d_{k}(l)=d_{k}(l-1)\); however, this may not be always true.  
Next, we calculate the stall time in a slot \(l\) for both classes of users.