## 3 Materials and Methods
### Markov Decision Process (MDP)  
Client-side adaptation necessitates making sequential decisions over time denoted as t. To formalize this adaptation process, we utilize a Markov Decision Process (MDP). The MDP is characterized by a quintuple of fundamental components: agent, environment, states, actions, and rewards, as illustrated in Figure 2.  
The agent embodies client-side adaptation responsibilities, with its primary task being to select an action denoted as at. This action entails the choice of an appropriate bitrate or quality \(q_{t}\) for the next subsequent segment \(S_{i}\) awaiting download. The agent's decision is based on current bandwidth measurements \(Bw_{t}\), the prevailing buffer state \(buff\_state_{t}\), and the quality \(q_{t-1}\) of the previous segment \(\text{s}_{i-1}\). The agent's bitrate selection (adaptation) results in the receipt of rewards or penalties.  
In the realm of adaptive video streaming, we define segments as a finite set of states, \(\text{S}=\{\text{s}_{0},\text{s}_{1},\ldots\text{s}_{\text{N}}\}\). It is important to note that a video is divided into N segments. Simultaneously, we establish a set of actions as \(\{\text{a}_{\text{t}}\}\), with t taking values from \(\{0,\,1,\,2\,\ldots\text{N}\}\), signifying the decision made at step **t**. The duration of each step, **t**, is determined by the download time of the segment. The action set for a specific state is denoted as \(\text{A}(\text{s})=\{\text{A}_{\text{s1}},\text{A}_{\text{s2}},\text{A}_{ \text{s3}}\}\). Here, \(\text{A}_{\text{s1}}\) characterizes the scenario when \(\text{q}_{\text{r}}\) and \(\text{q}_{t-1}\) reside in the same quality set, as defined previously. \(\text{A}_{\text{s2}}\) pertains to cases where \(\text{q}_{\text{t}}\) and \(\text{q}_{t-1}\) do not belong to the same quality set but are the closest in quality levels. \(\text{A}_{\text{s3}}\) represents instances where \(\text{q}_{\text{t}}\) falls within a lower quality set. State transitions at each step are defined as \(\text{s}_{\text{t}}=\{\text{q}_{\text{t-1}},\,\text{q}_{\text{t}},\,\text{buff \_state}_{t},\,\text{Bw}_{\text{t}}\}\), encompassing variables such as \(\text{q}_{\text{t-1}}\) denoting the quality of the preceding segment, \(\text{q}_{\text{t}}\) signifying the quality of the downloaded segment, \(\text{Bw}_{\text{t}}\) representing the current bandwidth, and \(\text{buff\_state}_{t}\) indicating the current buffer state. To ascertain the quality of the subsequent segment, the agent utilizes a policy \(\mathbf{\pi}\) as an approximation strategy. In this work, we incorporate a deep neural network (DNN) to approximate the policy \(\pi(a|s;\theta)\), responsible for mapping the current state \(\mathbf{S}\) to the next state \(\text{S}^{\prime}\). The agent is encapsulated within a deep neural network, which generates the policy for decision making, while \(\theta\) denotes the weights within the deep neural network.  
The state transition at time t is denoted as \(\text{S}=(\text{Bw},\,\text{buff\_state}_{\text{t}},\,\text{q}_{\text{t}})\) after implementing the policy \(\pi(a|s;\theta)\). The selected action is \(\text{a}_{\text{t}}=\text{A}_{\text{s1}}\). The resulting state is:  
\[\text{S}^{\prime}=(\text{Bw}^{\prime},\,\text{buff\_state}^{\prime},\text{q} _{\text{t}}^{\prime}), \tag{7}\]  
where the following are true:  
\(\text{q}_{\text{t}^{\prime}}=\Delta\text{q}_{\text{t}}\) if (\(\text{q}_{\text{t}}\text{q}_{\text{t-1}}\)) are in the same quality set.  
Figure 2: Reinforcement learning architecture.  
\[\text{Bw}_{t}{}^{\prime}=\Delta\text{Bw}_{t}\] \[\text{buff\_state}{}_{t}{}^{\prime}=\Delta\text{buff\_state}{}_{t}\]  
A reward function \(\text{R}_{t}(\text{S}_{t})\) is received after downloading each segment \(\text{S}_{t}\). The reward is as follows:  
\[\text{r}_{t}=\text{R}\left(\text{St}=\text{S}\right)=\text{f}(\text{q}_{t}, \text{q}_{t-1}), \tag{8}\]  
The following Table 2 illustrates the rewards received after each action taken by the agent: