# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 2 Materials and Methods  
### Reinforcement Learning Algorithms  
Reinforcement learning provides a computational framework for learning and decision-making from past experiences. The primary aim of RL is to find a policy to achieve a goal which maximizes a cumulative reward. It is usually accomplished by a cycle of 'perceive-reason-act' in the environment. Given the current state of the environment, an agent takes an action against the environment. This brings about a new state and returns a reward in due course. As the environment changes, the agent perceives the new states and repeats this process until it finds the optimal policy to achieve a goal. In this process, it requires a variety of experiences composed of state-action pairs through \"_exploration_\". Depending on the outcomes, the agent takes an action yielding the highest reward based on the agent'sexperiences, called \"_exploitation_\". In doing so, the agent builds state value, or state-action value, as a means to find an optimal policy in the environment. Q-learning is a representative off-policy RL algorithm for agents to learn the policy. Given the state-action values, the agent selects the action that has the highest Q-value [25]. Q-value can be obtained through Equation (1). \[\text{Q}(\text{s}_{\text{t}},\ \text{a}_{\text{t}})=(1-\alpha)\text{Q}(\text{s}_{ \text{t}},\text{a}_{\text{t}})+\alpha\Big{(}\text{r}_{\text{t}}+\gamma\underset {\text{a}}{\text{max}}\text{Q}(\text{s}_{\text{t}+1},\text{a})\Big{)}. \tag{1}\]  
All mathematical symbols in Equation (1) are shown in Table 1.  
The Q-value represents the maximum cumulative reward achievable from a given state and action pair. The Q-value is updated to reflect the reward obtained immediately, weighted by the learning rate and the maximum cumulative reward that can be obtained in the future. The discount factor determines the proportion of future rewards to be considered, compared to the reward obtained at the present time. It is an exclusive value between 0 and 1, such that the greater the reward obtained in the distant future, the less it is reflected in the current Q-value. The policy determines the action that achieves the maximum Q-value, as given by Equation (1). Q-learning can learn the optimal policy in a finite Markov decision process. Because the tabular version of Q-learning in the environment lacks the ability to deal with large state- and action-spaces, deep Q-networks (DQN) was proposed which combines Q-learning and deep-learning [26]. As deep learning showed a remarkable capacity to generalize the value over a certain part of a state(-action) space, it enables the scaling down of the problem to a manageable size. However, DQN is likely to overestimate the problem in which the Q-value of a specific action continues to increase. This overestimating problem stems from the bias caused by approximating the state-action value using the max operator. Additionally, due to the max operator, selecting and evaluating an action using the same value is prone to overestimation. To prevent this, a double-deep Q-network (DDQN) that combines double Q-learning and DQN was introduced [27]. DDQN uses two separated Q-networks: one for the action selection and the other for the evaluation [28]. It is expressed as Equation (2). \[Y_{t}=R_{t+1}+\gamma Q\bigg{(}S_{t+1},\underset{a}{\text{arg}}\underset{a}{ \text{max}}\text{Q}(S_{t+1},a;\theta_{t}),\theta_{t}^{-}\bigg{)}. \tag{2}\]  
All mathematical symbols in Equation (2) are also shown in Table 1.  
Equation (2) shows that there are two Q-networks, \(\theta\) and \(\theta^{-}\). DDQN obtains the Q-value using Equation (1) with \(Y_{t}\) calculated using Equation (2), and updates the theta parameter of the Q-network that determines the action. Although \(\theta\) changes during learning, the \(\theta^{-}\) parameter of the Q-network used to calculate the future Q-value remains constant, allowing the Q-value to change for each episode by adjusting only the currently obtained reward.