## 4 Deep Reinforcement Learning  
Reinforcement learning (RL) is a subfield in machine learning, in which the agent can learn from trial and error by interacting with the environment and observing reward [19]. As shown in Figure 4, the agent, also referred to as the decision-maker, obtains an immediate reward \(r\) from the environment according to the current action \(a_{t}\) when its current state is \(s_{t}\). Moreover, the agent's state transits to \(s_{t+1}\) after executing the action \(a_{t}\). The goal of RL is to lean an optimal policy for an agent that can make it choose the best action according to current state. We can use a tuple \(\langle S,A,R\rangle\) to present the RL, the action an agent can choose is \(a_{t}\in A\) and the state an agent can reach is \(s_{t}\in S\). \(R\) represents the space of reward value. \(Q\) learning is a model-free algorithm [20] for RL, which can be used to get the optimal policy. The evaluation function \(Q(s_{t},a_{t})\) represents the maximum discount cumulative reward when the agent starts with state \(s_{t}\) and uses \(a_{t}\) as the first action. Therefore, the optimal policy \(\pi^{*}\) could be denoted as:  
\[\pi^{*}(s_{t})=\underset{a_{t}}{\text{argmax}}\ Q(s_{t},a_{t}) \tag{4}\]  
According to the Formula (4), in order to obtain the optimal policy \(\pi^{*}\), an agent needs to select the action that maximizes \(Q(s_{t},a_{t})\) when agent is in state \(s_{t}\). In general, \(Q(s_{t},a_{t})\) could be iteratively updated by the following Formula:  
\[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha[r+\gamma\underset{a^{\prime}}{ \text{max}}Q\big{(}s_{t+1},a^{\prime}\big{)}-Q(s_{t},a_{t})] \tag{5}\]  
where \(r\) represents immediate reward when the agent is in state \(s_{t}\) and selects action \(a_{t}\) to execute. \(\gamma\) (\(0\leq\gamma<1\)) is a constant that determines the discount value of a delayed reward. \(\alpha(0\leq\alpha\leq 1)\) is learning rate, a larger value of \(\alpha\) will lead to a faster convergence for \(Q\) function. However, when the state space and the action space are too large, it is very hard to make \(Q(s_{t},a_{t})\) converge by traversing all states and actions. DRL can handle the complicated problems with large state space and action space [21]. It has been successfully applied to Alpha Go [22] and playing Atari [23]. When facing high-dimension state space and action space, it is difficult to obtain \(Q(s_{t},a_{t})\) according to the original method. Suppose that if \(Q(s_{t},a_{t})\) could be represented by a function, it will be regarded as a value function approximation problem for obtaining \(Q(s_{t},a_{t})\). Therefore, we can approximate the function \(Q(s_{t},a_{t})\) by using function \(Q^{\prime}(s_{t},a_{t},\omega)\), where \(\omega\) represents related parameters. \[Q(s_{t},a_{t})\approx Q^{\prime}(s_{t},a_{t},\omega) \tag{6}\]  
The loss function was defined by using the mean-square error for DRL that was proposed in paper [24]. Therefore, we follow their definition about the loss function in our paper:  
\[Loss(\omega)=E\bigg{[}(r+\gamma\underset{a^{\prime}}{\text{max}}Q^{\prime} \big{(}s_{t+1},a^{\prime},\omega\big{)}-Q^{\prime}(s_{t},a_{t},\omega)\big{)}^ {2}\bigg{]} \tag{7}\]  
Figure 4: The progress of an agent interacts with the environment. Then, the gradient of \(Loss(\omega)\) is:  
\[\frac{\partial Loss(\omega)}{\partial\omega}=E\bigg{[}(r+\gamma\underset{a^{ \prime}}{\max}Q^{\prime}\big{(}s_{t+1},a^{\prime},\omega\big{)}-Q^{\prime}(s_{t},a_{t},\omega))\frac{\partial Q^{\prime}(s_{t},a_{t},\omega)}{\partial\omega} \bigg{]} \tag{8}\]  
In Formula (8), \(\frac{\partial Q^{\prime}(s_{t},a_{t},\omega)}{\partial\omega}\) could be calculated by a deep neural network (DNN) [25]. Therefore, the DRL is composed of an offline DNN phase and an online \(Q\)-learning progress. Different from RL, a DNN for estimating the \(Q\) value is constructed according to each state-action pair and corresponding \(Q\) value.