# Deep Q-Network Algorithm-Based Cyclic Air Braking Strategy for Heavy-Haul Trains
## 3 DQN Control Algorithm
### DQN Algorithm Model
#### 3.2.4 Action Value Function Design  
Defined as \(Q(s,a;\theta)\), the action value function demonstrates the quality of the action taken by the agent in a given state. In the form of a formula, the value of an action can be represented as the sum of all rewards that the agent is able to accumulate in subsequent steps, commencing from the current state and action. During the learning process, the agent is coached to choose an action in a manner that optimizes the overall future reward, referred to as the expected return, as shown in the following formula:  
\[R_{k}=\sum_{\mu=k}^{N}\gamma^{\mu-k}r_{k}, \tag{25}\]  
The optimal \(Q\) function \(Q^{*}(s,\ a)\) is characterized as the highest cumulative reward attainable by adhering to the policy \(\pi\) upon entering the state s and executing the action \(a\). In the form of \(Q(s,a)\), it is a function consisting of two parts: the state value function and the advantage function. The action value function expresses the expected return obtained by executing the action in the specified states, that is, indicating that taking this action is good or bad. The formula for calculating the optimal \(Q\) function is presented in Equation (26):  
\[Q*(s,a)=\underset{\pi}{max}E[R_{k}|s_{k}=s,a_{k}=a,\pi], \tag{26}\]Theoretically, the optimal \(Q\) function obeys the Bellman equation, which is shown in Equation (27):  
\[Q*(s,a)=E_{s^{\prime}}[r(s,a)+\gamma max_{a^{\prime}}Q*(s^{\prime},a)], \tag{27}\]  
where \(s^{\prime}\) and \(a^{\prime}\) denote the state and the potential action at the subsequent time step, respectively. In the DQN framework, the neural network is employed to approximate the optimal function. The \(Q\) function is estimated by the neural network, with the weight parameters denoted as \(\theta\) (henceforth called the \(Q\)-network). Throughout the training process, the \(Q\)-network undergoes training through weight adjustments aimed at reducing the mean square error to a minimum, as defined by Equation (27). In this context, the best possible target value, \(r+\gamma max_{a^{\prime}}Q^{*}(st,\ a)\), is approximated with the target value \(y=r+\gamma max_{a^{\prime}}\hat{Q}(s^{\prime},\ at;\ \theta^{-})\). Thus, the loss function of updating the weight for the network is as follows:  
\[L_{i}(\theta_{i})=E_{s,a\sim p(),s^{\prime}\sim p}[r+\gamma maxQ(st,at,\theta^ {-})-Q(s,a,\theta)]^{2}, \tag{28}\]  
where \(L_{i}\) is the loss of the \(i\)th iteration; \(p\) is the joint probability distribution of states and actions; \(\theta\) is the weight parameter of the \(Q\)-network in the iteration; and \(\theta^{-}\) is the weight parameter of the target network. The target network has an identical structure to the \(Q\)-network. To revise the \(Q\)-network, it is necessary to initially compute the derivative of the loss function concerning \(\theta\), and the gradient \(\
abla_{\theta}L\) will be obtained. Then, the stochastic gradient descent (SGD) method is used to update the weight \(\theta\) as follows:  
\[\theta+\lambda\bullet\
abla_{\theta}L\rightarrow\theta, \tag{29}\]  
where \(\lambda\) is the learning rate. The weight of the target network \(\theta^{-}\) is updated periodically. When the parameters of the \(Q\)-network undergo J iterations of updating, a weight \(\theta^{-}\) will be assigned to the target network to generate a new target network. Then, \(\hat{Q}\) will be adopted to generate the target value y of the \(Q\)-learning to perform subsequent parameter updates J times for the \(Q\)-network. Through this approach, the overestimation or training instability caused by the correlation between the \(Q\)-network and the target network \(\hat{Q}\) can be avoided; thus, the algorithm can calculate more accurately and stably, and the convergence speed can be increased. Algorithm 1 summarizes the control method for heavy-haul trains based on the DQN algorithm. ``` //Initialization//// 1: Use weight \(\theta\) to randomly initialize the \(Q\)-network. 2: Use weight \(\theta^{-}=\theta\) to initialize target network \(\hat{Q}\).