# Deep Q-Network Algorithm-Based Cyclic Air Braking Strategy for Heavy-Haul Trains
## 3 DQN Control Algorithm
### DQN Algorithm Model
#### 3.2.4 Action Value Function Design
Figure 4: Change curves of cumulative reward at different learning rates: (**a**) the learning rate \(\lambda\) of DQN algorithm is 0.03; (**b**) the learning rate \(\lambda\) of DQN algorithm is 0.003; (**c**) the learning rate \(\lambda\) of DQN algorithm is 0.0003. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameter** & **Value** & **Parameter** & **Value** \\ \hline Maximum training episode \(M\) & 100,000 & Minimum air-refilling time \(T_{AI}\) & 100 \\ Update period of target network & 300 & Batch size of sampling \(n_{e}\) & 32 \\ Discount rate \(\gamma\) & 0.94 & Learning rate \(\lambda\) & 0.001 \\ Initial value of \(\varepsilon\) & 0.98 & Final value of \(\varepsilon\) & 0.1 \\ Capacity of memory buffer \(D\) & 5000 & Planned operation time \(T\) & 1500 s \\ Positive reward \(R_{d}\) & 5 & Negative reward \(R_{c}\) & \(-20\) \\ Minimum braking speed \(V^{\tau}_{min}\) & 40 km/h & Maximum braking speed \(V^{\tau}_{max}\) & 80 km/h \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Algorithm hyperparameters. Figure 5 shows the change curve of the total cumulative reward with the number of training episodes at a learning rate of 0.003 and a training period of 10,000 episodes. It also illustrates the change curve of the cumulative reward after the algorithm has converged. The other parameters of the DQN algorithm are the same as those shown in Table 3.

It can be seen from Figure 5 that in the 10,000-episode training experiment, the fluctuation range of the cumulative reward gradually decreases as the number of training episodes increases. From the change curve of the cumulative reward, it seems that the cumulative reward was maintained at a significant level within a designated range upon practice when executed about 5000 times, and the average cumulative reward converged to about 1400. After training was executed 10,000 times, the average cumulative reward remained at 1450, indicating that the proposed algorithm is convergent. #### 4.2.2 Practical Application Performance Test

Figure 6a,b exhibit the outputs of the air braking force and electric braking force during the control process of the DQN algorithm, respectively. The curve in Figure 7 shows the train speed controlled by the DQN algorithm. With the above parameter, i.e., the index Done was 1 with a learning rate of 0.003, the speed of the train was kept within the limitation range during the operation. This means that the agent had learned a safe and feasible control strategy during the test simulations to ensure the safe operation of the heavy-haul train. Moreover, it is evident from Figures 6 and 7 that during the process of train control, cyclic air braking was engaged when the train was running on the long and steep downhill section. Specifically, air braking was not applied during the initial 80 s. Since the overall braking force was not sufficient to decelerate the heavy-haul train, the train speed increased during this period. In the following 100 s, air braking and electric braking were utilized concurrently (\(h^{a}=1\) and \(h^{d}=0.8\)), resulting in the train's velocity dropping from 76 km/h to 52 km/h. Subsequently, air braking was discontinued, while electric braking was maintained constantly for the next 100 s (\(h^{a}=0\) and \(h^{d}=0.8\)). At 210 s, the train speed reached 78 km/h. At this moment, air braking was reactivated to reduce the train's velocity (\(h^{a}=1\) and \(h^{d}=0.8\)). Figure 5: Change curve of total cumulative reward of the DQN algorithm with \(\lambda=0.003\). Herein, an intelligent air braking strategy employing the DQN algorithm was used to implement five braking cycles for a heavy-haul train running on a long and steep downhill railway section. This approach ensured the safety of train operation. Furthermore, Figure 7 unequivocally illustrates that the train's safe operation indicator Y maintains a value of 1, which signifies that the train velocity consistently remains within the predefined safety limits during its entire operation.