# Deep Q-Network Algorithm-Based Cyclic Air Braking Strategy for Heavy-Haul Trains
## 3 DQN Control Algorithm
### DQN Algorithm Model
#### 3.2.4 Action Value Function Design
3: initialize the experience pool D with size C, greedy probability \(\varepsilon\), small-batch sample size \(n_{e}\), discount rate \(\gamma\), learning rate \(\lambda\) and the parameter update episode J of the target network //Process of training//
4: for episode = 1,...M do
5: initialize the state \(s_{0}\) of train through Equation (13)
6



## 4 Algorithm Simulation and Analysis

In this study, simulations were carried out with real data obtained for a section of the Shuohuang railway line in China to confirm the efficacy of the proposed algorithm. Firstly, the experimental parameter data setting was introduced. Then, the simulation results were presented and analyzed. The simulations were divided into three parts: the model training process, a practical application performance test, and performance comparison using different algorithms. ### Experimental Parameter Settings

To prove the effectiveness of the intelligent control method, a heavy-haul train weighing 20,000 tons was taken as the object of the simulations. The train formation consisted of 1 HXD1 electric locomotive + 108 freight cars + 1 HXD1 electric locomotive + 108 freight cars. The HXD1 electric locomotive had the abilities of traction and regenerative braking. The whole train was equipped with air braking. For specific train parameters, see Table 1.

Based on the data for the section spanning from Longgong Station to Beidaniu Station of the Shuozhou-Huanghua Line, simulations were carried out to obtain a speed curve of the heavy-haul train running on a long and steep downhill section. The operation section had a total length \(S\) = 23,800 m, with a gradient of 10-12% for the extended and precipitous downhill stretch. The speed limit on this line was 80 km/h. The specific data are presented in Table 2.

The parameter settings of Algorithm 1 used in this paper are listed in Table 3: \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dicomotive Parameters** & \multicolumn{3}{c}{**Freight Car Parameters**} \\
**Parameter Name** & **Value** & **Parameter Name** & **Value** \\ \hline Model & HXD1 & Model & C80 \\ Mass & 200 t & Mass & 100 t \\ Length & 35.2 m & Length & 13.2 m \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Train parameters. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Distance (m)** & **Gradient (-\%o)** & **Distance (m)** & **Gradient (-\%o)** \\ \hline
0–1000 & 1.5 & 12,430–14,080 & 10.5 \\
1000–1400 & 7.5 & 14,080–16,330 & 11.4 \\
1400–6200 & 10.9 & 16,330–19,130 & 10.6 \\
6200–6750 & 9 & 19,130–22,260 & 10.9 \\
6750–12,430 & 11.3 & 22,260–23,800 & 3.3 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Route information. ### Simulation Experiment Verification

#### 4.2.1 Model Training Process

In this study, simulations were carried out under different parameters and training cycles with the proposed DQN algorithm for verification purposes. Under the specified circumstances, the time interval \(\Delta t\) was 100 s. In addition, the initial speed \(V_{0}\) of the heavy-haul train running on a long and steep downhill section was 65 km/h. The change curve of the cumulative reward obtained using the proposed algorithm is shown in Figure 4. As shown in Figure 4a-c, the learning rates were determined to be 0.03, 0.003 and 0.0003, respectively, for the cumulative reward curve that changes depending on the number of episodes in different experiments. It is evident that during the short-term training period, the total cumulative reward gradually increases and converges in a good direction after training, as shown in Figure 4a-c. It should be noted that the best training performance is in Figure 4b, where the learning rate was determined to be 0.003. The average total reward's evolution curve exhibits a swifter and more consistent convergence rate, along with a higher convergence value. Consequently, a learning rate of 0.003 was selected for the other experiments.