# Deep Q-Network Algorithm-Based Cyclic Air Braking Strategy for Heavy-Haul Trains
## 3 DQN Control Algorithm
### Markov Decision Process  
Prior to implementing the DQN algorithm, it is essential to define the control process for train operation when running on a long and steep downhill section as a Markov decision process (MDP), which involves formalizing sequential decision-making. An illustration of the MDP interaction during the operation of heavy-haul trains running on long and steep downhill sections is shown in Figure 2.  
The locomotive of a heavy-haul train is described as the agent responsible for making control decisions, while the heavy-haul train's dynamics and the railway's underlying structure arrangements are defined as the environment. Assuming that the time interval between two consecutive states is \(\Delta t\), the control process is evenly divided into \(N\) steps with respect to the planned operating time \(T\), and the position, operating speed and operating time of the train are taken as the states of the heavy-haul train.  
\[s_{k}=[P_{k},V_{k},T_{k}],k=0,1,2,\cdots,n, \tag{14}\]  
where \(s_{k}\) is the status of the heavy-haul train at step \(k\), \(P_{k}\) is the position of the train, \(V_{k}\) is the train speed and \(T_{k}\) is the train running time. \(s_{0}\) and \(s_{n}\) represent, respectively, the initial  
Figure 2: Illustration of MDP interaction during operation of heavy-haul trains.  
state and final state of the heavy-haul train running on a long and steep downhill section, and their calculation is as follows:  
\[s_{0}=[0,V_{0},0], \tag{15}\]  
\[s_{n}=[P,V_{n},T], \tag{16}\]  
Starting from the initial state \(s_{0}\) of the train, succeeding states in the train control process are determined through actions until the final state \(s_{n}\) is reached. Specifically, at a given train state \(s_{k}\), the agent chooses the optimal action \(a_{k}\) from all feasible actions. After executing the action \(a_{k}\), the agent can obtain the next state \(s_{k+1}\) and receive the corresponding reward \(r_{k+1}\). The next state is \(s_{k+1}\), which is solely influenced by the current state \(s_{k}\) and action \(a_{k}\). Its calculation is as follows:  
\[s_{k+1}=\Phi(s_{k},a_{k}),k=0,1,\cdots,n-1, \tag{17}\]  
where \(\Phi()\) represents the functional dynamics of the system associated with the heavy-haul train dynamic model, while \((s_{k},a_{k})\) is denoted as the state-action pair. Notably, the agent lacks prior knowledge of train dynamics. Such knowledge is solely utilized for generating driving experience throughout the interaction and to supply the agent with training data. Herein, the control action refers to the setting of electric braking and the air braking notch.  
\[a_{k}=[h_{k}^{a},h_{k}^{d}],k=0,1,\cdots,n, \tag{18}\]  
where \(h_{k}^{a}\) is a binary variable, and \(h_{k}^{d}\) denotes the relative output ratio of the electric braking force generated by the train locomotive, which is subjected to the constraint condition outlined in Equation (11).  
The control output of a heavy-haul train in each cycle is determined solely by the speed and time of the current train. Thus, the control process of a heavy-haul train can be defined by reinforcement learning as a Markov decision process, which is expressed as follows:  
\[s_{0}\overset{a_{0}}{\rightarrow}s_{1},r_{1}\overset{a_{1}}{\rightarrow} \cdots s_{k},r_{k}\cdots\overset{a_{n-2}}{\rightarrow}s_{n-1},r_{n-1} \overset{a_{n-1}}{\rightarrow}s_{n}, \tag{19}\]