# Optimal Control Algorithm for Subway Train Operation by Proximal Policy Optimization
## 3 Methodology
### The Train Dynamic Model  
Figure 1 shows the force analysis while the train is running. \(f\) is the traction force output by the train's traction systems. \(G(p)\) represents the additional gradient resistance due to the gradient on the line track while \(p\) represents the position of the railway track. \(R(v)\) is the resistance produced by friction while \(v\) is the running speed of the train. \(b\) is the braking force generated by the braking system of the train. As shown in Figure 1, when the train is running, the resistance it receives includes \(G(p)\) and \(R(v)\). Taking the train traction and braking forces into account, the kinematic equation is shown in Equation (1). \[\left\{ \begin{array}{c}\frac{dp}{dt}=\frac{f-b-R(v)-G(p)}{(1+\tau M)}\\ \frac{dp}{dt}=v\end{array} \right. \tag{1}\]  
while \(M\) is the train mass and \(\tau\) represents the rotation mass coefficient of the train, and its value is determined by the total mass of the train and the converted mass of the rotation part. The energy consumption is calculated by Equation (2). \[E(T)=\frac{\int_{0}^{T}fv(t)dt}{\eta_{f}}+\int_{0}^{T}P_{au}(t)dt-\eta_{b}\int _{0}^{T}bv(t)dt \tag{2}\]  
In Equation (2), \(T\) is the travel time of the train, and \(P_{au}(t)\) is the power of train auxiliary systems, such as lights and air conditioners, at time \(t\). \(\eta_{f}\) is the efficiency of converting electrical energy into mechanical energy when the train traction system is working. \(\eta_{b}\) is the efficiency of converting mechanical energy into electrical energy during the regenerative braking of the train. The first part of Equation (2) represents the electric energy consumed by the traction system of the train; the second part is the energy consumed by auxiliary systems of the train, and the third part is the energy generated by train regenerative braking. Compared with the energy consumed by the traction system, the proportion of auxiliary energy consumption in the process of train operation is very small, so this paper takes the energy consumed by  
Figure 1: The force analysis while the train is running. the train traction drive system as the optimization object. That is, the optimization objective of the optimization problem is obtained by Equation (3). \[E(T)=\frac{\int_{0}^{T}fv(t)dt}{\eta_{f}}-\eta_{b}\int_{0}^{T}bv(t)dt \tag{3}\]  
The additional gradient resistance \(G(p)\) and the resistance produced by friction \(R(v)\) are calculated by Equations (4) and (5), respectively. \[G(p)=Mg\sin\left(\theta(p)\right) \tag{4}\]  
\[R(v)=r_{0}+r_{1}v+r_{2}v^{2} \tag{5}\]  
while \(g\) is the gravitational acceleration; \(\theta(p)\) is the gradient of the line on position \(p\); \(r_{0}\), \(r_{1}\) and \(r_{2}\) are basic resistance coefficients. In Equation (4), the gradient \(\theta(p)\) is measured by radians. To ensure the safety of train operation, the gradient of the line is strictly limited within a certain range, that is, the slope is at a small value. We can thus make the following approximation. \[\sin\left(\theta(p)\right)\approx\theta(p) \tag{6}\]  
Then we can change Equation (4) into Equation (7) for more convenient calculation. \[G(p)=Mg\theta(p) \tag{7}\]  
In addition to energy consumption, passenger comfort is defined by the negative integral of jerk rate over time as another optimization object, and it is defined by Equation (8). \[C(T)=-\int_{0}^{T}\left|\frac{du}{dt}\right|dt \tag{8}\]  
while \(u\) is the acceleration of the train. Equations (3) and (8) are the objects to be optimized. There are additional constraints to consider, such as travel time and speed limitations, in addition to the dynamic characteristics of the subway train as depicted in Equation (1). Then we have the multi-objective optimization function for train operation, as shown in Equation (9). \[ \begin{array}{c}\min Op(E(T),C)\\ \\ \begin{array}{c}\begin{cases}\frac{dv}{dt}=\frac{f-b-R(v)-G(p)}{(1+\tau M)}\\ \frac{dp}{dt}=v\end{cases}\\ \begin{array}{c}v(0)=v(T)=0\\ t\in[0,T]\\ v(t)\in[0,v_{\max}(t)]\end{cases}\end{array}\end{array}\end{array}