# Optimal Control Algorithm for Subway Train Operation by Proximal Policy Optimization
## 3 Methodology
### Optimization Model of Train Operation Based on Reinforcement Learning  
This subsection presents the reinforcement learning (RL) modules for train operation. The RL architecture includes the following items: _agent_, _environment_, _action_, _state_, and _reward_. _agent_ obtains _state_ from _environment_, and executes _action_. Then, _action_ is output into _environment_, which can cause the transfer of _state_ and _reward_ generated by _environment_ to _agent_. RL aims to obtain as much _reward_ as possible from _environment_. _Agent:_ As the output module of the train operation control command, the ATO subsystem plays the role of the agent in the RL architecture, which controls the speed and position of the train by changing the output level of the traction system. Action SpaceDuring the operation of the train, the output level of the traction/braking system is the only way to control the change in speed and position. Changes in traction/brake levels will result in corresponding changes in train speed and position. The traction and brake output of subway trains are continuous variables, and when the traction/brake reach the maximum, it will output the max traction/braking force (represented by \(f_{max}\) and \(b_{max}\), respectively). So, we can define the action space in the RL architecture as \([-1,1]\), and the action \(a_{t}\) taken by the agent at time \(t\) conforms to Equation (10). If \(a_{t}>0\), the train is under the traction condition with traction power \(a_{t}\times f_{max}\), and the train is under the brake condition with brake force \(a_{t}\times b_{max}\) if \(a_{t}<0\). If \(a_{t}=0\), the train is under the coasting condition. \[a_{t}\in[-1,1] \tag{10}\]  
State SpaceTwo crucial elements describe the operation characteristic of the train: speed and position. When the agent, known as the ATO system, takes an action, the speed and position of the train will be changed immediately. We define the speed and position of the train as the state of RL architecture, and the state \(s_{t}\) at time \(t\) can be described as Equation (11). Note that the speed and the position are bounded within a certain range, shown in Equation (12). \[s_{t}=(v_{t},p_{t}) \tag{11}\]  
\[ \begin{cases}v_{t}\in[0,v_{max}(t)]\\ \quad p_{t}\in[0,P]\end{cases}  \tag{12}\]  
EnvironmentThe environment in the RL architecture requires an accurate relationship between the agent's output and the state transfer. In this paper, the purpose of the environment is to describe the relationship between the output level of the train traction system and the speed and position of the train, i.e., to construct an accurate train dynamics model with the control level as the independent variable. We assumed that at time \(t\), the speed and the position of the train are \(v_{t}\) and \(p_{t}\), respectively. Meanwhile, the agent takes an action \(a_{t}\). The traction force and brake force are calculated by Equations (13) and (14), respectively. \[f_{t}= \begin{cases}a_{t}\times f_{max}&a_{t}>0\\ \quad 0&a_{t}\leq 0\end{cases}  \tag{13}\]  
\[b_{t}= \begin{cases}a_{t}\times b_{max}&a_{t}<0\\ \quad 0&a_{t}\geq 0\end{cases}  \tag{14}\]  
Then, we can get the acceleration of the train \(u_{t}\) using Equations (1), (5) and (7). After that, the state of RL architecture \((v_{t},p_{t})\) is transferred to \((v_{t+\Delta t},p_{t+\Delta t})\) by Equation (15) (assuming that the action execution time is \(\Delta t\)). \[ \begin{cases}\quad\quad v_{t+\Delta t}=v_{t}+u_{t}\Delta t\\ \quad p_{t+\Delta t}=p_{t}+v_{t}\Delta t+0.5u_{t}\Delta t^{2}\end{cases}  \tag{15}\]  
RewardThe reward of RL architecture comprises three components: the reward of energy saving, the reward of comfort, and the penalty term of time exceeded. We define the reward \(r(t)\) at time \(t\) as Equation (16). \[r(t)=-\mu_{1}E(t)+\mu_{2}C(t) \tag{16}\]  
where \(E(t)\) and \(C(t)\) are calculated by Equations (3) and (8), respectively, and \(\mu_{1}\) and \(\mu_{2}\) are the coefficients of the reward of energy saving and the reward of comfort.