# Optimal Control Algorithm for Subway Train Operation by Proximal Policy Optimization
## 3 Methodology
### Proximal Policy Optimization Method  
Proximal Policy Optimization Method [28] is one of the policy gradient methods which work by computing an estimator of the policy gradient and plugging it into a stochastic gradient ascent algorithm. The most commonly used gradient estimator has the form of Equation (18).  
\[\hat{g}=\hat{\mathbb{E}}_{t}[\
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\hat {A}_{t}] \tag{18}\]  
And \(\hat{g}\) is obtained by differentiating the objective function shown as Equation (19).  
\[L^{PG}(\theta)=\hat{\mathbb{E}}_{t}[\log\pi_{\theta}(a_{t}|s_{t})\hat{A}_{t}] \tag{19}\]  
where \(\pi_{\theta}\) is a stochastic policy, \(a_{t}\) and \(s_{t}\) are the action and state at time \(t\), respectively, \(\hat{A}_{t}\) is an estimator of the advantage function.  
PPO is optimized from the Trust Region Policy Optimization (TRPO) and in TRPO, the objective function is maximized subject to a constraint on the size of the policy update shown in Equation (20) [29].  
\[\begin{array}{ll}\max\limits_{\theta}&\hat{\mathbb{E}}_{t}\Big{[}\frac{\pi_{ \theta}(a_{t}|s_{t})}{\pi_{a_{old}}(a_{t}|s_{t})}\hat{A}_{t}\Big{]}\\ s.t.&\hat{\mathbb{E}}_{t}\big{[}KL\big{[}\pi_{\theta_{old}}(\cdot|s_{t}),\pi_{ \theta}(\cdot|s_{t})\big{]}\big{]}\leq\delta\end{array} \tag{20}\]  
where \(\theta_{old}\) is the parameters before the update. In TRPO, the optimization problem (20) is solved by using a penalty instead of the constraint. But it is hard and complicated for TRPO to implement. There are two ways to make the implementation easier: adaptive KL penalty coefficient and clipped surrogate objective [28]. In this paper, we focus on the clipped surrogate objective due to its simplicity and accuracy.  
We denote the objective function in Equation (20) as follows:  
\[L^{CPI}(\theta)=\hat{\mathbb{E}}_{t}\bigg{[}\frac{\pi_{\theta}(a_{t}|s_{t})}{ \pi_{\theta_{old}}(a_{t}|s_{t})}\hat{A}_{t}\bigg{]} \tag{21}\]  
where \(CPI\) represents the conservative policy iteration [30].  
In PPO, a hyper-parameter \(\epsilon\) is introduced to limit the distance between old and new strategies to a specific range shown as Equation (22), where \(radio_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{ t})}\). The first part is the probability ratio multiplied by the advantage function that is the unclipped objective \(L^{CPI}\). The second part of Equation (22) is to limit the probability ratio \(radio_{t}(\theta)\) to the interval \([1-\epsilon,1+\epsilon]\) which is the clipped objective. At last, we take the minimum of the unclipped and clipped objective to calculate \(L^{CLIP}(\theta)\).  
\[L^{CLIP}(\theta)=\hat{\mathbb{E}}_{t}\big{[}\min(radio_{t}(\theta)\hat{A}_{t},clip(radio_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\big{]} \tag{22}\]  
In order to speed up the training process and improve the efficiency of the PPO algorithm, we adopted Reward Scaling [31] in the training process, shown in Equation (23).  
\[R(t)=\frac{R(t)}{Std_{R}+\xi} \tag{23}\]  
where \(Std_{R}\) is the standard deviation of rewards \(R\) and \(\xi\) is a very small positive number to prevent a zero denominator.  
The optimal control process for subway train operation based on PPO with reward scaling is shown as Algorithm A1 in Appendix A. The comparison between the PPO algorithm with and without Reward Scaling is described in detail in Section 4.