# Optimal Control Algorithm for Subway Train Operation by Proximal Policy Optimization
## 1 Introduction  
The urban subway is the main line of urban public transportation and the main artery of passenger flow. It is a key system project for the rapid and smooth operation of urban traffic. It plays an important role in improving the happiness index and promoting social and economic development. With the rapid development of urban subways and the continuous expansion of network scale, the sustainable development of urban subways has attracted more and more attention. Although urban subways use less energy per unit of traffic than other forms of public transportation, they are massive. With the continuous expansion of urban subways, total energy consumption has increased dramatically. Among them, the traction energy consumption of train operation accounts for a relatively high proportion. Urban subway trains use on-board inverters to convert the DC power in the power supply system into AC power for the traction motor to drive the train, which requires a lot of power consumption. At the same time, auxiliary facilities such as lighting, air conditioning, and electric heating inside the carriage also require energy from the traction power supply system. In this situation, optimizing train operation energy consumption while maintaining comfort and punctuality is crucial for train operations.  
At present, many scholars have carried out relevant research on the train operation optimization problem by using heuristic methods such as genetic algorithms and differential evolution. The optimization process is roughly divided into two steps: first, optimize the train speed profiles, and then track the optimized speed profiles. Manual tuning, speed profiles, and operational constraints have been considered in existing work in order to improve performance. However, these solutions may lead to a gap between the control model and the actual operating process, thereby affecting the accuracy of the control model [1]. The emergence of reinforcement learning has found a feasible solution to this problem. It can directly optimize the train operation control strategy without the train speed profiles, to get rid of the limitations of the offline speed profiles and the precise model information of the train.  
In this thesis, we propose an optimal control algorithm for subway train operation based on Proximal Policy Optimization (PPO) to reduce train operation energy consumption, ensure passenger comfort, and at the same time meet the requirements of the train schedule for train operation time. In this paper, the position and speed of the train are used as the reinforcement learning state, the energy consumption and comfort are the optimization goals, the train running time is strongly constrained, and the output sequence of the train operation control is optimized based on PPO.  
This study is organized as follows: Section 2 presents a literature review about optimal control algorithms for subway train operation from the perspective of heuristic methods and reinforcement learning methods. Section 3 introduces the optimal control algorithm for subway train operation based on PPO, including the original PPO algorithm and the combination between PPO and the train operation model. Section 4 discusses the results of the PPO-based optimal control algorithm for subway train operation based on real data. Section 5 summarizes the innovation of this study and provides directions for future research.