## 2 Multiple Objective-Driven Energy-Saving Control Model of a Single Train
### Methodology
#### 3.3.2 DRLDA Algorithm for Multi-Train Energy-Saving Control Strategy  
The external force, speed, dwell time, passenger waiting time, traction energy consumption and other factors are multiple basic variables that are interrelated and mutually restricted in the train operation control process. The speed, passenger flow, position and distance are the cumulative results of external forces on a train over time. In turn, the choice is also affected by the state of other basic variables. Moreover, when a train arrives at the station, the speed should reach a specified value, such as zero, and the actual operation time must be less than or equal to the planned operation time. At the same time, the relationships among time-varying passenger flow prediction, train arrival time, dwell time and traction energy consumption should also be considered before the train enters the station. During the train's operation, the speed must not exceed the speed limit of the specific location. To deal with these complex constraints, the implementation of multi-train energy-saving control using DRL is a challenging cooperative optimization problem, because the learning process will involve a long-term trial and error process. Thus, the DRL is suitable to be combined with DA to handle the constraints. The learning system retains the global exploration and learning ability of reinforcement learning (RL), avoiding the locality and incompleteness of pure RL-guided learning. Under DA hybrid learning, it compares the current state of DRL with its own expectations. If they are inconsistent, it means that the agent has made an incorrect decision. Then, DA intervenes with the agent in time through weight distribution. The agent receives specific feedback actions and instructs the controller to make a reliable decision about the state. Finally, the input data for a given train/passenger flow and railway line are used to learn from the train energy consumption and passenger travel data to achieve the goals of safety, energy saving, comfort, punctuality, etc. RL can make the optimal decision by learning the optimal control strategy incrementally, so as to maximize the cumulative reward value and improve the system performance. DRL combines the perception ability of Deep Learning (DL) with the decision-making ability of RL. Before applying the DRLDA algorithm to the multi-train energy saving control, the basic elements are defined as state space, action space, reward mechanism setting, strategy and value function, respectively. _Step 1_: State space  
To accurately present the current train information, multiple parameters are selected as environment states, which are related to different attributes of the environment: speed (defined as \(v_{s_{j}^{*}}\)), passenger waiting time (defined as \(Y_{1}\)), traction energy consumption (defined as \(Y_{2}\)), current position (defined as \(x_{i}\)) and the remaining operating time of the train (defined as \(t_{rest_{i}}\)). The state space is defined as Equation (50):  
\[ \begin{split} s_{i}&=[v_{s_{i}^{g}},Y_{1},Y_{2},x_{i},t_{rest_{i}}]\\ s.t.&\left\{\begin{array}{l}v_{s_{i}^{g}}\in[v_{ \min},v_{\max}]\\ Y_{1}\in[T_{aq},\Delta T]\\ Y_{2}\in[0,E_{j}]\\ x_{i}\in[0,x_{end}]\\ t_{rest_{i}}\in[0,t_{plan}]\end{split} \right.\end{split} \tag{50}\]  
where \(x_{end}\) is the end position of the railway line. \(t_{rest_{i}}=t_{plan}-t_{operting}\) represents the rest operating time of a train. \(t_{plan}\) is the train travel planning time. \(t_{operting}\) indicates the time that the train has been running. The process of train operation is discrete; at each time point \(t\), there are two kinds of states for trains, i.e., the stop at station state and the running state. For the stop at station state, the value ranges for the variables are \(v_{s_{i}^{g}}=0\), \(\Delta T\geq Y_{1}\geq T_{aq}\), \(Y_{2}=0\), \(x_{i}\in x_{stop}\), \(0\leq t_{rest_{i}}\leq t_{ducll}=d_{\max}\) to satisfy the upper limits of dwell time and train speed.