## 2 Multiple Objective-Driven Energy-Saving Control Model of a Single Train
### Methodology
#### 3.3.2 DRLDA Algorithm for Multi-Train Energy-Saving Control Strategy
For the running state, when train runs at the railway line, the value ranges for the variables are \(v_{\min}\leq v_{s_{i}^{g}}\leq v_{\max}\), \(\Delta T\geq Y_{1}\geq T_{aq}\), \(Y_{2}\in E_{j}\), \(x_{i}\in x_{end}-x_{stop}\), \(t_{ducll}-\Delta t\leq t_{rest_{i}}\leq t_{rest_{i}^{\max}}^{\max}\) (where \(\Delta t\) is time intervals), and should satisfy the speed limit and the upper limit of the remaining operating time. During state transition, the state space \(s_{i}\) of multiple trains can be obtained by generating all candidate states within the value range. _Step 2_: Action space  
Trains generate corresponding traction or braking forces to control multiple operation based on the continuous output of the main control handle or URT system. Therefore, the action space \(A\) is composed of the variation tendency of the control force level (defined as \(CFL_{i}\)) according to the current operating states. The action set is defined as follows:  
\[ \begin{split} A&=\{a_{n_{i}}\}=|CFL_{i}|,CFL_{i}\in[ -100\%,100\%]\\ s.t.&\left\{\begin{array}{l}U_{1}=CFL_{i}*F_{ \text{traction\_max}}CFL_{i}>0\\ U_{2}=CFL_{i}*F_{\text{coasting}}CFL_{i}<0\\ U_{3}=CFL_{i}*F_{\text{secondary\_ traction}}CFL_{i}>0\\ U_{4}=0*CFL_{i}=0\\ U_{5}=CFL_{i}*B_{\max}CFL_{i}<0\end{array}\right.\end{split}  \tag{51}\]  
where \(U_{i}\) represents the different effects under different operating conditions and running states. \(F_{\text{traction\_max}}\) and \(B_{\max}\) are the maximum traction force and the maximum braking force of the train, respectively. \(F_{\text{secondary\_ traction}}\) is the secondary traction force. \(F_{\text{coasting}}\) is the train coasting force. The train with variation tendency of the control force level range is \([-100\%,100\%]\). The maximum traction force is \(100\%\), and the maximum braking force is \(-100\%\). A value of \(0\%\) means that train runs in a cruise state. _Step 3_: Reward mechanism setting  
An agent is able to randomly generate control strategies based on the definition of state and action. Reward function is of great importance in the realistic optimization objective field of RL applications. Different reward values can guide the algorithm to learn better control strategies in order to maximize the benefit. A reasonable reward and punishment mechanism can improve the convergence speed. Generally, under the cooperative setting, the common reward (defined as \(R_{\text{total}}(s_{i},a_{n_{i}})\)) is obtained by the independent individual reward and the cooperative individual reward based on the relationship between multiple trains. In order to reduce the energy consumption of all trains, \(R_{\text{total}}(s_{i},a_{n_{i}})\) is calculated by Equation (52). \[R_{\text{total}}(s_{i},a_{n_{i}})=\min Y=\theta Y_{1}+\omega Y_{2} \tag{52}\]_Step 4: Strategy and value function_  
In RL, the train selects an action mapping a probability distribution to execute in the current state, which is defined as strategy matching \(\pi\). The agent selects a dynamic state-action function \(Q\) with a maximum value. The current state-action function \(Q\) is defined as:  
\[Q[s_{i},\pi(s_{i})]=E\{(1-\kappa)R_{\text{total}}[s_{i},\pi(a_{n_{i}}|s_{i})]+ \kappa Q[s_{i+1},\pi(a_{n_{i}}|s_{i+1})]\} \tag{53}\]  
where \(\kappa\) is a discount factor; the larger \(\kappa\) is, the greater the focus on future cumulative returns. The cumulative return value of the action strategy is defined as a function \(J(\pi)\), calculated by \(J(\pi)=E_{\sim\tau}\{Q[s,\pi(a|s)]\}\). Thus, the optimal control strategy can be calculated by Equation (54). \[optimal_{\pi}=\text{argmax}_{\pi}J(\pi) \tag{54}\]  
_Step 5: DRLDA strategy design_  
This algorithm consists two kinds of networks, i.e., actor network and critic network, experience replay pool, multi-train energy-saving control model and reward function.