## 2 Multiple Objective-Driven Energy-Saving Control Model of a Single Train
### Methodology
#### 3.3.2 DRLDA Algorithm for Multi-Train Energy-Saving Control Strategy
The DRLDA algorithm learning framework is illustrated in Figure 3. The two main networks, including the actor network and the critic network, output the train state and control strategy to the train cooperative model to simulate the train operation process. The reward function evaluates the action strategy. Then, the reward values and train states are stored in an experience replay pool through a replay memory mechanism. When the pool is full of data, small batches of random samples are used to train the two main networks, which can effectively reduce the correlations among samples and improve the learning speed. The training process of the proposed algorithm is divided into a state-action function update and an action strategy update. Two main networks combine the advantages of value function update and strategy function update. The critic network adopts the single-step update method based on the value function to evaluate the current multi-train control strategy. The network parameter is defined as \(\xi^{Q}\). The actor network adopts the strategy gradient update method, which can better solve the problem of continuous action space, and is used to output the multi-train energy-saving control strategy. The network parameter is defined as \(\xi^{\pi}\). * Critic Network  
The critical network is similar to the Deep Q-Network (DQN) algorithm, which uses a fully connected neural network to fit each state action value, that is, \(Q(s,A\big{|}\xi^{Q})=Q(s,A)\). The state observations and the actions of sample data are taken as the input. The state action value \(Q\) is selected as the output. A larger \(Q\) value indicates a higher evaluation of  
Figure 3: The DRLDA algorithm learning framework. that action strategy. The new weight of the \(Q\) neural network parameters is calculated by Equation (55). \[\xi_{t+1}^{Q}=\xi_{t}^{Q}+\zeta_{Q}\Delta y\
abla Q\Big{(}s,a\Big{|}\xi_{t}^{Q} \Big{)} \tag{55}\]  
\[\Delta y=y^{\prime}_{t}-y_{t}\text{ is established, where }\psi^{\prime}_{t}=R_{total_{t}}+ \kappa Q_{t+1}\Big{[}s_{t+1},\pi(s_{i})_{t}\Big{(}s_{t+1}\Big{|}\xi^{\pi(s_{i })_{t}}\Big{)}\,\big{|}\xi^{Q}\Big{]},\]  
\[y_{t}=Q_{t}\Big{[}s_{t},\pi(s_{i})_{t}\Big{(}s_{t}\Big{|}\xi^{\pi(s_{i})_{t}} \Big{)}\,\big{|}\xi^{Q}\Big{]}.\]  
When calculating, the above equations are substituted into Equation (55). Where \(y^{\prime}_{t}\) is target value, \(\zeta_{Q}\) is the learning rate. \(\
abla Q\Big{(}s,a\Big{|}\xi_{t}^{Q}\Big{)}\) is the update gradient calculated by the critical neural network based on the \(Q\) value, which is output to the actor network as the basis for updating the action strategy. If the gradient is positive, it means that the optimal strategy is close to this control strategy. The larger the gradient, the faster the approach speed. * Actor Network  
The division of action space has a great impact on RL. If the space is too small, it is impossible to accurately describe the train movement and operation process. If the division space is too large, it leads to too large a dimensionality, so the train cannot learn all the actions. The actor network is applied to \(Q\) neural network parameters to estimate the train control strategy. The state observations of the sample data are taken as the input, and the action strategy is taken as the output. The different types of strategies can be divided into stochastic and deterministic strategies. In the process of train operation, there is a certain optimal control strategy in each state. Thus, this paper selects the deterministic strategy. The deterministic strategy outputs the corresponding optimal action in the current state, that is, \(a=\pi(s_{i})_{g}\). According to gradient ascent ([PERSON] et al. [54]), the derivative of \(J(\pi)\) is calculated for obtaining the strategy gradient, which is shown in Equation (56).