## 2 Analysis of Autonomous Control Algorithm for Mining Electric locomotives
### Theoretical Analysis
#### 2.2.1 Reinforcement Learning  
RL is a method for studying how an agent maximizes its reward in a complex and uncertain environment [15; 16]. Figure 4 shows the process of RL. The agent interacts with the environment, obtains the current state \(S_{t}\) and reward \(R_{t}\) at moment \(t\), and takes action \(A_{t}\) based on certain strategies. After the agent takes action \(A_{t}\), the environment obtains the latest state \(S_{t+1}\) and reward \(R_{t+1}\) at moment \(t+1\), and passes them to the agent. This interaction process can be represented by the Markov Decision Process (MDP).  
The MDP adds a decision layer to the commonly used Markov Process/Markov Reward Process, which is the action \(a\) shown in the Figure 5. This means that when the agent is in the state \(S_{t}\) at time \(t\), it must first decide on a specific action \(a\) to take in order to reach the middle layer, which is the black node in the diagram. After reaching the black node, the state of the agent at the moment \(t\)+1 also depends on the probability distribution.  
RL, as a trial and error learning method, continuously repeats the above interaction process between agents and the environment to find a mapping that can maximize the cumulative sum \(G_{t}\) of benefits \(R_{t}\) over time.  
\[G_{t}=\sum_{i=0}^{T-i-1}\gamma^{i}R_{t+i+1}. \tag{1}\]  
where, \(T\) is the total time, \(\gamma\) is the discount factor.  
However, the sum of cumulative returns, also known as value return \(G_{t}\), is not easily obtained. To solve this problem, researchers propose two value functions for estimating the sum of cumulative returns: the state value function \(V_{\pi}(s)\) and the action value function \(Q_{\pi}(s,a)\). The state value function \(V_{\pi}(s)\) is the expected value of the sum of the reward functions under \(\pi\) strategy and \(s\) state. The action value function \(Q_{\pi}(s,a)\) represents the expected value of the sum of the benefit functions of action \(a\) under the \(\pi\) strategy and \(s\) state.  
\[V_{\pi}(s)=\mathbb{E}_{\pi}[G_{t}|S_{t}=s]=\sum_{a\in\mathcal{A}}\pi(a|s)\cdot Q _{\pi}(s,a). \tag{2}\]  
Figure 4: Reinforcement Learning.  
Figure 5: The Markov Decision Process and Markov Process/Markov Reward Process.  
\[Q_{\pi}(s,a)=\mathbb{E}_{\pi}[G_{t}|S_{t}=s,A_{t}=a]=\sum_{r,s_{t+1}}p(s_{t+1},r|s, a)\cdot[R+\gamma V_{\pi}(s_{t+1})]. \tag{3}\]  
where, \(\mathcal{A}\) is the set of action \(a\).  
According to the value function, the Bellman equation is derived as follows:  
\[V(s)=R(s)+\gamma\sum_{s_{t+1}\in\mathcal{S}}p(s_{t+1}|s)\cdot V(s_{t+1}). \tag{4}\]  
where, \(\mathcal{S}\) is the set of state \(s\).  
The Bellman equation for the Q-function is as follows:  
\[Q(s,a)=R(s,a)+\gamma\underset{a_{t+1}}{\text{max}}Q(s_{t+1},a_{t+1}). \tag{5}\]  
We adopt Theorem - Cauchy's convergence Test to verify the convergence of RL we design to apply in the field of autonomous control of mining electric locomotives. The specific steps to prove the convergence of RL are detailed in Appendix A.  
The RL algorithm focuses on the feedback information obtained by the agent from the environment for decision-making, without relying on the agent and the environment model. This method does not require modeling of the intelligent agent, directly avoiding control bias caused by inaccurate modeling. In addition, this algorithm idea enables the algorithm proposed in this paper to be applicable to different types of mining electric locomotives driving on different road sections. So this method is universally applicable in complex mine conditions. The method proposed in this paper is applied in the actual field based on the results of laboratory simulation training, which greatly reduces the debugging time of the algorithm in the actual application process.