# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 3 Methodology
### PPO Algorithm  
1. Policy-Based Framework  
The interaction process between intelligent agents and the environment is a continuous decision-making process, and there is a correlation between actions and the next state. Markov chain can be used to simplify and model this process and is parameterized as (\(S\), \(A\), \(r\), \(P_{0}\), \(P\), \(\gamma\)). Among them, \(S\) is the state space of the environment, \(A\) is the action space of agent, \(r(s_{t})\) is the timely reward to the environment, \(P_{0}(s_{0})\) is the probability distribution of the environment initial state, \(P(s_{t+1}|s_{t},\ a_{t}\ )\) is the probability distribution of environment state transition, and \(\gamma\) is the discount factor for discounting future rewards. For sequential decision-making optimization, the policy-based method is an effective optimization framework, which firstly parameterizes the policy, sets the evaluation index  
Figure 4: OACP modeling framework. of the policy, and finally optimizes the parameters with the evaluation index as the objective function. The policy is parameterized by the neural network expressed as \(\pi(a_{t}|s_{t})\), and the evaluation index of the policy is the discounted episode reward expectation \(\eta(\pi)\) which is optimized to obtain the optimal policy, as shown in Formula (1). \[\eta(\pi)=\mathrm{E}_{s_{0},a_{0}s_{1},\dots\sim\pi}\left[\sum_{t=0}^{\infty} \gamma^{t}r(s_{t})\right] \tag{1}\]  
where \(s_{0}\sim P_{0}\), \(a_{t}\sim\pi(a_{t}|s_{t})\), \(s_{t}\sim P(s_{t+1}|s_{t},\ a_{t}\ )\). Since the environment was unknown, it was necessary to estimate the statistics variables through sampling for the optimization solution. To reduce the large estimation error of the direct sampling method by the temporal difference method (TD), the state value function \(V_{\pi}(s_{t})\), the action-state value function \(Q_{\pi}(s_{t},\ a_{t})\) and the advantage value function \(A_{\pi_{old}}(s_{t},\ a_{t})\) were introduced and applied to the objective function [24; 25]. The TD method can greatly reduce the estimated variance of the variables while appropriately increasing the estimated deviation, and the n-step bootstrap method can make a trade-off between the estimated deviation and variance according to the need. For the unstable caused by the fixed step size of the random gradient method, TRPO [26] introduced the trust region method, which transforms the optimization problem into the iterative subproblems. To ensure the monotonic improvement of the objective function, the objective function of the subproblem should meet three conditions: (1) be the lower bound of the original objective function; (2) approximate the original objective function within a certain region; and (3) be easy to solve. The relationship between the episode reward expectations from the two policies is established in Formula (2). By deducing and simplifying, the approximate episode reward expectation \(L_{\pi_{old}}(\pi)\) in Formula (3) was obtained, which can be easy to solve. \(L_{\pi_{old}}(\pi)\) can approximate the original objective function in a certain region. Further, the lower bound of the original objective function is constructed using \(L_{\pi_{old}}(\pi)\), as shown in Formula (4), which satisfies the above three conditions at the same time. The function can be used as the objective function of the subproblem for an iterative optimization, as shown in Formula (5), which can guarantee the monotonic improvement of the original objective function. To ensure the sufficient update step size on the premise of robustness, Formula (5) was equivalently transformed into an optimization form with constraints, and the maximum divergence constraint was simplified to an average divergence constraint to ensure that the problem can be solved, as shown in Formula (6). \[\eta(\pi)=\eta(\pi_{old})+\mathrm{E}_{s_{0},a_{0}s_{1},\dots\sim\pi}\left[\sum _{t=0}^{\infty}\gamma^{t}A_{\pi_{old}}(s_{t},\ a_{t})\right] \tag{2}\]  
where \(A_{\pi_{old}}(s_{t},\ a_{t})\) is the advantage function of \(\pi_{old}\).