# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 4 Case Study
### OACP Model  
Due to the different TBMAP response laws of different GEC environments, it was necessary to establish corresponding OACP for four GECs using reinforcement learning. For each GEC, the full-connect DNN was used to establish the attitude control policy, which outputted two continuous control parameters corresponding to DDST and DLST. To ensure the exploration, the distributions of control parameters for each TBMAP input were modeled as the normal distribution, defined by the mean and variance from the established full-connect DNN model, and the output control parameters were sampled from their distributions. The DNN model took four TBMAP parameters as input and outputted four values according to the means and variances of two control parameters. For some general hyperparameters of the PPO algorithm they were set as the same value as the original paper on the PPO algorithm, for example, the PPO clip coefficient and learning rate. For some non-general hyperparameters, they were optimized using grid search, for example, network structural parameters and total timesteps. For the output mean values, the DNN model adopted three hidden layers with Relu as the activation function and the symmetrical layer unit numbers, which were 64, 64, and 64, respectively. The model a the learnable values without connection with the input as the output variances. Under the PPO framework introduced in Section 3.2, the value function for the environment state was established by the DNN model, which took four TBMAP parameters as input and outputted one value corresponding to the environment state value. The value function adopted three hidden layers with Relu as the activation function and the symmetrical layer unit numbers, which were 64, 64, and 64, respectively. Under the PPO framework, the attitude control policy was gradually optimized during the alternating process between the interaction of policy and environment and policy training. The total number of interaction steps was set to one million. Eight environments were used to interact with policy in parallel for obtaining interaction data. The policy was trained using the interaction data every 2000 interaction steps, in which eight epoch updates were performed with the batch size of 1600. The GAE technology was used to estimate the state advantage to reduce its estimation bias, and batch normalization was performed on the state advantage to ensure a more stable learning process. The ADAM method was used to gradient update the policy and state value model, and the truncating of the update gradient was performed to prevent gradient explosion. The specific hyperparameter values are shown in Table 5. \begin{table}
\begin{tabular}{l l l l l} \hline \hline  & **TDA** & **TFA** & **HDTH** & **VDTH** \\ \hline GEC 2 & 0.913 & 0.932 & 0.928 & 0.943 \\ GEC 3 & 0.962 & 0.941 & 0.873 & 0.933 \\ GEC 4 & 0.921 & 0.865 & 0.928 & 0.923 \\ GEC 5 & 0.959 & 0.928 & 0.889 & 0.958 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: R2s of different representation parameters of different TBMAP models. \begin{table}
\begin{tabular}{l l} \hline \hline
**Hyperparameter Name** & **Hyperparameter Value** \\ \hline Total timesteps & 1,000,000 \\ Learning rate & \(3\times 10^{-4}\) \\ Parallel environment number & 8 \\ Policy updates frequency & 2000 \\ Lambda for GAE (\(\lambda\)) & 0.95 \\ Discount factor (\(\gamma\)) & 0.95 \\ Policy updates epochs & 8 \\ PPO clip coefficient & 0.2 \\ Coefficient of the value function loss & 0.5 \\ The maximum norm for the gradient clipping & 0.5 \\ Policy updates batch-size & 1600 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Hyperparameter values for training. Using the training strategy and hyperparameters introduced above, four attitude control policies were trained for one million interaction steps in their environments.