# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 4 Case Study
### TBMAP Predictive Model  
Due to the variation patterns of TBMAP being different in different GECs, the TBMAP predictive model needs to be established for four GECs using the same framework. For each GEC, the TBMAP prediction model was established using the full-connected DNN structure and trained by the matched data from this GEC environment introduced in Section 2.2, which is composed of the current TBMAP parameters, attitude control parameters, and the next moment TBMAP parameters. Because of four representation parameters of TBMAP, four independent DNN models with the same structure were established to predict them. Each DNN model took six parameters as input, including two attitude control parameters and four current TBMAP parameters, and outputted one value corresponding to one of the next moment TBMAP parameters. The DNN model adopted four hidden layers with Relu as an activation function, and the unit number of each layer adopted a symmetrical structure, which were 36, 72, 36, and 12, respectively. The Batchnorm layer was added before the Relu activation function of each layer to prevent gradient explosion and disappearance. The activation function of the output layer adopted the Tanh function to limit the output between \([-1,1]\), corresponding to the normalized TBMAP parameter. Four TBMAP prediction models used the same data preprocessing method and loss function. The corresponding training data for each TBMAP predictive model were selected from the total of the 151,447 sets of preprocessed data introduced in Section 2.2 and divided into training sets and test sets according to the ratio of 4:1. In order to eliminate the difference between different inputs; each input was normalized by the Min-Max normalization method to limit the input between \([-1,1]\), which can maintain the symmetry of original data. The output TBMAP parameters were also normalized by the Min-Max normalization to limit between \([-1,1]\) for the TBMAP predictive models with normalized parameters as output. For this supervised learning, the MSE introduced in Section 3.3 was selected as the loss function. The training hyperparameters of the TBMAP prediction models of different GECs were the same, while the training hyperparameters of the different representation parameter predictive models of the same GEC were different. The training hyperparameters of four representation parameter predictive models of GEC 2 are shown in Table 3.  
After training, four TBMAP predictive models were obtained for different GEC environments. The models had been well-fitted with the training data, and their MSE of the test set had also significantly decreased. R2s of different representation parameters of different TBMAP models were computed, shown in Table 4. From the table, it can be seen that all R2s exceed 0.85, which indicates that the predictive models all have obtained sufficient fitting. The predictive performance of the predictive models of four GECs on the test set are shown in Figures 6-9. From the figure, it can be seen that the prediction of various representation parameters of each GEC all had a good generalization effect, which indicates \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**GEC** & **Precision** & **Recall** & **F1-Score** & **Support** \\ \hline
2 & 0.9571 & 0.9691 & 0.9604 & 13,234 \\
3 & 0.9294 & 0.9254 & 0.9274 & 10,125 \\
4 & 0.8828 & 0.8366 & 0.8591 & 1585 \\
5 & 0.9769 & 0.9562 & 0.9664 & 5346 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Different statistical indexes of various GECs. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & **Epochs** & **Learning Rate** & **Batch Size** & \
\begin{tabular}{l} **Verification Set** \\ **Proportion** \\ \end{tabular} &
\begin{tabular}{l} **Optimization** \\ **Algorithm** \\ \end{tabular} \\ \hline TDA & 140 & 0.004 & 1000 & 0.1 & SGD \\ TFA & 140 & 0.004 & 2000 & 0.1 & SGD \\ HDTH & 200 & 0.004 & 2000 & 0.1 & SGD \\ VDTH & 200 & 0.004 & 1000 & 0.1 & SGD \\ \hline \hline \end{tabular}
\end{table}