# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 3 Methodology
### Base Model  
(1) GEC Predictive Model  
The rough sampling analysis of the geological environment before excavation cannot provide effective, accurate geological environment information for TBM tunneling. In order to realize the real-time optimal attitude control according to the current geological environment, it is necessary to predict the current geological environment in real time. Considering that the excavation parameters can be obtained in real-time during tunneling, the GEC prediction model can be built to predict the current GEC according to the current excavation parameters. The predictive model is established using a fully connected DNN, which has strong fitting ability. Twenty excavation parameters introduced in Section 2.2 are taken as input of the model, and the probabilities of four GECs are outputs. The training dataset matched by the 20 excavation parameters and GEC data introduced in Section 2.2, are used to train the GEC prediction model by supervised learning, and the cross entropy is used as the loss function for this multi-classification problem, as shown in Formula (8).  
\[\mathcal{L}(\theta_{L})=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{5}\left(y_{j}^{i }log\left(\hat{y}_{j}(x_{i})\right)\right) \tag{8}\]  
where \(\theta_{L}\) represents the parameters of the GEC model, \(x_{i}\) represents the 20 excavation parameters, \(y_{j}^{i}\) represents the real probability that the input \(x_{i}\) is GEC \(j\), \(\hat{y}_{j}(x_{i})\) represents the predictive probability that the input \(x_{i}\) is GEC \(j\).  
(2) TBMAP Predictive Model  
Considering the safety and feasibility, it is necessary to construct a simulated environment to replace the real excavation environment for training attitude control policy. This environment model should predict the next moment TBMAP based on the current TBMAP and the attitude control parameters. The TBMAP can be represented by the measuring parameters TDA, TFA, HDTH, and VDTH, which are introduced in Section 2.1. During tunneling, the attitude of the TBM left and right deflection angle is controlled by adjusting the displacement of the left and right support boot cylinders. The attitude of the TBM up and down deflection angle is controlled by adjusting the displacement of the torque cylinders. Therefore, the displacement difference between the left and right support boot cylinders (DDSB) and the displacement of the left torque cylinder (DLTC) are used as TBMattitude control parameters. With six parameters combined by TBMAP parameters and attitude control parameters as input, the TBMAP predictive model is established by fully connected DNN and output the next moment TBMAP parameters. Based on the matched dataset introduced in Section 2.2, composed of TBM attitude control parameters, current TBMAP parameters, and the next moment TBMAP parameters, the TBMAP predictive model is trained by supervised learning, with the mean square error as the loss function, as shown in Formula (9).  
\[\begin{split}\mathcal{L}(\theta_{TD},\ \theta_{TF},\ \theta_{HD},\ \theta_{VD})=\frac{1}{N}\sum_{i=1}^{N}\Bigl{[}\bigl{(}f_{TD}(x_{i})-y_{i}^{TD }\bigr{)}^{2}+\bigl{(}f_{TF}(x_{i})-y_{i}^{TF}\bigr{)}^{2}\\ +\bigl{(}f_{HD}(x_{i})-y_{i}^{HD}\bigr{)}^{2}+\bigl{(}f_{VD}(x_{i} )-y_{i}^{VD}\bigr{)}^{2}\Bigr{]}\end{split} \tag{9}\]  
where \(\theta_{TD},\ \theta_{TF},\ \theta_{HD},\ \theta_{VD}\) represent, respectively, the parameters of the measuring parameters TDA, TFA, HDTH and VDTH predictive model, \(x_{i}\) represents the input parameters of the model, \(f_{TD}(x_{i})\), \(f_{TF}(x_{i})\), \(f_{HD}(x_{i})\) and \(f_{VD}(x_{i})\) represent, respectively, the predictive value of these measuring parameters, and \(y_{i}^{P},y_{i}^{P},y_{i}^{P}\) and \(y_{i}^{A}\) represent, respectively, the real value of these measuring parameters.