# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 3 Methodology  
### Objective and General Idea  
The problems of the unknown geological environment, the hysteresis of TBM position response, and the unsolved overall optimization of the tunneling axis should be solved to avoid the snakelike motion and exceeding the deviation limit of manual attitude control. The TBM tunneling processes and geological environments of different engineering are complex and different, which are difficult to be analyzed and established to be a unified model. So, the data produced from specific engineering should be used to build the corresponding model to guide its excavation by deep learning method, which can simplify the modeling. For a better tunneling axis, the OACP should be established by learning from the generated excavation data and optimized for tunneling axis quality. The trained OACP should take current TBMAP and GEC information and output the optimal control parameters, which can achieve the real-time optimal attitude control and minimize the overall deviation between the tunneling axis and the designed tunnel axis. When collecting more new excavation data, the OACP model can adapt to the new geological environment by learning from the new data to provide the corresponding optimal attitude control.  
Figure 3: Statistical distributions of the excavation parameters: (**a**) statistical distribution of parameter TDA, (**b**) statistical distribution of parameter TFA, (**c**) statistical distribution of parameter HDTH, (**d**) statistical distribution of parameter VDTH, (**e**) statistical distribution of parameter DDSB, (**f**) statistical distribution of parameter DLTC.  
For these targets, this paper proposed the modeling framework for OACP based on the data introduced in Section 2, shown in Figure 4. The data generated during tunneling, including excavation parameters data, GEC data, TBMAP data, and attitude control parameters data, were collected and preprocessed. In order to obtain real-time geological information, the GEC predictive model was built using a deep neural network (DNN) and trained by the corresponding dataset of excavation parameters and GEC data. To solve the problems of the hysteresis of TBM position response and the unsolved overall optimization of the tunneling axis, the paper applied the reinforcement learning method to optimize the attitude control policy built by DNN. Due to security and feasibility, the real excavation interaction environment could not be used for optimizing OACP, which is needed in reinforcement learning. The TBMAP prediction models were established for four GECs based on the corresponding dataset of TBMAP parameters and attitude control parameters, which can be used as the simulated TBMAP interaction environment for optimizing OACP. With four established simulated interaction environments, four OACPs were optimized during the alternating process between the interaction of the policy and the environment and policy training using the PPO algorithm. When applied in actual engineering, the trained GEC prediction model was used to recognize the current GEC in real time, and the corresponding OACE was selected based on the current GEC to provide the real-time optimal attitude control parameters.