# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 1 Introduction
Although the above methods can obtain advanced geological information by analyzing the various feedback signals, they need the shutdown state of TBM when measuring, which cannot achieve convenience and real-time prediction. To this end, many scholars have studied the real-time prediction methods of geological information during tunneling, which can provide support for efficient excavation. [PERSON] et al. [12; 13] applied the forward propagation neural network and the improved support vector machine to build the rock mass parameters prediction model, which can predict the uniaxial compressive strength, brittleness index, and other rock mass parameters with the excavation parameters as the input. [PERSON] et al. [14] applied K++ means clustering algorithm to find the hidden classification pattern in the data and applied the support vector machine method to build the prediction model of the geological environment based on excavation parameters. [PERSON] et al. [15] applied an artificial neural network to build the prediction model of the geological environment category (GEC), which can predict the current GEC according to the TBM excavation parameters, which can achieve 96% prediction accuracy. In order to achieve the optimal control of the TBM attitude, a feasible solution is to establish the TBMAP prediction model first and adjust and control the TBM attitude based on it. Therefore, many scholars have conducted research on TBMAP prediction. [PERSON] et al. [16] established eight prediction models of shield machine attitude based on the data from five earth pressure balance shield machines and obtained two best algorithms, the LSTM and GRU with EVS \(>\) 0.9 and RMSE \(<\) 1.5. [PERSON] et al. [17] proposed the deep learning model with a graph convolutional network and long short-term memory, which can predict the vertical and horizontal deviations at the articulation and tail of TBM with high accuracy. [PERSON] et al. [18] presented a prediction framework for TBMAP in shield tunneling by applying a hybrid deep learning model, which contains a wavelet transform noise filter, convolutional neural network feature extractor, and long short-term memory. [PERSON] et al. [19] proposed an intelligent method based on a Bayesian-light gradient boosting machine model with 29 excavation parameters and 6 parameters about the shield attitude, which can predict the shield attitude and support attitude control by adjusting control parameters and conducting iterative prediction. In order to ensure that TBM can tunnel according to the designed tunnel axis within a certain deviation range, it is necessary to control TBM to the set attitude accurately. So, many scholars have conducted research on how to control TBM to the set attitude. [PERSON] et al. [20] proposed a cascade control system combining outer-loop trajectory tracking and inner-loop pressure control to enable unmanned automatic tunneling of TBM, in which the inner loop of the system is a cooperative control system of different hydraulic units while the outer loop is a fuzzy attitude correction system. [PERSON] et al. [21] constructed the tunneling axis deviation prediction model based on the improved XGboost and the multi-loop model of shield tunneling axis deviation correction based on the fusion of the geometric model and association rule, which can realize the accurate deviation prediction and deviation correction. [PERSON] et al. [22] proposed an integrated control system that consists of one trajectory planning controller for both cylinders and an individual cylinder controller for each hydraulic cylinder, and a cascade control strategy which comprises a feedforward controller of fixed-value compensation and feedback controller of Variable-gain PID, which can achieve the automatic control of the thrust trajectory.