# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 1 Introduction  
The tunnel is a key component of a series of major line projects, such as water conservancy, transportation, and energy transportation. Tunnels in different geological environments require different construction methods. The mountain tunnels generally adopt the mining method and the TBM method. The shallow buried and soft tunnels generally adopt the open excavation method, cover excavation method, shield tunneling method, and new Austrian tunneling method. The underwater tunnels generally adopt the sink-and-bury method and the shield tunneling method. TBMs [1, 2] have become the optimal tool for tunnel engineering due to their high efficiency, automation, economy, and environmental friendliness compared with conventional blasting tunneling. It is a large-scale comprehensive equipment integrating machinery, electronics, hydraulic pressure, and control. During excavation, TBM must pass a wide variety of geological environments that have significantly different properties [3, 4]. Based on the geological survey obtained by sampling, the engineers will design the optimal tunnel axis before construction, which can meet the requirements of the current demand and avoid unfavorable geology as much as possible. TBM needs to precisely tunnel according to the designed tunnel axis, which isthat the tunneling axis should be within the required deviation range from the designed tunnel axis and sufficiently smooth without snakelike motion. The efficient TBM attitude control meeting these requirements under frequently changeable and unforeseen geological environments is a complex and difficult problem. At present, the TBM attitude is mostly manually controlled during tunneling. The laser-based guidance system [5; 6] installed on TBM can measure and display TBM attitude and position deviation relative to the designed tunnel axis in real time. The TBM operators set the appropriate control parameters for TBM attitude based on the current attitude and position deviation and the rough geological information from a geological survey using their experience [7; 8]. Because of the complex and ever-changing geological environment, the operators cannot obtain enough current geological information according to the geological survey and personal perception. The current attitude control will affect the next attitude and position state and further affects the next attitude control; this correlation will lead to the hysteresis of position response of attitude control, which is difficult to predict by the operators. The optimization of the overall tunneling axis is difficult to solve by operators because of the huge search space composed of the long action sequence and continuous action value and the limited exploration. Therefore, in order to automatically and timely provide the optimal control for TBM attitude during tunneling, it is necessary to timely recognize the current geological environment, take hysteresis of TBM position response into consideration, and conduct sufficient exploration for the optimal overall tunneling axis. Geological information can provide important support for the optimal control of TBM attitude. However, due to the deep burying, complex and ever-changing geology, it is difficult to obtain sufficient geological information before excavation. Therefore, many scholars have studied the prediction methods of geological information. [PERSON] et al. [9] obtained a three-dimensional seismic ahead-prospecting method by optimizing the filtering method and imaging algorithm, which can accurately judge and locate the fault ahead of the tunnel face. [PERSON] et al. [10] proposed a method of electrical resistivity tomography survey, which can predict the abnormal strata ahead of the tunnel face. [PERSON] et al. [11] comprehensively applied the induced polarization and the resistance coefficient methods to analyze the induced polarization and resistivity measured in the tunnel face to predict the advanced geological state.