# The Real-Time Optimal Attitude Control of Tunnel Boring Machine Based on Reinforcement Learning
## 5 Conclusions  
In the current tunnel engineering constructed by TBM, the actual tunneling axis of manual control is often the snakelike motion around the design axis, even exceeding the deviation limit. This paper summarized three reasons for these problems: the unknown geological environment, the hysteresis of TBM position response, and the unsolved overall optimization of the tunneling axis. For these reasons, this paper proposed a real-time optimal attitude control framework based on the data obtained from the actual engineering, which contains the GEC predictive model, TBMAP predictive model, and OACP model. Based on these reasons, the control framework can effectively solve the problems of manual control. To verify the effectiveness of the proposed control framework, the Xinjiang Yiner Water Supply Phase II Project was adopted as a case study. This study has three major contributions to research and practice as follows:  
1. The paper proposes the GEC predictive model to obtain the real-time GEC for attitude control policy during tunneling. The GEC predictive model established using the DNN model was trained using the corresponding data of excavation parameters and GEC from the actual construction engineering. The accuracy of the trained GEC predictive model could reach 94%, and the model took excavation parameters as input, which indicates that the model can recognize the real-time GEC information from the excavation parameters as the input of the attitude control model. 2. The paper established the TBMAP predictive model for four GECs to be the interactive environment for training the attitude control policies. The TBMAP predictive model established by DNN was trained using the TBMAP parameters and attitude control parameters data of the corresponding GEC from the actual engineering. After training, R2s of different representing parameters prediction of different TBMAP models were computed, which all exceeded 0.85. It can be seen that the TBMAP predictive models have sufficient predictive accuracy and calculation efficiency, which can be used as the interactive environment to train the attitude control policy under the reinforcement learning framework. Figure 11: Episode rewards comparison of OACP and manual control: (**a**) episode rewards comparison in GEC 2, (**b**) episode rewards comparison in GEC 3, (**c**) episode rewards comparison in GEC 4, and (**d**) episode rewards comparison in GEC 5. 3. For the hysteresis of TBM position response and the overall optimization of the tunneling axis, the paper proposes the optimization framework of attitude control policy based on reinforcement learning. The attitude control policy for each GEC was established by the DNN model and was gradually optimized during the alternating process between the interaction of the policy and the established TBMAP predictive environment and policy training using the PPO algorithm, which can optimal the policy based on the episode deviation. To verify its effectiveness, the obtained OACP was compared with manual control based on practical engineering data. The results revealed that OACP can significantly reduce the accumulated deviation of the tunneling axis from the design tunnel axis by over 80% compared with the manual control. OACP combined with the GEC predictive model can easily provide real-time decision support for attitude control in actual engineering. Conceptualization, G.J. ; Methodology, G.J. and J.H. ; Software, J.H. and B.Y. ; Formal analysis, G.J. and J.H. ; Data curation, G.J. and J.H. ; Writing-original draft, B.Y. and Z.W. ; Writing-review & editing, B.Y. and Z.W. ; Visualization, B.Y. and Z.W. ; Supervision, Z.W. All authors have read and agreed to the published version of the manuscript. This work was supported in part by the National Natural Science Foundation of China (No. 52275236). Not applicable.