# Research on Wellbore Trajectory Optimization and Drilling Control Based on the TD3 Algorithm
## 4 Experimental Design and Result Analysis
### Three-Stage Wellbore Trajectory Design Based on Reinforcement Learning  
In order to systematically evaluate the spatial structure and controllability of the three-stage wellbore trajectory generated based on the reinforcement learning strategy, this paper comprehensively analyzes the distribution of the trajectory in three-dimensional space and its performance on the three main coordinate projection surfaces. In the process of trajectory generation, the position of the build-up point is randomly selected between 200 m and 400 m deep to enhance the generalization ability of the strategy at different build-up points. Figure 3(a) shows that all the trajectory tracks are drilled vertically from the ground during the training process. After a continuous deflection process, they finally smoothly transit to the horizontal section and accurately reach the preset target position, reflecting the typical vertical-deflection-horizontal three-stage structure. Figure 3(b) shows the North-East horizontal projection of all the trajectories. Each trajectory shows a uniform and orderly radiation distribution as a whole, and all tend to the direction of the target, indicating that the training strategy is stable and consistent in horizontal direction control. Figure 3(b),c show the vertical projections of all trajectories in the North-Height and East-Height directions, which clearly show the process of different trajectories gradually entering the deflecting section in the depth range of 200 m to 400 m.  
Figure 5: Key loss function diagram of the three-stage wellbore trajectory. The trajectories achieve smooth convergence near the target depth, indicating that the model has good bending control ability in vertical trajectory design. From the overall performance, the model can not only adapt to the random variation in the starting point in a certain range but also generate a physically reasonable and continuous curvature wellbore trajectory in all directions. The trajectory finally converges stably to the vicinity of the target, indicating strong generalization ability: a 1.2\({}^{\circ}\) average error in S-shaped wells (vs. DDPG's 1.8\({}^{\circ}\)), a 92% success rate in U-shaped well scenarios (DDPG: 78%), and a collision rate < 3% under random obstacle layouts with a 50% density increase. Table 3 shows the trajectory with the highest reward among all the trajectories extracted (the optimal trajectory data is no longer displayed in the following; only visual graphics are drawn). Figure 6 shows the three-dimensional map of the trajectory and the projection map of each coordinate plane. Specifically, after the trajectory is drilled vertically from the ground, it naturally enters the deflecting section at 215 m and finally extends steadily to the near-horizontal section at the target depth. On the horizontal projection plane, the trajectory is an approximate straight line, and the position control accuracy is high, which is conducive to the accurate docking of the target horizon and reduces the risk of trajectory offset. In the vertical projection, the curvature of the build-up section is continuous and stable, which meets the requirements of dogleg severity control in conventional wellbore trajectory planning and helps to reduce drilling tool wear and wellbore friction. With a training time of 2.3 s per episode on an RTX 3090 GPU and a convergence total of 50 h (2000 episodes), the model ensures real-time performance: the inference latency is <10 ms, meeting the 1 Hz control frequency requirement in field operations. The trajectory not only obtains the optimal evaluation in numerical value but also performs well in structural continuity, control accuracy, and engineering feasibility, which reflects the technical advantages of the reinforcement learning strategy in automatic wellbore trajectory planning. Compared with the DDPG baseline (identical state/action spaces and shared reward function structure, except for removing the dual Q network and delayed update mechanisms), TD3 demonstrates superior performance.