# Research on Wellbore Trajectory Optimization and Drilling Control Based on the TD3 Algorithm
## 4 Experimental Design and Result Analysis
### Three-Stage Wellbore Trajectory Design Based on Reinforcement Learning
TD3 convergees in 80 k steps (40 k steps faster than DDPG), with a final trajectory error of 0.78\({}^{\circ}\) (vs. DDPG's 1.32\({}^{\circ}\)) and a collision rate of 1.2% (4.5% for DDPG). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Serial Number** & **North** & **East** & **Height** & **Serial Number** & **North** & **East** & **Height** \\ \hline
1 & 0 & 0 & 0 & 14 & 531.86 & 443.22 & 800 \\
2 & 0 & 0 & 215 & 15 & 600.99 & 500.83 & 800 \\
3 & 1.15 & 0.96 & 244.96 & 16 & 670.12 & 558.44 & 800 \\
4 & 11.47 & 9.56 & 333.87 & 17 & 739.26 & 616.05 & 800 \\
5 & 31.89 & 26.57 & 419.77 & 18 & 808.39 & 673.66 & 800 \\
6 & 61.93 & 51.61 & 500.73 & 19 & 877.52 & 731.27 & 800 \\
7 & 100.94 & 84.11 & 574.94 & 20 & 946.66 & 788.88 & 800 \\
8 & 148.02 & 123.35 & 640.73 & 21 & 1015.79 & 846.49 & 800 \\
9 & 202.13 & 168.44 & 696.62 & 22 & 1084.92 & 904.1 & 800 \\
10 & 262.05 & 218.37 & 741.36 & 23 & 1154.05 & 961.71 & 800 \\
11 & 326.43 & 272.02 & 773.94 & 24 & 1181.71 & 984.76 & 800 \\
12 & 393.82 & 328.19 & 793.64 & 25 & 1188.62 & 990.52 & 800 \\
13 & 462.73 & 385.61 & 800 & 26 & 1200 & 1000 & 800 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Three-stage optimal wellbore trajectory data table. Five independent training runs (with 95% confidence intervals) show that the total training loss starts high and converges within 20,000 steps, with a final policy standard deviation of \(\pm\)0.15\({}^{\circ}\) for the well deviation and \(\pm\)0.2\({}^{\circ}\) for the azimuth. This statistical validation confirms the model's stable convergence and repeatability. As shown in Figure 4(a), the total training loss is at a high level in the early stage of training and then decreases rapidly and tends to converge within about 20,000 steps, indicating that the model quickly completes the basic strategy learning in the early stage and enters the stable optimization stage. This shows that the model has good training efficiency. As shown in Figure 4(b), the value function loss shows a smooth exponential downward trend from the higher initial value (about 230) and finally converges to close to 0 at about 40,000 steps. This shows that the model has significant learning ability in estimating the state value, and the estimation error is effectively corrected during the training process. As shown in Figure 4(c), the strategy gradient loss shows continuous fluctuations throughout the training process, and the values are mainly distributed in the range of \(-\)0.008 to 0.001. Although the fluctuation is large, there is no divergence, indicating that the strategy update process has certain instability but is generally controllable. This feature is consistent with the expected behavior of the policy gradient method in exploratory strategy adjustment. As shown in Figure 4(d), the entropy loss (negative entropy) gradually increases with the training progress, indicating that the strategy entropy continues to decline, and the strategy behavior gradually tends to be deterministic from the initial high randomness. This trend reflects the natural transition process from 'exploration' to 'utilization', which is an important manifestation of the gradual convergence of the strategy to the optimal behavior strategy. There is a trend of Evaluation Mean Reward with the number of training steps. As shown in the Figure 5, the model quickly achieves a significant improvement in strategy  
Figure 6: Three-stage reward evaluation of the highest wellbore trajectory. performance at the initial stage of training (about 5000 steps), and the average reward increases from about \(-1.80\) to about \(-1.76\), showing that reinforcement learning can quickly learn effective behavior strategies in the short term. In the subsequent training process, the average reward fluctuates slightly around \(-1.76\) and finally tends to be stable, indicating that the strategy has basically converged, and the subsequent training has a limited impact on the further improvement of performance.