# Research on Wellbore Trajectory Optimization and Drilling Control Based on the TD3 Algorithm
## 3 Reinforcement Learning Environment Modeling and Design
### Reward Function Design  
As a quantitative index of reinforcement learning training objectives, the design of the reward function needs to take into account trajectory accuracy, safety, and smoothness. In order to make the drill bit run smoothly along the target trajectory, the design of the reward function must be able to accurately express the system expectation and enable the agent to respond and adjust in time when there is a deviation so that the trajectory is as close as possible to the target trajectory. The design of the reward function takes into account the distance deviation between the drill bit and the target point and increases the penalty term. This restricts the situation of  
Figure 3: Obstacle setting diagram. greatly changing the action angle, ensuring that the drill bit action angle is continuously and stably adjusted, so that the drill bit can move steadily towards the target point. Combined with the actual engineering requirements of wellbore trajectory optimization, at each moment \(t\), the reinforcement learning agent receives the current state \(s_{t}\), performs the action \(a_{t}\), and obtains the following combination of rewards:  
\[r_{t}=\eta(\omega_{1}r_{\text{target}}+\omega_{2}r_{\text{distance}}+\omega_{3 }r_{\text{collision}}+\omega_{4}r_{\text{smooth}}+\omega_{5}r_{\text{progress}}) \tag{28}\]  
It is assumed that the current bit position \(X_{t}=[N_{t},E_{t},H_{t}]\), target point \(X_{\text{target}}=[N^{*},E^{*},H^{*}]\), and action \(a_{t}=[\Delta\theta_{t},\Delta\phi_{t}]\), \(\eta>0\), are reward scaling coefficients. \(\omega\) is the reward weight (hyperparameters: \(\omega_{1}=0.5\) for trajectory accuracy, accounting for 50% of field engineering costs; \(\omega_{2}=0.3\) for collision avoidance, prioritizing safety constraints; \(\omega_{3}=0.15\) for dogleg severity control, following the industry standard of <3\({}^{\circ}\)/30 m in Paragraphs 1-121; \(\omega_{4}=0.05\) for progress incentive to prevent conservative strategies; and \(\omega_{5}=0.05\) for energy consumption penalty as a secondary objective). The hyperparameters, other reward meanings, and calculation methods are as follows. (1) Hit accuracy reward (\(r_{\text{target}}\)). If the spatial distance GGGG between the current position of the drill bit and the target is less than the set threshold, a segmented reward is given:  
\[r_{\text{target}}= \begin{cases}1000,&d_{t}<3\\ 60,&3\leq d_{t}<5\\ 10,&5\leq d_{t}<10\\ 0,&\text{otherwise}\end{cases}  \tag{29}\]  
where \(d_{t}=\big{|}|X_{t}-X_{\text{target}}|\big{|}\). (2) Distance-guided reward (\(r_{\text{distance}}\)). The agent is guided to gradually approach the target, and the form of the exponential attenuation reward is adopted in the horizontal and vertical directions:  
\[r_{\text{distance}}=\text{e}^{-\frac{d_{xy}}{200}}+\text{e}^{-\frac{d_{z}}{100}} \tag{30}\]  
where  
\[d_{xy}=\big{|}\big{|}[N_{t},E_{t}]-[N^{*},E^{*}]\big{|}\big{|}_{2};d_{z}=|H_{t }-H^{*}| \tag{31}\]  
(3) Obstacle collision penalty (\(r_{collision}\)). If the distance between \(X_{t}\) and any obstacle body at the current position is less than the safe radius \(R_{safe}\), the penalty is immediately imposed, and the round is terminated. \[r_{\text{collision}}= \begin{cases}-10,&\min_{j}\big{\|}X_{t}-X_{\text{obs}j} \big{\|}<R_{\text{safe}}\\ 0,&\text{otherwise}\end{cases}  \tag{32}\]  
(4) Action smoothness award (\(r_{smooth}\)). Punishment is introduced into the amount of action change to prevent drastic changes, and control oscillations. \[r_{\text{smooth}}=\frac{1}{e^{\|a_{t}-a_{t-1}\|_{2}}} \tag{33}\]  
(5) Progress award (\(r_{\text{progress}}\)). When the agent is closer to the target than the previous state, a small positive reward is given.