# Borehole Trajectory Optimization Design Based on the SAC Algorithm with a Self-Attention Mechanism
## 3 Reinforcement Learning Environment Modeling and Design
### Reward Function Design  
The reward function is an important part of the trajectory design of the drilling hole. The reasonable design of the reward function will determine the learning effect of the agent, the accuracy of the trajectory tracking, and the smoothness of the adjustment process while punishing dangerous operations or excessive curvature. Therefore, this paper constructs a multi-objective weighted combination reward function system based on geological modeling, obstacle distribution, and trajectory characteristics. The form is as follows: The designed reward system integrates seven core components, each addressing a specific aspect of drilling control. The goal-approaching reward drives the trajectory toward the target position. The smoothness reward penalizes abrupt directional changes to ensure continuity and avoid mechanical stress. Obstacle avoidance introduces spatial safety constraints to prevent collisions. Geological adaptability encourages paths through soft, low-risk formations. The step penalty discourages unnecessarily long trajectories. Formation hardness is negatively weighted to avoid difficult-to-drill zones, while the drillability incentive encourages efficient path planning. Together, these terms form a balanced and adaptable reward framework that enables the reinforcement learning agent to plan trajectories that are not only accurate but also safe, efficient, and geologically feasible:  
\[R_{t}=w_{1}r_{target}+w_{2}r_{smooth}+w_{3}r_{safe}+w_{4}r_{go}+w_{5}r_{step }+w_{6}r_{hard}+w_{7r_{full}} \tag{14}\]where \(w_{6},w_{7}\) is the weight coefficient of hardness and drillability, which is usually set as a positive number less than 1, and is set according to engineering experience or strategy optimization. 1. Goal approaching reward  
Used to encourage the trajectory to progressively approach a preset target point, usually defined as a negative value for the distance to the target point:  
\[r_{\text{target}}=-\big{\|}P_{t}-P_{\text{target}}\big{\|}_{2} \tag{15}\]  
where \(P_{t}\) is the current drill position and \(P_{\text{target}}\) is the target position. 2. Trajectory Smoothness Bonus  
Used to constrain the local continuity of the trajectory and penalize movements with violent steering or sudden changes in curvature:  
\[r_{\text{smooth}}=\frac{1}{e^{\|\theta_{t}-\theta_{t-1}\|_{2}}}+\frac{1}{e^{ \|\phi_{t}-\phi_{t-1}\|_{2}}} \tag{16}\]  
It encourages a smooth range of motion while achieving goals, avoiding \"sharp turns\" or unrealistic trajectory adjustments. 3. Obstacle avoidance reward  
Used to punish the behavior of approaching or entering the dangerous area of the obstacle, reflecting the safety of the trajectory:  
\[r_{safe}=\left\{ \begin{array}{c}-C_{obs},d<d_{safe}\\ 0,else\end{array} \right. \tag{17}\]  
where \(C_{obs}\) is the penalty coefficient and \(d_{safe}\) is the safety boundary to ensure that the trajectory will not stray into the obstacle area. 4. Geological response reward  
Evaluates formation risks through real-time LWD data, encourages drill bits to pass through \"soft formation, low pressure zone\", and avoids entering \"high hardness, high pressure fault zone\":  
\[r_{geo}=-\gamma_{1}\cdot GR_{t}-\gamma_{2}\cdot DEN_{t}+\gamma_{3}\cdot(1/RT_{ t}) \tag{18}\]  
This equation indirectly reflects the drillability of the formation. When RT is high, it represents the tight layer or oil and gas layer, and when it is low, it may be mudstone or fault zone. This design enhances the adaptability of trajectory design to geological environment. To achieve obstacle avoidance, the proposed method integrates multiple components: (1) a continuous action space that allows fine-tuned directional adjustments in azimuth and inclination, (2) a safety-aware reward function that penalizes proximity to risk zones, and (3) a self-attention mechanism that enables the model to recognize spatial-temporal patterns in geological context.