# Borehole Trajectory Optimization Design Based on the SAC Algorithm with a Self-Attention Mechanism
## 3 Reinforcement Learning Environment Modeling and Design
### Reward Function Design
Obstacles are encoded as spatial constraints in the state space, and the reward function imposes penalties when the predicted trajectory approaches these constraints. During training, the reinforcement learning agent learns to associate certain state-action pairs with penalties or rewards based on proximity to obstacles, which drives the emergence of avoidance behavior. The result is a trajectory policy that proactively bypasses obstacle zones while maintaining overall path feasibility and smoothness. (5) Strategy efficiency reward  
To encourage reaching the target point as soon as possible, introduce a \"survival penalty\" for each step of the operation:  
\[r_{step}=-\delta \tag{19}\]  
where \(\delta\) is a fixed deduction item for each step, preventing the agent from adopting an inaction strategy such as \"standing still\". (6) Stratum hardness penalty term  
The hardness of the formation represents the anti-breaking ability of the rock. The higher the hardness, the greater the difficulty of drilling, the more serious the wear of the drill bit, the greater the decrease of control accuracy, and the higher the risk of sticking and deviation instability. According to the acoustic time difference (DT) or the equivalent hardness \(H_{t}\) calculated based on the geological regression model, the following penalty term can be constructed:  
\[r_{hard}=-\eta\cdot H_{t} \tag{20}\]  
where \(H_{t}\) is the hardness value of the formation at the current drilling location (which can be obtained from the joint inversion of DT, DEN, CNL, etc.) and \(\eta\) is a conditioning factor to control the intensity of the penalty for high hardness zones. (7) Drillability enhancements  
Drillability is a comprehensive measure of how easy a formation is to drill and is usually related to rock strength, degree of fragmentation, and well path stability. To encourage the drill bit to prioritize path areas with better drillability, the following positive incentives can be designed:  
\[r_{drill}=+\lambda\cdot D_{t} \tag{21}\]  
where \(D_{t}\) is the formation drillability rating, which can be obtained from logging data, geologic model prediction, or empirical index inversion, and \(\lambda\) is the drillability incentive coefficient, which is an adjustment parameter accompanying the hardness penalty term. The values of \(\lambda\) and \(\eta\) were determined through pretraining experiments using random geological models to evaluate their sensitivity to drilling difficulty and geological heterogeneity. A grid search method was employed, where different combinations of these coefficients were tested across training episodes. Values that led to stable convergence, high obstacle avoidance success rate, and minimal curvature fluctuation were selected. The final settings reflect a trade-off between safety (avoiding hard rock and unstable zones) and trajectory efficiency. The reward function system is capable of realizing the unification of target accuracy, safe obstacle avoidance, geological adaptation and trajectory smoothing, directly quantifying the engineering control objectives (e.g., distance, direction, obstacle avoidance), and flexibly introducing additional indexes according to the specific mission scenarios (e.g., deep-water drilling, traversing of fault zones) with scalability.