# Research on Wellbore Trajectory Optimization and Drilling Control Based on the TD3 Algorithm
## 3 Reinforcement Learning Environment Modeling and Design
### State Space Design  
The state space is an important component of the reinforcement learning agent for sensing the environment, and its structure should cover the key information that reflects the attitude of the combined eye and the current position of the drill bit. In wellbore trajectory control, it is very important to construct a reasonable state space, which determines whether the control system can accurately guide the drilling tool to run according to the predetermined trajectory. In order to achieve this goal, it is necessary to select an appropriate parameterization method that accurately expresses the dynamic characteristics of the wellbore trajectory and provides reliable input information for the deep reinforcement learning algorithm (TD3). An effective trajectory description method should not only cover the preset ideal drilling path but also reflect the actual trajectory of the drilling tool in the formation in real time. At the same time, it should be closely connected with the field control process to ensure that the algorithm can adapt to the complex drilling environment. A borehole trajectory is a kind of spatial curve that reflects the drilling tool. It is affected by some unknown factors in the drilling process. Usually, engineers determine the drilling trajectory according to the specific drilling intention and then record the actual migration trajectory of the drilling tool with the help of MWD data in the process of borehole trajectory drilling. The accurate characterization of the borehole trajectory usually requires the well depth \(L\), azimuth \(\varphi\), and deviation angle \(\alpha\). The well depth is used to reflect the total footage in the wellbore of the drilling tool, which refers to the length of the wellbore from the wellhead to the bit. The azimuth angle is used to reflect the direction of the wellbore trajectory on the projection of a horizontal plane; it is also used to describe the yaw direction of the drilling tool, which refers to a projection angle of the drilling tool from the geographic north to the well axis plane. The inclination angle of the wellbore trajectory is used to characterize the inclination angle of the current direction of the drilling tool relative to the vertical direction. As shown in Figure 3, these parameters work together to form a complete characterization of the wellbore trajectory, providing key data support for drilling trajectory adjustment and optimization. Combined with the engineering elements of trajectory control in the actual drilling process, this paper defines the state at the \(t\) moment as:  
\[s_{t}=[\theta_{t},\phi_{t},N_{t},E_{t},H_{t}] \tag{26}\]  
where \(\theta_{t}\) is the current deviation angle, which represents the angle between the bit direction and the north direction on the horizontal plane and is used to control the deflection degree of the bit relative to the vertical direction. The range is \([0,\pi/2]\). \(\phi_{t}\) is the current azimuth, which represents the angle between the direction of the drill bit on the horizontal plane and the north direction and the orientation of the wellbore on the horizontal plane, and the range is \([-\pi,\pi]\). \(N_{t}\) represents the current north position in meters; \(E_{t}\) represents the current east position in meters; and \(H_{t}\) represents the current vertical depth position in meters. \(\theta_{t}\) (well deviation angle) directly influences borehole stability; excessive values may cause wellbore collapse. \(\phi_{t}\) (azimuth angle) determines horizontal direction control, affecting target layer docking accuracy. The three-dimensional coordinates \(N_{t},E_{t},H_{t}\) calculate spatial distances to targets and obstacles, triggering collision penalties. The design fully reflects the coupling characteristics of the attitude and spatial position of the drill bit in the trajectory control process, as well as the attitude and position of the drill bit in the space.