# Research on Wellbore Trajectory Optimization and Drilling Control Based on the TD3 Algorithm
## 4 Experimental Design and Result Analysis  
### Experimental Settings  
In order to verify the effectiveness of the above methods, a well trajectory optimization experiment based on a simulated environment was designed. This section describes, in detail, the data sources, parameter settings, and evaluation criteria of the environment used in the experiment. (1) Environment and data sources. In this paper, a three-dimensional wellbore trajectory simulation environment was constructed, which mainly refers to the typical scenarios and parameter ranges in actual oil and gas well trajectory planning. The starting point of the borehole was located at a fixed point (wellhead) on the surface, and the target point was located at a predetermined underground reservoir location. In order to be representative, the target depth was set to about 800 m, and the horizontal offset was about 1000 m, similar to the well depth and displacement ratio of directional/horizontal wells. Relevant stratum and obstacle information were set based on the literature and experience. For example, a section of formation near the target was set up to limit the deviation of the well (the deviation angle should not be too large when entering the target), and a 'forbidden drilling area' (such as a high-pressure sag zone or abandoned wellbore position) was designed in the middle of the wellbore. Although we used simulated data, the parameter selection strived to conform to the order of magnitude of real wellbore planning, so as to ensure that the experimental conclusion has a reference value for practice. (2) State and action spaces. The state space included the current position (three-dimensional coordinates), the current deviation angle, and the current azimuth angle. The action space was a continuous two-dimensional action-azimuth steering angle and well deviation angle adjustment range, which indicated how much the current time step turned in the horizontal plane relative to the forward direction and how much the current angle changed relative to the current angle, respectively. After each step, the drill bit moved in a new direction with a fixed step length (e.g., 10 or 30 m) and then moved to the next time step. This setting simulated a steerable curved drilling tool that can fine-tune the azimuth and inclination to change the drilling direction at each step. (3) Obstacle layout logic. In order to investigate the obstacle avoidance ability of the algorithm, we placed two representative obstacles in the area where the wellbore path may pass, as shown in Figure 4.  
(a) Oblique crossing fault zone. The oblique crossing fault zone was represented by an inclined red cylinder with an inclined plane of about 45\({}^{\circ}\) passing through the wellbore path plane. If the drill bit did not change direction, the straight-line drilling penetrated into the area, and the agent needs to learn to detour in advance. Additionally, 'traversable obstacles' (e.g., low-strength mudstone interlayers) were defined with safety criteria (e.g., dogleg change rate \(<0.3^{\circ}/\text{m}\)), allowing controlled passage when the trajectory adjustment cost was lower than avoidance. (b) Target volume limitation. A target allowable area was set around the target, which was represented by a cylinder with a certain radius (vertical red cylinder in the Figure 7). Entering the bottom of the cylinder was regarded as the hit target. However, at the same time, the upper formation through which the column passes had requirements for well deviation (for example, it cannot be greater than a certain angle), and exceeding it was considered to be illegal. The agent must enter the target area and cannot enter from an inappropriate angle. In contrast, oblique fault zones and target volume constraints were classified as 'non-traversable obstacles' requiring complete avoidance. As shown in Table 2, the TD3 reinforcement learning wellbore trajectory design experimental environment is detailed.