# Research on Wellbore Trajectory Optimization and Drilling Control Based on the TD3 Algorithm
## 2 Related Work
### TD3 Network  
(1) The principle of the TD3 algorithm. The network of the TD3 algorithm is an actor-critic architecture. The actor network is responsible for generating actions, and the critic network is responsible for evaluating the value of actions (Q value) [32; 33; 34]. This architecture enables the algorithm to efficiently solve high-dimensional continuous action problems. The actor update goal is to maximize the Q value of the critic network, and the critic network optimization goal is to minimize the Q value prediction error. Based on the idea of double Q learning, TD3 uses two independent critic networks to reduce the bias of Q estimation. By sampling data from the experience playback pool to train the network, TD3 breaks the data correlation and improves the learning efficiency. The TD3 algorithm was proposed by [PERSON] et al. (2018) [PERSON] et al. (2018). A systematic improvement scheme was proposed for the two main defects of the deep deterministic policy gradient (DDPG) algorithm-overestimation bias and policy variance. By adding controllable noise to the target strategy, the strategy is considered to be too aggressive, and the generalization ability is improved. These improvements increase the sample efficiency of TD3 in continuous control tasks by 40% (compared to DDPG) and reduce the policy variance by 65% (based on the OpenAI Gym benchmark), making it a benchmark algorithm for continuous action space reinforcement learning. (2) Mathematical principles. In the framework of reinforcement learning, an irreversible Markov decision process (MDP) is considered, and its state transition probability is  
\[p(s_{t+1}\mid s_{t},\ a_{t}) \tag{14}\]  
The reward function is defined as  
\[r(s_{t},\ a_{t}) \tag{15}\]  
The deterministic strategy \(\pi(s;\phi)\) outputs the only action \(a=\pi(s;\phi)\), and the objective function is  
\[J(\phi)=\mathbb{E}_{s_{0}\sim\varphi_{0},a_{t}=\pi(s_{t})}\left[\sum_{t=0}^{ \infty}\gamma^{t}r(s_{t},a_{t})\right] \tag{16}\]  
According to the deterministic strategy gradient theorem ([PERSON] et al., 2014) [PERSON] et al. (2014); [PERSON] et al. (2016), the gradient calculation formula is as follows:  
\[\
abla_{\phi}J(\phi)=\mathbb{E}_{s\sim\varphi,\pi}\left[\
abla_{a}Q^{\pi}(s,a )\cdot\
abla_{\phi}\pi(s;\phi)\right]_{a=\pi(s)} \tag{17}\]  
The direction of strategy updating is determined by the sensitivity of the action to the \(Q\) value \(\
abla_{a}Q\) and the control ability of the strategy parameters to the action \(\
abla_{\phi}\pi\). In the DDPG algorithm, because of the fitting error of the single Q network, the Q value may be overestimated. To suppress this overestimation, TD3 introduces two Q networks, denoted by  
\[Q_{1}(s,a;\theta_{1}),\ Q_{2}(s,a;\theta_{2}) \tag{18}\]  
The target Q value is estimated by the minimum value. \[y=r+\gamma\text{min}_{i=1,2}\ Q_{i}(s^{\prime},\pi(s^{\prime};\phi^{\prime}), \theta_{i}^{\prime}) \tag{19}\]  
Through the consistency proof of the Bellman equation, it is concluded that the overestimation error of the double Q network satisfies  
\[\mathbb{E}[y]\leq\mathbb{E}[r+\gamma V^{*}(s^{\prime})] \tag{20}\]  
Compared with the single Q network, the double Q mechanism can effectively reduce the overestimation of the Q value, and the error variance reduction is:  
\[\mathcal{O}\left(\sigma_{Q}^{2}\right) \tag{21}\]  
where \(\sigma_{Q}^{2}\) represents the noise level estimated by the Q value. The strategy network is updated every \(d\) step (\(d=2\) in this study), while the Q network is updated in real time. This design is based on the 'bias-variance' trade-off principle:  
\[\text{Var}(\
abla_{\phi}J)\propto\frac{1}{d}\mathbb{E}[\|\
abla_{a}Q\|^{2}] \tag{22}\]  
By reducing the sampling frequency of the policy gradient, the gradient variance is reduced by \(d^{-1}\), thereby improving the training stability. A soft update (\(\tau=5\times 10^{-3}\)) is used to ensure the smooth tracking ability of the target Q network and avoid instability caused by too fast update.