## 4 DDPG Optimization of Control Parameters for Active Disturbance Rejection Controller
### Deep Deterministic Policy Gradient Algorithm Principles
The meanings of the various parameters in the above analysis are shown in Table 1.  
\[a_{t}=\mu_{\theta}(s_{t})+N \tag{33}\]  
The structure of the DDPG algorithm model is shown in Figure 4. The DDPG agent stores the sample data obtained from interacting with the LADRC control system in the experience pool. During the learning process, it randomly samples m pieces of data from the experience pool and continuously iterates to update the network gradient values to optimize the algorithm [29; 30]. \begin{table}
\begin{tabular}{c c} \hline \hline
**Algorithm Parameters** & **Meaning** \\ \hline \(Q\big{(}s_{t},a_{t}\mid\theta^{Q}\big{)}\) & The \(Q\) value output by the current value network at time \(t\) \\ \(Q^{\prime}\Big{(}s_{t+1},\pi^{\prime}\Big{(}s_{t+1}\mid\theta^{\mu^{\prime}} \Big{)}\mid\theta^{Q^{\prime}}\Big{)}\) & Input \(Q\) value of the target network \\ \(\pi^{\prime}\Big{(}S_{t+1}\mid\theta^{\pi^{\prime}}\Big{)}\) & Action variables output by the target Combining deep reinforcement learning with an active disturbance rejection controller, the control system is designed to obtain environmental state data through the interaction between the agent and the environment (underwater robot control system). Here, LSEC represents the state error feedback controller, and LESO represents the linear expanded observer. The structural block diagram of the active disturbance rejection controller based on deep reinforcement learning is shown in Figure 5.  
According to the control system structure block diagram designed in the above figure, we set the various parameters of the deep reinforcement learning agent. If the LADRC control has the Markov property, then when optimizing with DDPG, the future state transitions of the system depend only on the current state and control actions, without the need to explicitly model the state transition probabilities. The optimization problem of the LESO observation capability is modeled as a reinforcement learning task in a continuous action space. The DDPG algorithm is used to dynamically adjust the key parameter \(w_{0}\) of the LESO, enabling it to adapt to changes in external disturbances, thereby improving the disturbance estimation accuracy of the LESO and the robustness of the controller. The actor network is responsible for generating the adjustment of the LESO observer bandwidth \(w_{0}\), based on the current state. The critic network outputs the action value \(Q\) based on the current state and the action generated by the actor network, guiding the policy update of the actor network. The ROV studied in this paper is under umbilical cable control, effectively avoiding the issue of limited resources for the ROV, and the ROV's controller can meet the high computational demands of DDPG training. Figure 4: DDPG algorithm framework diagram. Figure 5: Block diagram of a self-disturbance rejection controller based on reinforcement learning. For the state space, select \(\left(e,\dot{e}\right)\), corresponding to the errors \(\left(e_{x},e_{y},e_{z},e_{\varphi}\right)\), and the differential of the error in each degree of freedom. For the action space, select the poles of the expanded state observer, that is \(w_{0}\), \(\beta_{1}=3\omega_{\alpha}\), \(\beta_{2}=3\omega_{\alpha}^{2}\), \(\beta_{3}=\omega_{\alpha}^{3}\). To reduce the final error, a reward function is set based on the error between the output of each degree of freedom and the expected value: \(R=-\left[\left(\sqrt{{{e_{x}}^{2}+{e_{y}}^{2}+{e_{z}}^{2}+{e_{\varphi}}^{2}}} \right)\right]\). In terms of the discount factor, the degree to which future rewards influence current decisions is determined, and in this article, the chosen discount factor \(\gamma=0.98\) is used to ensure that accurate trajectory tracking is given high priority. After multiple adjustments, the final selected DDPG parameters are shown in Table 2.  
The reward curve of the Deep Deterministic Policy Gradient (DDPG) algorithm is generally used to determine whether the agent's training has converged.