## 4 DDPG Optimization of Control Parameters for Active Disturbance Rejection Controller
### Stability Analysis  
To conduct the analysis, the following hypothesis is proposed based on engineering practice: the total disturbance observed by the observer in the self-disturbance rejection control is bounded within \(H\), \(H=\{D||D|\leqslant F_{h}\}\), where \(F_{h}\) is a positive constant. Theorem 4.1: _The estimation error of the observer constructed in Equation (15) is bounded [18], \(\underset{\omega_{0}\rightarrow\omega_{0}\rightarrow\omega_{1}\rightarrow \infty}{lim}||e||=0\)._  
Proof: Let \(q_{i}=\frac{e_{i}}{\omega_{\alpha}^{2}}\left(i=1,2,3\right)\), then Equation (18) can be rewritten as:  
\[\left\{ \begin{array}{l}\dot{q}_{1}=\omega_{o}(q_{2}-3q_{1})\\ \dot{q}_{2}=\omega_{o}(q_{3}-3q_{1})\\ \dot{q}_{3}=\omega_{o}\left(\frac{D}{\omega_{\alpha}^{2}}-q_{1}\right)\end{array} \right. \tag{34}\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Actor-network learning rate & 0.001 \\ Critics’ online learning rate & 0.0005 \\ Small batch sampling sample size & 64 \\ Discount factor & 0.98 \\ Noise variance & 0.2 \\ Noise attenuation coefficient & 0.00001 \\ Experience pool size & 100,000 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: DDPG algorithm parameters. Figure 6: Training reward change curve. The reference Formula (34) can be rewritten as:  
\[\dot{\eta}=\omega_{o}Aq+B\frac{D}{\omega_{o}^{3}} \tag{35}\]  
For simplicity, let \(q=[q_{1},q_{2},q_{3}]^{T}\), therefore:  
\[A= \begin{bmatrix}-3&1&0\\ -3&0&1\\ -1&0&0\end{bmatrix} ,\ B= \begin{bmatrix}0\\ 0\\ 1\end{bmatrix}  \tag{36}\]  
Observing the above equation, for any positive \(\omega_{o},\ A\) is Hurwitz, therefore, there exists a unique positive definite symmetric matrix \(P_{\eta}\) that satisfies the Lyapunov equation \(A^{T}P_{\eta}+P_{\eta}A=-Q_{\eta}\). By choosing the Lyapunov function as \(V(q)=q^{T}P_{\eta}q\), we can derive the following for \(V(q)\):  
\[ \begin{array}{l}\dot{V}(q)=q^{T}P_{\eta}\dot{q}+\dot{q}^{T}P_{\eta}q\\ =q^{T}P_{\eta}(\omega_{0}Aq+B\frac{D}{\omega_{0}^{3}})+(\omega_{0}Aq+B\frac{D }{\omega_{0}^{3}})^{T}P_{\eta}q\\ =-\omega_{0}q^{T}Q_{\eta}q+2q^{T}P_{\eta}B\frac{D}{\omega_{0}^{3}}\end{array}  \tag{37}\]  
Referring to Equation (37), using the Cauchy inequality and the property of the minimum eigenvalue of positive definite matrices, \(\dot{V}(q)\) can be rewritten as:  
\[ \begin{array}{l}\dot{V}(q)=-\omega_{0}q^{T}Q_{\eta}q+2q^{T}P_{\eta}B\frac{D }{\omega_{0}^{3}}\\ \leq\omega_{o}\lambda_{min}\big{(}Q_{\eta}\big{)}\parallel q\parallel^{2}+ \frac{2F_{h}\lambda_{max}\big{(}P_{\eta}\big{)}\parallel q\parallel}{\omega _{o}^{3}}\end{array}  \tag{38}\]  
Using the eigenvalues of the matrix, we can obtain the following bounds on the quadratic form: \(\lambda_{min}\big{(}P_{\eta}\big{)}\parallel q\parallel^{2}\leq q^{T}P_{ \eta}q\leq\lambda_{\max}\big{(}P_{\eta}\big{)}\parallel q\parallel^{2}\). This can be rewritten as: \(\frac{V(q)}{\lambda_{\max}\big{(}P_{\eta}\big{)}}\leq\parallel q\parallel^{2} \leq\frac{V(q)}{\lambda_{\min}\big{(}P_{\eta}\big{)}}\). Therefore, inequality (38) can be rewritten as:  
\[ \begin{array}{l}\dot{V}(q)\leq\omega_{o}\frac{\lambda_{min}\big{(}Q_{\eta} \big{)}}{\lambda_{max}\big{(}P_{\eta}\big{)}}V(q)\\ +\frac{2F_{h}\lambda_{max}\big{(}P_{\eta}\big{)}}{\omega_{o}^{3}\sqrt{\lambda _{min}\big{(}P_{\eta}\big{)}}}\sqrt{V(q)}\end{array}  \tag{39}\]  
To obtain the linear differential inequality, let \(W=\sqrt{V(q)}\), then \(\dot{W}=\frac{\dot{V}(q)}{2\sqrt{V(q)}}\) can be obtained, and inequality (39) can be rewritten as:  
\[\dot{W}\leq\omega_{o}\frac{\lambda_{min}\big{(}Q_{\eta}\big{)}}{2\lambda_{max }\big{(}P_{\eta}\big{)}}W+\frac{F_{h}\lambda_{max}\big{(}P_{\eta}\big{)}}{ \omega_{o}^{3}\sqrt{\lambda_{min}\big{(}P_{\eta}\big{)}}} \tag{40}\]  
When studying the state equation \(\dot{W}\), it is often necessary to obtain the boundary of its solution \(W\), rather than the solution itself. The Gronwall-Bellman method is one of the approaches used for this purpose [31].