## 4 DDPG Optimization of Control Parameters for Active Disturbance Rejection Controller
### Deep Deterministic Policy Gradient Algorithm Principles  
DDPG is a deep reinforcement learning algorithm based on the actor-critic framework. The actor-network outputs deterministic actions in a continuous action space based on environmental state feedback, while the critic-network calculates the corresponding \(Q\)-value based on the current state and action, which is used to evaluate the long-term expected return of the action. By adjusting the weights of the critic network according to the error between the reward output by the critic network and the actual received reward, the output estimates of the critic network can become more accurate. Using the policy gradient algorithm, the parameters of the actor-network are updated in the direction of increasing the action value. During the interaction between the agent and the environment, the learning parameters of both networks will be continuously updated until the policy converges [27]. At time \(t\), the mapping from state \(s\) to action \(a\) is referred to as policy \(\pi\). \[a_{t}=\pi(s_{t}) \tag{23}\]  
According to the actions generated by strategy \(\pi\), new states and reward values \(r\) are continuously obtained. The formula for calculating cumulative rewards is:  
\[G_{t}=\sum_{t=1}^{T}\gamma^{t}r^{t} \tag{24}\]  
The Bellman equation for the state value function is represented as:  
\[V_{\pi}(s)=E_{\pi}[\sum_{k=0}^{\infty}G_{t}|s=s_{t}]=E_{\pi}[r_{t+1}+\gamma V _{\pi}(s_{t+1})|s=s_{t}] \tag{25}\]  
Considering the impact of actions on the value function, the Bellman equation for the state-action value function is represented as:  
\[Q_{\pi}(s,a)=E_{\pi}[r_{t+1}+\gamma Q_{\pi}(s_{t+1},a_{t+1})|s=s_{t},a=a_{t}] \tag{26}\]  
The optimal Bellman equation can be expressed as:  
\[Q^{*}(s_{t},a_{t})=r_{s_{t}}^{a_{t}}+\gamma Q_{\pi}^{*}(s_{t+1},a_{t+1}) \tag{27}\]  
The optimal strategy \(\pi^{*}\) is obtained by maximizing the cumulative reward and its corresponding optimal Bellman equation:  
\[\pi(s)\to a_{t}=argmaxQ_{\pi}^{*}(s_{t},a_{t}) \tag{28}\]We calculate the loss function for the target \(Q\) value, using \(y_{t}\) to represent it:  
\[\left\{ \begin{matrix}y_{t}=r_{t}+\gamma Q^{\prime}\Big{(}s_{t+1},\pi^{\prime} \Big{(}s_{t+1}\mid\theta^{\pi^{\prime}}\Big{)}\mid\theta^{Q^{\prime}}\Big{)}\\ L\big{(}\theta^{Q}\big{)}=E\big{(}y_{t}-Q\big{(}s_{t},a_{t}\mid\theta^{Q}\big{)} \big{)}^{2}\end{matrix} \right. \tag{29}\]  
By calculating the gradient of the loss function, we update the current value network [28]. \[\left\{ \begin{matrix}\theta^{Q}_{k}=\theta^{Q}_{k-1}-\mu_{Q}\
abla_{\theta^{ Q}}L\Big{(}\theta^{Q}_{k-1}\Big{)}\\ \
abla_{\theta^{Q}}L\Big{(}\theta^{Q}_{k-1}\Big{)}=E\Big{(}2\Big{(}y_{t}-Q \Big{(}s,a\big{|}\theta^{Q}_{k-1}\Big{)}\big{|}s_{s=s_{t},a=a_{t}}\Big{)}\\ \
abla Q_{\theta^{Q}}\big{(}s,a\big{|}\theta^{Q}_{k-1}\big{)}\mid_{s=s_{t},a= a_{t}}\end{matrix} \right. \tag{30}\]  
The strategy network uses the \(Q\) function output from the value network as the loss function. By taking the policy gradient of the \(Q\) function, the update formula is obtained [28]. \[\left\{ \begin{array}{l}\theta^{\pi}_{k}=\theta^{\pi}_{k-1}-\mu_{\pi}\
abla_ {\theta^{\pi}}L\Big{(}\theta^{\pi}_{k-1}\Big{)}\\ \
abla_{\theta^{\pi^{\pi^{\pi^{\pi}}}_{k-1}}\pi=\
abla_{\theta}Q\big{(}s,a\mid \theta^{\pi}_{k-1}\Big{)}\\ \mid_{s=s_{t},a_{t}=\pi(s_{t})}\
abla_{\theta^{\pi}}\pi\Big{(}s\mid\theta^{ \pi}_{k-1}\Big{)}\mid_{s=s_{t}}\end{array} \right. \tag{31}\]  
The target network uses a soft update method as follows [29]. \[\left\{ \begin{matrix}\theta^{Q^{\prime}}_{k}=\pi\theta^{Q}_{k-1}+(1-\tau) \theta^{Q^{\prime}}_{k-1}\\ \theta^{\pi^{\prime}}_{k}=\pi\theta^{\pi}_{k-1}+(1-\tau)\theta^{\pi^{\prime}} _{k-1}\end{matrix} \right. \tag{32}\]  
The DDPG algorithm is a deterministic policy that adds noise to the deterministic policy, as shown in Equation (33), allowing the agent to explore the environment more effectively and preventing it from getting stuck in local optima. The deterministic policy gradient helps the critic converge and updates the network parameters [29].