## 4 Optimized LADRC Approach Using TD3
### The Basics of the TD3 Agent  
The TD3 algorithm fundamentally derives the optimal strategy by facilitating continuous interaction between the agent and the environment. This paper uses a powered parafoil system with controllers as the environment. At the same time, the agent functions as the algorithm's cognitive component, responsible for decision-making processes, as shown in Figure 3. Specifically, the agent is situated in a specific state, denoted as \(s_{t}\in S\), at each time step. Depending on the state transition probability \(P_{ss^{\prime}}^{a}=P[s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a]\) and the action value \(a\in A\), the agent undergoes a transition from the current state \(s_{t}\) to the subsequent state \(s_{t+1}\) with a certain probability. Simultaneously, the agent receives an immediate reward \(r_{t+1}\in R\) associated with this transition. \(S\) and \(A\) are the state space and action space, respectively.  
The goal of the TD3 algorithm is to find the optimal strategy. This, of course, necessitates receiving feedback in the form of reward values. Consequently, the optimal strategy is attained by maximizing the expected value of the cumulative reward, which can be expressed as  
\[\pi^{*}=\max\limits_{\pi}E_{\tau\sim\pi}[R_{c}(\tau)] \tag{26}\]  
where \(R_{c}=\sum\limits_{t=0}^{\infty}\gamma^{t}r_{t+1}\) is the cumulative reward, and the discount factor \(\gamma\in(0,1]\) is employed to represent the importance of future reward values.  
The TD3 agent consists of several key components, namely a replay buffer, two critic networks, an actor network, and their respective target critic networks and target actor network, as illustrated in Figure 4. On the one hand, the critic network estimates the \(Q\) value expressed in Equation (27) by taking the state and action as inputs and generating the corresponding \(Q\) value as output, a common approach in DRL algorithms. On the other hand, the actor network is updated using the policy gradient method, which involves computing the gradient of the expected \(Q\) value concerning the actor network parameters. The parameters of the networks are represented by \(\theta(\theta^{\prime})\) for the critic networks and \(\phi(\phi^{\prime})\) for the actor network.  
Figure 3: Schematic diagram of the interaction process.  
\[Q_{\pi}(s,a)=E_{\pi}[R_{c}|s_{t}=s,a_{t}=a\,] \tag{27}\]  
The training process of the TD3 algorithm is shown in Algorithm 1.  
```
Initialize critic networks \(Q_{\theta_{1}}(s,a)\), \(Q_{\theta_{2}}(s,a)\) and actor network \(\pi_{\varphi}(s)\) with random parameters \(\theta_{1}\), \(\theta_{2}\) and \(\phi\).  Initialize target networks \(Q_{\theta^{\prime}_{1}}(s,a)\), \(Q_{\theta^{\prime}_{2}}(s,a)\) and \(\pi_{\varphi^{\prime}}(s)\) with weights \(\theta^{\prime}_{1}\leftarrow\theta_{1}\), \(\theta^{\prime}_{2}\leftarrow\theta_{2}\) and \(\phi^{\prime}\leftarrow\phi\).  Initialize replay buffer \(\mathcal{D}\). if\(t\leq T\)then  Select action with exploration noise \(a\sim\pi_{\varphi}(s)+\epsilon\), \(\epsilon\sim\mathcal{N}(0,\sigma)\), observe reward \(r\) and new states \(s^{\prime}\).  Store transition tuple \((s,a,r,s^{\prime})\) in \(\mathcal{D}\).  Sample mini-batch of \(m\) transitions \((s,a,r,s^{\prime})\) from \(\mathcal{D}\). \(a^{\prime}\leftarrow\pi_{\varphi^{\prime}}(s^{\prime})+\epsilon\), \(\epsilon\sim\)clip\(\left(\mathcal{N}(0,\sigma^{\prime}),-c,c\right)\). \(y\gets r+\gamma\min_{i=1,2}Q_{\theta^{\prime}_{i}}(s^{\prime},a^{\prime})\).  Update critics by \(\theta_{i}\leftarrow\arg\min_{\theta_{i}}\frac{1}{m}\sum\left(y-Q_{\theta_{i}} (s,a)\right)^{2}\). if\(t\) mod \(x\)then  Update \(\phi\) by the deterministic policy gradient: \(\
abla_{\phi}J(\phi)=\frac{1}{m}\sum\
abla_{a}Q_{\theta_{1}}(s,a)\big{|}_{a= \pi_{\varphi}(s)}\
abla_{\phi}\pi_{\varphi}(s)\).  Update target networks by moving average method: \(\theta^{\prime}_{i}\leftarrow\tau\theta_{i}+(1-\tau)\theta^{\prime}_{i}\), \(\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}\). endif endif
```  
**Algorithm 1** TD3 Algorithm