## 3 Trajectory Optimization Method Based on Improved Twin Delayed Deep Deterministic Policy Gradient
### Improved Twin Delayed Deep Deterministic Policy Gradient Algorithm  
Since DDPG is an off-policy method based on the DQN algorithm, each time it selects the highest value in the current state instead of using the actual action of the next interaction, there may be an overestimation. In the Actor-Critic framework of continuous action control, if each step is estimated in this way, the error will accumulate step by step, resulting in failure to find the optimal strategy and, ultimately, making the algorithm unable to converge. The twin delayed deep deterministic policy gradient (TD3) algorithm is optimized for mitigating the overestimation error of the DDPG algorithm. The actor has two networks, an estimation network and a target network. The critic has two estimation networks and two target networks, respectively, as schematically illustrated in Figure 5. Thus, the critic has four networks with the same structure. The state quantity and action are the input of the critic network, and the output value is the value generated by the action executed in the current environment state. Regarding the optimization algorithm, TD3 adopts the Actor-Critic architecture similar to DDPG and is used to solve the problems in continuous action space. The improvement of the TD3 algorithm relative to the DDPG is mainly reflected in the following three aspects:  
The first is the double critic network structure. In TD3, the critic's estimated network and target network have two, respectively. The smaller value of the target network is selected as the update target to update Estimated Critic1 and Estimated Critic2, which can alleviate the phenomenon of overestimation. TD3 uses the same method as DDPG to construct the loss function:  
\[y_{i}=r_{i}+\gamma\min_{j=1,2}Q^{\prime}(S_{i+1},a_{i}^{\prime}|\theta^{Q^{ \prime}}) \tag{9}\]\[L=\frac{1}{n}\sum_{i}^{n}{(y_{i}-Q(S_{i},a_{i}|\theta^{Q}))^{2}} \tag{10}\]  
\(S\) and \(S_{i+1}\) are state quantities, as the input of the actor, and the output is the actions \(a_{i}\) and \(a_{i}^{\prime}\) generated in the current environment. The second is to delay updating the actor. In the TD3 algorithm, the critic network is updated once every step, the parameters of the actor are updated in a delayed manner, with a lower update frequency than the critic, that is, after the critic is updated multiple times, update the actor once. On the one hand, delaying updating the actor can reduce unnecessary repeated updates. On the other hand, it can also reduce errors accumulated in multiple updates. The third is the smooth regularization of the target strategy. By adding noise based on the normal distribution to the action of the target network selection as (11):  
\[a_{i}^{\prime}{=}\mu^{\prime}(S_{i+1}|\theta^{u^{\prime}})+clip(N(0,\sigma),- c,c) \tag{11}\]  
the value function is updated more smoothly, the network is more robust, and the robustness of the algorithm is improved. TD3 solves the problem of overvaluation of DDPG and facilitates the exploration of better strategies to improve the success rate and landing accuracy. However, applying the TD3 algorithm to the trajectory planning of the parafoil delivery system, combined with our existing simulation results and analyzing the experimental data, it is found that TD3 still has a larger landing error than the traditional trajectory optimization algorithm. This is difficult to solve only by increasing the number of training because the parafoil does not necessarily explore a better policy each time, or even a worse policy than the existing one, and stores it in the experience pool. This is due to the uncertainty of adding noise to the action. In the DDPG and the TD3 algorithms, in order to increase the randomness of the algorithm and the coverage of learning, they adopt the way of adding noise to the action to make it produce a certain fluctuation, hoping to explore more strategies.