## 5 Comparison
### Training and Results  
The DDPG, TD3 and improved TD3 algorithms have the same parameters during training. Table 2 shows the main super parameters of the three algorithms. The max episodes of the algorithm is 3000, which represents the maximum number of episodes of the three algorithms. \(\gamma\)ranges from 0 to 1, the larger the value of \(\gamma\), the greater the impact of the reward value of future steps on the current state. \(\tau\) is the soft update factor in (5). The learning rate setting should not be too large or too small to avoid failing to converge or falling into local optimization. Reply buffer size represents the maximum number of experiences that can be stored; when the reply buffer is full, the newly stored experience will replace the earliest experience. Batch size is the number of samples taken from the reply buffer during each training. In this work, the value of delay step is 3, which means that the parameter of the actor is updated only once after the parameter of the critic is updated three times. As shown in Figure 8, the initial point of each episode is initialized randomly in the task area to ensure that the training process covers as many positions as possible. The episode ends when the parafoil flight time ends or the parafoil touches the restricted zone or crosses the boundary. DDPG, TD3, and the improved TD3 algorithm are used to train the parafoil delivery system in the task area. Both the actor and the critic networks use the multilayer perceptron (MLP) network structure. Figure 9 shows the changes of the average reward values of the three algorithms during the training of 3000 episodes in simulation environment 1.  
Figure 6: The simulation environment 1.  
Figure 7: The simulation environment 2.  
As shown in Figure 9, the parafoil has not learned excellent strategies at the initial stage, so the probability of touching the no-fly zone or exceeding the flying region is high. It can be seen that the average return of the parafoil random exploration in the environment is low, usually a negative reward value. When the playback buffer size reaches a certain amount, the parafoil starts to train the network to update the policy. Using the method in this work, the noise to be added is selected according to the pre-evaluated reward value after the action is selected, and the noise decreases with the increase of times. It can be seen that the parafoil random exploration is not able to reach the policy.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Max Episodes & 3000 \\ Discount Factor \(\gamma\) & 0.99 \\ Soft Update Factor \(\tau\) & 0.01 \\ Critic Learning Rate & 0.0005 \\ Actor Leaning Rate & 0.001 \\ Reply Buffer Size & \(5\times 10^{5}\) \\ Batch Size & 256 \\ Delay Steps & 3 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Parameters of the training experiment.  
Figure 8: The initial positions of 3000 episodes.  
Figure 9: Average reward value of 3000 episodes.  
observed that, with the increase of episodes, the parafoil delivery system gradually learns excellent strategies for trajectory planning, so the average reward value gradually increases. The model trained by neural network can cover all the initial positions in the environment.