## 1 Introduction
However, by analysing all the above research, we can observe that most planning methods have to recalculate the trajectory after changing the initial position or target position. Therefore, the lack of computing power will seriously limit the real-time performance of trajectory planning. In recent years, deep reinforcement learning has been applied to autonomous trajectory optimization with its powerful learning ability. The deep reinforcement learning model is trained under the condition that the simulation environment and initial position are completely random. The simulation environment should include wind disturbance (including horizontal wind and vertical wind) and terrain change. Using the optimization model built by deep reinforcement learning, it can be launched at different initial positions without recalculating the flight trajectory. Based on this advantage, it is widely used in Unmanned Aerial Vehicles (UAVs) [32; 33; 34; 35; 36], automobile [37], and robots [38; 39]. [PERSON]. [40] explored an online path planning method based on deep reinforcement learning for UAV maneuvering target tracking and obstacle avoidance control. [PERSON]. [41] explored an approach based on deep reinforcement learning to enable drones to perform navigation tasks in a multi-obstacle environment with randomness and dynamics. [PERSON]. [42] explored a path planning method based on a deep neural network to solve the autonomous flight problem of quadrotor aircraft in unknown environments. In [43], an improved dual-delay deterministic strategy gradient is proposed for UAV energy-saving path planning. However, due to the monotonously decreasing height and fixed flight time of the parafoil system, it can be found from the above studies that ordinary UAV planning methods are not suitable for the parafoil system. Ref. [44] obtained a large number of training samples according to Kane equation (KE) and genetic algorithm (GA), and trained them by Back Propagation Neural Network (BPNN) by establishing the database of falling point trajectories. The trained neural network is used to calculate the trajectory parameters of airdrop under specific flight conditions. However, this method needs to prepare a large number of parameter samples in advance for network training, and requires samples to be effective. In contrast, deep reinforcement learning does not need to build a sample database in advance. Furthermore, by analyzing the simulation results of the above research, although the traditional reinforcement learning method is improved in real time, the landing accuracy is not as good as the traditional numerical computation method. Thus, we propose a real-time flight trajectory optimization method for a parafoil delivery system, which was based on an improved twin delayed deep deterministic policy gradient. First, based on the analysis of the actual flight data, combined with the flight environment and the characteristics of the parafoil, a 4-DOF model of the parafoil delivery system is built. Then, in the section of the optimization, we first introduce the principle of Deep Deterministic Policy Gradient (DDPG), explain the disadvantages of the DDPG algorithm, and introduce the improvement of Twin Delayed Deep Deterministic Policy Gradient (TD3) compared with DDPG. Based on the principle of TD3, the reason why the algorithm still leads to poor landing accuracy is explained. In order to reduce the impact of noise uncertainty on the exploration strategy and make better use of the good strategy of agent exploration, the existing TD3 method is improved by dynamically selecting the scale of noise in the action. By pre-evaluating the reward value of the action, different noise scales are selected according to the size of the pre-evaluated reward value. Simulation results show that the proposed method satisfies all trajectory constraints and shows the comparison results with the DDPG and TD3 algorithms. This method can achieve better real-time performance without losing landing accuracy.