## 3 Trajectory Optimization Method Based on Improved Twin Delayed Deep Deterministic Policy Gradient  
### Deep Deterministic Policy Gradient Algorithm  
The DDPG algorithm is proposed by the Google DeepMind team for realizing continuous action space control. It is composed of an Actor-Critic structure, combined with the Deep Q-learning Network (DQN) algorithm. Figure 4 shows that the Actor-Critic structure consists of two parts: the actor network and the critic network. Among them, the actor represents the strategy network, which takes the current state as input and then generates the action under the current state through the analysis of neural networks. It takes advantage of Policy Gradient's ability to select actions in a continuous interval, and then selects actions randomly based on the learned action distribution. However, DDPG is different from Policy Gradient in that it generates deterministic actions based on the output of the actor, instead of generating according to Policy Gradient. The critic is the value network with a single step update. This update method solves the problem of low learning efficiency caused by the strategy gradient of  
Figure 3: Results of the flight test. round update. Through the reward function to guide the learning direction of the network, the critic can obtain the potential rewards of the current state, and it takes the action output from the actor network as the input and outputs the evaluation value. Critic evaluates the action selected by the actor and guides the update direction of the network parameters of the actor, so that the actor after updating the network parameters can choose actions with a higher value as much as possible. The evaluation value Q is the reward for taking action \(a_{i}\) under \(S_{i}\). The formula is as follows:  
\[Q(S_{i},a_{i}|\theta^{Q}) \tag{2}\]  
where \(\theta^{Q}\) denotes the parameter of the critic network. The behavior of each state directly obtains a certain action value through the deterministic policy function \(\mu\):  
\[a_{t}=\mu(S_{t}|\theta^{\mu}) \tag{3}\]  
where \(\mu\) represents the deterministic behavior policy, which is defined as a function and simulated by a neural network, \(\theta^{\mu}\) represents the parameter of policy network, which is used to generate the determination action. \(a_{t}\) is the change rate of angular velocity in state \(S_{t}\), and the control input \(u\) in (1) is changed by \(a_{t}\). In order to make the DDPG algorithm more random and learning coverage, it is necessary to add random noise to the selected action to make the value of the action fluctuate. The action after adding noise can be expressed as:  
\[a_{t}\sim clip(N(\mu(S_{t}|\theta^{\mu}),\sigma^{2}),a_{low},a_{high}) \tag{4}\]  
where \(N\) denotes the gaussian noise, and the noise follows the normal distribution, where \(a_{t}\) is the expectation and \(\sigma\) is the variance, \(a_{low}\) is the minimum value of the action, and \(a_{high}\) is the maximum value of the action. The design of DDPG is based on the off-policy approach, which separates the behavioral strategies from the evaluation strategies. There are estimated networks and target networks in the actor and the critic. Their estimated network parameters need to be trained, and the target network is soft-updated. Therefore, the two network structures of the actor and the critic are the same, but the network parameters are updated asynchronously. The soft update formula of the target network of the actor and the critic is as follows:  
\[\left\{ \begin{array}{l}\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau) \theta^{Q^{\prime}}\\ \theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}} \end{array} \right. \tag{5}\]  
Figure 4: Schematic diagram of the Deep Deterministic Policy Gradient algorithm.