# UAV-Cooperative Penetration Dynamic-Tracking Interceptor Method Based on DDPG
## 3 Deep Deterministic Policy-Gradient Algorithm  
The DDPG algorithm is a branch of reinforcement learning [5]. The basic process of reinforcement-learning training is that the agent performs an action based on the current observation state. This action acts on the agent's training environment and returns a reward and a new state observation. The goal of training is to maximize the final reward. Reinforcement learning does not need to give any artificial strategies and guidance during training but only needs to give the reward function when the environment is in various states. This is also the only part of training that can be adjusted artificially. Figure 4 shows the basic process of reinforcement learning. The DDPG algorithm is an actor-critic framework algorithm that solves the problem of applying reinforcement learning in continuous space. There are two networks in the DDPG algorithm, namely, the state-action value function network \(Q(s,a|\theta^{Q})\) using \(\theta^{Q}\) parameters and the policy network \(\mu(s|\theta^{\mu})\) using \(\theta^{\mu}\) parameters. At the same time, two concepts are introduced, target network and experience replay. When the value function network is updated, the current value function is used to fit the future state-action value function. If both state-action value functions use the same network, it is difficult to fit during training. Therefore, the concept of the target network is introduced. The target network is used as  
Figure 4: The basic process of reinforcement learning. the future state-action value function, which is the same as the state-action value function network to be updated, except that it is not updated in real time but is updated according to the state-action value function network when the state-action value function network is updated to a certain extent. The policy network also adopts the same training method in DDPG. The experience replay is a function of storing state transfer \((s_{t},a_{t},r_{t},s_{t+1})\), and it will be stored in the experience replay pool every time the agent performs an action that causes the state to transfer. When the value function is updated, it will not be updated directly according to the action of the current policy, but the state transition value will be extracted from the experience playback pool for updating. The advantage of such an update is that the training and learning of the network is more efficient. Before training, the value function network, \(Q(s,a|\theta^{Q})\), and the policy network, \(\mu(s|\theta^{\mu})\), are first randomly initialized, and then the target network, \(Q^{\prime}\) and \(\mu^{\prime}\), are initialized. It is also necessary to initialize an action random noise, \(\mathcal{N}\), which is conducive to the agent's exploration. During training, the agent selects and executes actions, \(a_{t}=\mu(s_{t}\mid\theta^{\mu})+\mathcal{N}_{t}\), based on the current policy network and action noise and receives rewards, \(r_{t}\), and new state observations, \(s_{t+1}\), based on environment feedback. The state transition \((s_{t},a_{t},r_{t},s_{t+1})\) is stored in the experience replay pool. After that, N state transitions \((s_{i},a_{i},r_{i},s_{i+1})\) are randomly selected to update the value function network. The principle of updating the value function network is to minimize the loss function. The mathematical expression of the loss function is as follows:  
\[L=\frac{1}{N}\sum_{i}\Bigl{(}y_{i}-Q\Bigl{(}s_{i},a_{i}\mid\theta^{Q}\Bigr{)} \Bigr{)}^{2} \tag{7}\]  
where \(y_{i}=r_{i}+\gamma Q^{\prime}\Bigl{(}s_{i+1},\mu^{\prime}\Bigl{(}s_{i+1}\mid \theta^{\mu^{\prime}}\Bigr{)}\mid\theta^{Q^{\prime}}\Bigr{)}\), \(y_{i}\) is only related to \(\theta^{Q}\). Assuming the objective function of training is the following:  
\[J(\theta^{\mu})\ =E_{\theta^{\mu}}\Bigl{[}r_{1}+\gamma r_{2}+\gamma^{2}r_{3}+ \ldots\gamma^{N}r_{N}\Bigr{]} \tag{8}\]  
where \(\gamma\) is the discount factor.