# UAV-Cooperative Penetration Dynamic-Tracking Interceptor Method Based on DDPG
## 3 Deep Deterministic Policy-Gradient Algorithm
### Reward-Function Design  
The sparse reward problem is a common problem when designing the reward function. This problem will affect the training process of the UAV, prolong the training time of the UAV and even fail to achieve the training goal. To better achieve collaborative tasks and solve the problem of sparse reward, the reward-function design is divided into four parts,  
Figure 5: Multi-UAV DDPG algorithm.  
namely, the distance-reward function, \(R_{d}\), which is related to the distance between the UAV and the target, the cooperative reward, \(R_{co}\), which is used to constrain the position of the UAV, the mission success reward, \(R_{s}\), and the mission failure reward, \(R_{fail}\). The reward function is linearized to improve the efficiency of UAV cluster training. One of the UAVs is an example to introduce the design of the reward function.  
Assume \(d_{i}=\sqrt{\left(x_{i}-x_{t}\right)^{2}+\left(y_{i}-y_{t}\right)^{2}},i\in L\) represents the distance between one UAV and the target area, \(d_{target}\), representing the distance between the UAV's initial location and the target area.  
The distance reward, \(R_{d}\), is related to the distance from the UAV to the target area. The closer the distance, the greater the distance-reward value. This type of reward is the key reward on whether the UAV can reach the target area. The specific form is shown in (11):  
\[R_{d}=0.6-d_{i}/d_{target} \tag{11}\]  
The cooperative reward is related to the cooperative parameters in the UAV cluster. Here, the difference between the farthest and closest distance between the UAV and the target area is selected as the coordination parameter. Its specific expression is shown in (12). When there are two UAVs in the cluster, the distribution diagram is shown in Figure 6.  
\[R_{co}=R_{d}\times\left(1-\left(d_{max}-d_{min}\right)/d_{target}\right) \tag{12}\]  
where \(d_{max}=\left\{d_{1},d_{2},\ldots,d_{n}\right\}_{max}\), \(d_{min}=\left\{d_{1},d_{2},\ldots,d_{n}\right\}_{min}\), respectively, represent the maximum and minimum distances between the UAV and the target area in the environment. It can be seen from the mathematical expression and distribution diagram that there are two major distribution trends for synergistic rewards. When the maximum-distance difference of the UAV cluster is smaller, its value is larger, which will lead the UAVs to move towards time coordination. When the UAV is closer to the target area, its value is larger, and the UAV will be guided to reach the target area.  
When the UAV receives the success signal and finally reaches the target area, the mission is successful, and it will give a success reward. The success reward is also related to  
Figure 6: Co-reward distribution.  
the farthest distance between the UAV cluster and the target point. The closer the distance, the greater the success reward, as shown in (13).  
\[R_{fail}=-1000 \tag{13}\]  
When the final mission of the UAV fails, that is, when it is intercepted by an interceptor and fails to reach the target area, it will give a negative reward of failure, as shown in (14).  
\[R_{s}=250+2500/d_{max} \tag{14}\]  
By linearly superimposing the above multiple reward functions, the final reward function can be obtained. The reward function, R, of the UAV is shown in (15).  
\[R=\beta_{1}R_{d}+\beta_{2}R_{co}+\beta_{3}R_{s}+\beta_{4}R_{fail} \tag{15}\]  
where \(\beta_{1}+\beta_{2}+\beta_{3}+\beta_{4}=1\).