## 5 MDP Model for the Multi-UAV Task Planning
### Reward Function for UAV Flight Effect Evaluation  
In reinforcement learning, the agent aims to maximize its cumulative reward, so we need to design the reward function to specify what agents need to achieve. In this section, we describe the reward function in detail to guide the agent to move towards the target, shorten the range, avoid collisions, reach the target area, and stay within the task area.  
We designed a reward function \(r_{t}^{s}\) to guide the agent to the target, as Formula (16) shows.  
\[r_{t}^{s}=-\min\left(\left\|p_{t}^{\text{{\emph{uav}}}}-p_{t}^{\text{{\emph{ aim}}}}\right\|\right), \tag{16}\]  
After each step of the agent is updated, the multi-agent system will calculate the distance between each target and the agent to find the target closest to the agent. After the distance between the two is reversed, it will be added to the reward.  
Minimizing the flying distance is of great significance for the UAV cluster, such as saving resources, time, and so on. We added a negative penalty term for each time step with very small data, i.e., \(r_{t}^{s}<0\), to encourage the UAV to use a shorter path to achieve the goal.  
Next, we designed a reward function to avoid collisions. That is, when collisions occur, agents will obtain negative rewards, so they can avoid collisions. Inspired by [50], we added critical areas between agents and obstacles, and the critical areas are the extension of agent ontology. When there is a collision between critical areas, we define it as a pseudo-collision. At this time, the agent will obtain a negative reward, which is equivalent to increasing the collision warning mechanism, increasing the reaction time for agents to avoid collisions, and effectively reducing the probability of collisions. When agents or obstacles collide, we  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Value** & **Unit** \\ \hline Maximum flight speed \(v_{\max}\) & 10 & m/s \\ Maximum flight acceleration \(a_{\max}\) & 3 & m\({}^{2}\)/s \\ Minimum flight altitude \(h_{\min}\) & 0.1 & km \\ Maximum flight altitude \(h_{\max}\) & 9.2 & km \\ Maximum turning angle \(\theta_{\max}\) & 60 & \({}^{\circ}\) (Degree) \\ Maximum climbing angle \(\psi_{\max}\) & 45 & \({}^{\circ}\) (Degree) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Parameters of the UAV model in the experiment.  
regard them as real collisions, and the punishment of a real collision is more serious than that of a pseudo-collision. The schematic diagram of pseudo-collision, real collision and critical area is shown in Figure 7. The specific reward function is shown in Formula (17).  
\[r_{t}^{c}=\begin{cases}r_{pseudo},\left\|p_{t}^{uaw}-p_{t}^{udr}\right\|\leq D_ {critic}\\ r_{col},\left\|p_{t}^{uaw}-p_{t}^{udr}\right\|\leq D_{col}\\ 0\end{cases}, \tag{17}\]  
In Formula (17), \(D_{col}\) represents the range of collision and the \(D_{critic}\) represents the range of the pseudo-collision, where \(D_{critic}=D_{col}+2\cdot r_{critic}\), \(r_{critic}\) is the range of the critical area.  
We also design a \(r_{t}^{a}\) reward structure, which gives rewards to the agent who reach the target area.  
\[r_{t}^{a}=\begin{cases}r_{arr},\left\|p_{i}^{uav}-p_{i}^{aim}\right\|\leq D_{ aim},\\ 0,\quad\text{otherwise}\end{cases}, \tag{18}\]  
Meanwhile, we set up a reward function \(r_{t}^{o}\) to ensure that the agent stays within the task environment.  
\[r_{t}^{o}=\begin{cases}r_{out},\text{if out of range}\\ 0,\text{ if in range}\end{cases}, \tag{19}\]  
We add up all the reward structures to build a reward function \(r_{t}\) that can enable agents to achieve various tasks.  
\[r_{t}=r_{t}^{s}+r_{t}^{s}+r_{t}^{c}+r_{t}^{a}+r_{t}^{o}, \tag{20}\]  
In this work, we set \(r_{pseudo}=-10\), \(r_{col}=-20\), \(r_{arr}=30\),\(r_{out}=-2\), \(r_{t}^{s}=-0.1\) in the training procedure.