## 2 Materials and Methods
### MADDPG Solves UAV Path Planning and Task Offloading Problem  
Here, we first re-model the aforementioned problem as a Markov decision process (MDP) and then solve it using the MADDPG algorithm. 1. MDP  
The joint optimization problem of task offloading and UAV trajectory is modeled as a Markov decision process, denoted as \((N,S,\{A_{n}\}_{n\in N},P,\{R_{n}\}_{n\in N},\delta)\); the six elements are the set of agents, the set of agent states, the action of the nth agent, the state transition matrix, the reward obtained by the nth agent and the decay factor, whose range is \([0,1]\), respectively. 2. State  
According to the joint optimization problem of path planning and task offloading, the state (State) is composed of task parameters, UAV parameters, and edge server parameters within the time interval \(t\). The state of the t-th time interval is denoted as \(s_{t}=<\omega,O_{i},W,f,F>\), where \(\omega\) represents the coordinate matrix of \(n\) UAVs, as shown in Formula (28). \(O_{i}\) represents the observation of all user devices by UAV \(i\); if UAV \(i\) can not observe a user device \(q\), then \(O_{1,q}=[0,0,0,0]\), otherwise, it is the specific information, as shown in Formula (29). \(W\) is the set of tasks in the current time slot. \(f\) refers to the computing resources of the UAVs, which can be represented as \(f=(f_{1},f_{2},\ldots,f_{N})\). F represents the computing resources on the edge servers, denoted by the formula \(F=(F_{1},F_{2},\ldots,F_{K})\). \[ \begin{array}{c}\omega_{1}\left[\begin{array}{c}x_{1}\,y_{1}\,z_{1}\\ \omega_{2}\end{array}\left[\begin{array}{c}x_{2}\,y_{2}\,z_{2}\\ \ldots\end{array}\left[\begin{array}{c}\ldots\\ \ldots\end{array}\ldots\ldots\ldots\omega_{n}\right]\\ \omega_{n}\left[\begin{array}{c}x_{n}\,y_{n}\,z_{n}\end{array}\right]\end{array}  \tag{28}\]  
\[\begin{array}{c}O_{i,1}\left[ \begin{array}{c}i&1\\ O_{i,2}\end{array} \left[ \begin{array}{c}i&2\\ i&2\end{array} \left[ \begin{array}{c}\,d_{i,2}\\ \ldots\end{array} \left[ \begin{array}{c}\,\alpha_{i,2}\\ \ldots\end{array} \ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots \ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots \  
Action  
Each UAV needs to determine its motion state (horizontal flight distance \(l_{n}\), horizontal flight angle \(\theta_{n}\), vertical flight distance \(\Delta z_{n}\)), transmission power \(P_{n}\), task allocation ratio \(\gamma_{mk}^{n}\), and UAV transmission vector \(g_{n,s}\). The action of UAV n is represented as:  
\[a_{n}=\{l_{n},\theta_{n},\Delta z_{n},P_{n},\gamma_{mk}^{n},g_{n,s}\} \tag{30}\]  
Here, \(\gamma_{mk}^{n}\) is the proportion of user devices \(m\) in the coverage range of UAV \(n\) offloading tasks to edge server \(k\). \(g_{n,s}\) is the vector sent by UAV \(n\) to other UAVs, with dimensions of \(R_{s\times M}\) and specifically the probability inferred by UAV \(n\) of all other UAVs covering each mobile user device. 4. Reward Function  
In reinforcement learning algorithms, agents need to design appropriate reward functions to obtain the optimal policy. Based on the optimization goals described in formula (23), the reward function in this paper is set as:  
\[R_{n}=-U_{n}(t)=-w_{1}E_{n}(t)-w_{2}T_{n}(t) \tag{31}\]  
This paper aims to train the proposed reinforcement learning algorithm to obtain the optimal strategy and thus minimize the total cost of the system. We set the minimization of the total system cost as the objective because a well-trained reinforcement learning algorithm will generate the best strategy, i.e., the actions of the UAV. The reward values generated by the UAV will gradually increase and converge during training, so we set the negative value of the system total cost as the target reward for reinforcement learning. We hope that the trained algorithm can achieve the maximum reward value, i.e., the minimization of the total system cost. Security Reinforcement Learning Algorithm Integrating UAV Communication Mechanism 2.3.1.