## 2 Materials and Methods
### MADDPG Solves UAV Path Planning and Task Offloading Problem
#### 2.3.2 Security Reinforcement Learning
The safety constraint layer outputs the safety-constrained joint action \(a^{*}\), where \(a^{*}=(a^{*}_{1},a^{*}_{2},...,a^{*}_{n})\), and each component expression is \(a^{*}_{i}=\{l^{s}_{i},\theta^{*}_{i},\Delta z^{*}_{i},P^{*}_{i},\gamma^{i*}_{ mk},g^{*}_{i\beta}\}\), and \(l^{s}_{i}\), \(\theta^{*}_{i}\), \(\Delta z^{*}_{i}\) represent the horizontal flight distance, horizontal flight angle, and vertical flight distance of UAV \(i\) after safety constraints. \(P^{*}_{i}\) is the transmission power of the UAV, and \(\gamma^{i*}_{mk}\) represents the proportion of tasks offloaded to edge servers by the UAV. The first three elements determine the flight trajectory of the UAV, and \(\gamma^{i*}_{mk}\) determines the offloading situation of tasks generated by m objle devices. By putting these elements in the same action, joint optimization of task offloading and UAV trajectory is achieved. First, the safety constraint layer approximates the constraint function with respect to the action a to the first order, as shown in Formula (44). \[c_{j}(x^{\prime})=\hat{c}_{j}(x,a)\approx c_{j}(x)+g(x;\omega_{j})^{T}a \tag{44}\]  
Here, \(g\) is a neural network, and the single-step transformation data \(D=\{(x,a,x^{\prime})\}\) for training the neural network is generated by randomly initializing the agent's state and selecting actions according to random policies for multiple events. These generated data are used to train each constraint, and the loss function for each constraint is as shown in Formula (45), where each constraint can be trained separately. \[\underset{\omega_{i}}{argmin}\sum_{(s,a,s^{\prime})\in D}\left(\overline{c}_ {i}(s^{\prime})-\left(\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\right)\right)^ {2} \tag{39}\]0  
The safety signal introduced in Formula (44) adds an additional safety layer to strengthen the policy network. The safety layer constitutes a quadratic programming that computes the minimum distance of the actions taken by each policy network \(\pi_{\theta_{i}}(x_{i})\) on the linearized safety set, as shown specifically in Formula (46). \[\underset{\omega_{i}}{argmin}\sum_{(s,a,s^{\prime})\in D}\left(\overline{c}_ {i}(s^{\prime})-\left(\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\right)\right)^ {2} \tag{39}\]1  
\[\underset{\omega_{i}}{argmin}\sum_{(s,a,s^{\prime})\in D}\left(\overline{c}_ {i}(s^{\prime})-\left(\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\right)\right)^ {2} \tag{39}\]2  
\[\underset{\omega_{i}}{argmin}\sum_{(s,a,s^{\prime})\in D}\left(\overline{c}_ {i}(s^{\prime})-\left(\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\right)\right)^ {2} \tag{39}\]3 SMADDPG algorithm structure diagram. Here, \(\Pi(x)=(\pi_{\theta_{1}}(x_{1}),...,\pi_{\theta_{n}}(x_{n}))\) is the cascaded action of all agents without safety constraints. Due to the strong convexity of the optimization problem, a globally unique minimum exists whenever the feasible set is non-empty, which is the safest action closest to the original action. As the formula is generic, even though previous optimization iterations are feasible, there may not exist a recoverable action that can guarantee agents are in a safe state. To solve this problem, we adopt soft constraint formulas, that is, when an action that does not satisfy the safety constraint exists, it can also be executed, but the agent will be given certain penalties. The soft constraint formula is shown in (48). \[\underset{\omega_{i}}{argmin}\sum_{(s,a,s^{\prime})\in D}\left(\overline{c}_ {i}(s^{\prime})-\left(\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\right)\right)^ {2} \tag{39}\]4  
\[\underset{\omega_{i}}{argmin}\sum_{(s,a,s^{\prime})\in D}\left(\overline{c}_ {i}(s^{\prime})-\left(\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\right)\right)^ {2} \tag{39}\]5  
\(\varepsilon_{j}\geq 0\forall j=\{1,\ldots K\}\)  
Here, \(\varepsilon=(\varepsilon_{1},\ldots,\varepsilon_{K})\) is the relaxation variable, and \(\rho\) is the penalty weight after violating the constraint.