## 2 Materials and Methods
### MADDPG Solves UAV Path Planning and Task Offloading Problem
#### 2.3.2 Security Reinforcement Learning  
1. Introduction to Security Reinforcement Learning a. Linear Safety Signal Model In the absence of prior knowledge of its environment, intelligent agents initialized with random policies cannot guarantee meeting constraints for each state during the initial training phase. In order for agents to learn to avoid unsafe or unexpected actions, they must violate a sufficient number of times for the negative impact to propagate in the algorithm model. Therefore, the linear safety signal model combines some prior knowledge based on basic forms of single-step dynamics, which is implemented as follows in the linearization. \[\overline{c}(s^{\prime})=c_{i}(s,a)\approx\overline{c}_{i}(s)+g(s;\omega_{i})^ {T}a\] (38)  
Here, \(g(s;\omega_{i})\) represents a neural network that takes a state \(s\) as input and outputs a vector with the same dimension as action \(a\), which is the extracted feature of the state. \(\omega_{i}\) is the weight of the neural network. \(\overline{c}(s^{\prime})\) is the first-order approximation of \(c_{i}(s,a)\) with respect to \(a\), which explicitly represents the sensitivity of actions to changes in the safety signal using state features. Given a policy tuple \(D=\left\{(s_{i},a_{j},s^{\prime}_{j})\right\}\), the neural network \(g(s;\omega_{i})\) is trained using Formula (39). \[\underset{\omega_{i}}{argmin}\sum_{(s,a,s^{\prime})\in D}\left(\overline{c}_ {i}(s^{\prime})-\left(\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\right)\right)^ {2} \tag{39}\]  
b. Analysis and Optimization of the Safety Layer Adding a safety layer on top of the deep policy network addresses the action correction optimization issue for each forward pass, as illustrated in the specific model in Figure 3. Let \(s\) denote the current environmental state, \(\mu_{\theta}(s)\) denote the intelligent agent's action output by the deterministic policy gradient network. After passing through the safety layer, the resulting safe action \(\widetilde{\mu}_{\theta}(s)\) is obtained, which is closest to the original action \(\mu_{\theta}(s)\). The safety layer aims to minimize interference with the original action in the Euclidean norm to meet necessary constraints. The objective of the safety layer is to find safe intelligent agent actions that satisfy the constraints, as shown in Formula (40). \[\underset{a}{argmin}||a-\mu_{\theta}(s)||_{2}^{2} \tag{40}\]  
\[\text{s.t. }\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\leq C_{i}\;\;\forall i \in[K] \tag{41}\]  
To address problem (40), we substitute the safety signal \(c_{i}(s,a)\) with the linear model from Formula (38) to obtain a quadratic programming formulation, as shown in Formula (42). Due to the positive definiteness of the quadratic form and linear constraints, the global solution to this convex problem can be obtained. \[a^{*}=\underset{a}{argmin}||a-\mu_{\theta}(s)||_{2}^{2} \tag{42}\]  
\[\text{s.t. }\overline{c}_{i}(s)+g(s;\omega_{i})^{T}a\leq C_{i}\;\;\forall i \in[K] \tag{43}\]
2. MADDPG Algorithm with Integrated Security Layer. The actions obtained by the MADDPG algorithm do not consider the safety of the actions and may cause collisions between agents. Therefore, we integrated a safety layer  
Figure 3: Deep Policy Network with Added Security Layer. into the MADDPG algorithm to constrain the actions of agents, enabling agents to avoid unsafe actions and thereby prevent collisions. The reinforcement learning we propose with an integrated safety layer is a complete improvement based on the MADDPG algorithm. In the training process, unlike the MADDPG algorithm, which directly outputs the actions of multiple agents, the improved algorithm combines the actions of multiple agents and then imposes safety constraints on the combined action to obtain a safe combined action. The structure of the algorithm is shown in Figure 4. The algorithm combines the actions of all agents into an action vector \(\mathsf{a}\), i.e., \(a=(a_{1},a_{2},...,a_{n})\), and inputs this vector into the safety constraint layer.