## 4 The Proposed Algorithm
### Markov Decision Process Modeling
#### 4.1.4 MATD3-TORA-Based Task Offloading and Resource Allocation Algorithm
Initialization \[\mu(s_{1}^{1}|\theta_{1}),\ldots,\mu(s_{t}^{1}|\theta_{M}^{u}),\text{ the critic networks }Q_{1}(s_{t}\theta_{1}^{Q1}),\ldots,s\theta_{M}^{Q1}\] and \(Q_{2}(s_{t}\theta_{1}^{Q2}),\ldots,s\theta_{M}^{Q2}\), and their target networks. Initialize the experience replay buffer \(D\) and the simulation parameters of the multi-UAV MEC system. 8. Initialization \[\mu(s_{1}^{1}|\theta_{1}  
2. Environment Reset Reset  
Reset the environment state \(s_{t}\) of the multi-UAV MEC system and normalize the state \(s_{t}\) using Algorithm 1 to obtain \(s_{nor}\).
3. Action Execution  
Each UAV \(u\) selects its action \(a_{u}\) based on the current policy \(\mu(s_{nor}|\theta_{u})\) and Gaussian noise \(n_{t}\sim N(\mu,\theta^{2})\):  
\[a_{u}=\mu(s_{nor}|\theta_{u})+n_{t}. \tag{41}\]  
After executing action \(a_{u}\), UAV \(u\) observes the next state \(s^{\prime}\) and the immediate reward \(r\). The transition tuple \((s,a_{u},r,s^{\prime})\) is generated. Once all agents complete their actions and normalize their states, the global transition tuple \((S_{t},A_{t},R_{t},S^{\prime}_{nor})\) is stored in the experience replay buffer. 1. Sample Batch Extraction Randomly sample a batch of \(N_{b}\) transition tuples \((S_{b},A_{b},R_{b},S^{\prime}_{b})\) from the experience replay buffer to update the actor and critic networks. For agent \(u\), the corresponding sample is \((s_{u},a_{u},r_{u},s^{\prime}_{u})\). 2. Critic Network Update  
For each agent \(u\), MATD3-TORA employs a dual-critic structure to mitigate overestimation. The target Q-value \(y_{u}\) is computed using the minimum of the outputs from the two critic networks:  
\[y_{u}=r_{u}+\gamma min_{i=1,2}Q\theta^{\prime}_{i}(s^{\prime}_{u},a^{\prime}_ {u}), \tag{42}\]  
where \(\gamma\) is the discount factor. The parameters of the critic networks are updated by minimizing the loss function:  
\[\mu(s_{1}^{1}|\theta_{1}),\ldots,\mu(s_{t}^{1}|\theta_{M}^{u}),\text{ the critic networks }Q_{1}(s_{t}\theta_{1}^{Q1}),\ldots,s\theta_{M}^{Q1}\]0  
where \(\alpha_{critic}\) is the learning rate of the critic networks, and the loss function \(L(\theta_{Qi})\) is defined as  
\[\mu(s_{1}^{1}|\theta_{1}),\ldots,\mu(s_{t}^{1}|\theta_{M}^{u}),\text{ the critic networks }Q_{1}(s_{t}\theta_{1}^{Q1}),\ldots,s\theta_{M}^{Q1}\]1  
1. Actor Network Delayed Update MATD3-TORA introduces a delayed update mechanism for the actor networks to reduce fluctuations caused by critic network estimation errors, thereby enhancing learning stability. The parameters of the actor network \(u\) are updated as \[\mu(s_{1}^{1}|\theta_{1}),\ldots,\mu(s_{t}^{1}|\theta_{M}^{u}),\text{ the critic networks }Q_{1}(s_{t}\theta_{1}^{Q1}),\ldots,s\theta_{M}^{Q1}\]2 (45) where \(\alpha_{actor}\) is the learning rate of the actor network, and the gradient \(\
abla_{\theta_{u}}J_{u}\) can be computed as \[\mu(s_{1}^{1}|\theta_{1}),\ldots,\mu(s_{t}^{1}|\theta_{M}^{u}),\text{ the critic networks }Q_{1}(s_{t}\theta_{1}^{Q1}),\ldots,s\theta_{M}^{Q1}\]3 (46) Target Network Update MATD3-TORA employs soft updates to update the parameters of the target networks: \[\mu(s_{1}^{1}|\theta_{1}),\ldots,\mu(s_{t}^{1}|\theta_{M}^{u}),\text{ the critic networks }Q_{1}(s_{t}\theta_{1}^{Q1}),\ldots,s\theta_{M}^{Q1}\]4 (47) \[\mu(s_{1}^{1}|\theta_{1}),\ldots,\mu(s_{t}^{1}|\theta_{M}^{u}),\text{ the critic networks }Q_{1}(s_{t}\theta_{1}^{Q1}),\ldots,s\theta_{M}^{Q1}\]5 (48)where \(\tau\) is a small constant close to 0, controlling the update step size. 1. Iterative Update Repeat Steps 2 to 7 until the total number of training iterations \(I_{max}\) is reached. Finally, output the trained actor networks \(\mu(s|\theta_{1}),\dots,mu(s|\theta_{M})\). This algorithm framework ensures efficient task offloading and resource allocation in multi-UAV MEC systems, balancing coordination, stability, and performance.