## 3 MAPPO-Based Joint Caching and Computation Offloading Scheme
### Dec-POMDP Framework
\begin{array}{ll}-E_{u}^{\text{UAV}}(t),&\text{if \eqref{eq:r_2,u} and \eqref{eq:r_3,u} is satisfied,}\\ -\infty,&\text{else}\end{array} \right. \tag{44}\]  
The total energy consumption related reward \(r_{2}(t)\) is the sum of these individual rewards across all UAVs \(U\). \[r_{2}(t)=\sum_{u=1}^{U}r_{2,u}(t) \tag{45}\]  
The remaining caching space-related reward \(r_{3,u}(t)\) is defined as  
\[r_{3,u}(t)=\left\{ \begin{array}{ll}\sum_{d\in\mathcal{D}}x_{u,d}(t)s_{d}+ \sum_{k\in\mathcal{K}}x_{u,k}(t)s_{k},&\text{if \eqref{eq:r_3,u} is satisfied,}\\ -\infty,&\text{else}\end{array} \right. \tag{46}\]  
The total remaining caching space reward \(r_{3}(t)\) is the sum of \(r_{3,u}(t)\) over all UAVs \(U\). \[r_{3}(t)=\sum_{u=1}^{U}r_{3,u}(t) \tag{47}\]  
and interactive dynamics among agents, assisting the Critic network in forming a more precise assessment of the global state's value. During the training period, the central module leverages comprehensive global information (like the positions, caching statuses, and task states of all agents) to train the Critic network; it then uses the advantage measure derived from the Critic network to direct the policy gradient updates of the individual, decentralized Actor networks. During the actual deployment phase, each agent relies solely on its local observations and its trained Actor network to make decisions, eliminating the need for centralized coordination, which boosts the system's responsiveness and resilience. The detailed steps of the proposed MAPPO-based training framework are outlined in Algorithm 1. ```
Input: Max episodes \(M_{te}\), PPO epochs \(P_{ce}\), episode length \(E_{pl}\), learning rates \(\psi\) (Actor), \(\phi\) (Critic), discount factor \(\gamma\), clip parameter \(\epsilon\), reward weights \(\lambda_{E}\), \(\lambda_{Q}\), \(\lambda_{C}\), \(\lambda_{F}\) Output: Trained actor parameters \(\theta\), critic parameters \(\omega\)
1 Initialize actor parameters \(\theta_{u}\), critic parameters \(\omega_{u}\) for all agents \(u\in\mathcal{U}\cup\mathcal{N}\); for\(episode=1\) to \(M_{te}\)do
2 Reset environment, set \(t\gets 1\); fortime step \(t=1\) to \(E_{pl}\)do
3 foreach agent \(u\in\mathcal{U}\cup\mathcal{N}\)do
4 Observe local state \(o_{u}(t)\);
5 Select action \(a_{u}(t)\sim\pi_{\theta_{u}}(a|o_{u}(t))\)+N ;
6 Execute joint action \(a(t)=\{a_{u}(t)\}\) in environment;
7 Observe next state \(s(t+1)\) and compute reward \(R(t)\) using Equation (51);
8 Store transition \((s(t),o(t),a(t),R(t),s(t+1))\) in replay buffer;
9 forepoch \(=1\) to \(P_{ce}\)do
10 foreach agent \(u\in\mathcal{U}\cup\mathcal{N}\)do
11 Compute advantage estimate \(\hat{A}(s(t))\) using GAE;
12 Update actor network by maximizing the clipped surrogate objective: \(\theta_{u}\leftarrow\theta_{u}+\psi\
abla_{\theta_{u}}\mathbb{E}[\min(\rho_{t} \hat{A}(s(t)),\text{clip}(\rho_{t},1-e,1+e)\hat{A}(s(t)))+\beta H(\pi_{\theta_ {u}})]\);
13 where \(\rho_{t}=\frac{\pi_{\theta_{u}}(a(t)|s(t))}{\pi_{\theta_{u}}^{\text{old}}(a(t) |s(t))}\);
14 Update critic network by minimizing value loss: \(V_{t}arget=(1-\gamma)R(t)+\gamma V\omega u(s(t+1))\); \(\omega_{u}\leftarrow\omega_{u}-\phi\
abla_{\omega_{u}}\big{(}V_{\omega_{u}}(s (t))-\hat{V}(s(t))\big{)}^{2}\);
15
```  
**Algorithm 1**MAPPO-based joint caching and computation offloading training framework  
We advocate employing the MAPPO algorithm to refine the policies within our established framework. In this setup, the actor network, parameterized by \(\theta_{v}\), produces action choices guided by the policy \(\pi_{v}(a(t)|s(t);\theta_{v})\); this policy corresponds to the \(v\)-th agent type and can be shared across agents of the same category. The critic network, parameterized by \(\omega_{v}\), appraises the merit of these actions based on the received reward \(R(t)\) and the state \(s(t)\)[55].