## 3 MAPPO-Based Joint Caching and Computation Offloading Scheme
### Dec-POMDP Framework
To facilitate implementation in distributed network environments, we adopt a framework involving centralized training and decentralized execution, as illustrated in Figure 4. This architecture is designed to ensure robust overall system performance while preserving the decision-making autonomy of each individual agent. It is important to note that during the centralized training procedure, calculating rewards independently at each vehicle or UAV is challenging. Consequently, the involvement of a training hub is necessary; this hub centrally collects all observations and actions to compute the rewards. Concurrently, this training hub integrates the collected observations into a unified global state representation, which is then conveyed to the critic network associated with each agent. Subsequently, the critic network formulates the state-value function, formally defined as:  
\[ \begin{split} V_{v}(t)^{\pi}(s(t),\theta_{v})&=\mathbb{ E}_{a(t)\sim\pi_{v}(a(t)|s(t),\theta_{v})}[R(s(t)|a(t)),\pi_{v}]\\ &=\mathbb{E}_{a(t)\sim\pi_{v}(a(t)|s(t),\theta_{v})}\left[\sum_{i =0}^{\infty}\gamma_{v}^{i}R(s(t+i)|a(t+i))|s(t),\pi_{v}\right],\end{split}  \tag{52}\]  
Here, \(\mathbb{E}[\cdot]\) signifies the expectation operator, \(R(s(t)|a(t))\) represents the reward function for agent \(i\) belonging to the \(u\)-th agent type, \(\gamma_{v}\) serves as the reward discount factor, and \(\pi_{v}(a(t)|s(t);\theta_{v})\) denotes the collective policy employed by all agents. As derived from (52), the state-value function \(V_{v}(t)^{\pi}(s(t),\theta_{v})\) is influenced by both the prevailing state \(s(t)\) and the parameters \(\theta_{v}\) of the policy network \(\pi_{v}\). Specifically, as the state \(s(t)\) improves or becomes more favorable, the value \(V_{v}(t)^{\pi}\) increases, signaling higher anticipated future returns. Similarly, when the policy \(\pi_{v}\), characterized by parameters \(\theta_{v}\), demonstrates enhanced performance, the value \(V_{v}(t)^{\pi}\) also rises. If a policy \(\pi_{v}\) consistently achieves exceptional outcomes across all relevant states \(s(t)\) within the same time step, the averaged value of the state-value function \(V_{v}(t)^{\pi}\) is expected to be notably elevated. Consequently, we define the optimization objective for the actor network as:  
\[P(\theta_{v})=\mathbb{E}_{s(t)}[V_{v}(t)^{\pi}(s(t),\theta_{v})], \tag{53}\]  
This objective function is independent of the specific state \(s(t)\) and depends solely on the parameter set \(\theta_{v}\). We modify the parameters \(\theta_{v}\) of the policy network using a gradient ascent approach, seeking to maximize the objective function. Let \(\theta_{v}^{\text{now}}\) indicate the current parameter values of the policy network; following a gradient ascent update step, the refined parameters \(\theta_{v}^{\text{new}}\) are obtained via:  
\[\theta_{v}^{\text{new}}\leftarrow\theta_{v}^{\text{now}}+\psi\
abla_{\theta_ {v}}P(\theta_{v}^{\text{now}}), \tag{54}\]  
Figure 4: Architecture of the proposed MAPPO-based joint optimization training framework. where \(\psi\) is the learning rate for the actor network, and the gradient \(\
abla_{\theta_{v}}P(\theta_{v}^{\text{now}})\) can be expressed as:  
\[ \begin{split}\
abla_{\theta_{v}}P(\theta_{v}^{\text{now}})& =\frac{\partial P(\theta)}{\partial\theta}\bigg{|}_{\theta=\theta_{v}^{ \text{now}}}\\ &=\mathbb{E}_{s(t),a(t)\sim\pi_{v}(a(t)|s(t))\hat{a}_{v})}[\
abla_ {\theta_{v}}\ln(\pi_{v}(a(t)\mid s(t);\theta_{v}))\\ &\sum_{l=1}^{\infty}(\gamma_{v}\lambda)^{l}\big{(}Q_{v,u}^{\pi}( s(t);a(t))-V_{v,u}^{\pi}(s(t),\theta_{v})\big{)}\bigg{]}\end{split}  \tag{55}\]  
Here, \(\lambda\) is a coefficient that balances the trade-off between bias and variance in the estimation.