## 3 MAPPO-Based Joint Caching and Computation Offloading Scheme
### Dec-POMDP Framework  
Based on the MADRL algorithm introduced earlier, we build it upon a Dec-POMDP framework to address the joint caching and computation offloading optimization problem. This framework empowers individual agents (UAVs and vehicles) to make decentralized decisions in a dynamic IoV environment with partial state observability. Agents learn through interactions with the environment, adapting their actions to optimize collective behavior and maximize service efficiency, defined as the ratio of the cache hit rate to the total task processing delay. The Dec-POMDP framework is formally defined as a tuple \(\langle\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\mathcal{O},\mathcal{Z },\gamma\rangle\), where \(\mathcal{S}\) is the global state space; \(\mathcal{O}\) is the observation space; \(\mathcal{Z}(o(t)|s(t)):\mathcal{S}\to O\) denotes the mapping from global state \(s(t)\in\mathcal{S}\) to observation \(o(t)\in\mathcal{O}\); \(\mathcal{A}\) is the action space; \(\mathcal{P}\) is the transition probability distribution of the global state transition \(P(s(t+1)|s(t),a(t))\): \(\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}\) and \(a(t)\in\mathcal{A}\); \(\mathcal{R}\) is the instantaneous reward \(\mathcal{R}(r(t)|s(t),a(t))\): \(\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{R}\); and \(\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}\) is the discount factor. Assume that there is no global information exchange and each unit (UAV and vehicle) can only obtain its direct information. Then, we can obtain the state space, action space, and reward as follows. Observation (O):  
\[ \begin{split}\mathcal{O}=\{& o_{1}^{UAV}(t),o_{2}^{ UAV}(t),\ldots,o_{u}^{UAV}(t),\ldots,o_{U}^{UAV}(t),\\ & o_{1}^{Vehicle}(t),o_{2}^{Vehicle}(t),\ldots,o_{n}^{Vehicle}(t ),\ldots,o_{N}^{Vehicle}(t)\}\end{split}  \tag{38}\]  
Figure 3: Comparison of Multi-Agent Reinforcement Learning training and execution paradigms. The observation of UAV \(u\), \(o_{u}^{U}(t)\), contains the location of the UAV \(u\), the coverage range of the UAV \(u\), the remaining energy of the UAV \(u\), and the remaining caching space, i.e.,  
\[o_{u}^{U}(t)=\{\mathbf{I_{u}}(t),\mathbf{I_{u}}(t),E_{u}(t),C_{u}(t)\},u\in \mathcal{U} \tag{39}\]  
where \(\mathbf{I_{u}}(t)=\{I_{1,u}(t),\ldots,I_{n,u}(t),\ldots,I_{N,u}(t)\}\), \(C_{u}(t)=Q_{u}-(\sum_{d\in D}x_{u,d}(t)s_{d}+\sum_{k\in K}x_{u,k}(t)s_{k})\), and \(E_{u}(t)=E^{\max}-\sum_{t^{\prime}=1}^{t}E_{u}^{\text{UAV}}(t^{\prime})\). Partial observations are typical in decentralized MADRL [56]. The observation of vehicle \(n\), \(o_{n}^{N}(t)\), contains its location, i.e.,  
\[o_{n}^{N}(t)=\{\mathbf{I_{n}}(t)\},n\in\mathcal{N} \tag{40}\]  
Action Space (\(\mathcal{A}\)): The action space of the UAV \(u\) contains the cache decision of the UAV \(u\) and the location of the UAV \(u\), i.e.,  
\[a_{u}(t)=\{\mathbf{X}_{u}(t),\mathbf{I_{u}}(t)\} \tag{41}\]  
where \(\mathbf{X}_{u}(t)=\left[\mathbf{x}_{u,d}(t),\ \mathbf{x}_{u,k}(t)\right]_{d\in \mathcal{D},\ k\in\mathcal{K}}\) denotes the caching decision vector of UAV \(u\), and \(\mathbf{I_{u}}(t)=(X_{u}(t),\ Y_{u}(t),\ Z_{u}(t))\) represents the location of UAV \(u\) at time slot \(t\), and to satisfy the altitude constraint (35), the vertical coordinate must obey \(H_{\min}\leq Z_{u}(t)\leq H_{\max}\). The action space of vehicle \(n\) contains its offloading decision, i.e.,  
\[a_{n}(t)=\{\mathbf{A}_{n}(t)\} \tag{42}\]  
where \(\mathbf{A}_{n}(t)=\{a_{u,n}(t)\}_{u\in\mathcal{U}}\). Reward Function (\(\mathcal{R}\)): The reward function \(R(s,a)\) quantifies the immediate reward obtained by all agents after taking joint action \(a\) in state \(s\). We aim to maximize the system efficiency, defined as below. The service coefficient related reward \(r_{1}(t)\) is defined as  
\[r_{1}(t)=\frac{Pr_{h}(t)}{\sum_{n=1}^{N}T_{n}(t)} \tag{43}\]  
The UAV energy consumption and minimum safe distance related reward \(r_{2,u}(t)\) is defined as  
\[r_{2,u}(t)=\left\{