## 3 MAPPO-Based Joint Caching and Computation Offloading Scheme
### Dec-POMDP Framework
The action-value function is defined as \(Q_{v,u}^{\pi}(s(t);a(t))\), formulated as:  
\[Q_{v,u}^{\pi}(s(t);a(t))=\mathbb{E}\bigg{[}\sum_{i=1}^{\infty}\gamma_{v}^{i} \mathcal{R}_{v,u}(s_{t+i},a_{t+i})\mid s(t)=s,a(t)=a,\pi_{v}\bigg{]} \tag{56}\]  
To effectively gauge the comparative advantage of a specific action and concurrently mitigate variance and bias during the learning phase, we employ generalized advantage estimation (GAE) as a substitute for the advantage function-like term appearing in (53), in a manner that preserves generality. This approach bears resemblance to the use of surrogate objective functions and importance sampling techniques in conventional policy optimization methods. The policy gradient can thus be reformulated as:  
\[ \begin{split}\
abla_{\theta_{v}}P(\theta_{v}^{\text{now}})=& \mathbb{E}_{s(t),a(t)\sim\pi_{v}(a(t)|s(t);\theta_{v})}\bigg{]}\frac{\pi_{v_{v} ^{\text{now}}}(a(t)\mid s(t))}{\pi_{v_{v},\text{ave}}(a(t)\mid s(t))}\\ &\
abla_{\theta_{v}}\ln(\pi_{v}(a(t)\mid s(t);\theta_{v}))\hat{A }(s(t))\big{]}\end{split}  \tag{57}\]  
where \(\hat{A}(s(t))=\sum_{l=1}^{\infty}(\gamma_{v}\lambda)^{l}(r_{n}+\gamma_{v}V_{v }(s_{t+i+1})-V_{v}(s(t)))\). Furthermore, to constrain the magnitude of policy updates and thereby preclude excessive optimization steps that could compromise stability [57], we incorporate a clipping mechanism. The clip function serves to bind the ratio between the probabilities assigned by the new and old policies, ensuring that policy adjustments remain within an acceptable range. The definition of the clip function is given by:  
\[\text{clip}(x,1-\varsigma,1+\varsigma)=\left\{ \begin{array}{ll}1+\varsigma,&\text{if }x>1+\varsigma\\ x,&\text{if }1-\varsigma\leq x\leq 1+\varsigma\\ 1-\varsigma,&\text{if }x<1-\varsigma\end{array} \right. \tag{58}\]  
Here, \(\varsigma\) acts as a regularization parameter. Drawing upon concepts from (51), (55), and (56), we introduce \(\psi S_{t,u}\) to represent the policy entropy of the state. Consequently, the objective function for the actor network, which now incorporates the clip function, can be formulated as:  
\[ \begin{split} P(\theta_{v})=\mathbb{E}_{s(t)}\bigg{[}\min\bigg{[} \text{clip}\bigg{(}\frac{\pi_{v_{v}^{\text{now}}}(a(t)\mid s(t))}{\pi_{v_{v}^{ \text{now}}}(a(t)\mid s(t))},1-\epsilon,1+\epsilon\bigg{)}\hat{A}(s(t)),\\ \frac{\pi_{v_{v}^{\text{now}}}(a(t)\mid s(t))}{\pi_{v_{v}^{\text{now }}}(a(t)\mid s(t))}\hat{A}(s(t))\bigg{]}+\psi S_{t,v}\bigg{]}.\end{split}  \tag{59}\]  
In parallel, to update the critic network, which functions as a value estimator employed to evaluate the effectiveness of the actor network's policies, we specify its loss function as:  
\[P(\omega_{v})=\mathbb{E}_{s(t),a(t)}\Big{[}((1-\gamma)\mathcal{R}(s(t)\mid a(t ))+\gamma\mathcal{V}_{\omega_{v}}(s(t+1))-V_{\omega_{v}}(s(t)))^{2}\Big{]} \tag{60}\]Here, \(V_{\omega_{v}}(s(t))\) denotes the approximation of the state-value function produced by the value network with parameters \(\omega_{v}\), and \(\hat{V}_{\omega_{v}}(s(t+1))\) represents the estimated value of the subsequent state after the agent performs an action in the current state. Subsequently, we execute gradient descent to adjust the parameters \(\omega_{v}\) under the influence of the learning rate \(\varphi\), yielding the updated parameters:  
\[\omega_{v}^{\text{new}}\ \leftarrow\ \omega_{v}^{\text{now}}\ -\varphi\
abla_{ \omega_{v}}P(\omega_{v}^{\text{now}}\ ) \tag{61}\]