## 3 JPRL-Based Joint Resource Allocation Approach
### P-MADDPG-Based Uplink Power Allocation
Considering the transition samples of all agents, we designed the P-MADDPG algorithm to enhance the MADDPG by customizing a prioritized experience replay technique, where the more important transition samples have a higher probability of being replayed to participate in network updating. Specifically, in each step \(t\), the transition samples of all agents are measured by the corresponding importance denoted by \(V_{t}\), which is combined with \(\mathcal{S}_{t}\), \(\mathcal{A}^{p}_{t}\), \(\mathcal{R}_{t}\), and \(\mathcal{S}_{t+1}\) to form a tuple \(\left(\mathcal{S}_{t},\mathcal{A}^{p}_{t},R_{t},\mathcal{S}_{t+1},V_{t}\right)\) being stored in \(\mathcal{D}2\). Similar to the MADDQN, the goal of P-MADDPG updating is to minimize the magnitude between the joint Q-value and target joint Q-value, i.e., joint temporal-difference (JTD) error. The transitions with the large JTD error contain more information and are more necessary to the update of neural networks. Thus, the JTD error is a reasonable proxy measure of important value, and \(V_{t}\) is written as  
\[V_{t}=|Y_{t}-Q^{p}_{\text{sum}}\Big{(}\mathcal{S}_{t},\mathcal{A}^{p}_{t}\mid \omega^{Q}\Big{)}|. \tag{16}\]  
However, in the sampling process, initially stored transition samples with small JIDs may not be sampled to replay if the sampling only relies on the importance. This can result in over-fitting, since the system lacks the sampling diversity of transitions. To avoid the issue, a probability associated with the importance is assigned to each transition sample, which can overcome the above issues effectively. The probability of the arbitrary transition sample \(\varphi\) at the step \(t\) is expressed as  
\[P^{\varphi}_{t}=\frac{\left(V^{\varphi}_{t}\right)^{\alpha}}{\left(\sum_{ \varphi=1}^{|\mathcal{D}2|}(V^{\varphi}_{t})\right)^{\alpha}}, \tag{17}\]  
where \(\alpha\in[0,1]\) is a contribution factor that controls the impact of importance. In particular, \(\alpha=0\) means that all samples are equally distributed, i.e., no contribution is made according to importance (uniform sampling). Original samples are equally probability-distributed in the replay buffer, but prioritized experience replay introduces bias, since it changes the original distribution by assigning different probabilities to the transitions. The compensation weight is thus introduced to correct this bias, which is expressed as  
\[\lambda^{\varphi}_{t}=\left(\frac{1}{D}\frac{1}{P^{\varphi}_{t}}\right)^{ \beta}, \tag{18}\]  
where \(\beta\in[0,1]\) is a hyperparameter, which regulates the degree of bias compensation. In particular, there is no compensation for non-uniform probabilities \(P^{\varphi}_{t}\) if \(\beta=0\); there is partial compensation if \(0<\beta<1\); and thre is full compensation if \(\beta=1\). As a result, The loss of a mini-batch \(\varphi\) after weight compensation is rewritten as \[L=\mathbb{E}_{(\mathcal{S}_{t},\mathcal{A}_{t}^{p},\mathcal{S}_{t+1},R_{t},V_{t}^{ \varphi})\sim\mathcal{D}2}\bigg{[}\lambda_{t}^{\varphi}\Big{(}V_{t}^{\varphi} \Big{)}^{2}\bigg{]}. \tag{19}\]  
```
1: Initialize: Initialize the network parameters in MADDQN and P-MADDPG respectively, \(\omega\), \(\omega^{Q}\); Initialize the replay buffer \(\mathcal{D}1\) and the prioritized replay buffer \(\mathcal{D}2\), \(|\mathcal{D}_{1}|\), \(|\mathcal{D}_{2}|\); Initialize a sum tree for \(\mathcal{D}2\), \(\alpha\), \(\beta\). 2:
3:for episode \(i=1,\cdots,I\)do
4: Receive initial observation state of all agents \(K\), and input \(s_{k}(t)\) to agent \(k\). 5: Initialize the actions of all agents. 6:for step \(t=1,\cdots,T_{i}\)do
7:for agent \(k=1,\cdots,K\)do
8:if random number \(\zeta<\epsilon_{t}\)then
9: Randomly choose \(a_{k}^{b}(t)\) from bandwidth allocation action space. 10:else
11:endif
12: Choose power allocation \(a_{k}^{p}(t)=[\pi(s_{k}(t)+\sigma_{t})]_{\mathrm{Pmin}}^{\mathrm{P}}\). 13: Execute actions \(a_{k}^{b}(t)\), \(a_{k}^{p}(t)\) and observe next state \(s_{k}(t+1)\). 14:endfor
15: Calculate reward with all agents' actions by Equation (8).