## 3 JPRL-Based Joint Resource Allocation Approach
### P-MADDPG-Based Uplink Power Allocation  
For power allocation, a huge action space is not helpful for exploitation. In addition, although the discrete DRL algorithms can quantize power, they ignore the diversity of power choices. To this end, a novel P-MADDPG algorithm was proposed to solve the transmit power allocation subproblem. This is an enhancement of the DDPG with a prioritized replay buffer. In contrast to the power quantization, the P-MADDPG directly outputs the power of all the users in a continuous domain with infinite choices. Furthermore, by applying the prioritized replay buffer, it is more sensitive to the negative effect of the bad actions than the general MADDPG algorithms. Similar to DDPG, an actor-critic architecture [43] applies for learning and training; both the actor and critic networks of each agent contain two identical neural networks, which are named the online network and target network, respectively. For a multi-agent system, the actor network of agent \(k\) outputs the power allocation under the current state through a policy \(\pi\), i.e., \(a_{k}^{p}(t)=\pi(s_{k}(t))\). However, the inherent exploration-exploitation dilemma in the DRL is prevalent for an inflexible action policy. By taking advantage of the DQN, it is balanced by a stochastic noise whose function is similar to the \(\epsilon-greedy\) mechanism. Consequently, the actions of all agents are written as  
\[\mathcal{A}_{t}^{p}=[\pi(\mathcal{S}_{t}\mid\omega^{\mu})+\Sigma_{t}]_{P_{\text {min}}}^{P_{\text{max}}}, \tag{13}\]  
where \(\omega^{\mu}\) is the weight of the actor network, and \(\Sigma_{t}\) follows a Gaussian distribution \(\mathcal{N}(0,\varrho)\); \(\varrho\) is the variance of Gaussian noise and decreases linearly to zero as the iteration proceeds. Similarly, applying the individual action-value function to each agent ignores the features of others, which reduces the learning stability and weakens agent interaction. To this end, the critic network uses the joint action-value function \(Q_{\text{sum}}^{p}(\mathcal{S}_{t},\mathcal{A}_{t})\) to evaluate all actions. The specific \(Q_{\text{sum}}^{p}\) is defined as  
\[Q_{\text{sum}}^{p}(\mathcal{S}_{t},\mathcal{A}_{t})=\mathbb{E}_{R_{t}, \mathcal{S}_{t}\sim\mathcal{D}2}[R_{t}+\gamma_{2}Q_{\text{sum}}^{p}(\mathcal{S }_{t+1},\pi(\mathcal{S}_{t+1}))], \tag{14}\]  
where \(\mathcal{D}2\) is the experience replay buffer, and \(\gamma_{2}\in(0,1]\) is a discount factor. According to the deterministic policy gradient theorem, the action-value function \(Q_{\text{sum}}^{p}\) is used to update the actor parameters \(\omega^{\mu}\) in the direction of increasing the cumulative discounted reward with \(D\) samples of a mini-batch, that is  
\[ \begin{split}\
abla_{\omega^{\mu}}\pi\approx&\mathbb{E} _{\pi^{\prime}}[\
abla_{\omega^{\mu}}Q^{p}_{\text{snum}}(\mathcal{S},\mathcal{A} \mid\omega^{Q})\mid_{\mathcal{S}=\mathcal{S}_{t},\mathcal{A}=\pi(\mathcal{S}_ {t}\mid\omega^{p})}],\\ =&\mathbb{E}_{\pi^{\prime}}[\
abla_{\omega^{\mu}}Q^{ p}_{\text{snum}}(\mathcal{S},\mathcal{A}\mid\omega^{Q})\mid_{\mathcal{S}= \mathcal{S}_{t},\mathcal{A}=\pi(\mathcal{S}_{t})}\
abla_{\omega^{\mu}}\pi( \mathcal{S}\mid\omega^{\mu})\mid_{\mathcal{S}=\mathcal{S}_{t}}],\\ =&\frac{1}{D}\sum_{k}\
abla_{a^{p}_{k}(t)}Q^{p}_{ \text{snum}}\left(\mathcal{S}_{t},\mathcal{A}^{p}_{t}\mid\omega^{Q}\right)| \
abla_{\omega^{\mu}}\pi(\mathcal{S}_{k}(t)\mid\omega^{\mu})|_{\mathcal{S}_{k} (t)},\end{split}  \tag{15}\]  
where \(\omega^{Q}\) is the weight of critic network. A common method for training neural networks is to randomly and uniformly sample mini-batches from the buffer D2, which often results in a high probability of selecting bad actions among the vast combinations of different actions, thereby lowering performance. This method is inefficient and poorly helpful for guiding the networks to update in the correct direction.