## 3 JPRL-Based Joint Resource Allocation Approach  
The detailed structure of the joint uplink bandwidth and transmit power allocation is shown in Figure 2. Joint resource allocation often optimizes multiple variables consistently. However, for the problem of the joint allocation of the bandwidth and transmit power, there exist infinite combinations of joint assignment schemes that are influenced by the users interactions, thereby leading to unfortunate performance. In addition, the bandwidth assignment with limited choices is a discrete assignment scheme, rather than the continuous range such as for the power allocation. Thus, we de-coupled problem P1 into two sub-problems and designed an efficient JPRL method to solve the joint resource allocation problem in the considered large-scale multi-cell multi-user system. Specifically, the MADDQN algorithm was developed to solve the bandwidth allocation sub-problem with a discrete action space, and the P-MADDPG algorithm was designed to solve the transmit power allocation subproblem in the continuous domain. This resource assignment procedure satisfies the decentralized partially observable Markov decision process. Therefore, the proposed JPRL based on the RL method employed each user as an agent to model the optimization, which could solve large-scale resource allocation while meeting QoS constraints. The RL can be described as a stochastic game, which is defined by a tuple \(\langle\mathcal{K},\mathcal{S},\mathcal{A},R,P\rangle\), where \(\mathcal{K}\) is the set of agents, and \(\mathcal{S}\) and \(\mathcal{A}\) denote the set of states and the joint actions the space of all agents, respectively. The \(R\) is the reward function, and \(P\) is the state transition probability. The game is generally concerned with the interaction between the environment and one or more agents in a series of iterations. In each iteration, the agent observes the environmental state \(\mathcal{S}\) to take action from action space \(\mathcal{A}\). Thenm the agent receives an immediate reward \(R_{t}\) to reflect the quality of this iteration and observes a new state to the next step. Our goal was to maximize of the long-term rewards over various iterations. The details of the proposed framework are illustrated as follows. * Agent: All users \(K\). * State space: The state \(s_{k}(t)\) of agent \(k\) is denoted as its co-channel interference, and the global environment state is thus defined as a set including the state of all agents, i.e., \[ \begin{split}\mathcal{S}_{t}&=\{s_{1}(t),\dots,s_{ k}(t),\dots,s_{K}(t)\},\\ &=\{I_{l_{1,2}}(t),\dots,I_{l_{1,n}}(t),\dots,I_{l_{M,N}}(t)\}. \end{split} \] (6)
* Actions space: The actions of each agent consist of the bandwidth and power allocation and can be expressed as \[\mathcal{A}_{t}=\{\left(a_{1}^{b}(t),a_{1}^{p}(t)\right),\dots,\left(a_{K}^{b} (t),a_{K}^{p}(t)\right)\},\] (7) where \(\mathcal{A}_{t}^{b}=\{a_{1}^{b}(t),\dots,a_{K}^{b}(t)\}\) is defined as the bandwidth allocation, and \(\mathcal{A}_{t}^{p}=\{a_{1}^{p}(t),\dots,a_{K}^{p}(t)\}\) is defined as the power allocation of all agents. * Reward function: Since the whole performance is influenced by all users in the considered system, the sparse reward is a serious issue. Inspired by the entire long-term evaluation mechanism, in the learning process, previous lessons are indicative of the current learning. Therefore, a novel reward function is defined as \[R_{t}=\overline{T}H_{t}-\overline{T}H_{t,\tau}-c,\] (8) where \(\overline{T}H_{t}\) denotes the average throughput of the current step \(t\), \(\tau\) denotes the moving step, and \(\overline{T}H_{t,\tau}=\frac{1}{\tau}\sum_{\tau=1}^{\tau}\big{(}\overline{T}H _{t-\tau+1}\big{)}\) is the moving average of \(\overline{T}H_{t}\). \(c\) is a non-negative value. Especially, \(c=0\) if constraint C3 of Problem P1 is satisfied for all users; otherwise, \(c>0\).