## 4 Deep RL-Assisted Resource Allocation
### Proposed DQL-Based Approach  
In this subsection, we present our proposed approach that can perform the downlink resource allocation on multi-cell OFDMA systems. We consider a single DQL-based algorithm in which the agent makes interaction with the environment in order to learn the optimal power allocation of the users. The wireless environment contains everything in the OFDMA downlink transmission strategies except the agent. In the DQL algorithm, the serving BS in different cells is used as an agent \(i\) and interacts with the environment by mapping the observed state and performing possible actions. The agent \(i\) transmits confidential information to each user using the transmit power \(P_{i}\) and provides the required QoS guarantee. We denote the current state of agent \(i\) as \(s_{i}\in S_{i}\), which is comprised of environment features that are applicable to the possible actions \(a_{i}\in A_{i}\) of agent \(i\). As represented in Figure 2, at the time step \(t\), the two neighborhood BS sets are coupled together usingthe cross-coupling links \(h_{ij}(k)\). Observe that, even though the strategy sets of the BSs are dependent on each other, they are also cross coupled in the same fashion, i.e., by the same QoS conditions \(SINR_{i}(k)>\gamma_{o}\), for \(k=1,\ldots,K\). In this case, the interfering BS's transmit power \(P_{j}^{(t)}(k)\)\(h_{j\to i}^{(t)}(\text{k})\) will decrease due to its interference with neighboring BS-user pairs. Due to the cross coupling of BS-receiver pairs, our proposed DRL-based algorithm will be more complicated than the Nash equilibrium solution discussed in Section 2. Our aim is to measure the impact of each BS's interference on its neighbors' transmission systems, so we limit the exchange resources between neighboring BS-receiver pairs. Let the set of BSs whose \(SINR_{i}\) at user \(i\) is greater than a given threshold \(\gamma_{o}\) at each time step \(t\) be defined as  
\[I_{i}^{(t)}(\text{k})=\left\{j\in K,\;j\
eq i\backslash h_{j\to i}^{(t)}P_{j}^ {(t)}\right\}>\gamma_{o}{\sigma_{i}}^{2} \tag{16}\]  
where \(I_{i}^{(t)}(\text{k})\) denotes the set of \"interferers\" with respect to player \(i\) and \(h_{j\to i}^{(t)}P_{j}^{(t)}\) represents the received interference power from BS \(i\). Similarly, let the set of users whose SNR from BS \(i\) is also greater than the threshold value \(\gamma_{o}\) at each time step \(t\) be represented as  
\[U_{i}^{(t)}(\text{k})=\left\{j\in K,\;j\
eq i\backslash h_{i\to j}^{(t)}P_{i}^ {(t)}\right\}>\gamma_{o}{\sigma_{i}}^{2} \tag{17}\]  
From the neighboring BS-user pairs perspective, \(I_{i}^{(t)}(\text{k})\) is the set of \"interferers\" while in contrast \(U_{i}^{(t)}(\text{k})\) represents the set of the \"interfered\" neighbors' transmissions. Next, we consider the CSI the BS take over at each time step \(t\). Here, we firstly assume that the BS \(i\) learns via direct channel response \(h_{ij}^{(t)}(k)\) in the same cell. Moreover, the BS \(i\) also learns the corresponding received \(SINR_{i}\) at user \(i\) before the update of transmit power control, i.e., \(\sum_{j\
eq k,j\
eq i}^{K}h_{j\to i}^{(t)}P_{j}^{(t)}+{\sigma_{i}}^{2}(k)\). Moreover, at each time step \(t\), user \(i\) will inform interfering BS \(i\) of the received signal from its neighbors' transmissions \(j\in I_{i}^{(t)}(\text{k})\), i.e., \(h_{j\to i}^{(t)}P_{j}^{(t)}\). Note that these measurements can only be available at BS \(i\) at each time step \(t\). The objective of the training is to exploit a trained DQN to replay the experience in learning the power allocation strategy. We adopt DQL with experience replay to train our learning algorithm. The resource optimization scheme is performed in two phases, (1) in the learning or training stage, (2) in the testing or implementation phase. In the learning phase, the agent is in charge of the RB allocation and learns its best actions to achieve an optimal strategy by updating the RB allocated to each user.