## 4 Deep RL-Assisted Resource Allocation
### Proposed DQL-Based Approach
#### 4.2.3 Reward Function  
Reinforcement learning (RL) aims to solve problems that are hard to optimize using the traditional framework. This is typically tackled by considering the overall goal of the problem and designing the reward signal that correlates with the ultimate goal. In our proposed algorithm, the target function is defined to evaluate the agent's actions and outputs the result in a positive reward or penalty. Here, the maximum throughput of the system corresponds to a positive reward. Moreover, a higher received signal for each user benefits to improve the system throughput and award. On the other hand, to ensure the QoS requirement, the achievable rate of each user that satisfies the desired threshold aids to achieve a good reward, while the achievable rate that does not fulfil the required outcomes is given a negative reward or penalty. Taking into account the above points, we explain the reward function as how the strategy \(P_{i}^{(t)}\) affects the sum rate at time step \(t\). As described in (6), we set the design objective to maximize the sum throughput while ensuring the QoS conditions for each user. At each time step \(t\), for all agent \(i\in K\), the training network computes the achievable rate of each user \(i\) without the coupling interference from BS \(i\). Accordingly, we can describe the network trainer function for all users \(\mathcal{U}_{i}\) at time period \(t\) as follows:  
\[R_{i}^{(t)}(\text{k})=\log_{2}\left(1+\frac{P_{i}^{(t)}(k)\big{|}h_{i\to i}^{( t)}(k)\big{|}^{2}}{\sum_{j\
eq k}^{K}h_{j\to i}^{(t)}P_{j}^{(t)}+\sigma_{i}^{2}(k)}\right) \tag{19}\]  
Furthermore, the Q-network trainer estimates the cross coupling plus noise term (i.e., \(\sum_{j\
eq k,j\
eq i}^{K}h_{j\to i}^{(t)}P_{j}^{(t)}+\sigma_{i}^{2}(k)\)) in (18) by simply subtracting \(h_{i\to i}^{(t)}P_{i}^{(t)}\) from the total \(SINR\) power term in time step \(t\). Thus, the reward function in the absence of QoS at time step \(t\) is given by  
\[r_{i}^{t}=w_{i}^{t}\Big{(}{R_{i}}^{(t)},\ldots,{R_{N}}^{(t)}\Big{)} \tag{20}\]  
In addition, we also consider the reward signal design intending to optimize the overall capacity of the entire system under the relevant QoS conditions. The spectralefficiency of each BS \(i\) is its rate \(R_{i}^{(t)}(\mathrm{k})\), and each BS has to satisfy the constraints of power budget and QoS requirements. As mentioned in Problem (16), since the BS \(i\in I_{i}^{(t)}(\mathrm{k})\), its interference channel \(K\) in time step \(t\), i.e., \(h_{j\to i}^{(t)}p_{i}^{(t)}>\gamma_{o}\sigma_{i}^{2}\) is perfectly measurable by user each user \(i\) and can be generated to the training DQN. At each time step \(t\), we consider each BS's interference on its neighbors' transmissions that an interfering BS \(i\in I_{i}^{(t)}(\mathrm{k})\) causes to its neighboring BS-user pairs using interference pricing. Accordingly, the impact of each BS's interference on its neighbors' transmissions can be given by [52]\(\mathcal{U}\)  
\[\pi_{\bar{i}}^{(t)}=w_{i}^{t}\Big{(}R_{\bar{i}}^{(t)}(\mathrm{k})-R_{i}{}^{(t) }\Big{)} \tag{21}\]  
In this paper, we assume that \(w_{i}=1\) to maximize the sum throughput. To ensure that the QoS constraint is satisfied, the condition \(SINR_{i}(k)>\gamma_{o}(k)\) is checked by agent \(i\in K\) in the reward estimation. If the QoS does not meet this requirement, the algorithm simply declares the power \(P_{i}^{(t)}\) selection of that user as wrong and sets the reward to zero. Therefore, the QoS constrained reward design at time step \(t\) can be formulated as  
\[R_{i}^{t}=\left\{\begin{array}{c}\bigg{(}R  
The DQN takes the current observable state \(s^{(t)}\) as input and outputs the agent's possible action-value. Multiple episodes are executed to train the DQN function and accumulate experiences with its connection with the environment. Since we utilize a single DQN agent scenario, the state transition of the environment relies upon the agent's possible actions.