## 5 Numerical Simulations  
In this section, we intend to present the performance evaluation of the proposed DQL-based power allocation scheme via numerical simulations. We consider a downlink OFDMA transmission with random distribution of BSs and mobile devices over multiple channels in multi-cell network as shown in Figure 2. It has to be noted that the path loss and channel coefficients are modeled using the standards of independent and identically distributed complex Gaussian noise, while the power of the additive noise at each receiver has been normalized. According to [53], the path loss related to distance is given by \(128.1+37.6\ \log_{10}d\ \mathrm{dB}\), where \(d\) is the distance between the serving BS and each receiver in kms. The maximum transmit power budget for BS over multiple channels is given as 33 dBm, which is reusable in multi-cells. For the downlink OFDMA, all users share a bandwidth \(B\) and each user occupies of \(B/N\), where \(B\) is given as 10 MHz and \(N\) is the number of receivers. The capacity requirement for each receiver is \(1\mathrm{b/s/Hz}\). The additive noise power at each user is \(-114\ \mathrm{dBm}\) and the noise PSD is set to \(-174\ \mathrm{Bm}\). We next define the training parameters for our DQN model. Determination of these hyperparameter values becomes more challenging in DL-based resource allocation [42]. In this work, we do not over-parameterize the structure of a neural network. Our goal is to accelerate the learning process, so we adopt a small architecture of deep neural networks for training purposes. To train a DQN, we use a deep neural network with three hidden layers, including 200, 100, and 40 neurons, where both the input and output layers are taken as 1. We also use the ReLU, \(f(x)=\max(0,x)\) as an activation function for the hidden layers. We assume that the \(F\) RBs are equally distributed between \(N\) channels, which exhibiting the same bandwidth \(B\). Each linear unit has \(F\) number of RB allocation that may be grouped in \(N\) channels. ReLU is used to avoid the vanishing gradient problem and allows the DQL model to learn faster and perform better [54]. We also normalize the input layer size of the Q-network with some parameters relying on the maximum total power constraint, inter-cell path loss, etc., to optimize the performance metrics. Furthermore, we use the \(RMSProp\) optimizer method with learning rate of 0.001 for updating the weights of the DQN function. Given the value of discount factor \(\gamma=0.5\) and updating the DQN over 3000 episodes. However, as the value of discount factor \(\gamma\) increases, the outcome of DQL keeps improving for most of its applications [35]. We also apply the \(e\)-greedy algorithm to facilitate the training of DQN network. Herein, the DQN agent takes action randomly with probability \(e\) and can control the search algorithm by adjusting the value of \(e\). Performance evaluation in terms of sum throughput maximization, power consumption and spectral capacity will be carried out to compare the proposed method with several methods in the simulations. The proposed method uses several benchmarks, such as a 'WMMSE' method developed in [5] and an 'ideal WMMSE' with instantaneous full CSI. Furthermore, we also use the downlink OFDMA with random power allocation method ('OFDMA random') and 'full power allocation' scenarios. In the case of 'OFDMA random', the DQL agent will choose its transmitted power for multiple channels at random between 0 and \(P_{max}\) while in the 'full power allocation', the agent will use the maximum transmit power for all channels. Figure 3 illustrates the sum throughput performance of different power allocation methods in a 5-user scenario with QoS threshold \(\gamma_{o}=3\). It can be seen from the figure that the sum throughput of the proposed DQN method and the state-of-the-art power allocation algorithms increases with an increasing of transmit power, which contributes to improving the achievable weighted sum rate performance.