## 4 Simulation Results
### Performance Analysis  
In the simulations, we compare the proposed P-DQN for joint user clustering and power allocation with the following approaches:  
* **Nearest SBS + Random Power**: Each UE is associated with the nearest SBS. Random power means that each SBS serves the UEs in its cluster with random powers in a way that the resulting sum rate within the cluster cannot exceed the power and backhaul capacity limit. * **Best Channel + Random Power**: Each UE is associated with the SBS with the best received signal power, which depends on the UE-SBS distance as well as the small-scale fading effect. Furthermore, each SBS serves the UEs in its cluster with random power allocations under the power and backhaul capacity constraint. \begin{table}
\begin{tabular}{l l l} \hline \hline  & **Actor** & **Actor-Parameter** \\ \hline Learning rate & \(\alpha_{a}=10^{-5}\) & \(\alpha_{a,p}=10^{-5}\) \\ \hline Exploration & \(\epsilon\)-greedy & Ornstein-Uhlenbeck noise \\ \hline Number of Outputs & \(|\mathcal{A}_{d}|\) & \(K\cdot|\mathcal{A}_{d}|\) \\ \hline Hidden layer & ReLu, 16 & ReLu, 256 \\  & ReLu, 512 & \\ \hline Number of Inputs & \(K+K\cdot|\mathcal{A}_{d}|\) & \(K\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The settings for the deep neural networks used in the P-DQN. \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Value** \\ \hline Carrier frequency & \(f_{c}=2\,\mathrm{GHz}\) \\ \hline Subchannel bandwidth & \(B_{sub}=15\,\mathrm{kHz}\) \\ \hline Number of subchannels & \(N_{sub}=3\) \\ \hline Number of subchannels per user & \(|\mathcal{F}_{k}|=1\) \\ \hline MBS antenna array size & \(N_{T}=100\) \\ \hline MBS beamforming group size & \(N_{g}=20\) \\ \hline The radius of the entire network & \(500\,\mathrm{m}\) \\ \hline Number of SBS & \(J=3\) \\ \hline Number of UE & \(K=5\) \\ \hline SINR threshold of UE & \(\
u_{k}=1\) for each UE \\ \hline Path loss model & a. \(30.53+36.71\times\log_{10}d_{j,k}\) \\ (a. and b. indicate the model & in dB, \(d_{j,k}\) in km \\ for UE and SBS, respectively) & b. \(19.77+3.91\times\log_{10}d_{j}\) \\  & in dB, \(d_{j}\) in km \\ \hline Rayleigh channel coefficient & \(h\sim\mathcal{CN}(0,1)\) \\ \hline Noise power spectral density & \(N_{0}=-174\,\mathrm{dBm}/\mathrm{Hz}\) \\ \hline Maximum transmit power of SBS & \(P_{SBS,max}=24\,\mathrm{dBm}\) \\ \hline Maximum cluster size & \(|\mathcal{C}_{j}|_{max}=N_{sub}=3\) \\ \hline Transmit power of MBS & \(P_{MBS}=43\,\mathrm{dBm}\) \\ \hline Operational power of SBS & \(P_{o,SBS}=0.5\,\mathrm{W}\) \\ \hline Operational power of MBS & \(P_{o,MBS}=130\,\mathrm{W}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Simulation Parameters. * **P-DQN Association + Random Power**: The user association policy is accomplished by the proposed P-DQN, whereas each SBS serves the UEs in its cluster with random powers under the total power and backhaul capacity constraint. * **Random Association + P-DQN Power**: Each SBS allocates the power to its serving UEs based on the policy determined by the P-DQN. Each UE is randomly associated with one SBS in such a way that the random association policy obeys the backhaul link constraint. * **DQN with Five Discrete Power Levels**: The continuous power space is quantized into \(L\) non-uniform power intervals, with \(L\) discrete power levels \(\frac{P_{\text{SBS}_{j},max}}{10^{\mathcal{L}}}\) with \(\mathcal{L}\in\{0,\ldots,L-1\}\). In this simulation, \(L\) is set to 4. We set the weights \((\lambda_{1},\lambda_{2})=(0.43,0.16)\) for both the P-DQN with the cluster size constraint and the P-DQN without the cluster size constraint, which allows us to observe the effect of the cluster size constraint in \(C_{4}\). Figure 3 depicts the average normalized reward versus steps over 500 realizations during the training phase. It shows the convergence of the user association and power allocation algorithm using the proposed P-DQN.