## 4 Simulation Results
### Performance Analysis
Note that, while the P-DQN approach without cluster size constraint provides higher reward as can be seen from Figure 3, it cannot guarantee all UEs' QoS. Figure 4 shows the effectiveness of the proposed P-DQN algorithm, as it outperforms other approaches in terms of both the reward and the energy efficiency. The numerical values of the reward and energy efficiency obtained in Figure 4 and averaged over all time steps are summarized in Table 5. The improvement in the energy efficiency of the proposed P-DQN on average may reach 77.6% over the traditional DQN and 140.6% over the nearest-distance-based association approaches. Figure 6c, the P-DQN without the cluster size constraint tends to have all UEs served only by one SBS in the pursuit of small-system power consumption (as large operational power is consumed by any active SBS). Figure 3: Convergence property of the proposed P-DQN algorithm. Figure 4: Performance of the proposed reward function. (**a**) Reward. (**b**) Energy efficiency. Table 6 shows that each UE obtains 83.47% of the required QoS, and each SBS obeys the wireless backhaul capacity constraint at each timeslot through the proposed P-DQN with the cluster size constraint. However, the percentage of QoS satisfaction for the methods with random power or random association is about 50%. The percentage of QoS satisfaction of the proposed P-DQN with the cluster size constraint is higher than that of the other approaches, as a penalty term is used in the reward function to realize each UE's QoS as much as possible in the proposed P-DQN. For Random Association + P-DQN Association, the ratio is even smaller than 5%. For DQN with cluster size constraint, the ratio is even smaller than 1%, as the quantized power levels suffer from round-off imperfections, while the continuous action learned by the P-DQN allows for a better QoS with appropriate continuous power allocation. In Figure 5, we analyze the interference power experienced by each UE under various approaches. The results in Figure 5a,c,e demonstrate that the interference each user experiences in the \"Nearest SBS + Random Power\" approach is higher than that in the \"P-DQN with Cluster Size Constraint\" approach and \"P-DQN Association + Random Power\" approach. By comparing the results in Figure 5a,e, it can be seen that random power allocation does not have much impact on each user's received interference level. On the other hand, by comparing the results in Figure 5c,e, we observe a noticeably increased interference level in UE 3 and UE 4, which implies that the interference is largely dominated by the result of user association. The results here indicate that the user association policy learned by the proposed P-DQN with cluster size constraint generally tends to determine the matching between UEs and SBSs such that the inter-cluster interference can be managed to a lower level, as shown in Figure 5a, which results in a higher system throughput. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Percentage of Backhaul** & **Percentage of** \\
**Index** & **Constraint Satisfaction** & **QoS Satisfaction** \\ \hline (A) & 100.00\% & 83.47\% \\ \hline (B) & 100.00\% & 44.91\% \\ \hline (C) & 100.00\% & 44.42\% \\ \hline (D) & 100.00\% & 50.22\% \\ \hline (E) & 100.00\% & 4.87\% \\ \hline (F) & 100.00\% & 0.58\% \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The percentage of wireless backhaul link constraint satisfaction and QoS satisfaction. The method index in the table is as follows: (A) P-DQN with Cluster Size Constraint, (B) Nearest SBS + Random Power, (C) Best Channel + Random Power, (D) P-DQN Association + Random Power, (E) Random Association + P-DQN Power, and (F) DQN with Cluster Size Constraint.