## 4 Simulation Results
### Performance Analysis
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Method Index** & **Average Reward** & **Average Energy Efficiency** \\ \hline (A) & 0.8957 & 6.1770 \\ \hline (B) & \(-\)0.7453 & 2.5677 \\ \hline (C) & \(-\)0.7499 & 2.6169 \\ \hline (D) & 0.1968 & 4.6477 \\ \hline (E) & \(-\)0.1327 & 3.3923 \\ \hline (F) & 0.1186 & 3.4773 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The average standardized value of the reward and the energy efficiency, obtained from the results in Figure 4 and averaged over all time steps. The method index in the table is: (A) P-DQN with Cluster Size Constraint, (B) Nearest SBS + Random Power, (C) Best Channel + Random Power, (D) P-DQN Association + Random Power, (E) Random Association + P-DQN Power, and (F) DQN with Cluster Size Constraint. The increased level of interference in UEs (such as UE 3 and UE 4) observed in Figure 4(c) can be explained with the aid of Figure 6, which plots the locations of each UE and SBS with their association status being specified by colors. The association results and physical distances between each user and all SBSs in Figure 6 can provide insights into the interference level each UE experiences under different association strategies. For example, the increased interference in UE 4 in Figure 4(c) can be analyzed by comparing Figure 5(b) with Figure 5(a). More specifically, while UE 4 in Figure 5(a) associated with SBS 2 under the proposed P-DQN with cluster size constraint is interfered only by inter-cluster signals transmitted from SBS 1, this UE 4 associated with SBS 2 in Figure 5(b) under the \"Nearest SBS\" (i.e., nearest distance) association approach can potentially be interfered with by inter-cluster signals from SBS 0 and SBS 1. It can be seen from Figure 5(b) that SBS 0 could strongly interfere with UE 4, due to the short distance between SBS 0 and UE 4, thus leading to the increased interference level in UE 4 as shown in Figure 5(b). Finally, as for the case of P-DQN approach without cluster size constraint, the system suffers from intra-cluster interference, which significantly impacts the interference level in each user as shown in Figure 4(b). Since keeping an SBS active demands huge operational power, we can see from Figure 5(c) that the P-DQN without a cluster size constraint tends to have all UEs served only by one SBS in pursuit of less overall system power consumption. In contrast with the user association schemes based on the distance or the channel quality between a UE and an SBS, the P-DQN-based user association tends to activate fewer SBSs, which leads to less consumption of overall operational power in the SBSs and results in a higher energy efficiency. More specifically, as illustrated in Figure 5(a), where different colors refers to different clusters, we see thatUE 0, UE 2 and UE 3 are associated with SBS 1, and UE 1 and UE 4 are connected to SBS 2, whereas SBS 0 is not active when employing the policy learned by P-DQN with the cluster size constraint. Figure 5: _Cont._  
Figure 5: Average interference level of all power allocation and user association approaches. P-DQN association and P-DQN power refer to those of “P-DQN with Cluster Size Constraint”. Figure 6: The results of user association. (**a**) P-DQN with cluster size constraint. (**b**) The nearest SBS policy. (**c**) P-DQN without cluster size constraint. Each SBS and its associated UEs are shown in the same color.