# A Turbo Q-Learning (TQL) for Energy Efficiency Optimization in Heterogeneous Networks
## 6 Numerical Simulation  
The parameters of experimental simulation are set according to the 3 GPP LTE-A HetNets framework [37], and the wireless channel is modeled as deterministic path loss attenuation and random shadow fading models. In this part, the scenario we deployed is that each MBS covers the users in a 120\({}^{\circ}\) cell as shown as shadow part in Figure 4 and is interfered by three other MBSs. We deployed a field where six SBSs are randomly deployed within the coverage of the MBS in the green shaded part of Figure 4 and select working mode according to load conditions. The coverage radius of the MBS and the SBSs are 500 m and 100 m, respectively. The thermal noise power is \(-\)176 dBm, the system spectrum bandwidth is 10 MHz and the antenna gains of the MBSs, SBSs, and the UE are 14 dBi, 5 dBi, and 0 dBi, respectively. The maximum transmission power of the MBSs and the SBSs are set to 46 dBm and 30 dBm, respectively. The probability of a user entering a cell to access a MBS and a SBS are 1/3 and 2/3, respectively. The proportion \(\varphi\) of the BSs that wake up is set here to 0.5. Although LTE frame includes 10 subframes, the ABS mode has a periodicity of eight subframes. The ABS ratio of protected subframe to traditional subframe \(\alpha\in[0,1]\) belongs to the set  
Figure 3: The inputs are the users’ position, labels are the optimal action in each state. Figure 2: Multi-stage decision process. \(\{0/8,...,7/8\}\). CRE is denoted by \(\beta\in B\), where \(\text{B}=\{0,6,9,12,15,18\}\). The specific parameters are listed in Table 1.  
Figures 5 and 6 show the relationship during iterations and accuracy of Q-Learning algorithm and our TQL algorithm where learning rate, discount factor, and greed rate are all set to 0.1, and the number of users are set to 50, 100, 150, and 200, respectively. Figure 5 shows that the tabular method of the Q-Learning algorithm converges after \(80\times 1000=800\),000 iterations under different load conditions. Our proposed TQL algorithm converges after \(800\times 400=320\),000 iterations as shown in Figure 6. We can see that the convergence speed of our proposed TQL algorithm is increased by about 60% compared with the Algorithm 1. Note that the convergence speed of TQL algorithm proposed in this paper is still much faster than that of Algorithm 1, especially in the case where the action space cardinality is very large from the analysis of Algorithm 5. \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Value** \\ \hline Macro base station radius & 600 m \\ Small base station radius & 100 m \\ Minimum distance between small \\ base station & 40 m \\ Minimum distance from macro base & 70 m \\ station to small base station & 10 MHz \\ System bandwidth B & Normal: 46 dBm; Sleeping: 12 dBm; \(N_{TRX}=6\) \\ Transmit power of SBS & Normal: 30 dBm; \(N_{TRX}=2\) \\ Noise power density \(N_{0}\) & \(-174\) dBm \\ Path loss from MBS to UE & \(128.1+37.6\times log_{10}(R[\text{km}])\), \(R\) is the distance between MBS and UE \\ Path loss from SBS to UE & \(149.1+37.6\times log_{10}(R[\text{km}])\), \(R\) is the distance between MBS and UE \\ Carrier frequency & 2.0 Ghz \\ File size & 0.5 mbytes \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation Parameters  
Figure 4: Simulation scenario. Figure 7 shows that the comparison of sub-QLs (Algorithms 2-4) iterations in the TQL algorithm give different results where relative parameters are the same as that in analysis of Figure 5 and the number of users is 100. Take the greed line and red line as examples which indicate the number of sub-QL iterations of SBSs (_threshold\({}_{3}\)_ in Algorithm 4), ABS (_threshold\({}_{1}\)_ in Algorithm 2), and CRE (_threshold\({}_{2}\)_ in Algorithm 3) are all set to 400 and 2000, respectively. When the iteration number of TQL represented by the green line and red line is more than about 75 and 30, our proposed TQL algorithm is convergent, but the final accuracy rates are about 98% and 90%, respectively.