# A Turbo Q-Learning (TQL) for Energy Efficiency Optimization in Heterogeneous Networks
## 3 System Model  
We consider a two-layer HetNets scenario as shown in Figure 1, in which a cell contains the macro base stations (MBSs) and SBSs. The SBSs are randomly deployed within the coverage of MBSs. The sets of the SBSs and the MBSs are denoted as \(S\) and \(M\), respectively. The users (UEs) randomly enter the cell. According to a set of UEs association with BSs, UEs are divided into SBS UEs (SUEs) and MBS UEs (MUEs) who are associated with SBSs and MBS, respectively. In order to balance the load of the entire system network and reduce cross-layer interference by offloading the users of MBSs to SBSs, the enhanced Inter-cell Interference Coordination (eICIC) technology was proposed with two important parameters as ABS and CRE according to [34]. To reduce signal interference, MBSs and SBSs use radio resources in different time periods (subframes) according to eICIC. A frame is divided into some sub frames as ABS and non-ABS subframes, and MBSs normally transmit normal power in nABS (non-ABS) subframes and keep silent or transmit low power in ABS subframes, where the ratio of ABS in a frame is donated as \(\alpha\). The SBSs keep normal transmit power in the whole frames. In the time domain, since the MBSs are allowed to be muted in an ABS subframe period, the interference of the MBSs to the users serviced by SBSs is reduced. Therefore, the SINR  
Figure 1: HetNets system model. of UEs with poor channel condition is improved since there is no interference from MBSs in these ABS subframes. In general, the power of MBSs is much higher than that of SBSs, and some UEs should be accessed to the MBSs according to the reference signal receiving power (RSRP). This is because the UEs in LTE networks are associated with the BSs based on RSRP policy where the UEs are connected to the BSs with the highest reference signal. To balance load and improve the system capacity, SBSs are designed to enhance the frequency multiplexing of the network. CRE was proposed to support SBSs to extend their coverage by adding a bias to their RSRP in which users outside the edge of the SBSs can be connected to the SBSs. The UEs located in the extended area of the SBSs receive less interference from MBSs in ABS subframes and get better channel gain to improve their SINR. Due to the two operating modes of the MBSs in the ABS subframes and non-ABS subframes (nABS), there are also two interference modes for the UEs in the downlink in the HetNets. When the MBSs are in the ABS subframes, the UEs receive only the signal transmitted by the SBSs. However, the MBSs are in the nABS subframes, and the UEs are interfered by the transmitted signal from the SBSs and the MBSs. The \(\textit{SINR}_{k,n}\) of the UE \(n\) connected to the MBS \(k\) can be expressed as  
\[\textit{SINR}_{k,n}=\left\{ \begin{array}{ll}\frac{P_{M}^{k,n}G_{k,n}}{ \sum\limits_{j\in\mathcal{M},j\
eq k}P_{M}^{j,n}G_{j,n}+\sum\limits_{i\in S}P_ {S}^{j,n}G_{i,n}+N_{0}}&k\in M^{nABS}\\ \frac{P_{m}^{k,n}G_{k,n}}{\sum\limits_{j\in\mathcal{M},j\
eq k}P_{m}^{j,n}G_{ j,n}+\sum\limits_{i\in S}P_{S}^{j,n}G_{i,n}+N_{0}}&k\in M^{ABS}\end{array} \right. \tag{1}\]  
where \(P_{M}^{k,n}\) is the transmission power from the MBS \(k\) to the UE \(n\) in the nABS subframes, \(P_{m}^{j,n}\) is the transmission power from the MBS \(j\) to the UE \(n\) in the ABS subframes, \(G_{k,n}\) represents the channel gain from the MBS \(k\) to UE \(n\), \(P_{S}^{j,n}\) denotes the transmission power from the SBS \(j\) to the UE \(n\), \(N_{0}\) indicates noise variance of the additive white Gaussian, \(M^{ABS}\) and \(M^{nABS}\) are denoted as MBSs in the ABS and nABS subframes periods, respectively. Note that \(m\) and \(M\) are short for \(M^{ABS}\) and \(M^{nABS}\), respectively. The \(\textit{SINR}_{k,n}\) of the UE \(n\) connected in the SBS \(k\) can be written as  
\[\textit{SINR}_{k,n}=\left\{