# A Turbo Q-Learning (TQL) for Energy Efficiency Optimization in Heterogeneous Networks
## Solution with a Tql Algorithm
### Tql Algorithm
```
0:\(\alpha\in\text{A}\), \(\beta\in\text{B}\), \(|S|=\{0,1,...,|S|_{\max}\}\), Reward \(r\), Learning rate \(\lambda=\{\lambda_{1},\lambda_{2},\lambda_{3}\}\), Greedy probability \(\varepsilon=\{\varepsilon_{1},\varepsilon_{2},\varepsilon_{3}\}\) and Discount factor \(\gamma=\{\gamma_{1},\gamma_{2},\gamma_{3}\}\). 0: Optimal action configuration \(\{\alpha,\beta,|S|\}\) in each state. 1: Initialize \(U_{t}\), \(\alpha\), \(\beta\), \(|S|\),
2:while\(n<=threshold_{4}\)do
3: Fixed the CRE bias \(\beta\) and the number of SBS activation \(|S|\), calculate the ABS ratio \(\alpha\) according to Algorithm 2. Pass the solved \(\alpha\) to step (4) and step (5);
4: Fixing the ABS ratio \(\alpha\) and the number of SBS activation \(|S|\), calculate the CRE bias \(\beta\) according to Algorithm 3. Pass the solved \(\beta\) to step (3) and step (4);
5: Fix the ABS ratio \(\alpha\) and the CRE bias \(\beta\), calculate the number of SBS activation \(|S|\) according to Algorithm 4. Pass the solved \(|S|\) to step (4) and step (3);
6:\(n=n+1\);
7:endwhile
8: Output: \(\alpha\), \(\beta\), \(|S|\);
```  
**Algorithm 5**The algorithm for optimizing initial problems. In summary, our scheme has changed the size of action space from traditional exponential increase as \(X=(|P|+1)\times|\text{A}|\times|\text{B}|\) to linear increase as \(X=(|P|+1)+|\text{A}|+|\text{B}|\), which greatly reduces the dimension and size of the action space. Algorithm 5 can be considered as a multi-stage decision process optimization problem which is shown in Figure 2. The action spaces of the third, fourth, and fifth step in Algorithm 5, which are the optimization problem of Algorithms 2-4, are set to \(\mathcal{A}\), \(\mathcal{B}\), and \(\mathcal{C}\) where \(\mathcal{A},\mathcal{B},\mathcal{C}\) are limited and denoted as set \(A\), \(B\), and \(\{0,1,...,|P|\}\), respectively. It can be seen in Figure 2 that the state spaces of the third, fourth, and fifth step in Algorithm 5 are the Cartesian product of the other two action spaces and the state spaces size are \(|\mathcal{B}|\left|\mathcal{C}\right|\), \(|\mathcal{A}|\left|\mathcal{C}\right|\) and \(|\mathcal{A}|\left|\mathcal{B}\right|\), respectively. The state transition equation refers to the transition probability from state \(b_{i}c_{i}\) to state \(a_{i}b_{i}\) conditioned on taking action \(a_{i}\). The state transition probability is written as \(P\left(s^{{}^{\prime}}=a_{i}b_{i}|s=b_{i}c_{i},action=a_{i}\right)=1\), where \(a_{i}\in\mathcal{A}\), \(b_{i}\in\mathcal{B}\) and \(c_{i}\in\mathcal{C}\). We assume with the condition that there is no interference in the transition process, so the transition probability here is 1. If the multi-stage decision-making process is a closed loop that \(b_{i}c_{i}=b_{k}c_{k}\) or \(a_{i}c_{i}=a_{k}c_{k}\) or \(a_{i}b_{j}=a_{k}b_{i}\) exists, Algorithm 5 is stable. Since \(|\mathcal{A}|\), \(|\mathcal{B}|\), and \(|\mathcal{C}|\) are all bounded, \(b_{i}c_{i}=b_{k}c_{k}\) or \(a_{i}c_{i}=a_{k}c_{k}\) or \(a_{i}b_{j}=a_{k}b_{i}\) appears at most \(\min\left(|\mathcal{B}|\left|\mathcal{C}\right|,|\mathcal{A}|\left|\mathcal{C} \right|,|\mathcal{A}|\left|\mathcal{B}\right|\right)+