# A Turbo Q-Learning (TQL) for Energy Efficiency Optimization in Heterogeneous Networks
## Solution with a Tql Algorithm
### Tql Algorithm  
If QL is directly utilized to solve the original problem (12) as shown in Algorithm 1, we can see the action size \(X=(|P|+1)\times|\text{A}|\times|\text{B}|\) is too large, where \(|\bullet|\) represents a cardinality of set. Since the action is represented by a vector of three dimensions, the optimization problem can be decomposed into three subproblems. We propose the TQL algorithm which decomposes the objective optimization problem into three sub-problems as optimizing the ratio of ABS \(\alpha\), CRE bias \(\beta\), and the number of SBS activations \(|S|\) to reduce action space size. 2.1 Sub-Problem A: Given the Cre Bias \(\beta\) and the Number of Sbs Activation \(|S|\) for Optimizing the Abs Ratio \(\alpha\)  
The action of the sub-problem A is \(a_{t}=(\alpha,\beta,|S|)\) where \(\beta\) and \(|S|\) are given. \(\alpha\in\text{A}\) and the action space size is \(|A|\). State \(s=s_{t}\). The tabular method of Q-learning can be used to solve sub-problem A and the updating rule of sub-Q-value can be written as  
\[Q_{\alpha}(s,a)\gets Q_{\alpha}(s,a)+\lambda_{1}\left[r+\gamma_{1}\max_{a^ {{}^{\prime}}}Q_{\alpha}\left(s^{{}^{\prime}},a^{{}^{\prime}}\right)-Q_{\alpha }(s,a)\right] \tag{18}\]  
and shown in Algorithm 2. 2.2. Sub-Problem B: Given the Abs Ratio \(\alpha\) and the Number of Sbs Activation \(|S|\) for Optimizing the Cre Bias \(\beta\)  
The action of the sub-problem B is \(a_{t}=(\alpha,\beta,|S|)\) where \(\alpha\) and \(|S|\) are known. \(\beta\in\) B and the action space size is \(|B|\). State \(s=s_{t}\). Like Formula (18), the updating rule of sub-Q-value can be written as  
\[Q_{\beta}(s,a)\gets Q_{\beta}(s,a)+\lambda_{2}\left[r+\gamma_{2}\text{ max}_{a^{\prime}}Q_{\beta}\left(s^{{}^{\prime}},a^{{}^{\prime}}\right)-Q_{\beta}(s,a)\right] \tag{19}\]  
and shown in Algorithm 3. ```
0:\(a_{t}=(\alpha,\beta,|S|)\). 0: Optimized CRE bias \(\beta\). 1: Initialize \(Q_{\beta}(s,a)\), state \(s\) and \(n\)=0;
2: Setting learning rate \(\lambda_{2}\), greedy probability \(\varepsilon_{2}\), discount factor \(\gamma_{2}\), and threshold\({}_{2}\);
3:while\(n<=\emph{threshold}_{2}\)do
4: In state \(s\), select the optimal action \(a\) with greedy probability \(\varepsilon_{2}\);
5: Observe \(r\);
6: Update \(Q_{\beta}(s,a)\) according to Formula (19) ;
7:\(s\gets s^{{}^{\prime}}\);
8:\(n=n+1\);
9:endwhile
10: Output: \(\beta=a\);
```  
**Algorithm 3**The ABS ratio \(\alpha\) and the number of SBSs activation \(|S|\) are given to optimize the CRE bias \(\beta\). 2.3. Sub-Problem C: Given the Abs Ratio \(\alpha\) and the Cre Bias \(\beta\) for Optimizing the Number of Sbs Activation \(|S|\)  
The action of the sub-problem B is \(a_{t}=(\alpha,\beta,|S|)\) where \(\alpha\) and \(\beta\) are known. \(|S|\in\{0,1,...,|P|\}\) and the action space size is \(|P|+1\). State \(s=s_{t}\). It is similar to Formula (18) and the updating rule of sub-Q-value can be written as  
\[Q_{|S|}(s,a)\gets Q_{|S|}(s,a)+\lambda_{3}\left[r+\gamma_{3}\text{max}_{a^ {\prime}}Q_{|S|}\left(s^{{}^{\prime}},a^{{}^{\prime}}\right)-Q_{|S|}(s,a)\right] \tag{20}\]  
and shown in Algorithm 4. ```
0:\(a_{t}=(\alpha,\beta,|S|)\). 0:Optimized ABS ratio \(\alpha\). 1: Initialize \(Q_{|S|}(s,a)\), state \(s\) and \(n\)=0;
2: Setting learning rate \(\lambda_{3}\), greedy probability \(\varepsilon_{3}\), discount factor \(\gamma_{3}\), and \(threshold_{3}\)
3:while\(n<=threshold_{3}\)do
4: In state \(s\), select the optimal action \(a\) with greedy probability \(\varepsilon_{3}\);
5: Observe \(r\);
6: randomly transfer from \(s\) to \(s^{{}^{\prime}}\);
7: Update \(Q_{|S|}(s,a)\) according to Formula (20);
8:\(s\gets s^{{}^{\prime}}\);
9:\(n=n+1\);
10:endwhile
11: Output: \(|S|=a\);
```  
**Algorithm 4**The ABS ratio \(\alpha\) and the CRE bias \(\beta\) are given to optimize the number of SBS activation \(|S|\).