# A Turbo Q-Learning (TQL) for Energy Efficiency Optimization in Heterogeneous Networks
## Solution with a Tql Algorithm
### Q-Learning Algorithm  
The environment is typically formulated as a finite-state Markov Decision Process (MDP) and we set a finite discrete time series \(t\in\{0,1,...,\infty\}\). ABS and CRE are denoted by \(\alpha\in A\) and \(\beta\in B\), respectively. The activation state of the SBSs in the state \(u_{t}\) is \(e_{t}\in E\), where \(E=\left(0,1\right)^{|P|}\), and \(|P|\) represents the number of SBSs in a cell. According to the Control Space Augmentors (CSA) concept mentioned in [9], the SBSs states \(e_{t}\) can be derived based on the number of SBSs activations \(|S|\) in the cell, where A, B, and \(|S|\) represent a limited set of all parameter configurations. Let \(\mathcal{S}\) be a discrete set of environment states and \(\mathcal{A}\) be a discrete set of actions. At each step \(t\), the agent senses the environment state \(s_{t}=s\in\mathcal{S}\) and selects an action \(a_{t}=a\in\mathcal{A}\) to be performed, where \(s\) is the position of a certain number of UEs in the cell, \(\mathcal{S}\) represents the set of cell UEs positions, \(a\) is optimal \(\alpha\), \(\beta\) and \(|S|\) parameter configurations to optimize the energy efficiency of the system, and \(\mathcal{A}\) represents the set of parameter configurations. As a result, the environment makes a transition to the new state \(s_{t+1}=s^{\prime}\in\mathcal{S}\) according to probability \(\mathcal{P}(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)\) and thereby generates a reward \(r_{t}=r(s_{t},a)\in\mathcal{R}\) passing to the agent. MDP is denoted as a topic \(\left(\mathcal{S}\),\(\mathcal{A}\),\(\mathcal{P}\),\(\mathcal{R}\right)\), where  
\(\bullet\)\(\mathcal{S}\) is the set of finite state space;  
\(\bullet\)\(\mathcal{A}\) is the set of finite action space;  
\(\bullet\)\(\mathcal{P}\) is the set of transition probabilities;  
\(\bullet\)\(\mathcal{R}\) represents the set of reward funtion. (1) State: The position of users at step \(t\) is considered as state \(s_{t}=s\), and the set of states is denoted as \(\mathcal{S}\). (2) Action: The ABS configuration \(\alpha\), the CRE bias \(\beta\), and the number of SBS activations \(|S|\) are considered as action \(a_{t}=(\alpha,\beta,|S|)\) at state \(s_{t}\) where \(\alpha\in\text{A}\), \(\beta\in\text{B}\) and \(|S|\in\{0,1,...,|P|\}\). The action space size is \(X=\left(|P|+1\right)\times|\text{A}|\times|\text{B}|\). (3) State transition: The location of users in the cell changes is considered irregularly, and the state transition is random. (4) Reward function: The optimization problem is system energy efficiency which is used as reward function, but, in the actual simulation process, the reward is too large, which causes the system to fall into the local optimum easily. Our proposed solution is that negative reciprocal of energy efficiency is designed as the reward. The goal of RL is to find out the expectation of the strategy with the greatest cumulative reward, which can be expressed as  
\[\text{max}_{\pi}E\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})\right] \tag{13}\]  
where discount factor \(\gamma\) indicates the degree of influence of successor states on current state, and \(r\left(s_{t},a_{t},s_{t+1}\right)\) represents the reward of state \(s_{t}\) selecting \(a_{t}\) and then transiting to state \(s_{t+1}\). The best decision sequence of MAP is solved by the Bellman equation. The state-action value function \(q(s,a)\) can evaluate the current state. The value of each state-action is not only determined by the current state but also by the successor states. Therefore, the state-action value function \(q(s,a)\) of the current s can be obtained by the cumulative reward expectation of the state.