# QoS-Guaranteed Radio Resource Management in LTE-A Co-Channel Networks with Dual Connectivity
## 3 Proposed Schemes
### Q-Learning Dynamic Almost Blank Subframe
#### 3.3.2 Q-Learning-Based Dynamic ABS (QD-ABS) Scheme  
In this work, we proposed a Q-learning-based dynamic ABS scheme to learn the appropriate ABS configuration for a global ABS ratio setting. The key idea is inspired from the literature [33]. In [33], the ABS ratio was dynamically adjusted based on the throughput of Macro BS and pico BS. Since the UE distribution in each sector of a macro station is different, the ABS ratio will also be different. Furthermore, the difference in ABS ratios will affect the throughput of each BS. Thus, we can define the difference between the distribution of the throughput as the cost parameter in Q-learning. The agent finally chooses the best ABS ratio when Q-learning is converged to the minimum cost. The detailed steps of Q-learning-based dynamic ABS scheme consists of two steps:  
(1) It estimates the required ABS ratio in each sector of the Macro station first. Let the total bandwidth requirement of the UEs in Macro BS and pico BS be \(N_{MUE}\) and \(N_{PUE}\), respectively. Assume that the sum of the throughput allocated by the Macro BS and pico BS in the previous frame to the UEs is \(Th_{MUE}^{*}\) and \(Th_{PUE}^{*}\), respectively. Thus, the difference between the required bandwidth and the actual assigned throughput can be expressed as follows. \[\Delta Th_{MUE} = Th_{MUE}^{*}-N_{MUE} \tag{14}\] \[\Delta Th_{PUE} = Th_{PUE}^{*}-N_{PUE}. \tag{15}\]  
In order to meet the QoS requirements of the UE, the Equation (16) illustrates the opportunity of using ABS, which contains two conditions. The first one is that all of the MUEs must satisfy QoS requirement, and there are remaining resources in the Macro BS (\(\Delta Th_{MUE}>0\)). The other one is that some PUEs have not satisfied QoS requirement after resource allocation in pico BS (\(\Delta Th_{PUE}<0\)). Let the set of Macro BSs be denoted by \(M\). If the above two conditions are met, then we can first give the parameter \(\mu_{b}\) (\(b\in M\)) as the ABS ratio in each sector of the Macro station. \[ABS\ ratio\left\{ \begin{array}{cc}\mu_{b},if\Delta Th_{MUE}>0&\text{and}\ \Delta Th_{PUE}<\ 0\\ 0,&\text{others}.\end{array} \right.. \tag{16}\]  
Assuming that the throughput of the radio resource is evenly distributed across each TTI in a frame. For example, the allocation of 1000 bits by a frame is equivalent to an average transmission of 100 bits in 10 TTI. The formula of the ABS \(\mu_{b}\) is described as follows:  
\[\mu_{b} =\underset{\mu_{b}}{\text{argmax}}\left\{\begin{array}{cc}(1-\mu_{b} \ )Th_{MUE}^{*}-N_{MUE}\ \geq 0\\* **Action**: The action is defined as \(a_{t}\), which is the ABS ratio.
* **Cost**: The cost is defined as \(c\), which is the new calculated throughput of each sector of the Macro BS after using the unified ABS ratio.  
The cost \(c\) estimates the immediate return incurred due to action \(a_{t}\) at state \(s\). The cost function is calculated as:  
\[\Delta\mu=\ a_{t}-\mu_{b} \tag{18}\]  
\[Cost_{b}\left\{ \begin{array}{cl}0,&if\Delta\mu=0\\ \left|\Delta\mu Th_{\text{MUE}}\right|,&if\Delta\mu>0\\ \left|\Delta\mu\ (\varepsilon-1)\ Th_{\text{PUE}}\right|,&if\Delta\mu<0\end{array} \right. \tag{19}\]  
\[c=\sum Cost_{b} \tag{20}\]  
where Equation (18) represents the difference before and after adjusting the ABS ratio. In Equation (19), \(\Delta\mu>0\) indicates that the actual ABS ratio chosen by the system (\(a_{t}\)) is greater than that of the sector of the Macro BS estimated by itself (\(\mu_{b}\)), thus losing the excessive throughput of MUEs. On the contrary, \(\Delta\mu<0\) indicates that the actual ABS ratio chosen by the system is less than that of the sector of the Macro BS estimated by itself, thus losing the expected throughput of PUEs. Finally, the sum of these costs in each sector of the Macro BS is calculated in Equation (20). Given aforementioned parameters, the Q-value \(Q(s_{t},\ a_{t})\) can be defined as the expected cost of taking action \(a_{t}\) at state \(s_{t}\).