# Solution for Interference in Hotspot Scenarios Applying Q-Learning on FFR-Based ICIC Techniques
## 10 Proposed Solution: Background, Implementation, and Simulation Results
### The Q-Learning Implementation  
As a consequence of the discussions and exploratory analysis presented, the QL-based solution aims at maximizing the average SINR to mitigate the negative impact of ICI on a dynamic scenario. The algorithm controls the _RsrqThreshold_. According to the results shown in Section 8, the _RsrqThreshold_ is the parameter that has the biggest impact on average throughput. Hence, it was the first option for the QL algorithm. However, future works include the control of both the RsrqThreshold and the BandwidthDistribution, which is also an important parameter for the ICIC performance parameter, which dictates user allocation in the sub-bands defined by the Strict FR.  
The QL algorithm operates as a centralized solution, namely that every decision affects all cells in a cluster. The ICIC techniques presented in Section 3.1 split the bandwidth for clusters formed by three cells. Therefore, each change in the RsrqThreshold affects all three cells of the cluster. If each cell had an independent QL algorithm, they would be competing, as each action would affect the whole cluster, based only on the data from one cell.  
The QL algorithm needs the following parameters:  
* A set of available actions, \(A=a_{1},a_{2},...,a_{n}\);
* A set of possible states of the system, \(S=s_{1},s_{2},...,s_{n}\);
* A \(Q(s,a)\) matrix to store the estimated rewards;
* \(\alpha\) and \(\gamma\).  
The algorithm was configured with six states and six actions. The metric that defines the state is the SINR; however, this metric may vary. For example, depending on the service targeted, it could be throughput, PLR, delay, or a combination of those, as presented in Equation (4) [56]:  
\[M_{a}=\sum_{i=1}^{n}w_{i}f_{i}(M_{i}), \tag{4}\]  
where \(w_{1},\...,\ w_{n}\) are the weights of each metric (\(M_{1},\...,\ M_{n}\)) and \(f_{i}\) is a normalization function that maps the metrics to values between 0 and 1.  
The SINR was chosen because it directly indicates the impact of interference in users. Hence, the average SINR defines the states of the system and it is used as the reward from each state/action pair. Through experimental analysis to characterize the scenarios, the states are defined as follows:  
* State 1: \(SINR<24\ dB\);* State 2: 24 \(dB<SINR\leq 27\ dB\);
* State 3: 27 \(dB<SINR\leq 29\ dB\);
* State 4: 29 \(dB<SINR\leq 30\ dB\);
* State 5: 30 \(dB<SINR\leq 32\ dB\);
* State 6: \(SINR>32\ dB\).  
Additionally, the set of available actions corresponds to RsrqThreshold values: \(A~{}=~{}\{29,30,31,32,33,34\}\). These values were selected based on the results presented in Section 8, given that this parameter has a bigger impact on system performance when operating on that range. The matrix \(Q(s,a)\) initially has all values equal to zero and the values for \(\alpha\) and \(\gamma\) are 0.4 and 0.5, respectively. This choice is based on exploratory simulations and related works [43] and the \(Q(s,a)\) values are calculated according to Equation (5):  
\[Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha\Big{[}r+\gamma\max_{a}Q(s^{\prime},a) \Big{]} \tag{5}\]  
Algorithm 1 presents a pseudo-code of the proposed solution. The QL algorithm is executed each 40 ms. This interval is inspired by the study presented in [43]; however, exploratory simulations were conducted to assure that the SINR samples collected during this interval are sufficient to represent the average SINR of the system.  
As mentioned previously, the QL seeks an optimum solution through a succession of actions. Thus, at any given time, the algorithm may get stuck on a set of state/action pairs without exploring other options (similar to an optimization algorithm that gets stuck on a local minimum). To address this issue, the parameter \(\epsilon\) may be used to control the algorithm's level of exploration, making the agent choose a random action eventually.