# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 5 Proposed Coexistence Solutions for Uncoordinated LTE-U and Wi-Fi Networks
### Q-Learning Based Coexistence Scheme
Generally, during exploitation, the agent uses the experience from already investigated actions and selects the optimal action based on them, while during exploration, the agent investigates and experiences new actions. \(\varepsilon\)-greedy policy is a policy used to balance exploration and exploitation by using a decision value \(\varepsilon\). The decision value is selected in the range \(0\leq\varepsilon\leq 1\) and it is used to decide whether the agent will explore or exploit in every step. The agent uses exploration and exploitation with a probability of \(\varepsilon\) and \(1-\varepsilon\), respectively. Algorithm 2 shows how Q-learning can be used for optimally selecting the number of blank subframes (\(N_{b}\)), aiming at maximizing fairness and the aggregated throughput of LTE and WiFi networks. The procedures show how the agent selects an optimal value of \(N_{b}\) based on LTE throughput status and WiFi saturation status. The algorithm also shows how exploration and exploitation are balanced by an adjustable value of \(\varepsilon\). Initially, the \(\varepsilon\) is set to a value close to 1 so that the agent starts with a higher probability of exploration so that it quickly explores different states. After \(N_{\varepsilon}\) number of iterations, the value of \(\varepsilon\) is reduced by a factor of \(f_{\varepsilon}\), until a minimum value of \(\varepsilon_{\text{min}}\) is reached. The key elements of the Q-learning process for optimal \(N_{b}\) selection can be described as follows:  
* **Agent:** eNB, we assume the LTE-U eNBs belong to the same operator which cooperates together. * **State:** The state at time \(t+1\) is determined by the status of the environment after an action \(a\) is taken at time \(t\). The statuses of the WiFi and the LTE networks are used to represent the state after a frame configuration of LTE is selected. The new state \(s^{\prime}\) can be represented as: \[s^{\prime}=[WiFi_{\text{State}},LTE_{\text{State}},N_{\text{b}}].\] (1) _WiFi_State can be saturated (\(W_{\text{S}}\)) or unsaturated (\(W_{\text{U}}\)). \(LTE_{\text{State}}\) represents the state of LTE which is determined based on LTE target throughput and LTE obtained throughput. The state of the WiFi network is represented by its saturation status which is determined based on our previous work [8]. On the other hand, the state of the LTE network is determined by the \(LTE_{\text{obtained}}\) and \(LTE_{\text{Target}}\) throughput. The possible states of LTE are shown in Table 2. \(N_{\text{b}}\) is the number of blank LTE subframes in a single frame. The state-space \(S\) includes all combination of \(WiFi_{\text{State}}\), \(LTE_{\text{State}}\), and \(N_{\text{b}}\) values. * **Action:** The action \(a\) at time \(t\) represents the selection of \(N_{\text{b}}\) number of blank LTE subframes. * **Reward:** LTE network is required to attain the highest possible throughput by accessing the spectrum while maintaining fairness with WiFi. However, the performance of the WiFi can not be directly accessed by the agent as our goal is to propose a coexistence scheme that does not require signaling exchanges between the WiFi AP and LTE eNB. For this reason, we consider the CNN-based WiFi network saturation sensing model to determine the status of the co-located WiFi. The reward value for action \(a\) taken at a state \(s\) is computed based on the newly achieved state \(s^{\prime}\) as follows: \[r_{\text{Q}}= \begin{cases}100&\text{if }s^{\prime}=([W_{\text{U}}\!,\!L_{10}\!,\!N_{ \text{b}}])\text{ or}([W_{\text{S}}\!,\!LTE_{\text{state}}\
eq L_{10},5])\\ 100*\rho*\beta*\frac{N_{\text{b}}}{10}&\text{otherwise}\end{cases} .\] (2) where \(N_{\text{b}}\) is the number of blank LTE subframes and \(\beta\) is used based on the values in Table 2. \(\rho\) is 1 if WiFi state is \(W_{\text{U}}\) and 0.5 if WiFi state is \(W_{\text{S}}\). The value of \(\beta\) increases as the \(LTE_{\text{Obtained}}\) gets close to \(LTE_{\text{Target}}\). Similarly, the value of \(\rho\) is higher when the WiFi network is unsaturated.