# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 5 Proposed Coexistence Solutions for Uncoordinated LTE-U and Wi-Fi Networks
### Q-Learning Based Coexistence Scheme  
Reinforcement Learning (RL) is an area of machine learning where an agent learns from its interaction with the environment. Q-learning is a type of reinforcement learning algorithm that implements dynamic programming and Monte Carlo concepts to solve sequential decision-making problems in the form of Markov Decision Processes (MDP) [37]. In MDP, the agent interacts with the environment and makes decisions based on the environment states. The agent chooses actions and the environment on the other hand is used to update the responses to these actions in the form of observations or new situations. The agent aims to maximize the cumulative reward of taking an action in a particular state by learning a policy \(\pi\) from trial and error in the environment. In the process of learning iteration, the agent initially observes the current state \(s\) on the environment. The agent then selects the corresponding action \(a\) according to the policy \(\pi\) and it receives the reward value \(R\) from the environment. After the action is taken on the environment the state changes to \(s^{\prime}\). Finally, the optimal policy \(\pi^{*}\) is updated based on the current state and the reward value [37]. \begin{table}
\begin{tabular}{c c c} \hline
**LTE State** & **Description** & \(\beta\) \\ \hline L\({}_{1}\) & \(LTE_{\text{obtained}}\leq 10\%\) of \(LTE_{\text{Target}}\) & 0.1 \\ \hline L\({}_{2}\) & \(10\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 20\%\) of \(LTE_{\text{Target}}\) & 0.2 \\ \hline L\({}_{3}\) & \(20\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 30\%\) of \(LTE_{\text{Target}}\) & 0.3 \\ \hline L\({}_{4}\) & \(30\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 40\%\) of \(LTE_{\text{Target}}\) & 0.4 \\ \hline L\({}_{5}\) & \(40\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 50\%\) of \(LTE_{\text{Target}}\) & 0.5 \\ \hline L\({}_{6}\) & \(50\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 60\%\) of \(LTE_{\text{Target}}\) & 0.6 \\ \hline L\({}_{7}\) & \(60\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 70\%\) of \(LTE_{\text{Target}}\) & 0.7 \\ \hline L\({}_{8}\) & \(70\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 80\%\) of \(LTE_{\text{Target}}\) & 0.8 \\ \hline L\({}_{9}\) & \(80\%\) of \(LTE_{\text{Target}}<LTE_{\text{obtained}}\leq 90\%\) of \(LTE_{\text{Target}}\) & 0.9 \\ \hline L\({}_{10}\) & \(LTE_{\text{obtained}}>90\%\) of \(LTE_{\text{Target}}\) & 1 \\ \hline \end{tabular}
\end{table} 
Table 2: LTE states and \(\beta\) values. ```
1 Input: Set \(N_{b}\)=50%, \(N_{t}\)=0, Fairness=False Output: Best action
2 Update: s=[WiFistate,LTEstate,\(N_{b}\)], Fairness
3 ifFairness=Truethen
4 No action needed
5
6 else
7 if\((WiFistate=W_{U})\cap(LTE_{state}\
eq LTE_{10})\)then
8 whileWiFistate \(=W_{U}\)do
9 Take action \(N_{b}\gets N_{b-1}\) Update \(s\leftarrow[WiFistate,LTE_{state},N_{b}]\) Update \(N_{t}\gets N_{t}+1\)
10 end if
11 else
12 while\(LTE_{state}=L_{10}\)do
13 Take action \(N_{b}\gets N_{b+1}\) Update \(s\leftarrow[WiFistate,LTE_{state},N_{b}]\) Update \(N_{t}\gets N_{t}+1\)
14 end if
15
16 end if
17
18 end while
19//Keep updating the value of \(N_{b}\) based on traffic dynamics
20 whiletruedo
21 Periodically monitor the wireless environment
22 ifTraffic change is identifiedthen
23 Go to step 2
24 end if
25
26 end if
```  
**Algorithm 1**Rule based coexistence scheme  
In the Q-learning process, the agent balances between exploration and exploitation while it tries to find an optimal strategy in the selection of action \(a\) after observing the current state \(s\) of the environment. In the exploration case, the agent selects an action randomly, expecting that the random action will lead to a higher cumulative reward in the next iterations. On the contrary, in the exploitation process, an action is selected based on the latest expected optimal policy.