# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 5 Proposed Coexistence Solutions for Uncoordinated LTE-U and Wi-Fi Networks
### Experience Replay Based Q-Learning  
The traffic load of wireless networks is mostly dynamic. In such a dynamic environment state-action pairs are not consistent over time. This makes predicting future rewards more complex. For this reason, the Q-learning algorithm is mostly applied in stationary environments where state-action pairs are consistent. However, this problem can be solved by using Experience Replay (ER). ER is an approach used to minimize action-state pair oscillation. This is achieved by storing a large number of past experiences. A single stored experience is represented by a combination of the current state, action, reward, and next state \((s,a,r,s^{\prime})\). In traditional Q-learning, the Q-table is updated based on the single latest experience. However, Q-table values are updated by taking a random portion of the buffered experiences in case of experience replay-based Q-learning [9]. ```
Input: set learning rate (\(\eta\)), discount factor (\(\gamma\)), initial \(\epsilon\)-greedy value (\(\epsilon\)) Output: Optimal policy\(\pi^{\prime}\) for\(s\in\mathcal{S}\), \(a\in A\)do
1 Q(s,a) \(\gets 0\)
2 end while
3 whileConvergence is not achieveddo
4 Generate a random value \(\kappa\) where \(0\leq\kappa\leq 1\) if\(\kappa\leq\epsilon\)then
5 Select an action \(a\) randomly
6 else
7 Select an action \(a\) where
8\(Q(s,a)\leftarrow\max_{a}\,\left(Q(s,a)\right)\), \(\forall a\)
9 end if
10 Execute action \(a\) Update state \(s^{\prime}\), compute reward \(r\) at state \(s^{\prime}\) // Update Q value \(Q(s,a)\leftarrow(1-\eta)Q(s,a)+\eta[r+\gamma\max_{a}Q(s^{\prime},a)]\) Update \(s=s^{\prime}\)if(Number of iterations \(=N_{e})\cap(\epsilon\geq\epsilon_{min})\)then
11\(\epsilon\leftarrow\textsc{r}\epsilon*f_{\epsilon}\) // Update the value of \(\epsilon\) every \(N_{e}\) iteration
12 end if
13
14 end for
```  
**Algorithm 2**Q-learning based algorithm for dynamic selection of \(N_{\text{b}}\)  
In ER-based Q-learning, \(N_{E}\) number of experiences are recorded as a single batch from the experience buffer. The batch size remains fixed as newly recorded experiences keep pushing the oldest experience out of the list. The use of such an experience record enables the agent to learn more from important individual experiences, i.e., some important experiences can be used multiple times to achieve faster convergence. The importance of the reward can be defined based on different criterion such as it corresponding reward or time of occurrence. The experience record also helps to recall rare occurrences and harmful correlations. For this reason, ER is a promising approach to achieve an optimal policy from fewer experiences. Hence, we propose an ER-based Q-learning solution for LTE and WiFi network coexistence. The agent, action, and state elements of ER-based Q-learning are similar to the traditional Q-learning based solution described in Section 5.2. In traditional Q-learning, we can only estimate the reward value of a given action based on the current state \(s\) and the new state \(s^{\prime}\) after the action is taken. However, this equation only shows how good the action is in this state but with the knowledge of a single experience, we can not determine if the action is the best action among all the possible actions. For this reason, we propose ER based Q-learning solution to cope with the dynamic nature of the traffic load of the LTE and WiFi networks. For the dynamic traffic environment, we need to record the experiences by taking all possible actions for each traffic load combination and these observed experiences can be used to determine which action is the best. For each experience, a fairness factor (\(f_{f}\)) is included to indicate the fairness level in the experience. In other words, \(f_{f}\) value is selected in such a way that its value increases with the fairness level of the action. \(f_{f}\) is generated using the following equation:  
\[f_{f}(a)= \begin{cases}\rho*\beta&\text{if WiFi state is }W_{\text{U}}\\ \rho*\beta*\frac{N_{\text{b}}}{10}&\text{otherwise}\end{cases} .