# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 5 Proposed Coexistence Solutions for Uncoordinated LTE-U and Wi-Fi Networks
### Experience Replay Based Q-Learning
\tag{3}\]  
where \(N_{\text{b}}\) is the number of blank LTE subframes and \(\beta\) is used based on the values in Table 2. \(\rho\) is 1 if WiFi state is \(W_{\text{U}}\) and 0.5 if WiFi state is \(W_{\text{S}}\). The fairness factor equation shown in Equation (3) is set in such a way that its value increases as the two technologies utilize a fair share of the spectrum. As far as the WiFi the network is unsaturated, \(W_{\text{U}}\) the fairness factor value increases when the LTE network's obtained throughput increases. In the equation, the LTE network obtained throughput is reflected by the value of \(\beta\). On the other hand, when the WiFi network gets saturated, the value of the fairness factor depends on the LTE obtained throughput (reflected by \(\beta\)) and \(N_{\text{b}}\). Generally, the equation leads to highest fairness factor value if \(L_{10}\) and WiFi is at state \(W_{\text{U}}\) are obtained at a given action. However, if both LTE and WiFi networks have a higher traffic load, these \(L_{10}\) and \(W_{\text{U}}\) state can not be achieved simultaneously in all possible actions. For this case, the highest fairness factor is obtained when both technologies share the spectrum equally, i.e., \(N_{\text{b}}=5\). In case of experience replay based Q-learning, all possible actions are observed for each given traffic load. All actions are recorded with their corresponding \(WiF_{\text{itate}}\) and \(LTE_{\text{state}}\). The value of \(f_{f}(a)\) is then computed based on the LTE and WiFi states and the corresponding reward values are computed using:  
\[r_{\text{ER}}= \begin{cases}100&\text{if $F(a)$ is $\max_{a}F(a)$, $\forall a$}\\ 0&\text{otherwise}\end{cases} . \tag{4}\]  
The reward function presented in Equation (4) leads to the highest value when an action taken has the maximum possible fairness factor \(f_{f}(a)\) value as compared to all the other possible actions recorded as experiences. Without the use of an experience record, it is not possible to determine if a given action is the best action as compared to other possible actions. For this reason, the value of \(f_{f}(a)\) can not be used in the classical Q-learning reward function. In a dynamic environment, the best action can only be certainly determined after observing and recording all the possible actions. This is the reason behind proposing an ER based Q-learning solution for the LTE-U and WiFi network coexistence problem in a dynamic environment. Algorithm 3 shows how the ER based Q-learning can be used to predict an optimal policy \(\pi^{\star}\). The procedures show how \(N_{E}\) recorded experiences are used to update the Q-network. The dynamic behavior of the environment is represented by generating random traffic load on the LTE and WiFi network. For a given fixed traffic load all possible actions are taken and values \(\{s,a,f_{f},s^{\prime}\}\) are recorded in \(R_{f}\) for every action. Once the values of \(R_{f}\) are stored for all actions, the reward values are computed and the values \(\{s,a,r,s^{\prime}\}\) stored to \(R_{f}\) for every action \(a\). These values are then stored to \(R_{E}\) until the number of experiences reaches \(N_{E}\). Once \(N_{E}\) experiences are recorded, each stored element \(\{s,a,r,s^{\prime}\}\) is used to update the Q-network based on the procedures mentioned in Algorithm 2. This whole process is repeated until the Q-matrix converges.