# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 7 Conclusions and Future Work  
In cellular networks, broadband applications that employ licensed spectra are expanding and the demand for high-throughput services is expected to rise. As a result, licensed spectra are becoming increasingly scarce. LTE is expected to use the unlicensed band for some of its transmissions to alleviate the spectrum scarcity problem. To effectively use the unlicensed spectrum, however, a number of issues must be addressed. The most significant issue is establishing a harmonious coexistence with the WiFi networks that already exist on the unlicensed spectrum. Many coexistence strategies have been proposed to ensure that WiFi and LTE networks coexist together. Convolutional Neural Network (CNN) and Q-learning-based machine learning algorithms are mostly used in current coexistence methods to improve performance. Most existing coexistence systems take into account coordinated LTE and WiFi networks, in which a co-located LTE network picks its transmission time based on the quantity of WiFi traffic generated in its collision domain. The WiFi traffic demand is determined by a central coordinator which can communicate with co-located networks using a collaboration protocol to exchange status and requirements. Collaboration protocol-based information exchange to identify traffic status requires changes to the infrastructures of the co-located WiFi and LTE networks. Moreover, adopting this inter-technology collaboration protocol increases the coexistence schemes' complexity, traffic overhead, and reaction time. As a result, we present a coexistence scheme that can operate in uncoordinated LTE and WiFi networks without the necessity for a collaboration protocol to exchange traffic status between the technologies. In the proposed scheme, the LTE-U deploys a WiFi saturation sensing model to estimate the traffic demand of co-located WiFi networks. In particular, RER and ER Q-learning based non-cooperative coexistence schemes are proposed and evaluated. We also compare the performance of the \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Coexistence** & \multicolumn{6}{c}{**Distribution of Actions Taken in Each Rank (in \%)**} \\ \cline{2-9}
**Scheme** & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** \\ \hline QL & 66.7 & 28.1 & 0.5 & 0.2 & - & - & - & - & - \\ \hline ER QL & 86.3 & 13.1 & 0.6 & - & - & - & - & - & - \\ \hline RER QL & 89.4 & 9.7 & 0.9 & - & - & - & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Distribution of rank of actions taken based on the Q-learning, ER Q-learning, and RER Q-learning based schemes. proposed schemes with each other and with other non-coordinated schemes which use rule-based and Q-learning based approaches. The comparison is carried out in terms of model complexity, convergence, the accuracy of best action selection, fairness, and throughput. Our results show that the proposed RER Q-learning scheme converges faster than the ER Q-learning scheme and gives better aggregated throughput and fairness performance as compared to the rule-based and Q-learning schemes. In the future, a system that integrates the traffic prediction model and the proposed coexistence scheme can be developed to enhance the overall system performance. By integrating the traffic prediction model with the proposed coexistence scheme, the optimal execution frequency of the coexistence decision can be determined. In other words, the coexistence scheme is executed every period of time \(T\), where \(T\) is optimally selected considering computational complexity and the traffic dynamics of the co-located networks estimated by a network traffic prediction model. Conceptualization, M.G. ; A.S.; V.M. ; D.N. and I.M. ; methodology, M.G. ; A.S.; V.M. ; D.N. and I.M. ; software, M.G. ; A.S.; validation, V.M. ; D.N. and I.M. ; formal analysis, M.G. ; investigation, M.G. ; A.S.; V.M. ; D.N. and I.M. ; resources, I.M. ; data curation, M.G. ; writing--original draft preparation, M.G. ; writing--review and editing, A.S.; V.M. ; D.N. and I.M.