# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 1 Introduction
As it does not follow the same regulations as WiFi, CSAT is widely seen as more aggressive and less fair than LBT [7]. However, if CSAT is properly designed, it can provide the same level of fairness as compared to LBT. For this reason, we propose an efficient and fair CSAT-based LTE-U and WiFi coexistence scheme in this article. In our previous work [8], a Convolutional Neural Network (CNN) is proposed that senses WiFi saturation. This work uses this WiFi saturation sensing model to develop an LTE-U/WiFi coexistence mechanism that selects optimal LTE ON and OFF periods. We consider uncoordinated LTE-U and WiFi networks and we propose coexistence schemes that select an optimal muting period based on the saturation status of the WiFi network. In other words, the paper proposes coexistence schemes for uncoordinated LTE-U and WiFi networks which do not require a signaling protocol to exchange traffic status between the technologies. As a WiFi saturation sensing model which is capable to discriminate between saturated and unsaturated WiFi network behavior is used, the proposed coexistence scheme does not require decoding of the WiFi traffic. These features lead to considerably superior coexistence control decisions. In LTE-U and WiFi coexistence problem, the LTE-U and WiFi networks have dynamic traffic loads. For such a dynamic environment problems experience replay based reinforcement learning solutions are more efficient [9]. Hence, we propose two coexistence schemes: (a) Experience Replay (ER) based Q-learning (b) Reward selective Experience Replay (RER) based Q-learning. As uncoordinated LTE-U and WiFi networks are considered, the sensed saturation status of WiFi, which represents the traffic demand of WiFi, is stored in the experience records. In experience replay based solutions, it is recommended to use the most significant experiences for faster convergence [9]. For this reason, we also propose a RER based Q-learning coexistence scheme. In the RER based Q-learning scheme, experiences with highest reward value are used to update the Q-table. We present the performance comparison of the proposed schemes with each other and with other rule-based coexistence scheme and Q-learning based coexistence scheme which are formulated for comparison purposes. The main contributions of this work are summarized as follows:  
* Investigation on non-coordinated coexistence of LTE and WiFi networks. * Propose coexistence scheme that use ER based Q-learning and RER based Q-learning solutions for uncoordinated LTE-U and WiFi networks. * Performance analysis and comparison of the proposed coexistence schemes with each other and with a rule-based and Q-learning based coexistence schemes in terms of model complexity, convergence, the accuracy of best action selection, fairness, and throughput. We are able to observe that the RER Q-learning scheme converges faster than the ER Q-learning scheme and gives 19.1% and 5.2% better aggregated throughput performance than the rule-based and Q-learning schemes, respectively. The RER Q-learning scheme also achieves 16.4% and 10.9% better fairness performance as compared to the rule-based and Q-learning schemes, respectively. In general, this paper presents a coexistence scheme that does not require a collaboration protocol for the traffic status exchange between the co-located networks. In the proposed coexistence scheme, the LTE-U eNB uses a WiFi saturation sensing model to estimate the WiFi traffic load and selects an optimal configuration according to the WiFi saturation status. Therefore, the proposed scheme requires modifications on the LTE-U eNB side only and this makes it compatible with commercial off-the-shelf WiFi devices. This also enhances the deployment of real-time coexistence decisions as there is no delay introduced due to traffic load status exchange between the technologies.