# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 1 Introduction  
Recently, the modern industry has expanded the deployment of wireless networks in search of effective networking solutions that can improve network performance. The rapid increase of wireless network deployments in the industry, along with the quick penetration of wireless networks consumer devices such as smartphones and tablets, has resulted in an exponential increase in wireless traffic demand. Furthermore, the Internet-of-Things (IoT), which connect an unprecedented number of intelligent objects to next-generation mobile networks, consume a major chunk of the wireless spectrum [1]. As the wireless network industry keeps expanding, the licensed spectrum has become a scarce resource and the wireless network industry has shifted its focus towards exploiting unlicensed spectrum as an efficient approach of addressing spectrum shortages and customers' fast-expanding need for data traffic [2]. In this regard, many fourth-generation Long Term Evolution (LTE) versions, such as LTE unlicensed (LTE-U), LTE licensed assisted access (LAA), and MulteFire [3], have been proposed to use the unlicensed 5 GHz band, which is mostly used by the WiFi network. LTE-U is a technology developed as the first standard for unlicensed spectrum sharing and it has been proposed by the LTE-U forum in 3 GPP release 12 [4] of the LTE specifications. The LTE-U technology extends the LTE operation to the unlicensed spectrum; that is, the LTE standard protocol can be used to communicate on the unlicensed band, and the carrier aggregation technology can aggregatelicensed and unlicensed bands. Part of the data transmission in the licensed spectrum is thus shifted to the unlicensed band [3]. When LTE is introduced into a common unlicensed spectrum, it will certainly cause issues while competing and coexisting with other unlicensed communications technologies that use the same spectrum. In traditional communication technologies that use unlicensed spectrum for data transmission, such as WiFi, channel access can only be obtained competitively to achieve fair spectrum sharing. On the other hand, LTE technology, which is initially developed in the licensed spectrum, needs complete spectrum control during its data transmission [4]. LTE eNodeB (eNB) performs centralized scheduling of wireless resources to maximize spectral efficiency. Even in the absence of data traffic, LTE uses continuous signal transmission with minimum time gaps within the allocated resources. On the other hand, WiFi uses the carrier sense multiple access with collision avoidance (CSMA/CA) protocol to coexist with other unlicensed band wireless technologies. If the unlicensed spectrum is used as new LTE frequency bands, the transmission of LTE-U will cause significant interference to WiFi. As a result, it is important to design a reasonable and equitable coexistence scheme to ensure fair and efficient coexistence between the two technologies [2]. Carrier Sense Adaptive Transmission (CSAT) is one of the LTE/WiFi coexistence mechanisms which have been proposed by Qualcomm [5]. In CSAT, LTE uses ON and OFF duty-cycle periods to give transmission opportunities (TXOP) to co-located WiFi networks. During an OFF period (mute period), LTE does not transmit and this gives the opportunity to other WiFi networks operating in the same spectrum to transmit. On the other hand, LTE will access the channel during an ON period. The duration of the LTE ON and OFF periods are defined by the eNB to achieve higher aggregated throughput while maintaining fairness between the technologies. However, there is no standard algorithm used to select the optimal ON-OFF time ratio, and the selection of this optimal ON-OFF duration ratio is open for research. LTE Licensed Assisted Access (LAA), on the other hand, is another coexistence approach that employs a mechanism standardized by 3 GPP known as Listen Before Talk (LBT). Before any transmission in unlicensed spectrum, LBT performs Clear Channel Assessment (CCA) to determine transmission opportunities using energy detection [6].