# Q-Learning Based Fair and Efficient Coexistence of LTE in Unlicensed Band
## 3 LTE-U Coexistence Mechanism
### Q-Learning Based Joint ADC and DCS for LTE-U
The reward for action a of an agent is given through the following function: \[r^{k}=\ \frac{\sigma\ *\ \left(\theta_{k}^{target}\ -\theta_{k}^{target}-\theta_{k}^{action} \right)}{N_{k}}\ \text{for}\ \theta_{k}^{target}-\theta_{k}^{action}\ >00\ \text{for}\ \theta_{k}^{target}-\theta_{k}^{ action}\ <\ 0\] (9)  
where \(\sigma\) defines the fraction of positive rewards, and \(\ \theta_{k}^{target}\) and \(\theta_{k}^{action}\) are the predefined optimal duty cycling action and the chosen action values from the set of available duty cycling actions, respectively. In our system, the target duty cycling action in the network is the ratio of the sum of active LTE-U UEs to the total number of active users in the channel, and can be denoted as follows:  
\[\theta_{target}^{k}=\frac{M}{M+N_{k}} \tag{10}\]  
The proposed Q-learning algorithm is briefly described in Algorithm 1 below:Many LTE-U BSs available on the market are supported using a single or multiple Wi-Fi interface [32; 33; 34; 35] for monitoring the carrier and notification purposes in LTE-U. ULTRON [33], which operates in a LTE-U BS, employs Wi-Fi embedding to transmit Wi-Fi data over an unmodified LTE PHY. In addition, the same method is used to recognize a Wi-Fi preamble transmission directly applying LTE PHY. In addition, ULTRON also facilitates scalable Wi-Fi sensing to efficiently set up a single Wi-Fi sensing interface to jointly enhance the performance of both LTE and Wi-Fi [36]. Hence, we can use existing estimation techniques such as a Kalman filter [37], machine learning techniques [38], and power techniques [39] to approximate the number of active Wi-Fi STAs in each channel using LTE-U [40]. Here, we use a Wi-Fi preamble decoding and energy detection (ED) mechanism in the time domain [41; 42] without synchronization to the Wi-Fi STAs. During every muting period, LTE-U BS listens to the carrier to evaluate the collision probability (\(P_{k}^{coll}\)) and channel idle probability (\(P_{k}^{\textit{idle}}\)) among Wi-Fi STAs. The total number of observed slot times is denoted as \(C_{k}^{\textit{listen}}\) and the number of collisions in the observed period is represented by \(C_{k}^{\textit{coll}}\). Hence, \(P_{k}^{coll}\) can be calculated as \(P_{k}^{coll}=C_{k}^{coll}/C_{k}^{\textit{listen}}\). Furthermore, \(P_{k}^{\textit{idle}}\) can be acquired based on the ratio of the total number of idle slots to the total number of observed periods, i.e., \(P_{k}^{\textit{idle}}=C_{k}^{\textit{idle}}/C_{k}^{\textit{listen}}\), where \(C_{k}^{\textit{idle}}\) is the number of idle time slots in channel k.  
When an LTE-U BS is muted, the probability of channel \(k\) being idle is when all Wi-Fi STAs do not transmit is \(P_{k}^{\textit{idle}}=\ (1-\tau_{k}\ )^{N_{k}}\), where \(\tau_{k}\) represents the transmission probability of Wi-Fi STAs when LTE-U BS is muted. Similarly, the probability that the channel will experience a collision when at least one of the (\(N_{k}\)\(-\)1) remaining station transmits is \(P_{k}^{coll}=~{}(1-~{}\tau_{k})^{(N_{k}-1)}\). Hence, \(P_{k}^{coll}=~{}1-\frac{P_{k}^{\textit{pille}}}{1-\tau_{k}}\) and numerically solving the equation for \(\tau_{k}\), we obtain \(\tau_{k}=~{}1-\frac{P_{k}^{\textit{pille}}}{1-\tau_{k}^{\textit{pille}}}\). Now, the active number of Wi-Fi STAs is obtained by solving the equation, \(N_{k}=~{}log_{(1-\tau_{k})}~{}P_{k}^{\textit{pille}}\), which is \(\frac{log(P_{k}^{\textit{pille}})}{log(1-\tau_{k})}\).