# Q-Learning Based Fair and Efficient Coexistence of LTE in Unlicensed Band
## 3 LTE-U Coexistence Mechanism
### Q-Learning Based Joint ADC and DCS for LTE-U  
Q-learning has been used in various studies to enhance the coexistence mechanisms and use them individually to learn the best possible strategies to achieve the target [31]. In this paper, we use Q learning for monitoring the Wi-Fi STA traffic load on an unlicensed channel and adjust the LTE transmission accordingly. The objective of joint ADC and DCS reinforcement learning is to determine a policy by which the LTE-U BS will choose the channel and duty cycle period based on measurements observed during a muting period. Q-learning is a model-free reinforcement-learning algorithm. The Q-learning process is built on a Q-function (\(Q\left(k,a\right)\)), which is updated when it obtains a reward \(r\) from a state transition after the agent carries out a certain action \(a\). We use the single-state Q-learning approach with a null discount rate [31] given by  
\[Q(k,\ a)=\ (1-\alpha)*Q(k,\ a)+\ \alpha*r(k,\ a) \tag{6}\]  
Figure 4: LTE-U DC and CA model. where \(\alpha\in(0,1)\) is the learning rate and \(r(k,\ a)\) is the reward obtained as a result of the current action. At initialization, \(Q(k,\ a)\) is set to an arbitrary value \(Q_{init}\). Based on the value of \(Q(k,\ a)\), the proposed duty cycle decision making for LTE-BS follows the softmax policy [31]. The softmax approach is popular owing to its effective and popular means of balancing the exploration and exploitation in reinforcement learning. In a softmax policy, the duty cycling action \(a\) is chosen based on the following probability:  
\[p(k,a)=\frac{\textit{exp}Q(k,\ a)/\textit{Temp}_{k}}{\sum_{a^{\prime}=1}^{n} \textit{exp}\ ^{Q(k,\ a^{\prime})/\textit{Temp}_{k}}} \tag{7}\]  
where \(Temp_{k}\) is the temperature function, which helps reduce the temperature as the number of actions generated by LTE-U BS increase. The resulting amount of exploration will be progressively decreased as LTE-U learns the best solution, and can be expressed as follows:  
\[Temp_{k}=\frac{Temp_{k}^{init}}{log(1+Y_{k})} \tag{8}\]  
where \(Temp_{k}^{init}\) is the initial temperature and \(Y_{k}\) is the action counter. A high temperature causes different actions to be equiprobable whereas a low temperature causes a greater difference in the selection probability for actions that differ in their value estimates. Hence, our scope of Q-learning is to discover the optimal policy for choosing an action in a given state that maximizes the value of the overall reward. To learn this policy, an agent must estimate the value-function through experience. The main components of Q-learning are as follows:  
* An agent is the LTE-U BS. LTE-U BS can change its muting time period for each 10-ms duty cycle period;
* An action that an agent can take is a set of duty cycle patterns A = {0.2, 0.4, 0.6, and 0.8}. Herein, a duty-cycle pattern of 0.4 indicates that LTE-U mutes 0.8 portion of its frame time and transmits during the remaining 0.4 portion of 10 ms;
* Q-learning decisions are taken for every duty cycle duration, which is repeated every 10 ms;
* A state indicates the carrier that is selected for operation {1, 2,..., _K_};
* A reward function is a utility function that guarantees the selection of an appropriate duty-cycle action in the best available channel. This means the chosen action will be maintained close to the target duty-cycle value, offering fair coexistence with other co-located systems (Wi-Fi). At the same time, it compares the goodness of the selected channel with other available channels.