# A Multiarmed Bandit Approach for LTE-U/Wi-Fi Coexistence in a Multicell Scenario
## 4 Adaptive Duty Cycle Selection with Reinforcement Learning  
Our previous studies have demonstrated that there is a relationship between the DC value and the offered data rate. When the operators' offered data rate changes, the DC value that maximizes system throughput also changes. In a system where data rates frequently change, using a fixed DC value is not optimal, as it may not lead to maximum throughput. Instead, the system requires an algorithm that automatically selects the best DC value given the current state of the system. In this paper, we propose a framework that uses MAB to dynamically and autonomously choose the LTE-U DC value in a dynamic scenario where LTE-U coexists with Wi-Fi. The goal of this framework is to maximize system throughput (LTE-U + Wi-Fi) by choosing the best DC value for different combinations of offered data rates. By using this framework, the system is expected to always operate near its maximum achievable throughput, as the framework continuously selects the best DC value based on real-time conditions. To address the sequential decision-making process of choosing the optimal DC value as the demand changes over time, an Markov Decision Process (MDP) framework is required. From a machine learning perspective, the field of Reinforcement Learning is well-suited to address this behavior [39]. Moreover, dynamic coexistence scenarios require algorithms that can make decisions in real time. In Reinforcement Learning, the learning process centers around an agent and its interactions with its environment. The agent learns as it applies an action \(a_{t}\), at time step \(t\), and the environment responds by returning a reward and an observation/state at the next time step \(t+1\). With this reward and state, the agent can find relations between input (actions) and outputs as it works towards a goal. Typically, the goal of a RL problem is to maximize the cumulative reward over time. Once the agent is fully trained, it always chooses the best action on each time step \(t\) that maximizes the cumulative reward at the end of the learning process. As presented in [39], there are several ways to resolve \begin{table}
\begin{tabular}{l l} \hline \hline
**Wi-Fi parameter (802.11n-HT PHY/MAC)** & \\ \hline Bandwidth & 20 MHz \\ CCA-Energy Detection threshold & \(-\)62 dBm \\ CCA-Carrier sense threshold & \(-\)82 dBm \\ Bit Error Rate (BER) target & \(10^{-6}\) \\ \hline \hline
**LTE parameters** & \\ \hline Bandwidth & 20 MHz \\ Packet scheduler & Proportional fair \\ ABS pattern duration & 40 ms \\ Duty Cycle (DC) values & \{0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\} \\ \hline \hline
**Common parameters** & \\ \hline Tx power & \(-\)18 dBm \\ Traffic model & UDP full buffer \\ Mobility & Constant position \\ \hline \hline
**Scenario** & \\ \hline \hline \(d\) (distance between APs of different RATs) & 5 m \\ _bs\_space_ (distance between APs of same RAT) & 25 m \\ Number of LTE-U APs & 4 \\ Number of LTE-U stations & 20 \\ Number of Wi-Fi APs & 4 \\ Number of Wi-Fi stations & 20 \\ Path loss and Shadowing & ITU InH \\ Cell selection criteria & For Wi-Fi, AP with strongest RSS. \\  & For LTE-U, cell with the strongest RSRP. \\ UDPRate & \{0.5, 1, 2 Mbps, 4 Mbps\}. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters (adapted from [10]). a reinforcement learning MDP. In this work, we use an MAB approach, as it presents simplifications compared with other methods, resulting in a simple and flexible solution. In the MAB approach, the agent repeatedly chooses from the same set of actions in each time step. Each action results in a numerical reward based on a probability distribution, which is assumed to be independent and identically distributed (iid) for each action in the set. The goal of the agent is to maximize the expected total reward over a period. If the agent selects one of \(k\) available actions at each time step, the problem is called a \(k\)-multiarmed bandit. In this \(k\)-MAB, each action corresponds to a reward that follows a probability distribution.