# A Multiarmed Bandit Approach for LTE-U/Wi-Fi Coexistence in a Multicell Scenario
## 5 MAB-ADC: Multiarmed Bandit for Adaptive Duty Cycle Selection  
This section introduces the proposed Multi-armed Bandit for Adaptive Duty Cycle Selection (MAB-ADC) framework, which utilizes MAB for adaptive DC selection. The algorithm embedded in the LTE-U cell serves as the agent. Unlike conventional RL algorithms, MAB does not define any states. Instead, there is only one state, as the \(Q_{t}(a)\) depends only on the action itself.  
The whole framework is presented below, including each variable from the MAB theory mapped to its corresponding counterpart in the coexistence problem:1. The actions that the agent can choose from are the duty cycle values in the set \(\mathbb{A}=\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8\}\). Each value represents how much channel air time, during the ABS time, is given to each access technology. For example, a DC value of 0.4 means that 40% of the ABS time is reserved for LTE-U and 60% is reserved for Wi-Fi;
2. The reward is defined as the aggregated throughput: \(T_{x_{lite}}+T_{x_{wifi}}\), where \(T_{x_{lite}}\) is the LTE-U operator throughput over the ABS time, and \(T_{x_{wifi}}\) represents the throughput of the Wi-Fi operator;
3. The decision-making process happens for each duty cycle duration of 40 ms (ABS time);
4. To balance the exploration vs. exploitation trade-off in the \(\epsilon\)-greedy algorithm, the value of \(\epsilon\) is initially defined as 0.3, which corresponds to a 30% exploring probability. Then, this value decays with a factor of 1.015. This configuration ensures more exploration at the beginning and more exploitation after learning the best action. Besides, these values result in an exploring probability of 1.5% after 200 random selections.  
Algorithm 1 presents the pseudoalgorithm for the MAB-ADC. The variable \(N(a)\) is a vector that keeps track of how many times each action has been chosen up to the current time. The updating equation for \(Q(a)\) is modified to its incremental form, as opposed to Equation (1). The main advantage of using this incremental form is that it only needs to store \(Q(a)\) and \(N(a)\), minimizing the memory requirement.  
```
1 Initializefor\(a\in\mathbb{A}\)do
2 Start Q(a) = 0;
3 Start N(a) = 0;
4
5 end for
6 Learning:
7 Loop
8 Generate a random value \(r\in U(0,1)\)
9 if\(r<\epsilon\)then
10 Select an action \(a\in\mathbb{A}\) randomly;
11 else
12 Select an action \(a\in\mathbb{A}\) according to \(argmax\limits_{a}Q_{t}(a)\);
13
14 end if
15 Run \(a\);
16 Take the reward \(R\), \(T_{x_{lite}}+T_{x_{wifi}}\);
17 Update:
18\(N(a)\gets N(a)+1\);
19\(Q(a)\gets Q(a)+\frac{1}{N(a)}[R+Q(a)]\)
20 endloop
```  
**Algorithm 1**MAB-ADC: MAB application for adaptive DC selection  
To further enhance our proposed framework based on MAB, we add two operation modes: the independent and the centralized coordinated. In the independent mode, each LTE-U cell has its own embedded MAB-ADC, and the DC selection is independent among cells, i.e., each cell calculates its own \(\epsilon\), selects an action, and updates the \(Q(a)\) independently. The centralized coordinated mode has a central node where the proposed MAB-ADC is running and information from all LTE-U cells is exchanged. This central node calculates the \(\epsilon\), updates the variables, and selects the next actions. After defining the next action, that node sends the DC value to all connected LTE-U cells, and all of them set that same DC value until the central node defines a new one at the next ABS time.