## 3 Multi-Armed Bandit Algorithms Using Logarithmic Rates
### Thompson Sampling Using Logarithmic Rates  
In this subsection, we introduce another MAB algorithm for the transmission rate control in CSMA/CA wireless networks using the logarithmic values of the transmission rates based on Thompson sampling. We call it Thompson sampling with logarithmic rates (TS-logR). We assume that there are \(K\) MCS levels. Each transmission can be for aggregated (multiple) packets if the frame aggregation is enabled. Algorithm 2 shows the pseudocode of TS-logR.  
In line 2 of Algorithm 2, \(\alpha_{i}\) and \(\beta_{i}\) are lists used to monitor the success and failure events of each MCS level \(i\), respectively. They are initialized as empty lists. In line 6, the success probability estimate \(p_{i}(t)\) of each MCS level \(i\) is sampled from the beta distribution shown in Equation (4) using the histories of the success and failure events during the sliding window time \(\tau\). The two arguments used by the beta distribution, sum(\(\alpha_{i}\) during \([\max(0,t-\tau),t)\)) and sum(\(\beta_{i}\) during \([\max(0,t-\tau),t)\)), indicate that we only consider the numbers of success and failure events, respectively, that occur between \(t-\tau\) and \(t\), i.e.,\([t-\tau,t)\), if time \(t\) is greater than \(\tau\); otherwise, during \([0,t)\). Again, if \(\tau\) is set as \(\tau\geq t\), the sliding window is disabled, i.e., all the events in \([0,t)\) are considered.  
Line 8 shows the computation of the expected throughput of each MCS level using the log values of \(r_{i}\) and \(p_{i}(t)\). An MCS level \(i\) with the maximum expected throughput is selected for transmission. Ties are broken arbitrarily. Lines 10 and 12 track the success and failure events, respectively. If the data sender receives an ACK or BAck frame from the receiver before the timeout period expires, the event is considered a success and the number of successfully delivered packets in the transmission is appended to \(\alpha_{i}\). Otherwise, the transmission is considered a failure, and the number of the transmitted packets in the transmission is appended to \(\beta_{i}\). Finally, the time of each event is recorded on a separate list (not shown in the pseudocode) to be used for the sliding window.  
If the transmission rate \(r_{i}\) was used without the logarithm in line 8, we call the algorithm TS-R for performance comparison. The difference between TS-R and MTS is that TS-R accommodates the use of the frame aggregation. In addition, TS-R explicitly describes the use of the sliding window over time to adapt to wireless channel environments that can be rapidly changing.  
```
1:for each MCS level \(i\in\{0,...,K-1\}\)do
2:\(\alpha_{i}\leftarrow[],\beta_{i}\leftarrow[]\). \(\triangleright\) Prepare empty lists
3:endfor
4:for each transmission at time \(t\)do
5:for each \(i\)do
6: sample \(p_{i}(t)\) from beta(sum(\(\alpha_{i}\) during \([\max(0,t-\tau),t)\))+1, sum(\(\beta_{i}\) during \([\max(0,t-\tau),t)\))+1).
7:endfor
8: use \(i=\operatorname*{argmax}_{i}(\log(r_{i})\cdot p_{i}(t))\) for transmission and wait for the outcome.
9:if success then
10: append the number of successfully delivered packets to \(\alpha_{i}\).
11:else
12: append the number of transmitted packets to \(\beta_{i}\).
13:endif
14: record the time at which the outcome is observed.
15:endfor
```  
**Algorithm 2** TS-logR algorithm  
To evaluate the performance of the two proposed MAB algorithms in non-aggregated and aggregated packet transmission situations, we used IEEE 802.11g and IEEE 802.11 ax standards in experiments, respectively. For the experiments, we used the event-driven network simulator ns-3 (version 3.35). Moreover, we used ns3-gym [37] (version 1.0.1), which provides a simulation environment for reinforcement learning in wireless communication networks through cooperation with the ns-3.