# A Reinforcement-Learning-Based Distributed Resource Selection Algorithm for Massive IoT
## 3 Problem Formulation
### Channel Selection Problem as an MAB Problem  
According to the system model introduced in Section 2.2, the objective of the distributed channel selection problem for IoT devices is to maximize the total frames successfully transmitted via an optimal channel selection strategy. We assumed distributed channel selection processes for multiple IoT devices with no prior coordination. Each IoT device can select a set \(K=\{1,...,k\}\) of available channels. Here, \(k\) refers to the index of the channel. For simplicity, we assumed that each IoT device can access one channel at a time. The distributed channel selection problem of IoT device belongs to the class of MAB. The challenge in distributed channel selection problem is similar to that in the MAB problem: an IoT device (player) has \(K\) channels (slot machines), the \(k^{\text{th}}\) of which has successful frame transmission possibility \(p_{k},k\in K\) (reward probability). The IoT device does not know the values of the \(p_{k}\)s and must sequentially select a channel to send the data frame (select a machine to play). The target is to maximize the total successful frame transmission (overall gain) for a total of \(T\) transmissions (plays). The distributed channel selection problem and MAB problem share the same concept, which is summarized in Table 1.  
An IoT device selects channel \(k^{*}(t)\in\mathcal{K}\) at current time \(t\). If the IoT device sends the data frame over a selected channel \(k^{*}\) successfully, the selected channel, \(k^{*}\), receives a reward. We assume that the IoT devices follow a generalized version of the unslotted CSMA/CA algorithm introduced in Section 2.2 For more details, if channel \(k^{*}(t)\) is selected, an IoT device will wait the time specified by the generated random number. At the end of the waiting period, the IoT device senses the selected channel again, and if it is found to be busy, it will generate another random number following the CSMA/CA algorithm. It will then detect the selected channel, \(k^{*}(t)\), again until reaching the predetermined maximum challenge time. If the selected channel, \(k^{*}(t)\), is still found busy when reaching the maximum retry time, the IoT device will discard the data frame and recognize the current data frame transmission as failed. However, if the selected channel, \(k^{*}(t)\), is assessed as idle, the data frame will be transmitted by the IoT device. Moreover, if the ACK frame is received successfully from the destination of the data frame transmitted previously, then the IoT device will recognize the transmitted data frame over the selected channel, \(k^{*}(t)\), as a success. If the ACK frame is not received successfully from the destination of the data frame transmitted previously, then the IoT device will recognize the transmitted data frame over the selected channel, \(k^{*}(t)\), as failed. Regardless of the success of the current data transmission, the IoT device will enter sleep and possibly select a different channel to access when it wakes up the next time. The challenge stems from the uncertainty in the successful frame transmission probability, \(p_{k}(t)\), imposing a trade-off between the exploration learning \(p_{k}(t)\), and exploitation by selecting the channel with the highest estimated successful frame transmission probability, \(p_{k}(t)\), based on the currently available information. An IoT device employs a strategy that will select channel \(k\in K\) to access at current time \(t\) according to any possible causal information pattern obtained from the previous \(t-1\) observations. This is done to maximize the accumulated reward, \(X_{k}(t)\), i.e., the total successful frame transmission.