# A Reinforcement-Learning-Based Distributed Resource Selection Algorithm for Massive IoT
## 5 Proposal: TOW Dynamics-Based Channel Selection Algorithm and Its IoT Prototype Implementation  
In this section, the proposed TOW dynamics-based channel selection algorithm and its prototype implementation are described. The proposed algorithm works extremely simply. Accordingly, only minimal memory and computation capability are necessary to implement the algorithm on a resource-constrained IoT device. The IoT device conducts the proposed channel selection algorithm to update the accumulated reward estimates, \(X_{k}(t+1)\), of the all available channels simultaneously following the learning process described in Section 4. Based on the system model and problem formulation introduced in Sections 2.2 and 3.2 respectively, the IoT device initially wakes up and randomly selects a channel from the available \(K\) channels. As mentioned in Section 3.1, if the IoT device sends the data frame over selected channel \(k^{*}\) and receives the ACK frame from the destination, then the IoT device will recognize the data transmission as a success over the selected channel, \(k^{*}\), which implies that it is rewarded, and \(\Delta Q_{k^{*}}\) given by Equation (2) is set as \(+1\). Otherwise, if the IoT device does not receive the ACK frame, it will recognize the data transmission over the selected channel, \(k^{*}\), as failed, which implies that it is unrewarded, and \(\Delta Q_{k^{*}}\) is set as \(-\omega(t)\). Figure 4 shows the flowchart of the proposed procedure for each IoT device. Essentially, the proposed procedure follows the same procedure as that of the non-beacon-enabled mode of IEEE 802.15.4. Therefore, the IoT device periodically goes to the sleep mode to save power after sending a data frame. Each IoT device initializes \(Q_{k}(0)\), \(R_{k}(0)\), and \(N_{k}(0)\) at \(t=0\). Once the IoT device wakes up at \(t=0\), it sends the data frames over the selected channel \(k^{*}(0)=\arg\max_{k\in K}X_{k}(0)\). Because the \(X_{k}(0)\) is randomly set to the same value for all available channels initially, the IoT device actually randomly selects a channel to send data frames at \(t=0\). Meanwhile, the count of transmissions over channel \(k^{*}(0)\), denoted as \(N_{k^{*}(0)}\), is incremented by one. If the IoT device receives the ACK frame successfully, the channel \(k^{*}(0)\) is rewarded, and \(\Delta Q_{k^{*}}(0)=+1\) is added to \(Q_{k^{*}}(0)\). Meanwhile, the counts of rewards \(R_{k^{*}}(0)\) is incremented by one accordingly. Otherwise, the selected channel \(k^{*}(0)\) is unrewarded, and \(\Delta Q_{k^{*}}(0)=-\omega(0)\) is added to \(Q_{k^{*}}(0)\) for updating. Regarding those unselected channels \(k(k\
eq k^{*})\) at \(t=0\), \(\Delta Q_{k(k\
eq k^{*}(0))}(0)=0\). The reward estimates of the unselected channels are given by \(Q_{k(k\
eq k^{*}(0))}(1)=Q_{k}(0)\) accordingly. Then, \(X_{k}(1)\) of each available channel is updated by Equation (3) for the next time \(t=1\). Meanwhile, the reward probability estimate of the selected channel \(k^{*}\), i.e., \(p_{k^{*}(1)}\), is updated for the next time \(t=1\) according to Equation (11). In addition, the appropriate value of weight parameter \(\omega(1)\) is also updated adaptively based on Equation (10) for the next time \(t=1\) Then, the IoT device goes to sleep mode again and wakes up at \(t++(t=1\)_here_) to send a data frame over the selected channel with the highest \(X_{k}(1)\), which has been calculated at \(t=0\). We implemented the proposed channel selection algorithm on an SBC shown in Figure 5. As described above, the cognitive-IoT prototype only needs to receive the ACK frame to explore the channel selection strategy. Concurrently, the proposed TOW dynamics-based learning process only requires adding one or subtracting \(\omega\) to update the reward estimate. Therefore, to implement the proposal, only addition and subtraction are necessary based on the procedure of the proposal. Thus, minimal memory and computation capability are enough for prototyping the proposed reinforcement-learning-based channel selection algorithm.