# A Reinforcement-Learning-Based Distributed Resource Selection Algorithm for Massive IoT
## 2 IoT System  
### IEEE 802.15.4-Oriented Wireless Technology for IoT  
The IEEE 802.15.4 standard [4], characterized by low power, low cost, high reliability, and simplicity in the industrial, scientific, and medical (ISM) frequency band, is one of the key enabling technologies for IoT deployment. Therefore, the IEEE 802.15.4-oriented wireless devices have been widely deployed in many applications such as smart homes, environmental monitoring, and industrial automation.  
The IEEE 802.15.4g amendment [14] defines new PHY in the 920-MHz band for smart utility application and focuses on range. Three alternative PHYs, i.e., frequency shift keying (FSK), offset quadrature phase shift keying (O-QPSK), and orthogonal frequency division multiplexing (OFDM) are supported. FSK and O-QPSK are conventional well-known modulation techniques, while OFDM is commonly used in advanced systems. Any IEEE 802.15.4g-compliant chip must implement a PHYwith two-FSK modulation and a 50-kbps data rate. All other PHYs are optional. Each PHY allows further parametrization with data rates ranging from 6.25 kbps-800 kbps. In this work, the proposal only focuses on the MAC layer procedure, which is introduced as below.  
Depending on the massive IoT requirements, a PAN consists of one PAN coordinator or a gateway and multiple sensors and transmits the data to the coordinator or gateway with a simple star topology. In addition, the carrier sense multiple access algorithms with collision avoidance (CSMA/CA) algorithm is utilized as the radio channel access method for IoT devices. To satisfy the requirements of massive IoT, typically, a non-beacon-enabled mode, employing a simple unslotted CSMA/CA and direct transmission, is used to relieve the strict synchronization and minimize unnecessary protocol overheads. Three variables for attempting transmission, i.e., number of back-offs (\(NB\)), contention window (\(CW\)), and back-off exponent (\(BE\)), are maintained for each device. \(NB\) is the number of times required to back-off while attempting the current transmission based on the CSMA/CA algorithm. \(CW\) defines the number of back-off periods that need to be clear of the channel activity before transmitting. \(BE\) is the back-off exponent to determine the number of slots that a device has to wait before accessing a channel.  
Figure 1 describes an unslotted CSMA/CA algorithm. Initially, the \(NB\), \(CW\), and \(BE\) are initialized as 0, 2, and \(macMinBE\), respectively. Then, the device delays for a random back-off duration in the range from \(0\)-\(2^{BE}-1\). Subsequently, the PHY layer detects whether the channel is idle or not by performing a clear channel assessment (CCA). If the channel is assessed as idle, the MAC layer proceeds with the remaining CSMA/CA algorithm steps, including the frame transmission and acknowledgment (ACK) frame reception. If the ACK is received correctly, it terminates with a transmission success status, otherwise it terminates with a transmission failure status. If the channel is assessed to be busy, the MAC layer increments both the \(NB\) and \(BE\) by one, ensuring that the \(BE\leq aMaxBE\), and the \(CW\) is set as two. If \(NB\leq macMaxCSMABackoffs\), the CSMA/CA algorithm starts the back-off again. Otherwise, it terminates with a failure status.  
Figure 1: Unslotted CSMA/CA algorithm of IEEE 802.15.4. CCA, clear channel assessment.