## 3 Existing QL-Based GF RA Protocols  
Q-learning can be used as a multi-agent RL method to aid IoT devices in selecting the least congested RA pilots in a decentralized way. In typical mMTC networks, active IoT devices cannot coordinate either with the BS or with each other for pilot selection; therefore, IoT devices act as individual learning agents, using their previous experience to enhance the probability of selecting exclusive RA pilots, minimizing the occurrence of pilot collisions. In this section, we revisit two previous works where this framework has been applied in a _simple collision channel_, i.e., assuming that the only communication impairment is _pilot collision_, while neglecting multipath fading, shadowing, path loss, thermal noise, and ICI. If the chosen RA pilot is exclusively selected by that device (no interference), it is assumed that the payload data packet transmitted by it is successfully decoded at the BS, and the device proceeds to transmit the next data packet. Otherwise, if a collision occurs, the device repeats the transmission of the same packet in the next frame. Thus, _reinforcement learning_ techniques can be employed to guide the pilots' choice of devices towards the least congested ones, improving the connectivity performance of the network in the considered scenario. The interaction between IoT devices and the environment can be modeled as a Markov Decision Process, where at each time step, a device can change its current state \(x_{t}\in X\) to \(x_{t+1}\in X\) by taking action \(a_{t}\in A\) based on a transition probability function \(f(x_{t},a_{t},x_{t+1})\)[30]. Depending on its state-action pair, the device is rewarded with \(r_{t+1}\in R\) during the transition state. Besides, the expected return of a state-action pair is given by \(Q^{\pi}(x,a)=\mathbb{E}\left[\sum_{j=0}^{J}\gamma^{j}r_{t+j+1}|x_{t}=x,a_{t}=a, \pi\right]\), where \(\pi\) is the established policy, \(\gamma\in[0,1]\) is the discount factor and \(J\) is the length of one episode [30]. Satisfying the Bellman optimality equation, the Q-function can then be written as \(Q^{*}(x,a)=\max_{\pi}\mathcal{Q}^{\pi}(x,a)\). If a _greedy policy_\(\pi(x)=\arg\max_{a}\mathcal{Q}^{\pi}(x,a)\) is established for the  
Figure 1: Illustrative representation of the adopted cellular mMTC network with IoT devices employing GF RA protocol for their activation and payload data transmission. Q-function, then we have a QL algorithm that selects only the actions associated with the highest Q-value \(Q(x,a)\) at each state, calculated iteratively as:  
\[Q_{t+1}(x_{t},a_{t})=Q_{t}(x_{t},a_{t})\ +\delta_{t}\Big{[}r_{t+1}+\gamma\ \max_{a}Q_{t}(x_{t+1},a)-Q_{t}(x_{t},a_{t})\Big{]}, \tag{3}\]  
where \(\delta_{t}\) is the learning rate at the \(t\)th time step. This model can then be applied for _decentralized pilot selection_ as described in [28, 30], in which each device has a Q-table of \(\tau_{p}\) elements evaluating its experience in selecting the different pilots to transmit data. This Q-table can be updated according to:  
\[Q_{t+1}(k,\ell)=Q_{t}(k,\ell)+\delta[R(k,\ell)-Q_{t}(k,\ell)], \tag{4}\]  
where \(Q_{t}(k,\ell)\) is related to the experience of the \(k\)th device in choosing the pilot \(\ell=c(k)\), and \(R(k,\ell)\) indicates the reward function for this choice. It is worth emphasizing the differences between Equations (3) and (4). The former represents the general framework for establishing a policy for selecting the changing-state actions of a Markov Decision Process under a greedy perspective, i.e., seeking to maximize the associated rewards. Applying this model to the specific problem of decentralized pilot selection in mMTC, the state represents the pilot the device has chosen in the current access attempt, and the action represents the pilot it chooses for the next access attempt. As the action determines univocally the state, the Q-table of a single device can be represented in a single dimension.