## 5 Numerical Results  
In this section, we evaluate the performance of the proposed QL-based GF RA protocol in terms of: \((i)\) average latency, considered herein as the total number of attempts, \(A_{k}\), the device makes to transmit its \(L_{k}\) packets, \((ii)\) average network throughput, defined as the ratio between the number of successfully transmitted packets (without collisions) at certain time step and the number of available pilots \(\tau_{p}\), and \((iii)\) the per-device throughput, considered as the ratio of the total number of successfully sent packets by each device, \(L_{k}\), to the total number of attempts, \(A_{k}\), that the device has to make to send them, in such a way that \(L_{k}\leq A_{k}\). For the simulations, we consider a massive MIMO BS equipped with \(M\) antennas at the center of a hexagonal cell with a radius of \(250\,\mathrm{m}\), surrounded by six neighboring hexagonal cells with the same radius. Each neighboring cell has a fixed number of \(K_{\text{ici}}=400\) active interfering devices. The simulation parameters are set as \(\rho=27\) dBm, \(q=40\) dBm, \(\tau_{p}=400\), and \(\delta=0.1\). It is worth noting that the choice of the parameters is based on [29,30] in the case of \(\tau_{p}\) and \(\delta\), while being based on [8,31] in the case of \(\rho\) and \(q\). Furthermore, concerning the number of active interfering devices in the neighboring cells \(K_{\text{ici}}\), it is made equal to \(\tau_{p}\) similarly as in [8]. Table 1 summarizes the numerical parameters adopted in our simulations. With respect to \(L\) and \(P_{a}\), we investigate three different scenarios in this section: \((i)\)\(L_{k}=L\), \(\forall k\in\mathcal{K}\), and \(P_{a}=1\), such that \(K_{a}=K\); \((ii)\) random \(L_{k}\), and \(P_{a}=1\), such that \(K_{a}=K\); \((iii)\) random \(L_{k}\), \(\forall k\in\mathcal{K}\), and \(P_{a}=0.1\%\), such that \(K_{a}\leq K\) is also random. One can see that while scenario \((i)\) results equal to that evaluated in [29,30], scenarios \((ii)\) and \((iii)\) become gradually more realistic and challenging with a random number of packets and active devices. \begin{table}
\begin{tabular}{c c l} \hline \hline
**Parameter** & **Value** & **Description** \\ \hline \(M\) & \(100\) & Number of BS antennas in the center and neighboring cells \\ \hline \(\tau_{p}\) & \(400\) & Number of available RA pilot sequences \\ \hline \(q\) & \(40\) dBm & Transmit power of the BS \\ \hline \(\rho\) & \(27\) dBm & Transmit power of the UEs \\ \hline \(\sigma^{2}\) & \(-98.65\) dBm & Noise variance \\ \hline \(f_{c}\) & \(3\) GHz & Carrier frequency \\ \hline \(\delta\) & \(0.1\) & Learning rate \\ \hline \(K_{\text{ici}}\) & \(400\) & Number of active devices in each neighboring cell \\ \hline \(R\) & \(250\) m & Radius of the cells \\ \hline \(\sigma_{\text{sf}}\) & \(10\) dB & Shadow fading standard deviation \\ \hline \(\kappa\) & \(3.8\) & Path loss exponent \\ \hline \(g\) & \(-34.53\) dB & Path loss at the reference distance \\ \hline \multicolumn{3}{c}{10,000} & Number of Monte-Carlo realizations \\ \hline \multicolumn{3}{c}{27 dBm} & Transmit power of UEs in adjacent cells \\ \multicolumn{3}{c}{6} & Number of neighboring cells \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Numerical Parameters for Simulation Settings. We investigate four protocols: the (**a**) _baseline_ scheme, which is equivalent to the slotted ALOHA protocol, with the devices choosing the pilots uniformly at random; (**b**) the \(iQmMTC\) approach of [28], (**c**) the \(cQmMTC\) approach of [30], assuming that the actual values of \(|S_{t}|\) and \(K_{a}\) are perfectly known at the devices' side, like if a genie could inform this to them; and (**d**) our proposed two-step QL GF RA protocol leveraging the _massive_ MIMO propagation features to efficiently compute the negative rewards of the QL framework at the devices' side, which we denote as \(mQmMTC\).