# Random Access Using Deep Reinforcement Learning in Dense Mobile Networks
## 3 Problem Formulation and System Model
### Problem Formulation  
Any of the models proposed for RA enhancement can be assumed. In a four-way handshake RA, users send preambles at the first step. In three subsequent steps, TRxPs and users exchange control information leveraging resource reservation and collision resolution. For a one-shot-based and two-way RA, data can be sent along with the RA request in two steps. For these models, prior works required separate mathematical analyses to provide an analytical expression of the RA delay. However, in this work, we do not assume a predefined expression, and the analysis presented can be applied to any of these models. As shown in the system model in Figure 3, when a user moves through a TRxP's coverage area, it receives a signal from a single TRxP or some subset depending on the  
Figure 3: Users moving through the network have single or multiple selection options to perform a RA request depending on their position. position. It also contends with other users to seize an RA preamble opportunity. We consider a selection problem where the goal is to connect with a less congested TRxP; therefore, to reduce the amount of time the users spend to perform the RA. We also assessed other RA performance metrics. The problem statement is that we need to select an optimal policy that maximizes the probability of selecting a single best performing TRxP in a given spatial and temporal situation out of the \(K\) TRxPs available for the users to choose from. In such a situation where the user receives multiple signals from the available access points, the user can estimate the congestion level indirectly by ranking the TRxPs according to a delay variable representing an expectation of RA completion time for each. Considering this, users can select the optimal access point. We consider that as users move throughout a network of dense TRxPs, \(\{A_{1},A_{2},\ldots,A_{n}\}\), where at any given RAO time, \(t_{i}\), and position in the network, a user has some subset of TRxP choices of which to consider requesting RA, one TRxP is selected. Weights for each TRxP are assigned by the users for the purpose of the selection; thus, the higher the weight of a TRxP, the higher the chance of it being selected. Therefore, the probability that h-TRxP is selected depends on its weight and is given by:  
\[\frac{w_{i}}{\sum_{\forall i}w_{i}}. \tag{4}\]  
We consider four strategies to obtain a policy that maximizes the selection strategy, which is equivalent to maximizing the probability of selecting a better-performing TRxP. Four strategies for selection are considered, including (1) A strategy that randomly selects one of the available TRxPs, where each has an equal probability of selection, (2) a strategy that considers a channel quality metric based on reference signal received power (RSRP) measurements of users at the current time slot, ranking the TRxPs accordingly, (3) a weight ranking strategy, where weights are assigned based on the previous time slot experiences of a user, and (4) a strategy that considers any combination of the above strategies. A subset of the \(n\) TRxPs are available for a user to select at a given point, and the following weights are assigned to those members: \(\{w_{1},w_{2},\ldots,w_{n}\}\). If we consider the users' experience to rank TRxPs, a weight, \(w_{i}\), of any TRxP can be calculated from the equation given below. \[w_{i}=d_{t-1,a_{i}}=\left\{ \begin{array}{ll}T_{4a_{i}}-T_{1a_{i}},&\text{ in four way RA},\\ T_{2a_{i}}-T_{1a_{i}},&\text{in two way RA},\end{array} \right. \tag{5}\]  
where \(d_{t-1,a_{i}}\) is the user delay experience at a given RAO slot with TRxP \(i\), \(T_{4a_{i}}\) is the time the user completed step four, \(T_{2a_{i}}\) is the time the user completed step two, and \(T_{1a_{i}}\) is the time the user initiated the RA procedure. Let \(D_{t}\) represent a function encoding all the weights from all available TrxPs as follows.