# Random Access Using Deep Reinforcement Learning in Dense Mobile Networks
## 4 RL Based Selection of TRxPs for RA
### Proposed RL Based Selection for RA  
In the RL algorithm, users learn the selection policy through interaction with the RA environment, as depicted in Figure 4. When a user needs to establish/re-establish a network connection with a TRxP or the buffer is ready, the learning algorithm selects a TRxP before proceeding to the RA procedure. This is done before exchanging message 1 in the RA protocol procedure, illustrated in Figure 2. The user observes the state, takes an action, receives a reward, and observes the next state along with their current connection status. The reward is a factor of the RA's key performance indicators (KPIs), explained in detail in Section 4.4.2. In general, it combines values from both the state and action sets with KPIs. This can be represented by using a Markov decision process, as in a simple tuple \((S,A,r,S^{{}^{\prime}})\), where \(S\) is the state space, \(A\) is the action space, \(r\) is the reward, and \(S^{{}^{\prime}}\) is the new state. An agent is initially at a given state. It takes action and receives a reward. After that, it transits to the next state. Transition probabilities are not predetermined. The goal of such users is to maximize their expected long-term reward value. The KPI values can be accumulated to measure whether users are maximizing their reward. Starting from a certain position, where the network starts, the expected long-term reward can be formulated as in the following equation. \[E[r]=\sum_{i=0}^{\infty}(\gamma_{i}S_{i}|S=S_{0}), \tag{10}\]  
where \(S_{0}\) is the initial state, \(S_{i}\) is the \(i\)th state, and \(\gamma<1\). The action value function estimation is based on a neural network (thus deep RL) and allows the selection of the best action, which refers to the selection of the better performing access point. In the case of \(Q\)-based tabular learning, we can store the values in a \(Q\)-table initialized to null or infinity values. As the agent traverses through the network and gains information, the table is updated progressively. For deep RL, the algorithm itself approximates this action value function. We approach the problem by solving the deep RL task for two main reasons. First, the difficulty of the \(Q\)-based tabular algorithm in estimating new unseen positions, and second, the complexity of maintaining the table with increasing network size. Therefore, instead of using empty tables for the value-function approximation or using only \(Q\) tables, our neural network approach estimates the state-action-value function without the need for \(Q\) tables. The agent performs neural-network-based RL. It selects the optimal actions; that is, a user must be attached to a nearby cell based on the stored experience and current state. The agent also performs exploration with a certain probability and receives rewards in terms of the delay experienced by the user during the RA request. The reward is then recorded to update past experiences. In this manner, the agent makes predictions as well as learns from past experiences. Figure 4: The Deep \(Q\)-Network (DQN)-based agent interacts with the random access network environment to receive rewards. Finally, we can find the optimal policy by evaluating the optimal \(Q\) value, given in Equation (11) below. \(Q(S,a)\) measures the importance of taking action \(a\) in state \(S\). Out of the many actions listed, the optimal policy selects the optimal \(Q\) value in the following manner. During exploitation, it selects an action with a probability of \((1-\epsilon)\), and during exploration, with a probability of \(\epsilon\). Exploration is performed at the beginning of the training. As the agent advances through training, \(\epsilon\) decays with a decaying factor so that exploitation of the experiences observed so far begins to be utilized. To summarize, \(Q(S,a)\) is updated as follows.