# Random Access Using Deep Reinforcement Learning in Dense Mobile Networks
## 4 RL Based Selection of TRxPs for RA
### Algorithms  
In this section, we present the pseudocode for the four algorithms proposed and analyze their performance through a comparison. Algorithms 1 and 2 describe our proposals in this work. Algorithm 3 is a \(Q\)-based algorithm [39], and Algorithm 4 is a random TRxP selector. The baseline algorithm is the original 3 GPP selection algorithm, which selects a TRxP based solely on the RSRP measurement, i.e., a user ranks and assigns a numerical weight to each TRxP that it receives a pilot signal from. Finally, the TRxP that returned the highest rank was selected for the RA procedure.  
```
1:Initialize discount rate (\(\gamma\in(0,1)\)), \(\epsilon\)-greedy (\(\epsilon\in(0,1)\)) rate value, and the range of an episode
2:Start RA in a mobile network
3:while (Episode is not finished) do
4:while (Not every position is explored or step is reached) do
5: Get the visible TRxPs at the current position according to Equation (9)
6: Select a TRxP based on the \(\epsilon\)-greedy policy; predict a TRxP returning the minimum RA delay from the deep neural network or select a random TRxP.
7: Receive the reward according to Equation (20)
8: Remember (CurrentPosition, selectedTRxP, reward, nextSelectedPosition, trainingEndMarker)
9: Replay by sampling the experiences obtained from the above steps from 5 to 8
10: Train by updating the weights of the DQN
11:endwhile
```  
**Algorithm 1** DQN-based Intelligent TRxPs Selector Algorithm: Training.  
The online algorithm given in Algorithm 3 is based on a tabular \(Q\) value function. It updates each access point (TRxP here) according to Equation (25) given below. In Algorithm 1, the agent is trained for a number of episodes. In each step of a single episode, it aims to explore as many user positions as possible to gain an understanding of the network environment. Each selection decision moves the state of the agent from RRC_IDLE mode to RRC_Connected and then back to the former, to explore more positions.  
The Deep \(Q\)-Network (DQN) algorithm has two neural networks in the implementation. One estimates the \(Q\) value and transfers the learned weights to the other neural network. Finally, a mini-batch of experiences is sampled, and the neural network is trained on the updated information. The online algorithm given in Algorithm 2 makes use of the output obtained from Algorithm 1. Algorithm 2 executes when the user needs to perform the RA (i.e., it has a full buffer and passed system parameter criterion). To further minimize delay, additional training on the updated environment experiences is run only when the user is in RRC_IDLE mode. Here, \(Q(s,a)\) is updated as follows.  
\[Q(S,a)=\alpha r+(1-\alpha)Q(S,a). \tag{25}\]  
```
1:Initialize discount rate (\(\gamma\in(0,1)\)), \(\epsilon\)-greedy (\(\epsilon\in(0,1)\)) rate value, and the range of an episode
2:Start the RA network
3:while (Episode is not finished) do
4:while (Not every position is explored or steps is reached) do
5: Get the visible TRxPs at the current position according to the equation given in (9)
6:if (empty \(Q\) table) then
7: Select a random TRxP
8: Perform RA with selected TRxP, receive reward and update the \(Q\) value of that TRxP
9:else
10: Select a TRxP based on an \(\epsilon\)-greedy policy; get the TRxP that returned highest \(Q\) value or select a random TRxP.
11: Receive the reward according to Equation (20)
12: Update \(Q\) value and get the next state
13:endif
14:endwhile
```  
**Algorithm 3**\(Q\)-Based Intelligent TRxPs Selector Algorithm: Online.