# An Adaptive Hybrid Automatic Repeat Request (A-HARQ) Scheme Based on Reinforcement Learning
## 5 Performance Evaluation  
### Simulation Setup  
The procedures for constructing the simulation are described as follows. Set the channel bandwidth to 5 MHz and the sub-carrier space (SCS) to 15 kHz, as specified in Section 5.3.2 of the 3 GPP TS 38.104 [22]. The simulation employs a lookup table to map the received signal's interference-and-noise ratio (SINR) to the CQI index for a 0.1 BLER [19]. The lookup table corresponds to the CQI table as specified in table of 5.2.2.1-3 of the 3 GPP TS 38.214. The complete bandwidth is assumed to be allotted for the PUSCH/PDSCH. The channel quality is periodically improved or deteriorated by 1 every 0.2 s for all RBs of a UE. Whether the channel conditions for a particular UE improve or deteriorate is randomly determined. The initial value of the CQI for each RB and for each UE is given randomly and is limited by the maximum achievable CQI value corresponding to the distance of the UE from the gNB. According to the 3 GPP TS 38.323, the maximum radio-link control (RLC) session data unit (SDU) length is 9000 bytes [24]. The complete bandwidth is assumed to be allocated for the PUSCH or PDSCH. The above data will be used as the offline training data for this study. Other parameters in this study are set as follows:  
* The subcarrier spacing: 15 KHz
* Periodicity (in ms) at which the UL packets are generated by UEs: 30 ms
* Size of the UL packets (in bytes) generated by UEs: 5000 B
* Periodicity (in ms) at which the DL packets are generated for UEs at gNB: 20 ms* Size of the DL packets generated (in bytes) for UEs at gNB: 6000 B We create 4 mobile users in the simulation environment, and an equidistance between each UE and the base station. According to the state division in Section 4.1.2, after modifying and debugging the corresponding user \(Distance\) and \(NumRBs\) environment parameters in the built simulation environment, the corresponding downlink average CQI and downlink average BLER are obtained. The above data will be used as the offline training data in this study, and a total of 400 sets of simulation data will be obtained.  
The principle of the offline training and the online decision-making algorithm is basically the same. However, the initial Q-Tables of the two are different, and the learning rate in the online decision-making algorithm will be adjusted according to the system feedback. The initial state of each learning sets the random state selection strategy, so that all states have the possibility of traversing it. The Q-Table is updated according to the \(\varepsilon\)-greedy strategy, and when the set number of training times of end convergence is reached, the agent will stop training.  
Each state space in this study is an independent simulation experiment. The process of training is not a sequential accumulation of continuous actions and feedback, but each training iteration is relatively independent. The state selection process is shown in Figure 5.  
According to the Monte Carlo algorithm, the agent should be able to traverse 9 actions in 400 states to ensure the effectiveness of the algorithm, so the training episodes should meet at least 3600 times. Secondly, in accordance with the principle of Markov chains, it is expected that increasing the number of training times will result in more obvious signs of convergence. In order to make the agent more certain about the choice of the best strategy, the termination condition for the offline training is set at 100,000 training iterations.