# An Adaptive Hybrid Automatic Repeat Request (A-HARQ) Scheme Based on Reinforcement Learning
## 4 The Proposed Adaptive HARQ Q-Learning Model
### Q-Learning Model with the 5G NR-V2X System
#### 4.2.1 Offline Training  
Offline training in Algorithm 1 employs four modules: _Collector_, _Replay Buffer_, _State Classifier_, and _Trainer_, as illustrated in Figure 4. This stage aims to obtain a basic rule table, which is the basis for the online decision. The rule table, named Q-Table, represents the expected reward of each executable action for all states.  
The procedure of Algorithm 1 is presented as follows: Firstly, the state and action information is imported from the replay buffer, as well as the reward calculator function. The _episode_, \(BLER_{\text{MAX}}\), learning rate \(\alpha\), and discount factor \(\gamma\) are set respectively, and then an all-zero Q-Table is initialized. After the initial state is randomly selected, the training begins, the action is selected according to the \(\varepsilon\)-greedy strategy, and the corresponding reward \(R\) is obtained according to the reward rule. Then, enter the next state after executing the current action, obtain the current Q value from the Q-Table and calculate the new Q value, and then update the Q-Table according to the Bellman Update formula. The training is performed until the end of the number of iterations. The following section provides a comprehensive description of each module.  
In training, the agent collects data from the environment and stores them in the replay buffer. To keep the agent from making bad decisions, there is no prior knowledge at the beginning; the collector collects data from the environment and subsequently stores them in the replay buffer in the form (_BLER_, _CQI_, _Distance_, _NumRBs_). The parameters represent the average downlink block error rate downlink, average channel quality indication, user distance, and number of resource blocks, respectively.  
Figure 4: The asynchronous framework of the proposed mechanism.  
```
Input: The replay buffer Output: A basic Q-Table  Parameter set: \(episode\), \(BLER_{\text{MAX}}\), learning rate \(\alpha\), discount factor \(\gamma\);  Initialize the Q-Table;  Enter state \(S\) randomly from the replay buffer; for\(i=1\) toepisodedo ifs \(<\) e then  Select action A randomly; else  A = argmax (\(Q\) value); endif  Get reward \(R\) according to the Reward Calculator;  Update \(Q\) value for each state and action using Equation (1);  Enter the next state \(S\); endif endfor
```  
**Algorithm 1**Offline Training Algorithm for Agent  
Then, the _State Classifier_ sorts the data into different categories of the state space. According to the state division and action design in Section 4.1, 400 states (S1-S400) and 9 actions (A0-A8) are obtained to form the matrix of Q-Table of 400 \(\times\) 9, as illustrated in Table 4. The initial Q-Table is initialized with all zeros, and it is filled by conducting pretraining with the samples collected from the environment.  
Finally, train the samples and calculate a basic Q-Table, which is also the optimal action decision table. In order to make the algorithm converge and to maintain a certainly stability, the learning rate \(\alpha\) is set to the middle value of 0.5 and the discount factor \(\gamma\) is 0.8. In order to make the training effect play a greater role, the \(\varepsilon\)-greedy strategy is set to select the action. The model explores the system with probability \(\varepsilon\) to take random action and with probability \(\varepsilon\) to exploit the Q-Table. Also, set the attenuation factor to prevent the energy from being wasted in randomly selecting the non-optimal action after convergence. The \(\varepsilon\) decreases from 0.3 to 0.1 as the number of training episodes increases. The \(\varepsilon\)-greedy algorithm can ensure that the state of the state space is fully traversed to ensure that the optimal strategy is finally converged.