# An Adaptive Hybrid Automatic Repeat Request (A-HARQ) Scheme Based on Reinforcement Learning
## 4 The Proposed Adaptive HARQ Q-Learning Model
### Q-Learning Model with the 5G NR-V2X System
#### 4.2.2 Online Decision  
In the online decision procedure, the system will utilize the Q-Table obtained during the offline training stage to execute the selected action, and it will adjust the learning rate \(\alpha\) according to the system feedback. In addition, the system will continue to collect the data to optimize the Q-Table. The algorithm of the online decision is described in Algorithm 2. The agent first determines the action, and then performs the action within the network. Next, the system computes the reward \(R\) according to Equation (4) or Equation (5) and provides feedback after performing the action. The records of \(S\), \(A\), \(R\), \(S^{\prime}\) are saved and ready to be delivered to the buffer. The parameters represent the state, action, reward, and next state, respectively. Afterwards, update the Q-Table using Equation (1) based on the  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Q** & **A0** & **A1** & **A8** \\ \hline S1 &... &... &... \\... &... &... &... \\... &... &... & Q(S, A) &... \\ S400 &... &... &... &... \\ \hline \hline \end{tabular}
\end{table}
Table 4: Initial Q-Table.  
feedback of the environment. The online decision stage includes two modules: the module of decision making and the optimizer module, which will be introduced below.  
```
Input: The basic \(Q\)-Table Output: The optimal action decision Table
1 Parameter set: episode, \(BLER_{\text{MAX}}\), learning rate \(\alpha\), discount factor \(\gamma\);
2 Initialize the basic \(Q\)-Table;
3 for\(i=1\) to episode do
4 Select actions \(A\) based on \(\varepsilon\)-greedy strategy;
5 Compute reward \(R\) and observe next state \(S\);
6 Save record \(S\), \(A\), \(R\), \(S\)' into the replay buffer;
7 Update \(Q\) value for each state and action using Equation (1);
8 Enter the next state \(S\);
9 \(S\gets S\);
10 Adjust the learning rate \(\alpha\) according to the training results;
11 endfor
```  
**Algorithm 2**Online Decision Algorithm  
At the beginning, the agent will initialize the Q-Table in the same way as for the basic Q-Table that was calculated in the offline training stage. In decision making, we also take actions with the \(\varepsilon\)-policy, which is a strategy to weaken the contradiction in reinforcement learning between exploration and exploitation. The model explores the system with a probability \(\varepsilon\) of taking random action and with a probability of 1-\(\varepsilon\) of exploiting the Q-Table. After deciding the action, the 5G NR-V2X system will perform information transmissions according to this action.  
The optimizer can adjust the learning rate of the update algorithm according to the BLER optimization. The BLER feedback from each online training system is utilized to adapt the learning rate \(\alpha\) within a certain range of [0.2, 0.8]. The general idea is that when the system feedback regarding the BLER optimization effect performed by this action is satisfactory, the learning rate \(\alpha\) increases, while when the BLER optimization effect is more general, the learning rate \(\alpha\) decreases.