# An Adaptive Hybrid Automatic Repeat Request (A-HARQ) Scheme Based on Reinforcement Learning
## 4 The Proposed Adaptive HARQ Q-Learning Model
### Model Description
#### 4.1.3 Action Space  
A simple and efficient action-space design can reduce the difficulty of convergence and improve the training speed. The designed action space in this study is \(A_{k}\), \((k=0,\;1,\;\ldots,\;8)\), giving a total of nine discrete program actions, where A0 is the 5G HARQ transmission scheme shown in Figure 1, A1 and A2 are \(T\)-delay scheme schemes, A3 and A4 are \(K\)-repetition schemes, and A5-A8 are [T, K]-overlap schemes. Details are shown in Table 2, where A1-A8 correspond to the eight HARQ alternative transmission schemes shown in Figure 3.  
Although a larger number of retransmissions increases the success rate of packet transmission, the waiting time for the receiver to confirm the NACK signal and retransmitting will greatly affect the delay during retransmission. So, in order to save the time and resources required for retransmission, we limit the K-repetition scheme up to 3 times, that is, 1 normal transmission plus 2 consecutive retransmissions. The correct selection of the designed optimized transmission scheme can avoid the waste caused by the redundancy of channel resources when the channel quality is good and the long delay and high BLER caused by repeated retransmission when the channel quality is poor. For example, when CQI is poor, the probability of information transmission failure is large. In this case, the information can be sent with the \(T\)-delay scheme (delay 1 slot or delay 2 slots) to improve the success rate of the information transmission. Because an appropriate delay transmission scheme can avoid an environment with poor CQI, there is a greater probability of reducing the average BLER while releasing more limited channel resources at the same time. It is worth noting that although it seems that the \(T\)-delay scheme will temporarily waste time, the fact is that, the average number of retransmissions is reduced because it may avoid interference. Consequently, both the overall transmission time and the average delay are subsequently reduced. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**State** & **BLER** & **CQI** & **Distance** & **NumRBs** \\ \hline S1 & \(\ldots\) & \(\ldots\) & \([0,100]\) & \([0,20]\) \\ S2 & \(\ldots\) & \(\ldots\) & \([0,100]\) & \([20,25]\) \\ S3 & \(\ldots\) & \(\ldots\) & \([0,100]\) & \([25,30]\) \\ \(\ldots\) & \(\ldots\) & \(\ldots\) & \(\ldots\) & \(\ldots\) \\ S400 & \(\ldots\) & \(\ldots\) & \([1900,2000]\) & \([115,120]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: State space. \begin{table}
\begin{tabular}{c c} \hline \hline
**Action** & **A-HARQ Scheme** \\ \hline A0 & 5G HARQ scheme \\ A1 & \(T\)-delay scheme where T = 1 slot offset \\ A2 & \(T\)-delay scheme where T = 2 slot offset \\ A3 & \(K\)-repetition scheme where K = 1 \\ A4 & \(K\)-repetition scheme where K = 2 \\ A5 & [T, K]-overlap scheme where [T, K] = \([1,1]\) \\ A6 & [T, K]-overlap scheme where [T, K] = \([1,2]\) \\ A7 & [T, K]-overlap scheme where [T, K] = \([2,1]\) \\ A8 & [T, K]-overlap scheme where [T, K] = \([2,2]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Action space. When the user is far away, there exists a high probability of information transmission failure. In this case, the information can be sent with the \(K\)-repetition scheme (send 2 or 3 repetitions) to enhance the success rate of the information transmission and reduce the average transmission delay. Because the appropriate repetition transmission mechanism can save the waiting time after the receiver confirms the NACK signal when the original retransmission is required, this reduces the average BLER. It is worth noting that the use of a continuous repetition transmission mechanism will temporarily borrow channel resources; however, due to the improvement in the transmission success rate, the original required number of retransmissions is greatly reduced, and the channel resources will also make up for it. Moreover, when the distance is far and the CQI is poor at the same time, the [T, K]-overlap scheme of the repetition sent after delay can be selected.