# An Adaptive Hybrid Automatic Repeat Request (A-HARQ) Scheme Based on Reinforcement Learning
## 4 The Proposed Adaptive HARQ Q-Learning Model
### Model Description
#### 4.1.4 Reward Calculator  
The objective of the A-HARQ mechanism is to select the appropriate transmission scheme for each packet to reduce the entire retransmission in the 5G NR-V2X physical layer. The objective of Q-learning in the A-HARQ employs a learning model to choose the optimal scheme that can reduce the average BLER across various transmission environments. Based on our design, the selection of transmission scheme (A1-A8) is used in the case that the transmission fails to meet the BLER requirements, while the transmission scheme of the original 5G HARQ (A0) is used for cases that meet the BLER requirement. Therefore, it is possible to initially set the maximum tolerable BLER threshold, denoted as \(BLER_{\text{MAX}}\), which can be adjusted as necessary. In our observation, the BLER threshold is set as \(BLER_{\text{MAX}}=0.098\) in the simulation environment. The original 5G HARQ BLER parameter is denoted as \(BLER_{\text{HARQ}}\), and the proposed adaptive HARQ BLER parameter is denoted as \(BLER_{\text{A-HARQ}}\). The design of the Reward Calculator is mainly based on whether the BLER is not greater than the \(BLER_{\text{MAX}}\). When the value of the \(BLER_{\text{HARQ}}\) is smaller than the \(BLER_{\text{MAX}}\), two reward rules, \(R_{\text{I}}\) and \(R_{\text{II}}\), can be selected in the Reward Calculator, which are set as follows:  
* The reward rule corresponding to the state that meets the requirements of the BLER, i.e., \(BLER_{\text{HARQ}}\leq BLER_{\text{MAX}}\), is \(R_{\text{I}}\), defined as in Equation (3): \[R_{\text{I}}=\left\{ \begin{array}{rl}10,Action=\text{A}_{0}\\ -100,Action\
eq\text{A}_{0}\end{array} \right.\] (3) In this case, choosing the 5G HARQ scheme (A0) will result in a positive reward, while choosing other optimized transmission schemes (A1-A8) will result in a punishment. * The reward rule corresponding to the state that does not meet the requirements of the BLER, i.e., \(BLER_{\text{HARQ}}>BLER_{\text{MAX}}\), is \(R_{\text{II}}\), defined as in Equation (4): \[R_{\text{II}}=\left\{ \begin{array}{rl}-100,Action=\text{A}_{0}\\ R_{0}+R_{\text{p}},Action\
eq\text{A}_{0}\end{array} \right.\] (4) In the case of the unsatisfied BLER threshold, the real change in BLER (\(BLER_{\text{HARQ}}\)\(-\)\(BLER_{\text{A-HARQ}}\)) after execution of the A-HARQ decision is added as the main basis, and factors such as distance, CQI, and RB number are also considered. In this case, the selection of the 5G HARQ scheme (A0) will be punished, whereas the selection of other optimized transmission schemes (A1-A8) will be rewarded or punished correspondingly according to the key factor of the BLER change. The reward rule of \(R_{\text{II}}\) consists of the base reward \(R_{0}\) and the reinforcement reward \(R_{\text{p}}\). The base reward \(R_{0}\), reflecting the BLER reduction, can be expressed as Equation (5):  
\[R_{0}=\left\{ \begin{array}{rl}-100,BLER_{\text{A-HARQ}}\geq BLER_{\text{ HARQ}}\\ \tau(BLER_{\text{HARQ}}-BLER_{\text{AHARQ}}),BLER_{\text{A-HARQ}}<BLER_{\text{ HARQ}}\end{array} \right., \tag{5}\]  
where \(\tau\) is the reward coefficient, set at \(\tau=100\). The smaller the \(BLER_{\text{A-HARQ}}\) after the A-HARQ scheme, the greater the reward \(R_{0}\). If the transmission scheme only pursues the goal of reducing the BLER while ignoring the actual situation of the current network environment quality and available resources, it will cause a waste of limited time and space resources. Therefore, in the formulation of the reward function in this study, in addition to reducing the BLER, we also need to comprehensively consider the quality of the channel, UE distance, and available resource blocks under various circumstances. Only in this way can we achieve a reasonable allocation of transmission time slots and limited channel resources, reduce the total transmission time, and improve the reliability of data transmission.