# An Adaptive Hybrid Automatic Repeat Request (A-HARQ) Scheme Based on Reinforcement Learning
## 3 Problem Statement and System Framework
### System Overview  
As a technical document defined by the 3 GPP, the BLER is used to estimate the errors of the physical layer. In this study, the powerful learning ability of Q-learning in reinforcement  
Figure 1: 5G HARQ scheme of retransmission once (**a**) and retransmission twice (**b**).  
learning is utilized to obtain better communication schemes. The interaction between the agent and the environment in Q-learning will be determined by the update process of the Q-Table, which basically employs the TD Bellman Equation. In this study, the sequential difference method is used to set the Reward Calculator mechanism, and the update of the Bellman Equation is adopted to obtain the optimal strategy. The Bellman Update Equation is as in Equation (1):  
\[Q(S,A)\gets Q(S,A)+\alpha*\big{[}R+\gamma*\max Q(S^{\prime},A)-Q(S,A) \big{]}, \tag{1}\]  
where \(\alpha\) is the learning rate, \(R\) is the immediate benefits, and \(\gamma\) is the discount (or attenuation) rate. The larger \(\alpha\) is, the less the effect of retaining the previous training. And \(\max Q(S^{\prime},A)\) is the benefit in memory, and it refers to the maximum utility value in the action of the next state \(S^{\prime}\). The agent aims to obtain the maximum reward after choosing the action in the next state \(S^{\prime}\), so that the next time in state \(S\), it can continue to get the reward by choosing the correct action. The larger the value of \(\gamma\), the greater the role played by \(\max Q(S^{\prime},A)\), i.e., the more attention is paid to past experiences. In contrast, the smaller the \(\gamma\), the more attention is paid to the immediate benefits \(R\).  
Q-learning is a reinforcement learning approach that learns a policy that maximizes the expected reward through training and feedback. In the Adaptive HARQ Q-learning model, we design a retransmission program with a Q-learning model to determine the optimal retransmission time and frequency, thereby enhancing network performance.  
The functional component blocks of the adaptive HARQ (A-HARQ) scheme are depicted in Figure 2, where BLER is the average downlink block error rate, \(CQI\) is the average CQI, Distance is the distance between UEs, and NumRBs is the number of resource blocks. The A-HARQ scheme takes BLER, \(CQI\), Distance and NumRBs as input linguistic variables. A state classifier is responsible for classifying the current transmission state according to the input linguistic variables. The state identity denoted by \(Si\) can then be obtained from the state classifier. The optimal action \(Ak\) for \(Si\) is inferred from the \(\varepsilon\)-greedy strategy. By the action decision, a suitable optimized transmission scheme can be determined by the A-HARQ scheme. After the reward \(R(Si,Ak)\) is generated by the Reward Calculator based on the system feedback of the transmission result, the Q values \(q(Si,Ak)\) in the Q-Table are updated by the Q-Function Update. Moreover, the learning rate \(\alpha\) of the Q-Table will be adjusted by the Optimizer according to the feedback results of the system when making online decisions. The detailed design is given as Section 4.1.  
Figure 2: The architecture of the A-HARQ scheme.