# Q-LBR: Q-Learning Based Load Balancing Routing for UAV-Assisted VANET
## 4 Proposed Q-LBR Design
### Q-Learning-Based Load Balancing
#### 4.3.3 Reward Control Function Design for Rapid Convergence  
Reinforcement learning is a problem faced by an agent who must learn behavior through trial-and-error in a dynamic environment. However, the learning method can cause a convergence speed problem in terms of the time required to find the optimal state. In particular, the network environment is changed by various variables over time, and thus a method allowing the reinforcement learning system to respond quickly is required. Previous studies in which Q-learning was applied were generally proposed to control the learning rate through the value of \(\alpha\). However, if \(\alpha\) is too large, it is difficult to converge to the optimal value function and, if it is too small, it takes too long to learn. This shows that there is a limitation in coping with rapid changes in the network with the existing method through the reflection ratio of the learned results.  
Q-LBR proposes using an RCF to determine the reward according to \(U\!RCI_{u}(t)\) and \(G\!NCI_{ratio}(t)\) for the purpose of improving the convergence speed of the reward function. The RCF of Q-LBR dynamically determines the reward value according to the load-state of the URN and the ground network congestion with the rapidly changing network environment. Specifically, if the queue load of the URN is sufficient, a large positive reward value is given to increase the utilization of the URN. By contrast, under high congestion, a large negative value is given to reduce the URN and ground network congestion.  
The reward function (\(f_{r}(s_{t},\;a_{t})\)), given by Equation (8), is as follows:  
\[f_{r}(s_{t},\;a_{t})=\begin{cases}f_{r+}(s_{t},\;a_{t})&if\;U\!RCI_{u}(t)\leq U \!RCI_{th}\\ f_{r-}(s_{t},\;a_{t})&else\end{cases} \tag{8}\]  
The positive reward function (PRF), given by Equation (9), for action \(a\) is expressed as follows:  
\[f_{r+}(s_{t},\;a_{t})=-1/ln(k*(1-\lambda(t)), \tag{9}\]  
where \(\lambda(t)\), given by Equation (10), is a function \((\lambda(t)\in(0,1])\) for determining the reward values according to \(U\!RCI_{u}(t)\) and \(G\!NCI_{ratio}(t)\) (where \(U\!RCI_{u}(t)\leq U\!RCI_{th},G\!NCI_{ratio}\leq G\!NCI_{th}\)). Here, \(k\) is the scale parameter (k > 0). When \(\lambda(t)\) is high, the reward value is significantly increased. When the value of \(\lambda(t)\) is low, it gradually increases.  
\[\lambda(t)=w_{1}*\left(\frac{U\!RCI_{u}(t)}{U\!RCI_{th}}\right)+w_{2}*\left( \frac{G\!NCI_{ratio}(t)}{G\!NCI_{th}}\right) \tag{10}\]  
Figure 6: UAV routing policy area (URPA) example.  
The negative reward function (NRF) for action a is expressed as follows:  
\[f_{r-}(s_{t},\ a_{t})=ln(k*(1/r_{max}-\lambda(t)), \tag{11}\]  
where \(r_{max}\) is the maximum reward value (\(r_{max}>\lambda(t),\ \ r_{max}>0\)). The NRF is also controlled by \(\lambda(t)\) and the weight \(w\) of \(\mathit{URLI}(t)\) and \(\mathit{GNCI}(t)\). In contrast to the PRF, when \(\lambda(t)\) is high, the reward value is significantly decreased, and when the value of \(\lambda(t)\) is low, the reward value gradually decreases.