# Q-LBR: Q-Learning Based Load Balancing Routing for UAV-Assisted VANET
## 2 Related Studies
### Load-Balancing Routing Protocols  
With the rapid increase in the use of VANETs, and increased demand of networked vehicles for a wide range of services and better information, load balancing has become an essential and important research area. Efficient load balancing ensures efficient resource utilization and enhances the overall performance of the network system. UAV-aided cross-layer routing (UCLR) [7] is a cross-layer routing and load-balancing algorithm that considers the UAV relay based on Open Shortest Path First-MANET Designed Router (OSPF-MDR). The routing metric of UCLR is calculated using the packet error rate (PER), and load balancing is adjusted using a static threshold of the queue length. Although UCLR handles dynamic UAV traffic load issues between the URN and GVNs, the main drawback of the UCLR is its static load control with a dynamic network environment. In a UAV-assisted VANET environment there are several moving GVNs, and the changes in the traffic patterns are also extremely rapid. Therefore, there is a need for a dynamic load control scheme capable of responding to rapidly changing network environments. Moreover, UCLR does not consider a method for improving the utilization of the UAV throughput. A hierarchical routing scheme with load balancing (HRLB) has been proposed [8] as a hierarchical geography routing protocol for software-defined VANETs. HRLB constructs a path cost function with load balancing and maintains two paths with minimal costs from the selected grids. This protocol considers the load only from the GVN and disregards the UAV-assisted relay. A queue utilization routing algorithm (QURA) has been proposed [9] as a machine learning-based routing scheme for QoS routing. This protocol applies an artificial neural network (ANN) to routing and selects the next hop according to a queue utilization prediction (QUP). However, supervised learning has a problem in that it is difficult to create the training data and the label in dynamic network topology. Table 1 summarizes the characteristics of the UAV-assisted routing and load balancing protocols. By reviewing these protocols, we can state that most of the proposed routing protocol techniques designed for UAV-assisted VANET disregarded the traffic characteristics and dynamic load balancing in congested network environments. In addition, these routing protocols do not consider traffic bottlenecks owing to a better link quality and hop count compared to the ground network of a UAV relay node.  
To address the aforementioned problems, we propose a new load-balancing routing scheme that is capable of achieving efficient operation of UAV relay nodes in consideration of the traffic characteristics.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Features** & **LCAD 1** & **U2 RV 2** & **UCLR 3** & **HRLB 4** & **Q-LBR 5** \\ \hline Multipath & No & Yes & Yes & Yes & Yes \\ UAV-assisted Relay & Yes & Yes & Yes & No & Yes \\ Traffic Characteristics & No & No & No & No & Yes \\ Load Balancing & No & No & Yes & Yes & Yes \\ Dynamic Load Control & No & No & No & No & Yes \\ Machine Learning & No & No & No & No & Yes \\ Type of network & UAV/MANET & UAV/VANET & UAV/MANET & VANET & UAV/VANET \\ \hline \hline \end{tabular}  
* LCAD: Load Carry and Deliver Routing. 2 U2 RV: UAV-assisted Reactive Routing Protocol for VANET. 3 UCLR: UAV-aided Cross-Layer Routing. 4 HRLB: Hierarchical Routing Scheme with Load Balancing. 5 Q-LBR: Q-learning based Load Balancing Routing.  
\end{table}
Table 1: Comparative study between routing protocols.  
In addition, we use the Q-learning algorithm, which improves the convergence speed of the reward function for dynamic load control.