# Q-LBR: Q-Learning Based Load Balancing Routing for UAV-Assisted VANET
## 1 Introduction  
The vehicular ad hoc network (VANET), a special type of mobile ad hoc network (MANET), has been investigated to provide the infrastructure of a new service paradigm through self-organizing networks that exist between vehicles. However, it still experiences difficulty in routing with easily disconnected features that are associated with dynamic wireless environments in mobile network topologies. To overcome this problem, the deployment of unmanned aerial vehicles (UAVs) via the cooperation of vehicles has been considered. Several methods have recently been developed in the literature for UAV-assisted network protocols that address the issues of high mobility in the network and unpredictable change in topology of the mobile nodes [1; 2; 3; 4; 5; 6]. Unlike a fixed ground relay station, a UAV relay node (URN) moves along with the ground vehicular nodes (GVNs) to support a reliable network through a continuous line-of-sight (LoS) link. In addition, considering the characteristics through which MANET is temporarily constructed and operated, this is an extremely economical solution compared to the construction of a ground infrastructure. In the case of a VANET, in particular, the relay node is faced with the risk of a broken link that can be caused by mobility, and nonline-of-sight (NLoS) can occur more frequently than in a general MANET. Therefore, a UAV-assisted relay can be a more useful tool when operating in a VANET environment. Because the UAV relay path is most likely to be the best approach in terms of link quality and the number of hops, it is highly likely that a bottleneck of the URN will occur from the existing routing protocol when the network is congested. This bottleneck can degrade the transmission efficiency of the UAV and, in the case of Carrier Sense Multiple Access/Collision Avoidance (CSMA/CAMAC, Ex. 802.11p), the channel access opportunities of all the ground nodes can be lost. In addition, the spatial frequency reusability of the ground node can be lowered, resulting in a decrease in the overall network performance. However, if a UAV relay node (URN) is operated only as a backup path when a ground link disconnection occurs, the URN resources are wasted. Therefore, to increase the efficiency of a URN, it is necessary to design a structure that can handle the maximum acceptable traffic while maintaining a certain level of ground network load. In the design of the Q-learning based load balancing routing (Q-LBR), a URN uses an overhearing technique based on the broadcast nature of wireless media to recognize the ground network load in the message transmitted between the GVNs. The URN then distributes the UAV routing policy area (URPA) information based on the Q-learning method through broadcast messages such as Hello. The GVNs decide whether to use the air relay path through the received URPA and the current UAV relay load, and the URN continuously maximizes the network throughput even when the wireless environment is dynamically changing. We propose a Q-LBR system for load balancing in a UAV-assisted VANET. This provides a method for URN to handle the maximum traffic acceptable while maintaining a certain level of ground network load. The main contributions of this study are fourfold:  
1. We propose a low-overhead technique for estimating the network load through the queue status obtained by the URN from each GVN. To estimate a network load with low overhead, we design a technique using overhearing and broadcast messages received from the GVNs. This is possible because the URN can cover an area wider than that of the GVNs. 2. We propose a load balancing scheme based on Q-learning, which can enable dynamic network load control within the usable capacity of the URN. Q-LBR defines the URPA when considering the traffic characteristics and the existence of a route independently from the ground network routing. 3. We propose a reward control function (RCF) to enable rapid learning feedback of the reward values in consideration of a dynamic network environment.