# Q-LBR: Q-Learning Based Load Balancing Routing for UAV-Assisted VANET
## 5 Simulation Results and Analysis
### Performance Analysis  
The key element of Q-LBR is URPA, which induces a load balancing between the URN and the ground network. GNs determine the routing according to the URN load and ground network load based on the URPA. Therefore, if the URN grants the maximum allowable traffic through the proper URPA, a positive effect on the overall network performance can be expected because the URN path can provide a higher quality clear-LoS link than the ground path. Figure 8 shows the results of a comparative experiment when setting the URPA as a fixed value without a learning process and assigning a dynamic value through Q-learning from the perspective of the URN utilization (\(Q_{ground_{th}}\)= 70, \(URCI_{th}\)= 80, \(GNCI_{th}\)= 50, \(w_{1}\) = 0.7, and \(w_{2}\)= 0.3). URN utilization is a performance index that indicates the average queue length compared to the maximum queue length of the UAV per unit time, and is the same as \(URCI_{u}\), which indicates the queue load of the URN. This metric shows the degree of URN utilization in the network. A lower URN utilization means that \(URCI_{u}(t)\) is low because the load on the URN is idle. By contrast, in the case of the same traffic condition, a higher URN utilization means the UAV load is close to the maximum allowable queue length, and thus the URN is busy. However, if \(MQL_{u}\) is exceeded, it means that a queue drop occurs, and thus it is necessary to set the appropriate \(URCI_{th}\). In the case of Q-LBR (w/o QL), a fixed URPA policy is applied, and thus there is no coordination according to the ground network load and URN load conditions. Therefore, the overall URN utilization is relatively low (1%-40%). In the case of Q-LBR with Q-learning, the result shows that the URN utilization by dynamic URPA is improved by Q-learning. Therefore, the overall URN utilization is relatively high (40%-80%). If \(URCI_{th}\) and \(GNCI_{th}\) are increased, a higher URN utilization can be expected in Q-LBR with Q-learning. However, as the URN utilization increases, the possibility of a packet loss owing to an overload increases proportionally, and thus it is necessary to set an appropriate  
Figure 8: Q-LBR versus Q-LBR without Q-Learning for URN utilization. Figure 7: Basic network layout (10 GVNs and 1 URN) in the Riverbed Modeler. level (70%-80%). As a result, this experimental result shows that Q-LBR with Q-learning has a significant effect on the dynamic URPA  
Figure 9 shows the results of the comparative experiment according to the RCF of Q-LBR in the same environment as the above experiment. The purpose of the experiment was to find out how RCF affects the convergence speed through cumulated reward value (CRV). As a result, it was confirmed that there was a difference in the number of episodes required to reach the maximum reward value (\(r_{max}=5\)) depending on whether or not RCF or the reward value. Q-LBR (w/o RCF, PRF=+1, NRF=-1) approached \(r_{max}\) most quickly in the first 10 to 70 episodes, but the results were not converged even after 200 episodes. Q-LBR (w/o RCF, PRF = +0.3, NRF = -0.3) showed convergence after about 160 episodes. This result shows that the probability r_max of is high when the fluctuation of the reward value is small, but the probability of increasing the number of required episodes is high. On the other hand, since Q-LBR (with RCF) adjusted the reward value adaptively in consideration of the ground network load and URN load, it showed a rapid increase in the beginning and converged in a gentle curve. Finally, it converged to 110 episodes, which decreased by about 32% compared to Q-LBR (w/o RCF, PRF = +0.3, NRF = -0.3). From Figure 10 and Table 1, we can see that as the node speed increases the packet loss rate of the Q-LBR is lower than that of U2 RV. Q-LBR also performs better in terms of network utilization and latency. As the speed of the GVN increases, the probability of the topology changing increases and retransmission by routing control messages and route disconnection increases.