# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### [PERSON] et al. [69] have proposed a traffic-aware Q-network geographic routing scheme based on greedy perimeter stateless routing (GPSR) called TQNGPSR for flying ad hoc networks. This routing protocol introduces a traffic balancing strategy, which utilizes congestion information of neighboring nodes to evaluate the wireless link quality. The best route between the source and destination is selected based on this evaluation to reduce delay and  
\begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt} p{142.3 pt}} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline \multirow{6}{*}{QTAR [68]} & Designing a distributed routing process, controlling the size of state space, improving convergence rate, high scalability, adaptive adjustment of the hello broadcast period, designing a penalty mechanism to avoid falling into the routing holes, adjusting the learning parameters, including learning rate and discount factor dynamically, balancing energy consumption in the network, improving network lifetime, preventing the blind path problem by adjusting the link holding time based on the link lifetime. \\ \cline{1-1}  & & High communication overhead, slow convergence speed in large-scale networks despite trying to limit the state space, taking into account a flat network topology and ignoring the clustering process. \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \hline \hline \end{tabular}
\end{table}
Table 9: The most important advantages and disadvantages of QTAR.  
Figure 27: Learning process in QTAR.  
packet loss in the data transmission process. The Q-network algorithm is responsible for comparing the quality of the candidate links and selecting a link with the largest Q-value. Each node calculates the congestion level based on the buffer queue length received from its neighbors. It is proportional to the queuing delay. When the current node sends a data packet to the next-hop node. It returns an ACK message to the previous-hop node. The ACK message includes the buffer length information. In addition, each node periodically exchanges a hello message including the queue length information and spatial coordinates of the neighboring node, and updates its neighborhood table immediately after receiving any hello message from its neighbors. In the routing process, Q-values are first calculated regardless of congestion information. Next, the congestion penalty process is designed to update Q-values based on the buffer queue information. For this purpose, the reward function is recalculated based on the buffer queue information. If the buffer queue of a UAV is almost full, the node receives a high penalty in the congestion penalty process, and its corresponding Q-value will be reduced to lower the selection chance of this node as the next-hop node in the future. In TQNGPSR, each packet maintains a visit list (VL), which includes nodes visited by the packet. Therefore, when a UAV receives a packet, the node uses VL to know what UAVs were met. As a result, it sends the packet to a node, which is far from the visited area to prevent a local optimum. For this purpose, each UAV calculates the angle between its own and the neighboring nodes in VL and obtains the minimum angle. Given this angle, the Q-network can estimate Q-values corresponding to other neighbors. Therefore, the current node selects the next-hop node among nodes whose angles are larger than the available nodes. In the learning process, each data packet plays the agent role, and the state represents the node that holds the data packet. Additionally, the action indicates the selection of a neighboring node as the next-hop node. Figure 28 shows the learning process in TQNGPSR. In this method, the learning parameters, including the learning rate and the discount factor, are empirically selected. Table 10 presents the most important advantages and disadvantages of this routing scheme in summary.