# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### [PERSON] et al. [81] have proposed the topology-aware resilient routing strategy based on adaptive Q-learning (TARRAQ) for flying ad hoc networks. In this method, UAVs periodically exchange hello messages to share their status information with other nodes and form a neighborhood table. The purpose of TARRAQ is to find routes, which reduce delay and energy consumption, and increase the packet delivery rate. TARRAQ tries to accurately obtain topology changes and execute the routing process using a distributed, autonomous, and adaptive manner. To achieve this goal, the queuing theory is used to analyze the dynamic changes in topology. This analysis is used for calculating two parameters, including the neighbor change rate (NCR) and neighbors' change inter-arrival time (NCTT) distribution to describe the dynamic behavior of UAVs. Then, the sensing interval (SI) or the hello broadcast time is achieved based on NCR and NCTT, so that UAVs can adjust SI based on dynamic behavior and performance needs. This technique reduces routing overhead in the neighbor discovery process. In addition, the residual link duration (LD) is estimated based on the Kalman filter (KF) method. It indicates the link expiration time for the neighboring nodes. In TARRAQ, the routing process is modeled by the Markov decision algorithm, and the Q-learning algorithm is used to solve this routing problem. In this learning process, each packet plays the agent role, and the entire network is regarded as the environment. Furthermore, the state space includes all UAVs, and the action space indicates the selection of a neighboring node as the next-hop node. In this process, the reward function is defined based on the quality of the link, energy, and distance between neighbors. This learning process is represented in Figure 40. In this routing method, learning parameters, including learning rate and discount factor,  
\begin{table}
\begin{tabular}{p{14.2 pt} p{142.3 pt} p{142.3 pt}} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline \hline \multirow{7}{*}{ICRA} & Designing a distributed routing method, utilizing clustering technique in the network, balancing energy consumption, improving the network lifetime, increasing the stability of the network topology, reducing delay in the data transmission process, improving packet delivery rate, lowering routing overhead in the data transmission process, reducing network congestion in the routing process, managing Q-table size in the network, improving convergence speed, high scalability, considering the consumed energy of UAVs in the clustering process. & Designing a centralized clustering strategy, considering constant learning parameters, not providing a solution to prevent routing holes, considering a constant hello updating time, not determining the optimal number of clusters in the network, considering inadequate parameters to select inter-cluster forwarding nodes. \\ \hline \hline \end{tabular}
\end{table}
Table 21: The most important advantages and disadvantages of ICRA.  
Figure 39: Learning process in ICRA.  
are calculated based on the residual LD. Table 22 briefly expresses the most important advantages and disadvantages of TARRAQ.