# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 7 Discussion  
In this section, RL-based routing methods in FANETs are reviewed and compared in various aspects. The most important features of these routing methods are summarized in Table 23. According to this table, we can deduce that most RL-based routing schemes rely on local knowledge about the network environment. QSRP is the only RL-based routing method which relies on global knowledge obtained by the central controller. This global information includes the number of hops, location, and speed of UAVs in the network. However, the feasibility of this routing scheme is ambiguous, and it is very difficult or even impossible to implement this scheme in FANET because the topology of these networks changes rapidly and the global information obtained from the network will be invalid in a very short period. However, DQN-VR and QFL-MOR use both local and global knowledge in the routing process. In DQN-VR, researchers have presented an attractive idea. They believe that less dynamic information, such as residual energy that requires longer updating intervals, can be collected as global information. In contrast, high dynamic data such as location and speed, which change quickly, can be collected as local information. Then, global and local information are used in the routing process. This idea is more practical than QSRP and can be considered by researchers in the future. In addition, QTRV uses the information of single-hop and two-hop neighbors in its routing process to increase the local view of the agent relative to the network environment. However, this increases routing overhead in the network. Additionally, RL-based routing schemes usually use hello messages for obtaining local information about their neighbors in the network. However, the periodic broadcast of these messages between UAVs imposes a lot of communication overhead in the network and consumes network resources, including energy and bandwidth, and increases congestion in the network. In QTRAR, QMR, QGeo, QLGR, QSRP, and TARRAQ, researchers have adjusted the hello broadcast interval with regard to the speed of the network topology changes to manage communication overhead as much as possible. It is essential to pay attention to the energy problem in FANETs, which includes small drones with limited energy sources. DQN-VR, QTRAR, QMR, FEQ-routing-SA, QFL-MOR, ICRA, and TARRAQ have attempted to improve energy consumption on the network. Clustering is one useful solution that balances the energy consumption of nodes and improves network lifetime. ICRA and DQN-VR utilize the clustering technique to balance energy consumption in the network because this technique reduces communi  
Figure 40: Learning process in TARRAQ. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline TARRAQ [81] & \
\begin{tabular}{c} Designing a distributed routing method, determining \\ adaptive sensing interval, adjusting learning \\ parameters dynamically, reducing packet loss, paying \\ attention to energy in the routing process, reducing \\ delay in the data transmission process, predicting \\ dynamic behavior of nodes in the network, estimating \\ link lifetime using the Kalman filter. \\ \end{tabular} &
\begin{tabular}{c} High routing overhead, low scalability, enlarging \\ Q-table size in large-scale networks and decreasing \\ convergence speed, considering a flat network \\ topology and ignoring the clustering process, using \\ only RWP mobility model for simulating the motion of \\ UAVs and not adapting with other mobility models. \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 22: The most important advantages and disadvantages of TARRAQ. cation overhead and increases network scalability. Scalability is a major challenge when designing RL-based routing methods for large-scale networks because the size of the Q-table depends on the network size. This issue can greatly reduce the speed of the learning algorithm so that achieving an optimal response will be practically impossible.