# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 7 Discussion
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Reinforcement Learning**} & \multicolumn{4}{c}{**Deep Reinforcement Learning**} \\ \cline{2-9}
**Scheme** & **Single-Agent** & **Multi-Agent** & **Model-Based** & **Free-Model** & **Single-Agent** & **Multi-Agent** & **Model-Based** & **Free-Model** \\ \hline DQN-VR [67] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline QTAR [68] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline TQNGTSR [69] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline QMR [70] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline QGeor [71] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline QGRP [72] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline QLGR [73] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline FFOOnting: & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ SA [74] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline Q-FANET [75] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline PPMAC-RLSRP [76] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline PARRGT [77] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline QPL-MOR [78] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline
3 DQ [79] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline ICRA [80] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline TARAQ [81] & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) & \(\star\) \\ \hline \end{tabular}
\end{table} 
Table 27: Comparison of RL-based routing methods based on learning algorithm. Table 28 compares different RL-based routing protocols in terms of the routing path. According to this table, almost all RL-based routing methods are single-path. When finding the best route through the RL algorithm, almost all the routing paths between the source and destination are discovered. Then, they are prioritized based on the Q-value. However, in the route selection process, only one route (the path with maximum Q-value) is selected for the data transmission process. The management of the routing table (or Q-table) in a single-path routing method is easier than a multi-path routing scheme. However, it is not fault-tolerant, meaning that if the routing path is disconnected, the data transmission process will be delayed because a new path should be discovered. However, if the routing method selects alternative paths in the route selection process, it reduces packet loss and improves network performance. In Table 29, different routing methods are categorized based on network topology. According to this table, we can deduce that researchers perform a small number of studies in the field of hierarchical RL-based routing methods, and there are only two hierarchical RL-based routing methods, DQN-VR and ICRA, for FANETs. However, this topology is very suitable for large-scale networks because determining the different roles for nodes efficiently reduces the consumption of network resources in the route calculation process and lowers routing overhead. Researchers should consider this issue in the future to improve network performance. For example, in a clustered network, the RL algorithm is implemented in CHs to find the best path between different CHs, and each cluster is managed by the cluster head node. This reduces the state space and improves the convergence speed.