# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 7 Discussion
Moreover, in a tree-based network, each parent node is responsible for executing learning operations in its sub-tree to reduce the dimensions of the state space in the learning process. In addition, in a multi-level network, one or more nodes are selected for finding the best route at each network level and managing the relationships between different network levels to improve the routing process. However, the management of different roles in the network and the selection of parent nodes and CHs, especially in highly dynamic networks, are important challenges that should be considered in these methods. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Single-Path** & **Multi-Path** \\ \hline DQN-VR [67] & ✓ & \(\times\) \\ \hline QTAR [68] & ✓ & \(\times\) \\ \hline TQNGPSR [69] & ✓ & \(\times\) \\ \hline QMR [70] & ✓ & \(\times\) \\ \hline QGeo [71] & ✓ & \(\times\) \\ \hline QSRP [72] & ✓ & \(\times\) \\ \hline QLGR [73] & ✓ & \(\times\) \\ \hline FEQ-routing-SA [74] & ✓ & \(\times\) \\ \hline Q-FANET [75] & ✓ & \(\times\) \\ \hline PPMAC+RLSRP [76] & ✓ & \(\times\) \\ \hline PARRoT [77] & \(\times\) & ✓ \\ \hline QFL-MOR [78] & ✓ & \(\times\) \\ \hline
3 DQ [79] & ✓ & \(\times\) \\ \hline ICRA [80] & ✓ & \(\times\) \\ \hline TARRAQ [81] & ✓ & \(\times\) \\ \hline \hline \end{tabular}
\end{table} 
Table 28: Comparison of RL-based routing methods by data path. Table 30 compares different routing methods in terms of data delivery techniques. Most RL-based routing methods such as TQNGPSR, QLGR, PPMAC+RLSRP, PARRoT, 3 DQ, and ICRA use the greedy technique. However, this data delivery method deals with a major challenge (i.e., routing holes). In this case, the routing algorithm is trapped in the local optimum and cannot find any nodes to reach the destination. Among these routing methods, 3 DQ has presented an interesting idea. It combines the greedy technique and the store-carry-forward technique to solve the routing hole challenge in the greedy mode. However, this solution can increase delay in the data transmission process. Usually, the store-carry-forward technique is not alone used to create a route between the source and destination because it causes a high delay in the data transmission process. However, when this scheme is integrated with the greedy method, it can solve the disadvantages of both methods. Furthermore, DQN-VR, QTAR, QMR, QGeo, QSRP, FEQ-routing-SA, Q-FANET, QFL-MOR, and TARRAQ discover the best route between the source and destination based on their desired criteria discussed in Section 6. However, the route discovery technique increases communication overhead, bandwidth consumption, and delay in the routing methods. In Table 31, the routing methods are compared in terms of the routing process. According to this table, we can find that most RL-based routing methods are distributed. QSRP is the only centralized RL-based routing scheme. In this scheme, the central controller performs the routing process. QSRP assumes that this central server has global knowledge of the entire network and uses this knowledge in the routing process. However, in highly dynamic networks such as FANETs, it is very difficult or even impossible to gain global knowledge of the network by the central agent. For this reason, these methods are not successful in FANETs. Furthermore, they are not scalable. The most important advantage of this method is that the central controller obtains the best route at the lowest computational cost, and manages the routing process. As a result, UAVs do not consume energy for calculating routing paths. However, this technique is not fault-tolerant. On the other hand, obtaining global knowledge requires a lot of routing overhead. DQN-VR and ICRA use both centralized and distributed routing techniques. DQN-VR defines two types of data: global data and local data. Global data such as residual energy are less dynamic. Therefore, their update process is performed at longer time intervals.