# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 7 Discussion
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Scheme**} & \multirow{2}{*}{**Energy**} & \multirow{2}{*}{**Delay**} & \
\begin{tabular}{c} **Network** \\ **Lifetime** \\ \end{tabular} & **PDR** & **Throughput** & **Connectivity** &
\begin{tabular}{c} **Routing** \\ **Overhead** \\ \end{tabular} \\ \hline DQN-VR [67] & ✓ & x & ✓ & x & x & ✓ & x \\ \hline QTAR [68] & ✓ & ✓ & ✓ & ✓ & x & x & ✓ \\ \hline TQNGPSR [69] & x & ✓ & x & ✓ & ✓ & x & ✓ \\ \hline QMR [70] & ✓ & ✓ & x & ✓ & x & x & x \\ \hline QGeo [71] & x & ✓ & x & ✓ & x & x & ✓ \\ \hline QSRP [72] & x & ✓ & x & ✓ & x & x & x \\ \hline QLGR [73] & ✓ & ✓ & x & ✓ & ✓ & x & ✓ \\ \hline FEQ-routing-SA [74] & ✓ & x & x & ✓ & x & x & ✓ \\ \hline Q-FANET [75] & x & ✓ & x & ✓ & x & x & x \\ \hline PPMAC+RLSRP [76] & x & ✓ & x & ✓ & x & x & x \\ \hline PARRoT [77] & x & ✓ & x & ✓ & x & x & x \\ \hline QFL-MOR [78] & ✓ & ✓ & x & x & x & x \\ \hline
3 DQ [79] & x & ✓ & x & ✓ & ✓ & x & x \\ \hline ICRA [80] & ✓ & ✓ & ✓ & ✓ & x & x & x \\ \hline TARRAQ [81] & ✓ & ✓ & x & ✓ & x & x & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 25: Comparison of RL-based routing methods in terms of routing parameters. the most common reinforcement learning algorithm used to design the routing process in FANETs because this algorithm is simple and has an acceptable computational cost. However, if the state and action spaces are very large, the learning policy will significantly be complex. In this case, the curse of a dimensionality problem occurs. This problem causes a slow convergence speed in the Q-learning algorithm. Thus, this algorithm cannot reach an optimal response at an acceptable time. Many researchers have attempted to solve this problem when designing RL-based routing protocols in FANETs. For example, DQN-VR uses a clustering technique to reduce the state and action spaces in the routing process. It also uses a deep learning technique in the routing process that has a high convergence speed. TQNGPSR also uses a deep reinforcement learning algorithm, which accelerates the convergence speed and increases scalability. QMR and QTAR have also attempted to manage the state space through a suitable solution. In QTAR, only neighboring nodes that are closer to the destination node compared to the current node are inserted into the state space, and other nodes will be filtered. On the other hand, QMR defines a packet velocity constraint. Thus, only neighboring nodes that can meet this speed constraint are inserted into the state space and other neighboring nodes will be filtered. When designing RL-based routing methods, another important point is to determine how to adjust the learning parameters--namely, the learning rate and the discount factor. If these learning parameters have constant values, the selected action may be inaccurate. Therefore, in QTAR, QMR, QGeo, FEQ-routing-SA, PARRoT, and TARAQ, researchers have adjusted these learning parameters dynamically and based on the network conditions. This has improved the adaptability of the routing methods to the dynamic FANET environment. Moreover, Table 27 compares RL-based routing methods in terms of learning algorithms. As shown in this table, DQN-VR and TQNGPSR have used a deep reinforcement learning technique in their routing protocols. This learning technique performs complex computational operations in the FANET and has good performance for complex and large-scale networks because its learning speed is very high. Research on deep reinforcement learning algorithms for designing the routing protocols is still in the early steps, and researchers must study further research to solve the challenges related to this learning technique--namely, high computational complexity and its implementation in low-energy nodes. Most research in FANET such as QTAR, QMR, QGeo, QSRP, QLGR, FEQ-routing-SA, Q-FANET, PPMAC+RLSRP, PARRoT, QFL-MOR, 3 DQ, ICRA, and TARAQ use traditional reinforcement learning algorithms to design their routing process.