# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 7 Discussion
They are easier than DRL-based routing methods and have less computational complexity. Therefore, they are suitable for discovering the best routing path in small FANETs because the size of the state and action spaces are small, and the learning algorithm is converged to the optimal response with acceptable learning speed. However, if the flying ad hoc network is large, these routing methods deal with the curse of the dimensionality problem and cannot present a good performance for finding the best route in the network because their convergence speed decreases sharply. Among these methods, QLGR is the only multi-agent RL-based routing method in the flying ad hoc networks. In this approach, each UAV plays the agent role and tries to learn the best path between the source and destination through interactions with the network. An important challenge in this routing scheme is how to coordinate and cooperate between UAVs to find an optimal response because they are extremely dynamic. QLGR has a faster convergence speed than single-agent routing methods because the routing calculations are done in parallel in different UAVs. This has improved scalability in QLGR. However, its computational complexity is greater than single-agent routing methods. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Scheme** & **RL Algorithm** & **Agent** & **State Set** & **Action Set** & **Reward Function** & **Learning Rate** & **Discount Factor** \\ \hline \multirow{3}{*}{DQN-VR [67]} & \multirow{3}{*}{DQN} & \multirow{3}{*}{Central controller} & A 2D array, including mobility and residual energy & Selecting the best neighboring node & \multirow{3}{*}{Ficed congestion level} & \multirow{3}{*}{Ficed congestion level} \\  & & & & & & & \\  & & & & & & & \\ \hline \multirow{3}{*}{QTAR [68]} & \multirow{3}{*}{Q-learning} & \multirow{3}{*}{Each packet} & \multirow{3}{*}{Neighboring nodes towards destination} & Selecting the best next-hop node & Related to delay, energy, and velocity & Based on two-hop delay & \multirow{3}{*}{Based on two-hop changes between a UAV and its neighbors} \\  & & & & & & \\ \hline \multirow{3}{*}{TQNGPS [69]} & \multirow{3}{*}{DQN} & \multirow{3}{*}{Each packet} & \multirow{3}{*}{UAVs} & Selecting a candidate neighbor as the next-hop node & \multirow{3}{*}{Based on the queuing length} & \multirow{3}{*}{Ficed congestion level} \\  & & & & & & \\ \hline \multirow{3}{*}{QMR [70]} & \multirow{3}{*}{Q-learning} & \multirow{3}{*}{Each packet} & \multirow{3}{*}{UAVs} & Decision of the packet (agent) to be forwarded from the current node to a neighboring node & \multirow{3}{*}{Based on delay and energy} & \multirow{3}{*}{Based on single-hop movement of delay} \\  & & & & & & \\ \hline \multirow{3}{*}{QGeo [71]} & \multirow{3}{*}{Q-learning} & \multirow{3}{*}{Each packet} & \multirow{3}{*}{UAVs} & Transition from transmitter transmitter transmitter to neighbor node & Related to packet travel speed & \multirow{3}{*}{Fixed} & Related to distance and the problem \\  & & & & & & & \\ \hline \multirow{3}{*}{QSRP [72]} & \multirow{3}{*}{Q-learning} & \multirow{3}{*}{Central controller} & \multirow{3}{*}{UAVs} & Selecting a routing path & Related to link stability and hop count & \multirow{3}{*}{Fixed} & \multirow{3}{*}{Fixed} \\  & & & & & & \\ \hline \multirow{3}{*}{QLGR [73]} & \multirow{3}{*}{Q-learning} & \multirow{3}{*}{UAVs} & \multirow{3}{*}{State of UAVs} & Sending a packet to a neighboring node & Related to load capacity and link quality & \multirow{3}{*}{Fixed} & \multirow{3}{*}{Fixed} \\  & & & & & & \\ \hline \multirow{3}{*}{FBQ-routing-SA [74]} & \multirow{3}{*}{Q-learning} & \multirow{3}{*}{Each UAV} & \multirow{3}{*}{State of UAVs} & Sending a packet to a neighboring node & Related to transmission energy & Based on delivery time & \multirow{3}{*}{Fixed} \\  & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 26. Learning parameters in RL-based routing methods.