# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### [PERSON] et al. [78] have introduced a Q-learning-based fuzzy logic for multi-objective routing protocol (QFL-MOR) in FANETs. QFL-MOR uses various link-level factors and path-level parameters to evaluate the routing performance to find an optimal path between the source and destination. Link-level factors include transmission rate (TR), energy status (ES), and flight status (FS). Moreover, the route-level parameters include hop count (HC) and successful packet delivery time (SPDT). In this routing method, each UAV periodically shares its local information, such as speed, position, motion direction, and remaining energy with its neighboring UAVs to calculate link-level factors. Note that ES is calculated based on two parameters, including residual energy and the energy discharge rate, and FS is obtained from the speed and motion direction. In QFL-MOR, the routing process includes three steps. In the first step, each node (beginning from the source node) uses a fuzzy system to evaluate the quality of links between itself and neighboring nodes in terms of TR, ES, and FS to select an appropriate node with the best link quality as the next-hop node. This process continues until the data packet reaches the destination. Thus, a route is discovered between the source and destination at this step. In the second step, Q-learning evaluates the cost of the route created between the source and destination, and updates Q-values corresponding to hop count and successful packet delivery time in the selected path. In the last step, each node (beginning from the source node) uses a fuzzy system to improve the calculated path. Q-values related to the routing path--namely, HC and SPDT--and the link parameters such as TR, ES, and FS are considered as fuzzy inputs for this fuzzy system to improve the constructed route between the source and destination. The second and third steps are repeated to get the best route between the source and destination. In the learning process, the network is considered as an environment. Additionally, UAVs are corresponding to the state space, and the action indicates the selection of a neighboring node as the next-hop node. This learning process is shown in Figure 37. Note that in this process, the Q-learning parameters, including the learning rate and the discount factor, have constant values. Table 19 describes the most important advantages and disadvantages of QFL-MOR in summary.  
Figure 36: Learning process in PARRoT.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline \multirow{6}{*}{PARRoT [77]} & Designing a distributed routing method, reducing delay in the data transmission process, improving the packet delivery rate, adjusting the learning parameters dynamically, predicting the location of nodes in the network, preventing routing loops, designing a multi-path routing, improving fault-tolerance. & Not controlling the chirp broadcast interval, high routing overhead, low scalability, enlarging Q-table size in large-scale networks and reducing convergence speed, not considering energy of UAVs and link condition in the routing process, using a flat network topology and ignoring the clustering process, not considering the congestion control mechanism, not solving the routing hole problem. \\ \hline \hline \end{tabular}
\end{table}
Table 18: The most important advantages and disadvantages of PARRoT.