# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### PARRoT  
[PERSON] et al. [77] have designed the predictive ad hoc routing fueled by reinforcement learning and trajectory knowledge (PARRoT) for flying ad hoc networks. This routing method can reduce the end-to-end delay in the data transmission process. PARRoT consists of three steps: predicting cross-layer motion, distributing routing messages, and RL-based route maintenance. In the mobility prediction process, a cross-layer technique is used to acquire knowledge about the UAV motion pattern and predict relative mobility between different agents so that each agent can estimate its next position based on the current position. This information is broadcast through routing messages called chirp to obtain local knowledge of the network topology. The broadcast period of these chirp messages is a fixed time interval. As a result, PARRoT is not compatible with the dynamic network topology and may cause a high routing overhead. Chirp message contains location information and a sequel number to prevent routing loops and eliminate repeated messages. After receiving each chirp message, UAV first checks its sequence number to guarantee its freshness. Then, this information is used for updating the Q-table. In the Q-learning-based routing process, UAVs use the local information obtained from chirp messages to achieve a partial view of the network topology. To send a data packet to the destination node, each UAV only evaluates the fitness of its single-hop neighbors for reaching the destination node, and selects a neighbor with the highest Q-value as the next-hop node. PARRoT finds multiple paths to the destination, which improves fault tolerance and accelerates the route recovery process when failing the nodes in a path. In the routing process, each node plays the agent role and the set of UAVs is regarded as the state set. Moreover, the action indicates the selection of a neighboring node as the next hop. The Q-value update process is also performed by exchanging chirp messages. In the RL-based routing process, the learning rate has a constant amount, but the discount factor is dynamically evaluated based on the link expiry time (LET) and the change degree of neighbors at a specific time interval. This learning process is represented in Figure 36. Furthermore, Table 18 lists the most important advantages and disadvantages of PARRoT in summary.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline \multirow{6}{*}{\
\begin{tabular}{c} PPMAC+RLSRP [76] \\ \end{tabular} } & \multirow{6}{*}{
\begin{tabular}{c} Designing a distributed routing method, reducing delay in the routing process, improving the packet delivery rate, taking into account the routing problem as a POMDP problem, predicting the accurate location of nodes in the network. \\ \end{tabular} } & Not providing a mechanism for controlling the hello broadcast interval, high routing overhead, low scalability, enlarging Q-table size in large-scale networks and decreasing in convergence speed, considering constant learning parameters, ignoring the energy of UAVs, and link quality in the routing process, considering a flat network topology and not taking into account the clustering process, not considering a mechanism to prevent congestion in the network, not solving the routing hole problem. \\ \hline \hline \end{tabular}
\end{table}
Table 17: The most important advantages and disadvantages of PPMAC+RLSRP.  
Figure 35: Learning process in PPMAC+RLSRP.