# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### [PERSON] and [PERSON] [68] have suggested a Q-learning-based topology-aware routing method (QTAR) for flying ad hoc networks. QTAR tries to discover the best route between the source and destination using two-hop neighbors' information such as neighboring position, delay, speed, and energy. This method balances the load on the network because the energy level of nodes is used in the learning process. In addition, QTAR prevents routing loops because it uses the two-hop neighbors' information to prevent blind transmission. Moreover, reducing the number of hops in the created path is one of the key goals of two-hop neighboring information. Initially, each UAV obtains its location on the network using GPS and shares this information through the hello message with its two-hop and single  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline \hline \multirow{7}{*}{DQN-VR [67]} & & Not designing the adaptive broadcast mechanism for controlling beacon messages in the network, the ambiguity of how to update local and global information in CC and DCs, the ambiguity of how to calculate the reward function including congestion level and data transmission rate, ignoring the CH rotation process, not determining how to manage different network levels and how to change the level of UAVs in the network, not determining the optimal number of clusters and CHs at each level, fixing the DQN parameters. \\ \cline{1-1}  & & \\ \hline \hline \end{tabular}
\end{table}
Table 8: The most important advantages and disadvantages of DQN-VR.  
Figure 26: Learning process in DQN-VR.  
hop neighboring nodes. This message includes node position, link information, energy, speed, mobility model, and queuing delay. According to this message, a neighborhood table is formed in each node. It includes spatial information, energy level, mobility model, speed, and queuing delay. In QTAR, each UAV dynamically adjusts the hello broadcast period and the link holding time based on the shortest lifetime of links between itself and its neighboring nodes to adapt itself to the dynamic network topology. The link lifetime is calculated based on the distance between a UAV and its neighbors and their relative velocities. In QTAR, the Q-learning algorithm is responsible for learning the routing process in a distributed manner. In this process, each packet plays the agent role and neighboring nodes are considered as a state space. To manage the size of the state space, QTAR only puts nodes that are closer to the destination compared to the current node in the state set. This improves the convergence speed of this routing protocol. In this learning issue, the selection of the next-hop node is considered an action. Finally, the reward function is calculated based on three parameters: energy, delay, and speed. The learning process in QTAR is shown in Figure 27. Note that QTAR adjusts learning parameters, including learning rate and the reward factor dynamically, because if these parameters have constant values in dynamic networks, the selected action is not accurate. In this method, the learning rate is adjusted based on the delay information obtained from two-hop neighbors and the reward factor is calculated based on the speed and distance changes between each UAV and its neighbors. Additionally, QTAR considers a penalty mechanism to prevent routing holes. According to this mechanism, if the next-hop node is trapped in the routing holes or the previous-hop node does not receive the acknowledgment message from the next-hop node, it reduces the reward value, corresponding to the next-hop node to prevent the selection of this node in the routing process. Table 9 summarizes the most important advantages and disadvantages of this routing scheme.