# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### [PERSON] et al. (2020) have presented the Q-learning multi-objective optimization routing protocol (QMR) for FANETs. In the routing process, QMR attempts to reduce delay and energy consumption in the data transmission process between the source and destination. In order to balance energy consumption, QMR considers the energy factor in the routing process. In this method, UAVs periodically exchange their information, including geographic location, residual energy, mobility model, queuing delay, and discount factor  
\begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt}} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline TQNGPSR [69] & Designing a distributed routing process, utilizing deep reinforcement learning algorithm, high scalability, appropriate convergence speed, designing a traffic balancing strategy, preventing network congestion, reducing delay and packet loss in the routing process, designing a mechanism for avoiding to trap in a local optimum. & High routing overhead, not managing the size of the state space, not designing a mechanism to adjust the hello broadcast interval, considering a flat network topology, and ignoring the clustering process, not considering energy consumption in the routing process, considering fixed learning parameters. \\ \hline \hline \end{tabular}
\end{table}
Table 10: The most important advantages and disadvantages of TQNGPSR.  
Figure 28: Learning process in TQNGPSR.  
through hello messages. QMR also adjusts an adaptive hello broadcast interval based on node speed. In the learning process, the entire network is considered as the environment and each packet plays an agent role. The state set contains UAVs, and the action represents a decision to send the packet from the current node to a neighboring node. Figure 29 represents the learning process in this protocol. In QMR, the learning parameters are dynamically determined based on the network conditions. The learning rate is calculated based on one-hop delay, and the discount factor is characterized based on the movement of neighboring nodes in two consecutive time intervals. In addition, QMR has presented a new exploration and exploitation mechanism to balance exploration and exploitation based on the speed specified for the data packet. According to this mechanism, when selecting the next-hop node, the allowed speed for sending the packet from the current node to the next-hop node is calculated, and nodes that meet the allowed speed can be selected as the next-hop node. This idea filters the state space, reduces its size, and improves the convergence speed of the Q-learning algorithm. As a result, this reduces the end-to-end delay when sending the data packet to the destination. After calculating Q-values, a weight coefficient is computed based on the two parameters, including the link quality and the intimacy of the neighboring node and the current node. This coefficient is used to select the next-hop node among nodes, which have the highest weighted Q-value and meet the allowed packet speed. If the candidate node set is empty and there is no neighboring node, which meets the allowed packet speed, then the current node selects the next-hop node from the neighboring nodes, which have a speed greater than zero. In this case, it selects the neighboring node with the largest speed. Otherwise, if there is no node, which has a speed greater than zero, QMR sets up a penalty mechanism. The purpose of this mechanism is to prevent routing holes. According to this mechanism, if the next-hop node is encountered by a routing hole or does not send an ACK message to the previous-hop node, the previous-hop node reduces its reward to lower the selection chance of this node as the next-hop node in the future. Table 11 expresses the most important advantages and disadvantages of this routing scheme in summary.