# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### [PERSON] and [PERSON] in [72] have designed a Q-learning-based stepwise routing protocol (QSRP) for flying ad hoc networks. This method assumes that the network includes a central controller (CC) and a number of UAVs. First, CC executes a neighbor discovery process to calculate the link stability and the minimum number of hops to the destination node. Then, it uses a Q-learning algorithm to find paths with a minimum number of hops and high link quality. In the stepwise neighbor discovery process, the central controller broadcasts a discovery message on the network. This packet includes the ID of the central controller and hop count. Each UAV that receives this packet for the first time, increases the hop count to one unit and rebroadcasts this message on the network. In addition, it generates an ACK message including its ID, spatial information, speed, and the number of hops to CC, and sends this message to its parent node on the network. When the parent node receives the ACK message, it forms a neighborhood table to store this information. Then, it unicasts this message to its parent node until this message reaches CC. Note that the broadcast period of the discovery packet is adaptively adjusted based on speed and location of the nodes. After receiving ACK messages, CC creates a table that contains information about all network nodes. As a result, the central controller can obtain global knowledge of the \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline \multirow{6}{*}{QGeo [71]} & & Low scalability, high communication overhead, low convergence speed, enlarging the size of state space in large-scale networks, considering a flat network topology and ignoring clustering process, lack of attention to energy and the mobility pattern of nodes in the routing process, not balancing energy consumption in the network, reducing network lifetime, and not solving the routing hole problem. \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \end{tabular}
\end{table} 
Table 12: The most important advantages and disadvantages of QGeo. Figure 30: Learning process in QGeo. whole network. According to this information, it calculates a link stability scale based on node speed and the quality of the link between the nodes. In QSRP, the link quality is achieved based on the packet delivery rate and the transmission time. Then, the central controller executes a Q-learning algorithm to find the best route between the network nodes. In this process, each node is considered as the state, and the action indicates the selection of the route between the nodes. Furthermore, the reward function is defined based on two parameters, including the number of hops and link stability to guarantee the transmission quality and reduce the number of re-transmission in the network. In QSRP, learning parameters, including learning rate and the discount factor, have constant values. Figure 31 shows this learning process in QSRP. Moreover, Table 13 presents the most important advantages and disadvantages of QSRP. [PERSON] [PERSON] et al. [73] have offered a Q-learning geographic routing method (QLGR) for flying ad hoc networks. This method uses a multi-agent reinforcement learning technique in the routing process. QLGR reduces packet loss and routing overhead in the network. Initially, UAVs periodically share their information with each other by exchanging hello messages. This message includes spatial information, sequence number, message length, Q-value, maximum queue length, and occupied queue length. In QLGR, each UAV acts as a smart agent, and the network is regarded as the environment. Furthermore, the state of each node indicates the state space at any moment, and the action space is a set of neighbors of the current node. After taking an action, the agent receives two types of feedback (i.e., local reward (LR) and global reward (GR)) from the environment.