# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### FEQ-Routing-SA  
[PERSON] et al. [74] have proposed an improved version of the Q-routing algorithm called fully-echoed Q-routing with simulated annealing inference (FEQ-routing-SA) for FANETs. In FEQ-routing-SA, the purpose is to dynamically adjust the learning rate in the Q-learning algorithm to be compatible with the frequent changes in topology in FANETs to reduce energy consumption and packet loss. FEQ-routing-SA has major advantages, such as acceptable computational complexity, low routing overhead, and local decision-making about routing paths. In the first step, FEQ-routing-SA proposes a trajectory construction process based on a piece-wise linear mobility model. In this method, a hierarchical generative model is introduced to produce random parameters for each UAV based on its class to deduce its motion profile. Then, each UAV implements the Q-learning-based routing process to find the most appropriate route between source and destination and reduce the energy required for data transmission. This routing procedure allows UAVs to decide on the previous experience and minimize the transmission energy required for sending the packet. In FEQ-routing-SA, no information is exchanged between nodes. This significantly reduces the routing overhead. In this process, each node plays the agent role and the status of a node, including location and energy, and is considered the state space. Furthermore, the reward function is defined based on the transmission energy required for transferring packets from the source to destination. This learning process is shown in Figure 33. Note that this scheme defines two learning rates: the basic learning rate, which is constant, and the extra learning rate estimated based on  
Figure 32: Learning process in QLGR.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Advantage** & **Disadvantages** \\ \hline \multirow{7}{*}{QLGR [73]} & Designing a distributed routing method, adaptive & \\  & adjustment of hello broadcast interval for reducing & \\  & routing overhead, creating stable paths, reducing packet & \\  & loss, considering a multi-agent reinforcement learning & \\  & technique, improving convergence speed, high & \\  & scalability, using both local and global rewards, & \\  & managing Q-table size, solving the routing hole problem & \\  & by applying a routing recovery mechanism, preventing & \\  & congestion in the network. & \\ \hline \hline \end{tabular}
\end{table}
Table 14: The most important advantages and disadvantages of QLGR.  
the delivery time. In this method, the simulated annealing algorithm is used to determine the exploration rate. SA starts at a large exploration rate (high temperature) and learns better decisions by lowering the temperature. In FEQ-routing-SA, the temperature is adjusted according to the speed of the network nodes. In this process, when changing node speed, the temperature reaches its highest level and gradually is cooled during the interval. The fitness function in this method is calculated based on energy consumption changes (or Q-value) in the last ten iterations. Table 15 expresses the most important advantages and disadvantages of FEQ-routing-SA in summary.