# A Survey on Applications of Reinforcement Learning in Flying Ad-Hoc Networks
## 4 Applications of RL in FANET
### Routing Protocol
#### 4.1.3 Multiobjective Routing Protocol  
[PERSON] et al. [50] proposed a \(Q\)-learning-based fuzzy logic for multiobjective routing protocol. The source node determines the routing path using the proposed algorithm while considering the transmission rate, residual energy, energy drain rate, hop count, and successful packet delivery time. A fuzzy system is used to identify reliable links, and \(Q\)-learning supports the fuzzy system by providing a reward on the path [50]. The algorithm not only considers the single-link performance but also the whole path performance using two types of \(Q\)-values from two \(Q\)-learning algorithms. After obtaining the \(Q\)-values for the single links and the entire path, the fuzzy logic evaluates the \(Q\)-values and determines the optimal path for routing. Moreover, the learning parameters, such as the discount factor and learning rate, are fixed for the \(Q\)-learning algorithm.  
Similarly, in [51], [PERSON] et al. determined the routing path using a fuzzy logic-based RL algorithm, but they considered delay, stability, and bandwidth efficiency factors. Figure 11 summarizes the applications of RL in the routing protocol via block diagrams. Moreover, a comparative analysis of the aforementioned protocols is presented in Table 2.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline
**Routing Protocol** & **Algorithm** & **Advantages** & **Limitations** \\ \hline \multirow{8}{*}{QMR [12]} & & **1.** Multiple objectives, such as end-to-end delay, energy consumption are considered. & **1.** Re-establishing communication is uncertain if a node gets lost. \\  & \(Q\)-learning with dynamic learning rate, discount factor, and adaptive mechanism of explanation and exploitation & **2.** Dynamic and adaptive mechanism of explanation and exploitation. & **2.** Whole route stability is not considered. \\  & \(Q\)-learning with dynamic learning rate, discount factor, and adaptive mechanism of explanation and exploitation & **3.** An adaptive mechanism is used for balancing exploration and exploitation. & **3.** Computational energy consumption is not considered. \\  & & **4.** A penalty mechanism is used to combat “neighbor unavailability” problem. & **4.** Computational energy consumption is not considered. \\ \hline \multirow{8}{*}{RLSRP with PPMAC [49]} & & **1.** The positions of nodes are predictable. & **1.** Fixed RL parameters. \\  & Reinforcement learning with partially observable Markov decision process (POMDP) & **2.** Antenna direction can be changed towards the routing direction. & **2.** Only end-to-end delay is considered for route selection. \\  & Reinforcement learning with partially observable Markov decision process (POMDP) & **3.** Partially observable Markov decision process (POMDP) is used. & **3.** There is no adaptive mechanism for balancing exploration and exploitation. \\  & & **4.** Broadcasting is used for re-establishing the communication with other nodes. & **4.** Computational energy consumption is not considered. \\ \hline \multirow{8}{*}{Multiobjective Routing Protocol [50, 51]} & & **1.** Multiple factors, such as the transmission rate, residual energy, energy drain rate, hop count, and successful packet delivery time are considered. & **1.** There is no adaptive mechanism for balancing exploration and exploitation. \\  & \(Q\)-learning-based fuzzy logic & **2.** Both single and whole route performances are considered. & **2.** Fixed RL parameters. \\  & & **3.** Two \(Q\)-values are used from two \(Q\)-learning algorithm. & **3.** There is no mechanism to remedy the “neighbor unavailability” problem. & **3.** There is no mechanism to remedy the “neighbor unavailability” problem. \\  & & **4.** Computational energy consumption is not considered. & **4.** Computational energy consumption is not considered. \\ \hline \end{tabular}
\end{table}
Table 2: Comparative analysis of the routing protocols based on RL in FANET.  
Figure 11: Application of RL in Routing Protocols of FANET.