# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 4 Flying Ad Hoc Networks  
Recent advances in wireless technologies, easy access to radio interfaces, and other equipment such as positioning systems, sensors, and microcomputers, lead to the production of smart and small flying vehicles, especially unmanned aerial vehicles (UAVs) that form a new network called a flying ad hoc network (FANET) [28; 31]. In general, a flying ad hoc network consists of a group of UAVs that cooperate with each other and communicate without any infrastructure to perform a specific task without human intervention [32; 33]. In FANETs, all UAVs can establish UAV-to-UAV communication and only a small number of UAVs can communicate with the ground station (GS) [34; 35]. As a result, UAVs do not require complex hardware. When breaking communication links between UAVs, the connection between the ground station and ad hoc network is always active [36; 37]. However, these networks face several challenges:  
* **Connectivity:** In this network, UAVs have high mobility and low density, which cause the failure of communications between UAVs and affect network connectivity. These features lead to unstable links and high delay in the data transmission process [38]. * **Battery:** The biggest challenge in these networks is energy consumption because small UAVs use small batteries to supply their required energy for real-time data processing, communications, and flight [38]. * **Computational and storage capacity:** Flying nodes have limited resources in terms of storage and processing power. It is another challenge in FANETs that must be considered when designing a suitable protocol for sending data packets by UAVs to the ground station [38]. * **Delay:** Multi-hop communications are suitable for ad hoc networks such as FANETs, which are free-infrastructure, to guarantee end-to-end connectivity. However, this increases delay in the data transmission process. As a result, providing real-time services on these networks is a serious challenge [38]. * **Interference management:** UAVs are connected to each other through wireless communication. Due to the limited bandwidth capacity in this communication model and dynamic topology in FANETs, interference management is difficult and complex [38]. In FANETs, there are three types of communication between UAVs:
* **UAV-to-UAV communication (U2U):** As shown in Figure 2, UAVs communicate with each other using U2U communication in a multi-hop manner. This improves their communication range and increases the data rate. This communication is used when a UAV wants to send its data packet to another UAV or ground station beyond its communication radius [39]. * **UAV-to-GS communication GS (U2G):** As shown in Figure 2, UAVs communicate directly with GS through U2G communication when it is in their communication range. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** & **Advantage** & **Disadvantages** \\ \hline Dynamic programming (DP) & Acceptable convergence speed & Considering a complete model of the environment, high computational complexity \\ \hline Monte Carlo (MC) & A free-model reinforcement learning method & High return variance, low convergence speed, trapping in local optimum \\ \hline Q-learning & A free-model, off-policy, and forward reinforcement learning & Lack of generalization, inability to predict the optimal value for unseen states \\ \hline State-action–reward–state–action (SARSA) & A free-model, on-policy, and forward reinforcement learning & Lack of generalization, inability to predict the optimal value for unseen states \\ \hline Deep reinforcement learning (DRL) & \
\begin{tabular}{c} Suitable for solving problems with high dimensions, the ability to estimate \\ unseen states, the ability to generalization \\ \end{tabular} &
\begin{tabular}{c} Being unstable model, making rapid changes in the policy with little change in \\ Q-value \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparision of reinforcement learning algorithms [15].