# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
###### Abstract
Despite many efforts in this area, it is very difficult to design a routing protocol that guarantees efficient communication in these networks [12; 13]. Machine learning (ML) techniques can be used for solving various challenges, such as routing in FANETs. In general, machine learning is divided into three groups: supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, the designed model must explore dependencies between the training data (labeled data) to predict the correct answer to the requested problem (unlabeled data). In fact, this technique trains a model based on initial data to predict the label of new data. Unsupervised learning also tries to discover the patterns in data samples. In this learning process, the algorithm  
Figure 1: Flying ad hoc network. uses an unlabeled dataset, whereas reinforcement learning (RL) can learn the environment without any initial data samples. RL is similar to the human learning approach, meaning that this learning process does not require large datasets and the learning agent is not aware of the data label. Furthermore, reinforcement learning learns through interactions with the environment without the need for any dataset. The FANET is a very dynamic and complex network. For this reason, supervised and unsupervised learning techniques cannot find appropriate responses for routing in the FANET environment. For example, when failing each link or breaking each path, supervised and unsupervised learning methods must first have a dataset related to the failed path to find a new response (new paths) through learning this dataset. Reinforcement learning is suitable for FANET because it can control the dynamic and real-time environment in a desirable manner. RL can constantly learn new information about FANETs and communication links between nodes. Note that it is difficult to model the routing process in FANETs. To model this process, supervised and unsupervised learning methods must first execute a simulation process to produce a dataset because there is no dataset to train the model; then, they use this dataset to train the model. RL is the only machine learning technique which can learn the routing process without the need for a dataset. The reinforcement learning algorithms use the trial-and-error technique to learn a proper routing model in the network environment. This method can reduce the complexity of supervised and unsupervised learning methods to simulate and solve this problem in FANETs. Today, reinforcement learning (RL) techniques have been used in flying ad hoc networks to solve challenges related to the routing issue. RL is a branch of artificial intelligence (AI), which allows machines to become intelligent without human intervention and learn based on previous experiences [14; 15]. Reinforcement learning increases efficiency and reliability, and reduces computational costs compared to other AI techniques. In this process, the agent interacts with the dynamic environment to find its ideal behavior based on the reward-penalty feedback received from the environment [16; 17]. In FANETs, reinforcement learning allows drones to decide on various network operations, especially routing. In reinforcement learning, the agent should understand the environment by collecting data from the environment to find the best action for achieving a specific goal, such as creating a route with a maximum packet delivery rate (PDR) [17; 18; 19]. In fact, reinforcement learning algorithms have potential to improve routing protocols in FANETs. Therefore, it is necessary to study the RL applications in flying ad hoc networks. There are several works in this area. However, the number of review papers is not sufficient and further studies should be carried out. For example, ref. [20] has presented a comprehensive and useful review paper about artificial intelligence-based routing protocols in flying ad hoc networks.