# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 5 Proposed Classification
### Classification of RL-Based Routing Protocols Based on Learning Algorithm
#### 5.1.1 Traditional Reinforcement Learning-Based Routing Method  
In this routing protocol, the agent learns the network environment without any initial knowledge and finds the suitable path between the source and destination. This learning system is shown in Figure 7. Compared to deep learning reinforcement-based routing methods, these routing protocols are simpler and have less computational complexity. Therefore, they are easier to implement. Note that traditional reinforcement learning algorithms are suitable for discovering the best routing policy in small-scale FANETs because the dimensions of the state and action spaces are controllable, and the learning algorithm has an acceptable learning speed. However, if the flying ad hoc network is large-scale, traditional reinforcement learning algorithms cannot perform well to find the best routing policy in the network because the state and action spaces are large, and the learning algorithm has a slow convergence speed [49].  
In these routing methods, an agent alone learns the best behavior or the best route between the source and destination through interactions with the environment [50,51]. A single-agent reinforcement learning system is represented in Figure 8. The performance of these routing methods is not suitable for large-scale networks with large state and action spaces because the agent needs a long time to find an optimal response, meaning that their convergence speed is slow. In some cases, the agent may never find an optimal policy for the network. However, compared to multi-agent routing schemes, these methods are easier  
Figure 6: Classification of reinforcement learning-based routing protocols.  
Figure 7: Traditional reinforcement learning system.  
In these routing methods, a group of agents (for example, UAVs) tries to learn the optimal behavior through interactions with the network to find the best route between the source and destination. A multi-agent reinforcement learning system is shown in Figure 9. An important challenge in these routing methods is how to coordinate and cooperate among agents because the behavior of agents can affect the dynamics of the network environment [50]. In these routing methods, if the agents can communicate with each other and exchange their experiences, the routing calculations are performed parallel by the agents, and the learning ability of the multi-agent system is greatly improved. These routing protocols are fault-tolerant, meaning that if one or more agents fail in the network for any reason, other agents can perform their tasks to prevent an abnormal network performance [51; 52]. Furthermore, these protocols are suitable for networks with large state and action spaces because they have more learning ability. However, they have more computational complexity than single-agent methods.  
Model-Based Routing  
In these routing  
In these routing protocols, the task of the agent is to estimate the value function directly based on knowledge obtained from the environment and does not create any model of the network environment. Figure 11 shows a free-model learning system. Compared to model-based learning methods, these routing methods have appropriate computational complexity and are suitable for large and time-sensitive applications [53; 54]. However, these routing methods must perform more interactions with the environment to obtain more experiences for finding an optimal response and have less flexibility relative to the sudden changes made to the network.