# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 6 Investigating Reinforcement Learning-Based Routing Protocols
### ICRa  
[PERSON] et al. [80] have presented an intelligent clustering routing approach (ICRA) for flying ad hoc networks. This method increases the stability of the network topology and improves network lifetime by balancing energy consumption in the network. In ICRA, the nodes are equipped with GPS to obtain their position and movement information. They broadcast hello messages periodically on the network to exchange information including location, speed, movement direction, and timestamp with other network nodes, and form a neighborhood table. This information is used in the clustering process. Clustering balances energy consumption in the network. ICRA includes three phases: clustering, clustering strategy adjustment, and routing. In the clustering process, each node calculates a utility parameter based on four utility factors, including residual energy, centrality, speed similarity between a node and its neighbors, and the link holding time. Then, UAVs share their utilities with each other to select a node with the highest utility as the cluster head node. Then, CH nodes broadcast an advertisement message to announce their role. When non-CH nodes receive advertisement messages from different CH nodes, they are connected to a CH node with higher utility and a longer link lifetime. If a cluster member node receives a hello message from a CH node except its own CH, this cluster member node acts as an inter-cluster forwarding node. CH and inter-cluster forwarding nodes are responsible for creating routes between different clusters. When calculating the utility parameter, the weight coefficients corresponding to the four utility factors are characterized by a Q-learning-based clustering strategy adjustment process. This learning process follows a centralized strategy and is implemented by the ground station (GS). In this process, GS plays the agent role and the action space includes the selection of four weight coefficients corresponding to the utility factors. Furthermore, the state space represents four utility factors. The reward function is also calculated based on two parameters, including stability of the cluster structure (i.e., the number of times changing the node roles in the network) and the energy change rate of the nodes. In this process, learning parameters, including learning rate and discount factor, have fixed values. This learning process is represented in Figure 39. In the routing process, when one UAV receives a data packet from another UAV, and is not the destination of this packet, it checks the location of the destination node. If the destination node is its single-hop neighbor, it sends the data packet directly to the destination node. Otherwise, the following three modes should be considered.  
* Is the receiver node a CH? If yes, it sends the data packet to the closest CH node or inter-cluster forwarding node to the destination.  
Figure 38: Learning process in 3 DQ.  
* Is the receiver node an inter-cluster forwarding node? If yes, it selects the closest node to the destination among its CH node and its neighboring nodes in other clusters and sends the packet to the selected node.
* Is the receiver node a cluster member node? If yes, it sends this data packet directly to its CH.  
Table 21 briefly describes the most important advantages and disadvantages of ICRA.