# Survey on Q-Learning-Based Position-Aware Routing Protocols in Flying Ad Hoc Networks
## 4 Q-Learning-Based Position-Aware Routing Protocols in FANETs
### QL-Based Topology-Aware Routing Protocol (Qtar)  
[PERSON] et al. [23] proposed QL-based topology-aware routing (QTAR) for FANETs, which can make better routing decisions by extending the local view of each UAV using two-hop neighbor information. They updated the Q-values by adaptively adjusting the learning rate based on the exponential normalized two-hop delay and discount factor based on the one-hop current and previous neighbor similarity set of each UAV neighbor. Throughthis process, QTAR produces a more stable Q-value for better exploration, considering the dynamism in the FANET topology. The reward function includes the delay in terms of the MAC and queuing delay, UAV residual energy level, and PTS for two-hop links. They adaptively adjusted the hello interval based on the minimum link duration among one-hop neighbor UAVs.  
_Advantages_: The multi-objective reward function using two-hop neighbor information optimizes the delay and creates a proper load balance in the UAV energy consumption during multi-hop routing. The penalty mechanism in the reward function ensures avoidance of routing loops, routing holes, and local optima.  
_Limitations_: QTAR considers velocity constraints that jointly consider distance progress and channel conditions in terms of MAC and queuing delay for two-hop neighbor UAVs to ensure data packet delivery within the deadline PTT. Proper mobility control is necessary to satisfy this PTS constraint; otherwise, the selected relay UAV may leave the transmission range of the corresponding source UAV owing to the change in relative velocity. Maintaining two-hop neighbor information produces a high overhead for each UAV.  
Predictive Ad Hoc Routing Protocol Fueled by Reinforcement Learning and Trajectory Knowledge (PARRoT)  
[PERSON] et al. [29] proposed predictive ad hoc routing fueled by reinforcement learning and trajectory knowledge (PARRoT) for FANETs to perform the collaborative mission of a UAV swarm. In PARRoT, each UAV exchanges hello packets with a one-hop neighbor, which includes a unique UAV address, hello packet sequence number, current position, predicted future position, reward, and cohesion value. Based on the hello packet transmission and reception in reverse order (from the destination to the source node), each node updates the link-quality status represented as a Q-value. To address the dynamic topology in FANETs, the discount factor is calculated by considering two dynamic parameters: link duration and cohesion value. The link duration is calculated using a mobility controller that utilizes the trajectory knowledge (relative velocity, relative position, and transmission range) of two neighboring UAVs. The cohesion value is calculated based on the variation in the neighbor set at two different times. The neighbor link that provides a high link duration and cohesion value receives more rewards. To make the routing decision, each source node selects a neighbor link based on the maximum Q-value.  
To simulate the routing behavior, they considered both generic and realistic mobility models. For instance, they utilized the distributed dispersion detection (DDD) mobility model [54], which is suitable for exploring plume sources in disaster scenarios. In addition, they considered the dynamic cluster hovering (DCH) method [55], in which a UAV swarm dynamically adjusts its locations to provide network coverage of ground-based vehicles.  
_Advantages_: PARRoT introduces relative trajectory knowledge between two neighboring UAVs to make routing decisions in FANETs. Owing to the knowledge of relative mobility information, it selects a more stable neighbor link in FANETs and provides the highest survival time of the link to complete data transmission.  
_Limitations_: The residual energy of the neighbor node and routing loop are not considered in routing decision making. When making the routing decision, PARRoT always exploits the link with the maximum Q-value; thus, no exploration criteria are considered.