# Survey on Q-Learning-Based Position-Aware Routing Protocols in Flying Ad Hoc Networks
## 3 Q-Learning and Its Relationship with Routing in FANETs  
This section provides a brief overview of the effective QL algorithm and its relationship with position-based routing decisions in FANETs. In QL, agents iteratively adjust their action strategies through the reward that they achieve from the environmental feedback after performing a particular action (Figure 2). The agent selects an action in a particular state according to the reinforcement or simply the previous experience, known as the Q-value. The reinforcement comprises a direct reward and future Q-value expectation. Through reinforcement, agents can evaluate the effectiveness of an action in the current state and perform a better action in the next step. The objective of the agent is to maximize the expectations of the cumulative rewards over the sequential iteration. The advantage of QL is that the reward function can be designed using multiple weighted objectives to achieve multiple goals. Additionally, a suitable strategy for exploration and exploitation can aid in attaining the global optimal solution. In FANET routing, the data packets contained by each UAV node (source node) act as agents that function by relaying UAV selections from their neighbor to forward data packets to the desired destination (another UAV or BS). Through this sequential action, the \begin{table}
\begin{tabular}{c c c} \hline \hline
**Mobility Model** & **References** & **Applications in FANETs** \\ \hline \multirow{2}{*}{Boid flocking} & \multirow{2}{*}{[9]} & Surveillance, target searching, and communication in an emergency scenario, and simulating the routing and MAC protocol \\ \hline \multirow{2}{*}{Virtual force} & [4; 10; 11; 17] & Surveillance, ABS deployment, ground target searching and tracking, trajectory tracking, and leader–follower topology formation \\ \hline \multirow{2}{*}{Virtual spring} & \multirow{2}{*}{[18; 19]} & Aerial base station deployment, surveillance, aerial mesh network deployment, and target searching and tracking. \\ \hline \multirow{2}{*}{Artificial potential field} & \multirow{2}{*}{[12; 16; 46; 47]} & Surveillance, leader–follower topology formation, trajectory tracking, and target searching and tracking \\ \hline \multirow{2}{*}{Pheromone-based} & \multirow{2}{*}{[48; 49]} & Ground target searching and tracking, and reconnaissance \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Summary of realistic mobility models in FANETs according to applications. Figure 2: State, action, and environment relationship with an agent in the QL model in FANETs. data packets change their states from one UAV to another. This state transition continues until data packets are delivered to the desired destination node. For each action (relay node selection from the neighbor list), the UAV agent (source node) receives a reward or penalty. Gradually, each agent UAV gathers an experience represented by a Q-value that results in an optimal policy in which the cumulative reward is maximized over the iteration. The reward function is designed such that UAVs select an optimal routing path that minimizes the delay and creates a load balance in UAV energy consumption. The penalty mechanism in the reward function is triggered to avoid the local minimum problem if the next state of the data packets does not have any routing path forward toward the destination. It also aids in avoiding the routing loop and minimizes unnecessary detours of data packets during routing.