# Survey on Q-Learning-Based Position-Aware Routing Protocols in Flying Ad Hoc Networks
However, considerable theoretical disputes occur regarding UAV swarm control based on partial and relative local knowledge without the intervention of a central controller. Recently, significant developments have been achieved for the distributed control of UAV swarms, in which local optima are avoided by utilizing several metaheuristics [13] and game-theory-based optimization [14; 15]. The major limitation of metaheuristics and game theory is their high computational complexity with increased nodes and topology size. Therefore, to satisfy the scalability requirement and solve existing challenges such as delay, energy constraints, and mobility, an intelligent alternative approach that can perform multi-objective optimization adaptively in highly dynamic FANETs is required. Distributed algorithms such as the artificial potential field (APF) [12; 16], virtual force [10; 17], and bold flocking [9] have attracted the interest of researchers in constructing topology in FANETsby producing optimal mobility of UAVs in FANETs. These algorithms are highly efficient at performing external obstacle avoidance [11], generating a smooth trajectory for UAVs [10], and maximizing coverage under connectivity constraints [18; 19]. The adaptive adjustments of their attractive and repulsive force fields also support the maintenance of an optimal node density in FANETs to minimize interference in aerial links [12; 20]. In FANETs, link quality depends on several parameters, such as inter-UAV distance, node density, SINR, delay, relative mobility, and residual energy of relaying UAVs. The optimal node density and link SINR can be achieved by jointly optimizing the UAV mobility (position, velocity, and acceleration) and transmitting power according to the inter-UAV distance by adopting a topology control technique [12; 21]. The link delay includes MAC-layer channel access, queuing, propagation, processing, and transmission delays. The optimal resource allocation in resource-constrained FANETs, such as physical-layer UAV transmission power, MAC-layer time slots, or frequency resources, can significantly improve the SINR level in aerial links. Thus, this sequentially improves the network-layer performance (relay selection) as they are highly coupled. Owing to the above advantages, researchers have jointly considered the MAC layer delay, link SINR, relative mobility, position progress to the destination, and residual energy of neighboring UAVs, to design a multi-objective reward function in reinforcement learning (RL)-based algorithms [22; 23; 24]. RL is an area of machine learning concerned with how intelligent agents ought to take an action from a specific state by interacting with an environment to maximize rewards. Through the iterative state transitions, an agent learns how to choose an optimal action. Thus, RL-based action can be formulated as a Markov decision process (MDP) tuple consisting of state, action, and reward. The state represents the consequences that an agent faces in a dynamic environment by taking actions according to the learning policy. Through sequential action and utilizing previous experience, RL agents can make wiser decisions to reach a common objective. In communication theory, RL is applied in many scenarios such as channel modeling, resource allocation, and security [25; 26]. Recently, RL has been widely used in FANETs to design the smooth collaborative trajectory planning for UAV swarms with collision avoidance [27], and routing protocol design [23]. Q-learning (QL) is a model-free value-based off-policy RL approach, which can obtain an instant optimal policy based on historic experiences even without prior information of the environment or even without the intervention of any central controller [28]. Here, each agent makes an optimal decision based on its neighbor state information, which can be treated as partial MDP (PMDP).