# Survey on Q-Learning-Based Position-Aware Routing Protocols in Flying Ad Hoc Networks
## 3 Q-Learning and Its Relationship with Routing in FANETs
The QL-based routing decision process can be described as a partial MDP tuple \((S,A,P,R,\alpha,\gamma)\), where a set of states \(S\in\{s_{u_{i}}(t),i=(1,2,\ldots U)\}\) represents the \(u_{i}\in U\) active UAV node positions at each time \(t\), a set of actions \(A\in a_{u_{ij}}\) represents the selection of the relay UAV (\(u_{j}\)) from the neighbor list of the source UAV (\(u_{i}\)) to forward the data packets to the destination, \(P\) represents the probability of successful state transition of a particular data packet from one UAV to another UAV, and \(R\in r_{u_{ij}}\) is the reward function that can be obtained by a predefined reward function to evaluate the quality of the action (to select the link \(u_{ij}\)). The Q-values are iteratively updated at each UAV node using the following Bellman equation:  
\[Q\Big{(}s_{u_{ij}},a_{u_{ij}}\Big{)}\,\leftarrow\,Q\Big{(}s_{u_{ij}},a_{u_{ij} }\Big{)}+\alpha\left[r_{u_{ij}}+\gamma\,\underset{a_{u_{ij}}^{\prime}}{\max}\,Q \Big{(}s_{u_{ij}}^{\prime},a_{u_{ij}}^{\prime}\Big{)}-Q\Big{(}s_{u_{i}},a_{u_{ ij}}\Big{)}\right] \tag{1}\]  
where the term \(\underset{a_{u_{ij}}^{\prime}}{\max}\,Q\Big{(}s_{u_{ij}}^{\prime},a_{u_{ij}}^{ \prime}\Big{)}\) represent the future Q-value expectation in the next state \(s_{u_{ij}}^{\prime}\) after implementing the best action \(a_{u_{ij}}^{\prime}\). \(\alpha\) and \(\gamma\) represent the learning rate and discount factor, respectively, the values of which are defined within the range \([0\,1]\). The significance of \(\alpha\) is how rapidly the QL algorithm should learn, and \(\gamma\) defines how much the QL algorithm learns from its mistake owing to a bad action. The value of \(\alpha\) determines the degree to which the newly obtained information overrides the old information, and this parameter controls the convergence of the learning procedure. The value of \(\gamma\) controls the importance of future rewards. In FANETs, to obtain a better learning process and more stable Q-value, the values of \(\alpha\) and \(\gamma\) should be controlled adaptively with respect to the changes in the mobility of UAVs, considering the dynamic topology. The stability of the topology and the degree of change in the mobility of UAVs can be obtained with the aid of a topology control algorithm. Through proper mobility management defined by the topology controller, a UAV swarm can calculate the mobility factor and link duration [22; 23]. The parameter link duration defines how long two neighboring UAVs exist within transmission range of each other [21]. The link duration is a function of the UAV transmission range, relative velocity, and distance between two neighboring UAVs [21]. If the mobility factor is high and the link duration is low, the discount factor should be low, and vice versa. Thus, by incorporating a topology controller in the routing decision-making process, UAVs can reduce the number of retransmissions. Similarly, the delay, including the MAC-layer, queueing, and transmission delays, can be obtained from the designed MAC protocol and queuing model [22]. \(\alpha\) can be adjusted adaptively according to the delay. In the QL algorithm, an appropriate balance between exploration and exploitation is necessary to avoid local optima. Exploration refers to the seeking of new actions by the agent (source UAV) to gather new experiences. Exploration can provide good or bad rewards, but it can aid in determining the global optima. In contrast, exploitation involves taking action according to previous knowledge (Q-value). Owing to the limited energy of UAVs, excessive exploration can delay convergence and may include many bad actions that may increase the number of retransmissions. Therefore, proper guidelines are required to explore a better state (relay UAV) from the neighbor list.