# Q-Learning Empowered Highly Dynamic and Latency-Aware Routing Algorithm for Flying Ad-Hoc Networks
## 2 Related Works
The optimal routing path selection is realized by using the learning ability of a machine learning algorithm based on an accurate perception of network topology, channel state, user behavior, and traffic mobility. These algorithms can better satisfy the service quality requirements of dynamic UAV networks. [PERSON] proposed a Q-learning based multi-objective optimal routing algorithm, QMR [17], based on three routing protocols, QGrid [18], QLAR [19], and QGeo [20], which optimizes both end-to-end delay and energy consumption metrics. QMR obtains geolocation data through GPS and establishes the route exploration process by sending Hello packets. Based on QMR, a protocol [21] was also proposed. [PERSON] proposed a multi-objective Q-learning routing protocol based on fuzzy logic [22], which considers metrics such as transmission power and hop count. This protocol uses fuzzy logic to identify reliable data links and Q-learning to calculate the payoff value given to the path. Similarly, the fuzzy logic routing protocol proposed by [PERSON] [23] considers factors such as time delay, network stability, and bandwidth efficiency. [PERSON] proposed the PARRoT [24] routing protocol, which is based on [PERSON]'s B.A.T. Mobile [25] protocol. The learning rate of the protocol is fixed, while the discount factor is calculated based on the link failure time and the degree of aggregation of neighboring nodes. The packet delivery rate of the protocol is superior to that of B.A.T. Mobile and B.A.T.M.A.N. [26], but it faces significant computational complexity, and the actual protocol running time will increase sharply with the increase in the number of nodes. [PERSON] et al. utilized fuzzy logic [27] to determine neighboring nodes in real-time and designed a reinforcement learning-based future reward method that reduces the average hop count through continuous training. Compared with the Ant Colony Optimization (ACO) algorithm, this algorithm has a lower average hop count and higher link connectivity. [PERSON] also proposed a protocol named AR-GAL [28] that selects the route with the minimum end-to-end delay based on the continuous network conditions of FANETs. The protocol formulates the routing decision process as a Markov Decision Process (MDP) and designs a new MDP state composed of the current node state and the neighbor environment state. Table 3 summarizes the main contributions and problems of the above protocols. \begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt} p{142.3 pt}} \hline \hline
**Protocol** & **Contributions** & **Limitations** \\ \hline QMR [17] & Dynamic adjustment of Q-factor improves packet delivery rate and reduces latency & Designing alternatives for routing decisions based on speed alone is not good enough \\ \hline Q-FANET [21] & QMR-based improvements to provide lower latency and jitter & Did not verify protocol performance at different speeds \\ \hline QLFLMOR [22] & Maintain a lower hop count to reduce energy consumption and extend network survival time & Poor performance with a low number of nodes \\ \hline FLRLR [23] & Reduce average hop count and improve link connectivity & It does not verify the number of hops versus packet delivery rate as the number of nodes increases \\ \hline PARRoT [24] & Enables robust data delivery & Large computational complexity \\ \hline Fully-echoed Q-routing [27] & Introduced a new full echo Q protocol that avoids connection loss & Does not consider node mobility in the protocol \\ \hline AR-GAL [28] & A deeply reinforced routing protocol that introduces generative adversarial imitation learning can reduce latency and improve packet delivery rate & Routing failure exists at low flight density \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The main contributions and problems.