# Q-Learning Empowered Highly Dynamic and Latency-Aware Routing Algorithm for Flying Ad-Hoc Networks
## 4 QEMLR Routing Protocol
### Routing Establishment  
The routing establishment employs periodic flooding of Hello packets to construct and maintain the routing table. Each node periodically transmits and receives Hello packets from neighboring nodes, utilizing the contained information to establish the reverse link and continuously update the routing table and Q-table. The Q-learning algorithm is utilized in this protocol to calculate the Q-value and assess link stability. Each node transmits Hello packets containing routing information for route discovery and receives Hello packets from other nodes. The Q-value is then calculated based on the information extracted from these packets, and the corresponding nodes are added to the neighbor table and routing table. In the Q-model of this protocol, nodes initialize the Q-value to 1 and include it in the Hello message for transmission. The agent is the node that transmits the Hello packet in  
Figure 5: General flow chart of the protocol. the network, and the action represents the selection of the neighbor to transmit the current Hello packet. The new state is the forwarding of the Hello packet to the neighboring node. As depicted in Figure 6, the Hello packet was just forwarded to node \(i\) from source \(d\), and at this time, neighbor node \(j\) is selected for transmission. The Hello packet is forwarded to node \(j\), and the transmission feedback value is calculated based on the current network condition. The new feedback value is then utilized to compute the new Q value, using the Q value contained within the Hello message and the maximum Q value present in the Q table of node \(j\). The updated Q value is subsequently employed to update the Hello message and disseminate it to other nodes in the network. This propagation process \(d\to i\to j\to k\dots\dots\) involves the use of the Q value as a coefficient of reverse routing, which measures the suitability of node \(j\) to transmit data through the next hop \(i\) and the suitability of node \(k\) to receive data through node \(j\). Equation (1) represents the formula for calculating the Q value during the route establishment process. \[Q_{d}(i,j)=(1-\alpha)Q_{d}(i,j)+\alpha(reward(i,j)+\gamma\times\max Q_{d}(i,j)), \tag{1}\]  
where \(Q_{d}(i,j)\) is the reverse weight of node \(i\) choosing node \(j\) to transmit hello packets, which is the weight of \(j\) reaching source node \(d\) through \(i\). The current reward obtained by node \(i\) when it selects node \(j\) for transmission, is denoted as \(reward(i,j)\). The maximum Q value that can be achieved by the source node \(d\) through node \(i\) is denoted as \(\max Q_{d}(i,j)\). The parameter \(\gamma\) is used to determine the relative ratio of delayed returns to immediate returns, with larger values indicating the higher importance of delayed returns. Specifically, \(\gamma\) is given by Equation (2)  
\[\gamma=\gamma_{0}\times MF_{i}\times\varepsilon_{i,j}, \tag{2}\]  
with the presence of node mobility \(MF_{i}\) and the delay factor \(\varepsilon_{i,j}\) provided by Equations (3) and (4),  
\[MF_{i}=\left\{\sqrt{1-\frac{\left|(N_{i}(t)\cap N_{i}(t-\Delta t))\cup(N_{i}( t)\cap N_{i}(t-\Delta t))\right|}{|N_{i}(t)\cup N_{i}(t-\Delta t)|}}\right.,N_{i}(t) \cup N_{i}(t-\Delta t)\
eq\varnothing \tag{3}\]  
\[\varepsilon_{i,j}=e^{-delay\_onehop_{i,j}}. \tag{4}\]  
The mobility of neighboring nodes \(MF_{i}\) is utilized as a metric to quantify the change in node aggregation around node \(i\) over a given time period \(\Delta t\), thereby measuring the link stability factor around node \(i\). The current neighbor set of node \(i\) is denoted as \(N_{i}\), while \(N_{i}(t-\Delta t)\) represents the neighbor set of node \(i\) before time \(\Delta t\). The change in the neighbor set of nodes \(i\) after a period of time can be measured by performing corresponding calculations for these two sets of neighbor nodes. A relatively large value of \(MF_{i}\) indicates  
Figure 6: Q model of the protocol.