# Q-Learning Empowered Highly Dynamic and Latency-Aware Routing Algorithm for Flying Ad-Hoc Networks
## 4 QEMLR Routing Protocol
### Routing Maintenance  
Routing maintenance serves as a crucial component in ensuring link stability by calculating link failure time and performing timely deletion operations of failed links in both the routing table and Q-table. Routing maintenance supplements the route establishment, enabling the routing decision to identify the most suitable and stable link for outbound packet transmission based on the route entries in the routing table. The Friis free loss model [30] is employed to simulate this protocol, allowing for the inference of the effective communication range based on this model. \[P_{r}=\frac{P_{t}G_{t}G_{r}\lambda^{2}}{(4\pi d)^{2}L}, \tag{6}\]  
Figure 7: Flow chart for calculating the delay. In the following equation, \(P_{t}\) represents the transmission power, \(P_{r}\) denotes the receiving power, \(G_{t}\) and \(G_{r}\) are the antenna transmission and receiving gains, respectively. \(\lambda\) represents the wavelength (\(\lambda=c/f\), where \(c\) is the speed of light and\(f\) is the frequency), \(d\) is the communication radius, and \(L\) is the system loss factor. Specifically, the communication radius can be obtained using the following Equation (7):  
\[d=\left(\frac{1}{L}\frac{P_{t}G_{t}G_{r}}{P_{r}}(\frac{c}{4\pi f})^{2}\right)^ {\frac{1}{2}} \tag{7}\]  
Upon receiving a hello packet from node \(j\), node \(i\) can calculate the failure time of the link between them based on the location information provided in the hello message. If the time difference between the current time \(Time_{non}\) and the last time on receiving the hello packet \(Time_{last\_sent}\) exceeds the minimum link failure time \(Time_{LinkEpiry}\) and packet interval value as defined in Equation (8), the link is deemed to be in an unstable state. As a result, the neighboring node is removed from both the current node's neighbor table and Q table to ensure the stability of the final dynamic routing decision. \[Time_{now}-Time_{last\_sent}>\min(Time_{LinkEpiry},hello_{interval}) \tag{8}\]  
The position of node \(i\) and node \(j\) can be represented by \(p_{i}=(x_{i},y_{i},z_{i})\) and \(p_{j}=(x_{j},y_{j},z_{j})\) with \(\Delta p=p_{i}-p_{j}\) representing the position change in two nodes, \(\Delta v=v_{i}-v_{j}\) being the speed change of two nodes, the following Equation (9) can represent the position change \(\Delta p_{total}\) between \(i\) and \(j\),  
\[\Delta p_{total}=p_{i}-p_{j}+t\times\Delta v, \tag{9}\]  
where \(t\) is the link failure time between two nodes. Moreover, the communication distance between the two points is \(d\). (Equations (7) and (10) have the same \(d\))  
\[d=\sqrt{|\Delta p_{total}|} \tag{10}\]  
The link failure time can be deduced as follows:  
\[t=\frac{-c\pm\sqrt{c^{2}-ab}}{a}, \tag{11}\]  
where \(a\), \(b\), and \(c\) are:  
\[a= |\Delta v|^{2} \tag{12}\]  
\[b= \Big{|}\Delta p\Big{|}_{2}-d^{2} \tag{13}\]  
\[c=\sum_{k\in\{x,y,z\}}\Delta p_{k}\times\Delta v_{k} \tag{14}\]  
If the two values of \(t\) in Equation (11) exhibit negative values, as depicted in Status 3 in Figure 8, or if both values are positive, as illustrated in Status 1, it indicates that the connection between \(i\) and \(j\) may have been valid in the past or may be valid in the future, but is presently disabled and the \(Time_{LinkEpiry}\) is set to 0. In Status 2, the failure time of the current link (\(Time_{LinkEpiry}\)) is set to the positive \(t\) in Equation (11). The concept of node topology change degree, as introduced in [31], is a novel movement characteristic that integrates the relative position, relative movement direction, and relative rate of node movement to quantify the topology change between nodes. At moment \(t\), the position coordinates of node \(i\) are denoted as \(location_{i}^{t}=(x_{i}^{t},y_{i}^{t},z_{i}^{t})\), while the position coordinates of node \(j\) are denoted as \(location_{j}^{t}=(x_{j}^{t},y_{j}^{t},z_{j}^{t})\).