# Q-Learning Empowered Highly Dynamic and Latency-Aware Routing Algorithm for Flying Ad-Hoc Networks
## 4 QEMLR Routing Protocol
### Routing Maintenance
The velocity vector of node \(i\) at moment \(t\) is represented by \(\overset{\rightarrow}{v_{i}^{t}}\), and that of node \(j\) by \(\overset{\rightarrow}{v_{j}^{t}}\). The rate of node \(i\) at moment \(t\) is indicated by \(\left|\overset{\rightarrow}{v_{i}^{t}}\right|\), and that of node \(j\) by \(\left|\overset{\rightarrow}{v_{j}^{t}}\right|\). The distance between node \(i\) and node \(j\) at moment \(t\) is expressed as \(d_{i,j}^{t}=\sqrt{\left(x_{j}^{t}-x_{i}^{t}\right)^{2}+\left(y_{j}^{t}-y_{j}^ {t}\right)^{2}+\left(z_{j}^{t}-z_{i}^{t}\right)^{2}}\). The angle of the direction of motion between nodes \(i\) and \(j\) is then denoted by \(\theta_{i,j}^{t}=\arccos\left(\overset{\rightarrow}{v_{i}^{t}}\cdot\overset{ \rightarrow}{v_{j}^{t}}/\left|\overset{\rightarrow}{v_{i}^{t}}\right|\cdot \left|\overset{\rightarrow}{v_{j}^{t}}\right|\right)\). Finally, the relative rate of motion between nodes \(i\) and \(j\) at time \(t\) is denoted as \(relv_{i,j}^{t}=\left|\left|\overset{\rightarrow}{v_{j}^{t}}\right|-\left| \overset{\rightarrow}{v_{i}^{t}}\right|\right|\). (1) Degree of Distance Change  
The degree of distance change between node \(i\) and node \(j\) is denoted as \(Dis\_vari\), which represents the difference in distance between the two nodes at time \(t+T\) compared to the distance at time \(t\), taking into account the amount of change that occurred during this time interval. The amount of change in distance between the two nodes after \(T\) is denoted as \(\left|d_{i,j}^{t+T}-d_{i,j}^{t}\right|\) and is calculated using Equation (15). Figure 7:0  
Figure 7:1 Link failure time. The red arrows represent invalid communication ranges while the green arrows represent valid communication ranges. (2) Degree of Directional Change  
The degree of directional change between node \(i\) and node \(j\) is \(Dis\_vari\). This parameter reflects the degree of change in the direction of motion of the two nodes in time \(T\) and is calculated in Equation (16). Figure 7:2  
(3) Degree of Relative Rate Change  
The degree of relative rate change between the two nodes is represented by \(Velo\_vari\), which is defined as the ratio of the relative rate of the two nodes at time \(t+T\) to that at time \(t\). This parameter captures the variation in the rate of the two nodes over the time period \(T\), as expressed in Equation (17). Figure 7:3  
degree of topology change in two nodes is the topology change of two adjacent neighbors, \(i\) and \(j\) after experiencing time \(T\). See the following Equation (18). Figure 7:4  
The parameter \(TCD_{i,j}(t,t+T)\) is defined as a linear combination of the three aforementioned degrees of variation. The coefficients \(w1\), \(w2\), and \(w3\) are weight coefficients in the equation, and their values range from 0 to 1. These coefficients can be set based on the relative importance of the three factors in the topological change. In this paper, we assume that the three factors have equal importance and therefore set \(w1=w2=w3\). The speed variation, distance variation, and direction variation between nodes can be used to evaluate the network topology change. However, the mobility of neighboring nodes, represented by \(MF_{i}\), is an important factor in measuring the degree of node network topology. It is worth noting that \(MF_{i}\) alone may not be sufficient to completely evaluate the node network state. Therefore, a calculation factor that combines the above factors can be designed to assess the degree of node topology change. Once the node topology parameter \(TCD_{i,j}\) has been calculated using the formula in Equation (18), it is incorporated into the discount factor used for Q-value calculation during route establishment, as shown in Equation (19). Figure 7:5  
Subsequently, three metrics in the current node neighbor table are updated: \(d_{i,j}^{t}\theta_{i,j}^{t}\) and \(relv_{i,j}^{t}\). The effectiveness of various discount factors proposed in this paper is compared and analyzed in the performance test of the routing protocol.