# Q-Learning Empowered Highly Dynamic and Latency-Aware Routing Algorithm for Flying Ad-Hoc Networks
## 4 QEMLR Routing Protocol
### Routing Decision  
The following provides an illustrative example of a 7-node moving ad-hoc network to demonstrate the routing decision process. Figure 9 depicts a simplified illustration of the source node Sr propagating Hello packets.  
Assuming that the source node Sr initiates the transmission of Hello packets to the destination node De, the Q-value is initialized to 1.0 at this stage. Upon receiving the Hello packets, the intermediate node establishes a reverse route based on the information contained in the message. The routing table is then updated, and the Q-value is calculated and added to the Hello message to facilitate multi-hop transmission to node De. At this point, node De is aware of the presence of source node Sr outside the intermediate node area, thereby simplifying the route establishment process.  
Table 4 presents the information contained in the Hello message, which is used to maintain and establish the route. This information includes the originator and destination IP addresses, current and predicted positions, Q-value calculated based on the current routing information, observed real-time delay, sequence number, remaining hop count, and Mobility Factor (MF) of the current node's neighbors.  
When node D receives a Hello packet from node B, it first verifies whether B is registered in its neighbor table. If B is registered, D updates the neighbor table with information about this neighbor. If B is not registered, D needs to register the node in the neighbor table. The updated and registered information includes the time of the last received packet from B, the end-to-end delay on the last established link (De-B), the three-dimensional velocity of node B, the neighbor nodes' aggregation degree of node B, and the estimated failure time of the (De-B) link.  
The source IP address in the Hello message of node B belongs to source node Sr, which sent the Hello packet. The source node Sr passes through a series of intermediate nodes and finally hands the packet to De through node B. At this point, De knows that it can reach the destination Sr beyond an unknown region via B. The address of Sr is recorded as the destination address in the Q table of De. Then, [PERSON] can establish the reverse path in the Q table based on this information. At this time, there is an additional record in the Q table, which is shown in Table 5. When De needs to transmit data packets to Sr, it will find  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{3}{c}{**Packet Architecture (Total Bytes 48)**} \\ \hline
1 & 2 & 3 & 4 \\ \hline  & Originator IP Address & \\ \hline  & Destination IP Address & \\ \hline  & Position Vector3D & \\ \hline  & Predict Vector3D & \\ \hline  & Q value & \\ \hline  & Observed Delay & \\ \hline  & MF (Mobility Factor) & \\ \hline  & Sequence Number & TTL \\ \hline \hline \end{tabular}
\end{table}
Table 4: The structure of the Hello message.  
Figure 9: Hello propagation. Decimal fractions on the blue arrows represent the corresponding Q-value meanwhile red arrows represent optimal route obtained by the routing decision process.  
multiple routes recorded in the Q table and transmit data packets according to the route with the largest Q value.  
Upon establishing the Q-table, Node S is capable of forwarding data packets to Node D through either Node A or Node B. As a result, two records are created in the Q-table of Node D, corresponding to the destination address Sr. Node D selects the next hop for transmission based on the highest Q-value in the Q-table. In the configuration depicted in Figure 9, Node B is selected as the next hop for transmission. The red arrow in Figure 9 represents the routing path of the transmitted data packet from Node D to Sr, based on the reversed path.