## 2 Related Work  
Routing protocols for UANETs have been widely investigated in review articles [22; 23; 24]. The methods can be roughly divided into two categories. Conventional routing protocols and reinforcement-learning-based routing protocols. Conventional routing protocols include topology-based and geographical-location-based routing protocols. An example of topology-based routing protocols is the reactive routing protocol. Many routing protocols have been improved based on the original AODV. For example, the authors of [25] optimized the AODV protocol by considering resident energy and relative movement. The authors of [26] proposed an energy-aware, cross-layer AODV routing protocol to reduce link breakage and improve network performance. In [27], the authors proposed an HN-AODV routing algorithm. When the sequence number of the route request or route reply packet was equal to the sequence number of the recorded route request or route reply packet stored in a node, the number of neighbors and the hop count along the path were considered to decrease the end-to-end (E2E) delay during the path discovery process. Another example of topology-based routing protocols is proactive routing, which is a table-driven routing similar to reactive routing. The most typical proactive routing protocol is OLSR. Many improvements of OLSR have considered the characteristics of UANETs. In [28], the performance of the OLSR protocol was improved by adjusting the parameters and decreasing the holding times. In [29], the authors presented the novel and effective OLSR+, which calculated the link lifetime according to distance, link quality, relative velocity, and movement direction, and selected multipoint relays based on fuzzy logic based on residual energy, link lifetime, and the number of neighbors. Furthermore, the included route energy and route lifetime into the topology control message. In [30], the authors used multibeam directional antenna (MBDA) technology to forward a packet in different directions without radio frequency interference, which ensured that OLSR had a low probability of detection. In addition, a multipath-enhanced OLSR based on MADA was proposed, and a social network concept was applied to select the multipoint relays with small broadcast overheads. Geographical-based routing protocols do not need routing tables; instead, they require the maintenance of neighboring tables, which makes them suitable for highly dynamic networks. Since GPSR is capable of mitigating routing voids, the authors of [31] proposed a recovery strategy to resolve this problem. The recovery strategy involved three methods. In the first, when a routing void occurred, the current forwarding node waited and attempted to resend the packet. The second method forwarded packets to the farthest neighboring node. The third method forwarded packets to the node with the fastest movements, which was determined by predicting the distance information. The novel routing protocol in [32] used the location estimation of the destination to deliver packets in an appropriate direction while considering congestion information to forward a packet for delay-tolerant and energy-limited UAV networks. The authors of [33] presented an adaptive \"hello\" mechanism and a greedy forwarding mechanism according to the relative motion of the nodes. Overall, the conventional routing protocols have been relatively simple and easy to implement, but they lack the ability to self-adapt or to employ intelligent and autonomous decision-making. In recent years, multiagent autonomous, intelligent decision-making systems based on artificial intelligence have become a research hot spot. Many researchers have utilized RL to solve routing decision problems because RL-based routing protocols can adapt to highly dynamic topology changes in a UANET [34]. In [35], the authors proposed the Q-Fuzzy approach. Fuzzy logic was used to select the next-hop node by considering link-relatedparameters.