## 1 Introduction  
The development of unmanned aerial vehicles (UAVs) has grown substantially over the past decade, and UAVs are extensively used in military, commercial, and civilian applications, such as in emergency communications [1], meteorological surveillance [2], agriculture and forestry monitoring and assessment [3], mineral exploration [4], and task offloading for mobile edge computing [5], among others. In these applications, multiple UAVs form an ad hoc network (UANET), which is a self-organizing, distributed network architecture that is more flexible and scalable than centralized network architectures. The nodes can communicate with others to complete specific tasks efficiently in a UANET. The communication between UAVs is a challenge, however, due to frequent topology changes and link quality fluctuations as a result of the high mobility of UAVs, as well as congestion due to heavy network traffic. To accommodate the characteristics of a UANET, many protocols have improved upon conventional routing protocols. For example, in [6], the authors presented the improved OLSR based on the original optimized link-state routing (OLSR) protocol [7] to control the high overhead. In [8], the authors made use of the ad hoc on-demand distance vector (AODV), and Reynolds' boids were used for connectivity and path maintenance while a packet was being forwarded. These proactive and reactive routing protocols relied on network topology information and needed to establish and maintain a routing table, which is not appropriate for bandwidth-limited and highly dynamic UANETs. Therefore, a classical geolocation-routing protocol (GPSR) [9] was suggested that used geolocation information to assist in routing decisions. The routing protocol based on a geographical location was more suitable for highly dynamic UANETs because it was more scalable and had a lower overhead that only requiring the neighboring location information. The strategy of GPSR greedily selects the next-hop node, which provides the nearest distance to the destination node. It is essentially a shortest-route algorithm. If none of the neighboring nodes are closer to the destination node than itself, the target node forwards the packet by peripheral forwarding, leading to more overhead and delay. The authors of [10; 11; 12] presented a solution to address the routing-void problem. Although conventional routing protocols were simple and easily applied to UANETs, the aforementioned protocols were not intelligent and lacked autonomous routing decisions and adaptability, making them challenging to modify for a changeable UAV network environment with complex tasks. As artificial intelligence has advanced, reinforcement learning (RL) has become an important class of algorithms in artificial intelligence, and applications based on reinforcement learning have achieved effective results in edge computing [13], resource allocation [14], routing decisions [15; 16; 17], and so forth. Reinforcement learning is a methodological framework for learning, predicting, and making decisions. If a problem can be described or transformed into a sequential decision problem, then RL can be used to solve it. The advantage of RL-based routing protocols is that they have a long-term vision, i.e., long-term return, which allows them to select better decisions than conventional routing protocols, which only consider the immediate reward. The authors of [18] proposed a new Q-learning-based routing protocol for flying ad hoc networks, which considered two-hop neighboring nodes and adaptively selected the next-hop node based on the network characteristics, such as location, delay, velocity, and resident energy level.