## 6 AI-Based Approaches for Autonomous UAV Swarm Systems
### AI-Based Routing Protocol
#### 6.2.2 Self-Adaptive Learning-Based Routing Protocols  
Most learning-based routing protocols use reinforcement learning (RL) to make routing decisions. RL routing protocols allow agents to discover the best route between two points in the network even without prior knowledge of those locations. Standard RL algorithms are developed to find optimal routing strategies in small-scale UAVs. Through sequential actions by interacting with the dynamic environment and using previous experiences, RL agents can make more optimal decisions to maximize rewards. Several studies have introduced RL technology into location-based protocols to improve the performance of routing protocols for UAV networks such as end-to-end delay, packet transmission rate, and routing cost. Q-learning (QL) is a model-free off-policy value-based RL, suitable for performing multi-objective optimization in resource-constrained FANETs. QL evaluates the expected value of the cumulative reward and obtains the instantaneous optimal policy based on past experience, even in an unknown environment without a central controller. QL-based multi-objective optimization routing protocol (QMR): in [151], the authors propose a novel QL-based multi-objective optimization routing protocol for UAV networks with the advantages of low latency and low energy consumption. The QL learning parameters can be adaptively adjusted based on the network dynamics. In addition, the authors propose a novel discovery and exploitation mechanism for some unexplored potential optimal routing paths while exploiting the acquired knowledge. The proposed method re-estimates the neighbor relationships during the routing decision process to select a more reliable next hop. The simulation results show that the proposed method can provide higher packet arrival rates, lower latency, and lower energy consumption than the previous QL-based routing method. Improved QL-based routing protocol for UAV networks: in [152], the authors proposed an improved routing protocol based on QL to achieve reliable and real-time communication in highly dynamic UAV networks. The proposed scheme combines the techniques used in two different RL-based routing protocols, QMR and Q-Noise+, into a new protocol. By combining and adapting the components of these baseline protocols, the new protocol architecture is a better fit for the dynamic behavior of UAV networks. The protocol considers a finite number of final sets to update the Q values. Additionally, it also uses channel conditions to parameterize the enhanced protocol. The authors perform simulations to compare the performance of Q-FANET with other existing approaches, namely Q-Geo, Q-Noise+, and QMR, using the WSNet simulator. The results demonstrate that the protocol outperforms standard protocols in terms of reduced latency, along with improved packet delivery ratio. Adaptive and Reliable Routing Protocol with Deep Reinforcement Learning (ARdeep) was proposed by the authors in [153]. The algorithm decides the routing with the Markov decision process model to automatically describe the network variations. The link state is considered while making routing decisions to infer the network environment better. Additionally, the information about the packet error rate, expected connection time of the link, remaining energy of nodes, and distance between node and destination are the factors evaluated to make routing decisions. They accurately infer the network environment and make more appropriate forwarding decisions. The simulation results show that ARdeep performs better than existing QGeo routing protocols and conventional GPSR. QL-adaptive UAV-assisted geographic routing (QAGR) [141]: the authors proposed a QAGR algorithm based on the QL-adaptive UAV-assisted geographic routing method. The routing scheme is divided into two components. For the airborne component, the global routing path is computed using a fuzzy logic algorithm and Depth-First Search (DFS) using the information collected by the UAV and forwarded to the requesting vehicle on the ground.