## 6 AI-Based Approaches for Autonomous UAV Swarm Systems
### AI-Based Control Algorithm
#### 6.3.2 DRL Based Control Approaches  
DRL is a branch of deep learning and reinforcement learning, in which UAVs learn to optimize their behaviors by receiving rewards and penalties from the environment after each action. The applications of DRL in UAV formation control represent significant advancements in autonomous UAV swarms. DRL allows UAVs to learn optimal control policies through interactions with the environment, which is more efficient and robust compared to traditional methods. Based on DRL techniques, UAVs can autonomously coordinate to maintain formation patterns while ensuring collision avoidance and adapting to dynamic environments. Some common DRL models used in UAV swarm control are presented as follows. Deep Q-Networks (DQN) is an algorithm in deep reinforcement learning. It combines Q-learning and deep neural networks to solve decision-making problems in complex environments. This method was introduced by Google DeepMind, and it has achieved significant success in training intelligent agents to perform tasks autonomously without human supervision. DQN is based on the principle of Q-learning, in which the agent learns to optimize actions by maximizing the value of the received reward. Instead of using a Q-table as in traditional Q-learning, DQN uses a deep neural network to approximate the Q-value function. This neural network takes the state of the environment as input and output Q values corresponding to each possible action [197; 198]. Deep Q-Networks has been widely applied in UAV swarm control to optimize operational performance, save energy, and enhance coordination between UAVs in complex environments. One of the major challenges in UAV swarm control is formation switching due to the complex system structure and large computational requirements. To address this problem, a UAV formation switching method based on an improved Deep Q-Networks (DQN) algorithm has been proposed. The UAV formation switching problem is modeled as a Markov decision process (MDP), in which the DQN is used to determine the optimal actions for each UAV during the formation change. Next, an optimal dynamic target allocation algorithm is applied to optimize the positions of the UAVs in the new formation. It improves the formation switching performance and accelerates the convergence speed of the DQN [199]. DRL algorithms, including DQN are used to find energy-saving flight strategies through the learning mechanism from experience. DQN is applied in problems such as flight path planning, trajectory optimization, and obstacle avoidance to optimize energy consumption while ensuring mission completion [200]. A model combining DQN with a value decomposition network and a convolutional block attention module ensures communication quality with limited spectrum and power resources in UAV swarm systems. This method allows UAVs to optimize spectrum utilization in UAV-to-UAV (U2U) and UAV-to-Base Station (U2B) links, improving data transmission rates and increasing the probability of transmission success [201]. DQN is also used to optimize UAV flight paths in wireless sensor networks. When IoT devices use reflective transmission technology with limited transmission range, UAVs can assist in data collection by approaching and triggering sensors. The Prioritized DQN algorithm is proposed to reduce the total flight time of UAVs and to optimize energy and data collection efficiency compared to other DRL algorithms such as MADDPG, DDPG, and DQN [202]. A DQN model incorporating an attention mechanism was proposed to optimize the UAV control system, improve data transmission efficiency, and solve the scalability and mobility issues of UAVs. The experimental results show that this method helps to optimize the data transmission rate and ensure a reasonable throughput level in mobile networks using UAVs [203]. Momentum Policy Gradient (MPG) is an approach in reinforcement learning. It aims to improve the performance of gradient policy algorithms by reducing variance and increasing convergence speed.