## 6 AI-Based Approaches for Autonomous UAV Swarm Systems
### AI-Based Control Algorithm
#### 6.3.2 DRL Based Control Approaches
Traditional gradient policy methods often suffer from large fluctuations in gradient estimation, leading to unstable performance and requiring a large number of samples to achieve the desired accuracy [204; 205]. This algorithm is applied to improve the performance of UAV swarm control and multi-agent systems. In [206], an MPG algorithm was proposed to solve the UAV control problem with tasks such as target tracking, obstacle avoidance, and formation maintenance. The system is designed according to a two-part control model, including a sensing module and a control module. It helps to train a DRL agent without the need for a complex physical model. The experimental results show that MPG is capable of tracking the movement of the leading UAV stably, and it can be extended to control a UAV formation. In [207], a Decentralized Policy Gradient with Momentum (DPGM) algorithm was proposed to improve the performance of finding optimal policies for multi-agent systems. DPGM applies a policy update mechanism using gradients with momentum to increase the convergence speed while ensuring dispersion in a reinforcement learning environment. This algorithm achieves a convergence speed of O(1/T). It is on par with the most advanced distributed policy gradient methods today. In [208], a Distributed Adaptive Policy Gradient with Momentum (IS-DAPGM) algorithm was developed based on the Adam update mechanism and importance sampling technique to improve the UAV swarm control performance. Deep Deterministic Policy Gradient (DDPG) was developed to overcome the limitations of traditional Q-learning-based policy algorithms. They only work well with discrete action spaces [209]. By combining Deterministic Policy with deep neural networks, which can handle continuous state and action spaces, DDPG is capable of optimizing policies efficiently without sampling actions from a probability distribution like stochastic policy algorithms. DDPG uses experience recurrence to improve training efficiency and also applies an objective network to stabilize the weight update process, avoiding oscillation or divergence [209]. DDPG and its variants are used to optimize UAV control in complex missions. Multi-agent deep deterministic policy gradient (MADDPG) is an improved variant of the DDPG algorithm. This method is effective in situations where agents must learn to cooperate, compete, or coexist in a shared environment. Several improved versions of MADDPG aim to address challenges such as learning efficiency, collision avoidance, and enhancing cooperative behavior in UAV clusters, including GPR-MADDPG and COM-MADDPG. The GPR-MADDPG method combines Gaussian Process Regression (GPR) and Multi-Agent DDPG to optimize UAV control to track moving vehicles [210]. This method ensures that UAVs maintain reasonable overlapping observation ranges, minimize tracking errors, and increase surveillance performance in real and simulated environments with complex dynamics. In the target encirclement task with another variant, COM-MADDPG is used to improve the coordination between UAVs to encircle the target more effectively [211]. Table 10 summarizes the prominent control methods based on deep reinforcement learning in UAV swarms. In the above summary, it can be seen that the research on UAV swarm control focuses on improving and combining DRL-based algorithms to solve problems such as navigation in complex environments, tracking and avoiding obstacles, optimizing energy, or optimizing communication resources. The combination of basic RL algorithms with advanced techniques improves performance and adaptability in UAV swarm applications. However, there exist some limitations such as high implementation costs and complex computational requirements, especially with GPR [210] or multi-agent [211] integration methods. In ad dition, the ability to expand to large scales also needs to be verified to meet the practical requirements of the application.