# Advanced Cooperative Formation Control in Variable-Sweep Wing UAVs via the MADDPG-VSC Algorithm
## 3 Cooperative Control of Variable-Sweep Wing UAV via MADDPG
### Structure and Training Process of the MADDPG-VSC Algorithm
#### 3.4.3 Comparative Analysis of Learning Efficiency and Convergence  
To evaluate the training efficacy of the MADDPG-VSC algorithm, both MADDPG and MADDPG-VSC models were trained and compared. Figure 9 shows the reward curves, illustrating their performance and learning progress. In the complex simulation environment with multiple targets and obstacles, both models stabilized after around 3000 training episodes, indicating effective learning. However, stochastic noise during training caused some variability, seen as oscillations in the reward curves, which is common in reinforcement learning.  
Both algorithms stabilized after 2500 episodes, but the MADDPG-VSC algorithm achieved stability earlier, consistently yielding higher rewards throughout. This indicates a higher learning efficiency in handling tasks. The MADDPG algorithm exhibited significant fluctuations in later stages, particularly after approximately 1500 episodes, where reward values showed a slight decline. This suggests potential overfitting or instability issues during prolonged training. Although the algorithm eventually converged, the fluctuations indicate weaker adaptability and consistency when faced with complex tasks. In contrast, the MADDPG-VSC algorithm not only converged faster in the early training phase but also maintained a more stable reward value, demonstrating its advantages in multi-UAV formation and obstacle avoidance tasks. By incorporating adaptive sweep angle control, this algorithm significantly improved the stability and accuracy of task execution, particularly excelling in static complex environments.  
To evaluate the performance of the MADDPG and MADDPG-VSC algorithms, their Q-value and loss value trends were compared over 3000 training episodes. Figure 10 shows  
Figure 9: Training reward curves for MADDPG, MADDPG–VSC, and SAC algorithms.  
the results, providing insight into the learning progress of both models during multi-UAV formation control and obstacle avoidance tasks. The Q-value reflects the prediction of cumulative rewards, while the loss value indicates the prediction accuracy. Both models were tested under the same complex environment with multiple targets and obstacles.  
During training, both algorithms initially showed a sharp increase in Q-values, indicating effective early learning. However, the MADDPG algorithm experienced a decline in Q-values after around 500 episodes, suggesting potential overfitting or instability. This decline indicates that MADDPG struggled to maintain consistent performance in more complex scenarios. Although it eventually converged, the fluctuations suggest weaker adaptability. In contrast, the MADDPG-VSC algorithm maintained a stable Q-value trend throughout training. Its integration of adaptive sweep angle control helped optimize performance, allowing the algorithm to achieve higher and more stable Q-values. This stability demonstrates its improved efficiency in regards to multi-UAV coordination and obstacle avoidance.  
The loss value trends further support these findings. While both algorithms reduced their loss values over time, the MADDPG algorithm showed more variability, indicating less stable  
Figure 10: Network parameter variations using MADDPG, MADDPG–VSC, and SAC algorithms. (**a**) MADDPG algorithm; (**b**) MADDPG–VSC algorithm; (**c**) SAC algorithm.  
learning. On the other hand, the MADDPG-VSC algorithm exhibited more consistent and lower loss values, reflecting higher prediction accuracy and better learning stability.  
In summary, the MADDPG-VSC algorithm demonstrated greater stability and faster convergence, making it better suited for controlling variable-sweep wing UAVs in complex environments. The improvements in reward stability and loss reduction highlight its ability to handle dynamic and uncertain conditions, such as cooperative formation control and obstacle avoidance.