# Advanced Cooperative Formation Control in Variable-Sweep Wing UAVs via the MADDPG-VSC Algorithm
## 3 Cooperative Control of Variable-Sweep Wing UAV via MADDPG
### Structure and Training Process of the MADDPG-VSC Algorithm
#### 3.3.2 Training Process  
During the training process of the MADDPG-VSC algorithm, agents continuously optimize their control strategies through interaction with the environment and other agents. The following steps outline the key components of the training process:  
1. Initialization. 2. The critic network parameters \(\theta^{Q}\) and actor network parameters \(\theta^{\mu}\) for each agent are initialized. Additionally, the target network parameters \(\theta^{Q^{\prime}}\) and \(\theta^{\mu^{\prime}}\) are initialized, and the replay buffer U is set up for each agent. Figure 6: Structure of the MADDPG–VSC algorithm model. Policy selection. At each time step \(t\), agents select an action \(a_{t}\) based on its current state \(s_{t}\) using the policy \(\mu(s_{t}|\theta^{\mu})\). To encourage exploration within the continuous action space, stochastic noise \(\xi\) is added to the action, generating \(a^{\prime}=a+\xi\). Experience storage. After executing the action, each agent receives a reward \(r_{t}\) and observes the next state \(s_{t+1}\). The transition \((s_{t},a_{t},r_{t},s_{t+1})\) is stored in the replay buffer U.  
Parameter updates. When the replay buffer U reaches capacity, mini-batches of transitions are sampled for network updates. The target value \(y_{i}=r_{i}+\gamma Q^{\prime}\Big{(}s_{i+1},\mu^{\prime}\Big{(}s_{i+1}\Big{|} \theta^{\mu^{\prime}}\Big{)}|\theta^{Q^{\prime}}\Big{)}\) is computed, and the loss function is minimized to update the critic network parameters \(\theta^{Q}\). The actor network parameters \(\theta^{\mu}\) are updated through policy gradient backpropagation. Target network update. After the critic and actor networks are updated, the target networks are softly updated using the following equations: \(\theta^{\mu^{\prime}}=\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}\) and \(\theta^{Q^{\prime}}=\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}\). Repeat training. These steps are repeated throughout the training process until the specified number of episodes is completed or the algorithm converges, ultimately yielding a stable optimal policy. By providing a clear advantage in dynamic and unpredictable scenarios, the MADDPG-VSC algorithm enhances real-time trajectory planning and obstacle avoidance, where traditional algorithms often struggle. By incorporating multi-agent strategies, experience replay, and target networks, the training process is designed to enhance both stability and performance. As the critic and actor networks converge, the algorithm learns optimal control policies that adjust sweep angles and flight trajectories in response to changing environments. This ensures that UAV formations maintain stability and efficiency, enabling precise coordination and effective task execution, even in complex mission scenarios. Simulation Verification and Performance Analysis of the MADDPG-VSC  
Simulation Environment and Parameter Setting  
The simulation environment effectively tests the MADDPG-VSC algorithm's capabilities in obstacle avoidance and cooperative formation control. The simulation environment is configured within a 1000 m \(\times\) 1000 m area with a 2000 m altitude ceiling, creating a realistic and challenging space for testing these capabilities. Within this environment, three variable-sweep wing UAVs are initially positioned at specific locations and are tasked with navigating towards a stationary target while successfully avoiding seven strategically placed obstacles. This setup rigorously tests the effectiveness of the UAVs' control policies, particularly focusing on optimal sweep angle adjustments and coordinated movements necessary for obstacle avoidance and target acquisition. To evaluate the algorithm's robustness, two simulation scenarios were designed: a simple scenario with a regular quadrilateral flight path and no obstacles, serving as a baseline for basic coordination and trajectory tracking; and a complex scenario with a polygonal flight path, where UAVs must avoid obstacles, including simulated radar detection zones.