# Advanced Cooperative Formation Control in Variable-Sweep Wing UAVs via the MADDPG-VSC Algorithm
## 3 Cooperative Control of Variable-Sweep Wing UAV via MADDPG
### Application of MARL in Variable-Sweep Wing UAV  
In complex multi-agent systems, cooperative control is crucial for ensuring that individual agents, such as UAVs, can successfully accomplish their tasks together [18; 19; 20; 21]. However, this process poses several challenges, particularly when managing continuous action spaces in dynamic flight states. The MADDPG-VSC algorithm builds upon the MARL framework, which can be represented as a high-dimensional tuple \(<S,A_{1},A_{2},\ldots,A_{N},R_{1},R_{2},\ldots,R_{N},P_{\text{dyn}},\gamma>\)[22; 23; 24; 25]. In this framework, the state space \(S\) represents the key states of the UAVs, including position, velocity, and sweep angle; the action space \(A\) encompasses control inputs such as thrust, steering angle, and control voltage. The reward function \(R\) evaluates the contribution of each action to the success of the mission; the state transition dynamics \(P_{\text{dyn}}\) is defined as \(P_{\text{dyn}}:S\times A\times S\rightarrow\left[0,\,1\right],\) describing the probability of state changes; and the discount factor \(\gamma\) determines the current value of future rewards [26; 27; 28]. These elements together form the foundation of the MADDPG-VSC algorithm, providing the theoretical support needed for dynamic cooperative control of UAVs in complex environments. The MADDPG-VSC algorithm incorporates state-value and action-value functions, alongside Q-learning and policy gradient mechanisms, to optimize UAV control in real time. Their specific implementations in the MADDPG-VSC algorithm are as follows:  
1. State-value function \(V_{i}(s)\)  
The state-value function evaluates the total future reward that agent \(i\) may receive while in state \(s\). For UAV formation control tasks, predicting the long-term effects of different flight trajectories is crucial. Through the state-value function, the MADDPG-VSC algorithm can assess the UAV's flight state in real-time, enabling the selection of the optimal flight strategy at each moment. The state-value function \(V_{i}(s)\) is expressed as follows: \[V_{i}(s)=E_{i}\Big{[}r_{t+1}^{i}+\gamma V_{i}(s_{t+1})|s_{t}=s\Big{]}\] (11)
2. Action-value function \(Q_{i}(s,a)\)  
The action-value function evaluates the total reward that can be obtained by taking action \(a\) in state \(s\). This allows the MADDPG-VSC algorithm to guide the UAVs in selecting the optimal control strategy in complex environments, ensuring formation stability and mission success. The action-value function \(Q_{i}(s,a)\) is given by  
\[Q_{i}(s,a)=E_{i}\Big{[}r_{t+1}^{i}+\gamma Q_{i}(s_{t+1},a_{t+1})|s_{t}=s,a_{t}=a \Big{]} \tag{12}\]
3. Q-learning update  
The Q-learning update mechanism iteratively updates the Q-values to continuously optimize the agent's strategy, leading to convergence towards an optimal strategy. Through this mechanism, the MADDPG-VSC algorithm can dynamically adjust strategies during flight to adapt to changing environments and task demands. The Q-learning update mechanism is governed by the following equation:  
\[Q_{t+1}(s_{t},a_{t})=Q(s_{t},a_{t})+\eta\big{[}r_{t+1}+\gamma max_{a_{t+1}}Q_ {t+1}(s_{t+a},a_{t+1})-Q(s_{t},a_{t})\big{]} \tag{13}\]
4. Policy gradient update  
In dynamic environments, traditional Q-learning methods may struggle to handle continuous action spaces. To address this, the MADDPG-VSC algorithm employs a policy gradient update mechanism, which optimizes the policy parameters \(\theta^{\mu}\). This mechanism guides UAVs in selecting the optimal sweep angle adjustment strategy, thereby improving overall task success.