# Advanced Cooperative Formation Control in Variable-Sweep Wing UAVs via the MADDPG-VSC Algorithm
## 3 Cooperative Control of Variable-Sweep Wing UAV via MADDPG
### Optimization of the MADDPG-VSC Algorithm for Control Design
#### 3.2.1 Reward Function Design  
The MADDPG-VSC algorithm's reward function balances multiple objectives, including sweep angle adjustment, formation maintenance, energy efficiency, and obstacle avoidance. Each component contributes to optimal UAV performance in dynamic environments. 1. [label=(0)]
2. Sweep angle reward \(r_{\Lambda}\)  
This reward is given when the algorithm selects the sweep angle \(\Lambda\) based on the current state \(s\). It evaluates how closely the adjusted sweep angle matches the optimal sweep angle \(\Lambda_{\text{opt}}\), thus quantifying the effectiveness of the adjustment in specific flight states. The reward function optimizes sweep angle adjustments for better aerodynamic efficiency and UAV formation stability. Typically, the reward \(r_{\Lambda}^{+}\) is set to a positive value, such as 1 or 10, depending on the importance of maintaining optimal sweep angles. The parameter \(\beta_{\Lambda}\) is set to 0.1, allowing for a 10% deviation from the optimal sweep angle to ensure stability while optimizing performance. \[r_{\Lambda}= \begin{cases}r_{\Lambda}^{+}&\text{if}|\Lambda-\Lambda_{\text{opt }}|\leq\beta_{\Lambda}\\ 0&\text{otherwise}\end{cases}  \tag{15}\]
3. Formation reward \(r_{\text{formation}}\)  
Assuming that three UAVs form an equilateral triangle formation, the virtual center point \(C\) is the geometric center of the three UAVs and follows the designated flight trajectory. The formation reward is based on the deviation of each UAV from the center point \(C\), with smaller deviations indicating a more stable formation and thus, yielding a higher reward. This calculation ensures that the formation remains tight and stable, which is critical for maintaining formation performance [29]. The value of \(\alpha_{\text{formation}}\), which influences the importance of formation maintenance, is typically set to 10. This value balances the need for close formation with other mission objectives, such as energy efficiency and obstacle avoidance. The formation reward is then calculated as  
\[r_{\text{formation}}=\alpha_{\text{formation}}\cdot\left(1-\frac{d_{\text{avg }}}{d_{\text{max}}}\right) \tag{16}\]  
where \(d_{\text{avg}}\) represents the average deviation distance of the N UAVs from the center point \(C\), calculated as \(d_{\text{avg}}=\frac{1}{\text{N}}\sum\limits_{i=1}^{N}\left\|p_{i}-p_{C}\right\|\). Here, \(p_{i}\) represents the position of each UAV, and \(p_{C}\) represents the position of the center point \(C\). The term \(d_{\text{max}}\) denotes the maximum allowable deviation distance, beyond which the formation is considered unstable. Further, the relative distance \(d\) between UAVs is defined as:  
\[d=\sqrt{\left(x_{i+1}-x_{i}\right)^{2}+\left(y_{i+1}-y_{i}\right)^{2}+\left(z _{i+1}-z_{i}\right)^{2}} \tag{17}\]where \((x_{i},y_{i},z_{i})\) and \((x_{i+1},y_{i+1},z_{i+1})\) represent the positions of two UAVs in the formation. Additionally, the relative distance \(d\) between UAVs is subject to the constraint \(d\leq d_{\text{max}}\), ensuring that the formation remains stable. If the relative distance \(d\) between UAVs exceeds \(d_{\text{max}}\), the formation is considered unstable, and the reward decreases accordingly. Energy consumption penalty \(r_{\text{energy}}\)  
This penalty is incurred for unnecessary energy consumption, optimizing UAV endurance by minimizing energy wastage through thrust and flight trajectory optimization. The typical setting for \(\alpha_{\text{energy}}\), which discourages inefficient energy use, is 5, promoting a constant and efficient flight velocity. The energy consumption penalty is approximated by using the velocity \(\Delta v\) of the UAV, which is a proxy for energy usage. \(|\Delta v|\) can be defined as  
\[|\Delta v|=|v_{t}-v_{t-1}| \tag{18}\]  
where \(v_{t}\) and \(v_{t-1}\) represent the magnitude of the UAV's velocity vector at the current time step \(t\) and the previous time step \(t-1\), respectively.