## 4 Proposed Algorithm for Multi-Agent Configuration Change
### Reward Function  
In this paper, the reward function serves as a feedback mechanism to guide the agent's learning process and assess the efficacy of its actions, mainly for its generated potential position. The purpose of this process is to settle the local replanning strategy for each agent, which evaluates the generated potential state based on the environment. This article utilizes a segmented cost function to formulate the reward function, a choice motivated by its potential to accelerate the convergence of network updates, thereby enhancing the agents' learning efficiency in acquiring the optimal strategy. The reward function is designed as \(R_{t}=-(J_{i}(\hat{X}_{i}))\), whose aim is to use the four-layer cost function to reflect the potential state \((\hat{X}_{i})\) at the time step \(t\). Thus, the reward of each potential state \((\hat{X}_{i})\) would be transferred by the opposite of the cost function below, and the learning process can be explicitly guided so that the multi-agent system learns in the direction of cost reduction \(-J_{i}(\hat{X}_{i})\). The potential motion estimation of the agents is solved as minimization of the cost function \(J_{i}(\hat{X}_{i})\), subject to sets of constraints \(C_{inputs}\), \(C_{agent}\). The input constraints are as follows:  
\[C_{input}=\left\{ \begin{array}{l}v_{min}\leq\|\hat{\mathbf{v}}_{i}\|\leq v_{max} \\ |\hat{\mathbf{\omega}}_{i}|\leq v_{t}/R_{min}\end{array} \right., \tag{28}\]  
where the limits of \(v_{min}\) and \(v_{max}\) are given by the motion capabilities of the UAVs in the formation. The angular velocity \(\mathbf{\omega}_{k}\) is limited by the minimal allowed turning radius \(R_{min}\)of the formation that ensures smooth movement of all followers in the desired shape. \(R_{s}\) and \(R_{a}\) denote the safety distance of the agent and the obstacle. \[C_{agent}=\left\{ \begin{array}{l}\left\|\hat{p}_{j}(t)-\hat{p}_{i}(t)\right\| \geq R_{s}\\ \left\|\hat{p}_{o}(t)-\hat{p}_{i}(t)\right\|\geq 2R_{a}\end{array} \right., \tag{29}\]  
The cost function for the agents' potential state can provide feedback continuously during the maneuver. Thus, the four-layer \(J_{i}(\hat{X}_{i})\) is designed as below:  
\[ \begin{array}{ll}J_{i}(\hat{X}_{i})&=\sum\limits_{k=1,k\in N}\left\|\hat{p}_ {i}(k)-p_{d}^{*}(k)\right\|^{2}\\ &+\sigma_{1}\left(\min\left\{0,\frac{dist(\hat{X}_{i},\hat{X}_{j})-R_{s}}{ dist(\hat{X}_{i},\hat{X}_{j})-R_{a}}\right\}^{2}\right)+\sigma_{2}\left(\min \left\{0,\frac{dist_{obs}(\hat{X}_{i})-R_{s}}{dist_{obs}(\hat{X}_{i})-R_{a}} \right\}^{2}\right)\\ &+\sum\limits_{j\in N^{+},j\
eq i}\left[(\hat{v}_{i}-\hat{v}_{j})^{2}+(\hat{ \omega}_{i}-\hat{\omega}_{j})^{2}+(\hat{v}_{i}-\vec{v}_{d})^{2}\right]\\ &+\sum\limits_{i=1}^{N}pen_{i}\end{array} , \tag{30}\]  
where \(\sigma_{1}\) and \(\sigma_{2}\) represents the adjusting factor. Equation (30) mainly evaluates four aspects: the position distance to the reference value and the obstacles among agents in formation, the distance among the agents in reference sequences, the velocity distance in each linear and angular way and rotation maneuver penalty. This four-layer cost function totally considers displacement, distance and bearing direction of the agent in each time step, especially the rotation maneuver penalty to judge the turning maneuver during the flight. The details of Equation (30) are as follows. The first layer penalizes deviations between the reference \(p_{d}^{*}(k)\) and the current potential position \(\hat{p}_{i}(k)\) through the Euclidean distance \(\left\|\hat{p}_{i}(k)-p_{d}^{*}(k)\right\|^{2}\), which would show the basic relevant position at the current moment for all the agents \(k\in[1,N]\). The second layer represents the avoidance function between the potential state \(\hat{X}_{i}\) and its neighbors \(\hat{X}_{j}\), treating other agents as dynamic obstacles \(dist(\hat{X}_{i},\hat{X}_{j})\), while the other component indicates the distance to the nearest obstacle \(dist_{obs}(\hat{X}_{i})\).