## 4 Proposed Algorithm for Multi-Agent Configuration Change
### Improved Multi-Agent Proximal Policy Optimization
#### 4.2.1 Action Space  
This paper considers the continuous action space of the agent, denoted as velocity vector \((\hat{a}_{t}^{n}=(\hat{v}_{t}^{n},\hat{\omega}_{t}^{n})=[v_{x,t}^{i},v_{y,t}^{i },\omega_{x,t}^{i},\omega_{y,t}^{i}]^{T})\). \([v_{x,t}^{i},v_{y,t}^{i}]^{T}\) represents the linear velocity obtained from the APF technique, policy gradient function for the angular velocity \([\omega_{x,t}^{i},\omega_{y,t}^{i}]^{T}\). Furthermore, the output for each agent within the action space signifies the potential information, shown in Figure 7, and is conveyed to the reward function for evaluation in Section 4.4. Firstly, the APF technique is shown as below:  
Let \(U_{ij}(\left\|\mathbf{p}_{ij}\right\|)\) represent a non-negative smooth pairwise artificial potential function associated with the relative distance between agent \((i,j)\). It governs the collective and individual movement within the swarm, which satisfies the following:  
\[\left\{ \begin{array}{ll}U_{ij}(\left\|\mathbf{p}_{ij}\right\|)=\begin{cases}a \ln(\left\|\mathbf{p}_{ij}\right\|)+\dfrac{b}{c(\left\|\mathbf{p}_{ij}\right\|)^{2}}& \left\|\mathbf{p}_{ij}\right\|\leq d_{sen}\\ \\ \par a\ln(r)+\dfrac{b}{c(r)^{2}}&\left\|\mathbf{p}_{ij}\right\|>d_{sen}\\ \\ U_{ab}=\sum\limits_{k=1}^{m}\dfrac{b}{c(\left\|\mathbf{p}_{io}\right\|)^{2}},(b,c )_{ob}\end{cases},\end{array} \right. \tag{13}\]  
where \(i,j=1,2,\ldots,N,i\
eq j\), \(a\), \(b\) and \(c\) are positive constants, \(U_{ij}(\left\|\mathbf{p}_{ij}\right\|)\) is bounded and continuously differentiable over interval \(\
abla[b/(\left\|\mathbf{p}_{ij}\right\|)^{2}]\). \(\
abla[b/(\left\|\mathbf{p}_{ij}\right\|)^{2}]\) describes the gradient negative cube odd function, which is not defined at 0 point. Therefore, \(U_{ij}(\left\|\mathbf{p}_{ij}\right\|)\)  
Figure 6: Illustration of the MAPPO-CTDE framework. would obtain its boundary maximum value \(U_{0}\) when \(\left\|\mathbf{p}_{ij}\right\|\to 0\); minimum \(U_{d}\) when \((i,j)\) satisfies the desired position, that is, \(\left\|\mathbf{p}_{ij}\right\|=d\); the constant \(U_{r}\) when \(\left\|\mathbf{p}_{ij}\right\|>d_{sen}\). \(U_{ob}\) represents the obstacles' repulsion, which could generate a repulsive force solely to prevent collisions. Equation (13) effectively captures the essential forces of attraction and repulsion that exist between agents and between agents and obstacles. \(a\), \(b\) and \(c\) are the positive constants. Thus, the output from the action space is denoted as \(\hat{\mathbf{n}}_{local}^{i}\):  
\[\hat{\mathbf{n}}_{local}^{i} =\underbrace{-\sum_{j=1,j\
eq i}^{N}\
abla_{i}[\lambda_{ij}]U_{ ij}(\left\|\mathbf{\tilde{p}}_{ij}\right\|)+\lambda_{io}U_{io}(\left\|\mathbf{\tilde{p}}_{io} \right\|)}_{f_{i}^{a}} \tag{14}\] \[+\underbrace{C_{0}\sum_{j=1,j\
eq i}^{N}(\mathbf{\tilde{v}}_{ij}+\bm {\tilde{v}}_{id})}_{f_{i}^{a}}+\underbrace{C_{1}\sum_{j=1,j\
eq i}^{N}(\mathbf{ \tilde{\omega}}_{ij}+\mathbf{\tilde{\omega}}_{id})}_{f_{i}^{a^{\prime}}}\,\]  
where \(C_{0}\) and \(C_{1}\) are the constant value, and \(\lambda_{ij}\) and \(\lambda_{io}\) are different spring constants. \(\mathbf{f}_{i}^{a}\) is the gradient-based term, which regulates position consensus by employing local repulsive and attractive forces, ensuring collision avoidance and cohesion maintenance, and \(\mathbf{f}_{i}^{a}\) means the agent's velocity is adjusted by utilizing the weighted average velocity of its neighbors and desired velocity as a reference, same as \(\mathbf{f}_{i}^{a}\). Each agent's potential position is determined by following the steepest gradient toward the geometric point where the sum of potential forces is zero. Agents then adapt and locally track the potential trajectory, effectively integrating trajectory planning with formation control.