## 4 Proposed Algorithm for Multi-Agent Configuration Change
### Improved Multi-Agent Proximal Policy Optimization
#### 4.2.2 Action Selection
Considering the symmetry of \(U_{ij}\) with respect to \(\widetilde{\mathbf{p}}_{ij}\), we have the following:  
\[\frac{\partial U_{i}}{\partial\widetilde{\mathbf{p}}_{ij^{\prime}}}=\frac{ \partial U_{i}}{\partial\widetilde{\mathbf{p}}_{i}}=-\frac{\partial U_{i}}{ \partial\widetilde{\mathbf{p}}_{j^{\prime}}}, \tag{22}\]  
thus, we have  
\[\frac{d}{dt}\left(\frac{1}{2}{\sum}_{i=1,i\in O}^{N}V_{i}\right)={\sum}_{i=1, i\in O}^{N}\left\{\widetilde{v}_{i}^{\mathsf{T}}(\
abla_{p_{i}}U_{i})+C_{2} \widetilde{\mathbf{v}}_{i}^{\mathsf{T}}\widetilde{\mathbf{p}}_{i}\right\}, \tag{23}\]  
where \(U_{i}=\sum_{i\in O}^{N}U_{ij^{\prime}}(\Big{\|}\widetilde{\mathbf{p}}_{ij^{\prime }}\Big{\|})\), and \(j^{\prime}\) means other agents and obstacles. Then, the derivative to time \(t\) for \(Z(t)\) is obtained as follows:  
\[\dot{Z}(t)={\sum}_{i=1,i\in O}^{N}\left(\
abla_{p_{i}}U_{i}\right)+C_{2} \widetilde{\mathbf{v}}_{i}^{\mathsf{T}}\widetilde{\mathbf{p}}_{i}+\widetilde{\mathbf{v}}_{ i}^{\mathsf{T}}\widetilde{\mathbf{v}}_{i}, \tag{24}\]  
Thus, via substituting (27) to (32), we might have the following:  
\[\dot{Z}(t)={\sum}_{i=1}^{N}\widetilde{\mathbf{v}}_{i}^{\mathsf{T}}{\sum}_{j=1,j \
eq i}^{N}C_{1}a_{ij}(\mathbf{v}_{ij}-\mathbf{v}_{ip})+{\sum}_{i=1}^{N}C_{3} \widetilde{\mathbf{v}}_{i}^{\mathsf{T}}\dot{\widetilde{\mathbf{v}}}_{i}, \tag{25}\]  
and Equation (31) can be expressed in matrix form as follows:  
\[\dot{Z}(t)=-\widetilde{\mathbf{v}}^{\mathsf{T}}[(C_{1}Y(t)+C_{3}\Omega(t))\otimes I _{n}]\widetilde{\mathbf{v}}, \tag{26}\]  
where \(\Omega(t)\) is the stress matrix, \(Y(t)=diag[\gamma_{1},\gamma_{2},\ldots,\gamma_{N}]\), \(\widetilde{\mathbf{v}}=(\widetilde{\mathbf{v}}_{1}^{\mathsf{T}},\widetilde{\mathbf{v}}_{ 2}^{\mathsf{T}},\ldots,\widetilde{\mathbf{v}}_{N}^{\mathsf{T}})^{T}\). Apparently, \(\Omega(t)\) and \(Y(t)\) are positive semi-definite, and these two factors would explain that \(Z(t\geq 0)\) is a non-increasing function with \(\dot{Z}(t)\leq 0\). Thus, the level set \(\psi=\left\{\left|\widetilde{\mathbf{p}}^{T},\widetilde{\mathbf{v}}^{\mathsf{T}} \right|^{T}\in R^{2 nN}:Z(t)\leq Z(0),t\geq 0\right\}\) can be seen as a positively invariant set. Then, it is supposed that \(Z(0)<U_{r}\), and the equation can possibly meet \(U_{i}(\Big{\|}\widetilde{\mathbf{p}}_{ij^{\prime}}\Big{\|})\leq Z(t)\leq Z(0)<U_{r}\) (\(i\in O\), \(t\geq 0\)) in each edge \([e_{ij}]\). Therefore, the connectivity of graph \(G\) of all agents \(\mathbf{p}_{i}\) for swarm is shown within \(t\geq 0\) and \(i\in O\); Thus, the Lemma 1 has been proved. \(\square\)  
**Remark 3**. : _Despite the similarities between certain terms in Equation (14) and existing protocols, the linear speed of the formation guided by Equation (14) would be distinctively unique. This is due to the presence of the affine-based trajectory of each agent that can serve as an evaluation index against the local trajectory._  
**Remark 4**. : _Although the paper does not specifically analyze the impact of acceleration and turning rate as inputs, it is important to note that these two factors are influenced by the agents themselves. Therefore, assuming that the initial distance and interaction range between agents meet the requiredrange, \(Z(0)<U_{r}\) and \(Z(0)<U_{0}\) can effectively avoid collisions within the agent formation and ensure overall stability._