## 4 Proposed Algorithm for Multi-Agent Configuration Change
### Improved Multi-Agent Proximal Policy Optimization
#### 4.2.2 Action Selection  
The main problem would be the selection of the potential action vector \(\hat{\mathbf{a}}\); the components within action space \(\hat{\mathbf{a}}_{t}^{n}=(\hat{\mathbf{v}}_{t}^{n},\hat{\mathbf{\omega}}_{t}^{n})\) are all defined as continuous variables with a certain range. At each time step \(T_{i}\), the action of an agent is expressed as follows:  
\[\hat{\mathbf{a}}_{t}^{n}=(\hat{\mathbf{v}}_{t}^{n},\hat{\mathbf{\omega}}_{t}^{n}),(\hat\mathbf{{a}}_{t}^{n}\in\mathcal{A},\forall n\in\mathcal{N}), \tag{15}\]  
\[\hat{\mathbf{a}}_{t}^{n}=(\mathbf{a}_{t}^{1},\mathbf{a}_{t}^{2},\ldots,\mathbf{a}_{t}^{N}), \tag{16}\]  
\[\left\{ \begin{array}{l}v_{\min}\leq||\hat{\mathbf{v}}_{t}^{n}||\leq v_{\max}\\ u_{\min}\leq\hat{\mathbf{u}}_{t}^{n}\leq u_{\max}\\ |\hat{\mathbf{\omega}}_{t}|\leq\hat{\mathbf{v}}_{t}/R_{min}\end{array} \right., \tag{17}\]  
* Action of linear velocity \(\hat{\mathbf{v}}_{t}^{n}\): determined by the APF function from action space \(\hat{\mathbf{u}}_{local}^{i}\):
* Action of angular velocity \(\hat{\mathbf{\omega}}_{t}^{n}\): obtained from the policy gradient function \(\
abla\pi_{\theta}\) in Actor module. Supposing a disordered swarm of \(N\) agents is steered by protocol (14) in linear velocity, if the interaction between agents within the graph \((G,p)\) is undirected and connected for the whole time, \(Z_{0}\) is denoted as the sum of total artificial potential; the lemma and proof are as follows. **Lemma 1**. : _The graph \((G,p)\) is connected mutually if_  
\[Z_{0}<U_{r}\text{ for all }t\geq 0. \tag{18}\]  
Proof. : It is assumed that \(M(1\leq M\leq N)\) agents function as leaders. Thus, the kinematic error of the leaders in the system is defined as follows:  
\[\left\{ \begin{array}{l}\widetilde{\mathbf{p}}_{i}=\mathbf{p}_{i}-\mathbf{p}_{d}\\ \widetilde{\mathbf{v}}_{i}=\mathbf{v}_{i}-\mathbf{v}_{d}\end{array} \right.,i=1,2,\ldots,M, \tag{19}\]  
Figure 8: Illustration for potential position (red dotted circle) of one agent on local level. The forces in red dotted line among agents and obstacle. Potential gradient in black dotted line. The velocity direction in blue dotted line. based on the fact that the global trajectory is twice differentiable, it satisfies \(\dot{\mathbf{p}}_{d}=\mathbf{v}_{d}\), \(\dot{\mathbf{v}}_{d}=\mathbf{u}_{d}\) and we have \(\widetilde{\mathbf{p}}_{ij}=\widetilde{\mathbf{p}}_{i}-\widetilde{\mathbf{p}}_{j^{\prime}} \dot{\widetilde{\mathbf{p}}}=\widetilde{\mathbf{v}}_{i}\), \(\dot{\widetilde{\mathbf{v}}}_{i}=\widetilde{\mathbf{u}}_{i}\). Given that \(Z_{t}\) represents the summation of the total relative kinetic energy of an agent at time \(t\), it can be deduced that  
\[Z(t)=1/2{\sum}_{i=1,i\in O}^{N}\left\{[U_{i}(\Big{|}\Big{|}\widetilde{\mathbf{p}} _{ij^{\prime}}\Big{|}\Big{|})]+C_{2}(\mathbf{p}_{i}-\mathbf{p}_{d})^{\mathsf{T}}(\mathbf{ p}_{i}-\mathbf{p}_{d})+C_{3}(v_{i}-v_{d})^{\mathsf{T}}(v_{i}-v_{d})\right\}, \tag{20}\]  
let \(V_{i}=\sum_{i=1,j\in O}^{N}\left\{\widetilde{\mathbf{p}}_{i}^{\mathsf{T}} \widetilde{\mathbf{p}}_{i}+U(\|\widetilde{\mathbf{p}}_{i}\|)\right\}\) be the total artificial of agent \(i\). Then, in the error system, \(Z(t)\) can be rewritten as follows:  
\[Z(t)=\frac{1}{2}{\sum}_{i=1,i\in O}^{N}\left[V_{i}+\widetilde{\mathbf{v}}_{i}^{ \mathsf{T}}\widetilde{\mathbf{v}}_{i}\right], \tag{21}\]  
it can be easily demonstratedthat \(Z(t)\) is positive semi-definite, making it a suitable candidate function for Lyapunov analysis.