Performance-Based Human-in-the-Loop Optimal Bipartite Consensus Control for Multi-Agent Systems via Reinforcement Learning

Authors: zongsheng  huang
tieshan  li
yue  long
hanqing  yang
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-12 04:07:05

Abstract: This paper investigates the performance-based human-in-the-loop (HiTL) optimal bipartite consensus control problem for nonlinear multi-agent systems (MASs) under signed topology. First, to respond to any emergencies and guarantee the safety of MASs, the MASs are monitored by human operator sending command signals to the non-autonomous leader. Then, under the joint design architecture of prescribe-time performance function and error transformation, a novel performance index function involving transformed error and control input is developed to achieve optimal bipartite consensus with prescribed-time. Subsequently, the reinforcement learning (RL) method is utilized to learn the solution to Hamilton-Jacobian-Bellman (HJB) equation, in which the fuzzy logic systems (FLSs) are employed to implement the method. Finally, the simulation results depict the effectiveness of the constructed control scheme.

Main paper: 
 i. introduction: 
 In recent years, with the rapid development of multiple unmanned aerial vehicles (UAVs) [1] , multiple unmanned ground vehicles (UGVs) [2] and other fields, multi-agent systems (MASs) have been paid more and more attention by scholars. As one of the hot issues in control problems of MASs, consensus control problems have been widely studied. As a branch of consensus control, bipartite consensus was first introduced in [3] taking both competition and cooperation relationships between agents into consideration. For bipartite consensus, the agents eventually converge to two states of opposite sign but equal size. In [4] - [6] , the various control strategies of bipartite consensus have been designed broadly.This work was supported in part by the National Natural Science Foundation of China under Grant 51939001, Grant 62273072, and Grant 62203088, in part by the Natural Science Foundation of Sichuan Province under Grant 2022NSFSC0903.(Corresponding author: Tieshan Li) Notably, the MASs mentioned above are fully autonomous. However, incidents with Boeing 737 jetliners and Tesla's autonomous driving systems have raised serious concerns and highlighted the challenges that fully autonomous MASs face in making judgments during in uncertain and complex environments. Therefore, it is urgent to develop monitoring schemes to complete tasks when MASs encounter unexpected situations [7] . Fortunately, the human-in-the-loop (HiTL) control approach was introduced in MASs to supervise the entire system to respond to sudden changes by sending commands to the leader agent [8] . Later, many studies on HiTL control for MASs have emerged in [9] - [15] . In [9] , the HiTL formation tracking control scheme together with edge-based event-driven mechanism was constructed for MASs. Considering stochastic actuation attacks, in [13] , the prescribed-time and prescribedaccuracy HiTL cluster consensus control problem has been solved. In view of the ability to deal with emergencies, the HiTL control approach has also been favored by multi-UAV systems in [14] , [15] .Optimal control, a widely used control method, has garnered significant attention. For nonlinear systems, the optimal solution is derived from the Hamilton-Jacobian-Bellman (HJB) equation. However, obtaining the solution of HJB equation through numerical methods is infeasible. To overcome this challenge, reinforcement learning (RL) that motivated by animal behaviors was proposed as a powerful tool [16] . The core idea of RL is to approximate the solution of the HJB equation using a function approximation structure. The value iteration algorithm, one of the valuable algorithms in RL, was developed by Murray et al. in [17] , in which the convergence analysis was also detailed. In [18] , the policy iteration algorithm, as another equally important algorithm, was designed to obtain the optimal saturation controller for nonlinear systems. Based on the previous work, RL method has been used to solve the optimal problem for MASs. In [19] , an optimal control protocol based on RL was designed to achieve containment control without prior knowledge of the system dynamics. For unknown discrete-time MASs, in [20] , the optimal bipartite consensus control problem was solved. Nevertheless, the above results only conclude that the optimal controller is globally asymptotically stable. It is important to note that achieving specified accuracy within a given time is crucial in many fields.Fortunately, the prescribed-time control (PTC) was firstly proposed by Song et al. [21] . The PTC distinguishes from finite-time control and fixed-time control, in which the preset settling time is not related to the initial values of the system. Depending on [21] , in [22] , the convergence rate can be predetermined as needed, and a general method for constructing the time-varying rate function was provided. In [23] , a novel time-varying constraint function was devised to guarantee that the system remains operational beyond the prescribed time, leading to a global result. In particular, the PTC-based HiTL control scheme was developed to realize the cluster consensus within given time in [13] . However, to the best of the authors' knowledge, the bipartite consensus control scheme considering both optimal performance and prescribed-time performance under the framework of HiTL control has not been fully explored, which promotes our research.Driven by these observations, this paper focuses on investigating the performance-based HiTL optimal bipartite consensus control problem. The main contributions are summarized below.(1) Unlike the autonomous leader described in [4] - [6] which lacked intelligent decision-making, this paper aims to improve the security, stability, and emergency response capabilities of the system by designing the leader of the MASs to be non-autonomous, where the time-varying control input is governed by a human operator. (2) Compared with the existing optimal results for MASs in [19] , [20] , to realize both optimal performance and prescribed-time performance, a unified design framework of PTC and RL method is proposed, where the settling time and accuracy can be preset without initial values. The structure is given below. In Section II, the considered system and some assumptions are given. In Section III, the main results including the PTC performance function and optimal controller are designed. In Section IV, the convergence analysis is provided. The simulation results is given in Section V. Finally, the conclusion is presented in Section VI.The structurally balanced bipartition communication topology containing N followers is represented by a directed graphG = {V, ε, A}, where V = {V 1 , V 2 , • • • , V N } represents the vertex set, which is divided into the cooperative set V α and competitive set V β such that V α ∩ V β = 0 and V α ∪ V β = V. ε ⊆ V × V represents the edge set of N followers. Let A = [a ij ] ∈ R N ×N be the signed weight matrix, where a ij > 0 if (V i , V j ) ∈ V m , m ∈ {α, β} and a ij < 0 if V i ∈ V m , V j ∈ V n , m ̸ = n, m, n ∈ {α, β}. The neighbor set of ith follower is defined as N i = {j ∈ V : a ij ̸ = 0}. Define L = D -A ∈ R N ×N as the Laplacian matrix of G, where D = diag(d 1 , d 2 , • • • , d N ) ∈ R N ×N denotes the degree matrix with d i = N j=1 |a ij |.The argument graph consisting of one leader and N followers is denoted asG = { Ṽ, ε}, in which Ṽ = {V 0 , V 1 , V 2 , • • • , V N } and ε ⊆ Ṽ × Ṽ. Let B = diag{|b 1 |, |b 2 |, • • • , |b N |} ∈ R N ×N, where b i = 1 indicates that the information of the leader is available for the ith node and b i > 0 represents cooperative relation, b i < 0 represents competitive relation.Assume that the nonlinear MAS is composed of N (≥ 2) followers and one leader. The dynamics model of ith follower is provided asEQUATIONwherex i (t) ∈ R n denotes state, u i (t) ∈ R m is control input, f i (x i ) ∈ R n is internal dynamics and g i (x i ) ∈ R n×m is input dynamics.Next, the dynamics of the human-manipulated leader is given asEQUATIONwhere x h 0 (t) ∈ R n denotes state and u h 0 (t) ∈ R m is nonzero control input of human operator sending to leader, f h 0 (x h 0 ) ∈ R n represents internal dynamics.The following assumptions and lemma are imposed.Assumption 1. [19] The signed graph G has a directed spanning tree.Assumption 2. [24] The input of human operator always makes the leader (2) stable.The FLS can estimate a nonlinear continuous functionEQUATIONwith b > 0.To achieve prescribed-time (PT) performance for MASs, the PT performance function ϑ(t) is given asEQUATIONwhere h > 0, ι > 0, β > 0, ϑ Tr > 0, 0 < T r < ∞ and 0 < ϑ Tr < ∞ represent the user-defined settling time and steady-state tracking accuracy, respectively.Construct the bipartite consensus error ase i = N j=1 |a ij |(x i -sign(a ij )x j ) + |b i |(x i -sign(b i )x h 0 ), e i = [e i,1 , • • • , e i,n] T ∈ R n and adopt the error transformation function asEQUATIONwhere |e i,ı (0)| < ϑ(0).Based on (5) , it yieldsEQUATIONRemark 1. From ( 5), the inequality -ϑ ≤ e i,ı ≤ ϑ, ∀t ≥ 0 holds. Combined the definition in (4), it further observes that -ϑ Tr ≤ e i,ı ≤ ϑ Tr , ∀t ≥ T r if ϱ i,ı is bounded, which means the PT performance of e i can be ensured.Define the performance index function asEQUATIONwhere Q i and R i are symmetric positive definite matrices with suitable dimensions,A i = [A i,1 , • • • , A i,n ] T = [arctan(ϱ i,1 ), • • • , arctan(ϱ i,n )] T .Taking the time derivative of A i,ı , one hasEQUATIONwhereχ i,ı = π 2ϑ cos 2 ( π 2 e i,ı ϑ ) , ν i,ı = ei,ı θ ϑ , ėi = Γ i (f i + g i u i ) - N j=1 a ij ẋj -b i ẋh 0 and Γ i = d i + |b i |.Then, define the Hamiltonian function asEQUATIONwhere χi = diag{ χi,11+ϱ 2 i,1 , • • • , χi,n 1+ϱ 2 i,n }, ν i = [ν i,1 , • • • , ν i,n ] and χ i = diag{χ i,1 , • • • , χ i,n }.The corresponding HJB equation is given asEQUATIONDifferentiating the (10) with respect to u i , one hasEQUATIONSubstituting (11) into (10), ( 10) becomes( 2ϑ π A i ) T Q i ( 2ϑ π A i ) + ∂J * i ∂ϱ i [χ i (Γ i f i - N j=1 a ij ẋi -b i ẋh 0 -ν i )] + ∂J * i ∂ϑ ∂ϑ ∂t - Γ 2 i 4 ∂J * i ∂ϱ T i g i χ i R -1 i χ T i g T i ∂J * i ∂ϱ i = 0.Inspired by [26] ,EQUATIONwhereEQUATIONObviously, the HJB equation can not be acquired by numerical methods. Therefore, the PI approach is given in Algorithm 1 to find the optimal result. as Eq. ( 13). 4 Step 4:If ∥J (l+1) i -J (l)i ∥ ≤ ℵ with the predefined parameter ℵ > 0, stop; otherwise, set l = l + 1 and return to Step 2.The convergence and optimality of Algorithm 1 have been proved in [27] and are omitted here.In view of the unknown term F i (X i ) and J i (X i ) in ( 13), the FLSs is used to approximate these terms as.EQUATIONEQUATIONwhere ω Fi ∈ R hc1×n and ω Ji ∈ R hc2×n represent ideal weight matrices with h c1 and h c2 are the number of fuzzy rules; ϕ Fi ∈ R hc1 and ϕ Ji ∈ R hc2 are fuzzy basis functions; ϵ Fi (X i ) and ϵ Ji (X i ) denote bounded approximation errors. Thus, (13) becomesu * i = -k i R -1 i χ -1 i ϱ i -R -1 i χ -1 i (ω T Fi ϕ Fi (X i ) + ϵ Fi (X i )) - 1 2 R -1 i χ -1 i (ω T Ji ϕ Ji (X i ) + ϵ Ji (X i )).However, the ω Fi and ω Ji are unknown, the estimation forms of ( 14) and (15) areEQUATIONEQUATIONwhere ωFi ∈ R hc1×n and ωJi ∈ R hc2×n represent estimated weight matrices. According to ( 16) and ( 17), one hasEQUATIONThe updating laws are constructed asEQUATIONEQUATIONwhere C i ∈ R hc1×hc1 is a positive-definite matrix, r Fi > 0, r Ji > 0, r > 0 are design parameters.IV. STABILITY ANALYSIS Theorem 1. Consider the MAS consisting of followers (1) and leader (1) under Assumption 1-3, by choosing k i > 3 4 and adopting optimal control input (18) and adaptive law (19) and (20) , then the consensus error can converge to the prescribed accuracy within prescribed time.Proof. Develop the Lyapunov function asEQUATIONwhereϱ = [ϱ T 1 , • • • , ϱ T n ] T ∈ R N ×n , estimation error ωFi = ω Fi -ωFi and ωJi = ω Ji -ωJi .Invoking ( 5), (19) and (20) , it yieldsEQUATIONUsing Young's inequality, we haveEQUATIONEQUATIONEQUATIONEQUATIONCalculating (22) by bringing ( 23)-( 26), one hasEQUATION= N j=1 o 2 R -1 i ||ϵ Fi || 2 + N j=1 oR -1 i 4 ωT Ji ϕ Ji (X i ))ϕ T Ji (X i ))ω Ji + N j=1 oR -1 i 4 ||ϱ i || 2 + N j=1 r F i 2 ω T Fi ω Fi + N j=1 ωT J i 2 (ϕ T Ji (X i )ϕ Ji (X i ) + rI hc2 )ω Ji , κ 1 = min i=1,••• ,N {2oR -1 i (k i -3 4 )}, κ 2 = min i=1,••• ,N { r F i λmax(C -1 i ) }, κ 3 = min i=1,••• ,N {r Ji λ min (ϕ i )}, κ = min{κ 1 , κ 2 , κ 3 }, λ min (ϕ i ) is the minimal eigenvalue of ϕ T Ji (X i )ϕ Ji (X i ).V. SIMULATION A nonlinear MAS composed by four single-link robot arms (three followers and one human-controlled leader) is given to verify the effectiveness of the proposed control scheme. The model of agent is given as [12] J i qi + D i qi + M i gd i sin(q i ) = u i , i = 1, • • • , 3,the physical parameters of g, M i , D i , J i and d i can be found in [12] for details. u h 0 is set as As shown in Fig. 1 , it can be obtained thatEQUATIONA =   0 -1 1 0 0 0 0 0 0   , L =   2 1 -1 0 0 0 0 0 0   , B = diag{1, 0, 0, 0, 0}.For PT performance function, select ϑ Tr = 0.06, T r = 3s. The initial state values of followers and leader are presented in Table 1 . i = 0 i = 1 i = 2 i = 3 x i,1 (0) 1 0.8 0.5 0.8 x i,2 (0) -1 0.8 -0.5 -0.8For the unknown termF i (X i ), X i = [x i , x h 0 , ẋh 0 , ϑ, θ] T and defined over [-6, 6]. Choose X 0 i = [[-6 -L , -6 + L ] T , • • • , [-6 -L , -6 + L ] T 5 ] T and ϕ F L i (X i ) = exp(- (Xi-X 0 i ) T (Xi-X 0 i ) 2).J i (X i ), X i = [x i , ϱ i , x h 0 , ẋh 0 , ϑ, θ] T and defined over [-6, 6]. Choose X 0 i = [[-6 -L , -6 + L ] T , • • • , [-6 -L , -6 + L ] T 6 ] T and ϕ Ji (X i ) L (X i ) = exp(- (Xi-X 0 i ) T (Xi-X 0 i ) 2).For updating law (19) and (20) From Fig. 2 and Fig. 3 , the bipartite consensus can be achieved and the leader, followers 1 and 2 belong to a group while the follower 3 is geared to another group with opposite sign. Fig. 4 shows the bipartite consensus and the PT performance bounds. It can be obtained that the consensus error can reach the given accuracy 0.06 with the prescribed time 3s. The optimal control input for each agent is depicted in Fig. 5 , in which u i rapidly converges to a small region of zero. The norm of updating weights in unknown terms F i (X i )are given in Fig. 6 ., ωF1 (0) = ωF2 (0) = ωF3 (0) = [0.1] 12×2 , ωJ1 (0) = ωJ2 (0) = ωJ3 (0) = [0.92] 12×2 , C 1 = diag {0.5, • • • , 0.5} 12 , C 2 = diag {0.7, • • • , 0.7} 12 , C 3 = diag {0.3, • • • , 0.3} 12 , R i = diag{0.8, 0.8}, r Fi = 2, k i = 45, r Ji = 1.In this article, the problem of performance-based HiTL optimal bipartite consensus control for nonlinear MASs has been studied. First, the MASs have been monitored by human operator sending command signals to the non-autonomous leader to respond to any emergencies and guarantee the safety of MASs. Then, under the joint design architecture of prescribe-time performance function and error transformation, a novel performance index function has been developed to achieve optimal bipartite consensus with prescribed-time. Subsequently, the RL has been utilized to learn the solution to HJB equation, in which the FLSs are employed to implement the algorithm. The validity of the designed control scheme has been confirmed by simulation. 
 
 Appendix:  



Reviews: review: title: title: The obtained result is valuable and can be accepted after responding the following comments.
main_review: main_review: This paper  primarily investigates the design of an optimal bipartite consensus control scheme for nonlinear multi-agent systems (MASs) under signed topologies, with a human-in-the-loop (HiTL) component. The obtained result is valuable and can be accepted after responding the following comments.
1) In the introduction, it is not enough to state the current work. It should be expended and reconstructed. (2) In the simulation section, more analysis can be added to better explain the main results of this paper, that's not enough. (3) There are many typos and grammar errors. The authors should have a native English speaker or software packages to perform the editing check. (4) The conclusion of the article suggests using the present perfect tense for description
title: title: This article is quite fascinating and of high quality.
main_review: main_review: The paper titled "Performance-Based Human-in-the-Loop Optimal Bipartite Consensus Control for Multi-Agent Systems via Reinforcement Learning" investigates the performance-based human-in-the-loop optimal bipartite consensus control problem for nonlinear multi-agent systems under signed topology. Firstly, the MASs have been monitored by human operator sending command signals to the non-autonomous leader to respond to any emergencies and guarantee the safety of MASs. Then, under the joint design architecture of prescribe-time performance function and error transformation, a novel performance index function has been developed to achieve optimal bipartite consensus with prescribed-time. Finally, the RL has been utilized to learn the solution to HJB equation, in which the FLSs are employed to implement the algorithm. The article has clear logic and organization, my specific feedback is as follows: 1) In the introduction, the author’s analysis of the HiTL control is insufficient. 2) What role does the performance index function mentioned in the paper play in achieving the optimal bipartite consensus?
title: title: Accept
main_review: main_review: In this paper, " Distributed Unknown Input Observer-Based Global Fault-Tolerant Average Consensus Control for Linear Multi-Agent Systems", a global fault-tolerant average consistent control method is proposed. Firstly, a distributed unknown input observer based on relative estimation error is proposed. Then, using the relative estimation error, a global fault-tolerant mean consistent controller is proposed. The article has clear logic and organization, but there are still some problems. My specific feedback is as follows :1) What are the advantages of relative estimation error compared with estimation error? 2) What are the advantages of distributed unknown input observers compared to distributed observers?