# IntelligentL2-L∞ Consensus of Multiagent Systems under Switching Topologies via Fuzzy Deep Q Learning

Authors: Haoyu Cheng; Linpeng Xu; Ruijia Song; Yue Zhu; Yangwang Fang
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

The problem of intelligentL2-L∞ consensus design for leader-followers multiagent systems (MASs) under switching topologies is investigated based on switched control theory and fuzzy deep Q learning. It is supposed that the communication topologies are time-varying, and the model of MASs under switching topologies is constructed based on switched systems. By employing linear transformation, the problem of consensus of MASs is converted into the issue of L2-L∞ control. The consensus protocol is composed of the dynamics-based protocol and learning-based protocol, where the robust control theory and deep Q learning are applied for the two parts to guarantee the prescribed performance and improve the transient performance. The multiple Lyapunov function (MLF) method and mode-dependent average dwell time (MDADT) method are combined to give the scheduling interval, which ensures stability and prescribed attenuation performance. The sufficient existing conditions of consensus protocol are given, and the solutions of the dynamics-based protocol are derived based on linear matrix inequalities (LMIs). Then, the online design of the learning-based protocol is formulated as a Markov decision process, where the fuzzy deep Q learning is utilized to compensate for the uncertainties and achieve optimal performance. The variation of the learning-based protocol is modeled as the external compensation on the dynamics-based protocol. Therefore, the convergence of the proposed protocol can be guaranteed by employing the nonfragile control theory. In the end, a numerical example is given to validate the effectiveness and superiority of the proposed method.


---

## Body

## 1. Introduction

In recent years, the coordination control of MASs has attracted considerable attention for their broad applications in many fields [1, 2], such as formation control, cooperative attack, and attitude alignment. The MAS consists of a series of agents, which can communicate and interact with each other to realize multiple missions and adapt to the complex environment [3, 4]. In particular, much attention has been paid to the problem of consensus of MASs because of their great potential applications in both economic and military. The purpose of MASs is to construct a relationship between the agents to achieve an agreement for the state/output. In the past decades, fruitful research studies have emerged to contribute to the development in theory and applications. To mention a few, the problem of distributed formation control for MASs is studied in [5], the time-varying formation design for MASs with disturbances is proposed in [6], and the problem of finite-time consensus for switched nonlinear MASs is investigated in [7].In practical applications, it is well known that the communication topology among the agents may change dramatically over time to adjust to multiple missions and complex environments [8, 9], such as the MASs can realize obstacle avoidance and higher flight efficiency by formation transformation [10, 11]. The design flexibility, security, and performance of convergence will be improved, which motivated the studies on the switching topologies of MASs [1, 12]. Recently, because of the broad potential applications of switching topologies, considerable significant research studies have been proposed by scholar at home and abroad. The communication topologies among interacting agents will change according to the flight conditions and missions, which can be modeled as switched systems. The switched systems consist of a series of continuous-time (or discrete-time) subsystems and a switching signal, which determines the switching strategy between subsystems. It provides an efficient approach to deal with the problem of fast time-varying conditions. Therefore, it can be inferred that the switching of topologies can be viewed as the switching between subsystems, and it is essential to study the problem of consensus protocol design to make sure the state/output can converge to the given value. In [13], the problem of time-varying formation control of MASs is investigated. The communication topologies switching among given connected topologies and the switching signal depend on the Markovian process. The Lyapunov function method is utilized to analyze the convergence. In the work of [14], the problem of event-triggered leader-following consensus problem for multiagent systems with external disturbances is addressed under switching topologies. A novel distributed event-triggered protocol is proposed to realize disturbance rejection based on extended state observer. The average dwell time (ADT) method is utilized to ensure the stability of the event-triggered protocol. In [15], the time-varying practical formation problem is studied for spacecraft, where switching topologies and time-delays are taken into consideration. Sufficient conditions are provided to ensure that the error system is convergent, which are derived based on the ADT method. It is well known that the research studies mentioned above are proposed to deal with the problem of switching topologies. However, the convergence is guaranteed based on the ADT method. It can be inferred that the common parameters are applied for all subsystems in the ADT method, which will lead to conservativeness. To obtain tighter bounds on dwell time and improve the design flexibility of the algorithm, MDADT is applied during last decades. In [16], the MDADT method and multiple discontinuous Lyapunov function (MDLF) method are combined to analyze the stability of switched systems with unstable modes. The sufficient conditions are established, and the results in existing literature are covered as a special case. The fast switching and slow switching in the framework of MDADT are applied to unstable modes and stable modes. In [17], the global adaptive control algorithm for switched systems is proposed based on the MDADT method. The different properties of subsystems are taken into consideration. Then, the adaptive tracking controller is applied to the nonlinear switched systems with external disturbance and unmodeled dynamics, which illustrates the effectiveness and superiority of the MDADT method. In the work of [18], the event-triggered sliding mode controller is proposed. By employing the MDADT method and event-triggered strategy, less conservative and more practical results are obtained. Sufficient conditions are given to ensure stochastically exponential stability by the aid of the LMI technique. The literature mentioned has provided fruitful results on consensus protocol design for MASs under switching topologies. However, stability and convergence are ensured by the traditional ADT method. The different properties of subsystems cannot be considered, which will lead to conservativeness. Therefore, how to obtain less restrictive results is still an open and challenging problem, which has been fully investigated, and it has an important value and potential applications in practice.Moreover, in practical environment, there always exist uncertainties and disturbances, which will lead to performance degradation and even instability [19, 20]. Therefore, it is essential to investigate the robust consensus problem to improve the performance in the uncertain environment [21–23]. In the work of [24], the problem of distributed H∞ containment control for MASs with switching topologies is studied. An observer-based containment control scheme is proposed. The external disturbance and time delay in the environment are taken into consideration, which is more applicable than the traditional method. By employing the Lyapunov function method and LMIs technique, the sufficient existing conditions and solutions of control protocol are given in the form of LMIs. In [25], the problem of time-varying formation of second-order discrete-time MASs under switching topologies and the time delay is investigated. The sufficient conditions are given to ensure MASs accomplish the mission of time-varying formation based on the state transformation method. The time delay and uncertainties are considered. Compared with the existing literature, the proposed can overcome the undesirable response caused by time delay and improve the transient performance. In the work of [26], the problem of formation control for tail-sitters in flight mode transitions is studied. The nonlinear dynamics and uncertainties are considered, and the robust time-varying formation control protocol is proposed. It is proven that the tracking errors can converge to the origin in finite time. The problem of L2-gain robust protocol for time-varying output formation-containment of MASs is addressed in [27]. The PID-based output-feedback control protocol is provided to ensure that all followers can track a time-varying formation reference, where communication delays and external disturbance are taken into consideration. The asymptotic stability of MASs is proved by the Lyapunov function method. However, as well known, the transient performance and robustness cannot achieve simultaneously. Therefore, we need to make comprise of the transient performance and robustness, which still remains an open and challenging problem.In addition, with the development of computing ability, the intelligent technique has been an attractable problem during the last decades [28–30]. It is widely applied in the areas of target recognition, machine vision, robotic systems, and controller design [31, 32]. It provides an efficient method to improve the autonomy and design flexibility of the system [33]. The most widely used methods are the deep learning and reinforcement learning. As a combination of deep learning and reinforcement learning, the advantages of deep learning and reinforcement learning are adopted, which include the characteristics of self-fitting and self-learning. In the work of [34], the automatic completion of multiple peg-in-hole assemble tasks is realized. Because the traditional method requires an accurate contact model and complex analysis, the intelligent control method is formulated by constructing the task as a Markov decision process. The deep deterministic policy gradient (DDPG) algorithm is proposed to accomplish the task to achieve optimal policy and avoid risky actions. In [35], a noninteger PID controller is proposed based on the DDPG algorithm. The measurement noises and external disturbances are taken into consideration. The kinematic controller and dynamic controller are proposed to achieve optimal performance. The DDPG algorithm is given to compensate for the uncertainties and disturbances in the framework of actor-critic. A numerical example is given to illustrate the effectiveness of the proposed method. Cheng et al. [36] proposed the real-time controller for the problem of fuel-optimal moon landing. Because the traditional method cannot meet the demand of high requirements of real-time performance and autonomy, the deep reinforcement learning algorithm is proposed for the real-time optimal control based on actor-indirect method architecture. The deep neural networks are applied for initial guesses, and the efficiency of training data is guaranteed. The literature mentioned above has provided considerable meaningful results in the area of machine learning. However, to the best of the authors’ knowledge, the intelligent consensus design for MASs with considerations of stability, robustness, and optimal transient performance has not been fully studied yet. It is essential and important to achieve optimal comprise of robustness and transient performance.Based on the statement above, it can be inferred that the problem of the improvement of autonomy and design flexibility for the system needs to be studied. The problem of consensus protocol design for MASs under switching topologies has not been fully investigated yet. The design flexibility can be improved by employing tighter bounds on dwell time because less conservative results can be obtained, and it leaves more room to ensure the switching logic stays in the subsystems with better performance for long enough time. Moreover, it is of great importance to combine the advantages of the traditional method and intelligent technique, which can ensure convergence, robustness, and transient performance simultaneously. Therefore, the problem of intelligentL2-L∞ consensus design of MASs under switching topologies is investigated. The convergence and robustness are guaranteed by the Lyapunov function method and the MDADT method, which are more applicable. The transient performance is improved by fuzzy deep Q learning, in which the fuzzy reward function is proposed for the complex scheduling process. The main contributions of this study can be summarized as follows:(1)
TheL2-L∞ consensus protocol of MASs under switching topologies is designed. The problem of L2-L∞ consensus of MASs is converted into the problem of stability analysis for switched systems, which is more applicable than the traditional method. The MDADT method and multiple Lyapunov function method are combined to guarantee the stability and prescribed attenuation performance index, which can obtain tighter bounds on dwell time and less conservative results.(2)
The consensus protocol is composed of the dynamics-based consensus protocol and learning-based consensus protocol. Compared with the traditional method, the proposed strategy can ensure the stability, robustness, and transient performance simultaneously.(3)
The fuzzy reward function is utilized to improve the efficiency of the deep reinforcement learning algorithm. The design of reward function for the traditional method mainly depends on the experience of designer, which will lead to complexity. The fuzzy reward function can improve the data efficiency and ensure optimal performance.The rest of the study is organized as follows: the preliminaries and problem statement are provided in Section2; in Section 3, the main results of the study are given; the numerical example is given in Section 4, which is followed by the conclusion in Section 5.


## 2. Preliminaries and Problem Statement

In this study, it is supposed that MASs are composed of a leader labelled as 0 andn followers labelled as 1, 2, …, n. The connection topology among n followers can be described as a time-varying model with Nf topologies. We define Gσk=G1,G2,…,GNf as undirected connected graph, respectively. ℋ=1,2,…,n,n>1 represents the set of finite nodes. s=σk:0,∞⟶R=1,2,…,Nf denotes the switching signal, which is a piecewise continuous function of time and takes value in the finite set ℋ. Aσk=aijσkn×n and ℒσk=lijσkn×n are the adjacency matrices of the undirected graph Gσk and the Laplacian matrix at time instant k, where aijσk stands for the element of adjacency matrix, where aijσk=1 represents that the node i can obtain information from node j, and lijσk is defined in the following equation.(1)lijσk=−aijσki≠j,liiσk=∑j=1naijσki=j.Then, for given nodei∈ℋ, we can define the neighbors of node i as Ni,σk=j∈ℋ:aijσk=1.Another undirected connected graph is defined asG˜σk=G˜1,G˜2,…,G˜Nf to indicate the information transformation between the leader and the followers with n nodes. Define a diagonal matrix Θσk=diagθ1σk,θ2σk,…,θnσk, where θiσk=1 stands for that the node i∈ℋ can obtain information from the leader; otherwise, we define θiσk=0.Therefore, MASs with leader-followers can be described as in the following equations:(2)x0k+1=Ax0k,(3)xik+1=Axik+Buik+Dωikzik=Cυik,i∈ℋ,where A, B, C, and D are the system matrices with appropriate dimensions, x0k=x01k,x02k,…,x0pkT∈Rp represents the state vector of leader, xik=xi1k,xi2k,...,xipkT∈Rp is the state of the ith follower, uik=ui1k,ui2k,…,uilkT∈Rl is the input of the ith follower, zik=zi1k,zi2k,…,ziqkT∈Rq stands for the output of the ith follower, and ωik∈Rm denotes the external disturbance belonging to L20,∞. It is supposed that the agent i can obtain information from its neighbors and leader. Therefore, we define υik as relative state measurements of the ith agent, which can be described as follows:(4)υik=∑j∈Ni,σkxik−xjk+θiσkx0k−xik.In this study, the control input of theith agent to ensure the consensus of leader-followers is proposed.(5)uik=Kσk+Kc,σkυik,where Kσk is the control parameter to be determined by robust control theory, and Kc,σk is the compensated parameter obtained by deep Q learning. In this study, the gained parameters Kc,σk are supposed to vary in a finite set with given bounds. The Kc,σk can be viewed as additional perturbance of Kσk, which can be described as follows:(6)Kc,σk=MσkFσkNσk,where Mσk∈Rl×lΔ and Nσk∈RqΔ×q are the known matrices with appropriate dimensions, and Fσk∈RlΔ×qΔ are the unknown matrices with FσkTFσk≤I.For theith agent, the error of state is defined as eik=xik−x0k. Then, the closed-loop system can be rewritten as(7)ek+1=A˜σkek+ωB˜k,zk=C˜σkek,where ek=e1Tk,…,enTkT, zk=z1Tk,…,znTkT, ωk=ω1Tk,…,ωnTkT, A˜σk=In⊗A−ℒσk+Θσk⊗BKσk+Kc,σk, B˜=In⊗D, and C˜σk=−ℒσk+Θσk⊗C.To facilitate the proof, the definitions and lemmas are given as follows.Definition 1 (see [37]).
For given switching signalσk and k1>0, define Nσ,s0,k1 as the number of switching instants over the time interval 0,k1. Tσs0,k1 is set to be the activated time of undirected graph Gs during 0,k1. There exist constant scalars N0≥0 and τas>0, such that(8)Nσ,s0,k1≤N0+Tσs0,k1τas.
Then,τas is called the mode-dependent average dwell time and Nσ,s0,k1 is the mode-dependent chatter bound, respectively. In this study, we set N0=0.Definition 2.
(see [37]). If there exist control protocol in equation (5), all agents asymptotically track the state trajectory of the leader, such that(9)limk⟶∞xik−x0k2=0,for any initial conditionsxi0,i∈ℋDefinition 3.
(see [38]). For given constant scalars 0<δ<1 and γ>0, the prescribed L2−L∞ attenuation performance γ is satisfied such that(1)
The MASs in equations (2)-(3) are asymptotically stable when ωk=0.(2)
The following inequation holds for all nonzeroωk∈l20,∞.(10)zrL∞2≤γ2ωrL22.Lemma 1.
(see [35]). The matrices ℒσk+Θσk are symmetric and positive definite if and only if the graphs G˜σk are connected for t≥0. Moreover, there exist a transformation matrix Tσk, such that the following equation holds.(11)Tσkℒσk+ΘσkTσk−1=Zσk=diagλ1σk,…,λnσk,where λiσk,i∈ℋ are the nonzero eigenvalues of matrices ℒσk+Θσk.Lemma 2.
(see [39]). For given constant a>0 and real matrices Θ,U,V,W, it is concluded that equation (12) is equivalent to equation (13).(12)ΘU+aV∗−aW+WT<0,(13)Θ+UW−1VT+VW−TUT<0.Lemma 3 (see [39]).
For given symmetric matrixT and matricesℳ,N, if there exist constant scalar ε>0, such that(14)T+ℳℳTε−1+εNTN<0Then, the following equation holds for any appropriateℱ with ℱTℱ≤I.(15)T+ℳℱN+NTℱTℳ<0.


## 3. Main Results

### 3.1.L2-L∞ Consensus Protocol Design

In this section, theL2-L∞ consensus protocol is proposed, and the stability and prescribed performance are guaranteed.Lemma 4.
For given constant scalars0<δ<1, γ>0. The system in (7) with control input in (5) is asymptotic stable with L2-L∞ attenuation performance γ if and only if the following equation holds.(16)e˜k+1=In⊗A−Zσk⊗BKσk+ΔKσke˜k+In⊗Dω˜k,z˜k=−Zσk⊗Ce˜k,where(17)ω˜k=Tσk⊗Ilωk,z˜k=Tσk⊗Iqzk,e˜k=Tσk⊗Ipek.Proof.
Substituting equation (17) to (7), one can obtain equation (16). It can be inferred that the transformation matrix Tσk is unique; therefore, we have the following equations.(18)ω˜kL22=∑0∞ω˜k22,=∑0∞ωTkTσk⊗IlTTσk⊗Ilωk,=ωkL22,(19)z˜rL∞2=supz˜Tkz˜k,=supzTkTσk⊗IlTTσk⊗Ilzk,=zrL∞2.
It is obvious that the problem of robust consensus protocol design can be converted to the controller design of (16).Remark 1.
The system in equation (16) consists of the independent system in equation (20). Therefore, the stability of equation (7) is equivalent to the stability of n subsystems in equation (20); the attempt to ensure the prescribed attenuation performance of (7) can be converted to guarantee the attenuation performance of (16).(20)e˜ik+1=A¯ie˜ik+Dω˜ikz˜ik=C¯ie˜ik,i∈ℋ,where A¯i=A−λiσkBKσk+Kc,σk, and C¯i=−λiσkC.
In Theorem1, the sufficient conditions to guarantee the stability and prescribed attenuation performance index are presented.Theorem 1.
For given constant scalarsμs>1, 0<δs<1, γ>0, if there exist Lyapunov functions Vi,se˜ik,i∈ℋ, and class functions κ1, κ2, the switched systems in equation (20) with MDADT satisfying equation (25) are globally uniformly asymptotically stable with prescribed L2-L∞ attenuation performance γ, such that(21)κ1e˜ik≤Vi,se˜ik≤κ2e˜ik,(22)Vi,se˜ik≤μsVi,re˜ik,s≠r,(23)ΔVi,se˜ik≤−δsVi,se˜ik+ω˜k22,(24)z˜ik22≤γ2Vi,se˜ik,(25)τas≥τas∗=−lnμsln1−δs.Proof.
The entire proof can be divided into two steps.(1)
The stability of equation (20).The switching instants in the time interval0,k are set to be k1, k2, …, kt with kt+1=k. Then, (26) holds when ωik≡0.(26)Vi,σkte˜ik≤1−δσktk−ktVi,σkte˜ikt.Together with (22), we can conclude that(27)Vi,σkte˜ik≤μσkt1−δσktk−ktVi,σkte˜ikt.Based on equations (26)-(27), the following equation can be obtained by iteration.(28)Vi,σkte˜ik≤μσkt1−δσktk−ktVi,σkte˜ikt,≤μσktμσkt−11−δσktk−kt1−δσkt−1kt−kt−1Vi,σkt−1e˜ikt−1,⋅⋅⋅≤∏r=0tμσkr1−δσkrkr+1−krVi,σ0e˜i0,≤∏s=1NfμσksNσ,s0,k1−δσksTσs0,kVi,σ0e˜i0.Combining with Definition1, we have(29)Vi,σkte˜ik≤exp∑s=1NfNσ,s0,klnμσks+Tσs0,kln1−δσksVi,σ0e˜i0,≤exp∑s=1NfTσs0,kτaslnμσks+Tσs0,kln1−δσksVi,σ0e˜i0,≤exp∑s=1NfTσs0,klnμσksτas+ln1−δσksVi,σ0e˜i0.Then, we can obtain (29) based on (21).(30)e˜ik2≤κ1−1κ2exp∑s=1NfTσs0,klnμσksτas+ln1−δσkse˜i02.Therefore, the system in (20) with MDADT satisfying (25) is globally uniformly asymptotically stable.(2)
The system in equation (20) has prescribed L2-L∞ attenuation performance γ.
Together with equations (22)-(23), one has(31)Vi,σkte˜ik≤1−δσktk−ktVi,σkte˜ikt+∑r=ktk−11−δσktk−r−1ω˜r22,≤μσkt1−δσktk−ktVi,σkt−1e˜ikt+∑r=ktk−11−δσktk−r−1ω˜r22.
Then, one can obtain the equation as follows by iteration.(32)Vi,σkte˜ik≤μσkt1−δσktk−ktVi,σkt−1e˜ikt+∑r=ktk−11−δσktk−r−1ω˜r22,≤μσkt1−δσktk−ktμσkt−11−δσkt−1kt−kt−1Vi,σkt−2e˜ikt−1,+∑r=kt−1kt−11−δσkt−1k−r−1ω˜r22+∑r=ktk−11−δσktk−r−1ω˜r22,⋅⋅⋅≤μσk1⋅⋅⋅μσkt1−δσk0k1−k0⋅⋅⋅1−δσktk−ktVi,σ0e˜i0,+μσk1⋅⋅⋅μσkt1−δσk0k1−k0⋅⋅⋅1−δσktk−kt∑r=k0k1−11−δσk0k1−r−1ω˜r22,+μσk2⋅⋅⋅μσkt1−δσk1k2−k1⋅⋅⋅1−δσktk−kt∑r=k1k2−11−δσk1k2−r−1ω˜r22,+⋅⋅⋅+∑r=ktk−11−δσktk−r−1ω˜r22,≤1−δσktk−kt∏r=0t−1μσkr+11−δσkrkr+1−krVi,σ0e˜i0,+μσkt1−δσktk−kt∑q=1t−1∏p=qt−1μσkp1−δσkpkp+1−kp∑r=kq−1kq−11−δσkq−1kq−r−1ω˜r22,+μσkt1−δσktk−kt∑r=kt−1kt−11−δσkt−1kt−r−1ω˜r22,+∑r=ktk−11−δσktk−r−1ω˜r22,≤∏s=1NfμσksNσ,s0,k1−δσksTσs0,kVi,σ0e˜i0,+∑q=1t∏s=1NfμσksNσ,skq,k1−δσksTσskq,k∑r=kq−1kq−11−δσkq−1kq−r−1ω˜r22,+∑r=ktk−11−δσktk−r−1ω˜r22.
Substituting the equation above into (8), one can obtain that(33)Vi,σkte˜ik≤exp∑s=1NfTσs0,klnμσksτas+ln1−δσksVi,σ0e˜i0,+∑q=1texp∑s=1NfTσskq,klnμσksτas+ln1−δσks,×∑r=kq−1kq−11−δσkq−1kq−r−1ω˜r22,+∑r=ktk−11−δσktk−r−1ω˜r22.
According to the conditionsμs>1, 0<δs<1, and (25), we have(34)0<1−δσks<1,(35)0<exp∑s=1NfTσskq,klnμσksτas+ln1−δσks<1.
Combining equations (32)–(34), one can obtain equation (35).(36)Vi,σkte˜ik≤∑q=1t∑r=kq−1kq−1ω˜r22+∑r=ktk−1ω˜r22,=∑r=0kt−1ω˜r22+∑r=ktk−1ω˜r22,=∑r=0k−1ω˜r22.
Together with (24), it is obvious that(37)z˜ik22≤γ2Vi,se˜ik,≤γ2∑r=0k−1ω˜r22,which implies that supz˜r22≤γ2ω˜rL22, and the proof is complete.Corollary 1.
For given constant scalarsμs>1, 0<δs<1, γ>0, if there exist positive-definite matrices Pi,s∈Rp×p satisfying equations (37)–(39), the switched systems in equation (20) with MDADT satisfying equation (25) are globally uniformly asymptotically stable with prescribed L2-L∞ attenuation performance γ.(38)Pi,s≤μsPi,r,s≠r,(39)A¯iTPi,sA¯i−1−δsPi,sA¯iTPi,sD∗DTPi,sD−I≤0,(40)λis2CTC<γ2Pi,s.Proof.
The Lyapunov functionVi,se˜ik,i∈ℋ, is defined as follows:(41)Vi,se˜ik=e˜iTkPi,se˜ik.
According to (20) and (39), we can conclude that (38) is equivalent to (24). Along the trajectory of Vi,se˜ik, one has(42)ΔVi,se˜ik=e˜iTk+1Pi,se˜ik+1−e˜iTkPi,se˜ik,=A¯ie˜ik+Dω˜ikT×Pi,sA¯ie˜ik+Dω˜ik,−e˜iTkPi,se˜ik,=e˜iTkA¯iTPi,sA¯i−Pi,se˜ik+ω˜iTkDTPi,sDω˜ik,+e˜iTkA¯iTPi,sDω˜ik+ω˜iTkDTPi,sA¯ie˜ik.
Together with equations (40)-(41), we have(43)ΔVi,se˜ik+δsVi,se˜ik−ω˜iTkω˜ik,=e˜iTkA¯iTPi,sA¯i−1−δsPi,se˜ik+e˜iTkA¯iTPi,sDω˜ik,+ω˜iTkDTPi,sA¯ie˜ik+ω˜iTkDTPi,sDi−Iω˜ik,=e˜ikω˜ikTA¯iTPi,sA¯i−1−δsPi,sA¯iTPi,sD∗DTPi,sD−Ie˜ikω˜ik,≤0.
According to Theorem1, we can conclude that the system in (20) with MDADT satisfying (25) is globally uniformly asymptotically stable with prescribed L2-L∞ attenuation performance γ.
Based on Theorem1 and Corollary 1, the solutions of consensus protocol are given in Theorem 2.Theorem 2.
For given constant scalarsμs>1, 0<δs<1, γ>0, as>0, and εs>0, if there exist positive-definite matrices Pi,s∈Rp×p, matrices Xs∈Rl×l, Ys∈Rl×q, the MASs in (2)-(3) with control input in equation (5) are asymptotically stable with prescribed L2-L∞ attenuation performance γ such that equation (43) holds.(44)Ξ110Ξ13asλisYsT0∗−IDTPi,s00∗∗−In−1⊗PsBXs−Pi,sBPi,sBMs∗∗∗−asXs+XsT0∗∗∗∗−εsI<0.
The parameters of control protocol can be derived in (43).(45)Ks=Xs−1Ys,where Ξ11=−1−δsPi,s+εsλis2NsTNs, Ξ13=ATPi,s−λisYsTBT.Proof.
According to Schur complement, it is obvious that equation (43) is equivalent to equation (45).(46)Ξ110Ξ13asλisYsT∗−IDTPi,s0∗∗−Pi,sBXs−Pi,sB∗∗∗−asXs+XsT+εs−100Pi,sBMs000Pi,sBMs0T,=Ξ110Ξ13asλisYsT∗−IDTPi,s0∗∗−Pi,s+εs−1Pi,sBMsMsTBTPi,sBXs−Pi,sB∗∗∗−asXs+XsT,<0.
DefineΘs=Ts+εs−1ℳsT+εsNsTNs,Us=00XsTBT−BTPi,sT, and Vs=λisYs00T, where Ts=−1−δsPi,s0ATPi,s−λisYstBT∗−IDTPi,s∗∗−Pi,s, ℳs=00Pi,sBMs, and Ns=−λisNs00.
Together with Lemma2, we have(47)Θs+UsXs−1VsT+VsXs−TUsT,=Ts+εs−1ℳsℳsT+εsNsTNs+UsXs−1VsT+VsXs−TUsT<0.
Moreover, based on Lemma3, one has(48)T+ℳℱN+NTℱTℳT+UsXs−1VsT+VsXs−TUsT=−1−δsPi,s0A¯iTPi,s∗−IDTPi,s∗∗−Pi,s<0.
According to Schur complement, it is obvious that (46) is equivalent to (37), which completes the proof.


## 3.1.L2-L∞ Consensus Protocol Design

In this section, theL2-L∞ consensus protocol is proposed, and the stability and prescribed performance are guaranteed.Lemma 4.
For given constant scalars0<δ<1, γ>0. The system in (7) with control input in (5) is asymptotic stable with L2-L∞ attenuation performance γ if and only if the following equation holds.(16)e˜k+1=In⊗A−Zσk⊗BKσk+ΔKσke˜k+In⊗Dω˜k,z˜k=−Zσk⊗Ce˜k,where(17)ω˜k=Tσk⊗Ilωk,z˜k=Tσk⊗Iqzk,e˜k=Tσk⊗Ipek.Proof.
Substituting equation (17) to (7), one can obtain equation (16). It can be inferred that the transformation matrix Tσk is unique; therefore, we have the following equations.(18)ω˜kL22=∑0∞ω˜k22,=∑0∞ωTkTσk⊗IlTTσk⊗Ilωk,=ωkL22,(19)z˜rL∞2=supz˜Tkz˜k,=supzTkTσk⊗IlTTσk⊗Ilzk,=zrL∞2.
It is obvious that the problem of robust consensus protocol design can be converted to the controller design of (16).Remark 1.
The system in equation (16) consists of the independent system in equation (20). Therefore, the stability of equation (7) is equivalent to the stability of n subsystems in equation (20); the attempt to ensure the prescribed attenuation performance of (7) can be converted to guarantee the attenuation performance of (16).(20)e˜ik+1=A¯ie˜ik+Dω˜ikz˜ik=C¯ie˜ik,i∈ℋ,where A¯i=A−λiσkBKσk+Kc,σk, and C¯i=−λiσkC.
In Theorem1, the sufficient conditions to guarantee the stability and prescribed attenuation performance index are presented.Theorem 1.
For given constant scalarsμs>1, 0<δs<1, γ>0, if there exist Lyapunov functions Vi,se˜ik,i∈ℋ, and class functions κ1, κ2, the switched systems in equation (20) with MDADT satisfying equation (25) are globally uniformly asymptotically stable with prescribed L2-L∞ attenuation performance γ, such that(21)κ1e˜ik≤Vi,se˜ik≤κ2e˜ik,(22)Vi,se˜ik≤μsVi,re˜ik,s≠r,(23)ΔVi,se˜ik≤−δsVi,se˜ik+ω˜k22,(24)z˜ik22≤γ2Vi,se˜ik,(25)τas≥τas∗=−lnμsln1−δs.Proof.
The entire proof can be divided into two steps.(1)
The stability of equation (20).The switching instants in the time interval0,k are set to be k1, k2, …, kt with kt+1=k. Then, (26) holds when ωik≡0.(26)Vi,σkte˜ik≤1−δσktk−ktVi,σkte˜ikt.Together with (22), we can conclude that(27)Vi,σkte˜ik≤μσkt1−δσktk−ktVi,σkte˜ikt.Based on equations (26)-(27), the following equation can be obtained by iteration.(28)Vi,σkte˜ik≤μσkt1−δσktk−ktVi,σkte˜ikt,≤μσktμσkt−11−δσktk−kt1−δσkt−1kt−kt−1Vi,σkt−1e˜ikt−1,⋅⋅⋅≤∏r=0tμσkr1−δσkrkr+1−krVi,σ0e˜i0,≤∏s=1NfμσksNσ,s0,k1−δσksTσs0,kVi,σ0e˜i0.Combining with Definition1, we have(29)Vi,σkte˜ik≤exp∑s=1NfNσ,s0,klnμσks+Tσs0,kln1−δσksVi,σ0e˜i0,≤exp∑s=1NfTσs0,kτaslnμσks+Tσs0,kln1−δσksVi,σ0e˜i0,≤exp∑s=1NfTσs0,klnμσksτas+ln1−δσksVi,σ0e˜i0.Then, we can obtain (29) based on (21).(30)e˜ik2≤κ1−1κ2exp∑s=1NfTσs0,klnμσksτas+ln1−δσkse˜i02.Therefore, the system in (20) with MDADT satisfying (25) is globally uniformly asymptotically stable.(2)
The system in equation (20) has prescribed L2-L∞ attenuation performance γ.
Together with equations (22)-(23), one has(31)Vi,σkte˜ik≤1−δσktk−ktVi,σkte˜ikt+∑r=ktk−11−δσktk−r−1ω˜r22,≤μσkt1−δσktk−ktVi,σkt−1e˜ikt+∑r=ktk−11−δσktk−r−1ω˜r22.
Then, one can obtain the equation as follows by iteration.(32)Vi,σkte˜ik≤μσkt1−δσktk−ktVi,σkt−1e˜ikt+∑r=ktk−11−δσktk−r−1ω˜r22,≤μσkt1−δσktk−ktμσkt−11−δσkt−1kt−kt−1Vi,σkt−2e˜ikt−1,+∑r=kt−1kt−11−δσkt−1k−r−1ω˜r22+∑r=ktk−11−δσktk−r−1ω˜r22,⋅⋅⋅≤μσk1⋅⋅⋅μσkt1−δσk0k1−k0⋅⋅⋅1−δσktk−ktVi,σ0e˜i0,+μσk1⋅⋅⋅μσkt1−δσk0k1−k0⋅⋅⋅1−δσktk−kt∑r=k0k1−11−δσk0k1−r−1ω˜r22,+μσk2⋅⋅⋅μσkt1−δσk1k2−k1⋅⋅⋅1−δσktk−kt∑r=k1k2−11−δσk1k2−r−1ω˜r22,+⋅⋅⋅+∑r=ktk−11−δσktk−r−1ω˜r22,≤1−δσktk−kt∏r=0t−1μσkr+11−δσkrkr+1−krVi,σ0e˜i0,+μσkt1−δσktk−kt∑q=1t−1∏p=qt−1μσkp1−δσkpkp+1−kp∑r=kq−1kq−11−δσkq−1kq−r−1ω˜r22,+μσkt1−δσktk−kt∑r=kt−1kt−11−δσkt−1kt−r−1ω˜r22,+∑r=ktk−11−δσktk−r−1ω˜r22,≤∏s=1NfμσksNσ,s0,k1−δσksTσs0,kVi,σ0e˜i0,+∑q=1t∏s=1NfμσksNσ,skq,k1−δσksTσskq,k∑r=kq−1kq−11−δσkq−1kq−r−1ω˜r22,+∑r=ktk−11−δσktk−r−1ω˜r22.
Substituting the equation above into (8), one can obtain that(33)Vi,σkte˜ik≤exp∑s=1NfTσs0,klnμσksτas+ln1−δσksVi,σ0e˜i0,+∑q=1texp∑s=1NfTσskq,klnμσksτas+ln1−δσks,×∑r=kq−1kq−11−δσkq−1kq−r−1ω˜r22,+∑r=ktk−11−δσktk−r−1ω˜r22.
According to the conditionsμs>1, 0<δs<1, and (25), we have(34)0<1−δσks<1,(35)0<exp∑s=1NfTσskq,klnμσksτas+ln1−δσks<1.
Combining equations (32)–(34), one can obtain equation (35).(36)Vi,σkte˜ik≤∑q=1t∑r=kq−1kq−1ω˜r22+∑r=ktk−1ω˜r22,=∑r=0kt−1ω˜r22+∑r=ktk−1ω˜r22,=∑r=0k−1ω˜r22.
Together with (24), it is obvious that(37)z˜ik22≤γ2Vi,se˜ik,≤γ2∑r=0k−1ω˜r22,which implies that supz˜r22≤γ2ω˜rL22, and the proof is complete.Corollary 1.
For given constant scalarsμs>1, 0<δs<1, γ>0, if there exist positive-definite matrices Pi,s∈Rp×p satisfying equations (37)–(39), the switched systems in equation (20) with MDADT satisfying equation (25) are globally uniformly asymptotically stable with prescribed L2-L∞ attenuation performance γ.(38)Pi,s≤μsPi,r,s≠r,(39)A¯iTPi,sA¯i−1−δsPi,sA¯iTPi,sD∗DTPi,sD−I≤0,(40)λis2CTC<γ2Pi,s.Proof.
The Lyapunov functionVi,se˜ik,i∈ℋ, is defined as follows:(41)Vi,se˜ik=e˜iTkPi,se˜ik.
According to (20) and (39), we can conclude that (38) is equivalent to (24). Along the trajectory of Vi,se˜ik, one has(42)ΔVi,se˜ik=e˜iTk+1Pi,se˜ik+1−e˜iTkPi,se˜ik,=A¯ie˜ik+Dω˜ikT×Pi,sA¯ie˜ik+Dω˜ik,−e˜iTkPi,se˜ik,=e˜iTkA¯iTPi,sA¯i−Pi,se˜ik+ω˜iTkDTPi,sDω˜ik,+e˜iTkA¯iTPi,sDω˜ik+ω˜iTkDTPi,sA¯ie˜ik.
Together with equations (40)-(41), we have(43)ΔVi,se˜ik+δsVi,se˜ik−ω˜iTkω˜ik,=e˜iTkA¯iTPi,sA¯i−1−δsPi,se˜ik+e˜iTkA¯iTPi,sDω˜ik,+ω˜iTkDTPi,sA¯ie˜ik+ω˜iTkDTPi,sDi−Iω˜ik,=e˜ikω˜ikTA¯iTPi,sA¯i−1−δsPi,sA¯iTPi,sD∗DTPi,sD−Ie˜ikω˜ik,≤0.
According to Theorem1, we can conclude that the system in (20) with MDADT satisfying (25) is globally uniformly asymptotically stable with prescribed L2-L∞ attenuation performance γ.
Based on Theorem1 and Corollary 1, the solutions of consensus protocol are given in Theorem 2.Theorem 2.
For given constant scalarsμs>1, 0<δs<1, γ>0, as>0, and εs>0, if there exist positive-definite matrices Pi,s∈Rp×p, matrices Xs∈Rl×l, Ys∈Rl×q, the MASs in (2)-(3) with control input in equation (5) are asymptotically stable with prescribed L2-L∞ attenuation performance γ such that equation (43) holds.(44)Ξ110Ξ13asλisYsT0∗−IDTPi,s00∗∗−In−1⊗PsBXs−Pi,sBPi,sBMs∗∗∗−asXs+XsT0∗∗∗∗−εsI<0.
The parameters of control protocol can be derived in (43).(45)Ks=Xs−1Ys,where Ξ11=−1−δsPi,s+εsλis2NsTNs, Ξ13=ATPi,s−λisYsTBT.Proof.
According to Schur complement, it is obvious that equation (43) is equivalent to equation (45).(46)Ξ110Ξ13asλisYsT∗−IDTPi,s0∗∗−Pi,sBXs−Pi,sB∗∗∗−asXs+XsT+εs−100Pi,sBMs000Pi,sBMs0T,=Ξ110Ξ13asλisYsT∗−IDTPi,s0∗∗−Pi,s+εs−1Pi,sBMsMsTBTPi,sBXs−Pi,sB∗∗∗−asXs+XsT,<0.
DefineΘs=Ts+εs−1ℳsT+εsNsTNs,Us=00XsTBT−BTPi,sT, and Vs=λisYs00T, where Ts=−1−δsPi,s0ATPi,s−λisYstBT∗−IDTPi,s∗∗−Pi,s, ℳs=00Pi,sBMs, and Ns=−λisNs00.
Together with Lemma2, we have(47)Θs+UsXs−1VsT+VsXs−TUsT,=Ts+εs−1ℳsℳsT+εsNsTNs+UsXs−1VsT+VsXs−TUsT<0.
Moreover, based on Lemma3, one has(48)T+ℳℱN+NTℱTℳT+UsXs−1VsT+VsXs−TUsT=−1−δsPi,s0A¯iTPi,s∗−IDTPi,s∗∗−Pi,s<0.
According to Schur complement, it is obvious that (46) is equivalent to (37), which completes the proof.


## 4. Compensated Consensus Protocol Design Based on FDQL

In this section, the learning-based consensus protocol is proposed based on deep reinforcement learning, where fuzzy deepQ learning is utilized. The stability and prescribed attenuation performance are guaranteed by the robust control theory, and the learning-based control protocol is introduced to improve the transient performance and realize optimal control policy. The output of the learning-based control protocol can be viewed as an additional variation of robust consensus protocol. The online scheduling of control protocol is established as a Markovian process. Therefore, the advantages of robust control theory and deep reinforcement learning are combined.It is well known that reinforcement learning is composed of state, action, agent, and environment. The state ofkth step is defined as sk, and the chosen action is supposed to be ak; then, the reward function rk and the state sk+1 are generated based on the interaction with the environment. Therefore, the optimal control policy can be obtained by maximum the reward function.To improve the convergence of consensus protocol, the state is defined assk=e˜ikz˜ik and the action is defined as ak=Kc,σk.InQ learning, the deep neural network is utilized to approximate the action-state value function Q∗sk,ak, which can be described as(49)fsk,ak,ω=Q∗sk,ak,where fsk,ak,ω denotes the function of deep neural networks.The action is chosen based on the maximumQ value:(50)a∗=argmaxaQsk,ak.There exist two neural networks in the deepQ learning algorithm, whose structures are the same and can be called as the critic neural network and target neural network. The parameters of the critic neural network are updated based on temporal-difference learning. The output of the critic neural network is defined as Qsk,ak,ω and the output of the target neural network is defined as Qsk,ak,ω−. Therefore, the parameters of the critic neural network are updated based on the equation as follows:(51)Qsk,ak,ω−=LrR+γsmaxa′Qs′,a′,ω−+1−LrQsk,ak,ω,where Lr is the learning rate, γs denotes the discount factor, R represents the reward of state transition from sk to s′ through action ak, and maxa′Qs′,a′,ω− stands for the maximum Q value of the target neural network.It can be inferred that the reward function has an important influence on the final performance. The design of traditional deepQ learning mainly depends on the experience of designers, which can not achieve optimal performance and will improve the computational complexity. In this study, the reward function is applied to design the reward function. The input value of fuzzy reward function can be divided into five categories, which can be described as VB, B, N, G, and VG. The five categories represent very bad, bad, normal, good, and very good. In this study, it is supposed that there are four followers. Therefore, the inputs of the fuzzy reward system are set to be e1, e2, e3, and e4. It can be inferred that each fuzzy set includes 25 rules, and the total number of the fuzzy rules is 75. The output of the fuzzy reward function is limited in the interval [−1,0), and the defuzzifier of the fuzzy reward function is defined as(52)fz=∑p=125Ci∏q=1msqpzq∑p=125∏q=1msqpzq.Based on the statement above, the learning-based consensus protocol design algorithm can be summarized as follows:Algorithm 1: Learning-based control protocol design based on FDQN.
(1)
Design the dynamics-based consensus protocol according to Theorem2(2)
Define the bounds of learning-based consensus protocol parameters(3)
Initialize the weights of the Q value network(4)
Initialize the weights of the target Q value network(5)
Initialize the replay bufferR, episode = 0(6)
for episode = 1 to M do(7)
Initialize a random states1 and receive the initial observation(8)
Fort = 1 to K do(9)
Select an action based on the state and reward function(10)
Execute the actionak, and then, one can obtain the reward function rk and the state sk+1(11)
Store the pairsk,ak,rk,sk+1 in the replay buffer(12)
Sample a random minibatch of transitionssm,am,rm,sm+1 from the replay buffer(13)
Update the target Q value function(14)
Update the weights of the target Q value network(15)
end for(16)
end forRemark 2.
The FDQN algorithm proposed in this study can improve the transient convergence performance of MASs. The output of the deepQ network is supposed to be variation of parameters of consensus protocol. As well known, the design of reward function in the traditional method depends of the experience of the designers. To overcome the problem, the fuzzy reward function is developed to improve the learning efficiency in this study.


## 5. Numerical Example

In this section, an example is provided to illustrate the effectiveness of the method. The model of MASs is constructed as follows:(53)A=−1⋅3−1⋅51⋅31⋅2,B=1⋅21⋅5,C=1⋅10⋅9,D=0⋅130⋅12.The external disturbance is(54)dk=0⋅1e−0.2kcos0⋅4k.The switching topologies are shown in Figure1. Then, we can obtain the Laplace matrices as follows:(55)L1=100−101−100−110−1001,L2=100−1010−1001−1−1−1−13.Figure 1
Switching topologies of MASs. (a) Interacting topologyG1. (b) Interacting topology G2.
(a)(b)The parameters of switching topologies are given as follows:(56)a1=1⋅01a2=1⋅11a3=1⋅08,μ1=1⋅25μ2=1⋅31μ3=1⋅38,δ1=0⋅51δ2=0⋅52δ3=0⋅53.Therefore, we can obtain MDADT according to (25).(57)τ1=0⋅3128,τ2=0⋅3679,τ3=0⋅4266.It is well known that the ADT method can be viewed as a special case of the MDADT method. Therefore, it can be inferred thatτa=maxτai=0⋅4266. It is obvious that tighter bounds on dwell time and less conservative results can be obtained. Then, we define the attenuation performance index γ=0⋅9, and we can obtain the parameters of consensus protocol based on Theorem 2.The switching logic is shown in Figure2. In order to illustrate the effectiveness and superiority of the proposed method, the traditional ADT method and MDADT method are given as comparisons. From the statement above, we have realized that MDADT can obtain tighter bounds and less conservative results. Moreover, the comparisons of state response of the ADT method and MDADT method are shown in Figures 3–6. The state responses of MASs with ADT switching topologies are shown in Figures 3-4. The state responses of MASs with MDADT switching topologies are shown in Figures 5-6. We can see that the transient performance of the ADT method is better than that of the MDADT method because the different characteristics of subsystems are taken into consideration, which will no doubt improve the design flexibility and make it more applicable for practical conditions.Figure 2
The switching logic.Figure 3
The state response of (x)1 under the ADT method.Figure 4
The state response of (x)2 under the ADT method.Figure 5
The state response of (x)1 under the MDADT method.Figure 6
The state response of (x)2 under the MDADT method.Validate the superiority of the proposed method. The state response of the proposed method is shown in Figures7–11. The state responses of the proposed method are shown in Figures 7-8. We can conclude that the transient performance can be improved by the aid of fuzzy deep Q learning. The advantages of the traditional method and intelligent method are combined. Compared with the traditional method, the transient performance can be improved, and compared with the intelligent method, stability and training efficiency can be guaranteed. The attenuation performance index is shown in Figure 9, from which we can see that the robustness of the proposed is ensured. The episodes reward response is shown in Figure 10, and we can see that the reward function of the fuzzy deep Q learning algorithm can converge to the neighbor of the origin, which demonstrates the effectiveness of the algorithm in this study. In addition, the response of the action is shown in Figure 11, from which we can see that the learning-based consensus protocol is provided to compensate the additional input caused by the uncertainties.Figure 7
The state response of (x)1 with the proposed method.Figure 8
The state response of (x)2 with the proposed method.Figure 9
The response of attenuation performance index.Figure 10
The response of episodes reward.Figure 11
The response of the action.Based on the statement above, we can conclude that the convergence, robustness, and prescribed attenuation performance index are guaranteed. The less conservative results and tighter bounds on dwell time can be obtained by the MDADT method. The transient performance of the system can be improved based on the fuzzy deepQ learning algorithm. It is worth mentioning that the traditional robust cannot make comprised of robustness and transient performance, and the intelligent method always cannot guarantee convergence. By employing the proposed method, convergence, robustness, and transient performance are guaranteed simultaneously.


## 6. Conclusions

The problem of intelligentL2-L∞ consensus design for MASs under switching topologies is investigated in this study. The switching topologies of MASs are modeled as switched system theory by employing linear transformation. Then, the problem of consensus protocol design can be converted to the problem of L2-L∞ control. To ensure the convergence, robustness, and transient performance simultaneously, the proposed consensus protocol is composed of dynamics-based consensus protocol and learning-based consensus protocol, which provides baseline and compensation of uncertainties. The baseline of consensus protocol is obtained by dynamics-based consensus protocol, which is provided based on the MDADT method and MLF method. The scheduling interval of learning-based protocol is given by nonfragile control theory. Then, the learning-based consensus protocol is proposed based on the fuzzy deep Q learning algorithm to improve the transient performance and achieve optimal policy, where the fuzzy reward function is introduced to improve the learning efficiency.



---

*Source: 4105546-2022-02-16.xml*