# H∞ Consensus Design and Online Scheduling for Multiagent Systems with Switching Topologies via Deep Reinforcement Learning

Authors: Haoyu Cheng; Ruijia Song; Linpeng Xu; Di Zhang; Shengli Xu
Journal: International Journal of Aerospace Engineering
 (2022)

---

## Abstract

This paper is devoted toH∞ consensus design and online scheduling for homogeneous multiagent systems (MASs) with switching topologies via deep reinforcement learning. The model of homogeneous MASs with switching topologies is established based on switched systems theory, in which the switching of topologies is viewed as the switching among subsystems. By employing linear transformation, the closed-loop systems of MASs are converted into reduced-order systems. The problem of H∞ consensus design can be transformed to the issue of H∞ control. It is supposed that the consensus protocol is composed of two parts: dynamics-based protocol and learning-based protocol, where dynamics-based protocol is provided to guarantee the convergence and weighted attenuation and learning-based protocol is proposed to improve the transient performance. Then, the multiple Lyapunov function (MLF) method and mode-dependent average dwell time (MDADT) method are combined to ensure the stability and weighted H∞ disturbance attenuation index of reduced-order systems. The sufficient existing conditions of dynamics-based protocol are given through the feasible solutions of linear matrix inequalities (LMIs). Moreover, the online scheduling is formulated as a Markov decision process, and the deep deterministic policy gradient (DDPG) algorithm in the framework of actor-critic is proposed for the compensation of disturbance to explore optimal control policy. The online scheduling of parameters of MASs is viewed as bounded compensation of dynamics-based protocol, whose stability can be guaranteed by nonfragile control theory. Finally, simulation results are provided to illustrate the effectiveness and superiority of the proposed method.


---

## Body

## 1. Introduction

As a typical class of complex networks, MASs have attracted increasing attention because of its great potential applications in numerous areas of engineering society and modern industry, such as formation control, attitude alignment, multiple missile attack, and mobile sensor networks [1–3]. Especially in the area of aerospace engineering, the MASs can be applied to unmanned aerial vehicles, artificial satellite, and so on, which devotes an important and meaningful technique for cooperative behaviour. The problems of consensus and consensus tracking are core problems of coordination control of MASs, whose objective lies in establishing a relationship between the agents, so as to accomplish common goals jointly by designing appropriate control protocol to achieve an agreement for the states/outputs [4–6]. Recently, fruitful and encouraging results are reported on the consensus problem of MASs; to mention a few, the observer-based distributed consensus protocol for MASs is proposed in [7]; the fault-tolerant consensus tracking problem for MASs with switching network is studied in [8]; the problem of time-varying formation control for MASs with time delay and switching topologies is investigated in [9].Previous studies on the problem of consensus for MASs mainly concentrate on the MASs with nonlinear/linear systems, or first-order dynamics and second dynamics [10, 11]. However, in practice, there are many useful applications by realizing time-varying formation shape and relative positions [12, 13], such as obstacle avoidance to achieve greater security and formation transformation to realize higher searching efficiency, which motivates the researches on switching topologies for MASs. Switching topologies exist widely in practical situation because of its broad applications [14, 15]. It is supposed that the communication topology among each agent may change over given time interval. Accordingly, it is essential to guarantee the consensus problem of MASs with switching topologies [16]. Up to now, a number of important and meaningful results have immerged; see [17–20] and references therein. It can be inferred that the switching signal can be divided into arbitrary switching and constrained switching. It is often the case that the switching signal depends on given variables, such as frequency and positive. If the convergence of MASs is independent of switching topologies, we can conclude that the consensus can be ensured under arbitrary switching. In [21], the problem of finite-time consensus for switched nonlinear MASs is investigated. A novel adaptive protocol is provided to deal with the problems of nonlinearity and switching topologies. The sufficient conditions to guarantee the finite-time consensus of switched MASs under arbitrary switching is proposed. Also, constrained switching usually emerges naturally from the physical restrictions of communications. Therefore, the constrained switching is more applicable and preferable in practical applications. The latest literatures in analysis of constrained switching have been mainly concentrated on state-dependent, time-dependent, and a combination of them [22]. As one of the most components of constrained switching, the time-dependent switching has been extensively studied because of its wide applications. It is generally known that switching logic is one of the key issues for stability analysis for switched systems. The stability of switched systems can be guaranteed with proper switching logic even there exist unstable subsystems. The dwell time method is efficient and widely used. In the work of [23–25], the definition of MDADT is first proposed. Based on MDADT method, tighter bounds on dwell time and less restrictive results are obtained. The MLF method and MDADT method are combined to analyze the stability of switched systems, which is widely applied in existing literatures. In [26], the problem of stability analysis for switched systems with unstable modes is investigated. The multiple discontinuous Lyapunov function method and MDADT method are combined. The results illustrate that the stability of switched systems with unstable modes can be guaranteed if the switching logic satisfies given constraints. The literatures above focus on the stability analysis for single system. The switching topologies of MASs can be established as switched systems, where the communication topology is viewed as subsystem. The problem of consensus analysis and protocol design for MASs with switching topologies is studied in [27]. The problem of weighted H∞ consensus design for MASs with switching topologies is studied. The switching of topologies is viewed as the switching of subsystems. Therefore, the problem of robust consensus is converted into the problem of stability analysis for reduced-order system. The stability and weighted attenuation performance are guaranteed by the Lyapunov function method and average dwell time method. The sufficient conditions of stability analysis and the solution of protocol are given in the form of LMIs. In the work of [28], the issue of distributed containment control of MASs is studied. The asynchronous switching and stochastic disturbances are taken into consideration, in which it is supposed that the switching of controllers will lag behind the switching of system modes. By employing the Lyapunov function method and matrix transformation, the exponential stabilization in mean square sense is achieved. It can be inferred that how to obtain tighter bounds on dwell time and less restrictive results still remain interesting and challenging problems to be studied.Moreover, in many existing literatures, it is supposed the agent dynamics is precisely known, which will no doubt lead to restrictive results because of the inherent existence of uncertain dynamics and external disturbance [29–31]. Thus, the researches on robust consensus problem have attracted the attention of scholars at home and abroad. In the work of [32], the problem of robust H∞ consensus for linear MASs is addressed. The distributed dynamic output feedback protocol is proposed through model transformation, which converts the problem of H∞ consensus control of MASs to the problem of independent H∞ stability of reduced-order linear subsystems. The sufficient conditions of stability analysis and the solutions of feedback protocol are formulated by LMIs. In [33], the problem of robust consensus control for second-order MASs with switching topologies is investigated. The uncertain parameters are viewed as independent variables, which can be formulated as special polynomial systems of MASs. Thus, the robust consensus problem can be converted to the robust stability analysis problem. The consensus and disturbance attenuation performance can be guaranteed by solving LMIs, which bridge the connection between complex robust consensus and stability analysis for linear systems. Moreover, the robust consensus problem of fractional-order MASs with input saturation and disturbances is studied in [34]. It is supposed that the input saturation and external disturbances exist in the environment simultaneously. The Mittag-Leffler stability theorem and low gain feedback technique are applied to guarantee the robust consensus. In the work of [35], an unknown system dynamics estimator is developed to neutralize the adverse effect caused by exogenous disturbances and system uncertainties. Moreover, the fault case is considered in [36] and the event-triggered scheme is proposed in [37], which has developed fruitful results for the analysis of uncertainties. According to the literatures above, it can be inferred that the existing results about robust consensus cannot realize online-learning scheduling and the transient convergence performance cannot be guaranteed. It is essential to develop approaches to ensure the convergence, robustness, and transient performance simultaneously.On the other hand, with the development of computational ability, machine learning has attracted amount of attention and been applied in many fields during the last few years [38, 39]. As an important technique of machine learning, deep reinforcement learning provides efficient approach to deal with the problem of self-optimization and online scheduling [40–42]. Deep learning can promote the considerable performance by effective nonlinear approximate ability, and reinforcement learning can explore the optimal control policy efficiently, which is widely applied in many areas. In the work of [43], the switch-based adaptive dynamic programming is proposed in optimal control problem to minimize the cost, which proposes a distributed computational method for adaptive dynamic programming. In [44], a noninteger PID controller is proposed based on deep reinforcement learning. To accomplish the control object, a kinematic controller and a dynamic controller are established, which are based on the kinematic model and deep reinforcement learning. A DDPG algorithm in the framework of actor-critic is proposed for the supplementary dynamic controller. In addition, in the work of [45], the assembly task is accomplished by feedback deep deterministic policy gradient with fuzzy reward. The task is formulated as a Markov decision process, and a model-driven DDPG algorithm is proposed to learn the optimal policy. To improve the learning efficiency, the fuzzy reward system is utilized for the complex assembly process. Simulation results show that the optimal control policy can be achieved and the algorithm can apply to realistic systems efficiently. Reference [46] studies the problem of real-time optimal control, and the deep neural network is introduced to represent the optimal control action. The output of deep neural networks is close to the simulated optimal ones. Then, the on-board real-time optimal control approach is proposed. The developed approach can combine the advantages of optimal control approach and deep learning. The stability, optimality, and real-time ability can be guaranteed simultaneously. The literatures have made progress on the applications of machine learning. However, to the best of authors’ knowledge, the problem of H∞ consensus and online scheduling for MASs has not been fully investigated. It is great importance to make full use of traditional method and machine learning to ensure the convergence, robustness, and transient performance simultaneously, which motivates the study in the paper.Inspired by the reasons mentioned above and the developing related techniques, the problem ofH∞ consensus design and online scheduling for MASs with switching topologies is investigated. It should be pointed out that the model of MASs with switching topologies is established based on switched systems theory, where the switching of topologies can be viewed as subsystems of switched systems. Then, the problem of H∞ consensus is converted to the problem of H∞ controller design of reduced-order system, which is composed of two parts: the dynamics-based controller and learning-based controller. The MLF method and MDADT method are combined to analyze the stability and robustness. The tighter bounds on dwell time and less restrictive results are obtained. To improve the transient performance, the DDPG algorithm is proposed to deal with the problem of online scheduling and optimal control policy can be explored. The main contributions of this paper can be summarized as follows:
(1)
The model of MASs with switching topologies is established as switched systems, where the switching of topologies is viewed as the switching of subsystems. The problem ofH∞ consensus design can be converted to the problem of weighted H∞ control for reduced-order system(2)
The control scheme consists of two parts: the dynamics-based controller is presented based on robust control theory to guarantee the stability and robustness. The MLF method and MDADT method are combined to analyze the stability andH∞ attenuation performance. Sufficient conditions of stability and solutions of control protocol are given in the form of LMIs. It is shown that when the MDADT is chosen to be large enough, the convergence of MASs with switching topologies can be guaranteed. Moreover, compared to the traditional ADT method, the tighter bounds on dwell time and less restrictive results are obtained(3)
The learning-based controller is proposed as supplementary controller based on DDPG algorithm to improve the transient performance. It is provided in the framework of actor-critic and can be viewed as uncertain compensation of dynamics-based controller. The nonfragile control theory and robust control theory are applied to guarantee the stability of DDPG algorithm, where the advantages of traditional methods and machine learning are combined. Compared to the existing methods of machine learning, the stability, robustness, and optimality can be guaranteed simultaneouslyThe remainder of this paper is organized as follows: Section2 shows the model description and transformation. The control protocol of MASs with switching topologies is proposed in Section 3. The simulation results of MASs are provided in Section 4 to validate the effectiveness of proposed method, which is followed by the conclusion in Section 5.


## 2. Preliminaries and Problem Statement

In this paper, the MASs withn agents is considered, which is modeled by a discrete-time system and the ith (i=1,2,⋯,n) agent can be described in Equations (1)–(3).
(1)xik+1=A1xik+B1uik+D1dik,(2)yik=C1xik+D2dik,(3)zik=C2xik−1n∑j=1nxjk,where xik=xi1k,xi2k,⋯,xipkT∈Rp denote the state vector; yik=yi1k,yi2k,⋯,yiqkT∈Rq represents the output vector, respectively; zik=zi1k,zi2k,⋯,zipkT∈Rp is the measured output; uik=ui1k,ui2k,⋯,uilkT∈Rl is the control input; dik∈Rm denotes the external disturbance satisfying dik∈L20,∞; A1, B1, C1, C2, D1, and D2 are system matrices with appropriate dimensions.In this paper, it is supposed that there areNf possible communication topologies that are modeled by the connected balanced directed graphs G1,G2,⋯,GNf. The switching signal s=σk:0,∞⟶R=1,2,⋯,Nf is introduced to represent the switching of topologies, which is a left-continuous piecewise function and takes the value in the set 1,2⋯,Nf. As shown in [27], we define Aσk=aijσkn×n and Lσk=lijσkn×n as the adjacency matrix and the Laplacian matrix at time instant k.We set real matrixZc=ζijn×n with ζii=n−1/n and ζij=−1/n,i≠j. Given a matrix Γ∈Rn×n, we define JΓ=Γ+ΓT as the sum of matrix Γ and its transpose.To ensure the consensus and robustness of MASs, the consensus protocol is designed as follows:(4)uik=ur,ik+uc,ik,where ur,ik is the dynamics-based consensus protocol and uc,ik represents the learning-based consensus protocol.Remark 1.
In this paper, the consensus protocol consists of two parts: the dynamics-based is proposed based on robust control theory to ensure the convergence and robustness. Meanwhile, the learning-based consensus protocol is introduced as supplement compensation based on deep reinforcement learning to achieve optimal consensus protocol policy, through which the undesirable response caused by external disturbance can be overcome and the transient performance can be improved. The parameters ofur,ik can be obtained by LMI technique, and the parameters of uc,ik are scheduled online by DDPG algorithm in the framework of actor-critic.The output consensus protocol is defined as(5)ur,ik=Kr,σk∑j=1naijσkyik−yjk,uc,ik=Kc,σk∑j=1naijσkyik−yjk,where Kr,σk is the parameters of dynamics-based consensus protocol and Kc,σk is the parameters of learning-based consensus protocol obtained by DDPG algorithm. It is supposed that the variation interval of Kc,σk is set to be K¯c,σk,K¯c,σk, which is viewed as the additional bounded uncertainties of dynamics-based consensus protocol.We define the maximum value of learning-based protocol as:(6)K¯mc,σk=maxabsK¯c,σk,K¯c,σk.Therefore, the value ofKc,σk can be written as
(7)Kc,σk=MσkFσkNσk,where Mσk and Nσk are given matrices with appropriate dimensions and satisfying K¯mc,σk=MσkNσk; Fσk are unknown matrices satisfying
(8)FσkTFσk≤I.Remark 2.
As we all know, the robust control theory can ensure the convergence and satisfy prescribed attenuation performance index, but it cannot achieve optimal performance. Moreover, the results of deep reinforcement learning cannot guarantee the stability of the systems in existing researches. Therefore, it is difficult to guarantee convergence, robustness, and transient performance simultaneously. With the inspiration of robust control theory and deep reinforcement learning in recent literatures, the model-based approach and model-free approach are combined to realize convergence, robustness, and transient performance simultaneously.Based on the statement above, the systems of MASs can be rewritten as follows:(9)xk+1=In⊗A1+Lσk⊗B1Kr,σk+Kc,σkC1xk+Lσk⊗B1Kr,σk+Kc,σkD2+In⊗D1dk,zk=Zc⊗C2xk,where xk=x1Tk,⋯,xnTkT, zk=z1Tk,⋯,znTkT, and dk=d1Tk,⋯,dnTkT.To convenient the proof, the definitions are given as follows:Definition 3.
For given switching signalσk and k1>k0≥0, define Nσ,sk0,k1 as the number of its switching times of connected balanced directed graph Gs during time interval k0,k1 and Tσsk0,k1 as the total running time of sth subsystem during k0,k1. If there exist constants N0≥0 and τas>0, such that
(10)Nσ,sk0,k1≤N0+Tσsk0,k1τas,
thenτas is called mode-dependent average dwell time under switching logic σk.Definition 4.
The MASs are said to asymptotically achieve consensus in mean square, if Equation (11) holds when dik≡0.
(11)limk⟶∞xk−x¯k2=0,where xk=x1Tk,x2Tk,⋯,xnTkT, x¯k=1n/n⊗∑j=1nxjk, and 1n denotes n-dimensional column vector.Definition 5.
For given constants0<λ<1 and γ>0, the MASs in Equation (1) is said to achieve a weighted H∞ attenuation performance index γ with nonzero disturbance satisfying dk∈l20,∞.
(12)∑r=k0+∞1−λr−k0zr2≤γ2∑r=k0+∞dr2.Lemma 6.
For an orthogonal matrixΨ∈Rn×n with last column 1n/n satisfying
(13)ΨTZcΨ=diagIn−1,0,then for any connected balanced directed n-order graph Gσk, one has
(14)ΨTLΨ=diagL1,0,L1∈Rn−1×n−1.Lemma 7.
For given matrixΦ=ϕij∈Rn×n,ϕii=−∑j=1,j≠inϕij, one has
(15)ΦZc=Φ.Lemma 8.
For given constantα>0 and matrices Θ,U,V,W, if Equation (16) holds
(16)ΘU+αV∗−αJW<0,one has
(17)Θ+JUW−1VT<0.Lemma 9.
For given symmetric matrixT and matrices M,N, if there exists constant ε>0, such that
(18)T+ε−1MMT+εNTN<<0.Then we can obtain Equation (19) for unknown matrices F satisfying FTF≤I.
(19)T+JMFN<<0.According to Lemma6, it can be inferred that if there exists orthogonal matrix Ψ=Ψ1,Ψ2∈Rn×n, such that
(20)Ψ1TΨ1=In−1,Ψ1Ψ1T+Ψ2Ψ2T=In,(21)Ψ2TZc=0nT,ZcΨ2=0n,(22)Ψ1TZcΨ1=In−1,Ψ1TLσkΨ1=L1σk.Together with Lemma7, we have
(23)Lσk=LσkZc,(24)Zc=ZcZc.From Equations (20) and (21), it is obvious that
(25)Ψ1TZcLσk=Ψ1TZcΨ1Ψ1T+Ψ2Ψ2TLσk=Ψ1TZcΨ1Ψ1TLσk=Ψ1TLσk.Together with Equations (23)–(25), it can be obtained that
(26)Ψ1TZcLσk=Ψ1TLσkZc.Definingξk=Ψ1TZc⊗Imxk, one has
(27)ξk+1=A~σkξk+B~σkdk,zk=C~σkξk,where A~σk=In−1⊗A1+L1σk⊗B1Kr,σk+Kc,σkC1, B~σk=Ψ1TLσk⊗B1Kr,σk+Kc,σkD2+Ψ1TZc⊗D1, and C~σk=Ψ1⊗C2.Remark 10.
It can be inferred thatlimk⟶∞xk−x¯k2=0 is equivalent to limk⟶∞ξk2=0. Therefore, with the application of the transformation, the weighted H∞ consensus protocol design can be converted as the problem of weighted H∞ control design. The problem of consensus protocol design can be formulated as the switched systems in Equation (27) are asymptotically stable in mean square and satisfy the prescribed weighted attenuation performance index.


## 3. Main Results

### 3.1. Dynamics-Based Consensus Protocol Design

In this section, the stability criteria of switched systems in Equation (27) will be given, and the sufficient conditions to guarantee the prescribed H∞ attenuation performance of the switched systems will be presented. Firstly, the stability analysis of switched systems in Equation (27) will be given as follows.Theorem 11.
For given constant scalarsμs>1, 0<λs<1, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exists the Lyapunov function Vsξk and class K∞ functions κ1s and κ2s, then the switched systems in Equation (27) are mean square asymptotically stable such that
(28)κ1sξk≤Vsξk≤κ2sξk,(29)Vsξk≤μsVrξk,(30)ΔVsξk≤−λsVsξk,(31)τas≥τas∗=−lnμsln1−λs.Proof.
It is supposed that the switching instants in the intervalk0,k are defined as k1, k2,…, kt with kt+1=k. Then, Equation (32) can be obtained based on Equation (30) when dik≡0.
(32)Vσktξk≤1−λσktk−ktVσktξkt.
Substituting Equation (32) in Equation (29), one has
(33)Vσktξk≤μσkt1−λσktk−ktVσktξkt.
According to Equations (32) and (33), the equation can be obtained as follows by iteration.
(34)Vσktξk≤μσktƛσktk−ktVσktξkt≤μσktƛσktk−ktμσkt−1ƛσkt−1kt−kt−1Vσkt−1ξkt−1⋯≤∏r=0tμσkrƛσkrkr+1−krVσk0ξk0≤∏s=1NfμσksNσ,sk0,kƛσksTσsk0,kVσk0ξk0,where ƛs=1−λs; then, we can obtain Equation (35) based on Definition 3.
(35)Vσktξk≤exp∑s=1NfNσ,sk0,klnμσks+Tσsk0,klnƛσksVσk0ξk0≤exp∑s=1NfTσsk0,kτas+N0lnμσks+Tσsk0,klnƛσksVσk0ξk0≤μσksN0exp∑s=1NfTσsk0,klnμσksτas+lnƛσksVσk0ξk0.
Then, Equation (36) can be obtained according to Equation (28).
(36)ξk2≤κ1−1κ2μσksN0exp∑s=1NfTσsk0,klnμσksτas+lnƛσksξk02.
Therefore, it is obvious that the switched systems with MDADT satisfying Equation (31) are mean square asymptotically stable, which completes the proof.The stability analysis of switched systems in Equation (27) is presented in Theorem 11, and the sufficient conditions to guarantee the prescribed attenuation performance index will be given in Theorem 12 and Theorem 13.Theorem 12.
For given constant scalarsμs>1, 0<λs<1, γ>0, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exist the Lyapunov function Vsξk and class K∞ function κ1s and κ2s satisfying Equations (28) and (29), such that
(37)ΔVsξk+λsVsξk−Ωk≤0,then the switched systems in Equation (27) are mean square asymptotically stable with weighted H∞ attenuation performance index γd.
whereΩk=−zk2+γ2dk2, γd=γμmaxN0, and μmax=maxμs.Proof.
According to Equation (29), we have
(38)Vσktξk≤ƛσktk−ktVσktξkt+∑r=ktk−1ƛσktk−r−1Ωr≤μσktƛσktk−ktVσktξkt+∑r=ktk−1ƛσktk−r−1Ωr≤μσktƛσktk−ktμσkt−1ƛσkt−1kt−kt−1Vσkt−1ξkt−1+∑r=kt−1kt−1ƛσkt−1k−r−1Ωr+∑r=ktk−1ƛσktk−r−1Ωr⋯≤∏r=0tμσkrƛσkrkr+1−krVσk0ξk0+μσk0ƛσk0k1−k0⋯μσktƛσktk−kt∑r=k0k1−1ƛσk0k−r−1Ωr+μσk1ƛσk1k2−k1⋯μσktƛσktk−kt∑r=k1k2−1ƛσk1k−r−1Ωr+⋯+∑r=ktk−1ƛσktk−r−1Ωr≤∏s=1NfμσksNσ,sk0,kƛσksTσsk0,kVσk0ξk0+∑r=k0k−1∏m=1m≠nNfμσkmNσ,mr,kμσknNσ,nr,kƛσkmTσmr,kƛσknTσnr,k−1Ωr,where n represents the activated connected graph in rth step and m denotes the inactivated connected graph in rth step.Based on the fact0<ƛs<1, one has
(39)Vσktξk≤∏s=1NfμσksNσ,sk0,kƛσksTσsk0,kVσk0ξk0+∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1Ωr.Considering the zero-initial condition andVσktξk≥0, it is obvious that
(40)∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1Ωr≥0⇔∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1zr2≤γ2∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1dr2.Multiplying both sides of Equation (40) by ∏m=1Nfμσkm−Nσ,mk0,k, we can obtain that
(41)∑r=k0k−1∏m=1Nfμσkm−Nσ,mk0,rƛσkmTσmr,k−1zr2≤γ2∑r=k0k−1∏m=1Nfμσkm−Nσ,mk0,rƛσkmTσmr,k−1dr2.Together with Definition3 and Equation (31), it can be inferred that
(42)0≤Nσ,sk0,r≤N0+Tσsk0,rτas∗≤N0−Tσsk0,rlnƛσkslnμσks.Combining with the factμσkm>1 and Equation (42), one has
(43)∑r=k0k−1∏m=1Nfμσkm−N0+Tσmk0,rlnƛσkmlnμσkmƛσkmTσmr,k−1zr2≤γ2∑r=k0k−1∏m=1NfƛσkmTσmr,k−1dr2.It is obvious thatμσksTσsk0,rlnƛσks/lnμσks=ƛσksTσsk0,r; therefore, we can obtain that
(44)∑r=k0k−1∏m=1Nfμσkm−N0ƛσkmTσmk0,k−1zr2≤γ2∑r=k0k−1∏m=1NfƛσkmTσmr,k−1dr2⇔∑r=k0∞∑k=r∞∏m=1Nfμσkm−N0ƛσkmTσmk0,k−1zr2≤γ2∑r=k0∞∑k=r∞∏m=1NfƛσkmTσmr,k−1dr2⇔∑r=k0∞∏m=1Nfμσkm−N0zr2∑k=r∞∏m=1NfƛσkmTσmk0,k−1≤γ2∑r=k0∞dr2∑k=r∞∏m=1NfƛσkmTσmr,k−1⇔∑r=k0∞∏m=1Nfμσkm−N0zr2∑k=r∞∏m=1NfƛσkmTσmk0,r≤γ2∑r=k0∞dr2.Settingλmax=maxλs, one has
(45)∑r=k0∞1−λmaxr−k0zr2≤γd2∑r=k0∞dr2.Together with Definition4 and Definition 5, we can conclude that the switched systems (27) are mean square asymptotically stable with weighted H∞ attenuation performance index γd=γμmaxN0.Theorem 13.
For given constantsμs>1, 0<λs<1, γ>0, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exist positive definite matrices Ps, such that
(46)Ps≤μsPr,(47)T11sT12s∗T22s<0,then the switched systems in Equation (27) with MDADT satisfying Equation (31) are mean square asymptotically stable and have prescribed weighted H∞ attenuation performance index γd, where T11s=A~sTIn−1⊗PsA~s−1−λsIn−1⊗Ps+C~sTC~s, T12s=A~sTIn−1⊗PsB~s, and T22s=B~sTIn−1⊗PsB~s−γ2Inm.Proof.
Define the Lyapunov function in Equation (48):
(48)Vsξk=ξTkIn−1⊗Psξk.Then, we can obtain the equation as follows by differential operation:(49)ΔVsξk=ξTk+1In−1⊗Psξk+1−ξTkIn−1⊗Psξk=A~sξk+B~sdkTIn−1⊗PsA~sξk+B~sdk−ξTkIn−1⊗Psξk=ξTkA~sTIn−1⊗PsA~s−In−1⊗Psξk+ξTkA~sTIn−1⊗PsB~sdk+dTkB~sTIn−1⊗PsA~sξk+dTkB~sTIn−1⊗PsB~sdk.Together with Equations (27) and (47)–(49), one has
(50)ΔVsξk+λsVsξk−Ωk=ξTkA~sTIn−1⊗PsA~s−1−λsIn−1⊗Ps+C~sTC~sξk+ξTkA~sTIn−1⊗PsB~sdk+dTkB~sTIn−1⊗PsA~sξk+dTkB~sTIn−1⊗PsB~s−γ2Inmdk=ξkdkTT11sT12s∗T22sξkdk≤0.According to Theorem12, we can conclude that the system in Equation (27) is mean-square asymptotically stable with the prescribed weighted H∞ attenuation performance index γd, which completes the proof.
(51)Λ11Λ12Λ13Λ140∗Λ22Λ23Λ240∗∗−In−1⊗PsΛ34Λ53∗∗∗−αJIn−1⊗Xs0∗∗∗∗−εsIn−1lΔ<0,(52)Ks=Xs−1Ys,where Λ11=−1−λsIn−1⊗Ps+In−1⊗C2TC2+εsL1sTL1s⊗C1TNsTNsC1,Λ12=εsL1sTΨ1Ls⊗C1TNsTNsD2, Λ22=−γ2Inm+εsLsTLs⊗D2TNsTNsD2,Λ13=In−1⊗A1TPs+L1sT⊗C1TYsTB1T, Λ23=LsTΨ1⊗D2TYsTB1T+ZcTΨ1⊗D1TPs,Λ14=αL1sT⊗C1TYsT, Λ24=αLsTΨ1⊗D2TYsT, Λ34=In−1⊗PsB1−B1Xs, and Λ53=In−1⊗PsB1Ms.Theorem 14.
For given constant scalarsμs>1, 0<λs<1, γ>0, α>0, εs>0, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exist positive definite matrices Ps∈Rp×p and matrices Xs∈Rl×l and Ys∈Rl×q satisfying (46) and (51), then the weighted H∞ consensus design problem of MASs in Equations (1)–(3) can be solved if the MDADT satisfying (31) and the parameters of control protocol in (4) can be given by Equation (52).Proof.
According to the Schur complement, it is obvious that Equation (51) is equivalent to Equation (53).
(53)Ξ=Λ11Λ12Λ13Λ14∗Λ22Λ23Λ24∗∗−In−1⊗PsΛ34∗∗∗−αJIn−1⊗Xs+εs−100Λ53000Λ530T<0,which implies that
(54)Ξ=Λ11Λ12Λ13Λ14∗Λ22Λ23Λ24∗∗−In−1⊗Ps+εs−1Λ53Λ53TΛ34∗∗∗−αJIn−1⊗Xs=ΘsUs+αVs∗−αJWs<0,where,
(55)Θs=Ts+εs−1MsMsT+εsNsTNs<,Ns=L1s⊗NsC1Ψ1TLs⊗NsD20,Ts=−1−λsIn−1⊗Ps+In−1⊗C2TC20Λ13∗−γ2InmΛ23∗∗−In−1⊗Ps,Ms=00In−1⊗PsB1Ms,Us=00In−1⊗PsB1−B1Xs,Vs=L1sT⊗C1TYsTLsTΨ1⊗D2TYsT0,Ws=In−1⊗Xs.Based on Lemma8, it can be inferred that
(56)Θs+JUsWs‐1VsT=Ts+εs−1MsMsT+εsNsTNs+JUsWs‐1VsT<0.DefiningFs=In−1⊗Fs, one can obtain Equation (57) according to Lemma 9.
(57)Ts+JMsFsNs+JUsWs‐1VsT<0.That is,(58)−1−λsIn−1⊗Ps+C~sTC~s0A~sTIn−1⊗Ps∗−γ2InmB~sTIn−1⊗Ps∗∗−In−1⊗Ps<0.Based on the Schur complement, it can be concluded that Equation (58) is equivalent to (47), which completes the proof.Remark 15.
The problem of weightedH∞ consensus design problem of MASs in Equations (1)–(3) can be converted to the problem of stability analysis and H∞ controller design for switched systems in Equation (27) by model transformation. By employing MDADT method and MLF method, the stability analysis is given in Theorem 11, and the sufficient conditions to ensure the prescribed weighted attenuation performance index are given in Theorem 12 and Theorem 13, which is less restrictive, and tighter bounds on dwell time can be obtained. The existing conditions and solutions of weighted H∞ consensus protocol is given in Theorem 14 by the aid of LMI technique.


### 3.2. Learning-Based Consensus Protocol Design

The problem of weightedH∞ consensus design of MASs is studied based on robust control theory and nonfragile control theory. However, it is well-known that the optimal transient convergence performance cannot be realized by traditional control technique. Meanwhile, the intelligent control technique, such as deep learning and deep reinforcement learning cannot ensure the convergence and stability of closed-loop system. By employing traditional control method and deep reinforcement learning, the advantages of model-based method and model-free method are combined. The online scheduling of consensus protocol can be formulated as an infinite Markov decision process. The DDPG algorithm is introduced in the framework of actor-critic to improve the consensus performance.It can be known that the reinforcement is composed of an agent and the interacting environment. The state at time instantk is defined as sk and the chosen action is set to be ak, and then, the state sk+1 and reward function rk can be obtained by interacting with the environment. In this paper, the MASs are viewed as the agent, the state sk is set to be xikuik, and the action is defined as the parameters of learning-based consensus protocol Kc,σk, which can be described as follows:
(59)sk=xikuik,ak=Kc,σk.The DDPG algorithm is proposed based on deep Q learning and actor-critic, which tries to achieve optimal policy by updating the action in continuous space based on policy gradient theorem. The DDPG algorithm is composed of two actor networks and two critic networks. The action value is approximated through the critic networkQsk,akθQ, whose weight is defined as θQ, and the learning-based consensus protocol is generated by the actor network ϖskθϖ, whose weight is set to be θϖ. The weights of critic network are updated based on Equation (60).
(60)LθQ=Es,aQsk,akθQ−y¯k2,where y¯k=rksk,ak+γpQsk+1,ϖskθϖθQ.The weights of actor network are updated as follows:(61)θϖk+1=θϖk+Lae∇θϖJ,(62)∇θϖJ=Eπ∇θϖQπsk,πskθϖθQs=sk,a=πskθϖ=Eπ∇θϖQπsk,πskθQ∇θϖπskθQ,whereLae represents the learning rate.In the DDPG algorithm, the actor target networkϖ′skθϖ′ and the critic target network Q′sk,akθQ′ are shown, whose weights are updated as follows:
(63)θϖ′k+1=Latθϖk+1−Latθϖ′k,(64)θQ′k+1=LctθQk+1+1−LctθQ′k+1,where Lat and Lct are learning rates of network ϖ′skθϖ′ and network Q′sk,akθQ′.The reward function is defined as a function of convergence error, which can be described as follows:(65)Rk=rk+γprk+1+γp2rk+2+⋯+γpKf−krKf=rk+γpRk+1,rk=ρxk−x¯k2,where ρ is the weight of reward function.In order to improve the robustness of the DDPG algorithm, exploration noiseNa is introduced as compensation of the output of control policy, which can be described as follows:
(66)ak=πskθϖ+Na.Based on the statement above, the pseudocode of learning-based consensus protocol can be summarized in Algorithm1.Algorithm 1: Online scheduling based on DDPG algorithm.
1. Define the bounds of learning-based consensus protocol.2. Design the dynamics-based consensus protocol according to Theorem14.3. Initialize the weights of actor networkϖskθϖ and critic network Qsk,akθQ with randomly generated values.4. Initialize the weights of actor target network and critic target network:θϖ′⟵θϖ, θQ′⟵θQ.5. Initialize the Replay bufferR, episode=06.forepisode=1 to M do7.  Initialize an additional noise processNa for exploration with randomly generated values.8.  Initialize the states1 and receive the initial observation.9.fort=1 to K do10.    Apply the consensus protocol generated byak=πskθϖ+Na to the environment.11.    Calculate the reward functionrk and the state vector of next time instant sk+1.12.    Store the transition pairsk,ak,rk,sk+1 in the Replay buffer R.13.      Sample a random minibatch ofN transitions sm,am,rm,sm+1 from the Replay buffer R.14.     Defineyk=rksk,ak+γpQsk+1,ϖskθϖθQ.15.     Update the critic network based on Equation (60).16.     Update the actor network according to Equations (61) and (62):17.     Update the target networks based on Equations (63) and (64).18.end for19.end for


## 3.1. Dynamics-Based Consensus Protocol Design

In this section, the stability criteria of switched systems in Equation (27) will be given, and the sufficient conditions to guarantee the prescribed H∞ attenuation performance of the switched systems will be presented. Firstly, the stability analysis of switched systems in Equation (27) will be given as follows.Theorem 11.
For given constant scalarsμs>1, 0<λs<1, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exists the Lyapunov function Vsξk and class K∞ functions κ1s and κ2s, then the switched systems in Equation (27) are mean square asymptotically stable such that
(28)κ1sξk≤Vsξk≤κ2sξk,(29)Vsξk≤μsVrξk,(30)ΔVsξk≤−λsVsξk,(31)τas≥τas∗=−lnμsln1−λs.Proof.
It is supposed that the switching instants in the intervalk0,k are defined as k1, k2,…, kt with kt+1=k. Then, Equation (32) can be obtained based on Equation (30) when dik≡0.
(32)Vσktξk≤1−λσktk−ktVσktξkt.
Substituting Equation (32) in Equation (29), one has
(33)Vσktξk≤μσkt1−λσktk−ktVσktξkt.
According to Equations (32) and (33), the equation can be obtained as follows by iteration.
(34)Vσktξk≤μσktƛσktk−ktVσktξkt≤μσktƛσktk−ktμσkt−1ƛσkt−1kt−kt−1Vσkt−1ξkt−1⋯≤∏r=0tμσkrƛσkrkr+1−krVσk0ξk0≤∏s=1NfμσksNσ,sk0,kƛσksTσsk0,kVσk0ξk0,where ƛs=1−λs; then, we can obtain Equation (35) based on Definition 3.
(35)Vσktξk≤exp∑s=1NfNσ,sk0,klnμσks+Tσsk0,klnƛσksVσk0ξk0≤exp∑s=1NfTσsk0,kτas+N0lnμσks+Tσsk0,klnƛσksVσk0ξk0≤μσksN0exp∑s=1NfTσsk0,klnμσksτas+lnƛσksVσk0ξk0.
Then, Equation (36) can be obtained according to Equation (28).
(36)ξk2≤κ1−1κ2μσksN0exp∑s=1NfTσsk0,klnμσksτas+lnƛσksξk02.
Therefore, it is obvious that the switched systems with MDADT satisfying Equation (31) are mean square asymptotically stable, which completes the proof.The stability analysis of switched systems in Equation (27) is presented in Theorem 11, and the sufficient conditions to guarantee the prescribed attenuation performance index will be given in Theorem 12 and Theorem 13.Theorem 12.
For given constant scalarsμs>1, 0<λs<1, γ>0, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exist the Lyapunov function Vsξk and class K∞ function κ1s and κ2s satisfying Equations (28) and (29), such that
(37)ΔVsξk+λsVsξk−Ωk≤0,then the switched systems in Equation (27) are mean square asymptotically stable with weighted H∞ attenuation performance index γd.
whereΩk=−zk2+γ2dk2, γd=γμmaxN0, and μmax=maxμs.Proof.
According to Equation (29), we have
(38)Vσktξk≤ƛσktk−ktVσktξkt+∑r=ktk−1ƛσktk−r−1Ωr≤μσktƛσktk−ktVσktξkt+∑r=ktk−1ƛσktk−r−1Ωr≤μσktƛσktk−ktμσkt−1ƛσkt−1kt−kt−1Vσkt−1ξkt−1+∑r=kt−1kt−1ƛσkt−1k−r−1Ωr+∑r=ktk−1ƛσktk−r−1Ωr⋯≤∏r=0tμσkrƛσkrkr+1−krVσk0ξk0+μσk0ƛσk0k1−k0⋯μσktƛσktk−kt∑r=k0k1−1ƛσk0k−r−1Ωr+μσk1ƛσk1k2−k1⋯μσktƛσktk−kt∑r=k1k2−1ƛσk1k−r−1Ωr+⋯+∑r=ktk−1ƛσktk−r−1Ωr≤∏s=1NfμσksNσ,sk0,kƛσksTσsk0,kVσk0ξk0+∑r=k0k−1∏m=1m≠nNfμσkmNσ,mr,kμσknNσ,nr,kƛσkmTσmr,kƛσknTσnr,k−1Ωr,where n represents the activated connected graph in rth step and m denotes the inactivated connected graph in rth step.Based on the fact0<ƛs<1, one has
(39)Vσktξk≤∏s=1NfμσksNσ,sk0,kƛσksTσsk0,kVσk0ξk0+∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1Ωr.Considering the zero-initial condition andVσktξk≥0, it is obvious that
(40)∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1Ωr≥0⇔∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1zr2≤γ2∑r=k0k−1∏m=1NfμσkmNσ,mr,kƛσkmTσmr,k−1dr2.Multiplying both sides of Equation (40) by ∏m=1Nfμσkm−Nσ,mk0,k, we can obtain that
(41)∑r=k0k−1∏m=1Nfμσkm−Nσ,mk0,rƛσkmTσmr,k−1zr2≤γ2∑r=k0k−1∏m=1Nfμσkm−Nσ,mk0,rƛσkmTσmr,k−1dr2.Together with Definition3 and Equation (31), it can be inferred that
(42)0≤Nσ,sk0,r≤N0+Tσsk0,rτas∗≤N0−Tσsk0,rlnƛσkslnμσks.Combining with the factμσkm>1 and Equation (42), one has
(43)∑r=k0k−1∏m=1Nfμσkm−N0+Tσmk0,rlnƛσkmlnμσkmƛσkmTσmr,k−1zr2≤γ2∑r=k0k−1∏m=1NfƛσkmTσmr,k−1dr2.It is obvious thatμσksTσsk0,rlnƛσks/lnμσks=ƛσksTσsk0,r; therefore, we can obtain that
(44)∑r=k0k−1∏m=1Nfμσkm−N0ƛσkmTσmk0,k−1zr2≤γ2∑r=k0k−1∏m=1NfƛσkmTσmr,k−1dr2⇔∑r=k0∞∑k=r∞∏m=1Nfμσkm−N0ƛσkmTσmk0,k−1zr2≤γ2∑r=k0∞∑k=r∞∏m=1NfƛσkmTσmr,k−1dr2⇔∑r=k0∞∏m=1Nfμσkm−N0zr2∑k=r∞∏m=1NfƛσkmTσmk0,k−1≤γ2∑r=k0∞dr2∑k=r∞∏m=1NfƛσkmTσmr,k−1⇔∑r=k0∞∏m=1Nfμσkm−N0zr2∑k=r∞∏m=1NfƛσkmTσmk0,r≤γ2∑r=k0∞dr2.Settingλmax=maxλs, one has
(45)∑r=k0∞1−λmaxr−k0zr2≤γd2∑r=k0∞dr2.Together with Definition4 and Definition 5, we can conclude that the switched systems (27) are mean square asymptotically stable with weighted H∞ attenuation performance index γd=γμmaxN0.Theorem 13.
For given constantsμs>1, 0<λs<1, γ>0, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exist positive definite matrices Ps, such that
(46)Ps≤μsPr,(47)T11sT12s∗T22s<0,then the switched systems in Equation (27) with MDADT satisfying Equation (31) are mean square asymptotically stable and have prescribed weighted H∞ attenuation performance index γd, where T11s=A~sTIn−1⊗PsA~s−1−λsIn−1⊗Ps+C~sTC~s, T12s=A~sTIn−1⊗PsB~s, and T22s=B~sTIn−1⊗PsB~s−γ2Inm.Proof.
Define the Lyapunov function in Equation (48):
(48)Vsξk=ξTkIn−1⊗Psξk.Then, we can obtain the equation as follows by differential operation:(49)ΔVsξk=ξTk+1In−1⊗Psξk+1−ξTkIn−1⊗Psξk=A~sξk+B~sdkTIn−1⊗PsA~sξk+B~sdk−ξTkIn−1⊗Psξk=ξTkA~sTIn−1⊗PsA~s−In−1⊗Psξk+ξTkA~sTIn−1⊗PsB~sdk+dTkB~sTIn−1⊗PsA~sξk+dTkB~sTIn−1⊗PsB~sdk.Together with Equations (27) and (47)–(49), one has
(50)ΔVsξk+λsVsξk−Ωk=ξTkA~sTIn−1⊗PsA~s−1−λsIn−1⊗Ps+C~sTC~sξk+ξTkA~sTIn−1⊗PsB~sdk+dTkB~sTIn−1⊗PsA~sξk+dTkB~sTIn−1⊗PsB~s−γ2Inmdk=ξkdkTT11sT12s∗T22sξkdk≤0.According to Theorem12, we can conclude that the system in Equation (27) is mean-square asymptotically stable with the prescribed weighted H∞ attenuation performance index γd, which completes the proof.
(51)Λ11Λ12Λ13Λ140∗Λ22Λ23Λ240∗∗−In−1⊗PsΛ34Λ53∗∗∗−αJIn−1⊗Xs0∗∗∗∗−εsIn−1lΔ<0,(52)Ks=Xs−1Ys,where Λ11=−1−λsIn−1⊗Ps+In−1⊗C2TC2+εsL1sTL1s⊗C1TNsTNsC1,Λ12=εsL1sTΨ1Ls⊗C1TNsTNsD2, Λ22=−γ2Inm+εsLsTLs⊗D2TNsTNsD2,Λ13=In−1⊗A1TPs+L1sT⊗C1TYsTB1T, Λ23=LsTΨ1⊗D2TYsTB1T+ZcTΨ1⊗D1TPs,Λ14=αL1sT⊗C1TYsT, Λ24=αLsTΨ1⊗D2TYsT, Λ34=In−1⊗PsB1−B1Xs, and Λ53=In−1⊗PsB1Ms.Theorem 14.
For given constant scalarsμs>1, 0<λs<1, γ>0, α>0, εs>0, and s=σk:0,∞⟶R=1,2,⋯,Nf, if there exist positive definite matrices Ps∈Rp×p and matrices Xs∈Rl×l and Ys∈Rl×q satisfying (46) and (51), then the weighted H∞ consensus design problem of MASs in Equations (1)–(3) can be solved if the MDADT satisfying (31) and the parameters of control protocol in (4) can be given by Equation (52).Proof.
According to the Schur complement, it is obvious that Equation (51) is equivalent to Equation (53).
(53)Ξ=Λ11Λ12Λ13Λ14∗Λ22Λ23Λ24∗∗−In−1⊗PsΛ34∗∗∗−αJIn−1⊗Xs+εs−100Λ53000Λ530T<0,which implies that
(54)Ξ=Λ11Λ12Λ13Λ14∗Λ22Λ23Λ24∗∗−In−1⊗Ps+εs−1Λ53Λ53TΛ34∗∗∗−αJIn−1⊗Xs=ΘsUs+αVs∗−αJWs<0,where,
(55)Θs=Ts+εs−1MsMsT+εsNsTNs<,Ns=L1s⊗NsC1Ψ1TLs⊗NsD20,Ts=−1−λsIn−1⊗Ps+In−1⊗C2TC20Λ13∗−γ2InmΛ23∗∗−In−1⊗Ps,Ms=00In−1⊗PsB1Ms,Us=00In−1⊗PsB1−B1Xs,Vs=L1sT⊗C1TYsTLsTΨ1⊗D2TYsT0,Ws=In−1⊗Xs.Based on Lemma8, it can be inferred that
(56)Θs+JUsWs‐1VsT=Ts+εs−1MsMsT+εsNsTNs+JUsWs‐1VsT<0.DefiningFs=In−1⊗Fs, one can obtain Equation (57) according to Lemma 9.
(57)Ts+JMsFsNs+JUsWs‐1VsT<0.That is,(58)−1−λsIn−1⊗Ps+C~sTC~s0A~sTIn−1⊗Ps∗−γ2InmB~sTIn−1⊗Ps∗∗−In−1⊗Ps<0.Based on the Schur complement, it can be concluded that Equation (58) is equivalent to (47), which completes the proof.Remark 15.
The problem of weightedH∞ consensus design problem of MASs in Equations (1)–(3) can be converted to the problem of stability analysis and H∞ controller design for switched systems in Equation (27) by model transformation. By employing MDADT method and MLF method, the stability analysis is given in Theorem 11, and the sufficient conditions to ensure the prescribed weighted attenuation performance index are given in Theorem 12 and Theorem 13, which is less restrictive, and tighter bounds on dwell time can be obtained. The existing conditions and solutions of weighted H∞ consensus protocol is given in Theorem 14 by the aid of LMI technique.


## 3.2. Learning-Based Consensus Protocol Design

The problem of weightedH∞ consensus design of MASs is studied based on robust control theory and nonfragile control theory. However, it is well-known that the optimal transient convergence performance cannot be realized by traditional control technique. Meanwhile, the intelligent control technique, such as deep learning and deep reinforcement learning cannot ensure the convergence and stability of closed-loop system. By employing traditional control method and deep reinforcement learning, the advantages of model-based method and model-free method are combined. The online scheduling of consensus protocol can be formulated as an infinite Markov decision process. The DDPG algorithm is introduced in the framework of actor-critic to improve the consensus performance.It can be known that the reinforcement is composed of an agent and the interacting environment. The state at time instantk is defined as sk and the chosen action is set to be ak, and then, the state sk+1 and reward function rk can be obtained by interacting with the environment. In this paper, the MASs are viewed as the agent, the state sk is set to be xikuik, and the action is defined as the parameters of learning-based consensus protocol Kc,σk, which can be described as follows:
(59)sk=xikuik,ak=Kc,σk.The DDPG algorithm is proposed based on deep Q learning and actor-critic, which tries to achieve optimal policy by updating the action in continuous space based on policy gradient theorem. The DDPG algorithm is composed of two actor networks and two critic networks. The action value is approximated through the critic networkQsk,akθQ, whose weight is defined as θQ, and the learning-based consensus protocol is generated by the actor network ϖskθϖ, whose weight is set to be θϖ. The weights of critic network are updated based on Equation (60).
(60)LθQ=Es,aQsk,akθQ−y¯k2,where y¯k=rksk,ak+γpQsk+1,ϖskθϖθQ.The weights of actor network are updated as follows:(61)θϖk+1=θϖk+Lae∇θϖJ,(62)∇θϖJ=Eπ∇θϖQπsk,πskθϖθQs=sk,a=πskθϖ=Eπ∇θϖQπsk,πskθQ∇θϖπskθQ,whereLae represents the learning rate.In the DDPG algorithm, the actor target networkϖ′skθϖ′ and the critic target network Q′sk,akθQ′ are shown, whose weights are updated as follows:
(63)θϖ′k+1=Latθϖk+1−Latθϖ′k,(64)θQ′k+1=LctθQk+1+1−LctθQ′k+1,where Lat and Lct are learning rates of network ϖ′skθϖ′ and network Q′sk,akθQ′.The reward function is defined as a function of convergence error, which can be described as follows:(65)Rk=rk+γprk+1+γp2rk+2+⋯+γpKf−krKf=rk+γpRk+1,rk=ρxk−x¯k2,where ρ is the weight of reward function.In order to improve the robustness of the DDPG algorithm, exploration noiseNa is introduced as compensation of the output of control policy, which can be described as follows:
(66)ak=πskθϖ+Na.Based on the statement above, the pseudocode of learning-based consensus protocol can be summarized in Algorithm1.Algorithm 1: Online scheduling based on DDPG algorithm.
1. Define the bounds of learning-based consensus protocol.2. Design the dynamics-based consensus protocol according to Theorem14.3. Initialize the weights of actor networkϖskθϖ and critic network Qsk,akθQ with randomly generated values.4. Initialize the weights of actor target network and critic target network:θϖ′⟵θϖ, θQ′⟵θQ.5. Initialize the Replay bufferR, episode=06.forepisode=1 to M do7.  Initialize an additional noise processNa for exploration with randomly generated values.8.  Initialize the states1 and receive the initial observation.9.fort=1 to K do10.    Apply the consensus protocol generated byak=πskθϖ+Na to the environment.11.    Calculate the reward functionrk and the state vector of next time instant sk+1.12.    Store the transition pairsk,ak,rk,sk+1 in the Replay buffer R.13.      Sample a random minibatch ofN transitions sm,am,rm,sm+1 from the Replay buffer R.14.     Defineyk=rksk,ak+γpQsk+1,ϖskθϖθQ.15.     Update the critic network based on Equation (60).16.     Update the actor network according to Equations (61) and (62):17.     Update the target networks based on Equations (63) and (64).18.end for19.end for


## 4. Numerical Example

In this section, the numerical example is given to validate the effectiveness and superiority of the proposed method. The discrete-time MASs are taken into consideration, where each agent is modeled as follows:(67)A1=1.31.4−1.9−1.9,B1=1001,C1=1001,C2=1001,D1=0.20.3,D2=0.10.1.The external disturbance is defined as follows:(68)dk=0.1e−0.1kcos0.3k.The possible interaction topologies are presented in Figures1–3, and the Laplace matrices can be described in Equations (69)–(71). The switching logic is depicted in Figure 4. The parameters of switching topologies subject to ADT and MDADT are listed in Table 1. Accordingly, we can see that tighter bounds on dwell time and less restrictive results of consensus can be obtained. The ADT switching topologies can be viewed as a special case of MDADT switching topologies with τa∗≥τai∗. Based on Theorem 13, we can ensure that the H∞ consensus subject to MDADT switching topologies and the parameters of dynamics-based consensus protocol can be obtained based on Theorem 14.
(69)L1=100−10−110000−110000−12−100−101,(70)L2=100−10−1200−10−110000−12−1−10−102,(71)L3=2−100−101−100−10100−10−120000−11.Figure 1
Interaction topologyG1.Figure 2
Interaction topologyG2.Figure 3
Interaction topologyG3.Figure 4
The switching logic.Table 1
Comparison between MDADT switching topologies and ADT switching topologies in [27].
Switching logicParameterResultADTμ=1.15,λ=0.2τa∗=0.6263MDADTμ1=1.1λ1=0.3τa1∗=0.2672μ2=1.05λ2=0.25τa2∗=0.1696μ3=1.15λ3=0.2τa3∗=0.6263The responses of MASs subject to ADT switching topologies and MDADT switching topologies are given in Figures5–6, where the response of x subject to ADT switching is represented in Figure 5 and the response of x subject to MDADT switching topologies is described in Figure 6. We can see that the performance of MDADT method is better than that of ADT method.Figure 5
The response ofx subject to ADT switching topologies.
(a)
Response ofx1(b)
Response ofx2Figure 6
The response ofx subject to MDADT switching topologies.
(a)
Response ofx1(b)
Response ofx2Moreover, to demonstrate the superiority of the proposed method, the simulation results of the proposed method in this paper are shown in Figures7–10. The response of x with consensus protocol in this paper is depicted in Figure 7, and the response of attenuation performance is shown in Figure 8. We can see that the combination of dynamics-based consensus protocol and learning-based consensus protocol can ensure that the H∞ consensus and the transient performance can be improved by DDPG algorithm. The episode reward is shown in Figure 9; we can conclude that the DDPG algorithm can converge to a neighbor of the origin. The reward function in one episode is depicted in Figure 10, from which we can see that the transient performance and steady performance can be improved by DDPG algorithm.Figure 7
The response ofx of proposed method.
(a)
Response ofx1(b)
Response ofx2Figure 8
The response of attenuation performance of proposed method.Figure 9
The response of episode reward.Figure 10
The response of episode reward.In summary, according to the simulation results, we can see that the advantages of model-based method and model-free method are combined forH∞ consensus design of MASs in this paper. Firstly, compared to the existing literatures on traditional robust consensus protocol design, it can be inferred that tighter bounds on dwell time can be obtained by MDADT method. The online scheduling is realized, and the transient performance can be improved by employing DDPG algorithm. In addition, it is worth to mention that the stability of deep reinforcement learning cannot be guaranteed in theory in most results. The suggested method can ensure the H∞ consensus and achieve optimal policy simultaneously, which is proposed firstly.


## 5. Conclusions

The problem ofH∞ consensus design and online scheduling for MASs with switching topologies is studied in this paper based on robust control theory and deep reinforcement learning. The switching of topologies is viewed as the switching between subsystems, and then, the switched model of MASs is established. To convenient the designing process, a model transformation method is applied so that the problem of H∞ consensus design can be formulated as the problem of H∞ control for reduced-order switched systems. To ensure the H∞ consensus and achieve optimal consensus policy, the protocol is composed of two parts: the dynamics-based consensus protocol, which is proposed based on robust control theory, and the learning-based consensus protocol, which is developed by DDPG algorithm in the framework of actor-critic and can be viewed as compensation of dynamics-based consensus protocol to realize online scheduling. By employing MDADT method and MLF method, the stability analysis criteria and sufficient conditions to guarantee weighted H∞ attenuation performance are given and the solutions of dynamics-based protocol are presented by a series of solvable LMIs. The online scheduling is formulated as a Markov decision process, and the DDPG algorithm is proposed to explore the optimal policy, which can be viewed as supplement compensation, and the stability of DDPG algorithm can be guaranteed by nonfragile control theory. Numerical example in the end is given to show the effectiveness and superiority of proposed method.



---

*Source: 2650632-2022-03-15.xml*