# A Stable Distributed Neural Controller for Physically Coupled Networked Discrete-Time System via Online Reinforcement Learning

Authors: Jian Sun; Jie Li
Journal: Complexity
 (2018)

---

## Abstract

The large scale, time varying, and diversification of physically coupled networked infrastructures such as power grid and transportation system lead to the complexity of their controller design, implementation, and expansion. For tackling these challenges, we suggest an online distributed reinforcement learning control algorithm with the one-layer neural network for each subsystem or called agents to adapt the variation of the networked infrastructures. Each controller includes a critic network and action network for approximating strategy utility function and desired control law, respectively. For avoiding a large number of trials and improving the stability, the training of action network introduces supervised learning mechanisms into reduction of long-term cost. The stability of the control system with learning algorithm is analyzed; the upper bound of the tracking error and neural network weights are also estimated. The effectiveness of our proposed controller is illustrated in the simulation; the results indicate the stability under communication delay and disturbances as well.


---

## Body

## 1. Introduction

The increasing interconnection of physical systems through cybernetworks or physical networks has been observed in many infrastructures, such as power grid [1, 2], transportation networks, and unmanned systems. One critical issue of these called cyberphysical systems is complexity of the system when it grows very large, especially the control problem. Consequently, distributed schemes are suggested for reducing the communication and computational cost compared with centralized control scheme [3]. However, the coupling of subsystems and nonstatic environment in both cybernetworks and physics networks bring many challenges, such as physical interference among subsystems, time-varying plant parameters, communication delay, and expansibility of the cyberphysical system.To increase expansibility of the cyberphysical system, the multiagent concept is usually introduced. The cyberphysical system can be divided into many agents. Each agent has its own control policy and a unified framework for pursuing its target [4]. The expansion of the cyberphysical system turns into simply duplicating agents without accommodating control policy. To deal with the physical coupling of networked system, one common approach is to decouple subsystems in control design [5–8]. Each subsystem may utilize state information of neighbored subsystems for mitigating their physical interference, or the designer treats their physical interference as random disturbance [9, 10]. On the other hand, for addressing nonstatic environment with time-varying plants, online supervised learning, adaptive control, and reinforcement learning algorithm are suggested; they all enable adaptively adjusting their control parameters online, while the combination of neural network and reinforcement learning usually leads to better control performance compared with conventional supervised learning and adaptive control scheme [11]. Reinforcement learning constructs a long-run cost-to-go function to predict the consequence cost;, each control action takes the estimated future result into account [12], while, compared with adaptive control, the adaptive ability is limited in the number of time-varying parameters; the number of time-varying parameters of plant model may very large in practice.Recently, many researches are focused on reinforcement learning with neural network. These researches are classified into two categories. The first category is to simply utilize neural network to approximate unknown part about system model or control strategy, such as cost-to-go function and optimal control law. Prokhorov and Wunsch discussed three families of reinforcement learning control design [13], Heuristic dynamic programming (HDP), dual heuristic programming (DHP), and globalized dual heuristic programming (GDHP) and their application in optimal control. Xu et al. focus on experimental studies of real-time online learning control for nonlinear systems using kernel-based ADP methods [14]. Lee et al. focus on a class of reinforcement learning (RL) algorithms, named integral RL (I-RL), that solve continuous-time (CT) nonlinear optimal control problems with input affine system dynamics [15]. The second category is to combine the approach in the first category with supervised learning algorithm for guaranteeing convergence of the learning system; the supervised reinforcement learning also reduces a large number of trials by employing the error signal with domain knowledge [16–18]. It generates instinct feedback for correcting the control actions. Xu et al. suggest a novel adaptive-critic-based neural network (NN) controller which is investigated for nonlinear pure-feedback systems [19]. Liu et al. were concerned with a reinforcement learning-based adaptive tracking control technique to tolerate faults for a class of unknown multiple-input multiple-output nonlinear discrete-time systems with less learning parameters [20]. Besides these, researchers try to employing multilayer/deep neural network for approximating the functions in control, so that the precision of model is enhanced and the performance can be improved in a consequence [21, 22]. However, it is hard to analyze its stability of learning algorithm. Moreover, the learning rate may be slow as the number of tuned parameters is very large in the deep neural network [23].In this paper, we suggest a distributed neural controller for the physically coupled networked discrete-time system via online reinforcement learning. We model each subsystem as an agent; each agent can obtain its state and some physical neighbored subsystem state information to figure out optimal control action. One-layer adaptive critic neural network and action neural network are proposed for modeling the cost function and optimal action law. With deterministic learning algorithm, we incorporated supervised learning into our reinforcement learning algorithm for accelerating convergence rate. The stability of the learning algorithm is analyzed and the boundary of each parameter is also estimated. The contribution of this paper is two-fold.(1) We propose a distributed online reinforcement learning algorithm for controlling physically coupled networked discrete-time system.(2) Sufficient condition for guaranteeing learning algorithm stability and system stability are derived and the upper bound of parameters is estimated.The rest of the paper is organized as follows: We model the physically coupled networked system and control system in a mathematical dynamic equation in Section2, and some assumptions are made for simplifying the analysis; then, control system design via online reinforcement learning algorithm is depicted in Section 3; the stability analysis is detailedly discussed in Section 4; simulation results for illustrating the effectiveness and advantage of our algorithm are elaborated in Section 5. Section 6 is the conclusion part.


## 2. Physically Coupled Networked Control System and Problem Statement

In the physically coupled networked system, their subsystems may physically interfere with neighbored subsystems and change its state trajectory or dynamic. The structure is shown in Figure1. In order to improve the control system performance, some cyberconnections of communication infrastructures are installed for exchanging the states of neighbored subsystems [3]. The topology of cyberconnections and physical connections may not be the same for probably practical constraints in cyberresources.Figure 1
A physical-coupling networked system structure.


### 2.1. System Dynamic Equation

For a physically coupled networked system, consider that it consists ofn nonlinear dynamic subsystems, which are given in the discrete-time form:(1)xi,1k+1=xi,1k,xi,2k+1=xi,3k,⋮xi,pk+1=fixik+∑j∈Npifjixj,pk+gixikuik+dik,where i=1,…,n, xi,l∈Rm×1, l=1,2,…,p, and xi=xi,1T,…,xi,pTT∈Rpm×1. xi,ui∈Rq×1 and di∈Rm×1 are system state vector, control input vector, and disturbance vector for subsystem i. fixik∈Rm×1, fjixj,pk∈Rm×1, and gixik∈Rm×q are smooth vector function about local system dynamic, neighbor interference, and control input interference which are all unknown. Npi is physically connected neighbor set of subsystems i, which can interfere with the state trajectory of subsystem i. In order to simplify the analysis, the following reasonable assumptions are made [11].Assumption 1.
The disturbances are boundeddi≤dmax.Assumption 2.
g
i is an invertible matrix.d
m
a
x is a positive real number, it means the magnitude of disturbances are bounded. Assumption 2 is made for simplifying the analysis of action network which will be discussed in next section.The control objective is to track the state target vectoryi; then we have the error equation(2)ei,lk=xi,lk-yi,dk+l-p.Therefore, the subsystem dynamic in a form of error is(3)ei,1k+1=ei,2k,ei,2k+1=ei,3k,⋮ei,pk+1=fixik+∑j∈Npifjixj,pk+gixikuik+dik-yi,pk+1.


### 2.2. Distributed Control System and Control Objective

Distributed control system is more flexible and scalable than centralized control. Moreover, it divides a large system controller into many small subsystems controllers, which lead to the system state dimension reduction in a controller, so that much computational resource and time can be saved [24].The control objective is to decrease the error vectorei as fast as possible and bound in a small region for a given bounded disturbance. For subsystem controller, usually, an exponential damping rate of error is expected with a form of(4)eik+1=Γieik,where Γi<1. Therefore, the desired control input of subsystem i can be in a form of(5)ud,ik=-gixik-1∑j∈Npi∩Ncifjixj,pk+fixik-yi,pk+1-Γiei,pk.Nci is the cyberconnected neighbor set of subsystems i, which means the controller of subsystem i utilizes the received state information from neighbored subsystems via communication network.However,fixik, fjixj,pk, and gixik are unknown. A reinforcement learning scheme with neural network is proposed for approximating the desired control strategy and strategy utility function about long-term cost.


## 2.1. System Dynamic Equation

For a physically coupled networked system, consider that it consists ofn nonlinear dynamic subsystems, which are given in the discrete-time form:(1)xi,1k+1=xi,1k,xi,2k+1=xi,3k,⋮xi,pk+1=fixik+∑j∈Npifjixj,pk+gixikuik+dik,where i=1,…,n, xi,l∈Rm×1, l=1,2,…,p, and xi=xi,1T,…,xi,pTT∈Rpm×1. xi,ui∈Rq×1 and di∈Rm×1 are system state vector, control input vector, and disturbance vector for subsystem i. fixik∈Rm×1, fjixj,pk∈Rm×1, and gixik∈Rm×q are smooth vector function about local system dynamic, neighbor interference, and control input interference which are all unknown. Npi is physically connected neighbor set of subsystems i, which can interfere with the state trajectory of subsystem i. In order to simplify the analysis, the following reasonable assumptions are made [11].Assumption 1.
The disturbances are boundeddi≤dmax.Assumption 2.
g
i is an invertible matrix.d
m
a
x is a positive real number, it means the magnitude of disturbances are bounded. Assumption 2 is made for simplifying the analysis of action network which will be discussed in next section.The control objective is to track the state target vectoryi; then we have the error equation(2)ei,lk=xi,lk-yi,dk+l-p.Therefore, the subsystem dynamic in a form of error is(3)ei,1k+1=ei,2k,ei,2k+1=ei,3k,⋮ei,pk+1=fixik+∑j∈Npifjixj,pk+gixikuik+dik-yi,pk+1.


## 2.2. Distributed Control System and Control Objective

Distributed control system is more flexible and scalable than centralized control. Moreover, it divides a large system controller into many small subsystems controllers, which lead to the system state dimension reduction in a controller, so that much computational resource and time can be saved [24].The control objective is to decrease the error vectorei as fast as possible and bound in a small region for a given bounded disturbance. For subsystem controller, usually, an exponential damping rate of error is expected with a form of(4)eik+1=Γieik,where Γi<1. Therefore, the desired control input of subsystem i can be in a form of(5)ud,ik=-gixik-1∑j∈Npi∩Ncifjixj,pk+fixik-yi,pk+1-Γiei,pk.Nci is the cyberconnected neighbor set of subsystems i, which means the controller of subsystem i utilizes the received state information from neighbored subsystems via communication network.However,fixik, fjixj,pk, and gixik are unknown. A reinforcement learning scheme with neural network is proposed for approximating the desired control strategy and strategy utility function about long-term cost.


## 3. Control System Design by Reinforcement Learning and Neural Network

The proposed distributed control scheme with reinforcement learning consists of three parts: the first part will introduce a strategy utility function (also called long-term cost function); the second part depicts the critic neural network and online training algorithm; the last part of this section elaborates the action neural network and parameter updating algorithm.


### 3.1. Strategy Utility Function

The utility function defined for subsystemi is based on the current filtered state error eik; it is formulated as(6)pilk=0,if⁡eilk2≤cil1,otherwise,where l=1,2,…,mp, pilk∈R, and cil is a given constant positive scalar threshold forlth element of state error vector ei for subsystem i. pilk is also an indicator of current tracking performance; if pilk equals 1, it means the control system has a bad state, and the state deviates the desired value a lot. On the other hand, if pilk equals 0, it indicates well-tracking performance and thelth state error is in a small bounded region.The long-term cost is the sum of utility function at each sampling time. Based on the utility functionpilk, strategic utility function is defined as(7)Jik=αNpik+1+αN-1pik+2+⋯+αk+1piN+⋯,where 0<α<1, pik∈Rmp, andN is stage number. If N is infinite or very large, the strategy utility function is defined in a rolling horizon with a fixed number of stages. It is obvious that the control objective is to minimize Jik which improve the control performance.


### 3.2. Critic Network Design

In our proposed scheme, one-layer neural network is considered for approximating strategy utility functionJi. For simplifying the stability analysis, only output layer weights of neural network are designed to be adjustable in online training. A one-layer network is suggested to approximate strategy utility function; it is(8)Jik=Wc,ikϕc,ik-τ+εc,ik.The basis function ϕc,ik-τ is a Gaussian vector function which is defined as(9)ϕc,ik-τ=ϕc,i,1,ϕc,i,2,…,ϕc,i,h,…,ϕc,i,q,ϕc,i,hk-τ=exp⁡-eik-τ-cc,i,h2σc,i2,where τ is communication latency, cc,i,h∈Rmp is the Gaussian function center vector, and the centers should cover the system operation state region as much as possible. σc,i is width of Gaussian function. The approximation error εc,i would be very small if the dimension of basis function ϕc,i∈Rq×1 is large enough [11]. The relation betweenkth and k+1th optimal control action is(10)Jik=minuikαJik-1-αN+1pik,where uik∈Rm is control action for subsystem i. We estimate the strategy utility function by(11)J^ik=W^c,ikϕc,ik-τ.The prediction error of approximated strategy utility function J^i for critic NN is(12)ec,ik=J^ik-αJ^ik-1-αNpik.We define the objective function of critic NN for minimization at kth sampling as(13)Ea,ik=12ea,iTkea,ik.One common way to decrease the objective function is to update critic NN parameters along its gradient direction. Applying chain rule, partial derivative of objective function (13) with respect to W^c,ik is(14)∂Ea,ik∂W^c,ik=ec,ikϕc,ikT=J^ik-αJ^ik-1-αNpikϕc,ik-τT.Therefore, updating law for critic NN of subsystem i is(15)W^c,ik+1=W^c,ik-δiJ^ik-αJ^ik-1-αNpikϕc,ik-τT.δi is a given scalar, representing updating step size. The choice of δi is very important. If δi is too large, the online learning may diverge.


### 3.3. Action Neural Network Design

Our control objective is to minimize the tracking errorei and also to minimize the long-term cost function/strategy utility function Ji. They depend on the control action in each step. The desired control action (5) is an expected strategy for approaching this objective, and an action neural network is suggested for approximating the desired control action. The desired control action ud,i can be equal to(16)ud,ik=Wa,ikϕa,ik-τ+εa,ik,where Wa,i is the optimal weighting matrix for neural output which minimizes the residual εa,i; ϕa,i∈Rl×1 is the basis function which has the same form as (9). εa,i would be very small if the dimension of ϕa,i is very large. However, ud,i and Wa,i are unknown; the desired control action is proposed to be estimated by(17)u^ik=W^a,iϕa,ik-τ,where W^a,i is the estimated weighting matrix for Wa,i. And we have the estimated error u~i for desired control action.(18)u~ik=u^ik-ud,i=W~a,iϕa,ik-τ-εa,ik,where W~a,i=W^a,i-Wa,i, and we denote ςik=W~a,iϕa,ik-τ, which causes dynamic (3) to be(19)ei,1k+1=ei,2k,ei,2k+1=ei,3k,⋮ei,pk+1=Γiei,pk+giςik+∑j∈Npi\Ncifjixj,pk-giεik+dik.Npi and Nci are the neighbor subsystem sets of subsystem i which are connected to subsystem i in physical way and cyberway. In our proposed scheme, supervised learning is incorporated into the action neural network training for accelerating the convergence rate of online updating. The objective of the policy is not only to minimize long-term cost Ji but also to approximate the desired control output ud,i with supervised learning. Thus, the error vector of action network is defined as(20)eak=giςik+gi-1J^ik-Jd,i,where Jd,i is the desired utility function value for subsystem i, it can be set as 0 [20], and gi is principal mean square root. The following cost function is defined for each step:(21)Ea,ik=12ea,ikTea,ik.Then, the partial derivative of (21) with respect to W~a,i is obtained by chain rule.(22)∂Ea,ik∂W~a,ik=∂Ea,ik∂ea,ik∂ea,ik∂ςik∂ςik∂W~a,ik=ϕa,ikgiςik+J^ikT.Therefore, with gradient descent principle, the action NN weight matrix is updated by(23)W~a,ik+1=W~a,ik-βigiςik+J^ikϕa,ik-τT.βi is the updating step size for online learning of action neural network. The choice of βi will be discussed in the next section, which is associated with the stability of the online learning algorithm.


## 3.1. Strategy Utility Function

The utility function defined for subsystemi is based on the current filtered state error eik; it is formulated as(6)pilk=0,if⁡eilk2≤cil1,otherwise,where l=1,2,…,mp, pilk∈R, and cil is a given constant positive scalar threshold forlth element of state error vector ei for subsystem i. pilk is also an indicator of current tracking performance; if pilk equals 1, it means the control system has a bad state, and the state deviates the desired value a lot. On the other hand, if pilk equals 0, it indicates well-tracking performance and thelth state error is in a small bounded region.The long-term cost is the sum of utility function at each sampling time. Based on the utility functionpilk, strategic utility function is defined as(7)Jik=αNpik+1+αN-1pik+2+⋯+αk+1piN+⋯,where 0<α<1, pik∈Rmp, andN is stage number. If N is infinite or very large, the strategy utility function is defined in a rolling horizon with a fixed number of stages. It is obvious that the control objective is to minimize Jik which improve the control performance.


## 3.2. Critic Network Design

In our proposed scheme, one-layer neural network is considered for approximating strategy utility functionJi. For simplifying the stability analysis, only output layer weights of neural network are designed to be adjustable in online training. A one-layer network is suggested to approximate strategy utility function; it is(8)Jik=Wc,ikϕc,ik-τ+εc,ik.The basis function ϕc,ik-τ is a Gaussian vector function which is defined as(9)ϕc,ik-τ=ϕc,i,1,ϕc,i,2,…,ϕc,i,h,…,ϕc,i,q,ϕc,i,hk-τ=exp⁡-eik-τ-cc,i,h2σc,i2,where τ is communication latency, cc,i,h∈Rmp is the Gaussian function center vector, and the centers should cover the system operation state region as much as possible. σc,i is width of Gaussian function. The approximation error εc,i would be very small if the dimension of basis function ϕc,i∈Rq×1 is large enough [11]. The relation betweenkth and k+1th optimal control action is(10)Jik=minuikαJik-1-αN+1pik,where uik∈Rm is control action for subsystem i. We estimate the strategy utility function by(11)J^ik=W^c,ikϕc,ik-τ.The prediction error of approximated strategy utility function J^i for critic NN is(12)ec,ik=J^ik-αJ^ik-1-αNpik.We define the objective function of critic NN for minimization at kth sampling as(13)Ea,ik=12ea,iTkea,ik.One common way to decrease the objective function is to update critic NN parameters along its gradient direction. Applying chain rule, partial derivative of objective function (13) with respect to W^c,ik is(14)∂Ea,ik∂W^c,ik=ec,ikϕc,ikT=J^ik-αJ^ik-1-αNpikϕc,ik-τT.Therefore, updating law for critic NN of subsystem i is(15)W^c,ik+1=W^c,ik-δiJ^ik-αJ^ik-1-αNpikϕc,ik-τT.δi is a given scalar, representing updating step size. The choice of δi is very important. If δi is too large, the online learning may diverge.


## 3.3. Action Neural Network Design

Our control objective is to minimize the tracking errorei and also to minimize the long-term cost function/strategy utility function Ji. They depend on the control action in each step. The desired control action (5) is an expected strategy for approaching this objective, and an action neural network is suggested for approximating the desired control action. The desired control action ud,i can be equal to(16)ud,ik=Wa,ikϕa,ik-τ+εa,ik,where Wa,i is the optimal weighting matrix for neural output which minimizes the residual εa,i; ϕa,i∈Rl×1 is the basis function which has the same form as (9). εa,i would be very small if the dimension of ϕa,i is very large. However, ud,i and Wa,i are unknown; the desired control action is proposed to be estimated by(17)u^ik=W^a,iϕa,ik-τ,where W^a,i is the estimated weighting matrix for Wa,i. And we have the estimated error u~i for desired control action.(18)u~ik=u^ik-ud,i=W~a,iϕa,ik-τ-εa,ik,where W~a,i=W^a,i-Wa,i, and we denote ςik=W~a,iϕa,ik-τ, which causes dynamic (3) to be(19)ei,1k+1=ei,2k,ei,2k+1=ei,3k,⋮ei,pk+1=Γiei,pk+giςik+∑j∈Npi\Ncifjixj,pk-giεik+dik.Npi and Nci are the neighbor subsystem sets of subsystem i which are connected to subsystem i in physical way and cyberway. In our proposed scheme, supervised learning is incorporated into the action neural network training for accelerating the convergence rate of online updating. The objective of the policy is not only to minimize long-term cost Ji but also to approximate the desired control output ud,i with supervised learning. Thus, the error vector of action network is defined as(20)eak=giςik+gi-1J^ik-Jd,i,where Jd,i is the desired utility function value for subsystem i, it can be set as 0 [20], and gi is principal mean square root. The following cost function is defined for each step:(21)Ea,ik=12ea,ikTea,ik.Then, the partial derivative of (21) with respect to W~a,i is obtained by chain rule.(22)∂Ea,ik∂W~a,ik=∂Ea,ik∂ea,ik∂ea,ik∂ςik∂ςik∂W~a,ik=ϕa,ikgiςik+J^ikT.Therefore, with gradient descent principle, the action NN weight matrix is updated by(23)W~a,ik+1=W~a,ik-βigiςik+J^ikϕa,ik-τT.βi is the updating step size for online learning of action neural network. The choice of βi will be discussed in the next section, which is associated with the stability of the online learning algorithm.


## 4. Stability Analysis

This section discusses the stability of online learning algorithm and the tracking performance. It is necessary for control design. The upper bound of error and weight parameter of neural networks are analyzed. Firstly, a theorem about the stability of this scheme is proposed.Theorem 3.
For a given networked control system described in (3) and the parameter updating algorithm in (15) (23), if A1≤0, A2≤0, A3≤0, and A4≤0, where(24)A1=-μi,121-κiΓimax+∑i∈Npj\Ncjμj,1κjfijmax2,A2=-μi,22βi2gimin-1gimax-2βigimaxϕa,ik-τ2+μi,12κigimax,A3=μi,22βigimax+2βi2ϕa,ik-τ2-μi,32δi1-α2-α2N+1-3δiϕc,ik-τ2+μi,42,A4=μi,32δi+3αδi2ϕc,ik-τ2-μi,42,A5=μi,323δi2ϕc,ik-τ2+1,A6=κiμi,12,whereκi=3+Npi\Nci,gimax=λmaxgiTgi,Γimax=λmaxΓi2, gimin=λmaxgiTgi1/2, and fimax=λmax(∂fjiT/∂ej,p∂fji/∂ej,p).Then, there exist upper bounds for eik2ςik2J~ik2, when t→+∞,and they are(25)eik2≤-A5+A6dmax2A1,ςik2≤-A5+A6dmax2A2,J~ik2≤-A5+A6dmax2A3.And the system is stable.Proof of Theorem3.
For the dynamic system described in (3), (15), and (23), we first define a Lyapunov function which consisted of quadratic of tracking error, action network weight error, and the error of critic neural network. It is(26)Vk=∑i∈SVik,where(27)Vik=Vi,1k+Vi,2k+Vi,3k+Vi,4k,Vi,1k=μi,12ei,pkTei,pk,Vi,2k=μi,22tr⁡W~a,ikTW~a,ik,Vi,3k=μi,42tr⁡W~c,ikTW~c,ik,Vi,4k=μi,52tr⁡J~ik-1TJ~ik-1,where W~c,i=W^c,i-Wc,i and J~c,i=W~c,iϕc,i. For a subsystem i, we have(28)ΔVi,1=μi,12eik+1Teik+1-eikTeik=μi,12Γiei,pk+giςik+∑j∈Npi\Ncifjiej,pk+yi,dk-giεik+dik2-eik2≤μi,12-1-κiΓimaxei,pk2+∑j∈Npi\Nciκifjimaxej,pk2+κigimaxεik2+ςik2+κidik,where κi=3+Npi\Nci, gimax=λmaxgiTgi, fimax=λmax∂fjiT/∂ej,p∂fji/∂ej,p, Γimax=λmaxΓi2, and(29)ΔVi,2=μi,22tr⁡W~a,ik+1TW~a,ik+1-tr⁡W~a,ikTW~a,ik≤μi,22-2tr⁡βiςikTgiςik+J^ik+βi2ϕa,ik-τ2giςik+J^ik2≤μi,22-βi2gimin-1gimax-2βigimaxϕa,ik-τ2ςik2+βigimax+2βi2ϕa,ik-τ2J^ik2,where gimax=λmaxgiTgi and gimin=λmaxgiTgi1/2. For strategy utility function, (10) leads to(30)Jik=αJik-1-αN+1pi∗k,and pi∗ is the utility function under the optimal strategy.(31)ec,ik=J^ik-αJ^ik-1+αN+1pik=J^ik-αJ^ik-1+αN+1pik-Jik+αJik-1-αN+1pik=J~ik-αJ~ik-1+αN+1p~ik,and p~i(k)=pi(k)-pi∗(k), and p~i(k)≤1. The updating equation (15) yields(32)ΔVi,3=μi,32tr⁡W~c,ik+1TW~c,ik+1-tr⁡W~c,ikTW~c,ik≤μi,32-2δitr⁡W~c,ikJ~ik-αJ~ik-1+αN+1p~ikϕc,ik-τT+δi2ϕc,ik-τ2J~ik-αJ~ik-1+αN+1p~ik2≤μi,32-δi1-α2-α2N+1-3δiϕc,ik-τ2J~ik2+δi+3αδi2ϕc,ik-τ2J~ik-12+3δi2ϕc,ik-τ2+1.The last part of the variation of V is(33)ΔVi,4=μi,42J~ikTJ~ik-J~ik-1TJ~ik-1.With the sum of all the above variations, we get(34)ΔV=ΔVi,1+ΔVi,2+ΔVi,3+ΔVi,4=A1ei,pk2+A2ςik2+A3J~ik2+A4J~ik-12+A5+A6εik2+dmax2,where A1, A2, A3, A4, A5, and A6 are given in Theorem 3. Therefore, if A1≤0, A2≤0,A3≤0, and A4≤0, the upper boundary of xik2ςik2J^ik2 can be estimated, when t→+∞, εik2 is very small which can be neglected if the dimension of basis function ϕi is large enough. The upper bound can be estimated by(35)xik2≤-A5+A6dmax2A1,ςik2≤-A5+A6dmax2A2,J^ik2≤-A5+A6dmax2A3.Remark 4.
The stability of this system depends on the control parametersΓi, βi, δi, and α, system functions gi and fji, and the communication networks which affect parameter κi. It is obvious that if subsystem can obtain all state information from physically connected neighbors, the parameter κi would be smaller, it improves the system performance because the absolute value of A1 and A2 will be larger, and it decreases the upper bound of eik and ςi. Moreover, the sign of A5 and A6 cannot be necessarily definite, as they are not the coefficients of the estimated variable in the following Lyapunov function variation expression (34).


## 5. Simulation Results

This simulation illustrates the effectiveness and advantage of our proposed control scheme in four aspects: (1) The effectiveness of our proposed control scheme of physical coupling networked control system in tracking sine wave signal with disturbances; (2) its effectiveness with communication delay; (3) its advantages compared with conventional reinforcement learning; (4) its effectiveness in multicontrol input system.The first simulation considers a networked system called system I as shown in Figure2. System I consisted of four subsystems, each subsystem physically coupled with other subsystems. Each subsystem is a nonlinear system. Their equations are(36)xi,1k+1=xi,1k,xi,2k+1=-58xi,1k+xi,2k2+0.3xi,2k+uk+∑j∈Npifjixik+dik.i=1,2,3,4. The initial value of states for this simulation are x1=0.10.2T, x2=0.10.3T, x3=0.10.2T, and x4=0.50.2T. The target signals for xi,2k(i=1,2,3,4) are(37)y1k=0.2sin⁡0.1k+2,y2k=0.2sin⁡0.1k+1,y3k=0.2sin⁡0.1k+1.2,y4k=0.2sin⁡0.1k+1.4.The details of other functions and variables are listed in Table 1.Table 1
Function and parameters in networked system I.
Function/parameter name
Description
Mathematical expression
f
12
The interference function on subsystem 2 from subsystem 1
0.1
x
1,2
f
23
The interference function on subsystem 3 from subsystem 1
0.0
1
x
2,1
+
0.0
5
x
2,2
f
34
The interference function on subsystem 3 from subsystem 1
0.0
3
x
3,1
+
0.1
x
3,2
f
41
The interference function on subsystem 1 from subsystem 4
0.1
x
4,1
+
0.00
2
x
4,2
d
1
The disturbance on subsystem 1
Gaussian noise with magnitude of 0.01
d
2
The disturbance on subsystem 2
Gaussian noise with magnitude of 0.01
d
3
The disturbance on subsystem 3
Gaussian noise with magnitude of 0.01
d
4
The disturbance on subsystem 4
Gaussian noise with magnitude of 0.01Figure 2
The structure of networked system I.Figure2 illustrates both the physical connection and the cyberconnection of system I. The communication network can send state information from subsystems 1 to 2, 2 to 3, 3 to 4, and 4 to 1. The parameters of the proposed controller are illustrated in Table 2.Table 2
The parameters of controllers.
Parameter name
Description
Value
q
Dimension of basis vector functionφc,i for critic network. i=1,2,3,4.
40
l
Dimension of basis vector functionφa,i for action network. i=1,2,3,4.
40
σ
c
,
i
The width of radial basis function for critic network.i=1,2,3,4.
1.414
σ
a
,
i
The width of radial basis function for action network.i=1,2,3,4.
1.414
c
c
,
i
,
h
,
c
a
,
i
,
h
′
The center vector for basis function.i=1,2,3,4. h=1,…,q. h′=1,…,l.
Element distributed in[-1,+1]
N
The horizon length of strategy utility function.
100
δ
i
The update rate for critic network weight matrix.i=1,2,3,4.
0.05
β
i
The update rate for action network weight matrix.i=1,2,3,4.
0.05
Γ
i
The damping rate of tracking error.i=1,2,3,4.
0.05The simulation results are shown in Figures3 and 4. From Figure 3, it is obvious that all of the subsystems converge to the target state with small errors. The curves converge to the target curves at about 125th control actions, which mean the online learning algorithm successfully obtained the desired action network and critic network. From Figure 4, it can be seen that the fluctuant of control output is decreased along the time during the online learning process. They also illustrate the effectiveness of our proposed control scheme.Figure 3
The state curves of system I with the proposed control scheme.
(a)
(b)
(c)
(d)Figure 4
Control outputs of subsystem neural controllers with the proposed control scheme.
(a)
(b)
(c)
(d)In order to present the advantage of our suggested control scheme, we select conventional reinforcement learning without supervised learning scheme; the updating of action network solely depends on the backpropagation of critic network with the objective of minimizing the output of critic network [12]. The result is shown in Figure 5. The results explicitly indicate the divergence of the learning algorithm because of the fast changing of target signal. And the conventional reinforcement learning may need off-line learning in advance. The results illustrate our proposed control scheme is more stable and has more powerful online learning ability than the conventional method.Figure 5
The state curve of subsystem with conventional reinforcement learning.
(a)
(b)
(c)
(d)In practice, the controller usually encounters action delay or communication delay. It is also modeled in our suggested model. To illustrate the effectiveness of our proposed control scheme under communication delay, we chose three communication delay valuesτ=3,5,10 to carry out the simulation. The simulation results are shown in Figures 6–8. These results show our proposed control scheme is stable under communication delay. However, static error increases with the communication delay. It is clear that the error of simulation with τ=3 is relatively smallest and the error of simulation with τ=10 is largest in the results.Figure 6
The state curves of subsystems with control delayτ=3 under the proposed control scheme.
(a)
(b)
(c)
(d)Figure 7
The state curves of subsystems with control delayτ=5 under the proposed control scheme.
(a)
(b)
(c)
(d)Figure 8
The state curves of subsystems with control delayτ=10 under the proposed control scheme.
(a)
(b)
(c)
(d)For further demonstrating the effectiveness of our suggested scheme with multiple control input, we choose another system called system II for simulation. The model of system II is(38)xi,1k+1=xi,2k,xi,2k+1=xi,3k,xi,3k+1=-58xi,1k+xi,3k2+xi,1k+110+xi,1k2+xi,3k2ui,1k+di,1k+∑j∈Npifji,1xjk,xi,4k+1=-58xi,2k+xi,4k2+xi,1k+110+xi,2k2+xi,4k2ui,2k+di,2k+∑j∈Npifji,2xjk.i=1,2,3,4. The system structure is the same as shown in Figure 2. The target signals for xi,3 and xi,4 are(39)y1,3k=0.2sin⁡0.01k+2,y1,4k=0.2sin⁡0.01k+2,y2,3k=0.2sin⁡0.01k+1,y2,4k=0.2sin⁡0.01k+2.2,y3,3k=0.2sin⁡0.01k+1.2,y3,4k=0.2sin⁡0.01k+2.5,y4,3k=0.2sin⁡0.01k+1.4,y4,4k=0.2sin⁡0.01k+1.Other model parameters are illustrated in Table 3.Table 3
Model parameters of system II.
Function/parameter name
Description
Mathematical expression
f
12
The interference function on subsystem 2 from subsystem 1
f
12,3
f
12,4
=
0.05
0.1
0.2
0.3
0.33
0.2
0.01
0.07
x
1
f
23
The interference function on subsystem 3 from subsystem 1
f
23,3
f
23,4
=
0.01
0.05
0.01
0.07
0.04
0.04
0.1
0.3
x
2
f
34
The interference function on subsystem 3 from subsystem 1
f
34,3
f
34,4
=
0.03
0.1
0.1
0.3
0.02
0.07
0.2
0.04
x
3
f
41
The interference function on subsystem 1 from subsystem 4
f
41,3
f
41,4
=
0.1
0.002
0.2
0.043
0.01
0.3
0.2
0.1
x
4
d
1
The disturbance on subsystem 1
Gaussian noise with magnitude of 0.05
d
2
The disturbance on subsystem 2
Gaussian noise with magnitude of 0.05
d
3
The disturbance on subsystem 3
Gaussian noise with magnitude of 0.05
d
4
The disturbance on subsystem 4
Gaussian noise with magnitude of 0.05The controller parameters are set as shown in Table4.Table 4
Controller parameters of system II.
Parameter name
Description
Value
q
Dimension of basis vector functionφc,i for critic network. i=1,2,3,4.
100
l
Dimension of basis vector functionφa,i for action network. i=1,2,3,4.
100
σ
c
,
i
The width of radial basis function for critic network.i=1,2,3,4.
1.414
σ
a
,
i
The width of radial basis function for action network.i=1,2,3,4.
1.414
c
c
,
i
,
h
,
c
a
,
i
,
h
′
The center vector for basis function.i=1,2,3,4. h=1,…,q. h′=1,…,l.
Element distributed in[-1,+1]
N
The horizon length of strategy utility function.
100
δ
i
The update rate for critic network weight matrix.i=1,2,3,5.
0.001
β
i
The update rate for action network weight matrix.i=1,2,3,5.
0.03
Γ
i
The damping rate of tracking error.i=1,2,3,4.
0.01The simulation results are shown in Figure9. They show that all the subsystem states converge to the target signals within a small number of time steps (it is about 120). The tracking errors are small, and each of the state variables converges to its corresponding target signal. It illustrates the effectiveness of our suggested control scheme in application of multicontrol input systems with a relatively larger dimension compared with the previous simulation.Figure 9
The state curves of system II with the proposed control scheme.
(a)
(b)
(c)
(d)


## 6. Conclusion

This paper suggests online reinforcement learning with one-layer neural network for controlling physically coupled networked system. It is a distributed learning control scheme. The networked system is divided into many subsystems; each system is an individual agent with controller and reinforcement learning algorithm. The reinforcement learning algorithm consists of the learning of critic network and action network. The critic network approximates the strategy utility function and the action network approximate the defined desired optimal controller. The action network weights updating decreases long-term cost with supervised learning mechanism by incorporating the desired control errorς with long-term cost function J. The effectiveness of our proposed controller is illustrated in the simulation part. The simulation results also indicate that the proposed control scheme improves the tracking performance compared with conventional reinforcement learning with only objective of long-term cost (critic network). In the future research, we will investigate the application of our proposed control scheme in a large cyberphysical system such as smart grid.



---

*Source: 5950678-2018-02-07.xml*