# Optimal Control for Networked Control Systems with Markovian Packet Losses

Authors: Xiao Han; Zhijian Ji; Qingyuan Qi
Journal: Complexity
 (2020)

---

## Abstract

This paper is concerned with the optimal output feedback control problem for networked control systems (NCSs) with Markovian packet losses. In this paper, the packet losses occur both between the sensor and controller and between the controller and actuator. Moreover, the packet loss channels are described with two-state Markov chains. Since the precise state information cannot be obtained, thus an optimal recursive estimator is designed. Furthermore, by adopting the dynamic programming approach, we derive the optimal output feedback control, which is based on the solution to a given modified Riccati equation. The obtained results can be seen as an important implementation of the control theory for NCSs with unreliable communication channels.


---

## Body

## 1. Introduction

As it is well known, NCSs are spatially distributed systems in which the communication between sensors, actuators, and controllers occurs through a shared communication network (see, for e.g., [1, 2] and references therein). NCSs connect cyberspace to physical space and can perform many tasks over long distance, which have the practical application value. And NCSs have the advantages of low cost, light weight, and simple structure, which can improve the reliability of the system, so they are widely used in various fields, such as remote surgery [3], unmanned aerial vehicles [4], and artificial intelligence [5]. It is worth noting that the study of network control has become an important direction of control science in the recent years (see, for e.g., [6–9]).In communication networks, the impact of network communication bandwidth and limited load capacity, network congestion, and network connection interruption will result in data losses, retransmission, and communication links blocked or cut off (see, for e.g., [10,11] and references therein). Especially, data packet losses may occur in NCSs from sensors to controllers and from controllers to actuators. In addition, the communication between the sensor nodes and the remote state estimator is implemented via a shared network, where only one sensor node is permitted to transmit data at each time instant for the purpose of preventing data collisions [12]. In order to evaluate the estimation accuracy, various state estimation performance requirements have been introduced to quantify the engineering specifications such as H∞ constraint, minimum mean squared error (MMSE) index, ellipsoidal bound constraint, linear quadratic performance index, and ultimate boundedness requirement [13]. For example, Zou et al. [14] designed the moving horizon estimator for linear systems with unknown inputs and quantization effects.Although the closed-loop control system is robust to the change of structure and parameters in the system, the packet losses will inevitably lead to the performance degradation, which will seriously lead to the instability of the system. Thus, studying packet losses is a key problem to analyze and design the NCSs, which is the motivation of this paper. Many researchers showed their interest in the networked control for NCSs with unreliable packet loss channels, and great contributions have been made in this research field (see, for e.g., [15–18]). In [15], a switched system approach is adopted to deal with the problem of fault detection (FD) for uncertain delta operator NCSs with packet losses and time varying delays. And two independent Bernoulli distributed white sequences are introduced to account for packet losses. Qi and Zhang [18] concerned the optimal measurement feedback control and stabilization for NCSs with packet losses. However, they only considered the problem of packet losses in one channel, and the packet losses’ process is the Bernoulli process.In general, the process of data packet losses in communication networks is usually modeled as an independent identical distribution (i.i.d) Bernoulli process or a Markov chain. Obviously, the latter case of Markovian packet loss is more general and realistic. For example, Wang et al. [19] considered the case of data packet reception between the controller and the actuator network transmission at each time instant, which is described with Markovian packet losses. Zhou and Zhang [20] investigated the H-infinity fault detection for time-delay delta operator systems with random two-channel packet losses and limited communication. The random two-channel packet losses are described by the Markov chain process. Moreover, the stability of NCSs was studied for the NCSs with the simultaneous input delay and Markovian packet losses in [21]. However, the problem of two-channel packet losses is not considered. Overall, the next time instant of network transmission may depend on the current time instant, and the Markov process can be used to describe the dependency of events. Hence, it is more suitable to model the NCSs with this kind of packet losses than the i.i.d stochastic processes. In order to study the NCSs with packet losses, it is necessary to combine NCSs modeled by MJSs with optimal control.Furthermore, due to the existence of unreliable packet loss channels, the precise state information may not be available to the controller. Therefore, an observer or estimator is needed to be the feedback of regulating the system performance, i.e., an output feedback controller should be designed. It is noted that, in the past several decades, many literatures have been published on the output feedback control, especially for NCSs (see, for e.g., [22–29]). For example, in order to deal with this problem, Wu and Chen [22] designed the networked control systems with packet losses. While Tan et al. [25] investigated the stabilization of networked control systems with network-induced delay and packet losses. Then, Shah and Mehta [26] proposed a design method of the discrete-time sliding mode controller based on Thiran’s delay approximation, which considered the real time situation of networked medium and packet losses situation. Shah et al. [29] presented a discrete-time higher-order sliding mode controller for NCSs using event-triggered approach and time delay compensation to overcome the network abnormalities such as communication delay, congestion, and network utilization that degrade the performance of the NCSs.Referring to the previous work [30], we are devoted to deal with the optimal control of NCSs with two kinds of Markovian packet loss channels in this paper. The model of NCS under consideration is shown in Figure 1. θk depicts whether the state signal is lost when data is transmitted from the sensor to the controller. βk indicates the packet loss when data is transmitted from the controller to actuator. Furthermore, Song et al. [30] addressed the modeling and guaranteed cost control for a class of NCSs with packet dropouts. The packet dropout processes in the forward channel and feedback channel are modeled as two Markov chains, and the overall closed-loop NCS is modeled as a Markovian switched system with two modes. The differences between this paper and [30] are as follows. Firstly, the system model of this paper is different from the system model considered by [30]. There is additive noise in the system considered in this paper. However, the system proposed by [30] has no additive noise. Moreover, Song et al. [30] assumed vk=Kwk as the optimal controller for the design. In contrast, we did not assume the form of the controller when we considered the optimal control problem. Finally, Song et al. [30] studied the stochastic stability problem and gave a sufficient condition of stochastic stability with upper bound of the quadratic cost function. And in this paper, we investigated the optimal control problem without considering the stability.Figure 1
NCS model with two kinds of Markovian packet losses.For the considered NCSs model, we will solve a linear quadratic (LQ) optimal control problem. While previous works mainly focused on the i.i.d Bernoulli packet loss case and the single packet loss channel case. In addition, it should be pointed out that an estimator should be designed in the existence ofθk, which may cause the separation principle fail. As pointed out in [31], the controller was designed with linear minimum mean square error (LMMSE), but it was shown that the LMMSE was not ideal and the popular separation principle was invalid. Finally, it is noted the Kalman filter [32–34] cannot be directly adopted in designing the optimal estimator.In this paper, by adopting the optimality principle of dynamic programming approach, we solve the basic optimal LQ control problem for NCSs with Markovian packet losses, and also the optimal estimator is proposed in this paper. The main contributions can be summarized as follows. First of all, some preliminary results are presented and an optimal estimator (conditional expectation) is given for the NCSs with Markovian packet losses. Accordingly, the error covariance matrices with the optimal estimator can also be calculated recursively. Consequently, the value function is defined. Then, an induction method is used to derive the optimal control strategies and the associated value function by the optimality principle. We show that the optimal controls are given in terms of a modified Riccati equation, which is well defined and can be calculated backwardly under the basic assumption made in this paper. The obtained results are new to the best of our knowledge, which can be regarded as important implementation for NCSs with Markovain packet losses.The rest of this paper is as follows. Section2 introduces the system model and states the problems. Section 3 gives the main results. In Section 4, some numerical examples are given to verify the obtained results. Finally, we conclude this paper in Section 5.Throughout this paper, the following notations will be used.ℝn means the n-dimensional Euclidean space, AT indicates the transpose of matrix A, E· is the mathematical expectation, and E·|ℱk is the conditional expectation with respect to ℱk. PX denotes the probability if the event X occurs, PX|Y is the conditional probability. The real symmetric matrix A>0 or A≥0 signifies A is positive definite or positive semidefinite. TrB implies the trace of matrix B, and Nμ,Σ means the normal distribution with mean μ and covariance Σ.


## 2. Problem Statement

In this paper, we will consider the following NCS, which is a linear time-invariant system:(1)xk+1=Axk+βkBuk+Mωk,where xk∈ℝn is the system state, uk∈ℝm is the control input, and ωk∈ℝp denotes the system noise with zero mean and covariance Σω. A,B,andM are the given coefficient matrices with appropriate dimensions. βk∈0,1 is a two-state Markovian chain, which is used to denote the Markovian packet losses when data is transmitted from the controller to actuator. The transition probability matrix of βk is given by(2)Pβk+1=0|βk=0Pβk+1=1|βk=0Pβk+1=0|βk=1Pβk+1=1|βk=1,=1−ααλ1−λ,α∈0,1,λ∈0,1.The initial distribution isPβ0=0=a, Pβ0=1=1−a, and a∈0,1. Moreover, x0 is a random vector with mean μ and covariance Σ0.Due to the packet losses from the sensor to controller, the state information cannot be precisely obtained, the ‘raw’ information available to the controller is given by(3)hk=θkxk,where θk is a two-state Markovian chain with θk∈0,1, and the transition probability matrix is given by(4)Pθk+1=0|θk=0Pθk+1=1|θk=0Pθk+1=0|θk=1Pθk+1=1|θk=1=1−ppq1−q,p∈0,1,q∈0,1.Also, we assumePθ0=0=1−bandPθ0=1=b.To facilitate the discussions, we assumeβk, θk, ωk, and x0 are independent with each other.Associated with (1)–(4), the quadratic cost function to be minimized is given as follows:(5)JN=E∑k=0NxkTQxk+ukTRuk+ExN+1TSN+1xN+1,where Q∈ℝn×n,R∈ℝm×m,andSN+1∈ℝn×n are the given symmetric weighting matrices.In this paper, it is assumed thatuk is ℱk-measurable, and the information set ℱk is given by(6)ℱk≜h0,…,hk,β0,…,βk−1,…,u0,…,uk−1,θ0,…,θk.Therefore, the optimal control problem to be solved in this paper can be described as below.Problem 1.
Find theℱk-adapted controller uk to minimize cost function (5).Remark 1.
According to previous works [30, 35], the packet losses occur both from the sensor to controller and from the controller to actuator. And both the packet loss channels are described with two-state Markovian chains. In this paper, the difficulties and challenges can be concluded as follows. The precise state information is not available to the controller, and then to derive the optimal control and verify the ‘separation principle’ will remain challenging.


## 3. Main Results

### 3.1. Optimal Estimation

In this section, some preliminary results will be introduced.Lemma 1.
For simplicity, we denoteπk0=Pθk=0 and πk1=Pθk=1; then, πk0andπk1 can be calculated as follows:(7)πk0πk1=1−bb1−ppq1−qk,with π00=1−b and π10=b.
Similarly,τk0≜Pβk=0 and τk1≜Pβk=1 satisfy:(8)τk0τk1=a1−a1−ααλ1−λk,with τ00=a and τ10=1−a.Proof.
The proof can be found in [36], which can be omitted here.Lemma 2.
For NCS (1) and measurement (3), then the optimal estimator x^k=Exk|ℱk can be recursively calculated as follows:(9)x^k=1−θkAx^k−1+Buk−1+θkhk,with the initial estimator(10)x^0=1−θ0Ex0+θ0h0.
Furthermore, the error covariance matrixPk=Exk−x^kxk−x^kT|ℱk satisfies:(11)Pk=1−θkAPk−1AT+M∑ωMT,with P0=1−θ0Σ0.Proof.
The results can be deduced from Theorem 2 in [18], and the detailed proof is omitted here to avoid the repetition.


### 3.2. The Optimal Control Law

In order to guarantee the solvability of Problem1, we make the following standard assumption on the weighting matrices of (5).Assumption 1.
Q≥0,R>0,SN+1≥0.
For the sake of discussion, we denote the value functionVk,k=0,…,N as follows:(12)Vk=minuk,…,uNE∑j=kNxjTQxj+ujTRuj+xN+1TSN+1xN+1|ℱk.
Consequently, we will introduce the main results on the optimal control design for Problem1.Theorem 1.
Under Assumption1, the output feedback optimal controller can be given by(13)uk∗=βk−1uk1,∗+1−βk−1uk0,∗,where(14)uk0,∗=−R+λBTSk+1B−1λBTSk+1Ax^k,(15)uk1,∗=−R+1−λBTSk+1B−11−λBTSk+1Ax^k,in which x^k can be calculated from Lemma 2, and Sk satisfies the modified Riccati equation:(16)Sk=βk−1Sk1+1−βk−1Sk0,with(17)Sk0=Q+ATSk+1A−λ2ATSk+1B×R+λBTSk+1B−1BTSk+1A,(18)Sk1=Q+ATSk+1A−1−λ2ATSk+1B×R+1−λBTSk+1B−1BTSk+1A,with terminal condition SN+10=SN+11=SN+1 given in (5).
In this case, the value functionVk is given by(19)Vk=ExkTSkxk|ℱk+∑m=kNTrPmΛm+∑ωMTSm+1M,where Sk can be calculated from (16)–(18), and the error covariance Pk can be calculated from Lemma 2. Moreover, Λk is given by(20)Λk=βk−1Λk1+1−βk−1Λk0,with(21)Λk0=λ2ATSk+1BR+λBTSk+1B−1BTSk+1A,(22)Λk1=1−λ2ATSk+1BR+1−λBTSk+1B−1×BTSk+1A.
Furthermore, the optimal cost function can be obtained by(23)JN∗=TrΠ0S¯0+∑k=0NTrP¯kΛ¯k+∑ωMTS¯k+1M,in which Π0=Ex0x0T, and S¯k,Λ¯k are given by(24)S¯k=τk−10Sk0+τk−11Sk1,Λ¯k=τk−10Λk0+τk−11Λk1,Pk¯=πk0Pk,k=0,…,N,with terminal condition S¯N+1 is given by (5), and πk0,πk1,τk0,andτk1 can be calculated from (7) and (8).Proof.
Firstly, we will show thatSk given by (16) is positive semidefinite for k=0,…,N. In fact, from Assumption 1, we know SN+1≥0, R+λBTSN+1B>0, and R+1−λBTSN+1B>0. It can be easily verified that both SN0 and SN1 in (17) and (18) are positive semidefinite. Hence, SN≥0 can be derived. By repeating the above procedures, we can conclude that Sk≥0 for k=0,…,N. Moreover, it can also be obtained that R+λBTSkB>0 and R+1−λBTSkB>0. Therefore, the Riccati equations (16)–(18) are well defined under Assumption 1.
By following the well-known Bellman optimality principle from dynamic programming approach, it can be derived that the following relationship holds:(25)Vk=minukExkTQxk+ukTRuk+Vk+1|ℱk.
From (1), we know that VN can be calculated as(26)VN=minuNExNTQxN+uNTRuN+xN+1TSN+1xN+1|ℱN=minuNExNTQxN+uNTRuN+AxN+βNBuN+MωNT×SN+1AxN+βNBuN+MωN|ℱN=minuNExNTQxN+uNTRuN+xNTATSN+1AxN+2xNTATSN+1βNBuN+uNTBTβNSN+1βNBuN+ωNTMTSN+1MωN|ℱN=minuNExNTQ+ATSN+1AxN+uNTRuN|ℱN+2ExNT|ℱNEβN|βN−1ATSN+1BuN+EβN2|βN−1uNTBTSN+1BuN+Tr∑ωMTSN+1M.
The following two cases are under consideration:(1)
WhenβN−1=0, we can obtain that(27)VN0=minuNExNTQ+ATSN+1AxN+uNTRuN|ℱN+2λExNT|ℱNATSN+1BuN+λuNTBTSN+1BuN+Tr∑ωMTSN+1M.Furthermore, since we have shownR+λBTSN+1B is positive definite, then it holds(28)VN0=minuNEuN+R+λBTSN+1B−1×λBTSN+1AxNTR+λBTSN+1B×uN+R+λBTSN+1B−1λBTSN+1AxN+xNTQ+ATSN+1A−λ2ATSN+1B×R+λBTSN+1B−1BTSN+1AxN|ℱN+Tr∑ωMTSN+1M.By using completing square skill, we know that the optimal control can be presented as(29)uN0,∗=−R+λBTSN+1B−1λBTSN+1Ax^N.Then,VN can be calculated as follows:(30)VN0=ExNTSN0xN|ℱN+TrPNΛN0+ΣωMTSN+1M,whereSN0 and PN satisfy (17) and (11) for k=N, respectively. And ΛN0 is given by (21).(2)
WhenβN−1=1, it holds(31)VN1=minuNExNTQ+ATSN+1AxN+uNTRuN|ℱN+21−λExNT|ℱNATSN+1BuN+1−λuNTBTSN+1BuN+Tr∑ωMTSN+1M.
Also noting thatR+1−λBTSN+1B>0, so we have(32)VN1=minuNEuN+R+1−λBTSN+1B−1×1−λBTSN+1AxNTR+1−λBTSN+1B×uN+R+1−λBTSN+1B−1×1−λBTSN+1AxN+xNTQ+ATSN+1A−1−λ2ATSN+1B×R+1−λBTSN+1B−1×BTSN+1AxN|ℱN+Tr∑ωMTSN+1M.
Hence, the optimal controller of minimizingVN1 can be given by(33)uN1,∗=−R+1−λBTSN+1B−1×1−λBTSN+1Ax^N,from uN=uN1,∗, we get that(34)VN1=ExNTSN1xN|ℱN+TrPNΛN1+∑ωMTSN+1M,where SN1,ΛN1,andPN satisfy (18), (22), and (11), respectively.
In this case,VN can be written as follows:(35)VN=βN−1VN1+1−βN−1VN0=βN−1ExNTSN1xN|ℱN+1−βN−1ExNTSN0xN|ℱN+βN−1TrPNΛN1+∑ωMTSN+1M+1−βN−1TrPNΛN0+ΣωMTSN+1M=ExNTSNxN|ℱN+TrPNΛN+∑ωMTSN+1M.
To use the induction method, we assume the optimal controlsuk∗ are given by (13) for k=l+1,…,N, and relationships (14)–(18) hold. Moreover, it is assumed that Vk,k=l+1,…,N can be presented as (19), i.e.,(36)Vk=ExkTSkxk|ℱk+∑m=kNTrPmΛm+ΣωMTSm+1M.
So, we can know that(37)Vl=minul,…,uNE∑j=lNxjTQxj+ujTRuj+xN+1TSN+1xN+1|ℱl=minulExlTQxl+ulTRul+Vl+1|ℱl=minulExlTQxl+ulTRul+xl+1TSl+1xl+1|ℱl+∑k=l+1NTrPkΛk+ΣωMTSk+1M=minulExlTQxl+ulTRul+Axl+βlBul+MωlTSl+1×Axl+βlBul+Mωl|ℱl+∑k=l+1NTrPkΛk+ΣωMTSk+1M=minulExlTQ+ATSl+1Axl+ulTRul|ℱl+2ExlT|ℱlEβl|βl−1ATSl+1Bul+Eβl2|βl−1ulTBTSl+1Bul+∑k=l+1NTrPkATSk+1BEβk|ℱk×R+Eβk2|ℱkBTSk+1B−1Eβk|ℱk×BTSk+1A+∑k=lNTr∑ωMTSk+1M.
Also, two cases will be studied. Actually, ifβl−1=0, then it holds that(38)Vl0=minulExlTQ+ATSl+1Axl+ulTRul|ℱl+2ExlT|ℱlλATSl+1Bul+λulTBTSl+1Bul+∑k=l+1NTrPkΛk+∑k=lNTrΣωMTSk+1M.
Since the positive definiteness ofR+λBTSl+1B has been shown before, thus Vl0 can be written as follows:(39)Vl0=minulEul+R+λBTSl+1B−1λBTSl+1AxlT×R+λBTSl+1B×ul+R+λBTSl+1B−1λBTSl+1Axl+xlTQ+ATSl+1A−λATSl+1B×R+λBTSl+1B−1λBTSl+1Axl|ℱl+∑k=l+1NTrPkΛk+∑k=lNTrΣωMTSk+1M.
Hence, the optimal control of minimizingVl0 can be given as follows:(40)ul0,∗=−R+λBTSl+1B−1λBTSl+1Ax^l.
In the case oful=ul0∗, we can obtain(41)Vl0=ExlTSl0xl|ℱl+∑k=lNTrPkΛk0+ΣωMTSk+1M,where Sl0 satisfies (17) for k=l.
Besides, for the case ofβl−1=1, we can obtain(42)Vl1=minulExlTQ+ATSl+1Axl+ulTRul|ℱl+2ExlT|ℱl1−λATSl+1Bul+1−λulTBTSl+1Bul+∑k=l+1NTrPkΛk+∑k=lNTrΣωMTSk+1M.
SinceR+1−λBTSl+1B is strictly positive definite, so we have(43)Vl1=minulEul+R+1−λBTSl+1B−1×1−λBTSl+1AxlTR+1−λBTSl+1B×ul+R+1−λBTSl+1B−11−λBTSl+1Axl+xlTQ+ATSl+1A−1−λATSl+1B×R+1−λBTSl+1B−11−λBTSl+1Axl|ℱl+∑k=l+1NTrPkATSk+1B1−λ×R+1−λBTSk+1B−11−λBTSk+1A+∑k=lNTrΣωMTSk+1M.
And we can obtain(44)ul1,∗=−R+1−λBTSl+1B−1×1−λBTSl+1Ax^l,from ul=ul1,∗, we have(45)Vl1=ExlTSl1xl|ℱl+∑k=lNTrPkΛk1+ΣωMTSk+1M,in which Sl1 is given by (18).
From that, the optimal controlleruk∗ is valid for k=l.
Therefore, we can prove thatVl can be calculated as below:(46)Vl=βl−1Vl1+1−βl−1Vl0=βl−1ExlTSl1xl|ℱl+1−βl−1ExlTSl0xl|ℱl+βl−1∑k=lNTrPkΛk1+ΣωMTSk+1M+1−βl−1∑k=lNTrPkΛk0+ΣωMTSk+1M=ExlTSlxl|ℱl+∑k=lNTrPkΛk+ΣωMTSk+1M.
Until now, the procedures of the induction method are complete, and we have shown that (13)–(22) hold.
Finally, we will prove that the optimal cost functionJN∗ is given by (23).
In fact, noting thatJN∗=EV0, and the following assertions hold for k=0,…,N,(47)S¯k=ESk=τk−10Sk0+τk−11Sk1,Λ¯k=EΛk=τk−10Λk0+τk−11Λk1,Pk¯=EPk=πk0Pk,then it follows(48)JN∗=EEx0TS0x0|ℱl+∑k=0NTrPkΛk+∑ωMTSk+1M=Ex0TS0x0+∑k=0NETrPkΛk+ΣωMTSk+1M=TrΠ0S¯0+∑k=0NTrP¯kΛ¯k+ΣωMTS¯k+1M,in which Π0=Ex0x0T.
Thus, the optimal cost function (23) has been verified. This ends the proof.Remark 2.
For the strategy proposed in the paper, we analyze and clarify its advantages and disadvantages. For Problem1, the main technique adopted in deriving the main results is the optimality principle of dynamic programming approach. Furthermore, when the controller cannot obtain accurate state information, we give the optimal estimation (conditional expectation) of the NCSs with Markovian packet losses and use recursion to calculate the error covariance matrix. It is verified that the ‘separation principle’ holds, i.e., the control gain matrix and the estimation gain can be calculated separately. The obtained results are new to the best of our knowledge.Remark 3.
It can be easily seen from the derivations that the obtained results in Theorem1 include previous works as special cases (see, for e.g., [8, 18]). For example, if the packet losses processes βk and θk are independently identically distribution (i.i.d) Bernoulli processes, then the main results can be reduced to the case in [8, 18].Remark 4.
In this paper, we mainly investigate the finite horizon optimal control for NCSs with two kinds of Markovian packet loss channels. The infinite horizon will be reported in the future work. Thus, the convergence analysis of the proposed recursive estimator is not given in this paper. Actually, by defining the Lyapunov function candidate with the optimal cost function of infinite horizon, the stabilization problem can be solved. In this case, the finite horizon Riccati equations converge to algebraic Riccati equations.Remark 5.
The differences between this paper and [17] are as follows. Firstly, different from the measurement equation considered in this paper, the measurement equation in [17] is yk=Cxk+υk, where C is the system matrix and υk is the measurement noise. Secondly, the design of the estimator is different. In [17], parameters similar to the standard Kalman filter are used to obtain the estimation equation. In this paper, the optimal estimator and the error covariance matrix with the optimal estimator are obtained recursively. Finally, in [17], only the data loss of the control signal sent from the controller is considered, while this paper considers not only the packet loss from the controller to the actuator but also the packet loss from the sensor to the actuator. There are also differences in the information sets available to the two controllers at any given time.Remark 6.
By adopting the dynamic programming approach, we derive the optimal output feedback control, which can be regarded as a special case of output feedback control problem. As for the general output feedback control problem, i.e.,yk=Hxk+vk or yk=γkHxk+vk, we will study this case in the future. Obviously, the general case is more challenging and complicated. We will try to extent the obtained results and methods to the general case.Remark 7.
A pseudocode of obtaining the control signal of this paper is shown in Figure2.Figure 2
The pseudocode of obtaining the control signal.


## 3.1. Optimal Estimation

In this section, some preliminary results will be introduced.Lemma 1.
For simplicity, we denoteπk0=Pθk=0 and πk1=Pθk=1; then, πk0andπk1 can be calculated as follows:(7)πk0πk1=1−bb1−ppq1−qk,with π00=1−b and π10=b.
Similarly,τk0≜Pβk=0 and τk1≜Pβk=1 satisfy:(8)τk0τk1=a1−a1−ααλ1−λk,with τ00=a and τ10=1−a.Proof.
The proof can be found in [36], which can be omitted here.Lemma 2.
For NCS (1) and measurement (3), then the optimal estimator x^k=Exk|ℱk can be recursively calculated as follows:(9)x^k=1−θkAx^k−1+Buk−1+θkhk,with the initial estimator(10)x^0=1−θ0Ex0+θ0h0.
Furthermore, the error covariance matrixPk=Exk−x^kxk−x^kT|ℱk satisfies:(11)Pk=1−θkAPk−1AT+M∑ωMT,with P0=1−θ0Σ0.Proof.
The results can be deduced from Theorem 2 in [18], and the detailed proof is omitted here to avoid the repetition.


## 3.2. The Optimal Control Law

In order to guarantee the solvability of Problem1, we make the following standard assumption on the weighting matrices of (5).Assumption 1.
Q≥0,R>0,SN+1≥0.
For the sake of discussion, we denote the value functionVk,k=0,…,N as follows:(12)Vk=minuk,…,uNE∑j=kNxjTQxj+ujTRuj+xN+1TSN+1xN+1|ℱk.
Consequently, we will introduce the main results on the optimal control design for Problem1.Theorem 1.
Under Assumption1, the output feedback optimal controller can be given by(13)uk∗=βk−1uk1,∗+1−βk−1uk0,∗,where(14)uk0,∗=−R+λBTSk+1B−1λBTSk+1Ax^k,(15)uk1,∗=−R+1−λBTSk+1B−11−λBTSk+1Ax^k,in which x^k can be calculated from Lemma 2, and Sk satisfies the modified Riccati equation:(16)Sk=βk−1Sk1+1−βk−1Sk0,with(17)Sk0=Q+ATSk+1A−λ2ATSk+1B×R+λBTSk+1B−1BTSk+1A,(18)Sk1=Q+ATSk+1A−1−λ2ATSk+1B×R+1−λBTSk+1B−1BTSk+1A,with terminal condition SN+10=SN+11=SN+1 given in (5).
In this case, the value functionVk is given by(19)Vk=ExkTSkxk|ℱk+∑m=kNTrPmΛm+∑ωMTSm+1M,where Sk can be calculated from (16)–(18), and the error covariance Pk can be calculated from Lemma 2. Moreover, Λk is given by(20)Λk=βk−1Λk1+1−βk−1Λk0,with(21)Λk0=λ2ATSk+1BR+λBTSk+1B−1BTSk+1A,(22)Λk1=1−λ2ATSk+1BR+1−λBTSk+1B−1×BTSk+1A.
Furthermore, the optimal cost function can be obtained by(23)JN∗=TrΠ0S¯0+∑k=0NTrP¯kΛ¯k+∑ωMTS¯k+1M,in which Π0=Ex0x0T, and S¯k,Λ¯k are given by(24)S¯k=τk−10Sk0+τk−11Sk1,Λ¯k=τk−10Λk0+τk−11Λk1,Pk¯=πk0Pk,k=0,…,N,with terminal condition S¯N+1 is given by (5), and πk0,πk1,τk0,andτk1 can be calculated from (7) and (8).Proof.
Firstly, we will show thatSk given by (16) is positive semidefinite for k=0,…,N. In fact, from Assumption 1, we know SN+1≥0, R+λBTSN+1B>0, and R+1−λBTSN+1B>0. It can be easily verified that both SN0 and SN1 in (17) and (18) are positive semidefinite. Hence, SN≥0 can be derived. By repeating the above procedures, we can conclude that Sk≥0 for k=0,…,N. Moreover, it can also be obtained that R+λBTSkB>0 and R+1−λBTSkB>0. Therefore, the Riccati equations (16)–(18) are well defined under Assumption 1.
By following the well-known Bellman optimality principle from dynamic programming approach, it can be derived that the following relationship holds:(25)Vk=minukExkTQxk+ukTRuk+Vk+1|ℱk.
From (1), we know that VN can be calculated as(26)VN=minuNExNTQxN+uNTRuN+xN+1TSN+1xN+1|ℱN=minuNExNTQxN+uNTRuN+AxN+βNBuN+MωNT×SN+1AxN+βNBuN+MωN|ℱN=minuNExNTQxN+uNTRuN+xNTATSN+1AxN+2xNTATSN+1βNBuN+uNTBTβNSN+1βNBuN+ωNTMTSN+1MωN|ℱN=minuNExNTQ+ATSN+1AxN+uNTRuN|ℱN+2ExNT|ℱNEβN|βN−1ATSN+1BuN+EβN2|βN−1uNTBTSN+1BuN+Tr∑ωMTSN+1M.
The following two cases are under consideration:(1)
WhenβN−1=0, we can obtain that(27)VN0=minuNExNTQ+ATSN+1AxN+uNTRuN|ℱN+2λExNT|ℱNATSN+1BuN+λuNTBTSN+1BuN+Tr∑ωMTSN+1M.Furthermore, since we have shownR+λBTSN+1B is positive definite, then it holds(28)VN0=minuNEuN+R+λBTSN+1B−1×λBTSN+1AxNTR+λBTSN+1B×uN+R+λBTSN+1B−1λBTSN+1AxN+xNTQ+ATSN+1A−λ2ATSN+1B×R+λBTSN+1B−1BTSN+1AxN|ℱN+Tr∑ωMTSN+1M.By using completing square skill, we know that the optimal control can be presented as(29)uN0,∗=−R+λBTSN+1B−1λBTSN+1Ax^N.Then,VN can be calculated as follows:(30)VN0=ExNTSN0xN|ℱN+TrPNΛN0+ΣωMTSN+1M,whereSN0 and PN satisfy (17) and (11) for k=N, respectively. And ΛN0 is given by (21).(2)
WhenβN−1=1, it holds(31)VN1=minuNExNTQ+ATSN+1AxN+uNTRuN|ℱN+21−λExNT|ℱNATSN+1BuN+1−λuNTBTSN+1BuN+Tr∑ωMTSN+1M.
Also noting thatR+1−λBTSN+1B>0, so we have(32)VN1=minuNEuN+R+1−λBTSN+1B−1×1−λBTSN+1AxNTR+1−λBTSN+1B×uN+R+1−λBTSN+1B−1×1−λBTSN+1AxN+xNTQ+ATSN+1A−1−λ2ATSN+1B×R+1−λBTSN+1B−1×BTSN+1AxN|ℱN+Tr∑ωMTSN+1M.
Hence, the optimal controller of minimizingVN1 can be given by(33)uN1,∗=−R+1−λBTSN+1B−1×1−λBTSN+1Ax^N,from uN=uN1,∗, we get that(34)VN1=ExNTSN1xN|ℱN+TrPNΛN1+∑ωMTSN+1M,where SN1,ΛN1,andPN satisfy (18), (22), and (11), respectively.
In this case,VN can be written as follows:(35)VN=βN−1VN1+1−βN−1VN0=βN−1ExNTSN1xN|ℱN+1−βN−1ExNTSN0xN|ℱN+βN−1TrPNΛN1+∑ωMTSN+1M+1−βN−1TrPNΛN0+ΣωMTSN+1M=ExNTSNxN|ℱN+TrPNΛN+∑ωMTSN+1M.
To use the induction method, we assume the optimal controlsuk∗ are given by (13) for k=l+1,…,N, and relationships (14)–(18) hold. Moreover, it is assumed that Vk,k=l+1,…,N can be presented as (19), i.e.,(36)Vk=ExkTSkxk|ℱk+∑m=kNTrPmΛm+ΣωMTSm+1M.
So, we can know that(37)Vl=minul,…,uNE∑j=lNxjTQxj+ujTRuj+xN+1TSN+1xN+1|ℱl=minulExlTQxl+ulTRul+Vl+1|ℱl=minulExlTQxl+ulTRul+xl+1TSl+1xl+1|ℱl+∑k=l+1NTrPkΛk+ΣωMTSk+1M=minulExlTQxl+ulTRul+Axl+βlBul+MωlTSl+1×Axl+βlBul+Mωl|ℱl+∑k=l+1NTrPkΛk+ΣωMTSk+1M=minulExlTQ+ATSl+1Axl+ulTRul|ℱl+2ExlT|ℱlEβl|βl−1ATSl+1Bul+Eβl2|βl−1ulTBTSl+1Bul+∑k=l+1NTrPkATSk+1BEβk|ℱk×R+Eβk2|ℱkBTSk+1B−1Eβk|ℱk×BTSk+1A+∑k=lNTr∑ωMTSk+1M.
Also, two cases will be studied. Actually, ifβl−1=0, then it holds that(38)Vl0=minulExlTQ+ATSl+1Axl+ulTRul|ℱl+2ExlT|ℱlλATSl+1Bul+λulTBTSl+1Bul+∑k=l+1NTrPkΛk+∑k=lNTrΣωMTSk+1M.
Since the positive definiteness ofR+λBTSl+1B has been shown before, thus Vl0 can be written as follows:(39)Vl0=minulEul+R+λBTSl+1B−1λBTSl+1AxlT×R+λBTSl+1B×ul+R+λBTSl+1B−1λBTSl+1Axl+xlTQ+ATSl+1A−λATSl+1B×R+λBTSl+1B−1λBTSl+1Axl|ℱl+∑k=l+1NTrPkΛk+∑k=lNTrΣωMTSk+1M.
Hence, the optimal control of minimizingVl0 can be given as follows:(40)ul0,∗=−R+λBTSl+1B−1λBTSl+1Ax^l.
In the case oful=ul0∗, we can obtain(41)Vl0=ExlTSl0xl|ℱl+∑k=lNTrPkΛk0+ΣωMTSk+1M,where Sl0 satisfies (17) for k=l.
Besides, for the case ofβl−1=1, we can obtain(42)Vl1=minulExlTQ+ATSl+1Axl+ulTRul|ℱl+2ExlT|ℱl1−λATSl+1Bul+1−λulTBTSl+1Bul+∑k=l+1NTrPkΛk+∑k=lNTrΣωMTSk+1M.
SinceR+1−λBTSl+1B is strictly positive definite, so we have(43)Vl1=minulEul+R+1−λBTSl+1B−1×1−λBTSl+1AxlTR+1−λBTSl+1B×ul+R+1−λBTSl+1B−11−λBTSl+1Axl+xlTQ+ATSl+1A−1−λATSl+1B×R+1−λBTSl+1B−11−λBTSl+1Axl|ℱl+∑k=l+1NTrPkATSk+1B1−λ×R+1−λBTSk+1B−11−λBTSk+1A+∑k=lNTrΣωMTSk+1M.
And we can obtain(44)ul1,∗=−R+1−λBTSl+1B−1×1−λBTSl+1Ax^l,from ul=ul1,∗, we have(45)Vl1=ExlTSl1xl|ℱl+∑k=lNTrPkΛk1+ΣωMTSk+1M,in which Sl1 is given by (18).
From that, the optimal controlleruk∗ is valid for k=l.
Therefore, we can prove thatVl can be calculated as below:(46)Vl=βl−1Vl1+1−βl−1Vl0=βl−1ExlTSl1xl|ℱl+1−βl−1ExlTSl0xl|ℱl+βl−1∑k=lNTrPkΛk1+ΣωMTSk+1M+1−βl−1∑k=lNTrPkΛk0+ΣωMTSk+1M=ExlTSlxl|ℱl+∑k=lNTrPkΛk+ΣωMTSk+1M.
Until now, the procedures of the induction method are complete, and we have shown that (13)–(22) hold.
Finally, we will prove that the optimal cost functionJN∗ is given by (23).
In fact, noting thatJN∗=EV0, and the following assertions hold for k=0,…,N,(47)S¯k=ESk=τk−10Sk0+τk−11Sk1,Λ¯k=EΛk=τk−10Λk0+τk−11Λk1,Pk¯=EPk=πk0Pk,then it follows(48)JN∗=EEx0TS0x0|ℱl+∑k=0NTrPkΛk+∑ωMTSk+1M=Ex0TS0x0+∑k=0NETrPkΛk+ΣωMTSk+1M=TrΠ0S¯0+∑k=0NTrP¯kΛ¯k+ΣωMTS¯k+1M,in which Π0=Ex0x0T.
Thus, the optimal cost function (23) has been verified. This ends the proof.Remark 2.
For the strategy proposed in the paper, we analyze and clarify its advantages and disadvantages. For Problem1, the main technique adopted in deriving the main results is the optimality principle of dynamic programming approach. Furthermore, when the controller cannot obtain accurate state information, we give the optimal estimation (conditional expectation) of the NCSs with Markovian packet losses and use recursion to calculate the error covariance matrix. It is verified that the ‘separation principle’ holds, i.e., the control gain matrix and the estimation gain can be calculated separately. The obtained results are new to the best of our knowledge.Remark 3.
It can be easily seen from the derivations that the obtained results in Theorem1 include previous works as special cases (see, for e.g., [8, 18]). For example, if the packet losses processes βk and θk are independently identically distribution (i.i.d) Bernoulli processes, then the main results can be reduced to the case in [8, 18].Remark 4.
In this paper, we mainly investigate the finite horizon optimal control for NCSs with two kinds of Markovian packet loss channels. The infinite horizon will be reported in the future work. Thus, the convergence analysis of the proposed recursive estimator is not given in this paper. Actually, by defining the Lyapunov function candidate with the optimal cost function of infinite horizon, the stabilization problem can be solved. In this case, the finite horizon Riccati equations converge to algebraic Riccati equations.Remark 5.
The differences between this paper and [17] are as follows. Firstly, different from the measurement equation considered in this paper, the measurement equation in [17] is yk=Cxk+υk, where C is the system matrix and υk is the measurement noise. Secondly, the design of the estimator is different. In [17], parameters similar to the standard Kalman filter are used to obtain the estimation equation. In this paper, the optimal estimator and the error covariance matrix with the optimal estimator are obtained recursively. Finally, in [17], only the data loss of the control signal sent from the controller is considered, while this paper considers not only the packet loss from the controller to the actuator but also the packet loss from the sensor to the actuator. There are also differences in the information sets available to the two controllers at any given time.Remark 6.
By adopting the dynamic programming approach, we derive the optimal output feedback control, which can be regarded as a special case of output feedback control problem. As for the general output feedback control problem, i.e.,yk=Hxk+vk or yk=γkHxk+vk, we will study this case in the future. Obviously, the general case is more challenging and complicated. We will try to extent the obtained results and methods to the general case.Remark 7.
A pseudocode of obtaining the control signal of this paper is shown in Figure2.Figure 2
The pseudocode of obtaining the control signal.


## 4. Numerical Examples

### 4.1. Optimal Estimation and LMMSE

Without loss of generality, we consider the higher-order system with the following coefficients:A=10.101, B=10, M=1001, ωk∼N00,1001, μ=00, Σ0=1001, a=0.2, α=0.5, λ=0.6, p=0.4, q=0.3, b=0.4, R=1, Q=1001, SN+1=1001, and the time horizon N=50. It is noted that the controller uk is ℱk-measurable, which cannot affect the performance of the estimators. Furthermore, under Assumption 1, the output feedback optimal controller can be calculated by Theorem 1.In order to show the optimality of the estimator given in Lemma2, we will provide numerical examples to compare the proposed estimator in Lemma 2 and the linear minimum mean square error estimator (LMMSE).As shown in Figures3 and 4, as expected, it can be easily judged that the proposed estimator given in Lemma 2 can achieve better performance than the LMMSE given in [34].Figure 3
X1 comparison between optimal estimation and LMMSE.Figure 4
X2 comparison between optimal estimation and LMMSE.


### 4.2. Probability Display

In this simulation, we see more clearly the process of probability change.θk denotes the packet loss process, which obeys the two-state Markovian chain with θk∈0,1. The initial distribution is Pθ0=0=1−b=0.6. As shown in Figure 5, the dotted line represents the change of Pθk=0, and the solid line indicates the change of Pθk=1.Figure 5
The change of Markov probability.


### 4.3. The Effect of TPM on System Performance

In order to show the transition probability matrix (TPM) effect on the system performance, in this paper, numerical results can be carried out for different cases.We consider the higher-order system with the following coefficients:A=10.50−1, B=10, M=0.1000.1, ωk∼N00,1001, μ=00, Σ0=1001, R=1, Q=1001, SN+1=1001, a=0.3,b=0.2, the transition probability matrix of βk is 1−ααλ1−λ, the transition probability matrix of θk is 1−ppq1−q, and the time horizon N=30.Case 1: assume the transition probability matrix ofβk is 0.60.40.70.3, and the transition probability matrix of θk is 0.50.50.60.4, and we derive the cost function 1Case 2: assume the transition probability matrix ofβk is 0.80.20.70.3, and the transition probability matrix of θk is 0.50.50.60.4, and we obtain the cost function 2As shown in Figure6, we can see that different transition probability matrices have different effects on the performance of the system. When the transition probability matrix changes, the cost function of the system also changes.Figure 6
Cost functions.


## 4.1. Optimal Estimation and LMMSE

Without loss of generality, we consider the higher-order system with the following coefficients:A=10.101, B=10, M=1001, ωk∼N00,1001, μ=00, Σ0=1001, a=0.2, α=0.5, λ=0.6, p=0.4, q=0.3, b=0.4, R=1, Q=1001, SN+1=1001, and the time horizon N=50. It is noted that the controller uk is ℱk-measurable, which cannot affect the performance of the estimators. Furthermore, under Assumption 1, the output feedback optimal controller can be calculated by Theorem 1.In order to show the optimality of the estimator given in Lemma2, we will provide numerical examples to compare the proposed estimator in Lemma 2 and the linear minimum mean square error estimator (LMMSE).As shown in Figures3 and 4, as expected, it can be easily judged that the proposed estimator given in Lemma 2 can achieve better performance than the LMMSE given in [34].Figure 3
X1 comparison between optimal estimation and LMMSE.Figure 4
X2 comparison between optimal estimation and LMMSE.


## 4.2. Probability Display

In this simulation, we see more clearly the process of probability change.θk denotes the packet loss process, which obeys the two-state Markovian chain with θk∈0,1. The initial distribution is Pθ0=0=1−b=0.6. As shown in Figure 5, the dotted line represents the change of Pθk=0, and the solid line indicates the change of Pθk=1.Figure 5
The change of Markov probability.


## 4.3. The Effect of TPM on System Performance

In order to show the transition probability matrix (TPM) effect on the system performance, in this paper, numerical results can be carried out for different cases.We consider the higher-order system with the following coefficients:A=10.50−1, B=10, M=0.1000.1, ωk∼N00,1001, μ=00, Σ0=1001, R=1, Q=1001, SN+1=1001, a=0.3,b=0.2, the transition probability matrix of βk is 1−ααλ1−λ, the transition probability matrix of θk is 1−ppq1−q, and the time horizon N=30.Case 1: assume the transition probability matrix ofβk is 0.60.40.70.3, and the transition probability matrix of θk is 0.50.50.60.4, and we derive the cost function 1Case 2: assume the transition probability matrix ofβk is 0.80.20.70.3, and the transition probability matrix of θk is 0.50.50.60.4, and we obtain the cost function 2As shown in Figure6, we can see that different transition probability matrices have different effects on the performance of the system. When the transition probability matrix changes, the cost function of the system also changes.Figure 6
Cost functions.


## 5. Conclusion

In this paper, we have investigated the optimal control problem for NCSs with two kinds of Markovian packet losses. By adopting the optimality principle of dynamic programming approach, for the first time, we derive the optimal control strategy, which is based on the given modified Riccati equations. Moreover, we also have shown that the separation principle is applicable for the considered problem, and the optimal control gain and the optimal estimator can be calculated separately. For future research, we plan to extend the obtained results to investigate the stabilization problem of the infinite horizon.



---

*Source: 4804320-2020-11-21.xml*