# Indefinite LQ Control for Discrete-Time Stochastic Systems via Semidefinite Programming

Authors: Shaowei Zhou; Weihai Zhang
Journal: Mathematical Problems in Engineering
 (2012)

---

## Abstract

This paper is concerned with a discrete-time indefinite stochastic LQ problem
in an infinite-time horizon. A generalized stochastic algebraic Riccati equation (GSARE)
that involves the Moore-Penrose inverse of a matrix and a positive semidefinite constraint
is introduced. We mainly use a semidefinite-programming- (SDP-) based approach to study
corresponding problems. Several relations among SDP complementary duality, the GSARE,
and the optimality of LQ problem are established.


---

## Body

## 1. Introduction

Stochastic linear quadratic (LQ) control problem was pioneered by Wonham [1] and has become one of the most popular research field of modern control theory; see, for example, [2–12] and the references therein. In the most early literature about stochastic LQ issue, it is always assumed that the control weighting matrix R is positive definite and the state weighting matrix Q is positive semidefinite as the deterministic LQ problem does. However, a surprising fact was found that, different from deterministic LQ problem, for a stochastic LQ modeled by a stochastic Itô-type differential system, the original LQ optimization may still be well posed even if the cost weighting matrices Q and R are indefinite [5]. Follow-up research was carried out, and a lot of important results were obtained. In [6–9], continuous-time indefinite stochastic LQ control problem was studied. For the discrete-time case, there have been some works. For example, the system with only control-dependent noises was studied in [10]. The finite time and infinite horizon indefinite stochastic LQ control problem with state- and control-dependent noises were, respectively, studied in [11, 12].In this paper, we study discrete-time indefinite stochastic LQ control problem over an infinite time horizon. The system involves multiplicative noises in both the state and the control. We mainly use the SDP approach introduced in [9, 13] to discuss the corresponding problem. We first introduce a generalized stochastic algebraic Riccati equation (GSARE) that involves the Moore-Penrose inverse of a matrix. The potential relations among LQ problem, SDP, and GSARE are studied. What we have obtained extends the results of [9] from continuous-time case to discrete-time case.The remainder of this paper is organized as follows. In Section2, we formulate the discrete-time indefinite stochastic LQ problem and present some preliminaries including generalized stochastic algebraic Riccati equation, SDP, and some lemmas. Section 3 contains the main results. Some relations among the optimality of the LQ problem, the complementary optimal solutions of the SDP and its dual problem, and the solvability of the GSARE are established. Some comments are given in Section 4.Notations. ℛn: n-dimensional Euclidean space. ℛn×m: the set of all n×m matrices. 𝒮n: the set of all n×n symmetric matrices. A′: the transpose of matrix A. A≥0(A>0): A is positive semidefinite (positive definite). I: the identity matrix. ℛ: the set of all real numbers. N:={0,1,2,…} and Nt:={0,1,2,…,t}. Tr(M): the trace of a square matrix M. 𝒜adj: the adjoint mapping of a mapping 𝒜.


## 2. Preliminaries

### 2.1. Problem Statement

Consider the following discrete-time stochastic system:(2.1)x(t+1)=Ax(t)+Bu(t)+[Cx(t)+Du(t)]w(t),x(0)=x0,t=0,1,2,…,
where x(t)∈ℛn,  u(t)∈ℛm are, respectively, the system state and control input. x0∈ℛn is the initial state and w(t)∈ℛ is the noise. A,C∈ℛn×n and B,D∈ℛn×m are constant matrices. {w(t),t∈N} is a sequence of real random variables defined on a complete probability space (Ω,ℱ,ℱt,P) with ℱt=σ{w(s):s∈Nt}, which is a wide sense stationary, second-order process with E[w(t)]=0 and E[w(s)w(t)]=δst, where δst is the Kronecker function. u(t) belongs to ℒℱ2(ℛm), the space of all ℛm-valued, ℱt-adapted measurable processes satisfying E(∑t=0∞∥u(t)∥2)<∞. We assume that the initial state x0 is independent of the noise w(t),t∈N.We first give the following definitions.Definition 2.1.
System (2.1) is called mean square stabilizable if there exists a feedback control u(t)=Kx(t) such that for any initial state x0, the closed-loop system
(2.2)x(t+1)=(A+BK)x(t)+(C+DK)x(t)w(t),x(0)=x0,t=0,1,2,…,
is asymptotically mean square stable, that is, the corresponding state of (2.2) satisfies limt→∞E∥x(t)∥2=0, where K∈ℛm×n is a constant matrix.For system (2.1), we define the admissible control set
(2.3)Uad={u(t)∈LF2(Rm),u(t)ismeansquarestabilizingcontrol.
The cost functional associated with system (2.1) is
(2.4)J(x0,u)=∑t=0∞E[x′(t)Qx(t)+u′(t)Ru(t)],
where Q and R are symmetric matrices with appropriate dimensions and may be indefinite. The LQ optimal control problem is to minimize the cost functional J(x0,u) over u∈Uad. We define the optimal value function as
(2.5)V(x0)=inf⁡u∈UadJ(x0,u).
Since the weighting matrices Q and R may be indefinite, the LQ problem is called an indefinite stochastic LQ control problem.Definition 2.2.
The LQ problem is called well posed if(2.6)-∞<V(x0)<∞,∀x0∈Rn.
If there exists an admissible control u* such that V(x0)=J(x0,u*), the LQ problem is called attainable and V(x0) is the optimal cost value. u*(t),t∈N, is called an optimal control, and x*(t),t∈N, corresponding to u*(t) is called the optimal trajectory.Stochastic algebraic Riccati equation (SARE) is a primary tool in solving stochastic LQ control problems. In [12], the following discrete SARE:(2.7)-P+A′PA+C′PC+Q-(A′PB+C′PD)(R+B′PB+D′PD)-1(B′PA+D′PC)=0,R+B′PB+D′PD>0,
was studied. The constraint that R+B′PB+D′PD>0 is demanded in (2.7). In fact, the corresponding LQ problem may have optimal control even if the condition is not satisfied. In this paper, we introduce the following generalized stochastic algebraic Riccati equation (GSARE),
(2.8)R(P)≡-P+A′PA+C′PC+Q-(A′PB+C′PD)(R+B′PB+D′PD)+(B′PA+D′PC)=0,R+B′PB+D′PD≥0,
which weakens the positive definiteness constraint of R+B′PB+D′PD to positive semidefiniteness constraint and replaces the inverse by Moore-Penrose inverse. Hence, (2.8) is an extension of (2.7).


### 2.2. Semidefinite Programming

In this subsection, we will introduce SDP and its dual. SDP is a special conic optimization problem and is defined as follows.Definition 2.3 (see [14]).
Suppose that𝒱 is a finite-dimensional vector space with an inner product 〈·,·〉𝒱 and 𝒮 is a space of block diagonal symmetric matrices with an inner product 〈·,·〉𝒮. 𝒜:𝒱→𝒮 is a linear mapping, and A0∈𝒮. The following optimization problem:
(2.9)min⁡〈c,x〉V,s.t.A(x)=A(x)+A0≥0,
is called a semidefinite programming (SDP). From convex duality, the dual problem associated with the SDP is defined as
(2.10)max⁡-〈A0,Z〉S,s.t.Aadj=c,Z≥0.
In the context of duality, we refer to the SDP (2.9) as the primal problem associated with (2.10).Consider the following SDP problem:(2.11)(P)max⁡Tr⁡(P),s.t.A(P)=[-P+A′PA+C′PC+QA′PB+C′PDB′PA+D′PCR+B′PB+D′PD]≥0.
By the definition of SDP, we can get the dual problem of (2.11).Proposition 2.4.
The dual problem of (2.11) can be formulated as
(2.12)(D)min⁡Tr⁡(QS+RT),s.t.{-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+I=0,Z=[SU′UT]≥0.Proof.
The objective of the primal problem can be rewritten as maximizing〈I,P〉𝒮n. The dual variable Z=[SU′UT]≥0, where (S,T,U)∈𝒮n×𝒮m×ℛm×n. The LMI constraint in the primal problem can be represented as
(2.13)A(P)=A(P)+A0=[-P+A′PA+C′PCA′PB+C′PDB′PA+D′PCB′PB+D′PD]+[Q00R].
According to the definition of adjoint mapping, we have 〈𝒜(P),Z〉𝒮n+m=〈P,𝒜adj(Z)〉𝒮n, that is, Tr[𝒜(P)Z]=Tr[P𝒜adj(Z)]. It follows 𝒜adj(Z)=-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′. By Definition 2.3, the objective of the dual problem is to minimize 〈A0,Z〉𝒮n+m=Tr(A0Z)=Tr(QS+RT). On the other hand, we will state that the constraints of the dual problem (2.10) are equivalent to the constraints of (2.12). Obviously, 𝒜adj(Z)=-I is equivalent to the equality constraint of (2.12). This ends the proof.The primal problem (2.9) is said to satisfy the Slater condition if there exists a primal feasible solution x0 such that A(x0)>0, that is, the primal problem (2.9) is strictly feasible. The dual problem (2.10) is said to satisfy the Slater condition if there is a dual feasible solution Z0 satisfying Z0>0, that is, the dual problem (2.10) is strictly feasible.Letp* and d* denote the optimal values of SDP (2.9) and the dual SDP (2.10), respectively. Let Xopt and Zopt denote the primal and dual optimal sets. Then, we have the following proposition (see [13, Theorem  3.1]).Proposition 2.5.
p*=d* if either of the following conditions holds.(1)
The primal problem (2.9) satisfies Slater condition.(2)
The dual problem (2.10) satisfies Slater condition.If both conditions hold, the optimal setsXopt and Zopt are nonempty. In this case, a feasible point x is optimal if and only if there is a feasible point Z satisfying the complementary slackness condition:
(2.14)A(x)Z=0.


### 2.3. Some Definitions and Lemmas

The following definitions and lemmas will be used frequently in this paper.Definition 2.6.
For any matrixM, there exists a unique matrix M+, called the Moore-Penrose inverse of M, satisfying
(2.15)MM+M=M,M+MM+=M+,(MM+)′=MM+,(M+M)′=M+M.Lemma 2.7 (extended Schur’s lemma).
Let matricesM=M′, N, and R=R′ be given with appropriate dimensions. Then, the following conditions are equivalent:(1)
M-NR+N′≥0, R≥0, and N(I-RR+)=0,(2)
[MNN′R]≥0,(3)
[RN′NM]≥0.Lemma 2.8 (see [7]).
For a symmetric matrixS, we have(1)
S+=(S+)′,(2)
S≥0 if and only if S+≥0,(3)
SS+=S+S.Lemma 2.9 (see [12]).
In system (2.1), suppose T∈N is given, and P(t)∈𝒮n,t=0,1,…,T+1, is an arbitrary family of matrices, then, for any x(0)∈ℛn, we have
(2.16)∑t=0TE[x(t)u(t)]′Q[P(t)][x(t)u(t)]=E[x′(T+1)P(T+1)x(T+1)-x′(0)P(0)x(0)],
where
(2.17)Q[P(t)]=[-P(t)+A′P(t+1)A+C′P(t+1)CA′P(t+1)B+C′P(t+1)DB′P(t+1)A+D′P(t+1)CB′P(t+1)B+D′P(t+1)D].Lemma 2.10.
System (2.1) is mean square stabilizable if and only if one of the following conditions holds.(1)
There are a matrixK and a symmetric matrix P>0 such that
(2.18)-P+(A+BK)P(A+BK)′+(C+DK)P(C+DK)′<0.
Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(2)
For any matrixY>0, there is a matrix K such that the following matrix equation:
(2.19)-P+(A+BK)P(A+BK)′+(C+DK)P(C+DK)′+Y=0
has a unique positive definite solution P>0. Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(3)
The dual problem(D) satisfies the Slater condition.Proof.
(1) and (2) can be derived from Proposition  2.2 in [15]. (3) is a discrete edition of Theorem  6 in [7]. The proof is similar to Theorem  6 in [7] and is omitted.To this end, we need the following assumptions throughout the paper.Assumption 2.11.
System (2.1) is mean square stabilizable.Assumption 2.12.
The feasible set of(P) is nonempty.


## 2.1. Problem Statement

Consider the following discrete-time stochastic system:(2.1)x(t+1)=Ax(t)+Bu(t)+[Cx(t)+Du(t)]w(t),x(0)=x0,t=0,1,2,…,
where x(t)∈ℛn,  u(t)∈ℛm are, respectively, the system state and control input. x0∈ℛn is the initial state and w(t)∈ℛ is the noise. A,C∈ℛn×n and B,D∈ℛn×m are constant matrices. {w(t),t∈N} is a sequence of real random variables defined on a complete probability space (Ω,ℱ,ℱt,P) with ℱt=σ{w(s):s∈Nt}, which is a wide sense stationary, second-order process with E[w(t)]=0 and E[w(s)w(t)]=δst, where δst is the Kronecker function. u(t) belongs to ℒℱ2(ℛm), the space of all ℛm-valued, ℱt-adapted measurable processes satisfying E(∑t=0∞∥u(t)∥2)<∞. We assume that the initial state x0 is independent of the noise w(t),t∈N.We first give the following definitions.Definition 2.1.
System (2.1) is called mean square stabilizable if there exists a feedback control u(t)=Kx(t) such that for any initial state x0, the closed-loop system
(2.2)x(t+1)=(A+BK)x(t)+(C+DK)x(t)w(t),x(0)=x0,t=0,1,2,…,
is asymptotically mean square stable, that is, the corresponding state of (2.2) satisfies limt→∞E∥x(t)∥2=0, where K∈ℛm×n is a constant matrix.For system (2.1), we define the admissible control set
(2.3)Uad={u(t)∈LF2(Rm),u(t)ismeansquarestabilizingcontrol.
The cost functional associated with system (2.1) is
(2.4)J(x0,u)=∑t=0∞E[x′(t)Qx(t)+u′(t)Ru(t)],
where Q and R are symmetric matrices with appropriate dimensions and may be indefinite. The LQ optimal control problem is to minimize the cost functional J(x0,u) over u∈Uad. We define the optimal value function as
(2.5)V(x0)=inf⁡u∈UadJ(x0,u).
Since the weighting matrices Q and R may be indefinite, the LQ problem is called an indefinite stochastic LQ control problem.Definition 2.2.
The LQ problem is called well posed if(2.6)-∞<V(x0)<∞,∀x0∈Rn.
If there exists an admissible control u* such that V(x0)=J(x0,u*), the LQ problem is called attainable and V(x0) is the optimal cost value. u*(t),t∈N, is called an optimal control, and x*(t),t∈N, corresponding to u*(t) is called the optimal trajectory.Stochastic algebraic Riccati equation (SARE) is a primary tool in solving stochastic LQ control problems. In [12], the following discrete SARE:(2.7)-P+A′PA+C′PC+Q-(A′PB+C′PD)(R+B′PB+D′PD)-1(B′PA+D′PC)=0,R+B′PB+D′PD>0,
was studied. The constraint that R+B′PB+D′PD>0 is demanded in (2.7). In fact, the corresponding LQ problem may have optimal control even if the condition is not satisfied. In this paper, we introduce the following generalized stochastic algebraic Riccati equation (GSARE),
(2.8)R(P)≡-P+A′PA+C′PC+Q-(A′PB+C′PD)(R+B′PB+D′PD)+(B′PA+D′PC)=0,R+B′PB+D′PD≥0,
which weakens the positive definiteness constraint of R+B′PB+D′PD to positive semidefiniteness constraint and replaces the inverse by Moore-Penrose inverse. Hence, (2.8) is an extension of (2.7).


## 2.2. Semidefinite Programming

In this subsection, we will introduce SDP and its dual. SDP is a special conic optimization problem and is defined as follows.Definition 2.3 (see [14]).
Suppose that𝒱 is a finite-dimensional vector space with an inner product 〈·,·〉𝒱 and 𝒮 is a space of block diagonal symmetric matrices with an inner product 〈·,·〉𝒮. 𝒜:𝒱→𝒮 is a linear mapping, and A0∈𝒮. The following optimization problem:
(2.9)min⁡〈c,x〉V,s.t.A(x)=A(x)+A0≥0,
is called a semidefinite programming (SDP). From convex duality, the dual problem associated with the SDP is defined as
(2.10)max⁡-〈A0,Z〉S,s.t.Aadj=c,Z≥0.
In the context of duality, we refer to the SDP (2.9) as the primal problem associated with (2.10).Consider the following SDP problem:(2.11)(P)max⁡Tr⁡(P),s.t.A(P)=[-P+A′PA+C′PC+QA′PB+C′PDB′PA+D′PCR+B′PB+D′PD]≥0.
By the definition of SDP, we can get the dual problem of (2.11).Proposition 2.4.
The dual problem of (2.11) can be formulated as
(2.12)(D)min⁡Tr⁡(QS+RT),s.t.{-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+I=0,Z=[SU′UT]≥0.Proof.
The objective of the primal problem can be rewritten as maximizing〈I,P〉𝒮n. The dual variable Z=[SU′UT]≥0, where (S,T,U)∈𝒮n×𝒮m×ℛm×n. The LMI constraint in the primal problem can be represented as
(2.13)A(P)=A(P)+A0=[-P+A′PA+C′PCA′PB+C′PDB′PA+D′PCB′PB+D′PD]+[Q00R].
According to the definition of adjoint mapping, we have 〈𝒜(P),Z〉𝒮n+m=〈P,𝒜adj(Z)〉𝒮n, that is, Tr[𝒜(P)Z]=Tr[P𝒜adj(Z)]. It follows 𝒜adj(Z)=-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′. By Definition 2.3, the objective of the dual problem is to minimize 〈A0,Z〉𝒮n+m=Tr(A0Z)=Tr(QS+RT). On the other hand, we will state that the constraints of the dual problem (2.10) are equivalent to the constraints of (2.12). Obviously, 𝒜adj(Z)=-I is equivalent to the equality constraint of (2.12). This ends the proof.The primal problem (2.9) is said to satisfy the Slater condition if there exists a primal feasible solution x0 such that A(x0)>0, that is, the primal problem (2.9) is strictly feasible. The dual problem (2.10) is said to satisfy the Slater condition if there is a dual feasible solution Z0 satisfying Z0>0, that is, the dual problem (2.10) is strictly feasible.Letp* and d* denote the optimal values of SDP (2.9) and the dual SDP (2.10), respectively. Let Xopt and Zopt denote the primal and dual optimal sets. Then, we have the following proposition (see [13, Theorem  3.1]).Proposition 2.5.
p*=d* if either of the following conditions holds.(1)
The primal problem (2.9) satisfies Slater condition.(2)
The dual problem (2.10) satisfies Slater condition.If both conditions hold, the optimal setsXopt and Zopt are nonempty. In this case, a feasible point x is optimal if and only if there is a feasible point Z satisfying the complementary slackness condition:
(2.14)A(x)Z=0.


## 2.3. Some Definitions and Lemmas

The following definitions and lemmas will be used frequently in this paper.Definition 2.6.
For any matrixM, there exists a unique matrix M+, called the Moore-Penrose inverse of M, satisfying
(2.15)MM+M=M,M+MM+=M+,(MM+)′=MM+,(M+M)′=M+M.Lemma 2.7 (extended Schur’s lemma).
Let matricesM=M′, N, and R=R′ be given with appropriate dimensions. Then, the following conditions are equivalent:(1)
M-NR+N′≥0, R≥0, and N(I-RR+)=0,(2)
[MNN′R]≥0,(3)
[RN′NM]≥0.Lemma 2.8 (see [7]).
For a symmetric matrixS, we have(1)
S+=(S+)′,(2)
S≥0 if and only if S+≥0,(3)
SS+=S+S.Lemma 2.9 (see [12]).
In system (2.1), suppose T∈N is given, and P(t)∈𝒮n,t=0,1,…,T+1, is an arbitrary family of matrices, then, for any x(0)∈ℛn, we have
(2.16)∑t=0TE[x(t)u(t)]′Q[P(t)][x(t)u(t)]=E[x′(T+1)P(T+1)x(T+1)-x′(0)P(0)x(0)],
where
(2.17)Q[P(t)]=[-P(t)+A′P(t+1)A+C′P(t+1)CA′P(t+1)B+C′P(t+1)DB′P(t+1)A+D′P(t+1)CB′P(t+1)B+D′P(t+1)D].Lemma 2.10.
System (2.1) is mean square stabilizable if and only if one of the following conditions holds.(1)
There are a matrixK and a symmetric matrix P>0 such that
(2.18)-P+(A+BK)P(A+BK)′+(C+DK)P(C+DK)′<0.
Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(2)
For any matrixY>0, there is a matrix K such that the following matrix equation:
(2.19)-P+(A+BK)P(A+BK)′+(C+DK)P(C+DK)′+Y=0
has a unique positive definite solution P>0. Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(3)
The dual problem(D) satisfies the Slater condition.Proof.
(1) and (2) can be derived from Proposition  2.2 in [15]. (3) is a discrete edition of Theorem  6 in [7]. The proof is similar to Theorem  6 in [7] and is omitted.To this end, we need the following assumptions throughout the paper.Assumption 2.11.
System (2.1) is mean square stabilizable.Assumption 2.12.
The feasible set of(P) is nonempty.


## 3. Main Results

In this section, we will establish the relationship among the optimality of the LQ problem, the SDP, and the GSARE.The following theorem reveals the relation between the SDP complementary optimal solutions and the GSARE.Theorem 3.1.
If a feasible solution of(P), P*, satisfies ℛ(P*)=0, and the feedback control
(3.1)u(t)=K*x(t)=-(R+B′P*B+D′P*D)+(B′P*A+D′P*C)x(t),t∈N,
is stabilizing, then there exist complementary optimal solutions of (P) and (D). In particular, P* is optimal to (P), and there is a complementary dual optimal solution Z* of (D), such that S*>0.Proof.
By the stability assumption of the controlu(t)=K*x(t) and Lemma 2.10, the equation
(3.2)-Y+(A+BK*)Y(A+BK*)′+(C+DK*)Y(C+DK*)′+I=0
has a positive solution Y*>0. Let
(3.3)S*=Y*,U*=K*S*=K*Y*,T*=K*U*′=K*Y*K*′,
that is,
(3.4)Z*=[S*U*′U*T*]=[Y*Y*K*′K*Y*K*Y*K*′]=[I0K*I][Y*000][IK*′0I]≥0.
By (3.2) and (3.3), we have
(3.5)-S*+AS*A′+CS*C′+BU*A′+DU*C′+AU*′B′+CU*′D′+BT*B′+DT*D′+I=0,
which shows Z* is a feasible solution of (D). A(P*)≥0 because P* is a feasible solution of (P). By Lemmas 2.7 and 2.8,
(3.6)A′P*B+C′P*D=(A′P*B+C′P*D)(R+B′P*B+D′P*D)(R+B′P*B+D′P*D)+=(A′P*B+C′P*D)(R+B′P*B+D′P*D)+(R+B′P*B+D′P*D).
In addition, we have
(3.7)-P*+A′P*A+C′P*C+Q=K*′(R+B′P*B+D′P*D)K*
by ℛ(P*)=0 and K*=-(R+B′P*B+D′P*D)+(B′P*A+D′P*C). Therefore, we have
(3.8)A(P*)Z*=[-P*+A′P*A+C′P*C+QA′P*B+C′P*DB′P*A+D′P*CR+B′P*B+D′P*D][S*U*′U*T*]=[I-K*′0I][R(P*)00R+B′P*B+D′P*D][I0-K*I][S*U*′U*T*]=[I-K*′0I][0000]=[0000].
Obviously, P* and Z* are complementary optimal solutions to (P) and (D).P* is optimal to (P), and Z* is optimal to (D).S*>0 is trivial because S*=Y*>0.In above, the assumption that the control in (3.1) is stabilizing is not automatically satisfied. The following theorem reveals that we can obtain a stabilizing feedback control by the dual SDP.Theorem 3.2.
Suppose thatZ=[SU′UT] is a feasible solution of (D) with S>0, then the feedback control u(t)=US-1x(t) is stabilizing.Proof.
First, we haveZ≥0 because Z is feasible to (D). By Lemma 2.7, the inequality T-US-1U′≥0 holds. By simple calculations, we have
(3.9)ASA′+BUA′+AU′B′+BUS-1U′B′=(A+BUS-1)S(A+BUS-1)′,CSC′+DUC′+CU′D′+DUS-1U′D′=(C+DUS-1)S(C+DUS-1)′.
Hence,
(3.10)0=-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+I≥-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BUS-1U′B′+DUS-1U′D′+I>-S+(A+BUS-1)S(A+BUS-1)′+(C+DUS-1)S(C+DUS-1)′.
Above inequality shows (2.18) has a positive definite solution S>0 with K=US-1. According to Lemma 2.10, u(t)=Kx(t)=US-1x(t) is stabilizing.The following theorem shows the relationship between the optimality of the LQ problem and the solution of GSARE.Theorem 3.3.
If LQ problem (2.1)–(2.5) is attainable with respect to any x0∈ℛn, then (P) must have an optimal solution P* such that ℛ(P*)=0.Proof.
Since the LQ problem is attainable, then the optimal value must be of the quadratic form [16]:
(3.11)inf⁡u∈UadJ(x0,u)=x0′Mx0,∀x0∈Rn.
Let (x*(·),u*(·)) be an optimal pair for the initial state x0. Let T→∞ and P(t)=P in (2.16), where P is an any feasible solution of (P), then we have
(3.12)x0′Px0+∑t=0∞E[x(t)u(t)]′Q(P)[x(t)u(t)]=0.
Then, a completion square means
(3.13)J(x0,u*)=∑t=0∞E[x*′(t)Qx*(t)+u*′(t)Ru*(t)]=x0′Px0+∑t=0∞E{[u*(t)-Kx*(t)]′(R+B′PB+D′PD)[u*(t)-Kx*(t)]+x*′(t)R(P)x*(t)[u*(t)-Kx*(t)]′},
where K=-(R+B′PB+D′PD)+(B′PA+D′PC). Since P is feasible to (P), we have R+B′PB+D′PD≥0 and ℛ(P)≥0 by Lemma 2.7. Then, the inequality
(3.14)x0′Mx0≡J(x0,u*)≥x0′Px0
holds for any feasible solution P to (P). This shows that M must be optimal to (P). Moreover, taking P=M in (3.13) and considering J(x0,u*)=x0′Mx0, we know that Ex*′(t)ℛ(M)x*(t)=0 for t∈N. Setting t=0 and noticing that x0 is arbitrary, it follows that ℛ(M)=0.
Below, we will showM is a feasible solution of (P). We consider the following SDP and its dual under a perturbation ɛ>0:(3.15)(Pɛ)max⁡Tr⁡(P),s.t.[-P+A′PA+C′PC+Q+ɛIA′PB+C′PDB′PA+D′PCR+ɛI+B′PB+D′PD]≥0,(3.16)(Dɛ)min⁡Tr⁡[(Q+ɛI)S+(R+ɛI)T],s.t.{-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+I=0,Z=[SU′UT]≥0.
Obviously, (Pɛ) satisfies the Slater condition because we assume that the feasible set of (P) is nonempty and (Dɛ) also satisfies the Slater condition by the mean square stabilizability assumption and Lemma 2.10. Hence, the complementary optimal solutions exist by Proposition 2.5. Take any dual feasible solution Z0=[S0U0′U0T0]. By the weak duality in conic optimization problems, we have
(3.17)Tr⁡(P)≤Tr⁡[(Q+ɛI)S0+(R+ɛI)T0].
Let P0 be a feasible solution of (P), then P0 is feasible to (Pɛ) for all ɛ≥0. Similar to Theorem  10 in [7], we conclude that, for any ɛ>0, there exists the unique optimal solution of (Pɛ), denoted by Pɛ*, and Pɛ*≥P0.
Together with (3.17), we know that Pɛ* are contained in a compact set with 0≤ɛ≤ɛ0 (ɛ0>0 is a constant). Then, take a convergent subsequence satisfying limi→∞Pɛi*=P0* with ɛi→0 as i→∞. Obviously, P0* is feasible to (P) because the feasible region of (Pɛ) monotonically shrinks as ɛ↓0. Define the perturbed cost functional
(3.18)Jɛ(x0,u)=∑t=0∞E[x′(t)Qɛx(t)+u′(t)Rɛu(t)],
where Rɛ=R+ɛI,Qɛ=Q+ɛI. By (3.13), we have
(3.19)Jɛ(x0,u)=x0′Pɛ*x0+∑t=0∞E{[u(t)-Kɛx(t)]′(Rɛ+D′Pɛ*D+B′Pɛ*B)[u(t)-Kɛx(t)]+x′(t)Rɛ(Pɛ*)x(t)},
for any u∈Uad, where Kɛ=-(Rɛ+B′Pɛ*B+D′Pɛ*D)+(B′Pɛ*A+D′Pɛ*C) and ℛɛ(Pɛ*) is the form of ℛ(Pɛ*) with Q and R replaced by Qɛ and Rɛ. Then, by Theorems  10 and 12 in [7],(3.20)inf⁡u∈UadJɛ(x0,u)=x0′Pɛ*x0.
Furthermore,
(3.21)x0′Pɛi*x0=inf⁡u∈UadJɛi(x0,u)≥inf⁡u∈UadJ(x0,u)=x0′Mx0.
Taking limit, we have x0′P0*x0≥x0′Mx0. On the other hand, x0′Mx0≥x0′P0*x0 because P0* is feasible to (P) and (3.14). So M=P0*. The feasibility of M is proved. The proof is completed.The following theorem studies the converse of Theorem3.3.Theorem 3.4.
If a feasible solution of(P), P*, satisfies ℛ(P*)=0 and the feedback control u*(t)=-(R+B′P*B+D′P*D)+(B′P*A+D′P*C)x(t) is stabilizing, then it must be optimal for LQ problem (2.1)–(2.5).Proof.
For anyu∈Uad, we have
(3.22)J(x0,u)=x0′P*x0+∑t=0∞E[u(t)-K*x(t)]′(R+B′P*B+D′P*D)[u(t)-K*x(t)]
by (3.13) and ℛ(P*)=0, where K*=-(R+B′P*B+D′P*D)+(B′P*A+D′P*C). Because u*(t)=K*x(t) is stabilizing, u*(t) must be optimal.The following theorem shows we can get the optimal feedback control by SDP dual optimal solution.Theorem 3.5.
Assume that (P) and (D) have complementary optimal solutionsP* and Z* with S*>0. Then, ℛ(P*)=0 and LQ problem (2.1)–(2.5) has an attainable optimal feedback control given by u*(t)=U*(S*)-1x*(t).Proof.
From the proof of Theorem3.1, we have
(3.23)A(P*)=[I-K*′0I][R(P*)00R+B′P*B+D′P*D][I0-K*I],
where K*=-(R+B′P*B+D′P*D)+(B′P*A+D′P*C). By complementary slackness condition A(P*)Z*=0 and the invertibility of [I-K*′0I], we have
(3.24)[R(P*)00R+B′P*B+D′P*D][I0-K*I][S*U*′U*T*]=[R(P*)S*R(P*)U*′-(R+B′P*B+D′P*D)(K*S*-U*)-(R+B′P*B+D′P*D)(K*U*′-T*)]=[0000].
So ℛ(P*)S*=0,ℛ(P*)U*′=0. On the other hand, T*-U*(S*)+U*′≥0,S*≥0 and U*=U*S*(S*)+ from Z*≥0 and Lemma 2.7. From the equality constraint in (2.12) and the above results, we have
(3.25)0=R(P*)[-S*+AS*A′+CS*C′+BU*A′+DU*C′+AU*′B′+CU*′D′+BT*B′+DT*D′+I]R(P*)≥R(P*)[AS*A′+CS*C′+BU*A′+DU*C′+AU*′B′+CU*′D′+BU*(S*)+U*′B′+DU*(S*)+U*′D′+I]R(P*)=[R(P*)]2+R(P*)[(CS*+DU*)(S*)+(CS*+DU*)′+(AS*+BU*)(S*)+(AS*+BU*)′]R(P*)≥[R(P*)]2.
The last inequality holds because (S*)+≥0 from Lemma 2.8. It follows that ℛ(P*)=0.
For anyu∈Uad, by (3.13), we get(3.26)J(x0,u)=x0′Px0+∑t=0∞E{[u(t)-Kx(t)]′(R+B′PB+D′PD)[u(t)-Kx(t)]+x′(t)R(P)x(t)},
where P is any feasible solution of (P) and K=-(R+B′PB+D′PD)+(B′PA+D′PC).ℛ(P)≥0 because of the feasibility of P. Then,
(3.27)J(x0,u)≥x0′Px0.
On the other hand, u*(t)=U*(S*)-1x*(t) is stabilizing by Theorem 3.2. Let u(t)=u*(t) and P=P* in (3.26), then it follows that
(3.28)J(x0,u*)=x0′P*x0+∑t=0∞E[u*(t)-K*x*(t)]′(R+B′P*B+D′P*D)[u*(t)-K*x*(t)],
where K*=-(R+B′P*B+D′P*D)+(B′P*A+D′P*C). Below we prove J(x0,u*)=x0′P*x0. Applying complementary slackness condition A(P*)Z*=0 and above proof, we have
(3.29)[R(P*)00R+B′P*B+D′P*D][I0-K*I][S*U*′U*T*]=[R(P*)S*(R+B′P*B+D′P*D)U*+(B′P*A+D′P*C)S*R(P*)U*′(R+B′P*B+D′P*D)T+(B′P*A+D′P*C)U*′]=[0000].
Hence, (R+B′P*B+D′P*D)U*=-(B′P*A+D′P*C)S*. Then,
(3.30)[u*(t)-K*x*(t)]′(R+B′P*B+D′P*D)[u*(t)-K*x*(t)]=u*′(t)(R+B′P*B+D′P*D)u*(t)+2u*′(t)(B′P*A+D′P*C)x*(t)+x*′(t)(A′P*B+C′P*D)(R+B′P*B+D′P*D)+(B′P*A+D′P*C)x*(t)=u*′(t)(R+B′P*B+D′P*D)u*(t)-2u*′(t)(R+B′P*B+D′P*D)U*(S*)-1x*(t)+x*′(t)(S*)-1U*′(R+B′P*B+D′P*D)U*(S*)-1x*(t)=[u*(t)-U*(S*)-1x*(t)]′(R+B′P*B+D′P*D)[u*(t)-U*(S*)-1x*(t)]=0.
It follows from (3.27) and (3.28) that
(3.31)J(x0,u*)=x0′P*x0≤J(x0,u),∀u∈Uad.
The optimality of u*(t) is proved.


## 4. Conclusion

In this paper, we use the SDP approach to study discrete-time indefinite stochastic LQ control problem. Some relations are given and are summarized as follows. The condition that LQ problem is attainable can induce that(P) has an optimal solution P* satisfying GSARE (Theorem 3.3). Theorems 3.4 and 3.5 give two suffcient conditions for LQ problem attainability by GSARE and complementary optimal solutions of (P) and (D). Moreover, by dual SDP, we can get stabilized feedback control (Theorem 3.2). What we have obtained can be viewed as a discrete-time version of [9]. Of course, there are many open problems to be solved. For instance, the indefinite LQ problems for Markovian jumps or time-variant system merit further study.



---

*Source: 674087-2012-01-15.xml*