# Discrete-Time Indefinite Stochastic LQ Control via SDP and LMI Methods

Authors: Shaowei Zhou; Weihai Zhang
Journal: Journal of Applied Mathematics
 (2012)

---

## Abstract

This paper studies a discrete-time stochastic LQ problem over an infinite time horizon with state-and control-dependent noises, whereas the weighting matrices in the cost function  are  allowed to be indefinite. We mainly use semidefinite programming (SDP) and its duality to treat corresponding problems. Several relations among stability, SDP complementary duality, the existence of the solution to stochastic algebraic Riccati equation (SARE), and the optimality of LQ problem are established. We can test mean square stabilizability and solve SARE via SDP by LMIs method.


---

## Body

## 1.  Introduction

Stochastic linear quadratic (LQ) control problem was first studied by Wonham [1] and has become a popular research field of modern control theory,  which has been extensively studied by many researchers; see, for example, [2–12]. We should point out that, in the most early literature about stochastic LQ issue, it is always assumed that the control weighting matrix R is positive definite and the state weight matrix Q is positive semi-definite. A breakthrough belongs to [9], where a surprising fact was found that for a stochastic LQ modeled by a stochastic Itô-type differential system, even if the cost-weighting matrices Q and R are indefinite, the original LQ optimization may still be well-posed. This finding reveals the essential difference between deterministic and stochastic systems. After that, follow-up research was carried out and a lot of important results were obtained. In [10–12], continuous-time stochastic LQ control problem with indefinite weighting matrices was studied. The authors in [10] provided necessary and sufficient conditions for the solvability of corresponding generalized differential Riccati equation (GDRE). The authors introduced LMIs whose feasibility is shown to be equivalent to the solvability of SARE and developed a computational approach to the SARE by SDP in [11]. Furthermore, stochastic indefinite LQ problems with jumps in infinite time horizon and finite time horizon were, respectively, studied in [13, 14]. Discrete-time case was also studied in [15–17]. Among these, a central issue is solving corresponding SARE. A traditional method is to consider the so-called associated Hamiltonian matrix. However, this method does not work on when R is indefinite.In this paper, we use SDP approach introduced in [11, 18] to discuss discrete-time indefinite stochastic LQ control problem over an infinite time horizon. Several equivalent relations between the stabilization/optimality of the LQ problem and the duality of SDP are established. We show that the stabilization is equivalent to the feasibility of the dual SDP. Furthermore, we prove that the maximal solution to SARE associated with the LQ problem can be obtained by solving the corresponding SDP. What we have obtained extend the results of [11] from continuous-time case to discrete-time case and the results of [15] from finite time horizon to infinite time horizon.The organization of this paper is as follows. In Section2, we formulate the discrete-time indefinite stochastic LQ problem in an infinite time horizon and present some preliminaries including some definitions, lemmas, and SDP. Section 3 is devoted to the relations between stabilization and dual SDP. In Section 4, we develop a computational approach to the SARE via SDP and characterize the optimal LQ control by the maximal solution to the SARE. Some numerical examples are presented in Section 5.Notations 1.
ℛn: n-dimensional Euclidean space. ℛn×m: the set of all n×m matrices. 𝒮n: the set of all n×n symmetric matrices. A′: the transpose of matrix A. A≥0(A>0): A is positive semidefinite (positive definite). I: the identity matrix. σ(L): the spectrum set of the operator L. ℛ: the set of all real numbers. 𝒞: the set of all complex numbers. 𝒞-: the open left-hand side complex plane. Tr(M): the trace of a square matrix M. 𝒜adj: the adjoint mapping of 𝒜.


## 2. Preliminaries

### 2.1. Problem Statement

Consider the following discrete-time stochastic system:(2.1)x(t+1)=Ax(t)+Bu(t)+[Cx(t)+Du(t)]w(t),x(0)=x0,t=0,1,2,…,
where x(t)∈ℛn, u(t)∈ℛm are the system state and control input, respectively. x0∈ℛn is the initial state, and w(t)∈ℛ is the noise. A,C∈ℛn×n and B,D∈ℛn×m are constant matrices. {w(t),t=0,1,2,…} is a sequence of real random variables defined on a filtered probability space (Ω,ℱ,ℱt,P) with ℱt=σ{w(s):s=0,1,2,…,t}, which is a wide sense stationary, second-order process with E[w(t)]=0 and E[w(s)w(t)]=δst, where δst is the Kronecker function. u(t) belongs to ℒℱ2(ℛm), the space of all ℛm-valued, ℱt-adapted measurable processes satisfying(2.2)E(∑t=0∞‖u(t)‖2)<∞.
We assume that the initial state x0 is independent of the noise w(t).We first give the following definitions.Definition 2.1 (see [17]).
The following system(2.3)x(t+1)=Ax(t)+Cx(t)w(t),x(0)=x0,t=0,1,2,…,
is called asymptotically mean square stable if, for any initial state x0, the corresponding state satisfies limt→∞E∥x(t)∥2=0.Definition 2.2 (see [17]).
System (2.1) is called stabilizable in the mean square sense if there exists a feedback control u(t)=Kx(t) such that, for any initial state x0, the closed-loop system
(2.4)x(t+1)=(A+BK)x(t)+(C+DK)x(t)w(t),x(0)=x0,t=0,1,2,…,
is asymptotically mean square stable; that is, the corresponding state x(·) of (2.4) satisfies limt→∞E∥x(t)∥2=0, where K∈ℛm×n is a constant matrix.
For system (2.1), we define the admissible control set
(2.5)Uad={u(t)∈LF2(Rm),u(t)ismeansquarestabilizingcontrol.
The cost function associated with system (2.1) is
(2.6)J(x0,u)=∑t=0∞E[x′(t)Qx(t)+u′(t)Ru(t)],
where Q and R are symmetric matrices with appropriate dimensions and may be indefinite. The LQ optimal control problem is to minimize the cost functional J(x0,u) over u∈Uad. We define the optimal value function as
(2.7)V(x0)=inf⁡u∈UadJ(x0,u).
Since the weighting matrices Q and R may be indefinite, the LQ problem is called an indefinite LQ control problem.Definition 2.3.
The LQ problem is called well-posed if(2.8)-∞<V(x0)<∞,∀x0∈Rn.
If there exists an admissible control u* such that V(x0)=J(x0,u*), the LQ problem is called attainable and u* is called an optimal control.
Stochastic algebraic Riccati equation (SARE) is a primary tool in solving LQ control problems. Associated with the above LQ problem, there is a discrete SARE:(2.9)R(P)≡-P+A′PA+C′PC+Q-(A′PB+C′PD)(R+B′PB+D′PD)-1(B′PA+D′PC)=0,R+B′PB+D′PD>0.Definition 2.4.
A symmetric matrixPmax is called a maximal solution to (2.9) if Pmax is a solution to (2.9) and Pmax≥P for any symmetric solution P to (2.9).
Throughout this paper, we assume that system (2.1) is mean square stabilizable.


### 2.2. Some Definitions and Lemmas

The following definitions and lemmas will be used frequently in this paper.Definition 2.5.
For any matrixM, there exists a unique matrix M+, called the Moore-Penrose inverse, satisfying
(2.10)MM+M=M,M+MM+=M+,(MM+)′=MM+,(M+M)′=M+M.Definition 2.6.
Suppose that𝒱 is a finite-dimensional vector space and 𝒮 is a space of block diagonal symmetric matrices with given dimensions. 𝒜: 𝒱→𝒮 is a linear mapping and A0∈𝒮. Then the inequality
(2.11)A(x)+A0≥(>)0
is called a linear matrix inequality (LMI). An LMI is called feasible if there exists at least one x∈𝒱 satisfying the above inequality and x is called a feasible point.Lemma 2.7 (Schur’s lemma).
Let matricesM=M′, N and R=R′>0 be given with appropriate dimensions. Then the following conditions are equivalent:(1)
M-NR-1N′≥(>)0,(2)
[MNN′R]≥(>)0,(3)
[RN′NM]≥(>)0.Lemma 2.8 (extended Schur’s lemma).
Let matricesM=M′, N and R=R′ be given with appropriate dimensions. Then the following conditions are equivalent:(1)
M-NR+N′≥0, R≥0, and N(I-RR+)=0,(2)
[MNN′R]≥0,(3)
[RN′NM]≥0.Lemma 2.9 (see [11]).
For a symmetric matrixS, one has(1)
S+=(S+)′,(2)
S≥0 if and only if S+≥0,(3)
SS+=S+S.


### 2.3. Semidefinite Programming

Definition 2.10 (see [19]).
Suppose that𝒱 is a finite-dimensional vector space with an inner product 〈·,·〉𝒱 and 𝒮 is a space of block diagonal symmetric matrices with an inner product 〈·,·〉𝒮. The following optimization problem
(2.12)min⁡〈c,x〉VsubjecttoA(x)=A(x)+A0≥0
is called a semidefinite programming (SDP). From convex duality, the dual problem associated with the SDP is defined as
(2.13)max⁡-〈A0,Z〉SsubjecttoAadj=c,Z≥0.
In the context of duality, we refer to the SDP (2.12) as the primal problem associated with (2.13).Remark 2.11.
Definition2.10 is more general than Definition  6 in [11].
Letp* denote the optimal value of SDP (2.12); that is,
(2.14)p*=inf⁡{〈c,x〉V∣A(x)≥0},
and let d* denote the optimal value of the dual SDP (2.13); that is,
(2.15)d*=sup⁡{-〈A0,Z〉S∣Z≥0,Aadj=c}.
Let Xopt and Zopt denote the primal and dual optimal sets; that is,
(2.16)Xopt={x∣A(x)≥0,〈c,x〉V=p*},Zopt={Z∣Z≥0,Aadj=c,-〈A0,Z〉S=d*}.
About SDP, we have the following proposition (see [20, Theorem 3.1]).Proposition 2.12.
p*=d* if either of the following conditions holds.(1)
The primal problem (2.12) is strictly feasible; that is, there is an x with A(x)>0.(2)
The dual problem (2.13) is strictly feasible; that is, there is a Z with Z>0 and 𝒜adj=c.If both conditions hold, the optimal setsXopt and Zopt are nonempty. In this case, a feasible point x is optimal if and only if there is a feasible point Z satisfying the complementary slackness condition:(2.17)A(x)Z=0.


## 2.1. Problem Statement

Consider the following discrete-time stochastic system:(2.1)x(t+1)=Ax(t)+Bu(t)+[Cx(t)+Du(t)]w(t),x(0)=x0,t=0,1,2,…,
where x(t)∈ℛn, u(t)∈ℛm are the system state and control input, respectively. x0∈ℛn is the initial state, and w(t)∈ℛ is the noise. A,C∈ℛn×n and B,D∈ℛn×m are constant matrices. {w(t),t=0,1,2,…} is a sequence of real random variables defined on a filtered probability space (Ω,ℱ,ℱt,P) with ℱt=σ{w(s):s=0,1,2,…,t}, which is a wide sense stationary, second-order process with E[w(t)]=0 and E[w(s)w(t)]=δst, where δst is the Kronecker function. u(t) belongs to ℒℱ2(ℛm), the space of all ℛm-valued, ℱt-adapted measurable processes satisfying(2.2)E(∑t=0∞‖u(t)‖2)<∞.
We assume that the initial state x0 is independent of the noise w(t).We first give the following definitions.Definition 2.1 (see [17]).
The following system(2.3)x(t+1)=Ax(t)+Cx(t)w(t),x(0)=x0,t=0,1,2,…,
is called asymptotically mean square stable if, for any initial state x0, the corresponding state satisfies limt→∞E∥x(t)∥2=0.Definition 2.2 (see [17]).
System (2.1) is called stabilizable in the mean square sense if there exists a feedback control u(t)=Kx(t) such that, for any initial state x0, the closed-loop system
(2.4)x(t+1)=(A+BK)x(t)+(C+DK)x(t)w(t),x(0)=x0,t=0,1,2,…,
is asymptotically mean square stable; that is, the corresponding state x(·) of (2.4) satisfies limt→∞E∥x(t)∥2=0, where K∈ℛm×n is a constant matrix.
For system (2.1), we define the admissible control set
(2.5)Uad={u(t)∈LF2(Rm),u(t)ismeansquarestabilizingcontrol.
The cost function associated with system (2.1) is
(2.6)J(x0,u)=∑t=0∞E[x′(t)Qx(t)+u′(t)Ru(t)],
where Q and R are symmetric matrices with appropriate dimensions and may be indefinite. The LQ optimal control problem is to minimize the cost functional J(x0,u) over u∈Uad. We define the optimal value function as
(2.7)V(x0)=inf⁡u∈UadJ(x0,u).
Since the weighting matrices Q and R may be indefinite, the LQ problem is called an indefinite LQ control problem.Definition 2.3.
The LQ problem is called well-posed if(2.8)-∞<V(x0)<∞,∀x0∈Rn.
If there exists an admissible control u* such that V(x0)=J(x0,u*), the LQ problem is called attainable and u* is called an optimal control.
Stochastic algebraic Riccati equation (SARE) is a primary tool in solving LQ control problems. Associated with the above LQ problem, there is a discrete SARE:(2.9)R(P)≡-P+A′PA+C′PC+Q-(A′PB+C′PD)(R+B′PB+D′PD)-1(B′PA+D′PC)=0,R+B′PB+D′PD>0.Definition 2.4.
A symmetric matrixPmax is called a maximal solution to (2.9) if Pmax is a solution to (2.9) and Pmax≥P for any symmetric solution P to (2.9).
Throughout this paper, we assume that system (2.1) is mean square stabilizable.


## 2.2. Some Definitions and Lemmas

The following definitions and lemmas will be used frequently in this paper.Definition 2.5.
For any matrixM, there exists a unique matrix M+, called the Moore-Penrose inverse, satisfying
(2.10)MM+M=M,M+MM+=M+,(MM+)′=MM+,(M+M)′=M+M.Definition 2.6.
Suppose that𝒱 is a finite-dimensional vector space and 𝒮 is a space of block diagonal symmetric matrices with given dimensions. 𝒜: 𝒱→𝒮 is a linear mapping and A0∈𝒮. Then the inequality
(2.11)A(x)+A0≥(>)0
is called a linear matrix inequality (LMI). An LMI is called feasible if there exists at least one x∈𝒱 satisfying the above inequality and x is called a feasible point.Lemma 2.7 (Schur’s lemma).
Let matricesM=M′, N and R=R′>0 be given with appropriate dimensions. Then the following conditions are equivalent:(1)
M-NR-1N′≥(>)0,(2)
[MNN′R]≥(>)0,(3)
[RN′NM]≥(>)0.Lemma 2.8 (extended Schur’s lemma).
Let matricesM=M′, N and R=R′ be given with appropriate dimensions. Then the following conditions are equivalent:(1)
M-NR+N′≥0, R≥0, and N(I-RR+)=0,(2)
[MNN′R]≥0,(3)
[RN′NM]≥0.Lemma 2.9 (see [11]).
For a symmetric matrixS, one has(1)
S+=(S+)′,(2)
S≥0 if and only if S+≥0,(3)
SS+=S+S.


## 2.3. Semidefinite Programming

Definition 2.10 (see [19]).
Suppose that𝒱 is a finite-dimensional vector space with an inner product 〈·,·〉𝒱 and 𝒮 is a space of block diagonal symmetric matrices with an inner product 〈·,·〉𝒮. The following optimization problem
(2.12)min⁡〈c,x〉VsubjecttoA(x)=A(x)+A0≥0
is called a semidefinite programming (SDP). From convex duality, the dual problem associated with the SDP is defined as
(2.13)max⁡-〈A0,Z〉SsubjecttoAadj=c,Z≥0.
In the context of duality, we refer to the SDP (2.12) as the primal problem associated with (2.13).Remark 2.11.
Definition2.10 is more general than Definition  6 in [11].
Letp* denote the optimal value of SDP (2.12); that is,
(2.14)p*=inf⁡{〈c,x〉V∣A(x)≥0},
and let d* denote the optimal value of the dual SDP (2.13); that is,
(2.15)d*=sup⁡{-〈A0,Z〉S∣Z≥0,Aadj=c}.
Let Xopt and Zopt denote the primal and dual optimal sets; that is,
(2.16)Xopt={x∣A(x)≥0,〈c,x〉V=p*},Zopt={Z∣Z≥0,Aadj=c,-〈A0,Z〉S=d*}.
About SDP, we have the following proposition (see [20, Theorem 3.1]).Proposition 2.12.
p*=d* if either of the following conditions holds.(1)
The primal problem (2.12) is strictly feasible; that is, there is an x with A(x)>0.(2)
The dual problem (2.13) is strictly feasible; that is, there is a Z with Z>0 and 𝒜adj=c.If both conditions hold, the optimal setsXopt and Zopt are nonempty. In this case, a feasible point x is optimal if and only if there is a feasible point Z satisfying the complementary slackness condition:(2.17)A(x)Z=0.


## 3.  Mean Square Stabilization

The stabilization assumption of system (2.1) is basic for the study on the stochastic LQ problem for infinite horizon case. So, we will cite some equivalent conditions in verifying the stabilizability.Lemma 3.1 (see [16, 21]).
System (2.1) is mean square stabilizable if and only if one of the following conditions holds.(1)
There are a matrixK and a symmetric matrix P>0 such that
(3.1)-P+(A+BK)P(A+BK)′+(C+DK)P(C+DK)′<0.
Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(2)
There are a matrixK and a symmetric matrix P>0 such that
(3.2)-P+(A+BK)′P(A+BK)+(C+DK)′P(C+DK)<0.
Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(3)
For any matrixY>0, there is a matrix K such that the following matrix equation
(3.3)-P+(A+BK)′P(A+BK)+(C+DK)′P(C+DK)+Y=0
has a unique positive definite solution P>0. Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(4)
For any matrixY>0, there is a matrix K such that the following matrix equation
(3.4)-P+(A+BK)P(A+BK)′+(C+DK)P(C+DK)′+Y=0
has a unique positive definite solution P>0. Moreover, the stabilizing feedback control is given by u(t)=Kx(t).(5)
There exist matricesP>0 and U such that the following LMI
(3.5)[-PAP+BUCP+DUPA′+U′B′-P0PC′+U′D′0-P]<0
holds. Moreover, the stabilizing feedback control is given by u(t)=UP-1x(t).Below, we will construct the relation between the stabilization and the dual SDP. First, we assume that the interior of the set𝒫={P∈𝒮n|ℛ(P)≥0,R+B′PB+D′PD>0} is nonempty; that is, there is a P0∈𝒮n such that ℛ(P0)>0 and R+B′P0B+D′P0D>0.Consider the following SDP problem:(3.6)max⁡Tr⁡(P)subject toA(P)=[-P+A′PA+C′PC+QA′PB+C′PD0B′PA+D′PCR+B′PB+D′PD000P-P0]≥0.By the definition of SDP, we can get the dual problem of (3.6).Theorem 3.2.
The dual problem of (3.6) can be formulated as
(3.7)max⁡-Tr⁡(QS+RT)+Tr⁡(P0W)subject to{-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+W+I=0,[SU′UT]≥0,W≥0.Proof.
The objective of the primal problem can be rewritten as maximizing〈I,P〉𝒮n. Define the dual variable Z∈𝒮2n+m as
(3.8)Z=[SU′UTY′YW]≥0,
where (S,T,W,U,Y)∈𝒮n×𝒮m×𝒮n×ℛm×n×ℛn×(n+m). The LMI constraint in the primal problem can be represented as
(3.9)A(P)=A(P)+A0=[-P+A′PA+C′PCA′PB+C′PD0B′PA+D′PCB′PB+D′PD000P]+[Q000R000-P0],
that is,
(3.10)A(P)=[-P+A′PA+C′PCA′PB+C′PD0B′PA+D′PCB′PB+D′PD000P],A0=[Q000R000-P0].
According to the definition of adjoint mapping, we have 〈𝒜(P),Z〉𝒮2n+m=〈P,𝒜adj(Z)〉𝒮n, that is, Tr[𝒜(P)Z]=Tr[P𝒜adj(Z)]. It follows 𝒜adj(Z)=-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+W. By Definition 2.10, the objective of the dual problem is to maximize -〈A0,Z〉𝒮2n+m=-Tr(A0Z)=-Tr(QS+RT)+Tr(P0W). On the other hand, we will state that the constraints of the dual problem (2.13) are equivalent to the constraints of (3.7). Obviously, 𝒜adj(Z)=-I is equivalent to the equality constraint of (3.7). Furthermore, notice that the matrix variable Y does not work on in the above formulation and therefore can be treated as zero matrix. So, the condition Z≥0 is equivalent to
(3.11)[SU′UT]≥0,W≥0.
This ends the proof.Remark 3.3.
This proof is simpler than the proof in [11] because we use a more general dual definition.The following theorem reveals that the stabilizability of discrete stochastic system can be also regarded as a dual concept of optimality. This result is a discrete edition of Theorem  6 in [11].Theorem 3.4.
The system (2.1) is mean square stabilizable if and only if the dual problem (3.7) is strictly feasible.Proof.
First, we prove the necessary condition. Assume that system (2.1) is mean square stabilizable by the feedback u(t)=Kx(t). By Lemma 3.1, there is a unique S>0 satisfying
(3.12)-S+(A+BK)S(A+BK)′+(C+DK)S(C+DK)′+Y+I=0,
where Y>0 is a fixed matrix. Set U=KS, then U′=SK′. The above equality can be written as
(3.13)-S+ASA′+AU′B′+BUA′+BUS-1U′B′+CSC′+CU′D′+DUC′+DUS-1U′D′+Y+I=0.
Let ɛ>0,  T=ɛI+US-1U′ and W=-ɛBB′-ɛDD′+Y. Obviously, T and W satisfy
(3.14)-S+ASA′+AU′B′+BUA′+BTB′+CSC′+CU′D′+DUC′+DTD′+W+I=0.
We have W>0 for sufficiently small ɛ>0. By Lemma 2.7, T>US-1U′ is equivalent to [SU′UT]>0. We conclude that the dual problem (3.7) is strictly feasible.
Next, we prove the sufficient condition. Assume that the dual problem is strictly feasible; that is,-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+W+I=0, [SU′UT]>0, W>0. It implies that there are S>0, T and U such that
(3.15)-S+ASA′+AU′B′+BUA′+BTB′+CSC′+CU′D′+DUC′+DTD′<0,T>US-1U′.
It follows that
(3.16)-S+ASA′+AU′B′+BUA′+BUS-1U′B′+CSC′+CU′D′+DUC′+DUS-1U′D′<0.
Let K=US-1. The above inequality is equivalent to
(3.17)-S+(A+BK)S(A+BK)′+(C+DK)S(C+DK)′<0.
By Lemma 3.1, system (2.1) is mean square stabilizable.


## 4. Solutions to SARE and SDP

The following theorem will state the existence of the solution of the SARE (2.9) via SDP (3.6).Theorem 4.1.
The optimal set of (3.6) is nonempty, and any optimal solution P* must satisfy the SARE (2.9).Proof.
Since system (2.1) is mean square stabilizable, by Theorem 3.4, (3.7) is strictly feasible. Equation (3.6) is strictly feasible because P0 is a interior point of 𝒫. By Proposition 2.12, (3.6) is nonempty and P* satisfies A(P*)Z=0; that is,
(4.1)[-P*+A′P*A+C′P*C+QA′P*B+C′P*DB′P*A+D′P*CB′P*B+D′P*D+R00P*-P0][SU′UT00W]=0.
From the above equality, we have the following equalities:
(4.2)(-P*+A′P*A+C′P*C+Q)S+(A′P*B+C′P*D)U=0,(4.3)(-P*+A′P*A+C′P*C+Q)U′+(A′P*B+C′P*D)T=0,(4.4)(B′P*A+D′P*C)S+(B′P*B+D′P*D+R)U=0,(4.5)(B′P*A+D′P*C)U′+(B′P*B+D′P*D+R)T=0,(4.6)(P*-P0)W=0.
Moreover, R+D′P*D+B′P*B>0 because of R+D′P0D+B′P0B>0 and P*≥P0. Then by (4.4), U=-(B′P*B+D′P*D+R)-1(B′P*A+D′P*C)S. Substituting it into (4.2) yields ℛ(P*)S=0. Remember that S, T, U, W satisfy the equality constraint in (3.7). Multiplying both sides by ℛ(P*), we have
(4.7)R(P*)[ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BTB′+DTD′+W+I]R(P*)=0.
Considering [SU′UT]≥0, it follows from Lemma 2.8 that T≥US+U′ and U=USS+. By Lemma 2.9, S+≥0 and SS+=S+S. So we have
(4.8)R(P*)(ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BUS+U′B′+DUS+U′D′)R(P*)=R(P*)[(CS+DU)S+(SC′+U′D′)+(AS+BU)S+(SA′+U′B′)]R(P*)≤0.
Then it follows that ℛ(P*)ℛ(P*)≤0. It yields ℛ(P*)=0 due to ℛ(P*)≥0.The following theorem shows that any optimal solution of the primal SDP results in a stabilizing control for LQ problem.Theorem 4.2.
LetP* be an optimal solution to the SDP (3.6). Then the feedback control u(t)=-(R+B′P*B+D′P*D)-1(B′P*A+D′P*C)x(t) is mean square stabilizing for system (2.1).Proof.
Optimal dual variablesS, T, U, W satisfy (4.2)–(4.6). U=-(B′P*B+D′P*D+R)-1(B′P*A+D′P*C)S. Now we show S>0. Let Sx=0, x∈ℛn. The constraints in (3.7) imply
(4.9)x′(ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BUS+U′B′+DUS+U′D′+W+I)x≤0.
Similar to the proof of Theorem 4.1, we have x=0. We conclude that S>0 from S≥0. Again by the equality constraint in (3.7), we have
(4.10)-S+ASA′+CSC′+BUA′+DUC′+AU′B′+CU′D′+BUS-1U′B′+DUS-1U′D′=-S+(A+BUS-1)S(A+BUS-1)′+(C+DUS-1)S(C+DUS-1)′<0.
By Lemma 3.1, the above inequality is equivalent to the mean square stabilizability of system (2.1) with u(t)=US-1x(t)=-(R+B′P*B+D′P*D)-1(B′P*A+D′P*C)x(t). This ends the proof.Theorem 4.3.
There is a unique optimal solution to (3.6), which is the maximal solution to SARE (2.9).Proof.
The proof is similar to Theorem  9 in [11] and is omitted.Theorem 4.4.
Assume that𝒫 is nonempty, then SARE (2.9) has a maximal solution Pmax, which is the unique optimal solution to SDP:
(4.11)max⁡Tr⁡(P)subject toA(P)=[-P+A′PA+C′PC+QA′PB+C′PDB′PA+D′PCR+B′PB+D′PD]R+B′PB+D′PD>0.≥0,Proof.
The first assertion is Theorem  4 in [16]. As to the second part, we proceed as follows. By Lemma 2.7, Pmax satisfies the constraints in (4.11). Pmax is an optimal solution to (4.11) due to the maximality. Next we prove the uniqueness. Assume that Pmax′ is another optimal solution to (4.11). Then we have Tr(Pmax)=Tr(Pmax′). According to Definition 2.4, Pmax≥Pmax′. This yields Pmax=Pmax′. Hence, the proof of the theorem is completed.Remark 4.5.
Here we drop the assumption that the interior of𝒫 is nonempty.Remark 4.6.
Theorem4.4 presents that the maximal solution to SARE (2.9) can be obtained by solving SDP (4.11). The result provides us a computational approach to the SARE. Furthermore, as shown in [16], the relationship between the LQ value function and the maximal solution to SARE (2.9) can be established; that is, assuming that 𝒫 is nonempty, then the value function V(x0)=x0′Pmaxx0 and the optimal control can be expressed as u(t)=-(R+B′PmaxB+D′PmaxD)-1(B′PmaxA+D′PmaxC)x(t).
The above results represent SARE (2.9) may exist a solution even if R is indefinite (even negative definite). To describe the allowable negative degree, we give the following definition to solvability margin.Definition 4.7 (see [11]).
The solvability marginr* is defined as the largest nonnegative scalar r≥0 such that (2.9) has a solution for any R>-r*I.By Theorem4.4, the following conclusion is obvious.Theorem 4.8.
The solvability marginr* can be obtained by solving the following SDP:
(4.12)max⁡rsubject to{[-P+A′PA+C′PC+QA′PB+C′PDB′PA+D′PCB′PB+D′PD-rI]≥0,B′PB+D′PD-rI>0,r>0.


## 5.  Numerical Examples

Consider the system (2.1) with A, B, C, D as follows:(5.1)A=[-0.43262.28771.1892-1.6656-1.1465-0.03760.12531.19090.3273],B=[-0.8147-0.9134-0.90580.63240.12700.0975],C=[0.1746-0.58830.1139-0.18672.18321.06680.7258-0.13640.0593],D=[0.27850.96490.54690.15760.9575-0.9706].


### 5.1.  Mean Square Stabilizability

In order to test the mean square stabilizability of system (2.1), we only need to check weather or not LMI (3.5) is feasible by Theorem 3.2. Making use of LMI feasp solver [22], we find matrices P and U satisfying (3.5):(5.2)P=[4.82030.21052.07570.21051.2812-2.22872.0757-2.22875.3885],U=[-5.4185-0.9228-1.31134.54990.94260.5770],
and the stabilizing control u(t)=UP-1x(t); that is,(5.3)u(t)=[-0.6606-2.1116-0.86230.96790.4079-0.0970]x(t).


### 5.2.  Solutions of SARE

LetQ=I in SARE (2.9). Below, we solve SARE (2.9) via the SDP (4.11) (LMI mincx solver [22]).Case 1.
R is positive definite. Choose R=[2112], and we obtain the maximal solution to (2.9):
(5.4)Pmax⁡=[44.2780-100.5744-48.8271-100.5744360.6942182.4572-48.8271182.457294.6103].Case 2.
R is indefinite. Choose R=[0.20.30.30.2], and we obtain the maximal solution to (2.9):
(5.5)Pmax⁡=[17.7975-40.0973-19.1900-40.0973135.218967.6790-19.190067.679035.5117].Case 3.
R is negative definite. First we can get the solvability margin r*=0.4936 by solving SDP (4.12) (LMI gevp solver [22]). Hence (2.9) has a maximal solution when R>-0.4936I. Choose R=-0.4I, and we obtain the maximal solution to (2.9):
(5.6)Pmax⁡=[29.6283-66.7897-32.2431-66.7897234.0792117.9065-32.2431117.906561.3301].


## 5.1.  Mean Square Stabilizability

In order to test the mean square stabilizability of system (2.1), we only need to check weather or not LMI (3.5) is feasible by Theorem 3.2. Making use of LMI feasp solver [22], we find matrices P and U satisfying (3.5):(5.2)P=[4.82030.21052.07570.21051.2812-2.22872.0757-2.22875.3885],U=[-5.4185-0.9228-1.31134.54990.94260.5770],
and the stabilizing control u(t)=UP-1x(t); that is,(5.3)u(t)=[-0.6606-2.1116-0.86230.96790.4079-0.0970]x(t).


## 5.2.  Solutions of SARE

LetQ=I in SARE (2.9). Below, we solve SARE (2.9) via the SDP (4.11) (LMI mincx solver [22]).Case 1.
R is positive definite. Choose R=[2112], and we obtain the maximal solution to (2.9):
(5.4)Pmax⁡=[44.2780-100.5744-48.8271-100.5744360.6942182.4572-48.8271182.457294.6103].Case 2.
R is indefinite. Choose R=[0.20.30.30.2], and we obtain the maximal solution to (2.9):
(5.5)Pmax⁡=[17.7975-40.0973-19.1900-40.0973135.218967.6790-19.190067.679035.5117].Case 3.
R is negative definite. First we can get the solvability margin r*=0.4936 by solving SDP (4.12) (LMI gevp solver [22]). Hence (2.9) has a maximal solution when R>-0.4936I. Choose R=-0.4I, and we obtain the maximal solution to (2.9):
(5.6)Pmax⁡=[29.6283-66.7897-32.2431-66.7897234.0792117.9065-32.2431117.906561.3301].


## 6.  Conclusion

In this paper, we use the SDP approach to the study of discrete-time indefinite stochastic LQ control. It was shown that the mean square stabilization of system (2.1) is equivalent to the strict feasibility of the SDP (3.7). In addition, the relation between the optimal solution of (3.6) and the maximal solution of SARE (2.9) has been established. What we have obtained can be viewed as a discrete-time version of [11]. Of course, there are many open problems to be solved. For example, R+B′PB+D′PD>0 is a basic assumption in this paper. A natural question is whether or not we can weaken it to R+B′PB+D′PD≥0. This problem merits further study.



---

*Source: 638762-2012-02-16.xml*