# Discrete-Time Indefinite Stochastic Linear Quadratic Optimal Control with Second Moment Constraints

Authors: Weihai Zhang; Guiling Li
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

This paper studies the discrete-time stochastic linear quadratic (LQ) problem with a second moment constraint on the terminal state, where the weighting matrices in the cost functional are allowed to be indefinite. By means of the matrix Lagrange theorem, a new class of generalized difference Riccati equations (GDREs) is introduced. It is shown that the well-posedness, and the attainability of the LQ problem and the solvability of the GDREs are equivalent to each other.


---

## Body

## 1. Introduction

LQ control, initiated by Kalman [1] and extended to stochastic systems by Wonham [2], is one of the most important classes of optimal control issues from both theory and application point of view; we refer the reader to [2–8] for representative work in this area. Different from the classical LQ in modern control theory, it was found in [9, 10] that a stochastic LQ problem with indefinite control weighting matrices can still be well-posed, which evoked a series of subsequent researches; see, for example, [11, 12].It is well known that in practical engineering, the system state and control input are always subject to various constraints, so how to solve the constrained stochastic LQ issue is a more attractive topic; we refer the reader to [13–19]. Reference [14] presented a tractable approach for LQ controller design of the system with additive noise. Reference [16] was about the constrained LQ of deterministic systems with state equality constraints. Reference [13] studied the parametrization of the solutions of finite-horizon constrained LQ control. Reference [15] was devoted to a stochastic LQ optimal control and an application to portfolio selection, where the control variable is confined to a cone, and all the coefficients of the state equation are random processes. Reference [19] studied the indefinite stochastic LQ control problem of continuous-time Itô systems with a linear equality constraint M
x
(
T
)
=
ξ on the terminal state and gave a necessary condition for the existence of an optimal controller. Reference [20] generalized the results of [19] to discrete-time stochastic systems.In this paper, different from [19, 20] on the constraint conditions, we would like to deal with stochastic LQ control of discrete-time multiplicative noise systems with a second moment constraint E
[
x
(
T
)
T
x
(
T
)
]
=
c and such constraints are often encountered in H
∞ filtering design; see [21, 22]. By means of Lagrange theorem, we present a necessary condition for the existence of an optimal linear state feedback control with the second moment constraint on the terminal states. It is proved that the solvability of GDRE is necessary and sufficient for the existence of an optimal control under either of the state feedback case or of the open-loop forms. Moreover, we show that the well-posedness and the attainability of the constrained LQ problem, the feasibility of the LMI, and the solvability of the GDRE are equivalent to each other. The novel contribution of this paper is to consider a constrained discrete-time LQ optimal stochastic control, which includes some results of [23] as special cases. A new class of generalized difference Riccati equations (GDREs) is first introduced.The remainder of the paper is organized as follows. Section2 gives some definitions and preliminaries. In Section 3, the optimal state feedback control is studied using the matrix Lagrange theorem. We give a necessary and sufficient condition for the well-posedness of the constrained LQ control in Section 4. Section 5 shows the equivalence among the well-posedness and the attainability of the LQ problem, the feasibility of the LMI, and the solvability of the GDRE. The set of all optimal controls is determined. We conclude the paper in Section 6.Throughout the paper, the following notations are adopted:M
T denotes the transpose of M. M
>
0
(
M
≥
0
): M is a positive definite (positive semidefinite) symmetric matrix. tr
⁡
(
M
): the trace of a square matrix M. R
m
×
n: the space of all m
×
n matrices. S
n: the space of all n
×
n symmetric matrices.


## 2. Problem Setting

Consider the following constrained discrete-time stochastic LQ control problem.Problem 1.
Consider(1)
min
⁡
u
(
t
0
)
,
…
,
u
(
T
-
1
)
⁡
J
(
t
0
,
x
0
;
u
(
t
0
)
,
u
(
t
0
+
1
)
,
…
,
u
(
T
-
1
)
)
s
.
t
.
kkkkkk
x
(
t
+
1
)
=
[
A
(
t
)
x
(
t
)
+
B
(
t
)
u
(
t
)
]
kkkkkkkkkkkkkkkkk
+
[
C
(
t
)
x
(
t
)
+
D
(
t
)
u
(
t
)
]
w
(
t
)
,
s
.
t
.
kkkkkk
x
(
t
0
)
=
x
0
,
s
.
t
.
kkkkkk
E
{
x
(
T
)
T
x
(
T
)
}
=
c
,
s
.
t
.
kkkkkk
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}
,
where the state x
(
t
)
∈
R
n, the control input u
(
t
)
∈
R
m, and the noise w
(
t
)
∈
R
1, t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
},
(2)
J
(
t
0
,
x
0
;
u
(
t
0
)
,
u
(
t
0
+
1
)
,
…
,
u
(
T
-
1
)
)
∶
=
E
{
∑
t
=
t
0
T
-
1
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
]
x
(
T
)
T
Q
(
T
)
x
(
T
)
k
k
k
k
k
+
∑
t
=
t
0
T
-
1
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
]
}
.
The process {
w
(
t
0
)
,
w
(
t
0
+
1
)
,
…
,
w
(
T
-
1
)
} is a sequence of second-order stationary random variables defined on a complete probability space (
Ω
,
F
,
P
). Without loss of generality, we assume that
(3)
E
{
w
(
s
)
}
=
0
,
E
{
w
(
s
)
w
(
t
)
}
=
δ
s
t
,
where δ
s
t is the Kronecker delta, s
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}. c
≥
0 is a constant, A
(
t
)
,
B
(
t
)
,
C
(
t
)
,
D
(
t
)
,
Q
(
t
), and R
(
t
) are matrices having appropriate dimensions determined from context, and Q
(
t
) and R
(
t
) are real symmetric indefinite matrices. x
0 is a given deterministic vector.Definition 2.
Problem1 is called well-posed, if ∀
x
0
∈
R
n,
(4)
V
(
x
0
)
=
inf
⁡
u
(
t
0
)
,
u
(
t
0
+
1
)
,
…
,
u
(
T
-
1
)
J
(
t
0
,
x
0
;
u
(
t
0
)
,
…
,
u
(
T
-
1
)
)
>
-
∞
.Definition 3.
Problem1 is called attainable, if ∀
x
0
∈
R
n, there exists a sequence {
u
*
(
t
0
)
,
u
*
(
t
0
+
1
)
,
…
,
u
*
(
T
-
1
)
}, such that V
(
x
0
)
=
J
(
t
0
,
x
0
;
u
*
(
t
0
)
,
u
*
(
t
0
+
1
)
,
…
,
u
*
(
T
-
1
)
). In this case, {
u
0
*
,
u
1
*
,
…
,
u
N
-
1
*
} is called an optimal control sequence.
Now, let us consider a mathematical programming (MP) problem in a matrix space:(5)
min
⁡
f
(
X
)
i
s
.
t
.
ki
h
(
X
)
=
0
.Definition 4.
LetX
* be a point satisfying
(6)
h
(
X
*
)
=
(
h
1
(
X
*
)
,
…
,
h
p
(
X
*
)
)
T
=
0
,
and then X
* is said to be a constraintregular point if the gradient vectors ∇
h
j
(
X
*
)
,
j
=
1
,
…
,
p, are linearly independent.Lemma 5 (Lagrange theorem [24]).
Assume that the functionsf, h
1
,
…
,
h
p, are twice continuously differentiable. If a regular point X
* is also a relative minimum point for the original MP, then there exists a vector λ
∈
R
p such that
(7)
∇
X
L
(
X
*
,
λ
*
)
=
0
,
where the Lagrangian function L
(
X
,
λ
)
:
=
f
(
X
)
+
λ
T
h
(
X
).


## 3. A Necessary Condition for State Feedback Control

In this section, by the matrix Lagrange theorem, we present a necessary condition for Problem1 based on a new type of GDREs.LetX
(
t
)
=
E
[
x
(
t
)
x
(
t
)
T
]. Through a simple calculation, the following deterministic optimal control Problem 6 is equivalent to the original Problem 1 under the state feedback u
(
t
)
=
K
(
t
)
x
(
t
) for t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}.Problem 6.
Consider(8)
min
⁡
K
(
t
0
)
,
…
,
K
(
T
-
1
)
⁡
J
(
t
0
,
x
0
;
K
(
t
0
)
x
0
,
…
,
K
(
T
-
1
)
x
(
T
-
1
)
)
s
.
t
.
kkkkkk
i
X
(
t
+
1
)
=
(
A
(
t
)
+
B
(
t
)
K
(
t
)
)
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
i
×
X
(
t
)
(
A
(
t
)
+
B
(
t
)
K
(
t
)
)
T
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
i
+
(
C
(
t
)
+
D
(
t
)
K
(
t
)
)
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
i
×
X
(
t
)
(
C
(
t
)
+
D
(
t
)
K
(
t
)
)
T
,
s
.
t
.
kkkkkk
i
t
=
t
0
,
t
0
+
1
,
…
,
T
-
1
,
s
.
t
.
kkkkkk
X
(
t
0
)
=
X
0
=
x
0
x
0
T
,
s
.
t
.
kkkkkk
tr
⁡
[
X
(
T
)
]
=
c
with
(9)
J
(
t
0
,
x
0
;
K
(
t
0
)
x
0
,
…
,
K
(
T
-
1
)
x
(
T
-
1
)
)
=
∑
t
=
t
0
T
-
1
tr
⁡
{
[
Q
(
t
)
+
K
(
t
)
T
R
(
t
)
K
(
t
)
]
X
(
t
)
}
kkkki
i
+
tr
⁡
[
Q
(
T
)
X
(
T
)
]
[
Q
(
t
)
+
K
(
t
)
T
R
(
t
)
K
(
t
)
]
.Remark 7.
If Problem1 has a linear feedback optimal control solution u
*
(
t
)
=
K
*
(
t
)
x
(
t
), t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, then K
*
(
t
), t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, are the optimal solution of Problem 6.Theorem 8.
If Problem1 is attainable by u
(
t
)
=
K
*
(
t
)
x
(
t
), and the regular point (
K
*
(
t
)
,
X
*
(
t
)
) is a locally optimal solution of Problem 6, then there exist symmetric matrices P
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, and λ
∈
R
1 solving the following GDRE:
(10)
P
(
t
)
=
A
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
C
(
t
)
T
P
(
t
+
1
)
C
(
t
)
+
Q
(
t
)
-
H
(
t
)
T
G
(
t
)
†
H
(
t
)
,
H
(
t
)
=
B
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
C
(
t
)
,
G
(
t
)
=
R
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
B
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
D
(
t
)
≥
0
,
G
(
t
)
G
(
t
)
†
H
(
t
)
=
H
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}
,
P
(
T
)
=
Q
(
T
)
+
λ
I
,
where G
† is the Moore-Penrose generalized inverse of G. Moreover,
(11)
K
*
(
t
)
=
-
G
(
t
)
†
H
(
t
)
+
Y
(
t
)
-
G
(
t
)
†
G
(
t
)
Y
(
t
)
with Y
(
t
)
∈
R
m
×
n, t
=
t
0
,
t
0
+
1
,
…
,
T
-
1, being any given real matrices:
(12)
V
(
x
0
)
=
J
(
t
0
,
x
0
;
u
*
(
t
0
)
,
u
*
(
t
0
+
1
)
,
…
,
u
*
(
T
-
1
)
)
=
x
0
T
P
(
t
0
)
x
0
-
c
λ
.To prove Theorem8, we mainly use Lemma 5 to Problem 6 together with the following lemma to obtain GDRE (10) and then apply the technique of completing squares to show (12).Lemma 9 (see [12]).
LetA
,
B
,
C be given matrices with appropriate sizes; then the matrix equation
(13)
A
X
B
=
C
has a solution X if and only if
(14)
A
A
†
C
B
†
B
=
C
.
Moreover, any solution to A
X
B
=
C can be represented by
(15)
X
=
A
†
C
B
†
+
Y
-
A
†
A
Y
B
B
†
,
where Y is any matrix with appropriate size.Proof.
According to Remark7, K
*
(
t
) is also the optimal solution of Problem 6. Problem 6 is a typical MP problem about X
(
t
) and K
(
t
) as follows:(16)
min
⁡
f
[
X
(
t
)
,
K
(
t
)
]
s
.
t
.
kk
k
k
h
t
+
1
[
X
(
t
)
,
K
(
t
)
]
=
0
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}
,
s
.
t
.
kk
k
k
h
[
X
(
T
)
]
=
0
,
where
(17)
f
[
X
(
t
)
,
K
(
t
)
]
=
∑
t
=
t
0
T
-
1
tr
⁡
{
[
Q
(
t
)
+
K
(
t
)
T
R
(
t
)
K
(
t
)
]
X
(
t
)
}
kki
+
tr
⁡
[
Q
(
T
)
X
(
T
)
]
,
h
t
+
1
[
X
(
t
)
,
K
(
t
)
]
=
A
(
t
)
X
(
t
)
A
(
t
)
T
+
A
(
t
)
X
(
t
)
K
(
t
)
T
B
(
t
)
T
+
B
(
t
)
K
(
t
)
X
(
t
)
A
(
t
)
T
+
B
(
t
)
K
(
t
)
X
(
t
)
K
(
t
)
T
B
(
t
)
T
+
C
(
t
)
X
(
t
)
C
(
t
)
T
+
C
(
t
)
X
(
t
)
K
(
t
)
T
D
(
t
)
T
+
D
(
t
)
K
(
t
)
X
(
t
)
C
(
t
)
T
+
D
(
t
)
K
(
t
)
X
(
t
)
K
(
t
)
T
D
(
t
)
T
-
X
(
t
+
1
)
,
h
[
X
(
T
)
]
=
tr
⁡
[
X
(
T
)
]
-
c
.
Let matrices P
(
t
+
1
), t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, be the Lagrangian multipliers of
(18)
h
t
+
1
[
X
(
t
)
,
K
(
t
)
]
=
0
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}
,
and let λ
∈
R
1 be the Lagrangian multiplier of h
[
X
(
T
)
]
=
0; then the Lagrangian function
(19)
L
=
f
[
X
(
t
)
,
K
(
t
)
]
+
∑
t
=
t
0
T
-
1
‍
tr
⁡
{
P
(
t
+
1
)
h
t
+
1
[
X
(
t
)
,
K
(
t
)
]
}
+
λ
h
[
X
(
T
)
]
.
According to the the matrix Lagrange theorem, we obtain
(20)
∂
L
∂
(
K
t
)
=
0
,
t
=
t
0
,
t
0
+
1
,
…
,
T
-
1
,
(21)
∂
L
∂
(
X
t
)
=
0
t
=
t
0
,
t
0
+
1
,
…
,
T
.
Based on the partial rule of gradient matrices, (20) can be transformed into
(22)
[
R
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
B
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
D
(
t
)
]
K
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
C
(
t
)
=
0
.
Let
(23)
G
(
t
)
=
R
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
B
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
D
(
t
)
,
H
(
t
)
=
B
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
C
(
t
)
.
Then we obtain
(24)
G
(
t
)
G
(
t
)
†
H
(
t
)
=
H
(
t
)
,
G
(
t
)
K
(
t
)
+
H
(
t
)
=
0
.
Applying Lemma 9, we have
(25)
K
*
(
t
)
=
-
G
(
t
)
†
H
(
t
)
+
Y
(
t
)
-
G
(
t
)
†
G
(
t
)
Y
(
t
)
,
Y
(
t
)
∈
R
m
×
n
,
t
=
t
0
,
t
0
+
1
,
…
,
T
-
1
.
Equation (21) yields
(26)
P
(
T
)
=
Q
(
T
)
+
λ
I
,
P
(
t
)
=
Q
(
t
)
+
A
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
C
(
t
)
T
P
(
t
+
1
)
C
(
t
)
+
K
(
t
)
T
[
R
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
B
(
t
)
k
k
k
k
k
k
i
+
D
(
t
)
T
P
(
t
+
1
)
D
(
t
)
]
K
(
t
)
+
K
(
t
)
T
[
D
(
t
)
T
P
(
t
+
1
)
C
(
t
)
k
k
k
k
k
k
i
i
+
B
(
t
)
T
P
(
t
+
1
)
A
(
t
)
]
+
[
A
(
t
)
T
P
(
t
+
1
)
B
(
t
)
k
k
+
C
(
t
)
T
P
(
t
+
1
)
D
(
t
)
]
K
(
t
)
.
Substituting K
*
(
t
)
=
-
G
(
t
)
†
H
(
t
)
+
Y
(
t
)
-
G
(
t
)
†
G
(
t
)
Y
(
t
) into (26), it follows
(27)
P
(
t
)
=
A
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
C
(
t
)
T
P
(
t
+
1
)
C
(
t
)
+
Q
(
t
)
-
H
(
t
)
T
G
(
t
)
†
H
(
t
)
.
Without loss of generality, we can assume that P is symmetric. Otherwise, we can take P
¯
=
(
P
T
+
P
)
/
2. The objective functional
(28)
J
(
t
0
,
x
0
;
u
(
t
0
)
,
u
(
t
0
+
1
)
,
…
,
u
(
T
-
1
)
)
=
E
{
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
]
}
+
tr
⁡
[
X
(
T
)
Q
(
T
)
]
=
E
{
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
k
k
k
k
k
k
i
+
x
(
t
+
1
)
T
P
(
t
+
1
)
x
(
t
+
1
)
k
k
k
k
k
k
i
-
x
(
t
)
T
P
(
t
)
x
(
t
)
]
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
}
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
=
E
{
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
k
k
k
k
k
k
i
-
x
(
t
)
T
P
(
t
)
x
(
t
)
]
k
k
k
k
i
i
i
+
[
x
(
t
)
T
A
(
t
)
T
+
u
(
t
)
T
B
(
t
)
T
]
k
k
k
k
i
i
i
×
P
(
t
+
1
)
[
A
(
t
)
x
(
t
)
+
B
(
t
)
u
(
t
)
]
k
k
k
k
i
i
i
+
[
x
(
t
)
T
C
(
t
)
T
+
u
(
t
)
T
D
(
t
)
T
]
k
k
k
k
i
i
i
×
P
(
t
+
1
)
[
C
(
t
)
x
(
t
)
+
D
(
t
)
u
(
t
)
]
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
}
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
.
A completion of square implies
(29)
J
(
t
0
,
x
0
;
u
(
t
0
)
,
u
(
t
0
+
1
)
,
…
,
u
(
T
-
1
)
)
=
E
{
∑
t
=
t
0
T
-
1
‍
[
u
(
t
)
+
G
(
t
)
†
H
(
t
)
x
(
t
)
]
T
k
k
k
k
k
k
k
k
k
×
G
(
t
)
[
u
(
t
)
+
G
(
t
)
†
H
(
t
)
x
(
t
)
]
∑
t
=
t
0
T
-
1
‍
}
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
.
We assert that P
(
t
+
1
) must satisfy
(30)
G
(
t
)
=
R
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
B
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
D
(
t
)
≥
0
.
If it is not so, there is G
(
l
) for l
∈
{
t
0
,
…
,
T
-
1
} with a negative eigenvalue λ
<
0. Denote the unitary eigenvector with respect to λ by v
λ. Let δ
≠
0 be an arbitrary scalar; we construct a control sequence as follows:
(31)
u
~
(
t
)
=
{
-
G
(
t
)
†
H
(
t
)
x
(
t
)
,
t
≠
l
,
δ
|
λ
|
-
1
/
2
v
λ
-
G
(
t
)
†
H
(
t
)
x
(
t
)
,
t
=
l
.
The associated cost functional becomes
(32)
J
(
t
0
,
x
0
;
u
~
(
t
0
)
,
u
~
(
t
0
+
1
)
,
…
,
u
~
(
T
-
1
)
)
=
E
{
∑
t
=
t
0
T
-
1
‍
[
u
~
(
t
)
+
G
(
t
)
†
H
(
t
)
x
(
t
)
]
T
k
k
k
k
k
i
×
G
(
t
)
[
u
~
(
t
)
+
G
(
t
)
†
H
(
t
)
x
(
t
)
]
∑
t
=
t
0
T
-
1
‍
}
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
=
[
δ
|
λ
|
-
1
/
2
v
λ
]
T
G
(
l
)
[
δ
|
λ
|
-
1
/
2
v
λ
]
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
=
-
δ
2
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
.
Let δ
→
∞; then J
(
t
0
,
x
0
;
u
~
(
t
0
)
,
u
~
(
t
0
+
1
)
,
…
,
u
~
(
T
-
1
)
)
→
-
∞, which contradicts the attainability of Problem 1. So (30) holds.
In view of (29) and (30), (11) and (12) are easily derived. The proof is completed.Remark 10.
In Theorem8, in order to apply matrix Lagrange theorem, we assume the optimal solution (
K
*
(
t
)
,
X
*
(
t
)
)
T is a regular point. Generally speaking, for a given LQ control, it is easy to examine the regular condition.
Below, we present a numerical example to illustrate the effectiveness of Theorem8.Example 11.
In Problem1, we set
(33)
E
{
x
(
2
)
T
x
(
2
)
}
=
c
=
73
,
x
0
=
[
0
1
]
,
A
0
=
[
1
0
0
0
]
,
A
1
=
[
1
0
1
0
]
,
B
0
=
[
1
0
]
,
B
1
=
[
1
1
]
,
C
0
=
[
-
1
1
0
0
]
,
C
1
=
[
0
1
1
1
]
,
D
0
=
[
1
-
1
]
,
D
1
=
[
1
-
1
]
.
The state and control weighting matrices are as
(34)
Q
0
=
[
-
1
0
0
-
1
]
,
Q
1
=
[
-
1
0
0
0
]
,
Q
2
=
[
0
0
0
0
]
,
R
0
=
-
1
,
R
1
=
-
7
.
By the relationship between Problems 1 and 6, we know
(35)
X
0
=
[
0
0
0
1
]
,
tr
⁡
[
X
(
2
)
]
=
73
.
Applying Theorem 8, we obtain
(36)
X
(
0
)
=
[
0
0
0
1
]
,
X
(
1
)
=
[
5
-
2
-
2
1
]
,
X
(
2
)
=
[
34
-
26
-
26
39
]
,
P
(
2
)
=
λ
I
=
[
2
0
0
2
]
,
λ
=
2
.
Stage 2.Consider
(37)
G
(
1
)
=
1
>
0
,
H
(
1
)
=
(
2,0
)
,
P
(
1
)
=
[
1
2
2
4
]
,
K
(
1
)
=
(
-
2,0
)
.
Stage 1.Consider
(38)
G
(
0
)
=
1
>
0
,
H
(
0
)
=
(
2
,
-
1
)
,
K
(
0
)
=
(
-
2,1
)
,
P
(
0
)
=
[
5
-
3
-
3
0
]
.
The optimal cost value of Problem 1 is
(39)
V
(
x
0
)
=
E
[
x
0
T
P
(
t
0
)
x
0
-
c
λ
]
=
-
146
.
We are able to test the regular condition of (
K
*
(
t
)
,
X
*
(
t
)
)
T as follows. In Problem 6,
(40)
h
(
t
+
1
)
(
X
(
t
)
,
K
(
t
)
)
=
A
(
t
)
X
(
t
)
A
(
t
)
T
+
A
(
t
)
X
(
t
)
K
(
t
)
T
B
(
t
)
T
+
B
(
t
)
K
(
t
)
X
(
t
)
A
(
t
)
T
+
B
(
t
)
K
(
t
)
X
(
t
)
K
(
t
)
T
B
(
t
)
T
+
C
(
t
)
X
(
t
)
C
(
t
)
T
+
C
(
t
)
X
(
t
)
K
(
t
)
T
D
(
t
)
T
+
D
(
t
)
K
(
t
)
X
(
t
)
C
(
t
)
T
+
D
(
t
)
K
(
t
)
X
(
t
)
K
(
t
)
T
D
(
t
)
T
-
X
(
t
+
1
)
,
t
=
0,1
,
which is linear about X
(
t
) and quadratic about K
(
t
), while h
(
X
(
T
)
)
=
tr
⁡
[
X
(
T
)
]
-
c is linear about X
(
T
). By simple calculations, ∇
h
1
(
K
*
(
0
)
,
X
*
(
0
)
), ∇
h
2
(
K
*
(
1
)
,
X
*
(
1
)
), and ∇
h
(
X
*
(
T
)
) are all nonzero vectors and hence are linearly independent.


## 4. Well-Posedness

In this section, we first establish the link between the well-posedness of Problem1 and the feasibility of some LMIs and then prove that the solvability of GDRE (10) is not only necessary but also sufficient to the well-posedness of Problem 1. Moreover, the well-posedness and the attainability of Problem 1, the feasibility of some LMIs, and the solvability of GDRE (10) are equivalent to each other.Theorem 12.
Problem1 is well-posed if there exist symmetric matrices P
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, and λ
∈
R
1 solving the following LMIs:
(41)
M
¯
t
∶
=
[
A
(
t
)
T
P
(
t
+
1
)
A
(
t
)
-
P
(
t
)
+
C
(
t
)
T
P
(
t
+
1
)
C
(
t
)
+
Q
(
t
)
H
(
t
)
T
H
(
t
)
G
(
t
)
]
≥
0
,
(42)
P
(
T
)
≤
Q
(
T
)
+
λ
I
,
where
(43)
H
(
t
)
=
B
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
C
(
t
)
,
G
(
t
)
=
R
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
B
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
D
(
t
)
.Proof.
Note that(44)
J
(
t
0
,
x
0
;
u
(
t
0
)
,
u
(
t
0
+
1
)
,
…
,
u
(
T
-
1
)
)
=
E
{
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
]
k
k
k
k
k
k
i
+
x
(
T
)
T
Q
(
T
)
x
(
T
)
∑
t
=
t
0
T
-
1
‍
}
=
E
{
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
k
k
k
k
k
k
k
k
k
k
i
+
x
(
t
+
1
)
T
P
(
t
+
1
)
x
(
t
+
1
)
k
k
k
k
k
k
k
k
k
k
-
x
(
t
)
T
P
(
t
)
x
(
t
)
]
k
k
k
k
k
+
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
∑
t
=
t
0
T
-
1
‍
}
=
E
{
∑
t
=
t
0
T
-
1
‍
[
x
(
t
)
u
(
t
)
]
T
M
¯
t
[
x
(
t
)
u
(
t
)
]
k
k
k
k
k
+
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
∑
t
=
t
0
T
-
1
‍
}
.
By (41), it is easy to deduce that the cost functional is bounded from below by
(45)
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
≥
x
0
T
P
(
t
0
)
x
0
-
c
λ
.
Hence, Problem 1 is well-posed.Remark 13.
Theorem12 tells us that any symmetric matrices P
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, and λ
∈
R
1 satisfying LMIs (41)-(42) provide a lower bound
(46)
x
0
T
P
(
t
0
)
x
0
-
c
λ
for the cost function. In what follows, we will show that this lower bound is an exact optimal cost value if P
(
t
) and λ
∈
R
1 solve GDRE (10).
We have shown that if the LMIs (41)-(42) are satisfied, then the constrained LQ Problem 1 is well-posed. Below, we further show some other equivalent conditions.Lemma 14 (extended Schur’s lemma [25]).
Let the matricesM
=
M
T, H, G
=
G
T be given with appropriate sizes. Then, the following three conditions are equivalent:
(47)
(
1
)
M
-
H
G
†
H
T
≥
0
,
G
≥
0
,
H
(
I
-
G
G
†
)
=
0
.
(
2
)
[
M
H
H
T
G
]
≥
0
.
(
3
)
[
G
H
T
H
M
]
≥
0
.Theorem 15.
Problem1 is well-posed if and only if there exist symmetric matrices P
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, and λ
∈
R
1 satisfying GDRE (10). Furthermore, the optimal cost is
(48)
V
(
x
0
)
=
x
0
T
P
(
t
0
)
x
0
-
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
}
=
x
0
T
P
(
t
0
)
x
0
-
c
λ
.
A key to prove Theorem15 is the necessity part, where the stochastic optimization principle is used.Proof.
Necessity. For t
0
≤
l
≤
T
-
1, define
(49)
V
l
[
x
(
l
)
]
=
inf
⁡
u
(
l
)
,
…
,
u
(
T
-
1
)
E
{
∑
t
=
l
T
-
1
‍
[
x
(
t
)
T
Q
(
t
)
x
(
t
)
+
u
(
t
)
T
R
(
t
)
u
(
t
)
]
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
k
+
x
(
T
)
T
Q
(
T
)
x
(
T
)
∑
t
=
l
T
-
1
‍
}
.
By the stochastic optimization principle, when V
l
1
[
x
(
l
1
)
] is finite, then so is V
l
2
[
x
(
l
2
)
] for any l
1
≤
l
2. Since Problem 1 is assumed to be well-posed at t
0, V
l
[
x
(
l
)
] is finite at any stage 0
≤
l
≤
T
-
1. Now let us start with l
=
T
-
1, and let P
(
T
)
=
Q
(
T
)
-
λ
I, and we have
(50)
V
T
-
1
[
x
(
T
-
1
)
]
-
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
}
=
inf
⁡
u
(
T
-
1
)
{
E
[
x
(
T
-
1
)
T
Q
(
T
-
1
)
x
(
T
-
1
)
k
k
k
i
k
i
+
u
(
T
-
1
)
T
R
(
T
-
1
)
u
(
T
-
1
)
]
}
+
E
[
x
(
T
)
T
P
(
T
)
x
(
T
)
]
=
inf
⁡
u
(
T
-
1
)
E
{
x
(
T
-
1
)
T
k
k
k
k
k
k
i
k
i
×
[
Q
(
T
-
1
)
+
A
(
T
-
1
)
T
P
(
T
)
A
(
T
-
1
)
k
k
k
k
k
k
i
k
k
k
k
i
+
C
(
T
-
1
)
T
P
(
T
)
C
(
T
-
1
)
]
x
(
T
-
1
)
k
k
k
k
k
k
i
k
i
×
2
x
(
T
-
1
)
T
k
k
k
k
k
k
i
k
i
×
[
B
(
T
-
1
)
T
P
(
T
)
A
(
T
-
1
)
k
k
k
k
k
k
i
k
k
k
i
i
+
D
(
T
-
1
)
T
P
(
T
)
C
(
T
-
1
)
]
u
(
T
-
1
)
k
k
k
k
k
k
i
k
i
+
u
(
T
-
1
)
T
k
k
k
k
k
k
i
k
i
×
[
R
(
T
-
1
)
+
B
(
T
-
1
)
T
P
(
T
)
B
(
T
-
1
)
k
k
k
k
k
k
i
k
k
k
i
i
+
D
(
T
-
1
)
T
P
(
T
)
D
(
T
-
1
)
]
u
(
T
-
1
)
}
.
Since V
T
-
1
[
x
(
T
-
1
)
] is finite, using Lemma 4.3 of [23], there exists a symmetric matrix P
(
T
-
1
) such that
(51)
V
T
-
1
[
x
(
T
-
1
)
]
-
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
}
=
E
[
x
(
T
-
1
)
T
P
(
T
-
1
)
x
(
T
-
1
)
]
,
P
(
T
-
1
)
=
A
(
T
-
1
)
T
P
(
T
)
A
(
T
-
1
)
+
C
(
T
-
1
)
T
P
(
T
)
C
(
T
-
1
)
+
Q
(
T
-
1
)
-
H
(
T
-
1
)
T
×
G
(
T
-
1
)
†
H
(
T
-
1
)
,
H
(
T
-
1
)
=
B
(
T
-
1
)
T
P
(
T
)
A
(
T
-
1
)
+
D
(
T
-
1
)
T
P
(
T
)
C
(
T
-
1
)
,
G
(
T
-
1
)
=
R
(
T
-
1
)
+
B
(
T
-
1
)
T
P
(
T
)
B
(
T
-
1
)
+
D
(
T
-
1
)
T
P
(
T
)
D
(
T
-
1
)
≥
0
.
The obtained solution sequence of symmetric matrices P
(
t
), t
=
l
,
l
+
1
,
…
,
T
-
1, and λ
∈
R
1 to GDRE (10) satisfy
(52)
V
l
[
x
(
l
)
]
-
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
}
=
E
[
x
(
l
)
T
P
(
l
)
x
(
l
)
]
.
Then by the stochastic optimality principle, the following holds:
(53)
V
l
-
1
[
x
(
l
-
1
)
]
=
inf
⁡
u
(
l
-
1
)
E
{
x
(
l
-
1
)
T
Q
(
l
-
1
)
x
(
l
-
1
)
k
k
k
k
k
k
k
k
+
u
(
l
-
1
)
T
R
(
l
-
1
)
u
(
l
-
1
)
k
k
k
k
k
k
k
i
+
V
l
[
x
(
l
)
]
}
=
inf
⁡
u
(
l
-
1
)
E
[
x
(
l
-
1
)
T
Q
(
l
-
1
)
x
(
l
-
1
)
k
k
k
k
k
k
k
k
+
u
(
l
-
1
)
T
R
(
l
-
1
)
u
(
l
-
1
)
k
k
k
k
k
k
k
k
+
x
(
l
)
T
P
(
l
)
x
(
l
)
]
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
}
=
inf
⁡
u
(
l
-
1
)
E
{
x
(
l
-
1
)
T
k
k
k
k
k
k
k
k
×
[
Q
(
l
-
1
)
+
A
(
l
-
1
)
T
P
(
l
)
A
(
l
-
1
)
k
k
k
k
k
k
k
k
k
k
i
+
C
(
l
-
1
)
T
P
(
l
)
C
(
l
-
1
)
]
x
(
l
-
1
)
k
k
k
k
k
k
k
k
+
2
x
(
l
-
1
)
T
k
k
k
k
k
k
k
k
×
[
B
(
l
-
1
)
T
P
(
l
)
A
(
l
-
1
)
k
k
k
k
k
k
k
k
k
k
i
+
D
(
l
-
1
)
T
P
(
l
)
C
(
l
-
1
)
]
u
(
l
-
1
)
k
k
k
k
k
k
k
k
+
u
(
l
-
1
)
T
k
k
k
k
k
k
k
k
×
[
R
(
l
-
1
)
+
B
(
l
-
1
)
T
P
(
l
)
B
(
l
-
1
)
k
k
k
k
k
k
k
k
k
k
+
D
(
l
-
1
)
T
P
(
l
)
D
(
l
-
1
)
]
u
(
l
-
1
)
}
.
Lemma 4.3 of [23] provides necessary and sufficient conditions for the finiteness of V
l
-
1
[
x
(
l
-
1
)
]:
(54)
P
(
l
-
1
)
=
A
(
l
-
1
)
T
P
(
l
)
A
(
l
-
1
)
+
C
(
l
-
1
)
T
P
(
l
)
C
(
l
-
1
)
+
Q
(
l
-
1
)
-
H
(
l
-
1
)
T
G
(
l
-
1
)
†
H
(
l
-
1
)
,
H
(
l
-
1
)
=
B
(
l
-
1
)
T
P
(
l
)
A
(
l
-
1
)
+
D
(
l
-
1
)
T
P
(
l
)
C
(
l
-
1
)
,
G
(
l
-
1
)
=
R
(
l
-
1
)
+
B
(
l
-
1
)
T
P
(
l
)
B
(
l
-
1
)
+
D
(
l
-
1
)
T
P
(
l
)
D
(
l
-
1
)
≥
0
,
G
(
l
-
1
)
G
(
l
-
1
)
†
H
(
l
-
1
)
-
H
(
l
-
1
)
=
0
.
Moreover,
(55)
V
l
-
1
[
x
(
l
-
1
)
]
-
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
}
=
x
(
l
)
T
P
(
l
)
x
(
l
)
.
The above proves the necessity part via mathematical induction.
Sufficiency. From the proof of Theorem 8, if GDRE (10) admits a solution P
(
t
) and λ, Problem 1 is not only well-posed, but also attainable. The proof of this theorem is complete.


## 5. Other Equivalent Conditions

In this section, we present some other equivalent conditions for Problem1.Theorem 16.
For the constrained LQ Problem1, the following are equivalent:(i)
Problem1 is well-posed.
(ii)
Problem1 is attainable.
(iii)
The LMIs (41)-(42) are feasible.
(iv)
The GDRE (10) is solvable.
Furthermore, when any one of the above conditions is satisfied, Problem1 is attainable by
(56)
u
(
t
)
=
[
R
(
t
)
+
B
(
t
)
T
P
(
t
+
1
)
B
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
D
(
t
)
]
†
·
[
B
(
t
)
T
P
(
t
+
1
)
A
(
t
)
+
D
(
t
)
T
P
(
t
+
1
)
C
(
t
)
]
x
(
t
)
,
where P
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, are solutions to the GDRE (10).Proof.
Applying Theorems12–15, (
i
)
⇔
(
iii
)
⇔
(
iv
). (
ii
)
⇒
(
iv
) is shown by Theorem 8. The rest is to prove (
iv
)
⇒
(
ii
) and (56). Let P
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, solve the GDRE (10). In view of
(57)
∑
t
=
t
0
T
-
1
‍
[
x
(
t
+
1
)
T
P
(
t
+
1
)
x
(
t
+
1
)
-
x
(
t
)
T
P
(
t
)
x
(
t
)
]
=
E
[
x
(
T
)
T
P
(
T
)
x
(
T
)
-
x
0
T
P
(
t
0
)
x
0
]
,
a completion of squares yields
(58)
J
(
t
0
,
x
0
;
u
(
t
0
)
,
u
(
t
0
+
1
)
,
…
,
u
(
T
-
1
)
)
=
E
{
∑
t
=
t
0
T
-
1
‍
[
u
(
t
)
+
G
(
t
)
†
H
(
t
)
x
(
t
)
]
T
k
k
k
k
k
k
k
k
k
×
G
(
t
)
[
u
(
t
)
+
G
(
t
)
-
1
H
(
t
)
x
(
t
)
]
∑
t
=
t
0
T
-
1
‍
}
+
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
,
which shows
(59)
V
(
x
0
)
=
E
{
x
(
T
)
T
[
Q
(
T
)
-
P
(
T
)
]
x
(
T
)
+
x
0
T
P
(
t
0
)
x
0
}
,
u
*
(
t
)
=
-
G
(
t
)
†
H
(
t
)
x
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}
.
Finally, we present a general expression for the optimal control set based on the solution to GDRE (10).Theorem 17.
Assume that the GDRE (10) admits a solution. Then the set of all optimal controls is determined by
(60)
u
(
t
)
[
Y
(
t
)
,
Z
(
t
)
]
=
-
[
G
(
t
)
†
H
(
t
)
+
Y
(
t
)
-
G
(
t
)
†
G
(
t
)
Y
(
t
)
]
x
(
t
)
+
Z
(
t
)
-
G
(
t
)
†
G
(
t
)
Z
(
t
)
,
where Y
(
t
)
∈
R
m
×
n and Z
(
t
)
∈
R
m are arbitrary random variables defined on the probability space (
Ω
,
F
,
P
). Moreover, the optimal cost value is uniquely given by
(61)
V
(
x
0
)
=
x
0
T
P
(
t
0
)
x
0
-
c
λ
,
where P
(
t
)
,
t
∈
{
t
0
,
t
0
+
1
,
…
,
T
-
1
}, and λ are the solution to the GDRE (10).Proof.
This theorem can be proved by repeating the same procedure as in Theorem 5.1 of [23].


## 6. Conclusion

In this paper, we have investigated a class of indefinite stochastic LQ control problems with second moment constraints on the terminal state. By the matrix Lagrange theorem, we have established a new GDRE (10) associated with the constrained optimization Problem 1. In addition, by introducing LMIs (41)-(42), we show that the well-posedness and the attainability of Problem 1, the feasibility of the LMIs (41)-(42), and the solvability of GDRE (10) are equivalent to each other.



---

*Source: 278142-2014-05-18.xml*