# Study on Indefinite Stochastic Linear Quadratic Optimal Control with Inequality Constraint

Authors: Guiling Li; Weihai Zhang
Journal: Journal of Applied Mathematics
 (2013)

---

## Abstract

This paper studies the indefinite stochastic linear quadratic (LQ) optimal control problem with an inequality constraint
for the terminal state. Firstly, we prove a generalized Karush-Kuhn-Tucker (KKT) theorem under hybrid constraints. Secondly, a new type of generalized Riccati equations is obtained, based on which a necessary condition (it is also a sufficient condition under stronger assumptions) for the existence of an optimal linear state feedback control is given by means of KKT theorem. Finally, we
design a dynamic programming algorithm to solve the constrained indefinite stochastic LQ issue.


---

## Body

## 1. Introduction

The study on LQ control problems can be traced back to the pioneering work of Kalman [1] and Wonham [2] several decades ago. The LQ control theory is elegantly established and developed, and the main work can be seen in [3–11]. In particular, it is found [6] that a stochastic LQ problem with indefinite control weighting matrices may still be well-posed. This discovery evokes a series of subsequent researches, and many important achievements are obtained [5, 12–16]. Up to now, most work deals with the indefinite stochastic LQ problems without constraints. However, as a practical optimization problem, the indefinite stochastic LQ problem unavoidably has various constraints on the state or control; in particulary the inequality constraints often appear.For the constrained indefinite stochastic LQ problems, [17] studied the equally constrained stochastic LQ optimization for Itô systems. In this paper, we will study the stochastic LQ problem with inequality constraint.Firstly, we present and prove the generalized KKT theorem under hybrid constraints. Secondly, a necessary condition for the existence of an optimal linear state feedback control is given by means of the generalized KKT theorem. Thirdly, if we strengthen the condition, we can obtain a necessary and sufficient condition for the existence of the optimal linear feedback control to indefinite stochastic LQ optimal control problem with inequality constraint. Finally, we give a dynamic programming algorithm to solve the stochastic LQ problem with the inequality constraint. We provide an example to demonstrate the effectiveness of our main theoretical results.The outline of this paper is organized as follows. In Section2, we present a generalized KKT theorem under hybrid constraints. Section 3 proposes a KKT condition for the existence of an optimal linear state feedback control. In Section 4, we provide a necessary and sufficient condition and a dynamic programming algorithm for the stochastic LQ problem with inequality constraint. Section 5 concludes the paper.For convenience, throughout the paper, we adopt the following notations:AT denotes the transpose of a matrix A. A>0(A≥0): A is a positive definite (positive semidefinite) symmetric matrix. tr(A): trace of a square matrix. Rm×n: the space of all m×n real matrices. Sn: a  n×n symmetric matrix space.


## 2. Preliminaries

Consider the following indefinite stochastic LQ control.Problem 1.
Consider(1a)minJ(x0,u)=E∫0T[xT(t)Q(t)x(t)zzzzzzzzzzzzzzzzzz+uT(t)R(t)u(t)]dt,(1b)s.t.dx(t)=[A(t)x(t)+B(t)u(t)]dtzzzzzzzzzzz+[C(t)x(t)+D(t)u(t)]dw(t),(1c)…..zzcczx(0)=x0,(1d)zzzE{∥x(T)∥2}=E[xT(T)x(T)]≤c,where x(t)=[x1(t),x2(t),…,xn(t)]T is an n-dimensional state variable, u(t)∈Rm is a control input, w(·) is a one-dimensional standard Brownian motion defined on a filtered probability space (Ω,F,Ft,P). We denote the information flow ℱt=σ[w(s):0≤s≤t≤T]. u(·) belongs to ℒℱ2(Rm), where ℒℱ2(Rm) is a space of all Rm-valued, ℱt-adapted measurable processes satisfying E∫0T‍∥u(t)∥2dt<+∞. For each admissible control, the corresponding trajectory satisfies the constraint (1c). c in constraint (1c) is a given nonnegative constant. A(t), B(t), C(t), D(t), Q(t), and R(t) are time-varying matrices of suitable dimensions. Q(t) and R(t) in objective functional are symmetric matrices. To study the issue, we first put forward the following Assumption H1.
Assumption
H
1.  A(t),C(t)∈ℒ∞(0,T,Rn×n),Q(t)∈ℒ∞(0,T,Sn), B(t),D(t)∈ℒ∞(0,T,Rn×m), and R(t)∈ℒ∞(0,T,Sm), where L∞(0,T,X)∶={f(t):X-valued essential bounded measurable function and ess supt∈[0,T]∥f(t)∥<+∞}.
In this paper, the weighting matrices in the objective functional are not required to be definite. Therefore Problem1 is an indefinite stochastic LQ optimal control problem. For later use, we recall KKT theorem for this type of mathematical programming (MP) problems:
(2)minf(x)s.t.hhhg(x)≤0,hhhihhh(x)=0,
where g(x)=(g1(x),…,gp(x)), and h(x)=(h1(x),…,hq(x)).
The KKT conditions [18–20], which are also known as the Kuhn-Tucker (KT) conditions, are the first-order necessary conditions for a solution in nonlinear programming to be optimal, provided that some regularity conditions are satisfied. The Lagrange multiplier method, which allows only equality constraints, can be viewed as a special case of KKT conditions.
Regularity Condition (or Constraint Qualification). In MP above, let I*={i∣gi(x*)=0}(gi(x*)  i∈I* be active constraints at x*). The gradient vectors ∇gi(x*),i∈I*, and ∇hj(x*),j=1,…,q, are linearly independent, is known as a linear independent constraint qualification (LICQ).
Regular Point. In MP above, x* is said to be a regular point of the constraints if the gradient vectors ∇gi(x*),∇hj(x*),i∈I*,j=1,…,q, are linearly independent.
KKT Theorem. In MP above, we assume that the functions f,g=(g1,…,gp)T,h=(h1,…,hq)T are twice continuously differentiable and we assume that all the constraints satisfy the regularity condition LICQ. Let x* be a point satisfying all the constraints and let x* be a regular point of the above constraints. Now suppose that this regular point x* is also a relative minimum point for the original MP. Then it is shown that there exist a vector λ≥0∈R and a vector μ∈R, such that
(3)∇xL(x*,λ*,μ*)=0,λi*gi(x*)=0,i=1,…,p,
where L(x,λ,μ)=f(x)+λTg(x)+μTh(x) is the Lagrangian function and λi*gi(x*)=0,i=1,…,p, are complementary clackness condition.
It is particularly important to check the regularity condition before we apply the conclusion of KKT theorem. If it is not so, the conclusion of KKT theorem would not be valid, just as the following example shows.Example 2.
Consider(4)minhhf(x1,x2)=-2x1-3x2,s.t.hhhig(x1,x2)=x12+x22=0.
Obviously, the minimum point is x*(0,0)T. According to KKT theorem, we obtain
(5)2-λx1*=0,3-λx2*=0,x1*2+x2*2=0.
The conclusion of KKT theorem does not hold at point x*(0,0)T, because ∇g(0,0)=(0,0)T is not linearly independent. It does not satisfy the LICQ regularity condition.
In order for a minimum pointx* to satisfy the above KKT conditions, it should satisfy some regularity conditions. Except for LICQ regularity condition, the most used ones are listed below.
Constant Rank Constraint Qualification. For each subset of the gradients of the active inequality constraints and the gradients of the equality constraints the rank at a vicinity of x* is constant.
Mangasarian-Fromovitz Constraint Qualification. The gradients of the active inequality constraints and the gradients of the equality constraints are linear independent at x*.
Constant Positive Linear Dependence Constraint Qualification. For each subset of the gradients of the active inequality constraints and the gradients of the equality constraints, if it is positive-linear dependent at x*, then it is positive-linear dependent at a vicinity of x*.
The Slater condition for a convex MP is also a common regularity condition.Remark 3.
In this paper, for convenience, when we use the KKT theorem, we always assume that the local optimalx* meets the LICQ regularity condition. The same goes for other regularity conditions.Definition 4 (see [21]).
LetX be a vector space, Y a normed space, and T a transformation from X to Y. If the limit
(6)δT(x;h)=limα→0[T(x+αh)-T(x)]α
exists, it is called the Gateaux differential of T at x with increment h. If the limit exists for each h∈X, the transformation T is said to be Gateaux differentiable at x.Definition 5 (see [21]).
LetX be a vector space and Z a Banach space with a positive cone P having nonempty interior. Let G be a mapping from X to Z which has a Gateaux differential that is linear in its increment. A point x0∈X is said to be a regular point of the inequality G(x)≤0, if G(x0)≤0 and there is an h∈X such that G(x0)+δG(x0;h)<0.Definition 6 (see [21]).
LetX be a vector space and Z a Banach space. Let H(x)=[h1(x),…,h1(x)]T be a mapping from X to Z which has a Gateaux differential that is linear in its increment. A point x0∈X is said to be a regular point of the equality H(x)=0, if δh1(x0;h),…,δhn(x0;h) are linearly independent.
On the basis of the definitions above, let us discuss the KKT theorem in Banach space, where the objective function and the constraint functions in MP are functionals.
Let us consider(MPI)minf(x)s.t.lG(x)≤0hhhhllH(x)=0.
As a special case,
(MPII)minf(x)s.t.G(x)≤0
has the local necessary condition as follows.Lemma 7 (see [21] (generalized KKT theorem)).
LetX be a vector space and Z a Banach space having positive cone P. Assume that P contains an interior point. Let f be a Gateaux differentiable functional on X and G a Gateaux differentiable mapping from X to Z. Assume that the Gateaux differentials are linear in their increments. Suppose that x0 minimizes f subject to G(x)≤0 and that x0 is a regular point of the inequality G(x)≤0. Then there is a z0*≥0 in Z, such that the Lagrangian function f(x)+z0*G(x) is stationary at x0. Furthermore z0*G(x0)=0.
The following theorem is the local necessary condition of(MPI).Theorem 8.
LetX be a vector space and Z a Banach space having positive cone P. Assume that P contains an interior point. Let f be a Gateaux differentiable functional on X. Let G and H be Gateaux differentiable mappings from X to Z. Assume that the Gateaux differentials are linear in their increments. Suppose that x0 minimizes f subject to G(x)≤0, H(x)=0 and that x0 is a regular point of G(x)≤0, H(x)=0. Then there is a λ*≥0 in Z, μ*∈Z, such that the Lagrangian function f(x)+λ*G(x)+μ*H(x) is stationary at x0. Namely, δf(x0;h)+λ*δG(x0;h)+μ*δH(x0;h)=0. Furthermore, z0*G(x0)=0.Proof.
H
(
x
)
=
0 is equivalent to H(x)≤0 and -H(x)≤0. If x0 is a regular point of H(x)=0, then δh1(x0;h),…,δhn(x0;h) are linearly independent. So δh1(x0;h),…,δhn(x0;h) are all nonzero, because the Gateaux differentials are linear in their increments. Using Definition 5, it is easy to verify that x0 is a regular point of both -H(x)≤0 and H(x)≤0. According to Lemma 7, we know that the multiplier μ* of equality H(x)=0 has no nonnegative requirement.Definition 9 (see [22]).
Suppose thatf(X) is a scalar-valued function of the elements xij of X. Then the gradient matrix of f(X) is defined as
(7)∇[f(X)]=∂f(X)∂(X)
with
(8)[∂f(X)∂(X)]ij=∂f(X)∂xij.
Based on Definition 9, we can easily extend KKT theorem from Banach space to matrix space. Because X(t) can be treated as a vector [X1(t),…,Xn(t)]T, one can work out ∇[Xi(t)] and the KKT theorem holds.
When we apply the matrix KKT theorem, we need to give the partial list of gradient matrices [22] that we will use in this paper. In the following formulas, X is an n×m matrix. The formulas are not valid if the elements xij of X are not independent. A,B are assumed to have appropriate dimensions determined from context.
Consider the following:(9)…….∂∂Xtr(X)=I,….∂∂Xtr(XXT)=2X,….∂∂Xtr(AXT)=A,…..∂∂Xtr(AX)=AT,..∂∂Xtr(AXTB)=BA,0.∂∂Xtr(AXB)=ATBT,∂∂Xtr(AXBXT)=ATXBT+AXB,.∂∂Xtr(AXBX)=ATXTBT+BTXTAT.


## 3. KKT Conditions and a New Type of GDREs

Definition 10.
Problem1 is well-posed, if for any x0∈Rn, -∞<V(x0)=infu(·)∈Uad{J(x0,u(·))}. u* is called an optimal control, if V(x0)=J(x0,u*), and x* denotes the corresponding optimal trajectory.
Let the control law beu=Kx(K∈Cm×n[0,T]), and X(t)=E[x(t)xT(t)]. By substituting u=Kx into (1a) of Problem 1, we obtain the new objective functional:
(10)J(X,K)=tr∫0T[QX+KTRKX]dt,
where Cn×n[0,T] is the space of n-order square matrix whose elements are continuous functions. By substituting u=Kx into (1b) of Problem 1, we obtain a closed-loop system:
(11)dx=(A+BK)xdt+(C+DK)xdw,x(0)=x0.
By applying Itô’s formula to X(t), we obtain
(12)X˙=(A+BK)X+X(A+BK)T+(C+DK)X(C+DK)T,X(0)=X0=x0x0T.
Define the transformation H(X,K) from Cn×n[0,T]×Cm×n[0,T] to Cn×n[0,T]:
(13)H(X,K)≔X(t)-X(0)-∫0t‍[(A+BK)X+X(A+BK)T…….+(C+DK)X(C+DK)T]dt.
By substituting u=Kx into (1c) of Problem 1, we obtain
(14)trX(T)-c≤0.
Define the transformation G(X,K) from Cn×n[0,T] to R1:
(15)G(X(T))∶=trX(T)-c.
So the original stochastic Problem 1 can be transformed into the deterministic Problem 11 as follows.Problem 11.
Consider the following:(16a)
min
J
(
X
,
K
)
=
tr
∫
0
T
[
Q
X
+
K
T
R
K
X
]
d
t
,
(16b)
s
.
t
.
.
.
H
(
X
,
K
)
=
0
,
∀
t
∈
[
0
,
T
]
,
(16c)
z
z
z
z
z
G
(
X
(
T
)
)
≤
0
.Lemma 12.
J
(
X
,
K
), H(X,K), and G(X(T)) have continuous Gateaux derivative as follows:
(17)δJX(X,K;ΔX)=tr∫0T‍(Q+KTRK)ΔXdt,δJK(X,K;ΔX)=tr∫0T‍(ΔKTRKX+KTRΔKX)dt,δHX(X,K;ΔX)(t)=ΔX(t)-tr∫0t[(A+BK)ΔX+ΔX(A+BK)Tcssssssccccccccccccc+(C+DK)ΔX(C+DK)T]dt,δHK(X,K;ΔK)(t)=-tr∫0t[BΔKX+XΔKTBT+(C+DK)X(DΔK)T000000000000+(DΔK)X(C+DK)T]dt,δGX(X;ΔX(T))=trΔX(T).Proof.
We prove only the most complicated one. The rest can be verified in the same way. From Definition4,
(18)δHX(X,K;ΔX)=limα→0[H(X+αΔX,K)-H(X,K)]α,H(X,K)∶=X(T)-X(0)-∫0t[(A+BK)X+X(A+BK)T000000000000000000000+(C+DK)X(C+DK)T]dt.
Replace the X in H(X,K) with (X+αΔX) and then let α→0 showing the conclusion.Lemma 13 (see [21]).
Ifα(t) and β(t) are continuous in [t1,t2] and ∫t1t2‍[α(t)h(t)+β(t)h(t)]dt=0 for every continuously differentiable h(t) with h(t1)=h(t1)=0, then β is differentiable and α(t)≡β(t) in [t1,t2].Lemma 14 (see [21]).
Ifα(t) is continuous in [t1,t2] and ∫t1t2‍[α(t)h(t)]dt=0 for every continuously differentiable h(t) with h(t1)=h(t1)=0, then α(t)≡0 on [t1,t2].Theorem 15.
Assume thatK* is the optimal solution of Problem 1, and then there exist a symmetric matrix P∈NBVn×n[0,T] and a nonnegative λ∈R1, such that (19a)-P˙=(Q+K*TRK*)+P(A+BK*)+(A+BK*)TPhh+(C+DK*)P(C+DK*)T,(19b)P(T)=λIn,(19c)K*TR+PB+(C+DK*)TPD=0,(19d)λ[trX(T)-c]=0,where NBVn×n[0,T] is a matrix space whose elements are bounded functions in [0,T] with 0 value at point 0 and right continuous at (0,T).Proof.
K
* is also the optimal solution of Problem 11. Problem 11 is the type of (MPI). Assume that the optimal solution to Problem 11 is (X*,K*). Using Theorem 8 and Lemma 12, there exist a symmetric matrix P∈NBVn×n[0,T] and a nonnegative λ∈R1, such that
(20)tr∫0T(Q+K*TRK*)ΔXdtll+tr∫0T{∫0t[(A+BK*)ΔX+ΔX(A+BK*)TΔX(t)FFFFFFFffF-∫0t[(A+BK*)ΔX+ΔX(A+BK*)Tfffffffffff+(C+DK*)ΔX(C+DK*)T]dt∫0t[(A+BK*)ΔX+ΔX(A+BK*)T}dP+λtr[ΔX(T)]=0,(21)tr∫0T(ΔKTRK*X*+K*TRΔKK*)dt-tr∫0T{∫0t[BΔKX*+X*ΔKTBThhhhhhhhhlhh+(C+DK*)X*(DΔK)T+(DΔK)X*hhhhhhhhhhhh×(C+DK*)T]dt∫0t[BΔKX*+X*ΔKTBT}dP=0,(22)λ[trX(T)-c]=0.
For all (ΔX,ΔK)∈Cn×n[0,T]×Cm×n[0,T], (20)-(21) are established. According to Riesz representation theorem, we obtain the second item of (20) and the same of (21).
Without loss of generality, letp(T)=0. Integrate (20) by parts yielding
(23)tr∫0T(Q+K*TRK*)ΔXdt+tr∫0TΔXdP+tr∫0T{∫sP[(A+BK*)ΔX+ΔX(A+BK*)Thhhhhhhhhhh+(C+DK*)TΔX(C+DK*)T]∫s}dt+λtr[ΔX(T)]=0.
Clearly, P has no jump on [0,T). But P has a jump at T, and the value is -λIn. Because the above results are established for all continuous ΔX, then
(24)∫0TΔXdP=PΔX|0T-∫0TPΔX˙dt=-∫0TPΔX˙dt.
Thus, (20) becomes
(25)tr∫0T[(Q+K*TRK*)ΔX+P(A+BK*)ΔX0000l0+(A+BK*)TPΔX+(C+DK*)TP(C+DK*)TΔX0000l0-PΔX˙(Q+K*TRK*)]dt=0.
From Lemma 13, P is differential in [0,T) and (19a) is obtained.
In the same way, integrating (21) by parts, we obtain
(26)tr∫0T[(C+DK*)TK*TR(ΔKX*)+PB(ΔKX*)hihhh+(C+DK*)TPD(ΔKX*)]dt=0.
From Lemma 14, (19c) is obtained.
To ensure the continuity ofP, replace P(T)=0 with P(T)=λIn (i.e., (19b)).
Equation (19d) is called complementary slackness conditions.Remark 16.
Equations (16b)-(16c) of Problem 11 and (19a)–(19c) of Theorem 15 are 2n-dimensional, first-order differential equations including 2n terminal conditions and m algebraic equations. Equation (19d) is called a complementary slackness condition. By using these conditions, X*, K*, P, and λ are obtained.Remark 17.
As for the complementary slackness condition, if the inequality constraint of Problem11 is strict, then λ=0, and the problem becomes easier. If the inequality constraint of Problem 11 is an equality constraint, it simplifies Theorem 15 as Lagrange multiplier method.Definition 18.
M
∈
R
n
×
n is a given matrix. One calls M+ the Moore-Penrose generalized inverse of M, if
(27)MM†M=M,M†MM†=M†,(MM†)T=MM†,(M†M)T=M†M.
Based on Definition 18, we can rewrite Theorem 15 by expressing K* in terms of P.Lemma 19 (see [13]).
Let matricesL, M, and N be given with appropriate sizes. Then the matrix equation
(28)LXM=N
has a solution X if and only if
(29)LL†NM†M=N.
Moreover, any solution to LXM=N is represented by
(30)X=L†NM†+S-L†LSMM†,
where S is a matrix with an appropriate size.Theorem 20.
IfK* is optimal solution of Problem 1, then there exist a unique P∈NBVn×n[0,T] and a nonnegative λ∈R1, such that
(31)-P˙=(Q+K*TRK*)+P(A+BK*)00000+(A+BK*)TP+(C+DK*)P(C+DK*)T,P(T)=λIn,K*TR+PB+(C+DK*)TPD=0,λ[trX(T)-c]=0,K*=-(R+DTPD)†(BTP+DTPC)+Y-(R+DTPD)†(R+DTPD)Y,
where Y∈L2(0,T;Rm×n).Proof.
Form (19c) in Theorem 15, we obtain
(32)(R+DTPD)K*=-(BTP+DTPC).
According to Lemma 19,
(33)K*(R+DTPD)+(BTP+DTPC)+Y-(R+DTPD)+(BTP+DTPC)Y,
whereY∈L2(0,T;Rm×n).As a special case, let us consider the following discrete stochastic LQ control problem without inequality constraint.Problem 21.
Consider the following.(34)minJ(X,K)=tr∫0T[QX(t)+K(t)TRK(t)X(t)]dts.t.X˙=(A+BK)X+X(A+BK)Tjjjjjjjjjjjjjjj+(C+DK)X(C+DK)TjjjjjijX(0)=X0.Corollary 22.
IfK* is optimal solution of Problem 21, then there exists a unique P∈NBVn×n[0,T] to the following constrained GDRE [13]
(35)-P˙=PA+APT+CTPC+Qh-(PB+CTPD)(R+DTPD)(PB+CTPD)T,P(T)=0,(R+DTPD)(R+DTPD)+(BTP+DTPC)=BTP+DTPC,(R+DTPD)≥0,K*=-(R+DTPD)†(BTP+DTPC)+Y-(R+DTPD)†(R+DTPD)Y,
where Y∈L2(0,T;Rm×n).Proof.
Because of Problem21 without the equality constraint, λ=0, therefore P(T)=0. Theorem 20 yields this corollary directly.


## 4. Application

### 4.1. A Necessary and Sufficient Condition

In Theorem15, to ensure the uniqueness of λ and P, let us strengthen the condition (R+DTPD)≥0 as (R+DTPD)>0.Theorem 23.
IfK* is optimal solution of Problem 1, then P∈NBVn×n[0,T] and λ∈R1>0, such that
(36)-P˙=(Q+K*TRK*)+P(A+BK*)hhhl+(A+BK*)TP+(C+DK*)P(C+DK*)T,P(T)=λIn,K*TR+PB+(C+DK*)TPD=0,λ[trX(T)-c]=0,
and if (R+DTPD)>0 hold, then Problem 1 is well posed. The optimal solutions of Problem 1 are
(37)U*=-(R+DTPD)-1(BTP+DTPC)x,J*=x0TP(0)x0-λc.Proof.
In Remark3, in order for a minimum point x* of a mathematical programming to satisfy the KKT conditions, we have assumed that x* satisfies LICQ regularity condition. From (19c) of Theorem 15, we obtain
(38)K*(R+DTPD)=-(BTP+DTPC).
Because (R+DTPD)>0, Problem 11 has a unique solution; namely,
(39)K*=-(R+DTPD)-1(BTP+DTPC).
From the relationship between Problems 1 and 11 and by using Itô’s formula, we have
(40)J*=V(x0)=x0TP(0)x0-λc.


### 4.2. A Dynamic Programming Algorithm

A method for solving the indefinite stochastic linear quadratic (LQ) optimal control problem with unequal terminal state constraint is proposed as follows.Reconsider the following LQ Problem 1′(41)dx(t)=[A(t)x(t)+B(t)u(t)]dt+[C(t)x(t)+D(t)u(t)]dw(t),x(s)=y,E{∥x(T)∥2}=E{xT(T)x(T)}≤c,J(x0,u)=E{∫0T[xT(t)Q(t)x(t)+uT(t)R(t)u(t)]}dt,V(s,y)=infu(·)∈UadJ(s,y;u(·)).Then the backward dynamic programming optimality principle [8] yields the following theorem.Theorem 24.
Letx*(·) and u*(·) be an optimal solution of Problem 1′, then x*(·), and u*(·) satisfy the following Bellman equation:
(42)V(s,y)=minu(·)∈UadE{∫ss+h[x*T(t)Q(t)x*(t)………ll…+u*T(t)R(t)u*(t)]dt,,,,,,,+V(s+h,x*(s+h))∫ss+h[x*T(t)Q(t)x*(t)+u*T(t)R(t)(t)]dt}∀h>0,V(T-h,y)=minu(·)∈UadE{∫T-hT[x*T(t)Q(t)x*(t)……lllll…+u*T(t)R(t)u*(t)]∫T-hT[x*T(t)Q(t)x*(t)}dt,E{∥x(T)∥2}=E[xT(T)x(T)]≤c.Proof.
By applying the backward dynamic programming optimality principle, we obtain recursive relationships betweenV(s,x*(s)) and V(s+h,x*(s+h)) as follows:
(43)V(s,y)=minu(·)∈UadE{∫ss+h[x*T(t)Q(t)x*(t)………+u*T(t)R(t)u*(t)]dt………l+V(s+h,x*(s+h))∫},
where V(s,x*(s)) is the objective of [s,T] and V(s+h,x*(s+h)) of [s+h,T]. Let V(s,x*(s)) be the objective of [T-h,T] and then
(44)V(T-h,y)=minu(·)∈UadE∫T-hT[x*T(t)Q(t)x*(t)nnnnnnnnln+u*T(t)R(t)u*(t)]dt.
Combine the terminal state constraint
(45)E{∥x(T)∥2}=E[xT(T)x(T)]≤c,u(t)=Kx(t)(KisgivenbyTheorem20);
thus we can solve the above recursive relations. Furthermore, let h→0 generating the dynamic programming algorithm.Remark 25.
The dynamic programming algorithm can be applied to solve the stochastic LQ problem with inequality constraint except for a heavy computation. Nonetheless, it makes more sense in theory.Example 26.
In Problem1, let
(46)A=-1,B=1,C=-2,D=0,c=(1+e2)24e2,x0=1,Q=0,R=1,T=1.
By Theorem 15, we have
(47)-P˙=K*2+2P(-1+K*)+4P,P(1)=λ,K*+P=0,λ[X(1)-(1+e2)24e2]=0,λ≥0.
The constraints of Problem 11 are
(48)-X˙=2X+2KX,X(0)=1,X(1)≤(1+e2)24e2t.
Via a series of computations, we obtain
(49)X*=(1+e2)24e2t,P=21+e2t,λ=21+e2,K*=-21+e2t.
In Problem 11, the inequality constraint is
(50)G(X*(1))=X*(1)-c≤0.
When T=1,
(51)X*(1)=(1+e2)24e2=c.
Hence
(52)G(X*(1))=X*(1)-c=0.
This shows that the optimal solution of Problem 11 satisfies the inequality constraint. In this case, the inequality constraint is a nonactive constraint. Because (R+DTPD)=1>0, using Theorem 23, the optimal control of Problem 1 is
(53)u*(t)=-21+e2tx*(t),
and the optimal value is
(54)V(x0)=x0TP(0)x0-λc=1-e-22.


## 4.1. A Necessary and Sufficient Condition

In Theorem15, to ensure the uniqueness of λ and P, let us strengthen the condition (R+DTPD)≥0 as (R+DTPD)>0.Theorem 23.
IfK* is optimal solution of Problem 1, then P∈NBVn×n[0,T] and λ∈R1>0, such that
(36)-P˙=(Q+K*TRK*)+P(A+BK*)hhhl+(A+BK*)TP+(C+DK*)P(C+DK*)T,P(T)=λIn,K*TR+PB+(C+DK*)TPD=0,λ[trX(T)-c]=0,
and if (R+DTPD)>0 hold, then Problem 1 is well posed. The optimal solutions of Problem 1 are
(37)U*=-(R+DTPD)-1(BTP+DTPC)x,J*=x0TP(0)x0-λc.Proof.
In Remark3, in order for a minimum point x* of a mathematical programming to satisfy the KKT conditions, we have assumed that x* satisfies LICQ regularity condition. From (19c) of Theorem 15, we obtain
(38)K*(R+DTPD)=-(BTP+DTPC).
Because (R+DTPD)>0, Problem 11 has a unique solution; namely,
(39)K*=-(R+DTPD)-1(BTP+DTPC).
From the relationship between Problems 1 and 11 and by using Itô’s formula, we have
(40)J*=V(x0)=x0TP(0)x0-λc.


## 4.2. A Dynamic Programming Algorithm

A method for solving the indefinite stochastic linear quadratic (LQ) optimal control problem with unequal terminal state constraint is proposed as follows.Reconsider the following LQ Problem 1′(41)dx(t)=[A(t)x(t)+B(t)u(t)]dt+[C(t)x(t)+D(t)u(t)]dw(t),x(s)=y,E{∥x(T)∥2}=E{xT(T)x(T)}≤c,J(x0,u)=E{∫0T[xT(t)Q(t)x(t)+uT(t)R(t)u(t)]}dt,V(s,y)=infu(·)∈UadJ(s,y;u(·)).Then the backward dynamic programming optimality principle [8] yields the following theorem.Theorem 24.
Letx*(·) and u*(·) be an optimal solution of Problem 1′, then x*(·), and u*(·) satisfy the following Bellman equation:
(42)V(s,y)=minu(·)∈UadE{∫ss+h[x*T(t)Q(t)x*(t)………ll…+u*T(t)R(t)u*(t)]dt,,,,,,,+V(s+h,x*(s+h))∫ss+h[x*T(t)Q(t)x*(t)+u*T(t)R(t)(t)]dt}∀h>0,V(T-h,y)=minu(·)∈UadE{∫T-hT[x*T(t)Q(t)x*(t)……lllll…+u*T(t)R(t)u*(t)]∫T-hT[x*T(t)Q(t)x*(t)}dt,E{∥x(T)∥2}=E[xT(T)x(T)]≤c.Proof.
By applying the backward dynamic programming optimality principle, we obtain recursive relationships betweenV(s,x*(s)) and V(s+h,x*(s+h)) as follows:
(43)V(s,y)=minu(·)∈UadE{∫ss+h[x*T(t)Q(t)x*(t)………+u*T(t)R(t)u*(t)]dt………l+V(s+h,x*(s+h))∫},
where V(s,x*(s)) is the objective of [s,T] and V(s+h,x*(s+h)) of [s+h,T]. Let V(s,x*(s)) be the objective of [T-h,T] and then
(44)V(T-h,y)=minu(·)∈UadE∫T-hT[x*T(t)Q(t)x*(t)nnnnnnnnln+u*T(t)R(t)u*(t)]dt.
Combine the terminal state constraint
(45)E{∥x(T)∥2}=E[xT(T)x(T)]≤c,u(t)=Kx(t)(KisgivenbyTheorem20);
thus we can solve the above recursive relations. Furthermore, let h→0 generating the dynamic programming algorithm.Remark 25.
The dynamic programming algorithm can be applied to solve the stochastic LQ problem with inequality constraint except for a heavy computation. Nonetheless, it makes more sense in theory.Example 26.
In Problem1, let
(46)A=-1,B=1,C=-2,D=0,c=(1+e2)24e2,x0=1,Q=0,R=1,T=1.
By Theorem 15, we have
(47)-P˙=K*2+2P(-1+K*)+4P,P(1)=λ,K*+P=0,λ[X(1)-(1+e2)24e2]=0,λ≥0.
The constraints of Problem 11 are
(48)-X˙=2X+2KX,X(0)=1,X(1)≤(1+e2)24e2t.
Via a series of computations, we obtain
(49)X*=(1+e2)24e2t,P=21+e2t,λ=21+e2,K*=-21+e2t.
In Problem 11, the inequality constraint is
(50)G(X*(1))=X*(1)-c≤0.
When T=1,
(51)X*(1)=(1+e2)24e2=c.
Hence
(52)G(X*(1))=X*(1)-c=0.
This shows that the optimal solution of Problem 11 satisfies the inequality constraint. In this case, the inequality constraint is a nonactive constraint. Because (R+DTPD)=1>0, using Theorem 23, the optimal control of Problem 1 is
(53)u*(t)=-21+e2tx*(t),
and the optimal value is
(54)V(x0)=x0TP(0)x0-λc=1-e-22.


## 5. Conclusion

We have studied the indefinite stochastic LQ optimal control problem with unequal terminal state constraint, which can be transformed into a hybrid constrained mathematical programming problem. By applying KKT theorem, we have presented a necessary condition for the constrained indefinite stochastic LQ optimal control problem. By adding some conditions, we obtain a necessary and sufficient condition for indefinite stochastic LQ optimal control problem with inequality constraint. We demonstrate that the solvability of the generalized Riccati equation is sufficient for the well posedness of the indefinite LQ problem. Since this kind of LQ control problems can be transformed into a hybrid constrained mathematical programming, we have given a dynamic programming algorithm.



---

*Source: 805829-2013-12-24.xml*