# Study on Stochastic Linear Quadratic Optimal Control with Quadratic and Mixed Terminal State Constraints

Authors: Yang Hongli
Journal: Journal of Applied Mathematics
 (2013)

---

## Abstract

This paper studies the indefinite stochastic LQ control problem
with quadratic and mixed terminal state equality constraints, which can
be transformed into a mathematical programming problem. By means
of the Lagrangian multiplier theorem and Riesz representation theorem, the
main result given in this paper is the necessary condition for indefinite
stochastic LQ control with quadratic and mixed terminal equality
constraints. The result shows that the different terminal state constraints
will cause the endpoint condition of the differential Riccati equation
to be changed. It coincides with the indefinite stochastic LQ problem
with linear terminal state constraint, so the result given in this paper can
be viewed as the extension of the indefinite stochastic LQ problem
with the linear terminal state equality constraint. In order to guarantee
the existence and the uniqueness of the linear feedback control, a
sufficient condition is also presented in the paper. A numerical example
is presented at the end of the paper.


---

## Body

## 1. Introduction

Linear quadratic (LQ) control is an extremely important class of control problems in both theory and application. It is pioneered by Kalman [1] for deterministic systems and was extended to stochastic systems by Wonham [2]. In recent years, extensive research has been carried out in the so-called indefinite stochastic LQ control, in which the cost weighting matrices are allowed to be indefinite; refer to [3–6] for detailed accounts. A basic assumption in the LQ theory, both for deterministic and stochastic cases, is that the variable is unconstrained except for the differential equations constraint. As far as we know, very few results for constrained deterministic LQ can be found compared with the unconstrained one, not to mention the stochastic LQ control [7]. While in many real applications, constrained LQ control problem (such as nonnegativity and bound constraints for state and control variables) is a well-posed problem, constrained stochastic LQ control problem has a concrete application background, but the conventional LQ approach would collapse in the presence of any constraints. Study on the constrained stochastic LQ control will contribute to both theory and application a lot.Huang and Zhang [8] studied the indefinite stochastic LQ control problem with linear terminal state equality constraints. Necessary and sufficient conditions for indefinite stochastic LQ control problems were investigated based on the Lagrangian multiplier theorem and Riesz representation theorem. The result showed that the linear feedback optimal control can be obtained by solving systems of algebraic and differential equations. The previous results on unconstrained indefinite stochastic LQ can be viewed as a specified case of the main theorem in that paper.This paper studied the indefinite stochastic LQ control problem with quadratic terminal equality constraints and mixed constraints, which can be viewed as the extension of [8]. When the terminal state constraint is quadratic, the feasible region defined by the terminal constraint is nonconvex and multiple local minima abound, which makes the problem more complex to locate the optimum consistently. Developing a deeper understanding of the problems, as well as efficient algorithms for solving them, will have a big impact in many applications. Another reason for the study of this problem is that the methods used for solving this type of problem can be used to solve more general constrained optimal control problems. By means of the Lagrangian multiplier theorem and Riesz representation theorem, the main result in this paper is the necessary condition for indefinite stochastic LQ control with quadratic terminal constraints and mixed terminal constraints. The result showed that the difference of the terminal state constraints will cause the endpoint condition to be changed in the differential equations we obtained for the linear constraint control problem, which coincides with the reality. In order to guarantee the existence and the uniqueness of the linear feedback control, a sufficient condition is also presented in the paper. Numerical example is presented at the end of this paper.For the convenience, we make use of the following basic notation in this paper.Rn is the n-dimensional real column vector, x(t)=(x1(t),…,xn(t))′ is the n-dimensional statement column vector, u(t)∈Rm is the control input vector, ω(·) is the one-dimensional Brownian motion defined on filtered probability (Ω,F,P;Fl) with a standard Fl-adapted on [0,T], and Fl=σ(ω(s):0≤s≤t≤T) is an information fluid produced by Brownian motion. u(t) belongs to the admissible control set  Uad={u(t)∈Rm,u(t)∈LF2(0,T;Rm)} is the adapted stochastic process which satisfies E∫0T‍|u2(t)|dt<+∞ and the corresponding x(t) satisfies (3), ξi is the given FT measurable, integrable random variable; that is, E(ξi)<+∞,  and i=1,2,…,l+q is a known number. A,C∈L∞(0,T;Sn×n),Q∈L∞(0,T;Sn×n), B,D∈L∞(0,T;Rn×m), R∈L∞(0,T;Sm×m), Rm×m is m row n column matrix, and Sn×n is an n×n symmetric matrix. Ml∈Rl×n is the terminal state linear constraint coefficient matrix, and Mq∈Rq×n is the terminal state quadratic constraint matrix. Two assumptions are given in this paper.H1:
Ml, Mq (the coefficient matrix for terminal linear and quadratic constraints respectively in Problem 1 to Problem 21) are full row rank and the set defined by the terminal state constraints is not empty;H2:
L∞(0,T;X):={f(t):f(t) is an Fl-adapted, X-valued measurable process on [0,T], and E∫0T‍∥f(t)∥X2dt<+∞}.The rest of this paper is organized as follows. Section2 gives the problem statement and some preliminaries, Section 3 presents the main results of this paper. Numerical example is given in Section 4. Finally, Section 5 concludes this paper.


## 2. Problem Statement and Preliminaries

Problem 1.
We study the following indefinite stochastic LQ control problem:(1)inf{J(x0,u):=E∫0T‍x′Qx+u′Rudt}s.t.(2)dx(t)=(A(t)x(t)+B(t)u(t))+(C(t)x(t)+D(t)u(t))dω(t),(3)x(0)=x0,m11x112(T)+m12x122(T)+⋯+m1nx1n2(T)=ξ1,m21x212(T)+m22x222(T)+⋯+m2nx2n2(T)=ξ2,⋮mq1xq12(T)+mq2xq22(T)+⋯+mqnxqn2(T)=ξq.Suppose that the feasible set defined by the quadratic constraint is not empty. We investigate the necessary condition under the linear feedback optimal control(4)u(t)=K(t)x(t),in whichK(t)∈Cm×n[0,T]. The cost weight matrix in Problem 1 is not necessarily positive, which usually causes Problem 1 to be called indefinite stochastic LQ control problem with terminal constraint. First, we present some definitions and lemmas that will be used and then transform the optimal control Problem 1 into a deterministic control problem.Definition 2 (Gateaux differential).
LetX be a vector space, let Y be a normed space, and let T be a (possibly nonlinear) transformation defined on a domain D⊆X with a range R⊆Y. Given x∈D⊂X, and h is an arbitrary vector in X. If the limit
(5)δT(x;h)=limα→01α[T(x+αh)-T(x)]
exists, then it is called Gateaux differential at x with an increment h. If the limit exists for arbitrary h∈X, the transformation T is called Gateaux differentiable at x.Definition 3 (Frechet differential).
LetX and Y be normed linear spaces and let T be a transformation defined on an open domain D⊂X with a range R⊂Y. If, for a fixed x∈D⊂X, there exist δT(x;h)∈Y, which is linear and continuous with respect to h such that
(6)lim∥h(t)∥→0∥T(x+h)-T(x)-δT(x;h)∥∥h(t)∥=0,
then T is said to be Frechet differentiable at x, and δT(x;h) is said to be the Frechet differential of T at x with an increment h.Definition 4 (Frechet derivative and continuously Frechet differentiable).
Suppose that a transformationT defined on an open domain D⊂X is Frechet differentiable on D. The Frechet differential δT(x;h)=T′(x)h for a fixed x∈D, where T′(x) is a bounded linear operator from X to Y, one calls T′(x) the derivative of T(x). If the derivative T′(x) is continuous on some open ball S, then T is continuously Frechet differentiable on S.Definition 5 (regular point of transformation).
LetT(x) be a transformation defined on a Banach space X with a range Y, which is also a Banach space. T(x) is continuously Frechet differentiable if, for a given x0∈X,  δT(x0;h) is an onto mapping from X onto Y; then x0 is called a regular point.
Considering Problem1 under the linear feedback optimal control, substituting u(t) in Problem 1 with
(7)u(t)=K(t)x(t).
And taking the place of u(t) in (2), we have
(8)dx=(A+BK)xdt+(C+DK)xdω.
According to the Itô integrals formula for xx′, in which x(t) is the solution of (8), we have
(9)d(xx′)=[(A+BK)xx′+xx′(A+BK)′+(C+DK)xx′(C+DK)]dt.
Define E(xx′)=X; it is obvious that X is a symmetric matrix. Integrate both sides of (9) from 0 to t with variable t, then compute the derivatives of both sides of (9) after taking the expectation, then get the following matrix differential equation with the initial condition:
(10)X˙=(A+BK)X+X(A+BK)′+(C+DK)X(C+DK)′,X(0)=X0=x0x0′.
Substituting u(t) with u(t)=K(t)x(t) in (1) to
(11)inf{J(X,K)=tr∫0T‍[QX(t)+K′(t)RK(t)X(t)]dt},
(“tr” is the trace of a matrix, E(ξi)=li, i=1,2,…,q), the quadratic constraints are
(12)tr(Q1X(T))=l1,tr(Q2X(T))=l2,…,tr(QqX(T))=lq,
where Qi=diag[mi1,mi2,…,min], i=1,2,…,q, and L=(l1,l2,…,lq)′. Then Problem 1 can be transformed into the following deterministic optimal control Problem 6.Problem 6.
Consider(13)inf{J(X,K)=tr∫0T‍[QX(t)+K′(t)RK(t)X(t)]dt}
s.t. 							(14)X˙=(A+BK)X+X(A+BK)′+(C+DK)X(C+DK)′,(15)X(0)=X0=x0x0,(16)tr(Q1X(T))=l1,tr(Q2X(T))=l2,⋯tr(QqX(T))=lq.Remark 7.
It is obvious that if Problem1 has the optimal linear feedback control u*(t)=K*(t)x(t), then K*(t) must be the solution of Problem 6, while the inverse does not hold.
Take the objective functionalJ(X,K) as a functional defined on the space Cn×n[0,T]×Cn×n[0,T], in which Cn×n[0,T] is an n×n square matrix space, the element in this space is the continuous function defined on [0,T]:
(17)H(X,K)(t)=X(t)-X(0)-∫0t‍[+(C+DK)X(C+DK)′(A+BK)X+X(A+BK)i+(C+DK)X(C+DK)′]dt,
and terminal state constraints (16) defines a transformation G(X) from Cn×n[0,T] to Rq:(18)G(x(T))=(tr(Q1X(T))⋮tr(QqX(T))).
By the virtue of (17)-(18), constraints (14)–(16) can be reformulated as
(19)H(X,K)(t)=0,G(X(T))=L,∀t∈[0,T].Lemma 8.
FunctionalJ(X,K), H(X,K), G(X) are continuous Frechet differentiable functionals and have Frechet derivative
(20)δJX(X,K;ΔX)=tr∫0T‍(Q+K′RK)ΔXdt,(21)δJK(X,K;ΔK)=tr∫0T‍(ΔK′RKX+K′RΔKX)dt,(22)δHX(X,K;ΔX)(t)=ΔX(t)-∫0t‍[+(C+DK)ΔX(C+DK)′(A+BK)ΔX+ΔX(A+BK)+(C+DK)ΔX(C+DK)′]dt,(23)δHK(X,K;ΔK)(t)=-∫0t‍[BΔKX+XΔK′B′+(C+DK)X(DΔK)+DΔKX(C+DK)′]dt,(24)δGX(X;ΔX(T))=(tr(Q1ΔX(T))⋮tr(QqΔX(T))).Proof.
The proof is mainly based on Definition2. Apply Definition 2 to the functional constraints (19); the proofs for the first three results are the same to those in [8], and we only need the proof of (24) here:
(25)ΔG(X;ΔX)=limα→0G(X;X+αΔX-G(X))α,G(X;X+αX)-G(X)=(tr(Q1X(T)+αΔX(T))⋮tr(QqX(T)+αΔX(T)))-(tr(Q1X(T))⋮tr(QqX(T))),δG(X,ΔX)=limα→0G(X;X+ΔX)-G(X)α=(tr(Q1ΔX)⋮tr(QqΔX)).Lemma 9.
Constraint (19) satisfies the regular condition; that is,
(26)δH(X,K;ΔX,ΔK),δG(X;ΔX(T))
are onto mapping when
(27)ΔX×ΔK∈Cn×n×Cm×n.Proof.
The proof of the onto mapping forδH(X,K;ΔX,ΔK) has been given in [8]; we only need to prove that δG(X;ΔX) is a onto mapping when ΔX varies. For a given Y∈Rq, the following equation has a solution:
(28)(tr(Q1ΔX)⋮tr(QqΔX))=Y.
Because the coefficient matrix for quadratic terminal state constraints is full row rank (Assumption H1), equation
(29)(m11⋯m1n⋮⋮⋮mq1⋯mqn)(ΔX11⋮ΔXqn)=Y,
exists a solution, which finishes the proof.Definition 10 (the well-posedness).
The LQ optimal control problem is to minimize the cost functionalJ(x0,u) over u∈Uad. Define the optimal value function as V(x0)=infu∈UadJ(x0,u). The LQ problem is called well posed if
(30)-∞<V(x0)<+∞,∀x0∈Rn.
A well-posed problem is called attainable (with respect to x0) if there is a u*(·) control that achieves V(x0).Theorem 11 (Lagrange multiplier theorem).
If the continuously Frechet differentiable real functionalf defined on Banach space has a local extremum under the constraint H(x)=0 at the regular point x0, H(x) is a mapping from space X to Banach space Z, and then there exists an element z0*∈Z* such that the Lagrangian functional
(31)L(x)=f(x)+z0*H(x)
is stationary at x0; that is,
(32)δf(x0;h)+z0*δH(x0;h)=0
for each h∈X.Theorem 12 (Riesz representation theorem).
Letf(x) be a bounded linear functional on X=C[a,b], and then there is a bounded variation function ν on [a,b] such that, for all x∈X,
(33)f(x)=∫ab‍x(t)dν(t),
where the norm of f(x) is the total variation on [a,b]. Conversely, every function of bounded variation on [a,b] defines a bounded linear function on X in this way.


## 3. Main Results

Lemma 13 (see [9]).
Ifα(t) is continuous in [t1,t2] and ∫t1t2‍α(t)dh(t)=0  for every h∈D[t1,t2] (set of all continuously differentiable functions on [t1,t2]) with h(t1)=h(t2)=0, then α(t)=0 on [t1,t2].Lemma 14 (see [9]).
Ifα(t) is continuous in [t1,t2] and ∫t1t2‍α(t)dh(t)=0 for every h∈D[t1,t2] with h(t1)=h(t2)=0, then α(t)=c in [t1,t2], where c is a constant.Lemma 15 (see [9]).
Ifα(t) and β(t) are continuous in [t1,t2] and
(34)∫t1t2‍[α(t)h(t)+β(t)h′(t)]dt=0
for every h∈D[t1,t2] with h(t1)=h(t2)=0, then β is differentiable and β˙(t)=α(t) in [t1,t2].We make use ofNBVn×n[0,T] (nonnegative bounded variation functional on [0,T]) to express the matrix space with the element in [0,T]. The space is a bounded variation right continuous function that function takes value 0 at the point at the point t=0. Based on the Lagragian theorem and the Riesz representation theorem, we obtained a necessary condition for Problem 1.


### 3.1. Necessary Condition

Theorem 16 (necessary condition for indefinite stochastic LQ with quadratic constraints).
Suppose the set defined by the terminal state constraint is not empty and the optimal control Problem6 exists an optimal feedback control matrix K*; then there must exist a symmetric matrix P∈NBVn×n[0,T] and a vector λ∈Rq such that for all t∈[0,T], P and λ satisfy the following equations:
(35)-P˙=(Q+K*′RK*)+P(A+BK*)+(A+BK*)′P+(C+DK*)′P(C+DK*),(36)P(T)=∑i=1q‍λiQi,(37)K*′R+PB+(C+DK*)′PD=0.Proof.
From Remark7, we know that K* is the solution of deterministic optimal control Problem 6; take X and K in Problem 6 as the variables. Suppose that (X*,K*) is the optimal solution for Problem 6; according to Lemmas 14 and 15, Problem 6 satisfies the conditions of Lagrange multiplier theorem, and then there exists a symmetric matrix P∈NBVn×n[0,T] and λ∈Rq such that
(38)tr∫0T‍(Q+K*RK*)ΔXdt+tr∫0T‍(ΔX-∫0t‍(A+BK*)ΔX+ΔX(A+BK*)+(C+DK*)ΔX(C+DK*)′dt∫0t‍)dP+λ′(tr(Q1X)⋮tr(QqX))=0,(39)tr∫0T‍(ΔK′RK*X*+K*′RΔKX*)dt-tr∫0T‍(∫0t‍BΔKX*+X*ΔK′B′+(C+DK*)X*×(DΔK′)+(DΔK)X*(C+DK*)dt∫0t‍)dP=0.
The second parts of (38) and (39) are from Riesz representation theorem. In general, we take that P(T)=0, and then (38) becomes
(40)tr∫0T‍(Q+K*′RK*)ΔXdt+tr∫0T‍ΔXdP+tr∫0T‍(P[+(C+DK*)ΔX(C+DK*)′(A+BK*)ΔX+ΔX(A+BK*)+(C+DK*)ΔX(C+DK*)′])+tr(λ1Q1X+⋯+λqQqX)=0.
It is obvious that there is no jump in interval [0,T) for function P(t), otherwise we can choose ΔX that makes tr∫0T‍ΔXdP be far more than the other parts in the equality. But P(t) has jump at the point t=T, and the height is
(41)(λ1,…,λq)(tr(Q1X(T))⋮tr(QqX(T))).
Because the previous equalities (38) and (39) hold for all continuous functions ΔX, for specified function which has continuous derivative and ΔX(T)=0, all the previous equalities also hold, and then
(42)∫0T‍ΔXdP=PΔX|0T-∫0T‍PΔX˙dt=P(T)ΔX(T)-P(0)ΔX(0)-∫0T‍PΔX˙dt=P(T)ΔX(T)-∫0T‍PΔX˙dt.
Because P(T)=0, then
(43)∫0T‍ΔXdP=-∫0T‍PΔX˙dt,
and we have the following from (38)
(44)tr∫0T‍(Q+K*′RK*)ΔX+P(A+BK*)ΔX+(A+BK*)′×PΔX+(C+DK*)′P(C+DK*)ΔX-PΔX˙dt=0.
According to Lemma 15, we have
(45)-P˙=(Q+K*′RK*)+P(A+BK*)+(A+BK*′)P+(C+DK*)′P(C+DK*).
We take the integral by parts of the second part of (39), and then
(46)tr∫0T‍[K*′R(ΔK)X*+PB(ΔKX*)+(C+DK*)′PD(ΔKX*)]dt=0.
Based on Lemma 13, we have
(47)K*′R+PB+(C+DK*)′PD=0.
Change the endpoint conditions P(T)=0 with P(T)=λ1Q1+⋯+λqQq; that is, (36) holds.Remark 17.
The necessary condition in Theorem16 (35)–(37) and the constraints (14)–(16) in Problem 6 have 2n one order differential equations, 2n bound conditions, q terminal state conditions, and m algebra equations, and then we can determine X*(t),  K*(t),  P(t), and λ, respectively.Remark 18.
The result in Theorem12 is the same as that in [8] except for the terminal conditions.


### 3.2. Sufficient Condition

We have pointed out that the necessary conditions (14)–(16) and (35)–(37) are not sufficient for the existence and the uniqueness of the solution in Problem 6. In order to guarantee the uniqueness of λ and P(t), the conditions must be strengthened to
(48)P+D′PD>0;
that is, the matrix R+D′PD must be positive.Theorem 19 (sufficient condition for stochastic LQ problem with quadratic constraints).
If (14)–(16), (35)–(37), and (39) exist solutions X*(t), K*(t), P(t) , and λ, then Problem 6 is well posed and the optimal feedback control is
(49)u*(t)=-(R+D′PD)-1(B′P+D′PC)x(t);
the optimal cost value is
(50)V(x0)=infu(·)∈UadJ(x0,u(t))=x(0)′P(0)x(0)-λ′L.Proof.
Suppose thatK*(t), P(t), and λ are the solutions that satisfy (14)–(16), (35)–(37), and (48), and from (37), we have
(51)K*(t)=-(R+D′PD)-1(B′P+D′PC).
Substitute K*(t) in (35), and then
(52)P˙+PA+A′P+C′PC+Q-(PB+C′PD)(R+D′PD)-1(PB+C′PD)=0,J(x0,u(·))=E∫0T‍[x′Qx+u′Ru]dt+d(x′Px)-d(x′Px).
Apply Itô’s formula again, so
(53)=x′(0)P(0)x(0)-E(x′(T)P(T)x(T))+E∫0T‍x′(P˙+PA+A′P+C′PC+Q)x+2u′(B′P+D′PC)x+u′(R+D′PD)udt=x′(0)P(0)x(0)-λ′(tr(Q1X(T))⋮tr(QqX(T)))+E∫0T‍x′[(R+D′PD)-1P˙+PA+A′P+C′PC+Q-(PB+C′PD)i×(R+D′PD)-1(PB+C′PD)′P˙]xdt+E∫0T‍[u+(R+D′PD)-1(B′P+D′PC)x]′×(R+D′PD)[u+(R+D′PD)-1×(B′P+D′PC)x(R+D′PD)-1]dt=x′(0)P(0)x(0)-λ′(tr(Q1X(T))⋮tr(QqX(T)))+E∫0T‍[u+(R+D′PD)-1(B′P+D′PC)x]′×(R+D′PD)[u+(R+D′PD)-1i×(B′P+D′PC)x(R+D′PD)-1]dt,
where the last equality is from (52). It is obvious that linear feedback control (49) has the minimum cost J(x0,u(t)), and the cost value is
(54)x′(0)P(0)x(0)-λ′(tr(Q1X(T))⋯tr(QqX(T))).


### 3.3. Necessary Conditions for Stochastic LQ with Mixed Terminal State Constraint

We study the following indefinite stochastic LQ control problem with mixed constraints.Problem 20.
Consider(55)inf{J(x0,u):=E∫0T‍x′Qx+u′Rudt}
s.t. 									(56)dx(t)=(A(t)x(t)+B(t)u(t))+(C(t)x(t)+D(t)u(t))dω(t),x(0)=x0,m1,1x1,1(T)+m1,2x1,2(T)+⋯+m1,nx1,n(T)=ξ1,m2,1x2,1(T)+m2,2x2,2(T)+⋯+m2,nx2,n(T)=ξ2,⋯ml,1xl,1(T)+ml,2xl,2(T)+⋯+ml,nxl,n(T)=ξl,ml+1,1xl+1,12(T)+ml+1,2xl+1,22(T)+⋯+ml+1,nxl+1,n2(T)=ξl+1,ml+2,1xl+2,12(T)+ml+2,2xl+2,22(T)+⋯+ml+2,nxl+2,n2(T)=ξl+2,⋯ml+q,1xl+q,12(T)+ml+q,2xl+q,22(T)+⋯+ml+q,nxl+q,n2(T)=ξl+q.
Using the same method as the above, the stochastic LQ control problem with linear and quadratic terminal constraints can be transformed into the following Problem 21.Problem 21.
Consider(57)infJ(X,K)=tr∫0T‍[QX(t)+K′(t)RK(t)X(t)]dt
s.t. 									(58)X˙=(A+BK)X+X(A+BK)′+(C+DK)X(C+DK)′,X(0)=X0=x0x0,MlX(T)Ml′=N,tr(Q1X(T))=l1,tr(Q2X(T))=l2,⋯tr(QqX(T))=lq.Theorem 22 (necessary and sufficient conditions for stochastic LQ problem with mixed constraint).
Suppose that the terminal constraint set defined by the linear and quadratic state constraints is not empty and the optimal control Problem6 exists the optimal feedback control matrix K*. Then there must exist a symmetric matrix P∈NBVn×n[0,T] and a vector λ∈Rq, μ∈Rl×l, for all t∈[0,T], satisfing the following equalities:
(59)-P˙=(Q+K*′RK*)+P(A+BK*)+(A+BK*)′P+(C+DK*)′P(C+DK*),P(T)=∑i=1q‍λiQi+Ml′μMl,K*′R+PB+(C+DK*)′PD=0.If (14)–(16), (35)–(37), and (39) exist solutions X*(t),  K*(t), P(t), and λ, then Problem 6 is well posed and the optimal feedback control is
(60)u*(t)=-(R+D′PD)-1(B′P+D′PC)x(t).
The optimal value is
(61)V(x0)=infu(·)∈UadJ(x0,u(t))=x0′P(0)x(0)-λ′Lq-tr(μN).The proof of the theorem is the same as that of Theorem16 except for the terminal condition.


## 3.1. Necessary Condition

Theorem 16 (necessary condition for indefinite stochastic LQ with quadratic constraints).
Suppose the set defined by the terminal state constraint is not empty and the optimal control Problem6 exists an optimal feedback control matrix K*; then there must exist a symmetric matrix P∈NBVn×n[0,T] and a vector λ∈Rq such that for all t∈[0,T], P and λ satisfy the following equations:
(35)-P˙=(Q+K*′RK*)+P(A+BK*)+(A+BK*)′P+(C+DK*)′P(C+DK*),(36)P(T)=∑i=1q‍λiQi,(37)K*′R+PB+(C+DK*)′PD=0.Proof.
From Remark7, we know that K* is the solution of deterministic optimal control Problem 6; take X and K in Problem 6 as the variables. Suppose that (X*,K*) is the optimal solution for Problem 6; according to Lemmas 14 and 15, Problem 6 satisfies the conditions of Lagrange multiplier theorem, and then there exists a symmetric matrix P∈NBVn×n[0,T] and λ∈Rq such that
(38)tr∫0T‍(Q+K*RK*)ΔXdt+tr∫0T‍(ΔX-∫0t‍(A+BK*)ΔX+ΔX(A+BK*)+(C+DK*)ΔX(C+DK*)′dt∫0t‍)dP+λ′(tr(Q1X)⋮tr(QqX))=0,(39)tr∫0T‍(ΔK′RK*X*+K*′RΔKX*)dt-tr∫0T‍(∫0t‍BΔKX*+X*ΔK′B′+(C+DK*)X*×(DΔK′)+(DΔK)X*(C+DK*)dt∫0t‍)dP=0.
The second parts of (38) and (39) are from Riesz representation theorem. In general, we take that P(T)=0, and then (38) becomes
(40)tr∫0T‍(Q+K*′RK*)ΔXdt+tr∫0T‍ΔXdP+tr∫0T‍(P[+(C+DK*)ΔX(C+DK*)′(A+BK*)ΔX+ΔX(A+BK*)+(C+DK*)ΔX(C+DK*)′])+tr(λ1Q1X+⋯+λqQqX)=0.
It is obvious that there is no jump in interval [0,T) for function P(t), otherwise we can choose ΔX that makes tr∫0T‍ΔXdP be far more than the other parts in the equality. But P(t) has jump at the point t=T, and the height is
(41)(λ1,…,λq)(tr(Q1X(T))⋮tr(QqX(T))).
Because the previous equalities (38) and (39) hold for all continuous functions ΔX, for specified function which has continuous derivative and ΔX(T)=0, all the previous equalities also hold, and then
(42)∫0T‍ΔXdP=PΔX|0T-∫0T‍PΔX˙dt=P(T)ΔX(T)-P(0)ΔX(0)-∫0T‍PΔX˙dt=P(T)ΔX(T)-∫0T‍PΔX˙dt.
Because P(T)=0, then
(43)∫0T‍ΔXdP=-∫0T‍PΔX˙dt,
and we have the following from (38)
(44)tr∫0T‍(Q+K*′RK*)ΔX+P(A+BK*)ΔX+(A+BK*)′×PΔX+(C+DK*)′P(C+DK*)ΔX-PΔX˙dt=0.
According to Lemma 15, we have
(45)-P˙=(Q+K*′RK*)+P(A+BK*)+(A+BK*′)P+(C+DK*)′P(C+DK*).
We take the integral by parts of the second part of (39), and then
(46)tr∫0T‍[K*′R(ΔK)X*+PB(ΔKX*)+(C+DK*)′PD(ΔKX*)]dt=0.
Based on Lemma 13, we have
(47)K*′R+PB+(C+DK*)′PD=0.
Change the endpoint conditions P(T)=0 with P(T)=λ1Q1+⋯+λqQq; that is, (36) holds.Remark 17.
The necessary condition in Theorem16 (35)–(37) and the constraints (14)–(16) in Problem 6 have 2n one order differential equations, 2n bound conditions, q terminal state conditions, and m algebra equations, and then we can determine X*(t),  K*(t),  P(t), and λ, respectively.Remark 18.
The result in Theorem12 is the same as that in [8] except for the terminal conditions.


## 3.2. Sufficient Condition

We have pointed out that the necessary conditions (14)–(16) and (35)–(37) are not sufficient for the existence and the uniqueness of the solution in Problem 6. In order to guarantee the uniqueness of λ and P(t), the conditions must be strengthened to
(48)P+D′PD>0;
that is, the matrix R+D′PD must be positive.Theorem 19 (sufficient condition for stochastic LQ problem with quadratic constraints).
If (14)–(16), (35)–(37), and (39) exist solutions X*(t), K*(t), P(t) , and λ, then Problem 6 is well posed and the optimal feedback control is
(49)u*(t)=-(R+D′PD)-1(B′P+D′PC)x(t);
the optimal cost value is
(50)V(x0)=infu(·)∈UadJ(x0,u(t))=x(0)′P(0)x(0)-λ′L.Proof.
Suppose thatK*(t), P(t), and λ are the solutions that satisfy (14)–(16), (35)–(37), and (48), and from (37), we have
(51)K*(t)=-(R+D′PD)-1(B′P+D′PC).
Substitute K*(t) in (35), and then
(52)P˙+PA+A′P+C′PC+Q-(PB+C′PD)(R+D′PD)-1(PB+C′PD)=0,J(x0,u(·))=E∫0T‍[x′Qx+u′Ru]dt+d(x′Px)-d(x′Px).
Apply Itô’s formula again, so
(53)=x′(0)P(0)x(0)-E(x′(T)P(T)x(T))+E∫0T‍x′(P˙+PA+A′P+C′PC+Q)x+2u′(B′P+D′PC)x+u′(R+D′PD)udt=x′(0)P(0)x(0)-λ′(tr(Q1X(T))⋮tr(QqX(T)))+E∫0T‍x′[(R+D′PD)-1P˙+PA+A′P+C′PC+Q-(PB+C′PD)i×(R+D′PD)-1(PB+C′PD)′P˙]xdt+E∫0T‍[u+(R+D′PD)-1(B′P+D′PC)x]′×(R+D′PD)[u+(R+D′PD)-1×(B′P+D′PC)x(R+D′PD)-1]dt=x′(0)P(0)x(0)-λ′(tr(Q1X(T))⋮tr(QqX(T)))+E∫0T‍[u+(R+D′PD)-1(B′P+D′PC)x]′×(R+D′PD)[u+(R+D′PD)-1i×(B′P+D′PC)x(R+D′PD)-1]dt,
where the last equality is from (52). It is obvious that linear feedback control (49) has the minimum cost J(x0,u(t)), and the cost value is
(54)x′(0)P(0)x(0)-λ′(tr(Q1X(T))⋯tr(QqX(T))).


## 3.3. Necessary Conditions for Stochastic LQ with Mixed Terminal State Constraint

We study the following indefinite stochastic LQ control problem with mixed constraints.Problem 20.
Consider(55)inf{J(x0,u):=E∫0T‍x′Qx+u′Rudt}
s.t. 									(56)dx(t)=(A(t)x(t)+B(t)u(t))+(C(t)x(t)+D(t)u(t))dω(t),x(0)=x0,m1,1x1,1(T)+m1,2x1,2(T)+⋯+m1,nx1,n(T)=ξ1,m2,1x2,1(T)+m2,2x2,2(T)+⋯+m2,nx2,n(T)=ξ2,⋯ml,1xl,1(T)+ml,2xl,2(T)+⋯+ml,nxl,n(T)=ξl,ml+1,1xl+1,12(T)+ml+1,2xl+1,22(T)+⋯+ml+1,nxl+1,n2(T)=ξl+1,ml+2,1xl+2,12(T)+ml+2,2xl+2,22(T)+⋯+ml+2,nxl+2,n2(T)=ξl+2,⋯ml+q,1xl+q,12(T)+ml+q,2xl+q,22(T)+⋯+ml+q,nxl+q,n2(T)=ξl+q.
Using the same method as the above, the stochastic LQ control problem with linear and quadratic terminal constraints can be transformed into the following Problem 21.Problem 21.
Consider(57)infJ(X,K)=tr∫0T‍[QX(t)+K′(t)RK(t)X(t)]dt
s.t. 									(58)X˙=(A+BK)X+X(A+BK)′+(C+DK)X(C+DK)′,X(0)=X0=x0x0,MlX(T)Ml′=N,tr(Q1X(T))=l1,tr(Q2X(T))=l2,⋯tr(QqX(T))=lq.Theorem 22 (necessary and sufficient conditions for stochastic LQ problem with mixed constraint).
Suppose that the terminal constraint set defined by the linear and quadratic state constraints is not empty and the optimal control Problem6 exists the optimal feedback control matrix K*. Then there must exist a symmetric matrix P∈NBVn×n[0,T] and a vector λ∈Rq, μ∈Rl×l, for all t∈[0,T], satisfing the following equalities:
(59)-P˙=(Q+K*′RK*)+P(A+BK*)+(A+BK*)′P+(C+DK*)′P(C+DK*),P(T)=∑i=1q‍λiQi+Ml′μMl,K*′R+PB+(C+DK*)′PD=0.If (14)–(16), (35)–(37), and (39) exist solutions X*(t),  K*(t), P(t), and λ, then Problem 6 is well posed and the optimal feedback control is
(60)u*(t)=-(R+D′PD)-1(B′P+D′PC)x(t).
The optimal value is
(61)V(x0)=infu(·)∈UadJ(x0,u(t))=x0′P(0)x(0)-λ′Lq-tr(μN).The proof of the theorem is the same as that of Theorem16 except for the terminal condition.


## 4. Numerical Examples

LetA=-1, B=1,  C=-2, D=0,  Mq=1,  L=e2, X0=2, Q=0, R=1, and T=1 in Problem 6, according to (35)–(37) in Theorem 16 and the constraint (14)–(16) in Problem 1, and then
(62)-P˙=K*2+2P(K*-1)+4P,P(1)=λ,K*+P=0,X*=2(K*-1)X*+4X*,X(0)=2,X*2(1)=e2.
Solve the system equations, and then
(63)X*(t)=ee-1+e-2e-1e2t,K*(t)=2(2/e-1)e2t-1,P(t)=21-(2/e-1)e2t,λ=2e2-2e+1.
The optimal control for Problem 6 is u*(t)=2e(e-2)e2t/((2-e)(e-1)e2t-e(e-1)), and the optimal index cost value is
(64)J(x0,u*(t))=2e2-4e(e-1)2.


## 5. Conclusion

This paper studied a class of indefinite stochastic LQ control problems with quadratic terminal state constraints and mixed terminal state constraints. By means of the Lagrange multiplier theorem and Riesz representation theorem, this paper presented a necessary condition for indefinite stochastic LQ control problems with quadratic terminal state constraints and mixed terminal state constraints. The result shows that the necessary condition for quadratic terminal constraints is the same as for the linear terminal state constraints that is presented in [8] except for the terminal condition. This coincides with the reality. A sufficient condition also was presented for the existence and uniqueness of the optimal linear feedback control. Numerical example verified the main theorem in this paper.



---

*Source: 674327-2013-09-26.xml*