# Maximum Principle for Delayed Stochastic Linear-Quadratic Control Problem with State Constraint

Authors: Feng Zhang
Journal: Journal of Applied Mathematics
 (2013)

---

## Abstract

This paper is concerned with one kind of delayed stochastic linear-quadratic optimal control problems with state constraints. The control domain is not necessarily convex and the control variable does not enter the diffusion coefficient. Necessary conditions in the form of maximum principle as well as sufficient conditions are established.


---

## Body

## 1. Introduction

In the classical case, many random phenomena are described by stochastic differential equations (SDEs), such as the evolution of the stock prices. However, there also exist many phenomena which are characteristic of past dependence; that is, their present value depends not only on the present situation but also on the past history. Such models may be identified as stochastic differential delay equations (SDDEs). SDDEs have a wide range of applications in physics, biology, engineering, economics, and finance. See [1–4] and the references therein.A stochastic control system whose state function is described by the solution of an SDDE is called a delayed stochastic system. This kind of stochastic control problem appears widely in different research fields; see, for example, [3, 5]. It is worth pointing out that the delayed responses make it more difficult to deal with the system, not only for the infinite dimensional problem, but also for the absence of Itô's formula to deal with the delayed part of the trajectory.One fundamental research direction for stochastic optimal control problems is to establish necessary optimality conditions—Pontryagin maximum principle. By the duality between linear SDEs and backward stochastic differential equations (BSDEs), stochastic maximum principle for forward, backward, and forward-backward systems has been studied by many authors, including Peng [6, 7], Wu [8, 9], Xu [10], and Yong [11]. Recently, Peng and Yang [12] introduced a new type of BSDEs called anticipated BSDEs of the following form:
(1)-dYt=f(t,Yt,Zt,Yt+μ(t),Zt+ν(t))dt-ZtdWt,0≤t≤T,Yt=ξt,Zt=ηt,T≤t≤T+K,
in which the coefficient f contains not only the values of solutions of present but also those of the future. A duality between linear SDDEs and anticipated BSDEs was established in [12], which gave a new way to study the maximum principle for delayed stochastic control problems. Along this line, [13] studied the maximum principle for delayed stochastic optimal control problems in which the control domain is assumed to be convex and both the control variable and its delay part enter the diffusion coefficient. After that, [14] studied the optimal control problem in which the control system is described by a fully coupled anticipated forward-backward stochastic differential delayed equation, and then [15] generalized [13] to the case when the system involves both continuous and impulse controls and the coefficients are random.In practice, sometimes state constraints are inevitably encountered in stochastic optimal control problems; see, for example, [6, 10, 16, 17]. However, little attention was paid to the study of delayed stochastic control problem with state constraints by means of anticipated BSDEs.It is well known that the linear-quadratic (LQ) optimal control problem is an extremely important class of optimal control problems; it can model many problems in applications and many nonlinear control problems can be reasonably approximated by the LQ problems. This paper is concerned with a delayed stochastic LQ optimal control problem, in which the control system evolves by a linear SDDE and the cost functional has a quadratic criterion. We assume that the control domain is not necessarily convex and the control variable does not enter the diffusion coefficient. The coefficients may be random, and the delays enter both the state and the control variables. Besides, the terminal value of the state process is imposed to satisfy the following constraint:𝔼L(X(T))=0. Making use of Ekeland's variational principle and the duality between linear SDDEs and anticipated BSDEs, we establish necessary optimality conditions of the maximum principle type. Sufficient optimality conditions are also presented, which helps find optimal controls.Firstly, this paper involves many derivation details which were omitted in most existing literature. Secondly, whenL≡0, the state constraint disappears and the results in this paper degenerate to the corresponding ones without state constraints. Besides, in [6, 10], the function L(x) is assumed to have linear growth, while L is allowed to have quadratic growth in this paper. Thirdly, we can study unbounded control domain case. However, it is worth pointing out that when we apply Ekeland's variational principle to deal with the case when there are state constraints, we need the continuity of the state process X(·) and the lower semicontinuity of a penalty functional Jρ(v(·)) in the control variable v(·), which is impossible to prove when the control domain is unbounded. To overcome this difficulty, we adopt a convergence technique inspired by Tang and Li [16]. To be precise, we first study the optimal control problem with bounded control domain, and then extend the results to the case with unbounded control domain using a convergence technique. This method was also used in [9].In the classical LQ optimal control problem, a state feedback form of the optimal control can be obtained by virtue of the Riccati equations; for stochastic LQ problems with delays, see [18, 19]. On the one hand, we make use of the maximum principle method in this paper to investigate necessary conditions satisfied by the optimal control, which is different from the method of Riccati equations. Secondly, the study of LQ problems via Riccati equations is mostly carried on under the assumption that the admissible control can take values on the whole space, while we can study bounded control domain case as well as nonconvex control domain case in this paper.The organization of our paper is as follows. In Section2, we give the formulation of the problem. Section 3 is devoted to the study of the maximum principle when the control domain is bounded. In Section 4, we prove the maximum principle as well as the sufficient optimality condition for general control domain case.


## 2. Formulation of the Problem

Let (Ω,ℱ,ℙ) be a probability space and 𝔼 the expectation with respect to ℙ. {Wt,t≥0} is a one-dimensional standard Brownian motion, and {ℱt,t≥0} is its natural filtration augmented with the ℙ-null sets of ℱ. Let us denote by L2(ℱt) the set of real-valued ℱt-measurable random variables ξ’s such that 𝔼|ξ|2<∞. For a<b, we denote by M2(a,b) the set of one-dimensional progressively measurable processes {ϕ(t),a≤t≤b} such that 𝔼∫ab‍|ϕ(t)|2dt<∞, and by S2(a,b) the set of one-dimensional progressively measurable processes {ψ(t),a≤t≤b} such that 𝔼[supa≤t≤b|ψ(t)|2]<∞.In this paper, we only consider one-dimensional case for simplicity, and the results can be extended to multidimensional case without difficulty. Throughout this paper, we useC and C1,C2,… to represent positive constants which can be different from line to line.Assume thatT is a positive constant, and δ1 and δ2 are two nonnegative constants. Let U be a nonempty set in ℝ. We denote by 𝒰 the set of feasible controls, which is the collection of progressively measurable processes v(t):Ω×[-δ2,T]→U satisfying
(2)∥v(·)∥2≜𝔼∫-δ2T‍|v(t)|2dt<∞.
The control system considered in this paper evolves by the following linear SDDE:
(3)dX(t)=b(t,X(t),X(t-δ1),v(t),v(t-δ2))dt+σ(t,X(t),X(t-δ1))dWt,0≤t≤T,X(t)=ξt,-δ1≤t≤0,
where b(t,x,xδ,v,vδ)=At1x+At2xδ+Bt1v+Bt2vδ, σ(t,x,xδ)=Ct1x+Ct2xδ. We assume that ξt:[-δ1,0]→ℝ is continuous. The coefficients (A·i,B·i,C·i), i=1,2, are bounded progressively measurable processes, which are assumed to vanish outside [0,T].Let us mention that the initial pathξ· is independent of the control v(·), since v(·) can affect X(t) only for t≥0. It is easy to check that SDDE (3) admits a unique solution X(·)∈S2(0,T) for any v(·)∈𝒰 (for this, one can see Theorem 2.2 in [13] or Theorem 2.1 in [15]).In addition, we require that the state processX(·) satisfies the following constraint:
(4)𝔼L(X(T))=0,
where L(x):Ω×ℝ→ℝ satisfies that L(x) is ℱT-measurable for all x∈ℝ, 𝔼|L(0)|<∞, and L is continuously differentiable with |Lx(x)|≤C(1+|x|). Under these assumptions, L has a quadratic growth: |L(x)|≤C(1+|L(0)|+|x|2).Ifv(·)∈𝒰 also satisfies the state constraint (4), then v(·) is called an admissible control. The set of admissible controls is denoted by 𝒰ad.The cost functional is given as follows:(5)J(v(·))=𝔼[∫0T‍l(t,X(t),X(t-δ1),v(t),v(t-δ2))dt+Φ(X(T))∫0T‍l],
where l(t,x,xδ,v,vδ)=(1/2)(It1x2+It2xδ2+Mt1v2+Mt2vδ2), Φ(x)=(1/2)Nx2. We assume that N is a nonnegative bounded ℱT-measurable random variable, and the time-varying coefficients (I·i,M·i), i=1,2, are nonnegative bounded progressively measurable processes which are assumed to vanish outside [0,T]. It is easy to see that the functional J is well defined on 𝒰.The objective of the optimal control problem is to minimizeJ(v(·)) over 𝒰ad. An admissible control u*(·)∈𝒰ad is called optimal if it satisfies J(u*(·))=infv(·)∈𝒰adJ(v(·)). We use X*(·) to denote the optimal trajectory.Let us define a metricd on 𝒰 by
(6)d(v(·),u(·))≜𝔼∫-δ2T‍χv(t)≠u(t)dt,
where χ is an indicator function, that is, χE=1, if E holds, and χE=0 otherwise. It is well known that (𝒰,d) is a complete metric space.We will need the following Ekeland's variational principle.Lemma 1.
Let(S,d) be a complete metric space and F:S→ℝ lower semicontinuous and bounded from below. Assume that vε∈S satisfies F(vε)≤infv∈SF(v)+ε for some ε>0. Then for any λ>0, there exists vλ∈S such that F(vλ)≤F(vε), d(vλ,vε)≤λ, and F(vλ)≤F(v)+(ε/λ)d(v,vλ) for any v∈S.


## 3. Maximum Principle in the Case WhenU Is Bounded in ℝ

In this section, we only consider the case when the control domainU is a bounded set in ℝ. Let us denote by Xv(·) the trajectory corresponding to v(·)∈𝒰.The following results will play a crucial role in this section.Lemma 2.
There existsC>0, such that for any v(·), u(·)∈𝒰, it holds that
(7)𝔼[sup0≤t≤T|Xv(t)|2]≤C,(8)|J(v(·))|≤C,(9)𝔼[sup0≤t≤T|Xv(t)-Xu(t)|2]≤Cd(v(·),u(·)),(10)|J(v(·))-J(u(·))|≤Cd(v(·),u(·))1/2.Proof.
Recall that the coefficients are bounded and the control domainU is bounded. Let us first prove (7). By the basic inequality, the Cauchy-Schwartz inequality, and the BDG inequality we have, for 0≤r≤T,
(11)𝔼[sup0≤t≤r|Xv(t)|2]≤C+C|ξ0|2+C𝔼∫0r‍(|Xv(t)|2+|Xv(t-δ1)|2)dt.
Then by a change of variables we get
(12)𝔼[sup0≤t≤r|Xv(t)|2]≤C(1+|ξ0|2+∫-δ10‍|ξt|2dt)+C𝔼∫0r‍sup0≤t≤s|Xv(t)|2ds.
So, (7) can be obtained by the Gronwall inequality. Then result (8) is obvious. Next, let us prove (9). Denote X^(t)=Xv(t)-Xu(t), μ(t)=v(t)-u(t). In the classical way, we have
(13)𝔼[sup0≤t≤r|X^(t)|2]≤C𝔼∫0r‍(|X^(t)|2+|X^(t-δ1)|2)dt+C𝔼∫0r‍(χμ(t)≠0+χμ(t-δ2)≠0)dt.
Using a change of variables gives
(14)𝔼[sup0≤t≤r|X^(t)|2]≤C𝔼∫0r‍sup0≤t≤s|X^(t)|2ds+C𝔼∫-δ2T‍χμ(t)≠0dt.
Then applying the Gronwall inequality leads to (9). Finally we prove (10). Firstly, since
(15)12𝔼∫0T‍|It1[(Xv(t))2-(Xu(t))2]|dt≤C𝔼∫0T‍(|Xv(t)|+|Xu(t)|)|Xv(t)-Xu(t)|dt,
by (7) and (9), applying the Cauchy-Schwartz inequality gives
(16)12𝔼∫0T‍|It1[(Xv(t))2-(Xu(t))2]|dt≤Cd(v(·),u(·))1/2.
Next, since (1/2)|Mt1[(v(t))2-(u(t))2]|≤C1χμ(t)≠0, using the Cauchy-Schwartz inequality gives
(17)12𝔼∫0T‍|Mt1[(v(t))2-(u(t))2]|dt≤C1d(v(·),u(·))≤Cd(v(·),u(·))1/2.
In the same way, we can use a change of variables to get
(18)12𝔼∫0T‍|It2[(Xv(t-δ1))2-(Xu(t-δ1))2]|dt≤Cd(v(·),u(·))1/2,12𝔼∫0T‍|Mt2[(v(t-δ2))2-(u(t-δ2))2]|dt≤Cd(v(·),u(·))1/2.
Thus, (10) can be obtained.Let us define the following:Jρ(v(·))=|𝔼L(Xv(T))|2+|J(v(·))-J(u*(·))+ρ|2 for v(·)∈𝒰, where ρ>0 is small enough. Let us mention that the functional Jρ is defined on the feasible control set 𝒰, rather than just the admissible control set 𝒰ad. In other words, we are able to get rid of the sate constraint by introducing such a penalty functional. It is obvious that Jρ(u*(·))=ρ and Jρ(v(·))>0 for any v(·)∈𝒰. Thus, Jρ(u*(·))≤infv(·)∈𝒰Jρ(v(·))+ρ. The following lemma shows that Jρ(v(·)):𝒰→ℝ is continuous.Lemma 3.
There existsC>0 such that |Jρ(v(·))-Jρ(u(·))|≤Cd(v(·),u(·))1/4 holds for any v(·), u(·)∈𝒰.Proof.
Since(A-B)2≤|A2-B2| for A,B>0, we have
(19)|Jρ(v(·))-Jρ(u(·))|2≤|Jρ2(v(·))-Jρ2(u(·))|≤J^1+J^2,
where
(20)J^1=|[𝔼L(Xv(T))]2-[𝔼L(Xu(T))]2|,J^2=|[J(v(·))-J(u*(·))+ρ]2-[J(u(·))-J(u*(·))+ρ]2|.
We first consider J^1. On the one hand, by the growth condition of L, we can use (7) to get |𝔼[L(Xv(T))+L(Xu(T))]|≤C. On the other hand, since
(21)|𝔼[L(Xv(T))-L(Xu(T))]|≤C𝔼[(1+|Xv(T)|+|Xu(T)|)|Xv(T)-Xu(T)|],
by (7) and (9), we can use the Cauchy-Schwartz inequality to get
(22)|𝔼[L(Xv(T))-L(Xu(T))]|≤Cd(v(·),u(·))1/2.
Thus,
(23)J^1=|𝔼[L(Xv(T))+L(Xu(T))]|×|𝔼[L(Xv(T))-L(Xu(T))]|≤Cd(v(·),u(·))1/2.
Next, from (8) it follows that |J(v(·))+J(u(·))-2J(u*(·))+2ρ|≤C, so by (10) we have
(24)J^2=|J(v(·))+J(u(·))-2J(u*(·))+2ρ|×|J(v(·))-J(u(·))|≤Cd(v(·),u(·))1/2.
Thus |Jρ2(v(·))-Jρ2(u(·))|≤Cd(v(·),u(·))1/2. So |Jρ(v(·))-Jρ(u(·))|≤Cd(v(·),u(·))1/4.Now applying Lemma1 leads to the existence of vρ(·)∈𝒰 such that
(25)Jρ(vρ(·))≤Jρ(u*(·))=ρ,(26)d(vρ(·),u*(·))≤ρ,(27)Jρ(vρ(·))≤Jρ(v(·))+ρd(v(·),vρ(·)),∀v(·)∈𝒰.
In what follows, let us first derive the necessary conditions for vρ(·) and then take ρ↓0 to get proper conditions for u*(·).For anyτ∈[0,T) and v(·)∈𝒰, let us define
(28)vρϵ(t)={v(t),ifτ≤t≤τ+ϵ,vρ(t),otherwise,
where ϵ>0 is small enough such that we can always assume that τ+ϵ≤T. It is obvious that vρϵ(·)∈𝒰. Let us point out that we cannot get vρϵ(·)∈𝒰ad even if vρ(·)∈𝒰ad. This also shows why the functional Jρ is defined on 𝒰 rather than on𝒰ad. It is easy to see that
(29)d(vρ(·),vρϵ(·))≤ϵ.
Then, by (27),
(30)Jρ(vρϵ(·))-Jρ(vρ(·))≥-ρϵ.LetXρ(·), Xρϵ(·) be the trajectories corresponding to vρ(·), vρϵ(·), respectively. We introduce the following variational equation:
(31)dXρ1(t)=[At1Xρ1(t)+At2Xρ1(t-δ1)+Bt1(vρϵ(t)-vρ(t))+Bt2(vρϵ(t-δ2)-vρ(t-δ2))]dt+[Ct1Xρ1(t)+Ct2Xρ1(t-δ1)]dWt,0≤t≤T,Xρ1(t)=0,-δ1≤t≤0.
It is easy to check that this equation admits a unique solution Xρ1(·)∈S2(0,T). Moreover, we have the following.Lemma 4.
There existsC>0, which is independent of ϵ, such that
(32)𝔼[sup0≤t≤T|Xρ1(t)|2]≤Cϵ2.Proof.
By the basic inequality, the Cauchy-Schwartz inequality, and the BDG inequality, we can use a change of variables to get, for0≤r≤T,
(33)𝔼[sup0≤t≤r|Xρ1(t)|2]≤C∫0r‍𝔼[sup0≤t≤s|Xρ1(t)|2]ds+C𝔼(∫-δ2T‍χvρϵ(t)≠vρ(t)dt)2.
By the definition of vρϵ(·), we have
(34)∫-δ2T‍χvρϵ(t)≠vρ(t)dt=∫ττ+ϵ‍χvρϵ(t)≠vρ(t)dt.
Then by (29), applying the Cauchy-Schwartz inequality gives
(35)𝔼(∫-δ2T‍χvρϵ(t)≠vρ(t)dt)2≤ϵ𝔼∫ττ+ϵ‍χvρϵ(t)≠vρ(t)dt≤ϵd(vρ(·),vρϵ(·))≤ϵ2.
Finally, the result can be derived by the Gronwall inequality applied to (33).Let us denoteX~(·)=Xρϵ(·)-Xρ(·)-Xρ1(·). Then it's easy to check that X~(·) satisfies
(36)dX~(t)=[At1X~(t)+At2X~(t-δ1)]dt+[Ct1X~(t)+Ct2X~(t-δ1)]dWt,0≤t≤T,X~(t)=0,-δ1≤t≤0.
By the existence and uniqueness of the solution for this equation, we have X~(t)=0, a.s., a.e. That is, for a.e. t∈[0,T], ℙ—a.s.,
(37)Xρϵ(t)-Xρ(t)-Xρ1(t)=0.Lemma 5.
We have(38)Jρ2(vρϵ(·))-Jρ2(vρ(·))=2𝔼L(Xρ(T))×𝔼[Lx(Xρ(T))Xρ1(T)]+2Γ[J(vρ(·))-J(u*(·))+ρ]+o(ϵ),
where Γ is defined by
(39)Γ=𝔼{Mt2((vρϵ(t-δ2))2-(vρ(t-δ2))2)12Mt2((vρϵ(t-δ2))2-(vρ(t-δ2))2)NXρ(T)Xρ1(T)+∫0T‍[Mt2((vρϵ(t-δ2))2-(vρ(t-δ2))2)12Mt2((vρϵ(t-δ2))2-(vρ(t-δ2))2)It1Xρ(t)Xρ1(t)+It2Xρ(t-δ1)Xρ1(t-δ1)+12Mt1((vρϵ(t))2-(vρ(t))2)+12Mt2((vρϵ(t-δ2))2-(vρ(t-δ2))2)]dt}.Proof.
It is obvious thatJρ2(vρϵ(·))-Jρ2(vρ(·))=J-1+J-2, where
(40)J-1=[𝔼L(Xρϵ(T))]2-[𝔼L(Xρ(T))]2,J-2=[J(vρϵ(·))-J(u*(·))+ρ]2-[J(vρ(·))-J(u*(·))+ρ]2.
Firstly, J-1-2𝔼L(Xρ(T))×𝔼[Lx(Xρ(T))Xρ1(T)]=J-11+J-12, where
(41)J-11=(𝔼[L(Xρϵ(T))-L(Xρ(T))])2,J-12=2𝔼L(Xρ(T))×𝔼[L(Xρϵ(T))-L(Xρ(T))-Lx(Xρ(T))Xρ1(T)].
On the one hand, since
(42)|𝔼[L(Xρϵ(T))-L(Xρ(T))]|≤C𝔼[(1+|Xρϵ(T)|+|Xρ(T)|)|Xρϵ(T)-Xρ(T)|],
by (7), (32), and (37), we can use the Cauchy-Schwartz inequality to get J-11=o(ϵ). On the other hand, we can use the Cauchy-Schwartz inequality and the dominated convergence theorem to derive J-12=o(ϵ). Thus
(43)J-1=2𝔼L(Xρ(T))×𝔼[Lx(Xρ(T))Xρ1(T)]+o(ϵ).
Next we consider J-2. On the one hand, by (32) and (37) it’s easy to check that
(44)J(vρϵ(·))-J(vρ(·))=Γ+Cϵ2.
On the other hand, by (29) and (32) we can use the Cauchy-Schwartz inequality to derive |Γ|≤Cϵ, and thus
(45)|J(vρϵ(·))-J(vρ(·))|=|Γ+Cϵ2|≤Cϵ.
So, from the fact that |J(vρ(·))-J(u*(·))+ρ|≤C and
(46)J-2-2Γ[J(vρ(·))-J(u*(·))+ρ]=[J(vρϵ(·))-J(vρ(·))]2+2[J(vρϵ(·))-J(vρ(·))-Γ]×[J(vρ(·))-J(u*(·))+ρ],
we have |J-2-2Γ[J(vρ(·))-J(u*(·))+ρ]|≤Cϵ2, and thus
(47)J-2=2Γ[J(vρ(·))-J(u*(·))+ρ]+o(ϵ).
The proof is complete.Let us introduce the following Hamiltonian:(48)H(t,x,xδ,q,k,v,vδ,α,γ)=b(t,x,xδ,v,vδ)q+σ(t,x,xδ)k+γl(t,x,xδ,v,vδ).
The following is the maximum principle for the delayed stochastic LQ control problem with bounded control domain.Theorem 6.
Assume thatU is a bounded set in ℝ. Then for the optimal control u*(·), there exist (α*,γ*)∈ℝ2 satisfying
(49)|α*|2+|γ*|2=1,
and the solution (Q*(·),K*(·))∈S2(0,T)×M2(0,T) of the following adjoint equation:
(50)dQ*(t)=-{At1Q*(t)+Ct1K*(t)+γ*It1X*(t)+𝔼ℱt[At+δ12Q*(t+δ1)+Ct+δ12K*(t+δ1)+γ*It+δ12X*(t)]}dt+K*(t)dWt,0≤t≤T,Q*(T)=α*Lx(X*(T))+γ*NX*(T),Q*(t)=0,K*(t)=0,T<t≤T+δ1,
such that
(51)ℋ(t,v)≥ℋ(t,u*(t)),∀v∈U,a.e.t∈[0,T],ℙ—a.s.,
where ℋ is defined by
(52)ℋ(t,v)=H(Θ*(t),v,u*(t-δ2),α*,γ*)+𝔼ℱt[H(Θ*(t+δ2),u*(t+δ2),v,α*,γ*)]
with Θ*(t)=(t,X*(t),X*(t-δ1),Q*(t),K*(t)).Proof.
From (29) it follows that d(vρϵ(·),vρ(·))→0 as ϵ↓0. So by Lemma 3 we have Jρ(vρϵ(·))→Jρ(vρ(·))>0 as ϵ↓0. By Lemma 5 and (30), we have
(53)αρϵ𝔼[Lx(Xρ(T))Xρ1(T)]+γρϵΓ+ρϵ+o(ϵ)≥0,
where
(54)αρϵ=2𝔼L(Xρ(T))Jρ(vρϵ(·))+Jρ(vρ(·)),γρϵ=2[J(vρ(·))-J(u*(·))+ρ]Jρ(vρϵ(·))+Jρ(vρ(·)).
Besides, it's easy to check that limϵ↓0(|αρϵ|2+|γρϵ|2)=1. Therefore, there exists a subsequence, still denoted by (αρϵ,γρϵ), such that
(55)limϵ↓0(αρϵ,γρϵ)=(αρ,γρ),
for some (αρ,γρ), with
(56)|αρ|2+|γρ|2=1.
Let us introduce the following equation:(57)dQρϵ(t)=-{At1Qρϵ(t)+Ct1Kρϵ(t)+γρϵIt1Xρ(t)+𝔼ℱt[At+δ12Qρϵ(t+δ1)+Ct+δ12Kρϵ(t+δ1)+γρϵIt+δ12Xρ(t)]}dt+Kρϵ(t)dWt,0≤t≤T,Qρϵ(T)=αρϵLx(Xρ(T))+γρϵNXρ(T),Qρϵ(t)=0,Kρϵ(t)=0,T<t≤T+δ1.
It is easy to check that this equation admits a unique solution (Qρϵ(·),Kρϵ(·)) which belongs to S2(0,T)×M2(0,T). Applying Itô's formula to Xρ1(t)Qρϵ(t) and then taking expectations, we can use a change of variables to get
(58)αρϵ𝔼[Lx(Xρ(T))Xρ1(T)]+γρϵΓ=𝔼∫0T‍Qρϵ(t)[Bt1(vρϵ(t)-vρ(t))+Bt2(vρϵ(t-δ2)-vρ(t-δ2))]dt+12γρϵ𝔼∫0T‍{Mt1[(vρϵ(t))2-(vρ(t))2]+Mt2[(vρϵ(t-δ2))2-(vρ(t-δ2))2]}dt.
Then, by (53) and the definition of vρϵ(·) we have
(59)𝔼∫ττ+ϵ‍{[(v(t))2-(vρ(t))2]12γρϵ𝔼ℱt[Mt+δ22][(v(t))2-(vρ(t))2]Qρϵ(t)Bt1(v(t)-vρ(t))+12Mt1γρϵ[(v(t))2-(vρ(t))2]+𝔼ℱt[Qρϵ(t+δ2)Bt+δ22](v(t)-vρ(t))+12γρϵ𝔼ℱt[Mt+δ22][(v(t))2-(vρ(t))2]}dt+ρϵ+o(ϵ)≥0.
Let (Qρ(·),Kρ(·))∈S2(0,T)×M2(0,T) be the solution of
(60)dQρ(t)=-{+γρIt+δ12Xρ(t)]At1Qρ(t)+Ct1Kρ(t)+γρIt1Xρ(t)+𝔼ℱt[At+δ12Qρ(t+δ1)+Ct+δ12Kρ(t+δ1)+γρIt+δ12Xρ(t)]}dt+Kρ(t)dWt,0≤t≤T,Qρ(T)=αρLx(Xρ(T))+γρNXρ(T),Qρ(t)=0,Kρ(t)=0,T<t≤T+δ1.
Then, by subdividing the time interval [0,T], we can use (55) to derive
(61)𝔼[sup0≤t≤T|Qρϵ(t)-Qρ(t)|2+∫0T‍|Kρϵ(t)-Kρ(t)|2dt]⟶0,
as ϵ↓0. Consequently, considering the arbitrariness of τ∈[0,T), dividing (59) by ϵ, and then taking ϵ↓0 lead to
(62)𝔼{Qρ(t)Bt1(v(t)-vρ(t))+12Mt1γρ[(v(t))2-(vρ(t))2]+𝔼ℱt[Qρ(t+δ2)Bt+δ22](v(t)-vρ(t))+12γρ𝔼ℱt[Mt+δ22][(v(t))2-(vρ(t))2]}dt+ρ≥0.
Now let us takeρ↓0. On the one hand, by (56), there exists a subsequence of (αρ,γρ), which converges to (α*,γ*), and (49) holds. On the other hand, from (26) it follows that d(vρ(·),u*(·))→0 as ρ↓0, so we can use (9) to obtain
(63)𝔼[sup0≤t≤T|Xρ(t)-X*(t)|2]⟶0,
as ρ↓0. Consequently, we can check that
(64)𝔼[sup0≤t≤T|Qρ(t)-Q*(t)|2+∫0T‍|Kρ(t)-K*(t)|2dt]⟶0,
as ρ↓0, where (Q*(·),K*(·)) is the solution of the adjoint equation (50). Let us assume without loss of generality that Q*(t)=K*(t)=0 for t<0. Consequently, we can take ρ↓0 in (62) to get
(65)𝔼{Q*(t)Bt1(v(t)-u*(t))+12Mt1γ*[(v(t))2-(u*(t))2]+𝔼ℱt[Q*(t+δ2)Bt+δ22](v(t)-u*(t))+12γ*𝔼ℱt[Mt+δ22][(v(t))2-(u*(t))2]}≥0.
In order to obtain (65), we only need to prove that the terms on the left-hand side of (62) converge to the corresponding ones in (65) along a subsequence. For this, we first prove
(66)𝔼∫0T‍|Qρ(t)Bt1(v(t)-vρ(t))-Q*(t)Bt1(v(t)-u*(t))|dt⟶0,
as ρ↓0. In fact, since
(67)|Qρ(t)Bt1(v(t)-vρ(t))-Q*(t)Bt1(v(t)-u*(t))|=|(Qρ(t)-Q*(t))Bt1(v(t)-vρ(t))+Q*(t)Bt1(u*(t)-vρ(t))|≤C|Qρ(t)-Q*(t)|+C|Q*(t)|χvρ(t)≠u*(t),
by the Cauchy-Schwartz inequality we have
(68)𝔼∫0T‍|Qρ(t)Bt1(v(t)-vρ(t))-Q*(t)Bt1(v(t)-u*(t))|dt≤C𝔼∫0T‍|Qρ(t)-Q*(t)|dt+C𝔼∫0T‍|Q*(t)|2dtd(vρ(·),u*(·))⟶0,
as ρ↓0. With the same method and by a change of variables we can also prove
(69)𝔼∫0T‍|𝔼ℱt[Qρ(t+δ2)Bt+δ22](v(t)-vρ(t))-𝔼ℱt[Q*(t+δ2)Bt+δ22](v(t)-u*(t))|dt⟶0.
Next, since
(70)12|Mt1γρ[(v(t))2-(vρ(t))2]-Mt1γ*[(v(t))2-(u*(t))2]|=12|Mt1(γρ-γ*)[(v(t))2-(vρ(t))2]+Mt1γ*[(u*(t))2-(vρ(t))2]|≤C|γρ-γ*|+Cχvρ(t)≠u*(t),
we have
(71)12𝔼∫0T‍|Mt1γρ[(v(t))2-(vρ(t))2]-Mt1γ*[(v(t))2-(u*(t))2]Mt1γρ[(v(t))2-(vρ(t))2]|dt≤C|γρ-γ*|+Cd(vρ(·),u*(·))⟶0.
In a similar way, we can use a change of variables to get
(72)12𝔼∫0T‍|γρ𝔼ℱt[Mt+δ22][(v(t))2-(vρ(t))2]-γ*𝔼ℱt[Mt+δ22][(v(t))2-(u*(t))2]γρ𝔼ℱt[Mt+δ22][(v(t))2-(vρ(t))2]|dt⟶0.
Thus, we can derive (65).
Let us recall thatv(·)∈𝒰 is arbitrarily chosen, so result (65) holds for all v(·)∈𝒰. Next, we drop the expectation in (65). For any v∈U and E∈ℱt, let us define v(t)=vχE+u*(t)χE-. It is obvious that the defined v(·) is an element of 𝒰. Applying this v(·) in (65) gives
(73)𝔼{Q*(t)Bt1(v-u*(t))χE+12Mt1γ*[v2-(u*(t))2]χE+𝔼ℱt[Q*(t+δ2)Bt+δ22](v-u*(t))χE+12γ*𝔼ℱt[Mt+δ22][v2-(u*(t))2]χE}≥0.
Since E∈ℱt is arbitrarily chosen, it implies
(74)𝔼ℱt{Q*(t)Bt1(v-u*(t))+12Mt1γ*[v2-(u*(t))2]+𝔼ℱt[Q*(t+δ2)Bt+δ22](v-u*(t))+12γ*𝔼ℱt[Mt+δ22][v2-(u*(t))2]}≥0,
which leads to
(75){Q*(t)Bt1+𝔼ℱt[Q*(t+δ2)Bt+δ22]}(v-u*(t))+12γ*{Mt1+𝔼ℱt[Mt+δ22]}[v2-(u*(t))2]≥0,
which is just the conclusion of (51).


## 4. Maximum Principle for General Control DomainU

In this section, we study the maximum principle in the case whenU can be unbounded in ℝ. This case can be treated via the bounded case in Section 3 with a convergence technique.Let us defineUi={v∈U||v|≤∥u*(·)∥+i}, i=1,2,…. Then Ui is a bounded set in ℝ for fixed i. Besides,
(76)Ui⊂Ui+1,i=1,2,…,U=⋃i=1∞‍Ui.
We denote by 𝒰i the set of progressively measurable processes v(t):Ω×[-δ2,T]→Ui satisfying ∥v(·)∥<∞ and by 𝒰adi the collection of v(·)∈𝒰i satisfying the state constraint (4). Then from (76) it follows that
(77)𝒰i⊂𝒰i+1,i=1,2,…,𝒰=⋃i=1∞‍𝒰i;𝒰adi⊂𝒰adi+1,i=1,2,…,𝒰ad=⋃i=1∞‍𝒰adi.Sinceu*(·)∈𝒰ad, by (77) there exists i1 such that u*(·)∈𝒰adi for i>i1. Thus, u*(·) is still optimal when the original admissible control set 𝒰ad is replaced by 𝒰adi for i>i1. So, by Theorem 6, for i>i1, there exist (αi,γi)∈ℝ2, satisfying
(78)|αi|2+|γi|2=1,
and the solution (Qi(·),Ki(·))∈S2(0,T)×M2(0,T) of the following adjoint equation:
(79)dQi(t)=-{At1Qi(t)+Ct1Ki(t)+γiIt1X*(t)+𝔼ℱt[At+δ12Qi(t+δ1)+Ct+δ12Ki(t+δ1)+γiIt+δ12X*(t)]}dt+Ki(t)dWt,0≤t≤T,Qi(T)=αiLx(X*(T))+γiNX*(T),Qi(t)=0,Ki(t)=0,T<t≤T+δ1,
such that
(80){Qi(t)Bt1+𝔼ℱt[Qi(t+δ2)Bt+δ22]}(v-u*(t))+12γi{Mt1+𝔼ℱt[Mt+δ22]}[v2-(u*(t))2]≥0,∀v∈Ui,a.e.t∈[0,T],ℙ—a.s.
By (78), there exists a subsequence of (αi,γi), still denoted by (αi,γi), such that
(81)limi→∞(αi,γi)=(α,γ),
for some (α,γ), with
(82)|α|2+|γ|2=1.
Then, by (81) it's easy to check that
(83)𝔼[sup0≤t≤T|Qi(t)-Q(t)|2+∫0T‍|Ki(t)-K(t)|2dt]⟶0,
as i→∞, where (Q(·),K(·))∈S2(0,T)×M2(0,T) is the solution of
(84)dQ(t)=-{At1Q(t)+Ct1K(t)+γIt1X*(t)+𝔼ℱt[At+δ12Q(t+δ1)+Ct+δ12K(t+δ1)+γIt+δ12X*(t)]}dt+K(t)dWt,0≤t≤T,Q(T)=αLx(X*(T))+γNX*(T),Q(t)=0,K(t)=0,T<t≤T+δ1.
Let us assume without loss of generality that Q(t)=K(t)=0 for t<0.For any fixedv∈U, from (76) it follows that there exists i2 such that v∈Ui for i>i2, and consequently we see from (80) that
(85){Qi(t)Bt1+𝔼ℱt[Qi(t+δ2)Bt+δ22]}(v-u*(t))+12γi{Mt1+𝔼ℱt[Mt+δ22]}[v2-(u*(t))2]≥0,∀i>i1+i2,a.e.t∈[0,T],ℙ—a.s.
Then, similar to the proof of (65), by (81) and (83), taking i→∞ along a subsequence in (85) leads to
(86){Q(t)Bt1+𝔼ℱt[Q(t+δ2)Bt+δ22]}(v-u*(t))+12γ{Mt1+𝔼ℱt[Mt+δ22]}[v2-(u*(t))2]≥0,a.e.t∈[0,T],ℙ—a.s.
Note that the above inequality holds for all v∈U, and therefore we have the main result of this section.Theorem 7.
For the optimal controlu*(·), there exist (α,γ)∈ℝ2 satisfying (82) and the solution (Q(·),K(·))∈S2(0,T)×M2(0,T) of the adjoint equation (84) such that
(87)ℍ(t,v)≥ℍ(t,u*(t)),∀v∈U,a.e.t∈[0,T],ℙ—a.s.,
where ℍ is defined by
(88)ℍ(t,v)=H(Θ(t),v,u*(t-δ2),α,γ)+𝔼ℱt[H(Θ(t+δ2),u*(t+δ2),v,α,γ)]
with Θ(t)=(t,X*(t),X*(t-δ1),Q(t),K(t)).In what follows, let us investigate under what condition an admissible control turns out to be optimal. To this end, let us assume thatv(t)=η(t) for all v(·)∈𝒰ad and t∈[-δ2,0], where η(t):[-δ2,0]→U is a given function satisfying sup-δ1≤t≤0|η(t)|2<∞.Let us assume that(H)α≥0 and L is convex or α≤0 and L is concave.Theorem 8.
Assume (H). Assume thatu*(·)∈𝒰ad is an admissible control and X*(·) is the corresponding trajectory. Let α∈ℝ, γ>0 satisfy (82), and (Q(·),K(·))∈S2(0,T)×M2(0,T) satisfy (84). Then u*(·) is an optimal control if it satisfies (87).Proof.
Let us denoteX-(·)=Xv(·)-X*(·) for v(·)∈𝒰ad. Applying Itô's formula to X-(t)Q(t) for 0≤t≤T and then using a change of variables lead to
(89)α𝔼[Lx(X*(T))X-(T)]+γ𝔼[NX*(T)X-(T)]+γ𝔼∫0T‍[It1X*(t)X-(t)+It2X*(t-δ1)X-(t-δ1)]dt+12γ𝔼∫0T‍{Mt1[(v(t))2-(u*(t))2]+Mt2[(v(t-δ2))2-(u*(t-δ2))2]}dt=𝔼∫0T‍[ℍ(t,v(t))-ℍ(t,u*(t))]dt.
On the one hand, from (87) it follows that
(90)𝔼∫0T‍[ℍ(t,v(t))-ℍ(t,u*(t))]dt≥0.
On the other hand, since N, It1, and It2 are nonnegative, by the property of convex functions we have
(91)12𝔼[N(Xv(T))2-N(X*(T))2]≥𝔼[NX*(T)X-(T)],12𝔼∫0T‍It1[(Xv(t))2-(X*(t))2]dt≥𝔼∫0T‍It1X*(t)X-(t)dt,12𝔼∫0T‍It2[(Xv(t-δ1))2-(X*(t-δ1))2]dt≥𝔼∫0T‍It2X*(t-δ1)X-(t-δ1)dt.
Thus, it follows that α𝔼[Lx(X*(T))X-(T)]+γ[J(v(·))-J(u*(·))]≥0, so
(92)J(v(·))-J(u*(·))≥-αγ𝔼[Lx(X*(T))X-(T)].
Let us recall that 𝔼[L(Xv(T))-L(X*(T))]=0 holds for v(·)∈𝒰ad. Then, by (H), (92) leads to
(93)-αγ𝔼[Lx(X*(T))X-(T)]≥-αγ𝔼[L(Xv(T))-L(X*(T))]=0,
which gives J(v(·))-J(u*(·))≥0.Remark 9.
WhenL≡0, namely, there is no state constraint for X(·), we have α*=α=0. In this case, Theorems 6 and 7 degenerate to the maximum principle for stochastic LQ problem with delays and without state constraints. When L(x) is a linear function, the assumption (H) holds for any α∈ℝ.Example 10.
Let us takeU=(-∞,-1]∪[1,+∞) and L(x)=((K-x)+)2 with a fixed constant K. In this case, L(x) is a convex function and the state constraint (4) implies that X(T)≥K a.s. Assume that α>0, γ>0, and Mt1+𝔼ℱt[Mt+δ22]>0. Then we can solve the inequality (87) to obtain
(94)u*(t)={ν(t),if|ν(t)|≥1,-1,if-1<ν(t)≤0,1,if0<ν(t)≤1,
where ν(t) is defined by
(95)ν(t)=-Q(t)Bt1+𝔼ℱt[Q(t+δ2)Bt+δ22]γ{Mt1+𝔼ℱt[Mt+δ22]}.
By Theorem 8, this u*(·) is indeed an optimal control if it satisfies the state constraint (4).



---

*Source: 964765-2013-12-19.xml*