# Linear Feedback of Mean-Field Stochastic Linear Quadratic Optimal Control Problems on Time Scales

Authors: Yingjun Zhu; Guangyan Jia
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

This paper addresses a version of the linear quadratic control problem for mean-field stochastic differential equations with deterministic coefficients on time scales, which includes the discrete time and continuous time as special cases. Two coupled Riccati equations on time scales are given and the optimal control can be expressed as a linear state feedback. Furthermore, we give a numerical example.


---

## Body

## 1. Introduction

The linear quadratic control problem is one of the most important issues for optimal control problem. The study of the mean-field linear quadratic optimal control problem also has received much attention [1, 2], and it has a wide range of applications in engineering and finance [3, 4]. Until now, the mean-field linear quadratic control problem is well understood both from the continuous and discrete points of view. In this paper, the mean-field linear quadratic control problem is studied in the version of time scales.Time scales were first introduced by Hilger [5] in 1988 in order to unite differential and difference equations into a general framework. Recently, time scales theory is extensively studied in many works [6–14]. It is well known that the optimal control problems on time scales are an important field for both theory and applications. Since the calculus of variations on time scales was studied by Bohner [15], results on related topics and their applications has become more and more. The existence of optimal control for the dynamic systems on time scales was discussed [16–18]. Subsequently, maximum principles were studied in several work [19, 20], which specify the necessary conditions for optimality. In addition, Bellman dynamic programming on time scales for the deterministic optimal control problems was considered in [21]. At the same time, some results were obtained for the linear quadratic control problems for deterministic linear system on time scales in [22, 23]. In [24], the authors developed the linear quadratic control problems for stochastic linear system on time scales. To our best knowledge, the optimal control problems for the mean-field system on time scales have not been established.We are interested in the mean-field stochastic linear quadratic control problem on time scales (MF-SΔLQ for short). To deal with the well posedness of the state equation on time scales, we use the similar iteration method as [25]. Very similar to continuous and discrete cases, we can also get the associated Riccati equations (see [26, 27], for continuous and discrete cases) on time scales, and the optimal control can be expressed as a linear state feedback through the solutions of the coupled Riccati equations.The organization of this paper is as follows. In Section2, we show some preliminaries about time scales’ theory and MF-SΔLQ problem. We study the well posedness of the state equation on time scales and show the feedback representation of the optimal control by the associated Riccati equations on time scales in Section 3. Finally, an example is presented.


## 2. Preliminaries

A time scalesT is a nonempty closed subset of real numbers set ℝ and we denote 0,TT=0,T∩T. In this paper, we always suppose T is bounded. The forward jump operator σ and backward jump operator ρ are, respectively, defined by(1)σt=infs∈T:s>t,(2)ρt=sups∈T:s<t,(supplemented byinf∅≔supT and sup∅≔infT, where ∅ denotes the empty set). If σt=t (σt>t, ρt=t, and ρt=t), the point t is called right-dense (right-scattered, left-dense, and left-scattered). Moreover, a point is called isolated if it is both left-scattered and right-scattered. For a function f, we denote fσ=f⋅σ and fρ=f⋅ρ. The definition of the graininess function μ is as follows:(3)μt=σt−t.We now present some basic concepts and properties about time scales (see [10, 11]).Definition 1.
Letf be a function on time scales T, and f is called right-dense continuous function if f is continuous at every right-dense point and has finite left-sided limits at every left-dense point. Similarly, f is called left-dense continuous function if f is continuous at every left-dense point and has finite right-sided limits at every right-dense point. If f is right-dense continuous and also is left-dense continuous, then f is called continuous function.Remark 1.
If a functionf is right-dense continuous, then f has an antiderivative F.
Define the setTκ as follows:(4)Tκ=T\ρsupT,supT,ifsupT<∞,T,ifsupT=∞.Definition 2.
Letf: T⟶ℝ be a function and t∈Tκ, and if for all ε>0, there exist a neighborhood U of t such that(5)fσt−fs−fΔtσt−s≤εσt−s,for alls∈U.
We callfΔt the Δ derivative of f at t.Remark 2.
If the functionsf and g are differentiable at t, then the product fg is also differentiable at t and the product rule is given by(6)fgΔt=fΔtgt+fσtgΔt.
In this paper, we adopt the stochastic integral defined by Bohner et al. [25]. Let Ω,ℱ,ℱtt∈0,TT,P be a complete probability space with an increasing and continuous filtration ℱtt∈0,TT. We define that Lℱ20,TT;ℝ is the set of all ℱt-adapted, ℝ-valued measurable process Xt such that E∫0TXt2Δt<∞.
A Brownian motion indexed by time scalesT is defined by Grow and Sanyal [13]. Although the Brownian motion on time scales is very similar to that on continuous time, but there are also some differences between them. For example, the quadratic variation of a Brownian motion on time scales (see [14]) is an increasing process yet, but it is not deterministic. In fact, Wt=λ0,tT+∑s≤tWsσ−Ws2, where λ is the Lebesgue measure.
Next, we give the definition of the stochasticΔ integral and its properties.Definition 3 (see [25]).
The random processXt is stochastic Δ integrable on 0,TT if the corresponding X˜t is integrable, and define the integral value of Xt as(7)∫0TXtΔWt:=∫0TX˜tdWt,where(8)X˜t=Xsup0,tT,for allt∈0,T,and the Brownian motion on the right side of (7) is indexed by continuous time.
We also have the following properties.
LetXt,Yt∈Lℱ20,TT;ℝ and α,β∈ℝ. Then,(9)i∫0TαXt+βYtΔWt=α∫0TXtΔWt+β∫0TYtΔWt,iiE∫0TXtΔWt=0,iiiE∫0TXtΔWt2=E∫0TXt2ΔWt=E∫0TXt2Δt,where the integral X with respect to the quadratic variation of Brownian motion Wt is defined by Stieltjes integral as ∫0TXtωΔWtω.Notation 1.
The following notation will be used:M′: the transpose of any vector or matrix MM: = ∑i,jmij2 for any matrix or vector M=mijM≥0: M is a positive semidefinite matrixSn: the space of all n×n symmetric matricesX,Yt: the quadratic covariation process of X and YL∞0,TT;ℝ: the space of bounded, Δ-Lebesgue integrable, and ℝ-valued functions on 0,TTC10,TT;ℝ: the family of all ℝ-valued continuous functions ft defined on 0,TT such that they are Δ differentiable in t
Finally, we introduce our MF-SΔLQ problem. Consider the following stochastic Δ-differential equation:(10)ΔXt=AtXt+A¯tEXt+Btut+B¯tEutΔt+Dtut+D¯tEutΔW,X0=x,where the coefficients A⋅,A¯⋅,B⋅,B¯⋅,D⋅,andD¯⋅ are all deterministic matrix-valued functions and u⋅∈U0,TT=Lℱ20,TT;ℝm. The cost functional is(11)Jx;u⋅=E∫0TX′tQtXt+EXt′Q¯tEXt+u′tRtut+Eut′R¯tEutΔt+X′TGXT+EXT′G¯EXT,where GandG¯ are symmetric matrices and Q⋅,Q¯⋅,R⋅,andR¯⋅ are given deterministic matrix-valued functions.Problem (MF-SΔLQ).
For any given initial statex∈ℝn, find a u∗⋅∈U0,TT such that(12)Jx;u∗⋅=infu⋅∈U0,TTJx;u⋅.
u∗⋅ is called an optimal control of the MF-SΔLQ problems and the corresponding X⋅;x,u∗⋅ is called an optimal state process.


## 3. Main Results

First, we introduce the following assumptions which are necessary for the proofs of our main results.(H1) Assume that(13)A⋅,A¯⋅∈L∞0,TT;ℝn×n,B⋅,B¯⋅,D⋅,D¯⋅∈L∞0,TT;ℝn×m.(H2) Assume that(14)Q⋅,Q¯⋅∈L∞0,TT;Sn,R⋅,R¯⋅∈L∞0,TT;Sm,G,G¯∈Sn,and for some δ>0,(15)Qs≥0,Qs+Q¯s≥0,Rs≥δI,Rs+R¯s≥δI,s∈0,TT,G≥0,G+G¯≥0.Remark 3.
Assumption (H1) can guarantee the existence and uniqueness of the solution of the mean-field stochastic linear system (10). Under Assumptions (H1) and (H2), we can establish two coupled Riccati equations to show the feedback control.
Now, we show the well posedness of the state equation (10) by the iteration method, which is very similar to the way as in [25].Theorem 1.
LetΩ,ℱ,ℱtt∈0,TT,P be given and W be a standard ℱtt∈0,TT Brownian motion. Suppose that (H1) holds, then system (10) has a unique solution X∈Lℱ20,TT;ℝn for any x,u⋅∈ℝn×U0,TT.Proof.
For the existence, we adopt the iteration method and define(16)X0t=x,(17)Xnt=x+∫0tAsXn−1s+A¯sEXn−1s+Bsus+B¯sEusΔs+∫0tDsus+D¯sEusΔWs,n∈ℕ,t∈0,TT.
Letδnt=EXn+1t−Xnt2, and we claim that(18)δnt≤Mn+1hn+1t,0,n∈ℕ,t∈0,TT,where M is a generic constant and hn is the generalized monomials defined in [28]. When n=0, we obtain(19)δ0t=EX1t−X0t2=E∫0tAsX0t+A¯sEX0t+Bsus+B¯sEusΔs+∫0tDsus+D¯sEusΔWs2≤2E∫0tAsX0s+A¯sEX0s+Bsus+B¯sEusΔs2+2E∫0tDsus+D¯sEus2Δs≤tM=Mh1t,0.
Suppose inequality (14) holds for n−1, then(20)δnt=EXn+1t−Xnt2=E∫0tAsXns−Xn−1s+A¯tEXns−Xn−1sΔs2=2E∫0tAsXns−Xn−1sΔs2+2E∫0tA¯tEXns−Xn−1sΔs2≤M∫0tEXns−Xn−1s2Δs=M∫0tδn−1sΔs≤M∫0tMnhns,0Δs=Mn+1hn+1t,0,n∈ℕ,t∈0,TT.
This proves the claim.
Similarly, we have(21)supt∈0,TTXn+1t−Xnt2≤2TM∫0TXns−Xn−1s2+EXns−Xn−1s2Δs,n∈ℕ.By a martingale inequality and by inequality (18) (see [25], for details), one has(22)Esupt∈0,TTXn+1t−Xnt2≤CMnhnT,0,n∈ℕ,where C=4TM. Note that a simple probability inequality is obtained from Markov’s inequality, PY>a≤1/apEYp, where a>0,p>0, and Y is a random variable. Using the probability inequality, where Y=supt∈0,TTXn+1t−Xnt,a=1/2n and p=2, we obtain(23)Psupt∈0,TTXn+1t−Xnt>12n≤4nCMnhnT,0,n∈ℕ.According to Borel–Cantelli lemma, this implies that(24)Psupt∈0,TTXn+1t−Xnt>12ni.o.=0,where i.o. is the abbreviation of “infinitely often”. Consequently, X0+∑i=0n−1Xi+1−Xi converges uniformly. Let n⟶∞, we have(25)Xt=x+∫0tAsXs+A¯sEXs+Bsus+B¯sEusΔs+∫0tDsus+D¯sEusΔWs.
For the uniqueness, we assumeX1 and X2 are both solution. Then,(26)X1t−X2t=∫0tAsX1s−X2s+A¯sEX1s−X2sΔs.It follows that(27)EX1t−X2t2≤C∫0tEX1s−X2s2Δs.By Gronwall’s inequality [29], we obtain EX1t−X2t2=0. Thus, X1=X2.
We are in a position to give the main results of the MF-SΔLQ optimal control problem. For this, we need a useful lemma. By some simple calculations, it is not hard for us to get the following product rule for stochastic processes on time scales, which is very similar to Du and Dieu [12].Lemma 1.
For any twon-dimensional stochastic processes X1 and X2 with(28)ΔXit=ait,XitΔt+bit,XitΔWt,t∈0,TT,Xi0=ξi,where ai,bi:T×ℝn⟶ℝn, we have(29)ΔX1′tX2t=X1′tΔX2t+ΔX1′tX2t+ΔX1,X2t,t∈0,TT.In this case,(30)ΔX1,X2t=μta1′a2Δt+b1′b2ΔWt+μta1′b2+a2′b1ΔWt,t∈0,TT,where the function μ is the graininess function as defined in (3) on time scales.Remark 4.
Another form of the abovementioned product rule is as follows:(31)ΔX1′tX2t=X1′tΔX2t+ΔX1′tX2t+ΔX1′tΔX2t,where ΔtΔt=μtΔt, ΔtΔW=ΔWΔt=μtΔW, and ΔWΔW=ΔWt. When T=ℝ, it is consistent with Ito^’s formula.Remark 5.
As mentioned before, because the quadratic variation of a process depends on not only the process itself but also the structure of time, the quadratic variation of a process becomes a little more complicated than the classical one. For instance, the quadratic variation of a deterministic continuous process is no longer zero. Therefore, we can have different forms of the product rule on time scales. For example, the product rule (6) is equivalent to(32)fgΔt=fΔtgt+ftgΔt+μtfΔtgΔt.
Now, we use the square completion technique to present a state feedback optimal control via two coupled Riccati equations on time scales.Theorem 2.
Let (H1) and (H2) hold; then, the following Riccati equations on time scales (RΔEs) admit unique solution P⋅,P^⋅∈C10,TT;Sn:(33)−PΔt=Qt+A′tPσt+PσtAt+μtA′tPσtAt−I+μtA′tPσtBtK−1tB′tPσtI+μtAt,t∈0,ρTT,PT=G,(34)−P^Δt=Qt+Q¯t+At+A¯t′P^σt+P^σtAt+A¯t+μtAt+A¯t′P^σtAt+A¯t−I+μtAt+A¯t′P^σtBt+B¯tK^−1tBt+B¯t′P^σtI+μtAt+A¯t,P^T=G+G¯,t∈0,ρTT,where K and K^ are given as(35)Kt=Rt+μtB′tPσtBt+D′tPσtDt,(36)K^t=Rt+R¯t+μtBt+B¯t′PσtBt+B¯t+Dt+D¯t′PσtDt+D¯t.Furthermore, the optimal control of the MF-SΔLQ problems can be presented as(37)u∗t=−K−1tB′tPσtI+μtAtXt−EXt−K^−1tBt+B¯t′P^σtI+μtAt+A¯tEXt,t∈0,ρTT.In this case, the optimal cost functional is(38)J∗=x′P^0x.Proof.
From the state equation, we have(39)ΔEXt=At+A¯tEXt+Bt+B¯tEutΔt,(40)ΔXt−EXt=AtXt−EXt+Btut−EutΔt+Dtut−Eut+D+D¯EutΔWt.
Assume thatPΔt=Γt and P^Δt=Λt for some deterministic and differentiable functions P and P^ on time scales 0,TT. Applying Lemma 1 to X−EX′PX−EX and EX′P^EX, we can obtain(41)ΔXt−EXt′PtXt−EXt=Xt−EXt′ΓtXt−EXt+μtut−Eut′B′tPσtBtut−Eut+2ut−Eut′B′tPσtI+μtAtXt−EXtΔt+ut−Eut′D′tPσtDtut−Eut+Eut′Dt+D¯t′PσtDt+D¯tEutΔWt+2Xt−EXt+μtAtXt−EXt+Btut−Eut′PσtDtut−Eut+Xt−EXt+μtAtXt−EXt+Btut−Eut′PσtDt+D¯tEutΔWt,(42)ΔEX′tP^tEXt=EXt′ΛtEXt+μtEut′Bt+B¯t′P^σtBt+B¯tEut+2Eut′Bt+B¯t′P^σtI+μtAt+A¯tEXtΔt.
By property (9), integrating from 0 to T and taking expectation from the both sides of the above two equations (41) and (42), we see that(43)XT−EXT′PTXT−EXT=∫0TXt−EXt′ΓtXt−EXt+μtut−Eut′B′tPσtBtut−Eut+2ut−Eut′B′tPσtI+μtAtXt−EXt+ut−Eut′D′tPσtDtut−Eut+Eut′Dt+D¯t′PσtDt+D¯tEutΔt,(44)EX′TP^TEXT−x′P^0x=∫0TEXt′ΛtEXt+μtEut′Bt+B¯t′P^σtBt+B¯tEut+2Eut′Bt+B¯t′P^σtI+μtAt+A¯tEXtΔt.
Moreover, the cost functional can be rewritten as(45)Jx;u⋅=E∫0TXt−EXt′QtXt−EXt+EXt′Qt+Q¯tEXt+ut−Eut′Rtut−Eut+Eut′Rt+R¯tEutΔt+XT−EXT′GXT+EXT′G+G¯EXT.
Inserting (43) and (44) into the cost functional (45) gives(46)Jx;u⋅=E∫0TEut+K^−1tBt+B¯t′P^σtI+μtAt+A¯tEXt′K^t×Eut+K^−1tBt+B¯t′P^σtI+μtAt+A¯tEXtut−Eut+K−1tB′tPσtI+μtAtXt−EXt′Kt×ut−Eut+K−1B′tPσtI+μtAtXt−EXt+Xt−EXt′Γt+Qt+A′tPσt+PσtAt+μtA′tPσtAt−I+μtA′tPσtBtK−1tB′tPσtI+μtAtXt−EXt+EX′tΛt+Qt+Q¯t+At+A¯t′P^σt+P^σtAt+A¯t+μtAt+A¯t′P^σtAt+A¯t−I+μtAt+A¯t′P^σtBt+B¯tK^−1t×Bt+B¯t′P^σtI+μtAt+A¯tEXtΔt+x′P^0x+XT−EXT′G−PTXT−EXT+EX′TG+G¯−P^TEXT.
IfP and P^ satisfy the Riccati equations (33) and (34), then(47)Jx;u⋅=E∫0TEut+K^−1tBt+B¯t′P^σtI+μtAt+A¯tEXt′K^t×Eut+K^−1tBt+B¯t′P^σtI+μtAt+A¯tEXtΔt+E∫0Tut−Eut+K−1tB′tPσtI+μtAtXt−EXt′Kt×ut−Eut+K−1B′tPσtI+μtAtXt−EXtΔt+x′P^0x.
SinceK>0 and K^>0, the optimal control should satisfy(48)Eut+K^−1tBt+B¯t′P^σtI+μtAt+A¯tEXt=0,t∈0,ρTT,(49)ut−Eut+K−1B′tPσtI+μtAtXt−EXt=0,t∈0,ρTT.Making some calculations, we get the optimal control as (37). Substituting it into (47), we have the optimal cost functional can be expressed as (38). For the existence and uniqueness of the solutions to the RΔEs, it is assert [24] that Riccati equation (33) admits as a unique positive semidefinite solution P since (H2) holds. It follows that K^>0. Using the similar method, we can get the solvability of the RΔE (34).Remark 6.
WhenA¯,B¯,D¯,Q¯,R¯, and G¯ are all equal to zero, then P=P^. This recovers the result of the classical SΔLQ problem [24].Remark 7.
WhenT=ℝ+, the coupled RΔEs (33) and (34) reduce to the result in [26]. On the contrary, when T=Z+, the coupled RΔEs are consistent with the case in [27].
Similarly, we have the following theorem which can be regarded as an equivalent form of Theorem2.Theorem 3.
Let (H1) and (H2), then RΔE (33) and the following RΔE (50) admit unique solution P⋅,P¯⋅∈C10,TT;Sn:(50)−P¯Δt=Q¯t+A¯′tPσtI+μtAt+I+μtA′tPσtA¯t+μtA¯′tPσtA¯t+At+A¯t′P¯σt+P¯σtAt+A¯t+μtAt+A¯t′P¯σtAt+A¯t−I+μtAt+A¯t′P+P¯σtBt+B¯tK^−1t×Bt+B¯t′P+P¯σtI+μtAt+A¯t+I+μtA′tPσtBtK−1tB′tPσtI+μtAt,t∈0,ρTT,P^T=G¯,where K and K^ are given as before. For the MF-SΔLQ problem,(51)u∗t=−K^−1tBt+B¯t′P+P¯σtI+μtAt+A¯t−K−1tB′tPσtI+μtAtEXt−K−1tB′tPσtI+μtAtXt,t∈0,ρTT,is an optimal control. Moreover, the optimal cost functional with respect to u∗ is(52)J∗=x′P0x+x′P¯0x.Proof.
As the statement in previous Theorem2, we can obtain the solvability of the RΔEs (33) and (50). We need only to prove (51) and (42). Taking integral of ΔX′tPtXt+EXt′P¯tEXt from 0 to T and taking expectation, we obtain(53)EXT′PTXT+EXT′P¯TEXT−x′P0x−x′P¯0x=−E∫0TX′tQtXt+EXt′Q¯tEXt+u′tRtut+Eut′R¯tEutΔt+E∫0TEut+K^−1tBt+B¯t′P+P¯σtI+μtAt+A¯tEXt′K^t×Eut+K^−1tBt+B¯t′P+P¯σtI+μtAt+A¯tEXtΔt+E∫0Tut−Eut+K−1B′tPσtI+μtAtXt−EXt′Kt×ut−Eut+K−1tB′tPσtI+μtAtXt−EXtΔt.
Consequently, by completing the squares, one has(54)Jx;u⋅=E∫0TEut+K^−1tBt+B¯t′P+P¯σtI+μtAt+A¯tEXt′K^t×Eut+K^−1tBt+B¯t′P+P¯σtI+μtAt+A¯tEXtΔt+E∫0Tut−Eut+K−1tB′tPσtI+μtAtXt−EXt′Kt×ut−Eut+K−1B′tPσtI+μtAtXt−EXtΔt+x′P0x+x′P¯0x.
SinceK>0 and K^>0, we must select u such that(55)Eut=−K^−1tBt+B¯t′P+P¯σtI+μtAt+A¯tEXt,ut−Eut=−K−1tB′tPσtI+μtAtXt−EXt,t∈0,ρTT.Therefore, the optimal control satisfies (51). In this case, the optimal cost functional is (52). The conclusions are proved.Remark 8.
The solutionP^ of the RΔE (34) equals to the sum of P and P¯, where P and P¯ are the solutions of the RΔEs (33) and (50).


## 4. Example

The theorems in Section3 tell us that we can solve the MF-SΔLQ problems if the corresponding Riccati equations can be solved. Now, we discuss a numerical example based on the method developed in the previous sections and compare the difference among the time scales, continuous time, and discrete time.Consider the following example with one-dimensional state equation on time scalesT1=0,1/2∪1∪3/2,2:(56)ΔXt=EXt+12EutΔt+utΔWt,X0=1,and the cost functional(57)Ju⋅=E∫02ut2+Eut2Δt+EX22.The corresponding coupled Riccati equations are(58)PΔt=0,P2=0,(59)−P^Δt=2+μP^σt−181+μ2P^σt2,P^2=1.By solving the coupled Riccati equations and using Theorem 2, the optimal control can be expressed as(60)u1∗t=−41+0.97e2t−1EXt,t∈0,12,−3.05EXt,t=12,−3.99EXt,t=1,−41+15e−22−tEXt,t∈32,2.If we regard the time as the continuous timeT2=0,2, then the optimal control is(61)u2∗t=−41+15e−22−tEXt,t∈0,2.On the contrary, if we treat the time as discrete time T3=0,1,2,3,4, then the optimal control is(62)u3∗t=−2.87EXt,t=0,−3.61EXt,t=12,−3.13EXt,t=1,−1.21EXt,t=32.The comparison result of the optimal controls is shown in Figure1.Figure 1
Optimal controls of the dynamic systems in time scalesT1, continuous time T2, and discrete time T3, respectively.The example implies that we should take an impulsive control whent=1/2 and t=1 in the time scales setting T1. Although the optimal control in the interval 3/2,2 on the time scales T1 is the same as in the continuous case, they are different in the interval 0,1/2. It is to say that the time gap μ influences not only the impulsive control but also the optimal control in the interval 0,1/2. We can see that the optimal control depends on the structure of time domain. This interesting result is hidden in the classical continuous and discrete formulation.


## 5. Conclusions

The linear quadratic optimal control problems for mean-field stochastic differential equations on time scales are studied. It unifies and extends the mean-field optimal control problems in continuous and discrete time formulations. Via two coupled RΔEs on time scales, we get the corresponding optimal control with the state feedback representation. The optimal control problems established in this paper offer a more practical scheme in tackling directly the issue on the mixture of continuous time and discrete time.



---

*Source: 8051918-2020-05-30.xml*