# Dynamic Programming and Hamilton–Jacobi–Bellman Equations on Time Scales

Authors: Yingjun Zhu; Guangyan Jia
Journal: Complexity
 (2020)

---

## Abstract

Bellman optimality principle for the stochastic dynamic system on time scales is derived, which includes the continuous time and discrete time as special cases. At the same time, the Hamilton–Jacobi–Bellman (HJB) equation on time scales is obtained. Finally, an example is employed to illustrate our main results.


---

## Body

## 1. Introduction

The stochastic control problem is to find an optimal control such that a cost functional associated with a stochastic system reaches the minimum value. The method of dynamic programming is a powerful approach to solving the stochastic optimal control problems. The dynamic programming is a well-established subject [1–4] to deal with continuous and discrete optimal control problems, respectively, and it has great practical applications in various fields [5, 6]. It is generally assumed that the time is continuous or discrete in the dynamic systems. However, this cannot be guaranteed. In reality, the time scale could be neither continuous nor discrete. There are many processes which are the mixture of continuous time and discrete time, nonuniform discrete time, or the union of disjoint time interval, such as the production and storage process in economics, the investment process in finance, and the population for the seasonal insects. When the time is more complex, it makes the control problem more difficult. How to deal with this problem?Time scales were first introduced by Hilger [7] in 1988 in order to unite differential and difference equations into a general framework. This allows us to deal with the continuous and discrete analyses from the common point of view. Recently, time scale theory is extensively studied in many works [8–13]. It is well known that the optimal control problems on time scales are an important field for both theory and applications. Since the calculus of variations on time scales was studied by Bohner [14], the results on optimal control problems in the time scale setting and their applications have been rapidly growing. The existence of optimal control for the dynamic systems on time scales was discussed [15–17]. Subsequently, Pontryagin maximum principle on time scales was studied in several works [18, 19], which specifies the necessary conditions for optimality.The dynamic programming for dynamic systems on time scales is not a simple task to unite the continuous time and discrete time cases because the time scales contain more complex time cases. Seiffertt et al. [20] studied the approximate dynamic programming for the dynamic system in the isolated time scale setting. In addition, Bellman dynamic programming on general time scales for the deterministic optimal control problems was considered in [21, 22]. However, limited work [23, 24] has been done on the linear quadratic stochastic optimal control problem in the time scales setting. That is to say, the general setting of stochastic optimal control problems on time scales is completely open.Motivated by all these significant works, the purpose of this paper is to study the method of dynamic programming for the stochastic optimal control problems on time scales. As we know, the stochastic dynamic programming principle is different from the deterministic systems, which reflects the stochastic nature of the optimal problem. So, the method in the deterministic case on time scales cannot be applied to the stochastic case directly. In order to overcome this difficulty, we first give a new form of the chain rule on time scales. Based on this idea, we obtainIto^’s formula for the stochastic process on time scales. Second, we consider a family of optimal control problems with different initial times and states to establish the Bellman optimality principle in the time scale setting. Third, using Ito^’s formula and the Bellman optimality principle obtained in the time scales, we also get the associated Hamilton–Jacobi–Bellman (HJB for short) equation on time scales which is a nonlinear second-order partial differential equation involving expectation. If the HJB equation is solvable, then we can get an optimal feedback control. Our work will enrich the dynamic programming problem by providing a more general time framework and make dynamic programming theory to be a powerful tool in tackling the optimal control problem on complexity time.The organization of this paper is as follows. In Section2, we show some preliminaries about time scale theory. Section 3 focuses on the Bellman optimality principle and the HJB equations on time scales. By introducing a new symbol, we present Ito^’s formula in a new form. On this basis, we get the main results. Finally, an illustrative example is given to show the effectiveness of the proposed results.


## 2. Preliminaries

A time scaleT is a nonempty closed subset in real number set ℝ, and we denote 0,TT=0,T∩T. In this paper, we always suppose T is bounded. The forward jump operator σ and backward jump operator ρ are, respectively, defined by(1)σt=infs∈T:s>t,ρt=sups∈T:s<t,supplemented by inf∅≔supT and sup∅≔infT, where ∅ denotes the empty set. If σt=t and t<supT, the point t is called right-dense, while if σt>t, the point t is called right-scattered. Similarly, if ρt=t and t>infT, the point t is called left-dense, while if ρt<t, the point t is called left-scattered. Moreover, a point is called isolated if it is both left-scattered and right-scattered. For a function f, we denote fσt=fσt to represent the compositions of the functions f and σ. Similarly, we denote fρt=fρt. The definition of the graininess function μ is as follows:(2)μt=σt−t.We now present some basic concepts and properties about time scales (see [10, 11]).Definition 1 (see [10]).
Letf be a function on T. f is called a right-dense continuous function if f is continuous at every right-dense point and has finite left-sided limits at every left-dense point. Similarly, f is called a left-dense continuous function if f is continuous at every left-dense point and has finite right-sided limits at every right-dense point. If f is right-dense continuous and also is left-dense continuous, then f is called a continuous function.
Define the setTκ as(3)Tκ=T\ρsupT,supTifsupT<∞,TifsupT=∞.Definition 2 (see [11]).
Letf: T⟶ℝ be a function and t∈Tκ. If for all ε>0, there exists a neighborhood U of t such that(4)fσt−fs−fΔtσt−s≤εσt−s,for alls∈U,
we callfΔt the Δ-derivative of f at t.
We denote byC10,TT;ℝ the space of ℝ-valued continuously Δ-differentiable functions on 0,TT and denote by C1,2T×ℝn;ℝ the family of all functions ft,x defined on T×ℝn such that they are continuously Δ-differentiable in t and twice continuously differentiable in x.
Furthermore, we give the derivation rule of the compound function.Lemma 1 (see [25]).
Letg:T⟶ℝ be Δ-differentiable and f:ℝ⟶ℝ be continuously differentiable. Then fgt is Δ-differentiable, and(5)fΔgt=gΔt∫01fg′gt+hμtgΔtdh.In this paper, we adopt the stochastic integral defined by Bohner et al. [26]. Let Ω,ℱ,ℱtt∈0,TT,P be a complete probability space with increasing and continuous filtration ℱtt∈0,TT. We define that Lℱ20,TT;ℝ is the set of all ℱt-adapted, ℝ-valued measurable processes Xt such that E∫0TXt2Δt<∞.A Brownian motion indexed by time scalesT was defined by Grow and Sanyal [13]. Although the Brownian motion on time scales is very similar to that on continuous time, there are also some differences between them. For example, the quadratic variation of a Brownian motion on time scales (see [27]) is an increasing process yet, but it is not deterministic. In fact, Wt=λ0,tT+∑s≤tWsσ−Ws2, where λ is the Lebesgue measure.Now, we give the definition of the stochasticΔ-integral and its properties.Definition 3 (see [26]).
The random processXt is stochastic Δ-integrable on 0,TT if the corresponding X˜t is integrable. Define the integral value of Xt as(6)∫0TXtΔWt≔∫0TX˜tdWt,where(7)X˜t=Xsup0,tT,for allt∈0,T,and the Brownian motion on the right side of (6) is indexed by continuous time.
We also have the following properties.
LetXt,Yt∈Lℱ20,TT;ℝ and α,β∈ℝ. Then(8)i∫0TαXt+βYtΔWt=α∫0TXtΔWt+β∫0TYtΔWt,iiE∫0TXtΔWt=0,iiiE∫0TXtΔWt2=E∫0TXt2ΔWt=E∫0TXt2Δt,where the integral X with respect to the quadratic variation of Brownian motion Wt is defined by Stieltjes integral as ∫0TXtωΔWtω.
LetX be an n-dimensional stochastic process defined by(9)ΔXs=as,XsΔs+bs,XsΔWs,and we have the following Ito^’s formula.Lemma 2 (see [28]).
Letf∈C1,2T×ℝn;ℝ and X satisfy (9), then the following relation holds:(10)ft,Xt=f0,X0+∫0tfsΔs,XsΔs+∫0t∂f∂xs,XsTas,XsΔs+∫0t∂f∂xs,XsTbs,XsΔWs+12∫0tbs,XsT∂2f∂x∂xs,Xsbs,XsΔs+∑s∈0,tTfσs,Xσs−fσs,Xs−∑s∈0,tT∂f∂xs,XsTas,Xsσs−s−∑s∈0,tT∂f∂xs,XsTbs,XsWσs−Ws−12bs,XsT∂2f∂x∂xs,Xsbs,Xs.


## 3. Problem Statement and Main Results

LetΩ,ℱ,ℱtt∈T,P be a given filtered probability space satisfying the usual condition. Consider the stochastic control system(11)ΔXs=as,Xs,usΔs+bs,Xs,usΔWs,s∈0,ρTT,X0=x0∈ℝn.The controlu⋅ belongs to(12)U0,TT=u:0,TT×Ω⟶U|uis measurable andℱtt∈T−adapted,where U is a convex subset of ℝm. And the functions at,x,u and bt,x,u satisfy the Lipschitz condition and linear growth condition in x. Obviously, equation (11) admits a unique solution (see Bohner et al. [26]). The cost functional associated with (11) is(13)Ju⋅=E∫0Trs,Xs,usΔs+hXT,where the maps r:0,TT×ℝn×U⟶ℝ and h:ℝn⟶ℝ are continuous.The optimal control problem is to findu∗⋅∈U0,TT such that(14)Ju∗⋅=infu⋅∈U0,TTJu⋅.u∗⋅ is called the stochastic optimal control of the problem, and the corresponding X⋅;x0,u∗⋅ is called an optimal state process.Now, we consider a family of optimal control problems with different initial times and states. Lett,x∈0,TT×ℝn, consider the state equation(15)ΔXs=as,Xs,usΔs+bs,Xs,usΔWs,s∈t,TT,Xt=x,along with the cost functional(16)Jt,x;u⋅=E∫tTrs,Xs,usΔs+hXT.For anyt,x∈0,TT×ℝn, minimize (16) subject to (15) over Ut,TT. The value function of the optimal control problem is defined as(17)Vt,x=infu⋅∈Ut,TTJt,x;u⋅.We first introduce a symbol which is useful in the sequel. Letg:T⟶ℝ be Δ-differentiable and f:ℝ⟶ℝ be continuously differentiable. For any t∈Tκ, define fgΔ¯gt as follows:(18)fgΔ¯gt=lims⟶tfgt−fgsgt−gs,tis right−dense andgΔ≠0,fgσt−fgtgσt−gs,tis right−scattered andgΔ≠0,0,gΔ=0.Remark 1.
Note thatfgΔ¯gt depends not only on the functions f and g but also on the time scales T. If t is a right-dense point on time scales T, then fgΔ¯gt=fgΔgt=fg′gt. On the contrary, if t is a right-scattered point and gσt=σgt, we have fgΔ¯gt=fgΔgt.
With the help of the new symbol, we have the following lemma.Lemma 3.
Letf:ℝ×ℝ⟶ℝ be continuously differentiable and x:T⟶ℝ and y:T⟶ℝ be Δ-differentiable, then(19)fΔxt,yt=fxΔ¯xt,yσtxΔt+fyΔ¯xt,ytyΔt,t∈Tκ.Proof.
IfxΔt=0 or yΔt=0, it is easy to verify that (19) is true. We only give proofs under conditions xΔt≠0 and yΔt≠0. If t is right-dense, one has(20)fΔxt,yt=lims⟶tfxt,yt−fxs,yst−s=lims⟶tfxt,yt−fxt,ys+fxs,yt−fxs,yst−s=lims⟶tfxt,yt−fxs,ytxt−xs×xt−xst−s+fxs,yt−fxs,ysyt−ys×yt−yst−s=fxΔ¯xt,ytxΔt+fyΔ¯xt,ytyΔt=fxΔ¯xt,yσtxΔt+fyΔ¯xt,ytyΔt.
Whent is right-scattered, then(21)fΔxt,yt=fxσt,yσt−fxt,ytσt−t=fxσt,yσt−fxt,yσt+fxt,yσt−fxt,ytσt−t=fxσt,yσt−fxt,yσtxσt−xt×xσt−xtσt−t+fxt,yσt−fxt,ytyσt−yt×yσt−ytσt−t=fxΔ¯xt,yσtxΔt+fyΔ¯xt,ytyΔt.
This completes the proof.Remark 2.
Similarly, another form can be expressed as(22)fΔxt,yt=fxΔ¯xt,ytxΔt+fyΔ¯xσt,ytyΔt.Remark 3.
In particular, letg:T⟶ℝ be Δ-differentiable and f:ℝ⟶ℝ be continuously differentiable. Then fΔgt=fgΔ¯gtgΔt. It is easy to see that this equality is equivalent to (5).Remark 4.
It is not hard for us to get the following result of multidimensions:(23)FΔxt,yt=FxΔ¯xt,yσtTxΔt+FyΔ¯xt,ytTyΔt,where F:ℝn×ℝm⟶ℝ is continuously differentiable and x:T⟶ℝn and y:T⟶ℝm are Δ-differentiable.
Next, we showIto^’s formula in a new form on time scales.Proposition 1.
LetX satisfy (9) and f∈C1,2T×ℝn;ℝ, we have(24)ft,Xt=f0,X0+∫0tfsΔs,XsΔs+∫0tfxΔ¯σs,XsTas,Xss+∫0tfxΔ¯σs,XsTbs,XsΔWs+12∫0tIDsbs,XsTfxxΔ2σs,Xsbs,XsΔs,where I⋅ is an indicative function and D is the set of all right-dense points.Proof.
Because of Lemma2, it is enough to show that(25)∫0tfsΔs,XsΔs+∫0t∂f∂xs,XsTas,XsΔs+∫0t∂f∂xs,XsTbs,XsΔWs+12∫0tbs,XsT∂2f∂x∂xs,Xsbs,XsΔs+∑s∈0,tTfσs,Xσs−fσs,Xs−∑s∈0,tT∂f∂xs,XsTas,Xsσs−s−∑s∈0,tT∂f∂xs,XsTbs,XsWσs−Ws−12bs,XsT∂2f∂x∂xs,Xsbs,Xs=∫0tfsΔs,XsΔs+∫0tfxΔ¯σs,XsTas,XsΔs+∫0tfxΔ¯σs,XsTbs,XsΔWs+12∫0tIDsbs,XsTfxxΔ2σs,Xsbs,XsΔs.
By some manipulation, namely,(26)∑s∈0,tTfσs,Xσs−fσs,Xs=∑s∈0,tTfxΔ¯σs,XsTas,Xsσs−s+∑s∈0,tTfxΔ¯σs,XsTbs,XsWσs−Ws,it is straightforward to show that the above equation is true.
Now, we state the Bellman optimality principle on time scales.Theorem 1 (optimality principle).
Lett,x∈0,TT×ℝn. Then for any t^∈t,TT, we have(27)Vt,x=infu⋅∈Ut,t^TE∫tt^rs,Xs,usΔs+Vt^,Xt^.Proof.
For anyε>0, there exists a control u⋅∈Ut,TT such that(28)Vt,x+ε≥Jt,x;u⋅=E∫tTrs,Xs,usΔs+hXT=E∫tt^rs,Xs,usΔs+E∫t^Trs,Xs,usΔs+hXT|ℱt^t=E∫tt^rs,Xs,usΔs+Jt^,Xt^;u⋅≥E∫tt^rs,Xs,uss+Vt^,Xt^≥infu⋅∈Ut,t^TE∫tt^rs,Xs,usΔs+Vt^,Xt^,where ℱt^t is the sigma field generated by Wr−Wt:r∈t,t^T.
On the contrary, by the definition of value function (17), we obtain(29)Vt,x≤Jt,x;u⋅=E∫tt^rs,Xs,usΔs+Jt^,Xt^;u⋅.
Thus, taking infimum overu⋅∈Ut^,TT, one has(30)Vt,x≤E∫tt^rs,Xs,usΔs+Vt^,Xt^.
It follows that(31)Vt,x≤infu⋅∈Ut,t^TE∫tt^rs,Xs,usΔs+Vt^,Xt^.
Combining with (28) and (31), we get the result.
Furthermore, we give the HJB equation on time scales which is similar to continuous and discrete cases.Theorem 2.
Let the value functionV∈C1,20,TT×ℝn, then the value function V satisfies the following HJB equation:(32)VtΔt,x+infu∈UEtxHt,x,u,VxΔ¯σt,Xt,VxxΔ2σt,x=0,t,x∈0,ρTT×ℝn,VT,x=hx,where(33)Ht,x,u,p,P=rt,x,u+pTat,x,u+pTbt,x,uWΔt+12IDtbt,x,uTPbt,x,u,and we use the notation(34)EtxfxΔ¯Xt=EfxΔ¯Xt|Xt=x,WΔt=Wσt−Wtμt,tis right−scattered,0,tis right−dense.Proof.
According to the definition of the value function,VT,x=hx is satisfied. Fix u∈U, and let X⋅ be the state process corresponding to the control u⋅≡u. By the optimality principle and Ito^’s formula, for any t^>t and t,t^∈0,TT, we have(35)0≤EVt^,Xt^−Vt,xt^−t+1t^−tE∫tt^rs,Xs,uΔs=1t^−tE∫tt^VsΔs,Xs+Hs,Xs,u,VxΔ¯σs,Xs,VxxΔ2σs,XsΔs.
Ift is right-dense, then let t^⟶t, while if t is right-scattered, then let t^=σt. This leads to(36)0≤VtΔt,x+EtxHt,x,u,VxΔ¯σt,Xt,VxxΔ2σt,x.
It follows that(37)0≤VtΔt,x+infu∈UEtxHt,x,u,VxΔ¯σt,Xt,VxxΔ2σt,x.
Conversely, for anyε>0,t^>t, and t,t^∈0,TT, we can find a control u⋅ such that(38)ε≥EVt^,Xt^−Vt,xt^−t+1t^−tE∫tt^rs,Xs,usΔs=1t^−tE∫tt^VsΔs,Xs+Hs,Xs,us,VxΔ¯σs,Xs,VxxΔ2σs,XsΔs≥1t^−tE∫tt^VsΔs,Xs+infu∈UHs,Xs,u,VxΔσs,Xs,VxxΔ¯2σs,XsΔs.
By the same argument as that used in the above proof, we get(39)ε≥VtΔt,x+infu∈UEtxHt,x,u,VxΔ¯σt,Xt,VxxΔ2σt,x.
Therefore, (32) follows. The proof is completed.Remark 5.
It is not surprising that HJB equation (32) on time scales is very similar to the classical HJB equation in continuous and discrete time (see [29, 30]). An intriguing feature of the HJB equation on time scales is that the expectation is involved. When T=ℝ+ or T=ℤ+, we are able to reduce equation (32) into the classical ones.Remark 6.
Supposeb≡0, then HJB equation (32) becomes(40)VtΔt,x+infu∈UVxΔ¯σt,XtTat,x,u+rt,x,uXt=x=0.
In such a case, it is equivalent to the result in [22].Remark 7.
In particular, in Remark6, if we further let Xσt=σXt, HJB equation (32) degenerates into(41)VtΔt,x+infu∈UVxΔσt,xTat,xtu+rt,x,u=0,which is just the one given by Seiffertt et al. [20].
From the above, we end up with the following verification theorem.Theorem 3 (verification theorem).
LetF∈C1,20,TT×ℝn;ℝ be the solution of HJB equation (32), and there exists a function ϕt,Xt such that(42)EHt,Xt,ϕt,Xt,FxΔ¯σt,Xt,FxxΔ2σt,Xt=infu∈UEHt,Xt,u,FxΔ¯σt,Xt,FxxΔ2σt,Xt.
ThenFt,x=Vt,x, and ut=ϕt,Xt is an optimal control.Proof.
Letut=ϕt,Xt, we have(43)EhXT−Ft,x=E∫tTFsΔs,Xs+FxΔ¯σs,XsTas,Xs,us+FxΔ¯σs,XsTbs,Xs,usWΔs+12IDsbs,Xs,usTFxxΔ2σs,Xsbs,Xs,usΔs=−E∫tTrs,Xs,usΔs.
This yields(44)Ft,x=E∫tTrs,Xs,usΔs+hXT≥Vt,x.
In addition, for any admissible pairX⋅,v⋅, we have(45)EhXT−Ft,x=E∫tTFsΔs,Xs+FxΔ¯σs,XsTas,Xs,vs+FxΔσs,XsTbs,Xs,usWΔs+12IDsbs,Xs,vsTFxxΔ¯2σs,Xsbs,Xs,vsΔs=E∫tTFsΔs,Xs+Hs,Xs,u,FxΔ¯σs,Xs,FxxΔ2σs,Xs−rs,Xs,vsΔs≥E∫tTFsΔs,Xs+infu∈UHs,Xs,u,FxΔ¯σs,Xs,FxxΔ2σs,Xs−rs,Xs,vsΔs=−E∫tTrs,Xs,vsΔs.
Namely,(46)Ft,x≤E∫tTrs,Xs,vsΔs+hXT.
By arbitrary forX⋅,v⋅, it follows that(47)Ft,x≤Vt,x.
Hence, by (44) and (47), one has(48)Ft,x=Vt,x.
Finally, inequality (44) together with (48) proves the optimality of u⋅.


## 4. Example

The dynamic programming on time scales contains not only continuous and discrete cases but also other more general cases. In order to illustrate our result, we give an example. Consider the quantum time scaleT=t=qk:k∈ℕ,q>1. The state equation on time scales T is as follows:(49)ΔXs=XsusΔs+usΔWs,s∈T,X1=x0.To find the sequence of optimal control policyuqk,k=0,1,…,N, such that(50)minimizeJu⋅=E∫1qNX2s+u2sΔs.In this example, we haveμt=q−1t. By Theorem 2, the value function V satisfies the following equation:(51)Vt,xq−1=infu∈UEtxx2+u2+Vσt,Xσtq−1t,t∈T,VqN,x=0.We can see that the graininess functionμ affected the solution of HJB equation (51).(52)EX2t+ϕ2t,Xt+Vσt,Xσtq−1t=infu∈UEX2t+u2+Vσt,Xσtq−1t,t∈TBesides, by applying Theorem3, we can find the optimal control ut=ϕt,Xt through the above equation. Furthermore, it implies that the optimal strategy also depends on the structure of time scales.


## 5. Conclusions

In this paper, we developed the dynamic programming principle for the stochastic system on time scales, for which we presentedIto^’s formula on time scales in a new form by introducing a new symbol. Similar to the classical cases, we constructed the HJB equation on time scales and proved the verification theorem. The results of this paper are more general. The continuous and discrete analogues of the dynamic programming are special cases of this paper. An example has been given to demonstrate the effectiveness of our results.



---

*Source: 7683082-2020-11-19.xml*