# Optimal Control of Stochastic Dynamic Systems of a Random Structure with Poisson Switches and Markov Switching

Authors: Svitlana V. Antonyuk; Marian F. Byrka; Mykola Y. Gorbatenko; Taras O. Lukashiv; Igor V. Malyk
Journal: Journal of Mathematics
 (2020)

---

## Abstract

The problem of synthesis of the optimal control for a stochastic dynamic system of a random structure with Poisson perturbations and Markov switching is solved. To determine the corresponding functions for Bellman functional and optimal control the system of ordinary differential equation is investigated.


---

## Body

## 1. Introduction

Systems with Markov parameters belong to an important class of systems describing processes of the rapid changes in states that occur in many cases, for example, in industry, in queuing systems [1], in ecological systems [2], in economics and finance, and in modeling of a microgrid [3]. These systems are mathematical models of the hybrid dynamic systems, in which one part of the variables changes continuously and the other part changes discretely. The most used hybrid models are models described by differential equations. This article focuses on this area of research.Let us pay attention to the fact [4] that such abrupt changes in the systems are discrete events and are assumed to be modelled by a Markov chain taking values in a finite value set. Practical motivations as well as many theoretical results for the Markovian jump system can be found, for example, in [5–7].The linear system of the following form is considered in [4]:(1)x˙t=Aξt+ΔAξtxt+Bξt+ΔBξtut,t≥0,x0=x0,ξ0=ξ0.Here,x∈Rn is a state vector, ξ is the Markov chain with finite number of states, A,B are defined constant matrices for each fixed ξ, ΔA,ΔB are indefinite matrices that satisfy the specific conditions described in the following, and u∈Rm is a control. In this case, the problem of the synthesis of optimal control with constant inverse communication for such systems is solved.In [2], the optimal control problem for the Ito linear stochastic system with the uncertainty of the following form is solved:(2)x˙t=Aξt+ΔAξtxt+Bξt+ΔBξtut+Cξtxtwt,t≥0,x0=x0,ξ0=ξ0.Here and in the following,wt,t≥0, is a Wiener process.In [6], the optimal control problem for the Ito linear stochastic system of the following form is solved:(3)x˙t=A0t,ξtxt+B0t,ξtut+∑k=1rAkt,ξtxt+Bkt,ξtutwkt.The nonlinear case of the Ito stochastic system is considered in [4]. In this paper, the sufficient optimality conditions are obtained in the case of an infinite horizon for the system(4)x˙t=fx,ξt+g1x,ξt,utdt+g2x,ξtwt.In [7], optimality conditions were obtained in the case of an infinite horizon for a linear stochastic system with finite aftereffect and Markov parameters:(5)x˙t=A0ξtxt+A1ξtxt−ht+B0ξtut+B1ξtwt+gxt,xt−ht,ut,wt,ξt.A great contribution to the development of systems with Markov parameters was introduced by Katz [8]. In this work, a Markov process can be either a Markov chain or a continuous Markov process. In addition, it is advisable to consider the case of perturbations of the impulse type, this is when, for example, the moments of time in which the discontinuities of the phase trajectories of the process are possible, and they are known in advance. For deterministic and difference systems, this situation was studied in detail in [9].Stochastic systems of random structure according to Katz [8] with impulse Markov perturbations according to Tsarkov were considered in [10–13]. Stability in different probabilistic senses has been investigated, and the problem of optimal stabilization, solution of which is the control stabilizing the system to stochastically stable, is solved. In these papers, the absence of perturbation points was assumed, but in [14–16], the existence and uniqueness of the solution of a system of differential-difference equations with Markov parameters and switching in the presence of perturbation points were proved. Therefore, we can consider the problem of stability and optimal control for such systems.Furthermore, in the theory of control of classical problems, there are problems of constructing the control that minimizes a certain quality functional. This problem is wider than the problem of optimal stabilization because it does not include stabilization of the solutionx (t). The monograph [17] is one of the main papers in which the general theory of controllable processes and the theory of controlled stochastic differential equations are presented. Here, the classical problems of optimal stopping of random processes are considered, a rigorous proof of the derivation of Bellman’s equations [18] is presented, and applications of Bellman’s equations for constructing optimal controls are considered.Controllable systems with finite aftereffect of the deterministic and stochastic type are considered in [19]. The paper [20] is devoted to the synthesis of optimal control of linear stochastic dynamic systems with finite aftereffect and Poisson perturbations. The general form of the Bellman equation and the Bellman functional for a stochastic dynamical system of a random structure with Markov switching was obtained in [21], which is a sufficient condition for the existence of optimal control, and the problem of synthesis of optimal control for such systems was solved in [22].The switching times of Markov processes in [21, 22] are assumed to be known; however, in many cases, abrupt changes in the trajectories of the process at random moments are also possible. Such situations are adequately described by taking into account the term in the equation of motion, which is the integral of the Poisson measure. Stochastic systems with semi-Markov switching were introduced in [23]. In this paper, the conditions of weak convergence of semi-Markov random evolutions to the diffusion process are considered. Also, sufficient conditions for the stability of prelimited processes are discussed. In the articles [15, 16], Tsarkov et al. discussed the influence of Markov perturbation on the solutions and applied stochastic differential equations for applied problems.This paper is devoted to the synthesis of optimal control of stochastic dynamic systems of a random structure that are under the influence of impulse Markov switching at known moments of time with allowance for Poisson perturbations that allow describing discontinuities of trajectories at random moments.


## 2. Main Result

Consider the stochastic random structure system given by the Ito stochastic differential equation(6)dxt=at,ξt,xt,utdt+bt,ξt,xt,utdwt+∫Rmct,ξt,xt,u,zν˜dz,dt,t∈R+\K,with Markov switching(7)Δxtt=tk=gtk−,ξtk−,ηk,xtk−tk∈K=tn⇑,limn⟶∞tn=+∞,and initial conditions(8)xt0=x0∈Rm,ξt0=y∈Y,ηk0=h∈H.Here,ξt is the continuous Markov process with values in the space Y≔y1,…,yN, ηk,k≥0 is the Markov chain with values in space H; x:0,+∞×Ω⟶Rm;wt is the m-measurable standard Wiener process; ν˜dz,dt=νdz,dt−Eνdz,dt,Eνdz,dt=Πdzdt, is the centered Poisson measure [24, 25]; and the processes w,ν,ξ, and η are independent [24, 25] and defined in the probability basic (Ω,F,Ftt≥0,P).The trajectories of processxt,t≥0, belong to the Skorokhod space D [17], control ut≔ut,xt:0;T×Rm⟶Rm is the m-measurable function from a class of admissible controls U [19]; coefficients a:R+×Y×Rm×Rm⟶Rm,b:R+×Y×Rm×Rm⟶Rm×Rm, and c:R+×Y×Rm×Rm×Rm⟶Rm and function g:R+×Y×H×Rm⟶Rm are measurable functions which satisfy the conditions for the existence and uniqueness of a solution of problems (6)–(8). An example of the existence and uniqueness conditions is the Lipschitz condition and linear growth condition [17].Consider a sequence of functionsvkt,x:t0,T×Rm⟶R1, k≥0, and define the class of the functions:(9)V≔ft,x:f∈C1,2R+×Rm.The weak infinitesimal operator (WIO)vkt,x∈V [24](10)Lvkt,x=limΔ⟶0+1ΔEt,xvkt+Δ,xt+Δ,tk,y,h,u−vkt,x,is defined on the functionals vkt,x∈V.Here,xt=xt,tk,y,h,u is a strong solution of (6) on t∈tk,tk+1 by the control uk∈U, which is constructed on the interval tk,tk+1. The optimal control problem is to find a control uk0,k≥0, from the class U that minimizes the functional of quality [19]:(11)Iuk0=Iuk0t,x=infu∈U∑k=0NEtk,xkFxT+∫tTGs,xs,usds.For some fixedT>0, where Fx≥0,Gt,x,u≥0,Et,xf=Ef/xt=x, and N=maxk:tk<T.It is necessary to prove some auxiliary properties in order to obtain sufficient conditions of optimality. Let us consider the main statements about properties of the infinitesimal operator.Lemma 1.
Let(1)
Unique solution of problems (6)–(8) exists with finite second moment for each t(2)
Sequence of functionalsvk:0,T×Rm⟶R1, k≥0, from class V exists(3)
WIOLkvs,x be defined for s∈tk,T on solutions of (6)–(8)
Then,∀t˜1,t˜2∈t0,T and the quality(12)Etk,xkvkt˜2,xt˜2−Etk,xkvkt˜1,xt˜1=∫t˜1t˜2Etk,xkLvks,xsds,hold.Proof.
Then, the Markov processxt,tk,y,h,u with respect to the minimal σ-algebra Ft˜1,t˜2 that is built on the interval t˜1,t˜2, t˜1,t˜2∈t0;T, satisfies the Dynkin formula [24]:(13)Etk,xkvkt˜1+τt˜2,xt˜1+τt˜2,tk,y,h,u=vkt,x+Etk,xk∫t0τrt˜2Lvkt˜1+τ,xt˜1+τ,tk,y,h,udτ,where τr=inftxt>r and τrt˜2=minτr,t˜2.
Iflimr⟶∞τrt˜2=t˜2, then for the solution of (6)–(8), we can obtain the equal(14)Etk,xkvkt˜1+t˜2,xt˜1+t˜2,tk,y,h,u=vkt,x+Etk,xk∫t0t˜2Lvkt˜1+τ,xt˜1+τ,tk,y,h,udτ.
Analogously, we should write the Dynkin formula [24] on the interval t0,t˜1, and subtracting it from (14), we obtain (12). Lemma 1 is proved.Lemma 2.
Let(1)
Conditions (1) and (2) of Lemma1 be satisfied.(2)
For∀t∈t0,T, ∀vk∈V, k≥0, the equation(15)Lvkt,x+Gt,x,ukt,x=0,with boundary condition(16)vkT,x=0,k≥0,
hold, whereLvkt,x is the WIO defined by (10).Then,vk∈V, k≥0, can be given as(17)vkt,x=Etk,xk∫tTGs,x,us,xds,t∈tk,T.Proof.
Consider the solutionxt∈Rm of (6)–(8) for t∈tk,T, which is constructed according to the initial condition.Let us integrate (15) with respect to t from tk to T and calculate the expectation. We obtain(18)Etk,xk∫tkTLvks,xds+Etk,xk∫tkTGs,xds=0.According to Lemma1, the first term in (18) exists, and it is equal to increment (12):(19)Etk,xk∫tTLvks,xsds=Etk,xkvkT,xT−Etk,xkvkt,xt=−Etk,xkvkt,x=−vkt,x,where Etk,xkvkT,xT=vkT,x=0 by (16) and Etk,xkvkt,xt=Etk,xkvkt,x=vkt,x. So,(20)Etk,xk∫tTLvs,xds=−vkt,x.Substituting this equality into (18), we get the statement of Lemma 2.Theorem 1.
Let(1)
Unique solution of problems (6)–(8) exists with finite second moment for each t.(2)
Sequence of functionsvk∈V, k≥0, and control uk0∈U, k≥0, which satisfy for all t∈tk,T in the equation(21)Lvkt,x+Gt,x,uk0t,x=0,with boundary conditions(22)vkT,x=FxT,exist.(3)
∀t∈0,T, ∀uk∈U,k≥0, and the inequality hold:(23)Lvkt,x+Gt,x,ukt,x≥0,where L=Lt,x,u is WIO (10) on solutions of (6)–(8).Then, the controluk0 is optimal, and for ∀t∈t0,T,(24)Iuk0t,x=infu∈UIut,x=vkt,x.The sequence of functionsvkt,x is called the cost, or the Bellman function, and equation (21) can be written as the Bellman equation:(25)infu∈ULt,xk,uvkt,xk+Gt,xk,u=0.Proof.
The optimal controluk0∈U is an admissible control. So, there is a solution xtk,t0y,h,uk0 for which (21) takes the form(26)Lvkt,xtk,t0,y,h,uk0+Gt,xtk,t0,y,h,uk0,uk0=0.
uk0 is chosen at t,xt,t0,y,h,uk0,t∈tk,tk+1.
Let us integrate (26) from t to T, obtain the expectation, and, taking into account (22), obtain(27)vkt,x=Etk,xkFxT+∫tTLvkτ,xτ,t0,y,h,uk0dτ.
Now, letuk=ukt be any other admissible control from U. Тhen, by condition (3), the following inequality holds: for ∀uk∈U,k≥0,(28)Lvτ,xτ,t0,y,h,u+Gτ,xτ,t0,y,h,u,uτ≥0.
Let us integrate (28) with respect to τ∈tk,T and obtain the expectation E with fixed τ and initial value x. Taking into account Lemmas 1 and 2 and boundary condition (22), we obtain(29)vkt,xt,t0,y,h,uk0≤Etk,xkFxT,uk+∫tkTGτ,xτ,t0,y,h,uk,ukτ,xdτ=vkt,xt,t0,y,h,uk.
This is the definition of optimal controluk0t,x in the sense of the minimization of the functional Iut,x. The theorem is proved.


### 2.1. General Solution of the Problem of the Optimal Control

According to [10, 11, 26], WIO (10) has the form(30)Lvkt,x=∂vkt,x∂t+∇vkt,x,at,y,x,u+12SpbTt,y,x,u⋅∇2vkt,x⋅bt,y,x,u+∫Rmvkt,x+ct,y,x,u,z−vkt,x−∇vkt,xT⋅ct,y,x,u,zΠdz+∑j≠iN∫Rmvjt,xpijt,z/xdz−vit,xqij.⋅,⋅ is a scalar product, ∇vk≔∂vk/∂x1,…,∂vk/∂xmT, ∇2vk≔∂2vk/∂xi∂xji,j=1m, k≥0; «T» is the sign of transpose, Sp is the trace of a matrix, and pijt,z/x is the conditional density of distribution(31)Pxτ∈z,z+dzxτ−0=x=pijτz/xdz+οdz.By assumption,ξτ−0=yiandξτ=yj.The first equation forvk0t,x, k≥0, can be obtained by substituting (30) in (21). Then, the required equation takes the form(32)∂vk0t,x∂t+∂vk0t,x∂xT⋅at,yi,x,u+12SpbTt,yi,x,u⋅∂2vk0t,x∂x2⋅bt,yi,x,u+∫Rmvk0t,x+ct,yi,x,u,z−vk0t,x−∂vk0t,x∂xT⋅ct,yi,x,u,zΠdz+∑j≠iN∫Rmvj0t,xpijt,z/xdz−vi0t,xqij+Gt,x,u=0,with boundary condition(33)vk0T,x=FxT.The second equation for optimal controluk0t,x can be obtained from (32) by differentiating with respect to u because u=uk0, k≥0, gives a minimum of the left side of (32):(34)∂vk0t,x∂xT⋅∂at,yi,x,u∂u+12Sp∂bt,yi,x,u∂uT⋅∂2vk0t,x∂x2⋅bt,yi,x,u+bTt,yi,x,u⋅∂2vk0t,x∂x2⋅∂bt,yi,x,u∂u+∂∂u∫Rmvk0t,x+ct,yi,x,u,z−vk0t,x−∂vk0t,x∂xT⋅ct,yi,x,u,zΠdz+∂Gt,x,u∂uTu=uk0=0.∂a/∂u is m×m Jacobian, which consists of elements ∂an/∂us,n=1,m¯,s=1,m¯ (∂b/∂u—similarly), and ∂G/∂u≡∂G/∂u1,…,∂G/∂ur, k≥0.Solving systems (32) and (34), even in the presence of computing technology, is quite difficult. So, it is advisable to consider a simplified view of problems (6)–(8) and (10), such as a system with a quadratic quality functional. The following paragraphs will focus on such problems.


### 2.2. Optimal Control of Linear Stochastic Random Structure Dynamic Systems with Markov Switching

Let us consider the problem of optimal control for the stochastic dynamic system given by the following stochastic differential equation:(35)dxt=At,ξtxt+Bt,ξtutdt+σt,ξtxtdwt+∫RmCt,ξt,zxtν˜dz,dt,t∈R+\K,with Markov switching(36)Δxtt=tk=gtk−,ξtk−,ηk,xtk−,and initial conditions(37)x0=x0∈Rm,ξ0=y∈Y,η0=h∈H.Here,A,B,σ,andC are piecewise continuous integrable matrix functions of appropriate dimensions.The optimal control problem for systems (35)–(37) is to find the control uik0, i∈1,…N, k≥0, from set U of admissible controls such that it minimizes the functional of quality:(38)Iuik≔Iuikt,x≔∑k=0N¯Etk,xkxTTM0ξt,ηkxT+∫tTuTsM1s,ξs,ηkus+xTsM2s,ξs,ηkxsds.M1t,ξt,ηk is a uniformly positive definite m×m-matrix with respect to t∈0,T, and M0ξt,ηk and M2t,ξt,ηk are nonnegative definite m×m matrices. To simplify, let us denote(39)Ait≔At,yi,Bit≔Bt,yi,σit≔σt,yi,Cit,z≔Ct,yi,z,M0ik≔M0yi,ηk,M1ikt≔M1t,yi,ηk,M2ikt≔M2t,yi,ηk.Theorem 2.
The optimal control for problems (35)–(38) has the form(40)uik0t,x=−M1ik−1tBiTtPiktxt,where the nonnegative definite m×m matrix Pikt≔Pt,ξt,ηk belongs to the Bellman functional:(41)vik0t,x=xTtPiktxt,vik0T,x=xTtkM0ikxtk.Proof.
The Bellman equation for (35)–(37) has the form(42)infu∈ULvik0t,x+uikTt,xM1iktuikt,x+xTtM2iktxt=0,where(43)Lvik0t,x=∂vik0t,x∂t+Aitxt+Bituikt,xT∇vik0t,x+12SpσiTt∇2vik0t,xσit+∫Rmvik0t,x+Cit,z−vik0t,x−∂vik0t,x∂xT⋅Cit,zΠdz+∑j≠iNvjk0t,x−vik0t,xqij+∫Hvik0t,x+gtk−,ξtk−,ηk,xtk−Pkh,Z−vik0t,x.
Substitute (43) into (42):(44)∂vik0t,x∂t+Aitxt+Bituikt,xT∇vik0t,x+12SpσiTt∇2vik0t,xσit+∫Rmvik0t,x+Cit,z−vik0t,x−∂vik0t,x∂xT⋅Cit,zΠdz+∑j≠iNvjk0t,x−vik0t,xqij+∫Hvik0t,x+gtk−,ξtk−,ηk,xtk−Pkh,Z−vik0t,x+uikTt,xM1iktuikt,x+xTtM2iktxt=0.
The expression for the optimal control can be obtained by differentiating (44) because uikt,x=uik0t,x minimizes the left side of (44):(45)uik0t,x=−12M1ik−1tBiTt∇vik0t,x,where(46)∇vik0t,x=2Piktxt.
So,(47)uik0t,x=−M1ik−1tBiTtPiktxt.
The theorem is proved.


### 2.3. Construction of the Bellman Equation

Substituting (40) and (41) into (42), we obtain the following equation for ∀t∈tk,tk+1:(48)xtTdPiktdtxt+2Aitxt−BitM1ik−1tBiTtxtPiktxt+SpσiTtPiktσit+∫RmxTtCit,zPiktCiTt,zxtΠdz+∑j≠iNxTtPjktxt−xTtPiktxtqij+M1ik−1tBiTtPiktxtTM1iktM1ik−1tBiTtPiktxt+xTtM2iktxt=0.Equating the quadratic form with respect tox and expressions that do not depend on x to zero and taking into account the matrix equality 2xTPikAix=xTPikAi+AiTPikx, we obtain the system of differential equations for finding matrices Pikt, t∈tk,tk+1, i∈1,…N, and k≥0:(49)dPiktdt+AiTtPikt+PiktAit−BitM1ik−1tBiTtPikt+∫RmCit,zPiktCiTt,zΠdz+∑j≠iNPjkt−Piktqij+M1ik−1tBiTtPiktTBiTtPikt+M2ikt=0,(50)SpσiTtPiktσit=0,with boundary conditions(51)PikT=M0ik.Thus, the following theorem holds.Theorem 3.
If the control cost is found in the form (38) for systems (35)–(37), then the system of differential equations for finding the matrices Pikt, t∈tk,tk+1, i∈1,…N, and k≥0, has forms (49)–(51).Furthermore, the problem of the existence of solutions of problems (49)–(51) must be solved. Let us use the Bellman method of iterations [18]. For simplicity, consider the interval tk,tk+1, where ξt=yi, and omit all indices «ik» in u,v, and P. First, we define the zero approximation:(52)u0t,x=−M1−1tBTtP0txt,where P0t≥0 is the bounded and piecewise-continuous matrix. Let us substitute (52) into (34) and calculate the value of v1t,x for the resulting equation, which corresponds to the control (42).Then, substitutingv1t,x into Bellman equation (42), the control u1t,x which minimizes (42) will be found. Continuing this process, one can construct a sequence of controls unt,x and functionals vnt,x of the following form:(53)unt,x=−M1−1tBTtPntxt,vnt,x=xTtPntxt,vnT,x=xTtkM0xtk,where Pnt,t∈tk,tk+1 is the solution of boundary value problems (49)–(51) for T≔tk+1.The following estimate is valid for∀n≥1:(54)vn−1t,x≥vnt,x≥0,∀t∈tk,tk+1.The convergence of functionalsvnt,x to v0t,x, the convergence of controls unt,x to u0t,x, and the convergence of a sequence of matrices Pnt to Pt can be proved by using (54) [19, 20].The following estimate is valid:(55)maxt∈tk,tk+1Pt−Pnt≤Cn!,C<∞,k≥1.Thus, the following theorem holds.Theorem 4.
An approximate solution of the problem of synthesis of the optimal control for (35)–(38) is found using the Bellman successive approximation method for which the n-th approximation of the control and the Bellman functional for each interval tk,tk+1, k≥0, are found from formula (53), and the error is estimated by inequality (55).


### 2.4. Model Example

Let us consider the scalar case with two states for Markov chainsξ and η: ξ=1,2 and η=η1,η2. Define the following initial data for system (35):(1)
ξ=1, A1t=−1, B1t=1, σ1t=0.1, and C1t,z=z2(2)
ξ=2, A2t=−2.25, B2t=1, σ2t=0.2, and C2t,z=z3Generator of the Markov processξ is defined as Q=−0.50.50.5−0.5, and the transaction matrix for the Markov chain η is P=0.30.70.80.2.Then, defineM0, M1, and M2 for (38):(56)M011=M021=M012=M022=8,M111=0.7,M112=0.8,M121=0.9,M122=0.4,M211=0.5,M212=0.4,M221=1,M222=0.9.The Bellman functional considers the following form:vik0t,x=Pikx2t, i,k∈1,2.System (49) for finding Pik, i,k∈1,2, takes the form(57)2AiPik−Bi2M1ik−1Pik+∫R1Ci2zPikΠdz+∑j≠i2Pjk−Pikqij+M1ik−1Bi2Pik2+M2ik=0,i,k∈1,2.Substituting values ofA,B,M0,M1,andM2, we obtain(58)P11=0.1194,P12=0.1004,P21=0.1504,P22=0.1122.So, the optimal control for our case has a form(59)u0t,x=−0.08358xt,ξ=1,η=η1,−0.08032xt,ξ=1,η=η2,−0.13536xt,ξ=2,η=η1,−0.04488xt,ξ=2,η=η2.The results of simulation of two trajectories of the random processxt are shown in Figure 1. This figure shows two trajectories of solutions with a positive (blue line) and negative (red line) initial condition x0. As we can see from this figure, the solutions are stabilized because the optimal control satisfies the conditions of Theorem 1 with the quadratic functional.Figure 1
Two solutions of systems (35)–(37) with initial conditions −4 and 3 and parameters from the model example.


## 2.1. General Solution of the Problem of the Optimal Control

According to [10, 11, 26], WIO (10) has the form(30)Lvkt,x=∂vkt,x∂t+∇vkt,x,at,y,x,u+12SpbTt,y,x,u⋅∇2vkt,x⋅bt,y,x,u+∫Rmvkt,x+ct,y,x,u,z−vkt,x−∇vkt,xT⋅ct,y,x,u,zΠdz+∑j≠iN∫Rmvjt,xpijt,z/xdz−vit,xqij.⋅,⋅ is a scalar product, ∇vk≔∂vk/∂x1,…,∂vk/∂xmT, ∇2vk≔∂2vk/∂xi∂xji,j=1m, k≥0; «T» is the sign of transpose, Sp is the trace of a matrix, and pijt,z/x is the conditional density of distribution(31)Pxτ∈z,z+dzxτ−0=x=pijτz/xdz+οdz.By assumption,ξτ−0=yiandξτ=yj.The first equation forvk0t,x, k≥0, can be obtained by substituting (30) in (21). Then, the required equation takes the form(32)∂vk0t,x∂t+∂vk0t,x∂xT⋅at,yi,x,u+12SpbTt,yi,x,u⋅∂2vk0t,x∂x2⋅bt,yi,x,u+∫Rmvk0t,x+ct,yi,x,u,z−vk0t,x−∂vk0t,x∂xT⋅ct,yi,x,u,zΠdz+∑j≠iN∫Rmvj0t,xpijt,z/xdz−vi0t,xqij+Gt,x,u=0,with boundary condition(33)vk0T,x=FxT.The second equation for optimal controluk0t,x can be obtained from (32) by differentiating with respect to u because u=uk0, k≥0, gives a minimum of the left side of (32):(34)∂vk0t,x∂xT⋅∂at,yi,x,u∂u+12Sp∂bt,yi,x,u∂uT⋅∂2vk0t,x∂x2⋅bt,yi,x,u+bTt,yi,x,u⋅∂2vk0t,x∂x2⋅∂bt,yi,x,u∂u+∂∂u∫Rmvk0t,x+ct,yi,x,u,z−vk0t,x−∂vk0t,x∂xT⋅ct,yi,x,u,zΠdz+∂Gt,x,u∂uTu=uk0=0.∂a/∂u is m×m Jacobian, which consists of elements ∂an/∂us,n=1,m¯,s=1,m¯ (∂b/∂u—similarly), and ∂G/∂u≡∂G/∂u1,…,∂G/∂ur, k≥0.Solving systems (32) and (34), even in the presence of computing technology, is quite difficult. So, it is advisable to consider a simplified view of problems (6)–(8) and (10), such as a system with a quadratic quality functional. The following paragraphs will focus on such problems.


## 2.2. Optimal Control of Linear Stochastic Random Structure Dynamic Systems with Markov Switching

Let us consider the problem of optimal control for the stochastic dynamic system given by the following stochastic differential equation:(35)dxt=At,ξtxt+Bt,ξtutdt+σt,ξtxtdwt+∫RmCt,ξt,zxtν˜dz,dt,t∈R+\K,with Markov switching(36)Δxtt=tk=gtk−,ξtk−,ηk,xtk−,and initial conditions(37)x0=x0∈Rm,ξ0=y∈Y,η0=h∈H.Here,A,B,σ,andC are piecewise continuous integrable matrix functions of appropriate dimensions.The optimal control problem for systems (35)–(37) is to find the control uik0, i∈1,…N, k≥0, from set U of admissible controls such that it minimizes the functional of quality:(38)Iuik≔Iuikt,x≔∑k=0N¯Etk,xkxTTM0ξt,ηkxT+∫tTuTsM1s,ξs,ηkus+xTsM2s,ξs,ηkxsds.M1t,ξt,ηk is a uniformly positive definite m×m-matrix with respect to t∈0,T, and M0ξt,ηk and M2t,ξt,ηk are nonnegative definite m×m matrices. To simplify, let us denote(39)Ait≔At,yi,Bit≔Bt,yi,σit≔σt,yi,Cit,z≔Ct,yi,z,M0ik≔M0yi,ηk,M1ikt≔M1t,yi,ηk,M2ikt≔M2t,yi,ηk.Theorem 2.
The optimal control for problems (35)–(38) has the form(40)uik0t,x=−M1ik−1tBiTtPiktxt,where the nonnegative definite m×m matrix Pikt≔Pt,ξt,ηk belongs to the Bellman functional:(41)vik0t,x=xTtPiktxt,vik0T,x=xTtkM0ikxtk.Proof.
The Bellman equation for (35)–(37) has the form(42)infu∈ULvik0t,x+uikTt,xM1iktuikt,x+xTtM2iktxt=0,where(43)Lvik0t,x=∂vik0t,x∂t+Aitxt+Bituikt,xT∇vik0t,x+12SpσiTt∇2vik0t,xσit+∫Rmvik0t,x+Cit,z−vik0t,x−∂vik0t,x∂xT⋅Cit,zΠdz+∑j≠iNvjk0t,x−vik0t,xqij+∫Hvik0t,x+gtk−,ξtk−,ηk,xtk−Pkh,Z−vik0t,x.
Substitute (43) into (42):(44)∂vik0t,x∂t+Aitxt+Bituikt,xT∇vik0t,x+12SpσiTt∇2vik0t,xσit+∫Rmvik0t,x+Cit,z−vik0t,x−∂vik0t,x∂xT⋅Cit,zΠdz+∑j≠iNvjk0t,x−vik0t,xqij+∫Hvik0t,x+gtk−,ξtk−,ηk,xtk−Pkh,Z−vik0t,x+uikTt,xM1iktuikt,x+xTtM2iktxt=0.
The expression for the optimal control can be obtained by differentiating (44) because uikt,x=uik0t,x minimizes the left side of (44):(45)uik0t,x=−12M1ik−1tBiTt∇vik0t,x,where(46)∇vik0t,x=2Piktxt.
So,(47)uik0t,x=−M1ik−1tBiTtPiktxt.
The theorem is proved.


## 2.3. Construction of the Bellman Equation

Substituting (40) and (41) into (42), we obtain the following equation for ∀t∈tk,tk+1:(48)xtTdPiktdtxt+2Aitxt−BitM1ik−1tBiTtxtPiktxt+SpσiTtPiktσit+∫RmxTtCit,zPiktCiTt,zxtΠdz+∑j≠iNxTtPjktxt−xTtPiktxtqij+M1ik−1tBiTtPiktxtTM1iktM1ik−1tBiTtPiktxt+xTtM2iktxt=0.Equating the quadratic form with respect tox and expressions that do not depend on x to zero and taking into account the matrix equality 2xTPikAix=xTPikAi+AiTPikx, we obtain the system of differential equations for finding matrices Pikt, t∈tk,tk+1, i∈1,…N, and k≥0:(49)dPiktdt+AiTtPikt+PiktAit−BitM1ik−1tBiTtPikt+∫RmCit,zPiktCiTt,zΠdz+∑j≠iNPjkt−Piktqij+M1ik−1tBiTtPiktTBiTtPikt+M2ikt=0,(50)SpσiTtPiktσit=0,with boundary conditions(51)PikT=M0ik.Thus, the following theorem holds.Theorem 3.
If the control cost is found in the form (38) for systems (35)–(37), then the system of differential equations for finding the matrices Pikt, t∈tk,tk+1, i∈1,…N, and k≥0, has forms (49)–(51).Furthermore, the problem of the existence of solutions of problems (49)–(51) must be solved. Let us use the Bellman method of iterations [18]. For simplicity, consider the interval tk,tk+1, where ξt=yi, and omit all indices «ik» in u,v, and P. First, we define the zero approximation:(52)u0t,x=−M1−1tBTtP0txt,where P0t≥0 is the bounded and piecewise-continuous matrix. Let us substitute (52) into (34) and calculate the value of v1t,x for the resulting equation, which corresponds to the control (42).Then, substitutingv1t,x into Bellman equation (42), the control u1t,x which minimizes (42) will be found. Continuing this process, one can construct a sequence of controls unt,x and functionals vnt,x of the following form:(53)unt,x=−M1−1tBTtPntxt,vnt,x=xTtPntxt,vnT,x=xTtkM0xtk,where Pnt,t∈tk,tk+1 is the solution of boundary value problems (49)–(51) for T≔tk+1.The following estimate is valid for∀n≥1:(54)vn−1t,x≥vnt,x≥0,∀t∈tk,tk+1.The convergence of functionalsvnt,x to v0t,x, the convergence of controls unt,x to u0t,x, and the convergence of a sequence of matrices Pnt to Pt can be proved by using (54) [19, 20].The following estimate is valid:(55)maxt∈tk,tk+1Pt−Pnt≤Cn!,C<∞,k≥1.Thus, the following theorem holds.Theorem 4.
An approximate solution of the problem of synthesis of the optimal control for (35)–(38) is found using the Bellman successive approximation method for which the n-th approximation of the control and the Bellman functional for each interval tk,tk+1, k≥0, are found from formula (53), and the error is estimated by inequality (55).


## 2.4. Model Example

Let us consider the scalar case with two states for Markov chainsξ and η: ξ=1,2 and η=η1,η2. Define the following initial data for system (35):(1)
ξ=1, A1t=−1, B1t=1, σ1t=0.1, and C1t,z=z2(2)
ξ=2, A2t=−2.25, B2t=1, σ2t=0.2, and C2t,z=z3Generator of the Markov processξ is defined as Q=−0.50.50.5−0.5, and the transaction matrix for the Markov chain η is P=0.30.70.80.2.Then, defineM0, M1, and M2 for (38):(56)M011=M021=M012=M022=8,M111=0.7,M112=0.8,M121=0.9,M122=0.4,M211=0.5,M212=0.4,M221=1,M222=0.9.The Bellman functional considers the following form:vik0t,x=Pikx2t, i,k∈1,2.System (49) for finding Pik, i,k∈1,2, takes the form(57)2AiPik−Bi2M1ik−1Pik+∫R1Ci2zPikΠdz+∑j≠i2Pjk−Pikqij+M1ik−1Bi2Pik2+M2ik=0,i,k∈1,2.Substituting values ofA,B,M0,M1,andM2, we obtain(58)P11=0.1194,P12=0.1004,P21=0.1504,P22=0.1122.So, the optimal control for our case has a form(59)u0t,x=−0.08358xt,ξ=1,η=η1,−0.08032xt,ξ=1,η=η2,−0.13536xt,ξ=2,η=η1,−0.04488xt,ξ=2,η=η2.The results of simulation of two trajectories of the random processxt are shown in Figure 1. This figure shows two trajectories of solutions with a positive (blue line) and negative (red line) initial condition x0. As we can see from this figure, the solutions are stabilized because the optimal control satisfies the conditions of Theorem 1 with the quadratic functional.Figure 1
Two solutions of systems (35)–(37) with initial conditions −4 and 3 and parameters from the model example.


## 3. Conclusions

In this paper, the problem of synthesis of the optimal control of stochastic dynamical systems of a random structure with Poisson perturbations that are under the influence of pulse switching of the Markov chain type is solved. In the linear case, an algorithm for finding the optimal control is obtained, and its convergence is justified.



---

*Source: 9457152-2020-06-24.xml*