# Linearization of a Matrix Riccati Equation Associated to an Optimal Control Problem

Authors: Foued Zitouni; Mario Lefebvre
Journal: International Journal of Differential Equations
 (2014)

---

## Abstract

The matrix Riccati equation that must be solved to obtain the solution to stochastic optimal control problems known as LQG homing is linearized for a class of processes. The results generalize a theorem proved by Whittle and the one-dimensional case already considered by the authors. A particular two-dimensional problem is solved explicitly.


---

## Body

## 1. Introduction

Let{W(t),t≥0} be a one-dimensional standard Brownian motion and consider the controlled diffusion process {X(t),t≥0} that satisfies the stochastic differential equation
(1)dX(t)=f[X(t)]dt+b[X(t)]u[X(t)]dt+v[X(t)]1/2dW(t),
where u(·) is the control variable and f(·), b(·), and v(·)>0 are Borel measurable functions.The problem of finding the controlu* that minimizes the expected value of the cost function
(2)J(x)=∫0τ(x)‍{12q[X(t)]u2[X(t)]+λ}dt,
in which q(·)>0 is a Borel measurable function, λ≠0 is a real parameter, and τ(x) is afirst passage time defined by
(3)τ(x)=inf⁡{t>0:X(t)∉(d1,d2)∣X(0)=x},
with d1<x<d2, is a special case of what Whittle [1] (p. 289) termedLQG homing. This type of problem has applications in financial mathematics (see Makasu [2]). In Lefebvre and Zitouni [3], the authors considered an optimal landing problem. They also mentioned another possible application in which one wants to optimally control a dam.In order to obtainu*, we define the value function
(4)F(x)=inf⁡{u[X(t)],0≤t≤τ(x)}E[J(x)].
Using dynamic programming, one only has to determine the value of u*(x). We find that F(x) is such that
(5)F(x)=inf⁡u(x){12v(x)F′′(x)F(x)+12q(x)u2(x)+λ+[f(x)+b(x)u(x)]F′(x)+12v(x)F′′(x)};
that is,
(6)0=inf⁡u(x){12q(x)u2(x)+λ+[f(x)+b(x)u(x)]F′(x)+12v(x)F′′(x)}.
It follows that
(7)u*(x)=-b(x)q(x)F′(x),
so that
(8)λ+f(x)F′(x)-12b2(x)q(x)[F′(x)]2+12v(x)F′′(x)=0.
The boundary conditions are
(9)F(d1)=F(d2)=0.Whittle has shown that if the relation(10)αv[X(t)]=b2[X(t)]q[X(t)]
holds for a positive constant α, then the function
(11)Φ(x):=e-αF(x)
satisfies the linear second-order ordinary differential equation
(12)12v(x)Φ′′(x)+f(x)Φ′(x)-αλΦ(x)=0,
subject to
(13)Φ(d1)=Φ(d2)=1.
Equation (12) is actually the Kolmogorov backward equation satisfied by the moment-generating function (or the Laplace transform of the density function) of the random variable τ0(x) that corresponds to τ(x) but for the uncontrolled process obtained by setting u[X(t)]≡0 in (1). Moreover, the above boundary conditions are the appropriate ones. Thus, Whittle was able to sometimes transform the optimal control problem into a purely probabilistic problem.Remark 1.
Whenv[X(t)], b[X(t)], and q[X(t)] are (positive) constant functions, the relation in (10) is obviously satisfied. Therefore, it is then always possible to linearize (8) in such a case. However, if two (or all) of these functions are not constant, we can say that it is a special case when (10) does hold. When only one of these three functions is not a constant, the relation cannot be satisfied.
Next, notice that the optimal control is expressed in terms of the derivativeG(x):=F′(x) of the value function, which satisfies the Riccati equation
(14)λ+f(x)G(x)-12b2(x)q(x)[G(x)]2+12v(x)G′(x)=0.
However, in general, we do not have a condition that would enable us to determine the value of the arbitrary constant that appears in the solution of (14). Therefore, we must solve either the nonlinear second-order differential equation (8) or the Kolmogorov equation (12).
In Lefebvre and Zitouni [3], the authors generalized Whittle’s result. They showed that if b(·) is different from zero, then the function z(x) defined through
(15)G(x)=-q(x)v(x)b2(x)z′(x)z(x)
is a solution of the linear second-order ordinary differential equation
(16)λ-f(x)q(x)v(x)b2(x)z′(x)z(x)-12q(x)v2(x)b2(x)z′′(x)z(x)-12v(x)[q′(x)v(x)+q(x)v′(x)]b2(x)z′(x)z(x)+b′(x)q(x)v2(x)b3(x)z′(x)z(x)=0.
They then gave a method that can be used to obtain an explicit expression for G(x), hence the optimal control u*(x).
Now, Whittle actually considered LQG homing problems inn dimensions. {X(t),t≥0} is then an n-dimensional controlled diffusion process defined by
(17)dX(t)=f[X(t)]dt+B[X(t)]u[X(t)]dt+{N[X(t)]}1/2dW(t),
where the noise matrix N is symmetric and positive definite. The cost function J(x) is replaced by
(18)J(x)=∫0τ(x)‍{12u′[X(t)]Q[X(t)]u[X(t)]+λ}dt.
The matrix Q is positive definite and
(19)τ(x)=inf⁡{t>0:X(t)∉C∣X(0)=x},
where C denotes the continuation region.Remark 2.
In the general formulation,B is an n×m matrix, u is a (column) vector of dimension m, and Q is an m×m matrix. Here, we assume that m=n.
The optimal control is given by(20)u*(x)=-Q-1(x)B′(x)Fx(x)
and the value function F(x) satisfies
(21)λ+Fx′(x)f(x)-12Fx′(x)B(x)Q-1(x)B′(x)Fx(x)+12tr⁡[NFxx(x)]=0,
where Fx is the derivative of F with respect to the vector x. The equation is subject to
(22)F(x)=0ifx∈∂D,
in which ∂D denotes the boundary of the stopping region D (the complement of continuation region C).
The relation that generalizes (10) and that must hold between the control matrices B and Q and the noise matrix N in order to be able to linearize the nonlinear partial differential equation (21) is the following:
(23)αN[X(t)]=B[X(t)]Q-1[X(t)]B′[X(t)].
In practice, it is difficult to satisfy (exactly) the above relation for n≥2, especially if the matrices involved are not constant matrices. In fact, even in the case when the various matrices in (23) are indeed constant, it is rather rare that this relation is satisfied. Problems for which (23) holds must be symmetrical. For instance, an important particular case is the one whenN,B, andQ are all proportional to the identity matrix of order n (andf is identical to zero), so that we want to optimally control an n-dimensional Brownian motion.
Because of the importance of the matrix Riccati equation in many applications, the problem of linearizing this equation has been considered by a number of authors. Grasselli and Tebaldi [4], in particular, proposed a method that enabled them to transform the matrix Riccati equation that appeared in their work into 2n linear equations; see also Gourieroux and Sufana [5].
The aim of this paper is to generalize the theorem proved by Whittle [1] and, at the same time, the results in Lefebvre and Zitouni [3]. In the next section, first the two-dimensional case will be presented. Then, the results will be extended to the n-dimensional case. In Section 3, a particular two-dimensional problem will be solved explicitly. Finally, we will end with a few concluding remarks in Section 4.


## 2. Linearization of the Matrix Riccati Equation in Two Dimensions

Let(24)M[X(t)]=B[X(t)]Q-1[X(t)]B′[X(t)].
If the relation in (23) holds, then the matrix M is symmetric and invertible. To generalize Whittle’s theorem, we will assume that M is indeed symmetric and invertible but not necessarily proportional to N.For simplicity, we will present the linearization technique that we propose in the case of two-dimensional controlled diffusion processes. Equation (17) can then be rewritten as follows [omitting the dependence of all functions on X(t)=(X1(t),X2(t))]:
(25)(dX1(t)dX2(t))=(f1f2)dt+(b11b12b21b22)(u1u2)dt+(σ12σ12σ12σ22)1/2(dW1(t)dW2(t))
and the cost function becomes
(26)J(x1,x2)=∫0τ(x1,x2)‍{12(u1u2)(q11q12q21q22)(u1u2)+λ}dt.
The two standard Brownian motions are assumed to be independent.Next, let(27)Fx(x1,x2)=G(x1,x2)=(G1(x1,x2)G2(x1,x2)),Fxx(x1,x2)=Gx(x1,x2)=H(x1,x2)=(H11(x1,x2)H12(x1,x2)H12(x1,x2)H22(x1,x2)),
where we have assumed that Fx1x2(x1,x2)=Fx2x1(x1,x2).The optimal control is given by(28)(u1*u2*)=-(q11q12q21q22)-1(b11b21b12b22)(G1G2).
Moreover, the matrix Riccati equation satisfied by the derivative G(x1,x2) of the value function F(x1,x2) with respect to x and that we want to linearize is given by [see (21)]
(29)λ+G′(f1f2)-12G′MG+12tr⁡[NH]=0.Proposition 3.
Assume that the matrixM defined in (24) is symmetric and invertible. Then, the function z(x1,x2) defined by the transformation
(30)G′=1z(zx1zx2)K,
where zxi=∂z/∂xi  for i=1,2 and
(31)K=(k11k12k21k22):=-NM-1,
satisfies the linear partial differential equation
(32)λz+∑i=12‍∑j=12‍fizxjkji+12L(zx1x1,zx1x2,zx2x2)=0,
where
(33)L(zx1x1,zx1x2,zx2x2)=∑i=12‍∑j=12‍σi2{zxjxikji+zxj∂∂xikji}+2σ12{∑i=12‍zxix2ki1+zxi∂∂x2ki1}.
Furthermore, for the transformation to be valid, there must exist functions ϕ(x1) and ψ(x2) such that
(34)∫‍1z∑i=12‍ki1zxidx1+ψ(x2)=∫‍1z∑i=12‍ki2zxidx2+ϕ(x1).Proof.
We compute(35)G′(f1f2)=1z∑i=12‍∑j=12‍fizxjkji,G′MG=1z2[∑i=12‍zxi2(ki1ki2)M(ki1ki2)+2zx1zx2(k11k12)M(k21k22)],tr⁡[NH]=-zx12z2(k11k12)(σ12σ12)-zx22z2(k21k22)(σ12σ22)-zx1zx2z2((k11k12)(σ12σ22)+(k21k22)(σ12σ12))+L(zx1x1,zx1x2,zx2x2),
where the function L is defined in (33). Substituting these expressions into (29), we find that the differential equation satisfied by z(x1,x2) will indeed be the linear equation (32) if
(36)0=(k11k12)M(k11k12)+(k11k12)(σ12σ12),0=(k21k22)M(k21k22)+(k21k22)(σ12σ22),0=2(k11k12)M(k21k22)+(k11k12)(σ12σ22)+(k21k22)(σ12σ12).
Now, the three equations in this system hold simultaneously if(37)2KMK′+KN+NK′=(0000):=02.
But this relation is verified for all matrices B, Q, and N that satisfy the conditions mentioned above. Notice, however, that from (30) we deduce two expressions for the value function F(x1,x2). For the transformation to be valid, these two expressions must of course be compatible, which yields (34).Remarks 4.
(i) When (23) holds, so that we can apply Whittle’s theorem, the matrix K is given by
(38)K=-1αI2,
where I2 denotes the identity matrix of order 2. The condition in (34) becomes
(39)∫‍1zzx1dx1+ψ(x2)=∫‍1zzx2dx2+ϕ(x1);
that is,
(40)ln⁡[z(x1,x2)]+ψ(x2)=ln⁡[z(x1,x2)]+ϕ(x1).
We can obviously choose ψ(x2)=ϕ(x1)≡c0, a constant. Therefore, this condition is always satisfied when Whittle’s theorem can be used. Moreover, it is clearly more likely to satisfy (23) [and (34)] when all the matrices are constant.
(ii) The functionz(x1,x2) must be strictly positive. In one dimension, z(x) is indeed strictly positive since, making use of (15), it can be expressed as an exponential function. When (23) (in two dimensions) is satisfied, it is also easy to prove that z(x1,x2) is strictly positive.
(iii) The simplest problems that can be considered are such that the two controlled processes defined by (25) are independent, so that σ12=0, and all the matrices are constant, which implies that kij is also a constant, for i,j=1,2. The linear function L then reduces to
(41)L(zx1x1,zx1x2,zx2x2)=∑i=12‍∑j=12‍σi2zxjxikji.
(iv) Proposition3 does not give us the function G from which one deduces the optimal control. Similarly to Whittle’s theorem, it rather simplifies the optimal control problem. Indeed, it is generally easier to solve a linear than a nonlinear differential equation.
Since the proof is a simple extension of that of Proposition3, we can state the following corollary.Corollary 5.
In then-dimensional case, the matrix Riccati equation [see (21)]
(42)λ+G′(x)f(x)-12G′(x)B(x)Q-1(x)B′(x)G(x)+12tr⁡[NGx(x)]=0
is transformed into a linear partial differential equation for the function z(x1,…,xn) defined through
(43)G′=1z(zx1⋯zxn)K,
where
(44)K=(k11⋯k1n⋮⋮⋮kn1⋯knn):=-NM-1,
provided that the n expressions that we deduce from (43) for the value function F(x) are compatible.Remarks 6.
(i) For the sake of brevity, we did not give the linear equation satisfied byz(x1,…,xn), but it is a simple matter to derive it.
(ii) The largern is, the more difficult it should be to obtain n compatible expressions for the value function. Nevertheless, the result is clearly an improvement over Whittle’s theorem.
(iii) If we define insteadz(x1,…,xn) through the equation
(45)G′=1z(zx1⋯zxn)P,
with
(46)P:=-(M′)-1N,
then we find that z(x1,…,xn) satisfies a linear partial differential equation if
(47)2PMP′+PN+NP′=02,
which does not always hold true. Therefore, the transformation that we used is more appropriate.


## 3. Explicit Solution to a Particular Problem

In this section, we will make use of Proposition3 to help us solve a particular LQG homing problem in two dimensions.Assume that(48)(dX1(t)dX2(t))=(00)dt+(1002)(u1u2)dt+(1001)1/2(dW1(t)dW2(t))
and let
(49)J(x1,x2)=∫0τ(x1,x2)‍[12(u1u2)(1114)(u1u2)+λ]dt,
where
(50)τ(x1,x2)=inf⁡⁡{∣X1(0)=x1,X2(0)=x2t>0:X1(t)+X2(t)=0ord∣X1(0)=x1,X2(0)=x2},
with 0<x1+x2<d. We assume that λ is positive.We calculate(51)M:=BQ-1B′=(43-23-2343),K:=-NM-1=-(112121).Remark 7.
It is important to notice that, in this particular problem, the relation in (23) doesnot hold, since the matrix K is not proportional to the identity matrix of order 2. Hence, we could not appeal to Whittle’s theorem to linearize the differential equation satisfied by the value function.
From Proposition3, we deduce that the function z(x1,x2) here satisfies the linear partial differential equation
(52)λz-12(zx1x1+zx1x2+zx2x2)=0.
To solve this differential equation, we will use the method of similarity solutions. That is, we assume that the function z(x1,x2) can actually be written as
(53)z(x1,x2)=ρ(ξ),
with ξ:=x1+x2. Equation (52) is then transformed into the second-order ordinary differential equation
(54)-32ρ′′(ξ)+λρ(ξ)=0,
whose general solution can be written as
(55)ρ(ξ)=c1e2λ/3ξ+c2e-2λ/3ξ,
so that
(56)z(x1,x2)=c1e2λ/3(x1+x2)+c2e-2λ/3(x1+x2).
Let us takeλ=3/2, for simplicity. We then deduce from the preceding equation that
(57)zx1z=zx2z=c1ex1+x2-c2e-(x1+x2)c1e(x1+x2)+c2e-(x1+x2).
We can now compare the two expressions that we obtain for the value function F(x1,x2). First, we have
(58)Fx1(x1,x2)=G1(x1,x2)=-1z(x1,x2)(zx1(x1,x2)+12zx2(x1,x2))=-32zx1(x1,x2)z(x1,x2),
which implies that
(59)F(x1,x2)=-32ln⁡[z(x1,x2)]+ψ(x2).
Similarly, we obtain that(60)Fx2(x1,x2)=G2(x1,x2)=-32zx2(x1,x2)z(x1,x2),
so that
(61)F(x1,x2)=-32ln⁡[z(x1,x2)]+ϕ(x1).
Since z(x1,x2) depends on both x1 and x2 in our problem, we conclude that we must set ϕ(x1)=ψ(x2)≡c0.
Next, making use of the boundary conditionsF(x1,x2)=0 if x1+x2=0 or d, we can write that
(62)0=-32ln⁡(c1+c2)+c0,0=-32ln⁡(c1ed+c2e-d)+c0.
If c0=0, we find that
(63)c1=1-e-ded-e-d,c2=ed-1ed-e-d.
Hence,
(64)F(x1,x2)=-32ln⁡{1ed-e-d[(1-e-d)ex1+x2+(e-d-1)e-(x1+x2)]}.
Whenc0≠0, we deduce from (62) that
(65)c2=c1ed-11-e-d.
Thus, we obtain the following expression for the value function:
(66)F(x1,x2)=-32ln⁡{c1[ex1+x2+ed-11-e-de-(x1+x2)]}+c0,
which is valid for 0≤x1+x2≤d.
Now, remember that we do not need to determineF(x1,x2) explicitly to obtain the optimal controls u1*(x1,x2) and u2*(x1,x2). Only Fxi(x1,x2)=Gi(x1,x2) is needed. Here, we compute
(67)G1(x1,x2)=G2(x1,x2)=-32(1-e-d)ex1+x2-(ed-1)e-(x1+x2)(1-e-d)ex1+x2+(ed-1)e-(x1+x2)
for any constant c0 (and any c1). It follows that the optimal controls are given by [see (20)]
(68)(u1*(x1,x2)u2*(x1,x2))=-Q-1B′(G1(x1,x2)G2(x1,x2))=-13(4-2-12)(G1(x1,x2)G2(x1,x2))=-13(2G1(x1,x2)G1(x1,x2)).Remarks 8.
(i) Ifd tends to infinity, we find that Gi(x1,x2) tends to 3/2, for i=1,2. Notice that the optimal controls u1*(x1,x2) and u1*(x1,x2) are then negative, which is logical because λ>0 and the process X1(t)+X2(t) cannot reach the line X1(t)+X2(t)=d when d tends to infinity. Therefore, the optimizer tries to hit the line X1(t)+X2(t)=0 as soon as possible, but taking the quadratic control costs into account.
(ii) The reason whyu1*(x1,x2)=2u2*(x1,x2) is the fact that q11=1, while q22=4. Thus, controlling X2(t) is more expensive than controlling X1(t). Moreover, b22=2b11.


## 4. Conclusion

In this paper, we have generalized a theorem due to Whittle that sometimes enables us to linearize the matrix Riccati equation satisfied by the derivative of the value function in the so-called LQG homing problems. In Section3, we were able to use our result to obtain the explicit solution to such a problem in two dimensions, for which Whittle’s theorem did not apply. Solving this type of problem explicitly is usually very difficult. Furthermore, notice that the optimal control ui*(x1,x2) that we obtained depends on x1 and x2, for i=1,2, rather than being constant. Therefore, the solution was not evident at all.One cannot expect to be able to linearize the matrix Riccati equation (21) in all cases. Whittle’s theorem depends on a quite restrictive condition. Here, we showed that there are other cases than the ones to which Whittle’s result applies. We saw in Proposition 3 that the transformation that we propose will linearize (21) (with n=2), provided that (34) is satisfied. Since, as we mentioned above, our transformation will work every time Whittle’s theorem applies, we can at least claim that we have improved Whittle’s result.Apart from the case when we can transform the stochastic optimal control problem into a purely probabilistic problem, few particular cases have been solved so far in two or more dimensions. The authors (see Lefebvre and Zitouni [6]) treated the general case in one dimension. They used symmetry, when possible, to obtain the exact optimal control, and they proposed a technique that yields a very good approximation to this optimal control when symmetry arguments cannot be used. It would be interesting to generalize these results in the n-dimensional case.Finally, we could try to apply the technique developed by Grasselli and Tebaldi [4], as well as other techniques proposed by various authors, to linearize the matrix Riccati equation in the case of LQG homing problems.



---

*Source: 741390-2014-04-06.xml*