# LQG Homing in a Finite Time Interval

Authors: Mario Lefebvre
Journal: Journal of Control Science and Engineering
 (2011)

---

## Abstract

LetX(t) be a controlled one-dimensional diffusion process having constant infinitesimal variance. We consider the problem of optimally controlling X(t) until time T(x)=min{T1(x),t1}, where T1(x) is the first-passage time of the process to a given boundary and t1 is a fixed constant. The optimal control is obtained explicitly in the particular case when X(t) is a controlled Wiener process.


---

## Body

## 1. Introduction

Let{X(t),t≥0} be the one-dimensional controlled diffusion process defined by the stochastic differential equation(1)dX(t)=a[X(t)]dt+b0u(t)dt+σ0dB(t),
where a(·) is a real function, u(t) is the control variable, b0≠0, and σ0>0 are constants and {B(t),t≥0} is a standard Brownian motion. We define the first-passage time(2)T1(x)=inf⁡{t>0:X(t)=d∣X(0)=x},
where x<d, and the random variable(3)T(x)=min⁡{T1(x),t1},
where t1>0 is a constant.Next, we consider the cost criterion(4)J(x)=∫0T(x)12q0u2(t)dt+kln⁡[T(x)+1],
where q0>0 and k≠0 are constants. We want to find the control u* that minimizes the expected value of J(x). This type of problem is a special case of the ones that Whittle [1, page 289] termed LQG homing. Notice that if the constant k is negative, then the optimizer tries to maximize the survival time of the process in the interval (0,d), taking the quadratic control costs into account. LQG homing problems have been treated by various authors; see Kuhn [2], Lefebvre [3], and Makasu [4]. Kuhn and Makasu used a risk-sensitive cost criterion (see also Whittle [5, page 222]).In the general formulation given by Whittle,{X(t),t≥0} is an n-dimensional process and the random variable T(x) is the moment of first entry of the joint variable (X(t),t) into a stopping set D⊂ℝn×(0,∞). However, in practice, it is very difficult to obtain explicit solutions to problems in two or more dimensions (except in special instances). Moreover, in the papers published so far on homing problems, the hitting time T(x) was defined only in terms of X(t). Here, we consider the case when the optimizer stops controlling the diffusion process at most at time t1.Using a theorem in Whittle [1], we can state that the optimal control u* can be expressed as follows:(5)u*=σ02b0G′(x)G(x),
where(6)G(x)=E[exp⁡{-b02q0σ02kln⁡[τ(x)+1]}].
In the above formula, τ(x) is a random variable defined by(7)τ(x)=min⁡{τ1(x),t1},
with(8)τ1(x)=inf⁡{t>0:ξ(t)=d∣ξ(0)=x},
and {ξ(t),t≥0} is the uncontrolled process that satisfies the stochastic differential equation(9)dξ(t)=a[ξ(t)]dt+σ0dB(t).
That is, τ(x) is the random variable that corresponds to T(x) for the diffusion process obtained by setting u(t)=0 in (1).Hence, the optimal control problem is reduced to the computation of the mathematical expectationG(x). Actually, for this result to hold, we must have P[τ(x)<∞]=1. However, in our case this condition is trivially satisfied because τ(x)≤t1.In Section2, we will obtain an explicit solution u* in the case when a[X(t)]≡μ>0, so that(10)dX(t)=μdt+b0u(t)dt+σ0dB(t).
Notice that the uncontrolled process {ξ(t),t≥0} is then a Wiener process with drift μ and diffusion parameter σ02. Furthermore, we will choose the constant k=-q0σ02/b02. With this choice, the mathematical expectation G(x) simplifies to(11)G(x)=E[exp⁡{ln⁡[τ(x)+1]}]=1+E[τ(x)].


## 2. Optimal Control of a Wiener Process

Letm1(x) denote the expected value of the first-passage time τ1(x). In the case of the Wiener process defined by(12)dξ(t)=μdt+σ0dB(t)
the function m1(x) satisfies the ordinary differential equation(13)12σ02m1′′(x)+μm1′(x)=-1,
and is such that m1(x)=0 if x=d. We find (see Lefebvre [6, page 220]) that(14)m1(x)=d-xμ.
Therefore, in the case when t1 tends to infinity, the function G(x) is given by(15)G(x)=1+d-xμ.
It follows from (5) that(16)ut1=∞*=-σ02b01μ+d-x.Now, to obtain the expected value of the random variableτ(x), we can condition on τ1(x):(17)E[τ(x)]=E[τ(x)∣τ1(x)≤t1]P[τ1(x)≤t1]+E[τ(x)∣τ1(x)>t1]P[τ1(x)>t1].
We may write that (18)E[τ(x)∣τ1(x)>t1]=t1.
Moreover, because the conditional probability density function of τ1(x), given that τ1(x)≤t1, is given by(19)fτ1(x)(t∣τ1(x)≤t1)=fτ1(x)(t)P[τ1(x)≤t1]for0<t≤t1,
we have:(20)E[τ(x)]=∫0t1tfτ1(x)(t)dt+t1P[τ1(x)>t1].The functionfτ1(x)(t) is known to be (see Lefebvre [6, page 219])(21)fτ1(x)(t)=d-x2πσ02t3exp⁡{-(d-x-μt)22σ02t}fort>0.
Making use of this formula, we can obtain an explicit expression for the mathematical expectation E[τ(x)] and hence, for the optimal control u*.To illustrate the results, we computed (numerically) the optimal control whenb0=q0=σ0=μ=1 and d=t1=5. Looking at Figure 1, we see that the optimal control u* tends to zero much faster than ut1=∞*=1/(x-6) as x tends to -∞. However, for x close to the boundary at d=5, the two functions are similar.Figure 1
Optimal controlsut1=∞* (dotted line) and u* (solid line) when x∈(-10,5), b0=q0=σ0=μ=1, and d=t1=5.Next, to see the effect of the constantt1 on the optimal control, we computed the value of u* when x=0 and t1 varies from 0 to 20. This value is compared to ut1=∞*=-1/6 in Figure 2. When t1 decreases to 0, so does u*, as it should be. For t1≥15 (approximately), we have u*≃ut1=∞*.Figure 2
Optimal controlsut1=∞*=-1/6 (dotted line) and u* (solid line) when x=0, b0=q0=σ0=μ=1, d=5, and t1∈(0,20).Finally, in Figure3, we show the optimal control u* when d=15 and t1=5. Because E[τ1(x)]=15-x (see above), when x≤5 (approximately), it is very unlikely that the uncontrolled process will hit the boundary at d=15 before time t1=5. Therefore, the optimal control is close to 0. Notice that limd→∞u*=0 (for a finite value of x). Indeed, we can write that limd→∞P[τ1(x)>t1]=1. Hence, we deduce from (17) that E[τ(x)]=t1, which implies that G(x)=1+t1, and thus ud=∞*=0.Figure 3
Optimal controlu* when x∈(5,15), b0=q0=σ0=μ=1, d=15, and t1=5.


## 3. Conclusion

We have considered LQG homing problems for which the optimizer controls the diffusion process in the time interval(0,T(x)], where the random variable T(x) is smaller than or equal to a fixed constant t1. Moreover, the termination cost function was chosen so that the optimal control was expressed in terms of the expected value of a first-passage time for the corresponding uncontrolled process.An application of this type of problem is the following: suppose thatX(t) denotes the wear of a machine. The optimizer wants to maximize the lifetime of the machine. However, it is natural to assume that the machine will be replaced after a certain time, even if it is still in working order, because it might become obsolete.To obtain a more realistic model for the wear of a device, we could use a degenerate two-dimensional diffusion process, as in Lefebvre [7]. The most difficult problem would then be to compute the probability density function of the first-passage time τ1.



---

*Source: 561347-2011-05-14.xml*