# Some Results on Bellman Equations of Optimal Production Control in a Stochastic Manufacturing System

Authors: Azizul Baten; Anton Abdulbasah Kamil
Journal: Journal of Probability and Statistics
 (2009)

---

## Abstract

The paper studies the production inventory problem of minimizing
the expected discounted present value of production cost control in a manufacturing system with degenerate stochastic demand. We establish the existence of a
unique solution of the Hamilton-Jacobi-Bellman (HJB) equations associated with
this problem. The optimal control is given by a solution to the corresponding HJB
equation.


---

## Body

## 1. Introduction

Many manufacturing enterprisers use a production inventory system to manage fluctuations in consumer demand for the product. Such a system consists of a manufacturing plant and a finished goods warehouse to store those products which are manufactured but not immediately sold. The advantages of having products in inventory are as follows: first, they are immediately available to meet demand; second, by using the warehouse to store excess production during low demand periods to be available for sale during high demand periods. This usually permits the use of a smaller manufacturing plant than would otherwise be necessary, and also reduces the difficulties of managing the system.We are concerned with the optimization problem to minimize the expected discounted cost control of production planning in a manufacturing systems with degenerate stochastic demand:(1.1)J(p)=E[∫0∞e-ρt{h(xt)+pt2}dt]
subject to the dynamics of the state equation which says that the inventory at time t is increased by the production rate and decreased by the demand rate can be written according to
(1.2)dxt=(pt-yt)dt,x0=x,x>0,0≤pmin≤pmax,
and the demand equation with the production rate is described by the Brownian motion
(1.3)dyt=Aytdt+σytdwt,y0=y,y>0,
in the class 𝒫 of admissible controls of production processes pt with nonnegative constant
(1.4)pt≥0
defined on a complete probability space (Ω,ℱ,P) endowed with the natural filtration ℱt generated by σ(ws,s≤t) carrying a one-dimensional standard Brownian motion wt, xt is the inventory level for production rate at time t (state variable), yt is the demand rate at time t, pt is the production rate at time t (control variable), ρ>0 is the constant nonnegative discount rate, A is the nonzero constant, σ is nonzero constant diffusion coefficient, x0 is the initial value of inventory level, and  y0 is the initial value of demand rate.This optimization control problem of production planning in manufacturing systems has been studied by many authors like Fleming et al. [1], Sethi and Zhang [2], Sprzeuzkouski [3], Hwang et al.  [4], Hartl and Sethi [5], and Feichtinger and Hartl [6].  The Bellman equation associated with production inventory control problem is quite different from them and it is treated by Bensoussan et al. [7] for the one-dimensional manufacturing systems with the unbounded control region. Generally speaking, the similar type of linear control problems has been investigated for the stochastic deferential systems with invariant measures like Bensoussan [8], and Borkar [9]. The works of Bensoussan and Frehse [10] Da Prato and Ichikawa [11] on the Bellman equation of ergodic control without convex and polynomial growth hypothesis and the linear quadratic case are done for the linear ergodic control problem. This type of optimization problem has been studied also by Morimoto and Kawaguchi [12] for renewable resources as well as Baten and Sobhan Miah [13] for one-sector neoclassical growth model with the CES function. The optimality can be shown by an extension of the results given in Fujita and Morimoto [14], and for another setting of optimal control in manufacturing systems they are available in Morimoto and Okada [15] and Sethi et al. [16]. These papers treat the cases with bounded control regions. On the contrary, our control region is unbounded as in (1.4).The purpose of the paper is to give an optimal production cost control by an existence unique solution associated with the two-dimensional HJB equation. We apply the technique of dynamic programming principle [17] for the Riccati-based solution of the reduced (one-dimensional) HJB equation corresponding to production inventory control problem. This paper is organized as follows. In Section 2 by the principle of optimality Bellman [17], we have obtained the HJB equation and then the two-dimensional HJB equation has been reduced to one-dimensional second-order differential equation. We have derived the dynamics of inventory-demand ratio that evolves according to stochastic neoclassical differential equation through Itô’s lemma. We have finally found the Riccati-based solution of production inventory control problem that is satisfied by the value function of this optimization problem. In Section 3 we have established the properties of the value function and have shown the existence of an unique solution associated with the reduced (one-dimensional) Hamilton-Jacobi-Bellman (HJB) equation. Finally in Section 4 we present an application to production control of optimization problem (1.1) subject to (1.2) and (1.3).


## 2. Riccati-Based Solution of Hamilton-Jacobi Bellman Equation

### 2.1. The Hamilton-Jacobi-Bellman Equation

Supposeu(x,y,t):Rn×Rn×R→R is a function whose value is the minimum value of the objective function of the production inventory control problem for the manufacturing system given that we start it at time t in state x, and y. That is,
(2.1)u(x,y,t)=infpJ(p),
where the value function u is finite valued and twice continuously differentiable on (0,∞). We initially assume that u(x,y,t) exists for all x, y, and t in the relevant ranges.Since (1.2) and (1.3) is a scalar equation, the subscript t here means only time t. Thus, x and y will not cause any confusion and, at the same time, will eliminate the need of writing many parentheses. Thus, dwt is a scalar.To solve the problem defined by (1.1), (1.2), and (1.3), let u(x,y,t), known as the value function, be the expected value of the objective function (1.1) from t to infinity, when an optimal policy is followed from t to infinity, given xt=x,yt=y. Then by the principle of optimality [17],
(2.2)u(x,y,t)=minp[{h(xt)+pt2}dt+u(x+dxt,y+dyt,t+dt)].
We assume that u(x,y,t) is a continuously differentiable of its arguments. By Taylor’s expansion, we have
(2.3)u(x+dxt,y+dyt,t+dt)=-ρu(x,y,t)dt+uydyt+uxdxt+12uyy(dyt)2+higherorderterms.
From (1.2), we can formally write
(2.4)(dxt)2=(p)2(dt)2+(y)2(dt)2-2py(dt)2,(2.5)(dyt)2=(Ay)2(dt)2+(σy)2(dwt)2+2(Ay)(σy)dwtdt,(2.6)dxtdt=p(dt)2-y(dt)2,(2.7)dytdt=(Ay)(dt)2+(σy)dwtdt.
The exact meaning of these expressions comes from the theory of stochastic calculus; Arnold [18, chapter 5] and Karatzas and Shreve [19]. For our purposes, it is sufficient to know the multiplication rules of the stochastic calculus:
(2.8)(dwt)2=dt,dwtdt=0,dt2=0.
Substitute (2.3) into (2.2) and use (2.4), (2.5) (2.6), (2.7), and (2.8) to obtain
(2.9)u=min[-ρu(x,y,t)dt+12σ2y2uyydt+Ayuydt-yuxdt+{p2+pux}dt+h(x)dt+∘(dt)].
Note that we have suppressed the arguments of the functions involved in (2.3).Canceling the termu on both sides of (2.9), dividing the remainder by dt, and letting t→0, we obtain the dynamic programming partial differential equation or Hamilton-Jacobi-Bellman equation
(2.10)-ρu(x,y)+12σ2y2uyy+Ayuy-yux+F*(ux)+h(x)=0,u(0,y)=0,x,y>0,
where F*(x) is the Legendre transform of F(x), that is, F*(x)=minp>0{p2+px}=-x2/4 and ux, uy, uxx, uyy are partial derivatives of u(x,y,t) with respect to x and y.


### 2.2. A Reduction to 1-Dimensional Case

In this subsection, the general (two-dimensional) HJB equation has been reduced to a one-dimensional second-order differential equation. From the two-dimensional state space form (one statex for inventory level and the other state y for demand rate), it has been reduced to one-dimensional form for (z=x/y) the ratio of inventory to demand.There exists av∈C(0,∞) such that u(x,y)=y2v(x/y),y>0. Since ux=yv′(x/y),uy=2yv(x/y)-xv′(x/y),uyy=2v(x/y)-2(x/y)v′(x/y)+(x/y)2v”(x/y). Setting z=x/y and substituting these in (2.10), we have
(2.11)-ρv(z)+12σ2[2v(z)-2v′(z)z+z2v”(z)]+2Av(z)-Azv′(z)-v′(z)+minp≥0(p2+pyv′(z))+h(z)=0.
Since
(2.12)minp≥0(p2+pyv′(z))=y2minp≥0((py)2+(py)v′(z))=y2minq≥0(q2+qv′(z)),minq≥0(q2+qv′(z)-v′(z))=mink+1≥0((k+1)2+kv′(z)).
Then the HJB equation (2.11) becomes
(2.13)-ρ̃v(z)+12σ2z2v”(z)+Ãzv′(z)+mink≥-1((k+1)2+kv′(z))+h(z)=0,v(0)=0,z>0,
where ρ̃=-ρ+σ2+2A,Ã=-(A+σ2), and F*(z) is the Legendre transform of F(z), that is, F*(z)=min(k+1)≥0{(k+1)2+kz}=-z2/4-z.The main feature of the HJB equation (2.13) is the vanishing of the coefficient of uxx for x=0 in partial differential equation terminology, then the equation is degenerate elliptic. Generally speaking, the difficulty stems from the degeneracy in the second-order term of the HJB equation (2.13).


### 2.3. Value Function

Let us consider the minimum value of the payoff function is a function of this initial point. The value function can be defined as a function whose value is the minimum value of the objective function of the production inventory control problem (1.1) for the manufacturing system, that is,
(2.14)V(z)=infE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt]=infJ̃(kt);(k+1)≥0.
The value function V(z) is a solution to the reduced (one-dimensional) HJB equation (2.13) and the solution of this HJB equation is used to test controller for optimality or perhaps to construct a feedback controller. Again the HJB equation (2.13) arises in the production control problem (1.1), (1.2), (1.3) with constraint
(2.15)0≤pt≤kt,∀t≥0.


### 2.4. Stochastic Neoclassical Differential Equation for Dynamics of Inventory-Demand Ratio

As in the certainty optimal production control model, the dynamics of the state equation of inventory level (1.2) can be reduced to a one-dimensional process by working in intensive (per capita) variables. Define
(2.16)zt≡xtyt,inventory-demand  ratio,kt≡ptyt,per  capita  production.
To determine the stochastic differential for the inventory-demand ratio, z≡x/y, we apply Itô's lemma as follows:
(2.17)z=xy≡G(y,t),∂G∂y=-xy2=-zy,∂2G∂y2=2xy3=2zy2,∂G∂t=ẋy=(p-y)y=py-1=kt-1.
From Itô's lemma,
(2.18)dz=∂G∂ydy+∂G∂tdt+12∂2G∂y2(dy)2.
From (1.3), we have that (dy)2=σ2y2dt. Substituting the above expressions into (2.18), we have that the dynamics of zt to be the inventory-demand ratio at time t which evolves according to the stochastic neoclassical differential equation for demand
(2.19)dzt=(-ztyt)(Aytdt+σytdwt)+(kt-1)dt+122ztyt2σ2yt2dt=[-Azt+(kt-1)+σ2zt]dt-σztdwt=[(-A+σ2)zt+(kt-1)]dt-σztdwt≤[Ãzt+kt]dt-σztdwt,z0=z,z>0.


### 2.5. Riccati-Based Solution

This subsection deals with the Riccati-based solution of the reduced one-dimensional HJB equation (2.13) corresponding to the production inventory control problem (2.14) subject to (2.19) using the dynamic programming principle [17].To find the Riccati-based solution of HJB equation (2.13), we refer to Da Prato [20] and Da Prato and Ichikawa [11] for the degenerate linear control problems related to Riccati equation in case of convex function like h(z)=z2.By taking the derivative of (2.13) with respect to k and setting it to zero, we can minimize the expression inside the bracket of (2.13) (i.e., F*(v′(z))=mink≥-1((k+1)2+kv′(z))) with respect to k. This procedure yields
(2.20)k*=-v′(z)2-1.
Substituting (2.20) into (2.13) yields the equation
(2.21)-ρ̃v(z)+12σ2z2v”(z)+Ãzv′(z)-(v′(z))24-v′(z)+z2=0
known as the HJB equation. This is a partial differential equation which has a solution form
(2.22)v(z)=a(t)z2(t).
Then
(2.23)v′(z)=2a(t)z(t),v”(z)=2a(t).
Substituting (2.22) and (2.23) into (2.21) yields
(2.24)(1-ρ̃a+σ2a-a2+2aÃ)z2-2az=0.
Since (2.24) must hold for any value of z, we must have
(2.25)a2-a(2Ã+σ2+ρ̃)-1=0,
called a Riccati equation from which we obtain
(2.26)a=-(2Ã+σ2+ρ̃)±(2Ã+σ2+ρ̃)2+42=K1(say).
So, (2.22) is a solution form of (2.21).


## 2.1. The Hamilton-Jacobi-Bellman Equation

Supposeu(x,y,t):Rn×Rn×R→R is a function whose value is the minimum value of the objective function of the production inventory control problem for the manufacturing system given that we start it at time t in state x, and y. That is,
(2.1)u(x,y,t)=infpJ(p),
where the value function u is finite valued and twice continuously differentiable on (0,∞). We initially assume that u(x,y,t) exists for all x, y, and t in the relevant ranges.Since (1.2) and (1.3) is a scalar equation, the subscript t here means only time t. Thus, x and y will not cause any confusion and, at the same time, will eliminate the need of writing many parentheses. Thus, dwt is a scalar.To solve the problem defined by (1.1), (1.2), and (1.3), let u(x,y,t), known as the value function, be the expected value of the objective function (1.1) from t to infinity, when an optimal policy is followed from t to infinity, given xt=x,yt=y. Then by the principle of optimality [17],
(2.2)u(x,y,t)=minp[{h(xt)+pt2}dt+u(x+dxt,y+dyt,t+dt)].
We assume that u(x,y,t) is a continuously differentiable of its arguments. By Taylor’s expansion, we have
(2.3)u(x+dxt,y+dyt,t+dt)=-ρu(x,y,t)dt+uydyt+uxdxt+12uyy(dyt)2+higherorderterms.
From (1.2), we can formally write
(2.4)(dxt)2=(p)2(dt)2+(y)2(dt)2-2py(dt)2,(2.5)(dyt)2=(Ay)2(dt)2+(σy)2(dwt)2+2(Ay)(σy)dwtdt,(2.6)dxtdt=p(dt)2-y(dt)2,(2.7)dytdt=(Ay)(dt)2+(σy)dwtdt.
The exact meaning of these expressions comes from the theory of stochastic calculus; Arnold [18, chapter 5] and Karatzas and Shreve [19]. For our purposes, it is sufficient to know the multiplication rules of the stochastic calculus:
(2.8)(dwt)2=dt,dwtdt=0,dt2=0.
Substitute (2.3) into (2.2) and use (2.4), (2.5) (2.6), (2.7), and (2.8) to obtain
(2.9)u=min[-ρu(x,y,t)dt+12σ2y2uyydt+Ayuydt-yuxdt+{p2+pux}dt+h(x)dt+∘(dt)].
Note that we have suppressed the arguments of the functions involved in (2.3).Canceling the termu on both sides of (2.9), dividing the remainder by dt, and letting t→0, we obtain the dynamic programming partial differential equation or Hamilton-Jacobi-Bellman equation
(2.10)-ρu(x,y)+12σ2y2uyy+Ayuy-yux+F*(ux)+h(x)=0,u(0,y)=0,x,y>0,
where F*(x) is the Legendre transform of F(x), that is, F*(x)=minp>0{p2+px}=-x2/4 and ux, uy, uxx, uyy are partial derivatives of u(x,y,t) with respect to x and y.


## 2.2. A Reduction to 1-Dimensional Case

In this subsection, the general (two-dimensional) HJB equation has been reduced to a one-dimensional second-order differential equation. From the two-dimensional state space form (one statex for inventory level and the other state y for demand rate), it has been reduced to one-dimensional form for (z=x/y) the ratio of inventory to demand.There exists av∈C(0,∞) such that u(x,y)=y2v(x/y),y>0. Since ux=yv′(x/y),uy=2yv(x/y)-xv′(x/y),uyy=2v(x/y)-2(x/y)v′(x/y)+(x/y)2v”(x/y). Setting z=x/y and substituting these in (2.10), we have
(2.11)-ρv(z)+12σ2[2v(z)-2v′(z)z+z2v”(z)]+2Av(z)-Azv′(z)-v′(z)+minp≥0(p2+pyv′(z))+h(z)=0.
Since
(2.12)minp≥0(p2+pyv′(z))=y2minp≥0((py)2+(py)v′(z))=y2minq≥0(q2+qv′(z)),minq≥0(q2+qv′(z)-v′(z))=mink+1≥0((k+1)2+kv′(z)).
Then the HJB equation (2.11) becomes
(2.13)-ρ̃v(z)+12σ2z2v”(z)+Ãzv′(z)+mink≥-1((k+1)2+kv′(z))+h(z)=0,v(0)=0,z>0,
where ρ̃=-ρ+σ2+2A,Ã=-(A+σ2), and F*(z) is the Legendre transform of F(z), that is, F*(z)=min(k+1)≥0{(k+1)2+kz}=-z2/4-z.The main feature of the HJB equation (2.13) is the vanishing of the coefficient of uxx for x=0 in partial differential equation terminology, then the equation is degenerate elliptic. Generally speaking, the difficulty stems from the degeneracy in the second-order term of the HJB equation (2.13).


## 2.3. Value Function

Let us consider the minimum value of the payoff function is a function of this initial point. The value function can be defined as a function whose value is the minimum value of the objective function of the production inventory control problem (1.1) for the manufacturing system, that is,
(2.14)V(z)=infE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt]=infJ̃(kt);(k+1)≥0.
The value function V(z) is a solution to the reduced (one-dimensional) HJB equation (2.13) and the solution of this HJB equation is used to test controller for optimality or perhaps to construct a feedback controller. Again the HJB equation (2.13) arises in the production control problem (1.1), (1.2), (1.3) with constraint
(2.15)0≤pt≤kt,∀t≥0.


## 2.4. Stochastic Neoclassical Differential Equation for Dynamics of Inventory-Demand Ratio

As in the certainty optimal production control model, the dynamics of the state equation of inventory level (1.2) can be reduced to a one-dimensional process by working in intensive (per capita) variables. Define
(2.16)zt≡xtyt,inventory-demand  ratio,kt≡ptyt,per  capita  production.
To determine the stochastic differential for the inventory-demand ratio, z≡x/y, we apply Itô's lemma as follows:
(2.17)z=xy≡G(y,t),∂G∂y=-xy2=-zy,∂2G∂y2=2xy3=2zy2,∂G∂t=ẋy=(p-y)y=py-1=kt-1.
From Itô's lemma,
(2.18)dz=∂G∂ydy+∂G∂tdt+12∂2G∂y2(dy)2.
From (1.3), we have that (dy)2=σ2y2dt. Substituting the above expressions into (2.18), we have that the dynamics of zt to be the inventory-demand ratio at time t which evolves according to the stochastic neoclassical differential equation for demand
(2.19)dzt=(-ztyt)(Aytdt+σytdwt)+(kt-1)dt+122ztyt2σ2yt2dt=[-Azt+(kt-1)+σ2zt]dt-σztdwt=[(-A+σ2)zt+(kt-1)]dt-σztdwt≤[Ãzt+kt]dt-σztdwt,z0=z,z>0.


## 2.5. Riccati-Based Solution

This subsection deals with the Riccati-based solution of the reduced one-dimensional HJB equation (2.13) corresponding to the production inventory control problem (2.14) subject to (2.19) using the dynamic programming principle [17].To find the Riccati-based solution of HJB equation (2.13), we refer to Da Prato [20] and Da Prato and Ichikawa [11] for the degenerate linear control problems related to Riccati equation in case of convex function like h(z)=z2.By taking the derivative of (2.13) with respect to k and setting it to zero, we can minimize the expression inside the bracket of (2.13) (i.e., F*(v′(z))=mink≥-1((k+1)2+kv′(z))) with respect to k. This procedure yields
(2.20)k*=-v′(z)2-1.
Substituting (2.20) into (2.13) yields the equation
(2.21)-ρ̃v(z)+12σ2z2v”(z)+Ãzv′(z)-(v′(z))24-v′(z)+z2=0
known as the HJB equation. This is a partial differential equation which has a solution form
(2.22)v(z)=a(t)z2(t).
Then
(2.23)v′(z)=2a(t)z(t),v”(z)=2a(t).
Substituting (2.22) and (2.23) into (2.21) yields
(2.24)(1-ρ̃a+σ2a-a2+2aÃ)z2-2az=0.
Since (2.24) must hold for any value of z, we must have
(2.25)a2-a(2Ã+σ2+ρ̃)-1=0,
called a Riccati equation from which we obtain
(2.26)a=-(2Ã+σ2+ρ̃)±(2Ã+σ2+ρ̃)2+42=K1(say).
So, (2.22) is a solution form of (2.21).


## 3. Bellman Equations for Discounted Cost Control

### 3.1. Existence and Uniqueness

To solve the Bellman equation (2.13) let us consider this HJB equation associated with the discounted production control problem in the following form:
(3.1)-ρ̃v(z)+12σ2z2v”(z)+Ãzv′(z)+F(v′(z))+h(z)=0,0<ρ̃<1,
where
(3.2)F(z)={-z24-zifz≤0,-zifz≥0.
We make the following assumptions:
(3.3)h:continuous  function  onR,h:non-negative,  convex  onR,h satisfies the polynomial growth condition such that
(3.4)0≤h(z)≤K(1+|z|n),z∈R,
for some K>0,n≥2.In order to ensure the integrability ofJ(p), we assume that
(3.5)-ρ̃+2nÃ+n(2n-1)σ2<0.
This condition (3.5) is needed for the integrability of zt or J(p). Under (3.5), we have Lemmas 3.1, 3.2, and Theorem 3.5, which ensures the finiteness of J(p) and hence the finiteness of an existence unique solution of HJB equation of v.First we have established the properties of the value function of the optimal control problem.Lemma 3.1.
Under (3.5) and for each n∈N+, there exists K>0 such that
(3.6)E[e-ρ̃t|zt|2n]≤K(1+t).Proof.
We have given its proof here to need the same kind of calculations in the future. By Itô’s formula we have(3.7)e-ρ̃t|zt|2n=|z|2n+∫0t(-ρ̃)e-ρ̃s|zs|2nds+2n∫0te-ρ̃s|zs|2n-1sgn(zs)dzs+122n(2n-1)σ2∫0te-ρ̃szs2|zs|2n-2ds=|z|2n+∫0te-ρ̃s{-ρ̃|zs|2n+2nÃzs|zs|2n-1sgn(zs)+2nks|zs|2n-1sgn(zs)+n(2n-1)σ2zs2|zs|2n-2}ds-2nσ∫0te-ρ̃szs|zs|2n-1sgn(zs)dws=|z|2n+∫0te-ρ̃s{-ρ̃+2nÃ+n(2n-1)σ2}|zs|2nds+2n∫0te-ρ̃sks|zs|2n-1sgn(zs)ds-2nσ∫0te-ρ̃szs|zs|2n-1sgn(zs)dws.
Now by (2.15), (3.5) and taking expectation on the both sides, we obtain
(3.8)E[e-ρ̃t|zt|2n]≤|z|2n+2nE[∫0te-ρ̃s|zs|2nds]=|z|2n+E[∫0te-ρ̃sZ(zs)ds],
where Z(z)=2n|z|2n. Obviously, Z(z) is bounded above. Thus we can deduce (3.6).Lemma 3.2.
Under (3.5) and for each n∈N+, there exists K>0 such that
(3.9)ρ̃E[∫0∞e-ρ̃t|zt|2ndt]≤K(1+|z|2n+2).Proof.
By an application of Itô’s formula toe-ρ̃t|zt|2n+2, we have
(3.10)e-ρ̃t|zt|2n+2=|z|2n+2+∫0t(-ρ̃)e-ρ̃szs2n+2ds+(2n+2)∫0te-ρ̃s|zs|2n+1dzs+12(2n+2)(2n+1)σ2∫0te-ρ̃szs2zs2nds=|z|2n+2+∫0t(-ρ̃)e-ρ̃szs2n+2ds+(2n+2)∫0te-ρ̃s|zs|2n+1(Ãzs+ks)ds-2(n+1)σ∫0te-ρ̃szs|zs|2n+1dws+12(2n+2)(2n+1)σ2∫0te-ρ̃szs2zs2nds=|z|2n+2+∫0te-ρ̃s{-ρ̃+2(n+1)Ã+(n+1)(2n+1)σ2}zs2n+2ds+2(n+1)∫0te-ρ̃sks|zs|2n+1ds-2(n+1)σ∫0te-ρ̃szs|zs|2n+1dws.
Now by (2.15), (3.5) and taking expectation on the both sides we obtain
(3.11)E[e-ρ̃t|zt|2n+2]≤|z|2n+2+2(n+1)E[∫0te-ρ̃s|zs|2n+1ds].
Hence
(3.12)E[∫0∞e-ρ̃sG(zs)ds]≤|z|2n+2,
where G(z)=2(n+1)|z|2n+1.
We chooseζ<0 such that |z|2n≤G(z) for all z≥ζ. Then
(3.13)E[∫0∞e-ρ̃t|zt|2ndt]=∫0∞e-ρ̃tE[|zt|2n1(|zt|<ζ)+|zt|2n1(|zt|≥ζ)dt]≤∫0∞e-ρ̃t[{ζ2n+E[G(zt)]}dt]≤ζ2nρ̃+|z|2n+2.
Thus we get (3.9) with K>0 independent of sufficiently small ρ̃.Proposition 3.3.
We assume (3.3), (3.4). Then the value function v(z)isconvex.Proof.
For anyϵ>0, there exist k,k̂∈𝒫k such that
(3.14)E[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt]<v(z)+ϵ,E[∫0∞e-ρ̃t{h(ẑt)+(k̂t+1)2}dt]<v(ẑ)+ϵ,
where
(3.15)dzt=[Ãzt+kt]dt-σztdwt,z0=z∈R,dẑt=[Ãẑt+k̂t]dt-σẑtdwt,ẑ0=ẑ∈R.
We set
(3.16)k̃t=ξkt+(1-ξ)k̂t,z̃t=ξzt+(1-ξ)ẑt,z̃0=ξz+(1-ξ),ẑ≡z̃,
for   0<ξ<1. Clearly,
(3.17)dz̃t=[Ãz̃t+k̃t]dt-σz̃tdwt.
Hence, by convexity
(3.18)v(z̃)≤E[∫0∞e-ρ̃t{h(z̃t)+(k̃t+1)2}dt]≤ξE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt]+(1-ξ)E[∫0∞e-ρ̃t{h(ẑt)+(k̂t+1)2}dt]≤ξ(v(z)+ϵ)+(1-ξ)(v(ẑ)+ϵ).
Letting ϵ→0, we get
(3.19)v(z̃)=v(ξz+(1-ξ)ẑ)≤ξv(z)+(1-ξ)v(ẑ),
which completes the convexity of the value function v(z).Theorem 3.4.
Assume (3.4) and (3.5). Choose gr(z)=δ+|z|r for any 2≤r≤2n, then there exist δ≥1 and ζ>0 depending on M such that
(3.20)-ρ̃gr(z)+12σ2z2gr”(z)+Ãzgr′(z)+max|q|≤M(q2+qgr′(z))+ζgr(z)≤0.
Further
(3.21)E[∫0τe-ρ̃sζgr(zs)ds+e-ρ̃τgr(zτ)]≤gr(z)for2≤r≤2n,
where τ is any stopping time and zt is the response to (pt)∈𝒫M.
Then one has(3.22)0≤V(z)≤K̅(1+|z|r),
where K̅=Kδ/ζ, for some positive constant K.Proof.
By (3.5), we choose ζ∈(0,ρ̃) such that
(3.23)-ρ̃+rÃ+12r(r-1)σ2+ζ<0,
and then δ≥1 such that
(3.24)(-ρ̃+rÃ+12r(r-1)σ2+ζ)|z|r+Mr|z|r-1+(M2+ζδ-ρ̃δ)≤0.
Then (3.20) is immediate.
Applying Itô’s formula toe-ρ̃τgr(zτ), we have
(3.25)e-ρ̃τgr(zτ)=gr(z)+∫0τ(-ρ̃)e-ρ̃sgr(zs)ds+∫0τe-ρ̃sgr′(zs)dzs+12σ2∫0τe-ρ̃szs2gr”(zs)ds=gr(z)+∫0τe-ρ̃s{-ρ̃gr(zs)+Ãzsgr′(zs)+qgr′(zs)+12σ2zs2gr”(zs)}ds-σ∫0τe-ρ̃szsgr′(zs)dws=gr(z)+∫0τe-ρ̃s{-ρ̃gr(zs)+Ãzsgr′(zs)+12σ2zs2gr”(zs)+max|q|≤M(q2+qgr′(zs))+ζgr(zs)}ds-max|q|≤M∫oτq2e-ρ̃sds-∫0τe-ρ̃sζgr(zs)ds-σ∫0τe-ρ̃szsgr′(zs)dws.
Now by (3.20) and taking expectation on the both sides, we obtain
(3.26)E[∫0τe-ρ̃sζgr(zs)ds+e-ρ̃τgr(zτ)]≤gr(z)-M2ρ̃,
from which we deduced (3.21).
The convexity of the value functionV follows from the same line as (Proposition 3.3). Let (zt0) be the unique solution of
(3.27)dzt0=Ãzt0dt-σzt0dwt,z00=z.
Then by (3.4) and (3.21),
(3.28)V(z)≤E[∫0∞e-ρ̃th(zt0)dt]≤KE[∫0∞e-ρ̃tgr(zt0)dt]≤Kgr(z)ζ=K(δ+|z|r)ζ=Kδζ(1+|z|rδ)≤K̅(1+|z|r);2≤r≤2n,
which implies (3.22) and satisfies (3.4). Hence this completes the proof.Theorem 3.5.
Assume (3.3), (3.4), (3.5). Then there exists a unique solution v∈C2(R) of (3.1) such that
(3.29)ρ̃v(z)≤K(1+|z|n+3),z∈R,
for some constant K>0. Moreover, v admits a representation
(3.30)v(z)=infE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt];(kt+1)≥0.Proof.
SinceF*(z)=min{(k+1)2+kz} is Lipschitz continuous, this follows from Bensoussan [21] in case of Assumption (3.4) except convexity. For the general case, we take a nondecreasing sequence hn∈C(R) convergent to h with 0≤hn≤h. It is well known (Bensoussan [21]) that, for every n∈N+,  (3.1) has a unique solution vn for hn of the form
(3.31)vn(z)=infE[∫0∞e-ρ̃t{hn(zt)+(kt+1)2}dt];kt+1≥0
in the class C2(R) of continuous functions vanishing at infinitely, where zt is a solution of (2.19).
To prove (3.29), we recall (3.30). Hence by (3.4) and Lemma 3.2 we have
(3.32)0≤ρ̃vn(z)≤ρ̃E[∫0∞e-ρ̃t{hn(zt)+(kt+1)2∣}dt]≤ρ̃E[∫0∞e-ρ̃t{h(zt)+(kt+1)2∣}dt]≤K(1+ρ̃E[∫0∞e-ρ̃t{(1+|zt|n)+(kt+1)2}dt])≤K(1+|z|n+3).
This implies that v satisfies (3.29).
To estimatevn′ on [-s,s] for s∈R+, we remember the Taylor expansions of v∈C2(R):
(3.33)εv′(b)=v(b+ε)-v(b)-ε22v”(b+ϑ1ε),v′(z)=v′(b)+v”(b+ϑ2(z-b))(z-b),x∈Vε,0<ϑ1,ϑ2<1,
where Vε=Vε(b) is any ε-neighborhood of b∈[-s,s]. Then wa can obtain the Landau-Kolmogorov inequality:
(3.34)supVε|v′(z)|≤2εsupVε|v(z)|+3ε2supVε|v”(z)|.
Choosing 0<ε<1⋀(1/3(k+2Ã/σ2)), and by (3.1), (3.2), and (3.34) we have
(3.35)supVε|vn′(z)|≤2εsupVε|vn(z)|+3ε2supVε|vn”(z)|≤2εsupVε|vn(z)|+3ε2supVε[2ρ̃σ2|vn(z)z2|+2Ãσ2|vn′(z)z|+2σ2|F(vn′(z))z2|+2Ãσ2|hn(z)z|]≤(2ε+3ερ̃σ2)supVε|vn(z)|+3ε(k+2Ãσ2)supVε|vn′(z)|+3εÃσ2supVε|hn(z)|,
from which
(3.36)supVε|vn′(z)|≤11-3ε(k+2Ã/σ2)(2ε+3ερ̃σ2)supVε|vn(z)|+3ε(k+2Ãσ2)supVε|vn′(z)|+3εÃσ2supVε|hn(z)|.
Now by (3.34) and (3.4), we have
(3.37)supVε|vn′(z)|≤K1-3ε(k+2Ã/σ2)(2ε+3ερ̃σ2)(1+(s+1)n+3).
Thus, taking the finite covering Vε(bi),i=1,2,…,j, of [-s,s], we deduce
(3.38)sup-s≤z≤s|vn′(z)|≤∑i=1j‍supVε(bi)|vn′(z)|≤jK1-3ε(k+2Ã/σ2)(2ε+3ερ̃σ2)(1+(s+1)n+3),
and hence
(3.39)supnsup-s≤z≤s(|vn′(z)|+|vn”(z)|)<∞foreverys∈R+.
By the Ascoli-Arzelà theorem, we have
(3.40)vn→v,vn′→v′uniformlyon[-s,s],
taking a subsequence if necessary. Passing to the limit, we can obtain (3.1) and (3.30).
Following the inequality (3.29), we have
(3.41)E[e-ρ̃tv(zt)]→0ast→∞.
Hence by Itô’s formula to e-ρ̃tv(zt), for convex function [19, page 219], we have
(3.42)e-ρ̃tv(zt)=v(z)+∫0∞e-ρ̃t{-ρ̃v(z)+Ãzv′(z)+ktv′(z)+12σ2z2v”(z)}dt-∫0∞e-ρ̃tσztv′(z)dwt.
By virtue of (3.1) and taking expectation on the both sides we have
(3.43)E[e-ρ̃tv(zt)]=v(z)-E[∫0∞e-ρ̃t{h(zt+(kt+1)2)}dt].
Now by (3.41), we obtain
(3.44)v(z)=infE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt];kt≥-1,
which is similar to (3.30) and here the infimum is attained by the feedback law β(v′(z)) with β(θ)=argminF(θ). The proof is complete.Corollary 3.6.
Assume (3.3), (3.4), and (3.5). Then there exists a unique solution u∈C2(R) of (2.10) such that
(3.45)ρu(x,y)≤K(1+|x|n+3),(x,y)∈R,
for some constant K>0. Moreover, u admits a representation
(3.46)u(x,y)=infE[∫0∞e-ρt{h(xt)+pt2}dt];p≥0.


## 3.1. Existence and Uniqueness

To solve the Bellman equation (2.13) let us consider this HJB equation associated with the discounted production control problem in the following form:
(3.1)-ρ̃v(z)+12σ2z2v”(z)+Ãzv′(z)+F(v′(z))+h(z)=0,0<ρ̃<1,
where
(3.2)F(z)={-z24-zifz≤0,-zifz≥0.
We make the following assumptions:
(3.3)h:continuous  function  onR,h:non-negative,  convex  onR,h satisfies the polynomial growth condition such that
(3.4)0≤h(z)≤K(1+|z|n),z∈R,
for some K>0,n≥2.In order to ensure the integrability ofJ(p), we assume that
(3.5)-ρ̃+2nÃ+n(2n-1)σ2<0.
This condition (3.5) is needed for the integrability of zt or J(p). Under (3.5), we have Lemmas 3.1, 3.2, and Theorem 3.5, which ensures the finiteness of J(p) and hence the finiteness of an existence unique solution of HJB equation of v.First we have established the properties of the value function of the optimal control problem.Lemma 3.1.
Under (3.5) and for each n∈N+, there exists K>0 such that
(3.6)E[e-ρ̃t|zt|2n]≤K(1+t).Proof.
We have given its proof here to need the same kind of calculations in the future. By Itô’s formula we have(3.7)e-ρ̃t|zt|2n=|z|2n+∫0t(-ρ̃)e-ρ̃s|zs|2nds+2n∫0te-ρ̃s|zs|2n-1sgn(zs)dzs+122n(2n-1)σ2∫0te-ρ̃szs2|zs|2n-2ds=|z|2n+∫0te-ρ̃s{-ρ̃|zs|2n+2nÃzs|zs|2n-1sgn(zs)+2nks|zs|2n-1sgn(zs)+n(2n-1)σ2zs2|zs|2n-2}ds-2nσ∫0te-ρ̃szs|zs|2n-1sgn(zs)dws=|z|2n+∫0te-ρ̃s{-ρ̃+2nÃ+n(2n-1)σ2}|zs|2nds+2n∫0te-ρ̃sks|zs|2n-1sgn(zs)ds-2nσ∫0te-ρ̃szs|zs|2n-1sgn(zs)dws.
Now by (2.15), (3.5) and taking expectation on the both sides, we obtain
(3.8)E[e-ρ̃t|zt|2n]≤|z|2n+2nE[∫0te-ρ̃s|zs|2nds]=|z|2n+E[∫0te-ρ̃sZ(zs)ds],
where Z(z)=2n|z|2n. Obviously, Z(z) is bounded above. Thus we can deduce (3.6).Lemma 3.2.
Under (3.5) and for each n∈N+, there exists K>0 such that
(3.9)ρ̃E[∫0∞e-ρ̃t|zt|2ndt]≤K(1+|z|2n+2).Proof.
By an application of Itô’s formula toe-ρ̃t|zt|2n+2, we have
(3.10)e-ρ̃t|zt|2n+2=|z|2n+2+∫0t(-ρ̃)e-ρ̃szs2n+2ds+(2n+2)∫0te-ρ̃s|zs|2n+1dzs+12(2n+2)(2n+1)σ2∫0te-ρ̃szs2zs2nds=|z|2n+2+∫0t(-ρ̃)e-ρ̃szs2n+2ds+(2n+2)∫0te-ρ̃s|zs|2n+1(Ãzs+ks)ds-2(n+1)σ∫0te-ρ̃szs|zs|2n+1dws+12(2n+2)(2n+1)σ2∫0te-ρ̃szs2zs2nds=|z|2n+2+∫0te-ρ̃s{-ρ̃+2(n+1)Ã+(n+1)(2n+1)σ2}zs2n+2ds+2(n+1)∫0te-ρ̃sks|zs|2n+1ds-2(n+1)σ∫0te-ρ̃szs|zs|2n+1dws.
Now by (2.15), (3.5) and taking expectation on the both sides we obtain
(3.11)E[e-ρ̃t|zt|2n+2]≤|z|2n+2+2(n+1)E[∫0te-ρ̃s|zs|2n+1ds].
Hence
(3.12)E[∫0∞e-ρ̃sG(zs)ds]≤|z|2n+2,
where G(z)=2(n+1)|z|2n+1.
We chooseζ<0 such that |z|2n≤G(z) for all z≥ζ. Then
(3.13)E[∫0∞e-ρ̃t|zt|2ndt]=∫0∞e-ρ̃tE[|zt|2n1(|zt|<ζ)+|zt|2n1(|zt|≥ζ)dt]≤∫0∞e-ρ̃t[{ζ2n+E[G(zt)]}dt]≤ζ2nρ̃+|z|2n+2.
Thus we get (3.9) with K>0 independent of sufficiently small ρ̃.Proposition 3.3.
We assume (3.3), (3.4). Then the value function v(z)isconvex.Proof.
For anyϵ>0, there exist k,k̂∈𝒫k such that
(3.14)E[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt]<v(z)+ϵ,E[∫0∞e-ρ̃t{h(ẑt)+(k̂t+1)2}dt]<v(ẑ)+ϵ,
where
(3.15)dzt=[Ãzt+kt]dt-σztdwt,z0=z∈R,dẑt=[Ãẑt+k̂t]dt-σẑtdwt,ẑ0=ẑ∈R.
We set
(3.16)k̃t=ξkt+(1-ξ)k̂t,z̃t=ξzt+(1-ξ)ẑt,z̃0=ξz+(1-ξ),ẑ≡z̃,
for   0<ξ<1. Clearly,
(3.17)dz̃t=[Ãz̃t+k̃t]dt-σz̃tdwt.
Hence, by convexity
(3.18)v(z̃)≤E[∫0∞e-ρ̃t{h(z̃t)+(k̃t+1)2}dt]≤ξE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt]+(1-ξ)E[∫0∞e-ρ̃t{h(ẑt)+(k̂t+1)2}dt]≤ξ(v(z)+ϵ)+(1-ξ)(v(ẑ)+ϵ).
Letting ϵ→0, we get
(3.19)v(z̃)=v(ξz+(1-ξ)ẑ)≤ξv(z)+(1-ξ)v(ẑ),
which completes the convexity of the value function v(z).Theorem 3.4.
Assume (3.4) and (3.5). Choose gr(z)=δ+|z|r for any 2≤r≤2n, then there exist δ≥1 and ζ>0 depending on M such that
(3.20)-ρ̃gr(z)+12σ2z2gr”(z)+Ãzgr′(z)+max|q|≤M(q2+qgr′(z))+ζgr(z)≤0.
Further
(3.21)E[∫0τe-ρ̃sζgr(zs)ds+e-ρ̃τgr(zτ)]≤gr(z)for2≤r≤2n,
where τ is any stopping time and zt is the response to (pt)∈𝒫M.
Then one has(3.22)0≤V(z)≤K̅(1+|z|r),
where K̅=Kδ/ζ, for some positive constant K.Proof.
By (3.5), we choose ζ∈(0,ρ̃) such that
(3.23)-ρ̃+rÃ+12r(r-1)σ2+ζ<0,
and then δ≥1 such that
(3.24)(-ρ̃+rÃ+12r(r-1)σ2+ζ)|z|r+Mr|z|r-1+(M2+ζδ-ρ̃δ)≤0.
Then (3.20) is immediate.
Applying Itô’s formula toe-ρ̃τgr(zτ), we have
(3.25)e-ρ̃τgr(zτ)=gr(z)+∫0τ(-ρ̃)e-ρ̃sgr(zs)ds+∫0τe-ρ̃sgr′(zs)dzs+12σ2∫0τe-ρ̃szs2gr”(zs)ds=gr(z)+∫0τe-ρ̃s{-ρ̃gr(zs)+Ãzsgr′(zs)+qgr′(zs)+12σ2zs2gr”(zs)}ds-σ∫0τe-ρ̃szsgr′(zs)dws=gr(z)+∫0τe-ρ̃s{-ρ̃gr(zs)+Ãzsgr′(zs)+12σ2zs2gr”(zs)+max|q|≤M(q2+qgr′(zs))+ζgr(zs)}ds-max|q|≤M∫oτq2e-ρ̃sds-∫0τe-ρ̃sζgr(zs)ds-σ∫0τe-ρ̃szsgr′(zs)dws.
Now by (3.20) and taking expectation on the both sides, we obtain
(3.26)E[∫0τe-ρ̃sζgr(zs)ds+e-ρ̃τgr(zτ)]≤gr(z)-M2ρ̃,
from which we deduced (3.21).
The convexity of the value functionV follows from the same line as (Proposition 3.3). Let (zt0) be the unique solution of
(3.27)dzt0=Ãzt0dt-σzt0dwt,z00=z.
Then by (3.4) and (3.21),
(3.28)V(z)≤E[∫0∞e-ρ̃th(zt0)dt]≤KE[∫0∞e-ρ̃tgr(zt0)dt]≤Kgr(z)ζ=K(δ+|z|r)ζ=Kδζ(1+|z|rδ)≤K̅(1+|z|r);2≤r≤2n,
which implies (3.22) and satisfies (3.4). Hence this completes the proof.Theorem 3.5.
Assume (3.3), (3.4), (3.5). Then there exists a unique solution v∈C2(R) of (3.1) such that
(3.29)ρ̃v(z)≤K(1+|z|n+3),z∈R,
for some constant K>0. Moreover, v admits a representation
(3.30)v(z)=infE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt];(kt+1)≥0.Proof.
SinceF*(z)=min{(k+1)2+kz} is Lipschitz continuous, this follows from Bensoussan [21] in case of Assumption (3.4) except convexity. For the general case, we take a nondecreasing sequence hn∈C(R) convergent to h with 0≤hn≤h. It is well known (Bensoussan [21]) that, for every n∈N+,  (3.1) has a unique solution vn for hn of the form
(3.31)vn(z)=infE[∫0∞e-ρ̃t{hn(zt)+(kt+1)2}dt];kt+1≥0
in the class C2(R) of continuous functions vanishing at infinitely, where zt is a solution of (2.19).
To prove (3.29), we recall (3.30). Hence by (3.4) and Lemma 3.2 we have
(3.32)0≤ρ̃vn(z)≤ρ̃E[∫0∞e-ρ̃t{hn(zt)+(kt+1)2∣}dt]≤ρ̃E[∫0∞e-ρ̃t{h(zt)+(kt+1)2∣}dt]≤K(1+ρ̃E[∫0∞e-ρ̃t{(1+|zt|n)+(kt+1)2}dt])≤K(1+|z|n+3).
This implies that v satisfies (3.29).
To estimatevn′ on [-s,s] for s∈R+, we remember the Taylor expansions of v∈C2(R):
(3.33)εv′(b)=v(b+ε)-v(b)-ε22v”(b+ϑ1ε),v′(z)=v′(b)+v”(b+ϑ2(z-b))(z-b),x∈Vε,0<ϑ1,ϑ2<1,
where Vε=Vε(b) is any ε-neighborhood of b∈[-s,s]. Then wa can obtain the Landau-Kolmogorov inequality:
(3.34)supVε|v′(z)|≤2εsupVε|v(z)|+3ε2supVε|v”(z)|.
Choosing 0<ε<1⋀(1/3(k+2Ã/σ2)), and by (3.1), (3.2), and (3.34) we have
(3.35)supVε|vn′(z)|≤2εsupVε|vn(z)|+3ε2supVε|vn”(z)|≤2εsupVε|vn(z)|+3ε2supVε[2ρ̃σ2|vn(z)z2|+2Ãσ2|vn′(z)z|+2σ2|F(vn′(z))z2|+2Ãσ2|hn(z)z|]≤(2ε+3ερ̃σ2)supVε|vn(z)|+3ε(k+2Ãσ2)supVε|vn′(z)|+3εÃσ2supVε|hn(z)|,
from which
(3.36)supVε|vn′(z)|≤11-3ε(k+2Ã/σ2)(2ε+3ερ̃σ2)supVε|vn(z)|+3ε(k+2Ãσ2)supVε|vn′(z)|+3εÃσ2supVε|hn(z)|.
Now by (3.34) and (3.4), we have
(3.37)supVε|vn′(z)|≤K1-3ε(k+2Ã/σ2)(2ε+3ερ̃σ2)(1+(s+1)n+3).
Thus, taking the finite covering Vε(bi),i=1,2,…,j, of [-s,s], we deduce
(3.38)sup-s≤z≤s|vn′(z)|≤∑i=1j‍supVε(bi)|vn′(z)|≤jK1-3ε(k+2Ã/σ2)(2ε+3ερ̃σ2)(1+(s+1)n+3),
and hence
(3.39)supnsup-s≤z≤s(|vn′(z)|+|vn”(z)|)<∞foreverys∈R+.
By the Ascoli-Arzelà theorem, we have
(3.40)vn→v,vn′→v′uniformlyon[-s,s],
taking a subsequence if necessary. Passing to the limit, we can obtain (3.1) and (3.30).
Following the inequality (3.29), we have
(3.41)E[e-ρ̃tv(zt)]→0ast→∞.
Hence by Itô’s formula to e-ρ̃tv(zt), for convex function [19, page 219], we have
(3.42)e-ρ̃tv(zt)=v(z)+∫0∞e-ρ̃t{-ρ̃v(z)+Ãzv′(z)+ktv′(z)+12σ2z2v”(z)}dt-∫0∞e-ρ̃tσztv′(z)dwt.
By virtue of (3.1) and taking expectation on the both sides we have
(3.43)E[e-ρ̃tv(zt)]=v(z)-E[∫0∞e-ρ̃t{h(zt+(kt+1)2)}dt].
Now by (3.41), we obtain
(3.44)v(z)=infE[∫0∞e-ρ̃t{h(zt)+(kt+1)2}dt];kt≥-1,
which is similar to (3.30) and here the infimum is attained by the feedback law β(v′(z)) with β(θ)=argminF(θ). The proof is complete.Corollary 3.6.
Assume (3.3), (3.4), and (3.5). Then there exists a unique solution u∈C2(R) of (2.10) such that
(3.45)ρu(x,y)≤K(1+|x|n+3),(x,y)∈R,
for some constant K>0. Moreover, u admits a representation
(3.46)u(x,y)=infE[∫0∞e-ρt{h(xt)+pt2}dt];p≥0.


## 4. An Application to Production Control

In this section we will study the production control problem to minimize the cost (2.14) over the class 𝒫k of all progressively measurable processes pt such that 0≤pt≤kt and
(4.1)limt→∞1tE[e-ρ̃t|zt|n]=0
for the response zt to pt.Let us consider the stochastic differential equation(4.2)dzt*=[Ãzt*+βk(v′(zt*))]dt-σzt*dwt,z0*=z,z>0,
where βk(z)=argminF(z), that is,
(4.3)βk(z)={k-1ifz≤-2k,-z2-1if-2k<z≤0-1if0<z.,We need to establish the following lemmas.Lemma 4.1.
Under (3.5) and for each n∈N+, there exists K>0 such that
(4.4)E[e-ρ̃t|zt*|2n]≤K(1+t).Proof.
Since, we have by Itô’s formula(4.5)d(zt*)2=2zt*dzt*+12σ2(zt*)22dt=2zt*[(Ãzt*+βkv′(zt*))dt-σzt*dwt]+σ2(zt*)2dt,d(z̃t)2=2z̃tdzt+12σ2(z̃t)22dt≤2z̃t[Ãz̃tdt-σz̃tdwt]+σ2(z̃t)2dt.
Now by the assumptions of comparison theorem we have b1(z)=Ãz+βk(v′(zt))+σ2(z)2,   b2(z)=Ãz+σ2(z)2, then we have b1(z)<b2(z) and |σz-σz̃|=|σ||z-z̃|,  where σz=η(z),so∫0ϵη-2(z)dx=∫0ϵ(σz)-2dz=∞. Thus we can see (zt*)2≤(z̃t)2 by the comparison theorem of Ikeda and Watanabe [22]. Since the explosion time σ=inf{t:|zt*|=∞}, we have ∞=(zσ*)2≤(z̃σ)2. Hence σ=∞.
By the monotonicity ofβ(v′(z)), we have βk(v′(z̃))<β(v′(z))forz̃<z. Then
(4.6)dz̃t=[Ãz̃t+β(v′(z̃t))]dt-σz̃tdwt,z̃0=z*,dzt*=[Ãzt*+β(v′(zt*))]dt-σzt*dwt,z0*=z*,d(z̃t-zt*)=[Ã(z̃t-zt*)+βk(v′(z̃t))-βk(v′(zt*))]dt-σ(z̃t-zt*)dwt,
by Itô’s formula,
(4.7)d(z̃t-zt*)2=2(z̃t-zt*)d(z̃t-zt*)+12σ2(z̃t-zt*)22dt=2Ã(z̃t-zt*)2dt+2(z̃t-zt*)(βk(v′(z̃t))-βk(v′(zt*)))dt-2σ(z̃t-zt*)2dwt+σ2(z̃t-zt*)2dt≤(2Ã+σ2)(z̃t-zt*)2dt-2σ(z̃t-zt*)2dwt.
Hence
(4.8)(z̃t-zt*)2≤(2Ã+σ2)∫0t(z̃s-zs*)2ds-2σ∫0t(z̃s-zs*)2dws,E[(z̃t-zt*)2]≤(2Ã+σ2)∫0tE[(z̃s-zs*)2]ds.
Set ξ(t)=E[(z̃t-zt*)2], we obtain ξ(t)≤K∫0tξ(s)ds,∀t≥0 where K>2A+σ2. By Gronwall Lemma, we have ξ(t)=0a.s.∀t≥0. Therefore, E[(z̃t-zt*)2]=0a.s.∀t≥0, from which we have z̃t≤zt*. So, the uniqueness of (4.2) holds. Thus we conclude that (4.2) admits a unique strong solution (zt*), Ikeda and Watanabe [22, Chapter 4, Theorem 1.1], with E[|zt*|2n<∞].
By (3.5) and Itô’s formula,
(4.9)E[e-ρ̃t|zt*|2n]=|z|2n+E[∫0te-ρ̃s{-ρ̃+2nÃ+n(2n-1)σ2}|zs*|2nds]+2nE[∫0te-ρ̃sβk(v′(zs*))|zs*|2n-1sgn(zs*)ds]≤|z|2n+2nE[∫0te-ρ̃sβk(v′(zs*))|zs*|2n-1sgn(zs*)ds]=|z|2n+E[∫0te-ρ̃sZ(n)(s)ds],
where Z(n)(s)=2nβk(v′(zs*))|zs*|2n-1sgn(zs*).
By (4.3) it is easily seen that zβk(v′(z))≤(k-1)|z| if |z|≥a, for sufficiently large a>0. Clearly
(4.10)supsE[e-ρ̃sZ(n)(s)1(|zs*|<a)]<∞.
Also
(4.11)E[e-ρ̃sZ(n)(s)1(|zs*|≥a)]≤E[e-ρ̃s(k-1)|z|1(|zs*|≥a)].
By the same line as the proof of Lemma 3.1, we see that the right-hand side is bounded from above. This completes the proof.Theorem 4.2.
One assumes (3.3), (3.4), (3.5). Then the optimal cost control kt* is given by
(4.12)kt*=βk(v′(zt*)),
and the minimum value by
(4.13)J(k*)=v(z),
where zt* is defined by (4.2).Proof.
We first note by (4.3) that Fk(z)=min{(k+1)2+kz;k+1≥0}, and the minimum is attained by βk(z). We apply Itô’s formula for convex functions [19, page 219] to obtain
(4.14)e-ρ̃tv(zt*)=v(z)+∫0te-ρ̃s(-ρ̃v(zs)+Ãzv′(zs)+βk(v′(zs))+12σ2z2v”(zs))|z=zs*ds-∫0te-ρ̃sσzs*v′(zs*)dws.
Taking expectation on the both sides,
(4.15)E[e-ρ̃tv(zt*)]=v(z)+E[∫0te-ρ̃s(-ρ̃v(zs*)+Ãz*v′(zs*)+ks*v′(zs*)+12σ2z*2v”(zs*))ds].
By virtue of (2.13),
(4.16)E[e-ρ̃tv(zt*)]=v(z)-E[∫0te-ρ̃s{h(zs*)+(k+1)2}ds].
Choose n∈N+ such that 2n>m. By (3.4) and Lemma 3.1, we have
(4.17)1tE[e-ρ̃tv(zt*)]≤Kt(1+E[e-ρ̃mt|zt*|m])≤Kt(1+E([e-2nρ̃t|zt*|2n])m/2n)≤Kt(1+(K(1+t))m/2n)<∞,
which implies that
(4.18)liminft→∞1tE[e-ρ̃tv(zt*)]=0.
Hence zt* satisfies (4.1). Then we get E[∫0te-ρ̃s{h(zs*)+(k+1)2}ds]≤v(z) from which J(k*)≤v(z). By (3.4), we have J(k*)≤v(z)<∞, hence k*=(kt*)∈𝒫k.
ClearlyFk(z)≤(k+1)2+kz for every k∈𝒫k. Again following the same construction of (3.45) and by the HJB equation (2.13), we have
(4.19)E[e-ρ̃tv(zt)]≥v(z)-E[∫0te-ρ̃s{h(zs)+(k+1)2}ds],k=(kt)∈𝒫k.
By (4.1) we have
(4.20)E[∫0∞e-ρ̃s{h(zs)+(k+1)2}ds]≥v(z).
Thus we deduce J(k*)≥v(z). The proof is complete.Lemma 4.3.
Under (3.5), there exists a unique solution xt*≥0 of
(4.21)dxt*=[ψk(u′(xt*,yt))-yt]dt,x0*=x>0,
where
(4.22)ψk(x)={-x2ifx≤0,0if0<x,ψ(x):=-x2is  the  minimizer  ofminp≥0{p2+pux}.Proof.
Set(4.23)k(x,y)=ψ(ux(x,y))ifx,y>0.
Since k(x,y) is continuous in x and y, there exists a nonexplosive solution xt* of (4.21).
Now we will showxt*≥0∀t≥0 a.s. Suppose 0<v′(0+)<∞. Then L’Hospital’s rule gives
(4.24)limz→0+z2v′(z)z=limz→0+z2v”(z)=0.
Letting z→0+ in (3.1), we have F(v′(0+))=0, and hence v′(0+)=-(k+1)2/k≤0. This contradicts the assumption. Thus we get v′(0+)=∞, which implies that ψ(ux(0+,y))=0. In case T=inf{t≥0:xt*=0}=∞, we have at t=T,
(4.25)dxt*dt=ψ(ux(xt*,yt))-yt=0.
Therefore xt*≥0.
To prove uniqueness, letxi*(t),i=1,2 be two solutions of (4.21). Then x1*(t)-x2*(t) satisfies
(4.26)d(x1*(t)-x2*(t))=[(ψp(ux(x1*(t),yt))-yt)-(ψp(ux(x2*(t),yt))-yt)]dt,x1*(0)-x2*(0)=0.
We have
(4.27)d(x1*(t)-x2*(t))2=2(x1*(t)-x2*(t))d(x1*(t)-x2*(t))=2(x1*(t)-x2*(t))[ψ(ux(x1*(t),yt))-ψ(ux(x2*(t),yt))]dt.
Note that the function x→ψ(ux(x,y)) is increasing. Hence
(4.28)(x1*(t)-x2*(t))2≤2∫0t(x1*(s)-x2*(s))2ds.
By Gronwall's lemma, we have
(4.29)x1*(t)=x2*(t),∀t>0.
So, the uniqueness of (4.21) holds. The proof is complete.Theorem 4.4.
Under (3.5), the optimal production inventory cost control pt* is given by
(4.30)pt*=ψp(u′(xt*,yt)),
where xt* is defined by (4.21).Proof.
By Lemma4.3, we observe that (pt*) belongs to 𝒫. Now we apply Itô’s formula
(4.31)e-ρtu(xt*,yt)=u(x,y)+∫0te-ρs{12-ρu(x,y)+ux(x,y)(ps-ys)+Ayuy(x,y)+12σ2y2uyy(x,y)}|(x=xs*,y=ys)ds+∫0te-ρsσysuy(x*,y)dws.
By the HJB equation (2.10), we have
(4.32)e-ρtu(xt*,yt)=u(x,y)-∫0te-ρs{h(x*)+ps*2}ds+∫0te-ρsσysuy(x*,y)dws,
from which
(4.33)E[e-ρ(t⋀T)u(xt⋀T*,yt⋀T)]+E[∫0t⋀Te-ρs{h(x*)+ps*2}ds]=u(x,y).
Then we obtain by (4.18)
(4.34)E[∫0∞e-ρs{h(x*)+ps*2}ds]=u(x,y).
Letpt belong to 𝒫. By the same line as above, we have
(4.35)e-ρtu(xt,yt)=u(x,y)+∫0te-ρs{12-ρu(x,y)+ux(x,y)(ps-ys)+Ayuy(x,y)+12σ2y2uyy(x,y)}|(x=xs,y=ys)ds+∫0te-ρsσysuy(x,y)dws.
Again by the HJB equation (2.10), we can obtain
(4.36)E[e-ρtu(xt,yt)]≥u(x,y)-E[∫0te-ρs{h(xs)+ps2}ds].
By the same line as Lemma 4.1 and by (4.1),
(4.37)u(x,y)≤E[∫0∞e-ρs{h(xs)+ps2}ds],for  any(pt)∈𝒫.
Combining (4.34) with (4.37), we have
(4.38)J(p*)≤J(p).
Therefore the optimal production cost control p* which minimizes the production control problem (1.1) subject to (1.2) and (1.3). The proof is complete.



---

*Source: 370217-2009-07-02.xml*