# Asymptotic Optimality and Rates of Convergence of Quantized Stationary Policies in Continuous-Time Markov Decision Processes

Authors: Xiao Wu; Yanqiu Tang
Journal: Discrete Dynamics in Nature and Society
 (2022)

---

## Abstract

This paper is concerned with the asymptotic optimality of quantized stationary policies for continuous-time Markov decision processes (CTMDPs) in Polish spaces with state-dependent discount factors, where the transition rates and reward rates are allowed to be unbounded. Using the dynamic programming approach, we first establish the discounted optimal equation and the existence of its solutions. Then, we obtain the existence of optimal deterministic stationary policies under suitable conditions by more concise proofs. Furthermore, we discretize and incentivize the action space and construct a sequence of quantizer policies, which is the approximation of the optimal stationary policies of the CTMDPs, and get the approximation result and the rates of convergence on the expected discounted rewards of the quantized stationary policies. Also, we give an iteration algorithm on the approximate optimal policies. Finally, we give an example to illustrate the asymptotic optimality.


---

## Body

## 1. Introduction

This paper deals with the infinite horizon discounted continuous-time Markov decision processes (CTMDPs), as well as studies the asymptotic optimality of quantized stationary policies of CTMDPs, and gives the convergence rate results. The discount factors are state-dependent, and the transition rates and reward rates are allowed to be unbounded.It is well-known that the discounted CTMDPs have been widely studied as an important class of stochastic control problems. Generally speaking, according to the various forms of discount factors, the infinite horizon discounted CTMDPs can be classified into the following three groups: (i) MDPs with a fixed constant discount factorα, see, for instance, Doshi [1], Dynkin and Yushkevich [2], Feinberg [3], Guo [4, 5], Guo and Song [6], Guo and Hernndez-Lerma [7], Hernndez-Lerma and Lasserre [8, 9], Puterman [10], and the references therein; (ii) MDPs with varying (state-dependent or state-action dependent) discount factors, for instance, see Feinberg and Shwartz [11], Gonzlez-Hernndez et al. [12], Wu and Guo [13], Wu and Zhang [14], and the references therein; (iii) MDPs whose the discount factor is a function of the history, see Hinderer [15], for example. This paper will study the infinite horizon discounted CTMDPs in the case of the group.For the discounted criterion of MDPs, there are many works on the existence of solutions to the discounted optimality equation and of discounted optimal stationary policies, see, for instance, [1, 4, 6, 7, 16] for the CTMDPs and [8–10, 13–15] for the discrete-time Markov decision processes (DTMDPs). These references, however, are on the discounted MDPs with a constant discount factor or the discounted DTMDPs with varying discount factors. Recently, the discounted CDMDPs with state-dependent discount factors are studied in [16], in which the authors established the discounted reward optimality equation (DROE) and obtained the existence of discounted optimal stationary policies. However, in [16], the discussion is restricted to the class of all randomized stationary policies (i.e., the policies are time-independent). Following these ideas, still within the discounted continuous-time MDPs, models with Polish spaces are studied in this paper. We will extend some results in [16] to the case of all randomized Markov policies and obtain the existence of discounted optimal stationary policies by more concise proof.Although the existence of the optimal policies is proved, it is difficult to compute an optimal policy even in the stationary policies class for nonfinite Polish (i.e., complete and separable metric) state and action spaces. Furthermore, in applications to networked control, the transmission of such control actions to an actuator is not realistic when there is an information transmission constraint (imposed by the presence of a communication channel) between a plant, a controller, or an actuator. Thus, from a practical point of view, it is important to study the approximation of optimal stationary policies. Several approaches have been developed in the literature to solve this problem for finite or countable state spaces, see [17–20]. Lately, for infinite Borel state and action spaces, [21, 22] give the asymptotic optimality of quantized stationary policies in stochastic control for DTMDPs. Inspired by these, in this paper, we are concerned with the asymptotic optimality of quantized stationary policies in CTMDPs with Polish spaces. To the best of our knowledge, the corresponding asymptotic optimality for CTMDPs with varying (state-dependent) discount factors has not been studied.Therefore, this paper contains the following three main contributions:(a)
For the CDMDPs with state-dependent discount factors, we extend some results in [16] to the case of all randomized Markov policies, and the proof of the existence of discounted optimal stationary policies is simplified under mild conditions and gives an algorithm to get ε-optimal policies.(b)
We obtain that the deterministic stationary quantizer policies are able to approximate the optimal deterministic stationary policies under mild technical conditions and thus show that one can search for approximate optimal policies within the class of quantized control policies.(c)
For the asymptotic optimality, we give the corresponding convergence rates results.This paper is organized as follows. In section 2, we introduce the models of CDMDPs with the expected discounted reward criterion and state the discounted optimality problem. In section 3, under suitable conditions, we prove the main result on the existence of the solutions to the discounted optimal equation (DOE) and the existence of optimal stationary policies. In section 4, we give an iteration algorithm on theε-optimal policies. In section 5, we establish conditions under which quantized control policies are asymptotically optimal and give the corresponding convergence rate results and the rates of convergence on the expected discounted rewards of the quantized stationary policies. Finally, we illustrate the asymptotic optimality by an example in Section 6.


## 2. The Markov Decision Processes and Discounted Optimal Problem

Consider the model of continuous-time Markov decision processesℳ as follows:(1)ℳ≔S,Ax⊆A,x∈S,q•|x,a,αx,rx,a,where S is the state space, Ax are sets of admissible actions, and A is a compact action space. S and A are assumed to be Polish spaces (i.e., complete and separable metric spaces) with Borel σ-field ℬS and ℬA, respectively. Ax and K≔x,a|x∈S,a∈Ax are Borel subsets of A and S×A, respectively. q•|x,a denotes the function of transition rates, which satisfy the following properties:(P1)q•|x,a is a signed measure on ℬS for each fixed x,a∈K, and qB|•,• is a Borel-measurable function on K for each fixed B∈ℬS(P2)0≤qB|x,a<∞ for all x,a∈K and x∉B∈ℬS(P3)q•|x,a is conservative, that is, qS|x,a=0 for all x,a∈K, and then, 0≤−qx|x,a<∞(P4) the model in (1) is supposed to be stable, that is, for each x∈S, it holds that(2)q∗x≔supa∈Ax−qx|x,a<∞.The discount factorsαx are the nonnegative measurable functions on S. Finally, the reward rate function rx,a is assumed to be Borel-measurable on K. Note that, rx,a is allowed to be unbounded from both above and below, and it can be regarded as a cost rate rather than a reward rate only.The deﬁnitions of the randomized Markov policy≔πt,t≥0, randomized stationary policy φ, and (deterministic) stationary policy f are given by [8] [Deﬁnitions 2.2.3 and 2.3.2]. The sets of all randomized Markov policies, randomized stationary policies, and (deterministic) stationary policies are denoted by Π, Φ, and F, respectively. It is clear that F⊂Φ⊂Π, and for each π=πt,t≥0∈Π, x∈S, and B∈ℬS, we define the associated functions of transition rates qπ•|x,πt and reward rates rπx,πt by(3)qπB|x,πt≔∫AxqB|x,aπtda|x,rπx,πt≔∫Axrx,aπtda|x.In general, we also write asqB|x,πt and rx,πt, respectively. Furthermore, for each φ∈Φ, we define the functions of transition rates and reward rates by(4)qB|x,φ≔∫AxqB|x,aπtda|x,rπx,φ≔∫Axrx,aφda|x.In particular, we write them asqB|x,f and rx,f, respectively, when φ=f∈F, that is, qB|x,f=qB|x,fx and rx,f=rx,fx. Also, for each ∈S, we denote(5)qx,f≔−qx|x,fx.For any fixed policyπ=πt,t≥0∈Π, q•|x,πt is also called an infinitesimal generator (see Doshi [1]). As is well known, any transition function p˜πs,x,t,B depending on π such that(6)limτ⟶0+p˜πt,x,t+τ,B−δxBτ=qB|x,πt.for all x∈S and B∈ℬS is called a Q-process with transition rates q•|x,πt, where δxB is the Dirac measure at x∈S. By Guo [4], there exists a minimal Q-process pπmins,x,t,B with transition rates q⋅|x,πt, but such a Q-process might not be regular, that is, there may exist pπmins,x,t,S<1 for some x∈S and t≥s≥0. To ensure the regularity of the Q-process, we propose the following “drift conditions.”Assumption 1.
There exists a measurable functionw1≥1 on S, and constants c1≠0,b1≥0 and Mq>0 such that(a)
for allx,a∈K, ∫Sw1yqdy|x,a≤c1w1x+b1(b)
For eachx∈S, q∗x≤Mqw1xRemark 1.
(a)
The functionw in Assumption 1 (a) is used to guarantee the finiteness of the optimal value function as below, and by [4] [Remark 2.2(b)], it is an extension of the “drift condition” in Lund et al. [23] for a time-homogeneous Q-process. Moreover, Assumption 1 (b) is used to guarantee the regularity of the Q-process, and it is not required when the transition rates are bounded (i.e., supx∈Sq∗x<∞).(b)
Under Assumption1, it holds that pπmins,x,t,S≡1 by Guo [4] [Theorem 3.2]. Then, the Q-process with transition rates q•|x,πt is regular and unique. Hence, we write pπmins,x,t,B simply as pπs,x,t,B. Since it is time-homogeneous, we discuss the case that the initial time is s=0, and then, we write pπ0,x,t,B simply as pπx,t,B.
As it is well known (e.g., see Doshi [1] and Guo [5]), for each π=πt,t≥0∈Π and the initial state x∈S, there exists a unique probability space Ω,ℬΩ,Pxπ, where the probability measure Pxπ is completely determined by pπx,t,B (see Guo [6], Section 2.3]), and a state and action process xt,at,t≥0 with the transition probability function pπx,t,B such that (see Guo [5], Lemma 2.1])(7)Pxπxt∈B=pπx,t,B,Pxπat∈Γ|xt=j=πtΓ|j,∀Γ∈ℬA.
The expectation operator corresponding toPxπ can be denoted by Exπ. Moreover, for each x∈S, π∈Π and t≥0, the expected reward is given by(8)Exπrxt,πt≔∫Sry,πtpπx,t,dy.
Now, we state the discounted optimality problem. For eachπ∈Π and x∈S, the expected discounted reward criterion is defined as(9)Jx,π≔Exπ∫0∞e−∫0tαxsdsrxt,πtdt,and the corresponding optimal value function is given by(10)J∗x≔supπ∈ΠJx,π.
Also, a policyπ∗∈Π is called optimal policy if Jx,π∗≥Jx,π for all x∈S and π∈Π. Our main aim in Section 3 is to give conditions for the existence of optimal deterministic stationary policies.


## 3. The Existence of Optimal Stationary Policies

In this section, the existence and uniqueness of the discounted optimal equation (DOE) are shown, and the existence of the optimal policies is given for the CTMDPsM defined in (1).Note that, for any given measurable functionw≥1 on S, a function v on S is called w− bounded if the w− weighted norm vw≔supx∈Svx/wx is finite. Such a function w is called a weight function. It is clear that ℬwS≔v:vw<∞ is a Banach space for all real-valued measurable functions v on S. To guarantee the finiteness of the optimal value function, we need the following assumptions.Assumption 2.
Letw1 and c1 be as in Assumption 1. For each x∈S, suppose that the following conditions hold:(a)
Ax is a compact set(b)
The functionrx,a is continuous on a∈Ax, and for each x,a∈K, there exists a constant M1>0 such that rx,a≤M1w1x(c)
The discount factorαx is continuous on S, and there is a constant α0>c1 such that αx≥α0>c1(d)
For any bounded measurable functionux on S, the functions ∫Suyqdy|x,a and ∫Sw1yqdy|x,a are continuous on a∈Ax(e)
There exists a nonnegative measurable functionw2x on S, and constants c2>0,b2≥0 and M2>0 such that q∗xw1x≤M2w2x and ∫Suyqdy|x,a≤c2w2x+b2 for all x∈S and a∈Ax
For eachx∈S, let mx be any positive measurable function on S such that mx≥q∗x≥0, and(11)PB|x,a≔qB|x,amx+δxB,∀B∈ℬS,x,a∈K.where δxB is Dirac measure (i.e., it is equal to 1 if x∈B and 0 otherwise). It is clear that P⋅|x,a is a probability measure on S for each x,a∈K. For any u∈ℬw1S, define an operator T on ℬw1S as(12)Tux≔supa∈Axrx,aαx+mx+mxαx+mx∫SuyPdy|x,a,∀x∈S.
And, define a recursive sequenceun,n≥0 as(13)u0x≔−M1b1α0α0−c1−M1w1xα0−c1,unx≔Tun−1x.
Now, we give the discounted optimal equation (DOE).Theorem 1.
Under Assumptions1 and 2 (b)-(c), the following assertions hold.(a)
Jx,π≤b1M1/α0α0−c1+M1/α0−c1w1x for all x∈S and π∈Π, and J⋅,π∈ℬw1S(b)
Letu∗≔limn⟶∞un, then we have u∗∈ℬw1S, and it is the solution of the following discounted optimal equation (DOE):(14)αxux=supa∈Axrx,a+∫Suyqdy|x,a,∀x∈S.Proof.
(a)
By the assumptions, we have(15)Jx,π=Exπ∫0∞e−∫0tαxsdsrxt,πtdt≤∫0∞Exπe−α0trxt,πtdt,≤M1∫0∞e−α0tExπw1xtdt≤M1∫0∞e−α0tec1tw1x+b1c1ec1t−1dt,=b1M1α0α0−c1+M1α0−c1w1x,where the last inequality holds by Guo [4] [Theorem 3.2(b)]. Then, Jx,π∈ℬw1S for each x∈S, and part (a) holds.(b)
First, we obtainun is monotone and nondecreasing by a similar calculation as in Ye and Guo [16] [Equation (15)]. Furthermore, it is clear that the operator T is monotone and nondecreasing. Then, we have un is monotone and nondecreasing, which yields that u∗≔limn⟶∞un≥un for all n≥0.
Next, we show thatu∗∈ℬw1S. Note that w1x ≥1 by Assumption 1, which yields that(16)u0x≤M1b1α0α0−c1+M1w1xα0−c1≤M1b1+α0α0α0−c1w1x.
Then, by induction argument, for alln≥1, we have(17)unx≤supa∈AxM1w1xα0+mx+mxα0+mx∫Sb1M1α0α0−c1+M1α0−c1w1yPdy|x,a,=M1b1α0α0−c1+M1w1xα0−c1≤M1b1+α0α0α0−c1w1x.
Thus,u∗=limn⟶∞un≤M1b1+α0/α0α0−c1w1x, that is, u∗∈ℬw1S.
Last, we showTu∗=u∗. By the monotonicity of T and un, we have Tu∗≥Tun=un+1 for all n≥0, and so Tu∗≥u∗.
On the other hand, by the definition of the operatorT, we have(18)un+1x≥rx,aαx+mx+mxαx+mx∫SunyPdy|x,a.
Then, lettingn⟶∞, by Hern a′ Hernndez-Lerma and Lasserre [9], [Lemma 8.3.7], we obtain(19)u∗x≥rx,aαx+mx+mxαx+mx∫Su∗yPdy|x,a,which follows that u∗≥Tu∗. Thus, we have Tu∗=u∗, that is, u∗ is the solution of DOE in (14).Remark 2.
Theorem1 is not only the generalization of the control model with a constant discount factor in Guo [4] [Theorem 3.3(a)-(b)] but also the model in Ye and Guo [16] whose policies are restricted within the family Φ of all randomized stationary policies.
The following Lemma1 is a direct consequence of [16] [Theorem 3.2].Lemma 1.
Under Assumptions1 and 2, for each x∈S and φ∈Φ, the expected discounted reward criterion Jx,φ is the unique solution of the following equation:(20)αxux=rx,φ+∫Suyqdy|x,φ.Lemma 2.
Under Assumptions1 and 2, for each x∈S, φ∈Φ, and u∈ℬw1S, the following assertions hold.(a)
if(21)αxux≥rx,φ+∫Suyqdy|x,φ,then, we haveux≥Jx,φ.(b)
if(22)αxux≤rx,φ+∫Suyqdy|x,φ,then we haveux≤Jx,φ.Proof.
By (21), there exists a nonnegative measurable function vx on S such that(23)αxux=rx,φ+vx+∫Suyqdy|x,φ.
Now, letr¯x,a=rx,a+vx, and we get the new Markov decision processes:(24)​M¯≔​​​S,Ax⊆A,x∈S,q⋅|x,a,αx,r¯x,a,in which only the reward rate function is different from the model in (1). Moreover, for each x∈S, φ∈Φ, the expected discounted reward criterion is given by(25)J¯x,φ≔Jx,φ+∫0∞Exφe−∫0tαxsdsvxtdt≥0.
By Lemma1, we have ux=J¯x,φ≥Jx,φ, which gives part (a).
Similarly, we can prove (b).Remark 3.
Lemma2 is the generalization of Ye and Guo [16] [Lemma 6.3].Theorem 2.
Under Assumptions1 and 2, for each x∈S, the optimal value function J∗x is the solution of DOE in (12), and there exists a (deterministic) stationary policy f∗∈F such that(26)αxJ∗x≥rx,f∗+∫SJ∗yqdy|x,f∗.Proof.
By Theorem1(b), for each x∈S and π∈Π, we have(27)αxu∗x≥rx,πt+∫Su∗yqdy|x,πt,which together with Lemma 2(a) yields that u∗x≥Jx,π, and then, u∗x≥J∗x. Note that ∫Su∗yqdy|x,a is upper semicontinuous on a∈Ax; then, by [9] [Lemma 8.3.8], we can obtain that there exists a policy f∗∈F such that(28)αxu∗x=rx,f∗x+∫Su∗yqdy|x,f∗x,for all x∈S. Thus, by Lemma 1, we have u∗x=Jx,f∗.Remark 4.
(a)
Theorem2 shows that the optimal value function is a solution to the DOE ((21)]) and ensures the existence of an optimal (deterministic) stationary policy.(b)
By the construction of the new Markov decision processes, the proof of Theorem2 is more concise than in [16] Theorem 3.3.


## 4. An Iteration Algorithm forε-Optimal Policies

In this section, we provide an iteration algorithm forε-optimal policies.Algorithm 1: (An iteration algorithm).
Step 1. (Initialization). Choose anyε>0, let w1≥max1,2b1/α0−c1 in Assumption 1, and for each x∈S, letu0x≔−M1b1/α0α0−c1−M1w1x/α0−c1Step 2. (Iteration). For eachx∈S, letun+1x≔maxa∈Axrx,a/αx+q∗+1+q∗+1/αx+q∗+1∫SunyPdy|x,awherePdy|x,a≔qdy|x,a/q∗+1+δxy..Step 3. (Approximation value). Ifun+1−unw1≤ε1−λ/2λwhereλ≔α0+c1/2+q∗+1/α0+q∗+1≤1, go on step 4, otherwise increment n by 1 and return to step 2.Step 4. (ε-optimal policy). For each x∈S, choosefεx∈argmaxa∈Axrx,a/αx+q∗+1+q∗+1/αx+q∗+1∫Sun+1yPdy|x,a,andfε is ε-optimal policy.In fact, for the operatorT on ℬw1S in Section 3, with mx=q∗+1, it holds that(29)Tu−Tvw1≤λu−vw1,for all,u,v∈ℬw1S.Then, by Algorithm 1, we have(30)un+1−u∗≤un+1−Tun+1+Tun+1−u∗=Tun−Tun+1+Tun+1−Tu∗,≤λun+1−unw1w1x+λun+1−u∗w1w1x,which yields that(31)un+1−u∗w1≤λ1−λun+1−unw1.By a similar argument, we have(32)Jg,fε−un+1w1≤λ1−λun+1−unw1,and then,(33)Jg,fε−u∗w1≤2λ1−λun+1−unw1≤ε.


## 5. Asymptotic Optimality of Quantized Stationary Policies

### 5.1. Approximation of Deterministic Stationary Policies

In Section3, we give the existence of the deterministic stationary policies for the CTMDPs in (1) under suitable conditions. However, in practice, sometimes, the action space cannot satisfy the continuity conditions in theoretical research. Thus, in this section, we will discretize and incentivize the action space, so that we can construct a sequence of policies, namely “quantizer policies,” which is the approximation of the deterministic stationary policies of the CTMDPs in (1).To this end, we first give the definitions of quantizers and deterministic stationary quantizer policies.Deﬁnition 1.
A measurable functionf:S⟶A is called a quantizer from S to A, if fS≔fx∈A:x∈S is finite. Let ℱ denote the set of all quantizers from S to A.Deﬁnition 2.
A policy is called a deterministic stationary quantizer policy, if there exists a constant sequenceπ=πn,n≥0 of stochastic kernels on A given S such that πn⋅|x=δfx⋅ for all n for some f∈ℱ, where δfx⋅ is Dirac measure as in (11).
For any finite setΛ⊂A, let ℱΛ denotes the set of all quantizers having range Λ, and let SℱΛ denotes the set of all deterministic stationary quantizer policies induced by ℱΛ.
Denote the metric onA as dA, and then, the action space A is totally bounded by its compactness. For any fixed integer k≥1, there exists a finite point set aii=1nk such that for all a∈A,(34)min1≤i≤nkdAa,ai≤1k,where aii=1nk is called the 1/k− net in A. From this, for any a deterministic stationary policy f∈F, we can construct a sequence of quantizer policies to approximate to f by the following methods.Lemma 3 (The construction of quantizer policies).
LetΛk≔aii=1nk is the 1/k− net in A, for each x∈S and deterministic stationary policy f∈F, we define(35)fkx≔argmina∈ΛkdAfx,a.
Then,fkk≥1 is a deterministic stationary quantizer policy sequence, and fk converges uniformly to f as k⟶∞.Proof.
Lemma3 holds obviously by [21] [Section 3].
We also callfkk≥1 as the quantized approximations of f. Next, we show their expected discounted rewards also satisfy the approximation. For this purpose, we need the following conditions as follows.Assumption 3.
Letc1 be as in Assumption 1, for each x∈S, suppose that qx|x,a and qB|x,a are setwise continuous in a∈Ax for each B∈ℬS and x∉B, that is, if ak⟶a, then q⋅|x,an⟶q⋅|x,a setwise.Lemma 4.
Suppose that Assumptions2 and 3 hold. Let f∈F be a deterministic stationary policy of the control model in (1) and fkk≥1 be the quantized approximations of f as in Lemma 3, then for each x∈S, the strategic measures Pxfk induced by the quantized approximations fk of f converge to Pxfk in the weak topology. Therefore, Exfkrxt,at converges to Exfrxt,at.Proof.
The proof is similar to that of [21], Proposition 3.1, and by Assumption 3 and the definition of the strategic measures Pxf as in [6], [Section 2.3] or [5], [Section II], we can get Lemma 4 holds.
Now, we give the approximation result on the expected discounted rewards of the deterministic stationary quantizer policies.Theorem 3.
Suppose that Assumptions1–3 hold. Let f∈F be a deterministic stationary policy of the control model in (1), and fkk≥1 be the quantized approximations of f as in Lemma 3, then for each x∈S, we have(36)limk⟶∞Jx,fk=Jx,f.Proof.
By the definition of the expected discounted reward criterion, we can get(37)Jx,fk−Jx,f=∫0∞e−∫0tαxsdsExfkrxt,fkxt−Exfrxt,fxtdt,≤∫0Te−α0tExfkrxt,fkxt−Exfrxt,fxtdt,+∫0Te−α0tExfkrxt,fkxt−Exfrxt,fxtdt.
Note that, by Lemmas3 and 4, we have(38)Exfkrxt,fkxt−Exfrxt,fxt,≤Exfkrxt,fkxt−Exfkrxt,fxt,+Exfkrxt,fxt−Exfrxt,fxt,⟶0,which yields that(39)∫0Te−α0tExfkrxt,fkxt−Exfrxt,fxtdt⟶0k⟶∞.
On the other hand, we have(40)∫T∞e−α0tExfkrxt,fkxt−Exfrxt,fxtdt,≤∫T∞e−α0tExfkrxt,fkxt−Exfrxt,fxtdt,≤M1∫T∞e−α0tExfkw1xt+Exfw1xtdt,≤2M1∫T∞e−α0tec1tw1x+b1c1ec1t−1dt,=2M1w1xα0−c1e−α0−c1T+b1c1α0−c1e−α0−c1T−b1α0c1e−α0T,where the last inequality holds by [4] [Theorem 3.2(b)]. Then, we have(41)∫T∞e−α0tExfkrxt,fkxt−Exfrxt,fxtdt⟶0,
asT⟶∞. By (40), we can get(42)limk⟶∞Jx,fk=Jx,f.


### 5.2. Rates of Convergence

Deﬁnition 3.
Let•TV denote the total variation distance between measures P1 and P2 on the probability space Ω,F, which satisfies(43)P1−P2TV2supD∈FΩP1D−P2D.Assumption 4.
For eachx∈S, suppose that the model (1) satisfies the following conditions:(a)
A is a compact subset of ℝd for some d ≥1(b)
For allx,a1,x,a2∈K, there exists a constant K1>0 such that(44)rx,a1−rx,a2≤K1dAa1,a2.(c)
For allx,a1,x,a2∈K and the function of transition rates q⋅|x,a, there exists a constant K2>0 such that(45)q⋅|x,a1−q⋅|x,a2TV≤K2dAa1,a2.
By Lemma 3 and Assumption 4, the following Lemma holds.Lemma 5.
For any measurable functionf:S⟶A, we can construct a sequence of quantizers fkk≥1 from S to A, and there exists some constant K3>0 such that(46)supx∈SdAfx,fkx≤K31/k1/d.
Now, we give the convergence rates result.Theorem 4.
Suppose that Assumptions1–4 hold. Let f∈F be a deterministic stationary policy of the control model in (1) and fkk≥1 be the quantized approximations of f as in Lemma 3, then for each x∈S, it holds that(47)Jx,f−Jx,fk≤K31/k1/dα0K1+b1M1K2α0α0−c1+M1K2α0−c1w1x.Proof.
By Lemma1, we can get(48)αxJx,f=rx,f+∫Sqdy|x,f,αxJx,fk=rx,fk+∫SJy,fkqdy|x,f.
Then, by Lemma5 and Theorem 1, we have(49)α0Jx,f−Jx,fk≤αxJx,f−Jx,fk,≤rx,f−rx,fk+∫Sqdy|x,f−∫SJy,fkqdy|x,fk,≤K1dAfx,fkx+supx∈S,π∈Π≤K1K31/k1/d+b1M1α0α0−c1+M1α0−c1w1xK2dAfx,fkx,=K31/k1/dK1+b1M1K2α0α0−c1+M1K2α0−c1w1x,
which yields that(50)Jx,f−Jx,fk≤K31/k1/dα0K1+b1M1K2α0α0−c1+M1K2α0−c1w1x.


## 5.1. Approximation of Deterministic Stationary Policies

In Section3, we give the existence of the deterministic stationary policies for the CTMDPs in (1) under suitable conditions. However, in practice, sometimes, the action space cannot satisfy the continuity conditions in theoretical research. Thus, in this section, we will discretize and incentivize the action space, so that we can construct a sequence of policies, namely “quantizer policies,” which is the approximation of the deterministic stationary policies of the CTMDPs in (1).To this end, we first give the definitions of quantizers and deterministic stationary quantizer policies.Deﬁnition 1.
A measurable functionf:S⟶A is called a quantizer from S to A, if fS≔fx∈A:x∈S is finite. Let ℱ denote the set of all quantizers from S to A.Deﬁnition 2.
A policy is called a deterministic stationary quantizer policy, if there exists a constant sequenceπ=πn,n≥0 of stochastic kernels on A given S such that πn⋅|x=δfx⋅ for all n for some f∈ℱ, where δfx⋅ is Dirac measure as in (11).
For any finite setΛ⊂A, let ℱΛ denotes the set of all quantizers having range Λ, and let SℱΛ denotes the set of all deterministic stationary quantizer policies induced by ℱΛ.
Denote the metric onA as dA, and then, the action space A is totally bounded by its compactness. For any fixed integer k≥1, there exists a finite point set aii=1nk such that for all a∈A,(34)min1≤i≤nkdAa,ai≤1k,where aii=1nk is called the 1/k− net in A. From this, for any a deterministic stationary policy f∈F, we can construct a sequence of quantizer policies to approximate to f by the following methods.Lemma 3 (The construction of quantizer policies).
LetΛk≔aii=1nk is the 1/k− net in A, for each x∈S and deterministic stationary policy f∈F, we define(35)fkx≔argmina∈ΛkdAfx,a.
Then,fkk≥1 is a deterministic stationary quantizer policy sequence, and fk converges uniformly to f as k⟶∞.Proof.
Lemma3 holds obviously by [21] [Section 3].
We also callfkk≥1 as the quantized approximations of f. Next, we show their expected discounted rewards also satisfy the approximation. For this purpose, we need the following conditions as follows.Assumption 3.
Letc1 be as in Assumption 1, for each x∈S, suppose that qx|x,a and qB|x,a are setwise continuous in a∈Ax for each B∈ℬS and x∉B, that is, if ak⟶a, then q⋅|x,an⟶q⋅|x,a setwise.Lemma 4.
Suppose that Assumptions2 and 3 hold. Let f∈F be a deterministic stationary policy of the control model in (1) and fkk≥1 be the quantized approximations of f as in Lemma 3, then for each x∈S, the strategic measures Pxfk induced by the quantized approximations fk of f converge to Pxfk in the weak topology. Therefore, Exfkrxt,at converges to Exfrxt,at.Proof.
The proof is similar to that of [21], Proposition 3.1, and by Assumption 3 and the definition of the strategic measures Pxf as in [6], [Section 2.3] or [5], [Section II], we can get Lemma 4 holds.
Now, we give the approximation result on the expected discounted rewards of the deterministic stationary quantizer policies.Theorem 3.
Suppose that Assumptions1–3 hold. Let f∈F be a deterministic stationary policy of the control model in (1), and fkk≥1 be the quantized approximations of f as in Lemma 3, then for each x∈S, we have(36)limk⟶∞Jx,fk=Jx,f.Proof.
By the definition of the expected discounted reward criterion, we can get(37)Jx,fk−Jx,f=∫0∞e−∫0tαxsdsExfkrxt,fkxt−Exfrxt,fxtdt,≤∫0Te−α0tExfkrxt,fkxt−Exfrxt,fxtdt,+∫0Te−α0tExfkrxt,fkxt−Exfrxt,fxtdt.
Note that, by Lemmas3 and 4, we have(38)Exfkrxt,fkxt−Exfrxt,fxt,≤Exfkrxt,fkxt−Exfkrxt,fxt,+Exfkrxt,fxt−Exfrxt,fxt,⟶0,which yields that(39)∫0Te−α0tExfkrxt,fkxt−Exfrxt,fxtdt⟶0k⟶∞.
On the other hand, we have(40)∫T∞e−α0tExfkrxt,fkxt−Exfrxt,fxtdt,≤∫T∞e−α0tExfkrxt,fkxt−Exfrxt,fxtdt,≤M1∫T∞e−α0tExfkw1xt+Exfw1xtdt,≤2M1∫T∞e−α0tec1tw1x+b1c1ec1t−1dt,=2M1w1xα0−c1e−α0−c1T+b1c1α0−c1e−α0−c1T−b1α0c1e−α0T,where the last inequality holds by [4] [Theorem 3.2(b)]. Then, we have(41)∫T∞e−α0tExfkrxt,fkxt−Exfrxt,fxtdt⟶0,
asT⟶∞. By (40), we can get(42)limk⟶∞Jx,fk=Jx,f.


## 5.2. Rates of Convergence

Deﬁnition 3.
Let•TV denote the total variation distance between measures P1 and P2 on the probability space Ω,F, which satisfies(43)P1−P2TV2supD∈FΩP1D−P2D.Assumption 4.
For eachx∈S, suppose that the model (1) satisfies the following conditions:(a)
A is a compact subset of ℝd for some d ≥1(b)
For allx,a1,x,a2∈K, there exists a constant K1>0 such that(44)rx,a1−rx,a2≤K1dAa1,a2.(c)
For allx,a1,x,a2∈K and the function of transition rates q⋅|x,a, there exists a constant K2>0 such that(45)q⋅|x,a1−q⋅|x,a2TV≤K2dAa1,a2.
By Lemma 3 and Assumption 4, the following Lemma holds.Lemma 5.
For any measurable functionf:S⟶A, we can construct a sequence of quantizers fkk≥1 from S to A, and there exists some constant K3>0 such that(46)supx∈SdAfx,fkx≤K31/k1/d.
Now, we give the convergence rates result.Theorem 4.
Suppose that Assumptions1–4 hold. Let f∈F be a deterministic stationary policy of the control model in (1) and fkk≥1 be the quantized approximations of f as in Lemma 3, then for each x∈S, it holds that(47)Jx,f−Jx,fk≤K31/k1/dα0K1+b1M1K2α0α0−c1+M1K2α0−c1w1x.Proof.
By Lemma1, we can get(48)αxJx,f=rx,f+∫Sqdy|x,f,αxJx,fk=rx,fk+∫SJy,fkqdy|x,f.
Then, by Lemma5 and Theorem 1, we have(49)α0Jx,f−Jx,fk≤αxJx,f−Jx,fk,≤rx,f−rx,fk+∫Sqdy|x,f−∫SJy,fkqdy|x,fk,≤K1dAfx,fkx+supx∈S,π∈Π≤K1K31/k1/d+b1M1α0α0−c1+M1α0−c1w1xK2dAfx,fkx,=K31/k1/dK1+b1M1K2α0α0−c1+M1K2α0−c1w1x,
which yields that(50)Jx,f−Jx,fk≤K31/k1/dα0K1+b1M1K2α0α0−c1+M1K2α0−c1w1x.


## 6. An Example

In this section, we give an example to illustrate our main results.Consider a control problem of hypertension. As it is well known, we can describe the blood pressure with Gaussian distribution, and thus, the quantity of blood pressure may take values inS≔−∞,∞. When the current amount of blood pressure is at x∈S, a controlled amount a is given by a∈A=Ax≔L1,L2 for each x∈S with L1>0. The rate of change of blood pressure is given as follows:(51)qB|x,a≔ax+1∫B−x12πe−y−x2/2dy−ax+1δxB,for x,a∈Kx,a|x∈S,a∈Ax and B∈ℬS, where π is the circumference ratio, and δx⋅ is the Dirac measure at x. It is clear that q⋅|x,a is a transition rate function. We denote by rx,a≔n0x2+n1x−n2a2 the cost of taking control a when the current amount of blood pressure is at x∈S, and regard a as an action. The discount factor is defined by αx≔1/x+1+β for x∈S with a constant β>L1/2. Suppose that the constants nkk=0,1,2 satisfy that(i)
n0>0,n1>1+1/β and n2=ρ2/4βρ−1, where ρ≔n1β+1−β+2(ii)
ρL2>ρL1>2βρ−1Letw1x=x2+1, w2x=x4+1 and q∗x=L1x+1, and then, by Steps 1–4 of the iteration algorithm in Section 4, the approximate optimal value is(52)J∗x=ρn0x2+2ρ−n1+1n0x+ρ−n1+1n0,and the optimal stationary policy is(53)f∗x≔L1,x<U,2βρ−1ρx+1,U≤x≤V,L2,x>V,where U≔ρL1/2βρ−1−1, V≔ρL2/2βρ−1−1.Now, we can construct a sequence of quantizer policies off∗ as follows:(54)fk∗x≔L1,x<U,2βρ−1ρU+2j+12k0+1,k0j≤x​−U<k0j+1,j=0,1,…,k,L2,x>V,where k0≔V−U/k.Now, we computef∗, fk∗, and J∗ by assigning values to parameters β, L1, L2, n1, and n2 as follows:(55)β=12,L1=12,L2=67,n0=1,n1=4,then, the optimal value is J∗x=7/2x2+4x+1/2, and the optimal stationary policy is(56)f∗x≔12,x<16,37x+1,16≤x≤1,67,x>1.The quantizer policiesfk∗ of f∗ are(57)fk∗x≔​12,x<16,​​12+37k56j​+​512,56kj≤x−16<56kj+1,j=0,1,…,k,​67,x>1.Furthermore, the asymptotic approximation of the optimal policyf∗x is given by Figures 1 and 2 when k=15 and k=100, respectively. This verifies fk∗x⟶f∗x for each x∈S, and by Theorem 3, it holds that limk⟶∞Jx,fk=Jx,f.Figure 1
The optimal policyf∗x and fk∗x as k=15.Figure 2
The optimal policyf∗x and fk∗x as k=100.


## 7. Conclusions

In this paper, we are concerned with the asymptotic optimality of quantized stationary policies in CTMDPs with Polish spaces and varying (state-dependent) discount factors. First, we establish the discounted optimal equation (DOE) and give the existence of its solutions. Then, by a relatively simple proof, we obtain the existence of optimal deterministic stationary policies under suitable conditions in Theorem2. Meanwhile, we generalize the relevant conclusions of Ye and Guo [16] in Theorem 1 and Lemma 2. Next, we discretize and incentivize the action space, construct a sequence of policies, namely “quantizer policies,” and obtain the approximation results and the rates of convergence for the optimal policies on the CTMDPs in (1) as in Theorem 3. Finally, we give an example to illustrate the asymptotic optimality.



---

*Source: 1080946-2022-09-10.xml*