# Subgeometric Ergodicity Analysis of Continuous-Time Markov Chains under Random-Time State-Dependent Lyapunov Drift Conditions

Authors: Mokaedi V. Lekgari
Journal: Journal of Probability and Statistics
 (2014)

---

## Abstract

We investigate random-time state-dependent Foster-Lyapunov analysis on subgeometric rate ergodicity of continuous-time Markov chains (CTMCs). We are mainly concerned with making use of the available results on deterministic state-dependent drift conditions for CTMCs and on random-time state-dependent drift conditions for discrete-time Markov chains and transferring them to CTMCs.


---

## Body

## 1. Introduction

The ergodic theory of Markov processes in recent years has received quite substantial attention. With our focus on subgeometric ergodicity, which, loosely speaking, is a kind of convergence that is faster than ordinary ergodicity but slower than geometric ergodicity, much study is needed especially for continuous Markov processes [1]. It is for this reason that interest in this study developed.Meyn and Tweedie [2] pioneered the study of state-dependent drift for Ψ-irreducible Markov processes. Then Connor and Fort [3] found that state-dependent Foster-Lyapunov criteria can be employed not only to determine the ergodic properties of the Markov process Φ
t but also to infer a greater range of convergence rates for Φ
t. The study focused on the process Φ
t subsampled at some deterministic times. In [3] the results of the study were used for the classification of tame chains. Ergodicity in the context of state-dependent drift conditions on a deterministic time index was then studied by many authors, amongst them [4, 5]. Yüksel and Meyn [4] applied their results to “stochastic stabilization over an erasure channel.” The motivation for Zurkowski’s [5] study was its possible applications in network control and event triggered control systems. Although we cannot ignore the practical applications of our study in areas such as stability of stochastic networks, presenting such applications is beyond the scope of our work.Connor and Fort [3] and Yüksel and Meyn [4] studied ergodicity with a drift condition taking the form P
n
(
x
)
V
(
x
)
≤
β
V
(
x
)
+
b
1
C
(
x
) for some deterministic function n
:
X
→
[
1
,
∞
) and a constant β
∈
(
0,1
). According to Theorem  2.1(ii) of [6] a deterministic sequence of functions V
n exists, V
:
X
→
[
1
,
∞
), and satisfies a Foster-Lyapunov drift condition:
(1)
P
V
n
+
1
≤
V
n
-
r
(
n
)
f
+
b
r
(
n
)
1
C
,
n
∈
Z
+
,
for a petite set C
∈
B
(
X
) and a constant b
<
∞ such that V
0 is bounded on C.It has been proved that this Foster-Lyapunov condition holds not just for everyn
∈
Z
+ but also for a sequence of stopping times {
T
n
}, n
∈
Z
+ for some discrete time Markov chain (
Φ
n
)
n
∈
Z
+   [5]. The results of Zurkowski [5] relied heavily on the work of Connor and Fort [3], Meyn and Tweedie [7], and Tuominen and Tweedie [6].This study follows on the work of previous studies to investigate random-time state-dependent drift conditions results on the subgeometric ergodicity for CTMCs in an easy and reader-friendly manner. Amongst the previous authors Connor and Fort [3] studied state-dependent geometric Lyapunov drift condition for a general Markov chain subsampled at some deterministic times. Random-time state-dependent drift studies were then done by Yüksel and Meyn [4] and Zurkowski [5] for general Markov chains and discrete chains, respectively. The work in this study utilizes the results of the above mentioned studies with focus on subgeometric ergodicity for countable state space Q-processes.The paper is organized as follows. In Section2, we have the Preliminaries section which introduces basic notations, definitions, and theorems. The main results are availed in Section 3 which is mostly concerned with subgeometric ergodicity at rate r, in the f-norm. Section 4 is about the conclusions of this study.


## 2. Preliminaries

LetZ
+
=
{
0,1
,
2
,
…
} and let R
+
=
[
0
,
∞
). We let (
Φ
t
)
t
∈
R
+ denote a continuous-time Markov chain (CTMC) on a countable state space (
X
,
B
(
X
)
). The transition function of the continuous-time Markov process is denoted by P
t
(
i
,
j
)
=
P
i
(
Φ
t
=
j
)
=
E
i
[
1
Φ
t
=
j
], where here and hereafter 1
A is the indicator function of set A and P
x and E
x, respectively, denote the probability and expectation of the chain under the condition that Φ
0
=
i.


### 2.1. First Hitting Times

Given a subsetC
⊆
X we let
(2)
ξ
C
δ
=
inf
⁡
{
t
≥
δ
:
Φ
t
∈
C
}
be the first hitting time on C delayed by a constant δ
>
0. In the case when δ
=
0 we have ξ
C
0
=
ξ
C. We also have
(3)
ξ
C
+
=
inf
⁡
{
t
≥
J
1
:
Φ
t
∈
C
}
as the first hitting time on the set C after the first jump J
1 of the process. We also note that ξ
C
+
=
ξ
C if Φ
0
∉
C. If C is a singleton consisting only of state i, then we write ξ
i
δ for ξ
C
δ and equivalently ξ
i
+ for ξ
C
+.


### 2.2.l-Ergodicity

Forl
∈
R, such that l
≥
1, the l
th moment is denoted by m
i
j
(
l
)
:
=
E
i
[
ξ
j
(
l
)
] and m
j
(
l
)
:
=
E
j
[
ξ
j
+
(
l
)
]. By definition the Markov process Φ
t is said to be l-ergodic (or of ergodic degree l) if m
C
(
l
)
<
∞ for some (and then for all) finite nonempty C
⊆
X. For more studies on ergodic degrees, l-ergodicity, and related topics, the interested reader may consult Mao [8] and references therein.


### 2.3. Geometric Ergodicity

According to [3], Φ
t is said to be geometrically ergodic if there exist a function V
:
X
→
[
0
,
∞
) and constants γ
∈
(
0,1
), R
<
∞, such that, for any i
∈
X, ∥
P
t
(
i
,
·
)
-
π
(
·
)
∥
V
≤
R
V
(
i
)
γ
t, which is equivalent to the existence of a scale function V
:
X
→
[
1
,
∞
), a small set C, and constants λ
∈
(
0,1
), b
<
∞, such that P
V
(
i
)
≤
λ
V
(
i
)
+
b
1
C
(
i
).


### 2.4. Subgeometric Rate Function

Let functionr
∈
Λ
0, where Λ
0 is the family of measurable increasing functions r
:
R
+
→
[
1
,
∞
) satisfying (
log
⁡
r
(
t
)
/
t
)
↓
0 as t
↑
∞. Let Λ denote the class of positive functions r
¯
:
R
+
→
(
0
,
∞
) such that for some r
∈
Λ
0 we have
(4)
0
<
lim
⁡
t
inf
⁡
r
¯
(
t
)
r
(
t
)
≤
lim
⁡
t
sup
⁡
r
¯
(
t
)
r
(
t
)
<
∞
.
Then Λ is referred to as the class of subgeometric rate functions [9]. Indeed (4) implies the equivalence of the class of functions Λ
0 with the class of functions Λ. Example of the functions in the class r
∈
Λ is the rate r
(
t
)
=
exp
⁡
(
s
t
1
/
(
1
+
α
)
), α
>
0, s
>
0, which has been discussed only recently in literature. Without loss of generality, we suppose that r
(
0
)
=
1 whenever r
∈
Λ.The properties ofr
∈
Λ
0 which follow from (4) and are to be used frequently in this study are
(5)
r
(
x
+
y
)
≤
r
(
x
)
r
(
y
)
∀
x
,
y
∈
R
+
,
(6)
r
(
x
+
a
)
r
(
x
)
⟶
1
as
x
⟶
∞
,
for
each
a
∈
R
+
.


### 2.5. Modulated Moments

Sufficient conditions forf-ergodicity involve the modulated moments of the first hitting times. Having the subgeometric rate function r
∈
Λ given by (4), a function f
:
X
→
[
1
,
∞
), a subset C
⊆
X, and a constant δ
≥
0, we define the modulated moment of ξ
C
δ by
(7)
G
C
(
i
,
f
,
r
,
δ
)
:
=
E
i
[
∫
0
ξ
C
δ
r
(
t
)
f
(
Φ
t
)
d
t
]
,
i
∈
X
.
Analogous to the modulated moments of the first hitting times ξ
C
δ given as (7) above, the modulated moments of the first hitting times ξ
C
+, namely, G
C
+
(
i
,
f
,
r
), are defined as follows. For a subgeometric rate function r
∈
Λ, a function f
:
X
→
[
1
,
∞
), and a subset C
⊆
X, then
(8)
G
C
+
(
i
,
f
,
r
)
∶
=
E
i
[
∫
0
ξ
C
+
‍
r
(
t
)
f
(
Φ
t
)
d
t
]
,
i
∈
X
.


### 2.6. Subgeometric Rate Ergodicity

Letr
∈
Λ; then the ergodic chain Φ
t is said to be subgeometric ergodic of order r in the f-norm or simply (
f
,
r
)-ergodic if for all x
∈
X; then
(9)
lim
⁡
t
→
+
∞
r
(
t
)
∥
P
t
(
x
,
·
)
-
π
(
·
)
∥
f
=
0
,
where π is the unique invariant distribution of the process for a (signed) measure σ, where ∥
σ
∥
f
=
sup
⁡
|
g
|
≤
f
|
σ
(
g
)
| and f
:
X
→
[
1
,
∞
) is a measurable function.It is known that (9) holds if and only if the Foster-Lyapunov (or just Lyapunov) drift condition holds. It is for this reason that the Lyapunov drift conditions play a very crucial role in our study. The Lyapunov drift conditions provide bounds on the return time to accessible sets thereby availing some control on the Markov process dynamics by focusing on the hitting times on a particular set.We are aware that techniques that rely on small sets in some of the previous studies may become unavailable in the random-time drift setting because a small set forΦ
t may not necessarily be small for Φ
T
n. However, we also know that, for a Ψ-irreducible process, all finite subsets of X are petite and that an accessible closed petite set always exists [10], suggesting our work will be confined to working with petite sets. Then by Theorem  5.5.7 in Meyn and Tweedie [7] we know that if the Markov chain is Ψ-irreducible and aperiodic, then every petite set is small. Therefore all the petite sets in this study are assumed to be small because all chains are assumed to be Ψ-irreducible and aperiodic.In light of the foregoing definitions and notations we are ready to state the following subgeometric rate ergodicity propositions about(
f
,
1
)-ergodicity and (
1
,
r
)-ergodicity.Proposition 1 (see Proposition 2.11 in [11]).
LetΦ
t be an irreducible Markov process, and suppose that, for some ε
>
0 and some function f
:
X
→
[
1
,
∞
) on X, one has E
i
[
f
(
Φ
s
)
]
<
f
(
i
), for all i
∈
X and 0
≤
s
≤
ε. If there exist a petite set C
⊆
X and some constant δ
>
0 such that
(10)
sup
⁡
i
∈
C
G
C
(
i
,
f
,
1
,
δ
)
<
∞
then Φ
t is f-ergodic.Proposition 2 (see Theorem  3.3 in Liu et al. [12]).
An irreducible Markov chainΦ
t is subgeometrically ergodic at rate r in the total variation norm if and only if for some (and then for all) finite subset C
⊂
X one has
(11)
sup
⁡
i
∈
C
G
C
+
(
i
,
1
,
r
)
<
∞
.Proposition 3 (Theorem  3.2 in Liu et al. [12]).
For some finite nonempty setC, sup
⁡
j
∈
C
G
C
+
(
j
,
1
,
r
)
<
∞ if and only if for any finite set B and any i
∈
X, then G
B
+
(
i
,
1
,
r
)
<
∞.Throughout this study the nondecreasing sequence of stopping times{
T
n
:
n
∈
Z
+
} with T
0
=
0 is assumed to be similar to the sequence in Assumption 2.1 of Spieksma  [13] , which is restated as follows. Recursively we define T
0
=
0 and T
n
+
1
=
inf
⁡
{
t
>
T
n
:
Φ
t
≠
Φ
t
-
}, if Φ
T
n is not an absorbing state (i.e., q
Φ
T
n
≠
0). We put T
k
,
k
>
n if Φ
T
n is an absorbing state; then, we have Φ
T
k
=
Φ
T
n. In this sense the sequence {
T
n
}
n
∈
Z
+ is a nondecreasing sequence of stopping times representing successive jump times.


## 2.1. First Hitting Times

Given a subsetC
⊆
X we let
(2)
ξ
C
δ
=
inf
⁡
{
t
≥
δ
:
Φ
t
∈
C
}
be the first hitting time on C delayed by a constant δ
>
0. In the case when δ
=
0 we have ξ
C
0
=
ξ
C. We also have
(3)
ξ
C
+
=
inf
⁡
{
t
≥
J
1
:
Φ
t
∈
C
}
as the first hitting time on the set C after the first jump J
1 of the process. We also note that ξ
C
+
=
ξ
C if Φ
0
∉
C. If C is a singleton consisting only of state i, then we write ξ
i
δ for ξ
C
δ and equivalently ξ
i
+ for ξ
C
+.


## 2.2.l-Ergodicity

Forl
∈
R, such that l
≥
1, the l
th moment is denoted by m
i
j
(
l
)
:
=
E
i
[
ξ
j
(
l
)
] and m
j
(
l
)
:
=
E
j
[
ξ
j
+
(
l
)
]. By definition the Markov process Φ
t is said to be l-ergodic (or of ergodic degree l) if m
C
(
l
)
<
∞ for some (and then for all) finite nonempty C
⊆
X. For more studies on ergodic degrees, l-ergodicity, and related topics, the interested reader may consult Mao [8] and references therein.


## 2.3. Geometric Ergodicity

According to [3], Φ
t is said to be geometrically ergodic if there exist a function V
:
X
→
[
0
,
∞
) and constants γ
∈
(
0,1
), R
<
∞, such that, for any i
∈
X, ∥
P
t
(
i
,
·
)
-
π
(
·
)
∥
V
≤
R
V
(
i
)
γ
t, which is equivalent to the existence of a scale function V
:
X
→
[
1
,
∞
), a small set C, and constants λ
∈
(
0,1
), b
<
∞, such that P
V
(
i
)
≤
λ
V
(
i
)
+
b
1
C
(
i
).


## 2.4. Subgeometric Rate Function

Let functionr
∈
Λ
0, where Λ
0 is the family of measurable increasing functions r
:
R
+
→
[
1
,
∞
) satisfying (
log
⁡
r
(
t
)
/
t
)
↓
0 as t
↑
∞. Let Λ denote the class of positive functions r
¯
:
R
+
→
(
0
,
∞
) such that for some r
∈
Λ
0 we have
(4)
0
<
lim
⁡
t
inf
⁡
r
¯
(
t
)
r
(
t
)
≤
lim
⁡
t
sup
⁡
r
¯
(
t
)
r
(
t
)
<
∞
.
Then Λ is referred to as the class of subgeometric rate functions [9]. Indeed (4) implies the equivalence of the class of functions Λ
0 with the class of functions Λ. Example of the functions in the class r
∈
Λ is the rate r
(
t
)
=
exp
⁡
(
s
t
1
/
(
1
+
α
)
), α
>
0, s
>
0, which has been discussed only recently in literature. Without loss of generality, we suppose that r
(
0
)
=
1 whenever r
∈
Λ.The properties ofr
∈
Λ
0 which follow from (4) and are to be used frequently in this study are
(5)
r
(
x
+
y
)
≤
r
(
x
)
r
(
y
)
∀
x
,
y
∈
R
+
,
(6)
r
(
x
+
a
)
r
(
x
)
⟶
1
as
x
⟶
∞
,
for
each
a
∈
R
+
.


## 2.5. Modulated Moments

Sufficient conditions forf-ergodicity involve the modulated moments of the first hitting times. Having the subgeometric rate function r
∈
Λ given by (4), a function f
:
X
→
[
1
,
∞
), a subset C
⊆
X, and a constant δ
≥
0, we define the modulated moment of ξ
C
δ by
(7)
G
C
(
i
,
f
,
r
,
δ
)
:
=
E
i
[
∫
0
ξ
C
δ
r
(
t
)
f
(
Φ
t
)
d
t
]
,
i
∈
X
.
Analogous to the modulated moments of the first hitting times ξ
C
δ given as (7) above, the modulated moments of the first hitting times ξ
C
+, namely, G
C
+
(
i
,
f
,
r
), are defined as follows. For a subgeometric rate function r
∈
Λ, a function f
:
X
→
[
1
,
∞
), and a subset C
⊆
X, then
(8)
G
C
+
(
i
,
f
,
r
)
∶
=
E
i
[
∫
0
ξ
C
+
‍
r
(
t
)
f
(
Φ
t
)
d
t
]
,
i
∈
X
.


## 2.6. Subgeometric Rate Ergodicity

Letr
∈
Λ; then the ergodic chain Φ
t is said to be subgeometric ergodic of order r in the f-norm or simply (
f
,
r
)-ergodic if for all x
∈
X; then
(9)
lim
⁡
t
→
+
∞
r
(
t
)
∥
P
t
(
x
,
·
)
-
π
(
·
)
∥
f
=
0
,
where π is the unique invariant distribution of the process for a (signed) measure σ, where ∥
σ
∥
f
=
sup
⁡
|
g
|
≤
f
|
σ
(
g
)
| and f
:
X
→
[
1
,
∞
) is a measurable function.It is known that (9) holds if and only if the Foster-Lyapunov (or just Lyapunov) drift condition holds. It is for this reason that the Lyapunov drift conditions play a very crucial role in our study. The Lyapunov drift conditions provide bounds on the return time to accessible sets thereby availing some control on the Markov process dynamics by focusing on the hitting times on a particular set.We are aware that techniques that rely on small sets in some of the previous studies may become unavailable in the random-time drift setting because a small set forΦ
t may not necessarily be small for Φ
T
n. However, we also know that, for a Ψ-irreducible process, all finite subsets of X are petite and that an accessible closed petite set always exists [10], suggesting our work will be confined to working with petite sets. Then by Theorem  5.5.7 in Meyn and Tweedie [7] we know that if the Markov chain is Ψ-irreducible and aperiodic, then every petite set is small. Therefore all the petite sets in this study are assumed to be small because all chains are assumed to be Ψ-irreducible and aperiodic.In light of the foregoing definitions and notations we are ready to state the following subgeometric rate ergodicity propositions about(
f
,
1
)-ergodicity and (
1
,
r
)-ergodicity.Proposition 1 (see Proposition 2.11 in [11]).
LetΦ
t be an irreducible Markov process, and suppose that, for some ε
>
0 and some function f
:
X
→
[
1
,
∞
) on X, one has E
i
[
f
(
Φ
s
)
]
<
f
(
i
), for all i
∈
X and 0
≤
s
≤
ε. If there exist a petite set C
⊆
X and some constant δ
>
0 such that
(10)
sup
⁡
i
∈
C
G
C
(
i
,
f
,
1
,
δ
)
<
∞
then Φ
t is f-ergodic.Proposition 2 (see Theorem  3.3 in Liu et al. [12]).
An irreducible Markov chainΦ
t is subgeometrically ergodic at rate r in the total variation norm if and only if for some (and then for all) finite subset C
⊂
X one has
(11)
sup
⁡
i
∈
C
G
C
+
(
i
,
1
,
r
)
<
∞
.Proposition 3 (Theorem  3.2 in Liu et al. [12]).
For some finite nonempty setC, sup
⁡
j
∈
C
G
C
+
(
j
,
1
,
r
)
<
∞ if and only if for any finite set B and any i
∈
X, then G
B
+
(
i
,
1
,
r
)
<
∞.Throughout this study the nondecreasing sequence of stopping times{
T
n
:
n
∈
Z
+
} with T
0
=
0 is assumed to be similar to the sequence in Assumption 2.1 of Spieksma  [13] , which is restated as follows. Recursively we define T
0
=
0 and T
n
+
1
=
inf
⁡
{
t
>
T
n
:
Φ
t
≠
Φ
t
-
}, if Φ
T
n is not an absorbing state (i.e., q
Φ
T
n
≠
0). We put T
k
,
k
>
n if Φ
T
n is an absorbing state; then, we have Φ
T
k
=
Φ
T
n. In this sense the sequence {
T
n
}
n
∈
Z
+ is a nondecreasing sequence of stopping times representing successive jump times.


## 3. Main Results

The content presented in this section is on the subgeometric rate ergodicity ofΦ
t, where Φ
t is assumed to be an irreducible and aperiodic CTMC. We have results presented as Theorems 4, 6, and 7. Theorem 4 is a modified version of Theorem  3.1 in [12], while Theorem 6 is a continuous counterpart of Proposition  5.1.2 in Zurkowski [5]. Finally we have Theorem 7 which follows from Theorem  2.16 of [11].Theorem 4.
LetΦ
t be an irreducible and aperiodic chain. Further suppose that for some (and then for any) finite nonempty set C
⊆
X there exist a function V
:
X
→
(
0
,
∞
) and a constant b
∈
R such that, for an increasing sequence of stopping times {
T
i
,
i
∈
N
+
},
(12)
∑
Φ
T
n
+
1
∈
C
‍
q
Φ
T
n
,
Φ
T
n
+
1
V
(
Φ
T
n
+
1
)
≤
-
E
Φ
T
n
[
r
(
ξ
C
)
]
+
b
1
{
Φ
T
n
∈
C
}
,
∀
Φ
T
n
∈
C
;
then for any r
∈
Λ the chain Φ
t is (
1
,
r
)-ergodic.Proof.
It is worthwhile to realize that the chainΦ
T
n is also aperiodic and irreducible and that C is petite and hence small and that V is bounded on C. Let ζ
C
=
min
⁡
{
t
≥
0
:
Φ
T
n
∈
C
}. We know that ξ
B
≤
ζ
B  ∀
B
∈
B
+
(
X
), because the chain Φ
T
n can miss some of the visits of the chain Φ
t to the set B, so that, for any r
∈
Λ, we have
(13)
sup
⁡
j
∈
C
E
j
[
∫
0
ξ
C
‍
r
(
t
)
d
t
]
≤
sup
⁡
j
∈
C
E
j
[
∫
0
ζ
C
‍
r
(
t
)
d
t
]
which implies that
(14)
sup
⁡
j
∈
C
G
C
(
j
,
1
,
r
,
δ
)
≤
sup
⁡
j
∈
C
E
j
[
∫
0
ζ
C
‍
r
(
t
)
d
t
]
=
sup
⁡
j
∈
C
G
C
+
(
j
,
1
,
r
)
<
∞
for any j
∈
C. We get (14) owing to the equivalence of Λ
0 and Λ and supposing that r
∈
Λ
0 and hence satisfies the submultiplicative property (5). By employing Proposition 3 we get that (14) implies G
i
+
(
i
,
1
,
r
)
<
∞ for any i
∈
X. Then, by Proposition 2, the set C is (
1
,
r
)-regular; hence we conclude that the chain is (
1
,
r
)-ergodic.It is worth noting that, in practice, the drift condition in Theorem4 above is not easy to verify because it involves the unknown information E
Φ
T
n
[
r
(
ξ
C
)
]. The practically more favorable drift condition is stated in Corollary 5. It was Mao [8] who investigated l-ergodicity when l is restricted to Z
+; then Liu et al. [12] extended the results to the case when l
∈
R
+. Under random-time state-dependent drift function, we state the following corollary which is a modified version of Corollary 2.1 of Liu et al. [12].Corollary 5.
Letl
∈
R
+ with l
≥
1. Then the chain Φ
t is l-ergodic (i.e., sup
⁡
i
∈
C
E
i
[
ξ
C
+
(
l
)
]
<
∞) if and only if for some finite nonempty set C and an increasing sequence of stopping times {
T
i
} there exist ⌊
l
⌋
+
1 finite nonnegative functions V
(
k
)
(
Φ
T
n
), 0
≤
k
≤
⌊
l
⌋, and a constant b such that V
(
0
)
(
Φ
T
n
)
≥
E
Φ
T
n
[
ξ
0
l
^
] and
(15)
∑
Φ
T
n
+
1
∈
C
‍
q
Φ
T
n
,
Φ
T
n
+
1
V
(
m
+
1
)
(
Φ
T
n
+
1
)
≤
-
(
l
^
+
m
+
1
)
V
(
m
)
(
Φ
T
n
)
+
b
1
{
Φ
T
n
∈
C
}
for 0
≤
m
≤
⌊
l
⌋
-
1, where l
^
=
l
-
⌊
l
⌋ and ⌊
l
⌋ denotes the integer-part function, that is, the largest integer which is smaller than or equal to l.Proof.
The proof of this corollary is analogous to the proof of Corollary 2.1 of Liu et al. [12] and follows naturally therefrom.Remark. In a particular case of Corollary 5, the chain is said to be subgeometrically ergodic at rate r
(
t
)
=
t
l
∨
1, for some l
∈
Z
+, in the total variation norm if and only if we have nonnegative functions V
(
0
)
,
…
,
V
(
k
+
1
) on X, with C
⊂
X and b
∈
R such that V
(
0
)
≥
1, and
(16)
∑
Φ
T
n
+
1
∈
C
‍
q
Φ
T
n
,
Φ
T
n
+
1
V
(
m
+
1
)
(
Φ
T
n
+
1
)
≤
-
(
l
^
+
m
+
1
)
V
(
m
)
(
Φ
T
n
)
+
b
1
{
Φ
T
n
∈
C
}
for 0
≤
m
≤
n and any Φ
T
n
∈
C.Also the Lyapunov drift in Theorem4 and Corollary 5 which is of the
(17)
∑
j
∈
C
‍
q
i
j
V
(
j
)
≤
-
λ
W
(
i
)
+
b
1
C
(
i
)
,
∀
i
∈
C
,
where V
,
W
:
→
(
0
,
∞
) and λ
∈
R, is used here and in the rest of this paper for convenience. We note that for the Q-matrix Q
=
[
q
i
j
] and the real-valued function V on X we have (
Q
V
)
(
i
)
∶
=
∑
j
∈
C
‍
q
i
j
V
(
j
)
,
∀
i
∈
X.Theorem6 which follows deals with subgeometric rate convergence of f-ergodic chains, in the case when f is not a total variation norm. In a nutshell we will be dealing with (
f
,
r
)-regularity of the petite subset C
⊂
X.Theorem 6.
Suppose thatΦ
t is a ψ-irreducible and aperiodic Markov chain. Further suppose that there are functions V
:
X
→
(
0
,
∞
) and f
,
g
:
X
→
[
1
,
∞
), where V is bounded on the small set C, constant ε
>
0, b
∈
R, and r
∈
Λ such that, for an increasing sequence of stopping times {
T
n
,
n
∈
Z
+
},
(18)
E
[
V
(
Φ
T
n
+
1
)
∣
F
T
n
]
≤
V
(
Φ
T
n
)
-
g
(
Φ
T
n
)
+
b
1
{
Φ
T
n
∈
C
}
E
T
n
[
∫
0
ε
‍
f
(
Φ
T
s
)
r
(
T
s
)
d
T
s
]
≤
g
(
Φ
T
n
)
;
then Φ
t is (
f
,
r
)-ergodic.Proof.
For allΦ
T
n
∈
X and for all stopping time T, by Dynkin’s inequality, we get
(19)
E
[
∑
n
=
0
T
-
1
‍
g
(
Φ
T
n
)
∣
F
T
n
]
≤
V
(
Φ
T
0
)
+
b
E
[
∑
n
=
0
T
-
1
‍
1
{
Φ
T
n
∈
C
}
]
;
then for all Φ
T
n
∈
C we get
(20)
E
[
∑
n
=
0
ξ
C
-
1
‍
g
(
Φ
T
n
)
∣
F
T
n
]
≤
V
(
Φ
T
0
)
+
b
E
[
∑
n
=
0
ξ
C
-
1
‍
1
{
Φ
T
n
∈
C
}
]
≤
V
(
Φ
T
0
)
+
b
<
∞
because V is bounded on C. Again as in Theorem 4 we have that ξ
B
≤
ζ
B, for all B
∈
B
+
(
X
), where ζ
B
δ
=
min
⁡
{
t
>
δ
:
Φ
T
n
∈
B
} is the sampled hitting times for the chain Φ
T
n. Then for
all
i
∈
C
(21)
E
i
[
∫
0
ξ
C
‍
f
(
Φ
s
)
r
(
s
)
d
s
]
≤
E
i
[
∫
0
ξ
C
‍
g
(
Φ
u
)
d
u
]
≤
E
i
[
∫
0
ξ
~
C
‍
g
(
Φ
s
)
d
s
]
=
E
i
[
∑
k
=
1
ξ
˘
C
‍
g
(
Φ
˘
k
)
]
≤
E
i
[
∑
n
=
0
ξ
-
1
‍
g
(
Φ
T
n
)
∣
F
Φ
T
n
]
<
V
(
Φ
T
0
)
+
b
<
∞
which implies (
f
,
r
)-ergodicity of the chain Φ
t.The equation(22)
E
i
[
∫
0
ξ
~
C
‍
g
(
Φ
s
)
d
s
]
=
E
i
[
∑
k
=
1
ξ
˘
C
‍
g
(
Φ
˘
k
)
]
is taken from formula (15) in [14, Section  5], with the obvious notational changes of course.The following theorem generalizes the results of(
f
,
r
)-ergodicity in Theorem 6 which can be established by combining the results of Proposition 1 (i.e., (
f
,
1
)-ergodicity) with Theorem 4 (i.e., (
1
,
r
)-ergodicity). This theorem is similar to Proposition  5.1.4 in [5] and Theorem  2.8 in [15].Theorem 7 (borrowed from Theorem 2.16 in [11]).
LetΦ
t be an irreducible and aperiodic chain. Further suppose that for some (and then for any) finite nonempty set C
⊆
X there exist a function f
:
X
→
(
1
,
∞
) and constants s and ε such that, for an increasing sequence of stopping times {
T
n
,
n
∈
Z
+
}, E
Φ
T
n
[
f
(
Φ
(
s
)
)
]
≤
f
(
Φ
T
n
), for all Φ
T
n
∈
X and 0
≤
s
≤
ε. Then, for some r
∈
Λ and a constant δ
>
0,
(23)
sup
⁡
Φ
T
n
∈
C
G
C
(
Φ
T
n
,
f
,
1
,
δ
)
a
n
d
sup
⁡
Φ
T
n
∈
C
G
C
(
Φ
T
n
,
1
,
r
,
δ
)
.
Then Φ
t is subgeometrically ergodic at a rate Ψ
1
(
r
)
∨
1 in the (
Ψ
2
(
f
)
∨
1
)-norm, where the functions Ψ
1
,
Ψ
2
:
X
→
[
1
,
∞
) are a pair of inverse Young functions.The drift condition for the chain in Theorem7 is not easy to find but an easier way to verify drift is given in [11, Corollary  2.17]. It is restated as the following corollary.Corollary 8.
Let the chainΦ
t be as in Theorem 7, for a fixed constant α
∈
(
0,1
] and positive constants c
η such that, for an increasing sequence of stopping times {
T
n
,
n
∈
Z
+
},
(24)
∑
Φ
T
n
+
1
∈
C
‍
q
Φ
T
n
,
Φ
T
n
+
1
V
η
(
Φ
T
n
+
1
)
≤
-
c
η
V
η
-
α
(
Φ
T
n
)
+
b
1
{
Φ
T
n
∈
C
}
,
η
∈
[
α
,
1
]
for every p
∈
(
0,1
); then the chain Φ
t is subgeometrically ergodic at rate (
1
+
t
)
(
1
-
p
)
(
1
-
α
)
/
α
(
log
⁡
t
)
b
∨
1, in the (
(
V
(
1
-
α
)
p
(
log
⁡
V
)
-
b
)
∨
1
)-norm.


## 4. Conclusion

Through this study we have succeeded in the subgeometric rate ergodicity analysis of random-time state-dependent Foster-Lyapunov drift conditions for CTMCs. However there is still more work to be done which could pave way for more research especially on applications. The studies we relied on such as Connor and Fort [3] applied the results of their work to “tame” chains (which technically speaking are any chains with subgeometric drift ϕ
(
t
)
~
c
t
[
ln
⁡
t
]
-
α). Also the study on continuous-time controlled Markov chains on a countable state space by [11] was applied on discounted and average reward optimality criteria. Therefore our future research will focus on making any possible improvements to this study by refining our results and identifying applications of our results. Indeed interest on the same has developed from research fields such as “control and optimization theory,” “information theory,” and “queuing theory.”



---

*Source: 274535-2014-08-31.xml*