数理解析研究所講究録
第 835 巻 1993 年 99-109

99

可算状態マルコフ連鎖における定常絶対確率の連続性について
SUFFICIENT CONDITIONS FOR CONTINUITY OF
STATIONARY ABSOLUTE PROBABILITIES IN
COUNTABLE STATE MARKOV CHAINS

和歌山大教育 門田 良信 (Yoshinobu Kadota)
1. Introduction
We consider a space $oi$ discrete time stationary Markov chains, which are abbreviatedas(MC’s’

henthforth, with a countable state space. In the earlier paper[8], Kadota

proposed recurrent conditions which establish a Laurent series expansions of the MC.
The present paper has two purposes, defining the similar type conditions on a spaceof MC’s. The first purpose$i_{t)}\neg$

to examine the conditions by comparing with each otheror by finding counter examples. The second is to show that those conditions assure
continuity of cofficients of the Laurent series on a parameter space.
Our problems have been considered in the theory of Markov decision processes withaverage and sensitive discount criteria. The Laurent series expansion were first obtainedin a complete sense by Miller and$\backslash ^{r}einott[10]$and Veinott[12] in a finite state$spac\hat{c}$.

When the state space is countable, the series does not exist in general. It requires somerecurrence conditions. Wijngaard[13] and Dietz and Nallaw[3] have obtained the series,

using a quasi-compact condition for transition probabilities. Kadota[7] has obtained
it, using a Do\‘eblin condition which is equivalent to the quasi-compactness. Zijm[14]shows the existence and continuity of the first two terms of the serieses, in tha case that
all recurrent classes of the MC’s are aperiodic. Our conditions are so loose that those100
results are contained as special cases and they admit that each MC have countably

manv periodic recurrent classes. They supply a space of MC’s to the $cond\iota tons$ of
Dekker and Hordijk[2].
We now state our model. Let a state spase$1S$

denoted by$it^{)}s$$S$

be a non-empty countable set. A MC

transition probabilities $\{p_{ij}; i,j\in S\}$ , where$p_{ij}$is the conditional

probability that a system moves from $i\in S$ to $j\in S$ in a unit time. We assume

throughout the paper that $0\leqq p_{ij}\leqq 1$ for any $i,j\in S$ and that $\sum_{f\in s}p;;=1$ for
any $i\in S$ . Let
denote by$P$ “$p_{i^{1}j}=p_{\dot{\iota}j}$and$p_{ij}^{n}= \sum_{k\in S}p_{ik}p_{kj}^{n-1}$

the Markov matrix whose (i,j)-component is given by

the unit matrix. Assoclated$P^{0}=I$

for $i,j\in S$ and $n=1,2,$$w\iota thP^{n}$$p_{jj}^{n}$$\ldots$.

We. If $n=0$ , let. a transition probabilitv function is given

by $p^{n}(i, E)= \sum_{jE}p^{n}$ for any $i\in S$ and $E\subset S$ .
From the mean ergodic theorem of the MC, there exists

for any$p_{ij}^{k})/n$by$p_{1i}$.

$i,$$j\in S$ .

Denote by$P^{\cdot}$

the matrix whose $(i, j)$-component is given

Then we have $PP^{*}=PP=P^{*}=(P^{*})^{2}$ .

$p^{*}(i, E)= \lim\inf_{narrow\infty}(\sum_{k=1}^{n}p^{k}(i, E))/n$$p_{ij}^{*}= \lim_{narrow\infty}(\sum_{k=1}^{n}$for any $i\in S$ and

Associated with $P’$ , let$E\subset S$.

We need notation related to the ergodic theory of the MC’s. A MC separates

into the subsets $R$ of recurrent states and$T$of transient states. Thenrecurrent classes $\{E_{a}; a\in A\}$ . Each recurrent classclass$E_{a}$

is divided into penodic classes

If $d_{a}=1$ , we call$i\in E$

. If

sub-MC’s$E$$E_{a}$${}_{a}C_{1},{}_{a}C_{2},$$E_{a}$$R$$S$

consists ofis positive or null. A recurrent$\ldots,{}_{a}C_{d_{a}}$. We call$d_{a}$a period of$E_{a}$.

aperiodic. A subset $E\subset S$ is called closed $1fp(i, E)=1$ for any

is closed, we can consider a sub-MC$\{p_{\dot{\iota}j^{\alpha}}^{d} ; i, j\in {}_{a}C_{r}\}$for $r=1,2..$ . . ,$d_{a}$$\{p_{\iota_{J}}, i, j\in E\}$. Since. Then we have$\{p_{\iota j^{a}}^{d} ; i,j\in {}_{a}C_{r}\}$$d_{a}$-step

is aperiodic, we

101
have

$\lim_{n}arrow\infty P_{ij}^{nd_{a}}=p_{ij^{Q}}^{d*}=p_{i}^{d_{Q}}$for$i\in {}_{a}C_{r}$. We define $p^{d_{a}}(i, E)$ fromsimilar to $p(i, E)$ . If $d=1.c.m.\{d_{a} ; a\in A\}$ exists, letNote thatfor any$p^{d}{}^{t}(i, E)$$\{p_{j}^{d_{j}} ; i,j\in S\}$‘
$p_{i}^{d_{j^{a}}}$by$\lim$

inf

be a d-step MC.

is also defined for $i\in S$ and $E\subset S$ and that $p^{d*}(i, E)=p^{d_{a}}(i, E)$

and $E\subset S$ .

$i\in {}_{a}C_{\alpha}$

Let $A=(a_{ij} ; i.j\in S)$ be a countable-dimensional matrix. Define the norm of
by $||A||= \sup\{\sum_{j\in S}|a_{ij}|;i\in S\}$ . If 1 $A||<\infty,$by$\sigma(S)$is called bounded. We denotethe set of bounded matrices. Note that if $A,$ $B\in\sigma(S)$ , the matrix product$AB\in\sigma(S)$

If

$A$

$A$

and that 1 AB.

for any $E\subset S$ , then $P^{n}-P^{*}\in\sigma(S)$ . Denoting $S(i)_{n}^{+}=\{j\in$$p^{*}(i, E)= \sum_{;\epsilon E}p_{ij}$$S;p(f)_{jj}^{n}\geqq p_{ij}^{*}\}$$||\leqq\Vert A\Vert\Vert B||$

, we haveII$P^{n}-P^{\cdot}||=2 \sup_{i\in S}\{p^{n}(i, S(i)_{n}^{+})-p^{r}(i, S(i)_{n}^{+})\}$For the sub-MC’s $\{p_{\iota j}, i,j\in R\},$ $\{p;_{j} ; i, j\in E_{a}\}$ andnorms by 1I$\Vert_{R}$, Il$\Vert_{a}$and II$\{p_{ij^{a}}^{d} ; i,j\in {}_{a}C_{\alpha}\}$, respectively. For instance,$||_{a,\alpha}$.

, we denote their$\Vert P^{nd_{a}}-P^{\cdot}||_{a,\alpha}=$. Note that $||A||_{a,\alpha}\leqq||A||_{a}\leqq||A||R\leqq||A||$ for$\sup\{\sum_{i\epsilon {}_{a}C_{\alpha}}|p_{\iota j}^{nd_{a}}-p_{\iota j}|;i\in {}_{a}C_{\alpha}\}$any $A\in\sigma(S)$ .
We sometimes denote $p^{n}(i, E)-p(\dot{s}, E)$ by $(p^{n}-p’)(i, E)$ for simplicity. For $A=$$(a_{1f}),$$B=(b_{ij})\in\sigma(S)$, We often denote$\sum_{i\in s^{\urcorner}}a_{ij}b(j, E)$

by ab $(i, E)$ .

We are mainly concerned with a space of MC’s, denoting $\{P(f);f\in F\}$ with aparameter $f$ in a non-empty set $F$ . The definitions and notation above are applied

to each $P(f)$ .
$p^{d}$

Then, we should denote, etc, for any $f\in F$ by $P(f)^{n},$$d(f)_{a},$$p(f)^{d_{d}},$$p(f)^{d}$$P^{n},$$p(f)_{ij}^{*},$$p_{lj},$

$P,$

$R,$

$T,$

$E_{a},$$A,{}_{a}C_{r},$$d_{a},$$p^{d_{a}}$,$P(f),$ $R(f),$ $T(f),$ $E_{a}(f),$ $A(f),{}_{a}C(f)_{r}$ ,, etc, respectively. But those notation is bothersome, then we

102
often omit the notation$Rp(f)^{nd_{4}}(i, E\cap E_{a})$$f$.

For instance, we denote $R(f)p(f)^{nd(j)_{a}}(i, E\cap E(f)_{a})$ by.

2. Conditions and results.
In this section we set up our conditions and state our results. The proofs of them

are explalned in the next section.

Condition (A). (A-1) For a family of MC’s, there exist a positive integer $N$ , a
positive number 6 and a family of subsets $K(f)\subset S$ satisfying the following conditions.
(A-l-l) $p(f)^{N}(i, K(f))\geqq\delta>0$ for any $i\in S$ and $f\in F$ , where $N$ and$\delta$areconstants independent of $i\in S$ and $f\in F$ .

(A-1-2) $K(f)\cap T(f)=\emptyset$ for any $f\in F$ .
(A-1-3) Each ${}_{a}C(f)_{\alpha}\cap K(f)$ consists of only one state for $a\in A,$$d_{a}-1$$\alpha=0,$

1,)

and $f\in F$ .

(A-2) $\sup\{d(f)_{a}, a\in A(f), f\in F\}<\infty$ .
From (A-2) there is a least common multiplier of all $d(f)_{a}$ . We denote it by$d$.

According to Hordijk[6], we say a MC satisfies the Do\‘eblin condition, if there are an

integer $N\geqq 1$ , a number $S>0$ and a finite set $K$ such that$p^{N}(i, K)\geqq\delta$

for any $i\in S$ .

A family of MC’s satisfies simultaneous Do\‘eblin conditio $ns(simD)$ if evry MC satisfies

the Do\‘eblin condition and if $N,$$\delta,$$K$

are taken independent of $f\in F$ . Condition (A)does not assume $P(f)$ satisfies Do\‘eblin condotion, since $K(f)$ is not necessarily finite.

Condition (B). (B-1) The same condition with (A-2).

103(B-2) There exist a positive integer $N$ and$\epsilon_{0}>0$such that$|p(f)^{Nd}(i, E)-p(f)^{d*}(i, E)| \leqq\frac{1}{2}(1-\epsilon_{0})$for all $i\in S,$ $E\subset S$ and $f\in F$ .

Condition (C). There exists a constant $B>0$ such that$| \sum_{k=1}^{n}\{p(f)^{k}(i, E)-p(f)^{*}(i, E)\}|\leqq B$

for all $i\in S,$ $E\subset S,$ $f\in F$ and$n=1_{\rangle}2,$$\ldots$.

Condition (D). There exists $\lim_{narrow\infty}(\sum_{k=1}^{n}p(f)^{k}(i, E))/n=p(f)(i, E)$
uniformly in $i\in S,$ $E\subset S$ , and in $f\in F$ .
It is shown that Condition (D) implies (A-2) and$f\in F$ .

$\sum_{jS}p_{\dot{\iota}}=1$

for any $i\in S$ and

Next Theorem 1 gives the relations of Conditions (A) $\sim(D)$ . Corollary 2

gives equivalent conditions to (B). If $d=1$ , the (a-3) of Corollary 2 is reduced to to thecondition used by [14].
Theorem 1. Condition $(A)im$plies $(B),$ ($B\rangle$

implies $(C)$ , and $(C)$ implies $(D)$ .

Corollary 2. Supposes Condi tion (B-l) holds. Then,(a) following condi tions are $eq$ uivalen .
$t$(a-l) There are positive integers $1lf,$ $N$ an a positi$d$$\sup_{i\epsilon\tau}p(f)^{Af}(i, T(f))\leqq 1-\epsilon_{0}$$1-\epsilon_{0}$$ve$number$\epsilon_{0}$sa tisfyin th at$g$for all $f\in F$ and th at $\Vert P(f)^{Nd}-P(f)^{d}$ ‘

for all $f\in F,$ $a\in A(f)$ an $d\alpha=0,1,$$\ldots,$$d_{a}-1$$||_{\alpha}\leqq$.

(a-2) Condition (B-2).

(a-3) There exists $\lim_{narrow\infty}||P(f)^{nd}-P(f)^{d}$ ‘ $\Vert=0u$ niformly in $f\in F$ .
(b) If (a) holds, there exists a positi $ve$ number $BS$ tlch th at$\sum_{n}^{\infty_{=0}}\Vert\sum_{r=0}^{d-1}p(f)^{r}$

104$(P(f)^{nd}-P(f))||\leqq C$

.

$con$

vergin uniformly in $f\in F$ .
$g$

We give a Laurent series expansion which is seen in [8]. Let any $f\in F$ be fixed.
Then, we omit the notation$f$. For $0\leqq\beta<1$ , let$H_{\beta}(i, E)= \sum_{n=0}^{\infty}(p^{n}(i, E)-p(i, E))\beta^{n}$.

If Condition (C) holds, there exists $H(i, E)= \lim_{\betaarrow 1+}H_{\beta}(i, E)$ for any $i\in S$ and$E\subset S$. We denote $H_{ij}=H(i, \{j\})$ for $j\in S$ . Then it holds that $H(i, E)= \sum_{;\epsilon}{}_{E}H_{ij}$for any $i\in S$ and $E\subset S$ and that $||H||\leqq 2B$ . Letting $\beta=1/(1+\rho)$ for $0<\beta<1$ ,denote$\beta H_{\beta}$by$H_{\rho}$

and $H$ by$H_{0}= \lim_{\rhoarrow 0+}H_{\rho}$.

Theorem 3. Suppose $Condi$ tion $(C)$ holds.
(a) Let $\rho_{0}>0$ be $\rho_{0}||H||<1$ . Th en,$\sum_{n=0}^{\infty}\beta^{n+1}p^{n}(i, E)=\frac{\perp}{\rho}p(3, E)+\sum_{n=0}^{\infty}(-p)^{n}H^{n+1}(i,E)$(1)

for any $i\in S$ and $E\subset S$ an $d0<\rho\leqq\rho_{0}$ .
$(b)$

For $0\leqq\rho<\infty_{2}H_{\rho}$ satisfies uniq $uly$ in$=\beta(I-\beta P^{\cdot})$$\sigma(S)$that $(I-\beta P)H_{\rho}=H,(I-\beta P)$

and that $PH_{p}=H_{\rho}P^{\cdot}=O$ , where$O$

is the zero ma trix whose

components are all zero.

In order to show continuity of the coefficients on the right of (1), we assume $F$ is asubset of a metric space (X, dist), where dist $(f, g)$ is a metric from $(f, g)\in XxX$ tothe set of real numbers. We study two kinds of continuity conditions for the transition
probabilities.
Condition(I). For any$i,$$j\in S,$ $p(f)_{\iota j}$

is continious in $F\in F$ .

Condition(II). The Markov matrix $P(f)$ is continuous on $F$ with respect to the

105
norm

$||$

. . Namely,$||$$\lim_{garrow f}$

Il $P(g)-P(f)||=0$ for any $f\in F$ .

It is clear that Condition (II) implies Condition (I). Really, Condition (II) asserts

uniform continuity of $p(f)(i, E)$ in $i\in S$ and$E\subset S$.

Lemma 4. (a) $Su$ ppose $Con$ dition (I) holds. Then $p(f)^{n}(i, E)$ is $con$ tinuous uniformly in $E\subset S$ for any $i\in S$ and $n=1,2,$$(b)$$\ldots$.

Suppose $Condi$ tion (II) holds. Then $P(f)^{n}$ is cont in uous on $F$ with respect tothe norm.

Theorem 5. Suppose $Condi$ tion $(D)$ holds.
(a) If $\{P(f);f\in F\}$ satisfies $c_{011}di$ tion (I), then $p(f)^{*}(i, E)$ is cont in $uo$ us on $F$

uniformly in $E\subset S$ for any $i\in S$ .
$(b)$

If $\{P(f);f\in F\}$ sa tisfies $Con$ dition (II), then $p(f)$ is contin $uo$ us on $Fwi$ threspect to the norm.

Theorem 6. Suppose Condition $(C)$ holds.
$(a)$

If $\{P(f);f\in F\}$ satisfies Condi tion (I), then $H(f)^{n}(i, E)$ is cont inuous on $F$

uniformly in E C
$(b)$

$S$

for any $i\in S$ and $n=1.2,$$\ldots$.

ff $\{P(f);f\in F\}$ satisfies Condition (II), then $H(f)^{n}$ is cont inuo us on $F$ withrespect tothe norm for $n=1,2,$$\ldots$.

Note that if Condition (C) and Condition (I) (or (II)) hold, then, any MC $P(f)$ has

the Laurnt series of which all the coefficients are continuous in the sense of (a) (or $(b)$ )in Theorem 6 and 6.
3. Lemmas.

This section gives Lemmas and short comments for obtaining the results in section

1062.
Lemma 1 of [6], $wh_{1}ch$ shows equivalent conditions to$\sum_{i\epsilon s}p_{ij}=1$

for $i\in S$ , play afundamental role in our arguments. Next Lemma 7 gives an ergodic theorem. Lemma
8 considers properties of the norm.

Lemma 7. Suppose (A-2) and $\sum_{j\in S}p_{ij}=1$ for an.$vi\in S$ . Then,
and

$(a)$

$p(i, E)= \frac{1}{d}\sum_{r=0}^{d-1}p^{r}p^{d}(i, E)$$(b)$$\sum_{r=0}^{d-1}p^{r}(p^{nd}-p^{*})(i, E)=\sum_{r=0}^{d-1}p^{r}(p^{nd}-p^{d*})(\iota’, E)$for an$yi\in S$ an $dE\subset S$ ,Lemma 8. Suppose the am $econ$ ditions with Le $mma7$. Then, we have following$s$equaliti es and in $equ$ alities.

for $n=0,$

.

$(a)$

$|| \sum_{r=0}^{d-1}P^{r}(P^{nd}-P^{*})||\leq d||P^{nd}-P^{d}||$$(b)$$||P^{nd}-P^{d*}||_{a}= \max\{||P^{nd}-P^{d}||_{a,\alpha} ; \alpha=0,1, \ldots, d_{a}-1\}$$l,$ $2,$$\ldots$

and$||P^{nd}-P^{d}||R= \sup\{||P^{nd}-P^{d}||_{a)}a\in A\}$ .
$(c)$

For any $n>m\geq 1_{f}$ it holds that$||P^{nd}-P^{d}|| \leq 4\sup_{\dot{|}\epsilon\tau}p^{md}(i, T)+\{\sup_{i\in T}(1-p^{md}(i, T))\}\Vert P^{(n-m)d}-P^{d*}||_{R}$.

Next Lemma 9 is an application of Case (b) in Doob[4, p197] to the space of MC’s.
It can be proved in the same manner as that of [4]. (See also the proof of Theorem 2.4

in A. Federgruen, A. Hordijk and H. C. Tijms[5] and Lemma 2 of [8].)Lemma 9. Suppose Condition $(A)$ holds. Then there exist a posi tive integer $N$

an$d\epsilon_{0}>0$$||\cdot||$such that $||P(f)^{Nd}-P(f)^{d}||R\leq 1-\epsilon_{0}$ for any $f\in F$ .

$d$enotes the norm

taken on the restricted d-step $MC\{p_{j}^{d_{j}} ; i,j\in R\}$ .

Lemmas 7, 8, 9 are used in the proofs of Theorem 1, 3 and Corollary 2. We omit
the proof of Theorem 3, since it is lengthy. Theorem 5(a) is obtained using the ana-

107
lytic theorem that if a sequence of continuous functions unlformly converges, the limit

function is continuous. To prove the continuity of $H(f)$ in Theorem 6(a), we need next
Lemma 10 which follows from Theorem 5.

Lemma 10. Suppose Condition $(C)$ holds. Then,.

$H(i, E)= \lim_{Narrow\infty}\frac{1}{N+1}\sum_{n=0}^{N}\sum_{k=0}^{n}\{p^{k}(i, E)-p(i\rangle E)\}$

The $con$ vergence is $uni$ form $bo$ th in $i\in S$ an in E C .
$d$

$S$

The statements (b) in Theorem 5, 6 can be proved similar to (a) in them.

4. Examples.
Even when$S$

is finite, there exists a space of MC’s where $P(f)^{*}$ is discontinuous.

Among them the following Example 1, 2 have simple structure. Example 1 is seen in
Schweitzer[ll] and Example 2 is seen$\iota n$

Wijngaard[13] and Zijm[14].

Example 1. Let $S=\{1,2\}$ and $F=[0,1]x[0,1]$ , where $[0,1]$ is the closed intervalof real numbers. For any $f=(p, q)\in F$ , let $p(f)_{11}=1-p,$ $p(f)_{12}=p,$ $p(f)_{21}=q$ ,$p(f)_{22}=1-q$ .

For $0=(0,0)\in F,$ $P(0)=P(0)=I$ and $H(O)=O$ the zero-matrix. If $p+q>0$ , theMC makes one recurrent class for each $j\in F$ . Note that any subset $K(f)\subset S$ does notsatisfy both Condition (A-1-2) and (A-l-l). If $p+q>0$ , we have $p(f)_{\iota 1}^{\tau}=q/(p+q)$

and $p(f)_{\dot{\iota}2}=p/(p+q)$ for $i=1,2$ . It is clear thatany sequence1$\{f_{n}\}$$\lim_{narrow\infty}P(f_{n})$$\neq P(0)$forconverging to $0\in F$ . On the other-hand, it always holds that$\sum_{k=0}^{n}(P(f)^{k}-P(f))||\leqq B(f)$for any $f\in F$ , when$S$

is finite. We have $H(f)_{11}=$$-H(f)_{12}=p/(p+q)^{2}$ and $H(f)_{21}=-H(f)_{22}=-q/(p+q)^{2}$ .

Let $h=(p_{n}, q_{n})$108be a sequence converging$- \lim_{narrow\infty}H(f_{n})_{12}=\infty$$0$

satisfying$p_{n}\geqq q_{n}$.

Then we have $\lim_{narrow\infty}H(f_{n})_{11}=$, which contradicts to Condition (C).

Example 2. Let $S=\{1_{7}2_{t}3\}$ and $F=[0, (-1+\sqrt{5})/2)]$ , the closed interval. For
any $f=p\in F$ . let $p(f)_{11}=1-p-p^{2},$ $p(f)_{12}=p,$ $p(f)_{13}=p^{2},$ $p(f)_{22}=p(f)_{33}=1$

and $p(f)_{21}=p(f)_{23}=p(f)_{31}=p(f)_{32}=0$ .
If $f=p=0,$ $P(0)=P(O)^{*}=I$ and $H(O)=0$ . If $f=p\neq 0$ , the states 2, 3
make recurrent classes respectively and the state 1 is a transient. Note that any subset$K(f)\subset S$

does not satisfies both Condition (A-1-3) and (A-l-l). When $f=p\neq 0$ , wehave $p(f)_{12}^{*}=1/(1+p),$ $p(f)i_{3}=p/(1+p)$ and $p(f)_{22}^{*}=p(f)_{33}=1$ and other $p(f)_{ij}$ are

all zero. We have $H(f)_{11}=1/p(1+p),$ $H(f)_{12}=-1/p(1+p)^{2},$ $H(f)_{13}=-1/(1+p)^{2}$and other components are all zero. Then $\lim_{parrow 0}H(f)_{11}=-\lim_{parrow 0}H(f)_{12}=\infty$ ,which contradicts to Condition (C).
REFERENCES
[1]

[2]

[3]

[4]
[5]

[6]
[7]
[8]

K. A. Chung, Markov Chains with Stationary Transition Probabilities, Springerverlag Berlin, 1960.
R. Dekker and A. Hordijk, Avemge, sensitive and Blackwell optimal policies indenumerable Markov decision chains with unbounded rewards, Mathematics of Operations Reseach 13 (1988), $3?S-\ell 20$ .
H. M. Dietz and V. Nollau, Markov Decision Problems with Countable StateSpaces, Academie-Verlag, Berlin, 1983.
J. Doob, Stochastic Pmcesses, Wiley New York, 1953.
A. Federgruen, A. Hordijk and H. C. Tijms, Recurrence conditions in denumemble
state Markov decision processes (1978), Dynamic Programming and Its Applications,ed. by M. L. Puterman, 3-22.
A. Hordijk, Dynamic Programming and Potential Th eory (1974), Math. Centre
Tracts No. 51, Mathematisch Centrum, Amsterdam.
Y. Kadota, Countable state Markovian decision processes under the DoeblinConditions, Bull. MMath. Statist. 19 (1979), 85-94.
(Kadota), Kasan jyootai marukohu rensa no roomn tenkai ni tuite,
P9 fflSuurI $kaisek_{1}$ kenkyuu syo koukyuu roku 798 (1992), 180-190. (tn).
$\tilde{R}4_{\overline{\tilde{q}}}^{\simeq}$$\mu_{i^{a1u\Lambda Z}}/$

109
J. G. Kemeny, J. L. Snell and A. W. Knapp, Denumemble Markov Chains,
Springer-verlag New York Inc, 1976.
[10] B. L. Miller and A. F. Jr. Veinott, Discrete dynamic pmgmmming with a smallinterest mte, Ann. Math. Statist. 40 (1969), 366-370.
J. Schweitzer, Perturba tion theory and finite Markov chains, J. Appl. Prob. 5P.
[11](1968), 401-413.
[12] A. F. Jr. Veinott, On discrete dynamic progmmming with sensitive discount optimality cr;teria, Ann. Math. Statist. 40 (1969), 1635-1660.
[13] J. Wijngaard, Stationary Markovian decision pmblems and penurbation th eory of
quasi-compact linear opemtors, Math. Oper. Res. 2 (1977), 91-102.
[14] H. Zijm, The optimality equations in multichain denumemble state Markov decision poocesses with the avemge cost criterion: the bounded cost cace, Stat. &Decisions. 3 (1985), 143-165.

[9]