94

数理解析研究所講究録
第2078巻 2018年 94-101

Controlled Markov set‐chains and Bayesian
Decision Analysis
Masayuki HORIGUCHI
Professor of Department of Mathematics,
Faculty of Science,
Kanagawa University
Abstract

In this note, we summarize briefly our series of studies on Markov dccision
processes with unknown transition matrices and give an idea to show the rate ofconvergence of Bayesian updating of posterior distribution in a sampling problem.

1

Interval estimated Markov decision processes

MDPs consist of four tuples \{S, A, Q, r\} as follows. Let S\{1, 2, . . . , n\} be a finite
state space, A= \{a_{1}, a_{2}, . . . , a_{k}\} finite action space. The set of probabilities on S and=P(S) :=\{p= (p_{1}, p2, . . . , p_{n}) \displaystyle \in \mathbb{R}_{+}^{n}|\sum_{i\in S}p_{i}=1\}, P(S|S) :=\displaystyle \{q= (q_{ij} : i,j\in S) \in \mathbb{R}_{+}^{n\times n}|\sum_{j\in S}q_{ij} = 1 (i\in S)\} P(S|S\times A) :=\{Q=(q_{ij}(a) : i, j \in S, a \in A) \in \mathbb{R}_{+}^{kn\times n}|q_{i} (a) \in P(S) (i \in S, a \in A where \mathbb{R}_{+} is the setof nonnegative real numbers and \mathb {R}_{+}^{m} the set of m‐dimensional nonnegative real vectors.
Let B_{+}(D) denote the set of all non‐negative real functions on finite set D . For a finite
D of n elements, B_{+}(D) is identified with \mathb {R}_{+}^{n} . Let Q=(q_{ij}(a)) \in P(S|S\times A) denote aparameter space of k unknown transition matrices and r=(r(i, a)) \in B_{+}(S\times A) reward
transition kernels are defined as follows.

,

function .

For any stationary policy f\in F , discounted total expected reward $\phi$(f|Q) \in \mathbb{R}_{+}^{n} withdiscount factor $\beta$(0< $\beta$<1) is defined as a function of stochastic matrix Q\in P(S|S\times A)
as:$\phi$(f|Q)=\displaystyle \sum_{t=0}^{\infty}( $\beta$ Q(f) ^{t}r(f),(1)

where, r(f)=(r(1, f(1)), r(2, f(2)), \ldots, r(n, f(n)))'\in \mathbb{R}_{+}^{n}, Q(f)=(q_{ij}(f(i)))\in P(S|S) .
Q is estimated by interval matrix \mathcal{Q}=\langle\underline{Q}, \overline{Q} } , where\underline{Q}= (\underline{q}_{ij}(a) :i, j\in S, a\in A) \in \mathbb{R}_{+}^{kn\times n},\overline{Q}= (\overline{q}_{ij}(a) :i, j\in S, a\in A) \in \mathbb{R}_{+}^{kn\times n} ,\mathcal{Q}=\langle\underline{Q}, \overline{Q}\}=\{Q\in P(S|S\times A)|\underline{Q}\preceq Q\preceq\overline{Q}\}.

(2)

It should be noted that the partial orders \preceq, \prec \mathrm{o}\mathrm{n}\mathbb{R}^{m\times n} are defined by the components
orders for each element\underline{q}_{ij}(a) and \overline{q}_{ij}(a) (i, j \in S, a \in A) . We call \{S, A, \mathcal{Q}, r\} theinterval estimated MDPs.

95

For f\in F , we define discounted total expected‐set valued value function $\phi$(f|\mathcal{Q}) as
follows:$\phi$(f|\mathcal{Q})=\{ $\phi$(f|Q)|Q\in \mathcal{Q}\} \subset \mathbb{R}_{+}^{n}(3)where the value $\phi$(f|Q) of standard MDPs is defined in (1).
Let C(\mathbb{R}_{+}) denote the set of all bounded and closed intervals in \mathbb{R}_{+} and C(\mathbb{R}_{+})^{n} theset of all n‐dimensional column vectors whose elements are in C(\mathbb{R}_{+}) , i.e.,C(\mathbb{R}_{+})^{n}=\{D=(D_{1}, D2, . . . , D_{n})'|D_{i}\in C(\mathbb{R}_{+}) (1\leqq i\leqq n)\} .

(4)

We will denote by \mathcal{M}_{n} the set of all interval matrices with n\times n elements.

Lemma 1. (Hartfiel[3], Kurano, Song, Hosaka and Huang[5])(i) Any \mathcal{Q}\in \mathcal{M}_{n} is a convex polytope in \mathbb{R}^{n\times n}.
(ii) For any compact subset G\subset \mathbb{R}_{+}^{1\times n} and D\in C(\mathbb{R}_{+})^{n} , it holds GD\in C(\mathbb{R}_{+}) .
It can be shown that $\phi$(f|\mathcal{Q}) \in C(\mathbb{R}_{+})^{n} in the following. The map \mathcal{L} : C(\mathbb{R}_{+})^{n} \rightarrow

C(\mathbb{R}_{+})^{n} is defined by\mathcal{L}(f)v=r(f)+ $\beta$ \mathcal{Q}(f)v, v\in C(\mathbb{R}_{+})^{n} ,
where

(5)

\mathcal{Q}(f)=\{\underline{Q}(f), \overline{Q}(f)\},\underline{Q}(f)=(\underline{q}_{ij}(f(i) )\in \mathbb{R}_{+}^{n\times n}, \overline{Q}(f)=(\overline{q}_{ij}(f(i))) \in \mathbb{R}_{+}^{n\times n}.

Lemma 1, we have \mathcal{L}(f)v \in C(\mathbb{R}_{+})^{n} (v \in C(\mathbb{R}_{+})^{n}) . Moreover, we define\underline{L}(f) : \mathbb{R}_{+}^{n}\rightar ow \mathbb{R}_{+}^{n}, \overline{L}(f) : \mathbb{R}_{+}^{n}\rightar ow \mathbb{R}_{+}^{n} as follows: For x=(x_{1}, x2, . . . , x_{n})'\in \mathbb{R}_{+}^{n},$\Gamma$ \mathrm{k}\mathrm{o}\mathrm{m}\displaystyle \underline{L}(f)x=r(f)+ $\beta$\min_{Q\in \mathcal{Q}(f)} Qx ,(6)\displaystyle \overline{L}(f)x=r(f)+$\beta$_{Q}\max_{\in \mathcal{Q}(f)} Qx .

(7)

Then, we have the followings:
Lemma 2. For any f\in F,(i) \mathcal{L}(f) is monotone increasing and contractive mapping.

(ii) \underline{L}(f) and \overline{L}(f) are both monotone increasing and contractive mapping with respect
to \displaystyle \sup ‐norm.

Theorem 1. For any f\in F , it holds that(i) $\phi$(f|\mathcal{Q}) \in C(\mathbb{R}_{+})^{n} and $\phi$(f|\mathcal{Q}) is the unique fixed point of \mathcal{L}(f) . Moreover, for any
v\in C(\mathbb{R}_{+})^{n} , we have \mathcal{L}(f)^{\ell}v\rightarrow $\phi$(f|\mathcal{Q}) (\ell\rightarrow\infty) .
(ii) Let $\phi$(f|\mathcal{Q})=[\underline{ $\phi$}(f), \overline{ $\phi$}(f)] . Then, \underline{ $\phi$}(f) and \overline{ $\phi$}(f) are the unique fixed point of \underline{L}(f)and \overline{L}(f) , respectively.

96

Applying the result of De Robertis and Hartigan’s (cf. [9]) on Bayesian inference
method using intervals of prior measures, uncertain MDPs are formulated as interval
estimated MDPs.

Let P_{n}

:=

P(S)

=

\displaystyle \{p = (p_{1}, p2, . . . , p_{n})|p_{i} \geqq 0, \sum_{i=1}^{n}p_{i} = 1\} and \mathcal{B} the set of all

measurable set in \mathbb{R}^{n} , where \mathbb{R}^{n} denotes the set of n‐dimensional real vectors. For given
measures L and U on \mathcal{B} , we denote by L\leq U if L(A) \leq U(A) for all set A\in \mathcal{B} . Let us
denote by [L, U] the convex set of measures Q satisfying L(A) \leq Q(A) \leq U(A) .
For simplicity, let the prior measures [L, kL] (k\geqq 1) , where L
denotes a Lebesgue
mcasure on P_{n} . We denote by $\sigma$= ($\sigma$_{1}, $\sigma$_{2}, \ldots, $\sigma$_{n}) a data of indcpendent experiments,
where the i‐th state is observed with probability p_{i} and $\sigma$_{i} the number of outcomes ofstate i . Then, for a parameter p(p_{1}, p2, . . . , p_{n}) \in P_{n} , a data set $\sigma$ has probability
density function of multinomial distribution as follows:=f($\sigma$_{1},$\sigma$_{2},\displaystyle\ldots,$\sigma$_{n}|p)=\frac{($\sigma$_{1}+\cdot.\cdot.\cdot.+$\sigma$_{n})!{$\sigma$_{1}!$\sigma$_{n}!p_{1}^{$\sigma$_{1}p_{2}^{$\sigma$_{2}\cdotsp_{n}^{$\sigma$_{n} .

(8)

By using Bayesian inference for a prior measures [L, U] , intervals [\underline{$\lambda$}_{i},\overline{$\lambda$}_{i}] (i \in S) of
posterior measures of p_{i} is given by the range of integral ratios:\displaystyle \{\int_{P_{n} p_{i}Q(dp)/\int_{P_{n} Q(dp)|L_{ $\sigma$}\leq Q\leq U_{ $\sigma$}\} ,(9)where L_{ $\sigma$} and U_{ $\sigma$} are respectively lower and upper bounds of posterior measure for $\sigma$,and characterized as the unique solutions of (10) and (11):Theorem 2. Lower bound \underline{$\lambda$}_{i and upper bound \overline{$\lambda$}_{i of posterior intervals [\underline{$\lambda$}_{i},\overline{$\lambda$}_{i}] (i \in S)
are unique solutions as following equations:

U_{ $\sigma$}(p_{i}-\underline{ $\lambda$}_{i})^{-}+L_{ $\sigma$}(p_{i}-\underline{ $\lambda$}_{i})^{+}=0 ,U_{ $\sigma$}(p_{i}-\overline{ $\lambda$}_{i})^{+}+L_{ $\sigma$}(p_{i}-\overline{ $\lambda$}_{i})^{-}=0 ,

(10)
(11)

where Q(f) denotes the integral of function f w.r.t . measure Q, x^{+}=\displaystyle \max\{0, x\}, x^{-}=x-x^{+}=\displaystyle \min\{0_{J}x\}.

2

Bayesian control chart in uncertain MDPs

In this section, we describe a quality control model and formulate equivalent Bayesianmodel to the original problem.
Let X(t) denote states of system at time t(t\geqq 0) where X(t)=\left{bginary}{l0(\mathr{i}\mathr{n}-\mathr{c}\mathr{o}\mathr{n}\mathr{}\mathr{}\mathr{o}\mathr{l}),\ 1(mathr{o}\mathr{u}\mathr{}-\mathr{o}\mathr{f}-\mathr{c}\mathr{o}\mathr{n}\mathr{}\mathr{}\mathr{o}\mathr{l}).\end{ary}\ight.

The transition of state is occurred ex‐ponentially with distribution function Exp( $\theta$) , where$\Theta$ \ni $\theta$ is unknown.

We denote by $\theta$ a random variable of $\theta$ . The partially observationsof state X(t) of system are made by sampling Y_{i} (i=1,2, \ldots) of q‐dimensional data ofsample size n . The alternatives \triangle are selected at each inspcction step ih (i=1,2, . .
where\triangle=\left{bginary}{l0\mathr{c}\mathr{o}\mathr{n}\mathr{}\mathr{i}\mathr{n}\mathr{u}\mathr{e},\ 1mathr{s}\mathr{}\mathr{o}\mathr{p}\mathr{}\mathr{n}\mathr{d}\mathr{s}\mathr{e}\mathr{}\mathr{}\mathr{c}\mathr{}.\end{ary}\ight.

97

It is noted that by the information of sampling date the decision maker choices between
action 0 ” or 1” at each decision epoch ih(i=1,2, . .
We assume the following.

Assumption 1. If X_{ih}=0 (or 1), y_{1}^{i}, y_{2}^{i} , . . . , y_{n}^{i} are i.i. \mathrm{d} . random variables from N_{q}($\mu$_{0}, $\Sigma$)(or N_{q}($\mu$_{1}, $\Sigma$ where N_{q}($\mu$_{0}, $\Sigma$), N_{q}($\mu$_{1}, $\Sigma$) are \mathrm{q}‐dimensional normal distribution and $\Sigma$
is variance covariance matrix(positive definite) and $\mu$_{0}, $\mu$_{1} are mean vector, respectively.
For \mathrm{M} ‐distance d_{1} between $\mu$_{0} and $\mu$_{1} we assume the following:d_{1} := [($\mu$_{1}-$\mu$_{0})$\Sigma$^{-1}($\mu$_{1}-$\mu$_{0})^{t}]^{\frac{1}{2} >0 .

(12)

The cost structures of this inspection model are given in the followings: Investigation
cost A>0 will be occurred to stop and research the system whether there is failure ofsystem or not. When the state of system is failure, renewal cost R\geqq 0 will be charged tochange the state from 1” (out‐of‐control) to0 ” (in‐control).

If the system is operating

without knowing the state of system being failure, operating cost M>0 per unit timeis incurred while remaining the system in out‐of‐control. If the decision maker selects

an action “1” (stop and search), an observation cost b+nc(b, c\geqq 0) of sample size n isincurred.

$\Theta$ \times $\Omega$, $\Omega$
S \times (A \times S)^{\infty}\mathrm{a}\mathrm{n}\mathrm{d} random variables of
Let sample space be \overline{$\Omega$}
process \overline{ $\theta$},\tilde{p}_{0} , ã0, \tilde{p}_{1} , ãl, . . .. That is, for $\omega$ \in \overline{$\Omega$}
we have( $\theta$,p_{0}, a_{0},p_{1}, a_{1} , p2, . .
\tilde{ $\theta$}( $\omega$) $\theta$, \tilde{p}_{0}( $\omega$) =p_{0} , ã0 ( $\omega$) a_{0}, \tilde{p}_{1}( $\omega$) =p_{1} , . . ., where we set p_{0} 0 without loss of
=

=

=

=

=

=

generality. When the state is \tilde{p}_{m}=p at epoch mh, an action \~{a}_{m}=0 ( \mathrm{o}\mathrm{r}1 , respectively)
is selected and at the next epoch (m+1)h the information Y_{m+1}=y^{m+1} is obtained andat epoch (m+1)h the state moves to\tilde{p}_{m+1}=T (p, y^{m+1}, 0) (or T ( p, y^{m+1} , 1), respectively) ,(13)

where, by Bayes’ theorem, priori‐posteriori Bayesian operator T is defined as in (Lemma
1 in [8]):When $\theta$ is true parameter the costs c ) are given below:\left\{\begin{ar ay}{l} c(p, 0)=ME(\int_{0}^{h}I_{\{X_{s}=1\} d\mathrm{s})+b+nc=M[h-\frac{1-p}{ $\theta$}(1-e^{- $\theta$ h})] +b+nc,\ c(p, 1)=c_{1}(p)+c(0,0) , \end{ar ay}\right.

(14)

where c_{1}(p)=A+Rp.

Average expected cost $\varphi$( $\pi$| $\theta$,p_{0}) given by \tilde{ $\theta$}= $\theta$\in \mathcal{P}( $\Theta$) and initial state distributionp_{0}=p\in S is defined as follows:$\varphi$($\pi$| \theta$,p_{0})=\displaystyle\lim_{k\rightar ow}\sup_{\infty}\frac{1}{E($\tau$_{k})E[\sum_{m=0}^{$\tau$_{k}c(\tilde{p}_{m},\~{a}_{m})|$\theta$,p_{0}],where, $\pi$=($\tau$_{0}, $\tau$_{1} , $\tau$_{2}, . .

In addition, define discounted total expected cost v( $\pi$| $\theta$, p_{0}) as follows:

(15)

98

v($\pi$|$\theta$,p_{0})=\displaystyle\sum_{m=0}^{\infty}$\beta$^{m}E_{$\pi$}[c_{$\alpha$}(\tilde{p}_{m},\~{a}_{m})|$\theta$,p_{0}],(16)e^{- $\alpha$ h} denotes discount rate and E_{ $\pi$}[\cdot| $\theta$,p] is expectation with probability
where, $\beta$
measure P_{ $\pi$}(\cdot| $\theta$,p) on St given by parameters $\theta$,p and policy $\pi$.
Each policy $\pi$ \in $\Pi$ which minimize $\varphi$( $\pi$| $\theta$,p) , v( $\pi$| $\theta$,p) respectively call $\theta$‐average
optimal and $\theta$ ‐discounted optimal respectively. We have the following theorems.
=

optimal policy *of\displayte\frac{M}$\thea$} , there existssuch‐average
the control‐limit type. That is, there existsthat control policy following

Theorem 3 (V. Makis[8]). If A+R<$\theta$$\pi$

p_{ $\theta$}^{*} \in (0,1)
the decision function f_{ $\theta$} : S\rightarrow A as below is $\theta$ ‐average optimal.

\left{\begin{ar y}{l 0ifp<_{$\thea$}^{*,\ 1ifp\geq p_{$\thea$}^{*. \end{ar y}\right.

f_{ $\theta$}(p)=(17)

Theorem 4. (Sasaki, Horiguchi and Kurano ([10]))There exists $\theta$ ‐discounted optimal
policy of control‐limit type, that is, there exists \overline{p}_{$\theta$} \in (0,1) such that optimal decisionfunction g_{ $\theta$} : S\rightarrow A is given as below:

g_{ $\theta$}(p)=3\left{begin{ar y}l 0ifp<\overlin{p}_$\thea$},\ 1ifp\geq overlin{p}_$\thea$}. \end{ar y}\ight.

(18)

Repair problem with exponentially deteriorating
systemWe consider the following repair problem. Let $\theta$ \in $\Theta$\{$\theta$_{1}, . . . , $\theta$_{k}\} denote a finite
parameter space. For each $\theta$_{i}\in $\Theta$ , we denote by g(t| $\theta$) a pdf of dcteriorating timc \mathrm{t} with=exponential distribution asg(t| $\theta$)=\left{\begin{ar y}{l $\thea$e^{-$\thea$t},&$\iota$\geq 0,\ 0,&t<0. \end{ar y}\right.

Let constant T > 0 be a time of inspection intervals. Let X_{i} denote the state of
1 it means
0 it means the system is under control and if X_{i}system and if X_{i}the system is out of control. If the state X_{i} is 0 at the inspection epoch iT , by thememorylessness property of exponential distribution, it docs not affect the probabilityof the system being out of control from then on. On the other hand, if thc state X_{i} is 1
at epoch iT , the system is repaired immediately and starts as new one after that.
e^{- $\theta$ T} for each
It is easily seen that P (X_{i} = 0)
P(X_{i} > T)\displaystyle \int_{T}^{\infty} $\theta$ e^{- $\theta$ x}dx1P1-e^{$\theta$T}iinspection epoch iT,, 2, . . . , and (X_{i} = 1)
. Then we have a pdf ofstate of system given the parameter $\theta$_{i} by f(0|$\theta$_{i})=e^{-$\theta$_{i}T} , and f(1|$\theta$_{i})=1-e^{-$\theta$_{i}T}.
In this inspection and repair problem, we consider the rate of convergence aboutposterior probabilities by the sequences of states of system inspections.
=

=

=

=

=

=

=

99

inspection

Figure 1: The states of system are inspected at each epoch.
Let X_{1}, X_{2} , . . . are \mathrm{i}.\mathrm{i}.\mathrm{d} . random variables with X_{i}\sim f(x| $\theta$) where $\theta$\in $\Theta$=\{$\theta$_{1}, $\theta$_{2}, . . . , $\theta$_{k}\}.
We write q(q($\theta$_{\mathrm{I} ), q($\theta$_{2}), \ldots, q($\theta$_{k})) for apriori distribution of parameter $\theta$ \in $\Theta$ . Let
h_{n}= (x_{1}, x2, . . . , x_{n}) denote the sample of size n where X_{1} =x_{1}, X_{2}=x_{2} , . . . , X_{n}=x_{n},x_{i}=0 or 1, i=1 , 2,
, n . We will denote by q_{n}(q, h_{n})($\theta$_{i}) the posteriori distribution ofparameter $\theta$\in $\Theta$ as follows:=\cdotsq_{n}(q,h_{n})($\theta$_{i})=\displaystyle\frac{q($\theta$_{i})\prod_{l-1}^{n}f(x_{l}|$\theta$_{i}){\sum_{j=1}^{k}q($\theta$_{j})\prod_{l=1}^{n}f(x_{l}|$\theta$_{j}).
In order to show the rate of convergence of posteriori distribution, we give somewell‐known results.

Theorem 5 (Hölder’s Inequality). For p> 1, \displaystyle \frac{1}{p}+\frac{1}{q}=1, f\in\ovalbox{\t \small REJECT}, g\in L^{q} and $\mu$

Lebesguemeasure, it holds that|\displaystyle \int f(x)g(x)d $\mu$| (\displaystyle \int|f(x)|^{p}d $\mu$)^{p}1 (\displaystyle \int|g(x)|^{q}d $\mu$) ①\leqq

Moreover, for f'(x)(19):=\displaystyle \frac{|f(x)|^{p} {\int|\int(x)|^{p}d $\mu$}, g'(x) :=\displaystyle \frac{|g(x)|^{q} {\int|g(x)|^{q}d $\mu$} and D :=\{x|f(x)\neq g(x)\} , if $\mu$(D)>it holds that |\displaystyle \int f(x)g(x)d $\mu$| < ||f||_{p}. ||g||_{q} , i. e., the inequality (19) holds for strict
inequality ( < ) .

0,

If p=q=2 it is known as Schwartz’s inequality and for probability density functions
f and g for some distributions on probability space (X, \mathcal{B}, P) , we have the following.
Lemma 3. Let D := \{x|f(x) \neq g(x)\} and D' := \{x|f'(x) \neq g'(x)\} . For pdfs f(x) andg(x) , if $\mu$(D) >0 implies $\mu$(D')>0.
Proof.

\underline{f(x)^{2}

If $\mu$(D')
=

=

0,

i.e., f'(x)=g'(x) a.s., then, there exist c_{1}, c_{2} > 0 such that

\underline{g(x)^{2} . Hence there exists c>0 such that f(x) =cg(x) . Thc integrals overx\mathrm{f}\mathrm{o}\mathrm{}\mathrm{e}\mathrm{a}\mathrm{c}\mathrm{h}c_{1} sidcc2 function of equation, we have 1=c\times 1 . Thercforc we get c=1 . It means
f(x)=g(x) a.s, i.e., $\mu$(D)=0 , which proves the lemma by showing this contrapositivc.

1

Corollary 1. (Schwartz’s inequality for $\mu$(D) >0 ) For pdfs f and g , if $\mu$(D) >0 implies|\displaystyle \int(f x) ^{\frac{1}{2} (g(x) ^{\frac{1}{2} d $\mu$| <1.

100

We assume the following.
Assumption 2. $\mu$(D_{ij}) >0 for 1\leqq i<j\leqq k , where D_{l} =\{x|f(x|$\theta$_{i})\neq f(x|$\theta$_{j})\}.
Thcn, wc have the following.

Theorem 6. Under Assumption 2 and for a prior q(q($\theta$_{1}), q($\theta$_{2}), \ldots, q($\theta$_{k})) with
q($\theta$_{i}) >0 for all i(1\leqq i\leqq k) , there exists $\lambda$(0< $\lambda$< 1) such that=

P (q_{n}(q, h_{n})($\theta$_{i}) > $\epsilon$|\tilde{ $\theta$}=$\theta$_{i_{0} )\leqq K(q, n)$\lambda$^{n}for any $\varepsilon$>0 and i\neq i_{0} , whereK(q,n)=\displaystyle\max_{i\neqi0_{\tilde{$\varepsilon$2} ^{1}(\frac{q($\theta$_{l}) {q($\theta$_{i_{0} )} ^{\frac{1}{2}Proof. Since X_{1}, X_{2} , . . . are i.i. \mathrm{d}. , we haveHence,

q_{n}(q,h_{n})($\theta$_{i})=\displayst le\frac{q($\theta$_{i})\prod_{$\iota$_{-1}^{n}f(x_{l}|$\theta$_{i}){\sum_{j=1}^{k}q($\theta$_{j})\prod_{l=1}^{n}f(x_{l}|$\theta$_{j})=\frac{q($\theta$_{i})\prod_{l=1}^{n}f(x_{l}|$\theta$_{i}){q($\theta$_{i 0})\prod_{l=1}^{n}f(x_{l}|$\theta$_{i 0}).
P(q_{n}(q, h_{n})($\theta$_{i})> $\varepsilon$|\tilde{ $\theta$}=$\theta$_{i_{0} )=P(\displaystyle \frac{1}{ $\varepsilon$}q_{n}(q, h_{n})($\theta$_{i})>1|\tilde{ $\theta$}=$\theta$_{i_{0} )=P(\sqrt{\frac{1}{ $\varepsilon$} \sqrt{q_{n}(q,h_{n})($\theta$_{i}) >1|\tilde{ $\theta$}=$\theta$_{i_{0} )\leq E(\sqrt{\frac{1}{$\epsilon$} \sqrt{q_{n}(q,h_{n})($\theta$_{i}) >1|\tilde{$\theta$}=$\theta$_{i_{0} )=\displayst le\frac{1}$\epsilon$^{\frac{1}2}\int\sqrt{\frac{q($\theta$_{i})\prod_{l-1}^{n}f(x_{l}|$\theta$_{i}){q($\theta$_{i 0})\prod_{l-1}^{n}f(x_{l}|$\theta$_{i 0}) \prod_{l=1}^{n}f(x_{l}|$\theta$_{i 0})dx_{1}\cdotsdx_{n}=\displayst le\frac{q($\theta$_{i})^{\frac{1}2}{$\epsilon$^{\frac{1}2}\cdotq($\theta$_{i 0})^{\frac{1}2}\int(\prod_{l=1}^{n}f(x_{l}|$\theta$_{i})^{\frac{1}2}(\prod_{l=1}^{n}f(x_{l}|$\theta$_{i 0})^{\frac{1}2}dx_{1}\cdotsdx_{n}=\displayst le\frac{q($\thea$_{i})^{\frac{1}2}{$\varepsilon$^{\frac{1}2}\cdotq($\thea$_{i 0})^{\frac{1}2}\prod_{l=1}^{n}\intf(x_{l}|$\thea$_{i})^{\frac{1}2}f(x_{l}|$\thea$_{i 0})^{\frac{1}2}dx_{l}\displaystle\ q \frac{1}$\epsilon$^{\frac{1}2 }\frac{q($\thea$_{i})^{\frac{1}2 }{q($\thea$_{i 0})^{\frac{1}2 }$\lambda$^{n}=K(q,n)$\lambda$^{n}by Fubini’s Theorem

for \exists $\lambda$(0< $\lambda$<1) ,which completes the proof. 1
Finally, we apply this theorem to our inspection and repair problem.
Let $\lambda$_{i, _{0}\displaystyle \int f(x| $\theta$)^{\frac{1}{2} f(x|$\theta$_{i_{0} )^{\frac{1}{2} d $\mu$ for i \neq i_{0} . By Corollary 1, $\lambda$_{i, _{0} <.
Thenfrom Theorem 1, for \tilde{ $\theta$}=$\theta$_{i_{0} we have\displaystyle \max_{i\neq i_{0} $\lambda$_{i,i_{0}=q_{n}(q, h_{0})($\theta$_{k})\rightarrow
with exponentially fast asn\rightarrow\infty.

\left\{ begin{ar y}{l 1(k=i),\ 0(k\neqi_{0}) \end{ar y}\right.

1.

Let $\lambda$

=

101

References[1] M. H. DeGroot. Optimal statistical decisions. McGraw‐Hill Book Co., New York,1970.

[2] T. S. Ferguson. Mathematical Statistics. Academic Press, New York‐ London. 1967.
[3] D. J. Hartfiel Markov set‐chains, volume 1695 of Lecture Notes in Mathematics.
Springer‐Verlag, Berlin, 1998.

[4] O. Hernández‐Lerma. Adaptive Markov control processes, volume 79 of Applied
Mathematical Sciences. Springer‐Verlag, New York, 1989.

[5] M. Kurano, J. Song, M. Hosaka and Y. Huang. Controlled Markov set‐chains withdiscounting. J. Appl. Probab., pages 293‐302, 1998.

[6] M. Hosaka, M. Horiguchi and M. Kurano. Controlled Markov set‐chains under
average criteria. Appl. Math. Comput., pages 195‐209, 2001.

[7] M. Horiguchi. Bayesian Inference in Markov Decision Processes. In, Modern Trends
in Controlled Stochastic Processes: Theory and Applications, Vol. 2 (A.B. Pi‐
unovskiy ed Luniver Press (2015). (In press).

[8] Viliam Makis. Multivariate Bayesian control chart. Oper. Res., 56(2):487-496 , 2008.
[9] L. De Robertisand J. A. Hartigan, Bayesian inference using intervals of measures.
Ann. Statist., 9:235−244, 1981.

[10] M. Sasaki, M. Horiguchi and M. Kurano. Adaptive methods for multivariate
bayesian control chart. RIMS kokyuroku No. 1912 (In Japanese), pages 181‐192,
2014.

[11] K. M. van Hee. Bayesian control of Markov chains, volume 95 of MathematicalCentre Tracts. Mathematisch Centrum, Amsterdam, 1978.

Masayuki Horiguchi
Department of Mathematics,
Faculty of Science, Kanagawa University
Address: Tsuchiya 2946, Hiratsuka City,
Kanagawa Prefecture, 259‐1293, Japan\mathrm{E} ‐mail address: horiguchi@kanagawa‐u.ac.jp

神奈川大学

理学部

堀口正之