数理解析研究所講究録 1457 巻 2005 年 139-146

139
Fuzzy Perceptive Values for MDPs with Discounting
千葉大学教育学部 蔵野正美 (Masami Kurano)

Faculty of Education, Chiba University
千葉大学理学部 安田 正實 (Masami Yasuda)
千葉大学理学部 中神潤一 (Jun-ichi Nakagami)

Faculty of Science, Chiba University
北九州大学経済学部 吉田 祐治 (Yuji Yoshida)

Faculty of Economics and Business Administration, Kitakyushu University

Abstract

In this paper, we formulate the fuzzy perceptive model for discounted Markov decision
processes in which the perception for transition probabilities is described by fuzzy sets. The
optimal expected reward, called a fuzzy perceptive value, is characterized and calculated bya new fuzzy relation. As a numerical example, a machine maintenance problem is considered.
Keywords : Fuzzy perceptive model, Markov decision process,fuzzy perceptive reward, optimal policy function.

1. Introduction and notation
Many contributions to Markov decision processes(MDPs) have been made (cf. [1], [2], [4], [9],[10] , in which the transition probability of the state at each time is assumed to be uniquely
given. In a real application of MDPs, the transition probability will be estimated throughthe measurement of various phenomena. In such a case, the real value of the state transition
probability may be partially observed by dimness of perception or measurement imprecision.
For example, in a famous automobile replacement problem [4], the true value of the probability
that the car in within age after six months, given that the car is within age at that time,
may not be observed exactly. Usually, it is linguistically or roughly perceived, e.g., about 0.3,
the probability considerably larger than 0.3 etc. A possible approach to handle such a case is
to use the fuzzy set ([3], [12]), whose membership function can describe the perception value ofthe true probability. If the fuzzy perception of the transition probabilities for MDPs is given,
how can we estimate in advance the future expected reward, called a fuzzy perceptive value,
under the condition that we can know the true value of the transition probability immediately
before our decision making.
In our previous work [8], we have tried the perceptive analysis for an optimal stopping
problem. In this paper, we formulate the fuzzy perceptive model for MDPs and develop theperceptive analysis in which the fuzzy perceptive value for MDPs is characterized and calculated
by a new fuzzy relation.
In remainder of this section, we will give some notation and fundamental results on MDPs,
by which the fuzzy perceptive model is formulated in the sequel. For non-perception approaches
to MDPs with fuzzy imprecision refer to [7] . Recently Zadeh [13] wrote a summary paper ofperception based probability theory.
$)$

$j$

$q_{ij}$$\mathrm{i}$andbe the sets of real numbers, real -dimensional column vectors and,are endowed with the norm
andmatrices, respectively. The setsreal
, $x(n))’\in \mathbb{R}^{n}$ , $||x||= \sum_{j=1}^{n}|x(j)|$ and for $y=(y_{ij})\in \mathbb{R}^{m\cross n}$ ,where for $x=$ $(x(1), x(2)$ ,Let$\mathbb{R}$$m\mathrm{x}$,$\mathbb{R}^{n}$$\mathbb{R}^{m\}\langle n}$$n$$\mathbb{R}^{n}$$n$$\ldots$$||y||= \max_{1\leq i\leq m}\sum_{j=1}^{n}|y_{ij}|$.

$\mathbb{R}^{m\cross n}$$||\cdot$

$||$

140
is
. The a-cut of
For any set $X$ , let $F(X)$ be the set of all fuzzy sets,
where cl is
andgiven byismeans thata closure of a set. Let be the set of all fuzzy numbers, i.e.,be the set
normal, upper semicontinuous and fuzzy convex and has a compact support. Let
if and only if, it holds thatof all bounded and closed intervals of R. Then,).
(a, we write
for a $\in[0,1])$ . So, fornormal andif and only if
is defined as follows: For ,The binary relation on, there;(ii) for any, there exists
such that $x\leq y$ and(i) for any
satisfies the: Obviously, the binary relationsuch that $x\leq y$ andexists(cf. [6], [11]).
order relation on
axioms of aandare defined byFor$\tilde{x}\in \mathcal{F}(X)$$\tilde{x}:arrow[0, 1]$$\tilde{x}_{0}:=\mathrm{c}1\{x\in X|\overline{x}(x)>0\}$$\overline{x}_{\alpha}.--\{x\in X\lrcorner\tilde{x}(x)\geq\alpha\}(\alpha\in(0,1])$$\overline{r}\in\overline{\mathbb{R}}$$\overline{r}\in F(\mathbb{R})$$\mathbb{R}$$\mathbb{C}$$\tilde{r}\in\tilde{\mathbb{R}}$$\underline{\mathrm{f}}\mathrm{o}\mathrm{r}$$\overline{r}_{\alpha}=[\overline{r}_{\alpha}^{-},\tilde{r}_{\alpha}^{+}]$$\tilde{r}\in \mathbb{R}$$\tilde{r}_{\alpha}\in \mathbb{C}$$\overline{r}$$\tilde{r}\in \mathcal{F}(\mathbb{R})$$\tilde{r}$$\mathcal{F}(\mathbb{R})$$\neg\prec$$x\in \mathbb{R}$$1]$$y\in \mathbb{R}$$\overline{r}(x)\leq\overline{s}(y)$$y\in \mathbb{R}$$x\in \mathbb{R}$$\in[\mathrm{O},$$\overline{s}\in \mathcal{F}(\mathbb{R}),\overline{r}\prec_{\neg}\overline{s}$$\overline{s}(y)\leq\tilde{r}(x)$$\neg\prec$$F(\mathbb{R})$$\mathrm{p}\underline{\mathrm{a}}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}$$\overline{\min}\{\overline{r},\mathrm{s}\}$$\overline{r},\tilde{s}\in \mathbb{R},-\max\{\tilde{r}_{2}\urcorner s$$\max-\{\overline{r},\urcorner s$$(y):=$$\overline{r}(x_{1})$A$\overline{s}(x_{2})$}$(y\in \mathbb{R})$,$\overline{r}(x_{1})$A$\tilde{s}(x_{2})$}$(y\in \mathbb{R})$,{$\sup$$y=x_{1}\vee x_{2}x_{1},x_{2}\in \mathrm{R}$and$\overline{\min}\{\overline{r},\gamma s$$(y):=y=x_{1} \Lambda x_{2}\sup_{x_{1},x_{2}\in \mathbb{R}}${R. It is easy proved that forandR.
, (ii)
andare equivalent (cf. [6]): (i)Also, for, the followingF.
, (iv)(a); (iii)andandthe subsets of entrywise non-negative elements in
We denote by
the set of allandrespectively. Let
be the set of all bounded and closed intervals of.
-dimensional column vectors whose elem ents are in
We have the following.

where

$a$

A $b= \min\{a, b\}$ and

$a$

$a$

$b$

$\in$

$\tilde{r}\prec_{\neg}\tilde{s}\cdot$$(\mathrm{i})-(\mathrm{i}\mathrm{v})$$\overline{r},\overline{s}\in \mathbb{R}$$\in[\mathrm{O},$for any ,$\overline{\min}\{\overline{r},\hat{s}\}\in$$\overline{r},\overline{s}\in\overline{\mathbb{R}},-\max\{\tilde{r},s\gamma_{-}\in\tilde{\mathbb{R}}$$\overline{r}_{\alpha}^{+}\leq\overline{s}_{\alpha}^{+}$$\vee b=\max\{a, b\}$$1]$$\min\{\overline{r},\tilde{s}\}=$$\max-\{\tilde{r},\tilde{s}\}=\tilde{s}\cdot$$\mathbb{R}_{+}$$r-\alpha-\leq\sim.\mathrm{Q}\alpha-$$\mathbb{R}_{+}^{n}$$\mathbb{R}_{+}$$\mathbb{C}_{+}$$n$$\mathbb{C}$:$\mathbb{R}$$\mathbb{R}^{n}$$\mathbb{C}_{+}$

Lemma 1.1 ([5]) For any non-empty convex and compact set, it holds that$G\subset \mathbb{R}_{+}^{n}and$$D=(D_{1}, D_{2}, \ldots, D_{n})’$$\in \mathbb{C}_{+}^{n}$$GD=\{g’\cdot d|g\in G, d\in D\}\in \mathbb{C}_{+}$

where

$x’$

denotes the transpose of a vector$d=$ $(d_{1},d_{2}, \ldots ,d_{n})’\in D$,$g’\cdot$$d= \sum_{j=1}^{n}$$x\in \mathbb{R}^{n}$and for $g=$$(g_{1},g_{2}, \ldots,g_{n})’\in \mathbb{R}_{+}^{n}$

and

gjdj.

Here, we define MDPs whose extension to the fuzzy perceptive model will be done in Section
2. Consider finite state and action spaces, and , containing $n<\infty$ and $k<\infty$ elementswith $S=\{1,2, \ldots, n\}$ and $A=\{1,2, \ldots, k\}$ .
Let $P(S)$and $P(S|SA)$be the sets of all probabilities on and conditional
probabilities on given, that is,
$A$

$S$

$S$

$\subseteq \mathbb{R}^{n\mathrm{x}nk}$$\subset \mathbb{R}^{n}$$S\mathrm{x}A$$S$$P(S)$$:=$$\{q=(q(1), q(2), \ldots , q(n))’|q(\mathrm{i})\geq 0, \sum_{i=1}^{n}q(\mathrm{i})=1, \mathrm{i}\in S\}$

$P(S|SA)$$:=$$\{Q=(q_{ia}(\cdot) :,\mathrm{i}\in S, a\in A)|$$q_{ia}(\cdot)=(q_{\iota a}(1),q_{ia}(2)$,$\ldots$, $q_{ia}(n))’\in P(S)$ , $i\in S$ , $a\in A\}$ .

For any $Q=(q_{ia}(\cdot))\in P(S|SA)$ , we define a controlled dynamic system, called aMarkov decision process(MDP) , specified by $\{5, A, Q,r\}$ , where $r:S$ )is an immediate
reward function$\mathcal{M}(Q)$$<Aarrow \mathbb{R}_{+}$

141
When the system is in state
and action $a\in A$ is taken, then the system moves to a
. and the reward
new state $j\in S$ selected according to
is obtained. The process is$j\inS$
repeated from the new state.
We wish to maximize the expected total discounted reward over the infinite horizon.
Denote by $F$ the set of functions fromto . A policy is a sequenceof$f_{t}\in
F$
$(t
\geq
1)$
functions with
. Let II denote the class of policies. We denote by
the policy, , . . . with $f_{t}=f$ for all $t\geq 1$ and some $f\in F$ . Such a policy is called stationary.
We associate with each $f\in F$ and $Q\in P(S|SA)$ the column vector $r(f)$$(r(1, f(1))$ ,, $r(n, f(n)))’$ and thetransition matrix $Q(f)$ , whose
element is $q_{i,f(i)}(j)$,. Then, the expected total discounted reward from $\pi=$ , , . . . is the column
vector $\psi(\pi|Q)=(\psi(1, \pi|Q)$ ,, $\psi(n, \pi|Q))’$ , which is defined, as a function of $Q\in P(S|SA)$ ,by$\mathrm{i}\in S$$q_{ia}$$($$r(\mathrm{i}, a)$

$)$

$A$

$S$

$(f_{1}, f_{2}, \ldots)$$\pi$$f^{\infty}$$(f_{1}$$f_{2}$$)$$=$$n\mathrm{x}$$\ldots$$1\leq \mathrm{i}$$n$$(\mathrm{i},j)$$j\leq \mathrm{w}$$(f_{1}$$f_{2}$$)$$\ldots$(1.1)$\psi(\pi|Q)=\sum_{t=0}^{\infty}\beta^{t}Q(f_{1})Q(f_{2})\cdots$$Q(f_{t})r(f_{t+1})$,

where $0<\beta<1$ is a discount factor.
For any $Q\in P(S|SA)$ , a policy ’ satisfying that$\pi$

for all$\psi(\mathrm{i}, \pi^{*}|Q)=\sup_{\pi\in\Pi}\psi(\mathrm{i}, \pi|Q):=\psi(i|Q)$

is said to be -optimal and $\psi(Q):=(\psi(1|Q), \psi(2|Q)$ ,value vector.
We can state the well-known results.
$Q$

Theorem 1.1 (cf.
holds:$\cdots$, $\psi(n|Q))’$ is called the Q-optimal$[2],[9],[10]$ ) For any Q $=(q_{\iota a}(\cdot)$ : i $\in S,$a $\in A)$ $\in P(S|SA)$ ,(i) The -optimal value vector $\psi(Q):=(\psi(1|Q), \psi(2|Q)$ ,the optimality equations
$Q$

(1.2)

$\mathrm{i}\in S$$\cdots$, $\psi(n|Q))’$ is a unique solution to$(\mathrm{i}\in S)$$\psi(\mathrm{i}|Q)=\max_{a\in A}\{r(\mathrm{i}, a)+\beta\sum_{j\in S}q_{ia}(j)\psi(j|Q)\}$(ii) There exits an optimal stationary policy(1.2), i.e.,(1.3)$f_{*}^{\infty}$

such thatthe following$f_{*}(\mathrm{i})\in A$;

attains the minimum in$(\mathrm{i}\in S)$$\psi(\mathrm{i}|Q)=r(\mathrm{i}, \mathrm{f}\#(\mathrm{i})+\beta\sum_{j\in S}q_{if*(i)}(j)\psi(j|Q)$.

In Section 2, we define a fuzzy-perceptive model for MDPs, which is analyzed in Section 3
with a numerical example. The proof of the theorem is given in Section 4.

2. fuzzy-perceptive model
We define a fuzzy-perceptive model, in which fuzzy perception of the transition probabilities in
MDPs is accommodated. In a concrete form, we use the fuzzy set on $P(S|SA)$ whose membershipfunction describes the perception value of the transition probability.
$\overline{Q}$Firstly, for each
and $a\in A$ , we give a fuzzy perception of$(q_{ia}(1), q_{ia}(2)$ ,,, which is a fuzzy set on $P(S)$ and will be assumed to satisfy:
the following conditions$\mathrm{i}\in S$$q_{ia}(n))’,\overline{Q}_{ia}(\cdot)$$\ldots$$(\mathrm{i})-(\mathrm{i}\mathrm{i})$$q_{ia}(\cdot)=$142(i) (Normality) There exists a $q=q_{ia}(\cdot)\in P(S)$ with$\overline{Q}_{ia}(q)=1$;(ii) (Convexity and compactness) The
convex and compact subset in $P(S)$ (a $\in[0,1]$ ).

$\alpha- \mathrm{c}\mathrm{u}\mathrm{t}\overline{Q}_{\iota a,\alpha}(\cdot)=\{q=q_{ia}(\cdot)\in P(S)|\tilde{Q}_{ia}(q)\geq\alpha\}$is a, $a\in A$ }, we define the fuzzy set:
Secondly, from a family of fuzzy-perception
on $P(S|SA)$ , called fuzzy perception of the transition probability in MDPs, as follows:$\mathrm{i}\in S$$\mathrm{f}_{\backslash }\tilde{Q}_{ia}(\cdot)$$\mathrm{s}$$\overline{Q}$$(2,1)$$Q-(Q)= \min_{i\in S_{\}}a\in A}\overline{Q}_{ia}(q_{ia}$() ,$\cdot$

The a-cut of the fuzzy perception
(2.2)

where

$)$

$\tilde{Q}$$\in P(S|SA)$ .

$Q=(q_{\mathrm{i}a}(\cdot) :\mathrm{i}\in S, a\in A)$

is described explicitly in the following:$\tilde{Q}_{\alpha}=\{Q=(q_{ia}(\cdot) :$\in P(S|SA)|$\mathrm{i}\in S, a\in A)$$q_{\iota a}(\cdot)\in\tilde{Q}_{ia,\alpha}$$(\alpha\in[0, 1])$$= \prod_{i\in S,a\in A}\overline{Q}_{ia,\alpha}$for$\mathrm{i}\in S,a\in A$}.

on $P(S)$ , it
and $a\in A$ , in place of giving the fuzzy perception
Remark For each
on $[0, 1]$ , which represents the fuzzy
may be convenient to give the fuzzy set(the probability that the state moves to $j\in S$ when the action $a\in A$ isperception of).
taken in stateThen,is defined by$\overline{Q}ia$$\mathrm{i}\in S$$\tilde{q_{ia}}(j)\in\tilde{\mathbb{R}}(j\in S)$$q_{ia}(j)$$\mathrm{i}\in S$$\overline{Q}_{ia}(\cdot)$(2.3)$\overline{Q}_{\iota a}(q)=\min_{j\in S}\overline{q_{ia}}(j)(q_{\iota a}(j))$

,

where

$q=(q_{ia}(1), q_{ia}(2)$,$\ldots$, $q_{ia}(n))\in P(S)$ .

,For any fuzzy perception on $P(S|SA)$ , our fuzzy-perceptive model is denoted byis perceived with perception
in which for any $Q\in P(S|SA)$ the corresponding MDPs.
level$\mathcal{M}(\overline{Q})$$\overline{Q}$$A\Lambda(Q)$$\tilde{Q}(Q)$II for all $Q\in P(S|SA)$ is called a policy function. The
The map on $P(S|SA)$ with, the fuzzy perceptive reward
set of all policy functions will be denoted by A. For any
is a fuzzy set on denoted by$\delta(Q)\in$$\delta$$\delta\in\Delta$$\psi$$\mathbb{R}$(2.4)$\tilde{\psi}(\mathrm{i}, \delta)(x)=$$\overline{Q}(Q)$$\sup$$(i\in S)$.

$Q\in P(S|PS)$$x=\psi(i,\delta(Q)|Q)$, whereThe policy function
for allandis said to be optimal if
order
is defined in Section 1. If there exists an optimal policy function , we put$\psi=$.
,,
will be called a fuzzy perceptive value, where
Here, we can specify the fuzzy perceptive problem investigated in the next section: The
problem is to find the optimal policy function
and to characterize the fuzzy perceptive value.
$\delta^{*}\in\Delta$$\overline{\psi}(i, \delta)\neg\prec\overline{\psi}(\mathrm{i}, \delta^{*})$$\mathrm{i}\in S$$\delta\in\triangle$$\delta^{*}$$\underline{\mathrm{t}\mathrm{h}}\mathrm{e}$$\mathrm{p}\underline{\mathrm{a}\mathrm{r}}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}$$\neg\prec$$(\psi(1),\overline{\psi}(2)$$\ldots$$\psi-(n))$$\tilde{\psi}(\mathrm{i})=\overline{\psi}(\mathrm{i}, \delta^{*})(\mathrm{i}\in S)$$\delta^{*}$

3. Perceptive analysis
In this section, we derive a new fuzzy optimality relation for solving our perceptive problem.
The sufficient condition for the fuzzy perceptive reward
to be a fuzzy number is given in
the following lemma$\overline{\psi}(\mathrm{i}, \delta)$

143
For any

Lemma 3.1

Proof.

$\delta\in\triangle$, if

From normality of$\tilde{\psi}(\mathrm{i}, \delta)(x^{*})=1$

for$\psi(\mathrm{i}, \delta|Q)$$\tilde{Q}$is continuous in Q $\in P(S|SA)$ , then$\overline{\psi}(\mathrm{i}, \delta)\in$R.

, there exists $Q^{*}\in P(S|SA)$ with $\tilde{Q}(Q^{*})=1$ , such that
. For any a $\in[0,1]$ , we observed that$x’=\psi(\mathrm{i}, \delta|Q^{*})$.

$\overline{\psi}(\mathrm{i}, \delta)_{\alpha}=\{\psi(\mathrm{i}, \delta|Q)|Q\in\tilde{Q}_{\alpha}\}$

Since
of

$Q_{\alpha}$

is convex and compact, the continuity of.

$\tilde{\psi}(\mathrm{i}, \delta)_{\alpha}(\alpha\in[0.1])$$\psi(\mathrm{i}, \delta|\cdot)$means the convexty and compactness$\square$

Theorem 1.1 in Section 1 guarantees that for each $Q\in P(S|SA)$ there exists a -optimal sta. Thus, for each $Q\in P(S|SA)$ , we denote bytionary policy
the corresponding-optimal stationary policy, which is thought as a policy function.
$Q$$f_{*}^{\infty}(f_{*}\in F)$$\delta^{*}(Q)$$Q$

Lemma 3.2$\tilde{\psi}(\mathrm{i}, \delta^{*})\in\tilde{\mathbb{R}}$for all i$\in S$.

Applying Lemma 3.1, it is sufficient to prove thatis continuous in $Q\in$$P(S|SA)$ . For simplicity, for any $Q\in P(S|SA)$ , we put $\psi(Q)=(\psi_{1}(Q), \psi_{2}(Q)$ ,,. Let $Q=(q_{ia}(\cdot))$ ,there. By Theorem 1.1,
we have:
Proof.

$\tilde{\psi}(\mathrm{i}_{\dot{i}}\delta^{*}|Q)$$\psi_{n}(Q))’$$\ldots$$\overline{Q}=(\overline{q}_{iu}(\cdot))\in P(S|SA)$$\psi_{i}(Q)=\psi(\mathrm{i}, \delta^{*}|Q)(\mathrm{i}\in S)$(3.1)$\psi_{i}(Q)=\max_{a\in A}\{r(\mathrm{i}, a)+\beta\sum_{j\in S}q_{i_{\mathit{0}}}(j)\psi j(Q)\}$

(3.2)

;
.

$\psi_{i}(\overline{Q})=\max_{a\in A}\{r(\mathrm{i}, a)+\beta\sum_{j\in S}\overline{q}_{ia}(j)\psi_{j}(\overline{Q})\}$

Suppose that
respectively. Let $a=$$a_{i}=\delta^{*}(Q)(\mathrm{i})$and$\overline{a}_{i}=\delta^{*}(\overline{Q})(\mathrm{i})$$(a_{1}, a_{2}, \ldots, a_{n})’$$\psi(Q)-\psi(\overline{Q})$and$(\mathrm{i}\in S)$

give the minimum in (3.1) and (3.2). Then, it yields that$\overline{a}=(\overline{a}_{1},\overline{a}_{2}, \ldots,\overline{a}_{n})’$$\leq$$(r(a)+\beta Q(a)\psi(Q))-(r(a)+\beta\overline{Q}(a)\psi(\overline{Q}))$$=$$\beta$$=$( $Q(a)\psi(Q)-$ Qe(Q))$\beta(Q(a)-\overline{Q}(a))\psi(Q)+\beta\overline{Q}(a)(\psi(Q)-\psi(\overline{Q}))$

where $r(a)=(r(1, a_{1}),$ $r(2, a_{2})$ , . . . , $r(n, a_{n}))’$ and $Q(a)=(q_{ia_{i}}(j))$ andThus, we get$\overline{Q}(a)=(\overline{q}_{ia_{i}}(j))$$(I-\beta\overline{Q}(a))(\psi(Q)-\psi(\overline{Q}))\leq\beta(Q(a) -\overline{Q}(a))\psi(Q)$where I is an identity matrix. Since
(3.3)

,
.

,

, we have$(I- \beta\overline{Q}(a))^{-1}=\sum_{k=0}^{\infty}\beta^{k}\overline{Q}(a)^{k}\geq 0$$\psi(Q)-\psi(\overline{Q}))\leq\beta(I-\beta\overline{Q}(a))^{-1}(Q(a)-\overline{Q}(a))\psi(\overline{Q})$,$\psi(\overline{Q})-\psi(Q))\leq\beta(I-\beta Q(\overline{a}))^{-1}(\overline{Q}(\overline{a})-Q(\overline{a}))\psi(Q)$,Similarly we get
(3.4)

where

$Q(\overline{a})$and$\overline{Q}(\overline{a})$are defined similarly as the above. Observing that$0\leq\emptyset_{l}(Q),$ $\psi_{i}(\overline{Q})\leq\frac{1}{1-\beta}\max_{x\in S,a\in A}r(i, a)=:M$

and $||Q(a)-\overline{Q}(a)||\leq||Q-\overline{Q}||$ and
that
(3.5)

$||Q(\overline{a})-\overline{Q}(\overline{a})||\leq||Q-\overline{Q}||$, from (3.3) and (3.4), it holds$|| \psi(Q)-\psi(\overline{Q})||\leq\beta M\max\{||(I-\beta\overline{Q}(a))^{-1}||, ||(I-\beta Q(\overline{a}))^{-1}||\}\cdot||Q-\overline{Q}||$

.

144
When
in $P(S|SA)$ ,$||\psi(Q)-\psi(\overline{Q})||arrow
0$ .
that
$Qarrow\overline{Q}$$||(I-\beta\overline{Q}(a))^{-1}||$

Theorem 3.1

The policy function

Proof. Let

A. Since$\delta\in$

and

is

$\delta^{*}(Q)$

(3.6)

$Q$

$\delta$’, let a. By (3.6), y:=$(i)$x\in \mathbb{R}$$x=\psi(\mathrm{i}, \delta|Q)$

are bounded and (3.5) means

is optimal-optimal, for any Q $\in P(S|SA)$ it holds that
( , d’ $|Q$ )$\psi(\mathrm{i}, \delta|Q)\leq\psi$

For any$||(I-\beta Q(\overline{a}))^{-1}||$$\square$$\mathrm{i}.$$(\mathrm{i}\in S)$.

. Then, from the definition there exists.
, which implies. Then, there exists, let a$:=\overline{\psi}(\mathrm{i}, \mathrm{i}\mathrm{p}(\mathrm{x})$$y\in \mathbb{R}$$y=\psi(\mathrm{i}, \delta^{*}|Q)$with$\tilde{\psi}(\mathrm{i}, \delta^{*})(y)\geq\alpha$$\delta^{*}|Q)\geq x$

On the other hand, for. From (3.6), we have thatabove discussion yields that

$Q\in\tilde{Q}_{\alpha}$$Q\in\overline{Q}_{\alpha}$$:=\tilde{\psi}(\mathrm{i}, \delta^{*})(y)$$y\geq x:=\psi(\mathrm{i}, \delta|Q)$$\psi-(\mathrm{i},\delta)\neg\prec\tilde{\psi}(\mathrm{i},\delta^{*})$.

. This implies

such that. The$\tilde{\psi}(\mathrm{i}, \delta|Q)\leq\alpha$$\square$, so that we note by
From Lemma 3.2,.
the a-cut of
The fuzzy perceptive value
is characterized by a new fuzzy optimality
relation in Theorem 3.2, whose proof is omitted.
$\tilde{\psi}(\mathrm{i}):=\overline{\psi}(\mathrm{i}, \delta^{*})\in\overline{\mathbb{R}}(\mathrm{i}\in S)$$\overline{\psi}_{\alpha}(\mathrm{i}_{l});=[\tilde{\psi}_{\alpha}^{-}(i),\overline{\psi}_{\alpha}^{+}(\mathrm{i})]\}$$\overline{\psi}(\mathrm{i})$$\tilde{\psi}=$$(\overline{\psi^{/}}(1), \ldots,\overline{\psi}(n))’$Theorem 3.2 The fuzzy perceptive value
the following fuzzy optimality relations:
(3.7)

where

$\tilde{\psi}(\mathrm{i})=\overline{\max_{a\in A}}\{1_{\{r(i,a)\}}+\beta\tilde{Q}_{ia}\cdot\overline{\psi}\}(\mathrm{i}\in S)$$\overline{Q}_{ia}\cdot$$\sum_{j=1}^{n}q(j)\psi_{j}$$\overline{\psi}(x)=\sup\tilde{Q}_{ia}(q)$, $q=(q(1), q(2),$$\ldots$A$\tilde{\psi}(\psi)$

is a unique solution to$\tilde{\psi}=(\overline{\psi}(1), ’(\overline{\psi}(2), \ldots,\tilde{\psi}(n))’$,and the supremum is taken on the range $\{(q, \psi)|x=$,.

, $q(n))’\in P(S)$ , $\psi=(\psi(1), \psi(2),$$\ldots$$\psi(n)’\in \mathbb{R}^{n})\}$

The a-cut expression of (3.7) is as follows:(3.8)$\tilde{\psi}_{\alpha}^{-}(\mathrm{i})=\max_{a\in A}\{r(i,a)+\beta \mathrm{m}\mathrm{j}\mathrm{n}q_{ia}\in ,\alpha q_{ia}\cdot\overline{\psi}_{\alpha}^{-}\}$(3.9)$\overline{\psi}_{\alpha}^{+}(\mathrm{i})=\max_{a\in A}\{r(\mathrm{i}, a)+\beta \mathrm{m}_{\frac{\mathrm{a}}{Q}}\mathrm{x}q_{ia}\cdot\overline{\psi}_{\alpha}^{[perp]}\}q_{ia}\in_{ia_{\}}\alpha}(\mathrm{i}\in S)$

where$(i\in S)$;,$\overline{\psi}_{\alpha}^{\mp}=$$(\tilde{\psi}^{\mp}(1),\tilde{\psi}^{\mp}(2)$,$\ldots$, $\psi-\mp(n))’$ and$q_{ia} \cdot\overline{\psi}_{\alpha}^{\mp}=\sum_{j\in S}q_{\mathrm{i}a}(j)\overline{\psi}_{\alpha}^{\mp}(j)$We note that the a-cut ofin (3.7) is in
the right hand of (3.7) is well-defined.
$\tilde{Q}_{ia}\cdot\overline{\psi}$$\mathbb{C}$.

from Lemma 1.1, so that$\tilde{Q}_{ia}\cdot\overline{\psi}\in\tilde{\mathbb{R}}$. Thus,As a simple example, we consider a fuzzy perceptive model of a machine maintenance problem
dealt with in ([9],, p.17-18).
$\mathrm{p}.1$

An example (a machine maintenance problem). A machine can be operated synchronously,
say, once an hour. At each period there are two states; one is operating(state 1), and the otheris in failure(state 2). If the machine fails, it can be restored to perfect functioning by repair.
At each period, if the machine is running, we earn the return of $ 3.00 per period; the fuzzy setof probability of being in state 1 at the next step is (0.6/0.7/0.8) and that of the probability of

145
moving to state 2 is (0.2/0.3/0.4) , where for any $0\leq a<b<c\leq 1$ , the fuzzy number $(a/b/c)$

on $[0, 1]$ is defined by$(a/b/c)(x)=\{$

if $0\leq x\leq b$ ,if $b\leq x\leq 1$ .

$(x-a)/(b-a)\vee \mathrm{O}$$(x-c)/(b-c)\vee \mathrm{O}$If the machine is in failure, we have two actions to repair the failed machine; one is a usual
repair, denoted by 1, that yields the cost of $ LOO(that is, a return of -$1.00) with the fuzzy
set (0.3/0.4/0.5) of the probability moving in state 1 and the fuzzy set (0.5/0.6/0.7) of theprobability being in state 2; another is a rapid repair, denoted by 2, that requires the cost of$2.00(that is, a return of -$2,00) with the fuzzy set (0.6/0.7/0.8) of the probability moving instate 1 and the fuzzy set (0.2/0.3/0.4) of the probability being in state 2.
For the model considered, $S=\{1,2\}$ and there exists two stationary policies, $F=\{f_{1}, f_{2}\}$$=1$ and $f_{2}(2)=2$ , wherewithdenotes a policy of the usual repair anda policy of
the rapid repair. The state transition diagrams of two policies are shown in Figure 1.
$f_{1}(2)$$f_{1}$$f_{2}$(a) Usual repair$f_{1}$(b) Rapid repair$f_{2}$

Figure.l Transition diagrams.

Using (2.3), we obtain$\overline{Q}_{ia}(\cdot)(i\in S, a\in A)$, whose a-cut is given as follows(cf. [5]):{ (.6+.la, .4-.

), (.8–. la, $.2+.1\alpha)$ },$\overline{Q}_{11,\alpha}$

$=$

$co$

$\tilde{Q}_{21,\alpha}$$=$$co\{(.3+.1\alpha, .7 -.1\alpha), (.5-.1\alpha, .5+.1\alpha)\}$$\tilde{Q}_{22,\alpha}$

$=$

$co$

{

$(.6+.1\mathrm{a},$where $coX$ is a convex hull of a set $X$ .
So, putting,,(3.8) and (3.9) with $=0.9$ becom :$x_{1}=\tilde{\psi}_{\alpha}^{-}(1)$$\beta$$x_{2}=\overline{\psi}_{\alpha}^{-}(2)$$1\alpha$$.4-.1\alpha)$$y_{1}=\overline{\psi}_{\alpha}^{+}(1)$,, (.S–.la,$y_{2}=\tilde{\psi}_{\alpha}^{+}(2)$$.2+.1\alpha)$,},, the a-cut optimallyactions$\mathrm{e}$$x_{1}$$=$$3+.9 \min\{(.6+.1\mathrm{a})\mathrm{x}\mathrm{i}+(.4-.1\alpha)x_{2}, (.8-.1\alpha)x_{1}+(.2 +.1\alpha)x_{2}\}$$x_{2}$$=$$\max[-1+.9\min\{(.3+.1\alpha)x_{1}+(.7-.1\alpha)x_{2}, (.5 -.1\alpha)x_{1}+(.5 +.1\alpha)x_{2}\}$,$-2+.9$ $\min\{(.6+.1\alpha)x_{1}+(.4 -.1\alpha)x_{2}, (.8-0.1\alpha)x_{1}+(.2+.1\alpha)x_{2}\}]$ ,$y_{1}$$=$$3+.9$ $\max\{(.6+.1\alpha)y_{1}+(.4 -.1\alpha)y_{2}, (.8-.1\alpha)y_{1}+(.2+.1\alpha)y_{2}\}$$y_{2}$$=$$\max[-1+.9\max\{(.3+.1\alpha)y_{1}+(.7 -.1\alpha)y_{2}, (.5-.1\alpha)y_{1}+(.5 +.1\alpha)y_{2}\}$$-2+.9$ $\max$ { $(.6+.$ la)yi+(.4-.

$1\mathrm{a})\mathrm{y}_{2},$$(.8-0.1\alpha)y_{1}+(.2+.1\alpha)y_{2}$

} ,
$]$

,

148
After a simple calculation, we get$x_{1}=12+4.5\alpha$, $x_{2}=7+4.5\alpha$ , $y_{1}=21-4.5\alpha$ , $y_{2}=16-4.5\alpha$ .

Thus, we know the fuzzy perceptive value is$\tilde{\psi}(1)=$(12/16.5/21),$\overline{\psi}(2)=(7/11.5/16)$.

References[1] Blackwell,D., Discrete dynamic programming, Ann. Math. Statist, 33, (1962), 719-726.
[2] Derman,C, Finite State Markovian Decision Processes, Academic Press, New York, (1970).
[3] Dubois,D. and Prade,H.} Fuzzy Sets orad Systems : Theory and Applications, Academic
Press, (1980).
[4] Howard,R., Dynamic Programming and Markov Process, MIT Press, Cambrige, MA, (1960).

[5] Kurano,M., Song,J,, Hosaka,M. and Huang,Y., Controlled Markov set-chains with discounting, J. Appl. Prob., 35, (1998), 293-302.
[6] Kurano,M., Yasuda,M. Nakagami,J. and Yoshida,Y., Ordering of fuzzy sets A brief survey
and new results, J. Operations Research Society of Japan, 43, (2000), 138-148.
-[7] Kurano,M., Yasuda,M. Nakagami,J. and Yoshida,Y., A fuzzy treatm ent of uncertain
Markov decision process, 数理解析研究所講究録 1132, (2000), 221-229.
[8] Kurano,M., Yasuda,M. Nakagami,J. and Yoshida,Y., A fuzzy stopping problem with theconcept of perception, Fuzzy Optimization and Decision Making, 3, (2004), 367-374.
[9] Mine,H. and Osaki,S., Markov Decision Process, Elsevier, Amesterdam, (1970).
[10] Puterman,M.L., Markov Decision Process: Discrete Stochastic Dynamic Programming,
John Wiley & Sons, INC, (1994).
[11] Yoshida,Y. and Kerre,E.E., A fuzzy ordering on multi-dimensional fuzzy sets induced fromconvex cones, Fuzzy Sets and Systems, 130, (2002), 343-355.
[12] Zadeh,L.A., Fuzzy sets, Inform, and Control, 8, (1965), 338-353.
[13] Zadeh,L.A., Toward a perception-based theory of probabilistic reasoning with imprecise
probabilities, J. of Statistical Planning and Inference, 105, (2002), 233-264