This is the post-print version of the following article: Fumio Kanaya and Kenji Nakagawa, ``On the Practical Implication of Mutual Information for Statistical Decision Making,'' in IEEE Transactions on Information Theory, 1991. The original
publication is available at https://doi.org/10.1109/18.87006 (DOI: 10.1109/18.87006). (c) 1991 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/republishing thismaterial for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.

1.151

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 37, NO. 4, JULY 1991

Correspondence
On the Practical Implication of Mutual Information for
Statistical Decisionmaking
Fumio Kanaya and Kenji Nakagawa
Abstruct -A basic mathematical function that conjoins the two key
conceptions of mutual information and Bayes risk is defined. Thenbased on that function, some asymptotic theorems that verify an important implication of mutual information in the context of practical
Bayesian decisionmaking are proven.
Index Terms -Rate-distortion theory, statistical decision theory, Bayesrisk, mutual information, rate-distortion function.

I. INTRODUCTION
AND MOTIVATION
Mutual information that describes the amount of information
one random variable gives about a second random variable isone of the most fundamental information measures in information theory. In this discipline the concept of mutual information
has emerged in conjunction with efficient coding of sources andtheir reliable transmission over noisy channels. On the other
hand, it has also been known for some time that the concept ofmutual information is related closely to the discipline of statistical decision theory because in the context of statistical decisionmaking, it is common to utilize any information that is provided
by an observable random variable about the unknown true valueof an underlying random parameter. In the light of certain
common aspects emerging from the fundamental conception ofmutual information, it is fairly reasonable to attempt to look atthese two different disciplines from a unified perspective. This
point is supported, for instance, by Berger’s statement: “Ratedistortion theory provides knowledge about how the frequency
of faulty categorization will vary with the number and quality ofthe observations. More importantly, it also gives insight into
what set of observations would provide the most information
about the objects in question relative to the criterion of proper
categorization and, therefore, is of potential value in the designof efficient pattern recognition devices” [l,p. 91. This statement
raises the possibility that the rate distortion theory in the fieldof source encoding is applicable to the basic statistical decision
problem of pattern recognition.
However, to our knowledge, such speculations seem to haveremained heuristic, and thus it seems worthwhile to justify themby mathematically rigorous arguments. This is the motivation of
our work.
In the subsequent arguments we deal with the aforementioned interdisciplinary problem between information theory
Manuscript received May 31, 1989, revised January 3, 1991. This work
was presented in part at the 1985 IEEE International Symposium on
Information Theory, Brighton, England, June 1985 and at the 1988
IEEE International Symposium on Information Theory, Kobe, Japan,
June 1988.
The authors are with NTT Transmission Systems Laboratories, 1-2356
Take Yokosuka-shi, Kanagawa 238-03, Japan.
IEEE Log Number 9144609.

and statistical decision theory in rigorous mathematical formulations. First of all, we define a basic mathematical function toformulate a certain critical relationship that holds true between
the amount of information which is contained in the observations available to a decisionmaker and the Bayes risk that isachievable in the decisions made by utilizing these observations.
Then, after demonstrating some major properties of this basic
function, we prove two main theorems of this correspondence: adecisionmaking theorem and its converse. By the former theorem one is guaranteed that when the length of the decisionmaking procedure is sufficiently large and a proper observation isused, the probability of the actual average loss being greaterthan a certain prescribed value goes asymptotically to zero.
Proper observations can necessarily be found among those that
contain at least the amount of information required by the basic
function. Its converse theorem asserts the negation of the convergence to zero if the amount of information provided by theobservations is less than a certain permissible minimum bounddictated by the basic function, whatever decisionmaking procedure one devises. Finally, we give one theorem that suggests thatthe speed of this convergence is exponential for a sufficiently
large sample size.
11. DEFINITION
OF THE BASICFUNCTION
AND ITS PROPERTIESLet S be an unknown parameter taking values in the finite set
0 =(e}, and let X be a random variable taking values in thefinite set E = { x ) . It is assumed that a prior distribution P ( 0 )on 0 and a conditional probability matrix W = {W(xle):0 E 0,x E E }are given. Hence, the joint distribution Pr[S = 0, X = x ]= P(O)W(xlB).After having observed x, one has to choose anaction a from a finite set &={a}. Without loss of generality,
we assume that the loss function p: 0 X d+R+ a[O,m) satisfies the following condition:ayJ(e,a)0,ve E 0.
=(1)

Given a decision function $: Z+d,risk is defined byr ( $ , p ~ + ’ )= CC ~(e)w(xIe)p(w(x)).

(2)

B€@xE%

Let W be the set of all possible decision functions. Then, theminimum attainable risk defined by the subsequent equation isreferred to as the Bayes risk:r ( P , W ) P min r ( $ , P , W ) .
tJSP

(3)

In the context of statistical decision problems, one can obtain
information about 0 by observing x. This information is thenused for making decisions. The amount of information that isprovided by the observations about the unknown parameter isgiven by the well-defined mutual information between S and X,
that is,0018-9448/91/0700-1151$01.~ 01991 IEEE

1152

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 37, NO. 4, JULY 1991

Now we are ready to define a basic mathematical function Thus, considering the definition of d,(O, x) and of A, we obtain
R ( P , L). For a fixed nonnegative real number L , denote by TL
P(e)u(xle)d,(s,x)
the set of all L-admissible conditional probability matrices, i.e., L 2 C

T a ( W :r ( P , W ) < L } ,V L ~ O .
(5)
Then, the basic function R ( P , L ) is the function that is given by
the solution of the following extremum problem:
R ( P , L ) ~min z ( P , w ) ,w € rv,

V L ~ O .

(6)

It is evident from assumption (1) that the set TL is nonempty
for all L 2 0. It is also apparent that the set FL is compact,
since TLis the inverse image of a closed interval [O, L ] under acontinuous real-valued function r ( P , W ) of W. Thus, the existence of the basic function R ( P , L ) is ensured for all L 2 0
because the continuous real-valued .function Z(P, W ) must assume a minimum in any nonempty compact set of W.
Apparently in definition R ( P , L ) is very similar to the wellknown rate distortion function in the field of source encoding.
However, R ( P , L ) is more difficult to evaluate because theBayes risk r ( P , W ) is not a simple average distortion, but isitself the solution of a different functional optimization problem
that is essential to any decisionmaking procedure.
We are now in the position to demonstrate fundamental
properties of R ( P , L). First of all we give the following lemma.
Lemma 1: For each decision function $ E V, let R ( P , I,!I, D )be the rate-distortion function associated with the hypothetical
distortion measure d,: O x Z+R+ that is defined asd,(e , x ) 4% p ( e, $ ( x ) ) ,ve E 0, v x E x.

(7)

That is, let R( P , $, D ) be the solution to the following optimization problem:

i

(w: 1

K,(+)

P(e)w(xie)d,(e,x) 5 D ,f l S @ X € E

R(P,$,D)=min

Z(P,W).

R(P,L)= R(P,p,L),

V L > 0 , V P on 0.

(14)

This theorem can be derived as a rather immediate consequence of both the last lemma and Stjernvall's dominancetheory [2]. Prior to proof of this theorem, we will state therelevant definitions and theorem given by Stjernvall in appropriate terminology.
Definition 1: Let d ( x , y ) be a distortion measure between
finite alphabets .!X={1,2;..,1ZI} and 9=(1,2;..,IW/D. We
can define the vectors as

VY E 9.
d ( Y ) = (d(1, Y ) , 4 2 , Y ); . . ,d(lXl, Y ) ) ,Definition 2: Let the convex hull of a set d be denoted by
the symbol c o n v d . The characteristic set @ ( d ) of the distortion measure d x , y ) can then be defined as(9)@( d ) = convb( d ) + Z,

Then,

V L 2 0 , VPonO.

Theorem I : Let R ( P , p , 0 )be the rate distortion function thatregards 0 and d as a hypothetical source alphabet and reproduction alphabet, respectively, with the loss function p(0, a )adopted as the hypothetical distortion measure between these
two alphabets. Then the following equality holds:(8)

w 6 TDv,cS)

R ( P , L ) = min R ( P , $ , L ) ,Since this implies that U E TL,
it is proved that YLc YL.
This lemma assures us that the function R ( P , L ) can becomputed by determining the minimum among a finite-numberof rate-distortion functions, because (W\Ir( is equal to ldllzl,
which is obviously finite, Here we denote the cardinality of a set
by 1.1. Note that in the aforementioned case, respective distortion measures are induced from one and only one loss functionp ( 0 , a ) through decision functions as defined by (7). Next, wedemonstrate an important property of the basic function R ( P , L )that reveals the possibility of its evaluation without explicitly
undertaking the often very difficult and impractical solution of
the Bayes risk for a decision problem.

VE\lr

Proof: Let 'yL be the union of a collection of sets (VL($):where$ E 91, i.e.,

7 q d ) = ( d ( y ) :Y € 9 1
and
X = { k :k x 2 0,Vx E 2 ) .

The proof immediately follows from the proposition that
equation % = YL holds for the set YL that is previously
defined by (5). Since it is apparent that Y L 2 Y L ,proof iscompleted if we demonstrate the opposite relation.
Given any conditional probability matrix U E YL, it follows
from the definition of yL that there exists a nonempty subset
A c W consisting of every decision function $ that satisfies thefollowing:

P(e)u(xle)d,(e,x) 5 L.
fl€@

X

€

(11)

X

Since it is obvious that A is a finite set, a member S must be
found in it that satisfies the following minimal condition:c cP(e)u(xlw,(e>x>e € @x ~ ZI
f l € @ X € E

P(e)u(xle)d,(e,x),

V$EA.

(12)

Stjemuall's Theorem [2, p . SOOJ Assume the two distortion
measures d,: X x 9-R' and d,: LT x $2 --+ R i satisfy thecondition B ( d , ) c B ( d , ) . Then, for all distortion D > 0 and all
proability distributions P on X
R(P,d,,0 ) 2 R(P,d,, D ) ,where R ( P , d , D ) denotes the rate-distortion function associatedwith the distortion measure d.
Now we prove Theorem 1.

Proof of Theorem I: Let the distortion measure defined by
the loss function p(0,Cr) be denoted by do: 0 X d + R + andthe distortion measure defined by p(0, $(x)) for any given $ bedenoted by d,: O X $(.!X)+ R + , where $ ( E )denotes thed.
Then it is apparent
image of X under the mapping 4: Z-+from Definition 2 that B ( d , ) c B ( d , ) , since $(XI c d for all.!?Z and all $ E W.Thus in the light of Stjernvall's Theorem, for1153IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 37, NO. 4, JULY 1991

all Eand all I) E ?,R ( P , I ) ,D ) 2 R ( P , p , D ) ,V D > 0 , V P on 0 . (15)Now, consider some X with IX121dl.It is then obviousthat a decision function 4 must exist for which b ( X ) =dholds. We now prove that at least for this 4R( P , $ , D ) = R( P , p , D ) ,V D > 0 , V P on 0 . (16)For the purpose of this proof let WO be the optimal W that
achieves R ( P , p , D). ThenC C P ( e ) w o ( a l e ) p ( e , a )I D ,(17)

BE@ a€&

on the existence of the basic function that for all L 2 0 there
exists at least one stochastic matrix that achieves a point( L , R ( P , L)) on the R ( P , L ) curve. However, it is to be notedthat the aforementioned optimum stochastic matrix is not automatically computable by way of the extremum problem (6) thatdefines the basic function R ( P , L), because the cardinality ofthe observation data set .!Z is not specified and hence it may be
a n arbitrary finite integer. Now, by means of the next theorem,
which can be easily obtained from the proof of Theorem 1, it isassured that we can find at least one optimum stochastic matrix
achieving the point ( L , R( P , L)) within the L-admissible set
comprising only those matrices of a fixed size of I 0 l X Id[.
Theorem 2: To every stochastic matrix W: 0 - X that
achieves a point ( L , R( P , L ) ) on the curve of R ( P , L ) under the
> Id[
there
, exists a corresponding stochascondition thattic matrix V: 0 + 9 such that I/ also achieves the same point
91= Id1holds.
and 1
By combining this theorem with Theorem 1, we are led to theconclusion that every stochastic matrix achieving a point on therate-distortion R( P, p , L ) curve is a representative of the set of
all solution matrices to the extremum problem (6) that defines
the basic function R ( P , L ) . It should be noticed that the randomvariable Y that takes values in fV and is specified by the joint
distribution Pr[S = 0,Y = y ] = P(O)V(yle)is actually a sufficient
statistic of the corresponding observed random variable X forthe unknown parameter S (see, e.g., [4, pp. 303-3061).

1x1

111. MAINTHEOREMSThe basic function specifies by definition the minimum possible amount of information that must be given by the observations about the unknown parameter for one to achieve theBayes risk not greater than the prescribed value L. The purposeof the subsequent argument is to demonstrate the practical
implication of the basic function R ( P , L ) through proving main
theorems that relate it to the asymptotic behavior of an actual
decisionmaking procedure.
Now consider an actual decisionmaking context in which acomponent statistical decision problem having identical genericstructure is repeated n times. We assume that there is no
relationship among the n repetitions. As a practical matter, thequestion of major interest to the decisionmaker is the actual(random) averaging loss that accrues from the real decisionmaking procedure. Suppose that a compound decision function
0 " :X" d" is used. Then the actual average loss having
resulted from the n decisions is given by-

2 R(P,4,D).
(22)
Thus, by (22) and (15) we obtain (16). Finally it follows from(151, (161, and Lemma 1 that

R ( P ,L ) = R ( P , 4 ,L ) = R ( P , p , L ) ,VL > 0, V P on 0 .
(23)This completes the proof of Theorem 1.
By virtue of this theorem, it becomes possible to reduce thecomputation of the basic function R ( P ,L ) to that of the welldefined rate-distortion function in information theory, which
circumvents the cumbersome solution of the Bayes risk. Convexity, continuity and strictly decreasing property of the rate-distortion function is guaranteed also for the basic function R ( P , L )by the last theorem.
As a final consequence of this section, we give an important
property of the stochastic matrix that attains a point on the
R ( P , L ) curve. It is evident from the argument previously made7 -where an n-length sequence of observed data is denoted by avector x = x l x 2 . . . x , E 2"" and an underlying sequence oftrue parameters is denoted by a vector 0 = Ole2.. . O n E 0".We
also use for a compound decision function notation W " ( x ) =Wl(x)q2(x).. . ?Jx) E d".
Henceforth, we denote by E,.,,.[.]
the mathematical expectation of a random variable with respect
to the joint distribution Pr[S" = 8, X" = x] = P"(B)W"(xle) ofrandom variables S" and X".
Prior to stating the main theorems, we give two preliminarylemmas needed for their proof.
Lemma 2: To every R satisfying the condition that R ( P , L ) I
R I log 101, there exist a stochastic matrix W : 0 + X satisfying
Z ( P , W ) = R and, for every positive integer n , a compound1154

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 37, NO. 4, JULY 1991-

decision function 9":
X" d" such that

where S(a, b ) is defined as

E s . , x . [ f " ( @ , * " ( x > ) l ~L .
Proof Assume for the compound decision function 9"a
simple symmetric rule +"(x) = ~ ( x , ) + ( x , ) . . 4 ( x n ) with eachcomponent 4 being a Bayes rule against P that is determinedby (3). Then the lemma follows immediately from the definitionof the basic function R ( P , L ) and the hypothesis that an identical statistical decision problem is repeated independently.
Lemma 3: Suppose that for a given positive integer n there
exist a stochastic matrix W: 0 .5E" and a compound decision
X"+ d" such thatfunction 9":

Then for any given prior P on 0 and stochastic matrix W:
0 + X we denote the set of all jointly typical sequences by (cf.
t31)

V ( B , X ) E @ X . ~ E "(28)}.
Here the additional requirement must be met such that

Then we must haveN(B,xle, x) = 0 whenever P(B)W(xlB)= 0 and the sequence (6,)

I(P,W)>R(P,L).
Proof First it can be shown by some calculation that forevery positive integer n and every compound decision function

9":
X" d"
--f

where D, represents the next distortion associated with a properly defined stochastic matrix@ +d

is chosen such that

-

S,

--f

0,

h . 8 , -m,

as n -m.

(29)

Now, according to Lemma 2, we can choose W: 0 + X and9":
X" d" such that I ( P , W ) = R and 9 " ( x ) = c#~(x,)~ $ ( x , ). . . 4 ( x , ) with components 4 satisfying &E
P(B)W(xlB)p(B,+(x))iL . Then it follows from (27) and (28)that for every (e, x) E TFwls,

w:

D,P

c c

P(~)w(alqP(e,o.

8 € @ a ~ dHence, by using a data processing lemma (see, e.g., [3, pp.
55-56]) and a rate-distortion theorem (e.g., [ S , Theorem 9.2.1]),
we can easily obtain

I ( P , W ) 2 R( P , P ,E s . , x . [ P " ( ~ , ~ " ( x > > ] ) .(25)Then it follows by the strictly decreasing property of the ratedistortion function R ( P , p , D ) and from the postulate
E,n,Xn[pn(e,W"(x))l I L that

I(P,W)?R(P,p,L).
(26)
Finally the lemma follows immediately from both (26) andTheorem 1.
Now we are ready to prove two main theorems. First we provethe decisionmaking theorem. The significance of this theorem isthat it ensures that the probability of averaging loss greater thansome prescribed value L goes asymptotically to zero for sufficiently large values of n, if one uses proper observations thatyield the amount of information about unknown parameter notless than is required by the basic function R ( P , L).
Theorem 3 (Decisionmaking Theorem): T o every R satisfying
the condition that R ( P , L ) I R 2 log 101, there exists a stochastic matrix W : 0 + X satisfying Z(P, W )= R and a sequence ofn-length decisionmaking procedures (9":X"+ d"),
n=1,2, . . . such that for every 7 > 0 and E > 0
Pr[p"(e,P"(x)) I L + 771 2 I - E ,where we putiOn the other hand, by using Chebyshev's inequality, it is easily
shown that

Hence, considering (29), the theorem follows immediately from(30) and (31).

Due to the general feeling that in the practical statisticaldecisionmaking case the stochastic matrix W cannot be selected
arbitrarily as opposed to the test channel in the rate distortion
case, this theorem would not be considered of so much practicalsignificance. However, it should be noticed that in the context ofthe practical pattern recognition, the problem of feature selection has long been recognized and intensely studied. (For an
enlightening perspective of this field, see e.g., [6]J It could beconsidered from the statistical decision theoretic viewpoint thatthe essence of this problem virtually consists in the selection of
the optimal stochastic matrix W within the admissible set that isdetermined by some practical constraints. Thus, for instance, itwould be possible for this theorem to help prompt a new idea of
optimal feature selection.
Next we prove the converse to the decisionmaking theorem.

-

whenever n 2 n0(77,E).

Theorem 4 (Conuerse Theorem): For every stochastic matrix
Proof For a pair of sequences ~ = B , B 2 ~ ~ ~ 8 , ,andEO"
W , 0 X subject to the constraint that Z(P, W )< R ( P , L),x = x l x 2 . . . x , E X" we denote its joint type by
there exists a positive number p ( P , W ,L ) such that1
1 "Pr[p"(e,*"(x)) > L ] 2 P ( P , W , L ) ,--N(@,xi%x) 8 s((~,(e,,x,)),C

r=lv(e, x ) E o x x, (27)for every positive integer n and every n-length decisionmaking': X"+ d".
procedure W

1155

IEEE TRANSACTIONS ON INFORMATIONTHEORY, VOL. 37, NO. 4, JULY 1991

Proof: Suppose we are given an arbitrary compound deciThen we havesion function Q": E"-+ d".

W: X + W , i.e.,

D(VIlWIP) 4

P(x)D(V(.lx)llW(.Ix)).
X

A

Theorem 5: Let L be a positive number less than pM, and
W: 0 X an arbitrary stochastic matrix satisfying Z(P, W )2
R(P, L - 7)for some arbitrarily small 7 > 0. Then for every
X" d",
compound decision function 8":Pye)wyxie))r.

-+p n ( e , w ( x ) ) >L

Pfl(e)wn(xle)+PM

Z

Here notations V(.ln) and W ( . l x ) have been used for row
vectors of respective stochastic matrices.

Pfl(e)wfl(xle)pfl(e,Qfl(x))(e,x)< (1 -€--fpn(e,.rlrn(x))>L= L + ( ~ -, L )Pn(e)wfl(xle).
Lpfl(e,*n(~))>

=-

inf

D(VIlW1P).

v:I ( P , V ) < R ( P , L )Hence, for every L < pM,~r[p"(e,Q"(x))> L ] O

Proof: Let I/: 0 .+ 9"
be an arbitrary stochastic matrix suchthat Z(P, V )< R( P, L ) . According to Theorem 4 we have forany given Q": X" d",Pfl(e)wfl(xle)p"te,P"(x))> L-+~ S ~ , , . [ P " ( ~ ? Q " ( ~L) ) l 2. (32)

P f l V n ( { ( e , x ) p: f l ( e , Q f l ( x ) ) > L ) ) > ~ ( P , v , L ) > ~ ,pM-L

On the other hand, it follows from Lemma 3 that for every
positive integer a , we must have

Z(P,W)2R(P,E,,,,,[P"(e,Qfl(x))l).

(33)

Therefore, it follows immediately from Stein's Lemma (see.,e.g., [3, p. 281) that for every S > 0 and sufficiently large n
1- log pnwfl({ (e,x): p n ( e , 8 f l ( x ) ) > L } )Since by postulate of this theorem

R(P,L)> I(P,W),

for n = 1,2, . . . . (37)

> - D(P X VIIP X W )- 6 .

(34)

it follows from the last two inequalities that

R ( P , L ) > Z ( P , W )2 R ( P , E , , , , , [ p " ( 8 , Q f l ( x ) ) l ) . ( 3 5 )Let A ( P , R ) be the inverse of the basic function R ( P , L ) . Thenby using the strictly decreasing property of R ( P , L ) , it follows
from the last inequality that for every n and Q": X"-+ d",

L < A ( P , Z ( P , W ) ) IE y , x n [ ~ " ( e , Q " ( x ) ) ] .
Combining this inequality with (321, we have

(38)

Here P x V and P X W designate the joint distributions
P(B)V(xJB)and P(B)W(xlB), respectively. Thus, considering thefact that by definitionPr[p"(e,W"(x)) > L ] = P f l w f l ( { ( e , x ) p: f l ( e , Q f l ( x ) ) > L ) ) ,we obtain for every 8":
X" d"and sufficiently large n-+

1-logPr[pfl(e,Qfl(x))>L]2n-D(VlIWlP)-S,

V6>0.
(39)

The converse theorem then follows from this inequality immediately if we put p ( P , W ,L ) = ( A ( P ,Z(P, W ) ) - L ) / ( p M - L).
According to the converse theorem one can not always be
guaranteed of achieving the average loss of the prescribed value
L as the length n goes to infinity if one utilizes for decisionmaking the observations that provide less amount of information
about the unknown parameter than is dictated by the basic
function R ( P , L ) .
Consequently, through proving these two theorems it is possible to verify the practical implication that the mutual information has in the context of actual decisionmaking procedure.
Finally we give one theorem that concerns the speed ofconvergence in probability. We henceforth denote by D(PllQ)
informational divergence between two distributions P and Q on
a finite set X,i.e.,

and also denote by D ( V I ( W I P )conditional informational divergence between two stochastic matrices V: X + 9 and1Here we have used the identity D(P X VllP X W )= D(VIlW1J').
Since V was arbitrary subject to the constraint that Z(P, V )<R ( P , L ) and 6 > 0 can be made arbitrarily small, proof follows
immediately from (39).
It should be noted that by nonnegativity of informational
divergence and the postulate Z(P, W )2 R ( P , L - 7)for some
7 > 0, inf,D(VIIWIP) is guaranteed to be strictly positive in thelast theorem.
IV. CONCLUSION
We have presented an attempt to look at the two different
disciplines of Shannon theory and Bayes statistical decision
theory from a unifying perspective. We have viewed as key
notions the mutual information in the former and the Bayes riskin the latter. First we have defined a basic mathematical function that conjoins them. Then, through mathematically rigorousarguments based on major properties of this basic function, we
have proved a decisionmaking theorem and its converse whichenables us to verify that even for actual statistical decisionmaking procedures, the mutual information has important pragmatic
implications, as has long been recognized in conjunction withsource and channel coding procedures.

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 37, NO. 4, JULY 1991

1156

ACKNOWLEDGMENT
The authors are grateful to the anonymous referees for theirvaluable suggestions and comments that improved the presentation of the results in this correspondence.

nent of the original process at scale a. A consequence ofFlandrin’s computation is that fractional Brownian motion is
stationary at each fixed scale. In particular, when X ( t ) is afractional Brownian motion, the covariance of the process t rf
x X ( t ) is of the form

REFERENCES[l] T. Berger, Rate Distortion Theory. Englewood Cliffs, NJ: PrenticeHall, 1971.
[2] J. Stjernvall, “Dominance-A relation between distortion measures,”IEEE Trans. Inform. Theory, vol. IT-29, no. 6, pp. 798-807, Nov.
1983.
[31 I. Csiszir and J. Korner, Information Theory: Coding Theorems for
Discrete Memoryless Systems. New York Academic Press, 1981.
[41 S . Guiasu, Information Theory with Application. London: McGrawHill, 1977.
[5] R. G. Gallager, Information Theory and Reliable Communication.
New York John Wiley, 1968.
[61 S. D. Morgera and L. Datta, “Toward a fundamental theory of
optimal feature selection: Part I,” IEEE Trans. Pattern. Anal. Machine Intell., vol. PAMI-6, pp. 601-616, Sept. 1984.

On the Wavelet Transform of Fractional
Brownian Motion
J. Ramanathan and 0. Zeitouni
Abstract -A theorem characterizing fractional Brownian motion by
the covariance structure of its wavelet transform is established.

Index Terms -Wavelet transform, fractional Brownian motion.

I. INTRODUCTION
The wavelet transform of a function f(t) is defined by theformula(E [ K X ( t ) % X ( s ) ] = uAp - ,where p is a positive definite function determined in an explicit
manner by the order or the fractional Brownian motion and thedefining function g ( t ) of the wavelet transform. This fact is usedby Flandrin to make rigorous sense of the spectral content of
fractional Brownian motion.
It is natural to ask whether there are other Gaussian processes whose wavelet transforms have this natural covariancestructure. In addition, are there any Gaussian processes whose
wavelet transform is stationary with respect to the affine group(i.e., the statistics of the wavelet transform do not depend on
translations and dilations of the process)? The purpose of thispaper is to point out that the answers to both these questions
are negative. In particular, fractional Brownian motion is characterized by the property that its wavelet transform has the formshown above in (1). A consequence is that there are no nontrivial Gaussian processes on the real line whose wavelet transformproduces a random field that is stationary with respect to theaffine group. It should be remarked that the results presented
assume some fairly nonrestrictive growth conditions on thecovariance of the process X(t) and the kernel g ( t ) used todefine the wavelet transform (see Remark 2 at the end of thecorrespondence).
11. A CHARACTERIZATIONOF FRACTIONALBROWNIAN
MOTIONLet X ( t ) be a Gaussian random process defined for t E R
such that the covariance R(s,t ) = E [ X ( s ) X ( t ) ]is continuousand satisfies

R(0,O) = 0

where g ( t ) is a fixed function, t E R, and a E R+. This transform
yields a joint time-scale representation of the original input
function that has been of great recent interest. (See, e.g., [l],[2],and [4]).
where N E Z is fixed. In addition, we also impose the following
In a recent correspondence, Flandrin [3] proposed the use of conditions on the analyzing function g ( t ) :the wavelet transform to analyze the behavior of fractionala) /?Ag(t)I(1+ It12)N/2dt<m,Brownian motion, a highly nonstationary random process. (For ab) ~~~I6(5)l2/151d5<m, andbackground on fractional Brownian motion and some of itsc) 6 is smooth and has a simple zero at the origin.
applications, see [SI and [6]).The wavelet transform of a stochastic process X ( t ) is a random field Y X ( t , a ) on the upper half
(The Fourier transform of g is denoted by 6.) The growthplane. The process t x X ( t ) can be thought of as the compoconditions for the process X ( t ) and the analyzing wavelet g ( t )insure that the wavelet transform produces a random field with
Manuscript received December 19, 1989; revised January 6, 1991. J.
Ramanathan was supported by the MITRE sponsored Research Pro- finite covariance.
gram. 0. Zeitouni was supported in part by the Army Research Office
Theorem: Suppose the wavelet transform E X satisfies
under Contract DAALB3-86-K-0171.This work was performed while 0.
Zeitouni was visiting the Laboratory for Information and Decision
Systems at M.I.T.
E[~X(t)~X(s)]=aA
p ,J. Ramanathan was with the MITRE Corporation, Mail Stop E025,
Burlington Road, Bedford, MA 01730. He is now with the Departmentof Mathematics, Eastern Michigan University, Ypsilanti, MI 48197.
where A E R and p is a positive semidefinite function on the real
0. Zeitouni is with the Department of Electrical Engineering, Techline. Then X ( t ) is fractional Brownian motion of order H =nion-Israel Institute of Technology, Haifa 32000, Israel.
(A - l)/Z. In particular, A E (1,3).
IEEE Log Number 9100446.

-(Y)

0018-9~8/91/0700-1156$01.00 01991 IEEE