# A Strong Limit Theorem for Functions of Continuous Random Variables and an Extension of the Shannon-McMillan Theorem

Authors: Gaorong Li; Shuang Chen; Sanying Feng
Journal: Journal of Applied Mathematics
 (2008)

---

## Abstract

By means of the notion of likelihood ratio, the limit properties of the sequences of arbitrary-dependent continuous random variables are studied, and a kind of strong limit theorems represented by inequalities with random bounds for functions of continuous random variables is established. The Shannon-McMillan theorem is extended to the case of arbitrary continuous information sources. In the proof, an analytic technique, the tools of Laplace transform, and moment generating functions to study the strong limit theorems are applied.


---

## Body

## 1. Introduction

Let{Xn,n≥1} be a sequence of arbitrary continuous real
random variables on the probability space (Ω,ℱ,P) with the joint density
function(1.1)fn(x1,…,xn)>0,n=1,2,…,where xi∈(−∞,∞), 1≤i≤n.
Let Q be another probability measure on ℱ,
and {Xn,n≥1} is a sequence of independent random variables
on the probability space (Ω,ℱ,Q) with the marginal density functions gk(xk)  (1≤k≤n), and let(1.2)πn(x1,…,xn)=∏k=1ngk(xk).
In order to indicate the deviation between {Xn,n≥1} on the probability measure P and Q,
we first introduce the following definitions.Definition 1.1.
Let{Xn,n≥1} be a sequence of random variables with joint
distribution (1.1), and let gk(xk) (k=1,2,…,n) be defined by (1.2).
Let (1.3)Zn(ω)=fn(X1,…,Xn)πn(X1,…,Xn).In statistical terms, Zn(ω) is called the likelihood ratio, which is of
fundamental importance in the theory of testing the statistical hypotheses (cf.
[1, page 388];
[2, page 483]).The random variable(1.4)r(ω)=limsup⁡n→∞1nln⁡Zn(ω)=−liminf⁡n→∞1nln⁡[{∏k=1ngk(Xk)}/fn(X1,…,Xn)]is called asymptotic logarithmic
likelihood ratio, relative to the product of marginal distribution of (1.2), of Xn, n≥1,
where ln is the natural logarithm, ω is the sample point. For the sake of brevity,
we denote Xk(ω) by Xk.Althoughr(ω) is not a proper metric between probability
measures, we nevertheless think of it as a measure of  “dissimilarity”
between their joint distribution fn(x1,…,xn) and the product πn(x1,…,xn) of their marginals.Obviously,r(ω)=0,
a.s. if and only if {Xn,n≥1} are independent.A stochastic process of fundamental importance in the
theory of testing hypotheses is the sequence of likelihood ratio. In view of
the above discussion of the asymptotic logarithmic likelihood ratio, it is
natural to think ofr(ω) as a measure how far (the random deviation of) Xn is from being independent, how dependent they
are. The smaller r(ω) is, the smaller the deviation is (cf.
[3–5]).In [3], the strong deviation theorems for discrete random
variables were discussed by using the generating function method. Later, the
approach of Laplace transform to study the strong limit theorems was first
proposed by Liu [4].
Yang [6] further
studied the limit properties for Markov chains indexed by a homogeneous tree
through the analytic technique. Many comprehensive works may be found in Liu
[7]. The purpose of
this paper is to establish a kind of strong deviation theorems represented by
inequalities with random bounds for functions of arbitrary continuous random
variables, by combining the analytic technique with the method of Laplace
transform, and to extend the strong deviation theorems to the differential
entropy for arbitrary-dependent continuous information sources in more general
settings.Definition 1.2.
Let{hn(xn),n≥1} be a sequence of nonnegative ℬ orel measurable functions defined on ℛ,
the Laplace transform of random variables hn(Xn) on the probability space (Ω,ℱ,Q) is defined by(1.5)fn(s)=∫−∞∞e−shn(xn)gn(xn)dxn=EQe−shn(Xn),n=1,2,…,where EQ denotes the expectation under Q.We have the following assumptions in this paper.(1)
Assume
that there existss0∈(0,∞) such that(1.6)fn(s)<∞,s∈[−s0,s0],n=1,2,….(2)
AssumeM>0 is a constant, satisfying(1.7)sup⁡nEQhn(Xn)≤M<∞.In order to prove our main results, we first give a
lemma, and it will be shown that it plays a central role in the
proofs.Lemma 1.3.
Letfn(x1,…,xn), gn(x1,…,xn) be two probability functions on (Ω,ℱ,P),
let(1.8)Tn(ω)=gn(X1,…,Xn)fn(X1,…,Xn),then(1.9)limsup⁡n→∞1nln⁡Tn(ω)≤0a.s.Proof.
By [8], {Tn,ℱ,n≥1} is a nonnegative martingale and ETn=1, we have by the Doob martingale convergence
theorem, there exists an integral random variable T∞(ω),
such that Tn→T∞,
a.s. and (1.9) follows.


## 2. Main Results

Theorem 2.1.
Let{Xn,n≥1}, Zn(ω), r(ω), fn(s) be defined as before, and under the
assumptions of (1) and (2), let (2.1)limsup⁡n→∞1n∑k=1nEQes0hk(Xk)=limsup⁡n→∞1n∑k=1nfk(−s0)=M(s0)<∞.Then(2.2)liminf⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≥−β(r(ω)),a.s.(2.3)limsup⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≤β(r(ω)),a.s.where(2.4)β(x)=inf⁡{ϕ(s,x),−s0<s<0},x≥0,(2.5)ϕ(s,x)=−2e−2sM(s0)(s0−|s|)2−xs,x≥0,(2.6)0≤β(x)≤x(1+x)s0[2e−2M(s0)+1],lim⁡x→0+β(x)=β(0)=0.Remark 2.2.
Let(2.7)α(x)=sup⁡{ϕ(s,x),0<s<s0},x≥0,then(2.8)α(x)=−β(x).Proof.
Lets be an arbitrary real number in (−s0,s0),
let(2.9)gn(s,xn)=e−shn(xn)gn(xn)/fn(s),xn∈(−∞,∞),n=1,2,…,then ∫−∞∞gn(s,xn)dxn=1,
and let(2.10)qn(s;x1,…,xn)=∏k=1ngk(s,xk).Therefore, qn(s;x1,…,xn) is an n multivariate probability density function,
let(2.11)tn(s,ω)=qn(s;X1,…,Xn)fn(X1,…,Xn),n=1,2,….By Lemma 1.3, there exists a set A(s) such that P(A(s))=1, so we have(2.12)limsup⁡n→∞1nln⁡tn(s,ω)≤0,ω∈A(s).By (1.3), (2.9), (2.11), and
(2.12), we have(2.13)limsup⁡n→∞1n[−s∑k=1nhk(Xk)−∑k=1nln⁡fk(s)−ln⁡Zn(ω)]≤0,ω∈A(s).Therefore,(2.14)r(ω)≥0,ω∈A(0).By (2.13) and (1.4), the
property of the superior limit(2.15)limsup⁡n→∞(an−bn)≤0⟹limsup⁡n→∞an≤limsup⁡n→∞bn,and the inequality lnx≤x−1 (x>0), we have(2.16)limsup⁡n→∞1n(−s)∑k=1n[hk(Xk)−EQhk(Xk)]≤limsup⁡n→∞1n∑k=1n[ln⁡fk(s)+sEQhk(Xk)]+r(ω)≤limsup⁡n→∞1n∑k=1n[fk(s)−1+sEQhk(Xk)]+r(ω)=limsup⁡n→∞1n∑k=1n[fk(s)−1−(−s)EQhk(Xk)]+r(ω),ω∈A(s).By the inequality 0≤ex−1−x≤(1/2)x2e|x|,
which can be found in [9],
we have(2.17)0≤e−shk(Xk)−1−(−s)hk(Xk)≤(1/2)(shk(Xk))2e|shk(Xk)|.By (2.5) and (2.17), we
have(2.18)limsup⁡n→∞1n(−s)∑k=1n[hk(Xk)−EQhk(Xk)]≤(1/2)s2limsup⁡n→∞1n∑k=1nEQ[(hk(Xk))2e|s|hk(Xk)]+r(ω),ω∈A(s).
It is easy to see that φ(x)=txx2 (t>1)  attains its largest value φ(−2/lnt)=4e−2/(lnt)2 on the interval (−∞,0],
and φ(x)=txx2 (0<t<1) attains its largest value φ(−2/lnt)=4e−2/(lnt)2 on the interval [0,∞),
we have(2.19)sup⁡{e(s−s0)hk(Xk)[hk(Xk)]2,k≥1}≤4e−2(s−s0)2=4e−2(s0−s)2,0<s<s0,(2.20)sup⁡{e(s0+s)(−hk(Xk))[−hk(Xk)]2,k≥1}≤4e−2(s0+s)2,−s0<s<0.Let 0<s<s0 in (2.18), by (2.19) and (2.1), we
obtain(2.21)(−s)liminf⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≤(1/2)s2limsup⁡n→∞1n∑k=1nEQ[(hk(Xk))2e|s|hk(Xk)]+r(ω)=(1/2)s2limsup⁡n→∞1n∑k=1nEQ[es0hk(Xk)e(s−s0)hk(Xk)(hk(Xk))2]+r(ω)≤2e−2s2M(s0)(s0−s)2+r(ω),ω∈A(s).Dividing the two sides of (2.21)
by −s,
we obtain(2.22)liminf⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≥−2e−2sM(s0)(s0−s)2−r(ω)s=^ϕ(s,r(ω)),0<s<s0,ω∈A(s)∩A(0).By (2.14) and 0<s<s0,
obviously ϕ(s,r(ω))≤0,
hence α(r(ω))≤0.
Let Q+ be the set of rational numbers in the interval (0,s0),
and let A*=∩s∈Q+A(s), then P(A*)=1.
By (2.22), then we have(2.23)liminf⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≥ϕ(s,r(ω)),ω∈A*∩A(0),∀s∈Q+.
It is easy to see that ϕ(s,x) is a continuous function with respect to s on the interval (0,s0).
For each ω∈A*∩A(0) (0≤r(ω)<∞), take sn(ω)∈Q+,  n=1,2,…, such that(2.24)lim⁡n→∞ϕ(sn(ω),r(ω))=α(r(ω)).By (2.23), (2.24), and (2.8), we
have(2.25)liminf⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≥−β(r(ω)),ω∈A*∩A(0).Since P(A*∩A(0))=1,
(2.2) follows from (2.25).
Let−s0<s<0 in (2.18), by (2.20) and (2.1), we
have(2.26)limsup⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≤−(1/2)slimsup⁡n→∞1n∑k=1nEQ[(hk(Xk))2e|s|hk(Xk)]−r(ω)s=−(1/2)slimsup⁡n→∞1n∑k=1nEQ[es0hk(Xk)e(s0+s)(−hk(Xk))(−hk(Xk))2]−r(ω)s≤−2e−2sM(s0)(s0+s)2−r(ω)s=^ϕ(s,r(ω)),−s0<s<0,ω∈A(s)∩A(0).By (2.14) and −s0<s<0,
obviously ϕ(s,r(ω))≥0,
hence β(r(ω))≥0.
Let Q− be the set of rational numbers in the interval (−s0,0),
and let A*=∩s∈Q−A(s), then P(A*)=1.
Then we have by (2.26)(2.27)limsup⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≤ϕ(s,r(ω)),ω∈A*∩A(0),∀s∈Q−.It is clear that ϕ(s,x) is a continuous function with respect to s on the interval (−s0,0).
For each ω∈A*∩A(0) (0≤r(ω)<∞), take λn(ω)∈Q−, n=1,2,…, such that(2.28)lim⁡n→∞ϕ(λn(ω),r(ω))=β(r(ω)).By (2.27) and (2.28), we
have(2.29)limsup⁡n→∞1n∑k=1n[hk(Xk)−EQhk(Xk)]≤β(r(ω)),ω∈A*∩A(0).Since P(A*∩A(0))=1,
(2.3) follows from (2.29).
By (2.4), (2.5), and (2.14), if x>0,
we have(2.30)0≤β(x)≤ϕ(−s0x1+x,x)=x(1+x)s0[2e−2M(s0)+1].If x=0,
we have(2.31)β(0)≤ϕ(−n−1,0)=2e−2M(s0)n(s0−n−1)2,n≥1.Noticing that β(x)≥0, (x≥0),
(2.6) follows from (2.30) and (2.31).Corollary 2.3.
IfP=Q,
or {Xn,n≥1} is a sequence of independent random variables,
and under the assumptions of (1) and (2), then(2.32)lim⁡n→∞1n∑k=1n[hk(Xk)−Ehk(Xk)]=0a.s.Proof.
In this case,fn(x1,…,xn)=∏k=1ngk(xk), and r(ω)=0 a.s. Hence, (2.32) follows directly from
(2.2) and (2.3).


## 3. An Extension of the Shannon-McMillan Theorem

In order to
understand better, we first introduce some definitions in information theory in
this section.Let{Xn,n≥1} be a sequence produced by an arbitrary
continuous information source on the probability space (Ω,ℱ,P) with the joint density
function(3.1)fn(x1,…,xn)>0,xk∈(−∞,∞),1≤k≤n,n=1,2,….
For the sake of brevity, we denote fn>0,
and Xk stands for Xk(ω).
Let(3.2)pn(ω)=−(1/n)ln⁡fn(X1,…,Xn),where ω is the sample point, pn(ω) is called the sample entropy or the entropy
density of {Xk,1≤k≤n}.
Also let Q be another probability measure on ℱ with the density function(3.3)qn(x1,…,xn)>0,xk∈(−∞,∞),1≤k≤n,n=1,2,….Let(3.4)Ln(ω)=ln⁡[fn(X1,…,Xn)/qn(X1,…,Xn)],L(ω)=limsup⁡n→∞(1/n)Ln(ω)=−liminf⁡n→∞(1/n)ln⁡[qn(X1,…,Xn)/fn(X1,…,Xn)],D(fn∥qn)=EPLn=EPln⁡[fn(X1,…,Xn)/qn(X1,…,Xn)].Ln(ω), L(ω),
and D(fn∥qn) are called the sample relative entropy, the
sample relative entropy rate, and the relative entropy, respectively, relative
to the reference density function qn(x1,…,xn).
Indeed, they all are the measure of the deviation between the true joint
distribution density function fn(x1,…,xn) and the reference distribution density
function qn(x1,…,xn) (cf. [10, pages 12, 18]).A question of importance in information theory is the
study of the limit properties of the relative entropy densityfn(ω).
Since Shannon's initial work was published (cf. [11]), there has been a great
deal of investigation about this question (e.g., cf. [12–20]).In this paper, a class of small deviation theorems
(i.e., the strong limit theorems represented by inequalities) is established by
using the analytical technique, and an extension of the Shannon-McMillan
theorem to the arbitrary-dependent continuous information sources is given.
Especially, an approach of applying the tool of Laplace transform to the study
of the strong deviation theorems on the differential entropy is proposed.Lethk(xk)=−lngk(xk) (1≤k≤n,n=1,2,…) in (1.5), then we give the following
definitions.Definition 3.1.
The Laplace transform of−lngk(xk) is defined by (3.5)fk(s)=Ee−s(−ln⁡gk(Xk))=∫−∞∞e−s(−ln⁡gk(xk))gk(xk)dxk.Definition 3.2.
The differential entropy for continuous random
variablesXk is defined by (3.6)h(Xk)=E[−ln⁡gk(Xk)]=−∫−∞∞gk(xk)ln⁡gk(xk)dxk.In the following theorem, let{Xn,n≥1} be independent random variables with respect
to Q,
then the reference density function qn(x1,…,xn)=∏k=1ngk(xk),
and let hk(Xk)=−lngk(Xk) (1≤k≤n)  in Theorem 2.1.Theorem 3.3.
Let{Xn,n≥1}, Ln(ω), L(ω), fn(s) be given as above, and under the assumptions
of (1) and (2), let(3.7)limsup⁡n→∞1n∑k=1nEQes0(−ln⁡gk(Xk))=limsup⁡n→∞1n∑k=1nfk(−s0)=M(s0)<∞.Then(3.8)liminf⁡n→∞1n∑k=1n[−ln⁡gk(Xk)−h(Xk)]≥−β(L(ω)),a.s.limsup⁡n→∞1n∑k=1n[−ln⁡gk(Xk)−h(Xk)]≤β(L(ω)),a.s.where(3.9)β(x)=inf⁡{ϕ(s,x),−s0<s<0},x≥0,(3.10)ϕ(s,x)=−2e−2sM(s0)(s0−|s|)2−xs,x≥0,(3.11)0≤β(x)≤x(1+x)s0[2e−2M(s0)+1],lim⁡x→0+β(x)=β(0)=0.Remark 3.4.
Let(3.12)α(x)=sup⁡{ϕ(s,x),0<s<s0},x≥0,then(3.13)α(x)=−β(x).Corollary 3.5.
Letpn(ω) be defined by (3.2). Under the condition of
Theorem 3.3, then(3.14)liminf⁡n→∞[pn(ω)−(1/n)h(X1,…,Xn)]≥α(L(ω))−L(ω)+H*a.s.limsup⁡n→∞[pn(ω)−(1/n)h(X1,…,Xn)]≤β(L(ω))+H*a.s.where h(X1,…,Xn)=E[−lnfn(X1,…,Xn)] is the differential entropy for (X1,…,Xn),
and(3.15)H*=liminf⁡n→∞(1/n)[∑k=1nh(Xk)−h(X1,…,Xn)]=liminf⁡n→∞(1/n)D(fn∥qn),H*=limsup⁡n→∞(1/n)[∑k=1nh(Xk)−h(X1,…,Xn)]=limsup⁡n→∞(1/n)D(fn∥qn),where α(L(ω)), β(L(ω)) are denoted by (3.9)–(3.13).Corollary 3.6.
IfP=Q,
or {Xn,n≥1} are independent random variables, and there
exists s0>0,
such that (2.1) holds, then(3.16)lim⁡n→∞[pn(ω)−(1/n)h(X1,…,Xn)]=0a.s.



---

*Source: 639145-2008-05-11.xml*