# Generalized Cumulative Residual Entropy for  Distributions with Unrestricted Supports

Authors: Noomane Drissi; Thierry Chonavel; Jean Marc Boucher
Journal: Research Letters in Signal Processing
 (2008)

---

## Abstract

We consider the cumulative residual entropy (CRE) a recently introduced
measure of entropy. While in previous works distributions with positive support
are considered, we generalize the definition of CRE to the case of distributions
with general support. We show that several interesting properties
of the earlier CRE remain valid and supply further properties and insight to
problems such as maximum CRE power moment problems. In addition, we
show that this generalized CRE can be used as an alternative to differential
entropy to derive information-based optimization criteria for system identification
purpose.


---

## Body

## 1. Introduction

The concept of entropy is important for  studies in
many areas of engineering such as thermodynamics, mechanics, or digital
communications. An early definition of a measure of the entropy is the Shannon
entropy [1, 2]. In Shannon's approach, discrete values and
absolutely continuous distributions are treated in a somewhat different way
through entropy and differential entropy, respectively. Considering the
complementary cumulative distribution function (CCDF) instead of the
probability density function in the definition of differential entropy leads to
a new entropy measure named cumulative residual entropy (CRE) [3, 4]. In [3, 4], CRE is defined as(1)ℰ(X)=−∫ℝ+nP(|X|>u)logP(|X|>u)du,where n is the dimension of the random vector X.
Clearly, this formula is valid both for a discrete or an absolutely continuous
random variable (RV), or with both a discrete and an absolutely continuous
part, because it resorts to the CCDF of |X|.
In addition, unlike Shannon differential entropy it is always positive, while
preserving many interesting properties of Shannon entropy. The concept of CRE has found nice interpretations and
applications in the fields of reliability (see [5] where the concept of dynamic CRE is introduced) and images
alignment [3].Shannon entropy can be seen as a particular case of
exponential entropy, when entropy order tends to 1. Thus, following the work in
[4], a modified version of the exponential entropy, where
PDF is replaced by CCDF, has been introduced in [6], leading to new entropy-type measures, called
survival entropies.However, both Rao et al.'s
CRE and its exponential entropy generalization by Zografos and Nadarajah lead
to entropy-type
definitions that assume either positive valued RVs or apply to|X| otherwise. Although  the positive case is of
great interest for many applications, CRE and exponential entropies entail
difficulties when working with RVs with supports that are not restricted to
positive values.In this paper, we show that for an RVX, (1) remains a valid expression when P(|X|>u) is replaced by P(X>u) and integration is performed over ℝn,
without further hypothesis than in [4]. In addition, some desirable properties are enabled
by this CRE definition extension. We also complete the power moment constrained
maximum CRE distributions problem that was adressed in [7], for
classes of distributions that have
lower-unbounded supports. Finally, we illustrate the potential superiority of
the proposed generalized CRE (GCRE) against differential entropy in mutual
information-based estimation problems.The paper is organized as follows. Section2
introduces the GCRE definition. Some properties of GCRE are discussed in
Section 3. In Section 4, we introduce cumulative entropy rate and mutual
information rate. Section 5 deals with maximum GCRE distributions. With a view
to illustrate the potentiality of GCRE, in Section 6, we show on a simple
example a possible benefit of GCRE for systems identification.


## 2. Generalized Cumulative Residual Entropy (GCRE)

We will denote
byFXc(x) the complementary cumulative distribution
function (survival function) of a multivariate RV X=[X1,…,Xn]T of dimension n: FXc(X)=P(X>x)=P(Xi>xi,i=1,…,n).
We denote by HC(X) the GCRE of X that we define by(2)HC(X)=−∫ℝnFXc(u)logFXc(u)du.Clearly, like
the CRE, the GCRE is a positive and concave function of FXc.
In addition, existence of GCRE can be established without further assumption
upon distribution than those assumed for the CRE in [4].Theorem 1.
HC(X)<∞ if for some p>n, E[|X|p]<∞.Proof.
First let us remark that from the proof of the existence of CRE in
[4], it is sufficient to prove the result when X is a scalar RV, that is n=1,
and for p>1.
Then, letting p−1<α<1, we use the following inequality:
(3)−FXc(x)logFXc(x)≤e−11−α[FXc(x)]α1I]0,∞[(x)+(1−FXc(x))1I]−∞,0](x),where 1IA(x)=1 if x∈A and 1IA(x)=0 otherwise. The existence of ∫ℝ(e−1/(1−α))[FXc(x)]α1I[0,∞[(x)dx can be proven just as in [4]. Now, letting u=−x,
we have
(4)(1−FXc(x))1I]−∞,0](x)=FX(x)1I]−∞,0](x)≤1I[−1,0[(t)+FX(x)1I]−∞,−1[(x)≤1I]0,1](u)+FX(−u)1I]1,∞](u)≤1I]0,1](u)+F−X(u)1I]1,∞](u)≤1I]0,1](u)+F|−X|c(u)1I]1,∞](u)≤1I]0,1](u)+u−pE[|X|p]1I]1,∞](u).Thus,
(5)∫ℝ(1−FXc(x))1I]−∞,0](x)dx≤∫ℝ(1I]0,1](u)+u−pE[|X|p]1I]1,∞](u))du≤1+∫1∞u−pE[|X|p]du<∞. Finally,
putting all pieces together one finally proves convergence of right-hand side
of (2).


## 3. A Few Properties of GCRE

Let us now
exhibit a few more interesting properties of the GCRE. First, it is easy to
check that like Shannon entropy the GCRE remains constant with respect to
variable translation:(6)∀a∈ℝn,HC(X+a)=HC(X). In the same
way, it is clear that(7)∀a∈ℝ+,HC(aX)=aHC(X). When a<0,
we do not have such a nice property. However, let us consider the important
particular case where the distribution of X has a symmetry of the form(8)∃μ,∀x,FXc(μ−x)=1−FXc(μ+x).In this case,
we get the following result.Theorem 2.
For  an RVX that satisfies symmetry property (8), one has(9)∀a∈ℝ,HC(aX)=|a|HC(X).Proof.
Since
it is  clear that for alla∈ℝ+, HC(aX)=aHC(X),
we just have to check that HC(−X)=HC(X),
which can be established as follows:(10)−HC(−X)=∫ℝF−Xc(x)logF−Xc(x)dx=∫ℝF−Xc(x−μ)logF−Xc(x−μ)dx=∫ℝFX(−x+μ)logFX(−x+μ)dx=∫ℝ(1−FX(x+μ))log(1−FX(x+μ))dx=∫ℝFXc(x+μ)logFXc(x+μ)dx=∫ℝFXc(x)logFXc(x)dx=−HC(X).When the entries of vectorX are independent, it has been shown in [4] that if
the Xi are nonnegative, then(11)HC(X)=∑i(Πj≠i𝔼[Xj])HC(Xi).However, this
formula does not extend to RVs with distributions carried by ℝn because FC(X) can be integrated over ℝ+n in general but never over ℝn.
However, if the Xis are independent and have lower bounded
supports with respective lower bounds m1,…,mn,(12)HC(X)=∫Πi]mi,∞]−FXc(x)logFXc(x)dx=∑i(Πj≠i(𝔼[Xj]−mj))HC(Xi),because(13)∫mi∞FXic(u)du=[uFXic(u)]mi∞+∫mi∞uPXi(du)=−mi+𝔼[Xi].Conditional GCRE definition is a direct extension of
the definition of conditional CRE: the conditional GCRE ofX knowing that Y is equal to y is defined by(14)HC(X|Y=y)=∫ℝn−P(X>x|Y=y)logP(X>x|Y=y)dx.We recall an
important result from [4] that states that conditioning reduces
the entropy.Theorem 3.
For anyX and Y,(15)HC(X|Y)≤HC(X)equality holds if and only if X is independent of Y.As a consequence, ifX→Y→Z is a Markov chain, we have the data processing
inequality for GCRE:(16)HC(Z|X,Y)≤HC(Z|X).


## 4. Entropy and Mutual Information Rates

### 4.1. Entropy Rate

The GCRE of a
stochastic process{Xi} is defined by(17)HC(X)=limn→∞HC(Xn|Xn−1,Xn−2,…,X1)when the limit
exists.Theorem 4.
For stationary processes, the
limit exists.Proof.
Consider(18)HC(Xn+1|Xn,…,X1)≤HC(Xn+1|Xn,…,X2)≤HC(Xn|Xn−1,…,X1).The first line follows from the
fact that conditioning reduces entropy and the second follows from the
stationarity (see [2] for the equivalent proof in the case of Shannon
entropy).


### 4.2. Mutual Information

LetX and Y be two RVs. We define the cumulative mutual
information between X and Y as follows:(19)IC(X;Y)=HC(X)−HC(X|Y).Theorem 5.
IC is nonnegative and it vanishes if and only if X and Y are  independent.Proof.
It is
clear thatIC is nonnegative because of Theorem 3.For a random vectorX=(X1,X2,…,Xn) of size n,
mutual information is defined by(20)IC(X)=∑i=1nHC(Xi)−HC(Xn|Xn−1,…,X1).In the case of stochastic
processes {Xi},
we have HC(X)=limn→∞HC(Xn|Xn−1,…,X1) and the limit exists for stationary processes.
Then the mutual information rate for {Xi} is defined as(21)IC(X)=limT→∞1T∑t=1THC(Xt)−HC(X),where HC(Xt) is the marginal GCRE of the process X.


## 4.1. Entropy Rate

The GCRE of a
stochastic process{Xi} is defined by(17)HC(X)=limn→∞HC(Xn|Xn−1,Xn−2,…,X1)when the limit
exists.Theorem 4.
For stationary processes, the
limit exists.Proof.
Consider(18)HC(Xn+1|Xn,…,X1)≤HC(Xn+1|Xn,…,X2)≤HC(Xn|Xn−1,…,X1).The first line follows from the
fact that conditioning reduces entropy and the second follows from the
stationarity (see [2] for the equivalent proof in the case of Shannon
entropy).


## 4.2. Mutual Information

LetX and Y be two RVs. We define the cumulative mutual
information between X and Y as follows:(19)IC(X;Y)=HC(X)−HC(X|Y).Theorem 5.
IC is nonnegative and it vanishes if and only if X and Y are  independent.Proof.
It is
clear thatIC is nonnegative because of Theorem 3.For a random vectorX=(X1,X2,…,Xn) of size n,
mutual information is defined by(20)IC(X)=∑i=1nHC(Xi)−HC(Xn|Xn−1,…,X1).In the case of stochastic
processes {Xi},
we have HC(X)=limn→∞HC(Xn|Xn−1,…,X1) and the limit exists for stationary processes.
Then the mutual information rate for {Xi} is defined as(21)IC(X)=limT→∞1T∑t=1THC(Xt)−HC(X),where HC(Xt) is the marginal GCRE of the process X.


## 5. Maximum GCRE Distributions

In this
section, we only consider the case of one-dimensional RVs (n = 1). Maximum entropy principle is useful in
many scientific areas and most important distributions can be derived from it [8].
The maximum CRE distribution has been studied in [7]. For an RV X with a symmetric CCDF in the sense of (8), we are
looking for the maximum GCRE distribution, that is, the CCDF that solves the
following moment problem:(22)maxFcHC(Fc)∫ℝri(x)p(x)dx=ci,i=1,…,m,where p(x)=−(d/dx)Fc(x), (ri)i=1,m,
and (ci)i=1,m are fixed C1 real valued functions and real coefficients,
respectively. The solution of this problem is supplied by the following result.Theorem 6.
When the symmetry property (8) holds, the
solution of problem (22), when it can be reached, is of the form(23)Fc(x)=11+exp[∑1mλiri′(x−μ)](x≥μ).Proof.
(24)−HC(X)=∫−∞μFc(x)log(Fc(x))dx+∫μ∞Fc(x)log(Fc(x))dx=∫ℝ−Fc(x+μ)log(Fc(x+μ))dx+∫ℝ+Fc(x+μ)log(Fc(x+μ))dx=∫ℝ−(1−Fc(μ−x))log(1−Fc(μ−x))dx+∫ℝ+Fc(x+μ)log(Fc(x+μ))dx=∫ℝ+[(1−Fc(μ+x))log(1−Fc(μ+x))+Fc(μ+x)log(Fc(μ+x))]dx. Let us define f˜ by(25)f˜(Fc(x),−p)=−(1−Fc(μ+x))log(1−Fc(μ+x))−Fc(μ+x)log(Fc(μ+x))+∑1mλi(−p(x))ri(x). Then, since (Fc(x))′=−p(x),
the Euler-Lagrange equation [9] states that the solution Fc of problem (22) is a solution of equation(26)ddxf˜−p(x)=f˜Fc(x),where f˜u is the partial derivative of f˜ with respect to component u.
From (25), we get(27)ddxf˜−p(x)=∑1mλiri′(x),f˜Fc(x)=log(1−Fc(μ+x))−logFc(μ+x).Then,(28)log1−Fc(μ+x)Fc(μ+x)=∑1mλiri′(x),Fc(x)=11+exp(∑1mλiri′(x−μ)),for x∈[μ,∞[.


### 5.1. Example

We set the
constraints𝔼[X]=μ and 𝔼[X2]=σ2.
Then the maximum GCRE symmetric solution for the CCDF of X is given
by(29)FXc(x)=11+exp(λ1+2λ2x),for x>0,
which is the CCDF of a logistic distribution. The moment constraints lead to λ1+2λ2x=(σ3/π)(x−μ).
The  corresponding PDF is defined on ℝ by(30)pX(x)=σ3πexp(−(σ3/π)(x−μ))(1+exp(−(σ3/π)(x−μ)))2.


### 5.2. Positive Random Variables

It has been
shown in [7] that the maximum CRE (i.e., the maximum GCRE under
additional nonnegative constraint) distribution has CCDF in the
form(31)FC(x)=exp(−∑i=1mλiri′(x)),for x∈[0,∞[.
In [7], this result is derived from the log-sum inequality,
but of course it can also be derived from the Euler-Lagrange equation along the
same lines as in the proof of Theorem 6.With a positive support constraint and under first and
second moment constraints, it comes that the optimum CCDF is of the formFc(x)=exp(−λ1−2λ2x), for x>0.
Thus the solution, if it exists, is an exponential distribution. In fact, the
first and second power moment constraints must be such that 𝔼[X2]=2(𝔼[X])2,
otherwise the problem has no exact solution.


## 5.1. Example

We set the
constraints𝔼[X]=μ and 𝔼[X2]=σ2.
Then the maximum GCRE symmetric solution for the CCDF of X is given
by(29)FXc(x)=11+exp(λ1+2λ2x),for x>0,
which is the CCDF of a logistic distribution. The moment constraints lead to λ1+2λ2x=(σ3/π)(x−μ).
The  corresponding PDF is defined on ℝ by(30)pX(x)=σ3πexp(−(σ3/π)(x−μ))(1+exp(−(σ3/π)(x−μ)))2.


## 5.2. Positive Random Variables

It has been
shown in [7] that the maximum CRE (i.e., the maximum GCRE under
additional nonnegative constraint) distribution has CCDF in the
form(31)FC(x)=exp(−∑i=1mλiri′(x)),for x∈[0,∞[.
In [7], this result is derived from the log-sum inequality,
but of course it can also be derived from the Euler-Lagrange equation along the
same lines as in the proof of Theorem 6.With a positive support constraint and under first and
second moment constraints, it comes that the optimum CCDF is of the formFc(x)=exp(−λ1−2λ2x), for x>0.
Thus the solution, if it exists, is an exponential distribution. In fact, the
first and second power moment constraints must be such that 𝔼[X2]=2(𝔼[X])2,
otherwise the problem has no exact solution.


## 6. Simulation Results

With a view to
emphasize the potential practical interest of GCRE, we consider a simple system
identification problem. Here, we consider anMA(1) process, denoted by Y=(Yn)n∈ℤ,
generated by a white noise X=(Xn)n∈ℤ and corrupted by a white noise N:(32)Yn=Xn−aXn−1+Nn. The model input X and output Y are observed and the system model (MA(1)) is assumed to be known. We want to estimate
the coefficient a without prior knowledge upon the distributions
of X and Y.
Thus, we resort to mutual information (MI) to estimate a as the coefficient α such that RVs Y^nα=Xn−αXn−1 and Yn show the highest dependence. Shannon MI
between Y^nα and Yn is given by fS(α)=IS(Y^nα,Yn)=HS(Y^nα)−HS(Y^n|Yn),
where HS is Shannon differential entropy. Similarly,
for GCRE, MI will be defined as fC(α)=IC(Y^nα,Y)=HC(Y^nα)−HC(Y^nα|Yn).
We compare estimation performance for a by maximizing both fS(α) and fC(α).
Since true values of fS(α) and fC(α) are not available, they are estimated from
empirical distributions of (Yn,Y^nα).For simulations, we have chosenX Gaussian and N with a Laplace distribution: pN(x)=(λ/2)exp(−λ|x|).
We consider an experiment with a=0.5 and noise variance equal to 0.2. Estimation is
carried out from observation of (Xn,Yn)n=1,400.
Here, optimization of MIs is realized on a fixed regular grid of 200 points
over interval [0,1]. Estimation performance is calculated from 200 successive
experiments. Estimation of a from Shannon MI leads to bias and standard
deviation that are equal to 0.032 and 0.18, respectively, while they are equal
to 0.004 and 0.06, respectively, for GCRE MI.More important, we see on Figure1(a) that Shannon MI
estimates are much more irregular than GCRE MI (Figure 1(b)) estimates because
of smoothing brought by density integration in the calculation of CCDF. This
difference is important since the use of an iterative local optimization
technique would have failed in general to find Shannon's estimated MI global
optimum, because of its many local maxima.Ten estimates of (a)fS(α) and (b) fC(α).
Dotted line: mean estimate averaged from 200 realizations.
(a)(b)Of course, this drawback can be partly solved by kernel
smoothing of the empirical distribution of(Yn,Y^nα),
for instance by using the method proposed in [10]. However, we have checked that, for the above
example, very strong smoothing is necessary and then bias and variance
performance remain worse than with GCRE MI estimator.


## 7. Conclusion

We have shown
that the concept of cumulative residual entropy (CRE) introduced in [3, 4] can be extended
to distributions with general
supports. Generalized CRE (GCRE) shares many nice features of CRE. We also
pointed out specific properties of GCRE such as its maximum, moment
constrained, and distribution and we have illustrated practical interest of
GCRE by showing how it can be used in system identification procedures.



---

*Source: 790607-2008-07-15.xml*