# The Unifying Frameworks of Information Measures

Authors: Shiwei Yu; Ting-Zhu Huang
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

Information measures are capable of providing us with fundamental methodologies to analyze uncertainty and unveiling the substantive characteristics of random variables. In this paper, we address the issues of different types of entropies throughq-generalized Kolmogorov-Nagumo averages, which lead to the propositions of the survival Rényi entropy and survival Tsallis entropy. Therefore, we make an inventory of eight types of entropies and then classify them into two categories: the density entropy that is defined on density functions and survival entropy that is defined on survival functions. This study demonstrates that, for each type of the density entropy, there exists a kind of the survival entropy corresponding to it. Furthermore, the similarity measures and normalized similarity measures are, respectively, proposed for each type of entropies. Generally, functionals of different types of information-theoretic metrics are equally diverse, while, simultaneously, they also exhibit some unifying features in all their manifestations. We present the unifying frameworks for entropies, similarity measures, and normalized similarity measures, which helps us deal with the available information measures as a whole and move from one functional to another in harmony with various applications.


---

## Body

## 1. Introduction

Measures of probabilistic uncertainty and information have attracted growing attentions since Hartley introduced the practical measure of information as the logarithm of the amount of uncertainty associated with finite possible symbol sequences, where the distribution of events is considered to be equally probable [1]. Today, entropy plays a basic role in the definitions of information measures with various applications in different areas. It has been recognized as the fundamental important field intersecting with mathematics, communication, physics, computer science, economics, and so forth [2–5].The generalized information theory arising from the study of complex systems was intended to expand classical information theory based on probability. The additive probability measures, which are inherent in classical information theory, are extended to various types of nonadditive measures and thus result in different types of functionals that generalize Shannon entropy [6–8]. Generally, the formalization of uncertainty functions involves a considerable diversity. However, it also exhibits some unifying features [9].


### 1.1. Entropies Defined on Density Functions

We consider(X,Y) as the continuous random variables (r.v.) over a state space Ω×Ω with the joint density function f(x,y) and marginal density functions fX(x) and fY(y). We also consider the conditional density function fX∣Y(x∣y) of X given Y defined over Ω. Note that f(x), f(y), and f(x∣y) are also used to mean fX(x), fY(y), and fX∣Y(x∣y), respectively, if their meanings are clear in context.Letf(x) be a density function of r.v. X with ∫Rf(x)dx=1. The Khinchin axioms [10] are capable of obtaining the Shannon entropy in a unique way. However, this may be too narrow-minded if one wants to describe complex systems. Therefore, a generalized measure of an r.v. X with respect to Kolmogorov-Nagumo (KN) averages [11] can be deduced as (1)Xψ=ψ-1∫Rfxψlog⁡1fxdx, where ψ is a continuous and strictly monotonic KN function [12] and hence has an inverse ψ-1.The KN averages can be extended in different manners to propose more generalized information measures. We use theq-logarithm function [13] given as (2)lnqx=x1-q-11-q,x>0,q∈R to replace the logarithm function in (1). Note that log⁡x=limq→1lnqx and lnqx satisfies pseudoadditivity; for example, lnq(xy)=lnqx+lnqy+(1-q)lnqxlnqy. Hence we extend KN averages to a generalized measure of information X with respect to q-generalized KN averages defined by (3)Xψq=ψ-1∫Rfxψlnq1fxdx.In terms of Rényi’s generalization on axioms of KN averages [14], if q=1 and ψ(x)=x in (3), it yields Shannon entropy (SE) [15] defined as (4)HX=-∫Rfxlog⁡fxdx.Based on Shannon entropy, the Shannon mutual information (SMI) [15, 16] of r.v.s X and Y was given by (5)IX,Y=HX+HY-HX,Y=HX-HX∣Y, where H(X,Y) is the joint Shannon entropy of (X,Y) and H(X∣Y) is the conditional Shannon entropy of X given Y.Ifq=1 and ψ is chosen as ψ(x)=e(1-α)x in (3), it yields Rényi entropy (RE) [14] defined by (6)RαX=11-αlog⁡∫Rfαxdx, where α>0 and α≠1.Shannon entropy and Rényi entropy are additive. Ifq=α and ψ(x)=x in (3), we get the pseudoadditive entropy or Tsallis entropy (TE) [17] defined by (7)TαX=∫Rfxlnα1fxdx=1α-11-∫Rfαxdx, where α>0 and α≠1.We obtainH(X)=limα→1Rα(X) and H(X)=limα→1Tα(X). Therefore, Rényi entropy and Tsallis entropy can be viewed as interpolation formulas of the Shannon entropy (α=1) and Hartley entropy (α=0). A relation between Rényi and Tsallis entropies can be easily deduced as (8)TαX=1α-11-exp⁡1-αRαX.More recently, interest in generalized information measures increases dramatically in different manners. A respectable number of nonclassical entropies, rather than Shannon entropy, Rényi entropy, and Tsallis entropy, have already been developed in the study of complex systems.The exponential entropy (EE) of orderα [18] was defined by (9)EαX=∫Rfαxdx1/1-α, where α>0 and α≠1.We obtainRα(X)=log⁡Eα(X) and H(X)=limα→1log⁡Eα(X).


### 1.2. Entropies Defined on Survival Functions

As narrated in [19], information measures defined on the density function suffer from several drawbacks, since the distribution function is more regular than the density function. Therefore, the cumulative residual entropy, which was defined on the cumulative distribution function or equivalently the survival function, was proposed as an alternative information measure of uncertainty.LetX=(X1,X2,…,Xm) be a nonnegative r.v. in R+m. We use the notation X>x to mean that Xi>xi for xi≥0,i=1,2,…,m. The multivariate survival function of a nonnegative r.v. X is given as (10)F¯Xx=PX>x=PX1>x1,X2>x2,…,Xm>xm, where x=(x1,…,xm)∈R+m with R+m={x∈Rm:x=(x1,…,xm),xi≥0,i=1,…,m}.If the density function is replaced by the survival function,q is set as 1, and ψ(x)=x in (3), it yields the survival Shannon entropy (SSE) [19] defined as (11)H¯X=-∫R+mF¯Xxlog⁡F¯Xxdx.Since eight different types of entropies and their corresponding similarity measures will be discussed subsequently, it is worth pointing out that some notations and names of the existing information measures will be changed in harmony with the unifying frameworks throughout this paper.To consider the conditional survival entropy, we denoteFY∣X(x∣y) as the conditional distribution function of X given Y=y and also F¯X∣Y(x∣y) as the respective conditional survival function.The cross survival Shannon entropy (CSSE) of r.v.s(X,Y) was given by [19] (12)I¯X,Y=H¯X-H¯X∣Y, where H¯(X∣Y) is the conditional survival Shannon entropy of r.v.s X given Y defined as [19] (13)H¯X∣Y=-EY∫R+mF¯X∣Yx∣ylog⁡F¯X∣Yx∣ydx and EY(·) here is the expectation with respect to an r.v. Y. The nonnegativity of CSSE was proven in [19] and thus CSSE was used as a similarity measure in image registration [20]. The generalized versions of SSE in dynamic systems were discussed in [21, 22].If the density function in (9) is replaced by the survival function, this yields the survival exponential entropy (SEE) [23] of an r.v. X∈R+m with order α given by (14)E¯αX=∫R+mF¯Xαxdx1/1-α, where α>0 and α≠1.As an ongoing research program, generalized information measure offers us a steadily growing inventory of distinct entropy theories. Diversity and unity are two significant features of these theories. The growing diversity of information measures makes it increasingly more realistic to find a certain information measure suitable for a given condition. The unity allows us to view all available information measures as a whole and to move from one measure to another as needed. To that end, motivated by the researching approaches on Shannon entropy, Shannon mutual information [2], SSE [19], and SEE [23], we attempt to study information-theoretic metrics in their manifestations. On one hand, we propose several new types of entropies and their similarity measures; on the other hand, for each type of the existing entropies, except for Shannon entropy, we give the definitions of similarity measures (see Tables 1 and 2). Finally, we deduce the unifying frameworks for information measures emerging from the study of complex systems based on probability.Table 1
Entropies, conditional entropies, and joint entropies defined on density functions and survival functions.
Name Density entropy Name Survival entropy Shannon entropy H ( X ) = - ∫ R f x log ⁡ f ( x ) d x Survival Shannon entropy H ¯ ( X ) = - ∫ R + m F ¯ X x log ⁡ F ¯ X ( x ) d x  H X ∣ Y = - E y ∫ R f x ∣ y log ⁡ f x ∣ y d x H ¯ X ∣ Y = - E y ∫ R + m F ¯ X ∣ Y x ∣ y log ⁡ F ¯ X ∣ Y ( x ∣ y ) d x  H ( X , Y ) = H ( X ) + H Y ∣ X H ¯ ( X , Y ) = H ¯ ( X ) + H ¯ ( Y ∣ X )   Rényi entropy R α ( X ) = 1 1 - α log ⁡ ∫ R f α ( x ) d x Survival Rényi entropy R ¯ α ( X ) = 1 1 - α log ⁡ ∫ R + m F ¯ X α ( x ) d x  R α X ∣ Y = 1 1 - α E y log ⁡ ∫ R f α x ∣ y d x R ¯ α X ∣ Y = 1 1 - α E y log ⁡ ∫ R + m F ¯ X ∣ Y α x ∣ y d x  R α ( X , Y ) = R α ( X ) + R α Y ∣ X R ¯ α ( X , Y ) = R ¯ α ( X ) + R ¯ α Y ∣ X   Tsallis entropy T α ( X ) = 1 α - 1 1 - ∫ R f α x d x Survival Tsallis entropy T ¯ α ( X ) = 1 α - 1 ∫ R + m F ¯ X x d x - ∫ R + m F ¯ X α x d x  T α X ∣ Y = 1 α - 1 E y 1 - ∫ R f α x ∣ y d x T ¯ α X ∣ Y = 1 α - 1 E y ( E X X ∣ Y = y - ∫ R + m F ¯ X ∣ Y α x ∣ y d x )  T α ( X , Y ) = T α ( X ) + T α ( Y ∣ X ) T ¯ α ( X , Y ) = T ¯ α ( X ) + T ¯ α Y ∣ X   Exponential entropy E α ( X ) = ∫ R f α x d x 1 / ( 1 - α ) Survival exponential entropy E ¯ α ( X ) = ∫ R + m F ¯ X α x d x 1 / ( 1 - α )  E α ( X ∣ Y ) = E y ∫ R f α x ∣ y d x 1 / ( 1 - α ) E ¯ α ( X ∣ Y ) = E y ( ∫ R + m F ¯ X ∣ Y α x ∣ y d x ) 1 / ( 1 - α )  E α ( X , Y ) = E α ( X ) + E α ( Y ∣ X ) E ¯ α ( X , Y ) = E ¯ α ( X ) + E ¯ α Y ∣ XTable 2
The similarity measures and normalized similarity measures defined on density functions and survival functions.
Name Density similarity measure Name Survival similarity measure Shannon mutual information I ( X , Y ) = H ( X ) - H ( X | Y ) Cross survival Shannon entropy I ¯ ( X , Y ) = H ¯ ( X ) - H ¯ ( X | Y )  N I ( X , Y ) = H ( X ) + H ( Y ) H ( X , Y ) N I ¯ ( X , Y ) = H ¯ ( X ) + H ¯ ( Y ) H ¯ ( X , Y )   Rényi mutual information I R α ( X , Y ) = R α ( X ) - R α ( X | Y ) Cross survival Rényi entropy I R ¯ α ( X , Y ) = R ¯ α ( X ) - R ¯ α ( X | Y )  N I R α ( X , Y ) = R α ( X ) + R α ( Y ) R α ( X , Y ) N I R ¯ α ( X , Y ) = R ¯ α ( X ) + R ¯ α ( Y ) R ¯ α ( X , Y )   Tsallis mutual information I T α ( X , Y ) = T α ( X ) - T α ( X | Y ) Cross survival Tsallis entropy I T ¯ α ( X , Y ) = T ¯ α ( X ) - T ¯ α ( X | Y )  N I T α ( X , Y ) = T α ( X ) + T α ( Y ) T α ( X , Y ) I T ¯ α ( X , Y ) = T ¯ α ( X ) + T ¯ α ( Y ) T ¯ α ( X , Y )   Exponential mutual information I E α ( X , Y ) = E α ( X ) - E α ( X | Y ) Cross survival exponential entropy I E ¯ α ( X , Y ) = E ¯ α ( X ) - E ¯ α ( X | Y )  N I E α ( X , Y ) = E α ( X ) + E α ( Y ) E α ( X , Y ) NIE¯α(X,Y)=E¯α(X)+E¯α(Y)E¯α(X,Y)The remainder of this paper is organized as follows. Section2 will propose the similarity measures defined on the density function. In Section 3, the survival Rényi entropy and survival Tsallis entropy are presented. In Section 4, we address the similarity measures defined on the survival function. The unifying frameworks of information measures and examples are provided in Section 5. Finally, we conclude this paper in Section 6.


## 1.1. Entropies Defined on Density Functions

We consider(X,Y) as the continuous random variables (r.v.) over a state space Ω×Ω with the joint density function f(x,y) and marginal density functions fX(x) and fY(y). We also consider the conditional density function fX∣Y(x∣y) of X given Y defined over Ω. Note that f(x), f(y), and f(x∣y) are also used to mean fX(x), fY(y), and fX∣Y(x∣y), respectively, if their meanings are clear in context.Letf(x) be a density function of r.v. X with ∫Rf(x)dx=1. The Khinchin axioms [10] are capable of obtaining the Shannon entropy in a unique way. However, this may be too narrow-minded if one wants to describe complex systems. Therefore, a generalized measure of an r.v. X with respect to Kolmogorov-Nagumo (KN) averages [11] can be deduced as (1)Xψ=ψ-1∫Rfxψlog⁡1fxdx, where ψ is a continuous and strictly monotonic KN function [12] and hence has an inverse ψ-1.The KN averages can be extended in different manners to propose more generalized information measures. We use theq-logarithm function [13] given as (2)lnqx=x1-q-11-q,x>0,q∈R to replace the logarithm function in (1). Note that log⁡x=limq→1lnqx and lnqx satisfies pseudoadditivity; for example, lnq(xy)=lnqx+lnqy+(1-q)lnqxlnqy. Hence we extend KN averages to a generalized measure of information X with respect to q-generalized KN averages defined by (3)Xψq=ψ-1∫Rfxψlnq1fxdx.In terms of Rényi’s generalization on axioms of KN averages [14], if q=1 and ψ(x)=x in (3), it yields Shannon entropy (SE) [15] defined as (4)HX=-∫Rfxlog⁡fxdx.Based on Shannon entropy, the Shannon mutual information (SMI) [15, 16] of r.v.s X and Y was given by (5)IX,Y=HX+HY-HX,Y=HX-HX∣Y, where H(X,Y) is the joint Shannon entropy of (X,Y) and H(X∣Y) is the conditional Shannon entropy of X given Y.Ifq=1 and ψ is chosen as ψ(x)=e(1-α)x in (3), it yields Rényi entropy (RE) [14] defined by (6)RαX=11-αlog⁡∫Rfαxdx, where α>0 and α≠1.Shannon entropy and Rényi entropy are additive. Ifq=α and ψ(x)=x in (3), we get the pseudoadditive entropy or Tsallis entropy (TE) [17] defined by (7)TαX=∫Rfxlnα1fxdx=1α-11-∫Rfαxdx, where α>0 and α≠1.We obtainH(X)=limα→1Rα(X) and H(X)=limα→1Tα(X). Therefore, Rényi entropy and Tsallis entropy can be viewed as interpolation formulas of the Shannon entropy (α=1) and Hartley entropy (α=0). A relation between Rényi and Tsallis entropies can be easily deduced as (8)TαX=1α-11-exp⁡1-αRαX.More recently, interest in generalized information measures increases dramatically in different manners. A respectable number of nonclassical entropies, rather than Shannon entropy, Rényi entropy, and Tsallis entropy, have already been developed in the study of complex systems.The exponential entropy (EE) of orderα [18] was defined by (9)EαX=∫Rfαxdx1/1-α, where α>0 and α≠1.We obtainRα(X)=log⁡Eα(X) and H(X)=limα→1log⁡Eα(X).


## 1.2. Entropies Defined on Survival Functions

As narrated in [19], information measures defined on the density function suffer from several drawbacks, since the distribution function is more regular than the density function. Therefore, the cumulative residual entropy, which was defined on the cumulative distribution function or equivalently the survival function, was proposed as an alternative information measure of uncertainty.LetX=(X1,X2,…,Xm) be a nonnegative r.v. in R+m. We use the notation X>x to mean that Xi>xi for xi≥0,i=1,2,…,m. The multivariate survival function of a nonnegative r.v. X is given as (10)F¯Xx=PX>x=PX1>x1,X2>x2,…,Xm>xm, where x=(x1,…,xm)∈R+m with R+m={x∈Rm:x=(x1,…,xm),xi≥0,i=1,…,m}.If the density function is replaced by the survival function,q is set as 1, and ψ(x)=x in (3), it yields the survival Shannon entropy (SSE) [19] defined as (11)H¯X=-∫R+mF¯Xxlog⁡F¯Xxdx.Since eight different types of entropies and their corresponding similarity measures will be discussed subsequently, it is worth pointing out that some notations and names of the existing information measures will be changed in harmony with the unifying frameworks throughout this paper.To consider the conditional survival entropy, we denoteFY∣X(x∣y) as the conditional distribution function of X given Y=y and also F¯X∣Y(x∣y) as the respective conditional survival function.The cross survival Shannon entropy (CSSE) of r.v.s(X,Y) was given by [19] (12)I¯X,Y=H¯X-H¯X∣Y, where H¯(X∣Y) is the conditional survival Shannon entropy of r.v.s X given Y defined as [19] (13)H¯X∣Y=-EY∫R+mF¯X∣Yx∣ylog⁡F¯X∣Yx∣ydx and EY(·) here is the expectation with respect to an r.v. Y. The nonnegativity of CSSE was proven in [19] and thus CSSE was used as a similarity measure in image registration [20]. The generalized versions of SSE in dynamic systems were discussed in [21, 22].If the density function in (9) is replaced by the survival function, this yields the survival exponential entropy (SEE) [23] of an r.v. X∈R+m with order α given by (14)E¯αX=∫R+mF¯Xαxdx1/1-α, where α>0 and α≠1.As an ongoing research program, generalized information measure offers us a steadily growing inventory of distinct entropy theories. Diversity and unity are two significant features of these theories. The growing diversity of information measures makes it increasingly more realistic to find a certain information measure suitable for a given condition. The unity allows us to view all available information measures as a whole and to move from one measure to another as needed. To that end, motivated by the researching approaches on Shannon entropy, Shannon mutual information [2], SSE [19], and SEE [23], we attempt to study information-theoretic metrics in their manifestations. On one hand, we propose several new types of entropies and their similarity measures; on the other hand, for each type of the existing entropies, except for Shannon entropy, we give the definitions of similarity measures (see Tables 1 and 2). Finally, we deduce the unifying frameworks for information measures emerging from the study of complex systems based on probability.Table 1
Entropies, conditional entropies, and joint entropies defined on density functions and survival functions.
Name Density entropy Name Survival entropy Shannon entropy H ( X ) = - ∫ R f x log ⁡ f ( x ) d x Survival Shannon entropy H ¯ ( X ) = - ∫ R + m F ¯ X x log ⁡ F ¯ X ( x ) d x  H X ∣ Y = - E y ∫ R f x ∣ y log ⁡ f x ∣ y d x H ¯ X ∣ Y = - E y ∫ R + m F ¯ X ∣ Y x ∣ y log ⁡ F ¯ X ∣ Y ( x ∣ y ) d x  H ( X , Y ) = H ( X ) + H Y ∣ X H ¯ ( X , Y ) = H ¯ ( X ) + H ¯ ( Y ∣ X )   Rényi entropy R α ( X ) = 1 1 - α log ⁡ ∫ R f α ( x ) d x Survival Rényi entropy R ¯ α ( X ) = 1 1 - α log ⁡ ∫ R + m F ¯ X α ( x ) d x  R α X ∣ Y = 1 1 - α E y log ⁡ ∫ R f α x ∣ y d x R ¯ α X ∣ Y = 1 1 - α E y log ⁡ ∫ R + m F ¯ X ∣ Y α x ∣ y d x  R α ( X , Y ) = R α ( X ) + R α Y ∣ X R ¯ α ( X , Y ) = R ¯ α ( X ) + R ¯ α Y ∣ X   Tsallis entropy T α ( X ) = 1 α - 1 1 - ∫ R f α x d x Survival Tsallis entropy T ¯ α ( X ) = 1 α - 1 ∫ R + m F ¯ X x d x - ∫ R + m F ¯ X α x d x  T α X ∣ Y = 1 α - 1 E y 1 - ∫ R f α x ∣ y d x T ¯ α X ∣ Y = 1 α - 1 E y ( E X X ∣ Y = y - ∫ R + m F ¯ X ∣ Y α x ∣ y d x )  T α ( X , Y ) = T α ( X ) + T α ( Y ∣ X ) T ¯ α ( X , Y ) = T ¯ α ( X ) + T ¯ α Y ∣ X   Exponential entropy E α ( X ) = ∫ R f α x d x 1 / ( 1 - α ) Survival exponential entropy E ¯ α ( X ) = ∫ R + m F ¯ X α x d x 1 / ( 1 - α )  E α ( X ∣ Y ) = E y ∫ R f α x ∣ y d x 1 / ( 1 - α ) E ¯ α ( X ∣ Y ) = E y ( ∫ R + m F ¯ X ∣ Y α x ∣ y d x ) 1 / ( 1 - α )  E α ( X , Y ) = E α ( X ) + E α ( Y ∣ X ) E ¯ α ( X , Y ) = E ¯ α ( X ) + E ¯ α Y ∣ XTable 2
The similarity measures and normalized similarity measures defined on density functions and survival functions.
Name Density similarity measure Name Survival similarity measure Shannon mutual information I ( X , Y ) = H ( X ) - H ( X | Y ) Cross survival Shannon entropy I ¯ ( X , Y ) = H ¯ ( X ) - H ¯ ( X | Y )  N I ( X , Y ) = H ( X ) + H ( Y ) H ( X , Y ) N I ¯ ( X , Y ) = H ¯ ( X ) + H ¯ ( Y ) H ¯ ( X , Y )   Rényi mutual information I R α ( X , Y ) = R α ( X ) - R α ( X | Y ) Cross survival Rényi entropy I R ¯ α ( X , Y ) = R ¯ α ( X ) - R ¯ α ( X | Y )  N I R α ( X , Y ) = R α ( X ) + R α ( Y ) R α ( X , Y ) N I R ¯ α ( X , Y ) = R ¯ α ( X ) + R ¯ α ( Y ) R ¯ α ( X , Y )   Tsallis mutual information I T α ( X , Y ) = T α ( X ) - T α ( X | Y ) Cross survival Tsallis entropy I T ¯ α ( X , Y ) = T ¯ α ( X ) - T ¯ α ( X | Y )  N I T α ( X , Y ) = T α ( X ) + T α ( Y ) T α ( X , Y ) I T ¯ α ( X , Y ) = T ¯ α ( X ) + T ¯ α ( Y ) T ¯ α ( X , Y )   Exponential mutual information I E α ( X , Y ) = E α ( X ) - E α ( X | Y ) Cross survival exponential entropy I E ¯ α ( X , Y ) = E ¯ α ( X ) - E ¯ α ( X | Y )  N I E α ( X , Y ) = E α ( X ) + E α ( Y ) E α ( X , Y ) NIE¯α(X,Y)=E¯α(X)+E¯α(Y)E¯α(X,Y)The remainder of this paper is organized as follows. Section2 will propose the similarity measures defined on the density function. In Section 3, the survival Rényi entropy and survival Tsallis entropy are presented. In Section 4, we address the similarity measures defined on the survival function. The unifying frameworks of information measures and examples are provided in Section 5. Finally, we conclude this paper in Section 6.


## 2. Similarity Measures Defined on the Density Function

Shannon mutual information measures the information of an r.v.X conveying about another r.v. Y. It has been widely used in image registration [24, 25] and pattern recognition [26, 27]. Generally, as SMI is defined on Shannon entropy, each type of entropies would lead to corresponding similarity measures. In application, an idea similarity measure should be nonnegative. To that end, we take the way as [15, 19, 23] to define the similarity measures by linear expectation operator rather than KN average operator weighted by the escort distribution [28]. This section will present the similarity measures defined on the density function corresponding to Rényi entropy, Tsallis entropy, and the exponential entropy, respectively.


### 2.1. Rényi Mutual Information

Lemma 1.
LetX and Y be r.v.s and let φ(x) be a real convex function. Then (15)EY∫Rφfx∣ydx≥∫Rφfxdx.
Ifφ(x) is strictly convex, the equality holds if and only ifX andY are independent. If φ(x) is concave, the inequality is reversed.Proof.
For a real convex functionφ(x), using Jensen’s inequality [29], we obtain (16)EYφfx∣y≥φEYfx∣y.
The equality holds ifX andY are independent. Since f(x)=∫Rf(x∣y)f(y)dy, it is immediate that (17)EY∫Rφfx∣ydx=∫R∫Rφfx∣ydxfydy=∫R∫Rφfx∣yfydydx=∫REYφfx∣ydx≥∫RφEYfx∣ydx=∫Rφ∫Rfx∣yfydydx=∫Rφfxdx.
Now consider the "only if" part of the lemma. IfX andY are independent, then f(x∣y)=f(x), and thus the equality holds in (15). On the other hand, if the equality holds in (15), then the equality holds in Jensen’s inequality (16), which leads to EY(φ(f(x∣y)))=φ(EY(f(x∣y))). Then EY(f(x∣y))=f(x∣y) almost surely, since φ is strictly convex. We obtain EY(f(x∣y))=f(x) by (18); hence f(x∣y)=f(x), which leads to the independence ofX andY.
Lemma1 plays an important role to prove the nonnegativity for the similarity measure to be introduced, which is defined on the density function.Definition 2.
LetX andY be r.v.s; the conditional Rényi entropy of X given Y with order α is defined by (18)RαX∣Y=11-αEylog⁡∫Rfαx∣ydx, where α>0 and α≠1.
Motivated by the definitions of the joint Shannon entropyH(X,Y)=H(X)+H(Y∣X) and joint survival Shannon entropy H¯(X,Y)=H¯(X)+H¯(Y∣X), the joint Rényi entropy can be similarly introduced.Definition 3.
The joint Rényi entropy of r.v.sX andY with order α is defined as Rα(X,Y)=Rα(X)+Rα(Y∣X), where α>0 and α≠1.Theorem 4.
For r.v.sX andY, we obtain Rα(X)≥Rα(X∣Y) and Rα(X,Y)≤Rα(X)+Rα(Y) for all α>0 and α≠1.Proof.
Since1/(1-α)>0 and tα is concave of t>0 for all 0<α<1, using Lemma 1 and Jensen’s inequality, we have (19)RαX=11-αlog⁡∫Rfαxdx≥11-αlog⁡EY∫Rfαx∣ydx≥11-αEYlog⁡∫Rfαx∣ydx=RαX∣Y.
Since1/(1-α)<0 and tα is convex of t>0 for all α>1, using Lemma 1 and Jensen’s inequality, we obtain (20)RαX≥11-αlog⁡EY∫Rfαx∣ydx≥11-αEYlog⁡∫Rfαx∣ydx=RαX∣Y.
Similarly, we haveRα(Y)≥Rα(Y∣X), and thus Rα(X,Y)=Rα(X)+Rα(Y∣X)≤Rα(X)+Rα(Y).Definition 5.
The Rényi mutual information (RMI) of r.v.sX and Y with order α is defined as (21)IRαX,Y=RαX+RαY-RαY,X=RαX-RαX∣Y, where α>0 and α≠1.
It is worth pointing out that the definition of RMI parallels with the definitions of SMI (5) and CSSE (13). The nonnegativity of RMI is ensured by Theorem 4. Considering Theorem 4 that parallels with (5), we can give another form of the definition for RMI as (22)IRα∗X,Y=RαX+RαY-RαX,Y=RαY-RαY∣X.
There are no essential differences between these two forms of definitions for RMI. We only consider the similar definitions as (24) for similarity measures throughout this paper.
Using L’Hôpital’s rules, it is easy to obtainI(X,Y)=limα→1IRα(X,Y) and H(X,Y)=limα→1Rα(X,Y).
The normalized Shannon mutual information (NSMI) [16] of r.v.s X and Y was given as (23)NIX,Y=HX+HYHX,Y.
NSMI often acts as a robust similarity measure in image registration [16, 30], attribute abstraction [31], and clustering [32]. Note that 1≤NI(X,Y)≤2. In a similar way, different forms of the normalized mutual information will be deduced in this work.Definition 6.
The normalized Rényi mutual information (NRMI) of r.v.sX and Y with order α is defined by (24)NIRαX,Y=RαX+RαYRαX,Y, where α>0 and α≠1.
We immediately obtainNI(X,Y)=limα→1NIRα(X,Y) by L’Hôpital’s rules.


### 2.2. Tsallis Mutual Information

Definition 7.
The conditional Tsallis entropy of r.v.X given Y with order α is defined as (25)TαX∣Y=1α-1Ey1-∫Rfαx∣ydx, where α>0 and α≠1.Definition 8.
The joint Tsallis entropy of r.v.sX and Y with order α is defined as Tα(X,Y)=Tα(X)+Tα(Y∣X), where α>0 and α≠1.Theorem 9.
For two r.v.sX andY, we have Tα(X)≥Tα(X∣Y) and Tα(X,Y)≤Tα(X)+Tα(Y) for all α>0 and α≠1.Proof.
Since1/(α-1)<0 and tα is concave of t for all 0<α<1, using Lemma 1, we obtain (26)TαX-EYTαX∣Y=1-∫Rfαxdxα-1-EY1-∫Rfαx∣ydxα-1=1α-1EY∫Rfαx∣ydx-∫Rfαxdx≥1α-1∫Rfαxdx-∫Rfαxdx=0.
The inequality holds for allα>1, since 1/(α-1)>0 and tα is convex of t.
It is trivial to verify thatTα(X,Y)≤Tα(X)+Tα(Y).Definition 10.
The Tsallis mutual information (TMI) of r.v.sX and Y with order α is defined as ITα(X,Y)=Tα(X)-Tα(X∣Y), where α>0 and α≠1.
Using Theorem9, we have ITα(X,Y)≥0.Definition 11.
The normalized Tsallis mutual information (NTMI) of r.v.sX and Y with order α is defined by (27)NITαX,Y=TαX+TαYTαX,Y, where α>0 and α≠1.
It is easy to verify that(28)HX=limα→1⁡TαX,HX,Y=limα→1⁡TαX,Y,IX,Y=limα→1⁡ITαX,Y,NIX,Y=limα→1⁡NITαX,Y,by L’Hôpital’s rules, and Tα(X,X)=Tα(X),NITα(X,Y)≥1.


### 2.3. Exponential Mutual Information

Definition 12.
The conditional exponential entropy of r.v.X given Y with order α is defined by (29)EαX∣Y=Ey∫Rfαx∣ydx1/1-α, where α>0 and α≠1.Definition 13.
The joint exponential entropy of r.v.sX and Y with order α is defined as Eα(X,Y)=Eα(X)+Eα(Y∣X), where α>0 and α≠1.Theorem 14.
For two r.v.sX andY,(30)EY∫Rfαx∣ydx1/1-α≤∫Rfαxdx1/1-α for all α>0 and α≠1.Proof.
Sincetα is concave of t>0 for all 0<α<1, using Lemma 1, we obtain EY(∫Rfα(x∣y)dx)≤∫Rfα(x)dx. The equality is true, since 1/(1-α)>1 for all 0<α<1.
Similarly, sincetα is convex of t>0 for all α>1, using Lemma 1, we obtain EY(∫Rfα(x∣y)dx)≥∫Rfα(x)dx. We complete the proof by considering that t1/(1-α) is decreasing in t>0 for all α>1.Theorem 15.
For two r.v.sX andY, we obtain Eα(X)≥Eα(X∣Y) and Eα(X,Y)≤Eα(X)+Eα(Y) for all α>0 and α≠1.Proof.
Sincet1/(1-α) is convex of t>0 for all 0<α<1, using (30) and Jensen’s inequality, we obtain (31)EαX=∫Rfαxdx1/1-α≥EY∫Rfαx∣ydx1/1-α≥EY∫Rfαx∣ydx1/1-α=EαX∣Y.
Sincet1/(1-α) is concave and is decreasing in t>0 for all α>1, similarly we obtain (32)EαX≥EY∫Rfαx∣ydx1/1-α≥EY∫Rfαx∣ydx1/1-α=EαX∣Y.
It is trivial to verify thatEα(X,Y)≤Eα(X)+Eα(Y).Definition 16.
The exponential mutual information (EMI) of r.v.sX andY with order α is defined as IEα(X,Y)=Eα(X)-Eα(X∣Y), where α>0 and α≠1.
Using Theorem15, we have IEα(X,Y)≥0.Definition 17.
The normalized exponential mutual information (NEMI) of r.v.sX andY with order α is defined by (33)NIEαX,Y=EαX+EαYEαX,Y,where α>0 and α≠1.


## 2.1. Rényi Mutual Information

Lemma 1.
LetX and Y be r.v.s and let φ(x) be a real convex function. Then (15)EY∫Rφfx∣ydx≥∫Rφfxdx.
Ifφ(x) is strictly convex, the equality holds if and only ifX andY are independent. If φ(x) is concave, the inequality is reversed.Proof.
For a real convex functionφ(x), using Jensen’s inequality [29], we obtain (16)EYφfx∣y≥φEYfx∣y.
The equality holds ifX andY are independent. Since f(x)=∫Rf(x∣y)f(y)dy, it is immediate that (17)EY∫Rφfx∣ydx=∫R∫Rφfx∣ydxfydy=∫R∫Rφfx∣yfydydx=∫REYφfx∣ydx≥∫RφEYfx∣ydx=∫Rφ∫Rfx∣yfydydx=∫Rφfxdx.
Now consider the "only if" part of the lemma. IfX andY are independent, then f(x∣y)=f(x), and thus the equality holds in (15). On the other hand, if the equality holds in (15), then the equality holds in Jensen’s inequality (16), which leads to EY(φ(f(x∣y)))=φ(EY(f(x∣y))). Then EY(f(x∣y))=f(x∣y) almost surely, since φ is strictly convex. We obtain EY(f(x∣y))=f(x) by (18); hence f(x∣y)=f(x), which leads to the independence ofX andY.
Lemma1 plays an important role to prove the nonnegativity for the similarity measure to be introduced, which is defined on the density function.Definition 2.
LetX andY be r.v.s; the conditional Rényi entropy of X given Y with order α is defined by (18)RαX∣Y=11-αEylog⁡∫Rfαx∣ydx, where α>0 and α≠1.
Motivated by the definitions of the joint Shannon entropyH(X,Y)=H(X)+H(Y∣X) and joint survival Shannon entropy H¯(X,Y)=H¯(X)+H¯(Y∣X), the joint Rényi entropy can be similarly introduced.Definition 3.
The joint Rényi entropy of r.v.sX andY with order α is defined as Rα(X,Y)=Rα(X)+Rα(Y∣X), where α>0 and α≠1.Theorem 4.
For r.v.sX andY, we obtain Rα(X)≥Rα(X∣Y) and Rα(X,Y)≤Rα(X)+Rα(Y) for all α>0 and α≠1.Proof.
Since1/(1-α)>0 and tα is concave of t>0 for all 0<α<1, using Lemma 1 and Jensen’s inequality, we have (19)RαX=11-αlog⁡∫Rfαxdx≥11-αlog⁡EY∫Rfαx∣ydx≥11-αEYlog⁡∫Rfαx∣ydx=RαX∣Y.
Since1/(1-α)<0 and tα is convex of t>0 for all α>1, using Lemma 1 and Jensen’s inequality, we obtain (20)RαX≥11-αlog⁡EY∫Rfαx∣ydx≥11-αEYlog⁡∫Rfαx∣ydx=RαX∣Y.
Similarly, we haveRα(Y)≥Rα(Y∣X), and thus Rα(X,Y)=Rα(X)+Rα(Y∣X)≤Rα(X)+Rα(Y).Definition 5.
The Rényi mutual information (RMI) of r.v.sX and Y with order α is defined as (21)IRαX,Y=RαX+RαY-RαY,X=RαX-RαX∣Y, where α>0 and α≠1.
It is worth pointing out that the definition of RMI parallels with the definitions of SMI (5) and CSSE (13). The nonnegativity of RMI is ensured by Theorem 4. Considering Theorem 4 that parallels with (5), we can give another form of the definition for RMI as (22)IRα∗X,Y=RαX+RαY-RαX,Y=RαY-RαY∣X.
There are no essential differences between these two forms of definitions for RMI. We only consider the similar definitions as (24) for similarity measures throughout this paper.
Using L’Hôpital’s rules, it is easy to obtainI(X,Y)=limα→1IRα(X,Y) and H(X,Y)=limα→1Rα(X,Y).
The normalized Shannon mutual information (NSMI) [16] of r.v.s X and Y was given as (23)NIX,Y=HX+HYHX,Y.
NSMI often acts as a robust similarity measure in image registration [16, 30], attribute abstraction [31], and clustering [32]. Note that 1≤NI(X,Y)≤2. In a similar way, different forms of the normalized mutual information will be deduced in this work.Definition 6.
The normalized Rényi mutual information (NRMI) of r.v.sX and Y with order α is defined by (24)NIRαX,Y=RαX+RαYRαX,Y, where α>0 and α≠1.
We immediately obtainNI(X,Y)=limα→1NIRα(X,Y) by L’Hôpital’s rules.


## 2.2. Tsallis Mutual Information

Definition 7.
The conditional Tsallis entropy of r.v.X given Y with order α is defined as (25)TαX∣Y=1α-1Ey1-∫Rfαx∣ydx, where α>0 and α≠1.Definition 8.
The joint Tsallis entropy of r.v.sX and Y with order α is defined as Tα(X,Y)=Tα(X)+Tα(Y∣X), where α>0 and α≠1.Theorem 9.
For two r.v.sX andY, we have Tα(X)≥Tα(X∣Y) and Tα(X,Y)≤Tα(X)+Tα(Y) for all α>0 and α≠1.Proof.
Since1/(α-1)<0 and tα is concave of t for all 0<α<1, using Lemma 1, we obtain (26)TαX-EYTαX∣Y=1-∫Rfαxdxα-1-EY1-∫Rfαx∣ydxα-1=1α-1EY∫Rfαx∣ydx-∫Rfαxdx≥1α-1∫Rfαxdx-∫Rfαxdx=0.
The inequality holds for allα>1, since 1/(α-1)>0 and tα is convex of t.
It is trivial to verify thatTα(X,Y)≤Tα(X)+Tα(Y).Definition 10.
The Tsallis mutual information (TMI) of r.v.sX and Y with order α is defined as ITα(X,Y)=Tα(X)-Tα(X∣Y), where α>0 and α≠1.
Using Theorem9, we have ITα(X,Y)≥0.Definition 11.
The normalized Tsallis mutual information (NTMI) of r.v.sX and Y with order α is defined by (27)NITαX,Y=TαX+TαYTαX,Y, where α>0 and α≠1.
It is easy to verify that(28)HX=limα→1⁡TαX,HX,Y=limα→1⁡TαX,Y,IX,Y=limα→1⁡ITαX,Y,NIX,Y=limα→1⁡NITαX,Y,by L’Hôpital’s rules, and Tα(X,X)=Tα(X),NITα(X,Y)≥1.


## 2.3. Exponential Mutual Information

Definition 12.
The conditional exponential entropy of r.v.X given Y with order α is defined by (29)EαX∣Y=Ey∫Rfαx∣ydx1/1-α, where α>0 and α≠1.Definition 13.
The joint exponential entropy of r.v.sX and Y with order α is defined as Eα(X,Y)=Eα(X)+Eα(Y∣X), where α>0 and α≠1.Theorem 14.
For two r.v.sX andY,(30)EY∫Rfαx∣ydx1/1-α≤∫Rfαxdx1/1-α for all α>0 and α≠1.Proof.
Sincetα is concave of t>0 for all 0<α<1, using Lemma 1, we obtain EY(∫Rfα(x∣y)dx)≤∫Rfα(x)dx. The equality is true, since 1/(1-α)>1 for all 0<α<1.
Similarly, sincetα is convex of t>0 for all α>1, using Lemma 1, we obtain EY(∫Rfα(x∣y)dx)≥∫Rfα(x)dx. We complete the proof by considering that t1/(1-α) is decreasing in t>0 for all α>1.Theorem 15.
For two r.v.sX andY, we obtain Eα(X)≥Eα(X∣Y) and Eα(X,Y)≤Eα(X)+Eα(Y) for all α>0 and α≠1.Proof.
Sincet1/(1-α) is convex of t>0 for all 0<α<1, using (30) and Jensen’s inequality, we obtain (31)EαX=∫Rfαxdx1/1-α≥EY∫Rfαx∣ydx1/1-α≥EY∫Rfαx∣ydx1/1-α=EαX∣Y.
Sincet1/(1-α) is concave and is decreasing in t>0 for all α>1, similarly we obtain (32)EαX≥EY∫Rfαx∣ydx1/1-α≥EY∫Rfαx∣ydx1/1-α=EαX∣Y.
It is trivial to verify thatEα(X,Y)≤Eα(X)+Eα(Y).Definition 16.
The exponential mutual information (EMI) of r.v.sX andY with order α is defined as IEα(X,Y)=Eα(X)-Eα(X∣Y), where α>0 and α≠1.
Using Theorem15, we have IEα(X,Y)≥0.Definition 17.
The normalized exponential mutual information (NEMI) of r.v.sX andY with order α is defined by (33)NIEαX,Y=EαX+EαYEαX,Y,where α>0 and α≠1.


## 3. Entropies Defined on the Survival Function

The existing survival Shannon entropy and the survival exponential entropy extended the corresponding functionals from the density function to the survival function. In this section, we will propose the survival Rényi entropy and the survival Tsallis entropy defined on the survival function which, respectively, parallel with the classical Rényi entropy and Tsallis entropy defined on the density function.


### 3.1. Survival Rényi Entropy

If the density function is replaced by the survival function,q is set as 1, and ψ is chosen as ψ(x)=e(1-α)x in (3), it yields the survival Rényi entropy.Definition 18.
The survival Rényi entropy (SRE) of an r.v.X with order α is defined as (34)R¯αX=11-αlog⁡∫R+mF¯Xαxdx, where α>0 and α≠1.
It is immediate to see thatE¯αX=exp⁡(R¯α(X)).Definition 19.
The conditional survival entropy of r.v.X given Y with order α is defined as (35)R¯αX∣Y=11-αEylog⁡∫R+mF¯X∣Yαx∣ydx, where α>0 and α≠1.Definition 20.
The joint survival Rényi entropy of r.v.sX and Y with order α is defined as R¯α(X,Y)=R¯α(X)+R¯α(Y∣X), where α>0 and α≠1.Definition 21.
The joint survival exponential entropy of r.v.sX and Y with order α is defined as E¯α(X,Y)=E¯α(X)+E¯α(Y∣X), where α>0, α≠1, and E¯α(Y∣X)=exp(R¯α(Y∣X)) is the conditional survival exponential entropy of r.v. Y given X.
Theorem22 will show the relation between SRE and Shannon entropy.Theorem 22.
For an r.v.X, one has (36)R¯αX≥11-αHX-α,0<α<1,R¯αX≤11-αHX-α,α>1.Proof.
Using the log-sum inequality [2, 19], we obtain(37)∫R+mfxlog⁡fxF¯Xαxdx≥-log⁡∫R+mF¯Xαxdx and thus ∫R+mf(x)logf(x)dx-∫R+mf(x)logF¯Xα(x)dx≥-log∫R+mF¯Xα(x)dx. We obtain (38)HX+α∫R+mfxlog⁡F¯Xxdx≤log⁡∫R+mF¯Xαxdx.
SincedF¯X(x)/dx=-f(x), (38) can be written as (39)log⁡∫R+mF¯Xαxdx≥HX-α∫10log⁡udu=HX-α.
Note that1/(1-α)>0 for all 0<α<1 and 1/(1-α)<0 for all α>1. We complete the proof by multiplying 1/(1-α) on both sides of (39).Theorem 23.
LetX and Y be two r.v.s; then (40)R¯αX,Y≥11-αHX,Y-2α,0<α<1,(41)R¯αX,Y≤11-αHX,Y-2α,α>1.Proof.
If0<α<1, using Theorem 22, one has(42)R¯αX,Y=R¯αX+R¯αY∣X≥11-αHX+HY∣X-2α=11-αHX,Y-2α.
Similarly, the inequality in (41) holds for all α>1.
SinceR¯α(X)=log⁡E¯α(X), some properties of SRE can be similarly deduced by the proof approaches as SEE [23] and SSE [19]. We list these properties as propositions and neglect their proofs unless we can provide the improved, different, or more concise versions.Proposition 24.
IfEX(Xip)<∞, for some p>m/α, i=1,2,…,m, then R¯α(X)<∞ for all α>0 and α≠1.Proof.
For setsAi⊆R,i=1,2,…,m, using Hölder’s inequality [33], we have (43)PA1∩A2∩⋯∩Am≤∏i=1mPAi1/m.
For allα>0, we obtain (44)∫R+mF¯Xαxdx=∫R+mPX>xαdx=∫R+m∏i=1mPXi>xiαdx≤∫R+m∏i=1mPXi>xiα/mdx=∏i=1m∫0∞PXi>xiα/mdxi=∏i=1m∫01FXiα/mxidxi+∫1∞FXiα/mxidxi≤∏i=1m1+∫1∞1xipEXXipα/mdxi.
The inequality in (44) follows from Markov’s inequality [19, 34], where the integral on the right side exists if pα/m>1, that is, if p<m/α. Hence, ∫R+mF¯Xα(x)dx exists for all α>0.
(i) If∫R+mF¯Xα(x)dx≥1, note that log⁡t<t for all t≥1; then (45)R¯αX=11-αlog⁡∫R+mF¯Xαxdx<11-α∫R+mF¯Xαxdx<∞.
(ii) If0<∫R+mF¯Xα(x)dx<1, on one hand, for all α>1, since ∫R+mF¯Xα(x)dx exists for all α>0, we see 0<R¯α(X)<∞. On the other hand, for all 0<α<1, we obtain 1-α>0 and log⁡∫R+mF¯Xα(x)dx<0. We complete the proof, since -∞<R¯α(X)<0 for all 0<α<1.Proposition 25.
If the componentsXi,i=1,2,…,m, of r.v. X=(X1,X2,…,Xm) are independent, then R¯α(X)=∑i=1mR¯α(Xi) for all α>0 and α≠1.
Proposition25 is the immediate result of Theorem 15 in [23].
The Shannon entropy of a sum of independent variables is larger than that of either; for example,H(X+Y)≥max⁡{H(X),H(Y)}. SRE has this similar property.Proposition 26.
LetX andY be independent r.v.s; then R¯α(X+Y)≥max⁡R¯αX,R¯αY for all α>0 and α≠1.Proof.
SinceX and Y are independent, one has P(X+Y>u)=∫fY(v)P(X>u-v)dv. Since 1-α>0 and tα is concave of t>0 for all 0<α<1, using Jensen’s inequality, one has (46)PαX+Y>u≥∫fYvPαX>u-vdv.
Integrating both sides ofu from 0 to ∞, (47)∫PαX+Y>udu≥∫∫fYuPαX>u-vdvdu=∫fYvdv∫v∞PαX>u-vdu=∫PαX>xdx.
Then(48)log⁡∫R+mPαX+Y>udu≥log⁡∫R+mPαX>xdx.
Multiplying by1/(1-α) on both sides of the above leads to R¯α(X+Y)≥R¯α(X), and exchanging X and Y leads to R¯α(X+Y)≥R¯α(Y).
We complete the proof by using Jensen’s inequality and considering that1-α<0 and tα is convex of t>0 for all α>1 in a similar way.Proposition 27.
LetX=(X1,X2,…,Xm) and let Y=(Y1,Y2,…,Ym) be r.v.s with Yi=kiXi,ki≠0,i=1,2,…,m, for a constant vector k=(k1,k2,…,km)∈Rm; then R¯α(Y)=∑i=1mlogki+R¯α(X) for all 0<α<1.Proposition 28.
LetXn be a sequence ofm-dimensional nonnegative r.v.s converging in law to r.v.X. If all Xn are bounded in Lp for some p>m/α, then R¯α(X)=limn→∞R¯α(Xn) for all α>0 and α≠1.Proposition 29.
IfXi,i=1,2,…,m, of the r.v. X=(X1,X2,…,Xm) are independent, then R¯α(X)=∑i=1mR¯α(Xi) for all α>0 and α≠1.


### 3.2. Survival Tsallis Entropy

If the density function is replaced by the survival function,q=α, and ψ(x)=x in (3), or, equivalently, the density function is replaced by the survival function and logarithm function is replaced by q-logarithm function in (7), it yields the survival Tsallis entropy.Definition 30.
The survival Tsallis entropy (STE) of an r.v.X with order α is defined as (49)T¯αX=∫R+mF¯Xxlnα1F¯Xxdx=1α-1∫R+mF¯Xxdx-∫R+mF¯Xαxdx, where α>0 and α≠1.
Note thatF¯X(x)=∫x∞fX(t)dt. Using integration by parts formula, we obtain (50)∫R+mF¯Xxdx=xF¯Xx0∞-∫R+mxdF¯Xx=∫R+mxfXxdx=EXX.
Hence, the survival Tsallis entropy can also be written asT¯α(X)=(EX(X)-∫R+mF¯Xα(x)dx)/(α-1). It is easy to see that H¯(X)=limα→1T¯α(X) and (51)T¯αX=1α-1EXX-exp⁡1-αR¯αX=1α-1EXX-1-αlog⁡E¯αX.Definition 31.
The conditional survival Tsallis entropy of r.v.X given Y with order α is defined as (52)T¯αX∣Y=1α-1EyEXX∣Y=y-∫R+mF¯X∣Yαx∣ydx, where α>0 and α≠1.
Note thatEY(EX(X∣Y=y))=EY(∫R+mF¯X∣Yx∣ydx)=EX(X). Then T¯α(X∣Y)=1/α-1(EX(X)-Ey∫R+mF¯X∣Yα(x∣y)dx).Definition 32.
The joint survival Tsallis entropy of r.v.sX and Y with order α is defined as T¯α(X,Y)=T¯α(X)+T¯α(Y∣X), where α>0 and α≠1.
It is easy to see thatH¯(X)=limα→1T¯α(X) and H¯(X,Y)=limα→1T¯α(X,Y) using L’Hôpital’s rules.Theorem 33.
LetX be an r.v.; then (53)T¯αX≥1α-1EXX-eHX-α,0<α<1,T¯αX≤1α-1EXX-eHX-α,α>1.Proof.
It is easy to verify using (51) and Theorem 22.
Since there is a relation amongT¯α(X), R¯α(X), and E¯α(X) by (51), some properties of STE can be deduced by the theorems of SEE and SRE. We only list these properties as propositions and provide necessary explanations for their proofs.Proposition 34.
IfE(Xip)<∞ for some p>m/α, i=1,2,…,m, then T¯α(X)<∞ for all α>0 and α≠1.Proof.
It is easy to verify thatT¯α(X)≥0. By the proof of Proposition 24, we see that ∫R+mF¯X(x)dx and ∫R+mF¯Xα(x)dx exist. Thus, T¯α(X)<∞ for all α>0 and α≠1.Proposition 35.
LetX=(X1,X2,…,Xm) and let Y=(Y1,Y2,…,Ym) be r.v.s with Yi=kiXi,ki≠0,i=1,2,…,m, and let k=(k1,k2,…,km)∈Rm be a constant vector; then T¯α(Y)=∏i=1mkiT¯α(X) for all α>0 and α≠1.
This proposition can be proven using (30) in [23].Proposition 36.
LetXn be a sequence of m-dimensional nonnegative r.v.s converging toX: 0<α<1. If all Xn are bounded in Lp for some p>m/α, then T¯α(X)=limn→∞T¯α(Xn) for all α>0 and α≠1.
It is immediate using (38) in [23].Proposition 37.
LetX and Y be nonnegative and independent r.v.s; then T¯α(X+Y)≥max⁡{T¯α(X),T¯α(Y)} for all α>0 and α≠1.Proof.
It can be proven in a similar way as Proposition26 by considering α-1<0 and t-tα is convex of t>0 for all 0<α<1 and α-1>0 and t-tα is concave of t>0 for all α>1.


## 3.1. Survival Rényi Entropy

If the density function is replaced by the survival function,q is set as 1, and ψ is chosen as ψ(x)=e(1-α)x in (3), it yields the survival Rényi entropy.Definition 18.
The survival Rényi entropy (SRE) of an r.v.X with order α is defined as (34)R¯αX=11-αlog⁡∫R+mF¯Xαxdx, where α>0 and α≠1.
It is immediate to see thatE¯αX=exp⁡(R¯α(X)).Definition 19.
The conditional survival entropy of r.v.X given Y with order α is defined as (35)R¯αX∣Y=11-αEylog⁡∫R+mF¯X∣Yαx∣ydx, where α>0 and α≠1.Definition 20.
The joint survival Rényi entropy of r.v.sX and Y with order α is defined as R¯α(X,Y)=R¯α(X)+R¯α(Y∣X), where α>0 and α≠1.Definition 21.
The joint survival exponential entropy of r.v.sX and Y with order α is defined as E¯α(X,Y)=E¯α(X)+E¯α(Y∣X), where α>0, α≠1, and E¯α(Y∣X)=exp(R¯α(Y∣X)) is the conditional survival exponential entropy of r.v. Y given X.
Theorem22 will show the relation between SRE and Shannon entropy.Theorem 22.
For an r.v.X, one has (36)R¯αX≥11-αHX-α,0<α<1,R¯αX≤11-αHX-α,α>1.Proof.
Using the log-sum inequality [2, 19], we obtain(37)∫R+mfxlog⁡fxF¯Xαxdx≥-log⁡∫R+mF¯Xαxdx and thus ∫R+mf(x)logf(x)dx-∫R+mf(x)logF¯Xα(x)dx≥-log∫R+mF¯Xα(x)dx. We obtain (38)HX+α∫R+mfxlog⁡F¯Xxdx≤log⁡∫R+mF¯Xαxdx.
SincedF¯X(x)/dx=-f(x), (38) can be written as (39)log⁡∫R+mF¯Xαxdx≥HX-α∫10log⁡udu=HX-α.
Note that1/(1-α)>0 for all 0<α<1 and 1/(1-α)<0 for all α>1. We complete the proof by multiplying 1/(1-α) on both sides of (39).Theorem 23.
LetX and Y be two r.v.s; then (40)R¯αX,Y≥11-αHX,Y-2α,0<α<1,(41)R¯αX,Y≤11-αHX,Y-2α,α>1.Proof.
If0<α<1, using Theorem 22, one has(42)R¯αX,Y=R¯αX+R¯αY∣X≥11-αHX+HY∣X-2α=11-αHX,Y-2α.
Similarly, the inequality in (41) holds for all α>1.
SinceR¯α(X)=log⁡E¯α(X), some properties of SRE can be similarly deduced by the proof approaches as SEE [23] and SSE [19]. We list these properties as propositions and neglect their proofs unless we can provide the improved, different, or more concise versions.Proposition 24.
IfEX(Xip)<∞, for some p>m/α, i=1,2,…,m, then R¯α(X)<∞ for all α>0 and α≠1.Proof.
For setsAi⊆R,i=1,2,…,m, using Hölder’s inequality [33], we have (43)PA1∩A2∩⋯∩Am≤∏i=1mPAi1/m.
For allα>0, we obtain (44)∫R+mF¯Xαxdx=∫R+mPX>xαdx=∫R+m∏i=1mPXi>xiαdx≤∫R+m∏i=1mPXi>xiα/mdx=∏i=1m∫0∞PXi>xiα/mdxi=∏i=1m∫01FXiα/mxidxi+∫1∞FXiα/mxidxi≤∏i=1m1+∫1∞1xipEXXipα/mdxi.
The inequality in (44) follows from Markov’s inequality [19, 34], where the integral on the right side exists if pα/m>1, that is, if p<m/α. Hence, ∫R+mF¯Xα(x)dx exists for all α>0.
(i) If∫R+mF¯Xα(x)dx≥1, note that log⁡t<t for all t≥1; then (45)R¯αX=11-αlog⁡∫R+mF¯Xαxdx<11-α∫R+mF¯Xαxdx<∞.
(ii) If0<∫R+mF¯Xα(x)dx<1, on one hand, for all α>1, since ∫R+mF¯Xα(x)dx exists for all α>0, we see 0<R¯α(X)<∞. On the other hand, for all 0<α<1, we obtain 1-α>0 and log⁡∫R+mF¯Xα(x)dx<0. We complete the proof, since -∞<R¯α(X)<0 for all 0<α<1.Proposition 25.
If the componentsXi,i=1,2,…,m, of r.v. X=(X1,X2,…,Xm) are independent, then R¯α(X)=∑i=1mR¯α(Xi) for all α>0 and α≠1.
Proposition25 is the immediate result of Theorem 15 in [23].
The Shannon entropy of a sum of independent variables is larger than that of either; for example,H(X+Y)≥max⁡{H(X),H(Y)}. SRE has this similar property.Proposition 26.
LetX andY be independent r.v.s; then R¯α(X+Y)≥max⁡R¯αX,R¯αY for all α>0 and α≠1.Proof.
SinceX and Y are independent, one has P(X+Y>u)=∫fY(v)P(X>u-v)dv. Since 1-α>0 and tα is concave of t>0 for all 0<α<1, using Jensen’s inequality, one has (46)PαX+Y>u≥∫fYvPαX>u-vdv.
Integrating both sides ofu from 0 to ∞, (47)∫PαX+Y>udu≥∫∫fYuPαX>u-vdvdu=∫fYvdv∫v∞PαX>u-vdu=∫PαX>xdx.
Then(48)log⁡∫R+mPαX+Y>udu≥log⁡∫R+mPαX>xdx.
Multiplying by1/(1-α) on both sides of the above leads to R¯α(X+Y)≥R¯α(X), and exchanging X and Y leads to R¯α(X+Y)≥R¯α(Y).
We complete the proof by using Jensen’s inequality and considering that1-α<0 and tα is convex of t>0 for all α>1 in a similar way.Proposition 27.
LetX=(X1,X2,…,Xm) and let Y=(Y1,Y2,…,Ym) be r.v.s with Yi=kiXi,ki≠0,i=1,2,…,m, for a constant vector k=(k1,k2,…,km)∈Rm; then R¯α(Y)=∑i=1mlogki+R¯α(X) for all 0<α<1.Proposition 28.
LetXn be a sequence ofm-dimensional nonnegative r.v.s converging in law to r.v.X. If all Xn are bounded in Lp for some p>m/α, then R¯α(X)=limn→∞R¯α(Xn) for all α>0 and α≠1.Proposition 29.
IfXi,i=1,2,…,m, of the r.v. X=(X1,X2,…,Xm) are independent, then R¯α(X)=∑i=1mR¯α(Xi) for all α>0 and α≠1.


## 3.2. Survival Tsallis Entropy

If the density function is replaced by the survival function,q=α, and ψ(x)=x in (3), or, equivalently, the density function is replaced by the survival function and logarithm function is replaced by q-logarithm function in (7), it yields the survival Tsallis entropy.Definition 30.
The survival Tsallis entropy (STE) of an r.v.X with order α is defined as (49)T¯αX=∫R+mF¯Xxlnα1F¯Xxdx=1α-1∫R+mF¯Xxdx-∫R+mF¯Xαxdx, where α>0 and α≠1.
Note thatF¯X(x)=∫x∞fX(t)dt. Using integration by parts formula, we obtain (50)∫R+mF¯Xxdx=xF¯Xx0∞-∫R+mxdF¯Xx=∫R+mxfXxdx=EXX.
Hence, the survival Tsallis entropy can also be written asT¯α(X)=(EX(X)-∫R+mF¯Xα(x)dx)/(α-1). It is easy to see that H¯(X)=limα→1T¯α(X) and (51)T¯αX=1α-1EXX-exp⁡1-αR¯αX=1α-1EXX-1-αlog⁡E¯αX.Definition 31.
The conditional survival Tsallis entropy of r.v.X given Y with order α is defined as (52)T¯αX∣Y=1α-1EyEXX∣Y=y-∫R+mF¯X∣Yαx∣ydx, where α>0 and α≠1.
Note thatEY(EX(X∣Y=y))=EY(∫R+mF¯X∣Yx∣ydx)=EX(X). Then T¯α(X∣Y)=1/α-1(EX(X)-Ey∫R+mF¯X∣Yα(x∣y)dx).Definition 32.
The joint survival Tsallis entropy of r.v.sX and Y with order α is defined as T¯α(X,Y)=T¯α(X)+T¯α(Y∣X), where α>0 and α≠1.
It is easy to see thatH¯(X)=limα→1T¯α(X) and H¯(X,Y)=limα→1T¯α(X,Y) using L’Hôpital’s rules.Theorem 33.
LetX be an r.v.; then (53)T¯αX≥1α-1EXX-eHX-α,0<α<1,T¯αX≤1α-1EXX-eHX-α,α>1.Proof.
It is easy to verify using (51) and Theorem 22.
Since there is a relation amongT¯α(X), R¯α(X), and E¯α(X) by (51), some properties of STE can be deduced by the theorems of SEE and SRE. We only list these properties as propositions and provide necessary explanations for their proofs.Proposition 34.
IfE(Xip)<∞ for some p>m/α, i=1,2,…,m, then T¯α(X)<∞ for all α>0 and α≠1.Proof.
It is easy to verify thatT¯α(X)≥0. By the proof of Proposition 24, we see that ∫R+mF¯X(x)dx and ∫R+mF¯Xα(x)dx exist. Thus, T¯α(X)<∞ for all α>0 and α≠1.Proposition 35.
LetX=(X1,X2,…,Xm) and let Y=(Y1,Y2,…,Ym) be r.v.s with Yi=kiXi,ki≠0,i=1,2,…,m, and let k=(k1,k2,…,km)∈Rm be a constant vector; then T¯α(Y)=∏i=1mkiT¯α(X) for all α>0 and α≠1.
This proposition can be proven using (30) in [23].Proposition 36.
LetXn be a sequence of m-dimensional nonnegative r.v.s converging toX: 0<α<1. If all Xn are bounded in Lp for some p>m/α, then T¯α(X)=limn→∞T¯α(Xn) for all α>0 and α≠1.
It is immediate using (38) in [23].Proposition 37.
LetX and Y be nonnegative and independent r.v.s; then T¯α(X+Y)≥max⁡{T¯α(X),T¯α(Y)} for all α>0 and α≠1.Proof.
It can be proven in a similar way as Proposition26 by considering α-1<0 and t-tα is convex of t>0 for all 0<α<1 and α-1>0 and t-tα is concave of t>0 for all α>1.


## 4. Similarity Measures Defined on the Survival Function

Paralleling with the similarity measures and the normalized similarity measures defined on the density function in Section2, this section will focus on the corresponding similarity measures and the normalized similarity measures defined on the survival function. Traditionally, the kernel point is the proofs of the nonnegativity for the similarity measures to be introduced.


### 4.1. Cross Survival Rényi Entropy and Cross Survival Exponential Entropy

Lemma 38.
LetX andY be r.v.s and let φ(x) be a real convex function; then (54)EY∫R+mφF¯X∣Yx∣ydx≥∫R+mφF¯Xxdx.
If, moreover,φ(x) is strictly convex, then equality holds in (54) if and only ifX andY are independent. The inequality is reversed if φ(x) is concave.
Lemma38 was proven in [23]. It is a cornerstone to prove the nonnegativity of each form of the similarity measure to be introduced according to the survival function.Theorem 39.
LetX andY be r.v.s; then R¯α(X)≥R¯α(X∣Y) and E¯α(X)≥E¯α(X∣Y) for all α>0 and α≠1.Proof.
Sincetα is concave of t>0 and 1/(1-α)>0 for all 0<α<1, note that logt is strictly concave oft. Using Lemma 38 and Jensen’s inequality, we obtain (55)R¯αX=11-αlog⁡∫R+mF¯Xαxdx≥11-αlog⁡EY∫R+mF¯X∣Yαx∣ydx≥11-αEYlog⁡∫R+mF¯X∣Yαx∣ydx=EY11-αlog⁡∫R+mF¯X∣Yαx∣ydx=R¯αX∣Y.
Similarly, considering thattα is convex of t>0 and 1/(1-α)<0 for all α>1, the conclusion is the same.
It is trivial to verify thatE¯α(X)≥E¯α(X∣Y).
For r.v.sX andY, using Theorem 39, we have (56)∫R+mF¯Xαxdx1/1-α≥EY∫R+mF¯X∣Yαx∣ydx1/1-α for all 0<α and α≠1.Definition 40.
The cross survival Rényi entropy (CSRE) of r.v.sX and Y with order α is defined as IR¯α(X,Y)=R¯α(X)-R¯α(X∣Y), where α>0 and α≠1.Definition 41.
The cross survival exponential entropy (CSEE) of r.v.sX and Y with order α is defined as IE¯α(X,Y)=E¯α(X)-E¯α(X∣Y), where α>0 and α≠1.
Using Theorem39 and (56), we obtain IR¯α(X,Y)≥0 and IE¯α(X,Y)≥0.Definition 42.
The normalized cross survival Rényi entropy (NCSRE) of r.v.sX and Y with order α is defined as (57)NIR¯αX,Y=R¯αX+R¯αYR¯αX,Y, where α>0 and α≠1.Definition 43.
The normalized cross survival exponential entropy (NCSEE) of r.v.sX and Y with order α is defined as (58)NIE¯αX,Y=E¯αX+E¯αYE¯αX,Y,where α>0 and α≠1.Definition 44.
The normalized cross survival Shannon entropy (NCSSE) of r.v.sX and Y is defined as (59)NI¯X,Y=H¯X+H¯YH¯X,Y.Proposition 45.
LetX and Y be two r.v.s; then H¯(X,Y)≤H¯(X)+H¯(Y), R¯α(X,Y)≤R¯α(X)+R¯α(Y), and E¯α(X,Y)≤E¯α(X)+E¯α(Y), for α>0 and α≠1.Proof.
Since-tlog⁡t is convex of t for t>0, using Lemma 38, it is immediate that H¯(X,Y)≤H¯(X)+H¯(Y). The rest can be similarly proven by considering the range of α and using Lemma 38.
Using L’Hôpital’s rules, it is easy to see thatI¯(X,Y)=limα→1IR¯α(X,Y) and NI¯(X,Y)=limα→1NIR¯α(X,Y).


### 4.2. Cross Survival Tsallis Entropy

Theorem 46.
For two r.v.sX and Y, one has T¯α(X)≥T¯α(X∣Y) and T¯α(X,Y)≤T¯α(X)+T¯α(Y) for all α>0 and α≠1.Proof.
SinceF¯X(x)=EY(F¯X∣Y(x∣y)), using Lemma 38, considering 1/(α-1)<0 and tα is concave of t>0 for all 0<α<1, we obtain (60)T¯αX-T¯αX∣Y=1α-1EY∫R+mF¯X∣Yαx∣ydx-∫R+mF¯Xαxdx≥1α-1∫R+mF¯Xαxdx-∫R+mF¯Xαxdx=0.
Forα>1, similarly the conclusion is the same.
It is immediate to deduce the rest of conclusion by Definition32.Definition 47.
The cross survival Tsallis entropy (CSTE) of r.v.sX and Y with order α is defined as IT¯α(X,Y)=T¯α(X)-T¯α(X∣Y), where α>0 and α≠1.
Using Theorem46, we obtain IT¯α(X,Y)≥0 immediately.Definition 48.
The normalized cross survival Tsallis entropy (NCSTE) of r.v.sX and Y with order α is defined as (61)NIT¯αX,Y=T¯αX+T¯αYT¯αX,Y, where α>0 and α≠1.
We obtainI¯(X,Y)=limα→1IT¯α(X,Y) and NI¯(X,Y)=limα→1NIT¯α(X,Y) by L’Hôpital’s rules.
Note thatI¯(X,Y)≠I¯(Y,X), IR¯α(X,Y)≠IR¯α(Y,X), IE¯α(X,Y)≠IE¯α(Y,X), and IT¯α(X,Y)≠IT¯α(Y,X), whereas I(X,Y) is symmetric; for example, I(X,Y)=I(Y,X). We can define the symmetric versions of the similarity measures and the normalized similarity measures, taking the cross survival Rényi entropy as an example, as (62)SIR¯αX,Y=12IR¯αX,Y+IR¯αY,X,SNIR¯αX,Y=12NIR¯αX,Y+NIR¯αY,X.
The similar way can be used to define the symmetric similarity measures and normalized similarity measures for those defined on the density function.


## 4.1. Cross Survival Rényi Entropy and Cross Survival Exponential Entropy

Lemma 38.
LetX andY be r.v.s and let φ(x) be a real convex function; then (54)EY∫R+mφF¯X∣Yx∣ydx≥∫R+mφF¯Xxdx.
If, moreover,φ(x) is strictly convex, then equality holds in (54) if and only ifX andY are independent. The inequality is reversed if φ(x) is concave.
Lemma38 was proven in [23]. It is a cornerstone to prove the nonnegativity of each form of the similarity measure to be introduced according to the survival function.Theorem 39.
LetX andY be r.v.s; then R¯α(X)≥R¯α(X∣Y) and E¯α(X)≥E¯α(X∣Y) for all α>0 and α≠1.Proof.
Sincetα is concave of t>0 and 1/(1-α)>0 for all 0<α<1, note that logt is strictly concave oft. Using Lemma 38 and Jensen’s inequality, we obtain (55)R¯αX=11-αlog⁡∫R+mF¯Xαxdx≥11-αlog⁡EY∫R+mF¯X∣Yαx∣ydx≥11-αEYlog⁡∫R+mF¯X∣Yαx∣ydx=EY11-αlog⁡∫R+mF¯X∣Yαx∣ydx=R¯αX∣Y.
Similarly, considering thattα is convex of t>0 and 1/(1-α)<0 for all α>1, the conclusion is the same.
It is trivial to verify thatE¯α(X)≥E¯α(X∣Y).
For r.v.sX andY, using Theorem 39, we have (56)∫R+mF¯Xαxdx1/1-α≥EY∫R+mF¯X∣Yαx∣ydx1/1-α for all 0<α and α≠1.Definition 40.
The cross survival Rényi entropy (CSRE) of r.v.sX and Y with order α is defined as IR¯α(X,Y)=R¯α(X)-R¯α(X∣Y), where α>0 and α≠1.Definition 41.
The cross survival exponential entropy (CSEE) of r.v.sX and Y with order α is defined as IE¯α(X,Y)=E¯α(X)-E¯α(X∣Y), where α>0 and α≠1.
Using Theorem39 and (56), we obtain IR¯α(X,Y)≥0 and IE¯α(X,Y)≥0.Definition 42.
The normalized cross survival Rényi entropy (NCSRE) of r.v.sX and Y with order α is defined as (57)NIR¯αX,Y=R¯αX+R¯αYR¯αX,Y, where α>0 and α≠1.Definition 43.
The normalized cross survival exponential entropy (NCSEE) of r.v.sX and Y with order α is defined as (58)NIE¯αX,Y=E¯αX+E¯αYE¯αX,Y,where α>0 and α≠1.Definition 44.
The normalized cross survival Shannon entropy (NCSSE) of r.v.sX and Y is defined as (59)NI¯X,Y=H¯X+H¯YH¯X,Y.Proposition 45.
LetX and Y be two r.v.s; then H¯(X,Y)≤H¯(X)+H¯(Y), R¯α(X,Y)≤R¯α(X)+R¯α(Y), and E¯α(X,Y)≤E¯α(X)+E¯α(Y), for α>0 and α≠1.Proof.
Since-tlog⁡t is convex of t for t>0, using Lemma 38, it is immediate that H¯(X,Y)≤H¯(X)+H¯(Y). The rest can be similarly proven by considering the range of α and using Lemma 38.
Using L’Hôpital’s rules, it is easy to see thatI¯(X,Y)=limα→1IR¯α(X,Y) and NI¯(X,Y)=limα→1NIR¯α(X,Y).


## 4.2. Cross Survival Tsallis Entropy

Theorem 46.
For two r.v.sX and Y, one has T¯α(X)≥T¯α(X∣Y) and T¯α(X,Y)≤T¯α(X)+T¯α(Y) for all α>0 and α≠1.Proof.
SinceF¯X(x)=EY(F¯X∣Y(x∣y)), using Lemma 38, considering 1/(α-1)<0 and tα is concave of t>0 for all 0<α<1, we obtain (60)T¯αX-T¯αX∣Y=1α-1EY∫R+mF¯X∣Yαx∣ydx-∫R+mF¯Xαxdx≥1α-1∫R+mF¯Xαxdx-∫R+mF¯Xαxdx=0.
Forα>1, similarly the conclusion is the same.
It is immediate to deduce the rest of conclusion by Definition32.Definition 47.
The cross survival Tsallis entropy (CSTE) of r.v.sX and Y with order α is defined as IT¯α(X,Y)=T¯α(X)-T¯α(X∣Y), where α>0 and α≠1.
Using Theorem46, we obtain IT¯α(X,Y)≥0 immediately.Definition 48.
The normalized cross survival Tsallis entropy (NCSTE) of r.v.sX and Y with order α is defined as (61)NIT¯αX,Y=T¯αX+T¯αYT¯αX,Y, where α>0 and α≠1.
We obtainI¯(X,Y)=limα→1IT¯α(X,Y) and NI¯(X,Y)=limα→1NIT¯α(X,Y) by L’Hôpital’s rules.
Note thatI¯(X,Y)≠I¯(Y,X), IR¯α(X,Y)≠IR¯α(Y,X), IE¯α(X,Y)≠IE¯α(Y,X), and IT¯α(X,Y)≠IT¯α(Y,X), whereas I(X,Y) is symmetric; for example, I(X,Y)=I(Y,X). We can define the symmetric versions of the similarity measures and the normalized similarity measures, taking the cross survival Rényi entropy as an example, as (62)SIR¯αX,Y=12IR¯αX,Y+IR¯αY,X,SNIR¯αX,Y=12NIR¯αX,Y+NIR¯αY,X.
The similar way can be used to define the symmetric similarity measures and normalized similarity measures for those defined on the density function.


## 5. Unifying Frameworks and Examples

In this section, based on the generalized denotations on the entropies discussed previously, we will classify the fourteen types of entropies in two categories and then deduce the unifying presentations for entropies, similarity measures, and normalized similarity measures. Examples are also provided to unveil some properties of the information measures.As enumerated in Table1, different types of entropies have been discussed in this paper. There are three components in each item: entropy, conditional entropy, and joint entropy. In general, entropies in Table 1 can be classified into two categories: one is defined on the density function and the other is defined on the survival function. For simplicity, we refer to them as the density entropy and survival entropy, respectively. It is demonstrated that, for each type of the density entropy in Column 2, there is a survival entropy in Column 4 corresponding to it.


### 5.1. The Unifying Frameworks of Information Measures

For convenience, we view Shannon entropy, Rényi entropy, Tsallis entropy, and the exponential entropy as the classical density entropy and view their corresponding survival entropies as the classical survival entropy. We can see that the classical density entropy, the classical survival entropy, and their conditional entropy and joint entropy share similar presentations.LetHα(X) be one type of the generalized density entropy or the generalized survival entropy of r.v. X with order α. If α=1, then H1(X)=H(X) means the Shannon entropy or the survival Shannon entropy. In these notations, Hα(X∣Y) is the conditional entropy with order α>0. The corresponding joint entropy of r.v.s X and Y with order α>0 can be introduced as (63)HαX,Y=HαX+HαY∣X.For r.v.sX and Y, one has Hα(X,Y)≤Hα(X)+Hα(Y) for all α>0.Entropies, conditional entropies, and joint entropies are listed in Table1. The similarity measures and the normalized similarity measures are shown in Table 2 in detail, where the similarity measure is followed by the normalized one in each item. In a similar way, the similarity measure can be classified into the density similarity measure defined on the density function and the survival similarity measure defined on the survival function and so can the normalized similarity measures. Therefore, the unifying presentations for a similarity and a normalized similarity measure associated with a type of entropies can be deduced as (64)IαX,Y=HαX+HαX∣Y,(65)NIαX,Y=HαX+HαYHαX,Y.Note thatI1(X,Y)=I(X,Y) and NI1(X,Y)=NI(X,Y).We obtainIα(X,Y)≥0 for all α>0. Their symmetric versions can be, respectively, given by (66)SIαX,Y=12IαX,Y+IαY,X,(67)SNIαX,Y=12NIαX,Y+NIαY,X.The unifying frameworks make it possible to view all the available entropies listed in Table1 as a whole and to move from one to another as necessary. Subsequently, the similarity measures and the normalized similarity measures are simultaneously obtained.


### 5.2. Three Examples

Example 1.
LetX be an r.v. corresponding to the exponential distribution with mean 1/λ and density function f(x)=λe-λx. We obtain F¯(x)=∫x∞f(t)dt=e-λx, and (68)H¯X=1λ,R¯αX=1α-1log⁡λα,T¯αX=1λα,E¯αX=λα1/α-1.
The survival Shannon entropy, survival Rényi entropy, survival Tsallis entropy, and survival exponential entropy of r.v.X are plotted in Figure 1(a) with 0.5≤α≤0.9 and λ=1.1.
Ifλ=1, then F¯(x)=e-x=f(x). Hence, the survival entropies become the corresponding density entropies. Figure 1(b) shows Shannon entropy, Rényi entropy, Tsallis entropy, and the exponential entropy of an r.v. X with 0.5≤α≤1.5. We can see that some properties, such as concavity and monotonicity, may be changed when we generalize entropies by extending their definitions from the density function to the survival function.Figure 1
Plots of eight different types of entropies. (a) Survival entropies. (b) Density entropies.
(a) (b)Example 2.
Many literatures have pointed out that Shannon entropy has drawbacks: each frequency of the occurrence event contributes equally in summation or integral of its functional and, simultaneously, the spatial information is neglected. In fact, those entropies that are defined on density functions have similar drawbacks to Shannon entropy. However, the survival entropy can overcome this drawback. For instance, as shown in Figure2, Lena is used as a fixed image with size of 256×256. We exchange those pixels with the same occurrence frequency in Figure 2(a) and then generate the negative image as shown in Figure 2(b). Images (c) and (d) are their histograms, respectively. Thereby images (a) and (b) share the same occurrence frequency but specified to different pixel values or, equivalently, to different random variables.
The values of entropies of SE, RE, TE, SSE, SRE, and STE for Figures2(a) and 2(b) are shown in Table 3, respectively, where α=0.8. We can see that density entropies provide the same values, whereas survival entropies provide different values for these two images. In other words, these two images contain the same information in the view of the density entropy. But, to the survival entropy, the amount of information there is different. It is demonstrated that the survival entropy is capable of distinguishing the two images, whereas the density entropy is not. The reason is that spatial information is taken into account in survival entropy formulas.Table 3
The values acquired from different types of entropy for testing images.
Image SE RE TE SSE SRE STE (a) 5.1418 5.1755 9.0768 51.0285 22.9235 58.8816 (b) 5.1418 5.1755 9.0768 46.9452 25.9032 51.0613Figure 2
Testing images ((a) and (b)) and their histograms ((c) and (d)) accordingly.
(a) (b) (c) (d)Example 3.
We use a computerized tomography (CT) of brain slice as a testing image to evaluate the properties of similarity measures and normalized similarity measures. The density of CT ranges from 0 to 255 with the mean of 92.683 and size of256×256. It is originally fixed and viewed as a template. A series of rotation transformations are applied to it so as to obtain floating images for comparison with the template. The range of rotation angle is set as [-π,π]. We calculate the values of SMI, RMI, TMI, CSSE, CSRE, and CSTE between the template and floating images, as shown in Figure 3(a). Accordingly, as shown in Figure 3(b), the values of the normalized similarity measures of NSMI, NRMI, NTMI, NCSSE, NCSRE, and NCSTE acquired from the template and floating images in this experiment are also calculated with the same parameter settings, α=0.8, as Figure 3(a).
Figure3 demonstrates that when the rotation angle is zero or, equivalently, the template and the floating image are perfectly aligned, all the values of the considered similarity metrics reach their maximums. In other words, they all can be adopted as similarity metrics in image processing.
Generally, the similarity measures defined on survival function are more regular than those defined on density function, since the density is computed as the derivative of the survival function [19]. Therefore, the similarity measures defined on survival function are more robust in matching problems.
A kernel problem arising from matching problem is the choice of similarity metrics. In Table2, sixteen different types of similarity measures and normalized similarity measures are provided with unifying presentations by (64) and (65). Their symmetric versions are given by (66) and (67) simultaneously. The proposed similarity measures are capable of developing the means of matching in signal processing to some extent.Figure 3
The values of similarity measures (a) and normalized similarity measures (b) for image rotation transformations with angle range[-π,π].
(a) (b)


## 5.1. The Unifying Frameworks of Information Measures

For convenience, we view Shannon entropy, Rényi entropy, Tsallis entropy, and the exponential entropy as the classical density entropy and view their corresponding survival entropies as the classical survival entropy. We can see that the classical density entropy, the classical survival entropy, and their conditional entropy and joint entropy share similar presentations.LetHα(X) be one type of the generalized density entropy or the generalized survival entropy of r.v. X with order α. If α=1, then H1(X)=H(X) means the Shannon entropy or the survival Shannon entropy. In these notations, Hα(X∣Y) is the conditional entropy with order α>0. The corresponding joint entropy of r.v.s X and Y with order α>0 can be introduced as (63)HαX,Y=HαX+HαY∣X.For r.v.sX and Y, one has Hα(X,Y)≤Hα(X)+Hα(Y) for all α>0.Entropies, conditional entropies, and joint entropies are listed in Table1. The similarity measures and the normalized similarity measures are shown in Table 2 in detail, where the similarity measure is followed by the normalized one in each item. In a similar way, the similarity measure can be classified into the density similarity measure defined on the density function and the survival similarity measure defined on the survival function and so can the normalized similarity measures. Therefore, the unifying presentations for a similarity and a normalized similarity measure associated with a type of entropies can be deduced as (64)IαX,Y=HαX+HαX∣Y,(65)NIαX,Y=HαX+HαYHαX,Y.Note thatI1(X,Y)=I(X,Y) and NI1(X,Y)=NI(X,Y).We obtainIα(X,Y)≥0 for all α>0. Their symmetric versions can be, respectively, given by (66)SIαX,Y=12IαX,Y+IαY,X,(67)SNIαX,Y=12NIαX,Y+NIαY,X.The unifying frameworks make it possible to view all the available entropies listed in Table1 as a whole and to move from one to another as necessary. Subsequently, the similarity measures and the normalized similarity measures are simultaneously obtained.


## 5.2. Three Examples

Example 1.
LetX be an r.v. corresponding to the exponential distribution with mean 1/λ and density function f(x)=λe-λx. We obtain F¯(x)=∫x∞f(t)dt=e-λx, and (68)H¯X=1λ,R¯αX=1α-1log⁡λα,T¯αX=1λα,E¯αX=λα1/α-1.
The survival Shannon entropy, survival Rényi entropy, survival Tsallis entropy, and survival exponential entropy of r.v.X are plotted in Figure 1(a) with 0.5≤α≤0.9 and λ=1.1.
Ifλ=1, then F¯(x)=e-x=f(x). Hence, the survival entropies become the corresponding density entropies. Figure 1(b) shows Shannon entropy, Rényi entropy, Tsallis entropy, and the exponential entropy of an r.v. X with 0.5≤α≤1.5. We can see that some properties, such as concavity and monotonicity, may be changed when we generalize entropies by extending their definitions from the density function to the survival function.Figure 1
Plots of eight different types of entropies. (a) Survival entropies. (b) Density entropies.
(a) (b)Example 2.
Many literatures have pointed out that Shannon entropy has drawbacks: each frequency of the occurrence event contributes equally in summation or integral of its functional and, simultaneously, the spatial information is neglected. In fact, those entropies that are defined on density functions have similar drawbacks to Shannon entropy. However, the survival entropy can overcome this drawback. For instance, as shown in Figure2, Lena is used as a fixed image with size of 256×256. We exchange those pixels with the same occurrence frequency in Figure 2(a) and then generate the negative image as shown in Figure 2(b). Images (c) and (d) are their histograms, respectively. Thereby images (a) and (b) share the same occurrence frequency but specified to different pixel values or, equivalently, to different random variables.
The values of entropies of SE, RE, TE, SSE, SRE, and STE for Figures2(a) and 2(b) are shown in Table 3, respectively, where α=0.8. We can see that density entropies provide the same values, whereas survival entropies provide different values for these two images. In other words, these two images contain the same information in the view of the density entropy. But, to the survival entropy, the amount of information there is different. It is demonstrated that the survival entropy is capable of distinguishing the two images, whereas the density entropy is not. The reason is that spatial information is taken into account in survival entropy formulas.Table 3
The values acquired from different types of entropy for testing images.
Image SE RE TE SSE SRE STE (a) 5.1418 5.1755 9.0768 51.0285 22.9235 58.8816 (b) 5.1418 5.1755 9.0768 46.9452 25.9032 51.0613Figure 2
Testing images ((a) and (b)) and their histograms ((c) and (d)) accordingly.
(a) (b) (c) (d)Example 3.
We use a computerized tomography (CT) of brain slice as a testing image to evaluate the properties of similarity measures and normalized similarity measures. The density of CT ranges from 0 to 255 with the mean of 92.683 and size of256×256. It is originally fixed and viewed as a template. A series of rotation transformations are applied to it so as to obtain floating images for comparison with the template. The range of rotation angle is set as [-π,π]. We calculate the values of SMI, RMI, TMI, CSSE, CSRE, and CSTE between the template and floating images, as shown in Figure 3(a). Accordingly, as shown in Figure 3(b), the values of the normalized similarity measures of NSMI, NRMI, NTMI, NCSSE, NCSRE, and NCSTE acquired from the template and floating images in this experiment are also calculated with the same parameter settings, α=0.8, as Figure 3(a).
Figure3 demonstrates that when the rotation angle is zero or, equivalently, the template and the floating image are perfectly aligned, all the values of the considered similarity metrics reach their maximums. In other words, they all can be adopted as similarity metrics in image processing.
Generally, the similarity measures defined on survival function are more regular than those defined on density function, since the density is computed as the derivative of the survival function [19]. Therefore, the similarity measures defined on survival function are more robust in matching problems.
A kernel problem arising from matching problem is the choice of similarity metrics. In Table2, sixteen different types of similarity measures and normalized similarity measures are provided with unifying presentations by (64) and (65). Their symmetric versions are given by (66) and (67) simultaneously. The proposed similarity measures are capable of developing the means of matching in signal processing to some extent.Figure 3
The values of similarity measures (a) and normalized similarity measures (b) for image rotation transformations with angle range[-π,π].
(a) (b)


## 6. Conclusions

This research is conducted in two dimensions. On one hand, we extend KN averages toq-generalized KN averages to redefine the existing classical entropies and to define the survival Rényi entropy and survival Tsallis entropy on the survival function. On the other hand, for each type of entropies discussed, a similarity measure and a normalized similarity measure are proposed accordingly. Some properties of the information measures are researched.We make an inventory of sixteen types of entropies, similarity measures, and normalized similarity measures which exhibit diversity and unity. This leads to the proposition of the unifying frameworks for information measures. Therefore, our work addresses a broad spectrum of information measures as a whole through the unifying frameworks. Undoubtedly, some of them will be conceptual if they are not adapted for applications. More applications of the proposed information measures will dominate our research in the near future.



---

*Source: 1791954-2018-03-08.xml*