# A Unified Definition of Mutual Information with Applications in Machine Learning

Authors: Guoping Zeng
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

There are various definitions of mutual information. Essentially, these definitions can be divided into two classes: (1) definitions with random variables and (2) definitions with ensembles. However, there are some mathematical flaws in these definitions. For instance, Class 1 definitions either neglect the probability spaces or assume the two random variables have the same probability space. Class 2 definitions redefine marginal probabilities from the joint probabilities. In fact, the marginal probabilities are given from the ensembles and should not be redefined from the joint probabilities. Both Class 1 and Class 2 definitions assume a joint distribution exists. Yet, they all ignore an important fact that the joint or the joint probability measure is not unique. In this paper, we first present a new unified definition of mutual information to cover all the various definitions and to fix their mathematical flaws. Our idea is to define the joint distribution of two random variables by taking the marginal probabilities into consideration. Next, we establish some properties of the newly defined mutual information. We then propose a method to calculate mutual information in machine learning. Finally, we apply our newly defined mutual information to credit scoring.


---

## Body

## 1. Introduction

Mutual information has emerged in recent years as an important measure of statistical dependence. It has been used as a criterion for feature selection in engineering especially in machine learning (see [1–3] and references therein).Mutual information is a concept rooted in information theory. Its predecessor, called the rate of transmission, was first introduced by Shannon in 1948 in a classical paper [4] for the communication system. Shannon first introduced a concept called entropy for a single discrete chance variable. He then defined the joint entropy and conditional entropy for two discrete chance variables using the joint distribution. Finally, he defined the rate of transmission as the difference between the entropy and conditional entropy. While Shannon did not define a chance variable in his paper, it is understood to be a synonym of a random variable.Since Shannon’s pioneering work [4], there have been various definitions for mutual information. Essentially, these definitions can be divided into two classes: (1) definitions with random variables and (2) definitions with ensembles, that is, probability spaces in the mathematical literature.Class 1 definitions of mutual information depend on the joint distribution of two random variables. More specifically, Kullback ([5], 1959) defined entropy, conditional entropy, and joint entropy using compact mathematical formulas. Pinsker ([6], 1960 and 1964) treated the fundamental concepts of Shannon in a more advanced manner by employing probability theory. His definition of mutual information was more general in that he implicitly assumed the two random variables had different probability spaces. Ash ([7], 1965) explicitly assumed the two random variables had the same probability space and followed Shannon’s way to define mutual information. Cover and Thomas ([8], 2006) defined mutual information in a simple way by avoiding mentioning probability spaces.Class 2 definitions depend on the joint probability measure of the joint sample space of two ensembles. Among such definitions, Fano ([9], 1961), Abramson ([10], 1963), and Gallager ([11], 1968) developed their definitions in a similar way. They first defined the entropy of an ensemble, condition entropy, and the joint entropy of two ensembles. Next, they defined the mutual information of a joint event. Noting that the mutual information of a joint event is a random variable, they calculated the mean value of this random variable and called the result the mean information of two ensembles.However, there are some mathematical flaws in these various definitions of mutual information. Class 2 definitions redefine marginal probabilities from the joint probabilities. As a matter of fact, the marginal probabilities are given from the ensembles and hence should not be redefined from the joint probabilities. Moreover, except Pinsker’s definition, Class 2 definitions either neglect the probability spaces or assume the two random variables have the same probability space. Both Class 1 definitions and Class 2 definitions assume a joint distribution or a joint probability measure exists. Yet, they all ignore an important fact that the joint distribution or the joint probability measure is not unique.In this paper, we first present a unified definition for mutual information using random variables with different probability spaces. Our idea is to define the joint distribution of two random variables by taking the marginal probabilities into consideration. With our new definition of mutual information, different joint distributions will result in different mutual information. Next, we establish some properties of the newly defined mutual information. We then propose a method to calculate mutual information in machine learning. Finally, we apply our newly defined mutual information to credit scoring.The rest of the paper is organized as follows. In Section2, we briefly review the basic concepts in probability theory. In Section 3, we examine various definitions of mutual information. In Section 4, we first propose a new unified definition for mutual information and then establish some properties of the newly defined mutual information. In Section 5, we first propose a method to calculate mutual information in machine learning. We then apply the newly defined mutual information to credit scoring. The paper is concluded in Section 6.Throughout the paper, we will restrict our focus on mutual information for finite discrete random variables.


## 2. Basic Concepts in Probability Theory

Let us review some basic concepts of probability theory. They can be found in many books in probability theory such as [12].Definition 1.
A probability space is a triple(
Ω
,
F
,
P
), where (1)
Ω is a set, called a sample space, and elements of Ω are denoted by ω and are called outcomes,
(2)
F is an σ-field consisting of all subsets of Ω, and elements of F are called events,
(3)
P is called a probability measure, and it is a mapping from F to [
0
,
1
] with P
Ω
=
1 such that if A
1
,
A
2
,
… are pairwise disjoint, (1)
P
⋃
i
A
i
=
∑
i
P
(
A
i
)
.Definition 2.
A discrete probability space is a probability space(
Ω
,
F
,
P
) such that Ω is finite or countable: Ω
=
ω
1
,
ω
2
,
…
,. In this case, F is chosen to be all the subsets of Ω, and the probability measure P can be defined in terms of a series of nonnegative numbers p
1
,
p
2
,
… whose sum is 1. If A is any subset of Ω, then (2)
P
A
=
∑
ω
i
∈
A
p
i
.In particular,(3)
P
ω
i
=
p
i
.For simplicity, we will write P
ω as P
(
ω
). From Definition 2, we see that for a discrete probability space the probability measure is characterized by the pointwise mapping p: ω
1
,
ω
2
,
…
,
→
[
0
,
1
] in (2). The probability of an event A is computed simply by adding the probabilities of the individual points of A.Definition 3.
A randomvariableX on probability space (
Ω
,
F
,
P
) is a Borel measurable function from Ω to (
-
∞
,
∞
) such that, for every Borel set B, X
-
1
B
=
X
∈
B
∈
F. Here, we use notation X
∈
B
=
ω
∈
Ω
:
X
ω
∈
B.Definition 4.
IfX is a random variable, then for every Borel subset B of R, we define a function by μ
X
B
=
P
X
∈
B
=
P
X
-
1
B
. Then μ
X is a probability measure on R
,
B
R
,
μ
X and is called the probability distribution of X.Definition 5.
A random variableX is discrete if its range is finite or countable. In particular, any random variable on a discrete probability space is discrete, since Ω is countable.Definition 6.
A (discrete) random variableX on a discrete probability space (
Ω
,
F
,
P
) is a Borel measurable function from Ω to R, where Ω
=
ω
1
,
ω
2
,
…
, and R is the set of real numbers. If the range of X is x
1
,
x
2
,
…, then function f
X
:
x
1
,
x
2
,
…
→
0
,
1 defined by(4)
f
X
x
i
=
P
X
=
x
iis called the probability mass function of X, whereas probabilities P
X
=
x
1, P
X
=
x
2
,
… are called the probability distribution of X.Note that, in Definition2,(5)
∑
i
P
X
=
x
i
=
1
.Thus, a discrete random variable may be characterized by its probability mass function.


## 3. Various Definitions of Mutual Information

Since Shannon’s pioneering work [4], there have been various definitions for mutual information. Essentially, these definitions can be divided into two classes: (1) definitions with random variables and (2) definitions with ensembles, that is, probability spaces in the mathematical literature.


### 3.1. Shannon’s Original Definition

Definition 7.
Letx be a chance variable with probabilities p
1
,
p
2
,
…
,
p
n, whose sum is 1. Then(6)
H
x
=
-
∑
i
=
1
n
p
i
log
⁡
⁡
p
iis called the entropy of x.
Suppose two chance variables,x and y, have m and n possibilities, respectively. Let indices i and j range over all the m possibilities and all the n possibilities, respectively. Let p
(
i
) be the probability of i and p
(
i
,
j
) the probability of the joint occurrence of i and j. Denote the conditional probability of i given j by p
(
i
∣
j
) and conditional probability of j given i by p
(
j
∣
i
).Definition 8.
The joint entropy ofx and y is defined as(7)
H
x
,
y
=
-
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
i
,
j
.Definition 9.
The conditional entropy ofy, H
x
y, is defined as(8)
H
x
y
=
-
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
j
∣
i
=
-
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
i
,
j
∑
j
p
i
,
j
.The conditional entropy of x, H
y
x, can be defined similarly.Then the following relation holds:(9)
H
x
,
y
=
H
x
+
H
x
y
=
H
y
+
H
y
x
,
H
x
-
H
y
x
=
H
y
-
H
x
y
=
H
x
+
H
y
-
H
x
,
y
.Definition 10.
The rate of transmission of informationR is defined as the difference between H
x and H
y
x. Then R can be written in two other forms: (10)
R
=
H
x
-
H
y
x
=
H
y
-
H
x
y
=
H
x
+
H
y
-
H
x
,
y
.Remark 11.
Shannon did not derive the explicit formula forR:(11)
R
=
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
i
,
j
p
i
p
(
j
)
.However, he did imply it in Appendix 7 [4].


### 3.2. Class 1 Definitions

#### 3.2.1. Kullback’s Definition

Kullback [5] redefined entropy more mathematically in a standalone homework question as follows. Consider two discrete random variables x
,
y, where (12)
p
i
j
=
Prob
x
=
x
i
,
y
=
y
j
>
0
,
i
=
1
,
2
,
…
,
m
,
j
=
1
,
2
,
…
,
n
,
p
i
·
=
∑
j
=
1
n
p
i
j
,
p
·
j
=
∑
i
=
1
m
p
i
j
,
∑
i
=
1
m
∑
j
=
1
n
p
i
j
=
∑
i
=
1
m
p
i
·
=
∑
j
=
1
n
p
·
j
=
1
.Define the joint entropy, entropy, and conditional entropy as follows: (13)
H
x
,
y
=
-
∑
i
∑
j
p
i
j
log
⁡
⁡
p
i
j
,
H
x
=
-
∑
i
p
i
·
log
⁡
⁡
p
i
·
,
H
y
=
-
∑
j
p
·
j
log
⁡
⁡
p
·
j
,
H
y
∣
x
i
=
-
∑
j
p
i
j
p
i
·
log
⁡
⁡
p
i
j
p
i
·
,
H
y
∣
x
=
∑
i
p
i
H
y
∣
x
i
=
-
∑
i
∑
j
p
i
j
log
⁡
⁡
p
i
j
p
i
·
.Then H
x
,
y
=
H
x
+
H
y
∣
x
≤
H
x
+
H
(
y
) and H
y
≥
H
y
∣
x.


#### 3.2.2. Information Conveyed

Ash [7] began with two random variables X and Y and assumed X and Y had the same probability space. He systematically defined the entropy, conditional entropy, and joint entropy following Shannon’s path in [4]. At the end, he denoted H
X
-
H
(
X
∣
Y
) by I
(
X
∣
Y
) and called it the information conveyed about X by Y.


#### 3.2.3. Information of One Variable with respect to the Other

Pinsker [6] treated the fundamental concepts of Shannon in a more advanced manner by employing probability theory. Suppose ξ is a random variable defined on a probability space Ω
,
S
ω
,
P
ξ and is taking values in a measurable space X
,
S
x and η is a random variable defined on a probability space Ψ
,
S
ψ
,
P
η and is taking values in a measurable space Y
,
S
y. Then, the pair ξ, η of random variables may be regarded as a single random variable (
ξ
,
η
) with values in the product space X
×
Y of all pairs (
x
,
y
) with x
∈
X, y
∈
Y
. The distribution P
ξ
,
η
·
=
P
ξ
η
(
·
) of (
ξ
,
η
) is called the joint distribution of random variables ξ and η. By the product of the distributions P
ξ
(
·
) and P
η
(
·
), denoted by P
ξ
×
η
(
·
), we mean the distribution defined on S
x
×
S
y
(14)
P
ξ
×
η
E
×
F
=
P
ξ
(
E
)
×
P
η
(
F
)for E
∈
S
x and F
∈
S
y. If the joint distribution P
ξ
η
(
·
) coincides with the product distribution P
ξ
×
η
·
, the random variables ξ and η are said to be independent. If ξ and η are discrete random variables, say, X and Y contain countably many points, x
1
,
x
2
,
… and y
1
,
y
2
,
…, then(15)
I
ξ
,
η
=
∑
i
,
j
P
ξ
×
η
(
x
i
,
y
j
)
log
⁡
⁡
P
ξ
η
(
x
i
,
y
j
)
P
ξ
(
x
i
)
P
η
(
y
j
)
.
I is called the information of ξ and η with respect to the other.


#### 3.2.4. A Modern Definition in Information Theory

Of the various definitions of mutual information, the most widely accepted, of recent years, is the one by Cover and Thomas [8].LetX be a discrete variable with alphabet χ and probability mass function p
x
=
Pr
⁡
⁡
X
=
x, x
∈
χ. Let Y be a discrete variable with alphabet Υ and probability mass function p
y
=
Pr
⁡
⁡
Y
=
y, y
∈
Υ
. Suppose X and Y have a joint mass function (joint distribution) p
x
,
y. Then the mutual information I
(
X
,
Y
) can be defined as (16)
I
X
,
Y
=
∑
x
∈
χ
∑
y
∈
Υ
p
(
x
,
y
)
log
⁡
⁡
p
(
x
,
y
)
p
x
p
(
y
)
.


### 3.3. Class 2 Definitions

In Class 2 definitions, random variables are replaced by ensembles and mutual information is so-called average mutual information. Gallager [11] adopted a more general and more rigorous approach to introduce the concept of mutual information in communication theory. Indeed, he combined and compiled the results from Fano [9] and Abramson [10].Suppose that discrete ensembleX has a sample space a
1
,
a
2
,
…
,
a
K and discrete ensemble Y has a sample space b
1
,
b
2
,
…
,
b
L. Consider the joint sample space  a
k
,
b
j
:
1
≤
k
≤
K
,
1
≤
j
≤
J. A probability measure on the joint sample space is given by the join probability P
X
Y
a
k
,
b
j, defined for 1
≤
k
≤
K, 1
≤
j
≤
J. The combination of a joint sample space and probability measure for outcomes x and y is called a joint X
Yensemble. Then the marginal probabilities can be found as(17)
P
X
a
k
=
∑
j
=
1
J
P
X
Y
a
k
,
b
j
,
k
=
1
,
2
,
…
,
K
.In more abbreviated notation, this is written as(18)
P
x
=
∑
y
P
x
,
y
.Likewise, (19)
P
Y
b
j
=
∑
k
=
1
K
P
X
Y
(
a
k
,
b
j
)
,
j
=
1
,
2
,
…
,
J
.In more abbreviated notation, this is written as(20)
P
y
=
∑
x
P
x
,
y
.If P
X
a
k
>
0, the conditional probability that outcome y is b
j, given that outcome of x is a
k, is defined as(21)
P
Y
∣
X
b
j
∣
a
k
=
P
X
Y
a
k
,
b
j
P
X
a
k
.The mutual information between events x
=
a
k and y
=
b
j is defined as(22)
I
X
:
Y
a
k
,
b
j
=
log
⁡
P
X
∣
Y
a
k
∣
b
j
P
X
a
k
=
log
⁡
P
X
Y
a
k
,
b
j
P
X
a
k
P
Y
b
j
=
log
⁡
P
X
∣
Y
a
k
∣
b
j
P
X
a
k
=
I
Y
:
X
b
j
,
a
k
.Since the mutual information defined above is a random variable on the X
Y joint ensemble, the mean value, which is called the average mutual information denoted by I
(
X
,
Y
), is given by(23)
I
X
,
Y
=
∑
k
=
1
K
∑
j
=
1
L
P
X
Y
a
k
,
b
j
log
⁡
⁡
P
X
Y
a
k
,
b
j
P
X
a
k
P
Y
(
b
j
)
.Remark 12.
By means of an information channel consisting of a transmitter of alphabetA with elements a
i and total elements t and a receiver of alphabet B with elements b
j and total elements r, Abramson [10] denoted H
A
-
H
(
A
∣
B
)
=
∑
A
,
B
P
a
,
b
log
⁡
⁡
(
P
(
a
,
b
)
/
P
a
P
(
b
)
) by I
(
A
,
B
) and called it mutual information of A and B.The mutual informationI
X
,
Y between 2 continuous random variables X and Y [8] (also called rate of transmission in [1]) is defined as (24)
I
X
,
Y
=
∬
P
(
x
,
y
)
log
⁡
⁡
P
(
x
,
y
)
P
x
P
(
y
)
d
x
d
y
,where P
(
x
,
y
) is the joint probability density function of X and Y and P
(
x
) and P
(
y
) are the marginal density functions associated with X and Y, respectively. The mutual information between 2 continuous random variables is also called the differential mutual information.However, the differential mutual information is much less popular than its discrete counterpart. On the one hand, the joint density function involved is unknown in most cases and hence must be estimated [13, 14]. On the other hand, data in engineering and machine learning are mostly finite, and so mutual information between discrete random variables is used.


## 3.1. Shannon’s Original Definition

Definition 7.
Letx be a chance variable with probabilities p
1
,
p
2
,
…
,
p
n, whose sum is 1. Then(6)
H
x
=
-
∑
i
=
1
n
p
i
log
⁡
⁡
p
iis called the entropy of x.
Suppose two chance variables,x and y, have m and n possibilities, respectively. Let indices i and j range over all the m possibilities and all the n possibilities, respectively. Let p
(
i
) be the probability of i and p
(
i
,
j
) the probability of the joint occurrence of i and j. Denote the conditional probability of i given j by p
(
i
∣
j
) and conditional probability of j given i by p
(
j
∣
i
).Definition 8.
The joint entropy ofx and y is defined as(7)
H
x
,
y
=
-
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
i
,
j
.Definition 9.
The conditional entropy ofy, H
x
y, is defined as(8)
H
x
y
=
-
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
j
∣
i
=
-
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
i
,
j
∑
j
p
i
,
j
.The conditional entropy of x, H
y
x, can be defined similarly.Then the following relation holds:(9)
H
x
,
y
=
H
x
+
H
x
y
=
H
y
+
H
y
x
,
H
x
-
H
y
x
=
H
y
-
H
x
y
=
H
x
+
H
y
-
H
x
,
y
.Definition 10.
The rate of transmission of informationR is defined as the difference between H
x and H
y
x. Then R can be written in two other forms: (10)
R
=
H
x
-
H
y
x
=
H
y
-
H
x
y
=
H
x
+
H
y
-
H
x
,
y
.Remark 11.
Shannon did not derive the explicit formula forR:(11)
R
=
∑
i
,
j
p
i
,
j
log
⁡
⁡
p
i
,
j
p
i
p
(
j
)
.However, he did imply it in Appendix 7 [4].


## 3.2. Class 1 Definitions

### 3.2.1. Kullback’s Definition

Kullback [5] redefined entropy more mathematically in a standalone homework question as follows. Consider two discrete random variables x
,
y, where (12)
p
i
j
=
Prob
x
=
x
i
,
y
=
y
j
>
0
,
i
=
1
,
2
,
…
,
m
,
j
=
1
,
2
,
…
,
n
,
p
i
·
=
∑
j
=
1
n
p
i
j
,
p
·
j
=
∑
i
=
1
m
p
i
j
,
∑
i
=
1
m
∑
j
=
1
n
p
i
j
=
∑
i
=
1
m
p
i
·
=
∑
j
=
1
n
p
·
j
=
1
.Define the joint entropy, entropy, and conditional entropy as follows: (13)
H
x
,
y
=
-
∑
i
∑
j
p
i
j
log
⁡
⁡
p
i
j
,
H
x
=
-
∑
i
p
i
·
log
⁡
⁡
p
i
·
,
H
y
=
-
∑
j
p
·
j
log
⁡
⁡
p
·
j
,
H
y
∣
x
i
=
-
∑
j
p
i
j
p
i
·
log
⁡
⁡
p
i
j
p
i
·
,
H
y
∣
x
=
∑
i
p
i
H
y
∣
x
i
=
-
∑
i
∑
j
p
i
j
log
⁡
⁡
p
i
j
p
i
·
.Then H
x
,
y
=
H
x
+
H
y
∣
x
≤
H
x
+
H
(
y
) and H
y
≥
H
y
∣
x.


### 3.2.2. Information Conveyed

Ash [7] began with two random variables X and Y and assumed X and Y had the same probability space. He systematically defined the entropy, conditional entropy, and joint entropy following Shannon’s path in [4]. At the end, he denoted H
X
-
H
(
X
∣
Y
) by I
(
X
∣
Y
) and called it the information conveyed about X by Y.


### 3.2.3. Information of One Variable with respect to the Other

Pinsker [6] treated the fundamental concepts of Shannon in a more advanced manner by employing probability theory. Suppose ξ is a random variable defined on a probability space Ω
,
S
ω
,
P
ξ and is taking values in a measurable space X
,
S
x and η is a random variable defined on a probability space Ψ
,
S
ψ
,
P
η and is taking values in a measurable space Y
,
S
y. Then, the pair ξ, η of random variables may be regarded as a single random variable (
ξ
,
η
) with values in the product space X
×
Y of all pairs (
x
,
y
) with x
∈
X, y
∈
Y
. The distribution P
ξ
,
η
·
=
P
ξ
η
(
·
) of (
ξ
,
η
) is called the joint distribution of random variables ξ and η. By the product of the distributions P
ξ
(
·
) and P
η
(
·
), denoted by P
ξ
×
η
(
·
), we mean the distribution defined on S
x
×
S
y
(14)
P
ξ
×
η
E
×
F
=
P
ξ
(
E
)
×
P
η
(
F
)for E
∈
S
x and F
∈
S
y. If the joint distribution P
ξ
η
(
·
) coincides with the product distribution P
ξ
×
η
·
, the random variables ξ and η are said to be independent. If ξ and η are discrete random variables, say, X and Y contain countably many points, x
1
,
x
2
,
… and y
1
,
y
2
,
…, then(15)
I
ξ
,
η
=
∑
i
,
j
P
ξ
×
η
(
x
i
,
y
j
)
log
⁡
⁡
P
ξ
η
(
x
i
,
y
j
)
P
ξ
(
x
i
)
P
η
(
y
j
)
.
I is called the information of ξ and η with respect to the other.


### 3.2.4. A Modern Definition in Information Theory

Of the various definitions of mutual information, the most widely accepted, of recent years, is the one by Cover and Thomas [8].LetX be a discrete variable with alphabet χ and probability mass function p
x
=
Pr
⁡
⁡
X
=
x, x
∈
χ. Let Y be a discrete variable with alphabet Υ and probability mass function p
y
=
Pr
⁡
⁡
Y
=
y, y
∈
Υ
. Suppose X and Y have a joint mass function (joint distribution) p
x
,
y. Then the mutual information I
(
X
,
Y
) can be defined as (16)
I
X
,
Y
=
∑
x
∈
χ
∑
y
∈
Υ
p
(
x
,
y
)
log
⁡
⁡
p
(
x
,
y
)
p
x
p
(
y
)
.


## 3.2.1. Kullback’s Definition

Kullback [5] redefined entropy more mathematically in a standalone homework question as follows. Consider two discrete random variables x
,
y, where (12)
p
i
j
=
Prob
x
=
x
i
,
y
=
y
j
>
0
,
i
=
1
,
2
,
…
,
m
,
j
=
1
,
2
,
…
,
n
,
p
i
·
=
∑
j
=
1
n
p
i
j
,
p
·
j
=
∑
i
=
1
m
p
i
j
,
∑
i
=
1
m
∑
j
=
1
n
p
i
j
=
∑
i
=
1
m
p
i
·
=
∑
j
=
1
n
p
·
j
=
1
.Define the joint entropy, entropy, and conditional entropy as follows: (13)
H
x
,
y
=
-
∑
i
∑
j
p
i
j
log
⁡
⁡
p
i
j
,
H
x
=
-
∑
i
p
i
·
log
⁡
⁡
p
i
·
,
H
y
=
-
∑
j
p
·
j
log
⁡
⁡
p
·
j
,
H
y
∣
x
i
=
-
∑
j
p
i
j
p
i
·
log
⁡
⁡
p
i
j
p
i
·
,
H
y
∣
x
=
∑
i
p
i
H
y
∣
x
i
=
-
∑
i
∑
j
p
i
j
log
⁡
⁡
p
i
j
p
i
·
.Then H
x
,
y
=
H
x
+
H
y
∣
x
≤
H
x
+
H
(
y
) and H
y
≥
H
y
∣
x.


## 3.2.2. Information Conveyed

Ash [7] began with two random variables X and Y and assumed X and Y had the same probability space. He systematically defined the entropy, conditional entropy, and joint entropy following Shannon’s path in [4]. At the end, he denoted H
X
-
H
(
X
∣
Y
) by I
(
X
∣
Y
) and called it the information conveyed about X by Y.


## 3.2.3. Information of One Variable with respect to the Other

Pinsker [6] treated the fundamental concepts of Shannon in a more advanced manner by employing probability theory. Suppose ξ is a random variable defined on a probability space Ω
,
S
ω
,
P
ξ and is taking values in a measurable space X
,
S
x and η is a random variable defined on a probability space Ψ
,
S
ψ
,
P
η and is taking values in a measurable space Y
,
S
y. Then, the pair ξ, η of random variables may be regarded as a single random variable (
ξ
,
η
) with values in the product space X
×
Y of all pairs (
x
,
y
) with x
∈
X, y
∈
Y
. The distribution P
ξ
,
η
·
=
P
ξ
η
(
·
) of (
ξ
,
η
) is called the joint distribution of random variables ξ and η. By the product of the distributions P
ξ
(
·
) and P
η
(
·
), denoted by P
ξ
×
η
(
·
), we mean the distribution defined on S
x
×
S
y
(14)
P
ξ
×
η
E
×
F
=
P
ξ
(
E
)
×
P
η
(
F
)for E
∈
S
x and F
∈
S
y. If the joint distribution P
ξ
η
(
·
) coincides with the product distribution P
ξ
×
η
·
, the random variables ξ and η are said to be independent. If ξ and η are discrete random variables, say, X and Y contain countably many points, x
1
,
x
2
,
… and y
1
,
y
2
,
…, then(15)
I
ξ
,
η
=
∑
i
,
j
P
ξ
×
η
(
x
i
,
y
j
)
log
⁡
⁡
P
ξ
η
(
x
i
,
y
j
)
P
ξ
(
x
i
)
P
η
(
y
j
)
.
I is called the information of ξ and η with respect to the other.


## 3.2.4. A Modern Definition in Information Theory

Of the various definitions of mutual information, the most widely accepted, of recent years, is the one by Cover and Thomas [8].LetX be a discrete variable with alphabet χ and probability mass function p
x
=
Pr
⁡
⁡
X
=
x, x
∈
χ. Let Y be a discrete variable with alphabet Υ and probability mass function p
y
=
Pr
⁡
⁡
Y
=
y, y
∈
Υ
. Suppose X and Y have a joint mass function (joint distribution) p
x
,
y. Then the mutual information I
(
X
,
Y
) can be defined as (16)
I
X
,
Y
=
∑
x
∈
χ
∑
y
∈
Υ
p
(
x
,
y
)
log
⁡
⁡
p
(
x
,
y
)
p
x
p
(
y
)
.


## 3.3. Class 2 Definitions

In Class 2 definitions, random variables are replaced by ensembles and mutual information is so-called average mutual information. Gallager [11] adopted a more general and more rigorous approach to introduce the concept of mutual information in communication theory. Indeed, he combined and compiled the results from Fano [9] and Abramson [10].Suppose that discrete ensembleX has a sample space a
1
,
a
2
,
…
,
a
K and discrete ensemble Y has a sample space b
1
,
b
2
,
…
,
b
L. Consider the joint sample space  a
k
,
b
j
:
1
≤
k
≤
K
,
1
≤
j
≤
J. A probability measure on the joint sample space is given by the join probability P
X
Y
a
k
,
b
j, defined for 1
≤
k
≤
K, 1
≤
j
≤
J. The combination of a joint sample space and probability measure for outcomes x and y is called a joint X
Yensemble. Then the marginal probabilities can be found as(17)
P
X
a
k
=
∑
j
=
1
J
P
X
Y
a
k
,
b
j
,
k
=
1
,
2
,
…
,
K
.In more abbreviated notation, this is written as(18)
P
x
=
∑
y
P
x
,
y
.Likewise, (19)
P
Y
b
j
=
∑
k
=
1
K
P
X
Y
(
a
k
,
b
j
)
,
j
=
1
,
2
,
…
,
J
.In more abbreviated notation, this is written as(20)
P
y
=
∑
x
P
x
,
y
.If P
X
a
k
>
0, the conditional probability that outcome y is b
j, given that outcome of x is a
k, is defined as(21)
P
Y
∣
X
b
j
∣
a
k
=
P
X
Y
a
k
,
b
j
P
X
a
k
.The mutual information between events x
=
a
k and y
=
b
j is defined as(22)
I
X
:
Y
a
k
,
b
j
=
log
⁡
P
X
∣
Y
a
k
∣
b
j
P
X
a
k
=
log
⁡
P
X
Y
a
k
,
b
j
P
X
a
k
P
Y
b
j
=
log
⁡
P
X
∣
Y
a
k
∣
b
j
P
X
a
k
=
I
Y
:
X
b
j
,
a
k
.Since the mutual information defined above is a random variable on the X
Y joint ensemble, the mean value, which is called the average mutual information denoted by I
(
X
,
Y
), is given by(23)
I
X
,
Y
=
∑
k
=
1
K
∑
j
=
1
L
P
X
Y
a
k
,
b
j
log
⁡
⁡
P
X
Y
a
k
,
b
j
P
X
a
k
P
Y
(
b
j
)
.Remark 12.
By means of an information channel consisting of a transmitter of alphabetA with elements a
i and total elements t and a receiver of alphabet B with elements b
j and total elements r, Abramson [10] denoted H
A
-
H
(
A
∣
B
)
=
∑
A
,
B
P
a
,
b
log
⁡
⁡
(
P
(
a
,
b
)
/
P
a
P
(
b
)
) by I
(
A
,
B
) and called it mutual information of A and B.The mutual informationI
X
,
Y between 2 continuous random variables X and Y [8] (also called rate of transmission in [1]) is defined as (24)
I
X
,
Y
=
∬
P
(
x
,
y
)
log
⁡
⁡
P
(
x
,
y
)
P
x
P
(
y
)
d
x
d
y
,where P
(
x
,
y
) is the joint probability density function of X and Y and P
(
x
) and P
(
y
) are the marginal density functions associated with X and Y, respectively. The mutual information between 2 continuous random variables is also called the differential mutual information.However, the differential mutual information is much less popular than its discrete counterpart. On the one hand, the joint density function involved is unknown in most cases and hence must be estimated [13, 14]. On the other hand, data in engineering and machine learning are mostly finite, and so mutual information between discrete random variables is used.


## 4. A New Unified Definition of Mutual Information

In Section3, we reviewed various definitions of mutual information. Shannon’s original definition laid the foundation of information theory. Kullback’s definition used random variables for the first time and was more mathematical and more compact. Although Ash’s definition followed Shannon’s path, it was more systematic. Pinsker’s definition was most mathematical in that it employed probability theory. Gallager’s definition was more general and more rigorous in communication theory. Cover and Thomas’s definition is so succinct that it is now a standard definition in information theory.However, there are some mathematical flaws in these various definitions of mutual information. Class 2 definitions redefine marginal probabilities from the joint probabilities. As a matter of fact, the marginal probabilities are given from the ensembles and hence should not be redefined from the joint probabilities. Except Pinsker’s definition, Class 2 definitions either neglect the probability spaces or assume the two random variables have the same probability space. Both Class 1 definitions and Class 2 definitions assume a joint distribution or a joint probability measure exists. Yet, they all ignore an important fact that the joint distribution or the joint probability measure is not unique.


### 4.1. Unified Definition of Mutual Information

LetX be a finite discrete random variable on discrete probability space Ω
1
,
F
1
,
P
1 with Ω
1
=
ω
1
,
ω
2
,
…
,
ω
n and range x
1
,
x
2
,
…
,
x
K with K
≤
n. Let Y be a discrete random variable on probability space Ω
2
,
F
2
,
P
2 with Ω
2
=
ρ
1
,
ρ
2
,
…
,
ρ
m and range y
1
,
y
2
,
…
,
y
L with L
≤
m
.IfX and Y have the same probability space (
Ω
,
F
,
P
), then the joint distribution is simply (25)
P
X
Y
X
=
x
,
Y
=
y
=
P
ω
∈
Ω
:
X
ω
=
x
,
Y
ω
=
y
.However, when X and Y have different probability spaces and so different probability measures, the joint distribution is more complicated.Definition 13.
The joint sample space of random variablesX and Y is defined as the product Ω
1
×
Ω
2 of all pairs ω
i
,
ρ
j, i
=
1
,
2
,
…
,
n and j
=
1
,
2
,
…
,
m. The joint σ-field F
1
×
F
2 is defined as the product of all pairs A
1
,
A
2, where A
1 and A
2 are elements of F
1 and F
2
, respectively. A joint probability measure P
X
Y of P
1 and P
2 is a probability measure on F
1
×
F
2
:
P
X
Y
(
A
×
B
) such that for any A
⊆
Ω
1 and B
⊆
Ω
2
(26)
P
1
A
=
P
X
Y
A
×
Ω
2
=
∑
j
=
1
m
P
X
Y
A
×
ρ
j
,
P
2
B
=
P
X
Y
Ω
1
×
B
=
∑
i
=
1
m
P
X
Y
ω
i
×
B
,
Ω
1
×
Ω
2
,
F
1
×
F
2
,
P
X
Y is called the joint probability space of X and Y, and P
X
Y
(
X
=
x
i
×
Y
=
y
j
) for i
=
1
,
2
,
…
,
n and j
=
1
,
2
,
…
,
m the joint distribution of X and Y.Combining Definitions2 and 13, we immediately obtain the following results.Proposition 14.
A sequence of nonnegative numbers{
p
i
j
:
1
≤
i
≤
K
;
1
≤
j
≤
L
} whose sum is 1 can serve as a probability measure on F
1
×
F
2
:
P
X
Y
ω
i
,
ρ
j
=
p
i
j. The probability of any event A
×
B
⊆
Ω
1
×
Ω
2 is computed simply by adding the probabilities of the individual points of ω
,
ρ
∈
A
×
B. If, in addition, for i
=
1
,
2
,
…
,
K and j
=
1
,
2
,
…
,
L, the following hold:(27)
∑
j
=
1
L
p
i
j
=
P
X
ω
i
,
∑
i
=
1
K
p
i
j
=
P
Y
ρ
j
,then P
X
Y
ω
i
,
ρ
j
=
p
i
j is a joint distribution of X and Y.For convenience, from now on we will shortenP
X
Y
(
X
=
x
i
×
Y
=
y
j
) as P
X
Y
x
i
,
y
j.This two-dimensional measure should not be confused with one-dimensional joint distribution whenX and Y have the same probability space.Remark 15.
IfΩ
1
,
F
1
,
P
1
=
Ω
2
,
F
2
,
P
2, instead of using the two-dimensional measure P
X
Y
X
=
x
i
×
Y
=
y
j, we may use the one-dimensional measure P
1
X
=
x
i
and
Y
=
y
j. Then, (26) always hold. In this sense, our new definition of joint distribution reduces to the definition of joint distribution with the same probability space.Definition 16.
The conditional probabilityY
=
y
j, given X
=
x
i, is defined as (28)
P
Y
∣
X
Y
=
y
j
∣
X
=
x
i
=
P
X
Y
x
i
,
y
j
P
1
(
X
=
x
i
)
.Theorem 17.
For any two discrete random variables, there is at least one joint probability measure called the product probability measure or simply product distribution.Proof.
Let random variablesX and Y be defined as before. Define a function from Ω
1
×
Ω
2 to [
0
,
1
] as follows: (29)
P
X
Y
ω
i
,
ρ
j
=
P
1
ω
i
P
2
ρ
j
.Then(30)
∑
i
=
1
n
∑
j
=
1
m
P
X
Y
ω
i
,
ρ
j
=
∑
i
=
1
n
∑
j
=
1
m
P
1
ω
i
P
2
ρ
j
=
∑
i
=
1
n
P
1
ω
i
∑
j
=
1
m
P
2
ρ
j
=
1
.Hence, P
X
Y can serve as a probability measure on F
1
×
F
2 by Definition 2. The probability of any event A
×
B
⊆
Ω
1
×
Ω
2 is computed simply by adding the probabilities of the individual points of ω
,
ρ
∈
A
×
B. Moreover, for any A
=
ω
i
1
,
ω
i
2
,
…
,
ω
i
s
∈
Ω
1 of s elements,(31)
P
X
Y
A
×
Ω
2
=
∑
j
=
1
m
P
X
Y
(
A
×
{
ρ
j
}
)
=
∑
j
=
1
m
∑
u
=
1
s
P
X
Y
ω
i
u
×
ρ
j
=
∑
j
=
1
m
∑
u
=
1
s
P
1
ω
i
u
P
2
ρ
j
=
∑
j
=
1
m
P
2
ρ
j
∑
u
=
1
s
P
1
ω
i
u
=
∑
u
=
1
s
P
1
(
ω
i
u
)
=
P
1
(
A
)
.Similarly, P
X
Y
Ω
1
×
B
=
P
2
(
B
) for any B
∈
Ω
2. Hence, P
X
Y
(
X
=
x
i
×
Y
=
y
j
)
=
P
1
(
X
=
x
i
)
P
2
(
Y
=
y
j
) is a joint probability measure of X and Y by Definition 13.Definition 18.
Random variablesX and Y are said to be independent under a joint distribution P
X
Y
(
·
) if P
X
Y
(
·
) coincides with the product distribution P
X
×
Y
·
.Definition 19.
The joint entropyH
(
X
,
Y
) is defined as (32)
H
X
,
Y
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
.Definition 20.
The conditional entropyH
Y
∣
X is as follows:(33)
H
Y
∣
X
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
Y
=
y
j
∣
X
=
x
i
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
.Definition 21.
The mutual informationI
(
X
,
Y
) between X and Y is defined as (34)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
P
2
Y
=
y
j
.As other measures in information theory, the base of logarithm in (34) is left unspecified. Indeed, I
(
X
,
Y
) under one base is proportional to that under another base by the change-of-base formula. Moreover, we take 0
log
⁡
⁡
0 to be 0. This corresponds to the limit of x
log
⁡
⁡
x as x goes to 0.It is obvious that our new definition covers Class 2 definitions. It also covers Class 1 definitions by the following arguments. LetΩ
1
=
a
1
,
a
2
,
…
,
a
K and Ω
2
=
b
1
,
b
2
,
…
,
b
L. Define random variables X
:
Ω
1
→
R and Y
:
Ω
2
→
R as one-to-one mappings as (35)
X
a
i
=
x
i
,
i
=
1
,
2
,
…
,
K
,
Y
b
j
=
y
j
,
j
=
1
,
2
,
…
,
L
.Then we have(36)
P
X
Y
x
i
,
y
j
=
P
X
Y
a
i
,
b
j
.It is worth noting that our new definition of mutual information has some advantages over various existing definitions. For instance, it can be easily used to do feature selection as seen later. In addition, our new definition leads different values for different joint distribution as demonstrated in the following example.Example 22.
Assumerandom variablesX and Y have the following probability distributions: (37)
P
1
Y
=
0
=
1
3
,
P
1
Y
=
1
=
2
3
,
P
2
X
=
1
=
1
3
,
P
2
X
=
2
=
1
3
,
P
2
X
=
3
=
1
3
.We can generate four different joint probability distributions to lead 4 different values of mutual information. However, under all the existing definitions, a joint distribution must be given in order to find mutual information: (1)
P
1,0
=
0, P
1
,
1
=
1
/
3, P
2,0
=
1
/
3, P
2
,
1
=
0, P
3,0
=
0, P
3
,
1
=
1
/
3;
(2)
P
1,0
=
0, P
1
,
1
=
1
/
3, P
2,0
=
0, P
2
,
1
=
1
/
3, P
3,0
=
1
/
3, P
3
,
1
=
0;
(3)
P
1,0
=
1
/
3, P
1
,
1
=
0, P
2,0
=
0, P
2
,
1
=
1
/
3, P
3,0
=
0, P
3
,
1
=
1
/
3;
(4)
P
1,0
=
1
/
9, P
1
,
1
=
2
/
9, P
2,0
=
1
/
9, P
2
,
1
=
2
/
9, P
3,0
=
1
/
9, P
3
,
1
=
2
/
9.


### 4.2. Properties of Newly Defined Mutual Information

Before we discuss some properties of mutual information, we first introduce Kullback-Leibler distance [8].Definition 23.
The relative entropy or Kullback-Leibler distance between two discrete probability distributionsP
=
p
1
,
p
2
,
…
,
p
n and Q
=
q
1
,
q
2
,
…
,
q
n is defined as (38)
D
P
∥
Q
=
∑
i
p
i
log
⁡
⁡
p
i
q
i
.Lemma 24 (see [8]).
LetP and Q be two discrete probability distributions. Then D
P
∥
Q
≥
0 with equality if and only if p
i
=
q
i for all i.Remark 25.
The Kullback-Leibler distance is not a true distance between distributions since it is not symmetric and does not satisfy the triangle inequality either. Nevertheless, it is often useful to think of relative entropy as a “distance” between distributions.The following property shows that mutual information under a joint probability measure is the Kullback-Leibler distance between the joint distributionP
X
Y and the product distribution P
X
P
Y.Property 1.
Mutual information of random variablesX and Y is the Kullback-Leibler distance between the joint distribution P
X
Y and the product distribution P
1
P
2.Proof.
Using a mapping from 2-dimensional indices to one-dimensional index,(39)
i
,
j
⟶
i
-
1
∗
L
+
j
≜
n
for
i
=
1
,
…
,
K
,
j
=
1
,
2
,
…
,
L
,and using another mapping from one-dimensional index back to two-dimensional indices, (40)
i
=
n
L
,
j
=
n
-
i
-
1
∗
L
for
n
=
1
,
2
,
…
,
L
,
L
+
1
,
…
,
2
L
,
…
,
K
-
1
L
+
⋯
+
K
L
,we rewrite I
(
X
,
Y
) as (41)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
P
2
Y
=
y
j
=
∑
n
=
1
K
L
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
222
·
log
⁡
⁡
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L
.Since (42)
∑
n
=
1
K
L
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
=
1
,
∑
n
=
1
K
L
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L
=
∑
i
=
1
K
∑
j
=
1
L
P
1
X
=
x
i
P
2
Y
=
y
j
=
1
,we obtain(43)
I
X
,
Y
=
∑
n
=
1
K
L
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
222
·
log
⁡
⁡
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L
.Property 2.
LetX and Y be two discrete random variables. The mutual information between X and Y satisfies (44)
I
X
,
Y
≥
0with equality if and only if X and Y are independent.Proof.
Let us use the mappings between two-dimensional indices and one-dimensional index in the proof of Property1. By Lemma 24, I
X
,
Y
≥
0 with equality if and only if P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
=
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L for n
=
1
,
2
,
…
,
L
,
L
+
1
,
…
,
2
L
,
…
,
K
-
1
L
+
⋯
+
K, that is, P
X
Y
x
i
,
y
j
=
P
1
X
=
x
i
P
2
Y
=
y
j for i
=
1
,
…
,
K and j
=
1
,
2
,
…
,
L, or X and Y are independent.Corollary 26.
IfX is a constant random variable, that is, K
=
1, then, for any random variable Y, (45)
I
(
X
,
Y
)
=
0
.Proof.
Suppose the range ofX is a constant x and the sample space has only one point ω. Then, P
1
X
=
x
=
P
1
ω
=
1. For any j
=
1
,
2
,
…
,
L
,
(46)
P
X
Y
x
,
y
j
=
∑
i
=
1
1
P
X
Y
x
,
y
j
=
P
2
Y
=
y
j
=
P
1
X
=
x
P
2
Y
=
y
j
.Thus, X and Y are independent. By Property 2, I
(
X
,
Y
)
=
0.Lemma 27 (see [8]).
LetX be discrete random variables with K values. Then(47)
0
≤
H
X
≤
log
⁡
⁡
Kwith equality if and only if the K values are equally probable.Property 3.
LetX and Y be two discrete random variables. Then the following relationships among mutual information, entropy, and conditional entry hold:(48)
I
X
,
Y
=
H
X
-
H
X
∣
Y
=
H
Y
-
H
Y
∣
X
=
I
Y
,
X
.Proof.
Consider(49)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
P
2
Y
=
y
j
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
P
2
Y
=
y
j
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
2
Y
=
y
j
+
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
2
Y
=
y
j
-
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
-
∑
j
=
1
L
∑
i
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
2
Y
=
y
j
-
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
-
∑
j
=
1
L
P
2
Y
=
y
j
log
⁡
⁡
P
2
Y
=
y
j
-
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
H
Y
-
H
Y
∣
X
.Combining the above properties and noting that H
(
X
∣
Y
) and H
(
Y
∣
X
) are both nonnegative, we obtain the following properties.Property 4.
LetX and Y be 2 discrete random variables with K and L values, respectively. Then (50)
0
≤
I
X
,
Y
≤
H
Y
≤
log
⁡
⁡
L
,
0
≤
I
X
,
Y
≤
H
X
≤
log
⁡
⁡
K
.Moreover, I
(
X
,
Y
)
=
0 if and only if X and Y are independent.


## 4.1. Unified Definition of Mutual Information

LetX be a finite discrete random variable on discrete probability space Ω
1
,
F
1
,
P
1 with Ω
1
=
ω
1
,
ω
2
,
…
,
ω
n and range x
1
,
x
2
,
…
,
x
K with K
≤
n. Let Y be a discrete random variable on probability space Ω
2
,
F
2
,
P
2 with Ω
2
=
ρ
1
,
ρ
2
,
…
,
ρ
m and range y
1
,
y
2
,
…
,
y
L with L
≤
m
.IfX and Y have the same probability space (
Ω
,
F
,
P
), then the joint distribution is simply (25)
P
X
Y
X
=
x
,
Y
=
y
=
P
ω
∈
Ω
:
X
ω
=
x
,
Y
ω
=
y
.However, when X and Y have different probability spaces and so different probability measures, the joint distribution is more complicated.Definition 13.
The joint sample space of random variablesX and Y is defined as the product Ω
1
×
Ω
2 of all pairs ω
i
,
ρ
j, i
=
1
,
2
,
…
,
n and j
=
1
,
2
,
…
,
m. The joint σ-field F
1
×
F
2 is defined as the product of all pairs A
1
,
A
2, where A
1 and A
2 are elements of F
1 and F
2
, respectively. A joint probability measure P
X
Y of P
1 and P
2 is a probability measure on F
1
×
F
2
:
P
X
Y
(
A
×
B
) such that for any A
⊆
Ω
1 and B
⊆
Ω
2
(26)
P
1
A
=
P
X
Y
A
×
Ω
2
=
∑
j
=
1
m
P
X
Y
A
×
ρ
j
,
P
2
B
=
P
X
Y
Ω
1
×
B
=
∑
i
=
1
m
P
X
Y
ω
i
×
B
,
Ω
1
×
Ω
2
,
F
1
×
F
2
,
P
X
Y is called the joint probability space of X and Y, and P
X
Y
(
X
=
x
i
×
Y
=
y
j
) for i
=
1
,
2
,
…
,
n and j
=
1
,
2
,
…
,
m the joint distribution of X and Y.Combining Definitions2 and 13, we immediately obtain the following results.Proposition 14.
A sequence of nonnegative numbers{
p
i
j
:
1
≤
i
≤
K
;
1
≤
j
≤
L
} whose sum is 1 can serve as a probability measure on F
1
×
F
2
:
P
X
Y
ω
i
,
ρ
j
=
p
i
j. The probability of any event A
×
B
⊆
Ω
1
×
Ω
2 is computed simply by adding the probabilities of the individual points of ω
,
ρ
∈
A
×
B. If, in addition, for i
=
1
,
2
,
…
,
K and j
=
1
,
2
,
…
,
L, the following hold:(27)
∑
j
=
1
L
p
i
j
=
P
X
ω
i
,
∑
i
=
1
K
p
i
j
=
P
Y
ρ
j
,then P
X
Y
ω
i
,
ρ
j
=
p
i
j is a joint distribution of X and Y.For convenience, from now on we will shortenP
X
Y
(
X
=
x
i
×
Y
=
y
j
) as P
X
Y
x
i
,
y
j.This two-dimensional measure should not be confused with one-dimensional joint distribution whenX and Y have the same probability space.Remark 15.
IfΩ
1
,
F
1
,
P
1
=
Ω
2
,
F
2
,
P
2, instead of using the two-dimensional measure P
X
Y
X
=
x
i
×
Y
=
y
j, we may use the one-dimensional measure P
1
X
=
x
i
and
Y
=
y
j. Then, (26) always hold. In this sense, our new definition of joint distribution reduces to the definition of joint distribution with the same probability space.Definition 16.
The conditional probabilityY
=
y
j, given X
=
x
i, is defined as (28)
P
Y
∣
X
Y
=
y
j
∣
X
=
x
i
=
P
X
Y
x
i
,
y
j
P
1
(
X
=
x
i
)
.Theorem 17.
For any two discrete random variables, there is at least one joint probability measure called the product probability measure or simply product distribution.Proof.
Let random variablesX and Y be defined as before. Define a function from Ω
1
×
Ω
2 to [
0
,
1
] as follows: (29)
P
X
Y
ω
i
,
ρ
j
=
P
1
ω
i
P
2
ρ
j
.Then(30)
∑
i
=
1
n
∑
j
=
1
m
P
X
Y
ω
i
,
ρ
j
=
∑
i
=
1
n
∑
j
=
1
m
P
1
ω
i
P
2
ρ
j
=
∑
i
=
1
n
P
1
ω
i
∑
j
=
1
m
P
2
ρ
j
=
1
.Hence, P
X
Y can serve as a probability measure on F
1
×
F
2 by Definition 2. The probability of any event A
×
B
⊆
Ω
1
×
Ω
2 is computed simply by adding the probabilities of the individual points of ω
,
ρ
∈
A
×
B. Moreover, for any A
=
ω
i
1
,
ω
i
2
,
…
,
ω
i
s
∈
Ω
1 of s elements,(31)
P
X
Y
A
×
Ω
2
=
∑
j
=
1
m
P
X
Y
(
A
×
{
ρ
j
}
)
=
∑
j
=
1
m
∑
u
=
1
s
P
X
Y
ω
i
u
×
ρ
j
=
∑
j
=
1
m
∑
u
=
1
s
P
1
ω
i
u
P
2
ρ
j
=
∑
j
=
1
m
P
2
ρ
j
∑
u
=
1
s
P
1
ω
i
u
=
∑
u
=
1
s
P
1
(
ω
i
u
)
=
P
1
(
A
)
.Similarly, P
X
Y
Ω
1
×
B
=
P
2
(
B
) for any B
∈
Ω
2. Hence, P
X
Y
(
X
=
x
i
×
Y
=
y
j
)
=
P
1
(
X
=
x
i
)
P
2
(
Y
=
y
j
) is a joint probability measure of X and Y by Definition 13.Definition 18.
Random variablesX and Y are said to be independent under a joint distribution P
X
Y
(
·
) if P
X
Y
(
·
) coincides with the product distribution P
X
×
Y
·
.Definition 19.
The joint entropyH
(
X
,
Y
) is defined as (32)
H
X
,
Y
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
.Definition 20.
The conditional entropyH
Y
∣
X is as follows:(33)
H
Y
∣
X
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
Y
=
y
j
∣
X
=
x
i
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
.Definition 21.
The mutual informationI
(
X
,
Y
) between X and Y is defined as (34)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
P
2
Y
=
y
j
.As other measures in information theory, the base of logarithm in (34) is left unspecified. Indeed, I
(
X
,
Y
) under one base is proportional to that under another base by the change-of-base formula. Moreover, we take 0
log
⁡
⁡
0 to be 0. This corresponds to the limit of x
log
⁡
⁡
x as x goes to 0.It is obvious that our new definition covers Class 2 definitions. It also covers Class 1 definitions by the following arguments. LetΩ
1
=
a
1
,
a
2
,
…
,
a
K and Ω
2
=
b
1
,
b
2
,
…
,
b
L. Define random variables X
:
Ω
1
→
R and Y
:
Ω
2
→
R as one-to-one mappings as (35)
X
a
i
=
x
i
,
i
=
1
,
2
,
…
,
K
,
Y
b
j
=
y
j
,
j
=
1
,
2
,
…
,
L
.Then we have(36)
P
X
Y
x
i
,
y
j
=
P
X
Y
a
i
,
b
j
.It is worth noting that our new definition of mutual information has some advantages over various existing definitions. For instance, it can be easily used to do feature selection as seen later. In addition, our new definition leads different values for different joint distribution as demonstrated in the following example.Example 22.
Assumerandom variablesX and Y have the following probability distributions: (37)
P
1
Y
=
0
=
1
3
,
P
1
Y
=
1
=
2
3
,
P
2
X
=
1
=
1
3
,
P
2
X
=
2
=
1
3
,
P
2
X
=
3
=
1
3
.We can generate four different joint probability distributions to lead 4 different values of mutual information. However, under all the existing definitions, a joint distribution must be given in order to find mutual information: (1)
P
1,0
=
0, P
1
,
1
=
1
/
3, P
2,0
=
1
/
3, P
2
,
1
=
0, P
3,0
=
0, P
3
,
1
=
1
/
3;
(2)
P
1,0
=
0, P
1
,
1
=
1
/
3, P
2,0
=
0, P
2
,
1
=
1
/
3, P
3,0
=
1
/
3, P
3
,
1
=
0;
(3)
P
1,0
=
1
/
3, P
1
,
1
=
0, P
2,0
=
0, P
2
,
1
=
1
/
3, P
3,0
=
0, P
3
,
1
=
1
/
3;
(4)
P
1,0
=
1
/
9, P
1
,
1
=
2
/
9, P
2,0
=
1
/
9, P
2
,
1
=
2
/
9, P
3,0
=
1
/
9, P
3
,
1
=
2
/
9.


## 4.2. Properties of Newly Defined Mutual Information

Before we discuss some properties of mutual information, we first introduce Kullback-Leibler distance [8].Definition 23.
The relative entropy or Kullback-Leibler distance between two discrete probability distributionsP
=
p
1
,
p
2
,
…
,
p
n and Q
=
q
1
,
q
2
,
…
,
q
n is defined as (38)
D
P
∥
Q
=
∑
i
p
i
log
⁡
⁡
p
i
q
i
.Lemma 24 (see [8]).
LetP and Q be two discrete probability distributions. Then D
P
∥
Q
≥
0 with equality if and only if p
i
=
q
i for all i.Remark 25.
The Kullback-Leibler distance is not a true distance between distributions since it is not symmetric and does not satisfy the triangle inequality either. Nevertheless, it is often useful to think of relative entropy as a “distance” between distributions.The following property shows that mutual information under a joint probability measure is the Kullback-Leibler distance between the joint distributionP
X
Y and the product distribution P
X
P
Y.Property 1.
Mutual information of random variablesX and Y is the Kullback-Leibler distance between the joint distribution P
X
Y and the product distribution P
1
P
2.Proof.
Using a mapping from 2-dimensional indices to one-dimensional index,(39)
i
,
j
⟶
i
-
1
∗
L
+
j
≜
n
for
i
=
1
,
…
,
K
,
j
=
1
,
2
,
…
,
L
,and using another mapping from one-dimensional index back to two-dimensional indices, (40)
i
=
n
L
,
j
=
n
-
i
-
1
∗
L
for
n
=
1
,
2
,
…
,
L
,
L
+
1
,
…
,
2
L
,
…
,
K
-
1
L
+
⋯
+
K
L
,we rewrite I
(
X
,
Y
) as (41)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
P
2
Y
=
y
j
=
∑
n
=
1
K
L
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
222
·
log
⁡
⁡
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L
.Since (42)
∑
n
=
1
K
L
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
=
1
,
∑
n
=
1
K
L
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L
=
∑
i
=
1
K
∑
j
=
1
L
P
1
X
=
x
i
P
2
Y
=
y
j
=
1
,we obtain(43)
I
X
,
Y
=
∑
n
=
1
K
L
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
222
·
log
⁡
⁡
P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L
.Property 2.
LetX and Y be two discrete random variables. The mutual information between X and Y satisfies (44)
I
X
,
Y
≥
0with equality if and only if X and Y are independent.Proof.
Let us use the mappings between two-dimensional indices and one-dimensional index in the proof of Property1. By Lemma 24, I
X
,
Y
≥
0 with equality if and only if P
X
Y
x
n
/
L
,
y
n
-
n
/
L
-
1
∗
L
=
P
1
X
=
x
n
/
L
P
2
Y
=
y
n
-
n
/
L
-
1
∗
L for n
=
1
,
2
,
…
,
L
,
L
+
1
,
…
,
2
L
,
…
,
K
-
1
L
+
⋯
+
K, that is, P
X
Y
x
i
,
y
j
=
P
1
X
=
x
i
P
2
Y
=
y
j for i
=
1
,
…
,
K and j
=
1
,
2
,
…
,
L, or X and Y are independent.Corollary 26.
IfX is a constant random variable, that is, K
=
1, then, for any random variable Y, (45)
I
(
X
,
Y
)
=
0
.Proof.
Suppose the range ofX is a constant x and the sample space has only one point ω. Then, P
1
X
=
x
=
P
1
ω
=
1. For any j
=
1
,
2
,
…
,
L
,
(46)
P
X
Y
x
,
y
j
=
∑
i
=
1
1
P
X
Y
x
,
y
j
=
P
2
Y
=
y
j
=
P
1
X
=
x
P
2
Y
=
y
j
.Thus, X and Y are independent. By Property 2, I
(
X
,
Y
)
=
0.Lemma 27 (see [8]).
LetX be discrete random variables with K values. Then(47)
0
≤
H
X
≤
log
⁡
⁡
Kwith equality if and only if the K values are equally probable.Property 3.
LetX and Y be two discrete random variables. Then the following relationships among mutual information, entropy, and conditional entry hold:(48)
I
X
,
Y
=
H
X
-
H
X
∣
Y
=
H
Y
-
H
Y
∣
X
=
I
Y
,
X
.Proof.
Consider(49)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
X
Y
x
i
,
y
j
P
1
X
=
x
i
P
2
Y
=
y
j
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
P
2
Y
=
y
j
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
2
Y
=
y
j
+
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
2
Y
=
y
j
-
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
-
∑
j
=
1
L
∑
i
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
2
Y
=
y
j
-
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
-
∑
j
=
1
L
P
2
Y
=
y
j
log
⁡
⁡
P
2
Y
=
y
j
-
-
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
x
i
,
y
j
log
⁡
⁡
P
Y
∣
X
x
i
,
y
j
=
H
Y
-
H
Y
∣
X
.Combining the above properties and noting that H
(
X
∣
Y
) and H
(
Y
∣
X
) are both nonnegative, we obtain the following properties.Property 4.
LetX and Y be 2 discrete random variables with K and L values, respectively. Then (50)
0
≤
I
X
,
Y
≤
H
Y
≤
log
⁡
⁡
L
,
0
≤
I
X
,
Y
≤
H
X
≤
log
⁡
⁡
K
.Moreover, I
(
X
,
Y
)
=
0 if and only if X and Y are independent.


## 5. Newly Defined Mutual Information in Machine Learning

Machine learning is the science of getting machines (computers) to automatically learn from data. In a typical learning setting, a training setS contains N examples (also known as samples, observations, or records) from an input space X
=
X
1
,
X
2
,
…
,
X
M and their associated output values y from an output space Y (i.e., dependent variable). Here, X
1
,
X
2
,
…
,
X
M are called features, that is, independent variables. Hence, S can be expressed as (51)
S
=
x
i
1
,
x
i
2
,
…
,
x
i
M
,
y
i
,
i
=
1
,
2
,
…
,
N
,where feature X
j has values x
1
j
,
x
2
j
,
…
,
x
N
j for j
=
1
,
2
,
…
,
M.A fundamental objective in machine learning is to find a functional relationship between inputX and output Y. In general, there are a very large number of features, many of which are not needed. Sometimes, the output Y is not determined by the complete set of the input features X
1
,
X
2
,
…
,
X
M. Rather, it is decided by only a subset of them. This kind of reduction is called feature selection. Its purpose is to choose a subset of features to capture the relevant information. An easy and natural way for feature selection is as follows.(1)
Evaluate the relationship between each individual input featurex
i and the output Y.
(2)
Select the best set of attributes according to some criterion.


### 5.1.  Calculation of Newly Defined Mutual Information

Since mutual information measures dependency between random variables, we may use it to do feature selection in machine learning. Let us calculate mutual information between an input featureX and output Y. Assume X has K different values ω
1
,
ω
2
,
…
,
ω
K. If X has missing values, we will use ω
1 to represent all the missing values. Assume Y has L different values ρ
1
,
ρ
2
,
…
,
ρ
L.Let us build a two-way frequency or contingency table by makingX as the row variable and Y as the column variable like in [8]. Let O
i
j be the frequency (could be 0) of (ω
i
,
ρ
j) for i
=
1 to K and j
=
1 to L. Let the row and column marginal totals be n
i
· and n
·
j, respectively. Then (52)
n
i
·
=
∑
j
O
i
j
,
n
·
j
=
∑
i
O
i
j
,
N
=
∑
i
∑
j
O
i
j
=
∑
i
n
i
·
=
∑
j
n
·
j
.Let us denote the relative frequency O
i
j
/
N by p
i
j. We have the two-way relative frequency table; see Table 2.Since(53)
∑
i
=
1
K
∑
j
=
1
L
p
i
j
=
∑
i
=
1
K
p
i
·
=
∑
j
=
1
L
p
·
j
=
1
,
p
i
·
i
=
1
K, p
·
j
j
=
1
L, and p
i
j
i
=
1
K can each serve as a probability measure.Now we can define random variables forX and Y as follows. For convenience, we will use the same names X and Y for the random variables (54)
X
:
Ω
1
,
F
1
,
P
X
⟶
Ras X
ω
i
=
x
i, where Ω
1
=
ω
1
,
ω
2
,
…
,
ω
K and P
X
(
ω
i
)
=
n
i
·
/
N
=
p
i
· for i
=
1
,
2
,
…
,
K. Note that x
1
,
x
2
,
…
,
x
K could be any real numbers as long as they are distinct to guarantee that X is a one-to-one mapping. In this case, P
X
(
X
=
x
i
)
=
P
X
(
ω
i
).Similarly,(55)
Y
:
Ω
2
,
F
2
,
P
2
⟶
Ras Y
ρ
j
=
y
j, where Ω
2
=
ρ
1
,
ρ
2
,
…
,
ρ
L and P
Y
(
ρ
j
)
=
n
·
j
/
N
=
p
·
j for j
=
1
,
2
,
…
,
L. Also, y
1
,
y
2
,
…
,
y
K could be any real numbers as long as they are distinct to guarantee that Y is a one-to-one mapping. In this case, P
Y
(
Y
=
y
j
)
=
P
Y
(
ρ
j
).Now define a mappingP
X
Y from Ω
1
×
Ω
2 to R as follows:(56)
P
X
Y
ω
i
,
ρ
j
=
p
i
j
=
O
i
j
N
.Since (57)
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
ω
i
,
ρ
j
=
1
,
∑
j
=
1
L
P
X
Y
ω
i
,
ρ
j
=
∑
j
=
1
L
p
i
j
=
p
i
·
=
P
X
ω
i
,
∑
i
=
1
K
P
X
Y
ω
i
,
ρ
j
=
∑
i
=
1
K
p
i
j
=
p
·
j
=
P
Y
ρ
j
,
p
i
j
i
=
1
K is a joint probability measure by Proposition 14. Finally, we can calculate mutual information as follows: (58)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
ω
i
,
ρ
j
log
⁡
⁡
P
X
Y
ω
i
,
ρ
j
P
X
ω
i
P
Y
ρ
j
=
∑
i
=
1
K
∑
j
=
1
L
p
i
j
log
⁡
⁡
p
i
j
p
i
·
p
·
j
.It follows from Corollary 26 that if X has only one value, then I
(
X
,
Y
)
=
0. On the other hand, if X has all distinct values, the following result shows that mutual information will reach the maximum value.Proposition 28.
If all the values ofX are distinct, then I
(
X
,
Y
)
=
H
(
Y
).Proof.
If all the values ofX are distinct, then the number of different values of X equals the number of observations; that is, K
=
N. From Tables 1 and 2, we observe that(1)
O
i
j
=
0 or 1 for all i
=
1
,
2
,
…
,
K and j
=
1
,
2
,
…
,
L,
(2)
p
i
j
=
O
i
j
/
N
=
0 or 1
/
N for all i
=
1
,
2
,
…
,
K and j
=
1
,
2
,
…
,
L,
(3)
for eachj
=
1
,
2
,
…
,
L, since O
1
j
+
O
2
j
+
⋯
+
O
K
j
=
n
·
j, there are n
·
j nonzero O
i
j’s or equivalently n
·
j nonzero p
i
j’s,
(4)
p
i
·
=
1
/
N, i
=
1
,
2
,
…
,
K.Using the above observations and the fact that 0
log
⁡
0
=
0 we have(59)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
p
i
j
log
⁡
⁡
p
i
j
p
i
·
p
·
j
=
∑
i
=
1
K
p
i
1
log
⁡
⁡
p
i
1
p
i
·
p
·
1
+
∑
i
=
1
K
p
i
2
log
⁡
⁡
p
i
2
p
i
·
p
·
2
+
⋯
+
∑
i
=
1
K
p
i
L
log
⁡
⁡
p
i
L
p
i
·
p
·
L
=
∑
p
i
1
≠
0
1
N
log
⁡
⁡
1
/
N
p
·
1
/
N
+
∑
p
i
2
≠
0
1
N
log
⁡
⁡
1
/
N
p
·
2
/
N
+
⋯
+
∑
p
i
L
≠
0
1
N
log
⁡
⁡
1
/
N
p
·
L
/
N
=
∑
p
i
1
≠
0
1
N
log
⁡
⁡
1
p
·
1
+
∑
p
i
2
≠
0
1
N
log
⁡
⁡
1
p
·
2
+
⋯
+
∑
p
i
L
≠
0
1
N
log
⁡
⁡
1
p
·
L
=
n
·
1
N
log
⁡
⁡
1
p
·
1
+
n
·
2
N
log
⁡
⁡
1
p
·
2
+
⋯
+
n
·
L
N
log
⁡
⁡
1
p
·
L
=
p
·
1
log
⁡
⁡
1
p
·
1
+
p
·
2
log
⁡
⁡
1
p
·
2
+
⋯
+
p
·
L
log
⁡
⁡
1
p
·
L
=
H
Y
.Table 1
Frequency table.
ρ
1
ρ
2
⋯
ρ
j
⋯
ρ
L
Total
ω
1
O
11
O
12
⋯
O
1
j
⋯
O
1
L
n
1
∙
ω
2
O
21
O
22
⋯
O
2
j
⋯
O
2
L
n
2
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
i
O
i
1
O
i
2
⋯
O
i
j
⋯
O
i
L
n
i
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
K
O
K
1
O
K
2
⋯
O
K
j
⋯
O
K
L
n
K
∙
Total
n
∙
1
n
∙
2
⋯
n
∙
j
⋯
n
∙
L
NTable 2
Relative frequency table.
ρ
1
ρ
2
⋯
ρ
j
⋯
ρ
L
Total
ω
1
p
11
p
12
⋯
p
1
j
⋯
p
1
L
p
1
∙
ω
2
p
21
p
22
⋯
p
2
j
⋯
p
2
L
p
2
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
i
p
i
1
p
i
2
⋯
p
i
j
⋯
p
i
L
p
i
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
K
p
K
1
p
K
2
⋯
p
K
j
⋯
p
K
L
p
K
∙
Total
p
∙
1
p
∙
2
⋯
p
∙
j
⋯
p
∙
L
1


### 5.2. Applications of Newly Defined Mutual Information in Credit Scoring

Credit scoring is used to describe the process of evaluating the risk a customer poses of defaulting on a financial obligation [15–19]. The objective is to assign customers to one of two groups:good andbad. Machine learning has been successfully used to build models for credit scoring [20]. In credit scoring, Y is a binary variable,good andbad, and may be represented by 0 and 1.To apply mutual information to credit scoring, we first calculate mutual information for every pair of(
X
,
Y
) and then do feature selection based on values of mutual information. We propose three ways.


#### 5.2.1. Absolute Values Method

From Property4, we see that mutual information I
(
X
,
Y
) is nonnegative and upper bounded by log
⁡
(
L
) and that I
(
X
,
Y
)
=
0 if and only if X and Y are independent. In this sense, high mutual information indicates a large reduction in uncertainty, while low mutual information indicates a small reduction. In particular, zero mutual information means the two random variables are independent. Hence, we may select those features whose mutual information with Y is larger than some threshold based on needs.


#### 5.2.2. Relative Values

From Property4, we have 0
≤
I
(
X
,
Y
)
/
H
(
Y
)
≤
1. Note that I
(
X
,
Y
)
/
H
(
Y
) is relative mutual information, which measures how much information X catches from Y. Thus, we may select those features whose relative mutual information I
(
X
,
Y
)
/
H
(
Y
) is larger than some threshold between 0 and 1 based on needs.


#### 5.2.3. Chi-Square Test for Independency

For convenience, we will use the natural logarithm in mutual information. We first state an approximation formula for the natural logarithm function. It can be proved by the Taylor expansion like in Kullback’s book [5].Lemma 29.
Letp and q be two positive numbers less than or equal to 1. Then (60)
p
ln
⁡
⁡
p
q
≈
p
-
q
+
p
-
q
2
2
q
.The equality holds if and only if p
=
q. Moreover, the close p is to q, the better the approximation is.Now let us denoteN
×
I
(
X
,
Y
) by I
^
X
,
Y. Then, applying Lemma 29, we obtain(61)
2
I
^
X
,
Y
=
2
N
∑
i
=
1
K
∑
j
=
1
L
p
i
j
ln
⁡
⁡
p
i
j
p
i
·
p
·
j
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
ln
⁡
⁡
O
i
j
/
N
n
i
·
/
N
n
·
j
/
N
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
ln
⁡
⁡
O
i
j
n
i
·
n
·
j
/
N
≈
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
2
∑
i
n
i
·
N
∑
j
n
·
j
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
2
N
-
2
N
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
χ
2
.The last equation means the previous expression ∑
i
=
1
K
∑
j
=
1
L
(
O
i
j
-
n
i
·
n
·
j
/
N
)
2
/
(
n
i
·
n
·
j
/
N
) follows χ
2 distribution. According to [5], it follows χ
2 distribution with a degree of freedom of (
K
-
1
)
(
L
-
1
). Hence, 2
N
×
I
(
X
,
Y
) approximately follows χ
2 distribution with a degree of freedom of (
K
-
1
)
(
L
-
1
). This is the well-known Chi-square test for independence of two random variables. This allows using the Chi-square distribution to assign a significant level corresponding to the values of mutual information and (
K
-
1
)
(
L
-
1
).The null and alternative hypotheses are as follows.H
0
:X and Y are independent (i.e., there is no relationship between them).
H
1: X and Y are dependent (i.e., there is a relationship between them).The decision rule is to reject the null hypothesis at the α level of significance if the χ
2 statistic(62)
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
≈
2
N
×
I
(
X
,
Y
)is greater than χ
U
2, the upper-tail critical value from a Chi-square distribution with (
K
-
1
)
(
L
-
1
) degrees of freedom. That is,(63)
Select
feature
X
if
I
X
,
Y
>
χ
U
2
2
N
.Take credit scoring, for example. In this case, L
=
2. Assume feature X has 10 different values; that is, K
=
10. Using a level of significance of α
=
0.05
, we find χ
U
2 to be 16.9 from a Chi-square table with K
-
1
L
-
1
=
9, and select this feature only if I
X
,
Y
>
16.9
/
2
N.Assume a training set hasN examples. We can do feature selection by the following procedure.(i)
Step 1. Choose a level of significance ofα, say, 0.05.
(ii)
Step 2. FindK, the number of values of feature X.
(iii)
Step 3. Build the contingency table forX and Y.
(iv)
Step 4. CalculateI
(
X
,
Y
) from the contingency table.
(v)
Step 5. Findχ
U
2 with K
-
1
L
-
1 degrees of freedom from a Chi-square table or any other sources such as SAS.
(vi)
Step 6. SelectX if I
X
,
Y
>
16.9
/
2
N and discard it otherwise.
(vii)
Step 7. Repeat Steps 2–6 for all features.If the number of features selected from the above procedure is smaller or larger than what you want, you may adjust the level of significant α and reselect features using the procedure.


### 5.3. Adjustment of Mutual Information in Feature Selection

In Section5.2, we have proposed 3 ways to select feature based on mutual information. It seems that the larger the mutual information I
(
X
,
Y
), the more dependent X on Y. However, Proposition 28 says that if X has all distinct values, then I
(
X
,
Y
) will reach the maximum value H
(
Y
) and I
(
X
,
Y
)
/
H
(
Y
) will reach the maximum value 1.Therefore, ifX has too many different values, one may bin or group these values first. Based on the binned values, mutual information is calculated again. For numerical variables, we may adopt a three-step process.(i)
Step 1: select features by removing those with small mutual information.
(ii)
Step 2: do binning for the rest of numerical features.
(iii)
Step 3: select features by mutual information.


### 5.4.  Comparison with Existing Feature Selection Methods

There are many other feature selection methods in machine learning and credit scoring. An easy way is to build a logistic model for each feature with respect to the dependent variable and then select features withp values less than some specific values. However, this method does not apply to any nonlinear models in machine learning.Another easy way of feature selection is to calculate the covariance of each feature with respect to the dependent variable and then select features whose values are larger than some specific value. Yet, mutual information is better than the covariance method [21] in that mutual information measures the general dependence of random variables without making any assumptions about the nature of their underlying relationships.The most popular feature selection in credit scoring is done by information value [15–19]. To calculate information between an independent variable and the dependent variable, a binning algorithm is used to group similar attributes into a bin. Difference between the information of good accounts and that of bad accounts in each bin is then calculated. Finally, information value is calculated as the sum of information differences of all bins. Features with information value larger than 0.02 are believed to have strong predictive power. However, mutual information is a better measure than information value. Information value focuses only on the linear relationships of variables, whereas mutual information can potentially offer some advantages information value for nonlinear models such as gradient boosting model [22]. Moreover, information value depends on binning algorithms and the bin size. Different binning algorithms and/or different bin sizes will have different information value.


## 5.1.  Calculation of Newly Defined Mutual Information

Since mutual information measures dependency between random variables, we may use it to do feature selection in machine learning. Let us calculate mutual information between an input featureX and output Y. Assume X has K different values ω
1
,
ω
2
,
…
,
ω
K. If X has missing values, we will use ω
1 to represent all the missing values. Assume Y has L different values ρ
1
,
ρ
2
,
…
,
ρ
L.Let us build a two-way frequency or contingency table by makingX as the row variable and Y as the column variable like in [8]. Let O
i
j be the frequency (could be 0) of (ω
i
,
ρ
j) for i
=
1 to K and j
=
1 to L. Let the row and column marginal totals be n
i
· and n
·
j, respectively. Then (52)
n
i
·
=
∑
j
O
i
j
,
n
·
j
=
∑
i
O
i
j
,
N
=
∑
i
∑
j
O
i
j
=
∑
i
n
i
·
=
∑
j
n
·
j
.Let us denote the relative frequency O
i
j
/
N by p
i
j. We have the two-way relative frequency table; see Table 2.Since(53)
∑
i
=
1
K
∑
j
=
1
L
p
i
j
=
∑
i
=
1
K
p
i
·
=
∑
j
=
1
L
p
·
j
=
1
,
p
i
·
i
=
1
K, p
·
j
j
=
1
L, and p
i
j
i
=
1
K can each serve as a probability measure.Now we can define random variables forX and Y as follows. For convenience, we will use the same names X and Y for the random variables (54)
X
:
Ω
1
,
F
1
,
P
X
⟶
Ras X
ω
i
=
x
i, where Ω
1
=
ω
1
,
ω
2
,
…
,
ω
K and P
X
(
ω
i
)
=
n
i
·
/
N
=
p
i
· for i
=
1
,
2
,
…
,
K. Note that x
1
,
x
2
,
…
,
x
K could be any real numbers as long as they are distinct to guarantee that X is a one-to-one mapping. In this case, P
X
(
X
=
x
i
)
=
P
X
(
ω
i
).Similarly,(55)
Y
:
Ω
2
,
F
2
,
P
2
⟶
Ras Y
ρ
j
=
y
j, where Ω
2
=
ρ
1
,
ρ
2
,
…
,
ρ
L and P
Y
(
ρ
j
)
=
n
·
j
/
N
=
p
·
j for j
=
1
,
2
,
…
,
L. Also, y
1
,
y
2
,
…
,
y
K could be any real numbers as long as they are distinct to guarantee that Y is a one-to-one mapping. In this case, P
Y
(
Y
=
y
j
)
=
P
Y
(
ρ
j
).Now define a mappingP
X
Y from Ω
1
×
Ω
2 to R as follows:(56)
P
X
Y
ω
i
,
ρ
j
=
p
i
j
=
O
i
j
N
.Since (57)
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
ω
i
,
ρ
j
=
1
,
∑
j
=
1
L
P
X
Y
ω
i
,
ρ
j
=
∑
j
=
1
L
p
i
j
=
p
i
·
=
P
X
ω
i
,
∑
i
=
1
K
P
X
Y
ω
i
,
ρ
j
=
∑
i
=
1
K
p
i
j
=
p
·
j
=
P
Y
ρ
j
,
p
i
j
i
=
1
K is a joint probability measure by Proposition 14. Finally, we can calculate mutual information as follows: (58)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
P
X
Y
ω
i
,
ρ
j
log
⁡
⁡
P
X
Y
ω
i
,
ρ
j
P
X
ω
i
P
Y
ρ
j
=
∑
i
=
1
K
∑
j
=
1
L
p
i
j
log
⁡
⁡
p
i
j
p
i
·
p
·
j
.It follows from Corollary 26 that if X has only one value, then I
(
X
,
Y
)
=
0. On the other hand, if X has all distinct values, the following result shows that mutual information will reach the maximum value.Proposition 28.
If all the values ofX are distinct, then I
(
X
,
Y
)
=
H
(
Y
).Proof.
If all the values ofX are distinct, then the number of different values of X equals the number of observations; that is, K
=
N. From Tables 1 and 2, we observe that(1)
O
i
j
=
0 or 1 for all i
=
1
,
2
,
…
,
K and j
=
1
,
2
,
…
,
L,
(2)
p
i
j
=
O
i
j
/
N
=
0 or 1
/
N for all i
=
1
,
2
,
…
,
K and j
=
1
,
2
,
…
,
L,
(3)
for eachj
=
1
,
2
,
…
,
L, since O
1
j
+
O
2
j
+
⋯
+
O
K
j
=
n
·
j, there are n
·
j nonzero O
i
j’s or equivalently n
·
j nonzero p
i
j’s,
(4)
p
i
·
=
1
/
N, i
=
1
,
2
,
…
,
K.Using the above observations and the fact that 0
log
⁡
0
=
0 we have(59)
I
X
,
Y
=
∑
i
=
1
K
∑
j
=
1
L
p
i
j
log
⁡
⁡
p
i
j
p
i
·
p
·
j
=
∑
i
=
1
K
p
i
1
log
⁡
⁡
p
i
1
p
i
·
p
·
1
+
∑
i
=
1
K
p
i
2
log
⁡
⁡
p
i
2
p
i
·
p
·
2
+
⋯
+
∑
i
=
1
K
p
i
L
log
⁡
⁡
p
i
L
p
i
·
p
·
L
=
∑
p
i
1
≠
0
1
N
log
⁡
⁡
1
/
N
p
·
1
/
N
+
∑
p
i
2
≠
0
1
N
log
⁡
⁡
1
/
N
p
·
2
/
N
+
⋯
+
∑
p
i
L
≠
0
1
N
log
⁡
⁡
1
/
N
p
·
L
/
N
=
∑
p
i
1
≠
0
1
N
log
⁡
⁡
1
p
·
1
+
∑
p
i
2
≠
0
1
N
log
⁡
⁡
1
p
·
2
+
⋯
+
∑
p
i
L
≠
0
1
N
log
⁡
⁡
1
p
·
L
=
n
·
1
N
log
⁡
⁡
1
p
·
1
+
n
·
2
N
log
⁡
⁡
1
p
·
2
+
⋯
+
n
·
L
N
log
⁡
⁡
1
p
·
L
=
p
·
1
log
⁡
⁡
1
p
·
1
+
p
·
2
log
⁡
⁡
1
p
·
2
+
⋯
+
p
·
L
log
⁡
⁡
1
p
·
L
=
H
Y
.Table 1
Frequency table.
ρ
1
ρ
2
⋯
ρ
j
⋯
ρ
L
Total
ω
1
O
11
O
12
⋯
O
1
j
⋯
O
1
L
n
1
∙
ω
2
O
21
O
22
⋯
O
2
j
⋯
O
2
L
n
2
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
i
O
i
1
O
i
2
⋯
O
i
j
⋯
O
i
L
n
i
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
K
O
K
1
O
K
2
⋯
O
K
j
⋯
O
K
L
n
K
∙
Total
n
∙
1
n
∙
2
⋯
n
∙
j
⋯
n
∙
L
NTable 2
Relative frequency table.
ρ
1
ρ
2
⋯
ρ
j
⋯
ρ
L
Total
ω
1
p
11
p
12
⋯
p
1
j
⋯
p
1
L
p
1
∙
ω
2
p
21
p
22
⋯
p
2
j
⋯
p
2
L
p
2
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
i
p
i
1
p
i
2
⋯
p
i
j
⋯
p
i
L
p
i
∙
⋯
⋯
⋯
⋯
⋯
⋯
⋯
⋯
ω
K
p
K
1
p
K
2
⋯
p
K
j
⋯
p
K
L
p
K
∙
Total
p
∙
1
p
∙
2
⋯
p
∙
j
⋯
p
∙
L
1


## 5.2. Applications of Newly Defined Mutual Information in Credit Scoring

Credit scoring is used to describe the process of evaluating the risk a customer poses of defaulting on a financial obligation [15–19]. The objective is to assign customers to one of two groups:good andbad. Machine learning has been successfully used to build models for credit scoring [20]. In credit scoring, Y is a binary variable,good andbad, and may be represented by 0 and 1.To apply mutual information to credit scoring, we first calculate mutual information for every pair of(
X
,
Y
) and then do feature selection based on values of mutual information. We propose three ways.


### 5.2.1. Absolute Values Method

From Property4, we see that mutual information I
(
X
,
Y
) is nonnegative and upper bounded by log
⁡
(
L
) and that I
(
X
,
Y
)
=
0 if and only if X and Y are independent. In this sense, high mutual information indicates a large reduction in uncertainty, while low mutual information indicates a small reduction. In particular, zero mutual information means the two random variables are independent. Hence, we may select those features whose mutual information with Y is larger than some threshold based on needs.


### 5.2.2. Relative Values

From Property4, we have 0
≤
I
(
X
,
Y
)
/
H
(
Y
)
≤
1. Note that I
(
X
,
Y
)
/
H
(
Y
) is relative mutual information, which measures how much information X catches from Y. Thus, we may select those features whose relative mutual information I
(
X
,
Y
)
/
H
(
Y
) is larger than some threshold between 0 and 1 based on needs.


### 5.2.3. Chi-Square Test for Independency

For convenience, we will use the natural logarithm in mutual information. We first state an approximation formula for the natural logarithm function. It can be proved by the Taylor expansion like in Kullback’s book [5].Lemma 29.
Letp and q be two positive numbers less than or equal to 1. Then (60)
p
ln
⁡
⁡
p
q
≈
p
-
q
+
p
-
q
2
2
q
.The equality holds if and only if p
=
q. Moreover, the close p is to q, the better the approximation is.Now let us denoteN
×
I
(
X
,
Y
) by I
^
X
,
Y. Then, applying Lemma 29, we obtain(61)
2
I
^
X
,
Y
=
2
N
∑
i
=
1
K
∑
j
=
1
L
p
i
j
ln
⁡
⁡
p
i
j
p
i
·
p
·
j
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
ln
⁡
⁡
O
i
j
/
N
n
i
·
/
N
n
·
j
/
N
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
ln
⁡
⁡
O
i
j
n
i
·
n
·
j
/
N
≈
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
2
∑
i
n
i
·
N
∑
j
n
·
j
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
2
N
-
2
N
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
χ
2
.The last equation means the previous expression ∑
i
=
1
K
∑
j
=
1
L
(
O
i
j
-
n
i
·
n
·
j
/
N
)
2
/
(
n
i
·
n
·
j
/
N
) follows χ
2 distribution. According to [5], it follows χ
2 distribution with a degree of freedom of (
K
-
1
)
(
L
-
1
). Hence, 2
N
×
I
(
X
,
Y
) approximately follows χ
2 distribution with a degree of freedom of (
K
-
1
)
(
L
-
1
). This is the well-known Chi-square test for independence of two random variables. This allows using the Chi-square distribution to assign a significant level corresponding to the values of mutual information and (
K
-
1
)
(
L
-
1
).The null and alternative hypotheses are as follows.H
0
:X and Y are independent (i.e., there is no relationship between them).
H
1: X and Y are dependent (i.e., there is a relationship between them).The decision rule is to reject the null hypothesis at the α level of significance if the χ
2 statistic(62)
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
≈
2
N
×
I
(
X
,
Y
)is greater than χ
U
2, the upper-tail critical value from a Chi-square distribution with (
K
-
1
)
(
L
-
1
) degrees of freedom. That is,(63)
Select
feature
X
if
I
X
,
Y
>
χ
U
2
2
N
.Take credit scoring, for example. In this case, L
=
2. Assume feature X has 10 different values; that is, K
=
10. Using a level of significance of α
=
0.05
, we find χ
U
2 to be 16.9 from a Chi-square table with K
-
1
L
-
1
=
9, and select this feature only if I
X
,
Y
>
16.9
/
2
N.Assume a training set hasN examples. We can do feature selection by the following procedure.(i)
Step 1. Choose a level of significance ofα, say, 0.05.
(ii)
Step 2. FindK, the number of values of feature X.
(iii)
Step 3. Build the contingency table forX and Y.
(iv)
Step 4. CalculateI
(
X
,
Y
) from the contingency table.
(v)
Step 5. Findχ
U
2 with K
-
1
L
-
1 degrees of freedom from a Chi-square table or any other sources such as SAS.
(vi)
Step 6. SelectX if I
X
,
Y
>
16.9
/
2
N and discard it otherwise.
(vii)
Step 7. Repeat Steps 2–6 for all features.If the number of features selected from the above procedure is smaller or larger than what you want, you may adjust the level of significant α and reselect features using the procedure.


## 5.2.1. Absolute Values Method

From Property4, we see that mutual information I
(
X
,
Y
) is nonnegative and upper bounded by log
⁡
(
L
) and that I
(
X
,
Y
)
=
0 if and only if X and Y are independent. In this sense, high mutual information indicates a large reduction in uncertainty, while low mutual information indicates a small reduction. In particular, zero mutual information means the two random variables are independent. Hence, we may select those features whose mutual information with Y is larger than some threshold based on needs.


## 5.2.2. Relative Values

From Property4, we have 0
≤
I
(
X
,
Y
)
/
H
(
Y
)
≤
1. Note that I
(
X
,
Y
)
/
H
(
Y
) is relative mutual information, which measures how much information X catches from Y. Thus, we may select those features whose relative mutual information I
(
X
,
Y
)
/
H
(
Y
) is larger than some threshold between 0 and 1 based on needs.


## 5.2.3. Chi-Square Test for Independency

For convenience, we will use the natural logarithm in mutual information. We first state an approximation formula for the natural logarithm function. It can be proved by the Taylor expansion like in Kullback’s book [5].Lemma 29.
Letp and q be two positive numbers less than or equal to 1. Then (60)
p
ln
⁡
⁡
p
q
≈
p
-
q
+
p
-
q
2
2
q
.The equality holds if and only if p
=
q. Moreover, the close p is to q, the better the approximation is.Now let us denoteN
×
I
(
X
,
Y
) by I
^
X
,
Y. Then, applying Lemma 29, we obtain(61)
2
I
^
X
,
Y
=
2
N
∑
i
=
1
K
∑
j
=
1
L
p
i
j
ln
⁡
⁡
p
i
j
p
i
·
p
·
j
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
ln
⁡
⁡
O
i
j
/
N
n
i
·
/
N
n
·
j
/
N
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
ln
⁡
⁡
O
i
j
n
i
·
n
·
j
/
N
≈
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
2
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
2
∑
i
n
i
·
N
∑
j
n
·
j
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
2
N
-
2
N
N
+
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
=
χ
2
.The last equation means the previous expression ∑
i
=
1
K
∑
j
=
1
L
(
O
i
j
-
n
i
·
n
·
j
/
N
)
2
/
(
n
i
·
n
·
j
/
N
) follows χ
2 distribution. According to [5], it follows χ
2 distribution with a degree of freedom of (
K
-
1
)
(
L
-
1
). Hence, 2
N
×
I
(
X
,
Y
) approximately follows χ
2 distribution with a degree of freedom of (
K
-
1
)
(
L
-
1
). This is the well-known Chi-square test for independence of two random variables. This allows using the Chi-square distribution to assign a significant level corresponding to the values of mutual information and (
K
-
1
)
(
L
-
1
).The null and alternative hypotheses are as follows.H
0
:X and Y are independent (i.e., there is no relationship between them).
H
1: X and Y are dependent (i.e., there is a relationship between them).The decision rule is to reject the null hypothesis at the α level of significance if the χ
2 statistic(62)
∑
i
=
1
K
∑
j
=
1
L
O
i
j
-
n
i
·
n
·
j
/
N
2
n
i
·
n
·
j
/
N
≈
2
N
×
I
(
X
,
Y
)is greater than χ
U
2, the upper-tail critical value from a Chi-square distribution with (
K
-
1
)
(
L
-
1
) degrees of freedom. That is,(63)
Select
feature
X
if
I
X
,
Y
>
χ
U
2
2
N
.Take credit scoring, for example. In this case, L
=
2. Assume feature X has 10 different values; that is, K
=
10. Using a level of significance of α
=
0.05
, we find χ
U
2 to be 16.9 from a Chi-square table with K
-
1
L
-
1
=
9, and select this feature only if I
X
,
Y
>
16.9
/
2
N.Assume a training set hasN examples. We can do feature selection by the following procedure.(i)
Step 1. Choose a level of significance ofα, say, 0.05.
(ii)
Step 2. FindK, the number of values of feature X.
(iii)
Step 3. Build the contingency table forX and Y.
(iv)
Step 4. CalculateI
(
X
,
Y
) from the contingency table.
(v)
Step 5. Findχ
U
2 with K
-
1
L
-
1 degrees of freedom from a Chi-square table or any other sources such as SAS.
(vi)
Step 6. SelectX if I
X
,
Y
>
16.9
/
2
N and discard it otherwise.
(vii)
Step 7. Repeat Steps 2–6 for all features.If the number of features selected from the above procedure is smaller or larger than what you want, you may adjust the level of significant α and reselect features using the procedure.


## 5.3. Adjustment of Mutual Information in Feature Selection

In Section5.2, we have proposed 3 ways to select feature based on mutual information. It seems that the larger the mutual information I
(
X
,
Y
), the more dependent X on Y. However, Proposition 28 says that if X has all distinct values, then I
(
X
,
Y
) will reach the maximum value H
(
Y
) and I
(
X
,
Y
)
/
H
(
Y
) will reach the maximum value 1.Therefore, ifX has too many different values, one may bin or group these values first. Based on the binned values, mutual information is calculated again. For numerical variables, we may adopt a three-step process.(i)
Step 1: select features by removing those with small mutual information.
(ii)
Step 2: do binning for the rest of numerical features.
(iii)
Step 3: select features by mutual information.


## 5.4.  Comparison with Existing Feature Selection Methods

There are many other feature selection methods in machine learning and credit scoring. An easy way is to build a logistic model for each feature with respect to the dependent variable and then select features withp values less than some specific values. However, this method does not apply to any nonlinear models in machine learning.Another easy way of feature selection is to calculate the covariance of each feature with respect to the dependent variable and then select features whose values are larger than some specific value. Yet, mutual information is better than the covariance method [21] in that mutual information measures the general dependence of random variables without making any assumptions about the nature of their underlying relationships.The most popular feature selection in credit scoring is done by information value [15–19]. To calculate information between an independent variable and the dependent variable, a binning algorithm is used to group similar attributes into a bin. Difference between the information of good accounts and that of bad accounts in each bin is then calculated. Finally, information value is calculated as the sum of information differences of all bins. Features with information value larger than 0.02 are believed to have strong predictive power. However, mutual information is a better measure than information value. Information value focuses only on the linear relationships of variables, whereas mutual information can potentially offer some advantages information value for nonlinear models such as gradient boosting model [22]. Moreover, information value depends on binning algorithms and the bin size. Different binning algorithms and/or different bin sizes will have different information value.


## 6. Conclusions

In this paper, we have presented a unified definition for mutual information using random variables with different probability spaces. Our idea is to define the joint distribution of two random variables by taking the marginal probabilities into consideration. With our new definition of mutual information, different joint distributions will result in different values of mutual information. After establishing some properties of the new defined mutual information, we proposed a method to calculate mutual information in machine learning. Finally, we applied our newly defined mutual information to credit scoring.



---

*Source: 201874-2015-03-31.xml*