Shannon’s entropy

Shannon’s entropy

Definition (Discrete)

Let

X

be a

discrete random variable

on a

finite set

𝒳

=

{

x

1

,

…

,

x

n

}

, with

probability distribution
function

p

⁢

(

x

)

=

Pr

⁡

(

X

=

x

)

. The

entropy

H

⁢

(

X

)

of

X

is
defined as

H

⁢

(

X

)

=

-

∑

x

∈

𝒳

p

⁢

(

x

)

⁢

log

b

⁡

p

⁢

(

x

)

.

(1)

The convention

0

⁢

log

⁡

0

=

0

is adopted in the

definition

. The

logarithm

is usually taken to the base 2, in which case the
entropy is measured in “bits,” or to the base

e

, in which case

H

⁢

(

X

)

is measured in “nats.”

If

X

and

Y

are

random variables

on

𝒳

and

𝒴

respectively, the

joint entropy

of

X

and

Y

is

H

⁢

(

X

,

Y

)

=

-

∑

(

x

,

y

)

∈

𝒳

×

𝒴

p

⁢

(

x

,

y

)

⁢

log

b

⁡

p

⁢

(

x

,

y

)

,

where

p

⁢

(

x

,

y

)

denote the

joint distribution

of

X

and

Y

.

Discussion

The Shannon entropy was first introduced by Shannon in 1948 in his
landmark paper “A Mathematical Theory of Communication.” The
entropy is a

functional

of the probability distribution function

p

⁢

(

x

)

, and is sometime written as

H

⁢

(

p

⁢

(

x

1

)

,

p

⁢

(

x

2

)

,

…

,

p

⁢

(

x

n

)

)

.

It is noted that the entropy of

X

does not depend on the actual
values of

X

, it only depends on

p

⁢

(

x

)

. The definition of
Shannon’s entropy can be written as an

expectation

H

⁢

(

X

)

=

-

E

⁢

[

log

b

⁡

p

⁢

(

X

)

]

.

The quantity

-

log

b

⁡

p

⁢

(

x

)

is interpreted as the information
content of the outcome

x

∈

𝒳

, and is also called the

Hartley
information

of

x

. Hence the Shannon’s entropy is the

average

amount of

information

contained

in random variable

X

, it is also
the uncertainty removed after the actual outcome of

X

is
revealed.

Characterization

We write

H

⁢

(

X

)

as

H

n

⁢

(

p

1

,

…

,

p

n

)

. The Shannon entropy

satisfies

the following

properties

.

1.

For any

n

,

H

n

⁢

(

p

1

,

…

,

p

n

)

is a

continuous

and
symmetric function on

variables

p

1

,

p

2

,

…

,

p

n

.

2.

Event of

probability

zero does not contribute to the entropy, i.e. for any

n

,

H

n

+

1

⁢

(

p

1

,

…

,

p

n

,

0

)

=

H

n

⁢

(

p

1

,

…

,

p

n

)

.

3.

Entropy is maximized when the probability distribution is
uniform. For all

n

,

H

n

⁢

(

p

1

,

…

,

p

n

)

≤

H

n

⁢

(

1

n

,

…

,

1

n

)

.

This follows from

Jensen inequality

,

H

⁢

(

X

)

=

E

⁢

[

log

b

⁡

(

1

p

⁢

(

X

)

)

]

≤

log

b

⁡

(

E

⁢

[

1

p

⁢

(

X

)

]

)

=

log

b

⁡

(

n

)

.

4.

If

p

i

⁢

j

,

1

≤

i

≤

m

,

1

≤

j

≤

n

are
non-negative

real numbers

summing

up to one, and

q

i

=

∑

j

=

1

n

p

i

⁢

j

, then

H

m

⁢

n

⁢

(

p

11

,

…

,

p

m

⁢

n

)

=

H

m

⁢

(

q

1

,

…

,

q

m

)

+

∑

i

=

1

m

q

i

⁢

H

n

⁢

(

p

i

⁢

1

q

i

,

…

,

p

i

⁢

n

q

i

)

.

If we

partition

the

m

⁢

n

outcomes of the random experiment into

m

groups, each group contains

n

elements

, we can do the
experiment in two steps: first determine the group to which the
actual outcome belongs to, and second find the outcome in this
group. The probability that you will observe group

i

is

q

i

.
The conditional probability distribution

function

given group

i

is

(

p

i

⁢

1

/

q

i

,

…

,

p

i

⁢

n

/

q

i

)

. The entropy

H

n

⁢

(

p

i

⁢

1

q

i

,

…

,

p

i

⁢

n

q

i

)

is the entropy of the probability distribution conditioned on
group

i

. Property 4 says that the total information is the sum
of the information you gain in the first step,

H

m

⁢

(

q

1

,

…

,

q

m

)

, and a weighted sum of the entropies conditioned on each
group.

Khinchin in 1957 showed that the only function satisfying the
above

assumptions

is of the form:

H

n

⁢

(

p

1

,

…

,

p

n

)

=

-

k

⁢

∑

i

=

1

n

p

i

⁢

log

⁡

p

i

where

k

is a

positive

constant

, essentially a choice of unit of

measure

.

Definition (Continuous)

Entropy in the continuous case is called

differential entropy

(

http://planetmath.org/DifferentialEntropy

)

.

Discussion—Continuous Entropy

Despite its seductively analogous form, continuous entropy cannot be obtained as a limiting case of

discrete

entropy.

We wish to obtain a generally finite measure as the “bin size” goes to zero. In the discrete case, the bin size is the (implicit) width of each of the

n

(finite or

infinite

) bins/buckets/states whose probabilities are the

p

n

. As we generalize to the continuous domain, we must make this width explicit.

To do this, start with a

continuous function

f

discretized as shown in the figure:

Figure 1:

Discretizing the function

f

into bins of width

Δ

As the figure indicates, by the

mean-value theorem

there exists a value

x

i

in each bin such that

f

⁢

(

x

i

)

⁢

Δ

=

∫

i

⁢

Δ

(

i

+

1

)

⁢

Δ

f

⁢

(

x

)

⁢

𝑑

x

(2)

and thus the

integral

of the function

f

can be approximated (in the Riemannian sense) by

∫

-

∞

∞

f

⁢

(

x

)

⁢

𝑑

x

=

lim

Δ

→

0

⁡

∑

i

=

-

∞

∞

f

⁢

(

x

i

)

⁢

Δ

(3)

where this limit and “bin size goes to zero” are

equivalent

.

We will denote

H

Δ

:=

-

∑

i

=

-

∞

∞

Δ

⁢

f

⁢

(

x

i

)

⁢

log

⁡

Δ

⁢

f

⁢

(

x

i

)

(4)

and expanding the

log

we have

H

Δ

=

-

∑

i

=

-

∞

∞

Δ

⁢

f

⁢

(

x

i

)

⁢

log

⁡

Δ

⁢

f

⁢

(

x

i

)

(5)

=

-

∑

i

=

-

∞

∞

Δ

⁢

f

⁢

(

x

i

)

⁢

log

⁡

f

⁢

(

x

i

)

-

∑

i

=

-

∞

∞

f

⁢

(

x

i

)

⁢

Δ

⁢

log

⁡

Δ

.

(6)

As

Δ

→

0

, we have

∑

i

=

-

∞

∞

f

⁢

(

x

i

)

⁢

Δ

→

∫

f

⁢

(

x

)

⁢

𝑑

x

=

1

and

(7)

∑

i

=

-

∞

∞

Δ

⁢

f

⁢

(

x

i

)

⁢

log

⁡

f

⁢

(

x

i

)

→

∫

f

⁢

(

x

)

⁢

log

⁡

f

⁢

(

x

)

⁢

𝑑

x

(8)

This leads us to our definition of the differential entropy (continuous entropy):

h

⁢

[

f

]

=

lim

Δ

→

0

⁡

[

H

Δ

+

log

⁡

Δ

]

=

-

∫

-

∞

∞

f

⁢

(

x

)

⁢

log

⁡

f

⁢

(

x

)

⁢

𝑑

x

.

(9)

Title

Shannon’s entropy

Canonical

name

ShannonsEntropy

Date of creation

2013-03-22 12:00:53

Last modified on

2013-03-22 12:00:53

Owner

kshum (5987)

Last modified by

kshum (5987)

Numerical id

26

Author

kshum (5987)

Entry type

Definition

Classification

msc 94A17

Synonym

entropy

Synonym

Shannon entropy

Related topic

Logarithm

Related topic

ConditionalEntropy

Related topic

MutualInformation

Related topic

DifferentialEntropy

Related topic

HartleyFunction

Defines

joint entropy

Generated on Sat Feb 10 13:41:54 2018 by

LaTeXML