exponential family

exponential family

A

probability

(

density

)

function

f

X

(

x

∣

θ

)

given a

parameter

θ

is said to belong to the (one parameter)

exponential family

of

distributions

if it can be written in one of the following two

equivalent

forms:

1.

a

⁢

(

x

)

⁢

b

⁢

(

θ

)

⁢

exp

⁡

[

c

⁢

(

x

)

⁢

d

⁢

(

θ

)

]

2.

exp

⁡

[

a

⁢

(

x

)

+

b

⁢

(

θ

)

+

c

⁢

(

x

)

⁢

d

⁢

(

θ

)

]

where

a

,

b

,

c

,

d

are known functions.
If

c

⁢

(

x

)

=

x

, then the distribution is said to be in

canonical form

. When the distribution is in canonical form, the function

d

⁢

(

θ

)

is called a

natural parameter

. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called

nuisance parameters

.

Examples:

•

The

normal distribution

,

N

⁢

(

μ

,

σ

2

)

, treating

σ

2

as a nuisance parameter, belongs to the exponential family. To see this, take the

natural logarithm

of

N

⁢

(

μ

,

σ

2

)

to get

-

1

2

⁢

ln

⁡

(

2

⁢

π

⁢

σ

2

)

-

1

2

⁢

σ

2

⁢

(

x

-

μ

)

2

Rearrange the above expression and we have

x

⁢

μ

σ

2

-

μ

2

2

⁢

σ

2

-

1

2

⁢

[

x

2

σ

2

+

ln

⁡

(

2

⁢

π

⁢

σ

2

)

]

Set

c

⁢

(

x

)

=

x

,

d

⁢

(

μ

)

=

μ

/

σ

2

,

b

⁢

(

μ

)

=

-

μ

2

/

(

2

⁢

σ

2

)

, and

a

⁢

(

x

)

=

-

1

/

2

⁢

[

x

2

/

σ

2

+

ln

⁡

(

2

⁢

π

⁢

σ

2

)

]

. Then we see that

N

⁢

(

μ

,

σ

2

)

does indeed belong to the exponential family. Furthermore, it is in canonical form. The natural parameter is

d

⁢

(

μ

)

=

μ

/

σ

2

.

•

Similarly, the Poisson, binomial, Gamma, and

inverse Gaussian distributions

all belong to the exponential family and they are all in canonical form.

•

Lognormal and

Weibull distributions

also belong to the exponential family but they are not in canonical form.

Remarks

•

If the p.d.f of a

random variable

X

belongs to an exponential family, and it is expressed in the second of the two above forms, then

E

⁡

[

c

⁢

(

X

)

]

=

-

b

′

⁢

(

θ

)

d

′

⁢

(

θ

)

,

(1)

and

Var

⁡

[

c

⁢

(

X

)

]

=

d

′′

⁢

(

θ

)

⁢

b

′

⁢

(

θ

)

-

d

′

⁢

(

θ

)

⁢

b

′′

⁢

(

θ

)

d

′

⁢

(

θ

)

3

,

(2)

provided that functions

b

and

d

are appropriately conditioned.

•

Given a member from the exponential family of distributions, we have

E

⁡

[

U

]

=

0

and

I

=

-

E

⁡

[

U

′

]

, where

U

is the

score function

and

I

the

Fisher information

. To see this, first observe that the

log-likelihood function

from a member of the exponential family of distributions is given by

ℓ

(

θ

∣

x

)

=

a

(

x

)

+

b

(

θ

)

+

c

(

x

)

d

(

θ

)

,

and hence the score function is

U

⁢

(

θ

)

=

b

′

⁢

(

θ

)

+

c

⁢

(

X

)

⁢

d

′

⁢

(

θ

)

.

From (1),

E

⁡

[

U

]

=

0

.
Next, we obtain the Fisher information

I

. By definition, we have

I

=

E

[

U

2

]

-

E

[

U

]

2

=

E

⁡

[

U

2

]

=

d

′

⁢

(

θ

)

2

⁢

Var

⁡

[

c

⁢

(

X

)

]

=

d

′′

⁢

(

θ

)

⁢

b

′

⁢

(

θ

)

-

d

′

⁢

(

θ

)

⁢

b

′′

⁢

(

θ

)

d

′

⁢

(

θ

)

On the other hand,

∂

⁡

U

∂

⁡

θ

=

b

′′

⁢

(

θ

)

+

c

⁢

(

X

)

⁢

d

′′

⁢

(

θ

)

so

E

⁡

[

∂

⁡

U

∂

⁡

θ

]

=

b

′′

⁢

(

θ

)

+

E

⁡

[

c

⁢

(

X

)

]

⁢

d

′′

⁢

(

θ

)

=

b

′′

⁢

(

θ

)

-

b

′

⁢

(

θ

)

d

′

⁢

(

θ

)

⁢

d

′′

⁢

(

θ

)

=

b

′′

⁢

(

θ

)

⁢

d

′

⁢

(

θ

)

-

b

′

⁢

(

θ

)

⁢

d

′′

⁢

(

θ

)

d

′

⁢

(

θ

)

=

-

I

•

For example, for a

Poisson distribution

f

X

(

x

∣

θ

)

=

θ

x

⁢

e

-

θ

x

!

,

the natural parameter

d

⁢

(

θ

)

is

ln

⁡

θ

and

b

⁢

(

θ

)

=

-

θ

.

c

⁢

(

x

)

=

x

since Poisson is in canonical form. Then

U

⁢

(

θ

)

=

-

1

+

X

θ

⁢

and

⁢

I

=

-

E

⁡

[

-

X

θ

2

]

=

1

θ

as expected.

Title

exponential family

Canonical name

ExponentialFamily

Date of creation

2013-03-22 14:30:08

Last modified on

2013-03-22 14:30:08

Owner

CWoo (3771)

Last modified by

CWoo (3771)

Numerical id

7

Author

CWoo (3771)

Entry type

Definition

Classification

msc 62J12

Defines

canonical exponential family

Defines

nuisance parameter

Defines

natural parameter

Generated on Sat Feb 10 12:05:43 2018 by

LaTeXML