deviance

deviance

Background

In testing the fit of a

generalized linear model

𝒫

of some data (with

response variable

Y

and explanatory variable(s)

X

), one way is to compare

𝒫

with a

similar

model

𝒫

0

. By

similarity

we mean: given

𝒫

with the response variable

Y

i

∼

f

Y

i

and link function

g

such that

g

⁢

(

E

⁡

[

Y

i

]

)

=

𝐗

i

T

⁢

𝜷

, the model

𝒫

0

1.

is a generalized linear model of the same data,

2.

has the response variable

Y

distributed as

f

Y

, same as found in

𝒫

3.

has the same link function

g

as found in

𝒫

, such that

g

⁢

(

E

⁡

[

Y

i

]

)

=

𝐗

i

T

⁢

𝜷

𝟎

Notice that the only possible

difference

is found in the parameters

𝜷

.

It is desirable for this

𝒫

0

to be served as a base model in case when more than one models are being assessed. Two possible candidates for

𝒫

0

are the

null model

and the

saturated model

. The null model

𝒫

n

⁢

u

⁢

l

⁢

l

is one in which only one parameter

μ

is used so that

g

⁢

(

E

⁡

[

Y

i

]

)

=

μ

, all responses have the same predicted outcome. The saturated model

𝒫

m

⁢

a

⁢

x

is the other extreme where the maximum number of parameters are used in the model so that the observed response values equal to the predicted response values exactly,

g

⁢

(

E

⁡

[

Y

i

]

)

=

𝐗

i

T

⁢

𝜷

m

⁢

a

⁢

x

=

y

i

Definition

The

deviance

of a model

𝒫

(generalized linear model) is given by

dev

(

𝒫

)

=

2

[

ℓ

(

𝜷

^

m

⁢

a

⁢

x

∣

𝐲

)

-

ℓ

(

𝜷

^

∣

𝐲

)

]

,

where

ℓ

is the

log-likelihood function

,

𝜷

^

is the MLE of the parameter vector

𝜷

from

𝒫

and

𝜷

^

m

⁢

a

⁢

x

is the MLE of parameter vector

𝜷

m

⁢

a

⁢

x

from the saturated model

𝒫

m

⁢

a

⁢

x

.

Example

For a normal or

general linear model

, where the link function is the

identity

:

E

⁡

[

Y

i

]

=

𝐱

i

T

⁢

𝜷

,

where the

Y

i

’s are mutually

independent

and normally distributed as

N

⁢

(

μ

i

,

σ

2

)

. The log-likelihood function is given by

ℓ

(

𝜷

∣

𝐲

)

=

-

1

2

⁢

σ

2

∑

i

=

1

n

(

y

i

-

μ

i

)

2

-

n

⁢

ln

⁡

(

2

⁢

π

⁢

σ

2

)

2

,

where

μ

i

=

𝐱

i

T

⁢

𝜷

is the predicted response values, and

n

is the number of observations.

For the model in question, suppose

μ

^

i

=

𝐗

i

T

⁢

𝜷

^

is the expected mean calculated from the maximum likelihood estimate

𝜷

^

of the parameter vector

𝜷

. So,

ℓ

(

𝜷

^

∣

𝐲

)

=

-

1

2

⁢

σ

2

∑

i

=

1

n

(

y

i

-

μ

^

i

)

2

-

n

⁢

ln

⁡

(

2

⁢

π

⁢

σ

2

)

2

,

For the saturated model

𝒫

m

⁢

a

⁢

x

, the predicted value

(

μ

^

m

⁢

a

⁢

x

)

i

= the observed response value

y

i

. Therefore,

ℓ

(

𝜷

^

m

⁢

a

⁢

x

∣

𝐲

)

=

-

1

2

⁢

σ

2

∑

i

=

1

n

(

y

i

-

(

μ

^

m

⁢

a

⁢

x

)

i

)

2

-

n

⁢

ln

⁡

(

2

⁢

π

⁢

σ

2

)

2

=

-

n

⁢

ln

⁡

(

2

⁢

π

⁢

σ

2

)

2

.

So the deviance is

dev

(

𝒫

)

=

2

[

ℓ

(

𝜷

^

m

⁢

a

⁢

x

∣

𝐲

)

-

ℓ

(

𝜷

^

∣

𝐲

)

]

=

1

σ

2

∑

i

=

1

n

(

y

i

-

μ

^

i

)

2

,

which is exactly the residual sum of squares, or RSS, used in

regression models

.

Remarks

•

The deviance is necessarily non-negative.

•

The

distribution

of the deviance is asymptotically a chi square distribution (

http://planetmath.org/ChiSquaredRandomVariable

) with

n

-

p

degress of freedom, where

n

is the number of observations and

p

is the number of parameters in the model

𝒫

.

•

If two generalized linear models

𝒫

1

and

𝒫

2

are nested, say

𝒫

1

is nested within

𝒫

2

, we can perform

hypothesis testing

H

0

: the model for the data is

𝒫

1

with

p

1

parameters, against

H

1

: the model for the data is the more general

𝒫

2

with

p

2

parameters, where

p

1

<

p

2

. The deviance difference

Δ

(dev)

=

dev

⁡

(

𝒫

2

)

-

dev

⁡

(

𝒫

1

)

can be used as a

test statistic

and it is approximately a chi square distribution with

p

2

-

p

1

degrees of freedom

.

References

1

P. McCullagh and J. A. Nelder,

Generalized Linear Models

, Chapman & Hall/CRC, 2nd ed., London (1989).

2

A. J. Dobson,

An

Introduction

to Generalized Linear Models

, Chapman & Hall, 2nd ed. (2001).

Title

deviance

Canonical name

Deviance

Date of creation

2013-03-22 14:34:04

Last modified on

2013-03-22 14:34:04

Owner

CWoo (3771)

Last modified by

CWoo (3771)

Numerical id

8

Author

CWoo (3771)

Entry type

Definition

Classification

msc 62J12

Defines

null model

Defines

saturated model

Generated on Sat Feb 10 12:05:40 2018 by

LaTeXML