Discriminant

Analysis

by

Gaussian

Mixtures

Trevor

Hastie

Statistics

and

Data

Analysis

Research

Group

AT&T

Bell

Laboratories

Murray

Hill

New

Jersey

Robert

Tibshirani

Department

of

Preventive

Medicine

and

Biostatistics

and

Department

of

Statistics

University

of

Toronto

Toronto

Ontario

January

c

AT&T

Bell

Laboratories

and

University

of

Toronto

Abstract

Fisher-Rao

linear

discriminant

analysis

-LRB-

LDA

-RRB-

is

a

valuable

tool

for

multigroup

classification

LDA

is

equivalent

to

maximum

likelihood

classification

assuming

Gaussian

distributions

for

each

class

In

this

paper

we

fit

Gaussian

mixtures

to

each

class

to

facilitate

effective

classification

in

non-normal

settings

especially

when

the

classes

are

clustered

Low

dimensional

views

are

an

important

by-product

of

LDA

our

new

techniques

inherit

this

feature

We

are

able

to

control

the

within-class

spread

of

the

subclass

centers

relative

to

the

between-class

spread

Our

technique

for

fitting

these

models

permits

a

natural

blend

with

nonparametric

versions

of

LDA

Introduction

In

the

generic

classification

or

discrimination

problem

the

outcome

of

interest

G

falls

into

J

unordered

classes

which

for

convenience

we

denote

by

the

set

J

Jg

We

wish

to

build

a

rule

for

predicting

the

class

membership

of

an

item

based

on

p

measurements

of

predictors

or

features

X

Rp

Our

training

sample

consists

of

the

class

membership

and

predictors

for

N

items

This

is

an

important

practical

problem

with

applications

in

many

fields

Traditional

statistical

methods

for

this

problem

include

linear

discriminant

analysis

-LRB-

LDA

-RRB-

and

multiple

logistic

regression

nearest

neighbor

methods

and

classification

trees

Neural

network

classifiers

have

become

a

powerful

alternative

with

the

ability

to

incorporate

a

very

large

number

of

features

in

an

adaptive

nonlinear

model

Ripley

-LRB-

-RRB-

gives

an

informative

review

from

a

statisticians

viewpoint

LDA

can

be

derived

as

the

maximum

likelihood

method

for

normal

populations

with

different

means

and

common

covariance

matrix

It

is

natural

therefore

to

generalize

LDA

by

assuming

that

each

observed

class

is

in

fact

a

mixture

of

unobserved

normally

distributed

subclasses

This

approach

is

sometimes

mentioned

in

the

statistical

literature

-LRB-

McLachlan

Cheng

Titterington

for

example

-RRB-

and

pattern

recognition

literature

-LRB-

Taxt

Hjort

Eikvil

-RRB-

but

does

not

seem

to

have

generated

much

attention

In

this

paper

we

develop

the

mixture

approach

to

discrimination

in

a

number

of

interesting

and

useful

directions

For

brevity

we

refer

to

this

mixture

discriminant

analysis

technique

as

MDA

LDA

is

not

only

a

classification

procedure

but

also

a

data-reduction

tool

We

can

represent

multi-class

data

in

low

dimensional

projections

or

plots

that

highlight

their

class

differences

Our

MDA

procedure

has

this

feature

as

well

we

can

produce

a

hierarchy

of

coordinates

in

terms

of

their

abilities

to

separate

the

classes

-LRB-

and

subclasses

-RRB-

An

interesting

twist

occurs

here

for

two-class

data

LDA

produces

a

-LRB-

not

so

interesting

-RRB-

single

coordinate

while

MDA

will

produce

up

to

one

less

than

the

number

of

subclasses

in

the

mixture

representation

A

technique

known

as

Learning

Vector

Quantization

or

LVQ

has

received

a

lot

of

attention

in

the

pattern

recognition

literature

-LRB-

Kohonen

-RRB-

MDA

can

be

viewed

as

a

smooth

version

of

LVQ

LVQ

finds

a

set

of

cluster

centers

for

each

class

classification

is

performed

by

finding

the

closest

center

and

assigning

the

associated

class

The

online

learning

algorithms

for

LVQ

are

similar

to

the

k-means

algorithm

with

biases

built

in

to

encourage

good

classification

While

LVQ

generalizes

clustering

to

classification

problems

MDA

generalizes

mixture

density

estimation

to

classification

problems

An

additional

feature

of

our

model

is

subclass

shrinkage

we

can

regulate

the

within-class

spread

of

the

mixture

centers

relative

to

their

between-class

spread

This

emphasizes

our

bias

towards

classification

much

like

LVQ

biases

clustering

techniques

towards

classification

It

also

allows

us

to

use

many

subclass

centers

per

class

and

shrink

them

until

the

effective

number

of

centers

is

a

required

smaller

amount

Other

methods

have

been

proposed

to

generalize

LDA

to

allow

nonlinear

decision

boundaries

In

Hastie

Tibshirani

Buja

-LRB-

-RRB-

this

is

achieved

by

using

adaptive

nonparametric

regression

methods

The

link

between

nonparametric

regression

and

discriminant

analysis

is

provided

by

the

optimal

scoring

approach

first

suggested

by

Breiman

Ihaka

-LRB-

-RRB-

Hastie

et

al.

-LRB-

-RRB-

call

this

procedure

Flexible

Discriminant

analysis

-LRB-

FDA

-RRB-

The

generic

version

of

FDA

based

on

smoothing

splines

operates

by

expanding

the

predictors

into

a

large

-LRB-

adaptively

selected

-RRB-

basis

set

and

then

performing

a

penalized

dis

Linear

Discriminant

Analysis

Mixture

Discriminant

Analysis

Learning

Vector

Quantization

Flexible

Discriminant

Analysis

Mixture

Discriminant

Analysis

Rank

Model

MDA/FDA

Figure

A

two

class

problem

in

which

one

of

the

classes

occurs

in

two

separated

circular

clouds

The

panels

are

labeled

according

to

the

technique

that

produced

the

decision

boundary

criminant

analysis

in

the

enlarged

space

Intuitively

the

penalization

works

by

downweighting

rough

directions

relative

to

smooth

directions

in

this

enlarged

space

when

computing

Mahalanobis

distances

PDA

-LRB-

Hastie

Buja

Tibshirani

-RRB-

is

a

closely

related

technique

for

classifying

digitized

analog

signals

PDA

starts

with

a

high

dimensional

feature

set

such

as

pixels

from

a

digitized

image

or

spectral

values

on

a

grid

of

frequencies

Again

a

penalized

discriminant

analysis

is

used

but

here

to

ensure

spatial

smoothness

of

the

discriminant

coefficients

Both

of

these

techniques

adapt

naturally

to

MDA

in

the

-LRB-

enlarged

-RRB-

feature

space

we

use

a

mixture

of

Gaussians

rather

than

a

single

one

per

class

and

still

use

a

penalized

metric

when

computing

distances

Figure

illustrates

some

of

these

techniques

with

a

simple

example

Not

surprisingly

the

MDA

technique

has

the

most

satisfactory

boundary

-LRB-

the

problem

is

a

setup

for

MDA

-RRB-

and

hence

its

boundary

is

optimal

The

LVQ

boundary

seems

unnecessarily

biased

and

wiggly

The

FDA

boundary

is

a

reasonable

approximation

to

the

optimal

boundary

The

lower-left

panel

shows

a

version

of

the

MDA

fit

which

also

does

acceptably

well

The

lower

right

panel

shows

the

result

of

MDA

combined

with

the

nonlinear

transformations

of

FDA

there

is

no

noticeable

improvement

To

summarize

then

our

proposal

for

MDA

has

the

following

features

ffl

The

classes

are

modelled

as

mixtures

of

Gaussians

rather

than

a

single

Gaussian

as

in

LDA

ffl

Optimal

subspace

identification

is

possible

as

in

LDA

with

added

functionality

ffl

We

can

shrink

the

subclass

centers

for

example

towards

a

common

center

ffl

The

flexibility

of

FDA

and

PDA

is

easily

and

naturally

accommodated

The

paper

is

organized

as

follows

In

section

we

discuss

the

normal

mixture

model

and

an

EM

algorithm

for

its

estimation

In

section

we

discuss

reduced

rank

versions

of

the

procedures

which

we

demonstrate

in

section

on

the

well-known

waveform

example

of

Breiman

Friedman

Olshen

Stone

-LRB-

-RRB-

In

section

we

show

how

the

MDA

algorithm

can

be

expressed

as

a

repeated-regression

procedure

using

optimal

scoring

-LRB-

Hastie

et

al.

-RRB-

which

in

turn

allows

useful

nonparametric

extensions

Section

further

illustrates

these

techniques

on

a

handwritten

digit-recognition

problem

Section

is

about

the

centroid

shrinking

and

section

describes

some

further

comparative

examples

Extending

LDA

by

normal

mixtures

The

approach

to

classification

taken

here

is

to

model

the

class

densities

of

the

predictors

P

-LRB-

X

j

G

-RRB-

by

Gaussian

mixture

models

These

are

flipped

around

via

Bayes

theorem

and

class

priors

to

give

models

for

the

class

posterior

probabilities

P

-LRB-

G

j

X

-RRB-

a

basic

ingredient

for

classification

Suppose

we

have

training

data

-LRB-

xi

gi

-RRB-

Rp

J

i

N

We

divide

each

class

j

into

Rj

artificial

subclasses

denoted

by

cjr

r

Rj

and

define

R

PJ

j

Rj

Our

model

assumes

that

each

subclass

has

a

multivariate

normal

distribution

with

its

own

mean

vector

jr

and

common

covariance

matrix

This

is

not

the

only

possible

mixture

model

for

example

one

could

allow

each

subclass

to

have

a

different

covariance

matrix

or

force

the

subclasses

within

a

given

class

to

have

the

same

covariance

matrix

The

particular

model

chosen

here

is

attractive

because

it

keeps

the

total

number

of

parameters

under

control

and

as

we

will

see

it

has

the

right

structure

to

permit

the

other

generalizations

we

have

in

mind

Let

j

be

the

prior

for

class

j

and

within

class

j

let

ssjr

be

the

mixing

probability

for

the

rth

subclass

PRj

r

ssjr

Note

that

while

often

the

j

are

known

or

easily

estimated

from

the

training

data

the

ssjr

are

unknown

model

parameters

Let

D

-LRB-

x

-RRB-

-LRB-

x

-RRB-

T

-LRB-

x

u

-RRB-

be

the

Mahalanobis

distance

between

x

and

The

mixture

density

for

class

j

is

mj

-LRB-

x

-RRB-

P

-LRB-

X

xjG

j

-RRB-

Rj

X

r

ssjre?D

-LRB-

x

jr

-RRB-

-LRB-

-RRB-

and

the

conditional

log-likelihood

for

the

data

is

mix

-LRB-

rj

ssjr

-RRB-

N

X

i

log

mgi

-LRB-

xi

-RRB-

-LRB-

-RRB-

The

EM

algorithm

provides

a

convenient

method

for

maximizing

mix

-LRB-

-RRB-

The

EM

steps

are

p

-LRB-

cjrjx

j

-RRB-

Prob

-LRB-

x

rth

subclass

of

class

j

jx

j

-RRB-

ssjre?D

-LRB-

x

jr

-RRB-

PRj

k

ssjke?D

-LRB-

x

jk

-RRB-

-LRB-

-RRB-

ssjr

X

gi

j

p

-LRB-

cjrjxi

j

-RRB-

Rj

X

r

ssjr

-LRB-

-RRB-

jr

P

gi

j

xip

-LRB-

cjrjxi

j

-RRB-

P

gi

j

p

-LRB-

cjrjxi

j

-RRB-

-LRB-

-RRB-

-LRB-

N

-RRB-

J

X

j

X

gi

j

Rj

X

r

p

-LRB-

cjrjxi

j

-RRB-

-LRB-

xi

jr

-RRB-

-LRB-

xi

jr

-RRB-

T

-LRB-

-RRB-

The

notation

P

gi

j

means

summing

over

all

observations

belonging

to

the

jth

class

In

the

above

-LRB-

-RRB-

and

-LRB-

-RRB-

are

the

Estimation

step

while

-LRB-

-RRB-

and

-LRB-

-RRB-

are

the

Maximization

step

These

are

a

straightforward

generalization

of

the

EM

algorithm

for

estimating

normal

mixtures

-LRB-

e.g.

Titterington

Smith

and

Makov

page

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

have

the

same

appearance

as

the

maximumlikelihood

estimate

for

the

complete

normal

discriminant

problem

i.e.

the

situation

where

we

observe

the

subclass

membership

The

only

difference

is

that

the

subclass

indicator

is

replaced

p

-LRB-

cjrjxi

j

-RRB-

the

estimated

probability

that

observation

i

falls

in

subclass

cjr

given

that

it

is

observed

to

fall

into

class

j

Since

p

-LRB-

cjrjxi

j

-RRB-

is

a

function

of

jr

and

the

equations

-LRB-

-LCB-

-RRB-

must

be

iterated

The

posterior

class

probabilities

are

-LRB-

by

Bayes

theorem

-RRB-

P

-LRB-

G

jjX

x

-RRB-

jProb

-LRB-

xjj

-RRB-

j

Rj

X

r

ssjre?D

-LRB-

x

jr

-RRB-

-LRB-

-RRB-

normalized

so

that

PJ

j

P

-LRB-

G

jjX

x

-RRB-

The

classification

rule

chooses

j

to

maximize

p

-LRB-

jjx

-RRB-

Notice

that

this

does

not

have

the

same

form

as

a

linear

discriminant

rule

for

the

R

subclasses

and

in

particular

is

likely

to

be

nonlinear

Cluster

sizes

and

starting

values

The

EM

iteration

above

requires

a

choice

of

cluster

sizes

Rj

and

starting

values

for

the

means

jr

the

covariance

matrix

and

the

cluster

probabilities

p

-LRB-

cjrjx

j

-RRB-

We

currently

use

two

different

strategies

K-means

We

choose

a

fixed

number

of

clusters

-LRB-

say

r

-RRB-

for

each

class

and

then

use

a

k-means

clustering

algorithm

to

estimate

a

set

of

r

subclass

centroids

jr

for

each

class

Then

for

all

observations

in

class

j

p

-LRB-

cjrjxi

j

-RRB-

is

set

to

if

jr

is

the

closest

centroid

to

xi

and

otherwise

LVQ

We

run

the

LVQ

algorithm

on

the

training

data

and

let

it

select

the

Rj

and

jr

-LRB-

we

supply

R

-RRB-

As

in

the

previous

case

we

use

the

output

to

produce

p

-LRB-

cjrjxi

j

-RRB-

In

either

case

these

within

class

weights

determine

a

new

set

of

jrs

from

-LRB-

-RRB-

and

from

-LRB-

-RRB-

and

the

iterations

go

from

there

Both

k-means

and

LVQ

require

starting

centers

which

are

typically

randomly

selected

xi

It

is

known

-LRB-

Pal

Bezdek

Tsao

for

example

-RRB-

that

both

these

procedures

suffer

from

variability

due

to

random

starting

values

Our

strategy

is

to

try

a

number

of

different

starts

and

choose

the

best

We

use

either

a

likelihood

based

criterion

or

more

cheaply

training

sample

misclassification

to

guide

the

choice

Our

experience

has

been

that

MDA

always

outperforms

the

starting

procedure

-LRB-

LVQ

or

kmeans

-RRB-

with

respect

to

classification

of

the

training

data

In

fact

the

procedure

implemented

in

our

software

and

used

in

this

paper

is

the

EM

algorithm

above

but

with

the

M

step

carried

out

by

optimal

scoring

as

described

in

section

As

we

will

show

this

has

computational

advantages

and

facilitates

powerful

generalizations

of

the

model

Reduced

rank

discrimination

In

linear

discriminant

analysis

with

J

classes

one

can

choose

a

subspace

of

rank

r

J

that

maximally

separates

the

class

centroids

This

is

mainly

useful

for

descriptive

purposes

but

is

also

a

form

of

regularization

and

often

leads

to

improved

classification

performance

This

standard

Fisher-Rao

decomposition

-LRB-

Mardia

Kent

Bibby

for

example

-RRB-

is

derived

by

successively

maximizing

the

ratio

of

between

to

within-group

variance

of

linear

combinations

of

the

variables

vT

Bv

vT

Wv

where

B

is

the

between

class

covariance

-LRB-

of

the

class

centroids

-RRB-

and

W

the

pooled

within-class

covariance

In

this

section

we

show

that

reduced

rank

LDA

can

be

viewed

as

a

restricted

Gaussian

maximum

likelihood

solution

and

then

extend

this

concept

to

the

mixture

model

Proposition

Consider

maximizing

the

Gaussian

log-likelihood

-LRB-

j

d

-RRB-

J

X

j

X

gi

j

-LRB-

xi

j

-RRB-

T

-LRB-

xi

j

-RRB-

N

log

j?j

-LRB-

-RRB-

subject

to

the

constraints

rankf?jgJj

K

min

-LRB-

J

p

-RRB-

The

solution

is

j

WV

V

T

-LRB-

xj

x

-RRB-

x

-LRB-

-RRB-

W

J

X

j

Nj

N

-LRB-

xj

j

-RRB-

-LRB-

xj

j

-RRB-

T

W

WV?V

T

BV?V

T

W

-LRB-

-RRB-

where

V

is

a

matrix

consisting

of

the

leading

K

eigenvectors

of

W

This

solution

effectively

coincides

with

the

reduced

rank

LDA

solution

ffl

j

is

the

projection

of

the

jth

sample

mean

onto

the

discriminant

subspace

of

rank

K.

ffl

classification

based

on

-LRB-

x

j

-RRB-

T

-LRB-

x

j

-RRB-

is

equivalent

to

the

reduced

linear

discriminant

rule

of

rank

K.

Much

of

this

result

was

proved

by

Campbell

-LRB-

-RRB-

although

he

appears

to

have

overlooked

the

last

claim

we

outline

a

simple

proof

in

Appendix

A.

Now

consider

a

reduced

rank

version

of

the

mixture

model

We

take

the

Gaussian

mixture

log-likelihood

mix

-LRB-

equation

-RRB-

and

maximize

it

subject

to

the

rank

K

constraint

on

the

subclass

means

rankf?rjg

K

How

might

we

achieve

this

maximization

Once

again

we

can

use

the

EM

algorithm

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

remain

the

same

and

are

conditional

on

the

current

-LRB-

reduced

rank

-RRB-

versions

of

the

centers

and

the

corresponding

pooled

covariance

estimate

Steps

-LRB-

-RRB-

and

-LRB-

-RRB-

can

be

viewed

as

weighted

mean

and

pooled-covariance

maximum-likelihood

estimates

in

a

weighted

and

augmented

R-class

problem

We

augment

the

data

by

replicating

the

Nj

observations

in

class

j

Rj

times

with

the

th

such

replication

having

observation

weights

p

-LRB-

cj

jxi

j

-RRB-

This

is

done

for

each

of

the

J

classes

resulting

in

an

augmented

and

weighted

training

set

of

PJ

j

NjRj

observations

Note

that

the

sum

of

all

the

weights

is

N

We

now

impose

the

rank

restriction

By

analogy

with

the

early

part

of

this

section

this

can

be

achieved

by

a

weighted

version

of

LDA

We

postpone

the

details

of

how

this

final

step

is

carried

out

until

section

Two

details

are

worth

noting

right

away

ffl

Step

-LRB-

-RRB-

of

the

EM

algorithm

requires

only

differences

in

Mahalanobis

distances

from

the

estimated

centers

This

is

convenient

since

weighted

LDA

will

only

supply

distances

in

the

K

dimensional

subspace

of

spanf?jrg

ffl

The

solution

to

this

problem

can

not

be

obtained

by

a

simple

reduction

of

the

full

rank

mixture

solution

We

need

to

perform

a

reduced

rank

weighted

LDA

at

each

of

the

iterations

of

the

EM

algorithm

The

reason

is

that

the

weights

computed

in

step

-LRB-

-RRB-

should

depend

on

the

reducedrank

rather

than

the

full-rank

solution

We

collect

these

results

in

the

following

Proposition

Consider

maximization

of

the

constrained

mixture

density

loglikelihood

mix

-LRB-

jr

ssjr

-RRB-

J

X

j

X

gi

j

log

Rj

X

r

ssjre?D

-LRB-

x

jr

-RRB-

N

log

j?j

-LRB-

-RRB-

subject

to

rankf?jg

K

This

is

achieved

by

an

EM

algorithm

analogous

to

that

defined

in

-LRB-

-RRB-

-LRB-

-RRB-

with

steps

-LRB-

-RRB-

-LRB-

-RRB-

replaced

by

the

equations

for

maximizing

the

weighted

and

augmented

log-likelihood

weight

-LRB-

rj

-RRB-

J

X

j

X

gi

j

Rj

X

r

p

-LRB-

cjrjxi

j

-RRB-

D

-LRB-

x

jr

-RRB-

N

log

j?j

subject

to

rankf?jg

K

This

latter

maximization

can

be

achieved

via

a

similarly

augmented

and

weighted

rank-K

LDA

The

lower

left

panel

in

figure

shows

the

effect

of

rank

reduction

on

this

simple

example

Practically

speaking

the

approximate

reduced-rank

solution

obtained

by

a

weighted

rank

reduction

of

the

full

mixture

solution

is

more

attractive

than

the

exact

reduced

rank

solution

described

above

This

is

because

the

latter

requires

one

to

iteratively

fit

the

model

for

each

dimension

K

of

interest

We

have

found

in

a

simulation

experiment

-LRB-

not

shown

here

-RRB-

that

the

two

approaches

tend

to

agree

quite

well

Example

waveform

data

Discriminant

Var

Var

subclasses

penalized

Discriminant

Var

Var

subclasses

shrunk

to

penalized

Figure

A

two

dimensional

view

of

the

MDA

model

fit

to

a

sample

of

the

waveform

model

The

points

are

independent

test

data

projected

onto

the

leading

two

canonical

coordinates

The

subclass

centers

are

indicated

We

now

illustrate

some

of

these

ideas

on

a

popular

simulated

example

taken

from

Breiman

et

al.

-LRB-

pg

-RRB-

and

used

in

Hastie

et

al.

-LRB-

-RRB-

and

elsewhere

It

is

a

three

class

problem

with

variables

and

is

considered

to

be

a

difficult

pattern

recognition

problem

The

predictors

are

defined

by

xi

-LRB-

i

-RRB-

-LRB-

u

-RRB-

-LRB-

i

-RRB-

ffli

Class

xi

-LRB-

i

-RRB-

-LRB-

u

-RRB-

-LRB-

i

-RRB-

ffli

Class

-LRB-

-RRB-

xi

-LRB-

i

-RRB-

-LRB-

u

-RRB-

-LRB-

i

-RRB-

ffli

Class

where

i

u

is

uniform

on

-LRB-

-RRB-

ffli

are

standard

normal

variates

and

the

hi

are

the

shifted

triangular

waveforms

-LRB-

i

-RRB-

max

-LRB-

ji

-RRB-

-LRB-

i

-RRB-

-LRB-

i

-RRB-

and

-LRB-

i

-RRB-

-LRB-

i

-RRB-

Table

Results

for

waveform

data

The

values

are

averages

over

simulations

with

the

standard

error

of

the

average

in

parentheses

The

five

entries

above

the

line

are

taken

from

Hastie

et

al.

-LRB-

-RRB-

The

first

model

below

the

line

is

MDA

with

subclasses

per

class

The

next

line

is

the

same

except

that

the

discriminant

coefficients

are

penalized

via

a

roughness

penalty

to

effectively

The

third

is

the

same

as

the

second

with

the

centroids

shrunk

to

effectively

centroids

per

class

Technique

Error

Rates

Training

Test

LDA

-LRB-

-RRB-

-LRB-

-RRB-

QDA

-LRB-

-RRB-

-LRB-

-RRB-

CART

-LRB-

-RRB-

-LRB-

-RRB-

FDA/MARS

-LRB-

degree

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

FDA/MARS

-LRB-

degree

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

MDA

-LRB-

subclasses

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

MDA

-LRB-

subclasses

penalized

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

MDA

-LRB-

subclasses

shrunk

to

penalized

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Table

extends

a

simulation

of

Hastie

et

al.

-LRB-

-RRB-

to

include

the

methods

discussed

here

Each

training

sample

has

observations

and

equal

priors

were

used

so

there

are

roughly

observations

in

each

class

We

used

test

samples

of

size

The

three

MDA

models

are

described

in

the

caption

More

details

on

the

penalization

and

centroid

shrinkage

is

given

in

section

Figure

shows

the

leading

two

canonical

variates

for

the

last

two

models

evaluated

at

the

test

data

As

we

might

have

guessed

the

classes

appear

to

lie

on

the

edges

of

a

triangle

This

is

because

the

hj

-LRB-

i

-RRB-

are

represented

by

points

in

thereby

forming

vertices

of

a

triangle

and

each

class

is

represented

as

a

convex

combination

of

a

pair

of

vertices

and

hence

lie

on

an

edge

The

Bayes

risk

for

this

problem

is

about

-LRB-

Breiman

et

al.

-RRB-

MDA

comes

close

to

the

optimal

rate

which

is

not

surprising

since

the

structure

of

the

MDA

model

is

similar

to

the

generating

model

MDA

by

optimal

scoring

One

can

use

optimal

scoring

multiple

linear

regression

followed

by

an

eigen

analysis

to

fit

both

the

LDA

and

MDA

models

as

we

show

in

this

section

This

has

significant

computational

advantages

and

facilitates

some

useful

generalizations

of

both

techniques

Consider

first

the

LDA

model

If

YN?J

is

an

indicator

response

matrix

representing

the

classes

then

we

can

regress

Y

against

the

predictor

matrix

X

to

get

fitted

values

Y

HY

This

is

followed

by

a

suitably

normalized

eigendecomposition

of

Y

THY

Y

T

Y

which

contains

all

the

ingredients

for

an

LDA

of

any

rank

-LRB-

Breiman

Ihaka

-LRB-

-RRB-

Hastie

et

al.

-LRB-

-RRB-

-RRB-

This

simple

procedure

carries

over

to

the

M-step

of

the

MDA

algorithm

Instead

of

using

a

response

indicator

matrix

Y

we

use

a

blurred

response

matrix

ZN?R

whose

rows

consist

of

the

current

subclass

probabilities

for

each

observation

At

each

M-step

this

Z

is

used

in

a

multiple

linear

regression

followed

by

an

eigen-decomposition

just

as

in

the

LDA

case

above

Hastie

et

al.

-LRB-

-RRB-

called

this

procedure

optimal

scoring

which

we

now

describe

in

more

detail

for

MDA

Mixture

Discriminant

Analysis

by

Optimal

Scoring

MDA

Initialize

Start

with

a

set

of

Rj

subclasses

cjr

for

each

class

and

associated

subclass

probabilities

p

-LRB-

cjrjxi

j

-RRB-

These

can

be

derived

for

example

from

a

LVQ

or

K-means

preprocessing

of

the

data

Let

R

P

Rj

Iterate

Compute

Response

Define

the

blurred

response

matrix

ZN?R

as

follows

if

gi

j

then

fill

the

jth

block

of

Rj

entries

in

the

ith

row

with

the

values

p

-LRB-

cjrjxi

j

-RRB-

r

Rj

and

the

rest

with

zeros

Z

is

the

mixture

analog

of

an

indicator-response

matrix

except

observations

can

belong

to

several

-LRB-

sub

-RRB-

classes

with

associated

probabilities

Multivariate

linear

regression

Fit

a

multi-response

linear

regression

of

Z

on

X.

Let

Z

be

the

fitted

values

and

-LRB-

x

-RRB-

be

the

vector

of

fitted

regression

functions

Optimal

scores

Let

be

the

largest

K

nontrivial

eigenvectors

of

ZT

Z

with

normalization

TDp

IK

Here

Dp

is

a

diagonal

R

R

matrix

of

weights

with

rth

entry

the

sum

of

the

elements

of

the

rth

column

of

Z

-LRB-

the

total

weight

for

subclass

r

-RRB-

Update

the

fitted

model

from

step

using

the

optimal

scores

-LRB-

x

-RRB-

T

-LRB-

x

-RRB-

Update

p

-LRB-

cjrjxi

j

-RRB-

and

ssjr

using

formulas

-LRB-

-RRB-

and

-LRB-

-RRB-

To

show

that

this

MDA

algorithm

corresponds

to

the

EM

algorithm

in

proposition

we

need

to

show

that

steps

-LCB-

fit

the

augmented

and

weighted

rank-K

discriminant

analysis

A

proof

is

given

in

Appendix

A.

We

do

n't

actually

obtain

estimates

of

the

means

and

covariance

from

this

MDA

algorithm

The

K-dimensional

fit

-LRB-

x

-RRB-

produced

by

the

algorithm

is

actually

the

coordinate

of

the

discriminant

projection

of

x

onto

the

subspace

spanned

by

the

reduced-rank

subclass

centroids

-LRB-

up

to

known

scale

factors

-RRB-

Relative

Euclidean

distances

in

this

subspace

are

Mahalanobis

distances

in

the

implicitly

estimated

covariance

The

projected

means

themselves

are

scaled

rows

of

Since

relative

distances

are

all

that

are

needed

in

-LRB-

-RRB-

and

-LRB-

-RRB-

we

have

all

the

ingredients

needed

for

fitting

the

model

and

for

classification

Details

of

these

equivalences

are

given

in

Hastie

et

al

-LRB-

b

-RRB-

One

advantage

of

this

approach

is

immediately

clear

the

M-step

in

proposition

involved

a

weighted

LDA

of

P

j

NjRj

observations

while

here

we

only

have

N

observations

There

is

an

even

more

compelling

reason

for

using

optimal

scoring

to

fit

LDA

and

MDA

models

By

expressing

the

problem

as

a

multiple

regression

of

a

response

matrix

Y

or

Z

we

can

immediately

generalize

it

by

using

in

step

regression

methods

more

exotic

than

linear

regression

In

the

context

of

LDA

these

ideas

were

originally

proposed

by

Breiman

Ihaka

-LRB-

-RRB-

and

developed

further

by

Hastie

et

al

-LRB-

b

-RRB-

Two

classes

of

such

generalizations

emerge

FDA

Here

multivariate

adaptive

nonlinear

regression

replaces

the

linear

regression

and

the

linear

map

-LRB-

X

-RRB-

becomes

a

nonlinear

map

Hastie

et

al.

-LRB-

-RRB-

describe

the

class

of

nonlinear

regression

methods

that

can

be

used

including

adaptive

additive

models

the

multivariate

adaptive

regression

spline

-LRB-

MARS

-RRB-

model

of

Friedman

-LRB-

-RRB-

projection

pursuit

regression

and

neural

networks

They

call

the

resulting

procedure

FDA

for

flexible

discriminant

analysis

These

procedures

amount

to

expanding

the

original

predictors

x

in

an

adaptive

way

into

a

set

of

transformed

predictors

h

-LRB-

x

-RRB-

and

then

performing

LDA

or

MDA

in

the

new

space

In

some

cases

the

nonparametric

regressions

consist

of

a

basis

expansion

followed

by

a

penalized

regression

-LRB-

e.g.

additive

splines

-RRB-

In

this

case

the

transformed

space

h

-LRB-

x

-RRB-

is

large

and

the

penalized

regression

translates

into

a

penalized

version

of

LDA

or

MDA

PDA

Here

we

fit

a

more

restrictive

linear

map

via

penalized

least

squares

-LRB-

Hastie

et

al.

-RRB-

The

idea

is

that

the

feature

vectors

xi

arise

as

digitized

analog

signals

and

are

spatially

correlated

-LRB-

pixels

in

an

image

log-spectra

at

different

frequencies

-RRB-

The

penalization

amounts

to

regularizing

the

large

near

singular

covariance

matrix

by

adding

a

penalty

which

in

turn

ensures

that

the

coefficients

vk

are

spatially

smooth

-LRB-

and

hence

borrow

strength

from

neighboring

values

-RRB-

Again

we

do

n't

explicitly

compute

the

means

and

covariance

in

the

-LRB-

transformed

-RRB-

feature

space

all

that

is

needed

are

the

regression

fits

-LRB-

x

-RRB-

In

the

case

of

FDA

this

feature

space

is

often

a

high-dimensional

space

of

basis

functions

so

this

simplification

is

very

important

computationally

Class

Discriminant

Functions

Class

Discriminant

Functions

Class

Discriminant

Functions

Figure

Discriminant

functions

for

the

subclasses

in

the

waveform

example

each

panel

represents

the

subclasses

for

a

single

class

The

smooth

functions

are

-LRB-

-RRB-

jr

produced

by

the

MDA

algorithm

using

a

penalized

regression

method

The

wiggly

functions

are

the

unpenalized

versions

jr

Superimposed

on

the

figures

are

the

generating

waveforms

shifted

and

scaled

to

fit

into

the

plots

In

figure

and

section

we

demonstrate

the

use

of

the

FDA

extension

to

MDA

on

some

examples

Our

experience

there

and

on

other

examples

tried

so

far

suggests

ffl

Both

FDA

and

standard

MDA

are

capable

of

producing

non-linear

decision

boundaries

Adding

the

non-linear

capabilities

of

FDA

to

MDA

puts

these

components

in

competition

with

each

other

and

typically

does

not

yield

a

much

improved

decision

boundary

Figure

supports

this

claim

One

can

imagine

that

in

difficult

situations

the

two

components

can

support

each

other

-LCB-

if

too

few

centers

are

provided

nonlinearities

can

compensate

-LCB-

extreme

nonlinearities

which

are

costly

in

terms

of

fitted

degrees

of

freedom

can

be

avoided

by

using

mixtures

ffl

the

adaptive

aspect

of

FDA

plays

the

role

of

a

variable

selector

the

examples

in

section

support

this

claim

The

PDA

extension

to

MDA

is

justified

in

exactly

the

same

way

as

for

PDA

itself

Two

of

our

examples

support

penalization

ffl

The

waveform

predictors

in

section

represent

noisy

functions

sampled

at

evenly

spaced

abscissae

The

discriminant

coefficients

were

penalized

using

a

roughness

penalty

reducing

the

effective

number

of

parameters

from

down

to

and

gave

on

average

an

improvement

in

the

classification

performance

Figure

shows

the

fitted

penalized

discriminant

functions

-LRB-

-RRB-

jr

for

each

subclass

and

compares

them

to

the

unpenalized

versions

We

see

that

penalization

plays

two

roles

-LCB-

Variance

reduction

the

unpenalized

MDA

discriminant

functions

appear

unnecessarily

wiggly

for

which

we

pay

a

price

in

generalization

errors

-LCB-

Interpretation

by

superimposing

the

known

generating

functions

we

see

that

the

discriminant

functions

are

appropriately

placed

for

picking

out

the

peaks

and

valley

of

the

mixture

distribution

for

each

class

It

would

be

much

harder

and

less

convincing

to

attempt

the

same

interpretations

using

the

unpenalized

discriminant

functions

ffl

the

handwritten

digit

example

in

the

next

section

is

a

natural

for

both

mixture

models

and

for

penalization

-LCB-

There

are

several

characteristic

ways

for

writing

each

digit

calling

for

a

mixture

model

-LCB-

pixel

grayscale

values

have

strong

spatial

correlation

calling

for

regularization

for

the

same

reasons

as

given

in

the

waveform

example

Example

Handwritten

Digit

Recognition

Figure

A

random

selection

of

digitized

handwritten

and

Each

image

is

a

bit

grayscale

version

of

the

original

binary

image

size

and

orientation

normalized

to

pixels

In

Hastie

et

al.

-LRB-

-RRB-

we

illustrated

PDA

on

a

handwritten

digit

recognition

task

Here

we

focus

on

a

difficult

sub-task

that

of

distinguishing

handwritten

and

We

use

the

same

training

data

as

Le

Cun

et

al.

-LRB-

-RRB-

who

normalized

binary

images

for

size

and

orientation

resulting

in

grayscale

images

We

have

threes

fives

and

eights

in

our

training

set

and

roughly

a

quarter

the

number

of

test

examples

Figure

shows

some

examples

from

each

The

inputs

are

highly

correlated

because

of

their

spatial

arrangement

and

some

kind

of

smoothing

or

filtering

always

helps

We

used

a

M-dimensional

orthonormal

basis

of

smooth

functions

on

the

spatial

domain

of

the

image

These

were

derived

from

the

roughness

penalty

matrix

of

a

two-dimensional

degree

thin-plate

spline

smoother

they

are

the

M

trailing

eigenvectors

which

are

the

smoothest

and

penalized

the

least

Given

this

M

basis

matrix

P

an

input

pixel

vector

x

is

replaced

by

x

Px

The

performance

of

LDA

improved

by

nearly

because

of

the

filtering

alone

with

M

MDA

with

four

subclasses

per

digit

gave

no

better

performance

but

further

regularization

helped

ffl

reducing

M

from

to

yielded

a

improvement

ffl

with

M

and

shrinking

to

a

total

of

effective

centers

yielded

a

improvement

Table

Digit

classification

results

and

The

filtered

models

correspond

a

hierarchical

basis

of

smooth

two-dimensional

functions

derived

from

the

thin-plate

smoothing-spline

penalty

functional

The

shrunken

MDA

model

shrunk

the

centroids

to

an

effective

total

of

Technique

Error

rates

-LRB-

-RRB-

Training

Test

LDA

LDA

-LRB-

filtered

-LCB-

-RRB-

MDA

-LRB-

filtered

-LCB-

subclasses

-RRB-

MDA

-LRB-

filtered

-LCB-

subclasses

-RRB-

MDA

-LRB-

filtered

-LCB-

subclasses

shrunk

-RRB-

0.2660.1580.3790.1980.2690.3670.1680.1960.1520.2160.3250.306

Figure

The

subclass

centroids

for

the

filtered

MDA

fit

displayed

in

unfiltered

form

Above

each

image

the

within-class

mixing

parameter

ssjr

is

indicated

Not

all

the

centroids

are

very

different

which

suggests

that

different

numbers

could

be

used

with

different

classes

subclasses

filtered

Discriminant

Var

Var

subclasses

filtered

Discriminant

Var

Var

subclasses

filtered

shrunk

Discriminant

Var

Var

subclasses

filtered

shrunk

Discriminant

Var

Var

Figure

The

top

two

panels

show

the

the

first

discriminant

coordinates

for

the

MDA

fit

with

the

centroids

indicated

The

lower

two

panels

show

the

corresponding

coordinates

for

the

shrunken

MDA

model

We

see

that

in

the

latter

case

the

leading

two

coordinates

concentrate

on

group

separation

while

separation

in

the

subclass

centroids

plays

a

stronger

role

for

the

in

the

coordinate

For

the

top

left

plot

subclass

separation

is

given

as

much

weight

as

class

separation

and

we

see

more

class

overlap

Figure

displays

the

fitted

subclass

centroids

as

images

some

spatial

differences

are

apparent

within

a

class

Figure

illustrates

the

effect

of

shrinkage

on

the

eigenvectors

discussed

further

in

section

Shrinking

and

Penalization

In

this

section

we

make

precise

our

notion

of

shrinking

the

sub-class

centers

as

well

as

the

interpretation

of

penalized

optimal

scoring

in

the

context

of

the

present

model

MDA

with

centroid

shrinking

The

subspace

reduced

model

in

proposition

is

fit

iteratively

using

weighted

LDA

which

essentially

treats

all

classes

and

subclasses

interchangeably

Our

proposal

here

is

to

bias

this

decomposition

and

the

positions

of

the

means

themselves

in

such

a

way

that

within

each

class

the

variance

of

the

subclass

centroids

is

damped

relative

to

between-class

variance

We

do

this

by

penalizing

the

mixture

likelihood

for

between-subclass

variability

In

the

limit

-LRB-

infinite

penalty

-RRB-

we

expect

the

standard

single-Gaussian-per-class

LDA

model

to

emerge

Let

ssj

fssjkg

be

the

vector

of

Rj

subclass

probabilities

for

the

Jth

class

-LRB-

summing

to

-RRB-

and

j

-LRB-

I

-RRB-

where

I

is

the

identity

matrix

and

is

a

column

vector

of

ones

For

any

vector

u

with

Rj

elements

ju

is

a

vector

of

deviations

of

the

elements

of

u

about

their

weighted

mean

and

likewise

uT?Tj

ju

is

a

positive

scalar

measure

of

the

spread

of

u

about

its

mean

If

Qj

Tj

j

is

the

corresponding

penalty

matrix

then

Q

diag

-LRB-

J

QJ

-RRB-

is

an

appropriate

composite

penalty

matrix

for

penalizing

the

sub-components

of

an

R-vector

for

deviations

about

their

class

means

The

relative

strengths

of

the

penalties

on

the

diagonal

can

be

controlled

by

the

parameters

j

We

now

consider

maximizing

a

penalized

version

of

-LRB-

-RRB-

Definition

Let

U

be

the

R?p

rank-K

matrix

of

means

for

the

rank-K

MDA

problem

The

shrunken

MDA

estimate

of

rank

K

maximizes

the

penalized

mixture

log-likelihood

mix

Q

-LRB-

U

ssjr

-RRB-

mix

-LRB-

U

ssjr

-RRB-

tr

-LRB-

-RRB-

-LRB-

-RRB-

subject

to

rank

-LRB-

U

-RRB-

K

Our

use

in

-LRB-

-RRB-

of

to

weight

the

components

of

the

penalty

is

explained

below

proposition

below

An

EM

algorithm

emerges

just

as

in

the

unshrunk

case

and

the

M-step

corresponds

to

a

penalized

reduced

rank

augmented

and

weighted

R

class

Gaussian

log-likelihood

analogous

to

that

in

proposition

weight

Q

-LRB-

rj

-RRB-

J

X

j

X

gi

j

Rj

X

r

p

-LRB-

cjrjxi

j

-RRB-

D

-LRB-

x

jr

-RRB-

N

log

j?j?tr

-LRB-

-RRB-

-LRB-

-RRB-

subject

to

rank

-LRB-

U

-RRB-

K.

Unfortunately

the

maximizer

of

-LRB-

-RRB-

does

not

have

a

simple

solution

like

the

unshrunk

version

The

solution

is

characterized

by

an

expression

for

the

means

and

covariance

just

as

in

-LRB-

-RRB-

and

-LRB-

-RRB-

in

appendix

A.

But

unlike

there

the

solution

now

requires

iteration

between

the

mean

and

the

covariance

estimators

Apart

from

the

added

computational

complexity

this

destroys

the

valuable

link

with

optimal

scoring

We

prefer

to

use

a

restricted

version

where

we

fix

the

estimated

covariance

at

the

weighted

pooled

within-subclass

covariance

W

as

in

-LRB-

-RRB-

For

this

restricted

case

we

have

Proposition

Consider

maximizing

the

restricted

M-step

penalized

log-likelihood

weight

Q

-LRB-

rj

jW

-RRB-

J

X

j

X

gi

j

Rj

X

r

p

-LRB-

cjrjxi

j

-RRB-

D

-LRB-

x

jr

-RRB-

tr

-LRB-

W

-RRB-

-LRB-

-RRB-

subject

to

rank

-LRB-

U

-RRB-

K

Here

D

-LRB-

-RRB-

is

defined

in

terms

of

W

The

solution

corresponds

to

a

shrunken

LDA

If

M

is

the

R

p

weighted

subclass

centroid

matrix

and

Dp

the

diagonal

weight

matrix

with

rth

element

the

sum

of

the

weights

for

the

rth

subclass

then

the

shrunken

LDA

is

based

on

the

shrunken

between-subclass

matrix

BQ

MDp

-LRB-

Dp

Q

-RRB-

This

shrunken

LDA

can

be

fit

using

a

modified

MDA

algorithm

in

which

the

only

change

is

that

ZT

Z

is

decomposed

with

normalization

T

-LRB-

Dp

Q

-RRB-

IK

A

proof

is

given

in

Appendix

A.

The

form

of

the

penalty

in

-LRB-

-RRB-

-LRB-

-RRB-

requires

some

explanation

The

modelled

mean

matrix

U

has

R

rows

for

subclasses

and

p

columns

for

predictors

Our

penalty

matrix

Q

works

on

individual

columns

of

U

any

composite

penalty

has

to

add

these

up

in

a

sensible

way

since

the

predictors

are

correlated

The

rank-K

mean

model

can

be

written

U

V

T

-LRB-

-RRB-

with

R?K

and

Vp?K

-LRB-

V

T?V

IK

-RRB-

to

be

determined

This

is

the

form

in

which

the

unshrunken

solutions

emerge

Now

tr

-LRB-

-RRB-

tr

-LRB-

T

Q

-RRB-

where

-LRB-

U

-RRB-

V

the

projection

of

the

means

onto

the

subspace

spanned

by

the

orthonormal

basis

V

Hence

we

simply

add

the

penalties

for

the

uncorrelated

columns

of

which

is

equivalent

to

adding

the

penalties

for

the

columns

of

U

From

the

form

of

Q

it

is

clear

that

as

the

j

get

large

the

only

vectors

permitted

will

be

constant

within

a

class

and

the

problem

will

degenerate

to

the

standard

LDA

model

Choosing

the

amount

of

shrinking

In

practice

it

is

difficult

to

guess

suitable

values

for

j

One

strategy

is

to

let

j

and

then

choose

in

some

objective

way

for

example

by

crossvalidation

or

evaluation

on

a

test

data

set

Here

we

describe

a

strategy

that

helps

the

analyst

choose

j

in

a

more

subjective

way

In

the

proof

of

proposition

A.

it

emerges

that

the

shrunken

means

M

-LRB-

Dp

Q

-RRB-

are

used

in

the

subsequent

LDA

The

shrinkage

operator

S

-LRB-

Dp

Q

-RRB-

is

like

a

smoother

and

it

makes

sense

to

use

its

trace

as

the

effective

number

of

parameters

after

the

shrinking

-LRB-

Hastie

Tibshirani

-RRB-

Given

Dp

it

is

easy

to

work

backwards

and

derive

such

that

tr

-LRB-

S

-RRB-

df

for

some

nominal

value

of

df

A

subjective

strategy

is

to

select

a

-LRB-

large

-RRB-

number

of

subclass

centers

Rj

for

each

class

and

then

fit

the

shrunken

model

such

that

ffl

The

effective

overall

number

of

centers

is

df

with

J

df

R

P

j

Rj

or

ffl

The

effective

number

of

centers

per

class

is

dfj

with

dfj

Rj

Here

we

use

the

fact

that

Q

and

hence

S

is

block

diagonal

so

one

can

compute

dfj

for

each

class

Figure

illustrates

this

latter

case

where

subclasses

where

shrunk

to

an

effective

per

class

Similarly

in

figure

we

shrunk

from

a

total

of

centers

to

an

effective

total

of

Notice

how

the

leading

two

eigenvectors

focus

more

on

between-class

separation

-LRB-

than

for

the

unshrunk

case

-RRB-

while

the

third

starts

picking

up

within-subclass

variation

MDA

and

penalized

optimal

scoring

In

general

steps

of

the

MDA/FDA

or

MDA/PDA

optimal

scoring

algorithm

can

be

seen

to

minimize

K

X

k

J

X

j

X

gi

j

Rj

X

r

p

-LRB-

cjrjxi

j

-RRB-

-LRB-

k

-LRB-

crk

-RRB-

k

-LRB-

xi

-RRB-

-RRB-

J

-LRB-

k

-RRB-

-LRB-

-RRB-

where

k

-LRB-

x

-RRB-

xT

k

J

-LRB-

k

-RRB-

Tk

k

is

a

penalty

matrix

and

the

scoring

functions

k

are

suitable

normalized

Here

x

might

represent

the

original

predictors

and

the

penalty

enforces

smoothness

of

the

coefficients

vectors

k

themselves

or

else

might

be

a

basis

expansion

of

the

original

predictors

and

the

penalty

ensures

that

the

composition

k

-LRB-

x

-RRB-

is

smooth

in

the

original

domain

A

reasonable

goal

consistent

with

our

approach

so

far

would

be

to

derive

this

penalized

optimal

scoring

criterion

as

the

M-step

of

a

suitably

regularized

Gaussian

-LRB-

mixture

-RRB-

likelihood

Since

in

the

unpenalized

case

these

k

are

scaled

versions

of

the

right

eigenvectors

vk

in

-LRB-

-RRB-

-LRB-

Hastie

et

al.

-RRB-

it

might

seem

natural

to

maximize

the

mixture

likelihood

subject

to

an

appropriate

penalty

on

the

vk

This

was

the

approach

taken

by

Kiiveri

-LRB-

-RRB-

in

the

context

of

penalized

LDA

We

first

analyze

the

simpler

case

of

penalized

LDA

via

a

penalized

Gaussian

likelihood

Unfortunately

it

does

not

lead

to

a

simple

-LRB-

non-iterative

-RRB-

MLE

if

is

unknown

and

if

is

assumed

known

and

equal

to

W

it

still

differs

from

the

optimal

scoring

procedure

Hastie

et

al.

-LRB-

-RRB-

show

that

optimal

scoring

corresponds

to

LDA

with

a

fixed

penalized

within-class

covariance

W

In

other

words

Gaussian

maximum-likelihood

for

the

rank-reduced

means

with

assumed

known

and

equal

to

W

This

is

also

the

approach

taken

by

Leurgans

Moyeed

Silverman

-LRB-

-RRB-

For

MDA

things

get

more

complicated

since

the

corresponding

version

of

W

is

a

weighted

within-covariance

which

changes

at

each

iteration

We

are

thus

unable

to

write

down

an

explicit

likelihood

criterion

We

nevertheless

use

the

penalized

MDA

optimal

scoring

algorithm

which

corresponds

to

using

W

in

place

of

W

in

the

M-step

More

examples

Here

we

use

two

simulated

examples

and

some

real

data

to

compare

a

number

of

classification

procedures

The

methods

include

LDA

linear

discriminant

analysis

using

all

of

the

variables

MDA

mixture

discriminant

analysis

as

described

in

section

FDA

flexible

discriminant

analysis

using

adaptive

backfitting

as

described

in

Hastie

et

al.

-LRB-

-RRB-

MDA/FDA

mixture

discriminant

analysis

combined

with

flexible

discriminant

analysis

as

described

in

section

Neural

network

a

single

layer

perceptron

with

sigmoidal

outputs

crossentropy

cost

function

weight

decay

and

variable

metric

optimizer

-LRB-

Ripley

-RRB-

a

version

of

Kohonen

's

learning

vector

quantization

as

described

in

Hertz

Krogh

Palmer

-LRB-

-RRB-

Table

Results

for

simulation

The

values

are

averages

over

simulations

with

the

standard

error

of

the

average

in

parentheses

Technique

Error

rates

With

noise

variables

Training

Test

Training

Test

LDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

MDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

MDA/FDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

FDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Neural

net

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Simulation

separated

clouds

Here

we

examine

in

more

detail

an

example

like

that

of

Figure

There

are

two

classes

and

two

variables

The

first

class

has

observations

with

a

bivariate

normal

distribution

with

mean

-LRB-

-RRB-

and

more

with

mean

-LRB-

-RRB-

The

second

class

has

mean

-LRB-

-RRB-

The

covariance

matrix

is

the

identity

is

all

cases

The

left

part

of

Table

shows

a

summary

of

the

performance

of

a

number

of

procedures

over

realizations

The

MDA

procedure

performs

the

best

and

all

of

the

adaptive

procedures

outperform

LDA

FDA

works

here

by

transforming

the

predictor

space

so

that

the

two

subclasses

of

class

lie

on

top

of

one

another

The

right

half

of

the

table

shows

the

results

when

eight

N

-LRB-

-RRB-

noise

variables

were

added

the

predictor

set

The

deterioration

in

performance

of

MDA

and

the

neural

network

is

quite

striking

FDA

and

MDA/FDA

show

the

best

performance

probably

because

of

the

role

of

FDA

as

a

variable

selector

Simulation

unstructured

data

In

this

example

we

simulated

data

with

extremely

disconnected

class

structure

There

are

classes

each

with

spherical

bivariate

normal

subclasses

having

standard

deviation

The

means

of

the

subclasses

were

chosen

at

random

-LRB-

without

replacement

-RRB-

from

the

integers

-LSB-

-RSB-

-LSB-

-RSB-

Each

training

sample

had

observations

per

subclass

for

a

total

of

observations

For

testing

we

used

test

samples

of

size

Table

shows

the

error

rates

over

simulations

MDA

MDA/FDA

and

the

neural

net

clearly

outperform

the

other

procedures

FDA

and

also

offer

large

improvements

over

LDA

In

the

right

half

of

the

table

we

have

added

independent

N

-LRB-

-RRB-

noise

variables

The

performance

of

MDA

degrades

a

little

while

the

neural

net

and

especially

are

adversely

affected

MDA/FDA

is

able

to

select

the

important

input

Table

Results

for

Simulation

The

values

are

averages

over

simulations

with

the

standard

error

of

the

average

in

parentheses

Technique

Error

rates

With

noise

variables

Training

Test

Training

Test

LDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

MDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

MDA/FDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

FDA

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Neural

net

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

prototypes

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

dimensions

and

has

the

lowest

error

rate

Example

Thyroid

data

In

this

example

the

data

consist

of

observations

on

patients

The

object

is

to

predict

the

thyroid

status

-LRB-

normal

hyper

hypo

-RRB-

based

on

measurements

uptake

test

Total

Serum

thyroxin

Total

serum

triiodothyronine

Basal

thyroid-stimulating

hormone

Maximal

absolute

difference

of

TSH

value

after

injection

of

micro

grams

of

thyrotropin-releasing

hormone

as

compared

to

the

basal

value

They

are

described

in

Coomans

Broeckaert

M.

D.L.

-LRB-

-RRB-

We

took

a

random

training

sample

of

size

and

the

balance

-LRB-

observations

-RRB-

acted

as

the

test

sample

The

classification

results

are

shown

in

Table

The

nonlinear

methods

all

perform

very

well

with

MDA/FDA

winning

by

a

small

margin

Discussion

Classification

by

Gaussian

mixtures

is

not

a

new

idea

In

this

paper

we

have

added

to

the

functionality

of

this

approach

Table

Thyroid

data

results

Technique

Error

rates

Training

Test

LDA

MDA

MDA/FDA

FDA

Neural

net

-LRB-

hidden

units

-RRB-

ffl

Reduced

rank

versions

allow

valuable

low

dimensional

views

of

the

data

even

in

the

two

class

case

and

provide

a

natural

means

for

regularization

ffl

The

M-step

of

the

EM

algorithm

can

be

solved

via

a

weighted

optimal

scoring

algorithm

which

amounts

to

a

multiple

linear

regression

of

a

blurred

response

matrix

Z.

ffl

This

allows

natural

generalizations

by

replacing

the

linear

regression

by

more

exotic

forms

-LCB-

Adaptive

nonparametric

regressions

enrich

the

procedure

by

expanding

and

transforming

the

predictor

set

-LCB-

Penalized

regressions

regularize

the

coefficients

in

cases

where

the

predictors

are

sampled

analog

signals

over

some

spatial

domain

ffl

We

can

shrink

the

between-subclass

variability

relative

to

the

between

class

variability

This

often

leads

to

more

sensible

low

dimensional

views

and

is

a

natural

way

to

regularize

in

the

presence

of

many

subclasses

ffl

Our

procedure

provides

a

smooth

enhancement

to

the

LVQ

algorithm

and

in

fact

uses

LVQ

for

initialization

Further

comparisons

with

other

discrimination

techniques

in

a

variety

of

different

problems

would

be

useful

future

work

Software

We

have

written

a

set

of

functions

for

fitting

FDA

PDA

and

MDA

models

in

the

S/Splus

language

-LRB-

Becker

Chambers

Wilks

Chambers

Hastie

-RRB-

The

function

fda

-LRB-

-RRB-

fits

FDA

and

PDA

models

A

method

argument

allows

the

user

to

specify

the

multi-response

regression

method

to

be

used

the

default

is

linear

regression

and

thus

Fisher

's

LDA

Other

regression

methods

provided

for

FDA

are

polynomial

regression

ridge

regression

BRUTO

and

MARS

For

PDA

a

generalized

form

of

ridge

regression

is

included

Users

supply

a

penalty

matrix

and

target

df

and

the

procedure

derives

the

appropriate

penalty

constant

The

mda

-LRB-

-RRB-

function

has

additional

arguments

for

controlling

the

number

of

subclasses

their

initialization

and

df

parameters

to

control

the

shrinkage

A

variety

of

functions

are

provided

for

making

predictions

of

varying

types

from

these

models

-LRB-

class

predictions

posterior

probability

estimates

canonical

coordinates

-RRB-

based

on

models

of

specified

dimensions

Other

functions

are

provided

for

producing

plots

misclassification

-LRB-

confusion

-RRB-

matrices

and

for

extracting

coefficients

The

software

is

publicly

available

from

the

statistics

archive

at

Carnegie-Mellon

University

lib.stat.cmu.edu

Either

send

the

email

message

send

fda-mda

from

S

to

statlib@lib.stat.cmu.edu,

or

else

login

to

statlib

via

ftp

and

cd

S.

Acknowledgements

The

authors

thank

Andreas

Buja

Richard

Lippmann

Geoff

McLachlan

Daryl

Pregibon

Muni

Srivastava

and

Scott

VanderWiel

for

helpful

discussion

The

second

author

was

supported

by

the

Natural

Sciences

and

Engineering

Research

Council

of

Canada

A

Appendix

A.

Reduced

rank

models

In

proposition

in

section

we

claimed

that

maximizing

the

log-likelihood

-LRB-

-RRB-

-LRB-

j

K

-RRB-

J

X

j

X

gi

j

-LRB-

xi

j

-RRB-

T

-LRB-

xi

j

-RRB-

N

log

j?j

subject

to

rankf?jg

K

is

equivalent

to

reduced

rank

LDA

In

the

following

outline

of

a

proof

we

draw

heavily

on

Mardia

et

al.

-LRB-

hereafter

MKB

-RRB-

for

some

basic

multivariate

statistics

results

Let

B

be

the

between-class

covariance

matrix

and

for

fixed

let

V

denote

the

matrix

of

leading

K

eigenvectors

of

known

j

unknown

In

section

-LRB-

page

-RRB-

of

MKB

a

solution

to

-LRB-

-RRB-

is

given

assuming

is

known

This

has

the

form

of

the

usual

LDA

solution

except

with

W

replaced

by

One

can

write

the

estimated

means

as

j

V

V

T

-LRB-

xj

x

-RRB-

x

-LRB-

-RRB-

and

thus

the

estimated

-LRB-

rank-K

-RRB-

between

matrix

as

B

-LRB-

K

-RRB-

V

V

TBV

V

T

-LRB-

-RRB-

j

known

unknown

Although

this

case

is

not

explicitly

stated

in

MKB

one

deduces

-LRB-

and

easily

checks

-RRB-

from

their

equation

-LRB-

-RRB-

on

page

that

W

J

X

j

Nj

N

-LRB-

xj

j

-RRB-

-LRB-

xj

j

-RRB-

T

-LRB-

-RRB-

These

are

each

obtained

by

solving

the

partial

score

equations

for

j

or

assuming

the

other

is

known

The

full

maximum-likelihood

solution

requires

their

simultaneous

solution

and

suggests

iteration

However

the

solution

is

easier

We

plug

the

estimated

means

-LRB-

-RRB-

-LRB-

using

W

for

-RRB-

into

-LRB-

-RRB-

which

gives

W

J

X

j

Nj

N

-LRB-

xj

j

-RRB-

-LRB-

xj

j

-RRB-

T

W

B

B

-LRB-

K

-RRB-

W

B

WV

V

TBV

V

TW

W

WV?V

T

BV?V

T

W

-LRB-

-RRB-

where

V

T

WV

and

V

T

spans

the

complementary

p?K

dimensional

subspace

of

Rp

To

complete

the

proof

we

show

that

the

same

V

is

optimal

using

the

new

metric

First

of

all

note

that

V

T

V

V

TWV

IK

and

BV

WVDK

V

DK

where

the

first

equality

is

the

definition

of

V

and

DK

diag

-LRB-

K

-RRB-

We

have

thus

established

that

V

is

also

an

eigen-matrix

of

B

with

respect

to

we

have

still

to

show

that

it

has

remained

optimal

Note

that

V

T

V

Ip?K

V

T

BV

Ip?K

Dp?K

-LRB-

-RRB-

where

Dp?K

are

the

eigenvalues

of

B

corresponding

to

V

and

W

So

in

the

metric

the

columns

of

V

are

orthonormal

eigenvectors

of

B

V

is

orthogonal

and

orthogonal

to

V

Thus

the

columns

of

V

remain

eigenvectors

of

B

wrt

with

eigenvalues

-LRB-

Ip?K

Dp?K

-RRB-

K

Dp?K

and

thus

the

order

does

not

change

This

shows

that

the

constrained

maximum-likelihood

estimated

means

coincide

with

the

rank-K

LDA

means

Using

the

fact

that

V

V

T

V

-LRB-

D

p?K

-LRB-

Ip?K

Dp?K

-RRB-

V

T

is

not

hard

to

show

that

relative

Mahalanobis

distances

-LRB-

x

j

-RRB-

T

-LRB-

x

j

-RRB-

-LRB-

x

-RRB-

T

-LRB-

x

-RRB-

coincide

with

relative

Euclidean

distances

in

the

reduced

LDA

space

and

hence

classification

based

on

the

fitted

constrained

Gaussian

model

and

LDA

coincide

A.

The

MDA

algorithm

fits

a

reduced-rank

Gaussian

mixture

discriminant

model

In

proposition

we

have

that

the

reduced

rank

Gaussian

mixture

model

can

be

fit

by

an

EM

algorithm

with

the

M-step

a

reduced-rank

augmented/weighted

LDA

problem

In

Hastie

et

al.

-LRB-

-RRB-

we

show

that

optimal

scoring

solves

the

LDA

problem

It

is

immediate

that

the

augmented

weighted

LDA

problem

is

solved

by

a

similarly

augmented

and

weighted

optimal

scoring

procedure

but

that

is

not

what

is

presented

in

the

MDA

algorithm

in

section

For

example

in

MDA

we

still

have

N

observations

while

the

augmented

problem

has

N

P

NjRj

N

pseudo

observations

For

more

generality

we

justify

a

penalized

version

of

the

algorithm

The

augmented/weighted/penalized

optimal

scoring

criterion

for

a

single

score

is

J

X

j

X

gi

j

Rj

X

r

p

-LRB-

cjrjxi

j

-RRB-

-LRB-

jr

-LRB-

xi

-RRB-

-RRB-

J

-LRB-

-RRB-

-LRB-

-RRB-

where

-LRB-

x

-RRB-

xT

and

J

-LRB-

-RRB-

T

and

is

a

penalty

matrix

It

is

convenient

to

convert

to

a

matrix

notation

Let

Y

be

the

augmented

N

R

indicator

matrix

representing

the

observations

and

W

a

diagonal

N

N

weight

matrix

with

entries

the

values

p

-LRB-

cjrjxi

j

-RRB-

M

is

a

N

N

margin

matrix

with

a

row

for

each

-LRB-

original

-RRB-

observation

gi

consisting

of

a

for

each

of

the

Rgi

augmented

observations

associated

with

it

Notice

that

since

the

conditional

probabilities

sum

to

for

each

observation

MWMT

IN

Let

be

the

R

K

matrix

of

score

vectors

and

X

MTX

be

the

augmented

model

matrix

-LRB-

with

replicated

rows

-RRB-

Then

the

version

of

-LRB-

-RRB-

summed

over

all

K

scores

k

is

the

trace

of

RSS

-LRB-

B

-RRB-

-LRB-

Y

X?B

-RRB-

TW

-LRB-

Y

X?B

-RRB-

BT

B

-LRB-

-RRB-

which

is

to

be

minimized

wrt

B

and

subject

to

TY

TWY

IK

We

can

solve

for

B

for

fixed

-LRB-

Hastie

et

al.

-RRB-

to

get

RSS

-LRB-

-RRB-

TY

T

-LRB-

W

WS

-RRB-

Y

-LRB-

-RRB-

where

S

X

-LRB-

X?WX

-RRB-

TW

MT

-LRB-

XTX

-RRB-

MW

MTS

MW

-LRB-

-RRB-

where

S

is

the

collapsed

N

N

smoother

matrix

Thus

-LRB-

-RRB-

can

be

written

RSS

-LRB-

-RRB-

TY

T

-LRB-

W

WMTS

MW

-RRB-

Y

-LRB-

-RRB-

Now

Z

MWY

is

the

N

R

blurred

indicator

matrix

defined

in

section

and

Dp

Y

TWY

is

a

diagonal

R

R

weight

matrix

with

elements

the

sum

of

the

probabilities

in

the

columns

of

Z

Then

-LRB-

-RRB-

can

be

written

RSS

-LRB-

-RRB-

T

-LRB-

Dp

ZTS

Z

-RRB-

-LRB-

-RRB-

and

we

wish

to

minimize

trRSS

-LRB-

-RRB-

subject

to

TDp

IK

This

is

what

our

MDA

algorithm

solves

A.

Centroid

shrinking

Here

we

prove

proposition

Let

W

be

the

weighted

within

subclass

covariance

matrix

We

wish

to

maximize

weight

Q

-LRB-

U

jW

-RRB-

J

X

j

X

gi

j

Rj

X

r

p

-LRB-

cjrjxi

j

-RRB-

D

-LRB-

x

jr

-RRB-

tr

-LRB-

T

Q

-RRB-

where

U

V

TW

The

solution

minimizes

trDp

-LRB-

M

U

-RRB-

W

-LRB-

M

U

-RRB-

T

tr

-LRB-

T

Q

-RRB-

-LRB-

-RRB-

where

Dp

is

R?R

diagonal

with

rth

element

the

sum

of

all

the

weights

associated

with

the

rth

subclass

It

is

easily

shown

that

xT

and

without

loss

of

generality

we

assume

the

overall

mean

x

Let

V

W

V

M

MW

then

-LRB-

-RRB-

can

be

written

trDp

-LRB-

M

V

T

-RRB-

-LRB-

M

V

T

-RRB-

T

tr

-LRB-

T

Q

-RRB-

-LRB-

-RRB-

For

fixed

V

and

after

some

simple

manipulations

this

leads

to

-LRB-

Dp

Q

-RRB-

V

-LRB-

Dp

Q

-RRB-

-LRB-

-RRB-

Plugging

this

back

into

-LRB-

-RRB-

and

simplifying

we

get

tr

-LRB-

MTDpMW

-RRB-

tr

-LRB-

V

T

MTDp

-LRB-

Dp

Q

-RRB-

-RRB-

-LRB-

-RRB-

The

second

term

involves

V

and

it

corresponds

to

the

eigendecomposition

of

the

shrunken

between-subclass

covariance

matrix

BQ

MTDp

-LRB-

Dp

Q

-RRB-

-LRB-

while

the

unshrunk

version

is

defined

B

MTDpM

-RRB-

We

now

use

the

results

of

Hastie

et

al.

-LRB-

section

-RRB-

to

tie

this

decomposition

to

optimal

scoring

This

solution

corresponds

to

a

generalized

SVD

of

-LRB-

Y

T

X

-RRB-

N

DpM

with

left

and

right

metrics

Dp

Q

and

W

respectively

This

corresponds

to

optimal

scoring

with

score-metric

Dp

Q.

Remark

An

interesting

duality

emerges

Subtracting

a

penalty

from

the

loglikelihood

to

shrink

the

left

eigen-vectors

leads

to

a

Dp

Q

normalization

for

the

optimal

scoring

problem

Adding

a

penalty

to

the

optimal

scoring

criterion

to

smooth

the

right

eigen-vectors

leads

to

a

W

normalization

for

the

likelihood/LDA

problem

Both

these

can

be

used

simultaneously

and

amount

to

a

left

and

right-penalized

canonical

correlation

analysis

-LRB-

Leurgans

et

al.

Hastie

et

al.

-RRB-

References

Becker

R.

Chambers

J.

Wilks

A.

-LRB-

-RRB-

The

New

S

Language

Wadsworth

International

Group

Breiman

L.

Ihaka

R.

-LRB-

-RRB-

Nonlinear

discriminant

analysis

via

scaling

and

ACE

Technical

report

Univ.

of

California

Berkeley

Breiman

L.

Friedman

J.

Olshen

R.

Stone

C.

-LRB-

-RRB-

Classification

and

Regression

Trees

Wadsworth

Campbell

N.

-LRB-

-RRB-

Canonical

variate

analysis

a

general

formulation

Australian

Journal

of

Statistics

-LCB-

Chambers

J.

Hastie

T.

-LRB-

-RRB-

Statistical

Models

in

S

Wadsworth/Brooks

Cole

Pacific

Grove

Cheng

B.

Titterington

D.

-LRB-

-RRB-

Neural

networks

and

statistical

perspectives

submitted

Coomans

D.

Broeckaert

M.

M.

J.

D.L.

M.

-LRB-

-RRB-

Comparison

of

multivariate

discriminant

techniques

for

clinical

data

application

to

the

thyroid

functional

state

Meth

Inform

Med

-LCB-

Friedman

J.

-LRB-

-RRB-

Multivariate

adaptive

regression

splines

-LRB-

with

discussion

-RRB-

Annals

of

Statistics

-LRB-

-RRB-

-LCB-

Hastie

T.

Tibshirani

R.

-LRB-

-RRB-

Generalized

Additive

Models

Chapman

and

Hall

Hastie

T.

Buja

A.

Tibshirani

R.

-LRB-

-RRB-

Penalized

discriminant

analysis

To

be

published

Hastie

T.

Tibshirani

R.

Buja

A.

-LRB-

-RRB-

Flexible

discriminant

analysis

by

optimal

scoring

To

be

published

Hertz

J.

Krogh

A.

Palmer

R.

-LRB-

-RRB-

Introduction

to

the

theory

of

neural

computation

Addison

Wesley

Redwood

City

Kiiveri

H.

-LRB-

-RRB-

Canonical

variate

analysis

of

high

dimensional

data

Smoothing

canonical

vectors

Technical

Report

WA

DMS

CSIRO

Kohonen

T.

-LRB-

-RRB-

Self-Organization

and

Associative

Memory

-LRB-

edition

-RRB-

Springer-Verlag

Berlin

Le

Cun

Y.

Boser

B.

Denker

J.

S.

Henderson

D.

Howard

R.

Hubbard

W.

Jackel

L.

-LRB-

-RRB-

Handwritten

digit

recognition

with

a

back-propogation

network

in

D.

Touretzky

ed.

Advances

in

Neural

Information

Processing

Systems

Vol

Morgan

Kaufman

Denver

CO.

Leurgans

S.

Moyeed

R.

Silverman

B.

-LRB-

-RRB-

Canonical

correlation

analysis

when

the

data

are

curves

J.

Royal

Statist

Soc

-LRB-

Series

B

-RRB-

-LCB-

Mardia

K.

Kent

J.

Bibby

J.

-LRB-

-RRB-

Multivariate

Analysis

Academic

Press

McLachlan

G.

J.

-LRB-

-RRB-

Discriminant

Analysis

and

Statistical

Pattern

Recognition

Wiley

New

York

Pal

N.

Bezdek

J.

Tsao

E.

-LRB-

-RRB-

Generalized

clustering

networks

and

kohonen

self-organizing

scheme

IEEE

trans

on

Neural

Networks

-LCB-

Ripley

B.

-LRB-

-RRB-

Classification

and

clustering

in

spatial

and

image

data

To

appear

proc

of

Jahrestagung

von

Gesellschaft

fur

Klassification

H.

Goehl

and

M.

Schader

ed

Ripley

B.

-LRB-

-RRB-

Neural

networks

and

related

methods

for

classification

to

appear

-LRB-

with

discussion

-RRB-

Taxt

T.

Hjort

N.

Eikvil

L.

-LRB-

-RRB-

Statistical

classification

using

a

linear

mixture

of

multinormal

probability

densities

Pattern

Recognition

Letters

-LCB-

Titterington

D.M.

Smith

A.F.M.

and

Makov

U.E.

-LRB-

-RRB-

Statistical

Analysis

of

Finite

Mixture

Distributions

Wiley

New

York

classification
clustering
nonparametric
pattern recognition
penalized