Hierarchical

Non-Emitting

Markov

Eric

Sven

Ristad

Robert

G.

Thomas

Research

Report

May

Revised

January

Abstract

We

describe

a

simple

variant

of

the

interpolated

Markov

model

with

nonemitting

state

transitions

and

prove

that

it

is

strictly

more

powerful

than

any

Markov

model

More

importantly

the

non-emitting

model

outperforms

the

classic

interpolated

model

on

the

natural

language

texts

under

a

wide

range

of

experimental

conditions

with

only

a

modest

increase

in

computational

requirements

The

non-emitting

model

is

also

much

less

prone

to

overfitting

authors

are

partially

supported

by

Young

Investigator

Award

to

the

first

author

from

the

National

Science

Foundation

Our

implementation

of

the

statistical

language

models

described

herein

used

the

library

of

practical

abstractions

-LSB-

-RSB-

An

abbreviated

version

of

this

report

was

presented

at

the

Machines

that

Learn

workshop

-LRB-

Snowbird

April

-LCB-

-RRB-

and

at

the

Annual

Meeting

of

the

ACL

-LRB-

Madrid

July

-LCB-

-RRB-

-LSB-

-RSB-

Introduction

The

Markov

model

has

long

been

the

core

technology

of

statistical

language

modeling

Many

other

models

have

been

proposed

but

has

offered

a

better

combination

of

predictive

performance

computational

efficiency

and

ease

of

implementation

Here

we

add

hierarchical

non-emitting

state

transitions

to

the

Markov

model

Although

the

states

in

our

model

remain

Markovian

the

model

itself

is

no

longer

Markovian

because

it

can

represent

unbounded

dependencies

in

the

state

order

distribution

Consequently

the

non-emitting

Markov

model

is

strictly

more

powerful

than

any

Markov

model

including

the

context

model

-LSB-

-RSB-

the

backoff

model

-LSB-

-RSB-

and

the

interpolated

Markov

model

-LSB-

-RSB-

More

importantly

the

non-emitting

model

consistently

outperforms

the

interpolated

Markov

model

on

natural

language

texts

under

a

wide

range

of

experimental

conditions

The

non-emitting

model

is

also

nearly

as

computationally

efficient

and

easy

to

implement

as

the

interpolated

Markov

model

The

remainder

of

our

report

consists

of

five

sections

and

one

appendix

In

section

motivate

the

fundamental

problem

of

time

series

prediction

which

is

to

combine

the

probabilities

of

events

of

different

orders

Section

reviews

the

interpolated

Markov

model

and

briefly

demonstrates

the

equivalence

of

interpolated

models

and

basic

Markov

models

of

the

same

model

order

Next

we

introduce

the

hierarchical

non-emitting

Markov

model

in

section

and

prove

that

even

a

second

order

non-emitting

model

is

strictly

more

powerful

than

any

Markov

model

of

any

model

order

Section

provides

efficient

algorithms

to

optimize

the

parameters

of

a

non-emitting

model

on

data

In

section

we

report

empirical

results

for

the

interpolated

model

and

the

non-emitting

model

on

the

Brown

corpus

and

Wall

Street

Journal

Finally

in

section

we

conjecture

that

the

non-emitting

model

excels

empirically

because

it

imposes

a

pseudoBayesian

discipline

on

maximum

likelihood

techniques

Appendix

A

reviews

the

backoff

model

and

explains

how

to

construct

a

non-emitting

backoff

model

that

is

strictly

more

powerful

than

any

backoff

model

Our

notation

is

as

follows

Let

A

be

a

finite

alphabet

of

distinct

symbols

jAj

k

and

let

xT

AT

denote

an

arbitrary

string

of

length

T

over

the

alphabet

A

Then

xji

denotes

the

substring

of

xT

that

begins

at

position

i

and

ends

at

position

j.

For

convenience

we

abbreviate

the

unit

length

substring

xii

as

xi

and

the

length

t

prefix

of

xT

as

xt

Time

Series

Prediction

A

time

series

model

must

assign

accurate

probabilities

to

strings

of

unbounded

length

Yet

unbounded

strings

do

n't

occur

in

recorded

histories

which

are

always

finite

Therefore

in

order

to

estimate

the

probabilities

of

unbounded

strings

from

a

finite

corpus

we

must

assume

that

each

symbol

in

a

given

string

depends

only

on

a

finite

number

of

-LRB-

equivalence

classes

of

-RRB-

contexts

The

most

widely

adopted

independence

assumption

is

the

order

n

Markov

assumption

which

states

that

each

symbol

depends

only

on

the

immediately

preceding

n

symbols

and

is

conditionally

independent

of

the

distant

past

p

-LRB-

xT

jT

-RRB-

QTt

p

-LRB-

xtjxt

-RRB-

ss

QTt

p

-LRB-

xtjxt

t?n

-RRB-

The

simplest

statistical

model

to

incorporate

an

order

n

Markov

assumption

is

the

basic

Markov

model

A

basic

Markov

model

OE

hA

n

ffini

consists

of

an

alphabet

A

a

model

order

n

n

and

the

state

transition

probabilities

ffin

An?A

-LSB-

-RSB-

With

probability

ffin

-LRB-

yjxn

-RRB-

a

Markov

model

in

the

state

xn

will

emit

the

symbol

y

and

transition

to

the

state

y.

Therefore

the

probability

pm

-LRB-

xtjxt

OE

-RRB-

assigned

by

an

order

n

basic

Markov

model

OE

to

a

symbol

xt

in

the

history

xt

depends

only

on

the

last

n

symbols

of

the

history

pm

-LRB-

xtjxt

OE

-RRB-

ffin

-LRB-

xtjxt

t?n

-RRB-

-LRB-

-RRB-

Since

the

Markov

model

contains

only

a

finite

number

of

parameters

it

is

in

principal

possible

to

estimate

their

values

directly

from

data

All

that

remains

is

to

choose

the

model

order

In

real-world

time

series

problems

the

future

depends

on

the

entire

past

even

if

only

weakly

In

order

to

more

closely

approximate

a

real-world

source

we

would

like

our

model

order

to

be

as

large

as

possible

Yet

we

have

only

a

finite

amount

of

training

data

from

which

to

estimate

our

model

parameters

An

order

n

Markov

model

over

an

alphabet

of

k

symbols

has

kn

events

while

a

corpus

of

length

T

has

at

most

T

n

distinct

events

of

order

n

The

exponential

growth

in

events

quickly

exceeds

the

size

of

all

available

training

data

and

nearly

all

the

higher-order

events

do

not

occur

in

the

training

data

This

tension

between

model

complexity

and

data

sparsity

is

fundamental

to

time

series

modeling

The

probabilities

of

the

lower

order

events

can

be

more

accurately

estimated

from

the

available

training

data

while

the

higher

order

events

are

better

able

to

model

complex

real-world

sources

An

effective

model

then

must

include

individual

events

of

both

higher

and

lower

orders

The

two

most

widely-used

techniques

for

combining

individual

events

of

varying

orders

are

backoff

and

interpolation

In

an

interpolated

model

the

transition

probabilities

from

lower

and

higher

order

states

are

combined

stochastically

using

mixing

parameters

In

a

backoff

model

the

event

probabilities

are

combined

according

to

a

partial

order

which

typically

favors

higher

order

events

over

lower

order

events

In

section

and

appendix

A

we

show

that

backoff

models

and

interpolated

models

are

formally

equivalent

to

basic

Markov

models

Therefore

backoff

and

interpolation

are

simply

parameter

estimation

schemes

for

basic

Markov

models

Interpolation

Here

we

introduce

the

interpolated

Markov

model

and

explain

why

the

interpolated

model

class

is

equivalent

to

the

class

of

basic

Markov

models

In

the

next

section

we

introduce

hierarchical

non-emitting

state

transitions

to

the

Markov

model

and

prove

that

the

new

non-emitting

models

are

no

longer

Markovian

even

though

their

states

are

In

the

interpolated

Markov

model

the

transition

probabilities

from

states

of

different

orders

are

combined

using

state-conditional

mixing

parameters

The

mixing

parameters

smooth

the

transition

probabilities

from

higher

order

states

with

those

from

lower

order

states

-LSB-

-RSB-

Mixing

the

transition

probabilities

from

states

of

different

orders

results

in

more

accurate

predictions

than

can

be

obtained

from

any

fixed

model

order

Formally

an

interpolated

Markov

model

OE

hA

n

ffi

i

consists

of

a

finite

alphabet

A

a

maximal

model

order

n

the

state

transition

probabilities

ffi

ffin

ffii

Ai

A

-LSB-

-RSB-

and

the

state-conditional

interpolation

parameters

An

-LSB-

n

-RSB-

-LSB-

-RSB-

The

state

order

is

a

hidden

variable

The

probability

assigned

by

an

interpolated

model

is

a

linear

combination

of

the

probabilities

assigned

by

all

the

lower

order

Markov

models

pc

-LRB-

yjxn

OE

-RRB-

n

X

i

ffii

-LRB-

yjxi

-RRB-

-LRB-

ijxn

-RRB-

-LRB-

-RRB-

An

interpolated

model

is

a

valid

probability

model

if

every

ffii

-LRB-

jxi

-RRB-

and

every

-LRB-

ijxn

-RRB-

is

valid

It

is

nonzero

for

all

strings

A

if

-LRB-

-RRB-

is

strictly

positive

for

all

symbols

A

and

no

-LRB-

ijxn

-RRB-

is

unity

when

ffi

-LRB-

jxi

-RRB-

is

zero

for

some

symbol

Estimating

the

O

-LRB-

nkn

-RRB-

state

interpolation

probabilities

is

considerably

easier

than

estimating

the

O

-LRB-

kn

-RRB-

state

transition

probabilities

in

an

order

n

Markov

model

To

begin

with

we

set

-LRB-

ijxn

-RRB-

to

if

the

order

i

state

xi

is

novel

Now

we

need

only

to

estimate

the

O

-LRB-

nT

-RRB-

interpolation

parameters

that

have

been

observed

in

the

training

data

Nonetheless

there

are

still

too

many

interpolation

parameters

to

be

accurately

estimated

Further

refinements

are

necessary

to

improve

predictive

performance

One

refinement

is

to

group

similar

parameters

into

equivalence

classes

and

then

constrain

them

to

take

the

same

values

This

is

called

parameter

tying

At

one

extreme

each

state-conditional

interpolation

distribution

is

its

own

equivalence

class

At

the

other

extreme

all

interpolation

probabilities

are

tied

together

and

we

have

the

state-independent

interpolated

Markov

model

pc

-LRB-

yjxn

OE

-RRB-

n

X

i

ffii

-LRB-

yjxi

-RRB-

i

-LRB-

-RRB-

with

only

n

interpolation

parameters

While

parameter

tying

can

improve

performance

reducing

state-conditional

interpolation

to

state-independent

interpolation

results

in

poor

performance

A

hierarchical

parameterization

of

the

full

state-conditional

interpolation

is

more

effective

Let

i

Ai

-LSB-

-RSB-

be

the

set

of

ith

order

state

interpolation

parameters

where

i

-LRB-

xi

-RRB-

is

the

probability

of

using

the

ithorder

state

transition

probability

ffii

-LRB-

jxi

-RRB-

conditioned

on

the

decision

not

to

use

any

higher

order

state

transition

probability

-LRB-

ijxn

-RRB-

i

-LRB-

xi

-RRB-

i

Y

j

n

-LRB-

j

-LRB-

xj

-RRB-

-RRB-

Then

the

probability

pc

-LRB-

yjxn

OE

-RRB-

that

the

state

xn

will

emit

the

symbol

y

has

a

particularly

simple

form

pc

-LRB-

yjxi

OE

-RRB-

i

-LRB-

xi

-RRB-

ffii

-LRB-

yjxi

-RRB-

-LRB-

i

-LRB-

xi

-RRB-

-RRB-

pc

-LRB-

OE

-RRB-

-LRB-

-RRB-

where

i

-LRB-

xi

-RRB-

for

i

n

and

therefore

pc

-LRB-

xtjxt

OE

-RRB-

pc

-LRB-

xtjxt

t?n

OE

-RRB-

ie.

the

prediction

depends

only

on

the

last

n

symbols

of

the

history

A

quick

glance

at

the

form

of

-LRB-

-RRB-

and

-LRB-

-RRB-

reveals

the

fundamental

simplicity

of

the

interpolated

Markov

model

Every

interpolated

model

is

equivalent

to

a

basic

Markov

model

of

the

same

order

and

every

basic

Markov

model

is

an

interpolated

model

of

the

same

order

We

may

convert

an

interpolated

model

OE

into

a

basic

model

of

the

same

model

order

n

simply

by

setting

-LRB-

yjxn

-RRB-

equal

to

pc

-LRB-

yjxn

OE

-RRB-

for

all

states

xn

An

and

symbols

y

A.

Thus

the

class

interpolated

Markov

models

is

extensionally

equivalent

to

the

class

of

basic

Markov

models

Non-Emitting

Transitions

In

the

previous

section

we

explained

how

to

combine

events

of

varying

orders

using

interpolation

and

backoff

Interpolation

and

backoff

both

use

the

probabilities

of

lower

events

to

estimate

the

probabilities

of

higher

order

events

As

a

result

interpolated

and

backoff

models

are

extensionally

equivalent

to

each

other

and

to

basic

Markov

models

of

the

same

order

In

this

section

we

explain

how

to

combine

events

of

varying

orders

using

non-emitting

state

transitions

The

central

idea

is

to

allow

actual

non-emitting

transitions

between

events

of

different

orders

Unlike

interpolation

and

backoff

non-emitting

transitions

are

not

merely

an

estimation

method

-LCB-

they

actually

increase

the

expressive

power

of

the

model

class

As

a

result

non-emitting

models

are

strictly

more

powerful

than

the

class

of

basic

Markov

models

The

next

section

provides

efficient

algorithms

to

evaluate

the

probability

of

a

string

according

to

a

non-emitting

model

and

to

optimize

the

parameters

of

a

non-emitting

model

on

data

A

non-emitting

mixture

Markov

model

OE

hA

n

ffi

i

consists

of

a

finite

alphabet

A

a

maximal

model

order

n

the

emitting

state

transition

probabilities

ffii

Ai

A

-LSB-

-RSB-

and

the

non-emitting

state

transition

probabilities

i

Ai

-LSB-

n

-RSB-

-LSB-

-RSB-

The

non-emitting

model

alternates

between

non-emitting

and

emitting

transitions

according

to

the

and

ffi

parameters

respectively

The

parameter

-LRB-

jjxi

-RRB-

specifies

the

probability

that

the

model

will

transition

from

the

state

xi

to

the

state

xj

without

emitting

a

symbol

The

parameter

ffij

-LRB-

yjxj

-RRB-

specifies

the

probability

that

the

model

will

emit

the

symbol

y

form

the

state

xj

and

transition

to

the

successor

state

xjy

Then

the

probability

pffl

-LRB-

yj

jxi

OE

-RRB-

assigned

to

a

string

yj

in

the

state

xi

has

the

form

pffl

-LRB-

yj

jxi

OE

-RRB-

iX

l

-LRB-

ljxi

-RRB-

ffil

-LRB-

-RRB-

pffl

-LRB-

OE

-RRB-

-LRB-

-RRB-

When

the

model

order

is

sufficiently

high

then

a

hierarchical

parameterization

of

the

non-emitting

transition

probabilities

may

improve

performance

With

probability

i

-LRB-

xi

-RRB-

a

hierarchical

non-emitting

model

will

transition

from

the

state

xi

to

the

state

without

emitting

a

symbol

With

probability

i

-LRB-

xi

-RRB-

ffii

-LRB-

yjxi

-RRB-

the

model

will

transition

from

the

state

xi

to

the

state

xiy

and

emit

the

symbol

y.

Therefore

the

probability

pffl

-LRB-

yj

jxi

OE

-RRB-

assigned

to

a

string

yj

in

the

history

xi

by

a

hierarchical

non-emitting

model

OE

has

the

recursive

form

-LRB-

-RRB-

pffl

-LRB-

yj

jxi

OE

-RRB-

i

-LRB-

xi

-RRB-

ffii

-LRB-

-RRB-

pffl

-LRB-

OE

-RRB-

-LRB-

i

-LRB-

xi

-RRB-

-RRB-

pffl

-LRB-

yj

OE

-RRB-

-LRB-

-RRB-

where

i

-LRB-

xi

-RRB-

for

i

n

and

-LRB-

ffl

-RRB-

Note

that

unlike

the

basic

Markov

model

pffl

-LRB-

xtjxt

OE

-RRB-

pffl

-LRB-

xtjxt

t?n

OE

-RRB-

because

the

state

distribution

of

the

nonemitting

model

depends

on

the

prefix

xi?n

This

simple

fact

will

allow

us

to

establish

that

there

exists

a

non-emitting

model

that

is

not

equivalent

to

any

Markov

model

Lemma

states

that

there

exists

a

non-emitting

model

OE

that

can

not

be

converted

into

an

equivalent

basic

model

of

any

order

There

will

always

be

a

string

xT

that

distinguishes

the

non-emitting

model

OE

from

any

given

basic

model

because

the

non-emitting

model

can

encode

unbounded

dependencies

in

its

state

distribution

Lemma

A

-LSB-

pffl

-LRB-

xT

jOE

T

-RRB-

pm

-LRB-

xT

T

-RRB-

-RSB-

Proof

The

idea

of

the

proof

is

that

our

non-emitting

model

will

encode

the

first

symbol

of

the

string

xT

in

its

state

distribution

for

an

unbounded

distance

This

will

allow

it

to

predict

the

last

symbol

xT

using

its

knowledge

of

the

first

symbol

The

basic

model

will

only

be

able

predict

the

last

symbol

xT

using

the

preceding

n

symbols

and

therefore

when

T

is

greater

than

n

we

can

arrange

for

pffl

-LRB-

xT

jOE

T

-RRB-

to

differ

from

any

pm

-LRB-

xT

T

-RRB-

simply

by

our

choice

of

The

smallest

non-emitting

model

capable

of

exhibiting

the

required

behavior

has

order

Lower

order

non-emitting

models

are

equivalent

to

interpolated

models

of

the

same

order

with

the

same

parameters

The

non-emitting

transition

probabilities

and

the

interior

of

the

string

xT

will

be

chosen

so

that

the

non-emitting

model

is

either

in

an

order

state

or

an

order

state

with

no

way

to

transition

from

one

to

the

other

The

first

symbol

will

determine

whether

the

non-emitting

model

goes

to

the

order

state

or

stays

in

the

order

state

No

matter

what

probability

the

basic

model

assigns

to

the

final

symbol

xT

the

non-emitting

model

can

assign

a

different

probability

by

the

appropriate

choice

of

-LRB-

xT

-RRB-

and

-LRB-

xT

jxT

T

-RRB-

Consider

the

second

order

non-emitting

model

over

a

binary

alphabet

with

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

on

strings

in

When

then

will

be

predicted

using

the

order

model

-LRB-

-RRB-

and

all

subsequent

xt

will

be

predicted

by

the

second

order

model

-LRB-

xtjxt

t

-RRB-

When

then

all

subsequent

xt

will

be

predicted

by

the

order

model

-LRB-

xt

-RRB-

Thus

for

all

t

p

pffl

-LRB-

xtjxt

-RRB-

pffl

-LRB-

xtjxt

t?p

-RRB-

for

any

fixed

p

and

no

basic

model

is

equivalent

to

this

simple

non-emitting

model

Every

basic

model

is

a

non-emitting

model

with

the

appropriate

choice

of

non-emitting

transition

probabilities

Lemma

A

-LSB-

pffl

-LRB-

xT

T

-RRB-

pm

-LRB-

xT

jOE

T

-RRB-

-RSB-

Proof

A

basic

model

OE

hA

n

ffini

is

equivalent

to

a

non-emitting

model

hA

n

where

ffin

and

-LRB-

njxn

-RRB-

for

all

xn

In

the

hierarchical

parameterization

-LRB-

xn

-RRB-

for

all

xn

Therefore

the

class

Pffl

of

non-emitting

Markov

distributions

is

strictly

more

powerful

than

the

class

Pm

of

basic

Markov

distributions

Theorem

Pm

Pffl

Proof

Pm

Pffl

by

lemma

and

Pm

Pffl

by

lemma

Since

interpolated

models

and

backoff

models

are

equivalent

to

basic

Markov

models

we

have

as

a

corollary

that

non-emitting

Markov

models

are

strictly

more

powerful

than

interpolated

and

backoff

models

Note

that

non-emitting

Markov

models

are

considerably

less

powerful

than

the

full

class

of

stochastic

finite

state

automaton

because

their

states

are

Markovian

For

the

same

reason

non-emitting

models

are

also

less

powerful

than

the

full

class

of

hidden

Markov

models

Let

us

now

turn

to

the

algorithms

required

to

evaluate

the

probability

of

a

string

according

to

a

non-emitting

mixture

model

and

to

optimize

the

nonemitting

state

transitions

on

a

training

corpus

Estimation

Here

we

present

an

efficient

expectation-maximization

-LRB-

EM

-RRB-

algorithm

to

optimize

the

parameters

of

a

hierarchical

non-emitting

mixture

model

on

data

An

EM

algorithm

iteratively

maximizes

the

probability

of

the

training

data

according

to

the

model

by

computing

the

expectation

of

model

parameters

on

the

data

and

then

updating

the

model

parameters

in

order

to

maximize

those

expectations

-LSB-

-RSB-

The

non-emitting

mixture

model

is

sufficiently

expressive

that

any

maximum

likelihood

estimator

will

overfit

its

parameters

to

the

training

corpus

Unseen

events

will

be

assigned

zero

probability

and

the

overfit

model

will

fail

to

accurately

predict

the

future

The

traditional

solution

to

this

problem

for

interpolated

Markov

models

is

cross-estimation

-LSB-

-RSB-

Cross-estimation

repeatedly

partitions

the

training

data

into

two

blocks

and

optimizes

the

mixing

parameters

on

one

block

after

initializing

the

state

transition

parameters

on

the

other

block

We

present

a

traditional

cross-estimation

algorithm

for

hierarchical

nonemitting

models

We

begin

by

partitioning

the

training

corpus

into

a

fixed

set

of

blocks

B.

Ideally

our

partition

is

linguistically

meaningful

and

roughly

uniform

but

neither

condition

is

essential

For

example

we

might

divide

a

natural

language

text

corpus

on

sentence

paragraph

or

article

boundaries

Next

we

call

crossestimate

-LRB-

B

OE

-RRB-

on

our

hierarchical

non-emitting

model

OE

cross-estimate

-LRB-

B

OE

-RRB-

Until

convergence

Initialize

to

zero

For

each

block

Bi

in

B

Initialize

ffi

using

B?Bi

expectation-step

-LRB-

Bi

OE

-RRB-

maximization-step

-LRB-

OE

-RRB-

Initialize

ffi

using

B

The

variables

-LRB-

xi

-RRB-

and

-LRB-

xi

-RRB-

accumulate

expectations

for

the

non-emitting

state

transition

parameter

-LRB-

xi

-RRB-

-LRB-

xi

-RRB-

contains

the

expectation

of

emitting

a

symbol

in

state

xi

conditioned

on

being

in

state

xi

while

-LRB-

xi

-RRB-

contains

the

expectation

of

transitioning

to

without

emitting

a

symbol

conditioned

on

being

in

state

xi

Lines

enumerate

all

one-block

partitions

of

the

training

corpus

The

emitting

state

transitions

ffi

are

initialized

to

their

maximum

likelihood

estimates

on

the

larger

block

B?Bi

and

then

the

non-emitting

state

transitions

are

optimized

on

the

smaller

withheld

block

Bi

The

heart

of

the

algorithm

is

the

expectation-step

-LRB-

-RRB-

procedure

which

calculates

the

expectation

of

the

non-emitting

transitions

on

the

string

xb

and

then

increments

the

accumulators

expectation-step

-LRB-

xb

OE

-RRB-

ff

forward

-LRB-

xb

OE

-RRB-

backward

-LRB-

xb

OE

-RRB-

for

t

b

downto

for

i

upto

min

-LRB-

n

t

-RRB-

t

-LRB-

i

-RRB-

fft

-LRB-

i

-RRB-

-LRB-

t

-LRB-

i

-RRB-

-RRB-

t

-LRB-

i

-RRB-

t

-LRB-

i

-RRB-

fft

-LRB-

i

-RRB-

t

-LRB-

i

-RRB-

ffit

-LRB-

i

-RRB-

t

-LRB-

i

-RRB-

if

-LRB-

t

n

-RRB-

-LSB-

t

-LRB-

n

-RRB-

fft

-LRB-

n

-RRB-

t

-LRB-

n

-RRB-

ffit

-LRB-

n

-RRB-

t

-LRB-

n

-RRB-

-RSB-

The

forward

variable

fft

-LRB-

i

-RRB-

contains

the

probability

p

-LRB-

xt

ot

ijOE

-RRB-

that

the

model

OE

generated

the

prefix

xt

and

terminated

in

the

order

i

state

The

backward

variable

t

-LRB-

i

-RRB-

contains

the

probability

p

-LRB-

xt

bT

jxt

ot

i

OE

-RRB-

that

the

model

OE

generated

the

suffix

xbt

given

that

it

was

in

the

order

i

state

at

time

t.

To

simplify

the

notation

we

define

t

-LRB-

i

-RRB-

to

be

the

probability

-LRB-

xtt

i

-RRB-

of

emitting

a

symbol

from

the

ith

order

state

at

time

t

given

that

we

are

in

that

state

We

also

define

ffit

-LRB-

i

-RRB-

to

be

the

probability

ffii

-LRB-

xt

jxtt

i

-RRB-

of

the

emitting

transition

from

state

xtt

i

-RRB-

to

state

xt

t

i

The

expectation-step

-LRB-

-RRB-

algorithm

requires

O

-LRB-

nb

-RRB-

time

and

space

for

an

order

n

non-emitting

model

on

a

string

xb

of

length

b

A

comparable

interpolated

model

can

take

an

expectation

step

in

O

-LRB-

nb

-RRB-

time

and

O

-LRB-

-RRB-

space

-LSB-

-RSB-

While

the

difference

between

O

-LRB-

nb

-RRB-

and

O

-LRB-

-RRB-

space

can

be

considerable

the

additional

space

requirements

of

the

non-emitting

algorithm

are

small

when

compared

to

the

cost

of

storing

all

the

model

parameters

An

order

n

mixture

model

has

O

-LRB-

nT

-RRB-

parameters

for

a

training

corpus

of

size

T

and

the

training

corpus

is

typically

an

order

of

magnitude

larger

than

the

withheld

block

forward

-LRB-

xT

OE

-RRB-

-LRB-

-RRB-

for

t

upto

T

for

i

min

-LRB-

n

t

-RRB-

downto

fft

-LRB-

i

-RRB-

fft

-LRB-

i

-RRB-

-LRB-

t

-LRB-

i

-RRB-

-RRB-

fft

-LRB-

i

-RRB-

fft

-LRB-

i

-RRB-

t

-LRB-

i

-RRB-

ffit

-LRB-

i

-RRB-

if

-LRB-

t

n

-RRB-

-LSB-

fft

-LRB-

n

-RRB-

fft

-LRB-

n

-RRB-

t

-LRB-

n

-RRB-

ffit

-LRB-

n

-RRB-

-RSB-

return

-LRB-

ff

-RRB-

backward

-LRB-

xT

OE

-RRB-

for

i

upto

min

-LRB-

n

T

-RRB-

T

-LRB-

i

-RRB-

T

-LRB-

i

-RRB-

ffiT

-LRB-

i

-RRB-

if

-LRB-

T

n

-RRB-

-LSB-

T

-LRB-

n

-RRB-

T

-LRB-

n

-RRB-

ffiT

-LRB-

n

-RRB-

-RSB-

for

t

T

downto

for

i

upto

min

-LRB-

n

t

-RRB-

t

-LRB-

i

-RRB-

-LRB-

t

-LRB-

i

-RRB-

-RRB-

t

-LRB-

i

-RRB-

t

-LRB-

i

-RRB-

t

-LRB-

i

-RRB-

ffit

-LRB-

i

-RRB-

t

-LRB-

i

-RRB-

if

-LRB-

t

n

-RRB-

-LSB-

t

-LRB-

n

-RRB-

t

-LRB-

n

-RRB-

ffit

-LRB-

n

-RRB-

T

-LRB-

n

-RRB-

-RSB-

return

-LRB-

-RRB-

The

forward

-LRB-

-RRB-

and

backward

-LRB-

-RRB-

algorithms

each

require

O

-LRB-

nT

-RRB-

time

and

space

It

is

possible

to

evaluate

the

probability

pffl

-LRB-

xT

jOE

-RRB-

of

a

string

xT

according

to

an

order

n

non-emitting

model

OE

in

O

-LRB-

nT

-RRB-

time

and

O

-LRB-

n

-RRB-

space

In

contrast

it

is

possible

to

evaluate

the

probability

pc

-LRB-

xT

jOE

-RRB-

according

to

an

interpolated

model

in

O

-LRB-

nT

-RRB-

time

and

O

-LRB-

-RRB-

space

Again

the

small

additional

cost

in

space

is

negligible

when

compared

to

the

cost

of

storing

the

model

parameters

Having

done

all

the

work

in

the

expectation

step

the

maximization

step

is

straightforward

maximization-step

-LRB-

OE

-RRB-

Forall

states

xi

in

A

n

-LRB-

xi

-RRB-

-LRB-

xi

-RRB-

=-LRB-

-LRB-

xi

-RRB-

-LRB-

xi

-RRB-

-RRB-

Line

reestimates

each

non-emitting

state

transition

parameter

-LRB-

xi

-RRB-

as

the

expectation

of

emitting

a

symbol

from

that

state

divided

by

the

expectation

of

being

in

that

state

In

order

to

ensure

that

no

non-emitting

state

transition

parameter

-LRB-

xi

-RRB-

is

ever

reestimated

to

or

we

typically

initialize

each

accumulator

to

a

small

positive

number

-LRB-

eg.

-RRB-

instead

of

zero

When

parameters

are

tied

then

their

and

expectations

must

be

pooled

before

they

are

updated

Let

o

-LRB-

xi

-RRB-

be

the

equivalence

class

of

xi

under

the

tying

scheme

o

For

simplicity

imagine

o

-LRB-

xi

-RRB-

to

be

an

index

All

algorithms

in

this

section

would

use

the

tied

parameter

-LRB-

o

-LRB-

xi

-RRB-

-RRB-

instead

of

the

untied

parameter

-LRB-

xi

-RRB-

The

tied-expectation-step

-LRB-

-RRB-

algorithm

would

increment

the

-LRB-

o

-LRB-

xi

-RRB-

-RRB-

and

-LRB-

o

-LRB-

xi

-RRB-

-RRB-

accumulators

and

the

tied-maximization-step

-LRB-

-RRB-

algorithm

would

be

as

follows

tied-maximization-step

-LRB-

OE

ss

-RRB-

Forall

classes

i

in

o

-LRB-

A

n

-RRB-

-LRB-

i

-RRB-

-LRB-

i

-RRB-

=-LRB-

-LRB-

i

-RRB-

-LRB-

i

-RRB-

-RRB-

In

some

situations

cross-estimation

may

be

approximated

by

forward-estimation

Like

cross-estimation

forward-estimation

initializes

the

ffi

parameters

on

one

text

block

and

optimizes

the

parameters

on

another

block

The

only

difference

is

that

forward-estimation

uses

only

a

single

text

partition

whereas

crossestimation

uses

all

one-block

text

partitions

As

result

forward-estimation

is

considerably

faster

than

cross-estimation

both

in

the

amount

of

time

required

per

iteration

and

in

the

number

of

iterations

until

convergence

Unfortunately

it

can

lead

to

inferior

results

when

there

are

too

many

mixing

parameters

forward-estimate

-LRB-

Bffi

B

OE

-RRB-

Until

convergence

Initialize

to

zero

Initialize

ffi

using

Bffi

expectation-step

-LRB-

B

OE

-RRB-

maximization-step

-LRB-

OE

-RRB-

Initialize

ffi

using

Bffi

-LSB-

B

Implementation

Note

Unless

the

corpus

and

the

alphabet

size

are

very

small

then

the

fft

-LRB-

i

-RRB-

and

t

-LRB-

i

-RRB-

values

used

in

the

expectation-step

-LRB-

-RRB-

procedure

will

exceed

the

representational

range

of

double

precision

IEEE

floating

point

numbers

When

this

happens

a

floating

point

exception

will

occur

and

an

alternate

representation

must

be

used

for

the

probability

values

The

simplest

approach

is

to

use

a

logarithmic

representation

Multiplication

and

division

of

probability

values

is

straightforward

in

a

logarithmic

representation

log

-LRB-

x

y

-RRB-

log

-LRB-

x

-RRB-

log

-LRB-

y

-RRB-

log

-LRB-

x

y

-RRB-

log

-LRB-

x

-RRB-

log

-LRB-

y

-RRB-

Addition

of

logarithmic

probability

values

is

more

costly

and

care

must

be

taken

to

avoid

underflow

log

-LRB-

x

y

-RRB-

ae

log

-LRB-

x

-RRB-

if

log

-LRB-

y

-RRB-

log

-LRB-

x

-RRB-

log

-LRB-

x

-RRB-

log

-LRB-

exp

-LRB-

log

-LRB-

y

-RRB-

log

-LRB-

x

-RRB-

-RRB-

-RRB-

otherwise

Here

is

the

smallest

representable

exponent

for

example

for

IEEE

double

precision

floating

point

numbers

when

the

logarithms

are

natural

-LRB-

ie.

base

e

-RRB-

This

test

is

necessary

to

avoid

underflow

in

the

call

to

exp

-LRB-

-RRB-

While

it

is

simple

to

implement

logarithmic

arithmetic

can

be

times

slower

than

straight

probability

arithmetic

depending

on

the

speed

of

the

floating

point

unit

and

the

math

library

provided

with

the

operating

system

For

this

reason

our

implementation

used

an

extended

exponent

representation

from

the

library

of

practical

abstractions

-LSB-

-RSB-

This

module

provides

single

precision

floating

point

numbers

with

bit

exponents

It

is

to

times

faster

than

the

logarithmic

representation

depending

on

the

machine

When

computation

time

is

at

a

premium

then

the

most

effective

solution

is

to

periodically

scale

the

probability

values

in

the

fft

-LRB-

i

-RRB-

and

t

-LRB-

i

-RRB-

arrays

to

keep

them

in

an

acceptable

range

Scaling

is

more

difficult

to

implement

than

logarithmic

arithmetic

or

arithmetic

and

it

is

inherently

nonmodular

Empirical

Results

The

ultimate

measure

of

a

statistical

model

is

its

predictive

performance

in

the

domain

of

interest

To

take

the

true

measure

of

non-emitting

models

for

natural

language

texts

we

evaluate

their

performance

as

character

models

on

the

Brown

corpus

-LSB-

-RSB-

and

as

word

models

on

the

Wall

Street

Journal

Our

results

show

that

the

non-emitting

Markov

model

consistently

gives

better

predictions

than

the

traditional

interpolated

Markov

model

under

equivalent

experimental

conditions

In

all

cases

we

compare

non-emitting

and

interpolated

models

of

identical

model

orders

with

the

same

number

of

parameters

Note

that

the

non-emitting

bigram

and

the

interpolated

bigram

are

equivalent

Corpus

Alphabet

Size

Blocks

Brown

WSJ

WSJ

All

values

were

initialized

uniformly

to

and

then

optimized

using

crossestimation

on

the

first

of

each

corpus

The

remaining

percent

of

each

corpus

was

used

to

evaluate

model

performance

We

consider

a

single

parameter

tying

scheme

in

which

all

states

with

the

same

frequency

and

diversity

are

considered

equivalent

The

frequency

c

-LRB-

xi

-RRB-

of

a

state

is

the

number

of

times

that

the

string

xi

occurred

in

the

training

corpus

The

diversity

q

-LRB-

xi

-RRB-

jfy

c

-LRB-

xiy

-RRB-

gj

of

a

state

is

the

number

of

distinct

symbols

observed

in

the

state

Experience

with

multinomial

prediction

suggests

that

frequency

and

diversity

are

necessary

to

accurately

estimate

the

likelihood

of

novel

symbols

-LSB-

-RSB-

In

related

work

-LSB-

-RSB-

Thomas

compares

the

performance

of

the

interpolated

and

non-emitting

models

on

the

Brown

corpus

and

Wall

Street

Journal

with

ten

different

parameter

tying

schemes

His

experiments

confirm

that

some

parameter

tying

schemes

improve

model

performance

although

to

a

lesser

degree

when

cross-estimation

is

used

The

non-emitting

model

consistently

outperformed

the

interpolated

model

on

both

corpora

for

all

ten

parameter

tying

schemes

Thomas

shows

that

our

frequency-diversity

parameter

tying

scheme

is

one

of

the

more

effective

parameter

schemes

Brown

Corpus

Our

first

set

of

experiments

were

with

character

models

on

the

Brown

corpus

-LSB-

-RSB-

The

Brown

corpus

is

an

eclectic

collection

of

English

prose

containing

characters

partitioned

into

files

We

performed

iterations

of

cross

estimation

on

blocks

Results

are

reported

as

per-character

test

message

entropies

-LRB-

bits/char

-RRB-

p

-LRB-

yvjv

-RRB-

The

non-emitting

model

outperforms

the

interpolated

model

for

all

nontrivial

model

orders

particularly

for

larger

model

orders

The

non-emitting

model

is

considerably

less

prone

to

overfitting

After

EM

iterations

the

untied

order

non-emitting

model

scores

bits/char

while

the

untied

order

interpolated

model

scores

bits/char

The

untied

non-emitting

model

even

outperforms

the

tied

interpolated

model

for

all

nontrivial

model

orders

Model

Interpolation

Non-Emitting

order

untied

tied

untied

tied

We

also

compared

the

performance

of

our

techniques

with

two

new

interpolation

schemes

recently

proposed

by

Potamianos

and

Jelinek

-LSB-

-RSB-

Their

DI-TD

scheme

uses

hierarchical

state-conditional

interpolation

-LRB-

xi

-RRB-

variable-width

frequency

order

parameter

tying

and

top-down

optimization

on

one

withheld

block

Their

DI-BU

scheme

uses

general

state-conditional

interpolation

-LRB-

jjxi

-RRB-

variable-width

frequency

order

parameter

tying

and

bottom-up

optimization

on

one

withheld

block

The

comparison

is

performed

on

a

modified

version

of

the

Brown

corpus

which

they

provided

to

us

This

modified

corpus

eliminates

the

unusual

punctuation

of

the

original

Brown

corpus

reduces

the

alphabet

size

from

to

and

separates

distinct

linguistic

tokens

with

single

spaces

Corpus

Alphabet

Size

Train

Test

Blocks

Brown

-LRB-

std

-RRB-

Brown

-LRB-

JHU

-RRB-

Another

difference

between

the

Potamianos-Jelinek

validation

paradigm

and

ours

lies

in

how

the

corpus

is

partitioned

into

training

and

testing

blocks

In

our

experiments

the

test

block

was

the

last

of

the

Brown

corpus

-LCB-

the

last

characters

from

plus

all

files

from

through

inclusive

In

Potamianos

and

Jelinek

's

experiments

the

test

block

consisted

of

complete

sentences

chosen

uniformly

from

the

entire

-LRB-

modified

-RRB-

Brown

corpus

To

this

comparison

we

added

the

original

interpolation

schemes

of

Jelinek

and

Mercer

-LSB-

-RSB-

under

iterations

of

forward-estimation

-LRB-

DI-FE

-RRB-

and

cross-estimation

-LRB-

DI-CE

-RRB-

Both

models

used

hierarchical

state-conditional

interpolation

-LRB-

xi

-RRB-

and

straight

frequency

diversity

parameter

tying

We

also

added

the

hierarchical

non-emitting

model

with

straight

frequency

diversity

parameter

tying

and

iterations

of

forward-estimation

-LRB-

NE-FE

-RRB-

and

crossoptimization

-LRB-

NE-CE

-RRB-

The

results

are

summarized

in

the

following

table

as

mean

test

message

entropies

-LRB-

bits/char

-RRB-

Model

Interpolation

Non-Emitting

order

DI-TD

DI-BU

DI-FE

DI-CE

NE-FE

NE-CE

The

non-emitting

model

consistently

outperforms

all

interpolation

schemes

at

all

model

orders

above

by

a

significant

margin

The

original

Jelinek-Mercer

interpolation

scheme

also

tends

to

outperform

the

two

new

DI-TD

and

DI-BU

schemes

at

higher

model

orders

for

both

forward-estimation

-LRB-

DI-FE

-RRB-

and

crossestimation

-LRB-

DI-CE

-RRB-

Note

also

that

the

best

order

result

in

the

Potamianos-Jelinek

paradigm

-LRB-

bits/char

-RRB-

is

considerably

better

than

the

best

order

result

in

our

validation

paradigm

-LRB-

bits/char

-RRB-

We

believe

this

is

partially

attributable

to

the

reduced

alphabet

size

of

the

modified

corpus

and

principally

due

to

the

difference

in

the

two

train-test

partitions

The

prediction

problem

posed

by

our

paradigm

is

more

difficult

because

the

last

of

the

Brown

files

are

appreciably

different

than

the

first

of

the

files

WSJ

The

second

set

of

experiments

was

on

the

Wall

Street

Journal

corpus

which

contains

words

Our

vocabulary

consisted

of

the

words

that

occurred

at

least

times

in

the

entire

WSJ

corpus

All

out-of-vocabulary

words

were

mapped

to

a

unique

OOV

symbol

We

performed

iterations

of

cross

estimation

on

blocks

Following

standard

practice

in

the

speech

recognition

community

results

are

reported

as

per-word

test

message

perplexities

p

-LRB-

yv

jv

-RRB-

The

perplexity

represents

the

effective

alphabet

size

Again

the

non-emitting

model

outperforms

the

interpolated

model

for

all

nontrivial

model

orders

even

without

parameter

tying

Model

Interpolation

Non-Emitting

order

untied

tied

untied

tied

WSJ

The

third

set

of

experiments

was

on

the

Wall

Street

Journal

corpus

which

contains

words

Our

vocabulary

consisted

of

the

words

that

occurred

at

least

times

in

the

entire

WSJ

corpus

Again

all

out-of-vocabulary

words

were

mapped

to

a

unique

OOV

symbol

We

performed

iterations

of

cross

estimation

on

blocks

Results

are

reported

as

test

message

perplexities

As

with

the

WSJ

corpus

the

non-emitting

model

outperforms

the

interpolated

model

for

all

nontrivial

model

orders

even

without

parameter

tying

Model

Interpolation

Non-Emitting

order

untied

tied

untied

tied

Posthoc

Analysis

In

order

to

understand

the

striking

empirical

advantage

of

the

non-emitting

model

over

the

interpolated

model

we

conducted

the

following

experiment

We

induced

order

interpolated

and

non-emitting

models

from

the

Brown

corpus

using

forward

estimation

with

no

parameter

tying

This

configuration

was

chosen

to

maximize

the

performance

difference

between

the

two

models

The

resulting

interpolated

model

predicts

the

Brown

test

corpus

with

bits/char

while

the

resulting

non-emitting

model

predicts

the

Brown

test

corpus

with

bits/char

Figure

visualizes

the

overwhelming

advantage

of

the

nonemitting

model

in

this

experiment

e

e

Prediction

Ratio

Time

Brown

Corpus

-LRB-

Iteration/None

-RRB-

Figure

The

relative

performance

of

the

non-emitting

and

interpolated

models

through

the

Brown

test

corpus

on

a

character

by

character

basis

The

horizontal

access

represents

time

through

the

test

corpus

proceeding

from

left

to

right

The

vertical

access

represents

the

ratio

pffl

-LRB-

xtjxt

-RRB-

pc

-LRB-

xtjxt

-RRB-

when

pffl

-LRB-

xtjxt

-RRB-

pc

-LRB-

xtjxt

-RRB-

and

the

ratio

pc

-LRB-

xtjxt

-RRB-

pffl

-LRB-

xtjxt

-RRB-

otherwise

Thus

each

positive

spike

represents

a

win

for

the

non-emitting

model

and

each

negative

spike

represents

a

win

for

the

interpolated

model

The

magnitude

of

the

win

is

encoded

by

the

magnitude

of

the

spike

The

largest

win

for

the

non-emitting

model

is

a

factor

of

-LRB-

bits

-RRB-

The

largest

win

for

the

interpolated

model

is

a

factor

of

-LRB-

bits

-RRB-

The

following

table

shows

the

mean

state

order

occupancy

statistics

for

the

two

models

on

the

Brown

corpus

Order

Interpolated

Non-Emitting

As

might

be

expected

the

interpolated

model

spends

more

time

than

the

nonemitting

model

in

the

higher

order

states

-LRB-

orders

-RRB-

It

is

arguably

more

surprising

however

that

the

interpolated

model

also

spends

more

time

in

the

lower

order

states

-LRB-

orders

-RRB-

One

of

the

smaller

positive

spikes

in

the

figure

occurs

when

both

models

try

to

predict

the

space

that

follows

the

string

but

now

Keith

in

the

Brown

test

corpus

Unfortunately

the

string

Keith

does

not

occur

in

the

training

corpus

Nonetheless

the

non-emitting

model

assigns

times

more

probability

than

the

interpolated

model

to

the

event

that

a

space

will

follow

the

string

Keith

According

to

the

non-emitting

model

a

space

will

follow

the

string

Keith

with

probability

The

interpolated

model

assigns

probability

to

the

same

event

The

reason

is

somewhat

subtle

On

the

training

corpus

the

string

eith

is

followed

by

the

letter

e

with

near

certainty

-LRB-

-RRB-

As

a

result

-LRB-

eith

-RRB-

approaches

unity

in

both

the

interpolated

and

non-emitting

models

Since

the

model

order

is

sufficiently

high

the

interpolated

model

will

use

the

eith

state

whenever

it

occurs

and

no

higher

order

state

is

preferred

-LRB-

see

figure

-RRB-

The

hierarchical

non-emitting

model

has

no

such

freedom

-LRB-

see

figure

-RRB-

In

order

to

reach

the

eith

state

it

must

accurately

predict

every

symbol

in

the

string

eith

Otherwise

it

will

be

forced

to

a

lower

order

state

along

the

way

The

transition

to

a

lower

order

state

occurs

when

the

non-emitting

model

attempts

to

predict

the

symbol

t

from

the

state

ei

Since

ei

is

rarely

followed

by

t

in

the

training

corpus

-LRB-

-RRB-

the

non-emitting

model

is

forced

into

the

lower

order

state

i

from

which

it

is

able

to

predict

the

symbol

t

with

greater

probability

-LRB-

-RRB-

As

a

result

the

non-emitting

model

is

never

able

to

reach

the

eith

state

Instead

it

must

predict

the

space

after

Keith

using

the

state

ith

This

works

quite

well

because

ith

is

followed

by

with

high

probability

in

the

training

corpus

-LRB-

-RRB-

K

e

i

t

h

Figure

State

occupancy

probabilities

for

the

order

interpolated

model

on

part

of

the

Brown

test

corpus

-LRB-

bits/char

-RRB-

The

horizontal

axis

represents

the

position

in

the

test

string

and

the

vertical

access

represents

the

hidden

state

order

The

bottom

column

shows

the

conditional

probability

of

the

symbol

given

the

hidden

state

distribution

Thus

the

interpolated

model

is

in

the

order

state

eith

with

probability

at

least

when

predicting

the

final

symbol

and

it

assigns

probability

to

this

symbol

K

e

i

t

h

Figure

State

occupancy

probabilities

for

the

order

non-emitting

model

on

part

of

the

Brown

test

corpus

-LRB-

bits/char

-RRB-

The

horizontal

axis

represents

the

position

in

the

test

string

and

the

vertical

access

represents

the

hidden

state

order

The

bottom

column

shows

the

conditional

probability

of

the

symbol

given

the

hidden

state

distribution

Thus

the

non-emitting

model

is

in

the

order

state

ith

with

probability

when

predicting

the

final

symbol

and

it

assigns

probability

to

this

symbol

Quality

Tying

This

posthoc

analysis

led

John

Lafferty

-LRB-

personal

communication

-RRB-

to

suggest

that

the

interpolated

model

might

be

able

to

approximate

the

empirical

performance

of

the

non-emitting

model

with

a

suitable

parameter

tying

scheme

According

to

the

non-emitting

model

two

states

should

be

considered

equivalent

if

they

are

equally

effective

at

predicting

the

future

and

they

are

equally

well

predicted

by

the

model

A

state

is

well-predicted

if

the

string

that

it

represents

is

assigned

high

probability

relative

to

the

other

states

available

at

the

time

A

state

provides

strong

predictions

if

the

entropy

of

its

emitting

state

transition

probabilities

is

low

Let

the

quality

of

a

state

be

a

measure

of

how

well

the

state

is

predicted

by

the

model

The

simplest

quality

measures

ignore

the

contribution

of

nonemitting

transitions

to

the

probability

of

a

state

One

such

quality

measure

is

the

mean

empirical

posterior

probability

of

the

state

The

mean

empirical

posterior

is

the

empirical

expectation

ffi

-LSB-

xi

-RSB-

of

the

state

divided

by

its

frequency

c

-LRB-

xi

-RRB-

The

empirical

expectation

ffi

-LSB-

xijyT

-RSB-

of

an

ith

order

state

xi

in

an

order

n

mixture

Markov

model

with

respect

to

a

string

yT

is

computed

as

follows

ffi

-LSB-

xijyT

-RSB-

X

ft

xi

ytt

ig

ffi

-LRB-

ot

ijyt

-RRB-

with

the

empirical

posterior

ffi

-LRB-

o

ijyt

-RRB-

ffi

-LRB-

ytt

i

-RRB-

Pnj

ffi

-LRB-

ytt

j

-RRB-

Note

that

ffi

-LSB-

xijyT

-RSB-

may

be

calculated

for

all

states

in

O

-LRB-

nT

-RRB-

time

Alternately

the

empirical

posterior

ffi

-LRB-

o

ijyt

-RRB-

of

the

ith

order

state

at

time

t

could

be

weighted

by

its

predictive

success

log

ffi

-LRB-

yt

jytt

i

-RRB-

A

further

refinement

is

to

compute

the

quality

of

a

state

on

withheld

data

as

a

kind

of

cross-validation

In

order

to

construct

a

parameter

tying

scheme

the

quality

measure

must

be

quantized

to

a

finite

number

of

levels

Once

quantized

it

may

be

conjoined

with

another

equivalence

classification

to

form

a

joint

equivalence

classification

where

two

states

are

equivalent

iff

they

are

equivalent

in

both

classifications

Conclusion

In

this

report

we

propose

a

time

series

model

that

combines

Markovian

events

of

varying

orders

using

non-emitting

transitions

We

prove

that

the

resulting

class

of

non-emitting

Markov

models

is

strictly

more

powerful

than

the

class

of

Markov

models

including

interpolated

and

backoff

models

More

importantly

our

empirical

investigation

reveals

that

the

non-emitting

model

consistently

outperforms

the

strongest

interpolated

Markov

models

on

natural

language

texts

with

only

a

modest

increase

in

computational

requirements

The

expressive

power

of

the

non-emitting

model

comes

from

its

ability

to

represent

additional

information

in

its

state

order

distribution

To

prove

that

the

non-emitting

model

was

strictly

more

powerful

than

any

Markov

model

we

used

the

state

order

distribution

to

represent

an

unbounded

dependency

In

our

posthoc

analysis

we

revealed

how

the

model

uses

its

hidden

state

order

distribution

to

remember

the

short-term

effectiveness

of

all

available

Markovian

states

The

non-emitting

model

succeeds

empirically

because

it

imposes

a

pseudoBayesian

discipline

on

maximum

likelihood

techniques

The

interpolated

model

will

favor

a

high-order

state

if

it

provides

strong

predictions

on

withheld

data

The

non-emitting

model

will

favor

a

high-order

state

if

the

state

provides

strong

predictions

on

withheld

data

and

it

is

well-predicted

by

the

model

In

order

to

reach

a

high

order

state

the

non-emitting

model

must

assign

high

probability

to

each

symbol

in

that

state

Otherwise

the

non-emitting

model

will

be

forced

to

transition

to

a

lower

order

state

at

a

previous

time

step

and

will

not

be

able

to

reach

the

high

order

state

Thus

the

state

occupancies

of

the

non-emitting

model

are

influenced

as

much

by

their

prior

probabilities

-LRB-

pseudo-Bayes

-RRB-

as

their

past

ability

to

predict

the

future

-LRB-

maximum

likelihood

-RRB-

Finally

we

note

the

use

of

non-emitting

transitions

is

a

general

modeling

technique

that

may

be

employed

in

any

time

series

model

for

symbolic

domains

and

for

continuous

domains

References

-LSB-

-RSB-

Bahl

L.

R.

Brown

P.

F.

de

Souza

P.

V.

Mercer

R.

L.

and

Nahamoo

D

A

fast

algorithm

for

deleted

interpolation

In

Proc

EU

ROSPEECH

-LRB-

Genoa

-RRB-

pp.

-LCB-

-LSB-

-RSB-

Baum

L.

and

Eagon

J

An

inequality

with

applications

to

statistical

estimation

for

probabilistic

functions

of

a

Markov

process

and

to

models

for

ecology

Bull

AMS

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Baum

L.

Petrie

T.

Soules

G.

and

Weiss

N

A

maximization

technique

occurring

in

the

statistical

analysis

of

probabilistic

functions

of

Markov

chains

Ann

Math

Stat

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Brown

P.

Pietra

V.

D.

Pietra

S.

D.

Lai

J.

and

Mercer

R

An

estimate

of

an

upper

bound

for

the

entropy

of

English

Computational

Linguistics

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Cleary

J.

and

Witten

I.

Data

compression

using

adaptive

coding

and

partial

string

matching

IEEE

Trans

Comm

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Dempster

A.

Laird

N.

and

Rubin

D.

Maximum

likelihood

from

incomplete

data

via

the

EM

algorithm

J.

Royal

Statist

Soc

Ser

B

-LRB-

methodological

-RRB-

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Francis

W.

N.

and

Kucera

H.

Frequency

analysis

of

English

usage

lexicon

and

grammar

Houghton

Mifflin

Boston

-LSB-

-RSB-

Good

I.

J

The

population

frequencies

of

species

and

the

estimation

of

population

parameters

Biometrika

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Good

I.

J

The

Estimation

of

Probabilities

MIT

Press

Cambridge

MA

-LSB-

-RSB-

H.

Ney

U.

Essen

R.

K.

On

the

estimation

of

small

probabilities

by

leaving-one-out

IEEE

Trans

PAMI

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Howard

P.

and

Vitter

J.

Practical

implementations

of

arithmetic

coding

In

Image

and

Text

Compression

J.

Storer

Ed

Kluwer

Academic

Norwell

MA

pp.

-LCB-

-LSB-

-RSB-

Jelinek

F.

and

Mercer

R.

L.

Interpolated

estimation

of

Markov

source

parameters

from

sparse

data

In

Pattern

Recognition

in

Practice

-LRB-

Amsterdam

May

-LCB-

-RRB-

E.

S.

Gelsema

and

L.

N.

Kanal

Eds.

North

Holland

pp.

-LCB-

-LSB-

-RSB-

Katz

S.

Estimation

of

probabilities

from

sparse

data

for

the

language

model

component

of

a

speech

recognizer

IEEE

Trans

ASSP

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

MacKay

D.

J.

and

Peto

L.

C.

B

A

hierarchical

Dirichlet

language

model

Natural

Language

Engineering

-LRB-

-RRB-

-LSB-

-RSB-

Moffat

A.

Implementing

the

PPM

data

compresion

scheme

IEEE

Trans

Communications

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Potamianos

G.

and

Jelinek

F

The

study

of

n-gram

and

decision

tree

letter

language

modeling

methods

CLSP

Johns

Hopkins

University

Baltimore

MD

April

-LSB-

-RSB-

Riccardi

G.

Pieraccini

R.

and

Bocchieri

E.

Stochastic

automata

for

language

modeling

Computer

Speech

and

Language

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Rissanen

J.

Probability

estimation

for

symbols

observed

or

not

In

International

Symposium

on

Information

Theory

-LRB-

Quebec

CN

September

-RRB-

IEEE

-LSB-

-RSB-

Rissanen

J

A

universal

data

compression

system

IEEE

Trans

Inform

Theory

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Rissanen

J.

Complexity

of

strings

in

the

class

of

Markov

sources

IEEE

Trans

Inform

Theory

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Ristad

E.

S

A

natural

law

of

succession

Tech

Rep.

Department

of

Computer

Science

Princeton

University

Princeton

NJ

May

-LSB-

-RSB-

Ristad

E.

S.

and

Thomas

R.

G.

New

techniques

for

context

modeling

In

Proc

Annual

Meeting

of

the

ACL

-LRB-

Cambridge

MA

June

-RRB-

-LSB-

-RSB-

Ristad

E.

S.

and

Thomas

R.

G.

Hierarchical

non-emitting

Markov

models

In

Proc

Annual

Meeting

of

the

ACL

-LRB-

Madrid

July

-LCB-

-RRB-

pp.

-LCB-

-LSB-

-RSB-

Ristad

E.

S.

and

Yianilos

P.

N.

Library

of

practical

abstractions

release

ftp

-LSB-

-RSB-

Thomas

R.

G.

Mixture

Models

for

Natural

Language

Processing

PhD

thesis

Department

of

Computer

Science

Princeton

University

Princeton

NJ

January

-LSB-

-RSB-

Willems

F.

M.

J.

Shtarkov

Y.

M.

and

Tjalkens

T.

J

The

context-tree

weighting

method

basic

properties

IEEE

Trans

Inform

Theory

-LRB-

-RRB-

-LCB-

A

Backoff

The

backoff

model

is

arguably

the

most

widely

used

statistical

language

model

due

in

large

part

to

its

ease

of

implementation

computational

efficiency

reasonable

performance

at

lower

model

orders

and

an

influential

paper

-LSB-

-RSB-

Backoff

models

are

also

widely

used

in

the

data

compression

community

in

large

part

due

to

their

computational

efficiency

-LSB-

-RSB-

Here

we

review

the

backoff

model

establish

the

equivalence

of

backoff

models

and

basic

Markov

models

and

then

specify

a

class

of

non-emitting

backoff

models

that

is

strictly

more

powerful

than

the

class

of

traditional

backoff

models

In

a

backoff

model

event

probabilities

are

combined

according

to

a

partial

order

Typically

higher

order

events

are

preferred

over

lower

order

events

The

event

probabilities

are

rescaled

as

we

move

through

the

partial

order

so

that

the

derived

probability

function

is

valid

The

efficacy

of

the

backoff

model

depends

the

events

that

are

included

in

the

model

their

individual

probabilities

and

the

order

in

which

they

are

combined

Formally

a

hierarchical

backoff

model

hA

E

ffii

consists

of

an

alphabet

A

a

dictionary

E

of

selected

state

transitions

E

A

A

and

the

state

transition

probabilities

ffi

E

-LSB-

-RSB-

The

state

transition

probabilities

ffi

are

extended

to

an

unbounded

domain

by

selecting

the

maximal

suffix

of

the

history

that

appears

with

the

relevant

symbol

in

the

dictionary

E

of

state

transitions

pb

-LRB-

yjxt

-RRB-

ae

ffi

-LRB-

yjxt

-RRB-

if

hxt

yi

E

-LRB-

xt

-RRB-

pb

-LRB-

-RRB-

otherwise

-LRB-

-RRB-

where

-LRB-

xt

-RRB-

rescales

the

conditional

probability

distribution

as

we

backoff

from

higher

order

events

to

lower

order

events

-LRB-

xi

-RRB-

-LRB-

ffi

-LRB-

E

-LRB-

xi

-RRB-

jxi

-RRB-

-RRB-

-LRB-

pb

-LRB-

E

-LRB-

xi

-RRB-

-RRB-

-RRB-

and

E

-LRB-

xi

-RRB-

is

the

set

of

symbols

available

in

the

context

xi

E

-LRB-

xi

-RRB-

fy

xiy

Eg

The

rescalar

-LRB-

xi

-RRB-

is

computed

directly

form

the

transition

probabilities

ffi

-LRB-

jxi

-RRB-

in

conjunction

with

the

transition

dictionary

E

It

is

not

a

free

parameter

A

hierarchical

backoff

model

is

a

valid

probability

model

if

the

dictionary

E

includes

every

th

order

state

transition

-LCB-

ffflg

A

ae

E

-LCB-

and

every

ffi

-LRB-

E

-LRB-

xi

-RRB-

jxi

-RRB-

is

a

valid

probability

function

A

backoff

model

is

nonzero

for

all

strings

A

if

every

ffi

-LRB-

yjxi

-RRB-

is

nonzero

and

no

ffi

-LRB-

E

-LRB-

xi

-RRB-

jxi

-RRB-

is

unity

when

E

-LRB-

xi

-RRB-

ae

A

In

order

to

induce

a

hierarchical

backoff

model

from

data

we

must

select

the

state

transition

dictionary

and

estimate

its

probabilities

One

simple

-LCB-

but

highly

effective

-LCB-

selection

technique

is

to

include

every

state

transition

whose

frequency

exceeds

a

fixed

threshold

that

may

depend

on

the

state

order

More

effective

selection

techniques

require

significant

computational

resources

-LSB-

-RSB-

The

state

transition

probabilities

ffi

-LRB-

yjxt

-RRB-

are

typically

assigned

by

multinomial

estimates

either

as

conditional

events

yjxi

in

the

symbol

alphabet

A

or

as

joint

events

xiy

in

the

string

alphabet

Ai

The

most

widely

used

multinomial

estimates

for

statistical

language

modeling

employ

some

form

of

discounting

-LSB-

-RSB-

although

other

estimators

have

also

been

shown

to

be

effective

-LSB-

-RSB-

A

valid

backoff

model

whose

event

dictionary

E

is

a

subset

of

An

can

be

converted

into

an

equivalent

basic

Markov

model

of

order

n

simply

by

setting

-LRB-

xtjxt

t?n

-RRB-

equal

to

pb

-LRB-

xtjxt

t?n

-RRB-

Every

basic

model

is

a

backoff

model

with

a

complete

state

transition

dictionary

Consequently

the

class

of

backoff

models

is

extensionally

equivalent

to

the

class

of

basic

Markov

models

The

hierarchical

non-emitting

backoff

model

hA

E

ffii

has

the

same

parameterization

as

the

traditional

backoff

model

The

only

difference

is

that

the

backoff

from

the

state

xi

to

its

maximal

proper

suffix

is

permanent

in

the

non-emitting

backoff

model

pffl

-LRB-

yj

jxi

-RRB-

ae

ffi

-LRB-

-RRB-

pffl

-LRB-

-RRB-

if

hxi

E

-LRB-

xi

-RRB-

pb

-LRB-

yj

OE

-RRB-

otherwise

-LRB-

-RRB-

The

rescalar

-LRB-

xi

-RRB-

is

identical

in

both

version

of

the

backoff

model

The

class

of

non-emitting

backoff

models

is

strictly

more

powerful

than

the

class

of

basic

Markov

models

by

a

similar

argument

as

in

lemma

The

only

difference

is

that

the

presence

or

absence

of

a

state

transition

yjxi

in

the

dictionary

E

must

be

used

to

control

the

hidden

state

order

There

exists

an

order

non-emitting

backoff

model

that

is

not

equivalent

to

any

basic

model

OE

for

any

fixed

order

The

proof

is

virtually

identical

to

the

proof

of

lemma

above

The

only

difference

is

that

the

presence

or

absence

of

a

state

transition

yjxi

in

the

dictionary

E

must

be

used

to

control

the

hidden

state

order

since

the

backoff

model

does

not

have

any

mixing

parameters

Conversely

every

order

n

backoff

model

can

be

converted

into

an

equivalent

non-emitting

backoff

model

with

a

complete

state

transition

dictionary

E

An

Therefore

the

class

of

non-emitting

backoff

models

is

strictly

more

powerful

than

the

class

of

simple

backoff

models

brown corpus
interpolated markov model
markov model
statistical language model
wall street journal

discrete time series
hidden markov model
mixture modeling
non-emitting state transitions
state-conditional interpolation