Estimation

and

Use

of

Uncertainty

in

Pseudo-relevance

Feedback

Kevyn

Collins-Thompson

and

Jamie

Callan

Language

Technologies

Institute

School

of

Computer

Science

Carnegie

Mellon

University

Pittsburgh,

PA

U.S.A.

{kct

callan}@cs.cmu.edu

ABSTRACT

Existing

pseudo-relevance

feedback

methods

typically

perform

averaging

over

the

top-retrieved

documents,

but

ignore

an

important

statistical

dimension:

the

risk

or

variance

associated

with

either

the

individual

document

models,

or

their

combination.

Treating

the

baseline

feedback

method

as

a

black

box,

and

the

output

feedback

model

as

a

random

variable,

we

estimate

a

posterior

distribution

for

the

feedback

model

by

resampling

a

given

query"s

top-retrieved

documents,

using

the

posterior

mean

or

mode

as

the

enhanced

feedback

model.

We

then

perform

model

combination

over

several

enhanced

models,

each

based

on

a

slightly

modified

query

sampled

from

the

original

query.

We

find

that

resampling

documents

helps

increase

individual

feedback

model

precision

by

removing

noise

terms,

while

sampling

from

the

query

improves

robustness

(worst-case

performance)

by

emphasizing

terms

related

to

multiple

query

aspects.

The

result

is

a

meta-feedback

algorithm

that

is

both

more

robust

and

more

precise

than

the

original

strong

baseline

method.

Categories

and

Subject

Descriptors:

[Information

Retrieval]:

Retrieval

Models

General

Terms:

Algorithms,

Experimentation

INTRODUCTION

Uncertainty

is

an

inherent

feature

of

information

retrieval.

Not

only

do

we

not

know

the

queries

that

will

be

presented

to

our

retrieval

algorithm

ahead

of

time,

but

the

user"s

information

need

may

be

vague

or

incompletely

specified

by

these

queries.

Even

if

the

query

were

perfectly

specified,

language

in

the

collection

documents

is

inherently

complex

and

ambiguous

and

matching

such

language

effectively

is

a

formidable

problem

by

itself.

With

this

in

mind,

we

wish

to

treat

many

important

quantities

calculated

by

the

retrieval

system,

whether

a

relevance

score

for

a

document,

or

a

weight

for

a

query

expansion

term,

as

random

variables

whose

true

value

is

uncertain

but

where

the

uncertainty

about

the

true

value

may

be

quantified

by

replacing

the

fixed

value

with

a

probability

distribution

over

possible

values.

In

this

way,

retrieval

algorithms

may

attempt

to

quantify

the

risk

or

uncertainty

associated

with

their

output

rankings,

or

improve

the

stability

or

precision

of

their

internal

calculations.

Current

algorithms

for

pseudo-relevance

feedback

(PRF)

tend

to

follow

the

same

basic

method

whether

we

use

vector

space-based

algorithms

such

as

Rocchio"s

formula

or

more

recent

language

modeling

approaches

such

as

Relevance

Models

First,

a

set

of

top-retrieved

documents

is

obtained

from

an

initial

query

and

assumed

to

approximate

a

set

of

relevant

documents.

Next,

a

single

feedback

model

vector

is

computed

according

to

some

sort

of

average,

centroid,

or

expectation

over

the

set

of

possibly-relevant

document

models.

For

example,

the

document

vectors

may

be

combined

with

equal

weighting,

as

in

Rocchio,

or

by

query

likelihood,

as

may

be

done

using

the

Relevance

The

use

of

an

expectation

is

reasonable

for

practical

and

theoretical

reasons,

but

by

itself

ignores

potentially

valuable

information

about

the

risk

of

the

feedback

model.

Our

main

hypothesis

in

this

paper

is

that

estimating

the

uncertainty

in

feedback

is

useful

and

leads

to

better

individual

feedback

models

and

more

robust

combined

models.

Therefore,

we

propose

a

method

for

estimating

uncertainty

associated

with

an

individual

feedback

model

in

terms

of

a

posterior

distribution

over

language

models.

To

do

this,

we

systematically

vary

the

inputs

to

the

baseline

feedback

method

and

fit

a

Dirichlet

distribution

to

the

output.

We

use

the

posterior

mean

or

mode

as

the

improved

feedback

model

estimate.

This

process

is

shown

in

Figure

As

we

show

later,

the

mean

and

mode

may

vary

significantly

from

the

single

feedback

model

proposed

by

the

baseline

method.

We

also

perform

model

combination

using

several

improved

feedback

language

models

obtained

by

a

small

number

of

new

queries

sampled

from

the

original

query.

A

model"s

weight

combines

two

complementary

factors:

the

model"s

probability

of

generating

the

query,

and

the

variance

of

the

model,

with

high-variance

models

getting

lower

weight.

For

example,

an

expected

parameter

vector

conditioned

on

the

query

observation

is

formed

from

top-retrieved

documents,

which

are

treated

as

training

strings

(see

p.

Figure

Estimating

the

uncertainty

of

the

feedback

model

for

a

single

query.

SAMPLING-BASED

FEEDBACK

In

Sections

we

describe

a

general

method

for

estimating

a

probability

distribution

over

the

set

of

possible

language

models.

In

Sections

and

we

summarize

how

different

query

samples

are

used

to

generate

multiple

feedback

models,

which

are

then

combined.

Modeling

Feedback

Uncertainty

Given

a

query

Q

and

a

collection

C,

we

assume

a

probabilistic

retrieval

system

that

assigns

a

real-valued

document

score

f(D,

Q)

to

each

document

D

in

C,

such

that

the

score

is

proportional

to

the

estimated

probability

of

relevance.

We

make

no

other

assumptions

about

f(D,

Q).

The

nature

of

f(D,

Q)

may

be

complex:

for

example,

if

the

retrieval

system

supports

structured

query

languages

then

f(D,

Q)

may

represent

the

output

of

an

arbitrarily

complex

inference

network

defined

by

the

structured

query

operators.

In

theory,

the

scoring

function

can

vary

from

query

to

query,

although

in

this

study

for

simplicity

we

keep

the

scoring

function

the

same

for

all

queries.

Our

specific

query

method

is

given

in

Section

We

treat

the

feedback

algorithm

as

a

black

box

and

assume

that

the

inputs

to

the

feedback

algorithm

are

the

original

query

and

the

corresponding

top-retrieved

documents,

with

a

score

being

given

to

each

document.

We

assume

that

the

output

of

the

feedback

algorithm

is

a

vector

of

term

weights

to

be

used

to

add

or

reweight

the

terms

in

the

representation

of

the

original

query,

with

the

vector

normalized

to

form

a

probability

distribution.

We

view

the

the

inputs

to

the

feedback

black

box

as

random

variables,

and

analyze

the

feedback

model

as

a

random

variable

that

changes

in

response

to

changes

in

the

inputs.

Like

the

document

scoring

function

f(D,

Q),

the

feedback

algorithm

may

implement

a

complex,

non-linear

scoring

formula,

and

so

as

its

inputs

vary,

the

resulting

feedback

models

may

have

a

complex

distribution

over

the

space

of

feedback

models

(the

sample

space).

Because

of

this

potential

complexity,

we

do

not

attempt

to

derive

a

posterior

distribution

in

closed

form,

but

instead

use

simulation.

We

call

this

distribution

over

possible

feedback

models

the

feedback

model

distribution.

Our

goal

in

this

section

is

to

estimate

a

useful

approximation

to

the

feedback

model

distribution.

For

a

specific

framework

for

experiments,

we

use

the

language

modeling

(LM)

approach

for

information

retrieval

The

score

of

a

document

D

with

respect

to

a

query

Q

and

collection

C

is

given

by

p(Q|D)

with

respect

to

language

models

ˆθQ

and

ˆθD

estimated

for

the

query

and

document

respectively.

We

denote

the

set

of

k

top-retrieved

documents

from

collection

C

in

response

to

Q

by

DQ(k,

C).

For

simplicity,

we

assume

that

queries

and

documents

are

generated

by

multinomial

distributions

whose

parameters

are

represented

by

unigram

language

models.

To

incorporate

feedback

in

the

LM

approach,

we

assume

a

model-based

scheme

in

which

our

goal

is

take

the

query

and

resulting

ranked

documents

DQ(k,

C)

as

input,

and

output

an

expansion

language

model

ˆθE,

which

is

then

interpolated

with

the

original

query

model

ˆθQ:

ˆθNew

α)

ˆθQ

α

ˆθE

This

includes

the

possibility

of

α

where

the

original

query

mode

is

completely

replaced

by

the

feedback

model.

Our

sample

space

is

the

set

of

all

possible

language

models

LF

that

may

be

output

as

feedback

models.

Our

approach

is

to

take

samples

from

this

space

and

then

fit

a

distribution

to

the

samples

using

maximum

likelihood.

For

simplicity,

we

start

by

assuming

the

latent

feedback

distribution

has

the

form

of

a

Dirichlet

distribution.

Although

the

Dirichlet

is

a

unimodal

distribution,

and

in

general

quite

limited

in

its

expressiveness

in

the

sample

space,

it

is

a

natural

match

for

the

multinomial

language

model,

can

be

estimated

quickly,

and

can

capture

the

most

salient

features

of

confident

and

uncertain

feedback

models,

such

as

the

overall

spread

of

the

distibution.

Resampling

document

models

We

would

like

an

approximation

to

the

posterior

distribution

of

the

feedback

model

LF

To

accomplish

this,

we

apply

a

widely-used

simulation

technique

called

bootstrap

sampling

p.

on

the

input

parameters,

namely,

the

set

of

top-retrieved

documents.

Bootstrap

sampling

allows

us

to

simulate

the

approximate

effect

of

perturbing

the

parameters

within

the

black

box

feedback

algorithm

by

perturbing

the

inputs

to

that

algorithm

in

a

systematic

way,

while

making

no

assumptions

about

the

nature

of

the

feedback

algorithm.

Specifically,

we

sample

k

documents

with

replacement

from

DQ(k,

C),

and

calculate

an

expansion

language

model

θb

using

the

black

box

feedback

method.

We

repeat

this

process

B

times

to

obtain

a

set

of

B

feedback

language

models,

to

which

we

then

fit

a

Dirichlet

distribution.

Typically

B

is

in

the

range

of

to

samples,

with

performance

being

relatively

stable

in

this

range.

Note

that

instead

of

treating

each

top

document

as

equally

likely,

we

sample

according

to

the

estimated

probabilities

of

relevance

of

each

document

in

DQ(k,

C).

Thus,

a

document

is

more

likely

to

be

chosen

the

higher

it

is

in

the

ranking.

Justification

for

a

sampling

approach

The

rationale

for

our

sampling

approach

has

two

parts.

First,

we

want

to

improve

the

quality

of

individual

feedback

models

by

smoothing

out

variation

when

the

baseline

feedback

model

is

unstable.

In

this

respect,

our

approach

resembles

bagging

an

ensemble

approach

which

generates

multiple

versions

of

a

predictor

by

making

bootstrap

copies

of

the

training

set,

and

then

averages

the

(numerical)

predictors.

In

our

application,

top-retrieved

documents

can

be

seen

as

a

kind

of

noisy

training

set

for

relevance.

Second,

sampling

is

an

effective

way

to

estimate

basic

properties

of

the

feedback

posterior

distribution,

which

can

then

be

used

for

improved

model

combination.

For

example,

a

model

may

be

weighted

by

its

prediction

confidence,

estimated

as

a

function

of

the

variability

of

the

posterior

around

the

model.

foo2-401.map-Dim:5434,Size:12*12units,gaussianneighborhood

(a)

Topic

Foreign

minorities,

Germany

foo2-402.map-Dim:5698,Size:12*12units,gaussianneighborhood

(b)

Topic

Behavioral

genetics

foo2-459.map-Dim:8969,Size:12*12units,gaussianneighborhood

(c)

Topic

When

can

a

lender

foreclose

on

property

Figure

Visualization

of

expansion

language

model

variance

using

self-organizing

maps,

showing

the

distribution

of

language

models

that

results

from

resampling

the

inputs

to

the

baseline

expansion

method.

The

language

model

that

would

have

been

chosen

by

the

baseline

expansion

is

at

the

center

of

each

map.

The

similarity

function

is

JensenShannon

divergence.

Visualizing

feedback

distributions

Before

describing

how

we

fit

and

use

the

Dirichlet

distribution

over

feedback

models,

it

is

instructive

to

view

some

examples

of

actual

feedback

model

distributions

that

result

from

bootstrap

sampling

the

top-retrieved

documents

from

different

TREC

topics.

Each

point

in

our

sample

space

is

a

language

model,

which

typically

has

several

thousand

dimensions.

To

help

analyze

the

behavior

of

our

method

we

used

a

Self-Organizing

Map

(via

the

SOM-PAK

package

to

‘flatten"

and

visualize

the

high-dimensional

density

The

density

maps

for

three

TREC

topics

are

shown

in

Figure

above.

The

dark

areas

represent

regions

of

high

similarity

between

language

models.

The

light

areas

represent

regions

of

low

similarity

the

‘valleys"

between

clusters.

Each

diagram

is

centered

on

the

language

model

that

would

have

been

chosen

by

the

baseline

expansion.

A

single

peak

(mode)

is

evident

in

some

examples,

but

more

complex

structure

appears

in

others.

Also,

while

the

distribution

is

usually

close

to

the

baseline

feedback

model,

for

some

topics

they

are

a

significant

distance

apart

(as

measured

by

JensenShannon

divergence),

as

in

Subfigure

In

such

cases,

the

mode

or

mean

of

the

feedback

distribution

often

performs

significantly

better

than

the

baseline

(and

in

a

smaller

proportion

of

cases,

significantly

worse).

Fitting

a

posterior

feedback

distribution

After

obtaining

feedback

model

samples

by

resampling

the

feedback

model

inputs,

we

estimate

the

feedback

distribution.

We

assume

that

the

multinomial

feedback

models

ˆθB}

were

generated

by

a

latent

Dirichlet

distribution

with

parameters

αN

To

estimate

the

αN

we

fit

the

Dirichlet

parameters

to

the

B

language

model

samples

according

to

maximum

likelihood

using

a

generalized

Newton

procedure,

details

of

which

are

given

in

Minka

We

assume

a

simple

Dirichlet

prior

over

the

αN

setting

each

to

αi

μ

p(wi

C),

where

μ

is

a

parameter

and

p(·

C)

is

the

collection

language

model

estimated

from

a

set

of

documents

from

collection

C.

The

parameter

fitting

converges

very

quickly

typically

just

or

Because

our

points

are

language

models

in

the

multinomial

simplex,

we

extended

SOM-PAK

to

support

JensenShannon

divergence,

a

widely-used

similarity

measure

between

probability

distributions.

iterations

are

enough

so

that

it

is

practical

to

apply

at

query-time

when

computational

overhead

must

be

small.

In

practice,

we

can

restrict

the

calculation

to

the

vocabulary

of

the

top-retrieved

documents,

instead

of

the

entire

collection.

Note

that

for

this

step

we

are

re-using

the

existing

retrieved

documents

and

not

performing

additional

queries.

Given

the

parameters

of

an

N-dimensional

Dirichlet

distribution

Dir(α)

the

mean

μ

and

mode

x

vectors

are

easy

to

calculate

and

are

given

respectively

by

μi

αiP

αi

and

xi

αi−N

We

can

then

choose

the

language

model

at

the

mean

or

the

mode

of

the

posterior

as

the

final

enhanced

feedback

model.

(We

found

the

mode

to

give

slightly

better

performance.)

For

information

retrieval,

the

number

of

samples

we

will

have

available

is

likely

to

be

quite

small

for

performance

reasons

usually

less

than

ten.

Moreover,

while

random

sampling

is

useful

in

certain

cases,

it

is

perfectly

acceptable

to

allow

deterministic

sampling

distributions,

but

these

must

be

designed

carefully

in

order

to

approximate

an

accurate

output

variance.

We

leave

this

for

future

study.

Query

variants

We

use

the

following

methods

for

generating

variants

of

the

original

query.

Each

variant

corresponds

to

a

different

assumption

about

which

aspects

of

the

original

query

may

be

important.

This

is

a

form

of

deterministic

sampling.

We

selected

three

simple

methods

that

cover

complimentary

assumptions

about

the

query.

No-expansion

Use

only

the

original

query.

The

assumption

is

that

the

given

terms

are

a

complete

description

of

the

information

need.

Leave-one-out

A

single

term

is

left

out

of

the

original

query.

The

assumption

is

that

one

of

the

query

terms

is

a

noise

term.

Single-term

A

single

term

is

chosen

from

the

original

query.

This

assumes

that

only

one

aspect

of

the

query,

namely,

that

represented

by

the

term,

is

most

important.

After

generating

a

variant

of

the

original

query,

we

combine

it

with

the

original

query

using

a

weight

αSUB

so

that

we

do

not

stray

too

‘far".

In

this

study,

we

set

αSUB

For

example,

using

the

Indri

query

language,

a

leave-oneout

variant

of

the

initial

query

that

omits

the

term

‘ireland"

for

TREC

topic

is:

#combine(ireland

peace

talks)

#combine(peace

talks))

Combining

enhanced

feedback

models

from

multiple

query

variants

When

using

multiple

query

variants,

the

resulting

enhanced

feedback

models

are

combined

using

Bayesian

model

combination.

To

do

this,

we

treat

each

word

as

an

item

to

be

classified

as

belonging

to

a

relevant

or

non-relevant

class,

and

derive

a

class

probability

for

each

word

by

combining

the

scores

from

each

query

variant.

Each

score

is

given

by

that

term"s

probability

in

the

Dirichlet

distribution.

The

term

scores

are

weighted

by

the

inverse

of

the

variance

of

the

term

in

the

enhanced

feedback

model"s

Dirichlet

distribution.

The

prior

probability

of

a

word"s

membership

in

the

relevant

class

is

given

by

the

probability

of

the

original

query

in

the

entire

enhanced

expansion

model.

EVALUATION

In

this

section

we

present

results

confirming

the

usefulness

of

estimating

a

feedback

model

distribution

from

weighted

resampling

of

top-ranked

documents,

and

of

combining

the

feedback

models

obtained

from

different

small

changes

in

the

original

query.

General

method

We

evaluated

performance

on

a

total

of

queries

derived

from

four

sets

of

TREC

topics:

and

We

chose

these

for

their

varied

content

and

document

properties.

For

example,

documents

are

Web

pages

with

a

wide

variety

of

subjects

and

styles

while

documents

are

more

homogeneous

news

articles.

Indexing

and

retrieval

was

performed

using

the

Indri

system

in

the

Lemur

toolkit

Our

queries

were

derived

from

the

words

in

the

title

field

of

the

TREC

topics.

Phrases

were

not

used.

To

generate

the

baseline

queries

passed

to

Indri,

we

wrapped

the

query

terms

with

Indri"s

#combine

operator.

For

example,

the

initial

query

for

topic

is:

#combine(ireland

peace

talks)

We

performed

Krovetz

stemming

for

all

experiments.

Because

we

found

that

the

baseline

(Indri)

expansion

method

performed

better

using

a

stopword

list

with

the

feedback

model,

all

experiments

used

a

stoplist

of

common

English

words.

However,

an

interesting

side-effect

of

our

resampling

approach

is

that

it

tends

to

remove

many

stopwords

from

the

feedback

model,

making

a

stoplist

less

critical.

This

is

discussed

further

in

Section

Baseline

feedback

method

For

our

baseline

expansion

method,

we

use

an

algorithm

included

in

Indri

as

the

default

expansion

method.

This

method

first

selects

terms

using

a

log-odds

calculation

described

by

Ponte

but

assigns

final

term

weights

using

Lavrenko"s

relevance

We

chose

the

Indri

method

because

it

gives

a

consistently

strong

baseline,

is

based

on

a

language

modeling

approach,

and

is

simple

to

experiment

with.

In

a

TREC

evaluation

using

the

corpus

the

method

was

one

of

the

topperforming

runs,

achieving

a

gain

in

MAP

compared

to

using

unexpanded

queries.

In

this

study,

it

achieves

an

average

gain

in

MAP

of

over

the

four

collections.

Indri"s

expansion

method

first

calculates

a

log-odds

ratio

o(v)

for

each

potential

expansion

term

v

given

by

o(v)

X

D

log

p(v|D)

p(v|C)

over

all

documents

D

containing

v,

in

collection

C.

Then,

the

expansion

term

candidates

are

sorted

by

descending

o(v),

and

the

top

m

are

chosen.

Finally,

the

term

weights

r(v)

used

in

the

expanded

query

are

calculated

based

on

the

relevance

model

r(v)

X

D

p(q|D)p(v|D)

p(v)

p(D)

The

quantity

p(q|D)

is

the

probability

score

assigned

to

the

document

in

the

initial

retrieval.

We

use

Dirichlet

smoothing

of

p(v|D)

with

μ

This

relevance

model

is

then

combined

with

the

original

query

using

linear

interpolation,

weighted

by

a

parameter

α.

By

default

we

used

the

top

documents

for

feedback

and

the

top

expansion

terms,

with

the

feedback

interpolation

parameter

α

unless

otherwise

stated.

For

example,

the

baseline

expanded

query

for

topic

is:

#combine(ireland

peace

talks)

ireland

peace

northern

Expansion

performance

We

measure

our

feedback

algorithm"s

effectiveness

by

two

main

criteria:

precision,

and

robustness.

Robustness,

and

the

tradeoff

between

precision

and

robustness,

is

analyzed

in

Section

In

this

section,

we

examine

average

precision

and

precision

in

the

top

documents

We

also

include

recall

at

documents.

For

each

query,

we

obtained

a

set

of

B

feedback

models

using

the

Indri

baseline.

Each

feedback

model

was

obtained

from

a

random

sample

of

the

top

k

documents

taken

with

replacement.

For

these

experiments,

B

and

k

Each

feedback

model

contained

terms.

On

the

query

side,

we

used

leave-one-out

(LOO)

sampling

to

create

the

query

variants.

Single-term

query

sampling

had

consistently

worse

performance

across

all

collections

and

so

our

results

here

focus

on

LOO

sampling.

We

used

the

methods

described

in

Section

to

estimate

an

enhanced

feedback

model

from

the

Dirichlet

posterior

distribution

for

each

query

variant,

and

to

combine

the

feedback

models

from

all

the

query

variants.

We

call

our

method

‘resampling

expansion"

and

denote

it

as

RS-FB

here.

We

denote

the

Indri

baseline

feedback

method

as

Base-FB.

Results

from

applying

both

the

baseline

expansion

method

(Base-FB)

and

resampling

expansion

(RS-FB)

are

shown

in

Table

We

observe

several

trends

in

this

table.

First,

the

average

precision

of

RS-FB

was

comparable

to

Base-FB,

achieving

an

average

gain

of

compared

to

using

no

expansion

across

the

four

collections.

The

Indri

baseline

expansion

gain

was

Also,

the

RS-FB

method

achieved

consistent

improvements

in

over

Base-FB

for

every

topic

set,

with

an

average

improvement

of

over

Base-FB

for

all

topics.

The

lowest

gain

over

Base-FB

was

for

and

the

highest

was

for

Finally,

both

Base-FB

and

RS-FB

also

consistently

improved

recall

over

using

no

expansion,

with

Base-FB

achieving

better

recall

than

RS-FB

for

all

topic

sets.

Retrieval

robustness

We

use

the

term

robustness

to

mean

the

worst-case

average

precision

performance

of

a

feedback

algorithm.

Ideally,

a

robust

feedback

method

would

never

perform

worse

than

using

the

original

query,

while

often

performing

better

using

the

expansion.

To

evaluate

robustness

in

this

study,

we

use

a

very

simple

measure

called

the

robustness

index

For

a

set

of

queries

Q,

the

RI

measure

is

defined

as:

RI(Q)

n+

n−

|Q|

where

n+

is

the

number

of

queries

helped

by

the

feedback

method

and

n−

is

the

number

of

queries

hurt.

Here,

by

‘helped"

we

mean

obtaining

a

higher

average

precision

as

a

result

of

feedback.

The

value

of

RI

ranges

from

a

minimum

This

is

sometimes

also

called

the

reliability

of

improvement

index

and

was

used

in

Sakai

et

al.

Collection

NoExp

Base-FB

RS-FB

TREC

AvgP

Recall

TREC

AvgP

Recall

TREC

AvgP

Recall

AvgP

Recall

Table

Comparison

of

baseline

(Base-FB)

feedback

and

feedback

using

re-sampling

(RS-FB).

Improvement

shown

for

BaseFB

and

RS-FB

is

relative

to

using

no

expansion.

(a)

TREC

(upper

curve);

TREC

(lower

curve)

(b)

TREC

(upper

curve);

(lower

curve)

Figure

The

trade-off

between

robustness

and

average

precision

for

different

corpora.

The

x-axis

gives

the

change

in

MAP

over

using

baseline

expansion

with

α

The

yaxis

gives

the

Robustness

Index

(RI).

Each

curve

through

uncircled

points

shows

the

RI/MAP

tradeoff

using

the

simple

small-α

strategy

(see

text)

as

α

decreases

from

to

zero

in

the

direction

of

the

arrow.

Circled

points

represent

the

tradeoffs

obtained

by

resampling

feedback

for

α

Collection

N

Base-FB

RS-FB

n−

RI

n−

RI

TREC

TREC

TREC

Combined

Table

Comparison

of

robustness

index

(RI)

for

baseline

feedback

(Base-FB)

vs.

resampling

feedback

(RS-FB).

Also

shown

are

the

actual

number

of

queries

hurt

by

feedback

(n−)

for

each

method

and

collection.

Queries

for

which

initial

average

precision

was

negligible

were

ignored,

giving

the

remaining

query

count

in

column

N.

of

when

all

queries

are

hurt

by

the

feedback

method,

to

when

all

queries

are

helped.

The

RI

measure

does

not

take

into

account

the

magnitude

or

distribution

of

the

amount

of

change

across

the

set

Q.

However,

it

is

easy

to

understand

as

a

general

indication

of

robustness.

One

obvious

way

to

improve

the

worst-case

performance

of

feedback

is

simply

to

use

a

smaller

fixed

α

interpolation

parameter,

such

as

α

placing

less

weight

on

the

(possibly

risky)

feedback

model

and

more

on

the

original

query.

We

call

this

the

‘small-α"

strategy.

Since

we

are

also

reducing

the

potential

gains

when

the

feedback

model

is

‘right",

however,

we

would

expect

some

trade-off

between

average

precision

and

robustness.

We

therefore

compared

the

precision/robustness

trade-off

between

our

resampling

feedback

algorithm,

and

the

simple

small-α

method.

The

results

are

summarized

in

Figure

In

the

figure,

the

curve

for

each

topic

set

interpolates

between

trade-off

points,

beginning

at

where

α

and

continuing

in

the

direction

of

the

arrow

as

α

decreases

and

the

original

query

is

given

more

and

more

weight.

As

expected,

robustness

continuously

increases

as

we

move

along

the

curve,

but

mean

average

precision

generally

drops

as

the

gains

from

feedback

are

eliminated.

For

comparison,

the

performance

of

resampling

feedback

at

α

is

shown

for

each

collection

as

the

circled

point.

Higher

and

to

the

right

is

better.

This

figure

shows

that

resampling

feedback

gives

a

somewhat

better

trade-off

than

the

small-α

approach

for

of

the

collections.

Figure

Histogram

showing

improved

robustness

of

resampling

feedback

(RS-FB)

over

baseline

feedback

(Base-FB)

for

all

datasets

combined.

Queries

are

binned

by

change

in

AP

compared

to

the

unexpanded

query.

Collection

DS

QV

DS

No

QV

TREC

AvgP

RI

TREC

AvgP

RI

TREC

AvgP

RI

AvgP

RI

Table

Comparison

of

resampling

feedback

using

document

sampling

(DS)

with

(QV)

and

without

(No

QV)

combining

feedback

models

from

multiple

query

variants.

Table

gives

the

Robustness

Index

scores

for

Base-FB

and

RS-FB.

The

RS-FB

feedback

method

obtained

higher

robustness

than

Base-FB

on

three

of

the

four

topic

sets,

with

only

slightly

worse

performance

on

A

more

detailed

view

showing

the

distribution

over

relative

changes

in

AP

is

given

by

the

histogram

in

Figure

Compared

to

Base-FB,

the

RS-FB

method

achieves

a

noticable

reduction

in

the

number

of

queries

significantly

hurt

by

expansion

(i.e.

where

AP

is

hurt

by

or

more),

while

preserving

positive

gains

in

AP.

Effect

of

query

and

document

sampling

methods

Given

our

algorithm"s

improved

robustness

seen

in

Section

an

important

question

is

what

component

of

our

system

is

responsible.

Is

it

the

use

of

document

re-sampling,

the

use

of

multiple

query

variants,

or

some

other

factor?

The

results

in

Table

suggest

that

the

model

combination

based

on

query

variants

may

be

largely

account

for

the

improved

robustness.

When

query

variants

are

turned

off

and

the

original

query

is

used

by

itself

with

document

sampling,

there

is

little

net

change

in

average

precision,

a

small

decrease

in

for

out

of

the

topic

sets,

but

a

significant

drop

in

robustness

for

all

topic

sets.

In

two

cases,

the

RI

measure

drops

by

more

than

We

also

examined

the

effect

of

the

document

sampling

method

on

retrieval

effectiveness,

using

two

different

strategies.

The

‘uniform

weighting"

strategy

ignored

the

relevance

scores

from

the

initial

retrieval

and

gave

each

document

in

the

top

k

the

same

probability

of

selection.

In

contrast,

the

‘relevance-score

weighting"

strategy

chose

documents

with

probability

proportional

to

their

relevance

scores.

In

this

way,

documents

that

were

more

highly

ranked

were

more

likely

to

be

selected.

Results

are

shown

in

Table

The

relevance-score

weighting

strategy

performs

better

overall,

with

significantly

higher

RI

and

scores

on

of

the

topic

sets.

The

difference

in

average

precision

between

the

methods,

however,

is

less

marked.

This

suggests

that

uniform

weighting

acts

to

increase

variance

in

retrieval

results:

when

initial

average

precision

is

high,

there

are

many

relevant

documents

in

the

top

k

and

uniform

sampling

may

give

a

more

representative

relevance

model

than

focusing

on

the

highly-ranked

items.

On

the

other

hand,

when

initial

precision

is

low,

there

are

few

relevant

documents

in

the

bottom

ranks

and

uniform

sampling

mixes

in

more

of

the

non-relevant

documents.

For

space

reasons

we

only

summarize

our

findings

on

sample

size

here.

The

number

of

samples

has

some

effect

on

precision

when

less

than

but

performance

stabilizes

at

around

to

samples.

We

used

samples

for

our

experiments.

Much

beyond

this

level,

the

additional

benefits

of

more

samples

decrease

as

the

initial

score

distribution

is

more

closely

fit

and

the

processing

time

increases.

The

effect

of

resampling

on

expansion

term

quality

Ideally,

a

retrieval

model

should

not

require

a

stopword

list

when

estimating

a

model

of

relevance:

a

robust

statistical

model

should

down-weight

stopwords

automatically

depending

on

context.

Stopwords

can

harm

feedback

if

selected

as

feedback

terms,

because

they

are

typically

poor

discriminators

and

waste

valuable

term

slots.

In

practice,

however,

because

most

term

selection

methods

resemble

a

tf

idf

type

of

weighting,

terms

with

low

idf

but

very

high

tf

can

sometimes

be

selected

as

expansion

term

candidates.

This

happens,

for

example,

even

with

the

Relevance

Model

approach

that

is

part

of

our

baseline

feedback.

To

ensure

as

strong

a

baseline

as

possible,

we

use

a

stoplist

for

all

experiments

reported

here.

If

we

turn

off

the

stopword

list,

however,

we

obtain

results

such

as

those

shown

in

Table

where

four

of

the

top

ten

baseline

feedback

terms

for

TREC

topic

(said,

but,

their,

not)

are

stopwords

using

the

BaseFB

method.

(The

top

expansion

terms

were

selected

to

generate

this

example.)

Indri"s

method

attempts

to

address

the

stopword

problem

by

applying

an

initial

step

based

on

Ponte

to

select

less-common

terms

that

have

high

log-odds

of

being

in

the

top-ranked

documents

compared

to

the

whole

collection.

Nevertheless,

this

does

not

overcome

the

stopword

problem

completely,

especially

as

the

number

of

feedback

terms

grows.

Using

resampling

feedback,

however,

appears

to

mitigate

Collection

QV

Uniform

QV

Relevance-score

weighting

weighting

TREC

AvgP

RI

TREC

AvgP

RI

TREC

AvgP

RI

AvgP

RI

Table

Comparison

of

uniform

and

relevance-weighted

document

sampling.

The

percentage

change

compared

to

uniform

sampling

is

shown

in

parentheses.

QV

indicates

that

query

variants

were

used

in

both

runs.

Baseline

FB

p(wi|R)

Resampling

FB

p(wi|R)

said

court

court

pay

pay

federal

but

education

employees

teachers

their

employees

not

case

federal

their

workers

appeals

education

union

Table

Feedback

term

quality

when

a

stoplist

is

not

used.

Feedback

terms

for

TREC

topic

merit

pay

vs

seniority.

the

effect

of

stopwords

automatically.

In

the

example

of

Table

resampling

feedback

leaves

only

one

stopword

(their)

in

the

top

ten.

We

observed

similar

feedback

term

behavior

across

many

other

topics.

The

reason

for

this

effect

appears

to

be

the

interaction

of

the

term

selection

score

with

the

top-m

term

cutoff.

While

the

presence

and

even

proportion

of

particular

stopwords

is

fairly

stable

across

different

document

samples,

their

relative

position

in

the

top-m

list

is

not,

as

sets

of

documents

with

varying

numbers

of

better,

lower-frequency

term

candidates

are

examined

for

each

sample.

As

a

result,

while

some

number

of

stopwords

may

appear

in

each

sampled

document

set,

any

given

stopword

tends

to

fall

below

the

cutoff

for

multiple

samples,

leading

to

its

classification

as

a

high-variance,

low-weight

feature.

RELATED

WORK

Our

approach

is

related

to

previous

work

from

several

areas

of

information

retrieval

and

machine

learning.

Our

use

of

query

variation

was

inspired

by

the

work

of

YomTov

et

al.

Carpineto

et

al.

and

Amati

et

al.

among

others.

These

studies

use

the

idea

of

creating

multiple

subqueries

and

then

examining

the

nature

of

the

overlap

in

the

documents

and/or

expansion

terms

that

result

from

each

subquery.

Model

combination

is

performed

using

heuristics.

In

particular,

the

studies

of

Amati

et

al.

and

Carpineto

et

al.

investigated

combining

terms

from

individual

distributional

methods

using

a

term-reranking

combination

heuristic.

In

a

set

of

TREC

topics

they

found

wide

average

variation

in

the

rank-distance

of

terms

from

different

expansion

methods.

Their

combination

method

gave

modest

positive

improvements

in

average

precision.

The

idea

of

examining

the

overlap

between

lists

of

suggested

terms

has

also

been

used

in

early

query

expansion

approaches.

Xu

and

Croft"s

method

of

Local

Context

Analysis

(LCA)

includes

a

factor

in

the

empirically-derived

weighting

formula

that

causes

expansion

terms

to

be

preferred

that

have

connections

to

multiple

query

terms.

On

the

document

side,

recent

work

by

Zhou

Croft

explored

the

idea

of

adding

noise

to

documents,

re-scoring

them,

and

using

the

stability

of

the

resulting

rankings

as

an

estimate

of

query

difficulty.

This

is

related

to

our

use

of

document

sampling

to

estimate

the

risk

of

the

feedback

model

built

from

the

different

sets

of

top-retrieved

documents.

Sakai

et

al.

proposed

an

approach

to

improving

the

robustness

of

pseudo-relevance

feedback

using

a

method

they

call

selective

sampling.

The

essence

of

their

method

is

that

they

allow

skipping

of

some

top-ranked

documents,

based

on

a

clustering

criterion,

in

order

to

select

a

more

varied

and

novel

set

of

documents

later

in

the

ranking

for

use

by

a

traditional

pseudo-feedback

method.

Their

study

did

not

find

significant

improvements

in

either

robustness

(RI)

or

MAP

on

their

corpora.

Greiff,

Morgan

and

Ponte

explored

the

role

of

variance

in

term

weighting.

In

a

series

of

simulations

that

simplified

the

problem

to

documents,

they

found

that

average

precision

degrades

as

term

frequency

variance

high

noiseincreases.

Downweighting

terms

with

high

variance

resulted

in

improved

average

precision.

This

seems

in

accord

with

our

own

findings

for

individual

feedback

models.

Estimates

of

output

variance

have

recently

been

used

for

improved

text

classification.

Lee

et

al.

used

queryspecific

variance

estimates

of

classifier

outputs

to

perform

improved

model

combination.

Instead

of

using

sampling,

they

were

able

to

derive

closed-form

expressions

for

classifier

variance

by

assuming

base

classifiers

using

simple

types

of

inference

networks.

Ando

and

Zhang

proposed

a

method

that

they

call

structural

feedback

and

showed

how

to

apply

it

to

query

expansion

for

the

TREC

Genomics

Track.

They

used

r

query

variations

to

obtain

R

different

sets

Sr

of

top-ranked

documents

that

have

been

intersected

with

the

top-ranked

documents

obtained

from

the

original

query

qorig.

For

each

Si,

the

normalized

centroid

vector

ˆwi

of

the

documents

is

calculated.

Principal

component

analysis

(PCA)

is

then

applied

to

the

ˆwi

to

obtain

the

matrix

Φ

of

H

left

singular

vectors

φh

that

are

used

to

obtain

the

new,

expanded

query

qexp

qorig

ΦT

Φqorig.

In

the

case

H

we

have

a

single

left

singular

vector

φ:

qexp

qorig

(φT

qorig)φ

so

that

the

dot

product

φT

qorig

is

a

type

of

dynamic

weight

on

the

expanded

query

that

is

based

on

the

similarity

of

the

original

query

to

the

expanded

query.

The

use

of

variance

as

a

feedback

model

quality

measure

occurs

indirectly

through

the

application

of

PCA.

It

would

be

interesting

to

study

the

connections

between

this

approach

and

our

own

modelfitting

method.

Finally,

in

language

modeling

approaches

to

feedback,

Tao

and

Zhai

describe

a

method

for

more

robust

feedback

that

allows

each

document

to

have

a

different

feedback

α.

The

feedback

weights

are

derived

automatically

using

regularized

EM.

A

roughly

equal

balance

of

query

and

expansion

model

is

implied

by

their

EM

stopping

condition.

They

propose

tailoring

the

stopping

parameter

η

based

on

a

function

of

some

quality

measure

of

feedback

documents.

CONCLUSIONS

We

have

presented

a

new

approach

to

pseudo-relevance

feedback

based

on

document

and

query

sampling.

The

use

of

sampling

is

a

very

flexible

and

powerful

device

and

is

motivated

by

our

general

desire

to

extend

current

models

of

retrieval

by

estimating

the

risk

or

variance

associated

with

the

parameters

or

output

of

retrieval

processes.

Such

variance

estimates,

for

example,

may

be

naturally

used

in

a

Bayesian

framework

for

improved

model

estimation

and

combination.

Applications

such

as

selective

expansion

may

then

be

implemented

in

a

principled

way.

While

our

study

uses

the

language

modeling

approach

as

a

framework

for

experiments,

we

make

few

assumptions

about

the

actual

workings

of

the

feedback

algorithm.

We

believe

it

is

likely

that

any

reasonably

effective

baseline

feedback

algorithm

would

benefit

from

our

approach.

Our

results

on

standard

TREC

collections

show

that

our

framework

improves

the

robustness

of

a

strong

baseline

feedback

method

across

a

variety

of

collections,

without

sacrificing

average

precision.

It

also

gives

small

but

consistent

gains

in

precision.

In

future

work,

we

envision

an

investigation

into

how

varying

the

set

of

sampling

methods

used

and

the

number

of

samples

controls

the

trade-off

between

robustness,

accuracy,

and

efficiency.

Acknowledgements

We

thank

Paul

Bennett

for

valuable

discussions

related

to

this

work,

which

was

supported

by

NSF

grants

and

and

U.S.

Dept.

of

Education

grant

Any

opinions,

findings,

and

conclusions

or

recommendations

expressed

in

this

material

are

the

authors.

and

do

not

necessarily

reflect

those

of

the

sponsors.

REFERENCES

The

Lemur

toolkit

for

language

modeling

and

retrieval.

G.

Amati,

C.

Carpineto,

and

G.

Romano.

Query

difficulty,

robustness,

and

selective

application

of

query

expansion.

In

Proc.

of

the

European

Conf.

on

Information

Retrieval

(ECIR

pages

R.

K.

Ando

and

T.

Zhang.

A

high-performance

semi-supervised

learning

method

for

text

chunking.

In

Proc.

of

the

Annual

Meeting

of

the

ACL,

pages

June

L.

Breiman.

Bagging

predictors.

Machine

Learning,

C.

Carpineto,

G.

Romano,

and

V.

Giannini.

Improving

retrieval

feedback

with

multiple

term-ranking

function

combination.

ACM

Trans.

Info.

Systems,

K.

Collins-Thompson,

P.

Ogilvie,

and

J.

Callan.

Initial

results

with

structured

queries

and

language

models

on

half

a

terabyte

of

text.

In

Proc.

of

Text

REtrieval

Conference.

NIST

Special

Publication.

R.

O.

Duda,

P.

E.

Hart,

and

D.

G.

Stork.

Pattern

Classification.

Wiley

and

Sons,

edition,

W.

R.

Greiff,

W.

T.

Morgan,

and

J.

M.

Ponte.

The

role

of

variance

in

term

weighting

for

probabilistic

information

retrieval.

In

Proc.

of

the

Intl.

Conf.

on

Info.

and

Knowledge

Mgmt.

(CIKM

pages

T.

Kohonen,

J.

Hynninen,

J.

Kangas,

and

J.

Laaksonen.

SOMPAK:

The

self-organizing

map

program

package.

Technical

Report

Helsinki

University

of

Technology,

V.

Lavrenko.

A

Generative

Theory

of

Relevance.

PhD

thesis,

University

of

Massachusetts,

Amherst,

C.-H.

Lee,

R.

Greiner,

and

S.

Wang.

Using

query-specific

variance

estimates

to

combine

Bayesian

classifiers.

In

Proc.

of

the

Intl.

Conf.

on

Machine

Learning

(ICML

pages

D.

Metzler

and

W.

B.

Croft.

Combining

the

language

model

and

inference

network

approaches

to

retrieval.

Info.

Processing

and

Mgmt.,

T.

Minka.

Estimating

a

Dirichlet

distribution.

Technical

report,

minka/papers/dirichlet.

J.

Ponte.

Advances

in

Information

Retrieval,

chapter

Language

models

for

relevance

feedback,

pages

W.B.

Croft,

ed.

J.

M.

Ponte

and

W.

B.

Croft.

A

language

modeling

approach

to

information

retrieval.

In

Proc.

of

the

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

pages

J.

Rocchio.

The

SMART

Retrieval

System,

chapter

Relevance

Feedback

in

Information

Retrieval,

pages

Prentice-Hall,

G.

Salton,

ed.

T.

Sakai,

T.

Manabe,

and

M.

Koyama.

Flexible

pseudo-relevance

feedback

via

selective

sampling.

ACM

Transactions

on

Asian

Language

Information

Processing

(TALIP),

T.

Tao

and

C.

Zhai.

Regularized

estimation

of

mixture

models

for

robust

pseudo-relevance

feedback.

In

Proc.

of

the

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

pages

J.

Xu

and

W.

B.

Croft.

Improving

the

effectiveness

of

information

retrieval

with

local

context

analysis.

ACM

Trans.

Inf.

Syst.,

E.

YomTov,

S.

Fine,

D.

Carmel,

and

A.

Darlow.

Learning

to

estimate

query

difficulty.

In

Proc.

of

the

ACM

SIGIR

Conf.

on

Research

and

Development

in

Information

Retrieval,

pages

Y.

Zhou

and

W.

B.

Croft.

Ranking

robustness:

a

novel

framework

to

predict

query

performance.

In

Proc.

of

the

ACM

Intl.

Conf.

on

Information

and

Knowledge

Mgmt.

(CIKM

pages

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O B I O O B I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

pseudo-relevance feedback
vector space-based algorithm

feedback method
posterior distribution
enhanced feedback model
information retrieval
query expansion
probability distribution
risk
feedback model
estimating uncertainty
language modeling
feedback distribution