A

Support

Vector

Method

for

Optimizing

Average

Precision

Yisong

Yue

Cornell

University

Ithaca,

NY,

USA

yyue@cs.cornell.edu

Thomas

Finley

Cornell

University

Ithaca,

NY,

USA

tomf@cs.cornell.edu

Filip

Radlinski

Cornell

University

Ithaca,

NY,

USA

filip@cs.cornell.edu

Thorsten

Joachims

Cornell

University

Ithaca,

NY,

USA

tj@cs.cornell.edu

ABSTRACT

Machine

learning

is

commonly

used

to

improve

ranked

retrieval

systems.

Due

to

computational

difficulties,

few

learning

techniques

have

been

developed

to

directly

optimize

for

mean

average

precision

(MAP),

despite

its

widespread

use

in

evaluating

such

systems.

Existing

approaches

optimizing

MAP

either

do

not

find

a

globally

optimal

solution,

or

are

computationally

expensive.

In

contrast,

we

present

a

general

SVM

learning

algorithm

that

efficiently

finds

a

globally

optimal

solution

to

a

straightforward

relaxation

of

MAP.

We

evaluate

our

approach

using

the

TREC

and

TREC

Web

Track

corpora

comparing

against

SVMs

optimized

for

accuracy

and

ROCArea.

In

most

cases

we

show

our

method

to

produce

statistically

significant

improvements

in

MAP

scores.

Categories

and

Subject

Descriptors

[Information

Search

and

Retrieval]:

Retrieval

Models

General

Terms

Algorithm,

Theory,

Experimentation

INTRODUCTION

State

of

the

art

information

retrieval

systems

commonly

use

machine

learning

techniques

to

learn

ranking

functions.

However,

most

current

approaches

do

not

optimize

for

the

evaluation

measure

most

often

used,

namely

Mean

Average

Precision

(MAP).

Instead,

current

algorithms

tend

to

take

one

of

two

general

approaches.

The

first

approach

is

to

learn

a

model

that

estimates

the

probability

of

a

document

being

relevant

given

a

query

(e.g.,

If

solved

effectively,

the

ranking

with

best

MAP

performance

can

easily

be

derived

from

the

probabilities

of

relevance.

However,

achieving

high

MAP

only

requires

finding

a

good

ordering

of

the

documents.

As

a

result,

finding

good

probabilities

requires

solving

a

more

difficult

problem

than

necessary,

likely

requiring

more

training

data

to

achieve

the

same

MAP

performance.

The

second

common

approach

is

to

learn

a

function

that

maximizes

a

surrogate

measure.

Performance

measures

optimized

include

accuracy

ROCArea

or

modifications

of

ROCArea

and

NDCG

Learning

a

model

to

optimize

for

such

measures

might

result

in

suboptimal

MAP

performance.

In

fact,

although

some

previous

systems

have

obtained

good

MAP

performance,

it

is

known

that

neither

achieving

optimal

accuracy

nor

ROCArea

can

guarantee

optimal

MAP

In

this

paper,

we

present

a

general

approach

for

learning

ranking

functions

that

maximize

MAP

performance.

Specifically,

we

present

an

SVM

algorithm

that

globally

optimizes

a

hinge-loss

relaxation

of

MAP.

This

approach

simplifies

the

process

of

obtaining

ranking

functions

with

high

MAP

performance

by

avoiding

additional

intermediate

steps

and

heuristics.

The

new

algorithm

also

makes

it

conceptually

just

as

easy

to

optimize

SVMs

for

MAP

as

was

previously

possible

only

for

accuracy

and

ROCArea.

In

contrast

to

recent

work

directly

optimizing

for

MAP

performance

by

Metzler

Croft

and

Caruana

et

al.

our

technique

is

computationally

efficient

while

finding

a

globally

optimal

solution.

Like

our

method

learns

a

linear

model,

but

is

much

more

efficient

in

practice

and,

unlike

can

handle

many

thousands

of

features.

We

now

describe

the

algorithm

in

detail

and

provide

proof

of

correctness.

Following

this,

we

provide

an

analysis

of

running

time.

We

finish

with

empirical

results

from

experiments

on

the

TREC

and

TREC

Web

Track

corpus.

We

have

also

developed

a

software

package

implementing

our

algorithm

that

is

available

for

public

THE

LEARNING

PROBLEM

Following

the

standard

machine

learning

setup,

our

goal

is

to

learn

a

function

h

X

Y

between

an

input

space

X

(all

possible

queries)

and

output

space

Y

(rankings

over

a

corpus).

In

order

to

quantify

the

quality

of

a

prediction,

ˆy

h(x),

we

will

consider

a

loss

function

Y

Y

∆(y,

ˆy)

quantifies

the

penalty

for

making

prediction

ˆy

if

the

correct

output

is

y.

The

loss

function

allows

us

to

incorporate

specific

performance

measures,

which

we

will

exploit

for

optimizing

MAP.

We

restrict

ourselves

to

the

supervised

learning

scenario,

where

input/output

pairs

(x,

y)

are

available

for

training

and

are

assumed

to

come

from

some

fixed

distribution

P(x,

y).

The

goal

is

to

find

a

function

h

such

that

the

risk

(i.e.,

expected

loss),

R∆

P

(h)

Z

X×Y

∆(y,

h(x))dP(x,

y),

is

minimized.

Of

course,

P(x,

y)

is

unknown.

But

given

a

finite

set

of

training

pairs,

S

{(xi,

yi)

X

Y

i

n},

the

performance

of

h

on

S

can

be

measured

by

the

empirical

risk,

R∆

S

(h)

n

nX

∆(yi,

h(xi)).

In

the

case

of

learning

a

ranked

retrieval

function,

X

denotes

a

space

of

queries,

and

Y

the

space

of

(possibly

weak)

rankings

over

some

corpus

of

documents

C

,d|C|}.

We

can

define

average

precision

loss

as

∆map(y,

ˆy)

MAP(rank(y),

rank(ˆy)),

where

rank(y)

is

a

vector

of

the

rank

values

of

each

document

in

C.

For

example,

for

a

corpus

of

two

documents,

with

having

higher

rank

than

rank(y

We

assume

true

rankings

have

two

rank

values,

where

relevant

documents

have

rank

value

and

non-relevant

documents

rank

value

We

further

assume

that

all

predicted

rankings

are

complete

rankings

(no

ties).

Let

p

rank(y)

and

ˆp

rank(ˆy).

The

average

precision

score

is

defined

as

MAP(p,

ˆp)

rel

X

j:pj

Prec@j,

where

rel

|{i

pi

is

the

number

of

relevant

documents,

and

Prec@j

is

the

percentage

of

relevant

documents

in

the

top

j

documents

in

predicted

ranking

ˆy.

MAP

is

the

mean

of

the

average

precision

scores

of

a

group

of

queries.

MAP

vs

ROCArea

Most

learning

algorithms

optimize

for

accuracy

or

ROCArea.

While

optimizing

for

these

measures

might

achieve

good

MAP

performance,

we

use

two

simple

examples

to

show

it

can

also

be

suboptimal

in

terms

of

MAP.

ROCArea

assigns

equal

penalty

to

each

misordering

of

a

relevant/non-relevant

pair.

In

contrast,

MAP

assigns

greater

penalties

to

misorderings

higher

up

in

the

predicted

ranking.

Using

our

notation,

ROCArea

can

be

defined

as

ROC(p,

ˆp)

rel

(|C|

rel)

X

X

j:pj

where

p

is

the

true

(weak)

ranking,

ˆp

is

the

predicted

ranking,

and

is

the

indicator

function

conditioned

on

b.

Doc

ID

p

Table

Toy

Example

and

Models

Suppose

we

have

a

hypothesis

space

with

only

two

hypothesis

functions,

and

as

shown

in

Table

These

two

hypotheses

predict

a

ranking

for

query

x

over

a

corpus

of

eight

documents.

Hypothesis

MAP

ROCArea

Table

Performance

of

Toy

Models

Table

shows

the

MAP

and

ROCArea

scores

of

and

Here,

a

learning

method

which

optimizes

for

ROCArea

would

choose

since

that

results

in

a

higher

ROCArea

score,

but

this

yields

a

suboptimal

MAP

score.

MAP

vs

Accuracy

Using

a

very

similar

example,

we

now

demonstrate

how

optimizing

for

accuracy

might

result

in

suboptimal

MAP.

Models

which

optimize

for

accuracy

are

not

directly

concerned

with

the

ranking.

Instead,

they

learn

a

threshold

such

that

documents

scoring

higher

than

the

threshold

can

be

classified

as

relevant

and

documents

scoring

lower

as

nonrelevant.

Doc

ID

p

Table

Toy

Example

and

Models

We

consider

again

a

hypothesis

space

with

two

hypotheses.

Table

shows

the

predictions

of

the

two

hypotheses

on

a

single

query

x.

Hypothesis

MAP

Best

Acc.

Table

Performance

of

Toy

Models

Table

shows

the

MAP

and

best

accuracy

scores

of

and

The

best

accuracy

refers

to

the

highest

achievable

accuracy

on

that

ranking

when

considering

all

possible

thresholds.

For

instance,

with

a

threshold

between

documents

and

gives

errors

(documents

incorrectly

classified

as

non-relevant),

yielding

an

accuracy

of

Similarly,

with

a

threshold

between

documents

and

gives

errors

(documents

incorrectly

classified

as

relevant,

and

document

as

non-relevant),

yielding

an

accuracy

of

A

learning

method

which

optimizes

for

accuracy

would

choose

since

that

results

in

a

higher

accuracy

score,

but

this

yields

a

suboptimal

MAP

score.

OPTIMIZING

AVERAGE

PRECISION

We

build

upon

the

approach

used

by

for

optimizing

ROCArea.

Unlike

ROCArea,

however,

MAP

does

not

decompose

linearly

in

the

examples

and

requires

a

substantially

extended

algorithm,

which

we

describe

in

this

section.

Recall

that

the

true

ranking

is

a

weak

ranking

with

two

rank

values

(relevant

and

non-relevant).

Let

Cx

and

C¯x

denote

the

set

of

relevant

and

non-relevant

documents

of

C

for

query

x,

respectively.

We

focus

on

functions

which

are

parametrized

by

a

weight

vector

w,

and

thus

wish

to

find

w

to

minimize

the

empirical

risk,

R∆

S

(w)

R∆

S

(h(·;

w)).

Our

approach

is

to

learn

a

discriminant

function

F

X

Y

over

input-output

pairs.

Given

query

x,

we

can

derive

a

prediction

by

finding

the

ranking

y

that

maximizes

the

discriminant

function:

h(x;

w)

argmax

y∈Y

F(x,

y;

w).

We

assume

F

to

be

linear

in

some

combined

feature

representation

of

inputs

and

outputs

Ψ(x,

y)

RN

i.e.,

F(x,

y;

w)

wT

Ψ(x,

y).

The

combined

feature

function

we

use

is

Ψ(x,

y)

|Cx|

|C¯x|

X

i:di∈Cx

X

j:dj

∈C¯x

[yij

(φ(x,

di)

φ(x,

dj))]

where

φ

X

C

N

is

a

feature

mapping

function

from

a

query/document

pair

to

a

point

in

N

dimensional

We

represent

rankings

as

a

matrix

of

pairwise

orderings,

Y

For

any

y

Y,

yij

if

di

is

ranked

ahead

of

dj,

and

yij

−1

if

dj

is

ranked

ahead

of

di,

and

yij

if

di

and

dj

have

equal

rank.

We

consider

only

matrices

which

correspond

to

valid

rankings

(i.e,

obeying

antisymmetry

and

transitivity).

Intuitively,

Ψ

is

a

summation

over

the

vector

differences

of

all

relevant/non-relevant

document

pairings.

Since

we

assume

predicted

rankings

to

be

complete

rankings,

yij

is

either

or

−1

(never

Given

a

learned

weight

vector

w,

predicting

a

ranking

(i.e.

solving

equation

given

query

x

reduces

to

picking

each

yij

to

maximize

wT

Ψ(x,

y).

As

is

also

discussed

in

this

is

attained

by

sorting

the

documents

by

wT

φ(x,

d)

in

descending

order.

We

will

discuss

later

the

choices

of

φ

we

used

for

our

experiments.

Structural

SVMs

The

above

formulation

is

very

similar

to

learning

a

straightforward

linear

model

while

training

on

the

pairwise

difference

of

relevant/non-relevant

document

pairings.

Many

SVM-based

approaches

optimize

over

these

pairwise

differences

(e.g.,

although

these

methods

do

not

optimize

for

MAP

during

training.

Previously,

it

was

not

clear

how

to

incorporate

non-linear

multivariate

loss

functions

such

as

MAP

loss

directly

into

global

optimization

problems

such

as

SVM

training.

We

now

present

a

method

based

on

structural

SVMs

to

address

this

problem.

We

use

the

structural

SVM

formulation,

presented

in

Optimization

Problem

to

learn

a

w

RN

Optimization

Problem

(Structural

SVM)

min

w

C

n

nX

ξi

s.t.

∀i,

∀y

Y

yi

wT

Ψ(xi,

yi)

wT

Ψ(xi,

y)

∆(yi,

y)

ξi

The

objective

function

to

be

minimized

is

a

tradeoff

between

model

complexity,

w

and

a

hinge

loss

relaxation

of

MAP

loss,

P

ξi.

As

is

usual

in

SVM

training,

C

is

a

For

example,

one

dimension

might

be

the

number

of

times

the

query

words

appear

in

the

document.

Algorithm

Cutting

plane

algorithm

for

solving

OP

within

tolerance

Input:

(xn,

yn),

C,

Wi

for

all

i

n

repeat

for

i

n

do

H(y;

w)

∆(yi,

y)

wT

Ψ(xi,

y)

wT

Ψ(xi,

yi)

compute

ˆy

argmaxy∈Y

H(y;

w)

compute

ξi

maxy∈Wi

H(y;

w)}

if

H(ˆy;

w)

ξi

then

Wi

Wi

{ˆy}

w

optimize

over

W

S

i

Wi

end

if

end

for

until

no

Wi

has

changed

during

iteration

parameter

that

controls

this

tradeoff

and

can

be

tuned

to

achieve

good

performance

in

different

training

tasks.

For

each

(xi,

yi)

in

the

training

set,

a

set

of

constraints

of

the

form

in

equation

is

added

to

the

optimization

problem.

Note

that

wT

Ψ(x,

y)

is

exactly

our

discriminant

function

F(x,

y;

w)

(see

equation

During

prediction,

our

model

chooses

the

ranking

which

maximizes

the

discriminant

If

the

discriminant

value

for

an

incorrect

ranking

y

is

greater

than

for

the

true

ranking

yi

(e.g.,

F(xi,

y;

w)

F(xi,

yi;

w)),

then

corresponding

slack

variable,

ξi,

must

be

at

least

∆(yi,

y)

for

that

constraint

to

be

satisfied.

Therefore,

the

sum

of

slacks,

P

ξi,

upper

bounds

the

MAP

loss.

This

is

stated

formally

in

Proposition

Proposition

Let

ξ∗

(w)

be

the

optimal

solution

of

the

slack

variables

for

OP

for

a

given

weight

vector

w.

Then

n

Pn

ξi

is

an

upper

bound

on

the

empirical

risk

R∆

S

(w).

(see

for

proof)

Proposition

shows

that

OP

learns

a

ranking

function

that

optimizes

an

upper

bound

on

MAP

error

on

the

training

set.

Unfortunately

there

is

a

problem:

a

constraint

is

required

for

every

possible

wrong

output

y,

and

the

number

of

possible

wrong

outputs

is

exponential

in

the

size

of

C.

Fortunately,

we

may

employ

Algorithm

to

solve

OP

Algorithm

is

a

cutting

plane

algorithm,

iteratively

introducing

constraints

until

we

have

solved

the

original

problem

within

a

desired

tolerance

The

algorithm

starts

with

no

constraints,

and

iteratively

finds

for

each

example

(xi,

yi)

the

output

ˆy

associated

with

the

most

violated

constraint.

If

the

corresponding

constraint

is

violated

by

more

than

we

introduce

ˆy

into

the

working

set

Wi

of

active

constraints

for

example

i,

and

re-solve

using

the

updated

W.

It

can

be

shown

that

Algorithm

outer

loop

is

guaranteed

to

halt

within

a

polynomial

number

of

iterations

for

any

desired

precision

Theorem

Let

¯R

maxi

maxy

Ψ(xi,

yi)

Ψ(xi,

y)

maxi

maxy

∆(yi,

y),

and

for

any

Algorithm

terminates

after

adding

at

most

max

ff

constraints

to

the

working

set

W.

(see

for

proof)

However,

within

the

inner

loop

of

this

algorithm

we

have

to

compute

argmaxy∈Y

H(y;

w),

where

H(y;

w)

∆(yi,

y)

wT

Ψ(xi,

y)

wT

Ψ(xi,

yi),

or

equivalently,

argmax

y∈Y

∆(yi,

y)

wT

Ψ(xi,

y),

since

wT

Ψ(xi,

yi)

is

constant

with

respect

to

y.

Though

closely

related

to

the

classification

procedure,

this

has

the

substantial

complication

that

we

must

contend

with

the

additional

∆(yi,

y)

term.

Without

the

ability

to

efficiently

find

the

most

violated

constraint

(i.e.,

solve

argmaxy∈Y

H(y,

w)),

the

constraint

generation

procedure

is

not

tractable.

Finding

the

Most

Violated

Constraint

Using

OP

and

optimizing

to

ROCArea

loss

(∆roc),

the

problem

of

finding

the

most

violated

constraint,

or

solving

argmaxy∈Y

H(y,

w)

(henceforth

argmax

H),

is

addressed

in

Solving

argmax

H

for

∆map

is

more

difficult.

This

is

primarily

because

ROCArea

decomposes

nicely

into

a

sum

of

scores

computed

independently

on

each

relative

ordering

of

a

relevant/non-relevant

document

pair.

MAP,

on

the

other

hand,

does

not

decompose

in

the

same

way

as

ROCArea.

The

main

algorithmic

contribution

of

this

paper

is

an

efficient

method

for

solving

argmax

H

for

∆map.

One

useful

property

of

∆map

is

that

it

is

invariant

to

swapping

two

documents

with

equal

relevance.

For

example,

if

documents

da

and

db

are

both

relevant,

then

swapping

the

positions

of

da

and

db

in

any

ranking

does

not

affect

∆map.

By

extension,

∆map

is

invariant

to

any

arbitrary

permutation

of

the

relevant

documents

amongst

themselves

and

of

the

non-relevant

documents

amongst

themselves.

However,

this

reshuﬄing

will

affect

the

discriminant

score,

wT

Ψ(x,

y).

This

leads

us

to

Observation

Observation

Consider

rankings

which

are

constrained

by

fixing

the

relevance

at

each

position

in

the

ranking

(e.g.,

the

document

in

the

ranking

must

be

relevant).

Every

ranking

which

satisfies

the

same

set

of

constraints

will

have

the

same

∆map.

If

the

relevant

documents

are

sorted

by

wT

φ(x,

d)

in

descending

order,

and

the

non-relevant

documents

are

likewise

sorted

by

wT

φ(x,

d),

then

the

interleaving

of

the

two

sorted

lists

which

satisfies

the

constraints

will

maximize

H

for

that

constrained

set

of

rankings.

Observation

implies

that

in

the

ranking

which

maximizes

H,

the

relevant

documents

will

be

sorted

by

wT

φ(x,

d),

and

the

non-relevant

documents

will

also

be

sorted

likewise.

By

first

sorting

the

relevant

and

non-relevant

documents,

the

problem

is

simplified

to

finding

the

optimal

interleaving

of

two

sorted

lists.

For

the

rest

of

our

discussion,

we

assume

that

the

relevant

documents

and

non-relevant

documents

are

both

sorted

by

descending

wT

φ(x,

d).

For

convenience,

we

also

refer

to

relevant

documents

as

{dx

dx

|Cx|}

Cx

and

non-relevant

documents

as

{d¯x

d¯x

|C¯x|}

C¯x

We

define

with

as

the

change

in

H

from

when

the

highest

ranked

relevant

document

ranked

after

d¯x

j

is

dx

to

when

it

is

dx

For

we

have

δj(i,

i

|Cx|

j

j

i

j

j

i

(sx

i

s¯x

j

where

si

wT

φ(x,

di).

The

first

term

in

is

the

change

in

∆map

when

the

ith

relevant

document

has

j

non-relevant

documents

ranked

before

it,

as

opposed

to

j

The

second

term

is

the

change

in

the

discriminant

score,

wT

Ψ(x,

y),

when

yij

changes

from

to

dx

i

d¯x

j

dx

d¯x

j

dx

i

dx

Figure

Example

for

δj(i,

i

Figure

gives

a

conceptual

example

for

δj(i,

i

The

bottom

ranking

differs

from

the

top

only

where

d¯x

j

slides

up

one

rank.

The

difference

in

the

value

of

H

for

these

two

rankings

is

exactly

δj(i,

i

For

any

we

can

then

define

as

X

δj(k,

k

or

equivalently,

X

|Cx|

j

j

k

j

j

k

(sx

k

s¯x

j

Let

o|C¯x|

encode

the

positions

of

the

non-relevant

documents,

where

dx

oj

is

the

highest

ranked

relevant

document

ranked

after

the

jth

non-relevant

document.

Due

to

Observation

this

encoding

uniquely

identifies

a

complete

ranking.

We

can

recover

the

ranking

as

yij

if

i

j

sign(si

sj)

if

di,

dj

equal

relevance

sign(oj

i

if

di

dx

i

dj

d¯x

j

sign(j

oi

if

di

d¯x

i

dj

dx

j

We

can

now

reformulate

H

into

a

new

objective

function,

H

o|C¯x||w)

H(¯y|w)

|C¯x

X

δk(ok,

|Cx

where

¯y

is

the

true

(weak)

ranking.

Conceptually

H

starts

with

a

perfect

ranking

¯y,

and

adds

the

change

in

H

when

each

successive

non-relevant

document

slides

up

the

ranking.

We

can

then

reformulate

the

argmax

H

problem

as

argmax

H

argmax

|C¯x

X

δk(ok,

|Cx

s.t.

o|C¯x|.

Algorithm

describes

the

algorithm

used

to

solve

equation

Conceptually,

Algorithm

starts

with

a

perfect

ranking.

Then

for

each

successive

non-relevant

document,

the

algorithm

modifies

the

solution

by

sliding

that

document

up

the

ranking

to

locally

maximize

H

while

keeping

the

positions

of

the

other

non-relevant

documents

constant.

Proof

of

Correctness

Algorithm

is

greedy

in

the

sense

that

it

finds

the

best

position

of

each

non-relevant

document

independently

from

the

other

non-relevant

documents.

In

other

words,

the

algorithm

maximizes

H

for

each

non-relevant

document,

d¯x

j

Algorithm

Finding

the

Most

Violated

Constraint

(argmax

H)

for

Algorithm

with

∆map

Input:

w,

Cx

C¯x

sort

Cx

and

C¯x

in

descending

order

of

wT

φ(x,

d)

sx

i

wT

φ(x,

dx

i

i

|Cx

s¯x

i

wT

φ(x,

d¯x

i

i

|C¯x

for

j

|C¯x

do

optj

argmaxk

δj(k,

|Cx

end

for

encode

ˆy

according

to

return

ˆy

without

considering

the

positions

of

the

other

non-relevant

documents,

and

thus

ignores

the

constraints

of

In

order

for

the

solution

to

be

feasible,

the

jth

non-relevant

document

must

be

ranked

after

the

first

j

non-relevant

documents,

thus

satisfying

opt|C¯x|.

If

the

solution

is

feasible,

the

it

clearly

solves

Therefore,

it

suffices

to

prove

that

Algorithm

satisfies

We

first

prove

that

δj(·,

is

monotonically

decreasing

in

j.

Lemma

For

any

|Cx

and

j

|C¯x

it

must

be

the

case

that

Proof.

Recall

from

that

both

and

are

summations

of

terms.

We

will

show

that

each

term

in

the

summation

of

is

no

greater

than

the

corresponding

term

in

or

k

δj(k,

k

for

k

Each

term

in

δj(k,

k

and

k

can

be

further

decomposed

into

two

parts

(see

We

will

show

that

each

part

of

k

is

no

greater

than

the

corresponding

part

in

δj(k,

k

In

other

words,

we

will

show

that

both

j

j

k

j

j

k

j

j

k

j

j

k

and

−2

(sx

k

s¯x

−2

(sx

k

s¯x

j

are

true

for

the

aforementioned

values

of

j

and

k.

It

is

easy

to

see

that

is

true

by

observing

that

for

any

two

positive

integers

a

b,

a

b

a

b

a

b

a

b

and

choosing

a

j

and

b

j

k.

The

second

inequality

holds

because

Algorithm

first

sorts

d¯x

in

descending

order

of

s¯x

implying

s¯x

s¯x

j

Thus

we

see

that

each

term

in

is

no

greater

than

the

corresponding

term

in

δj,

which

completes

the

proof.

The

result

of

Lemma

leads

directly

to

our

main

correctness

result:

Theorem

In

Algorithm

the

computed

values

of

optj

satisfy

implying

that

the

solution

returned

by

Algorithm

is

feasible

and

thus

optimal.

Proof.

We

will

prove

that

optj

holds

for

any

j

|C¯x

thus

implying

Since

Algorithm

computes

optj

as

optj

argmax

k

δj(k,

|Cx

then

by

definition

of

δj

for

any

i

optj,

δj(i,

optj)

δj(i,

|Cx

δj(optj,

|Cx

Using

Lemma

we

know

that

optj)

δj(i,

optj)

which

implies

that

for

any

i

optj,

|Cx

|Cx

Suppose

for

contradiction

that

optj.

Then

|Cx

|Cx

which

contradicts

Therefore,

it

must

be

the

case

that

optj

which

completes

the

proof.

Running

Time

The

running

time

of

Algorithm

can

be

split

into

two

parts.

The

first

part

is

the

sort

by

wT

φ(x,

d),

which

requires

O(n

log

n)

time,

where

n

|Cx

|C¯x

The

second

part

computes

each

optj,

which

requires

O(|Cx

|C¯x

time.

Though

in

the

worst

case

this

is

the

number

of

relevant

documents,

|Cx

is

often

very

small

(e.g.,

constant

with

respect

to

n),

in

which

case

the

running

time

for

the

second

part

is

simply

O(n).

For

most

real-world

datasets,

Algorithm

is

dominated

by

the

sort

and

has

complexity

O(n

log

n).

Algorithm

is

guaranteed

to

halt

in

a

polynomial

number

of

iterations

and

each

iteration

runs

Algorithm

Virtually

all

well-performing

models

were

trained

in

a

reasonable

amount

of

time

(usually

less

than

one

hour).

Once

training

is

complete,

making

predictions

on

query

x

using

the

resulting

hypothesis

h(x|w)

requires

only

sorting

by

wT

φ(x,

d).

We

developed

our

software

using

a

Python

to

SVMstruct

since

the

Python

language

greatly

simplified

the

coding

process.

To

improve

performance,

it

is

advisable

to

use

the

standard

C

of

SVMstruct

EXPERIMENT

SETUP

The

main

goal

of

our

experiments

is

to

evaluate

whether

directly

optimizing

MAP

leads

to

improved

MAP

performance

compared

to

conventional

SVM

methods

that

optimize

a

substitute

loss

such

as

accuracy

or

ROCArea.

We

empirically

evaluate

our

method

using

two

sets

of

TREC

Web

Track

queries,

one

each

from

TREC

and

TREC

(topics

and

both

of

which

used

the

corpus.

For

each

query,

TREC

provides

the

relevance

judgments

of

the

documents.

We

generated

our

features

using

the

scores

of

existing

retrieval

functions

on

these

queries.

While

our

method

is

agnostic

to

the

meaning

of

the

features,

we

chose

to

use

existing

retrieval

functions

as

a

simple

yet

effective

way

of

acquiring

useful

features.

As

such,

our

Dataset

Base

Funcs

Features

TREC

Indri

TREC

Indri

TREC

Submissions

TREC

Submissions

Table

Dataset

Statistics

experiments

essentially

test

our

method"s

ability

to

re-rank

the

highly

ranked

documents

(e.g.,

re-combine

the

scores

of

the

retrieval

functions)

to

improve

MAP.

We

compare

our

method

against

the

best

retrieval

functions

trained

on

(henceforth

base

functions),

as

well

as

against

previously

proposed

SVM

methods.

Comparing

with

the

best

base

functions

tests

our

method"s

ability

to

learn

a

useful

combination.

Comparing

with

previous

SVM

methods

allows

us

to

test

whether

optimizing

directly

for

MAP

(as

opposed

to

accuracy

or

ROCArea)

achieves

a

higher

MAP

score

in

practice.

The

rest

of

this

section

describes

the

base

functions

and

the

feature

generation

method

in

detail.

Choosing

Retrieval

Functions

We

chose

two

sets

of

base

functions

for

our

experiments.

For

the

first

set,

we

generated

three

indices

over

the

corpus

using

The

first

index

was

generated

using

default

settings,

the

second

used

Porter-stemming,

and

the

last

used

Porter-stemming

and

Indri"s

default

stopwords.

For

both

TREC

and

TREC

we

used

the

description

portion

of

each

query

and

scored

the

documents

using

five

of

Indri"s

built-in

retrieval

methods,

which

are

Cosine

Similarity,

TFIDF,

Okapi,

Language

Model

with

Dirichlet

Prior,

and

Language

Model

with

Jelinek-Mercer

Prior.

All

parameters

were

kept

as

their

defaults.

We

computed

the

scores

of

these

five

retrieval

methods

over

the

three

indices,

giving

base

functions

in

total.

For

each

query,

we

considered

the

scores

of

documents

found

in

the

union

of

the

top

documents

of

each

base

function.

For

our

second

set

of

base

functions,

we

used

scores

from

the

TREC

and

TREC

Web

Track

submissions.

We

used

only

the

non-manual,

non-short

submissions

from

both

years.

For

TREC

and

TREC

there

were

and

such

submissions,

respectively.

A

typical

submission

contained

scores

of

its

top

documents.

b

ca

wT

φ(x,d)

f(d|x)

Figure

Example

Feature

Binning

Generating

Features

In

order

to

generate

input

examples

for

our

method,

a

concrete

instantiation

of

φ

must

be

provided.

For

each

TREC

TREC

Model

MAP

W/L

MAP

W/L

SVM∆

map

Func.

Best

Best

Table

Comparison

with

Indri

Functions

ument

d

scored

by

a

set

of

retrieval

functions

F

on

query

x,

we

generate

the

features

as

a

vector

φ(x,

d)

∀f

F,

∀k

Kf

where

f(d|x)

denotes

the

score

that

retrieval

function

f

assigns

to

document

d

for

query

x,

and

each

Kf

is

a

set

of

real

values.

From

a

high

level,

we

are

expressing

the

score

of

each

retrieval

function

using

|Kf

bins.

Since

we

are

using

linear

kernels,

one

can

think

of

the

learning

problem

as

finding

a

good

piecewise-constant

combination

of

the

scores

of

the

retrieval

functions.

Figure

shows

an

example

of

our

feature

mapping

method.

In

this

example

we

have

a

single

feature

F

{f}.

Here,

Kf

{a,

b,

c},

and

the

weight

vector

is

w

wa,

wb,

wc

For

any

document

d

and

query

x,

we

have

wT

φ(x,

d)

if

f(d|x)

a

wa

if

a

f(d|x)

b

wa

wb

if

b

f(d|x)

c

wa

wb

wc

if

c

f(d|x)

This

is

expressed

qualitatively

in

Figure

where

wa

and

wb

are

positive,

and

wc

is

negative.

We

ran

our

main

experiments

using

four

choices

of

F:

the

set

of

aforementioned

Indri

retrieval

functions

for

TREC

and

TREC

and

the

Web

Track

submissions

for

TREC

and

TREC

For

each

F

and

each

function

f

F,

we

chose

values

for

Kf

which

are

reasonably

spaced

and

capture

the

sensitive

region

of

f.

Using

the

four

choices

of

F,

we

generated

four

datasets

for

our

main

experiments.

Table

contains

statistics

of

the

generated

datasets.

There

are

many

ways

to

generate

features,

and

we

are

not

advocating

our

method

over

others.

This

was

simply

an

efficient

means

to

normalize

the

outputs

of

different

functions

and

allow

for

a

more

expressive

model.

EXPERIMENTS

For

each

dataset

in

Table

we

performed

trials.

For

each

trial,

we

train

on

randomly

selected

queries,

and

select

another

queries

at

random

for

a

validation

set.

Models

were

trained

using

a

wide

range

of

C

values.

The

model

which

performed

best

on

the

validation

set

was

selected

and

tested

on

the

remaining

queries.

All

queries

were

selected

to

be

in

the

training,

validation

and

test

sets

the

same

number

of

times.

Using

this

setup,

we

performed

the

same

experiments

while

using

our

method

(SVM∆

map),

an

SVM

optimizing

for

ROCArea

(SVM∆

roc)

and

a

conventional

classification

SVM

(SVMacc)

All

SVM

methods

used

a

linear

kernel.

We

reported

the

average

performance

of

all

models

over

the

trials.

Comparison

with

Base

Functions

In

analyzing

our

results,

the

first

question

to

answer

is,

can

SVM∆

map

learn

a

model

which

outperforms

the

best

base

TREC

TREC

Model

MAP

W/L

MAP

W/L

SVM∆

map

Func.

Best

Best

Table

Comparison

with

TREC

Submissions

TREC

TREC

Model

MAP

W/L

MAP

W/L

SVM∆

map

Func.

Best

Best

Table

Comparison

with

TREC

Subm.

(w/o

best)

functions?

Table

presents

the

comparison

of

SVM∆

map

with

the

best

Indri

base

functions.

Each

column

group

contains

the

macro-averaged

MAP

performance

of

SVM∆

map

or

a

base

function.

The

W/L

columns

show

the

number

of

queries

where

SVM∆

map

achieved

a

higher

MAP

score.

Significance

tests

were

performed

using

the

two-tailed

Wilcoxon

signed

rank

test.

Two

stars

indicate

a

significance

level

of

All

tables

displaying

our

experimental

results

are

structured

identically.

Here,

we

find

that

SVM∆

map

significantly

outperforms

the

best

base

functions.

Table

shows

the

comparison

when

trained

on

TREC

submissions.

While

achieving

a

higher

MAP

score

than

the

best

base

functions,

the

performance

difference

between

SVM∆

map

the

base

functions

is

not

significant.

Given

that

many

of

these

submissions

use

scoring

functions

which

are

carefully

crafted

to

achieve

high

MAP,

it

is

possible

that

the

best

performing

submissions

use

techniques

which

subsume

the

techniques

of

the

other

submissions.

As

a

result,

SVM∆

map

would

not

be

able

to

learn

a

hypothesis

which

can

significantly

out-perform

the

best

submission.

Hence,

we

ran

the

same

experiments

using

a

modified

dataset

where

the

features

computed

using

the

best

submission

were

removed.

Table

shows

the

results

(note

that

we

are

still

comparing

against

the

best

submission

though

we

are

not

using

it

for

training).

Notice

that

while

the

performance

of

SVM∆

map

degraded

slightly,

the

performance

was

still

comparable

with

that

of

the

best

submission.

Comparison

w/

Previous

SVM

Methods

The

next

question

to

answer

is,

does

SVM∆

map

produce

higher

MAP

scores

than

previous

SVM

methods?

Tables

and

present

the

results

of

SVM∆

map,

SVM∆

roc,

and

SVMacc

when

trained

on

the

Indri

retrieval

functions

and

TREC

submissions,

respectively.

Table

contains

the

corresponding

results

when

trained

on

the

TREC

submissions

without

the

best

submission.

To

start

with,

our

results

indicate

that

SVMacc

was

not

competitive

with

SVM∆

map

and

SVM∆

roc,

and

at

times

underperformed

dramatically.

As

such,

we

tried

several

approaches

to

improve

the

performance

of

SVMacc.

Alternate

SVMacc

Methods

One

issue

which

may

cause

SVMacc

to

underperform

is

the

severe

imbalance

between

relevant

and

non-relevant

docTREC

TREC

Model

MAP

W/L

MAP

W/L

SVM∆

map

roc

SVMacc

Table

Trained

on

Indri

Functions

TREC

TREC

Model

MAP

W/L

MAP

W/L

SVM∆

map

roc

SVMacc

Table

Trained

on

TREC

Submissions

uments.

The

vast

majority

of

the

documents

are

not

relevant.

addresses

this

problem

by

assigning

more

penalty

to

false

negative

errors.

For

each

dataset,

the

ratio

of

the

false

negative

to

false

positive

penalties

is

equal

to

the

ratio

of

the

number

non-relevant

and

relevant

documents

in

that

dataset.

Tables

and

indicate

that

still

performs

significantly

worse

than

SVM∆

map.

Another

possible

issue

is

that

SVMacc

attempts

to

find

just

one

discriminating

threshold

b

that

is

query-invariant.

It

may

be

that

different

queries

require

different

values

of

b.

Having

the

learning

method

trying

to

find

a

good

b

value

(when

one

does

not

exist)

may

be

detrimental.

We

took

two

approaches

to

address

this

issue.

The

first

method,

converts

the

retrieval

function

scores

into

percentiles.

For

example,

for

document

d,

query

q

and

retrieval

function

f,

if

the

score

f(d|q)

is

in

the

top

of

the

scores

f(·|q)

for

query

q,

then

the

converted

score

is

f

(d|q)

Each

Kf

contains

evenly

spaced

values

between

and

Tables

and

show

that

the

performance

of

was

also

not

competitive

with

SVM∆

map.

The

second

method,

normalizes

the

scores

given

by

f

for

each

query.

For

example,

assume

for

query

q

that

f

outputs

scores

in

the

range

to

Then

for

document

d,

if

f(d|q)

the

converted

score

would

be

f

(d|q)

Each

Kf

contains

evenly

spaced

values

between

and

Again,

Tables

and

show

that

was

not

competitive

with

SVM∆

map

MAP

vs

ROCArea

SVM∆

roc

performed

much

better

than

SVMacc

in

our

experiments.

When

trained

on

Indri

retrieval

functions

(see

Table

the

performance

of

SVM∆

roc

was

slight,

though

not

significantly,

worse

than

the

performances

of

SVM∆

map.

However,

Table

shows

that

SVM∆

map

did

significantly

outperform

SVM∆

roc

when

trained

on

the

TREC

submissions.

Table

shows

the

performance

of

the

models

when

trained

on

the

TREC

submissions

with

the

best

submission

removed.

The

performance

of

most

models

degraded

by

a

small

amount,

with

SVM∆

map

still

having

the

best

performance.

TREC

TREC

Model

MAP

W/L

MAP

W/L

SVM∆

map

roc

SVMacc

Table

Trained

on

TREC

Subm.

(w/o

Best)

CONCLUSIONS

AND

FUTURE

WORK

We

have

presented

an

SVM

method

that

directly

optimizes

MAP.

It

provides

a

principled

approach

and

avoids

difficult

to

control

heuristics.

We

formulated

the

optimization

problem

and

presented

an

algorithm

which

provably

finds

the

solution

in

polynomial

time.

We

have

shown

empirically

that

our

method

is

generally

superior

to

or

competitive

with

conventional

SVMs

methods.

Our

new

method

makes

it

conceptually

just

as

easy

to

optimize

SVMs

for

MAP

as

was

previously

possible

only

for

Accuracy

and

ROCArea.

The

computational

cost

for

training

is

very

reasonable

in

practice.

Since

other

methods

typically

require

tuning

multiple

heuristics,

we

also

expect

to

train

fewer

models

before

finding

one

which

achieves

good

performance.

The

learning

framework

used

by

our

method

is

fairly

general.

A

natural

extension

of

this

framework

would

be

to

develop

methods

to

optimize

for

other

important

IR

measures,

such

as

Normalized

Discounted

Cumulative

Gain

and

Mean

Reciprocal

Rank.

ACKNOWLEDGMENTS

This

work

was

funded

under

NSF

Award

NSF

CAREER

Award

and

a

gift

from

Yahoo!

Research.

The

third

author

was

also

partly

supported

by

a

Microsoft

Research

Fellowship.

REFERENCES

B.

T.

Bartell,

G.

W.

Cottrell,

and

R.

K.

Belew.

Automatic

combination

of

multiple

ranked

retrieval

systems.

In

Proceedings

of

the

ACM

Conference

on

Research

and

Development

in

Information

Retrieval

(SIGIR),

C.

Burges,

T.

Shaked,

E.

Renshaw,

A.

Lazier,

M.

Deeds,

N.

Hamilton,

and

G.

Hullender.

Learning

to

rank

using

gradient

descent.

In

Proceedings

of

the

International

Conference

on

Machine

Learning

(ICML),

C.

J.

C.

Burges,

R.

Ragno,

and

Q.

Le.

Learning

to

rank

with

non-smooth

cost

functions.

In

Proceedings

of

the

International

Conference

on

Advances

in

Neural

Information

Processing

Systems

(NIPS),

Y.

Cao,

J.

Xu,

T.-Y.

Liu,

H.

Li,

Y.

Huang,

and

H.-W.

Hon.

Adapting

ranking

SVM

to

document

retrieval.

In

Proceedings

of

the

ACM

Conference

on

Research

and

Development

in

Information

Retrieval

(SIGIR),

B.

Carterette

and

D.

Petkova.

Learning

a

ranking

from

pairwise

preferences.

In

Proceedings

of

the

ACM

Conference

on

Research

and

Development

in

Information

Retrieval

(SIGIR),

R.

Caruana,

A.

Niculescu-Mizil,

G.

Crew,

and

A.

Ksikes.

Ensemble

selection

from

libraries

of

models.

In

Proceedings

of

the

International

Conference

on

Machine

Learning

(ICML),

J.

Davis

and

M.

Goadrich.

The

relationship

between

precision-recall

and

ROC

curves.

In

Proceedings

of

the

International

Conference

on

Machine

Learning

(ICML),

D.

Hawking.

Overview

of

the

web

track.

In

Proceedings

of

D.

Hawking

and

N.

Craswell.

Overview

of

the

web

track.

In

Proceedings

of

Nov.

R.

Herbrich,

T.

Graepel,

and

K.

Obermayer.

Large

margin

rank

boundaries

for

ordinal

regression.

Advances

in

large

margin

classifiers,

A.

Herschtal

and

B.

Raskutti.

Optimising

area

under

the

ROC

curve

using

gradient

descent.

In

Proceedings

of

the

International

Conference

on

Machine

Learning

(ICML),

K.

Jarvelin

and

J.

Kekalainen.

Ir

evaluation

methods

for

retrieving

highly

relevant

documents.

In

Proceedings

of

the

ACM

Conference

on

Research

and

Development

in

Information

Retrieval

(SIGIR),

T.

Joachims.

A

support

vector

method

for

multivariate

performance

measures.

In

Proceedings

of

the

International

Conference

on

Machine

Learning

(ICML),

pages

New

York,

NY,

USA,

ACM

Press.

J.

Lafferty

and

C.

Zhai.

Document

language

models,

query

models,

and

risk

minimization

for

information

retrieval.

In

Proceedings

of

the

ACM

Conference

on

Research

and

Development

in

Information

Retrieval

(SIGIR),

pages

Y.

Lin,

Y.

Lee,

and

G.

Wahba.

Support

vector

machines

for

classification

in

nonstandard

situations.

Machine

Learning,

D.

Metzler

and

W.

B.

Croft.

A

markov

random

field

model

for

term

dependencies.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

pages

K.

Morik,

P.

Brockhausen,

and

T.

Joachims.

Combining

statistical

learning

with

a

knowledge-based

approach.

In

Proceedings

of

the

International

Conference

on

Machine

Learning,

S.

Robertson.

The

probability

ranking

principle

in

ir.

journal

of

documentation.

Journal

of

Documentation,

I.

Tsochantaridis,

T.

Hofmann,

T.

Joachims,

and

Y.

Altun.

Large

margin

methods

for

structured

and

interdependent

output

variables.

Journal

of

Machine

Learning

Research

(JMLR),

V.

Vapnik.

Statistical

Learning

Theory.

Wiley

and

Sons

Inc.,

L.

Yan,

R.

Dodier,

M.

Mozer,

and

R.

Wolniewicz.

Optimizing

classifier

performance

via

approximation

to

the

Wilcoxon-Mann-Witney

statistic.

In

Proceedings

of

the

International

Conference

on

Machine

Learning

(ICML),

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O

ranked retrieval system
learning technique
map relaxation
information retrieval system
machine learn for information retrieval
support vector machine

machine learning
mean average precision
optimal solution
relaxation of map
probability
surrogate measure
loss function
supervised learning
rank