HITS

Hits

TRECExploring

IR

Evaluation

Results

with

Network

Analysis

Stefano

Mizzaro

Dept.

of

Mathematics

and

Computer

Science

University

of

Udine

Via

delle

Scienze,

Udine,

Italy

mizzaro@dimi.uniud.it

Stephen

Robertson

Microsoft

Research

JJ

Thomson

Avenue

Cambridge

UK

ser@microsoft.com

ABSTRACT

We

propose

a

novel

method

of

analysing

data

gathered

from

TREC

or

similar

information

retrieval

evaluation

experiments.

We

define

two

normalized

versions

of

average

precision,

that

we

use

to

construct

a

weighted

bipartite

graph

of

TREC

systems

and

topics.

We

analyze

the

meaning

of

well

known

and

somewhat

generalized

indicators

from

social

network

analysis

on

the

Systems-Topics

graph.

We

apply

this

method

to

an

analysis

of

TREC

data;

among

the

results,

we

find

that

authority

measures

systems

performance,

that

hubness

of

topics

reveals

that

some

topics

are

better

than

others

at

distinguishing

more

or

less

effective

systems,

that

with

current

measures

a

system

that

wants

to

be

effective

in

TREC

needs

to

be

effective

on

easy

topics,

and

that

by

using

different

effectiveness

measures

this

is

no

longer

the

case.

Categories

and

Subject

Descriptors

[Information

Storage

and

Retrieval]:

Information

Search

and

Retrieval

General

Terms

Measurement,

Experimentation

INTRODUCTION

Evaluation

is

a

primary

concern

in

the

Information

Retrieval

(IR)

field.

TREC

(Text

REtrieval

Conference)

is

an

annual

benchmarking

exercise

that

has

become

a

de

facto

standard

in

IR

evaluation:

before

the

actual

conference,

TREC

provides

to

participants

a

collection

of

documents

and

a

set

of

topics

(representations

of

information

needs).

Participants

use

their

systems

to

retrieve,

and

submit

to

TREC,

a

list

of

documents

for

each

topic.

After

the

lists

have

been

submitted

and

pooled,

the

TREC

organizers

employ

human

assessors

to

provide

relevance

judgements

on

the

pooled

set.

This

defines

a

set

of

relevant

documents

for

each

topic.

System

effectiveness

is

then

measured

by

well

established

metrics

(Mean

Average

Precision

being

the

most

used).

Other

conferences

such

as

NTCIR,

INEX,

CLEF

provide

comparable

data.

Network

analysis

is

a

discipline

that

studies

features

and

properties

of

(usually

large)

networks,

or

graphs.

Of

particular

importance

is

Social

Network

Analysis

that

studies

networks

made

up

by

links

among

humans

(friendship,

acquaintance,

co-authorship,

bibliographic

citation,

etc.).

Network

analysis

and

IR

fruitfully

meet

in

Web

Search

Engine

implementation,

as

is

already

described

in

textbooks

Current

search

engines

use

link

analysis

techniques

to

help

rank

the

retrieved

documents.

Some

indicators

(and

the

corresponding

algorithms

that

compute

them)

have

been

found

useful

in

this

respect,

and

are

nowadays

well

known:

Inlinks

(the

number

of

links

to

a

Web

page),

PageRank

and

HITS

(Hyperlink-Induced

Topic

Search)

Several

extensions

to

these

algorithms

have

been

and

are

being

proposed.

These

indicators

and

algorithms

might

be

quite

general

in

nature,

and

can

be

used

for

applications

which

are

very

different

from

search

result

ranking.

One

example

is

using

HITS

for

stemming,

as

described

by

Agosti

et

al.

In

this

paper,

we

propose

and

demonstrate

a

method

for

constructing

a

network,

specifically

a

weighted

complete

bidirectional

directed

bipartite

graph,

on

a

set

of

TREC

topics

and

participating

systems.

Links

represent

effectiveness

measurements

on

system-topic

pairs.

We

then

apply

analysis

methods

originally

developed

for

search

applications

to

the

resulting

network.

This

reveals

phenomena

previously

hidden

in

TREC

data.

In

passing,

we

also

provide

a

small

generalization

to

Kleinberg"s

HITS

algorithm,

as

well

as

to

Inlinks

and

PageRank.

The

paper

is

organized

as

follows:

Sect.

gives

some

motivations

for

the

work.

Sect.

presents

the

basic

ideas

of

normalizing

average

precision

and

of

constructing

a

systemstopics

graph,

whose

properties

are

analyzed

in

Sect.

Sect.

presents

some

experiments

on

TREC

data;

Sect.

discusses

some

issues

and

Sect.

closes

the

paper.

MOTIVATIONS

We

are

interested

in

the

following

hypotheses:

Some

systems

are

more

effective

than

others;

tn

MAP

tn)

sm

AP(sm,

AP(sm,

tn)

MAP(sm)

AAP

AAP(tn)

(a)

MAP

AAP

(b)

Table

AP,

MAP

and

AAP

Some

topics

are

easier

than

others;

Some

systems

are

better

than

others

at

distinguishing

easy

and

difficult

topics;

Some

topics

are

better

than

others

at

distinguishing

more

or

less

effective

systems.

The

first

of

these

hypotheses

needs

no

further

justification

every

reported

significant

difference

between

any

two

systems

supports

it.

There

is

now

also

quite

a

lot

of

evidence

for

the

second,

centered

on

the

TREC

Robust

Track

Our

primary

interest

is

in

the

third

and

fourth.

The

third

might

be

regarded

as

being

of

purely

academic

interest;

however,

the

fourth

has

the

potential

for

being

of

major

practical

importance

in

evaluation

studies.

If

we

could

identify

a

relatively

small

number

of

topics

which

were

really

good

at

distinguishing

effective

and

ineffective

systems,

we

could

save

considerable

effort

in

evaluating

systems.

One

possible

direction

from

this

point

would

be

to

attempt

direct

identification

of

such

small

sets

of

topics.

However,

in

the

present

paper,

we

seek

instead

to

explore

the

relationships

suggested

by

the

hypotheses,

between

what

different

topics

tell

us

about

systems

and

what

different

systems

tell

us

about

topics.

We

seek

methods

of

building

and

analysing

a

matrix

of

system-topic

normalised

performances,

with

a

view

to

giving

insight

into

the

issue

and

confirming

or

refuting

the

third

and

fourth

hypotheses.

It

turns

out

that

the

obvious

symmetry

implied

by

the

above

formulation

of

the

hypotheses

is

a

property

worth

investigating,

and

the

investigation

does

indeed

give

us

valuable

insights.

THE

IDEA

step:

average

precision

table

From

TREC

results,

one

can

produce

an

Average

Precision

(AP)

table

(see

Tab.

each

AP(si,

tj)

value

measures

the

AP

of

system

si

on

topic

tj.

Besides

AP

values,

the

table

shows

Mean

Average

Precision

(MAP)

values

i.e.,

the

mean

of

the

AP

values

for

a

single

system

over

all

topics,

and

what

we

call

Average

Average

Precision

(AAP)

values

i.e.,

the

average

of

the

AP

values

for

a

single

topic

over

all

systems:

MAP(si)

n

nX

AP(si,

tj),

AAP(tj)

m

mX

AP(si,

tj).

MAPs

are

indicators

of

systems

performance:

higher

MAP

means

good

system.

AAP

are

indicators

of

the

performance

on

a

topic:

higher

AAP

means

easy

topic

a

topic

on

which

all

or

most

systems

have

good

performance.

Critique

of

pure

AP

MAP

is

a

standard,

well

known,

and

widely

used

IR

effectiveness

measure.

Single

AP

values

are

used

too

(e.g.,

in

AP

histograms).

Topic

difficulty

is

often

discussed

(e.g.,

in

TREC

Robust

track

although

AAP

values

are

not

used

and,

to

the

best

of

our

knowledge,

have

never

been

proposed

(the

median,

not

the

average,

of

AP

on

a

topic

is

used

to

produce

TREC

AP

histograms

However,

the

AP

values

in

Tab.

present

two

limitations,

which

are

symmetric

in

some

respect:

Problem

They

are

not

reliable

to

compare

the

effectiveness

of

a

system

on

different

topics,

relative

to

the

other

systems.

If,

for

example,

can

we

infer

that

is

a

good

system

(i.e.,

has

a

good

performance)

on

and

a

bad

system

on

The

answer

is

no:

might

be

an

easy

topic

(with

high

AAP)

and

a

difficult

one

(low

AAP).

See

an

example

in

Tab.

is

outperformed

(on

average)

by

the

other

systems

on

and

it

outperforms

the

other

systems

on

Problem

Conversely,

if,

for

example,

can

we

infer

that

is

considered

easier

by

than

by

No,

we

cannot:

might

be

a

good

system

(with

high

MAP)

and

a

bad

one

(low

MAP);

see

an

example

in

Tab.

These

two

problems

are

a

sort

of

breakdown

of

the

well

known

high

influence

of

topics

on

IR

evaluation;

again,

our

formulation

makes

explicit

the

topics

systems

symmetry.

step:

normalizations

To

avoid

these

two

problems,

we

can

normalize

the

AP

table

in

two

ways.

The

first

normalization

removes

the

influence

of

the

single

topic

ease

on

system

performance.

Each

AP(si,

tj)

value

in

the

table

depends

on

both

system

goodness

and

topic

ease

(the

value

will

increase

if

a

system

is

good

and/or

the

topic

is

easy).

By

subtracting

from

each

AP(si,

tj)

the

AAP(tj)

value,

we

obtain

normalized

AP

values

(APA(si,

tj),

Normalized

AP

according

to

AAP):

APA(si,

tj)

AP(si,

tj)

AAP(tj),

that

depend

on

system

performance

only

(the

value

will

increase

only

if

system

performance

is

good).

See

Tab.

The

second

normalization

removes

the

influence

of

the

single

system

effectiveness

on

topic

ease:

by

subtracting

from

each

AP(si,

tj)

the

MAP(si)

value,

we

obtain

normalized

AP

values

(APM(si,

tj),

Normalized

AP

according

to

MAP):

APM(si,

tj)

AP(si,

tj)

MAP(si),

that

depend

on

topic

ease

only

(the

value

will

increase

only

if

the

topic

is

easy,

i.e.,

all

systems

perform

well

on

that

topic).

See

Tab.

In

other

words,

APA

avoids

Problem

APA(s,

t)

values

measure

the

performance

of

system

s

on

topic

t

normalized

tn

MAP

tn)

sm

APA(sm,

APA(sm,

tn)

MAP(sm)

(a)

tn

tn)

sm

APM(sm,

APM(sm,

tn)

AAP

AAP(tn)

(b)

MAP

AAP

(c)

(d)

Table

Normalizations:

APA

and

MAP:

normalized

AP

(APA)

and

MAP

(MAP)

(a);

normalized

AP

(APM)

and

AAP

(AAP)

(b);

a

numeric

example

(c)

and

(d)

according

to

the

ease

of

the

topic

(easy

topics

will

not

have

higher

APA

values).

Now,

if,

for

example,

we

can

infer

that

is

a

good

system

on

and

a

bad

system

on

(see

Tab.

Vice

versa,

APM

avoids

Problem

APM(s,

t)

values

measure

the

ease

of

topic

t

according

to

system

s,

normalized

according

to

goodness

of

the

system

(good

systems

will

not

lead

to

higher

APM

values).

If,

for

example,

we

can

infer

that

is

considered

easier

by

than

by

(see

Tab.

On

the

basis

of

Tables

and

we

can

also

define

two

new

measures

of

system

effectiveness

and

topic

ease,

i.e.,

a

Normalized

MAP

(MAP),

obtained

by

averaging

the

APA

values

on

one

row

in

Tab.

and

a

Normalized

AAP

(AAP),

obtained

by

averaging

the

APM

values

on

one

column

in

Tab.

MAP(si)

n

nX

APA(si,

tj)

AAP(tj)

m

mX

APM(si,

tj).

Thus,

overall

system

performance

can

be

measured,

besides

by

means

of

MAP,

also

by

means

of

MAP.

Moreover,

MAP

is

equivalent

to

MAP,

as

can

be

immediately

proved

by

using

Eqs.

and

MAP(si)

n

nX

(AP(si,

tj)

AAP(tj))

MAP(si)

n

nX

AAP(tj)

(and

n

Pn

AAP(tj)

is

the

same

for

all

systems).

And,

conversely,

overall

topic

ease

can

be

measured,

besides

by

tn

APM

sm

tn

APA

sm

sm

tn

APM

sm

APA

T

tn

MAP

AAP

Figure

Construction

of

the

adjacency

matrix.

APA

T

is

the

transpose

of

APA.

means

of

AAP,

also

by

means

of

AAP,

and

this

is

equivalent

(the

proof

is

analogous,

and

relies

on

Eqs.

and

The

two

Tables

and

are

interesting

per

se,

and

can

be

analyzed

in

several

different

ways.

In

the

following

we

propose

an

analysis

based

on

network

analysis

techniques,

mainly

Kleinberg"s

HITS

algorithm.

There

is

a

little

further

discussion

of

these

normalizations

in

Sect.

step:

Systems-Topics

Graph

The

two

tables

and

can

be

merged

into

a

single

one

with

the

procedure

shown

in

Fig.

The

obtained

matrix

can

be

interpreted

as

the

adjacency

matrix

of

a

complete

weighted

bipartite

graph,

that

we

call

Systems-Topics

graph.

Arcs

and

weights

in

the

graph

can

be

interpreted

as

follows:

(weight

on)

arc

s

t:

how

much

the

system

s

thinks

that

the

topic

t

is

easy

assuming

that

a

system

has

no

knowledge

of

the

other

systems

(or

in

other

words,

how

easy

we

might

think

the

topic

is,

knowing

only

the

results

for

this

one

system).

This

corresponds

to

APM

values,

i.e.,

to

normalized

topic

ease

(Fig.

(weight

on)

arc

s

t:

how

much

the

topic

t

thinks

that

the

system

s

is

good

assuming

that

a

topic

has

no

knowledge

of

the

other

topics

(or

in

other

words,

how

good

we

might

think

the

system

is,

knowing

only

the

results

for

this

one

topic).

This

corresponds

to

APA

(normalized

system

effectiveness,

Fig.

Figs.

and

show

the

Systems-Topics

complete

weighted

bipartite

graph,

on

a

toy

example

with

systems

and

topics;

the

graph

is

split

in

two

parts

to

have

an

understandable

graphical

representation:

arcs

in

Fig.

are

labeled

with

APM

values;

arcs

in

Fig.

are

labeled

with

APA

values.

ANALYSIS

OF

THE

GRAPH

Weighted

Inlinks,

Outlinks,

PageRank

The

sum

of

weighted

outlinks,

i.e.,

the

sum

of

the

weights

on

the

outgoing

arcs

from

each

node,

is

always

zero:

The

outlinks

on

each

node

corresponding

to

a

system

s

(Fig.

is

the

sum

of

all

the

corresponding

APM

values

on

one

row

of

the

matrix

in

Tab.

The

outlinks

on

each

node

corresponding

to

a

topic

t

(Fig.

is

the

sum

of

all

the

corresponding

APA

(a)

(b)

(c)

(d)

Figure

The

relationships

between

systems

and

topics

(a)

and

(b);

and

the

Systems-Topics

graph

for

a

toy

example

(c)

and

(d).

Dashed

arcs

correspond

to

negative

values.

h

(a)

sm

tn

sm

tn

APM

(APA)

sm

APA

T

tn

(APM

T

a

(h)

sm

tn

Figure

Hub

and

Authority

computation

values

on

one

row

of

the

transpose

of

the

matrix

in

Tab.

The

of

weighted

inlinks

is:

MAP

for

each

node

corresponding

to

a

system

s;

this

corresponds

to

the

average

of

all

the

corresponding

APA

values

on

one

column

of

the

APA

T

part

of

the

adjacency

matrix

(see

Fig.

AAP

for

each

node

corresponding

to

a

topic

t;

this

corresponds

to

the

average

of

all

the

corresponding

APM

values

on

one

column

of

the

APM

part

of

the

adjacency

matrix

(see

Fig.

Therefore,

weighted

inlinks

measure

either

system

effectiveness

or

topic

ease;

weighted

outlinks

are

not

meaningful.

We

could

also

apply

the

PageRank

algorithm

to

the

network;

the

meaning

of

the

PageRank

of

a

node

is

not

quite

so

obvious

as

Inlinks

and

Outlinks,

but

it

also

seems

a

sensible

measure

of

either

system

effectiveness

or

topic

ease:

if

a

system

is

effective,

it

will

have

several

incoming

links

with

high

Usually,

the

sum

of

the

weights

on

the

incoming

arcs

to

each

node

is

used

in

place

of

the

average;

since

the

graph

is

complete,

it

makes

no

difference.

weights

(APA);

if

a

topic

is

easy

it

will

have

high

weights

(APM)

on

the

incoming

links

too.

We

will

see

experimental

confirmation

in

the

following.

Hubs

and

Authorities

Let

us

now

turn

to

more

sophisticated

indicators.

Kleinberg"s

HITS

algorithm

defines,

for

a

directed

graph,

two

indicators:

hubness

and

authority;

we

reiterate

here

some

of

the

basic

details

of

the

HITS

algorithm

in

order

to

emphasize

both

the

nature

of

our

generalization

and

the

interpretation

of

the

HITS

concepts

in

this

context.

Usually,

hubness

and

authority

are

defined

as

h(x)

P

x→y

a(y)

and

a(x)

P

y→x

h(y),

and

described

intuitively

as

a

good

hub

links

many

good

authorities;

a

good

authority

is

linked

from

many

good

hubs.

As

it

is

well

known,

an

equivalent

formulation

in

linear

algebra

terms

is

(see

also

Fig.

h

Aa

and

a

AT

h

(where

h

is

the

hubness

vector,

with

the

hub

values

for

all

the

nodes;

a

is

the

authority

vector;

A

is

the

adjacency

matrix

of

the

graph;

and

AT

its

transpose).

Usually,

A

contains

and

only,

corresponding

to

presence

and

absence

of

unweighted

directed

arcs,

but

Eq.

can

be

immediately

generalized

to

(in

fact,

it

is

already

valid

for)

A

containing

any

real

value,

i.e.,

to

weighted

graphs.

Therefore

we

can

have

a

generalized

version

(or

rather

a

generalized

interpretation,

since

the

formulation

is

still

the

original

one)

of

hubness

and

authority

for

all

nodes

in

a

graph.

An

intuitive

formulation

of

this

generalized

HITS

is

still

available,

although

slightly

more

complex:

a

good

hub

links,

by

means

of

arcs

having

high

weights,

many

good

authorities;

a

good

authority

is

linked,

by

means

of

arcs

having

high

weights,

from

many

good

hubs.

Since

arc

weights

can

be,

in

general,

negative,

hub

and

authority

values

can

be

negative,

and

one

could

speak

of

unhubness

and

unauthority;

the

intuitive

formulation

could

be

completed

by

adding

that

a

good

hub

links

good

unauthorities

by

means

of

links

with

highly

negative

weights;

a

good

authority

is

linked

by

good

unhubs

by

means

of

links

with

highly

negative

weights.

And,

also,

a

good

unhub

links

positively

good

unauthorities

and

negatively

good

authorities;

a

good

unauthority

is

linked

positively

from

good

unhubs

and

negatively

from

good

hubs.

Let

us

now

apply

generalized

HITS

to

our

Systems-Topics

graph.

We

compute

a(s),

h(s),

a(t),

and

h(t).

Intuitively,

we

expect

that

a(s)

is

somehow

similar

to

Inlinks,

so

it

should

be

a

measure

of

either

systems

effectiveness

or

topic

ease.

Similarly,

hubness

should

be

more

similar

to

Outlinks,

thus

less

meaningful,

although

the

interplay

between

hub

and

authority

might

lead

to

the

discovery

of

something

different.

Let

us

start

by

remarking

that

authority

of

topics

and

hubness

of

systems

depend

only

on

each

other;

similarly

hubness

of

topics

and

authority

of

systems

depend

only

on

each

other:

see

Figs.

and

Thus

the

two

graphs

in

Figs.

and

can

be

analyzed

independently.

In

fact

the

entire

HITS

analysis

could

be

done

in

one

direction

only,

with

just

APM(s,

t)

values

or

alternatively

with

just

APA(s,

t).

As

discussed

below,

probably

most

interest

resides

in

the

hubness

of

topics

and

the

authority

of

systems,

so

the

latter

makes

sense.

However,

in

this

paper,

we

pursue

both

analyses

together,

because

the

symmetry

itself

is

interesting.

Considering

Fig.

we

can

state

that:

Authority

a(t)

of

a

topic

node

t

increases

when:

if

h(si)

APM(si,

t)

increases

(or

if

APM(si,

t)

h(si)

increases);

if

h(si)

APM(si,

t)

decreases

(or

if

APM(si,

t)

h(si)

decreases).

Hubness

h(s)

of

a

system

node

s

increases

when:

if

a(tj)

APM(s,

tj)

increases

(or,

if

APM(s,

tj)

a(tj)

increases);

if

a(tj)

APM(s,

tj)

decreases

(or,

if

APM(s,

tj)

a(tj)

decreases).

We

can

summarize

this

as:

a(t)

is

high

if

APM(s,

t)

is

high

for

those

systems

with

high

h(s);

h(s)

is

high

if

APM(s,

t)

is

high

for

those

topics

with

high

a(t).

Intuitively,

authority

a(t)

of

a

topic

measures

topic

ease;

hubness

h(s)

of

a

system

measures

system"s

capability

to

recognize

easy

topics.

A

system

with

high

unhubness

(negative

hubness)

would

tend

to

regard

easy

topics

as

hard

and

hard

ones

as

easy.

The

situation

for

Fig.

i.e.,

for

a(s)

and

h(t),

is

analogous.

Authority

a(s)

of

a

system

node

s

measures

system

effectiveness:

it

increases

with

the

weight

on

the

arc

(i.e.,

APA(s,

tj))

and

the

hubness

of

the

incoming

topic

nodes

tj.

Hubness

h(t)

of

a

topic

node

t

measures

topic

capability

to

recognize

effective

systems:

if

h(t)

it

increases

further

if

APA(s,

tj)

increases;

if

h(t)

it

increases

if

APA(s,

tj)

decreases.

Intuitively,

we

can

state

that

A

system

has

a

higher

authority

if

it

is

more

effective

on

topics

with

high

hubness;

and

A

topic

has

a

higher

hubness

if

it

is

easier

for

those

systems

which

are

more

effective

in

general.

Conversely

for

system

hubness

and

topic

authority:

A

topic

has

a

higher

authority

if

it

is

easier

on

systems

with

high

hubness;

and

A

system

has

a

higher

hubness

if

it

is

more

effective

for

those

topics

which

are

easier

in

general.

Therefore,

for

each

system

we

have

two

indicators:

authority

(a(s)),

measuring

system

effectiveness,

and

hubness

(h(s)),

measuring

system

capability

to

estimate

topic

ease.

And

for

each

topic,

we

have

two

indicators:

authority

(a(t)),

measuring

topic

ease,

and

hubness

(h(t)),

measuring

topic

capability

to

estimate

systems

effectiveness.

We

can

define

them

formally

as

a(s)

X

t

h(t)

APA(s,

t),

h(t)

X

s

a(s)

APA(s,

t),

a(t)

X

s

h(s)

APM(s,

t),

h(s)

X

t

a(t)

APM(s,

t).

We

observe

that

the

hubness

of

topics

may

be

of

particular

interest

for

evaluation

studies.

It

may

be

that

we

can

evaluate

the

effectiveness

of

systems

efficiently

by

using

relatively

few

high-hubness

topics.

EXPERIMENTS

We

now

turn

to

discuss

if

these

indicators

are

meaningful

and

useful

in

practice,

and

how

they

correlate

with

standard

measures

used

in

TREC.

We

have

built

the

Systems-Topics

graph

for

TREC

data

(featuring

systems

actually,

Actually,

TREC

data

features

systems;

due

to

some

bug

in

our

scripts,

we

did

not

include

one

system

but

the

results

should

not

be

affected.

NAPM

NAPA

AP

Figure

Distributions

of

AP,

APA,

and

APM

values

in

TREC

data

MAP

In

PR

H

A

MAP

Inlinks

PageRank

Hub

(a)

AAP

In

PR

H

A

AAP

Inlinks

PageRank

Hub

(b)

Table

Correlations

between

network

analysis

measures

and

MAP

(a)

and

AAP

(b)

runs

on

topics).

This

section

illustrates

the

results

obtained

mining

these

data

according

to

the

method

presented

in

previous

sections.

Fig.

shows

the

distributions

of

AP,

APA,

and

APM:

whereas

AP

is

very

skewed,

both

APA

and

APM

are

much

more

symmetric

(as

it

should

be,

since

they

are

constructed

by

subtracting

the

mean).

Tables

and

show

the

Pearson"s

correlation

values

between

Inlinks,

PageRank,

Hub,

Authority

and,

respectively,

MAP

or

AAP

(Outlinks

values

are

not

shown

since

they

are

always

zero,

as

seen

in

Sect.

As

expected,

Inlinks

and

PageRank

have

a

perfect

correlation

with

MAP

and

AAP.

Authority

has

a

very

high

correlation

too

with

MAP

and

AAP;

Hub

assumes

slightly

lower

values.

Let

us

analyze

the

correlations

more

in

detail.

The

correlations

chart

in

Figs.

and

demonstrate

the

high

correlation

between

Authority

and

MAP

or

AAP.

Hubness

presents

interesting

phenomena:

both

Fig.

(correlation

with

MAP)

and

Fig.

(correlation

with

AAP)

show

that

correlation

is

not

exact,

but

neither

is

it

random.

This,

given

the

meaning

of

hubness

(capability

in

estimating

topic

ease

and

system

effectiveness),

means

two

things:

(i)

more

effective

systems

are

better

at

estimating

topic

ease;

and

(ii)

easier

topics

are

better

at

estimating

system

effectiveness.

Whereas

the

first

statement

is

fine

(there

is

nothing

against

it),

the

second

is

a

bit

worrying.

It

means

that

system

effectiveness

in

TREC

is

affected

more

by

easy

topics

than

by

difficult

topics,

which

is

rather

undesirable

for

quite

obvious

reasons:

a

system

capable

of

performing

well

on

a

difficult

topic,

i.e.,

on

a

topic

on

which

the

other

systems

perform

badly,

would

be

an

important

result

for

IR

effectiveness;

(a)

(b)

(c)

(d)

Figure

Correlations:

MAP

(x

axis)

and

authority

(y

axis)

of

systems

(a);

AAP

and

authority

of

topics

(b);

MAP

and

hub

of

systems

(c)

and

AAP

and

hub

of

topics

(d)

versely,

a

system

capable

of

performing

well

on

easy

topics

is

just

a

confirmation

of

the

state

of

the

art.

Indeed,

the

correlation

between

hubness

and

AAP

(statement

(i)

above)

is

higher

than

the

correlation

between

hubness

and

MAP

(corresponding

to

statement

(ii)):

vs.

However,

this

phenomenon

is

quite

strong.

This

is

also

confirmed

by

the

work

being

done

on

the

TREC

Robust

Track

In

this

respect,

it

is

interesting

to

see

what

happens

if

we

use

a

different

measure

from

MAP

(and

AAP).

The

GMAP

(Geometric

MAP)

metric

is

defined

as

the

geometric

mean

of

AP

values,

or

equivalently

as

the

arithmetic

mean

of

the

logarithms

of

AP

values

GMAP

has

the

property

of

giving

more

weight

to

the

low

end

of

the

AP

scale

(i.e.,

to

low

AP

values),

and

this

seems

reasonable,

since,

intuitively,

a

performance

increase

in

MAP

values

from

to

should

be

more

important

than

an

increase

from

to

To

use

GMAP

in

place

of

MAP

and

AAP,

we

only

need

to

take

the

logarithms

of

initial

AP

values,

i.e.,

those

in

Tab.

(zero

values

are

modified

into

ε

We

then

repeat

the

same

normalization

process

(with

GMAP

and

GAAP

Geometric

AAP

replacing

MAP

and

AAP):

whereas

authority

values

still

perfectly

correlate

with

GMAP

and

GAAP

the

correlation

with

hubness

largely

disappears

(values

are

and

slightly

negative

but

not

enough

to

concern

us).

This

is

yet

another

confirmation

that

TREC

effectiveness

as

measured

by

MAP

depends

mainly

on

easy

topics;

GMAP

appears

to

be

a

more

balanced

measure.

Note

that,

perhaps

surprisingly,

GMAP

is

indeed

fairly

well

balanced,

not

biased

in

the

opposite

direction

that

is,

it

does

not

overemphasize

the

difficult

topics.

In

Sect.

below,

we

discuss

another

transformation,

replacing

the

log

function

used

in

GMAP

with

logit.

This

has

a

similar

effect:

the

correlation

of

mean

logitAP

and

average

logitAP

with

hubness

are

now

small

positive

numbers

and

respectively),

still

comfortably

away

from

the

high

correlations

with

regular

MAP

and

AAP,

i.e.,

not

presenting

the

problematic

phenomenon

(ii)

above

(over-dependency

on

easy

topics).

We

also

observe

that

hub

values

are

positive,

whereas

authority

assumes,

as

predicted,

both

positive

and

negative

values.

An

intuitive

justification

is

that

negative

hubness

would

indicate

a

node

which

disagrees

with

the

other

nodes,

e.g.,

a

system

which

does

better

on

difficult

topics,

or

a

topic

on

which

bad

systems

do

better;

such

systems

and

topics

would

be

quite

strange,

and

probably

do

not

appear

in

TREC.

Finally,

although

one

might

think

that

topics

with

several

relevant

documents

are

more

important

and

difficult,

this

is

not

the

case:

there

is

no

correlation

between

hub

(or

any

other

indicator)

and

the

number

of

documents

relevant

to

a

topic.

DISCUSSION

Related

work

There

has

been

considerable

interest

in

recent

years

in

questions

of

statistical

significance

of

effectiveness

comparisons

between

systems

(e.g.

and

related

questions

of

how

many

topics

might

be

needed

to

establish

differences

(e.g.

We

regard

some

results

of

the

present

study

as

in

some

way

complementary

to

this

work,

in

that

we

make

a

step

towards

answering

the

question

Which

topics

are

best

for

establishing

differences?.

The

results

on

evaluation

without

relevance

judgements

such

as

show

that,

to

some

extent,

good

systems

agree

on

which

are

the

good

documents.

We

have

not

addressed

the

question

of

individual

documents

in

the

present

analysis,

but

this

effect

is

certainly

analogous

to

our

results.

Are

normalizations

necessary?

At

this

point

it

is

also

worthwhile

to

analyze

what

would

happen

without

the

MAP-

and

AAP-normalizations

defined

in

Sect.

Indeed,

the

process

of

graph

construction

(Sect.

is

still

valid:

both

the

APM

and

APA

matrices

are

replaced

by

the

AP

one,

and

then

everything

goes

on

as

above.

Therefore,

one

might

think

that

the

normalizations

are

unuseful

in

this

setting.

This

is

not

the

case.

From

the

theoretical

point

of

view,

the

AP-only

graph

does

not

present

the

interesting

properties

above

discussed:

since

the

AP-only

graph

is

symmetrical

(the

weight

on

each

incoming

link

is

equal

to

the

weight

on

the

corresponding

outgoing

link),

Inlinks

and

Outlinks

assume

the

same

values.

There

is

symmetry

also

in

computing

hub

and

authority,

that

assume

the

same

value

for

each

node

since

the

weights

on

the

incoming

and

outgoing

arcs

are

the

same.

This

could

be

stated

in

more

precise

and

formal

terms,

but

one

might

still

wonder

if

on

the

overall

graph

there

are

some

sort

of

counterbalancing

effects.

It

is

therefore

easier

to

look

at

experimental

data,

which

confirm

that

the

normalizations

are

needed:

the

correlations

between

AP,

Inlinks,

Outlinks,

Hub,

and/or

Authority

are

all

very

close

to

one

(none

of

them

is

below

Are

these

normalizations

sufficient?

It

might

be

argued

that

(in

the

case

of

APA,

for

example)

the

amount

we

have

subtracted

from

each

AP

value

is

topicdependent,

therefore

the

range

of

the

resulting

APA

value

is

also

topic-dependent

(e.g.

the

maximum

is

AAP(tj)

and

the

minimum

is

AAP(tj)).

This

suggests

that

the

cross-topic

comparisons

of

these

values

suggested

in

Sect.

may

not

be

reliable.

A

similar

issue

arises

for

APM

and

comparisons

across

systems.

One

possible

way

to

overcome

this

would

be

to

use

an

unconstrained

measure

whose

range

is

the

full

real

line.

Note

that

in

applying

the

method

to

GMAP

by

using

log

AP,

we

avoid

the

problem

with

the

lower

limit

but

retain

it

for

the

upper

limit.

One

way

to

achieve

an

unconstrainted

range

would

be

to

use

the

logit

function

rather

than

the

log

We

have

also

run

this

variant

(as

already

reported

in

Sect.

above),

and

it

appears

to

provide

very

similar

results

to

the

GMAP

results

already

given.

This

is

not

surprising,

since

in

practice

the

two

functions

are

very

similar

over

most

of

the

operative

range.

The

normalizations

thus

seem

reliable.

On

AAT

and

AT

A

It

is

well

known

that

h

and

a

vectors

are

the

principal

left

eigenvectors

of

AAT

and

AT

A,

respectively

(this

can

be

easily

derived

from

Eqs.

and

that,

in

the

case

of

citation

graphs,

AAT

and

AT

A

represent,

respectively,

bibliographic

coupling

and

co-citations.

What

is

the

meaning,

if

any,

of

AAT

and

AT

A

in

our

Systems-Topics

graph?

It

is

easy

to

derive

that:

AAT

[i,

j]

if

i

S

j

T

or

i

T

j

S

P

k

A[i,

k]

A[j,

k]

otherwise

AT

A[i,

j]

if

i

S

j

T

or

i

T

j

S

P

k

A[k,

i]

A[k,

j]

otherwise

(where

S

is

the

set

of

indices

corresponding

to

systems

and

T

the

set

of

indices

corresponding

to

topics).

Thus

AAT

and

AT

A

are

block

diagonal

matrices,

with

two

blocks

each,

one

relative

to

systems

and

one

relative

to

topics:

(a)

if

i,

j

S,

then

AAT

[i,

j]

P

k∈T

APM(i,

k)·APM(j,

k)

measures

how

much

the

two

systems

i

and

j

agree

in

estimating

topics

ease

(APM):

high

values

mean

that

the

two

systems

agree

on

topics

ease.

(b)

if

i,

j

T,

then

AAT

[i,

j]

P

k∈S

APA(k,

i)·APA(k,

j)

measures

how

much

the

two

topics

i

and

j

agree

in

estimating

systems

effectiveness

(APA):

high

values

mean

that

the

two

topics

agree

on

systems

effectiveness

(and

that

TREC

results

would

not

change

by

leaving

out

one

of

the

two

topics).

(c)

if

i,

j

S,

then

AT

A[i,

j]

P

k∈T

APA(i,

k)

APA(j,

k)

measures

how

much

agreement

on

the

effectiveness

of

two

systems

i

and

j

there

is

over

all

topics:

high

values

mean

that

many

topics

quite

agree

on

the

two

systems

effectiveness;

low

values

single

out

systems

that

are

somehow

controversial,

and

that

need

several

topics

to

have

a

correct

effectiveness

assessment.

(d)

if

i,

j

T,

then

AT

A[i,

j]

P

k∈S

APM(k,

i)·APM(k,

j)

measures

how

much

agreement

on

the

ease

of

the

two

topics

i

and

j

there

is

over

all

systems:

high

values

mean

that

many

systems

quite

agree

on

the

two

topics

ease.

Therefore,

these

matrices

are

meaningful

and

somehow

interesting.

For

instance,

the

submatrix

(b)

corresponds

to

a

weighted

undirected

complete

graph,

whose

nodes

are

the

topics

and

whose

arc

weights

are

a

measure

of

how

much

two

topics

agree

on

systems

effectiveness.

Two

topics

that

are

very

close

on

this

graph

give

the

same

information,

and

therefore

one

of

them

could

be

discarded

without

changes

in

TREC

results.

It

would

be

interesting

to

cluster

the

topics

on

this

graph.

Furthermore,

the

matrix/graph

(a)

could

be

useful

in

TREC

pool

formation:

systems

that

do

not

agree

on

topic

ease

would

probably

find

different

relevant

documents,

and

should

therefore

be

complementary

in

pool

formation.

Note

that

no

notion

of

single

documents

is

involved

in

the

above

analysis.

Insights

As

indicated,

the

primary

contribution

of

this

paper

has

been

a

method

of

analysis.

However,

in

the

course

of

applying

this

method

to

one

set

of

TREC

results,

we

have

achieved

some

insights

relating

to

the

hypotheses

formulated

in

Sect.

We

confirm

Hypothesis

above,

that

some

topics

are

easier

than

others.

Differences

in

the

hubness

of

systems

reveal

that

some

systems

are

better

than

others

at

distinguishing

easy

and

difficult

topics;

thus

we

have

some

confirmation

of

Hypothesis

There

are

some

relatively

idiosyncratic

systems

which

do

badly

on

some

topics

generally

considered

easy

but

well

on

some

hard

topics.

However,

on

the

whole,

the

more

effective

systems

are

better

at

distinguishing

easy

and

difficult

topics.

This

is

to

be

expected:

a

really

bad

system

will

do

badly

on

everything,

while

even

a

good

system

may

have

difficulty

with

some

topics.

Differences

in

the

hubness

of

topics

reveal

that

some

topics

are

better

than

others

at

distinguising

more

or

less

effective

systems;

thus

we

have

some

confirmation

of

Hypothesis

If

we

use

MAP

as

the

measure

of

effectiveness,

it

is

also

true

that

the

easiest

topics

are

better

at

distinguishing

more

or

less

effective

systems.

As

argued

in

Sect.

this

is

an

undesirable

property.

GMAP

is

more

balanced.

Clearly

these

ideas

need

to

be

tested

on

other

data

sets.

However,

they

reveal

that

the

method

of

analysis

proposed

in

this

paper

can

provide

valuable

information.

Selecting

topics

The

confirmation

of

Hypothesis

leads,

as

indicated,

to

the

idea

that

we

could

do

reliable

system

evaluation

on

a

much

smaller

set

of

topics,

provided

we

could

select

such

an

appropriate

set.

This

selection

may

not

be

straightforward,

however.

It

is

possible

that

simply

selecting

the

high

hubness

topics

will

achieve

this

end;

however,

it

is

also

possible

that

there

are

significant

interactions

between

topics

which

would

render

such

a

simple

rule

ineffective.

This

investigation

would

therefore

require

serious

experimentation.

For

this

reason

we

have

not

attempted

in

this

paper

to

point

to

the

specific

high

hubness

topics

as

being

good

for

evaluation.

This

is

left

for

future

work.

CONCLUSIONS

AND

FUTURE

DEVELOPMENTS

The

contribution

of

this

paper

is

threefold:

we

propose

a

novel

way

of

normalizing

AP

values;

we

propose

a

novel

method

to

analyse

TREC

data;

the

method

applied

on

TREC

data

does

indeed

reveal

some

hidden

properties.

More

particularly,

we

propose

Average

Average

Precision

(AAP),

a

measure

of

topic

ease,

and

a

novel

way

of

normalizing

the

average

precision

measure

in

TREC,

on

the

basis

of

both

MAP

(Mean

Average

Precision)

and

AAP.

The

normalized

measures

(APM

and

APA)

are

used

to

build

a

bipartite

weighted

Systems-Topics

graph,

that

is

then

analyzed

by

means

of

network

analysis

indicators

widely

known

in

the

(social)

network

analysis

field,

but

somewhat

generalised.

We

note

that

no

such

approach

to

TREC

data

analysis

has

been

proposed

so

far.

The

analysis

shows

that,

with

current

measures,

a

system

that

wants

to

be

effective

in

TREC

needs

to

be

effective

on

easy

topics.

Also,

it

is

suggested

that

a

cluster

analysis

on

topic

similarity

can

lead

to

relying

on

a

lower

number

of

topics.

Our

method

of

analysis,

as

described

in

this

paper,

can

be

applied

only

a

posteriori,

i.e.,

once

we

have

all

the

topics

and

all

the

systems

available.

Adding

(removing)

a

new

system

topic

would

mean

re-computing

hubness

and

authority

indicators.

Moreover,

we

are

not

explicitly

proposing

a

change

to

current

TREC

methodology,

although

this

could

be

a

by-product

of

these

and

further

analyses.

This

is

an

initial

work,

and

further

analyses

could

be

performed.

For

instance,

other

effectiveness

metrics

could

be

used,

in

place

of

AP.

Other

centrality

indicators,

widely

used

in

social

network

analysis,

could

be

computed,

although

probably

with

similar

results

to

PageRank.

It

would

be

interesting

to

compute

the

higher-order

eigenvectors

of

AT

A

and

AAT

The

same

kind

of

analysis

could

be

performed

at

the

document

level,

measuring

document

ease.

Hopefully,

further

analyses

of

the

graph

defined

in

this

paper,

according

to

the

approach

described,

can

be

insightful

for

a

better

understanding

of

TREC

or

similar

data.

Acknowledgments

We

would

like

to

thank

Nick

Craswell

for

insightful

discussions

and

the

anonymous

referees

for

useful

remarks.

Part

of

this

research

has

been

carried

on

while

the

first

author

was

visiting

Microsoft

Research

Cambridge,

whose

financial

support

is

acknowledged.

REFERENCES

M.

Agosti,

M.

Bacchin,

N.

Ferro,

and

M.

Melucci.

Improving

the

automatic

retrieval

of

text

documents.

In

Proceedings

of

the

CLEF

Workshop,

volume

of

LNCS,

pages

C.

Buckley

and

E.

Voorhees.

Evaluating

evaluation

measure

stability.

In

SIGIR,

pages

S.

Chakrabarti.

Mining

the

Web.

Morgan

Kaufmann,

G.

V.

Cormack

and

T.

R.

Lynam.

Statistical

precision

of

information

retrieval

evaluation.

In

SIGIR,

pages

J.

Kleinberg.

Authoritative

sources

in

a

hyperlinked

environment.

J.

of

the

ACM,

M.

Levene.

An

Introduction

to

Search

Engines

and

Web

Navigation.

Addison

Wesley,

L.

Page,

S.

Brin,

R.

Motwani,

and

T.

Winograd.

The

PageRank

Citation

Ranking:

Bringing

Order

to

the

Web,

S.

Robertson.

On

GMAP

and

other

transformations.

In

CIKM,

pages

M.

Sanderson

and

J.

Zobel.

Information

retrieval

system

evaluation:

effort,

sensitivity,

and

reliability.

In

SIGIR,

pages

I.

Soboroff,

C.

Nicholas,

and

P.

Cahan.

Ranking

retrieval

systems

without

relevance

judgments.

In

SIGIR,

pages

TREC

Common

Evaluation

Measures,

(Last

visit:

Jan.

Text

REtrieval

Conference

(TREC).

(Last

visit:

Jan.

E.

Voorhees

and

C.

Buckley.

The

effect

of

topic

set

size

on

retrieval

experiment

error.

In

SIGIR,

pages

E.

M.

Voorhees.

Overview

of

the

TREC

Robust

Retrieval

Track.

In

TREC

Proceedings,

E.

M.

Voorhees

and

D.

K.

Harman.

TRECExperiment

and

Evaluation

in

Information

Retrieval.

MIT

Press,

S.

Wasserman

and

K.

Faust.

Social

Network

Analysis.

Cambridge

University

Press,

Cambridge,

UK,

O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I I O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O

information retrieval evaluation experiment

ir evaluation
network analysis
trec
weighted bipartite graph
mean average precision
web search engine implementation
pagerank
stemming