Learn

from

Web

Search

Logs

to

Organize

Search

Results

Xuanhui

Wang

Department

of

Computer

Science

University

of

Illinois

at

Urbana-Champaign

Urbana,

IL

ChengXiang

Zhai

Department

of

Computer

Science

University

of

Illinois

at

Urbana-Champaign

Urbana,

IL

czhai@cs.uiuc.edu

ABSTRACT

Effective

organization

of

search

results

is

critical

for

improving

the

utility

of

any

search

engine.

Clustering

search

results

is

an

effective

way

to

organize

search

results,

which

allows

a

user

to

navigate

into

relevant

documents

quickly.

However,

two

deficiencies

of

this

approach

make

it

not

always

work

well:

the

clusters

discovered

do

not

necessarily

correspond

to

the

interesting

aspects

of

a

topic

from

the

user"s

perspective;

and

the

cluster

labels

generated

are

not

informative

enough

to

allow

a

user

to

identify

the

right

cluster.

In

this

paper,

we

propose

to

address

these

two

deficiencies

by

learning

interesting

aspects

of

a

topic

from

Web

search

logs

and

organizing

search

results

accordingly;

and

generating

more

meaningful

cluster

labels

using

past

query

words

entered

by

users.

We

evaluate

our

proposed

method

on

a

commercial

search

engine

log

data.

Compared

with

the

traditional

methods

of

clustering

search

results,

our

method

can

give

better

result

organization

and

more

meaningful

labels.

Categories

and

Subject

Descriptors:

[Information

Search

and

Retrieval]:

Clustering,

Search

process

General

Terms:

Algorithm,

Experimentation

INTRODUCTION

The

utility

of

a

search

engine

is

affected

by

multiple

factors.

While

the

primary

factor

is

the

soundness

of

the

underlying

retrieval

model

and

ranking

function,

how

to

organize

and

present

search

results

is

also

a

very

important

factor

that

can

affect

the

utility

of

a

search

engine

significantly.

Compared

with

the

vast

amount

of

literature

on

retrieval

models,

however,

there

is

relatively

little

research

on

how

to

improve

the

effectiveness

of

search

result

organization.

The

most

common

strategy

of

presenting

search

results

is

a

simple

ranked

list.

Intuitively,

such

a

presentation

strategy

is

reasonable

for

non-ambiguous,

homogeneous

search

results;

in

general,

it

would

work

well

when

the

search

results

are

good

and

a

user

can

easily

find

many

relevant

documents

in

the

top

ranked

results.

However,

when

the

search

results

are

diverse

(e.g.,

due

to

ambiguity

or

multiple

aspects

of

a

topic)

as

is

often

the

case

in

Web

search,

the

ranked

list

presentation

would

not

be

effective;

in

such

a

case,

it

would

be

better

to

group

the

search

results

into

clusters

so

that

a

user

can

easily

navigate

into

a

particular

interesting

group.

For

example,

the

results

in

the

first

page

returned

from

Google

for

the

ambiguous

query

jaguar

(as

of

Dec.

contain

at

least

four

different

senses

of

jaguar

(i.e.,

car,

animal,

software,

and

a

sports

team);

even

for

a

more

refined

query

such

as

jaguar

team

picture,

the

results

are

still

quite

ambiguous,

including

at

least

four

different

jaguar

teams

a

wrestling

team,

a

jaguar

car

team,

Southwestern

College

Jaguar

softball

team,

and

Jacksonville

Jaguar

football

team.

Moreover,

if

a

user

wants

to

find

a

place

to

download

a

jaguar

software,

a

query

such

as

download

jaguar

is

also

not

very

effective

as

the

dominating

results

are

about

downloading

jaguar

brochure,

jaguar

wallpaper,

and

jaguar

DVD.

In

these

examples,

a

clustering

view

of

the

search

results

would

be

much

more

useful

to

a

user

than

a

simple

ranked

list.

Clustering

is

also

useful

when

the

search

results

are

poor,

in

which

case,

a

user

would

otherwise

have

to

go

through

a

long

list

sequentially

to

reach

the

very

first

relevant

document.

As

a

primary

alternative

strategy

for

presenting

search

results,

clustering

search

results

has

been

studied

relatively

extensively

The

general

idea

in

virtually

all

the

existing

work

is

to

perform

clustering

on

a

set

of

topranked

search

results

to

partition

the

results

into

natural

clusters,

which

often

correspond

to

different

subtopics

of

the

general

query

topic.

A

label

will

be

generated

to

indicate

what

each

cluster

is

about.

A

user

can

then

view

the

labels

to

decide

which

cluster

to

look

into.

Such

a

strategy

has

been

shown

to

be

more

useful

than

the

simple

ranked

list

presentation

in

several

studies

However,

this

clustering

strategy

has

two

deficiencies

which

make

it

not

always

work

well:

First,

the

clusters

discovered

in

this

way

do

not

necessarily

correspond

to

the

interesting

aspects

of

a

topic

from

the

user"s

perspective.

For

example,

users

are

often

interested

in

finding

either

phone

codes

or

zip

codes

when

entering

the

query

area

codes.

But

the

clusters

discovered

by

the

current

methods

may

partition

the

results

into

local

codes

and

international

codes.

Such

clusters

would

not

be

very

useful

for

users;

even

the

best

cluster

would

still

have

a

low

precision.

Second,

the

cluster

labels

generated

are

not

informative

enough

to

allow

a

user

to

identify

the

right

cluster.

There

are

two

reasons

for

this

problem:

The

clusters

are

not

corresponding

to

a

user"s

interests,

so

their

labels

would

not

be

very

meaningful

or

useful.

Even

if

a

cluster

really

corresponds

to

an

interesting

aspect

of

the

topic,

the

label

may

not

be

informative

because

it

is

usually

generated

based

on

the

contents

in

a

cluster,

and

it

is

possible

that

the

user

is

not

very

familiar

with

some

of

the

terms.

For

example,

the

ambiguous

query

jaguar

may

mean

an

animal

or

a

car.

A

cluster

may

be

labeled

as

panthera

onca.

Although

this

is

an

accurate

label

for

a

cluster

with

the

animal

sense

of

jaguar,

if

a

user

is

not

familiar

with

the

phrase,

the

label

would

not

be

helpful.

In

this

paper,

we

propose

a

different

strategy

for

partitioning

search

results,

which

addresses

these

two

deficiencies

through

imposing

a

user-oriented

partitioning

of

the

search

results.

That

is,

we

try

to

figure

out

what

aspects

of

a

search

topic

are

likely

interesting

to

a

user

and

organize

the

results

accordingly.

Specifically,

we

propose

to

do

the

following:

First,

we

will

learn

interesting

aspects

of

similar

topics

from

search

logs

and

organize

search

results

based

on

these

interesting

aspects.

For

example,

if

the

current

query

has

occurred

many

times

in

the

search

logs,

we

can

look

at

what

kinds

of

pages

viewed

by

the

users

in

the

results

and

what

kind

of

words

are

used

together

with

such

a

query.

In

case

when

the

query

is

ambiguous

such

as

jaguar

we

can

expect

to

see

some

clear

clusters

corresponding

different

senses

of

jaguar.

More

importantly,

even

if

a

word

is

not

ambiguous

(e.g.,

car),

we

may

still

discover

interesting

aspects

such

as

car

rental

and

car

pricing

(which

happened

to

be

the

two

primary

aspects

discovered

in

our

search

log

data).

Such

aspects

can

be

very

useful

for

organizing

future

search

results

about

car.

Note

that

in

the

case

of

car,

clusters

generated

using

regular

clustering

may

not

necessarily

reflect

such

interesting

aspects

about

car

from

a

user"s

perspective,

even

though

the

generated

clusters

are

coherent

and

meaningful

in

other

ways.

Second,

we

will

generate

more

meaningful

cluster

labels

using

past

query

words

entered

by

users.

Assuming

that

the

past

search

logs

can

help

us

learn

what

specific

aspects

are

interesting

to

users

given

the

current

query

topic,

we

could

also

expect

that

those

query

words

entered

by

users

in

the

past

that

are

associated

with

the

current

query

can

provide

meaningful

descriptions

of

the

distinct

aspects.

Thus

they

can

be

better

labels

than

those

extracted

from

the

ordinary

contents

of

search

results.

To

implement

the

ideas

presented

above,

we

rely

on

search

engine

logs

and

build

a

history

collection

containing

the

past

queries

and

the

associated

clickthroughs.

Given

a

new

query,

we

find

its

related

past

queries

from

the

history

collection

and

learn

aspects

through

applying

the

star

clustering

algorithm

to

these

past

queries

and

clickthroughs.

We

can

then

organize

the

search

results

into

these

aspects

using

categorization

techniques

and

label

each

aspect

by

the

most

representative

past

query

in

the

query

cluster.

We

evaluate

our

method

for

result

organization

using

logs

of

a

commercial

search

engine.

We

compare

our

method

with

the

default

search

engine

ranking

and

the

traditional

clustering

of

search

results.

The

results

show

that

our

method

is

effective

for

improving

search

utility

and

the

labels

generated

using

past

query

words

are

more

readable

than

those

generated

using

traditional

clustering

approaches.

The

rest

of

the

paper

is

organized

as

follows.

We

first

review

the

related

work

in

Section

In

Section

we

describe

search

engine

log

data

and

our

procedure

of

building

a

history

collection.

In

Section

we

present

our

approach

in

details.

We

describe

the

data

set

in

Section

and

the

experimental

results

are

discussed

in

Section

Finally,

we

conclude

our

paper

and

discuss

future

work

in

Section

RELATED

WORK

Our

work

is

closely

related

to

the

study

of

clustering

search

results.

In

the

authors

used

Scatter/Gather

algorithm

to

cluster

the

top

documents

returned

from

a

traditional

information

retrieval

system.

Their

results

validate

the

cluster

hypothesis

that

relevant

documents

tend

to

form

clusters.

The

system

Grouper

was

described

in

In

these

papers,

the

authors

proposed

to

cluster

the

results

of

a

real

search

engine

based

on

the

snippets

or

the

contents

of

returned

documents.

Several

clustering

algorithms

are

compared

and

the

Suffix

Tree

Clustering

algorithm

(STC)

was

shown

to

be

the

most

effective

one.

They

also

showed

that

using

snippets

is

as

effective

as

using

whole

documents.

However,

an

important

challenge

of

document

clustering

is

to

generate

meaningful

labels

for

clusters.

To

overcome

this

difficulty,

in

supervised

learning

algorithms

were

studied

to

extract

meaningful

phrases

from

the

search

result

snippets

and

these

phrases

were

then

used

to

group

search

results.

In

the

authors

proposed

to

use

a

monothetic

clustering

algorithm,

in

which

a

document

is

assigned

to

a

cluster

based

on

a

single

feature,

to

organize

search

results,

and

the

single

feature

is

used

to

label

the

corresponding

cluster.

Clustering

search

results

has

also

attracted

a

lot

of

attention

in

industry

and

commercial

Web

services

such

as

Vivisimo

However,

in

all

these

works,

the

clusters

are

generated

solely

based

on

the

search

results.

Thus

the

obtained

clusters

do

not

necessarily

reflect

users"

preferences

and

the

generated

labels

may

not

be

informative

from

a

user"s

viewpoint.

Methods

of

organizing

search

results

based

on

text

categorization

are

studied

in

In

this

work,

a

text

classifier

is

trained

using

a

Web

directory

and

search

results

are

then

classified

into

the

predefined

categories.

The

authors

designed

and

studied

different

category

interfaces

and

they

found

that

category

interfaces

are

more

effective

than

list

interfaces.

However

predefined

categories

are

often

too

general

to

reflect

the

finer

granularity

aspects

of

a

query.

Search

logs

have

been

exploited

for

several

different

purposes

in

the

past.

For

example,

clustering

search

queries

to

find

those

Frequent

Asked

Questions

(FAQ)

is

studied

in

Recently,

search

logs

have

been

used

for

suggesting

query

substitutes

personalized

search

Web

site

design

Latent

Semantic

Analysis

and

learning

retrieval

ranking

functions

In

our

work,

we

explore

past

query

history

in

order

to

better

organize

the

search

results

for

future

queries.

We

use

the

star

clustering

algorithm

which

is

a

graph

partition

based

approach,

to

learn

interesting

aspects

from

search

logs

given

a

new

query.

Thus

past

queries

are

clustered

in

a

query

specific

manner

and

this

is

another

difference

from

previous

works

such

as

in

which

all

queries

in

logs

are

clustered

in

an

oﬄine

batch

manner.

SEARCH

ENGINE

LOGS

Search

engine

logs

record

the

activities

of

Web

users,

which

reflect

the

actual

users"

needs

or

interests

when

conducting

ID

Query

URL

Time

win

zip

xxxx

win

zip

xxxx

time

zones

xxxx

Table

Sample

entries

of

search

engine

logs.

Different

ID"s

mean

different

sessions.

Web

search.

They

generally

have

the

following

information:

text

queries

that

users

submitted,

the

URLs

that

they

clicked

after

submitting

the

queries,

and

the

time

when

they

clicked.

Search

engine

logs

are

separated

by

sessions.

A

session

includes

a

single

query

and

all

the

URLs

that

a

user

clicked

after

issuing

the

query

A

small

sample

of

search

log

data

is

shown

in

Table

Our

idea

of

using

search

engine

logs

is

to

treat

these

logs

as

past

history,

learn

users"

interests

using

this

history

data

automatically,

and

represent

their

interests

by

representative

queries.

For

example,

in

the

search

logs,

a

lot

of

queries

are

related

to

car

and

this

reflects

that

a

large

number

of

users

are

interested

in

information

about

car.

Different

users

are

probably

interested

in

different

aspects

of

car.

Some

are

looking

for

renting

a

car,

thus

may

submit

a

query

like

car

rental;

some

are

more

interested

in

buying

a

used

car,

and

may

submit

a

query

like

used

car;

and

others

may

care

more

about

buying

a

car

accessory,

so

they

may

use

a

query

like

car

audio.

By

mining

all

the

queries

which

are

related

to

the

concept

of

car,

we

can

learn

the

aspects

that

are

likely

interesting

from

a

user"s

perspective.

As

an

example,

the

following

is

some

aspects

about

car

learned

from

our

search

log

data

(see

Section

car

rental,

hertz

car

rental,

enterprise

car

rental,

car

pricing,

used

car,

car

values,

car

accidents,

car

crash,

car

wrecks,

car

audio,

car

stereo,

car

speaker,

In

order

to

learn

aspects

from

search

engine

logs,

we

preprocess

the

raw

logs

to

build

a

history

data

collection.

As

shown

above,

search

engine

logs

consist

of

sessions.

Each

session

contains

the

information

of

the

text

query

and

the

clicked

Web

page

URLs,

together

with

the

time

that

the

user

did

the

clicks.

However,

this

information

is

limited

since

URLs

alone

are

not

informative

enough

to

tell

the

intended

meaning

of

a

submitted

query

accurately.

To

gather

rich

information,

we

enrich

each

URL

with

additional

text

content.

Specifically,

given

the

query

in

a

session,

we

obtain

its

top-ranked

results

using

the

search

engine

from

which

we

obtained

our

log

data,

and

extract

the

snippets

of

the

URLs

that

are

clicked

on

according

to

the

log

information

in

the

corresponding

session.

All

the

titles,

snippets,

and

URLs

of

the

clicked

Web

pages

of

that

query

are

used

to

represent

the

session.

Different

sessions

may

contain

the

same

queries.

Thus

the

number

of

sessions

could

be

quite

huge

and

the

information

in

the

sessions

with

the

same

queries

could

be

redundant.

In

order

to

improve

the

scalability

and

reduce

data

sparseness,

we

aggregate

all

the

sessions

which

contain

exactly

the

same

queries

together.

That

is,

for

each

unique

query,

we

build

a

pseudo-document

which

consists

of

all

the

descriptions

of

its

clicks

in

all

the

sessions

aggregated.

The

keywords

contained

in

the

queries

themselves

can

be

regarded

as

brief

summaries

of

the

pseudo-documents.

All

these

pseudo-documents

form

our

history

data

collection,

which

is

used

to

learn

interesting

aspects

in

the

following

section.

OUR

APPROACH

Our

approach

is

to

organize

search

results

by

aspects

learned

from

search

engine

logs.

Given

an

input

query,

the

general

procedure

of

our

approach

is:

Get

its

related

information

from

search

engine

logs.

All

the

information

forms

a

working

set.

Learn

aspects

from

the

information

in

the

working

set.

These

aspects

correspond

to

users"

interests

given

the

input

query.

Each

aspect

is

labeled

with

a

representative

query.

Categorize

and

organize

the

search

results

of

the

input

query

according

to

the

aspects

learned

above.

We

now

give

a

detailed

presentation

of

each

step.

Finding

Related

Past

Queries

Given

a

query

q,

a

search

engine

will

return

a

ranked

list

of

Web

pages.

To

know

what

the

users

are

really

interested

in

given

this

query,

we

first

retrieve

its

past

similar

queries

in

our

preprocessed

history

data

collection.

Formally,

assume

we

have

N

pseudo-documents

in

our

history

data

set:

H

QN

Each

Qi

corresponds

to

a

unique

query

and

is

enriched

with

clickthrough

information

as

discussed

in

Section

To

find

q"s

related

queries

in

H,

a

natural

way

is

to

use

a

text

retrieval

algorithm.

Here

we

use

the

OKAPI

method

one

of

the

state-of-the-art

retrieval

methods.

Specifically,

we

use

the

following

formula

to

calculate

the

similarity

between

query

q

and

pseudo-document

Qi:

w∈q

Qi

c(w,

q)

IDF(w)

c(w,

Qi)

b)

b

|Qi|

avdl

c(w,

Qi)

where

and

b

are

OKAPI

parameters

set

empirically,

c(w,

Qi)

and

c(w,

q)

are

the

count

of

word

w

in

Qi

and

q

respectively,

IDF(w)

is

the

inverse

document

frequency

of

word

w,

and

avdl

is

the

average

document

length

in

our

history

collection.

Based

on

the

similarity

scores,

we

rank

all

the

documents

in

H.

The

top

ranked

documents

provide

us

a

working

set

to

learn

the

aspects

that

users

are

usually

interested

in.

Each

document

in

H

corresponds

to

a

past

query,

and

thus

the

top

ranked

documents

correspond

to

q"s

related

past

queries.

Learning

Aspects

by

Clustering

Given

a

query

q,

we

use

Hq

dn}

to

represent

the

top

ranked

pseudo-documents

from

the

history

collection

H.

These

pseudo-documents

contain

the

aspects

that

users

are

interested

in.

In

this

subsection,

we

propose

to

use

a

clustering

method

to

discover

these

aspects.

Any

clustering

algorithm

could

be

applied

here.

In

this

paper,

we

use

an

algorithm

based

on

graph

partition:

the

star

clustering

algorithm

A

good

property

of

the

star

clustering

in

our

setting

is

that

it

can

suggest

a

good

label

for

each

cluster

naturally.

We

describe

the

star

clustering

algorithm

below.

Star

Clustering

Given

Hq,

star

clustering

starts

with

constructing

a

pairwise

similarity

graph

on

this

collection

based

on

the

vector

space

model

in

information

retrieval

Then

the

clusters

are

formed

by

dense

subgraphs

that

are

star-shaped.

These

clusters

form

a

cover

of

the

similarity

graph.

Formally,

for

each

of

the

n

pseudo-documents

dn}

in

the

collection

Hq,

we

compute

a

TF-IDF

vector.

Then,

for

each

pair

of

documents

di

and

dj

(i

j),

their

similarity

is

computed

as

the

cosine

score

of

their

corresponding

vectors

vi

and

vj

that

is

sim(di,

dj

cos(vi,

vj)

vi

vj

|vi|

|vj

A

similarity

graph

Gσ

can

then

be

constructed

as

follows

using

a

similarity

threshold

parameter

σ.

Each

document

di

is

a

vertex

of

Gσ.

If

sim(di,

dj)

σ,

there

would

be

an

edge

connecting

the

corresponding

two

vertices.

After

the

similarity

graph

Gσ

is

built,

the

star

clustering

algorithm

clusters

the

documents

using

a

greedy

algorithm

as

follows:

Associate

every

vertex

in

Gσ

with

a

flag,

initialized

as

unmarked.

From

those

unmarked

vertices,

find

the

one

which

has

the

highest

degree

and

let

it

be

u.

Mark

the

flag

of

u

as

center.

Form

a

cluster

C

containing

u

and

all

its

neighbors

that

are

not

marked

as

center.

Mark

all

the

selected

neighbors

as

satellites.

Repeat

from

step

until

all

the

vertices

in

Gσ

are

marked.

Each

cluster

is

star-shaped,

which

consists

a

single

center

and

several

satellites.

There

is

only

one

parameter

σ

in

the

star

clustering

algorithm.

A

big

σ

enforces

that

the

connected

documents

have

high

similarities,

and

thus

the

clusters

tend

to

be

small.

On

the

other

hand,

a

small

σ

will

make

the

clusters

big

and

less

coherent.

We

will

study

the

impact

of

this

parameter

in

our

experiments.

A

good

feature

of

the

star

clustering

algorithm

is

that

it

outputs

a

center

for

each

cluster.

In

the

past

query

collection

Hq,

each

document

corresponds

to

a

query.

This

center

query

can

be

regarded

as

the

most

representative

one

for

the

whole

cluster,

and

thus

provides

a

label

for

the

cluster

naturally.

All

the

clusters

obtained

are

related

to

the

input

query

q

from

different

perspectives,

and

they

represent

the

possible

aspects

of

interests

about

query

q

of

users.

Categorizing

Search

Results

In

order

to

organize

the

search

results

according

to

users"

interests,

we

use

the

learned

aspects

from

the

related

past

queries

to

categorize

the

search

results.

Given

the

top

m

Web

pages

returned

by

a

search

engine

for

q:

sm},

we

group

them

into

different

aspects

using

a

categorization

algorithm.

In

principle,

any

categorization

algorithm

can

be

used

here.

Here

we

use

a

simple

centroid-based

method

for

categorization.

Naturally,

more

sophisticated

methods

such

as

SVM

may

be

expected

to

achieve

even

better

performance.

Based

on

the

pseudo-documents

in

each

discovered

aspect

Ci,

we

build

a

centroid

prototype

pi

by

taking

the

average

of

all

the

vectors

of

the

documents

in

Ci:

pi

|Ci|

l∈Ci

vl.

All

these

pi"s

are

used

to

categorize

the

search

results.

Specifically,

for

any

search

result

sj,

we

build

a

TF-IDF

vector.

The

centroid-based

method

computes

the

cosine

similarity

between

the

vector

representation

of

sj

and

each

centroid

prototype

pi.

We

then

assign

sj

to

the

aspect

with

which

it

has

the

highest

cosine

similarity

score.

All

the

aspects

are

finally

ranked

according

to

the

number

of

search

results

they

have.

Within

each

aspect,

the

search

results

are

ranked

according

to

their

original

search

engine

ranking.

DATA

COLLECTION

We

construct

our

data

set

based

on

the

MSN

search

log

data

set

released

by

the

Microsoft

Live

Labs

in

In

total,

this

log

data

spans

days

from

to

There

are

queries,

distinct

queries,

and

distinct

URLs

in

the

raw

data.

To

test

our

algorithm,

we

separate

the

whole

data

set

into

two

parts

according

to

the

time:

the

first

data

is

used

to

simulate

the

historical

data

that

a

search

engine

accumulated,

and

we

use

the

last

to

simulate

future

queries.

In

the

history

collection,

we

clean

the

data

by

only

keeping

those

frequent,

well-formatted,

English

queries

(queries

which

only

contain

characters

‘a",

‘b",

‘z",

and

space,

and

appear

more

than

times).

After

cleaning,

we

get

unique

queries

in

our

history

data

collection

totally.

On

average,

each

query

has

distinct

clicks.

We

build

the

pseudo-documents

for

all

these

queries

as

described

in

Section

The

average

length

of

these

pseudo-documents

is

words

and

the

total

data

size

of

our

history

collection

is

We

construct

our

test

data

from

the

last

data.

According

to

the

time,

we

separate

this

data

into

two

test

sets

equally

for

cross-validation

to

set

parameters.

For

each

test

set,

we

use

every

session

as

a

test

case.

Each

session

contains

a

single

query

and

several

clicks.

(Note

that

we

do

not

aggregate

sessions

for

test

cases.

Different

test

cases

may

have

the

same

queries

but

possibly

different

clicks.)

Since

it

is

infeasible

to

ask

the

original

user

who

submitted

a

query

to

judge

the

results

for

the

query,

we

follow

the

work

and

opt

to

use

the

clicks

associated

with

the

query

in

a

session

to

approximate

relevant

documents.

Using

clicks

as

judgments,

we

can

then

compare

different

algorithms

for

organizing

search

results

to

see

how

well

these

algorithms

can

help

users

reach

the

clicked

URLs.

Organizing

search

results

into

different

aspects

is

expected

to

help

informational

queries.

It

thus

makes

sense

to

focus

on

the

informational

queries

in

our

evaluation.

For

each

test

case,

i.e.,

each

session,

we

count

the

number

of

different

clicks

and

filter

out

those

test

cases

with

fewer

than

clicks

under

the

assumption

that

a

query

with

more

clicks

is

more

likely

to

be

an

informational

query.

Since

we

want

to

test

whether

our

algorithm

can

learn

from

the

past

queries,

we

also

filter

out

those

test

cases

whose

queries

can

not

retrieve

at

least

pseudo-documents

from

our

history

collection.

Finally,

we

obtain

and

test

cases

in

the

first

and

second

test

sets

respectively.

On

average,

we

have

and

clicks

for

each

test

case

in

the

two

test

sets

respectively.

EXPERIMENTS

In

the

section,

we

describe

our

experiments

on

the

search

result

organization

based

past

search

engine

logs.

Experimental

Design

We

use

two

baseline

methods

to

evaluate

the

proposed

method

for

organizing

search

results.

For

each

test

case,

the

first

method

is

the

default

ranked

list

from

a

search

engine

(baseline).

The

second

method

is

to

organize

the

search

results

by

clustering

them

(cluster-based).

For

fair

comparison,

we

use

the

same

clustering

algorithm

as

our

logbased

method

(i.e.,

star

clustering).

That

is,

we

treat

each

search

result

as

a

document,

construct

the

similarity

graph,

and

find

the

star-shaped

clusters.

We

compare

our

method

(log-based)

with

the

two

baseline

methods

in

the

following

experiments.

For

both

cluster-based

and

log-based

methods,

the

search

results

within

each

cluster

is

ranked

based

on

their

original

ranking

given

by

the

search

engine.

To

compare

different

result

organization

methods,

we

adopt

a

similar

method

as

in

the

paper

That

is,

we

compare

the

quality

(e.g.,

precision)

of

the

best

cluster,

which

is

defined

as

the

one

with

the

largest

number

of

relevant

documents.

Organizing

search

results

into

clusters

is

to

help

users

navigate

into

relevant

documents

quickly.

The

above

metric

is

to

simulate

a

scenario

when

users

always

choose

the

right

cluster

and

look

into

it.

Specifically,

we

download

and

organize

the

top

search

results

into

aspects

for

each

test

case.

We

use

Precision

at

documents

in

the

best

cluster

as

the

primary

measure

to

compare

different

methods.

is

a

very

meaningful

measure

as

it

tells

us

the

perceived

precision

when

the

user

opens

a

cluster

and

looks

at

the

first

documents.

We

also

use

Mean

Reciprocal

Rank

(MRR)

as

another

metric.

MRR

is

calculated

as

MRR

|T|

q∈T

rq

where

T

is

a

set

of

test

queries,

rq

is

the

rank

of

the

first

relevant

document

for

q.

To

give

a

fair

comparison

across

different

organization

algorithms,

we

force

both

cluster-based

and

log-based

methods

to

output

the

same

number

of

aspects

and

force

each

search

result

to

be

in

one

and

only

one

aspect.

The

number

of

aspects

is

fixed

at

in

all

the

following

experiments.

The

star

clustering

algorithm

can

output

different

number

of

clusters

for

different

input.

To

constrain

the

number

of

clusters

to

we

order

all

the

clusters

by

their

sizes,

select

the

top

as

aspect

candidates.

We

then

re-assign

each

search

result

to

one

of

these

selected

aspects

that

has

the

highest

similarity

score

with

the

corresponding

aspect

centroid.

In

our

experiments,

we

observe

that

the

sizes

of

the

best

clusters

are

all

larger

than

and

this

ensures

that

is

a

meaningful

metric.

Experimental

Results

Our

main

hypothesis

is

that

organizing

search

results

based

on

the

users"

interests

learned

from

a

search

log

data

set

is

more

beneficial

than

to

organize

results

using

a

simple

list

or

cluster

search

results.

In

the

following,

we

test

our

hypothesis

from

two

perspectives

organization

and

labeling.

Method

Test

set

Test

set

MMR

MMR

Baseline

Cluster-based

Log-based

Cluster/Baseline

Log/Baseline

Log/Cluster

Table

Comparison

of

different

methods

by

MMR

and

We

also

show

the

percentage

of

relative

improvement

in

the

lower

part.

Comparison

Test

set

Test

set

Impr./Decr.

Impr./Decr.

Cluster/Baseline

Log/Baseline

Log/Cluster

Table

Pairwise

comparison

w.r.t

the

number

of

test

cases

whose

are

improved

versus

decreased

w.r.t

the

baseline.

Overall

performance

We

compare

three

methods,

basic

search

engine

ranking

(baseline),

traditional

clustering

based

method

(clusterbased),

and

our

log

based

method

(log-based),

in

Table

using

MRR

and

We

optimize

the

parameter

σ"s

for

each

collection

individually

based

on

values.

This

shows

the

best

performance

that

each

method

can

achieve.

In

this

table,

we

can

see

that

in

both

test

collections,

our

method

is

better

than

both

the

baseline

and

the

cluster-based

methods.

For

example,

in

the

first

test

collection,

the

baseline

method

of

MMR

is

the

cluster-based

method

is

and

our

method

is

We

achieve

higher

accuracy

than

both

cluster-based

method

improvement)

and

the

baseline

method

improvement).

The

values

are

for

the

baseline,

for

cluster-based

method,

but

for

our

method.

Our

method

improves

over

the

baseline

by

while

the

cluster-based

method

even

decreases

the

accuracy.

This

is

because

cluster-based

method

organizes

the

search

results

only

based

on

the

contents.

Thus

it

could

organize

the

results

differently

from

users"

preferences.

This

confirms

our

hypothesis

of

the

bias

of

the

cluster-based

method.

Comparing

our

method

with

the

cluster-based

method,

we

achieve

significant

improvement

on

both

test

collections.

The

p-values

of

the

significance

tests

based

on

on

both

collections

are

and

respectively.

This

shows

that

our

log-based

method

is

effective

to

learn

users"

preferences

from

the

past

query

history,

and

thus

it

can

organize

the

search

results

in

a

more

useful

way

to

users.

We

showed

the

optimal

results

above.

To

test

the

sensitivity

of

the

parameter

σ

of

our

log-based

method,

we

use

one

of

the

test

sets

to

tune

the

parameter

to

be

optimal

and

then

use

the

tuned

parameter

on

the

other

set.

We

compare

this

result

(log

tuned

outside)

with

the

optimal

results

of

both

cluster-based

(cluster

optimized)

and

log-based

methods

(log

optimized)

in

Figure

We

can

see

that,

as

expected,

the

performance

using

the

parameter

tuned

on

a

separate

set

is

worse

than

the

optimal

performance.

However,

our

method

still

performs

much

better

than

the

optimal

results

of

cluster-based

method

on

both

test

collections.

Test

set

Test

set

cluster

optimized

log

optimized

log

tuned

outside

Figure

Results

using

parameters

tuned

from

the

other

test

collection.

We

compare

it

with

the

optimal

performance

of

the

cluster-based

and

our

logbased

methods.

Bin

number

#Queries

Improved

Decreased

Figure

The

correlation

between

performance

change

and

result

diversity.

In

Table

we

show

pairwise

comparisons

of

the

three

methods

in

terms

of

the

numbers

of

test

cases

for

which

is

increased

versus

decreased.

We

can

see

that

our

method

improves

more

test

cases

compared

with

the

other

two

methods.

In

the

next

section,

we

show

more

detailed

analysis

to

see

what

types

of

test

cases

can

be

improved

by

our

method.

Detailed

Analysis

To

better

understand

the

cases

where

our

log-based

method

can

improve

the

accuracy,

we

test

two

properties:

result

diversity

and

query

difficulty.

All

the

analysis

below

is

based

on

test

set

Diversity

Analysis:

Intuitively,

organizing

search

results

into

different

aspects

is

more

beneficial

to

those

queries

whose

results

are

more

diverse,

as

for

such

queries,

the

results

tend

to

form

two

or

more

big

clusters.

In

order

to

test

the

hypothesis

that

log-based

method

help

more

those

queries

with

diverse

results,

we

compute

the

size

ratios

of

the

biggest

and

second

biggest

clusters

in

our

log-based

results

and

use

this

ratio

as

an

indicator

of

diversity.

If

the

ratio

is

small,

it

means

that

the

first

two

clusters

have

a

small

difference

thus

the

results

are

more

diverse.

In

this

case,

we

would

expect

our

method

to

help

more.

The

results

are

shown

in

Figure

In

this

figure,

we

partition

the

ratios

into

bins.

The

bins

correspond

to

the

ratio

ranges

and

respectively.

([i,

j)

means

that

i

ratio

j.)

In

each

bin,

we

count

the

numbers

of

test

cases

whose

are

improved

versus

decreased

with

respect

to

the

ranking

baseline,

and

plot

the

numbers

in

this

figure.

We

can

observe

that

when

the

ratio

is

smaller,

the

log-based

method

can

improve

more

test

cases.

But

when

Bin

number

#Queries

Improved

Decreased

Figure

The

correlation

between

performance

change

and

query

difficulty.

the

ratio

is

large,

the

log-based

method

can

not

improve

over

the

baseline.

For

example,

in

bin

test

cases

are

improved

and

are

decreased.

But

in

bin

all

the

test

cases

are

decreased.

This

confirms

our

hypothesis

that

our

method

can

help

more

if

the

query

has

more

diverse

results.

This

also

suggests

that

we

should

turn

off

the

option

of

re-organizing

search

results

if

the

results

are

not

very

diverse

(e.g.,

as

indicated

by

the

cluster

size

ratio).

Difficulty

Analysis:

Difficult

queries

have

been

studied

in

recent

years

Here

we

analyze

the

effectiveness

of

our

method

in

helping

difficult

queries.

We

quantify

the

query

difficulty

by

the

Mean

Average

Precision

(MAP)

of

the

original

search

engine

ranking

for

each

test

case.

We

then

order

the

test

cases

in

test

set

in

an

increasing

order

of

MAP

values.

We

partition

the

test

cases

into

bins

with

each

having

a

roughly

equal

number

of

test

cases.

A

small

MAP

means

that

the

utility

of

the

original

ranking

is

low.

Bin

contains

those

test

cases

with

the

lowest

MAP"s

and

bin

contains

those

test

cases

with

the

highest

MAP"s.

For

each

bin,

we

compute

the

numbers

of

test

cases

whose

are

improved

versus

decreased.

Figure

shows

the

results.

Clearly,

in

bin

most

of

the

test

cases

are

improved

vs

while

in

bin

log-based

method

may

decrease

the

performance

vs

This

shows

that

our

method

is

more

beneficial

to

difficult

queries,

which

is

as

expected

since

clustering

search

results

is

intended

to

help

difficult

queries.

This

also

shows

that

our

method

does

not

really

help

easy

queries,

thus

we

should

turn

off

our

organization

option

for

easy

queries.

Parameter

Setting

We

examine

parameter

sensitivity

in

this

section.

For

the

star

clustering

algorithm,

we

study

the

similarity

threshold

parameter

σ.

For

the

OKAPI

retrieval

function,

we

study

the

parameters

and

b.

We

also

study

the

impact

of

the

number

of

past

queries

retrieved

in

our

log-based

method.

Figure

shows

the

impact

of

the

parameter

σ

for

both

cluster-based

and

log-based

methods

on

both

test

sets.

We

vary

σ

from

to

with

step

Figure

shows

that

the

performance

is

not

very

sensitive

to

the

parameter

σ.

We

can

always

obtain

the

best

result

in

range

σ

In

Table

we

show

the

impact

of

OKAPI

parameters.

We

vary

from

to

with

step

and

b

from

to

with

step

From

this

table,

it

is

clear

that

is

also

not

very

sensitive

to

the

parameter

setting.

Most

of

the

values

are

larger

than

The

default

values

and

b

give

approximately

optimal

results.

We

further

study

the

impact

of

the

amount

of

history

similarity

threhold:

sigma

cluster-based

log-based

cluster-based

log-based

Figure

The

impact

of

similarity

threshold

σ

on

both

cluster-based

and

log-based

methods.

We

show

the

result

on

both

test

collections.

b

Table

Impact

of

OKAPI

parameters

and

b.

information

to

learn

from

by

varying

the

number

of

past

queries

to

be

retrieved

for

learning

aspects.

The

results

on

both

test

collections

are

shown

in

Figure

We

can

see

that

the

performance

gradually

increases

as

we

enlarge

the

number

of

past

queries

retrieved.

Thus

our

method

could

potentially

learn

more

as

we

accumulate

more

history.

More

importantly,

as

time

goes,

more

and

more

queries

will

have

sufficient

history,

so

we

can

improve

more

and

more

queries.

An

Illustrative

Example

We

use

the

query

area

codes

to

show

the

difference

in

the

results

of

the

log-based

method

and

the

cluster-based

method.

This

query

may

mean

phone

codes

or

zip

codes.

Table

shows

the

representative

keywords

extracted

from

the

three

biggest

clusters

of

both

methods.

In

the

clusterbased

method,

the

results

are

partitioned

based

on

locations:

local

or

international.

In

the

log-based

method,

the

results

are

disambiguated

into

two

senses:

phone

codes

or

zip

codes.

While

both

are

reasonable

partitions,

our

evaluation

indicates

that

most

users

using

such

a

query

are

often

interested

in

either

phone

codes

or

zip

codes.

since

the

values

of

cluster-based

and

log-based

methods

are

and

respectively.

Therefore

our

log-based

method

is

more

effective

in

helping

users

to

navigate

into

their

desired

results.

Cluster-based

method

Log-based

method

city,

state

telephone,

city,

international

local,

area

phone,

dialing

international

zip,

postal

Table

An

example

showing

the

difference

between

the

cluster-based

method

and

our

log-based

method

#queries

retrieved

Test

set

Test

set

Figure

The

impact

of

the

number

of

past

queries

retrieved.

Labeling

Comparison

We

now

compare

the

labels

between

the

cluster-based

method

and

log-based

method.

The

cluster-based

method

has

to

rely

on

the

keywords

extracted

from

the

snippets

to

construct

the

label

for

each

cluster.

Our

log-based

method

can

avoid

this

difficulty

by

taking

advantage

of

queries.

Specifically,

for

the

cluster-based

method,

we

count

the

frequency

of

a

keyword

appearing

in

a

cluster

and

use

the

most

frequent

keywords

as

the

cluster

label.

For

log-based

method,

we

use

the

center

of

each

star

cluster

as

the

label

for

the

corresponding

cluster.

In

general,

it

is

not

easy

to

quantify

the

readability

of

a

cluster

label

automatically.

We

use

examples

to

show

the

difference

between

the

cluster-based

and

the

log-based

methods.

In

Table

we

list

the

labels

of

the

top

clusters

for

two

examples

jaguar

and

apple.

For

the

cluster-based

method,

we

separate

keywords

by

commas

since

they

do

not

form

a

phrase.

From

this

table,

we

can

see

that

our

log-based

method

gives

more

readable

labels

because

it

generates

labels

based

on

users"

queries.

This

is

another

advantage

of

our

way

of

organizing

search

results

over

the

clustering

approach.

Label

comparison

for

query

jaguar

Log-based

method

Cluster-based

method

jaguar

animal

jaguar,

auto,

accessories

jaguar

auto

accessories

jaguar,

type,

prices

jaguar

cats

jaguar,

panthera,

cats

jaguar

repair

jaguar,

services,

boston

jaguar

animal

pictures

jaguar,

collection,

apparel

Label

comparison

for

query

apple

Log-based

method

Cluster-based

method

apple

computer

apple,

support,

product

apple

ipod

apple,

site,

computer

apple

crisp

recipe

apple,

world,

visit

fresh

apple

cake

apple,

ipod,

amazon

apple

laptop

apple,

products,

news

Table

Cluster

label

comparison.

CONCLUSIONS

AND

FUTURE

WORK

In

this

paper,

we

studied

the

problem

of

organizing

search

results

in

a

user-oriented

manner.

To

attain

this

goal,

we

rely

on

search

engine

logs

to

learn

interesting

aspects

from

users"

perspective.

Given

a

query,

we

retrieve

its

related

queries

from

past

query

history,

learn

the

aspects

by

clustering

the

past

queries

and

the

associated

clickthrough

information,

and

categorize

the

search

results

into

the

aspects

learned.

We

compared

our

log-based

method

with

the

traditional

cluster-based

method

and

the

baseline

of

search

engine

ranking.

The

experiments

show

that

our

log-based

method

can

consistently

outperform

cluster-based

method

and

improve

over

the

ranking

baseline,

especially

when

the

queries

are

difficult

or

the

search

results

are

diverse.

Furthermore,

our

log-based

method

can

generate

more

meaningful

aspect

labels

than

the

cluster

labels

generated

based

on

search

results

when

we

cluster

search

results.

There

are

several

interesting

directions

for

further

extending

our

work:

First,

although

our

experiment

results

have

clearly

shown

promise

of

the

idea

of

learning

from

search

logs

to

organize

search

results,

the

methods

we

have

experimented

with

are

relatively

simple.

It

would

be

interesting

to

explore

other

potentially

more

effective

methods.

In

particular,

we

hope

to

develop

probabilistic

models

for

learning

aspects

and

organizing

results

simultaneously.

Second,

with

the

proposed

way

of

organizing

search

results,

we

can

expect

to

obtain

informative

feedback

information

from

a

user

(e.g.,

the

aspect

chosen

by

a

user

to

view).

It

would

thus

be

interesting

to

study

how

to

further

improve

the

organization

of

the

results

based

on

such

feedback

information.

Finally,

we

can

combine

a

general

search

log

with

any

personal

search

log

to

customize

and

optimize

the

organization

of

search

results

for

each

individual

user.

ACKNOWLEDGMENTS

We

thank

the

anonymous

reviewers

for

their

valuable

comments.

This

work

is

in

part

supported

by

a

Microsoft

Live

Labs

Research

Grant,

a

Google

Research

Grant,

and

an

NSF

CAREER

grant

REFERENCES

E.

Agichtein,

E.

Brill,

and

S.

T.

Dumais.

Improving

web

search

ranking

by

incorporating

user

behavior

information.

In

SIGIR,

pages

J.

A.

Aslam,

E.

Pelekov,

and

D.

Rus.

The

star

clustering

algorithm

for

static

and

dynamic

information

organization.

Journal

of

Graph

Algorithms

and

Applications,

R.

A.

Baeza-Yates.

Applications

of

web

query

mining.

In

ECIR,

pages

D.

Beeferman

and

A.

L.

Berger.

Agglomerative

clustering

of

a

search

engine

query

log.

In

KDD,

pages

D.

Carmel,

E.

Yom-Tov,

A.

Darlow,

and

D.

Pelleg.

What

makes

a

query

difficult?

In

SIGIR,

pages

H.

Chen

and

S.

T.

Dumais.

Bringing

order

to

the

web:

automatically

categorizing

search

results.

In

CHI,

pages

S.

Cronen-Townsend,

Y.

Zhou,

and

W.

B.

Croft.

Predicting

query

performance.

In

Proceedings

of

ACM

SIGIR

pages

S.

T.

Dumais,

E.

Cutrell,

and

H.

Chen.

Optimizing

search

by

showing

results

in

context.

In

CHI,

pages

M.

A.

Hearst

and

J.

O.

Pedersen.

Reexamining

the

cluster

hypothesis:

Scatter/gather

on

retrieval

results.

In

SIGIR,

pages

T.

Joachims.

Optimizing

search

engines

using

clickthrough

data.

In

KDD,

pages

T.

Joachims.

Evaluating

Retrieval

Performance

Using

Clickthrough

Data.,

pages

Physica/Springer

Verlag,

in

J.

Franke

and

G.

Nakhaeizadeh

and

I.

Renz,

Text

Mining.

R.

Jones,

B.

Rey,

O.

Madani,

and

W.

Greiner.

Generating

query

substitutions.

In

WWW,

pages

K.

Kummamuru,

R.

Lotlikar,

S.

Roy,

K.

Singal,

and

R.

Krishnapuram.

A

hierarchical

monothetic

document

clustering

algorithm

for

summarization

and

browsing

search

results.

In

WWW,

pages

Microsoft

Live

Labs.

Accelerating

search

in

academic

research,

Search

RFP.aspx.

P.

Pirolli,

P.

K.

Schank,

M.

A.

Hearst,

and

C.

Diehl.

Scatter/gather

browsing

communicates

the

topic

structure

of

a

very

large

text

collection.

In

CHI,

pages

F.

Radlinski

and

T.

Joachims.

Query

chains:

learning

to

rank

from

implicit

feedback.

In

KDD,

pages

S.

E.

Robertson

and

S.

Walker.

Some

simple

effective

approximations

to

the

model

for

probabilistic

weighted

retrieval.

In

SIGIR,

pages

G.

Salton,

A.

Wong,

and

C.

S.

Yang.

A

vector

space

model

for

automatic

indexing.

Commun.

ACM,

X.

Shen,

B.

Tan,

and

C.

Zhai.

Context-sensitive

information

retrieval

using

implicit

feedback.

In

SIGIR,

pages

C.

J.

van

Rijsbergen.

Information

Retrieval,

second

edition.

Butterworths,

London,

V.

N.

Vapnik.

The

Nature

of

Statistical

Learning

Theory.

Springer-Verlag,

Berlin,

Vivisimo.

X.

Wang,

J.-T.

Sun,

Z.

Chen,

and

C.

Zhai.

Latent

semantic

analysis

for

multiple-type

interrelated

data

objects.

In

SIGIR,

pages

J.-R.

Wen,

J.-Y.

Nie,

and

H.

Zhang.

Clustering

user

queries

of

a

search

engine.

In

WWW,

pages

E.

Yom-Tov,

S.

Fine,

D.

Carmel,

and

A.

Darlow.

Learning

to

estimate

query

difficulty:

including

applications

to

missing

content

detection

and

distributed

information

retrieval.

In

SIGIR,

pages

O.

Zamir

and

O.

Etzioni.

Web

document

clustering:

A

feasibility

demonstration.

In

SIGIR,

pages

O.

Zamir

and

O.

Etzioni.

Grouper:

A

dynamic

clustering

interface

to

web

search

results.

Computer

Networks,

H.-J.

Zeng,

Q.-C.

He,

Z.

Chen,

W.-Y.

Ma,

and

J.

Ma.

Learning

to

cluster

web

search

results.

In

SIGIR,

pages

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

meaningful cluster label
search result snippet
pseudo-document
centroid-based method
log-based method
interest aspect

retrieval model
ranking function
ambiguity
clustering view
history collection
past query
clickthrough
star clustering algorithm
suffix tree clustering algorithm
monothetic clustering algorithm
pairwise similarity graph
similarity threshold parameter
cosine similarity
centroid prototype
reciprocal rank
mean average precision
search result organization
search engine log