New

Event

Detection

Based

on

Indexing-tree

and

Named

Entity

Zhang

Kuo

Tsinghua

University

Beijing,

China

Li

Juan

Zi

Tsinghua

University

Beijing,

China

ljz@keg.cs.tsinghua.edu.cn

Wu

Gang

Tsinghua

University

Beijing,

China

ABSTRACT

New

Event

Detection

(NED)

aims

at

detecting

from

one

or

multiple

streams

of

news

stories

that

which

one

is

reported

on

a

new

event

(i.e.

not

reported

previously).

With

the

overwhelming

volume

of

news

available

today,

there

is

an

increasing

need

for

a

NED

system

which

is

able

to

detect

new

events

more

efficiently

and

accurately.

In

this

paper

we

propose

a

new

NED

model

to

speed

up

the

NED

task

by

using

news

indexing-tree

dynamically.

Moreover,

based

on

the

observation

that

terms

of

different

types

have

different

effects

for

NED

task,

two

term

reweighting

approaches

are

proposed

to

improve

NED

accuracy.

In

the

first

approach,

we

propose

to

adjust

term

weights

dynamically

based

on

previous

story

clusters

and

in

the

second

approach,

we

propose

to

employ

statistics

on

training

data

to

learn

the

named

entity

reweighting

model

for

each

class

of

stories.

Experimental

results

on

two

Linguistic

Data

Consortium

(LDC)

datasets

and

show

that

the

proposed

model

can

improve

both

efficiency

and

accuracy

of

NED

task

significantly,

compared

to

the

baseline

system

and

other

existing

systems.

Categories

and

Subject

Descriptors

[Information

Systems]:

Information

Search

and

Retrieval;

[Information

Systems

Applications]:

Types

of

Systemsdecision

support.

General

Terms

Algorithms,

Performance,

Experimentation

INTRODUCTION

Topic

Detection

and

Tracking

(TDT)

program

aims

to

develop

techniques

which

can

effectively

organize,

search

and

structure

news

text

materials

from

a

variety

of

newswire

and

broadcast

media

New

Event

Detection

(NED)

is

one

of

the

five

tasks

in

TDT.

It

is

the

task

of

online

identification

of

the

earliest

report

for

each

topic

as

soon

as

that

report

arrives

in

the

sequence

of

documents.

A

Topic

is

defined

as

a

seminal

event

or

activity,

along

with

directly

related

events

and

activities

An

Event

is

defined

as

something

(non-trivial)

happening

in

a

certain

place

at

a

certain

time

For

instance,

when

a

bomb

explodes

in

a

building,

the

exploding

is

the

seminal

event

that

triggers

the

topic,

and

other

stories

on

the

same

topic

would

be

those

discussing

salvaging

efforts,

the

search

for

perpetrators,

arrests

and

trial

and

so

on.

Useful

news

information

is

usually

buried

in

a

mass

of

data

generated

everyday.

Therefore,

NED

systems

are

very

useful

for

people

who

need

to

detect

novel

information

from

real-time

news

stream.

These

real-life

needs

often

occur

in

domains

like

financial

markets,

news

analysis,

and

intelligence

gathering.

In

most

of

state-of-the-art

(currently)

NED

systems,

each

news

story

on

hand

is

compared

to

all

the

previous

received

stories.

If

all

the

similarities

between

them

do

not

exceed

a

threshold,

then

the

story

triggers

a

new

event.

They

are

usually

in

the

form

of

cosine

similarity

or

Hellinger

similarity

metric.

The

core

problem

of

NED

is

to

identify

whether

two

stories

are

on

the

same

topic.

Obviously,

these

systems

cannot

take

advantage

of

topic

information.

Further

more,

it

is

not

acceptable

in

real

applications

because

of

the

large

amount

of

computation

required

in

the

NED

process.

Other

systems

organize

previous

stories

into

clusters

(each

cluster

corresponds

to

a

topic),

and

new

story

is

compared

to

the

previous

clusters

instead

of

stories.

This

manner

can

reduce

comparing

times

significantly.

Nevertheless,

it

has

been

proved

that

this

manner

is

less

accurate

This

is

because

sometimes

stories

within

a

topic

drift

far

away

from

each

other,

which

could

lead

low

similarity

between

a

story

and

its

topic.

On

the

other

hand,

some

proposed

NED

systems

tried

to

improve

accuracy

by

making

better

use

of

named

entities

However,

of

the

systems

have

considered

that

terms

of

different

types

(e.g.

Noun,

Verb

or

Person

name)

have

different

effects

for

different

classes

of

stories

in

determining

whether

two

stories

are

on

the

same

topic.

For

example,

the

names

of

election

candidates

(Person

name)

are

very

important

for

stories

of

election

class;

the

locations

(Location

name)

where

accidents

happened

are

important

for

stories

of

accidents

class.

So,

in

NED,

there

still

exist

following

three

problems

to

be

investigated:

How

to

speed

up

the

detection

procedure

while

do

not

decrease

the

detection

accuracy?

How

to

make

good

use

of

cluster

(topic)

information

to

improve

accuracy?

How

to

obtain

better

news

story

representation

by

better

understanding

of

named

entities.

Driven

by

these

problems,

we

have

proposed

three

approaches

in

this

paper.

make

the

detection

procedure

faster,

we

propose

a

new

NED

procedure

based

on

news

indexing-tree

created

dynamically.

Story

indexing-tree

is

created

by

assembling

similar

stories

together

to

form

news

clusters

in

different

hierarchies

according

to

their

values

of

similarity.

Comparisons

between

current

story

and

previous

clusters

could

help

find

the

most

similar

story

in

less

comparing

times.

The

new

procedure

can

reduce

the

amount

of

comparing

times

without

hurting

accuracy.

use

the

clusters

of

the

first

floor

in

the

indexing-tree

as

news

topics,

in

which

term

weights

are

adjusted

dynamically

according

to

term

distribution

in

the

clusters.

In

this

approach,

cluster

(topic)

information

is

used

properly,

so

the

problem

of

theme

decentralization

is

avoided.

on

observations

on

the

statistics

obtained

from

training

data,

we

found

that

terms

of

different

types

(e.g.

Noun

and

Verb)

have

different

effects

for

different

classes

of

stories

in

determining

whether

two

stories

are

on

the

same

topic.

And

we

propose

to

use

statistics

to

optimize

the

weights

of

the

terms

of

different

types

in

a

story

according

to

the

news

class

that

the

story

belongs

to.

On

dataset,

the

new

NED

model

just

uses

comparing

times

of

the

basic

model,

while

its

minimum

normalized

cost

is

which

is

better

than

the

basic

model,

and

also

better

than

any

other

results

previously

reported

for

this

dataset

The

rest

of

the

paper

is

organized

as

follows.

We

start

off

this

paper

by

summarizing

the

previous

work

in

NED

in

section

Section

presents

the

basic

model

for

NED

that

most

current

systems

use.

Section

describes

our

new

detection

procedure

based

on

news

indexing-tree.

In

section

two

term

reweighting

methods

are

proposed

to

improve

NED

accuracy.

Section

gives

our

experimental

data

and

evaluation

metrics.

We

finally

wrap

up

with

the

experimental

results

in

Section

and

the

conclusions

and

future

work

in

Section

RELATED

WORK

Papka

et

al.

proposed

Single-Pass

clustering

on

NED

When

a

new

story

was

encountered,

it

was

processed

immediately

to

extract

term

features

and

a

query

representation

of

the

story"s

content

is

built

up.

Then

it

was

compared

with

all

the

previous

queries.

If

the

document

did

not

trigger

any

queries

by

exceeding

a

threshold,

it

was

marked

as

a

new

event.

Lam

et

al

build

up

previous

query

representations

of

story

clusters,

each

of

which

corresponds

to

a

topic

In

this

manner

comparisons

happen

between

stories

and

clusters.

Recent

years,

most

work

focus

on

proposing

better

methods

on

comparison

of

stories

and

document

representation.

Brants

et

al.

extended

a

basic

incremental

TF-IDF

model

to

include

sourcespecific

models,

similarity

score

normalization

based

on

document-specific

averages,

similarity

score

normalization

based

on

source-pair

specific

averages,

term

reweighting

based

on

inverse

event

frequencies,

and

segmentation

of

documents.

Good

improvements

on

TDT

bench-marks

were

shown.

Stokes

et

al.

utilized

a

combination

of

evidence

from

two

distinct

representations

of

a

document"s

content.

One

of

the

representations

was

the

usual

free

text

vector,

the

other

made

use

of

lexical

chains

(created

using

WordNet)

to

build

another

term

vector.

Then

the

two

representations

are

combined

in

a

linear

fashion.

A

marginal

increase

in

effectiveness

was

achieved

when

the

combined

representation

was

used.

Some

efforts

have

been

done

on

how

to

utilize

named

entities

to

improve

NED.

Yang

et

al.

gave

location

named

entities

four

times

weight

than

other

terms

and

named

entities

DOREMI

research

group

combined

semantic

similarities

of

person

names,

location

names

and

time

together

with

textual

similarity

UMass

research

group

split

document

representation

into

two

parts:

named

entities

and

non-named

entities.

And

it

was

found

that

some

classes

of

news

could

achieve

better

performance

using

named

entity

representation,

while

some

other

classes

of

news

could

achieve

better

performance

using

non-named

entity

representation.

Both

and

used

text

categorization

technique

to

classify

news

stories

in

advance.

In

news

stories

are

classified

automatically

at

first,

and

then

test

sensitivities

of

names

and

non-name

terms

for

NED

for

each

class.

In

frequent

terms

for

each

class

are

removed

from

document

representation.

For

example,

word

election

does

not

help

identify

different

elections.

In

their

work,

effectiveness

of

different

kinds

of

names

(or

terms

with

different

POS)

for

NED

in

different

news

classes

are

not

investigated.

We

use

statistical

analysis

to

reveal

the

fact

and

use

it

to

improve

NED

performance.

BASIC

MODEL

In

this

section,

we

present

the

basic

New

Event

Detection

model

which

is

similar

to

what

most

current

systems

apply.

Then,

we

propose

our

new

model

by

extending

the

basic

model.

New

Event

Detection

systems

use

news

story

stream

as

input,

in

which

stories

are

strictly

time-ordered.

Only

previously

received

stories

are

available

when

dealing

with

current

story.

The

output

is

a

decision

for

whether

the

current

story

is

on

a

new

event

or

not

and

the

confidence

of

the

decision.

Usually,

a

NED

model

consists

of

three

parts:

story

representation,

similarity

calculation

and

detection

procedure.

Story

Representation

Preprocessing

is

needed

before

generating

story

representation.

For

preprocessing,

we

tokenize

words,

recognize

abbreviations,

normalize

abbreviations,

add

part-of-speech

tags,

remove

stopwords

included

in

the

stop

list

used

in

InQuery

replace

words

with

their

stems

using

K-stem

and

then

generate

word

vector

for

each

news

story.

We

use

incremental

TF-IDF

model

for

term

weight

calculation

In

a

TF-IDF

model,

term

frequency

in

a

news

document

is

weighted

by

the

inverse

document

frequency,

which

is

generated

from

training

corpus.

When

a

new

term

occurs

in

testing

process,

there

are

two

solutions:

simply

ignore

the

new

term

or

set

df

of

the

term

as

a

small

const

(e.g.

df

The

new

term

receives

too

low

weight

in

the

first

solution

and

too

high

weight

in

the

second

solution.

In

incremental

TF-IDF

model,

document

frequencies

are

updated

dynamically

in

each

time

step

t:

)t

t

D

tdf

w

df

w

df

w−=

where

Dt

represents

news

story

set

received

in

time

t,

and

dfDt(w)

means

the

number

of

documents

that

term

w

occurs

in,

and

dft(w)

means

the

total

number

of

documents

that

term

w

occurs

in

before

time

t.

In

this

work,

each

time

window

includes

news

stories.

Thus,

each

story

d

received

in

t

is

represented

as

follows:

)}nd

weight

d

t

w

weight

d

t

w

weight

d

t

w→

where

n

means

the

number

of

distinct

terms

in

story

d,

and

)weight

d

t

w

means

the

weight

of

term

w

in

story

d

at

time

t:

log(

log((

log(

log((

t

t

t

t

w

d

tf

d

w

N

df

w

weight

d

t

w

tf

d

w

N

df

w

where

Nt

means

the

total

number

of

news

stories

before

time

t,

and

tf(d,w)

means

how

many

times

term

w

occurs

in

news

story

d.

Similarity

Calculation

We

use

Hellinger

distance

for

the

calculation

of

similarity

between

two

stories,

for

two

stories

d

and

d"

at

time

t,

their

similarity

is

defined

as

follows:

w

d

d

sim

d

d

t

weight

d

t

w

weight

d

t

w

Detection

Procedure

For

each

story

d

received

in

time

step

t,

the

value

time

d

time

d

n

d

max

sim

d

d

t

is

a

score

used

to

determine

whether

d

is

a

story

about

a

new

topic

and

at

the

same

time

is

an

indication

of

the

confidence

in

our

decision

time(d)

means

the

publication

time

of

story

d.

If

the

score

exceeds

the

thresholdθ

new,

then

there

exists

a

sufficiently

similar

document,

thus

d

is

a

old

story,

otherwise,

there

is

no

sufficiently

similar

previous

document,

thus

d

is

an

new

story.

New

NED

Procedure

Traditional

NED

systems

can

be

classified

into

two

main

types

on

the

aspect

of

detection

procedure:

S-S

type,

in

which

the

story

on

hand

is

compared

to

each

story

received

previously,

and

use

the

highest

similarity

to

determine

whether

current

story

is

about

a

new

event;

S-C

type,

in

which

the

story

on

hand

is

compared

to

all

previous

clusters

each

of

which

representing

a

topic,

and

the

highest

similarity

is

used

for

final

decision

for

current

story.

If

the

highest

similarity

exceeds

thresholdθ

new,

then

it

is

an

old

story,

and

put

it

into

the

most

similar

cluster;

otherwise

it

is

a

new

story

and

create

a

new

cluster.

Previous

work

show

that

the

first

manner

is

more

accurate

than

the

second

one

Since

sometimes

stories

within

a

topic

drift

far

away

from

each

other,

a

story

may

have

very

low

similarity

with

its

topic.

So

using

similarities

between

stories

for

determining

new

story

is

better

than

using

similarities

between

story

and

clusters.

Nevertheless,

the

first

manner

needs

much

more

comparing

times

which

means

the

first

manner

is

low

efficient.

We

propose

a

new

detection

procedure

which

uses

comparisons

with

previous

clusters

to

help

find

the

most

similar

story

in

less

comparing

times,

and

the

final

new

event

decision

is

made

according

to

the

most

similar

story.

Therefore,

we

can

get

both

the

accuracy

of

S-S

type

methods

and

the

efficiency

of

S-C

type

methods.

The

new

procedure

creates

a

news

indexing-tree

dynamically,

in

which

similar

stories

are

put

together

to

form

a

hierarchy

of

clusters.

We

index

similar

stories

together

by

their

common

ancestor

(a

cluster

node).

Dissimilar

stories

are

indexed

in

different

clusters.

When

a

story

is

coming,

we

use

comparisons

between

the

current

story

and

previous

hierarchical

clusters

to

help

find

the

most

similar

story

which

is

useful

for

new

event

decision.

After

the

new

event

decision

is

made,

the

current

story

is

inserted

to

the

indexing-tree

for

the

following

detection.

The

news

indexing-tree

is

defined

formally

as

follows:

S-Tree

{r,

NC

NS

E}

where

r

is

the

root

of

S-Tree,

NC

is

the

set

of

all

cluster

nodes,

NS

is

the

set

of

all

story

nodes,

and

E

is

the

set

of

all

edges

in

S-Tree.

We

define

a

set

of

constraints

for

a

S-Tree:

ⅰ

is

an

non-terminal

node

in

the

treeC

i

i

N

i∀

ⅱ

is

a

terminal

node

in

the

treeS

i

i

N

i∀

ⅲ

out

degree

of

is

at

least

i

i

N

i∀

ⅳ

is

represented

as

the

centroid

of

its

desendantsC

i

i

iN∀

For

a

news

story

di,

the

comparison

procedure

and

inserting

procedure

based

on

indexing-tree

are

defined

as

follows.

An

example

is

shown

by

Figure

and

Figure

Figure

Comparison

procedure

Figure

Inserting

procedure

Comparison

procedure:

Step

compare

di

to

all

the

direct

child

nodes

of

r

and

select

λ

nodes

with

highest

similarities,

e.g.,

and

in

Figure

Step

for

each

selected

node

in

the

last

step,

e.g.

compare

di

to

all

its

direct

child

nodes,

and

select

λ

nodes

with

highest

similarities,

e.g.

and

Repeat

step

for

all

non-terminal

nodes.

Step

record

the

terminal

node

with

the

highest

similarty

to

di,

e.g.

and

the

similarity

value

Inserting

di

to

the

S-tree

with

r

as

root:

Find

the

node

n

which

is

direct

child

of

r

in

the

path

from

r

to

the

terminal

node

with

highest

similarity

s,

e.g.

If

s

is

smaller

than

θ

then

add

di

to

the

tree

as

a

direct

child

of

r.

Otherwise,

if

n

is

a

terminal

node,

then

create

a

cluster

node

instead

of

n,

and

add

both

n

and

di

as

its

direct

children;

if

n

is

an

non-terminal

node,

then

repeat

this

procedure

and

insert

di

to

the

sub-tree

with

n

as

root

recursively.

Here

h

is

the

length

between

n

and

the

root

of

S-tree.

The

more

the

stories

in

a

cluster

similar

to

each

other,

the

better

the

cluster

represents

the

stories

in

it.

Hence

we

add

no

constraints

on

the

maximum

of

tree"s

height

and

degree

of

a

node.

Therefore,

we

cannot

give

the

complexity

of

this

indexing-tree

based

procedure.

But

we

will

give

the

number

of

comparing

times

needed

by

the

new

procedure

in

our

experiments

in

Term

Reweighting

Methods

In

this

section,

two

term

reweighting

methods

are

proposed

to

improve

NED

accuracy.

In

the

first

method,

a

new

way

is

explored

for

better

using

of

cluster

(topic)

information.

The

second

one

finds

a

better

way

to

make

use

of

named

entities

based

on

news

classification.

Term

Reweighting

Based

on

Distribution

Distance

TF-IDF

is

the

most

prevalent

model

used

in

information

retrieval

systems.

The

basic

idea

is

that

the

fewer

documents

a

term

appears

in,

the

more

important

the

term

is

in

discrimination

of

documents

(relevant

or

not

relevant

to

a

query

containing

the

term).

Nevertheless,

in

TDT

domain,

we

need

to

discriminate

documents

with

regard

to

topics

rather

than

queries.

Intuitively,

using

cluster

(topic)

vectors

to

compare

with

subsequent

news

stories

should

outperform

using

story

vectors.

Unfortunately,

the

experimental

results

do

not

support

this

intuition

Based

on

observation

on

data,

we

find

the

reason

is

that

a

news

topic

usually

contains

many

directly

or

indirectly

related

events,

while

they

all

have

their

own

sub-subjects

which

are

usually

different

with

each

other.

Take

the

topic

described

in

section

as

an

example,

events

like

the

explosion

and

salvage

have

very

low

similarities

with

events

about

criminal

trial,

therefore

stories

about

trial

would

have

low

similarity

with

the

topic

vector

built

on

its

previous

events.

This

section

focuses

on

how

to

effectively

make

use

of

topic

information

and

at

the

same

time

avoid

the

problem

of

content

decentralization.

At

first,

we

classify

terms

into

classes

to

help

analysis

the

needs

of

the

modified

model:

Term

class

A:

terms

that

occur

frequently

in

the

whole

corpus,

e.g.,

year

and

people.

Terms

of

this

class

should

be

given

low

weights

because

they

do

not

help

much

for

topic

discrimination.

Term

class

B:

terms

that

occur

frequently

within

a

news

category,

e.g.,

election,

storm.

They

are

useful

to

distinguish

two

stories

in

different

news

categories.

However,

they

cannot

provide

information

to

determine

whether

two

stories

are

on

the

same

or

different

topics.

In

another

words,

term

election

and

term

storm

are

not

helpful

in

differentiate

two

election

campaigns

and

two

storm

disasters.

Therefore,

terms

of

this

class

should

be

assigned

lower

weights.

Term

class

C:

terms

that

occur

frequently

in

a

topic,

and

infrequently

in

other

topics,

e.g.,

the

name

of

a

crash

plane,

the

name

of

a

specific

hurricane.

News

stories

that

belong

to

different

topics

rarely

have

overlap

terms

in

this

class.

The

more

frequently

a

term

appears

in

a

topic,

the

more

important

the

term

is

for

a

story

belonging

to

the

topic,

therefore

the

term

should

be

set

higher

weight.

Term

class

D:

terms

that

appear

in

a

topic

exclusively,

but

not

frequently.

For

example,

the

name

of

a

fireman

who

did

very

well

in

a

salvage

action,

which

may

appears

in

only

two

or

three

stories

but

never

appeared

in

other

topics.

Terms

of

this

type

should

receive

more

weights

than

in

TF-IDF

model.

However,

since

they

are

not

popular

in

the

topic,

it

is

not

appropriate

to

give

them

too

high

weights.

Term

class

E:

terms

with

low

document

frequency,

and

appear

in

different

topics.

Terms

of

this

class

should

receive

lower

weights.

Now

we

analyze

whether

TF-IDF

model

can

give

proper

weights

to

the

five

classes

of

terms.

Obviously,

terms

of

class

A

are

lowly

weighted

in

TF-IDF

model,

which

is

conformable

with

the

requirement

described

above.

In

TF-IDF

model,

terms

of

class

B

are

highly

dependant

with

the

number

of

stories

in

a

news

class.

TF-IDF

model

cannot

provide

low

weights

if

the

story

containing

the

term

belongs

to

a

relative

small

news

class.

For

a

term

of

class

C,

the

more

frequently

it

appears

in

a

topic,

the

less

weight

TFIDF

model

gives

to

it.

This

strongly

conflicts

with

the

requirement

of

terms

in

class

C.

For

terms

of

class

D,

TF-IDF

model

gives

them

high

weights

correctly.

But

for

terms

of

class

E,

TF-IDF

model

gives

high

weights

to

them

which

are

not

conformable

with

the

requirement

of

low

weights.

To

sum

up,

terms

of

class

B,

C,

E

cannot

be

properly

weighted

in

TF-IDF

model.

So,

we

propose

a

modified

model

to

resolve

this

problem.

When

θ

init

andθ

new

are

set

closely,

we

assume

that

most

of

the

stories

in

a

first-level

cluster

(a

direct

child

node

of

root

node)

are

on

the

same

topic.

Therefore,

we

make

use

of

a

first-level

cluster

to

capture

term

distribution

(df

for

all

the

terms

within

the

cluster)

within

the

topic

dynamically.

KL

divergence

of

term

distribution

in

a

first-level

cluster

and

the

whole

story

set

is

used

to

adjust

term

weights:

cw

tw

cw

tw

w

d

D

weight

d

t

w

KL

P

P

weight

d

t

w

weight

d

t

w

KL

P

P

γ

γ

where

cw

c

c

c

c

df

w

df

w

p

y

p

y

N

N

t

tw

tw

t

t

df

w

df

w

p

y

p

y

N

N

where

dfc(w)

is

the

number

of

documents

containing

term

w

within

cluster

C,

and

Nc

is

the

number

of

documents

in

cluster

C,

and

Nt

is

the

total

number

of

documents

that

arrive

before

time

step

t.

γ

is

a

const

parameter,

now

is

manually

set

KL

divergence

is

defined

as

follows

log

)x

p

x

KL

P

Q

p

x

q

x

The

basic

idea

is:

for

a

story

in

a

topic,

the

more

a

term

occurs

within

the

topic,

and

the

less

it

occurs

in

other

topics,

it

should

be

assigned

higher

weights.

Obviously,

modified

model

can

meet

all

the

requirements

of

the

five

term

classes

listed

above.

Term

Reweighting

Based

on

Term

Type

and

Story

Class

Previous

work

found

that

some

classes

of

news

stories

could

achieve

good

improvements

by

giving

extra

weight

to

named

entities.

But

we

find

that

terms

of

different

types

should

be

given

different

amount

of

extra

weight

for

different

classes

of

news

stories.

We

use

to

recognize

named

entity

types

and

part-ofspeech

tags

for

terms

that

appear

in

news

stories.

Named

entity

types

include

person

name,

organization

name,

location

name,

date,

time,

money

and

percentage,

and

five

POSs

are

selected:

(NN),

verb

(VB),

adjective

(JJ),

adverb

(RB)

and

cardinal

number

(CD).

Statistical

analysis

shows

topic-level

discriminative

terms

types

for

different

classes

of

stories.

For

the

sake

of

convenience,

named

entity

type

and

part-of-speech

tags

are

uniformly

called

term

type

in

subsequent

sections.

Determining

whether

two

stories

are

about

the

same

topic

is

a

basic

component

for

NED

task.

So

at

first

we

use

χ

statistic

to

compute

correlations

between

terms

and

topics.

For

a

term

t

and

a

topic

T,

a

contingence

table

is

derived:

Table

A

Contingence

Table

Doc

Number

belong

to

topic

T

not

belong

to

topic

T

include

t

A

B

not

include

t

C

D

The

χ

statistic

for

a

specific

term

t

with

respect

to

topic

T

is

defined

to

be

w

T

A

B

C

D

AD

CB

A

C

B

D

A

B

C

D

χ

News

topics

for

the

TDT

task

are

further

classified

into

rules

of

interpretations

(ROIs)

The

ROI

can

be

seen

as

a

higher

level

class

of

stories.

The

average

correlation

between

a

term

type

and

a

topic

ROI

is

computed

as:

avg

)k

m

m

km

kT

R

w

P

w

TP

R

p

w

T

R

P

χ

χ

where

K

is

the

number

of

term

types

(set

constantly

in

the

paper).

M

is

the

number

news

classes

(ROIs,

set

in

the

paper).

Pk

represents

the

set

of

all

terms

of

type

k,

and

Rm

represents

the

set

of

all

topics

of

class

m,

p(t,T)

means

the

probability

that

t

occurs

in

topic

T.

Because

of

limitation

of

space,

only

parts

of

the

term

types

term

types)

and

parts

of

news

classes

classes)

are

listed

in

table

with

the

average

correlation

values

between

them.

The

statistics

is

derived

from

labeled

data

in

corpus.

(Results

in

table

are

already

normalized

for

convenience

in

comparison.)

The

statistics

in

table

indicates

the

usefulness

of

different

term

types

in

topic

discrimination

with

respect

to

different

news

classes.

We

can

see

that,

location

name

is

the

most

useful

term

type

for

three

news

classes:

Natural

Disasters,

Violence

or

War,

Finances.

And

for

three

other

categories

Elections,

Legal/Criminal

Cases,

Science

and

Discovery,

person

name

is

the

most

discriminative

term

type.

For

Scandals/Hearings,

date

is

the

most

important

information

for

topic

discrimination.

In

addition,

Legal/Criminal

Cases

and

Finance

topics

have

higher

correlation

with

money

terms,

while

Science

and

Discovery

have

higher

correlation

with

percentage

terms.

Non-name

terms

are

more

stable

for

different

classes.

From

the

analysis

of

table

it

is

reasonable

to

adjust

term

weight

according

to

their

term

type

and

the

news

class

the

story

belongs

to.

New

term

weights

are

reweighted

as

follows:

class

d

D

type

w

T

class

d

D

type

w

w

d

weight

d

t

w

weight

d

t

w

weight

d

t

w

α

α

where

type(w)

represents

the

type

of

term

w,

and

class(d)

represents

the

class

of

story

d,

c

kα

is

reweighting

parameter

for

news

class

c

and

term

type

k.

In

the

work,

we

just

simply

use

statistics

in

table

as

the

reweighting

parameters.

Even

thought

using

the

statistics

directly

may

not

the

best

choice,

we

do

not

discuss

how

to

automatically

obtain

the

best

parameters.

We

will

try

to

use

machine

learning

techniques

to

obtain

the

best

parameters

in

the

future

work.

In

the

work,

we

use

BoosTexter

to

classify

all

stories

into

one

of

the

ROIs.

BoosTexter

is

a

boosting

based

machine

learning

program,

which

creates

a

series

of

simple

rules

for

building

a

classifier

for

text

or

attribute-value

data.

We

use

term

weight

generated

using

TF-IDF

model

as

feature

for

story

classification.

We

trained

the

model

on

the

judged

English

stories

in

and

classify

the

rest

of

the

stories

in

and

all

stories

in

Classification

results

are

used

for

term

reweighting

in

formula

Since

the

class

labels

of

topic-off

stories

are

not

given

in

TDT

datasets,

we

cannot

give

the

classification

accuracy

here.

Thus

we

do

not

discuss

the

effects

of

classification

accuracy

to

NED

performance

in

the

paper.

EXPERIMENTAL

SETUP

Datasets

We

used

two

LDC

datasets

and

for

our

experiments.

contains

news

stories

from

January

to

June

It

contains

around

stories

from

sources

like

ABC,

Associated

Press,

CNN,

New

York

Times,

Public

Radio

International,

Voice

of

America

etc.

Only

English

stories

in

the

collection

were

considered.

contains

approximately

English

stories

collected

from

October

to

December

In

addition

to

the

sources

used

in

it

also

contains

stories

from

NBC

and

MSNBC

TV

broadcasts.

We

used

transcribed

versions

of

the

TV

and

radio

broadcasts

besides

textual

news.

dataset

is

labeled

with

about

topics,

and

approximately

English

stories

belong

to

at

least

one

of

these

topics.

dataset

is

labeled

with

about

topics,

and

approximately

English

stories

belong

to

at

least

one

of

these

topics.

All

the

topics

are

classified

into

Rules

of

Interpretation:

Cases,

Disasters,

Violence

or

War,

and

Discovery

News,

Law,

News,

News.

Evaluation

Metric

TDT

uses

a

cost

function

CDet

that

combines

the

probabilities

of

missing

a

new

story

and

a

false

alarm

*Det

Miss

Miss

Target

FA

FA

NontargetC

C

P

P

C

P

P=

Table

Average

correlation

between

term

types

and

news

classes

where

CMiss

means

the

cost

of

missing

a

new

story,

PMiss

means

the

probability

of

missing

a

new

story,

and

PTarget

means

the

probability

of

seeing

a

new

story

in

the

data;

CFA

means

the

cost

of

a

false

alarm,

PFA

means

the

probability

of

a

false

alarm,

and

PNontarget

means

the

probability

of

seeing

an

old

story.

The

cost

CDet

is

normalized

such

that

a

perfect

system

scores

and

a

trivial

system,

which

is

the

better

one

of

mark

all

stories

as

new

or

old,

scores

Det

Det

Miss

Target

FA

Nontarget

C

Norm

C

min

C

P

C

P

New

event

detection

system

gives

two

outputs

for

each

story.

The

first

part

is

yes

or

no

indicating

whether

the

story

triggers

a

new

event

or

not.

The

second

part

is

a

score

indicating

confidence

of

the

first

decision.

Confidence

scores

can

be

used

to

plot

DET

curve,

i.e.,

curves

that

plot

false

alarm

vs.

miss

probabilities.

Minimum

normalized

cost

can

be

determined

if

optimal

threshold

on

the

score

were

chosen.

EXPERIMENTAL

RESULTS

Main

Results

To

test

the

approaches

proposed

in

the

model,

we

implemented

and

tested

five

systems:

this

system

is

used

as

baseline.

It

is

implemented

based

on

the

basic

model

described

in

section

i.e.,

using

incremental

TF-IDF

model

to

generate

term

weights,

and

using

Hellinger

distance

to

compute

document

similarity.

Similarity

score

normalization

is

also

employed

S-S

detection

procedure

is

used.

this

system

is

the

same

as

except

that

S-C

detection

procedure

is

used.

this

system

is

the

same

as

except

that

it

uses

the

new

detection

procedure

which

is

based

on

indexing-tree.

implemented

based

on

the

approach

presented

in

section

i.e.,

terms

are

reweighted

according

to

the

distance

between

term

distributions

in

a

cluster

and

all

stories.

The

new

detection

procedure

is

used.

implemented

based

on

the

approach

presented

in

section

i.e.,

terms

of

different

types

are

reweighted

according

to

news

class

using

trained

parameters.

The

new

detection

procedure

is

used.

The

following

are

some

other

NED

systems:

for

each

pair

of

stories,

it

computes

three

similarity

values

for

named

entity,

non-named

entity

and

all

terms

respectively.

And

employ

Support

Vector

Machine

to

predict

new

or

old

using

the

similarity

values

as

features.

it

extended

a

basic

incremental

TF-IDF

model

to

include

source-specific

models,

similarity

score

normalization

based

on

document-specific

averages,

similarity

score

normalization

based

on

source-pair

specific

averages,

etc.

it

split

document

representation

into

two

parts:

named

entities

and

non-named

entities,

and

choose

one

effective

part

for

each

news

class.

Table

and

table

show

topic-weighted

normalized

costs

and

comparing

times

on

and

datasets

respectively.

Since

no

heldout

data

set

for

fine-tuning

the

threshold

θ

new

was

available

for

experiments

on

we

only

report

minimum

normalized

costs

for

our

systems

in

table

outperforms

all

other

systems

including

and

it

performs

only

comparing

times

in

detection

procedure

which

is

only

of

Table

NED

results

on

Systems

Min

Norm(CDet)

Cmp

times

-①

θ

②

θ

λ

When

evaluating

on

the

normalized

costs

on

we

use

the

optimal

thresholds

obtained

from

data

set

for

all

systems.

reduces

comparing

times

to

which

is

just

of

but

at

the

same

time

it

also

gets

a

deteriorated

minimum

normalized

cost

which

is

higher

than

uses

the

new

detection

procedure

based

on

news

indexing-tree.

It

requires

even

less

comparing

times

than

This

is

because

story-story

comparisons

usually

yield

greater

similarities

than

story-cluster

ones,

so

stories

tend

to

be

combined

Location

Person

Date

Organization

Money

Percentage

NN

JJ

CD

Elections

Scandals/Hearings

Legal/Criminal

Cases

Natural

Disasters

Violence

or

War

Science

and

Discovery

Finances

Sports

together

in

And

is

basically

equivalent

to

in

accuracy

results.

adjusts

term

weights

based

on

the

distance

of

term

distributions

between

the

whole

corpus

and

cluster

story

set,

yielding

a

good

improvement

by

compared

to

The

best

system

has

a

minimum

normalized

cost

which

is

better

than

and

also

better

than

any

other

results

previously

reported

for

this

dataset

Further

more,

only

needs

comparing

times

which

is

of

Table

NED

results

on

Systems

Norm(CDet)

Min

Norm(CDet)

Cmp

times

-①

θ

②

θ

λ

shows

the

five

DET

curves

for

our

systems

on

data

set

achieves

the

minimum

cost

at

a

false

alarm

rate

of

and

a

miss

rate

of

We

can

observe

that

and

obtain

lower

miss

probability

at

regions

of

low

false

alarm

probabilities.

The

hypothesis

is

that,

more

weight

value

is

transferred

to

key

terms

of

topics

from

non-key

terms.

Similarity

score

between

two

stories

belonging

to

different

topics

are

lower

than

before,

because

their

overlapping

terms

are

usually

not

key

terms

of

their

topics.

Parameter

selection

for

indexing-tree

detection

Figure

shows

the

minimum

normalized

costs

obtained

by

on

using

different

parameters.

Theθ

init

parameter

is

tested

on

six

values

spanning

from

to

And

the

λ

parameter

is

tested

on

four

values

and

We

can

see

that,

whenθ

init

is

set

to

which

is

the

closest

one

toθ

new,

the

costs

are

lower

than

others.

This

is

easy

to

explain,

because

when

stories

belonging

to

the

same

topic

are

put

in

a

cluster,

it

is

more

reasonable

for

the

cluster

to

represent

the

stories

in

it.

When

parameter

λ

is

set

to

or

the

costs

are

better

than

other

cases,

but

there

is

no

much

difference

between

and

θ-initλ

MinCost

Figure

Min

Cost

on

(δ

x

θ-init

λ

Comparingtimes

x

Figure

Comparing

times

on

(δ

Figure

gives

the

comparing

times

used

by

on

with

the

same

parameters

as

figure

The

comparing

times

are

strongly

dependent

onθ

init.

Because

the

greaterθ

init

is,

the

less

stories

combined

together,

the

more

comparing

times

are

needed

for

new

event

decision.

So

we

useθ

init

for

and

In

this

parameter

setting,

we

can

get

both

low

minimum

normalized

costs

and

less

comparing

times.

CONCLUSION

We

have

proposed

a

news

indexing-tree

based

detection

procedure

in

our

model.

It

reduces

comparing

times

to

about

one

seventh

of

traditional

method

without

hurting

NED

accuracy.

We

also

have

presented

two

extensions

to

the

basic

TF-IDF

model.

The

first

extension

is

made

by

adjust

term

weights

based

on

term

distributions

between

the

whole

corpus

and

a

cluster

story

set.

And

the

second

extension

to

basic

TF-IDF

model

is

better

use

of

term

types

(named

entities

types

and

part-of-speed)

according

to

news

categories.

Our

experimental

results

on

and

datasets

show

that

both

of

the

two

extensions

contribute

significantly

to

improvement

in

accuracy.

We

did

not

consider

news

time

information

as

a

clue

for

NED

task,

since

most

of

the

topics

last

for

a

long

time

and

TDT

data

sets

only

span

for

a

relative

short

period

(no

more

than

months).

For

the

future

work,

we

want

to

collect

news

set

which

span

for

a

longer

period

from

internet,

and

integrate

time

information

in

NED

task.

Since

topic

is

a

relative

coarse-grained

news

cluster,

we

also

want

to

refine

cluster

granularity

to

event-level,

and

identify

different

events

and

their

relations

within

a

topic.

Acknowledgments

This

work

is

supported

by

the

National

Natural

Science

Foundation

of

China

under

Grant

No.

Any

opinions,

findings

and

conclusions

or

recommendations

expressed

in

this

material

are

the

author(s)

and

do

not

necessarily

reflect

those

of

the

sponsor.

REFERENCES

In

Topic

Detection

and

Tracking.

Event-based

Information

Organization.

Kluwer

Academic

Publishers,

False

Alarm

Probability

(in

MissProbability(in%)

Topic

Weighted

Curve

Min

Norm(Cost)

Topic

Weighted

Curve

Min

Norm(Cost)

Topic

Weighted

Curve

Min

Norm(Cost)

Topic

Weighted

Curve

Min

Norm(Cost)

Topic

Weighted

Curve

Min

Norm(Cost)

Random

Performance

Figure

DET

curves

on

Y.

Yang,

J.

Carbonell,

R.

Brown,

T.

Pierce,

B.T.

Archibald,

and

X.

Liu.

Learning

Approaches

for

Detecting

and

Tracking

News

Events.

In

IEEE

Intelligent

Systems

Special

Issue

on

Applications

of

Intelligent

Information

Retrieval,

volume

Y.

Yang,

T.

Pierce,

and

J.

Carbonell.

A

Study

on

Retrospective

and

On-line

Event

Detection.

In

Proceedings

of

Melbourne,

Australia,

J.

Allan,

V.

Lavrenko,

D.

Malin,

and

R.

Swan.

Detections,

Bounds,

and

Timelines:

Umass

and

In

Proceedings

of

Topic

Detection

and

Tracking

Workshop

Vienna,

VA,

R.

Papka

and

J.

Allan.

On-line

New

Event

Detection

Using

Single

Pass

Clustering

Technical

Report

W.

Lam,

H.

Meng,

K.

Wong,

and

J.

Yen.

Using

Contextual

Analysis

for

News

Event

Detection.

International

Journal

on

Intelligent

Systems,

B.

Thorsten,

C.

Francine,

and

F.

Ayman.

A

System

for

New

Event

Detection.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference,

New

York,

NY,

USA.

ACM

Press.

S.

Nicola

and

C.

Joe.

Combining

Semantic

and

Syntactic

Document

Classifiers

to

Improve

First

Story

Detection.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference,

New

York,

NY,

USA.

ACM

Press.

Y.

Yang,

J.

Zhang,

J.

Carbonell,

and

C.

Jin.

Topicconditioned

Novelty

Detection.

In

Proceedings

of

the

ACM

SIGKDD

International

Conference,

ACM

Press.

M.

Juha,

A.M.

Helena,

and

S.

Marko.

Applying

Semantic

Classes

in

Event

Detection

and

Tracking.

In

Proceedings

of

International

Conference

on

Natural

Language

Processing

(ICON

pages

M.

Juha,

A.M.

Helena,

and

S.

Marko.

Simple

Semantics

in

Topic

Detection

and

Tracking.

Information

Retrieval,

K.

Giridhar

and

J.

Allan.

Text

Classification

and

Named

Entities

for

New

Event

Detection.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference,

New

York,

NY,

USA.

ACM

Press.

J.

P.

Callan,

W.

B.

Croft,

and

S.

M.

Harding.

The

INQUERY

Retrieval

System.

In

Proceedings

of

International

Conference

on

Database

and

Expert

Systems

Applications,

R.

Krovetz.

Viewing

Morphology

as

An

Inference

Process.

In

Proceedings

of

ACM

Y.

Yang

and

J.

Pedersen.

A

Comparative

Study

on

Feature

Selection

in

Text

Categorization.

In

J.

D.

H.

Fisher,

editor,

The

Fourteenth

International

Conference

on

Machine

Learning

Morgan

Kaufmann,

T.

M.

Cover,

and

J.A.

Thomas.

Elements

of

Information

Theory.

Wiley.

The

linguistic

data

consortium,

The

TDT

task

definition

and

evaluation

plan,

R.

E.

Schapire

and

Y.

Singer.

Boostexter:

A

Boosting-based

System

for

Text

Categorization.

In

Machine

Learning

Kluwer

Academic

Publishers,

K.

Giridhar

and

J.

Allan.

Using

Names

and

Topics

for

New

Event

Detection.

In

Proceedings

of

Human

Technology

Conference

and

Conference

on

Empirical

Methods

in

Natural

Language,

Vancouver,

B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O B I I O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O

stream of news story
news volume
news indexing-tree
term reweighting approach
named entity reweighting mode
existing system
topic detection and track
name entity
real-time index

new event detection
news story stream
volume of news
speed up the ned task
ned accuracy
term weight
statistics
training data
class of story
story class
linguistic data consortium
baseline system