Regularized

Clustering

for

Documents

Fei

Wang,

Changshui

Zhang

State

Key

Lab

of

Intelligent

Tech.

and

Systems

Department

of

Automation,

Tsinghua

University

Beijing,

China,

Tao

Li

School

of

Computer

Science

Florida

International

University

Miami,

FL

U.S.A.

taoli@cs.fiu.edu

ABSTRACT

In

recent

years,

document

clustering

has

been

receiving

more

and

more

attentions

as

an

important

and

fundamental

technique

for

unsupervised

document

organization,

automatic

topic

extraction,

and

fast

information

retrieval

or

filtering.

In

this

paper,

we

propose

a

novel

method

for

clustering

documents

using

regularization.

Unlike

traditional

globally

regularized

clustering

methods,

our

method

first

construct

a

local

regularized

linear

label

predictor

for

each

document

vector,

and

then

combine

all

those

local

regularizers

with

a

global

smoothness

regularizer.

So

we

call

our

algorithm

Clustering

with

Local

and

Global

Regularization

(CLGR).

We

will

show

that

the

cluster

memberships

of

the

documents

can

be

achieved

by

eigenvalue

decomposition

of

a

sparse

symmetric

matrix,

which

can

be

efficiently

solved

by

iterative

methods.

Finally

our

experimental

evaluations

on

several

datasets

are

presented

to

show

the

superiorities

of

CLGR

over

traditional

document

clustering

methods.

Categories

and

Subject

Descriptors

[Information

Storage

and

Retrieval]:

Information

Search

and

Retrieval-Clustering;

[Artificial

Intelligence]:

Learning-Concept

Learning

General

Terms

Algorithms

INTRODUCTION

Document

clustering

has

been

receiving

more

and

more

attentions

as

an

important

and

fundamental

technique

for

unsupervised

document

organization,

automatic

topic

extraction,

and

fast

information

retrieval

or

filtering.

A

good

document

clustering

approach

can

assist

the

computers

to

automatically

organize

the

document

corpus

into

a

meaningful

cluster

hierarchy

for

efficient

browsing

and

navigation,

which

is

very

valuable

for

complementing

the

deficiencies

of

traditional

information

retrieval

technologies.

As

pointed

out

by

the

information

retrieval

needs

can

be

expressed

by

a

spectrum

ranged

from

narrow

keyword-matching

based

search

to

broad

information

browsing

such

as

what

are

the

major

international

events

in

recent

months.

Traditional

document

retrieval

engines

tend

to

fit

well

with

the

search

end

of

the

spectrum,

i.e.

they

usually

provide

specified

search

for

documents

matching

the

user"s

query,

however,

it

is

hard

for

them

to

meet

the

needs

from

the

rest

of

the

spectrum

in

which

a

rather

broad

or

vague

information

is

needed.

In

such

cases,

efficient

browsing

through

a

good

cluster

hierarchy

will

be

definitely

helpful.

Generally,

document

clustering

methods

can

be

mainly

categorized

into

two

classes:

hierarchical

methods

and

partitioning

methods.

The

hierarchical

methods

group

the

data

points

into

a

hierarchical

tree

structure

using

bottom-up

or

top-down

approaches.

For

example,

hierarchical

agglomerative

clustering

(HAC)

is

a

typical

bottom-up

hierarchical

clustering

method.

It

takes

each

data

point

as

a

single

cluster

to

start

off

with

and

then

builds

bigger

and

bigger

clusters

by

grouping

similar

data

points

together

until

the

entire

dataset

is

encapsulated

into

one

final

cluster.

On

the

other

hand,

partitioning

methods

decompose

the

dataset

into

a

number

of

disjoint

clusters

which

are

usually

optimal

in

terms

of

some

predefined

criterion

functions.

For

instance,

K-means

is

a

typical

partitioning

method

which

aims

to

minimize

the

sum

of

the

squared

distance

between

the

data

points

and

their

corresponding

cluster

centers.

In

this

paper,

we

will

focus

on

the

partitioning

methods.

As

we

know

that

there

are

two

main

problems

existing

in

partitioning

methods

(like

Kmeans

and

Gaussian

Mixture

Model

(GMM)

the

predefined

criterion

is

usually

non-convex

which

causes

many

local

optimal

solutions;

the

iterative

procedure

(e.g.

the

Expectation

Maximization

(EM)

algorithm)

for

optimizing

the

criterions

usually

makes

the

final

solutions

heavily

depend

on

the

initializations.

In

the

last

decades,

many

methods

have

been

proposed

to

overcome

the

above

problems

of

the

partitioning

methods

Recently,

another

type

of

partitioning

methods

based

on

clustering

on

data

graphs

have

aroused

considerable

interests

in

the

machine

learning

and

data

mining

community.

The

basic

idea

behind

these

methods

is

to

first

model

the

whole

dataset

as

a

weighted

graph,

in

which

the

graph

nodes

represent

the

data

points,

and

the

weights

on

the

edges

correspond

to

the

similarities

between

pairwise

points.

Then

the

cluster

assignments

of

the

dataset

can

be

achieved

by

optimizing

some

criterions

defined

on

the

graph.

For

example

Spectral

Clustering

is

one

kind

of

the

most

representative

graph-based

clustering

approaches,

it

generally

aims

to

optimize

some

cut

value

(e.g.

Normalized

Cut

Ratio

Cut

Min-Max

Cut

defined

on

an

undirected

graph.

After

some

relaxations,

these

criterions

can

usually

be

optimized

via

eigen-decompositions,

which

is

guaranteed

to

be

global

optimal.

In

this

way,

spectral

clustering

efficiently

avoids

the

problems

of

the

traditional

partitioning

methods

as

we

introduced

in

last

paragraph.

In

this

paper,

we

propose

a

novel

document

clustering

algorithm

that

inherits

the

superiority

of

spectral

clustering,

i.e.

the

final

cluster

results

can

also

be

obtained

by

exploit

the

eigen-structure

of

a

symmetric

matrix.

However,

unlike

spectral

clustering,

which

just

enforces

a

smoothness

constraint

on

the

data

labels

over

the

whole

data

manifold

our

method

first

construct

a

regularized

linear

label

predictor

for

each

data

point

from

its

neighborhood

as

in

and

then

combine

the

results

of

all

these

local

label

predictors

with

a

global

label

smoothness

regularizer.

So

we

call

our

method

Clustering

with

Local

and

Global

Regularization

(CLGR).

The

idea

of

incorporating

both

local

and

global

information

into

label

prediction

is

inspired

by

the

recent

works

on

semi-supervised

learning

and

our

experimental

evaluations

on

several

real

document

datasets

show

that

CLGR

performs

better

than

many

state-of-the-art

clustering

methods.

The

rest

of

this

paper

is

organized

as

follows:

in

section

we

will

introduce

our

CLGR

algorithm

in

detail.

The

experimental

results

on

several

datasets

are

presented

in

section

followed

by

the

conclusions

and

discussions

in

section

THE

PROPOSED

ALGORITHM

In

this

section,

we

will

introduce

our

Clustering

with

Local

and

Global

Regularization

(CLGR)

algorithm

in

detail.

First

let"s

see

the

how

the

documents

are

represented

throughout

this

paper.

Document

Representation

In

our

work,

all

the

documents

are

represented

by

the

weighted

term-frequency

vectors.

Let

W

wm}

be

the

complete

vocabulary

set

of

the

document

corpus

(which

is

preprocessed

by

the

stopwords

removal

and

words

stemming

operations).

The

term-frequency

vector

xi

of

document

di

is

defined

as

xi

xim]T

xik

tik

log

n

idfk

where

tik

is

the

term

frequency

of

wk

W,

n

is

the

size

of

the

document

corpus,

idfk

is

the

number

of

documents

that

contain

word

wk.

In

this

way,

xi

is

also

called

the

TFIDF

representation

of

document

di.

Furthermore,

we

also

normalize

each

xi

i

n)

to

have

a

unit

length,

so

that

each

document

is

represented

by

a

normalized

TF-IDF

vector.

Local

Regularization

As

its

name

suggests,

CLGR

is

composed

of

two

parts:

local

regularization

and

global

regularization.

In

this

subsection

we

will

introduce

the

local

regularization

part

in

detail.

Motivation

As

we

know

that

clustering

is

one

type

of

learning

techniques,

it

aims

to

organize

the

dataset

in

a

reasonable

way.

Generally

speaking,

learning

can

be

posed

as

a

problem

of

function

estimation,

from

which

we

can

get

a

good

classification

function

that

will

assign

labels

to

the

training

dataset

and

even

the

unseen

testing

dataset

with

some

cost

minimized

For

example,

in

the

two-class

classification

(in

which

we

exactly

know

the

label

of

each

document),

a

linear

classifier

with

least

square

fit

aims

to

learn

a

column

vector

w

such

that

the

squared

cost

J

n

(wT

xi

is

minimized,

where

yi

is

the

label

of

xi.

By

taking

∂J

/∂w

we

get

the

solution

w∗

n

xixT

i

−1

n

xiyi

which

can

further

be

written

in

its

matrix

form

as

w∗

XXT

−1

Xy,

where

X

xn]

is

an

m

n

document

matrix,

y

yn]T

is

the

label

vector.

Then

for

a

test

document

t,

we

can

determine

its

label

by

l

sign(w∗T

u),

where

sign(·)

is

the

sign

function.

A

natural

problem

in

is

that

the

matrix

XXT

may

be

singular

and

thus

not

invertable

(e.g.

when

m

n).

To

avoid

such

a

problem,

we

can

add

a

regularization

term

and

minimize

the

following

criterion

J

n

n

(wT

xi

λ

w

where

λ

is

a

regularization

parameter.

Then

the

optimal

solution

that

minimize

J

is

given

by

w∗

XXT

λnI

−1

Xy,

where

I

is

an

m

m

identity

matrix.

It

has

been

reported

that

the

regularized

linear

classifier

can

achieve

very

good

results

on

text

classification

problems

However,

despite

its

empirical

success,

the

regularized

linear

classifier

is

on

earth

a

global

classifier,

i.e.

w∗

is

estimated

using

the

whole

training

set.

According

to

this

may

not

be

a

smart

idea,

since

a

unique

w∗

may

not

be

good

enough

for

predicting

the

labels

of

the

whole

input

space.

In

order

to

get

better

predictions,

proposed

to

train

classifiers

locally

and

use

them

to

classify

the

testing

points.

For

example,

a

testing

point

will

be

classified

by

the

local

classifier

trained

using

the

training

points

located

in

the

vicinity

In

the

following

discussions

we

all

assume

that

the

documents

coming

from

only

two

classes.

The

generalizations

of

our

method

to

multi-class

cases

will

be

discussed

in

section

of

it.

Although

this

method

seems

slow

and

stupid,

it

is

reported

that

it

can

get

better

performances

than

using

a

unique

global

classifier

on

certain

tasks

Constructing

the

Local

Regularized

Predictors

Inspired

by

their

success,

we

proposed

to

apply

the

local

learning

algorithms

for

clustering.

The

basic

idea

is

that,

for

each

document

vector

xi

i

n),

we

train

a

local

label

predictor

based

on

its

k-nearest

neighborhood

Ni,

and

then

use

it

to

predict

the

label

of

xi.

Finally

we

will

combine

all

those

local

predictors

by

minimizing

the

sum

of

their

prediction

errors.

In

this

subsection

we

will

introduce

how

to

construct

those

local

predictors.

Due

to

the

simplicity

and

effectiveness

of

the

regularized

linear

classifier

that

we

have

introduced

in

section

we

choose

it

to

be

our

local

label

predictor,

such

that

for

each

document

xi,

the

following

criterion

is

minimized

Ji

ni

xj

∈Ni

wT

i

xj

qj

λi

wi

where

ni

|Ni|

is

the

cardinality

of

Ni,

and

qj

is

the

cluster

membership

of

xj.

Then

using

we

can

get

the

optimal

solution

is

w∗

i

XiXT

i

λiniI

−1

Xiqi,

where

Xi

xini

and

we

use

xik

to

denote

the

k-th

nearest

neighbor

of

xi.

qi

qini

]T

with

qik

representing

the

cluster

assignment

of

xik.

The

problem

here

is

that

XiXT

i

is

an

m

m

matrix

with

m

ni,

i.e.

we

should

compute

the

inverse

of

an

m

m

matrix

for

every

document

vector,

which

is

computationally

prohibited.

Fortunately,

we

have

the

following

theorem:

Theorem

w∗

i

in

can

be

rewritten

as

w∗

i

Xi

XT

i

Xi

λiniIi

−1

qi,

where

Ii

is

an

ni

ni

identity

matrix.

Proof.

Since

w∗

i

XiXT

i

λiniI

−1

Xiqi,

then

XiXT

i

λiniI

w∗

i

Xiqi

XiXT

i

w∗

i

λiniw∗

i

Xiqi

w∗

i

Xi

qi

XT

i

w∗

i

Let

β

qi

XT

i

w∗

i

then

w∗

i

Xiβ

λiniβ

qi

XT

i

w∗

i

qi

XT

i

Xiβ

qi

XT

i

Xi

λiniIi

β

β

XT

i

Xi

λiniIi

−1

qi.

Therefore

w∗

i

Xiβ

Xi

XT

i

Xi

λiniIi

−1

qi

Using

theorem

we

only

need

to

compute

the

inverse

of

an

ni

ni

matrix

for

every

document

to

train

a

local

label

predictor.

Moreover,

for

a

new

testing

point

u

that

falls

into

Ni,

we

can

classify

it

by

the

sign

of

qu

w∗T

i

u

uT

wi

uT

Xi

XT

i

Xi

λiniIi

−1

qi.

This

is

an

attractive

expression

since

we

can

determine

the

cluster

assignment

of

u

by

using

the

inner-products

between

the

points

in

{u

Ni},

which

suggests

that

such

a

local

regularizer

can

easily

be

kernelized

as

long

as

we

define

a

proper

kernel

function.

Combining

the

Local

Regularized

Predictors

After

all

the

local

predictors

having

been

constructed,

we

will

combine

them

together

by

minimizing

Jl

n

w∗T

i

xi

qi

which

stands

for

the

sum

of

the

prediction

errors

for

all

the

local

predictors.

Combining

with

we

can

get

Jl

n

w∗T

i

xi

qi

n

xT

i

Xi

XT

i

Xi

λiniIi

−1

qi

qi

Pq

q

where

q

qn]T

and

the

P

is

an

n

n

matrix

constructing

in

the

following

way.

Let

αi

xT

i

Xi

XT

i

Xi

λiniIi

−1

then

Pij

αi

j,

if

xj

Ni

otherwise

where

Pij

is

the

(i,

j)-th

entry

of

P,

and

αi

j

represents

the

j-th

entry

of

αi

Till

now

we

can

write

the

criterion

of

clustering

by

combining

locally

regularized

linear

label

predictors

Jl

in

an

explicit

mathematical

form,

and

we

can

minimize

it

directly

using

some

standard

optimization

techniques.

However,

the

results

may

not

be

good

enough

since

we

only

exploit

the

local

informations

of

the

dataset.

In

the

next

subsection,

we

will

introduce

a

global

regularization

criterion

and

combine

it

with

Jl,

which

aims

to

find

a

good

clustering

result

in

a

local-global

way.

Global

Regularization

In

data

clustering,

we

usually

require

that

the

cluster

assignments

of

the

data

points

should

be

sufficiently

smooth

with

respect

to

the

underlying

data

manifold,

which

implies

the

nearby

points

tend

to

have

the

same

cluster

assignments;

the

points

on

the

same

structure

(e.g.

submanifold

or

cluster)

tend

to

have

the

same

cluster

assignments

Without

the

loss

of

generality,

we

assume

that

the

data

points

reside

(roughly)

on

a

low-dimensional

manifold

and

q

is

the

cluster

assignment

function

defined

on

M,

i.e.

We

believe

that

the

text

data

are

also

sampled

from

some

low

dimensional

manifold,

since

it

is

impossible

for

them

to

for

∀x

M,

q(x)

returns

the

cluster

membership

of

x.

The

smoothness

of

q

over

M

can

be

calculated

by

the

following

Dirichlet

integral

D[q]

M

q(x)

dM,

where

the

gradient

q

is

a

vector

in

the

tangent

space

T

Mx,

and

the

integral

is

taken

with

respect

to

the

standard

measure

on

M.

If

we

restrict

the

scale

of

q

by

q,

q

M

(where

M

is

the

inner

product

induced

on

M),

then

it

turns

out

that

finding

the

smoothest

function

minimizing

D[q]

reduces

to

finding

the

eigenfunctions

of

the

Laplace

Beltrami

operator

L,

which

is

defined

as

Lq

−div

q,

where

div

is

the

divergence

of

a

vector

field.

Generally,

the

graph

can

be

viewed

as

the

discretized

form

of

manifold.

We

can

model

the

dataset

as

an

weighted

undirected

graph

as

in

spectral

clustering

where

the

graph

nodes

are

just

the

data

points,

and

the

weights

on

the

edges

represent

the

similarities

between

pairwise

points.

Then

it

can

be

shown

that

minimizing

corresponds

to

minimizing

Jg

qT

Lq

n

(qi

wij,

where

q

qn]T

with

qi

q(xi),

L

is

the

graph

Laplacian

with

its

(i,

j)-th

entry

Lij

di

wii,

if

i

j

−wij,

if

xi

and

xj

are

adjacent

otherwise,

where

di

j

wij

is

the

degree

of

xi,

wij

is

the

similarity

between

xi

and

xj.

If

xi

and

xj

are

wij

is

usually

computed

in

the

following

way

wij

e

xi−xj

where

σ

is

a

dataset

dependent

parameter.

It

is

proved

that

under

certain

conditions,

such

a

form

of

wij

to

determine

the

weights

on

graph

edges

leads

to

the

convergence

of

graph

Laplacian

to

the

Laplace

Beltrami

operator

In

summary,

using

with

exponential

weights

can

effectively

measure

the

smoothness

of

the

data

assignments

with

respect

to

the

intrinsic

data

manifold.

Thus

we

adopt

it

as

a

global

regularizer

to

punish

the

smoothness

of

the

predicted

data

assignments.

Clustering

with

Local

and

Global

Regularization

Combining

the

contents

we

have

introduced

in

section

and

section

we

can

derive

the

clustering

criterion

is

minq

J

Jl

λJg

Pq

q

λqT

Lq

s.t.

qi

where

P

is

defined

as

in

and

λ

is

a

regularization

parameter

to

trade

off

Jl

and

Jg.

However,

the

discrete

fill

in

the

whole

high-dimensional

sample

space.

And

it

has

been

shown

that

the

manifold

based

methods

can

achieve

good

results

on

text

classification

tasks

In

this

paper,

we

define

xi

and

xj

to

be

adjacent

if

xi

N(xj)

or

xj

N(xi).

constraint

of

pi

makes

the

problem

an

NP

hard

integer

programming

problem.

A

natural

way

for

making

the

problem

solvable

is

to

remove

the

constraint

and

relax

qi

to

be

continuous,

then

the

objective

that

we

aims

to

minimize

becomes

J

Pq

q

λqT

Lq

qT

(P

I)T

(P

I)q

λqT

Lq

qT

(P

I)T

(P

I)

λL

q,

and

we

further

add

a

constraint

qT

q

to

restrict

the

scale

of

q.

Then

our

objective

becomes

minq

J

qT

(P

I)T

(P

I)

λL

q

s.t.

qT

q

Using

the

Lagrangian

method,

we

can

derive

that

the

optimal

solution

q

corresponds

to

the

smallest

eigenvector

of

the

matrix

M

(P

I)T

(P

I)

λL,

and

the

cluster

assignment

of

xi

can

be

determined

by

the

sign

of

qi,

i.e.

xi

will

be

classified

as

class

one

if

qi

otherwise

it

will

be

classified

as

class

Multi-Class

CLGR

In

the

above

we

have

introduced

the

basic

framework

of

Clustering

with

Local

and

Global

Regularization

(CLGR)

for

the

two-class

clustering

problem,

and

we

will

extending

it

to

multi-class

clustering

in

this

subsection.

First

we

assume

that

all

the

documents

belong

to

C

classes

indexed

by

L

C}.

qc

is

the

classification

function

for

class

c

c

C),

such

that

qc

(xi)

returns

the

confidence

that

xi

belongs

to

class

c.

Our

goal

is

to

obtain

the

value

of

qc

(xi)

c

C,

i

n),

and

the

cluster

assignment

of

xi

can

be

determined

by

{qc

(xi)}C

using

some

proper

discretization

methods

that

we

will

introduce

later.

Therefore,

in

this

multi-class

case,

for

each

document

xi

i

n),

we

will

construct

C

locally

linear

regularized

label

predictors

whose

normal

vectors

are

wc∗

i

Xi

XT

i

Xi

λiniIi

−1

qc

i

c

C),

where

Xi

xini

with

xik

being

the

k-th

neighbor

of

xi,

and

qc

i

[qc

qc

qc

ini

]T

with

qc

ik

qc

(xik).

Then

(wc∗

i

)T

xi

returns

the

predicted

confidence

of

xi

belonging

to

class

c.

Hence

the

local

prediction

error

for

class

c

can

be

defined

as

J

c

l

n

(wc∗

i

T

xi

qc

i

And

the

total

local

prediction

error

becomes

Jl

C

J

c

l

C

n

(wc∗

i

T

xi

qc

i

As

in

we

can

define

an

n×n

matrix

P

(see

and

rewrite

Jl

as

Jl

C

J

c

l

C

Pqc

qc

Similarly

we

can

define

the

global

smoothness

regularizer

in

multi-class

case

as

Jg

C

n

(qc

i

qc

j

wij

C

(qc

)T

Lqc

Then

the

criterion

to

be

minimized

for

CLGR

in

multi-class

case

becomes

J

Jl

λJg

C

Pqc

qc

λ(qc

)T

Lqc

C

(qc

)T

(P

I)T

(P

I)

λL

qc

trace

QT

(P

I)T

(P

I)

λL

Q

where

Q

qc

is

an

n

c

matrix,

and

trace(·)

returns

the

trace

of

a

matrix.

The

same

as

in

we

also

add

the

constraint

that

QT

Q

I

to

restrict

the

scale

of

Q.

Then

our

optimization

problem

becomes

minQ

J

trace

QT

(P

I)T

(P

I)

λL

Q

s.t.

QT

Q

I,

From

the

Ky

Fan

theorem

we

know

the

optimal

solution

of

the

above

problem

is

Q∗

[q∗

q∗

q∗

C

]R,

where

q∗

k

k

C)

is

the

eigenvector

corresponds

to

the

k-th

smallest

eigenvalue

of

matrix

(P

I)T

(P

I)

λL,

and

R

is

an

arbitrary

C

C

matrix.

Since

the

values

of

the

entries

in

Q∗

is

continuous,

we

need

to

further

discretize

Q∗

to

get

the

cluster

assignments

of

all

the

data

points.

There

are

mainly

two

approaches

to

achieve

this

goal:

As

in

we

can

treat

the

i-th

row

of

Q

as

the

embedding

of

xi

in

a

C-dimensional

space,

and

apply

some

traditional

clustering

methods

like

kmeans

to

clustering

these

embeddings

into

C

clusters.

Since

the

optimal

Q∗

is

not

unique

(because

of

the

existence

of

an

arbitrary

matrix

R),

we

can

pursue

an

optimal

R

that

will

rotate

Q∗

to

an

indication

The

detailed

algorithm

can

be

referred

to

The

detailed

algorithm

procedure

for

CLGR

is

summarized

in

table

EXPERIMENTS

In

this

section,

experiments

are

conducted

to

empirically

compare

the

clustering

results

of

CLGR

with

other

representitive

document

clustering

algorithms

on

datasets.

First

we

will

introduce

the

basic

informations

of

those

datasets.

Datasets

We

use

a

variety

of

datasets,

most

of

which

are

frequently

used

in

the

information

retrieval

research.

Table

summarizes

the

characteristics

of

the

datasets.

Here

an

indication

matrix

T

is

a

n×c

matrix

with

its

(i,

j)th

entry

Tij

such

that

for

each

row

of

Q∗

there

is

only

one

Then

the

xi

can

be

assigned

to

the

j-th

cluster

such

that

j

argjQ∗

ij

Table

Clustering

with

Local

and

Global

Regularization

(CLGR)

Input:

Dataset

X

{xi}n

Number

of

clusters

C;

Size

of

the

neighborhood

K;

Local

regularization

parameters

{λi}n

Global

regularization

parameter

λ;

Output:

The

cluster

membership

of

each

data

point.

Procedure:

Construct

the

K

nearest

neighborhoods

for

each

data

point;

Construct

the

matrix

P

using

Construct

the

Laplacian

matrix

L

using

Construct

the

matrix

M

(P

I)T

(P

I)

λL;

Do

eigenvalue

decomposition

on

M,

and

construct

the

matrix

Q∗

according

to

Output

the

cluster

assignments

of

each

data

point

by

properly

discretize

Q∗

Table

Descriptions

of

the

document

datasets

Datasets

Number

of

documents

Number

of

classes

CSTR

Reuters

WebACE

CSTR.

This

is

the

dataset

of

the

abstracts

of

technical

reports

published

in

the

Department

of

Computer

Science

at

a

university.

The

dataset

contained

abstracts,

which

were

divided

into

four

research

areas:

Natural

Language

Processing(NLP),

Robotics/Vision,

Systems,

and

Theory.

WebKB.

The

WebKB

dataset

contains

webpages

gathered

from

university

computer

science

departments.

There

are

about

documents

and

they

are

divided

into

categories:

student,

faculty,

staff,

course,

project,

department

and

other.

The

raw

text

is

about

Among

these

categories,

student,

faculty,

course

and

project

are

four

most

populous

entity-representing

categories.

The

associated

subset

is

typically

called

Reuters.

The

Text

Categorization

Test

collection

contains

documents

collected

from

the

Reuters

newswire

in

It

is

a

standard

text

categorization

benchmark

and

contains

categories.

In

our

experiments,

we

use

a

subset

of

the

data

collection

which

includes

the

most

frequent

categories

among

the

topics

and

we

call

it

Reuters-top

WebACE.

The

WebACE

dataset

was

from

WebACE

project

and

has

been

used

for

document

clustering

The

WebACE

dataset

contains

documents

consisting

news

articles

from

Reuters

new

service

via

the

Web

in

October

These

documents

are

divided

into

classes.

The

dataset

used

in

our

experiments

are

selected

from

the

famous

The

topic

rec

containing

autos,

motorcycles,

baseball

and

hockey

was

selected

from

the

version

The

dataset

contains

document

vectors.

To

pre-process

the

datasets,

we

remove

the

stop

words

using

a

standard

stop

list,

all

HTML

tags

are

skipped

and

all

header

fields

except

subject

and

organization

of

the

posted

articles

are

ignored.

In

all

our

experiments,

we

first

select

the

top

words

by

mutual

information

with

class

labels.

Evaluation

Metrics

In

the

experiments,

we

set

the

number

of

clusters

equal

to

the

true

number

of

classes

C

for

all

the

clustering

algorithms.

To

evaluate

their

performance,

we

compare

the

clusters

generated

by

these

algorithms

with

the

true

classes

by

computing

the

following

two

performance

measures.

Clustering

Accuracy

(Acc).

The

first

performance

measure

is

the

Clustering

Accuracy,

which

discovers

the

one-toone

relationship

between

clusters

and

classes

and

measures

the

extent

to

which

each

cluster

contained

data

points

from

the

corresponding

class.

It

sums

up

the

whole

matching

degree

between

all

pair

class-clusters.

Clustering

accuracy

can

be

computed

as:

Acc

N

max

Ck,Lm

T(Ck,

Lm)

where

Ck

denotes

the

k-th

cluster

in

the

final

results,

and

Lm

is

the

true

m-th

class.

T(Ck,

Lm)

is

the

number

of

entities

which

belong

to

class

m

are

assigned

to

cluster

k.

Accuracy

computes

the

maximum

sum

of

T(Ck,

Lm)

for

all

pairs

of

clusters

and

classes,

and

these

pairs

have

no

overlaps.

The

greater

clustering

accuracy

means

the

better

clustering

performance.

Normalized

Mutual

Information

(NMI).

Another

evaluation

metric

we

adopt

here

is

the

Normalized

Mutual

Information

NMI

which

is

widely

used

for

determining

the

quality

of

clusters.

For

two

random

variable

X

and

Y,

the

NMI

is

defined

as:

NMI(X,

Y)

I(X,

Y)

H(X)H(Y)

where

I(X,

Y)

is

the

mutual

information

between

X

and

Y,

while

H(X)

and

H(Y)

are

the

entropies

of

X

and

Y

respectively.

One

can

see

that

NMI(X,

X)

which

is

the

maximal

possible

value

of

NMI.

Given

a

clustering

result,

the

NMI

in

is

estimated

as

NMI

C

C

nk,mlog

n·nk,m

nk

ˆnm

C

nklog

nk

n

C

ˆnmlog

ˆnm

n

where

nk

denotes

the

number

of

data

contained

in

the

cluster

Ck

k

C),

ˆnm

is

the

number

of

data

belonging

to

the

m-th

class

m

C),

and

nk,m

denotes

the

number

of

data

that

are

in

the

intersection

between

the

cluster

Ck

and

the

m-th

class.

The

value

calculated

in

is

used

as

a

performance

measure

for

the

given

clustering

result.

The

larger

this

value,

the

better

the

clustering

performance.

Comparisons

We

have

conducted

comprehensive

performance

evaluations

by

testing

our

method

and

comparing

it

with

other

representative

data

clustering

methods

using

the

same

data

corpora.

The

algorithms

that

we

evaluated

are

listed

below.

Traditional

k-means

(KM).

Spherical

k-means

(SKM).

The

implementation

is

based

on

Gaussian

Mixture

Model

(GMM).

The

implementation

is

based

on

Spectral

Clustering

with

Normalized

Cuts

(Ncut).

The

implementation

is

based

on

and

the

variance

of

the

Gaussian

similarity

is

determined

by

Local

Scaling

Note

that

the

criterion

that

Ncut

aims

to

minimize

is

just

the

global

regularizer

in

our

CLGR

algorithm

except

that

Ncut

used

the

normalized

Laplacian.

Clustering

using

Pure

Local

Regularization

(CPLR).

In

this

method

we

just

minimize

Jl

(defined

in

and

the

clustering

results

can

be

obtained

by

doing

eigenvalue

decomposition

on

matrix

(I

P)T

(I

P)

with

some

proper

discretization

methods.

Adaptive

Subspace

Iteration

(ASI).

The

implementation

is

based

on

Nonnegative

Matrix

Factorization

(NMF).

The

implementation

is

based

on

Tri-Factorization

Nonnegative

Matrix

Factorization

(TNMF)

The

implementation

is

based

on

For

computational

efficiency,

in

the

implementation

of

CPLR

and

our

CLGR

algorithm,

we

have

set

all

the

local

regularization

parameters

{λi}n

to

be

identical,

which

is

set

by

grid

search

from

The

size

of

the

k-nearest

neighborhoods

is

set

by

grid

search

from

For

the

CLGR

method,

its

global

regularization

parameter

is

set

by

grid

search

from

When

constructing

the

global

regularizer,

we

have

adopted

the

local

scaling

method

to

construct

the

Laplacian

matrix.

The

final

discretization

method

adopted

in

these

two

methods

is

the

same

as

in

since

our

experiments

show

that

using

such

method

can

achieve

better

results

than

using

kmeans

based

methods

as

in

Experimental

Results

The

clustering

accuracies

comparison

results

are

shown

in

table

and

the

normalized

mutual

information

comparison

results

are

summarized

in

table

From

the

two

tables

we

mainly

observe

that:

Our

CLGR

method

outperforms

all

other

document

clustering

methods

in

most

of

the

datasets;

For

document

clustering,

the

Spherical

k-means

method

usually

outperforms

the

traditional

k-means

clustering

method,

and

the

GMM

method

can

achieve

competitive

results

compared

to

the

Spherical

k-means

method;

The

results

achieved

from

the

k-means

and

GMM

type

algorithms

are

usually

worse

than

the

results

achieved

from

Spectral

Clustering.

Since

Spectral

Clustering

can

be

viewed

as

a

weighted

version

of

kernel

k-means,

it

can

obtain

good

results

the

data

clusters

are

arbitrarily

shaped.

This

corroborates

that

the

documents

vectors

are

not

regularly

distributed

(spherical

or

elliptical).

The

experimental

comparisons

empirically

verify

the

equivalence

between

NMF

and

Spectral

Clustering,

which

Table

Clustering

accuracies

of

the

various

methods

CSTR

Reuters

WebACE

KM

SKM

GMM

NMF

Ncut

ASI

TNMF

CPLR

CLGR

Table

Normalized

mutual

information

results

of

the

various

methods

CSTR

Reuters

WebACE

KM

SKM

GMM

NMF

Ncut

ASI

TNMF

CPLR

CLGR

has

been

proved

theoretically

in

It

can

be

observed

from

the

tables

that

NMF

and

Spectral

Clustering

usually

lead

to

similar

clustering

results.

The

co-clustering

based

methods

(TNMF

and

ASI)

can

usually

achieve

better

results

than

traditional

purely

document

vector

based

methods.

Since

these

methods

perform

an

implicit

feature

selection

at

each

iteration,

provide

an

adaptive

metric

for

measuring

the

neighborhood,

and

thus

tend

to

yield

better

clustering

results.

The

results

achieved

from

CPLR

are

usually

better

than

the

results

achieved

from

Spectral

Clustering,

which

supports

Vapnik"s

theory

that

sometimes

local

learning

algorithms

can

obtain

better

results

than

global

learning

algorithms.

Besides

the

above

comparison

experiments,

we

also

test

the

parameter

sensibility

of

our

method.

There

are

mainly

two

sets

of

parameters

in

our

CLGR

algorithm,

the

local

and

global

regularization

parameters

({λi}n

and

λ,

as

we

have

said

in

section

we

have

set

all

λi"s

to

be

identical

to

λ∗

in

our

experiments),

and

the

size

of

the

neighborhoods.

Therefore

we

have

also

done

two

sets

of

experiments:

Fixing

the

size

of

the

neighborhoods,

and

testing

the

clustering

performance

with

varying

λ∗

and

λ.

In

this

set

of

experiments,

we

find

that

our

CLGR

algorithm

can

achieve

good

results

when

the

two

regularization

parameters

are

neither

too

large

nor

too

small.

Typically

our

method

can

achieve

good

results

when

λ∗

and

λ

are

around

Figure

shows

us

such

a

testing

example

on

the

WebACE

dataset.

Fixing

the

local

and

global

regularization

parameters,

and

testing

the

clustering

performance

with

different

−5

−4

−3

−5

−4

−3

local

regularization

para

(log

value)

global

regularization

para

(log

value)

clusteringaccuracy

Figure

Parameter

sensibility

testing

results

on

the

WebACE

dataset

with

the

neighborhood

size

fixed

to

and

the

x-axis

and

y-axis

represents

the

value

of

λ∗

and

λ.

sizes

of

neighborhoods.

In

this

set

of

experiments,

we

find

that

the

neighborhood

with

a

too

large

or

too

small

size

will

all

deteriorate

the

final

clustering

results.

This

can

be

easily

understood

since

when

the

neighborhood

size

is

very

small,

then

the

data

points

used

for

training

the

local

classifiers

may

not

be

sufficient;

when

the

neighborhood

size

is

very

large,

the

trained

classifiers

will

tend

to

be

global

and

cannot

capture

the

typical

local

characteristics.

Figure

shows

us

a

testing

example

on

the

WebACE

dataset.

Therefore,

we

can

see

that

our

CLGR

algorithm

can

achieve

satisfactory

results

and

is

not

very

sensitive

to

the

choice

of

parameters,

which

makes

it

practical

in

real

world

applications.

CONCLUSIONS

AND

FUTURE

WORKS

In

this

paper,

we

derived

a

new

clustering

algorithm

called

clustering

with

local

and

global

regularization.

Our

method

preserves

the

merit

of

local

learning

algorithms

and

spectral

clustering.

Our

experiments

show

that

the

proposed

algorithm

outperforms

most

of

the

state

of

the

art

algorithms

on

many

benchmark

datasets.

In

the

future,

we

will

focus

on

the

parameter

selection

and

acceleration

issues

of

the

CLGR

algorithm.

REFERENCES

L.

Baker

and

A.

McCallum.

Distributional

Clustering

of

Words

for

Text

Classification.

In

Proceedings

of

the

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

M.

Belkin

and

P.

Niyogi.

Laplacian

Eigenmaps

for

Dimensionality

Reduction

and

Data

Representation.

Neural

Computation,

June

M.

Belkin

and

P.

Niyogi.

Towards

a

Theoretical

Foundation

for

Laplacian-Based

Manifold

Methods.

In

Proceedings

of

the

Conference

on

Learning

Theory

(COLT).

size

of

the

neighborhood

clusteringaccuracy

Figure

Parameter

sensibility

testing

results

on

the

WebACE

dataset

with

the

regularization

parameters

being

fixed

to

and

the

neighborhood

size

varing

from

to

M.

Belkin,

P.

Niyogi

and

V.

Sindhwani.

Manifold

Regularization:

a

Geometric

Framework

for

Learning

from

Examples.

Journal

of

Machine

Learning

Research

D.

Boley.

Principal

Direction

Divisive

Partitioning.

Data

mining

and

knowledge

discovery,

L.

Bottou

and

V.

Vapnik.

Local

learning

algorithms.

Neural

Computation,

P.

K.

Chan,

D.

F.

Schlag

and

J.

Y.

Zien.

Spectral

K-way

Ratio-Cut

Partitioning

and

Clustering.

IEEE

Trans.

Computer-Aided

Design,

Sep.

D.

R.

Cutting,

D.

R.

Karger,

J.

O.

Pederson

and

J.

W.

Tukey.

Scatter/Gather:

A

Cluster-Based

Approach

to

Browsing

Large

Document

Collections.

In

Proceedings

of

the

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

I.

S.

Dhillon

and

D.

S.

Modha.

Concept

Decompositions

for

Large

Sparse

Text

Data

using

Clustering.

Machine

Learning,

vol.

pages

January

C.

Ding,

X.

He,

and

H.

Simon.

On

the

equivalence

of

nonnegative

matrix

factorization

and

spectral

clustering.

In

Proceedings

of

the

SIAM

Data

Mining

Conference,

C.

Ding,

X.

He,

H.

Zha,

M.

Gu,

and

H.

D.

Simon.

A

min-max

cut

algorithm

for

graph

partitioning

and

data

clustering.

In

Proc.

of

the

International

Conference

on

Data

Mining

(ICDM),

pages

C.

Ding,

T.

Li,

W.

Peng,

and

H.

Park.

Orthogonal

Nonnegative

Matrix

Tri-Factorizations

for

Clustering.

In

Proceedings

of

the

Twelfth

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining,

R.

O.

Duda,

P.

E.

Hart,

and

D.

G.

Stork.

Pattern

Classification.

John

Wiley

Sons,

Inc.,

T.

Li,

S.

Ma,

and

M.

Ogihara.

Document

Clustering

via

Adaptive

Subspace

Iteration.

In

Proceedings

of

the

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

T.

Li

and

C.

Ding.

The

Relationships

Among

Various

Nonnegative

Matrix

Factorization

Methods

for

Clustering.

In

Proceedings

of

the

International

Conference

on

Data

Mining

(ICDM).

X.

Liu

and

Y.

Gong.

Document

Clustering

with

Cluster

Refinement

and

Model

Selection

Capabilities.

In

Proc.

of

the

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

E.

Han,

D.

Boley,

M.

Gini,

R.

Gross,

K.

Hastings,

G.

Karypis,

V.

Kumar,

B.

Mobasher,

and

J.

Moore.

WebACE:

A

Web

Agent

for

Document

Categorization

and

Exploration.

In

Proceedings

of

the

International

Conference

on

Autonomous

Agents

ACM

Press,

M.

Hein,

J.

Y.

Audibert,

and

U.

von

Luxburg.

From

Graphs

to

Manifolds

Weak

and

Strong

Pointwise

Consistency

of

Graph

Laplacians.

In

Proceedings

of

the

Conference

on

Learning

Theory

(COLT),

J.

He,

M.

Lan,

C.-L.

Tan,

S.-Y.

Sung,

and

H.-B.

Low.

Initialization

of

Cluster

Refinement

Algorithms:

A

Review

and

Comparative

Study.

In

Proc.

of

Inter.

Joint

Conference

on

Neural

Networks,

A.

Y.

Ng,

M.

I.

Jordan,

Y.

Weiss.

On

Spectral

Clustering:

Analysis

and

an

algorithm.

In

Advances

in

Neural

Information

Processing

Systems

B.

Sch¨olkopf

and

A.

Smola.

Learning

with

Kernels.

The

MIT

Press.

Cambridge,

Massachusetts.

J.

Shi

and

J.

Malik.

Normalized

Cuts

and

Image

Segmentation.

IEEE

Trans.

on

Pattern

Analysis

and

Machine

Intelligence,

A.

Strehl

and

J.

Ghosh.

Cluster

Ensembles

A

Knowledge

Reuse

Framework

for

Combining

Multiple

Partitions.

Journal

of

Machine

Learning

Research,

V.

N.

Vapnik.

The

Nature

of

Statistical

Learning

Theory.

Berlin:

Springer-Verlag,

Wu,

M.

and

Sch¨olkopf,

B.

A

Local

Learning

Approach

for

Clustering.

In

Advances

in

Neural

Information

Processing

Systems

S.

X.

Yu,

J.

Shi.

Multiclass

Spectral

Clustering.

In

Proceedings

of

the

International

Conference

on

Computer

Vision,

W.

Xu,

X.

Liu

and

Y.

Gong.

Document

Clustering

Based

On

Non-Negative

Matrix

Factorization.

In

Proceedings

of

the

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

H.

Zha,

X.

He,

C.

Ding,

M.

Gu

and

H.

Simon.

Spectral

Relaxation

for

K-means

Clustering.

In

NIPS

T.

Zhang

and

F.

J.

Oles.

Text

Categorization

Based

on

Regularized

Linear

Classification

Methods.

Journal

of

Information

Retrieval,

L.

Zelnik-Manor

and

P.

Perona.

Self-Tuning

Spectral

Clustering.

In

NIPS

D.

Zhou,

O.

Bousquet,

T.

N.

Lal,

J.

Weston

and

B.

Sch¨olkopf.

Learning

with

Local

and

Global

Consistency.

NIPS

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

global regularization
hierarchical method
document cluster

document clustering
regularization
cluster hierarchy
spectrum
specified search
partitioning method
label prediction
function estimation
manifold