Machine

Learning

for

Information

Architecture

in

a

Large

Governmental

Website

Miles

Efron

School

of

Information

Library

Science

Manning

Hall

University

of

North

Carolina

Chapel

Hill,

NC

efrom@ils.unc.edu

Jonathan

Elsas

School

of

Information

Library

Science

Manning

Hall

University

of

North

Carolina

Chapel

Hill,

NC

jelsas@email.unc.edu

Gary

Marchionini

School

of

Information

Library

Science

Manning

Hall

University

of

North

Carolina

Chapel

Hill,

NC

march@ils.unc.edu

Junliang

Zhang

School

of

Information

Library

Science

Manning

Hall

University

of

North

Carolina

Chapel

Hill,

NC

junliang@email.unc.edu

ABSTRACT

This

paper

describes

ongoing

research

into

the

application

of

machine

learning

techniques

for

improving

access

to

governmental

information

in

complex

digital

libraries.

Under

the

auspices

of

the

GovStat

Project,

our

goal

is

to

identify

a

small

number

of

semantically

valid

concepts

that

adequately

spans

the

intellectual

domain

of

a

collection.

The

goal

of

this

discovery

is

twofold.

First

we

desire

a

practical

aid

for

information

architects.

Second,

automatically

derived

documentconcept

relationships

are

a

necessary

precondition

for

realworld

deployment

of

many

dynamic

interfaces.

The

current

study

compares

concept

learning

strategies

based

on

three

document

representations:

keywords,

titles,

and

full-text.

In

statistical

and

user-based

studies,

human-created

keywords

provide

significant

improvements

in

concept

learning

over

both

title-only

and

full-text

representations.

Categories

and

Subject

Descriptors

[Information

Storage

and

Retrieval]:

Digital

Libraries-Systems

Issues,

User

Issues;

[Information

Storage

and

Retrieval]:

Information

Search

and

RetrievalClustering

General

Terms

Design,

Experimentation

INTRODUCTION

The

GovStat

Project

is

a

joint

effort

of

the

University

of

North

Carolina

Interaction

Design

Lab

and

the

University

of

Maryland

Human-Computer

Interaction

Citing

end-user

difficulty

in

finding

governmental

information

(especially

statistical

data)

online,

the

project

seeks

to

create

an

integrated

model

of

user

access

to

US

government

statistical

information

that

is

rooted

in

realistic

data

models

and

innovative

user

interfaces.

To

enable

such

models

and

interfaces,

we

propose

a

data-driven

approach,

based

on

data

mining

and

machine

learning

techniques.

In

particular,

our

work

analyzes

a

particular

digital

library-the

website

of

the

Bureau

of

Labor

(BLS)-in

efforts

to

discover

a

small

number

of

linguistically

meaningful

concepts,

or

bins,

that

collectively

summarize

the

semantic

domain

of

the

site.

The

project

goal

is

to

classify

the

site"s

web

content

according

to

these

inferred

concepts

as

an

initial

step

towards

data

filtering

via

active

user

interfaces

(cf.

Many

digital

libraries

already

make

use

of

content

classification,

both

explicitly

and

implicitly;

they

divide

their

resources

manually

by

topical

relation;

they

organize

content

into

hierarchically

oriented

file

systems.

The

goal

of

the

present

research

is

to

develop

another

means

of

browsing

the

content

of

these

collections.

By

analyzing

the

distribution

of

terms

across

documents,

our

goal

is

to

supplement

the

agency"s

pre-existing

information

structures.

Statistical

learning

technologies

are

appealing

in

this

context

insofar

as

they

stand

to

define

a

data-driven-as

opposed

to

an

agency-drivennavigational

structure

for

a

site.

Our

approach

combines

supervised

and

unsupervised

learning

techniques.

A

pure

document

clustering

approach

to

such

a

large,

diverse

collection

as

BLS

led

to

poor

results

in

early

tests

But

strictly

supervised

techniques

are

inappropriate,

too.

Although

BLS

designers

have

defined

high-level

subject

headings

for

their

collections,

as

we

discuss

in

Section

this

scheme

is

less

than

optimal.

Thus

we

hope

to

learn

an

additional

set

of

concepts

by

letting

the

data

speak

for

themselves.

The

remainder

of

this

paper

describes

the

details

of

our

concept

discovery

efforts

and

subsequent

evaluation.

In

Section

we

describe

the

previously

existing,

human-created

conceptual

structure

of

the

BLS

website.

This

section

also

describes

evidence

that

this

structure

leaves

room

for

improvement.

Next

(Sections

we

turn

to

a

description

of

the

concepts

derived

via

content

clustering

under

three

document

representations:

keyword,

title

only,

and

full-text.

Section

describes

a

two-part

evaluation

of

the

derived

conceptual

structures.

Finally,

we

conclude

in

Section

by

outlining

upcoming

work

on

the

project.

STRUCTURING

ACCESS

TO

THE

BLS

WEBSITE

The

Bureau

of

Labor

Statistics

is

a

federal

government

agency

charged

with

compiling

and

publishing

statistics

pertaining

to

labor

and

production

in

the

US

and

abroad.

Given

this

broad

mandate,

the

BLS

publishes

a

wide

array

of

information,

intended

for

diverse

audiences.

The

agency"s

website

acts

as

a

clearinghouse

for

this

process.

With

over

text/html

documents

(and

many

more

documents

if

spreadsheets

and

typeset

reports

are

included),

providing

access

to

the

collection

provides

a

steep

challenge

to

information

architects.

The

Relation

Browser

The

starting

point

of

this

work

is

the

notion

that

access

to

information

in

the

BLS

website

could

be

improved

by

the

addition

of

a

dynamic

interface

such

as

the

relation

browser

described

by

Marchionini

and

Brunk

The

relation

browser

allows

users

to

traverse

complex

data

sets

by

iteratively

slicing

the

data

along

several

topics.

In

Figure

we

see

a

prototype

instantiation

of

the

relation

browser,

applied

to

the

FedStats

The

relation

browser

supports

information

seeking

by

allowing

users

to

form

queries

in

a

stepwise

fashion,

slicing

and

re-slicing

the

data

as

their

interests

dictate.

Its

motivation

is

in

keeping

with

Shneiderman"s

suggestion

that

queries

and

their

results

should

be

tightly

coupled

Thus

in

Figure

Relation

Browser

Prototype

ure

users

might

limit

their

search

set

to

those

documents

about

energy.

Within

this

subset

of

the

collection,

they

might

further

eliminate

documents

published

more

than

a

year

ago.

Finally,

they

might

request

to

see

only

documents

published

in

PDF

format.

As

Marchionini

and

Brunk

discuss,

capturing

the

publication

date

and

format

of

documents

is

trivial.

But

successful

implementations

of

the

relation

browser

also

rely

on

topical

classification.

This

presents

two

stumbling

blocks

for

system

designers:

Information

architects

must

define

the

appropriate

set

of

topics

for

their

collection

Site

maintainers

must

classify

each

document

into

its

appropriate

categories

These

tasks

parallel

common

problems

in

the

metadata

community:

defining

appropriate

elements

and

marking

up

documents

to

support

metadata-aware

information

access.

Given

a

collection

of

over

documents,

these

hurdles

are

especially

daunting,

and

automatic

methods

of

approaching

them

are

highly

desirable.

A

Pre-Existing

Structure

Prior

to

our

involvement

with

the

project,

designers

at

BLS

created

a

shallow

classificatory

structure

for

the

most

important

documents

in

their

website.

As

seen

in

Figure

the

BLS

home

page

organizes

top-level

documents

into

categories.

These

include

topics

such

as

Employment

and

Unemployment,

Productivity,

and

Inflation

and

Spending.

Figure

The

BLS

Home

Page

We

hoped

initially

that

these

pre-defined

categories

could

be

used

to

train

a

document

classifier,

thus

automating

the

process

of

populating

the

relation

browser

altogether.

However,

this

approach

proved

unsatisfactory.

In

personal

meetings,

BLS

officials

voiced

dissatisfaction

with

the

existing

topics.

Their

form,

it

was

argued,

owed

as

much

to

the

institutional

structure

of

BLS

as

it

did

to

the

inherent

topology

of

the

website"s

information

space.

In

other

words,

the

topics

reflected

official

divisions

rather

than

semantic

clusters.

The

BLS

agents

suggested

that

re-designing

this

classification

structure

would

be

desirable.

The

agents"

misgivings

were

borne

out

in

subsequent

analysis.

The

BLS

topics

comprise

a

shallow

classificatory

structure;

each

of

the

top-level

categories

is

linked

to

a

small

number

of

related

pages.

Thus

there

are

pages

associated

with

Inflation.

Altogether,

the

link

structure

of

this

classificatory

system

contains

documents;

that

is,

excluding

navigational

links,

there

are

documents

linked

from

the

BLS

home

page,

where

each

hyperlink

connects

a

document

to

a

topic

(pages

can

be

linked

to

multiple

topics).

Based

on

this

hyperlink

structure,

we

defined

M,

a

symmetric

matrix,

where

mij

counts

the

number

of

topics

in

which

documents

i

and

j

are

both

classified

on

the

BLS

home

page.

To

analyze

the

redundancy

inherent

in

the

pre-existing

structure,

we

derived

the

principal

components

of

M

(cf.

Figure

shows

the

resultant

scree

Because

all

documents

belong

to

at

least

one

BLS

topic,

A

scree

plot

shows

the

magnitude

of

the

kth

eigenvalue

versus

its

rank.

During

principal

component

analysis

scree

plots

visualize

the

amount

of

variance

captured

by

each

component.

Eigenvalue

RankMEigenvalue

RankM

Eigenvalue

Rank

Eigenvlue

MagnitudeMEigenvlue

MagnitudeM

EigenvlueMagnitude

Figure

Scree

Plot

of

BLS

Categories

the

rank

of

M

is

guaranteed

to

be

less

than

or

equal

to

(hence,

eigenvalues

What

is

surprising

about

Figure

however,

is

the

precipitous

decline

in

magnitude

among

the

first

four

eigenvalues.

The

four

largest

eigenvlaues

account

for

of

the

total

variance

in

the

data.

This

fact

suggests

a

high

degree

of

redundancy

among

the

topics.

Topical

redundancy

is

not

in

itself

problematic.

However,

the

documents

in

this

very

shallow

classificatory

structure

are

almost

all

gateways

to

more

specific

information.

Thus

the

listing

of

the

Producer

Price

Index

under

three

categories

could

be

confusing

to

the

site"s

users.

In

light

of

this

potential

for

confusion

and

the

agency"s

own

request

for

redesign,

we

undertook

the

task

of

topic

discovery

described

in

the

following

sections.

A

HYBRID

APPROACH

TO

TOPIC

DISCOVERY

To

aid

in

the

discovery

of

a

new

set

of

high-level

topics

for

the

BLS

website,

we

turned

to

unsupervised

machine

learning

methods.

In

efforts

to

let

the

data

speak

for

themselves,

we

desired

a

means

of

concept

discovery

that

would

be

based

not

on

the

structure

of

the

agency,

but

on

the

content

of

the

material.

To

begin

this

process,

we

crawled

the

BLS

website,

downloading

all

documents

of

MIME

type

text/html.

This

led

to

a

corpus

of

documents.

Based

on

this

corpus,

we

hoped

to

derive

k

topical

categories,

such

that

each

document

di

is

assigned

to

one

or

more

classes.

Document

clustering

(cf.

provided

an

obvious,

but

only

partial

solution

to

the

problem

of

automating

this

type

of

high-level

information

architecture

discovery.

The

problems

with

standard

clustering

are

threefold.

Mutually

exclusive

clusters

are

inappropriate

for

identifying

the

topical

content

of

documents,

since

documents

may

be

about

many

subjects.

Due

to

the

heterogeneity

of

the

data

housed

in

the

BLS

collection

(tables,

lists,

surveys,

etc.),

many

documents"

terms

provide

noisy

topical

information.

For

application

to

the

relation

browser,

we

require

a

small

number

(k

of

topics.

Without

significant

data

reduction,

term-based

clustering

tends

to

deliver

clusters

at

too

fine

a

level

of

granularity.

In

light

of

these

problems,

we

take

a

hybrid

approach

to

topic

discovery.

First,

we

limit

the

clustering

process

to

a

sample

of

the

entire

collection,

described

in

Section

Working

on

a

focused

subset

of

the

data

helps

to

overcome

problems

two

and

three,

listed

above.

To

address

the

problem

of

mutual

exclusivity,

we

combine

unsupervised

with

supervised

learning

methods,

as

described

in

Section

FOCUSING

ON

CONTENT-RICH

DOCUMENTS

To

derive

empirically

evidenced

topics

we

initially

turned

to

cluster

analysis.

Let

A

be

the

n×p

data

matrix

with

n

observations

in

p

variables.

Thus

aij

shows

the

measurement

for

the

ith

observation

on

the

jth

variable.

As

described

in

the

goal

of

cluster

analysis

is

to

assign

each

of

the

n

observations

to

one

of

a

small

number

k

groups,

each

of

which

is

characterized

by

high

intra-cluster

correlation

and

low

inter-cluster

correlation.

Though

the

algorithms

for

accomplishing

such

an

arrangement

are

legion,

our

analysis

focuses

on

k-means

during

which,

each

observation

oi

is

assigned

to

the

cluster

Ck

whose

centroid

is

closest

to

it,

in

terms

of

Euclidean

distance.

Readers

interested

in

the

details

of

the

algorithm

are

referred

to

for

a

thorough

treatment

of

the

subject.

Clustering

by

k-means

is

well-studied

in

the

statistical

literature,

and

has

shown

good

results

for

text

analysis

(cf.

However,

k-means

clustering

requires

that

the

researcher

specify

k,

the

number

of

clusters

to

define.

When

applying

k-means

to

our

document

collection,

indicators

such

as

the

gap

statistic

and

an

analysis

of

the

mean-squared

distance

across

values

of

k

suggested

that

k

was

optimal.

This

paramterization

led

to

semantically

intelligible

clusters.

However,

clusters

are

far

too

many

for

application

to

an

interface

such

as

the

relation

We

have

focused

on

k-means

as

opposed

to

other

clustering

algorithms

for

several

reasons.

Chief

among

these

is

the

computational

efficiency

enjoyed

by

the

k-means

approach.

Because

we

need

only

a

flat

clustering

there

is

little

to

be

gained

by

the

more

expensive

hierarchical

algorithms.

In

future

work

we

will

turn

to

model-based

clustering

as

a

more

principled

method

of

selecting

the

number

of

clusters

and

of

representing

clusters.

browser.

Moreover,

the

granularity

of

these

clusters

was

unsuitably

fine.

For

instance,

the

solution

derived

a

cluster

whose

most

highly

associated

words

(in

terms

of

log-odds

ratio

were

drug,

pharmacy,

and

chemist.

These

words

are

certainly

related,

but

they

are

related

at

a

level

of

specificity

far

below

what

we

sought.

To

remedy

the

high

dimensionality

of

the

data,

we

resolved

to

limit

the

algorithm

to

a

subset

of

the

collection.

In

consultation

with

employees

of

the

BLS,

we

continued

our

analysis

on

documents

that

form

a

series

titled

From

the

Editor"s

These

are

brief

articles,

written

by

BLS

employees.

BLS

agents

suggested

that

we

focus

on

the

Editor"s

Desk

because

it

is

intended

to

span

the

intellectual

domain

of

the

agency.

The

column

is

published

daily,

and

each

entry

describes

an

important

current

issue

in

the

BLS

domain.

The

Editor"s

Desk

column

has

been

written

daily

(five

times

per

week)

since

As

such,

we

operated

on

a

set

of

N

documents.

Limiting

attention

to

these

documents

not

only

reduced

the

dimensionality

of

the

problem.

It

also

allowed

the

clustering

process

to

learn

on

a

relatively

clean

data

set.

While

the

entire

BLS

collection

contains

a

great

deal

of

nonprose

text

(i.e.

tables,

lists,

etc.),

the

Editor"s

Desk

documents

are

all

written

in

clear,

journalistic

prose.

Each

document

is

highly

topical,

further

aiding

the

discovery

of

termtopic

relations.

Finally,

the

Editor"s

Desk

column

provided

an

ideal

learning

environment

because

it

is

well-supplied

with

topical

metadata.

Each

of

the

documents

contains

a

list

of

one

or

more

keywords.

Additionally,

a

subset

of

the

documents

contained

a

subject

heading.

This

metadata

informed

our

learning

and

evaluation,

as

described

in

Section

COMBINING

SUPERVISED

AND

UNSUPERVISED

LEARNING

FORTOPIC

DISCOVERY

To

derive

suitably

general

topics

for

the

application

of

a

dynamic

interface

to

the

BLS

collection,

we

combined

document

clustering

with

text

classification

techniques.

Specifically,

using

k-means,

we

clustered

each

of

the

documents

into

one

of

k

clusters,

with

the

number

of

clusters

chosen

by

analyzing

the

within-cluster

mean

squared

distance

at

different

values

of

k

(see

Section

Constructing

mutually

exclusive

clusters

violates

our

assumption

that

documents

may

belong

to

multiple

classes.

However,

these

clusters

mark

only

the

first

step

in

a

two-phase

process

of

topic

identification.

At

the

end

of

the

process,

documentcluster

affinity

is

measured

by

a

real-valued

number.

Once

the

Editor"s

Desk

documents

were

assigned

to

clusters,

we

constructed

a

k-way

classifier

that

estimates

the

strength

of

evidence

that

a

new

document

di

is

a

member

of

class

Ck.

We

tested

three

statistical

classification

techniques:

probabilistic

Rocchio

(prind),

naive

Bayes,

and

support

vector

machines

(SVMs).

All

were

implemented

using

McCallum"s

BOW

text

classification

library

Prind

is

a

probabilistic

version

of

the

Rocchio

classification

algorithm

Interested

readers

are

referred

to

Joachims"

article

for

further

details

of

the

classification

method.

Like

prind,

naive

Bayes

attempts

to

classify

documents

into

the

most

probable

class.

It

is

described

in

detail

in

Finally,

support

vector

machines

were

thoroughly

explicated

by

Vapnik

and

applied

specifically

to

text

in

They

define

a

decision

boundary

by

finding

the

maximally

separating

hyperplane

in

a

high-dimensional

vector

space

in

which

document

classes

become

linearly

separable.

Having

clustered

the

documents

and

trained

a

suitable

classifier,

the

remaining

documents

in

the

collection

are

labeled

by

means

of

automatic

classification.

That

is,

for

each

document

di

we

derive

a

k-dimensional

vector,

quantifying

the

association

between

di

and

each

class

Ck.

Deriving

topic

scores

via

naive

Bayes

for

the

entire

collection

required

less

than

two

hours

of

CPU

time.

The

output

of

this

process

is

a

score

for

every

document

in

the

collection

on

each

of

the

automatically

discovered

topics.

These

scores

may

then

be

used

to

populate

a

relation

browser

interface,

or

they

may

be

added

to

a

traditional

information

retrieval

system.

To

use

these

weights

in

the

relation

browser

we

currently

assign

to

each

document

the

two

topics

on

which

it

scored

highest.

In

future

work

we

will

adopt

a

more

rigorous

method

of

deriving

documenttopic

weight

thresholds.

Also,

evaluation

of

the

utility

of

the

learned

topics

for

users

will

be

undertaken.

EVALUATION

OF

CONCEPT

DISCOVERY

Prior

to

implementing

a

relation

browser

interface

and

undertaking

the

attendant

user

studies,

it

is

of

course

important

to

evaluate

the

quality

of

the

inferred

concepts,

and

the

ability

of

the

automatic

classifier

to

assign

documents

to

the

appropriate

subjects.

To

evaluate

the

success

of

the

two-stage

approach

described

in

Section

we

undertook

two

experiments.

During

the

first

experiment

we

compared

three

methods

of

document

representation

for

the

clustering

task.

The

goal

here

was

to

compare

the

quality

of

document

clusters

derived

by

analysis

of

full-text

documents,

documents

represented

only

by

their

titles,

and

documents

represented

by

human-created

keyword

metadata.

During

the

second

experiment,

we

analyzed

the

ability

of

the

statistical

classifiers

to

discern

the

subject

matter

of

documents

from

portions

of

the

database

in

addition

to

the

Editor"s

Desk.

Comparing

Document

Representations

Documents

from

The

Editor"s

Desk

column

came

supplied

with

human-generated

keyword

metadata.

Additionally,

The

titles

of

the

Editor"s

Desk

documents

tend

to

be

germane

to

the

topic

of

their

respective

articles.

With

such

an

array

of

distilled

evidence

of

each

document"s

subject

matter,

we

undertook

a

comparison

of

document

representations

for

topic

discovery

by

clustering.

We

hypothesized

that

keyword-based

clustering

would

provide

a

useful

model.

But

we

hoped

to

see

whether

comparable

performance

could

be

attained

by

methods

that

did

not

require

extensive

human

indexing,

such

as

the

title-only

or

full-text

representations.

To

test

this

hypothesis,

we

defined

three

modes

of

document

representation-full-text,

title-only,

and

keyword

only-we

generated

three

sets

of

topics,

Tfull,

Ttitle,

and

Tkw,

respectively.

Topics

based

on

full-text

documents

were

derived

by

application

of

k-means

clustering

to

the

Editor"s

Desk

documents,

where

each

document

was

represented

by

a

vector.

These

dimensions

captured

the

TF.IDF

weights

of

each

term

ti

in

document

dj,

for

all

terms

that

occurred

at

least

three

times

in

the

data.

To

arrive

at

the

appropriate

number

of

clusters

for

these

data,

we

inspected

the

within-cluster

mean-squared

distance

for

each

value

of

k

As

k

approached

the

reduction

in

error

with

the

addition

of

more

clusters

declined

notably,

suggesting

that

k

would

yield

good

divisions.

To

select

a

single

integer

value,

we

calculated

which

value

of

k

led

to

the

least

variation

in

cluster

size.

This

metric

stemmed

from

a

desire

to

suppress

the

common

result

where

one

large

cluster

emerges

from

the

k-means

algorithm,

accompanied

by

several

accordingly

small

clusters.

Without

reason

to

believe

that

any

single

topic

should

have

dramatically

high

prior

odds

of

document

membership,

this

heuristic

led

to

kfull

Clusters

based

on

document

titles

were

constructed

similarly.

However,

in

this

case,

each

document

was

represented

in

the

vector

space

spanned

by

the

terms

that

occur

at

least

twice

in

document

titles.

Using

the

same

method

of

minimizing

the

variance

in

cluster

membership

ktitle-the

number

of

clusters

in

the

title-based

representation-was

also

set

to

The

dimensionality

of

the

keyword-based

clustering

was

very

similar

to

that

of

the

title-based

approach.

There

were

keywords

in

the

data,

all

of

which

were

retained.

The

median

number

of

keywords

per

document

was

where

a

keyword

is

understood

to

be

either

a

single

word,

or

a

multiword

term

such

as

consumer

price

index.

It

is

worth

noting

that

the

keywords

were

not

drawn

from

any

controlled

vocabulary;

they

were

assigned

to

documents

by

publishers

at

the

BLS.

Using

the

keywords,

the

documents

were

clustered

into

classes.

To

evaluate

the

clusters

derived

by

each

method

of

document

representation,

we

used

the

subject

headings

that

were

included

with

of

the

Editor"s

Desk

documents.

Each

of

these

documents

was

assigned

one

or

more

subject

headings,

which

were

withheld

from

all

of

the

cluster

applications.

Like

the

keywords,

subject

headings

were

assigned

to

documents

by

BLS

publishers.

Unlike

the

keywords,

however,

subject

headings

were

drawn

from

a

controlled

vocabulary.

Our

analysis

began

with

the

assumption

that

documents

with

the

same

subject

headings

should

cluster

together.

To

facilitate

this

analysis,

we

took

a

conservative

approach;

we

considered

multi-subject

classifications

to

be

unique.

Thus

if

document

di

was

assigned

to

a

single

subject

prices,

while

document

dj

was

assigned

to

two

subjects,

international

comparisons,

prices,

documents

di

and

dj

are

not

considered

to

come

from

the

same

class.

Table

shows

all

Editor"s

Desk

subject

headings

that

were

assigned

to

at

least

documents.

As

noted

in

the

table,

Table

Top

Editor"s

Desk

Subject

Headings

Subject

Count

prices

unemployment

occupational

safety

health

international

comparisons,

prices

manufacturing,

prices

employment

productivity

consumer

expenditures

earnings

wages

employment

unemployment

compensation

costs

earnings

wages,

metro.

areas

benefits,

compensation

costs

earnings

wages,

occupations

employment,

occupations

benefits

earnings

wage,

regions

work

stoppages

earnings

wages,

industries

Total

Table

Contingecy

Table

for

Three

Document

Representations

Representation

Right

Wrong

Accuracy

Full-text

Title

Keyword

there

were

such

subject

headings,

which

altogether

covered

of

the

documents

with

subjects

assigned.

These

document-subject

pairings

formed

the

basis

of

our

analysis.

Limiting

analysis

to

subjects

with

N

kept

the

resultant

χ2

tests

suitably

robust.

The

clustering

derived

by

each

document

representation

was

tested

by

its

ability

to

collocate

documents

with

the

same

subjects.

Thus

for

each

of

the

subject

headings

in

Table

Si,

we

calculated

the

proportion

of

documents

assigned

to

Si

that

each

clustering

co-classified.

Further,

we

assumed

that

whichever

cluster

captured

the

majority

of

documents

for

a

given

class

constituted

the

right

answer

for

that

class.

For

instance,

There

were

documents

whose

subject

heading

was

prices.

Taking

the

BLS

editors"

classifications

as

ground

truth,

all

of

these

documents

should

have

ended

up

in

the

same

cluster.

Under

the

full-text

representation

of

these

documents

were

clustered

into

category

while

were

in

category

and

documents

were

in

category

Taking

the

majority

cluster

as

the

putative

right

home

for

these

documents,

we

consider

the

accuracy

of

this

clustering

on

this

subject

to

be

Repeating

this

process

for

each

topic

across

all

three

representations

led

to

the

contingency

table

shown

in

Table

The

obvious

superiority

of

the

keyword-based

clustering

evidenced

by

Table

was

borne

out

by

a

χ2

test

on

the

accuracy

proportions.

Comparing

the

proportion

right

and

Table

Keyword-Based

Clusters

benefits

costs

international

jobs

plans

compensation

import

employment

benefits

costs

prices

jobs

employees

benefits

petroleum

youth

occupations

prices

productivity

safety

workers

prices

productivity

safety

earnings

index

output

health

operators

inflation

nonfarm

occupational

spending

unemployment

expenditures

unemployment

consumer

mass

spending

jobless

wrong

achieved

by

keyword

and

title-based

clustering

led

to

p

Due

to

this

result,

in

the

remainder

of

this

paper,

we

focus

our

attention

on

the

clusters

derived

by

analysis

of

the

Editor"s

Desk

keywords.

The

ten

keyword-based

clusters

are

shown

in

Table

represented

by

the

three

terms

most

highly

associated

with

each

cluster,

in

terms

of

the

log-odds

ratio.

Additionally,

each

cluster

has

been

given

a

label

by

the

researchers.

Evaluating

the

results

of

clustering

is

notoriously

difficult.

In

order

to

lend

our

analysis

suitable

rigor

and

utility,

we

made

several

simplifying

assumptions.

Most

problematic

is

the

fact

that

we

have

assumed

that

each

document

belongs

in

only

a

single

category.

This

assumption

is

certainly

false.

However,

by

taking

an

extremely

rigid

view

of

what

constitutes

a

subject-that

is,

by

taking

a

fully

qualified

and

often

multipart

subject

heading

as

our

unit

of

analysis-we

mitigate

this

problem.

Analogically,

this

is

akin

to

considering

the

location

of

books

on

a

library

shelf.

Although

a

given

book

may

cover

many

subjects,

a

classification

system

should

be

able

to

collocate

books

that

are

extremely

similar,

say

books

about

occupational

safety

and

health.

The

most

serious

liability

with

this

evaluation,

then,

is

the

fact

that

we

have

compressed

multiple

subject

headings,

say

prices

international

into

single

subjects.

This

flattening

obscures

the

multivalence

of

documents.

We

turn

to

a

more

realistic

assessment

of

document-class

relations

in

Section

Accuracy

of

the

Document

Classifiers

Although

the

keyword-based

clusters

appear

to

classify

the

Editor"s

Desk

documents

very

well,

their

discovery

only

solved

half

of

the

problem

required

for

the

successful

implementation

of

a

dynamic

user

interface

such

as

the

relation

browser.

The

matter

of

roughly

fourteen

thousand

unclassified

documents

remained

to

be

addressed.

To

solve

this

problem,

we

trained

the

statistical

classifiers

described

above

in

Section

For

each

document

in

the

collection

di,

these

classifiers

give

pi,

a

k-vector

of

probabilities

or

distances

(depending

on

the

classification

method

used),

where

pik

quantifies

the

strength

of

association

between

the

ith

document

and

the

kth

class.

All

classifiers

were

trained

on

the

full

text

of

each

document,

regardless

of

the

representation

used

to

discover

the

initial

clusters.

The

different

training

sets

were

thus

constructed

simply

by

changing

the

Table

Cross

Validation

Results

for

Classifiers

Method

Av.

Percent

Accuracy

SE

Prind

Naive

Bayes

SVM

class

variable

for

each

instance

(document)

to

reflect

its

assigned

cluster

under

a

given

model.

To

test

the

ability

of

each

classifier

to

locate

documents

correctly,

we

first

performed

a

cross

validation

on

the

Editor"s

Desk

documents.

During

cross-validation

the

data

are

split

randomly

into

n

subsets

(in

this

case

n

The

process

proceeds

by

iteratively

holding

out

each

of

the

n

subsets

as

a

test

collection

for

a

model

trained

on

the

remaining

n

subsets.

Cross

validation

is

described

in

Using

this

methodology,

we

compared

the

performance

of

the

three

classification

models

described

above.

Table

gives

the

results

from

cross

validation.

Although

naive

Bayes

is

not

significantly

more

accurate

for

these

data

than

the

SVM

classifier,

we

limit

the

remainder

of

our

attention

to

analysis

of

its

performance.

Our

selection

of

naive

Bayes

is

due

to

the

fact

that

it

appears

to

work

comparably

to

the

SVM

approach

for

these

data,

while

being

much

simpler,

both

in

theory

and

implementation.

Because

we

have

only

documents

and

classes,

the

number

of

training

documents

per

class

is

relatively

small.

In

addition

to

models

fitted

to

the

Editor"s

Desk

data,

then,

we

constructed

a

fourth

model,

supplementing

the

training

sets

of

each

class

by

querying

the

Google

search

and

applying

naive

Bayes

to

the

augmented

test

set.

For

each

class,

we

created

a

query

by

submitting

the

three

terms

with

the

highest

log-odds

ratio

with

that

class.

Further,

each

query

was

limited

to

the

domain

For

each

class

we

retrieved

up

to

documents

from

Google

(the

actual

number

varied

depending

on

the

size

of

the

result

set

returned

by

Google).

This

led

to

a

training

set

of

documents

in

the

augmented

model,

as

we

call

it

Cross

validation

suggested

that

the

augmented

model

decreased

classification

accuracy

(accuracy=

with

standard

error=

As

we

discuss

below,

however,

augmenting

the

training

set

appeared

to

help

generalization

during

our

second

experiment.

The

results

of

our

cross

validation

experiment

are

encouraging.

However,

the

success

of

our

classifiers

on

the

Editor"s

Desk

documents

that

informed

the

cross

validation

study

may

not

be

good

predictors

of

the

models"

performance

on

the

remainder

to

the

BLS

website.

To

test

the

generality

of

the

naive

Bayes

classifier,

we

solicited

input

from

human

judges

who

were

familiar

with

the

BLS

website.

The

sample

was

chosen

by

convenience,

and

consisted

of

faculty

and

graduate

students

who

work

on

the

GovStat

project.

However,

of

the

reviewers

had

prior

knowledge

of

the

outcome

of

the

classification

before

their

participation.

For

the

experiment,

a

random

sample

of

documents

was

drawn

from

the

entire

BLS

collection.

On

average

each

A

more

formal

treatment

of

the

combination

of

labeled

and

unlabeled

data

is

available

in

Table

Human-Model

Agreement

on

Sample

Docs.

Human

Judge

Choice

Model

Model

Choice

Model

Choice

N.

Bayes

(aug.)

N.

Bayes

Human

Judge

Choice

Model

Model

Choice

Model

Choice

N.

Bayes

(aug.)

N.

Bayes

viewer

classified

documents,

placing

each

document

into

as

many

of

the

categories

shown

in

Table

as

he

or

she

saw

fit.

Results

from

this

experiment

suggest

that

room

for

improvement

remains

with

respect

to

generalizing

to

the

whole

collection

from

the

class

models

fitted

to

the

Editor"s

Desk

documents.

In

Table

we

see,

for

each

classifier,

the

number

of

documents

for

which

it"s

first

or

second

most

probable

class

was

voted

best

or

second

best

by

the

human

judges.

In

the

context

of

this

experiment,

we

consider

a

first-

or

second-place

classification

by

the

machine

to

be

accurate

because

the

relation

browser

interface

operates

on

a

multiway

classification,

where

each

document

is

classified

into

multiple

categories.

Thus

a

document

with

the

correct

class

as

its

second

choice

would

still

be

easily

available

to

a

user.

Likewise,

a

correct

classification

on

either

the

most

popular

or

second

most

popular

category

among

the

human

judges

is

considered

correct

in

cases

where

a

given

document

was

classified

into

multiple

classes.

There

were

multiclass

documents

in

our

sample,

as

seen

in

Figure

The

remaining

documents

were

assigned

to

or

classes.

Under

this

rationale,

The

augmented

naive

Bayes

classifier

correctly

grouped

documents,

while

the

smaller

model

(not

augmented

by

a

Google

search)

correctly

classified

The

resultant

χ2

test

gave

p

suggesting

that

increasing

the

training

set

improved

the

ability

of

the

naive

Bayes

model

to

generalize

from

the

Editor"s

Desk

documents

to

the

collection

as

a

whole.

However,

the

improvement

afforded

by

the

augmented

model

comes

at

some

cost.

In

particular,

the

augmented

model

is

significantly

inferior

to

the

model

trained

solely

on

Editor"s

Desk

documents

if

we

concern

ourselves

only

with

documents

selected

by

the

majority

of

human

reviewers-i.e.

only

first-choice

classes.

Limiting

the

right

answers

to

the

left

column

of

Table

gives

p

in

favor

of

the

non-augmented

model.

For

the

purposes

of

applying

the

relation

browser

to

complex

digital

library

content

(where

documents

will

be

classified

along

multiple

categories),

the

augmented

model

is

preferable.

But

this

is

not

necessarily

the

case

in

general.

It

must

also

be

said

that

accuracy

under

a

fairly

liberal

test

condition

leaves

room

for

improvement

in

our

assignment

of

topics

to

categories.

We

may

begin

to

understand

the

shortcomings

of

the

described

techniques

by

consulting

Figure

which

shows

the

distribution

of

categories

across

documents

given

by

humans

and

by

the

augmented

naive

Bayes

model.

The

majority

of

reviewers

put

Number

of

Human-Assigned

ClassesMNumber

of

Human-Assigned

ClassesM

Number

of

Human-Assigned

Classes

FrequencyMFrequencyM

Frequency

Figure

Number

of

Classes

Assigned

to

Documents

by

Judges

documents

into

only

three

categories,

jobs,

benefits,

and

occupations.

On

the

other

hand,

the

naive

Bayes

classifier

distributed

classes

more

evenly

across

the

topics.

This

behavior

suggests

areas

for

future

improvement.

Most

importantly,

we

observed

a

strong

correlation

among

the

three

most

frequent

classes

among

the

human

judges

(for

instance,

there

was

correlation

between

benefits

and

occupations).

This

suggests

that

improving

the

clustering

to

produce

topics

that

were

more

nearly

orthogonal

might

improve

performance.

CONCLUSIONS

AND

FUTURE

WORK

Many

developers

and

maintainers

of

digital

libraries

share

the

basic

problem

pursued

here.

Given

increasingly

large,

complex

bodies

of

data,

how

may

we

improve

access

to

collections

without

incurring

extraordinary

cost,

and

while

also

keeping

systems

receptive

to

changes

in

content

over

time?

Data

mining

and

machine

learning

methods

hold

a

great

deal

of

promise

with

respect

to

this

problem.

Empirical

methods

of

knowledge

discovery

can

aid

in

the

organization

and

retrieval

of

information.

As

we

have

argued

in

this

paper,

these

methods

may

also

be

brought

to

bear

on

the

design

and

implementation

of

advanced

user

interfaces.

This

study

explored

a

hybrid

technique

for

aiding

information

architects

as

they

implement

dynamic

interfaces

such

as

the

relation

browser.

Our

approach

combines

unsupervised

learning

techniques,

applied

to

a

focused

subset

of

the

BLS

website.

The

goal

of

this

initial

stage

is

to

discover

the

most

basic

and

far-reaching

topics

in

the

collection.

Based

mjobsjobsMjobsM

jobs

benefitsunemploymentpricespricesMpricesM

prices

safetyinternationalspendingspendingMspendingM

spending

occupationscostscostsMcostsM

costs

productivityHuman

ClassificationsMHuman

ClassificationsM

Human

Classifications

jobs

prices

spending

occupationscostscostsMcostsM

costs

productivityMachine

ClassificationsMMachine

ClassificationsM

Machine

Classifications

Figure

Distribution

of

Classes

Across

Documents

on

a

statistical

model

of

these

topics,

the

second

phase

of

our

approach

uses

supervised

learning

(in

particular,

a

naive

Bayes

classifier,

trained

on

individual

words),

to

assign

topical

relations

to

the

remaining

documents

in

the

collection.

In

the

study

reported

here,

this

approach

has

demonstrated

promise.

In

its

favor,

our

approach

is

highly

scalable.

It

also

appears

to

give

fairly

good

results.

Comparing

three

modes

of

document

representation-full-text,

title

only,

and

keyword-we

found

accuracy

as

measured

by

collocation

of

documents

with

identical

subject

headings.

While

it

is

not

surprising

that

editor-generated

keywords

should

give

strong

evidence

for

such

learning,

their

superiority

over

fulltext

and

titles

was

dramatic,

suggesting

that

even

a

small

amount

of

metadata

can

be

very

useful

for

data

mining.

However,

we

also

found

evidence

that

learning

topics

from

a

subset

of

the

collection

may

lead

to

overfitted

models.

After

clustering

Editor"s

Desk

documents

into

categories,

we

fitted

a

naive

Bayes

classifier

to

categorize

the

remaining

documents

in

the

collection.

While

we

saw

fairly

good

results

(classification

accuracy

of

with

respect

to

a

small

sample

of

human

judges),

this

experiment

forced

us

to

reconsider

the

quality

of

the

topics

learned

by

clustering.

The

high

correlation

among

human

judgments

in

our

sample

suggests

that

the

topics

discovered

by

analysis

of

the

Editor"s

Desk

were

not

independent.

While

we

do

not

desire

mutually

exclusive

categories

in

our

setting,

we

do

desire

independence

among

the

topics

we

model.

Overall,

then,

the

techniques

described

here

provide

an

encouraging

start

to

our

work

on

acquiring

subject

metadata

for

dynamic

interfaces

automatically.

It

also

suggests

that

a

more

sophisticated

modeling

approach

might

yield

better

results

in

the

future.

In

upcoming

work

we

will

experiment

with

streamlining

the

two-phase

technique

described

here.

Instead

of

clustering

documents

to

find

topics

and

then

fitting

a

model

to

the

learned

clusters,

our

goal

is

to

expand

the

unsupervised

portion

of

our

analysis

beyond

a

narrow

subset

of

the

collection,

such

as

The

Editor"s

Desk.

In

current

work

we

have

defined

algorithms

to

identify

documents

likely

to

help

the

topic

discovery

task.

Supplied

with

a

more

comprehensive

training

set,

we

hope

to

experiment

with

model-based

clustering,

which

combines

the

clustering

and

classification

processes

into

a

single

modeling

procedure.

Topic

discovery

and

document

classification

have

long

been

recognized

as

fundamental

problems

in

information

retrieval

and

other

forms

of

text

mining.

What

is

increasingly

clear,

however,

as

digital

libraries

grow

in

scope

and

complexity,

is

the

applicability

of

these

techniques

to

problems

at

the

front-end

of

systems

such

as

information

architecture

and

interface

design.

Finally,

then,

in

future

work

we

will

build

on

the

user

studies

undertaken

by

Marchionini

and

Brunk

in

efforts

to

evaluate

the

utility

of

automatically

populated

dynamic

interfaces

for

the

users

of

digital

libraries.

REFERENCES

A.

Agresti.

An

Introduction

to

Categorical

Data

Analysis.

Wiley,

New

York,

C.

Ahlberg,

C.

Williamson,

and

B.

Shneiderman.

Dynamic

queries

for

information

exploration:

an

implementation

and

evaluation.

In

Proceedings

of

the

SIGCHI

conference

on

Human

factors

in

computing

systems,

pages

R.

Baeza-Yates

and

B.

Ribeiro-Neto.

Modern

Information

Retrieval.

ACM

Press,

A.

Blum

and

T.

Mitchell.

Combining

labeled

and

unlabeled

data

with

co-training.

In

Proceedings

of

the

eleventh

annual

conference

on

Computational

learning

theory,

pages

ACM

Press,

H.

Chen

and

S.

Dumais.

Hierarchical

classification

of

web

content.

In

Proceedings

of

the

annual

international

ACM

SIGIR

conference

on

Research

and

development

in

information

retrieval,

pages

M.

Efron,

G.

Marchionini,

and

J.

Zhang.

Implications

of

the

recursive

representation

problem

for

automatic

concept

identification

in

on-line

governmental

information.

In

Proceedings

of

the

ASIST

Special

Interest

Group

on

Classification

Research

(ASIST

SIG-CR),

C.

Fraley

and

A.

E.

Raftery.

How

many

clusters?

which

clustering

method?

answers

via

model-based

cluster

analysis.

The

Computer

Journal,

A.

K.

Jain,

M.

N.

Murty,

and

P.

J.

Flynn.

Data

clustering:

a

review.

ACM

Computing

Surveys,

September

T.

Joachims.

A

probabilistic

analysis

of

the

Rocchio

algorithm

with

TFIDF

for

text

categorization.

In

D.

H.

Fisher,

editor,

Proceedings

of

International

Conference

on

Machine

Learning,

pages

Nashville,

US,

Morgan

Kaufmann

Publishers,

San

Francisco,

US.

T.

Joachims.

Text

categorization

with

support

vector

machines:

learning

with

many

relevant

features.

In

C.

N´edellec

and

C.

Rouveirol,

editors,

Proceedings

of

European

Conference

on

Machine

Learning,

pages

Chemnitz,

DE,

Springer

Verlag,

Heidelberg,

DE.

I.

T.

Jolliffe.

Principal

Component

Analysis.

Springer,

edition,

L.

Kaufman

and

P.

J.

Rosseeuw.

Finding

Groups

in

Data:

an

Introduction

to

Cluster

Analysis.

Wiley,

G.

Marchionini

and

B.

Brunk.

Toward

a

general

relation

browser:

a

GUI

for

information

architects.

Journal

of

Digital

Information,

A.

K.

McCallum.

Bow:

A

toolkit

for

statistical

language

modeling,

text

retrieval,

classification

and

clustering.

T.

Mitchell.

Machine

Learning.

McGraw

Hill,

E.

Rasmussen.

Clustering

algorithms.

In

W.

B.

Frakes

and

R.

Baeza-Yates,

editors,

Information

Retrieval:

Data

Structures

and

Algorithms,

pages

Prentice

Hall,

R.

Tibshirani,

G.

Walther,

and

T.

Hastie.

Estimating

the

number

of

clusters

in

a

dataset

via

the

gap

statistic,

V.

N.

Vapnik.

The

Nature

of

Statistical

Learning

Theory.

Springer,

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

supervised and unsupervised learning technique

access
complex digital library
bureau of labor statistics
eigenvalue
bls collection
multiway classification
information architecture
interface design


Demonstration

of

Grid-Enabled

Ensemble

Kalman

Filter

Data

Assimilation

Methodology

for

Reservoir

Characterization

Ravi

Vadapalli

High

Performance

Computing

Center

Texas

Tech

University

Lubbock,

TX

Ravi.Vadapalli@ttu.edu

Ajitabh

Kumar

Department

of

Petroleum

Engineering

Texas

A&M

University

College

Station,

TX

akumar@tamu.edu

Ping

Luo

Supercomputing

Facility

Texas

A&M

University

College

Station,

TX

pingluo@sc.tamu.edu

Shameem

Siddiqui

Department

of

Petroleum

Engineering

Texas

Tech

University

Lubbock,

TX

Shameem.Siddiqui@ttu.edu

Taesung

Kim

Supercomputing

Facility

Texas

A&M

University

College

Station,

TX

tskim@sc.tamu.edu

ABSTRACT

Ensemble

Kalman

filter

data

assimilation

methodology

is

a

popular

approach

for

hydrocarbon

reservoir

simulations

in

energy

exploration.

In

this

approach,

an

ensemble

of

geological

models

and

production

data

of

oil

fields

is

used

to

forecast

the

dynamic

response

of

oil

wells.

The

Schlumberger

ECLIPSE

software

is

used

for

these

simulations.

Since

models

in

the

ensemble

do

not

communicate,

message-passing

implementation

is

a

good

choice.

Each

model

checks

out

an

ECLIPSE

license

and

therefore,

parallelizability

of

reservoir

simulations

depends

on

the

number

licenses

available.

We

have

Grid-enabled

the

ensemble

Kalman

filter

data

assimilation

methodology

for

the

TIGRE

Grid

computing

environment.

By

pooling

the

licenses

and

computing

resources

across

the

collaborating

institutions

using

GridWay

metascheduler

and

TIGRE

environment,

the

computational

accuracy

can

be

increased

while

reducing

the

simulation

runtime.

In

this

paper,

we

provide

an

account

of

our

efforts

in

Gridenabling

the

ensemble

Kalman

Filter

data

assimilation

methodology.

Potential

benefits

of

this

approach,

observations

and

lessons

learned

will

be

discussed.

Categories

and

Subject

Descriptors

C

[Distributed

Systems]:

Distributed

applications

General

Terms

Algorithms,

Design,

Performance

INTRODUCTION

Grid

computing

is

an

emerging

collaborative

computing

paradigm

to

extend

institution/organization

specific

high

performance

computing

(HPC)

capabilities

greatly

beyond

local

resources.

Its

importance

stems

from

the

fact

that

ground

breaking

research

in

strategic

application

areas

such

as

bioscience

and

medicine,

energy

exploration

and

environmental

modeling

involve

strong

interdisciplinary

components

and

often

require

intercampus

collaborations

and

computational

capabilities

beyond

institutional

limitations.

The

Texas

Internet

Grid

for

Research

and

Education

(TIGRE)

is

a

state

funded

cyberinfrastructure

development

project

carried

out

by

five

(Rice,

A&M,

TTU,

UH

and

UT

Austin)

major

university

systems

collectively

called

TIGRE

Institutions.

The

purpose

of

TIGRE

is

to

create

a

higher

education

Grid

to

sustain

and

extend

research

and

educational

opportunities

across

Texas.

TIGRE

is

a

project

of

the

High

Performance

Computing

across

Texas

(HiPCAT)

consortium.

The

goal

of

HiPCAT

is

to

support

advanced

computational

technologies

to

enhance

research,

development,

and

educational

activities.

The

primary

goal

of

TIGRE

is

to

design

and

deploy

state-of-the-art

Grid

middleware

that

enables

integration

of

computing

systems,

storage

systems

and

databases,

visualization

laboratories

and

displays,

and

even

instruments

and

sensors

across

Texas.

The

secondary

goal

is

to

demonstrate

the

TIGRE

capabilities

to

enhance

research

and

educational

opportunities

in

strategic

application

areas

of

interest

to

the

State

of

Texas.

These

are

bioscience

and

medicine,

energy

exploration

and

air

quality

modeling.

Vision

of

the

TIGRE

project

is

to

foster

interdisciplinary

and

intercampus

collaborations,

identify

novel

approaches

to

extend

academic-government-private

partnerships,

and

become

a

competitive

model

for

external

funding

opportunities.

The

overall

goal

of

TIGRE

is

to

support

local,

campus

and

regional

user

interests

and

offer

avenues

to

connect

with

national

Grid

projects

such

as

Open

Science

Grid

and

TeraGrid

Within

the

energy

exploration

strategic

application

area,

we

have

Grid-enabled

the

ensemble

Kalman

Filter

(EnKF)

approach

for

data

assimilation

in

reservoir

modeling

and

demonstrated

the

extensibility

of

the

application

using

the

TIGRE

environment

and

the

GridWay

metascheduler.

Section

provides

an

overview

of

the

TIGRE

environment

and

capabilities.

Application

description

and

the

need

for

Grid-enabling

EnKF

methodology

is

provided

in

Section

The

implementation

details

and

merits

of

our

approach

are

discussed

in

Section

Conclusions

are

provided

in

Section

Finally,

observations

and

lessons

learned

are

documented

in

Section

TIGRE

ENVIRONMENT

The

TIGRE

Grid

middleware

consists

of

minimal

set

of

components

derived

from

a

subset

of

the

Virtual

Data

Toolkit

(VDT)

which

supports

a

variety

of

operating

systems.

The

purpose

of

choosing

a

minimal

software

stack

is

to

support

applications

at

hand,

and

to

simplify

installation

and

distribution

of

client/server

stacks

across

TIGRE

sites.

Additional

components

will

be

added

as

they

become

necessary.

The

PacMan

packaging

and

distribution

mechanism

is

employed

for

TIGRE

client/server

installation

and

management.

The

PacMan

distribution

mechanism

involves

retrieval,

installation,

and

often

configuration

of

the

packaged

software.

This

approach

allows

the

clients

to

keep

current,

consistent

versions

of

TIGRE

software.

It

also

helps

TIGRE

sites

to

install

the

needed

components

on

resources

distributed

throughout

the

participating

sites.

The

TIGRE

client/server

stack

consists

of

an

authentication

and

authorization

layer,

Globus

job

submission

via

web

services

(pre-web

services

installations

are

available

up

on

request).

The

tools

for

handling

Grid

proxy

generation,

Grid-enabled

file

transfer

and

Grid-enabled

remote

login

are

supported.

The

pertinent

details

of

TIGRE

services

and

tools

for

job

scheduling

and

management

are

provided

below.

Certificate

Authority

The

TIGRE

security

infrastructure

includes

a

certificate

authority

(CA)

accredited

by

the

International

Grid

Trust

Federation

(IGTF)

for

issuing

X.

user

and

resource

Grid

certificates

The

Texas

Advanced

Computing

Center

(TACC),

University

of

Texas

at

Austin

is

the

TIGRE"s

shared

CA.

The

TIGRE

Institutions

serve

as

Registration

Authorities

(RA)

for

their

respective

local

user

base.

For

up-to-date

information

on

securing

user

and

resource

certificates

and

their

installation

instructions

see

ref

The

users

and

hosts

on

TIGRE

are

identified

by

their

distinguished

name

(DN)

in

their

certificate

provided

by

the

CA.

A

native

Grid-mapfile

that

contains

a

list

of

authorized

DNs

is

used

to

authenticate

and

authorize

user

job

scheduling

and

management

on

TIGRE

site

resources.

At

Texas

Tech

University,

the

users

are

dynamically

allocated

one

of

the

many

generic

pool

accounts.

This

is

accomplished

through

the

Grid

User

Management

System

(GUMS)

Job

Scheduling

and

Management

The

TIGRE

environment

supports

job

submission

via

web

services.

The

job

submission

scripts

are

generated

using

XML.

The

web

services

GRAM

translates

the

XML

scripts

into

target

cluster

specific

batch

schedulers

such

as

LSF,

PBS,

or

SGE.

The

high

bandwidth

file

transfer

protocols

such

as

GridFTP

are

utilized

for

staging

files

in

and

out

of

the

target

machine.

The

login

to

remote

hosts

for

compilation

and

debugging

is

only

through

GSISSH

service

which

requires

resource

authentication

through

certificates.

The

authentication

and

authorization

of

Grid

jobs

are

managed

by

issuing

Grid

certificates

to

both

users

and

hosts.

The

certificate

revocation

lists

(CRL)

are

updated

on

a

daily

basis

to

maintain

high

security

standards

of

the

TIGRE

Grid

services.

The

TIGRE

portal

documentation

area

provides

a

quick

start

tutorial

on

running

jobs

on

TIGRE.

Metascheduler

The

metascheduler

interoperates

with

the

cluster

level

batch

schedulers

(such

as

LSF,

PBS)

in

the

overall

Grid

workflow

management.

In

the

present

work,

we

have

employed

GridWay

metascheduler

a

Globus

incubator

project

to

schedule

and

manage

jobs

across

TIGRE.

The

GridWay

is

a

light-weight

metascheduler

that

fully

utilizes

Globus

functionalities.

It

is

designed

to

provide

efficient

use

of

dynamic

Grid

resources

by

multiple

users

for

Grid

infrastructures

built

on

top

of

Globus

services.

The

TIGRE

site

administrator

can

control

the

resource

sharing

through

a

powerful

built-in

scheduler

provided

by

GridWay

or

by

extending

GridWay"s

external

scheduling

module

to

provide

their

own

scheduling

policies.

Application

users

can

write

job

descriptions

using

GridWay"s

simple

and

direct

job

template

format

(see

Section

for

details)

or

standard

Job

Submission

Description

Language

(JSDL).

See

section

for

implementation

details.

Customer

Service

Management

System

A

TIGRE

portal

was

designed

and

deployed

to

interface

users

and

resource

providers.

It

was

designed

using

GridPort

and

is

maintained

by

TACC.

The

TIGRE

environment

is

supported

by

open

source

tools

such

as

the

Open

Ticket

Request

System

(OTRS)

for

servicing

trouble

tickets,

and

MoinMoin

Wiki

for

TIGRE

content

and

knowledge

management

for

education,

outreach

and

training.

The

links

for

OTRS

and

Wiki

are

consumed

by

the

TIGRE

portal

the

gateway

for

users

and

resource

providers.

The

TIGRE

resource

status

and

loads

are

monitored

by

the

Grid

Port

Information

Repository

(GPIR)

service

of

the

GridPort

toolkit

which

interfaces

with

local

cluster

load

monitoring

service

such

as

Ganglia.

The

GPIR

utilizes

cron

jobs

on

each

resource

to

gather

site

specific

resource

characteristics

such

as

jobs

that

are

running,

queued

and

waiting

for

resource

allocation.

ENSEMBLE

KALMAN

FILTER

APPLICATION

The

main

goal

of

hydrocarbon

reservoir

simulations

is

to

forecast

the

production

behavior

of

oil

and

gas

field

(denoted

as

field

hereafter)

for

its

development

and

optimal

management.

In

reservoir

modeling,

the

field

is

divided

into

several

geological

models

as

shown

in

Figure

For

accurate

performance

forecasting

of

the

field,

it

is

necessary

to

reconcile

several

geological

models

to

the

dynamic

response

of

the

field

through

history

matching

Figure

Cross-sectional

view

of

the

Field.

Vertical

layers

correspond

to

different

geological

models

and

the

nails

are

oil

wells

whose

historical

information

will

be

used

for

forecasting

the

production

behavior.

(Figure

The

EnKF

is

a

Monte

Carlo

method

that

works

with

an

ensemble

of

reservoir

models.

This

method

utilizes

crosscovariances

between

the

field

measurements

and

the

reservoir

model

parameters

(derived

from

several

models)

to

estimate

prediction

uncertainties.

The

geological

model

parameters

in

the

ensemble

are

sequentially

updated

with

a

goal

to

minimize

the

prediction

uncertainties.

Historical

production

response

of

the

field

for

over

years

is

used

in

these

simulations.

The

main

advantage

of

EnKF

is

that

it

can

be

readily

linked

to

any

reservoir

simulator,

and

can

assimilate

latest

production

data

without

the

need

to

re-run

the

simulator

from

initial

conditions.

Researchers

in

Texas

are

large

subscribers

of

the

Schlumberger

ECLIPSE

package

for

reservoir

simulations.

In

the

reservoir

modeling,

each

geological

model

checks

out

an

ECLIPSE

license.

The

simulation

runtime

of

the

EnKF

methodology

depends

on

the

number

of

geological

models

used,

number

of

ECLIPSE

licenses

available,

production

history

of

the

field,

and

propagated

uncertainties

in

history

matching.

The

overall

EnKF

workflow

is

shown

Figure

Figure

Ensemble

Kaman

Filter

Data

Assimilation

Workflow.

Each

site

has

L

licenses.

At

START,

the

master/control

process

(EnKF

main

program)

reads

the

simulation

configuration

file

for

number

(N)

of

models,

and

model-specific

input

files.

Then,

N

working

directories

are

created

to

store

the

output

files.

At

the

end

of

iteration,

the

master/control

process

collects

the

output

files

from

N

models

and

post

processes

crosscovariances

to

estimate

the

prediction

uncertainties.

This

information

will

be

used

to

update

models

(or

input

files)

for

the

next

iteration.

The

simulation

continues

until

the

production

histories

are

exhausted.

Typical

EnKF

simulation

with

and

field

histories

of

years,

in

time

steps

ranging

from

three

months

to

a

year,

takes

about

three

weeks

on

a

serial

computing

environment.

In

parallel

computing

environment,

there

is

no

interprocess

communication

between

the

geological

models

in

the

ensemble.

However,

at

the

end

of

each

simulation

time-step,

model-specific

output

files

are

to

be

collected

for

analyzing

cross

covariances

and

to

prepare

next

set

of

input

files.

Therefore,

master-slave

model

in

messagepassing

(MPI)

environment

is

a

suitable

paradigm.

In

this

approach,

the

geological

models

are

treated

as

slaves

and

are

distributed

across

the

available

processors.

The

master

Cluster

or

(TIGRE/GridWay)

START

Read

Configuration

File

Create

N

Working

Directories

Create

N

Input

files

Model

l

Model

Model

N.

ECLIPSE

on

site

A

ECLIPSE

on

Site

B

ECLIPSE

on

Site

Z

Collect

N

Model

Outputs,

Post-process

Output

files

END

process

collects

model-specific

output

files,

analyzes

and

prepares

next

set

of

input

files

for

the

simulation.

Since

each

geological

model

checks

out

an

ECLIPSE

license,

parallelizability

of

the

simulation

depends

on

the

number

of

licenses

available.

When

the

available

number

of

licenses

is

less

than

the

number

of

models

in

the

ensemble,

one

or

more

of

the

nodes

in

the

MPI

group

have

to

handle

more

than

one

model

in

a

serial

fashion

and

therefore,

it

takes

longer

to

complete

the

simulation.

A

Petroleum

Engineering

Department

usually

procures

ECLIPSE

licenses

while

at

least

ten-fold

increase

in

the

number

of

licenses

would

be

necessary

for

industry

standard

simulations.

The

number

of

licenses

can

be

increased

by

involving

several

Petroleum

Engineering

Departments

that

support

ECLIPSE

package.

Since

MPI

does

not

scale

very

well

for

applications

that

involve

remote

compute

clusters,

and

to

get

around

the

firewall

issues

with

license

servers

across

administrative

domains,

Grid-enabling

the

EnKF

workflow

seems

to

be

necessary.

With

this

motivation,

we

have

implemented

Grid-enabled

EnKF

workflow

for

the

TIGRE

environment

and

demonstrated

parallelizability

of

the

application

across

TIGRE

using

GridWay

metascheduler.

Further

details

are

provided

in

the

next

section.

IMPLEMENTATION

DETAILS

To

Grid-enable

the

EnKF

approach,

we

have

eliminated

the

MPI

code

for

parallel

processing

and

replaced

with

N

single

processor

jobs

(or

sub-jobs)

where,

N

is

the

number

of

geological

models

in

the

ensemble.

These

model-specific

sub-jobs

were

distributed

across

TIGRE

sites

that

support

ECLIPSE

package

using

the

GridWay

metascheduler.

For

each

sub-job,

we

have

constructed

a

GridWay

job

template

that

specifies

the

executable,

input

and

output

files,

and

resource

requirements.

Since

the

TIGRE

compute

resources

are

not

expected

to

change

frequently,

we

have

used

static

resource

discovery

policy

for

GridWay

and

the

sub-jobs

were

scheduled

dynamically

across

the

TIGRE

resources

using

GridWay.

Figure

represents

the

sub-job

template

file

for

the

GridWay

metascheduler.

Figure

GridWay

Sub-Job

Template

In

Figure

REQUIREMENTS

flag

is

set

to

choose

the

resources

that

satisfy

the

application

requirements.

In

the

case

of

EnKF

application,

for

example,

we

need

resources

that

support

ECLIPSE

package.

ARGUMENTS

flag

specifies

the

model

in

the

ensemble

that

will

invoke

ECLIPSE

at

a

remote

site.

is

prepared

by

the

EnKF

main

program

(or

master/control

process)

and

is

transferred

by

GridWay

to

the

remote

site

where

it

is

untared

and

is

prepared

for

execution.

Finally,

specifies

the

name

and

location

where

the

output

files

are

to

be

written.

The

command-line

features

of

GridWay

were

used

to

collect

and

process

the

model-specific

outputs

to

prepare

new

set

of

input

files.

This

step

mimics

MPI

process

synchronization

in

master-slave

model.

At

the

end

of

each

iteration,

the

compute

resources

and

licenses

are

committed

back

to

the

pool.

Table

shows

the

sub-jobs

in

TIGRE

Grid

via

GridWay

using

gwps

command

and

for

clarity,

only

selected

columns

were

shown

USER

JID

DM

EM

NAME

HOST

pingluo

wrap

pend

enkf.jt

antaeus.hpcc.ttu.edu/LSF

pingluo

wrap

pend

enkf.jt

antaeus.hpcc.ttu.edu/LSF

pingluo

wrap

actv

enkf.jt

minigar.hpcc.ttu.edu/LSF

pingluo

wrap

pend

enkf.jt

minigar.hpcc.ttu.edu/LSF

pingluo

wrap

done

enkf.jt

cosmos.tamu.edu/PBS

pingluo

wrap

epil

enkf.jt

cosmos.tamu.edu/PBS

Table

Job

scheduling

across

TIGRE

using

GridWay

Metascheduler.

DM:

Dispatch

state,

EM:

Execution

state,

JID

is

the

job

id

and

HOST

corresponds

to

site

specific

cluster

and

its

local

batch

scheduler.

When

a

job

is

submitted

to

GridWay,

it

will

go

through

a

series

of

dispatch

(DM)

and

execution

(EM)

states.

For

DM,

the

states

include

pend(ing),

prol(og),

wrap(per),

epil(og),

and

done.

DM=prol

means

the

job

has

been

scheduled

to

a

resource

and

the

remote

working

directory

is

in

preparation.

DM=warp

implies

that

GridWay

is

executing

the

wrapper

which

in

turn

executes

the

application.

DM=epil

implies

the

job

has

finished

running

at

the

remote

site

and

results

are

being

transferred

back

to

the

GridWay

server.

Similarly,

when

EM=pend

implies

the

job

is

waiting

in

the

queue

for

resource

and

the

job

is

running

when

EM=actv.

For

complete

list

of

message

flags

and

their

descriptions,

see

the

documentation

in

ref

We

have

demonstrated

the

Grid-enabled

EnKF

runs

using

GridWay

for

TIGRE

environment.

The

jobs

are

so

chosen

that

the

runtime

doesn"t

exceed

more

than

a

half

hour.

The

simulation

runs

involved

up

to

jobs

between

A&M

and

TTU

sites

with

TTU

serving

licenses.

For

resource

information,

see

Table

I.

One

of

the

main

advantages

of

Grid-enabled

EnKF

simulation

is

that

both

the

resources

and

licenses

are

released

back

to

the

pool

at

the

end

of

each

simulation

time

step

unlike

in

the

case

of

MPI

implementation

where

licenses

and

nodes

are

locked

until

the

completion

of

entire

simulation.

However,

the

fact

that

each

sub-job

gets

scheduled

independently

via

GridWay

could

possibly

incur

another

time

delay

caused

by

waiting

in

queue

for

execution

in

each

simulation

time

step.

Such

delays

are

not

expected

EXECUTABLE=runFORWARD

REQUIREMENTS=HOSTNAME=cosmos.tamu.edu

HOSTNAME=antaeus.hpcc.ttu.edu

HOSTNAME=minigar.hpcc.ttu.edu

in

MPI

implementation

where

the

node

is

blocked

for

processing

sub-jobs

(model-specific

calculation)

until

the

end

of

the

simulation.

There

are

two

main

scenarios

for

comparing

Grid

and

cluster

computing

approaches.

Scenario

I:

The

cluster

is

heavily

loaded.

The

conceived

average

waiting

time

of

job

requesting

large

number

of

CPUs

is

usually

longer

than

waiting

time

of

jobs

requesting

single

CPU.

Therefore,

overall

waiting

time

could

be

shorter

in

Grid

approach

which

requests

single

CPU

for

each

sub-job

many

times

compared

to

MPI

implementation

that

requests

large

number

of

CPUs

at

a

single

time.

It

is

apparent

that

Grid

scheduling

is

beneficial

especially

when

cluster

is

heavily

loaded

and

requested

number

of

CPUs

for

the

MPI

job

is

not

readily

available.

Scenario

II:

The

cluster

is

relatively

less

loaded

or

largely

available.

It

appears

the

MPI

implementation

is

favorable

compared

to

the

Grid

scheduling.

However,

parallelizability

of

the

EnKF

application

depends

on

the

number

of

ECLIPSE

licenses

and

ideally,

the

number

of

licenses

should

be

equal

to

the

number

of

models

in

the

ensemble.

Therefore,

if

a

single

institution

does

not

have

sufficient

number

of

licenses,

the

cluster

availability

doesn"t

help

as

much

as

it

is

expected.

Since

the

collaborative

environment

such

as

TIGRE

can

address

both

compute

and

software

resource

requirements

for

the

EnKF

application,

Grid-enabled

approach

is

still

advantageous

over

the

conventional

MPI

implementation

in

any

of

the

above

scenarios.

CONCLUSIONS

AND

FUTURE

WORK

TIGRE

is

a

higher

education

Grid

development

project

and

its

purpose

is

to

sustain

and

extend

research

and

educational

opportunities

across

Texas.

Within

the

energy

exploration

application

area,

we

have

Grid-enabled

the

MPI

implementation

of

the

ensemble

Kalman

filter

data

assimilation

methodology

for

reservoir

characterization.

This

task

was

accomplished

by

removing

MPI

code

for

parallel

processing

and

replacing

with

single

processor

jobs

one

for

each

geological

model

in

the

ensemble.

These

single

processor

jobs

were

scheduled

across

TIGRE

via

GridWay

metascheduler.

We

have

demonstrated

that

by

pooling

licenses

across

TIGRE

sites,

more

geological

models

can

be

handled

in

parallel

and

therefore

conceivably

better

simulation

accuracy.

This

approach

has

several

advantages

over

MPI

implementation

especially

when

a

site

specific

cluster

is

heavily

loaded

and/or

the

number

licenses

required

for

the

simulation

is

more

than

those

available

at

a

single

site.

Towards

the

future

work,

it

would

be

interesting

to

compare

the

runtime

between

MPI,

and

Grid

implementations

for

the

EnKF

application.

This

effort

could

shed

light

on

quality

of

service

(QoS)

of

Grid

environments

in

comparison

with

cluster

computing.

Another

aspect

of

interest

in

the

near

future

would

be

managing

both

compute

and

license

resources

to

address

the

job

(or

processor)-to-license

ratio

management.

OBSERVATIONS

AND

LESSIONS

LEARNED

The

Grid-enabling

efforts

for

EnKF

application

have

provided

ample

opportunities

to

gather

insights

on

the

visibility

and

promise

of

Grid

computing

environments

for

application

development

and

support.

The

main

issues

are

industry

standard

data

security

and

QoS

comparable

to

cluster

computing.

Since

the

reservoir

modeling

research

involves

proprietary

data

of

the

field,

we

had

to

invest

substantial

efforts

initially

in

educating

the

application

researchers

on

the

ability

of

Grid

services

in

supporting

the

industry

standard

data

security

through

role-

and

privilege-based

access

using

standard.

With

respect

to

QoS,

application

researchers

expect

cluster

level

QoS

with

Grid

environments.

Also,

there

is

a

steep

learning

curve

in

Grid

computing

compared

to

the

conventional

cluster

computing.

Since

Grid

computing

is

still

an

emerging

technology,

and

it

spans

over

several

administrative

domains,

Grid

computing

is

still

premature

especially

in

terms

of

the

level

of

QoS

although,

it

offers

better

data

security

standards

compared

to

commodity

clusters.

It

is

our

observation

that

training

and

outreach

programs

that

compare

and

contrast

the

Grid

and

cluster

computing

environments

would

be

a

suitable

approach

for

enhancing

user

participation

in

Grid

computing.

This

approach

also

helps

users

to

match

their

applications

and

abilities

Grids

can

offer.

In

summary,

our

efforts

through

TIGRE

in

Grid-enabling

the

EnKF

data

assimilation

methodology

showed

substantial

promise

in

engaging

Petroleum

Engineering

researchers

through

intercampus

collaborations.

Efforts

are

under

way

to

involve

more

schools

in

this

effort.

These

efforts

may

result

in

increased

collaborative

research,

educational

opportunities,

and

workforce

development

through

graduate/faculty

research

programs

across

TIGRE

Institutions.

ACKNOWLEDGMENTS

The

authors

acknowledge

the

State

of

Texas

for

supporting

the

TIGRE

project

through

the

Texas

Enterprise

Fund,

and

TIGRE

Institutions

for

providing

the

mechanism,

in

which

the

authors

(Ravi

Vadapalli,

Taesung

Kim,

and

Ping

Luo)

are

also

participating.

The

authors

thank

the

application

researchers

Prof.

Akhil

Datta-Gupta

of

Texas

A&M

University

and

Prof.

Lloyd

Heinze

of

Texas

Tech

University

for

their

discussions

and

interest

to

exploit

the

TIGRE

environment

to

extend

opportunities

in

research

and

development.

REFERENCES

Foster,

I.

and

Kesselman,

C.

(eds.)

The

Grid:

Blueprint

for

a

new

computing

infrastructure

(The

Elsevier

series

in

Grid

computing)

TIGRE

Portal:

Vadapalli,

R.

Sill,

A.,

Dooley,

R.,

Murray,

M.,

Luo,

P.,

Kim,

T.,

Huang,

M.,

Thyagaraja,

K.,

and

Chaffin,

D.

Demonstration

of

TIGRE

environment

for

Grid

enabled/suitable

applications.

IEEE/ACM

Int.

Conf.

on

Grid

Computing,

Sept

Austin

The

High

Performance

Computing

across

Texas

Consortium

Pordes,

R.

Petravick,

D.

Kramer,

B.

Olson,

D.

Livny,

M.

Roy,

A.

Avery,

P.

Blackburn,

K.

Wenaus,

T.

Würthwein,

F.

Foster,

I.

Gardner,

R.

Wilde,

M.

Blatecky,

A.

McGee,

J.

and

Quick,

R.

The

Open

Science

Grid,

J.

Phys

Conf

Series

and

Reed,

D.A.

Grids,

the

TeraGrid

and

Beyond,

Computer,

vol

no.

and

Evensen,

G.

Data

Assimilation:

The

Ensemble

Kalman

Filter,

Springer

Herrera,

J.

Huedo,

E.

Montero,

R.

S.

and

Llorente,

I.

M.

Scientific

Programming,

vol

No.

pp

Avery,

P.

and

Foster,

I.

The

GriPhyN

project:

Towards

petascale

virtual

data

grids,

technical

report

and

The

PacMan

documentation

and

installation

guide

Caskey,

P.

Murray,

M.

Perez,

J.

and

Sill,

A.

Case

studies

in

identify

management

for

virtual

organizations,

EDUCAUSE

Southwest

Reg.

Conf.,

Feb

Austin,

TX.

The

Grid

User

Management

System

(GUMS)

Thomas,

M.

and

Boisseau,

J.

Building

grid

computing

portals:

The

NPACI

grid

portal

toolkit,

Grid

computing:

making

the

global

infrastructure

a

reality,

Chapter

Berman,

F.

Fox,

G.

Thomas,

M.

Boisseau,

J.

and

Hey,

T.

(eds),

John

Wiley

and

Sons,

Ltd,

Chichester

Open

Ticket

Request

System

The

MoinMoin

Wiki

Engine

Vasco,

D.W.

Yoon,

S.

and

Datta-Gupta,

A.

Integrating

dynamic

data

into

high

resolution

reservoir

models

using

streamline-based

analytic

sensitivity

coefficients,

Society

of

Petroleum

Engineers

(SPE)

Journal,

Emanuel,

A.

S.

and

Milliken,

W.

J.

History

matching

finite

difference

models

with

streamlines,

SPE

Proc

of

the

Annual

Technical

Conf

and

Exhibition,

Sept

New

Orleans,

LA.

Nævdal,

G.

Johnsen,

L.M.

Aanonsen,

S.I.

and

Vefring,

E.H.

Reservoir

monitoring

and

Continuous

Model

Updating

using

Ensemble

Kalman

Filter,

SPE

Proc

of

the

Annual

Technical

Conf

and

Exhibition,

Oct

Denver,

CO.

Jafarpour

B.

and

McLaughlin,

D.B.

History

matching

with

an

ensemble

Kalman

filter

and

discrete

cosine

parameterization,

SPE

Proc

of

the

Annual

Technical

Conf

and

Exhibition,

Nov

Anaheim,

CA

Li,

G.

and

Reynolds,

A.

C.

An

iterative

ensemble

Kalman

filter

for

data

assimilation,

SPE

Proc

of

the

SPE

Annual

Technical

Conf

and

Exhibition,

Nov

Anaheim,

CA

Arroyo-Negrete,

E.

Devagowda,

D.

Datta-Gupta,

A.

Streamline

assisted

ensemble

Kalman

filter

for

rapid

and

continuous

reservoir

model

updating.

Proc

of

the

Int.

Oil

Gas

Conf

and

Exhibition,

SPE

Dec

China

ECLIPSE

Reservoir

Engineering

Software

sp

O O O B I I O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O B I I I O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I I O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O B O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I I O O O O B I I B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O B O O O O O O O O B O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O

datum assimilation methodology
hydrocarbon reservoir simulation
pooling license
grid-enabling
and gridway

ensemble kalman filter
energy exploration
tigre grid computing environment
grid computing
cyberinfrastructure development project
high performance computing
tigre grid middleware
strategic application area
gridway metascheduler
reservoir model
enkf
tigre