PDQ

Zach Miller UW, Tevik UW, Mike Wilde, Nika Nefedova

Natalia Maltsev, Alek Rodrigues, Ian Foster, Robert Brunner, Ray Plante,
Bob Wilhelmson, Phoebe Lenear, Sudhakar, Al Rossi, Tom Defanti

MEAD -- BW

Test reliability of data movement from to x to y in context of MEAD

Gridftp testing, gridftp and unitree incorporation, java cog as base for
gridftp implementation

Issues:

1 useability and reliability

2 invocation (clients

3 esg experience?

4 upgrade to globus 2.2

5 communication...

Q2 --

1)  continue integration and testing

2)  joint mcs design with earth system grid

3)  chimera design meeting and expts

4)  establish initial mead MCS for grid data mgt

note: esg documents reviewed and initial chimera design meeting today --

project repository isssues

-   use case themes

    -   register executable with run or set of runs (project)

    -   associate input/output data with run or project

    -   must be able to process existing data

    -   automated mechanism for accomplishing all of these

    -   allow for user queries

-   design goals

    -   easy setup and administration to set up mcs by research groups

        -   as well as potentially for national community

    -   xml in/xml out

m&dm -- issues and organization

-   issues

    -   how to store and access (including finding relevant info) --
        large data volumes (use dods, etc)

    -   interactive distributed parallel metadata searches
        through/across metadata databases

    -   which databases to use?

    -   Organization of runs/projects/groups

    -   Secure user publishing and access (more than pipeline) including
        user notebook

    -   Replica mgt

    -   Movement of data chunks between sites for post pipeline services
        (analysis, datamining, vis)

    -   Stable, portable middleware and m&dm related software on
        platforms starting with teragrid

        -   Push for simple instantiations --

Data structure

Initial suite output

-   only need logical name

    -   app reads/writes to logical name

    -   uri s for each file

-   5d file (field, time, data(3d))

    -   specify 2 of these when accessing

    -   optimize for sequential access

    -   data subset

-   other dimensions

MCS requirements

-   storing attributes

-   querying

-   extensibility

-   consistency

-   support for authentication/authorization

-   support for logical collections

-   support for logical views

-   creation information

-   annotations

-   audit records

-   transformation history

-   master copy support versioning.

Current mcs level of efforts

-   fte level in esg and outside esg

(jay talk on gridftp and mcs \--)

level of mcs -- 1 and a bit ftes working on it --

certainly the functionality --

mcs came out of frustration with not being able to get at mcat in SRB --

would be good to register how much support this should get --

Radioastronomy (ray plante)

RAI progress on targets

Use gridftp to deliver vla data to NCSA

-   testing

    -   gridftp server on bima archive machine (nmi release)

    -   client on vla archive machine

    -   install client/server on other target machine

    -   some firewall issues

-   initial bulk transfer -- by disk or by network (1 terabyte...)

    -   estimate -- 1 month to transfer 1TB by network (not bad

    -   VLA has \< 100Mb/s connection to Abilene

    -   New mexic concerned about flooding Abilene \--

-   Completion expected by end of month in some form

Incorporate gridftp into bima pipeline to transfer data between archive
and compute platforms

-   java cog is critical here

-   hope that

mirroring sw for vla

-   2 modes

    -   between nrao sites

    -   between nrao and ncsa

-   nrao prefers simple/secure/robust off the shelf

-   how does globus solution compare with rsync

-   [www.rsync.org](http://www.rsync.org/)

-   used for transfer from telescope for bima

rsync v globus

-   rsync

    -   ssh based

    -   incremental mirroring

    -   checksum verification of mirrored files

    -   retries until success

-   globus rft/gridftp

    -   security compatible with grid infrastructure

    -   ability to copy into ncsa mass storage

    -   recording of replication in catalog (not yet automated)

    -   retries until success (implemented in rft)

    -   no checksum verification? (only tcp checksum -- not sufficient
        in long run)

    -   Gridftp works only on a file-basis -- need to go in and get a
        list of files in target directory)

-   Questions

    -   Correct assessment?

    -   When does rsync/globus make sense

Main feature -- checksum, functioning at directory level, and also
replica catalog integration --

Should this be a grid-enabled rsync --

Miron -- thinks challenge of mirroring is good --

Where should control of effort be?

What should be in server --

Should it be checkpointing? Or subsetting?

Having ability to process the data (eg checksumming) this is a good
capability that we'll find many additional uses for ..

Robert Brunner

Virtual astronomy --

Current status/revised milestones/parting thoughts

Current status:

-   data movement

    -   ncsa cache setup on quest 2 machine

    -   data movement from SRB keeps crashing

    -   waiting on condor group to get their approach working

-   data archives

    -   initial archives up for 2mass/first/roast/and nvss

    -   dposs catalog data movement tricky

-   quest2???

    -   Not yet up and running -- need to take off the front line --
        instrument isn't generating data yet.

    -   Keep on back burner..

M3

-   initial deployment of astronomy data archives

    -   dposs, 2mass, nvss, first, roast

-   good progress on initial movement work

help from tevik and miron has been exemplary -- promote this as a model
for interaction

m6

-   complete catalog archives

    -   database tests

        -   currently on sql server

        -   would like to try oracle odm

    -   sdss data mirroring tests

-   initial data mining tools evaluation

    -   d2k

    -   oracle odm

-   initial image data archive work

    -   dposs

-   complete dposs image transfer tests

-   initial dposs image processing tests...

Miron: would be good to instrument our tools -- so that we can record
how much data we move (and perhaps the rate) --

Robert -- want to get out ahead of the all-hands meeting -- start
thinking about this now rather than end of march --

Scidac -- has data management -- for efficient queries -- being
developed at Lawrence Berkley lab --

<http://sdm.lbl.gov/sdmcenter/>

RB -- think that if we can get other people convinced that data items
are useable and indeed use them -- this would be an impressive metric of
success for the expedition..

M9

-   complete dposs image processing pipeline

-   initiate web service tools for astro archives (from m6)

    -   data mining tools

    -   ogsa migration path plan

-   complete sdss data mirror

m12

-   complete dposs image processing

-   complete astronomy archives /mirrors

    -   sdss, ukirt, noao, quest2'

-   continue web service tool deployment

-   initial chimera tests...

parting thoughts

-   bag of tools

    -   users need help!

-   Pdq extent?

    -   

-   Broader involvement

    -   Among the alliance...

-   No telecons! Use access grid

    -   Thinks these are a failure...

    -   Weekly conference is too frequent

    -   ½ month or 1 month meetings -- can have sufficient progress for
        reporting to the group...

    -   already talking with people needs to talk to ...

-   Technology liaisons!

    -   Need a co-trench inhabitant to really try things out --

Going through the issues --

Less frequently ??

Access grid?? More hassle, but can share slides --

Clearly designated technology partners

Biology -- people at ANL and argonne

MEAD --

A little unclear

Working with gregor for java cog

But need to step this up for MCS -- higher level

One other issue --

How do we communicate to the rest of the alliance --

Want really good presentations at the all-hands meeting --

Topic oriented meetings --

Natalia Maltsev

Bioinformatics has a large number collaborators -- Zach Miller, Alex
Rodrigez, ...

Biology overview...

Goals of project

Develop

-   integrated computational infrastructure gwiz/gadu

-   tools and algorithsms for...

q4 --

-   robustness of dataflow

-   working plan for chimerization..

-   grid-based backend for high-throughput computations..

with respect to condor:

accomplishments

-   gadu-condor pipeline established

    -   grid-ftp installed

    -   gadu submits blas jobs to condor via gridftp

-   100 genomes submitted to condor for blast

remaining condor tasks

-   automate pipeline between gadu and condor for periodic automated
    blast submissions...

-   add analysis pipeline for interpro and other bioinformatics tools

-   add condor pools at u of C to gadu..

data storage requirements

one genome

-   flat file 1 kb to 15 MB

-   100 genomes now

-   635 genomes...in fututre

-   \< one minute processing time...

need 4 TB working...

plans for q1 2003

-   run protoype under chimera (working with Mike and Nika...)

-   add additional distributed resources to gadu

    -   u of C condor pool

    -   datagrid cluster

    -   teragrid (jazz?)

-   dev approaches for recording data provenance

-   find solutions -- to data storage needs.

Zach -- using condor cluster --and will also work with condor-g

Milestones for

Gadu infrastructure development

Chimerize gadu

Utilize large distributed resources --

Other ...

Need to solve gadu space mgt problems...

Nagiza --at ORNL...

SVMMER

Tool for automated annotation -- of data...

<http://www-wit.mcs.anl.gov/svmmer/>

Sudhakar --

Want to mirror db on mass store at NCSA --

With automatic mirroring of such data --

Data grid

Starting with gridFTP

Gregor -- not sufficient to test c toolkit against c toolkit --

Need to test java cog against machines for portal expeditions --

Wrote test framework to do this in CoG kit --

Can load in a matrix of machines to test --

Client to c-server --

Need to be hardening this --

Done in 24 hours nonstop mode --

Ian --

Long term -- cog needs to be part of the full globus distribution --

Definitely for gt3 -- but perhaps for gt2 --

Have the tests now --

Probably will eliminate client protocol incompatibilities --

Timelines: need to be addressing this in January (testing) (not
integration) --

Integration -- discussed in January -- for end of spring integration in
globus --

Gt3 -- middle of next year --

Mass storage problem --

Set of experts need to get together --

On capabilities of mass store device --

Michelle Butler/Bill Allcock/

Timeout -- occurs on fetching of of large files --

ftp protocol doesn't have provision for dealing with time out --

is ftp protocol suitable for this --

Jay -- talk with michelle --

To come up for gridftp powwow --in next couple of weeks --

Extended listing

Device dependent data --

List or get \-- /dev???

Want test in gridftp suite --

ftp server is not supposed to get from /dev --

also symbolic link feature --

this is very well documented -- ftp doesn't know about paths, just cds
--

rsync and rft -- designed to do very different things --

large data sets have been primarily read only -- so updates not usually
a problem.

Rft -- have large chunk of data to move -- or partially move --

Rsync -- does diff thing and moves bytes that it changes --

Tutorial -- see the slides ... notes are on the slides...

Q&a

Master script and job scripts -- spit out by shell planner --

For Pegasus --condor cmd files generated rather than job scripts --

Cannot mix shell and grid planners --

But it is possible to include submit side into the processing -- in
theory --

Local has special handle -- not through condor-g but standard or vanilla
universe --

Would be good to do some early planning to see what sites we would like
to run on -- any gaps in the site --

Good to try out dummy jobs.

Have tried on condor and pbs --

Haven't thought about mpi programs, for example --

Think of a challenge problem in Q1 of next year --

Currently chimera gang is pretty tied up --

Pdq targets -- alliance and teragrid production resources

May be good to form a chimera working group

One reason for evl to be involved -- staging at starlight --

Have several clusters -- one for bob grossman data mining, one unused,
one for Caltech datatag expts -- an extra pc with a big disk is easy

Connectivity is pretty easy with one caveat --

Can do at one to ten gigabit/s

Only issue is what do you do once get the data there --

Talking with abiline -- starlight is on other side of fence -- want us
to pay for abiline connection (300,000 for 1gigabit connection) -- want
to get to do experiments --

Could do other experiments but want to get rid of this bottleneck --

Also zero sum gain with equipment and power --

Have plan to expand the site --

If put something in have to take something out...

Could accommodate anything we really need...

Place that this will matter -- UW -- with OC3 -- a few megabits is
conceived to be a denial of service attack --

fours of terabytes -- sounds like a job for NCSA --

6 months from now -- will have the space for this ..

short term --want access to some space for experiments --

one disk should be sufficient --

mcs working group, gridftp working group, and chimera working group --

all hands -- apr 30 to may 1

Next face to face March 19^th^, in Urbana --

AG meeting in January -- let's try Jan 28 10-12 AM

Perhaps using netmeeting or vnc --