# Presentation: 407289

## Atlas and the Grid

- Jennifer Schopf
- Argonne National Lab
- Northwestern University

- GriPhyN Coordination Meeting
- University of Chicago
- August 2-3, 2001

## Atlas Structure

- Very large, international project (~2000 people, ~40 countries)
- Many different funding sources, lines between them often unclear
- Currently - Various independent forays into grid technologies
- Next step – Unification of approaches

- Griphyn Coord.  8/2/01

## Two software “trees”

- Suite of applications that make up the Atlas SW – 2 pieces currently in focus
- ATLSim
- Older, Fortran based
- Used by physicists for results today
- Generally CPU bound-1K to 10M events, several min per event
- Athena and ATLFast
- OO framework based on Gaudi (LHCb)

- Griphyn Coord.  8/2/01

## Virtual Data in the Atlas Arena

- In Atlas, an input file is often a previous output file
- Need for a data signature-
- Fully specify the environment, conditions, algorithm components, inputs etc. required to produce a given file
- Given a fully specified data signature, a data set could be reproduced
- NOT CURRENTLY POSSIBLE

- Griphyn Coord.  8/2/01

## Summary

- GRAPPA/
- Gardner

- Grid Data
- Access/Malon

- DBYA/
- Wenaus

- GridView/
- De

- Condor (G)

- GRAM

- GSI

- MDS/
- GIIS/GRIS

- GridFTP

- Replica Cat

- Replica Mgr

- Saul
- Youssef

- Griphyn Coord.  8/2/01

## Outline

- 1. Background
- _2. GRAPPA – portal_
- 3. DBYA - MySQL data replica
- 4. Grid Data Access - Athena with Data Replica Cat or GDMP
- 5. Grid View - monitoring
- 6. Next steps

- Griphyn Coord.  8/2/01

## GRAPPA: Grid Access Portal for Physics Applications

- Provide a point of access to ATLAS grid resources
- Collaborative effort
- Indiana (Physics and CS)
- Northwestern (ECE)
- Argonne (CS)
- Boston (Physics)

- Griphyn Coord.  8/2/01

## GRAPPA Plan

- Provide a simple interface for physicists to submit and monitor jobs on the Grid
- Web-based as well as script-based
- Ability to “replay”
- Compatible with
- ATLSim
- Athena architecture
- Adaptable and/or “extensible” to new developments in Grid software, Athena, etc

- Griphyn Coord.  8/2/01

## GRAPPA Components

- User Interface
- Job submission
- Monitoring
- Bookkeeping
- Resource selection

- Griphyn Coord.  8/2/01

## Use of IU Active Notebook project

- Tool to help design and manage grid experiments
- Five pieces:
- The Java Tomcat web server
- A "database" of notebooks
- web pages, appl. configuration,  and launch scripts.
- A Script Execution Engine (GridJPython)
- interfaces to GSISSH, GSIFTP, COGKit, etc
- A performance monitoring utility
- A SOAP-based Grid Event publish and subscription service

- Griphyn Coord.  8/2/01

## Slide 11

- Griphyn Coord.  8/2/01

## Current GRAPPA

- Uses Globus GRAM, GSI, etc
- Executes ATLSim jobs
- Provides bookkeeping
- Prototype running at IU
- Available for general beta-testing early September.

- Griphyn Coord.  8/2/01

## Next Steps

- Run with ATLFast/Athena jobs
- Incorporate existing tools for resource discovery
- _Store user annotations with job metadata_
- Display redirected stdout and stderr
- Stage input files with GSIftp
- Integrate Condor
- Interface with Athena Python scripting
- Interface with PPDG Job description language

- Griphyn Coord.  8/2/01

## Web Links

- Main GRAPPA page
- _[http://lexus.physics.indiana.edu/~griphyn/grappa/](http://lexus.physics.indiana.edu/~griphyn/grappa/)_
- Active Notebook page
- _[http://www.extreme.indiana.edu/an/index.html](http://www.extreme.indiana.edu/an/index.html)_
- List of goals and objectives
- _[http://lexus.physics.indiana.edu/griphyn/grappa/GRAPPA_Project_PhaseI_Updated.html](http://lexus.physics.indiana.edu/griphyn/grappa/GRAPPA_Project_PhaseI_Updated.html)__ _
- Draft GRAPPA architecture
- _[www.cs.indiana.edu/hyplan/bramley/arch.html](http://www.cs.indiana.edu/hyplan/bramley/arch.html)__ _
- Rob’s GRAPPA talk from Atlas telecon 7/01
- _[lexus.physics.indiana.edu/~griphyn/grappa/grappa-status-7-01.ppt](http://lexus.physics.indiana.edu/~griphyn/grappa/grappa-status-7-01.ppt)__ _

- Griphyn Coord.  8/2/01

## Outline

- 1. Background
- 2. GRAPPA – portal
- _3. DBYA - MySQL data replica_
- 4. Grid Data Access - Athena with Data Replica Cat or GDMP
- 5. Grid View - monitoring
- 6. Next steps

- Griphyn Coord.  8/2/01

## DBYA- Database for y’all

- Database rapid prototyping tool for design and development of distributed data management services
- Part of former NOVA project
- Some pieces will use Globus work
- Replica catalog, replica manager, grid ftp
- Some pieces are separate
- tool to crawl over ATLAS data stores and send their contents to the catalog
- metadata catalog

- Griphyn Coord.  8/2/01

## Status

- File catalog for CERN and BNL
- Resident ATLAS data
- Stable and running since May 2001
- 40k files, representing 2TB data
- Automated file replication tool between CERN and BNL
- Since July 2001

- Griphyn Coord.  8/2/01

## Next Steps

- Definition of logical file name/physical file name schemas
- Integration with Globus tools
- _Separate project – Algorithmic Virtual Data_

- Griphyn Coord.  8/2/01

## Web Links

- DBYA info
- _[http://atlassw1.phy.bnl.gov/dbya/info](http://atlassw1.phy.bnl.gov/dbya/info)_
- Running version of the system
- /afs/rhic/usatlas/users/wenaus/dbya
- /afs/cern.ch/user/w/wenaus/public/dbya

- Griphyn Coord.  8/2/01

## Outline

- 1. Background
- 2. GRAPPA – portal
- 3. DBYA - MySQL data replica
- _4. Grid Data Access - Athena with Data Replica Cat or GDMP_
- 5. Grid View - monitoring
- 6. Next steps

- Griphyn Coord.  8/2/01

## Grid-Enabled Data Access in Athena

- David Malon, ANL
- CHEP paper
- Integrate Grid Data Access techniques (Globus replica catalog and/or GDMP) into the Athena event selection module
- When a file is needed, a check is done to see if the file is local
- if not, use Grid Data Access tools to make it local

- Griphyn Coord.  8/2/01

## Athena and Globus

- Search Globus replica catalog and select
- Transfer file using the protocol associated with the location object in the catalog (gsiftp, https/globus-url-copy from a remote gass_server, ...)

- Griphyn Coord.  8/2/01

## Athena and GDMP

- Use the GDMP import/export catalogs
- Supports certain subscription-based approaches
- Use GDMP to copy files

- Griphyn Coord.  8/2/01

## Status

- Paper submitted
- Prototype up and running on ANL systems
- Next Steps:
- GDMP will use Globus data replica work
- Further integration with Grid tools (e.g., EDG WP1 co-allocation work)
- _Metadata work – extending the Athena Event Selector properties to allow for data signature/virtual data_

- Griphyn Coord.  8/2/01

## Outline

- 1. Background
- 2. GRAPPA – portal
- 3. DBYA - MySQL data replica
- 4. Grid Data Access - Athena with Data Replica Cat or GDMP
- _5. Grid View - monitoring_
- 6. Next steps

- Griphyn Coord.  8/2/01

## GridView (Kaushik De)

- Tool to show the status of the 8 test bed machines on the web
- Uses Globus GRAM to query sites every 30 mins
- Hostname, Uptime, Idletime, # users, and Load average.

- Griphyn Coord.  8/2/01

## Slide 27

- Griphyn Coord.  8/2/01

## Next Steps

- Add pointer to additional information stored in the Globus MDS
- Java and/or Python implementation

- Griphyn Coord.  8/2/01

## Outline

- 1. Background
- 2. GRAPPA – portal
- 3. DBYA - MySQL data replica
- 4. Grid Data Access - Athena with Data Replica Cat or GDMP
- 5. Grid View - monitoring
- _6. Next steps_

- Griphyn Coord.  8/2/01

## Summary

- GRAPPA/
- Gardner

- Grid Data
- Access/Malon

- DBYA/
- Wenaus

- GridView/
- De

- Condor (G)

- GRAM

- GSI

- MDS/
- GIIS/GRIS

- GridFTP

- Replica Cat

- Replica Mgr

- Saul
- Youssef

- Griphyn Coord.  8/2/01

## Plan for Unification

- Developed a critical mass of people
- Various independent forays into grid technologies (necessary step in the direction of a unified program of work)
- We aren't completely there yet – but soon...

- Griphyn Coord.  8/2/01

## Next steps

- Working on both a software and a facilities plan
- Craig Tull (LBL, CERN) – mapping of Athena functions to Globus
- Preliminary integration of Athena with
- Virtual data
- Resource management
- Gsi
- Design and requirement documents
- Testbed involving Athena/ATLFast
- 8 sites already running basic Globus

- Griphyn Coord.  8/2/01

## Additional Information

- Contact Info
- _[jms@mcs.anl.gov](mailto:jms@mcs.anl.gov)_
- USAtlas Computing
- _[http://www.usAtlas.bnl.gov/computing/](http://www.usatlas.bnl.gov/computing/)_

- Griphyn Coord.  8/2/01

- Extra slides follow this

- Griphyn Coord.  8/2/01

## In this example, a parameter file is generated and saved on the host where execution will occur.

- Griphyn Coord.  8/2/01

## Then the job is executed with user supplied command line parameters. For remote execution GRAM is used

- Griphyn Coord.  8/2/01

## Parameters are automatically stored under the job name for future use

- Griphyn Coord.  8/2/01

## Slide 38

- Griphyn Coord.  8/2/01

## Slide 39

- Griphyn Coord.  8/2/01