# Presentation: 081448

## US Atlas, Grid Computing, and the DCs

- Jennifer Schopf
- Argonne National Lab
- Northwestern University

- Atlas SW Week
- CERN
- Dec 4, 2001

## US Grid Projects

- GriPhyN: NSF funded, 4 physics exps., strong CS research component
- PPDG: DOE funded, 5-7 physics exps., deployment oriented
- iVDGL: NSF funded, GriPhyN part 2, testbed focused
- Essential to have a coordinated plan between these efforts and the ATLAS data challenges

- J. Schopf, ANL

## Overlapping Objectives

- Testbed deployment, packaging, basic sequential services
- Data management
- replicas, reliable file transfers, catalogs
- Resource management
- job submission, scheduling, fault tolerance
- Globus is underlying framework

- J. Schopf, ANL

## Globus ToolkitTM

- Core protocols and services
- Grid Security Infrastructure
- Grid Resource Access & Management
- MDS information service & monitoring
- GridFTP data access & transfer
- Data Grid technologies
- Replica catalog, replica mgmt. service
- Reliable file transfer
- Defacto-standard for Grid projects:
- GriPhyN, PPDG, NEES, EU DataGrid, ESG, iVDGL, Fusion Collaboratory, DISCOM, NASA IPG, NSF TeraGrid, DOE Science Grid, UK Grid Center, U.S. GRIDS Center, Access Grid, GridPort, MPICH-G2, Condor-G, GrADS, and others

- J. Schopf, ANL

## Globus Status

- GlobusToolkit 2.0 release
- Beta Release 11/14/01
- New packaging: enables modular binary and source distributions
- GRAM 1.5 (job submission): enhanced robustness
- MDS-2.1 (information service): security, better perf
- GridFTP- secure large file transfer
- Replica Mgt-  Data mgmt, catalogs for replicas
- Continuing work
- Community Authorization Services
- Reliable file transfer
- Java, other “Commodity Grid” toolkits

- J. Schopf, ANL

## Outline

- Project overview
- PPDG, GriPhyN, iVDGL
- Tool overview
- Magda
- Grappa
- PacMan
- Monitoring
- Focus on DC work

- J. Schopf, ANL

## ATLAS PPDG

- Existing Particle Physics Data Grid program newly funded (July 01) for 3 years at ~$3M/yr
- ATLAS support: 1.5 FTE BNL-Software, .5FTE BNL-Tier 1, 1FTE ANL (0.8 with ANL cost structure)
- Support at this level for three years
- T. Wenaus is the ATLAS lead, J. Schopf is the CS liaison for ATLAS
- Proposal emphasizes delivering useful capability to experiments (ATLAS, CMS, BaBar, D0, STAR, JLab) through close collaboration between experiments and CS
- Develop and deploy grid tools in vertically integrated services within the experiments

- J. Schopf, ANL

## ATLAS GriPhyN/iVDGL

- GriPhyN funded Fall 2000, $11.9M/5 years
- iVDGL funded Fall 2001, $13.65M/5 year
- Both involve ATLAS, CMS, Ligo and SDSS
- ATLAS support: 2.5 FTE IU, 1.5 FTE BU, $331K/5yr IU HW
- ATLAS lead R. Gardener, CS liaison J. Schopf
- GriPhyN proposal emphasizes virtual data requirements, collaboration between experiments
- iVDGL proposal emphasizes testbed and infrastructure issues across experiment

- J. Schopf, ANL

## Tools

- Magda: MAnager for Grid-based Data
- GRAPPA: Grid Access Portal for Physics Applications
- PacMan: PACkage MANager
- Monitoring - GridView, etc.

- J. Schopf, ANL

## USAtlas Tool Development

- GRAPPA/
- Gardner,
- Bramley

- Monitoring/
- De, Schopf,
- Taylor, Yu, etc

- Condor (G)

- GRAM

- GSI

- MDS/
- GIIS/GRIS

- GridFTP

- Replica Cat

- Replica Mgr

- PacMan/
- Youssef

- Packaging

- Magda/
- Wenaus

- J. Schopf, ANL

## MAnager for Grid-based DAta
Focused on the principal PPDG year 1 deliverable of a production distributed data manager
Designed for rapid development of components to support users quickly, with components later replaced by Grid Toolkit elements
Deploy as an evolving production tool and as a testing ground for Grid Toolkits
Under development at BNL

Info:              http://www.usatlas.bnl.gov/magda/info
The system: http://www.usatlas.bnl.gov/magda/dyShowMain.pl

- Focused on the principal PPDG year 1 deliverable of a production distributed data manager
- Designed for rapid development of components to support users quickly, with components later replaced by Grid Toolkit elements
- Deploy as an evolving production tool and as a testing ground for Grid Toolkits
- Under development at BNL
**Info:              http://www.usatlas.bnl.gov/magda/info**
**The system: http://www.usatlas.bnl.gov/magda/dyShowMain.pl**

- Magda

- J. Schopf, ANL

## Architecture Diagram

- Location

- Location

- Location

- Site

- Host 2

- Location

- Location

- Cache

- Disk Site

- Location

- Location

- Location

- Mass StoreSite

- Source to cache
- stagein

- Source to dest
- transfer

- MySQL

- Synch  via DB

- Host 1

- Collection of logical
- files to replicate

- Spider

- Spider

- scp, gsiftp

- Register replicas

- Catalog updates

- J. Schopf, ANL

## Distributed Catalog and Metadata

- Based on MySQL database
- Catalog of ATLAS data at CERN, BNL (also ANL, LBNL)
- Supported data stores: CERN Castor, CERN stage, BNL HPSS (rftp service), disk, code repositories, ...
- Current content: physics TDR data, test beam data, ntuples, ...
- About 150k files currently cataloged representing >2TB data
- Has run without problems with ~1.5M files cataloged
- Globus replica catalog to be integrated and evaluated
- Integrate with Grenoble catalog for appl. metadata
- Beginning: Integration as metadata layer into ‘hybrid’ (ROOT+RDBMS) implementation of ATLAS DB arch.
- To come: Data signature (‘object histories’), object cataloging

- J. Schopf, ANL

## File Replication

- Supports multiple replication tools as needed and available
- Automated CERN-BNL replication
- CERN stage  cache  **scp**  cache  BNL HPSS
*stagein*, *transfer*, *archive* scripts coordinated via db
- Replication in USATLAS testbed using Globus gsiftp
- Currently supported testbed sites are ANL, LBNL, BU
- BNL HPSS  cache  **gsiftp ** testbed disk
**gsiftp **not usable to CERN;  no grid link until CA issues resolved
- GDMP (flat file version) will be integrated soon
- GDMP being developed by CMS, PPDG and EDG

- J. Schopf, ANL

## Data Access and Production Support

- Command line tools usable in production jobs to access data
- getfile, releasefile, putfile
- Adaptation to support ATLAS Data Challenge production environment in progress
- Drawing on STAR production experience
- Near term focus -- application in DC0, DC1
- Accepted as the file cat. and replication tool for DC0
- Objective is to integrate it with GDMP for ATLAS-wide use in DC1 also
- Callable APIs for cat. usage and update to come
- Collaboration with David Malon on Athena integration

- J. Schopf, ANL

## Slide 16

- J. Schopf, ANL

## Slide 17

- J. Schopf, ANL

## Magda in DC0 Update

- ‘putfile’ for use in production scripts ready
- Archives data in Castor or HPSS and registers archived copies in catalog
- Ready for try-out; will work with Pavel to integrate
- Command line access ‘catfind’ improved
- ‘LFN://atlas/filename’ syntax supported (Still have to change atlas to atlas.org)
- ‘Add site’ procedure tested by Fred Luehring (thanks), and doc improved
- Details will be posted shortly

- J. Schopf, ANL

## Magda putfile, catfind

**$ putfile**
**Usage:  putfile <filename> <location> [path] [--host=<hostname>]**
**           ****[--logical=<lfn>] [--keys=<keys>]**
**Parameters:**
**  ****<filename>          Name of physical file to archive. Used as logical**
**                      ****filename unless --logical option is used.**
**  ****<location>          Magda location where the file should be archived,**
**                      ****in format site:location**
**  ****<path>              Full pathname of directory where file should be**
**                      ****archived, if the pathname is different from (ie a**
**                      ****deeper directory level than) the location name.**
**                      ****If <path> is not present, file will be put in**
**                      ****the directory specified by the location.**
**  ****--host=<hostname>   Local hostname to use (overrides DBYA_HOST)**
**  ****--logical=<lfn>     Logical filename if different from physical name**
**  ****--keys=<keys>       User-defined key strings in form "key1,key2,key3"**
**$ catfind rd13_run214699.dat**
**LFN://atlas/rd13_run214699.dat site=cerncastor path=/castor/cern.ch/atlas/testbeam/emebh6/2001 size=3564000**
**LFN://atlas/rd13_run214699.dat site=usatlasrftp path=/home/wenaus/lardata size=3564000**
**$ catfind LFN://atlas/Y00034.6**
**LFN://atlas/Y00034.6 site=usatlasrftp path=/home/wenaus/tdrdata size=170942400 **
**       ****keys=tape,CalorimJetProd_Eta=0.3_E0=1000_GeV,dijet_1000_030_2**

- J. Schopf, ANL

## GRAPPA: Grid Access Portal for Physics Applications

- Provide a point of access to ATLAS grid resources
- IU (Physics, CS), Northwestern (ECE), ANL (CS), BU (Physics),
- Provide a simple interface for physicists to submit and monitor jobs on the Grid
- Web-based as well as script-based
- Ability to “replay”
- Compatible with both ATLSim and Athena architecture
- Adaptable and/or “extensible” to new developments in Grid software, Athena, etc

- J. Schopf, ANL

## GRAPPA Components

- User Interface
- Job submission
- Monitoring
- Bookkeeping
- Resource selection

- J. Schopf, ANL

## Use of IU Active Notebook project

- Tool to help design and manage grid experiments
- Five pieces:
- The Java Tomcat web server
- A "database" of notebooks
- web pages, appl. configuration,  and launch scripts.
- A Script Execution Engine (GridJPython)
- interfaces to GSISSH, GSIFTP, COGKit, etc
- A performance monitoring utility
- A SOAP-based Grid Event publish and subscription service

- J. Schopf, ANL

## Slide 23

- J. Schopf, ANL

## Current GRAPPA

- Uses Globus GRAM, GSI, etc
- Executes ATLSim jobs
- Provides bookkeeping
- Prototype running at IU
- Available for general beta-testing early September.

- J. Schopf, ANL

## In this example, a parameter file is generated and saved on the host where execution will occur.

- J. Schopf, ANL

## Then the job is executed with user supplied command line parameters. For remote execution GRAM is used

- J. Schopf, ANL

## Parameters are automatically stored under the job name for future use

- J. Schopf, ANL

## Current Status

- Simple prototype that allows users to submit an Athena job from a web interface to the Condor pool on the Atlas IU cluster (via Globus)
- Next steps
- Adding more Athena functionality to the interface (e.g., user's defined libraries)
- Experiment with other job launch mechanisms
- Condor-G and DAGMAN description language
- Web Services Flow Language as a more general workflow description
- Explore interfaces for multiple metadata and replica catalog systems.
- http://lexus.physics.indiana.edu/~griphyn/grappa/

- J. Schopf, ANL

## Next Steps

- Run with ATLFast/Athena jobs
- Incorporate existing tools for resource discovery
- _Store user annotations with job metadata_
- Display redirected stdout and stderr
- Stage input files with GSIftp
- Integrate Condor
- Interface with Athena Python scripting
- Interface with PPDG Job description language

- J. Schopf, ANL

## Web Links

- Main GRAPPA page
- _[http://lexus.physics.indiana.edu/~griphyn/grappa/](http://lexus.physics.indiana.edu/~griphyn/grappa/)_
- Active Notebook page
- _[http://www.extreme.indiana.edu/an/index.html](http://www.extreme.indiana.edu/an/index.html)_
- List of goals and objectives
- _[http://lexus.physics.indiana.edu/griphyn/grappa/GRAPPA_Project_PhaseI_Updated.html](http://lexus.physics.indiana.edu/griphyn/grappa/GRAPPA_Project_PhaseI_Updated.html)__ _
- Draft GRAPPA architecture
- _[www.cs.indiana.edu/hyplan/bramley/arch.html](http://www.cs.indiana.edu/hyplan/bramley/arch.html)__ _
- Rob’s GRAPPA talk from Atlas telecon 7/01
- _[lexus.physics.indiana.edu/~griphyn/grappa/grappa-status-7-01.ppt](http://lexus.physics.indiana.edu/~griphyn/grappa/grappa-status-7-01.ppt)__ _

- J. Schopf, ANL

## pacman

- Package manager for the grid in development by Saul Youssef (Boston U, GriPhyN/iVDGL)
- Single tool to easily manage installation and environment setup for the *long* list of ATLAS, grid and other software components needed to ‘Grid-enable’ a site
- fetch, install, configure, add to login environment, update
- Sits over top of (and is compatible with) the many software packaging approaches (rpm, tar.gz, etc.)
- Uses dependency hierarchy, so one command can drive the installation of a complete environment of many packages

- J. Schopf, ANL

## Details addressed by pacman

- Where do I get the software?
- Which version is right for my system?
- Should I take the latest release or a more stable release?
- Are there dependent packages that I have to install first?
- Do I have to be root to do the installation?
- What is the exact procedure for building the installation?
- How do I setup whatever environment variables, paths, etc. once the software is installed?
- How can I set up the same environment on multiple machines?
- How can I find out when a new version comes out and when should I upgrade?

- J. Schopf, ANL

## pacman is distributed

- Packages organized into *caches* hosted at various sites, where responsible persons manage the local cache and individual *packages* hosted by that cache
- Support responsibility is distributed among sites according to where the maintainers are
- Many people share the pain
- Includes a web interface (for each cache) as well as command line tools

- J. Schopf, ANL

- J. Schopf, ANL

## Monitoring

- Joint working group set up between PPDG and GriPhyN to investigate monitoring issues
- Led by J. Schopf and D. Yu
- Monitoring currently defined very broadly:
- Is this router configured correctly?
- Has the application finished using that file yet?
- What information do I need to determine where to run my application?

- J. Schopf, ANL

## GridView (Kaushik De)

- Tool to show the status of the 8 test bed machines on the web
- Uses Globus GRAM to query sites every 30 mins
- Hostname, Uptime, Idletime, # users, and Load average
- Next steps include
- Integration with Globus information service (MDS)
- Visualization as part of cross PPDG/GriPhyN monitoring work

- J. Schopf, ANL

## Slide 37

- J. Schopf, ANL

## Application Level Monitoring (Taylor)

- Initial testbed to incorporate GRID monitoring capabilities into Athena
- Collaboration with Valerie Taylor, David Quarrie, and others
- Very long ramp-up due to difficulty of “outsiders” running an Athena application
- Working on developing an Auditor for Athena
- http://www.ece.nwu.edu/EXTERNAL/vtaylorlab/prophesy.html

- J. Schopf, ANL

## Current Status

- Mailing list has been set up
- pg-monitoring@mcs.anl.gov
- Defining usage cases
- Sensors
- Predictors
- Archiving
- Will gather requirements and look at extending Globus MDS as a common framework to meet these requirements

- J. Schopf, ANL

## Tool Use Toward DC

- J. Schopf, ANL

## ATLAS PPDG Program

- Principal ATLAS Particle Physics Data Grid deliverables:
**Year 1: Production distributed data service deployed to users.** Between CERN, BNL, and US grid testbed sites
**Year 2:** Production distributed **job management service**
**Year 3:** Create **‘transparent’ distributed processing** capability integrating distributed services into ATLAS software

- J. Schopf, ANL

## Year 1 Plan

- Year 1 plan draws on grid middleware development while delivering immediately useful capability to ATLAS
- Data management has received little attention in ATLAS up to now
- This is changing with the onset of Data Challenges, and the PPDG program is designed to help the DCs

- J. Schopf, ANL

## ATLAS PPDG Activity in Year 1

- Principal Project Activity: Production distributed data management (Magda/Globus)
- Other efforts:
- US ATLAS grid testbed -- Ed May et al
- Monitoring -- J. Schopf, D. Yu, co-chair WG
- Distributed job management -- preparatory to year 2 focus
- Data signature

- J. Schopf, ANL

## GriPhyN

- Principal ATLAS GriPhyN/iVDGL deliverables:
- 2001: Testbed with GriPhyN VDT 1.0, packaged with PacMan
- 2002: Serving DC1 data, simple job submission
- 2003: Dataset re-creation/Data signature
- Additional efforts:
- Monitoring: joint with PPDG

- J. Schopf, ANL

## GriPhyN Testbed Issues

- GriPhyN is defining VDT 1.0
- SW install for GriPhyN/PPDG (compatible with EDG as well)
- Globus 2.0 beta
- GSI, GridFTP, MDS, repl. cat stuff, etc (Gram 1.5)
- GDMP 2.0 (supports flat files)
- Condor 6.3.1 (also Condor-G, Dagman)
- Extra tools for ATLAS
- Objectivity 6.1
- Magda
- Still need to resolve CA issue between EDG and US test sites - being worked on by PPDG/ESNet/EDG

- J. Schopf, ANL

## GriPhyN ATLAS Goal 1Serving DC1 Data (July-Dec 2002)

- Limited reconstruction analysis job using grid job submission interface
- Serving the data results form DC1
- As part of DC1, data must be tagged with meta data for ease of access
- Minimal keywords would be sufficient
- Magda already implements portions of this
- Job submission with minimal smarts
- Extend GRAPPA work
- Move compute resources to data sites

- J. Schopf, ANL

## GriPhyN ATLAS Goal 2Dataset Re-creation (Jan-Sept 2003)

- Goal: be able to re-create a data file
- Need to evaluate what parameters need to be kept track of
- Need to evaluate data needed for full data signature
- Need to develop a metric for evaluating success - what is good enough?

- J. Schopf, ANL

## Joint Time Table

- Now-June02	PG1		Data management using MAGDA, Tier  Tier1, Tier2
- Dec02	 		GG0.1   	VDT 1.0 deployed (basic infrastructure)
- Jan 02 			GG0.2   	Integration of CERN testbed node into US Atlas testbed
- Jan02-Jul02	DC1     	Data creation, use of MAGDA, Tier 0-2
- Jul02-Jun03 	PG2    	Basic job management, grid job submission
- July02-Dec02 GG1		Serving data from DC1 to universities, simple grid job sub.
- Dec02-Sep03 DC2		Grid resource mgmt, data usage, smarter scheduling
- Dec02-Sep03	GG2		Dataset re-creation, metadata, advanced data grid tools
- Jul03-Jun04	PG3		Smart job submission, resource usage
- PG - PPDG Goal; GG - GriPhyN/iVDGL goal; DC - ATLAS Data Challenge

- J. Schopf, ANL

## Joint Time Table

- Now-June02	PG1		Data management using MAGDA, Tier  Tier1, Tier2
- Dec02	 		GG0.1   	VDT 1.0 deployed (basic infrastructure)
- Jan 02 			GG0.2   	Integration of CERN testbed node into US Atlas testbed
- Jan02-Jul02	DC1     	Data creation, use of MAGDA, Tier 0-2
- Jul02-Jun03 	PG2    	Basic job management, grid job submission
- July02-Dec02 GG1		Serving data from DC1 to universities, simple grid job sub.
- Dec02-Sep03 DC2		Grid resource mgmt, data usage, smarter scheduling
- Dec02-Sep03	GG2		Dataset re-creation, metadata, advanced data grid tools
- Jul03-Jun04	PG3		Smart job submission, resource usage
- Basic testbed/deployment 	Data-centric goals   		Job/Scheduling-centric goals

- J. Schopf, ANL

## Combined Planning

- PG - PPDG Goal; GG - GriPhyN/iVDGL goal; DC - ATLAS Data Challenge

- J. Schopf, ANL