# Presentation: 328292

## iVDGL Integration Issues

**Rob Gardner**
**Indiana University**
- Teragrid Applications Workshop
- Argonne National Laboratory
- APRIL 30, 2002

## Data Grid Projects

- Particle Physics Data Grid (PPDG)
- Year 1 of 3-year project to deliver vertically integrated grid services to experiments
- GriPhyN
- CS and Physics collaboration to develop virtual data concept
- Main goal is to develop a virtual data catalog and data language
- iVDGL
- Platform on which to design, implement, integrate and test grid software
- Infrastructure for ATLAS, CMS, LIGO, SDSS prototype Tier2 centers
- Forum for grid interoperability – collaborate with EU DataGrid, DataTag, ...
- EDG
- European Data Grid project – HEP, Earth Sciences, Biology
- DataTAG
- Transatlantic grid interoperability

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## iVDGLInternational Virtual Data Grid Laboratory

**iVDGL****International Virtual Data Grid Laboratory**

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## iVDGL Background

- Themes:
- Distributed platform to design, develop, test, release the VDT, fundamental CS research, grid operations
- Internationally integrated virtual data grid system, interoperable, controlled sharing of data and compute resources
- Common grid infrastructure, export to other application domains
- iVDGL has identified two demonstration projects of US and international scope to be executed this year
***We would like to show interoperability with TeraGrid as part of these demonstrations***

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## iVDGL Architecture

- iSite

**iVDGL Architecture**

- Compute Platform

- Storage Platform

-  iVDGL Monitoring Interface

- iVDGL Mgmt. Interface

- iVDGL Control Interface

- Local Management Interface

- Experiment Scheduler

- Health and Status Monitoring

- iVDGL Configuration Information

- Access Control and Policy Services

- iGOC

- Experiment Management

- Experiment Data Collection

- Applications

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

##  Experiment test grid domains

- CMS

- LIGO

- SDSS/NVO

- ANL

- BNL

- BU

- IU

- UM

- OU

- UTA

- HU

- LBL

- CIT

- UCSD

- UF

- FNAL

- FNAL

- JHU

- CS Research

- ANL

- UCB

- UC

- IU

- NU

- UW

- ISI

- CIT

- UTB

- PSU

- UWM

- UC

- iGOC

- Outreach

- HU

- SKU

-  Experiment test grid domains

- BTeV

- ALICE

- D0

- SKU

- CDF

- BaBar

- Star

- JLAB

- US iVDGL

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## iVDGL is starting the process of

- integrating the experiment-specific
- testbed efforts to create a grid
- laboratory

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## iVDGL Milestones

- Short time scale – August:  US-iVDGL-1
- US component of iVDGL:  one or more sites from core experiments
- Tier 1 centers, Prototype Tier 2 centers, outreach sites
- Initial deployment and sustained operation for a few days
- Near term – November SC02:  iVDGL-2
- Integrated international functionality, partnering with DataTAG and EDG
- Use of GLUE schema, overlapping VO’s, policy development and execution
- The short time scale implies:
- Use of VDT 1.x
- Near term core application software releases as identified by experiments
- Heavy involvement of core experiment testbeds – configuration and deployment

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## US iVDGL Interoperability

- US-iVDGL-1 Milestone  (August 02)

- ATLAS

- CMS

- LIGO

- SDSS/NVO

-   US-iVDGL-1
- Aug 2002

- 1

- 2

- iGOC

- 1

- 2

- 1

- 2

- 1

- 2

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Transatlantic Interoperability

- iVDGL-2 Milestone  (November 02)

- ATLAS

- CMS

- LIGO

- SDSS/NVO

-   iVDGL-2
- Nov 2002

- ANL

- BNL

- BU

- IU

- UM

- OU

- UTA

- HU

- LBL

- CIT

- UCSD

- UF

- FNAL

- FNAL

- JHU

- CS Research

- ANL

- UCB

- UC

- IU

- NU

- UW

- ISI

- CIT

- UTB

- PSU

- UWM

- UC

- iGOC

- Outreach

- DataTAG

- CERN

- INFN

- UK PPARC

- U of A

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Issues

- Cross project domains
- Collaborative Grid projects – PPDG, GriPhyN, iVDGL, DataTAG, EDG, UK GridPP and e-Science
- Experiment testbed groups: LIGO,SDSS,CMS,ATLAS
- HICB, HIJBT  GLUE working group, schema document, testbed
- Effort
- iVDGL work teams: (facilities, VDT, operations, applications, EOR)
- Project coordination group and Interoperability teams, DT WP4
- Experimental testbed groups developing application demonstrations, users
- Identify iVDGL components and functionality
- Hardening, deploying the VDT
- Interoperability across US domains for security and resource discovery
- Controlled sharing of resources
- Transatlantic grid operations, bulk file transfers, end-to-end network connectivity tests

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Goals

- Dynamic  web pages showing US-iVDGL-1 applications being requested, scheduled and executed at all US-iVDGL-1 sites.
- Web pages showing automated installation and execution of different experiment and/or demonstration applications at a single US-iVDGL-1 site.
- Web pages showing problems and/or alarms encountered at each site where applications are deployed, a tracking system for operational support and problem resolution over the Laboratory.
- Application specific monitoring displays, graphical output, job reporting for grid-wide jobs.

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Tasks

- Identify iVDGL team contacts, sites and effort for this purpose
- Create agreements between experiment testbed efforts and iVDGL for development and support of US iVDGL-1, iVDGL-2 demonstrations
- Refine demonstration objectives and use cases, success metrics
- Collect requirements, define responsibilities for iVDGL teams
- iVDGL Site AAA effort
- Coordinate with GLUE (Grid Laboratory Uniform Environment) schema effort to deploy a common information service.
- This is a subproject of the HENP Inter-Grid Collaboration Board, or HICB.
- More info can be found at:  http://www.hicb.org/

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Tasks, II

- Setup system for service discovery
- Setup, support and maintain iVDGL information infrastructure linking those from individual experiments.
- Identify, evaluate, test and deploy information service viewing systems (mapcenter, gridview, gridsearcher, ...)
- Setup operations trouble-ticket system
- Identify, evaluate, test and deploy grid monitoring systems.

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Responsibilities

- Facilities –
- one or more sites belonging each experiment. Support for local installation and operation of the US-iVDGL-1 sites. Receive and resolve problem reports. Publishing resources as services to iVDGL
- Core Software –
- support for installation and use  of VDT 1.x on all iVDGL sites
- Operations –
- development and support for the Laboratory operation and demonstration web pages; iVDGL RA
- Applications –
- supply  the applications to run in the Laboratory - Ensure each application runs on all sites of the Laboratory. Construct and support  the application queues and demonstration applications.
- Outreach/Education - documentation of the Laboratory

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## TeraGrid - iVDGL

- TeraGrid-lite platform suitable for GriPhyN apps
- Within iVDGL, two application domains (CMS, ATLAS) have grid interoperability across testbeds
- Setting up joint iVDGL infrastructure the next step
- Infrastructure tools (eg. Grid map file management) will be developed and used, some from EU projects
- Connecting iVDGL sites to Teragrid as a test-case would be valuable for both sides

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## addendum

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Data Flow from ATLAS

**Data Flow from ATLAS**

***level 1**** - **special hardware*

*40 MHz   ****(40 TB/sec)***

***level 2**** - **embedded processors*

***level 3**** – **PC *

*75 KHz ****(75 GB/sec)***

*5 KHz ****(5 GB/sec)***

*100 Hz*
*(100 MB/sec)*

*data recording &*
*offline analysis*

- Minimum requirements
- at design luminosity

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Production Flow

- Raw data

- Reconstruction

- Data Acquisition

- Level 3 trigger

- Trigger Tags

- Event Summary Data
- ESD

-  Event Tags

- Calibration Data

- Run Conditions

- Trigger System

- Coordinated
- by collaboration

- Collaboration
- Group or
- Physicist

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Analysis Flow

-  Event Tags

- Event Selection

- Calibration Data

**Analysis**

- Raw Data

- Collaboration
- wide

- Analysis
- Groups

- Physicists

- Physics Analysis

- PhysObjs

- Analysis
- Objects

- PhysObjs

- PhysObjs

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## LHC Computing Model

- Hierarchical, distributed tiers

**Tier 2**

**Tier 1**

- Tier 0

**Tier 2**

- universities

- physicists

**CERN**

- Lab, University

- Fermilab - CMS
- Brookhaven - ATLAS

**Tier 3**

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Grappa

- Grid user interface being developed at Indiana U
- Flexible, use existing XCAT portal technology
- Submit Athena jobs to grid computing elements
- Manage JobOptions, record sessions
- Staging and output collection supported
- Preliminary tests on US ATLAS Grid Testbed
- See Shava Smallen’s talk/demo following

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Pacmanhttp://physics.bu.edu/~youssef/pacman/index.html

- Package manager for the grid in development by Saul Youssef (Boston U, GriPhyN/iVDGL)
- Single tool to easily manage installation and environment setup for the *long* list of ATLAS, grid and other software components needed to ‘Grid-enable’ a site
- fetch, install, configure, add to login environment, update
- Sits over top of (and is compatible with) the many software packaging approaches (rpm, tar.gz, etc.)
- Uses dependency hierarchy, so one command can drive the installation of a complete environment of many packages

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## Details addressed by pacman

- Where do I get the software?
- Which version is right for my system?
- Should I take the latest release or a more stable release?
- Are there dependent packages that I have to install first?
- Do I have to be root to do the installation?
- What is the exact procedure for building the installation?
- How do I setup whatever environment variables, paths, etc. once the software is installed?
- How can I set up the same environment on multiple machines?
- How can I find out when a new version comes out and when should I upgrade?

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002

## pacman is distributed

- Packages organized into *caches* hosted at various sites, where responsible persons manage the local cache and individual *packages* hosted by that cache
- Support responsibility is distributed among sites according to where the maintainers are
- Many people share the pain
- Includes a web interface (for each cache) as well as command line tools

- R. Gardner        Teragrid Applications Workshop, Argonne

- April 30, 2002