# Presentation: 016097

## GriPhyN, iVDGL and LHC  Computing

- University of Florida
- http://www.phys.ufl.edu/~avery/
- avery@phys.ufl.edu

**GriPhyN, iVDGL and LHC  Computing**

**DOE/NSF Computing Review of LHC Computing**
**Lawrence Berkeley Laboratory****Jan. 14-17 2003**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## GriPhyN/iVDGL Summary

- Both funded through NSF ITR program
- GriPhyN: 	$11.9M (NSF) + $1.6M (matching)	(2000 – 2005)
- iVDGL:		$13.7M (NSF) + $2M (matching)	(2001 – 2006)
- Basic composition
- GriPhyN:	12 funded universities, SDSC, 3 labs	(~80 people)
- iVDGL:		16 funded institutions, SDSC, 3 labs	(~70 people)
- Expts:		US-CMS, US-ATLAS, LIGO, SDSS/NVO
- Large overlap of people, institutions, management
- Grid research vs Grid deployment
- GriPhyN:	2/3 “CS” + 1/3 “physics”		(  0% H/W)
- iVDGL:		1/3 “CS” + 2/3 “physics”		(20% H/W)
- iVDGL:		$2.5M Tier2 hardware			($1.4M LHC)
- Physics experiments provide frontier challenges
- Virtual Data Toolkit (VDT) in common

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## GriPhyN Institutions

- U Florida
- U Chicago
- Boston U
- Caltech
- U Wisconsin, Madison
- USC/ISI
- Harvard
- Indiana
- Johns Hopkins
- Northwestern
- Stanford
- U Illinois at Chicago
- U Penn
- U Texas, Brownsville
- U Wisconsin, Milwaukee
- UC Berkeley

- UC San Diego
- San Diego Supercomputer Center
- Lawrence Berkeley Lab
- Argonne
- Fermilab
- Brookhaven

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## U Florida	CMS
Caltech		CMS, LIGO
UC San Diego	CMS, CS
Indiana U	ATLAS, iGOC
Boston U	ATLAS
U Wisconsin, Milwaukee	LIGO
Penn State	LIGO
Johns Hopkins	SDSS, NVO
U Chicago	CS
U Southern California	CS
U Wisconsin, Madison	CS
Salish Kootenai	Outreach, LIGO
Hampton U	Outreach, ATLAS
U Texas, Brownsville	Outreach, LIGO
Fermilab		CMS, SDSS, NVO
Brookhaven	ATLAS
Argonne Lab	ATLAS, CS

- Caltech		CMS, LIGO
- UC San Diego	CMS, CS
- Indiana U	ATLAS, iGOC
- Boston U	ATLAS
- U Wisconsin, Milwaukee	LIGO
- Penn State	LIGO
- Johns Hopkins	SDSS, NVO
- U Chicago	CS
- U Southern California	CS
- U Wisconsin, Madison	CS
- Salish Kootenai	Outreach, LIGO
- Hampton U	Outreach, ATLAS
- U Texas, Brownsville	Outreach, LIGO
- Fermilab		CMS, SDSS, NVO
- Brookhaven	ATLAS
- Argonne Lab	ATLAS, CS

- iVDGL Institutions

**T2 / Software**

**CS support**

**T3 / Outreach**

**T1 / Labs****(not funded)**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Driven by LHC Computing Challenges

**1800 Physicists**
**150 Institutes**
**32 Countries**

- Complexity:	Millions of detector channels, complex events
- Scale:	PetaOps (CPU), Petabytes (Data)
- Distribution:	Global distribution of people & resources

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Goals: PetaScale Virtual-Data Grids

***Virtual Data Tools***

**Request Planning &**** Scheduling Tools**

**Request Execution &**** Management Tools**

***Transforms***

**Distributed resources****(code, storage, CPUs,****networks)**

**Security and**** Policy****Services**

**Other Grid****Services**

***Interactive User Tools***

**Production Team**

**Single Investigator**

**Workgroups**

**Raw data****source**

**Petaflops**
**Petabytes**
**Performance**

**Resource****Management****Services **

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Global LHC Data Grid

- Online System

- CERN Computer Center > 20 TIPS

- 100-200 MBytes/s

- 2.5-10 Gbps

- > 1 Gbps

- 2.5-10 Gbps

- ~0.6 Gbps

***Tier 0***

***Tier 1***

***Tier 3***

***Tier 4***

- Tier0/( Tier1)/( Tier2)  ~ 1:1:1

***Tier 2***

- Physics cache

- PCs, other portals

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Coordinating U.S. Grid Projects: Trillium

- Trillium: GriPhyN + iVDGL + PPDG
- Large overlap in project leadership & participants
- Large overlap in experiments, particularly LHC
- Joint projects (monitoring, etc.)
- Common packaging, use of VDT & other GriPhyN software
- Organization from the “bottom up”
- With encouragement from funding agencies NSF & DOE
- DOE (OS) & NSF (MPS/CISE) working together
- Complementarity: DOE (labs), NSF (universities)
- Collaboration of computer science/physics/astronomy encouraged
- Collaboration strengthens outreach efforts

**See Ruth Pordes talk**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## iVDGL: Goals and Context

- International Virtual-Data Grid Laboratory
- A global Grid laboratory (US, EU, Asia, South America, ...)
- A place to conduct Data Grid tests “at scale”
- A mechanism to create common Grid infrastructure
- A laboratory for other disciplines to perform Data Grid tests
- A focus of outreach efforts to small institutions
- Context of iVDGL in US-LHC computing program
- Mechanism for NSF to fund proto-Tier2 centers
- Learn how to do Grid operations (GOC)
- International participation
- DataTag
- UK e-Science programme: support 6 CS Fellows per year in U.S.
- None hired yet. Improve publicity?

****

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## iVDGL: Management and Coordination

- Project Coordination Group

- US External Advisory Committee

**GLUE Interoperability Team**

- Collaborating Grid Projects

- US ProjectDirectors

- Outreach Team

- Core Software Team

- Facilities Team

- Operations Team

- Applications Team

**International Piece**

- US Project Steering Group

**U.S. Piece**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## iVDGL: Work Teams

- Facilities Team
- Hardware (Tier1, Tier2, Tier3)
- Core Software Team
- Grid middleware, toolkits
- Laboratory Operations Team (GOC)
- Coordination, software support, performance monitoring
- Applications Team
- High energy physics, gravity waves, digital astronomy
- New groups: Nuc. physics? Bioinformatics? Quantum Chemistry?
- Education and Outreach Team
- Web tools, curriculum development, involvement of students
- Integrated with GriPhyN, connections to other projects
- Want to develop further international connections

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## US-iVDGL Sites (Sep. 2001)

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## New iVDGL Collaborators

- New experiments in iVDGL/WorldGrid
- BTEV, D0, ALICE
- New US institutions to join iVDGL/WorldGrid
- Many new ones pending
- Participation of new countries (different stages)
- Korea, Japan, Brazil, Romania, ...

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## US-iVDGL Sites (Spring 2003)

**Partners?**
- EU
- CERN
- Brazil
- Korea
- Japan

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## An Inter-Regional Center for High Energy Physics Research and Educational Outreach (CHEPREO) at Florida International University

**  ****E/O Center in Miami area**
**  ****iVDGL Grid Activities**
**  ****CMS Research**
**  ****AMPATH network**
**  ****Int’l Activities (Brazil, etc.)**

**Status:**
**  ****Proposal submitted Dec. 2002**
**  ****Presented to NSF review panel****    Jan. 7-8, 2003**
**  ****Looks very positive**

## US-LHC Testbeds

- Significant Grid Testbeds deployed by US-ATLAS & US-CMS
- Testing Grid tools in significant testbeds
- Grid management and operations
- Large productions carried out with Grid tools

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## US-ATLAS Grid Testbed

**Grappa:** *Manages overall grid experience*
**Magda:**  *Distributed data management and replication*
**Pacman:**  *Defines and installs software environments*
**DC1 production with grat:***Data challenge ATLAS simulations*
**Instrumented Athena:**  *Grid monitoring of Atlas analysis apps.*
**vo-gridmap:**  *Virtual organization management*
**Gridview:**  *Monitors U.S. Atlas resources*

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## US-CMS Testbed

- Brazil

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Commissioning the CMS Grid Testbed

- A complete prototype
- CMS Production Scripts
- Globus, Condor-G, GridFTP
- Commissioning: Require production quality results!
- Run until the Testbed "breaks"
- Fix Testbed with middleware patches
- Repeat procedure until the entire Production Run finishes!
- Discovered/fixed many Globus and Condor-G problems
- Huge success from this point of view alone
- ... but very painful

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## CMS Grid Testbed Production

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Production Success on CMS Testbed

- Recent results
- 150k events generated: 1.5 weeks continuous running
- 1M event run just completed on larger testbed: 8 weeks

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

**Notes:**
Modify MCRunJob?

## US-LHC Proto-Tier2 (2001)

**>1 RAID**

**WAN**

***FEth/GEth******Switch***

**“****Flat” switching topology**

**20-60 nodes****Dual 0.8-1 GHz, P3****1 TByte RAID**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## US-LHC Proto-Tier2 (2002/2003)

***GEth/FEth***

*** ******Switch***

***GEth******Switch***

**>1 RAID**

**WAN**

**“****Hierarchical” switching topology**

***GEth/FEth***

**40-100 nodes****Dual 2.5 GHz, P4****2-6 TBytes RAID**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Creation of WorldGrid

- Joint iVDGL/DataTag/EDG effort
- Resources from both sides (15 sites)
- Monitoring tools (Ganglia, MDS, NetSaint, ...)
- Visualization tools (Nagios, MapCenter, Ganglia)
- Applications: ScienceGrid
- CMS:	CMKIN, CMSIM
- ATLAS:	ATLSIM
- Submit jobs from US or EU
- Jobs can run on any cluster
- Demonstrated at IST2002 (Copenhagen)
- Demonstrated at SC2002 (Baltimore)

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## WorldGrid 

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## WorldGrid Sites

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## GriPhyN Progress

- CS research
- Invention of DAG as a tool describing workflow
- System to describe, execute workflow: DAGMan
- Much new work on planning, scheduling, execution
- Virtual Data Toolkit + Pacman
- Several major releases this year: VDT 1.1.5
- New packaging tool: Pacman
- VDT + Pacman vastly simplify Grid software installation
- Used by US-ATLAS, US-CMS
- LCG will use VDT for core Grid middleware
- Chimera Virtual Data System (more later)

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Virtual Data Concept

- Data request may
- Compute locally/remotely
- Access local/remote data
- Scheduling based on
- Local/global policies
- Cost

**Major facilities, archives**

**Regional facilities, caches**

**Local facilities, caches**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Virtual Data: Derivation and Provenance

- Most scientific data are not simple “measurements”
- They are computationally corrected/reconstructed
- They can be produced by numerical simulation
- Science & eng. projects are more CPU and data intensive
- Programs are significant community resources (transformations)
- So are the executions of those programs (derivations)
- Management of dataset transformations important!
- Derivation: 	Instantiation of a potential data product
- Provenance: 	Exact history of any existing data product

**Programs are valuable, like data. They should be community resources.**
**We already do this, but manually!**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Virtual Data Motivations (1)

**Transformation**

**Derivation**

**Data**

*product-of*

*execution-of*

*consumed-by/*
*generated-by*

- “I’ve detected a muon calibration error and want to know which derived data products need to be recomputed.”

- “I’ve found some interesting data, but I need to know exactly what corrections were applied before I can trust it.”

- “I want to search a database for 3 muon SUSY events. If a program that does this analysis exists, I won’t have to write one from scratch.”

- “I want to apply a forward jet analysis to 100M events. If the results already exist, I’ll save weeks of computation.”

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Virtual Data Motivations (2)

- Data track-ability and result audit-ability
- Universally sought by scientific applications
- Facilitates tool and data sharing and collaboration
- Data can be sent along with its recipe
- Repair and correction of data
- Rebuild data products—c.f., “make”
- Workflow management
- A new, structured paradigm for organizing, locating, specifying, and requesting data products
- Performance optimizations
- Ability to re-create data rather than move it

** ****Needed: Automated, robust system**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## “Chimera” Virtual Data System

- Virtual Data API
- A Java class hierarchy to represent transformations & derivations
- Virtual Data Language
- Textual for people & illustrative examples
- XML for machine-to-machine interfaces
- Virtual Data Database
- Makes the objects of a virtual data definition persistent
- Virtual Data Service (future)
- Provides a service interface (e.g., OGSA) to persistent objects
- Version 1.0 available
- To be put into VDT 1.1.6?

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Virtual Data Catalog Object Model

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

**Notes:**
Feel free to skip.

## Virtual Data Language (VDL)
Describes virtual data products

Virtual Data Catalog (VDC)
Used to store VDL

Abstract Job Flow Planner
Creates a logical DAG (dependency graph)

Concrete Job Flow Planner 
Interfaces with a Replica Catalog
Provides a physical DAG submission file to     Condor-G

Generic and flexible
As a toolkit and/or a framework
In a Grid environment or locally 

- Describes virtual data products
- Virtual Data Catalog (VDC)
- Used to store VDL
- Abstract Job Flow Planner
- Creates a logical DAG (dependency graph)
- Concrete Job Flow Planner
- Interfaces with a Replica Catalog
- Provides a physical DAG submission file to     Condor-G
- Generic and flexible
- As a toolkit and/or a framework
- In a Grid environment or locally

**Abstract****Planner**

**VDC**

**Replica****Catalog**

**Concrete****Planner**

**DAX**

**DAGMan**

**DAG**

**VDL**

**XML**

- Chimera as a Virtual Data System

**XML**

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Chimera Application: SDSS Analysis

- galaxy clusters?

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Virtual Data and LHC Computing

- US-CMS (Rick Cavanaugh talk)
- Chimera prototype tested with CMS MC (~200K test events)
- Currently integrating Chimera into standard CMS production tools
- Integrating virtual data into Grid-enabled analysis tools
- US-ATLAS (Rob Gardner talk)
- Integrating Chimera into ATLAS software
- HEPCAL document includes first virtual data use cases
- Very basic cases, need elaboration
- Discuss with LHC expts: requirements, scope, technologies
- New ITR proposal to NSF ITR program($15M)
*Dynamic Workspaces for Scientific Analysis Communities*
- Continued progress requires collaboration with CS groups
- Distributed scheduling, workflow optimization, ...
- Need collaboration with CS to develop robust tools

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)

## Summary

- Very good progress on many fronts in GriPhyN/iVDGL
- Packaging: Pacman + VDT
- Testbeds (development and production)
- Major demonstration projects
- Productions based on Grid tools using iVDGL resources
- WorldGrid providing excellent experience
- Excellent collaboration with EU partners
- Looking to collaborate with more international partners
- Testbeds, monitoring, deploying VDT more widely
- New directions
- Virtual data a powerful paradigm for LHC computing
- Emphasis on Grid-enabled analysis
- Extending Chimera virtual data system to analysis

- Paul Avery

- LHC Computing Review (Jan. 14, 2003)