# Presentation: 328282

## US-ATLAS Grid Efforts

**John Huth**
**Harvard University**
- Agency Review of LHC Computing
- Lawrence Berkeley Laboratory
- January 14-17, 2003

## Outline

- Goals
- Data Challenge experience and plans
- Relation to external groups
- US ATLAS grid management

- Name       Title of this Talk

- November  14-17, 2000

## Goals

- Primary: establish a robust and efficient platform (fabric and software) for US LHC and International LHC simulation, production and data analysis
- Data challenge support
- User analysis support
- Production support
- Ready for data taking on “day one”
- Realizing common grid goals
- Local control
- Democratic access to data
- Support of autonomous analysis communities (private grids)
- Value added to other disciplines
- Serving as a knowledgeable user community for testing middleware

- Name       Title of this Talk

- November  14-17, 2000

## US ATLAS Testbed

- Develop a tier’ed, scalable computing platform
- Test grid middleware integrated with software
- Test interoperability
- CMS
- EDG
- Other groups
- Support physics efforts
- Data Challenges
- Production
- User analysis

- Name       Title of this Talk

- November  14-17, 2000

## Testbed Fabric

- Production gatekeepers at ANL, BNL, LBNL, BU, IU, UM, OU, UTA
- Large clusters at BNL, LBNL, IU, UTA, BU
- Heterogeneous system - Condor, LSF, PBS
- Currently > 100 nodes available
- Could double capacity quickly, if needed
- + Multiple R&D gatekeepers
- gremlin@bnl - iVDGL GIIS
- heppc5@uta - ATLAS hierarchical GIIS
- atlas10/14@anl - EDG testing
- heppc6@uta+gremlin@bnl - glue schema
- heppc17/19@uta - GRAT development
- few sites - Grappa portal
- bnl - VO server
- few sites - iVDGL testbed

- Name       Title of this Talk

- November  14-17, 2000

## Grid Testbed Sites

- US -ATLAS testbed launched February 2001
- Two new sites joining - UNM, SMU

**Grid Testbed Sites**

- Name       Title of this Talk

- November  14-17, 2000

## Testbed Tools

- Many tools developed by the U.S. ATLAS testbed group during past 2 years
**GridView**** - simple tool to monitor status of testbed ****Kaushik De,**** ****Patrick McGuigan**
**Gripe**** - unified user accounts ****Rob Gardner**
**Magda**** - MAnager for Grid DAta ****Torre Wenaus, Wensheng Deng (see Gardner & Wenaus talks)**
**Pacman**** - package management and distribution tool ****Saul Youssef**
**Grappa**** - web portal using active notebook technology ****Shava Smallen, Dan Engh**
**GRAT**** - GRid Application Toolkit**
**Gridsearcher**** - MDS browser ****Jennifer Schopf**
**GridExpert**** - Knowledge Database ****Mark Sosebee**
**VO Toolkit**** - Site AA**** Rich Baker**

- Name       Title of this Talk

- November  14-17, 2000

## Recent Accomplishments

- May 2002
- Globus 2.0 beta RPM developed at BNL
- Athena-atlfast grid package developed at UTA
- Installation using Pacman developed at BU
- GRAT toolkit for job submission on grid, developed at UTA & OU
- June 2002
- Tested interoperability - successfully ran ATLAS MC jobs on CMS & D0 grid sites
- ANL demonstrated that U.S. testbed package can run successfully at EDG sites
- July 2002
- New production software released & deployed
- 2 week Athena-atlfast MC production run using GRAT & GRAPPA
- Generated 10 million events, thousand files catalogued in Magda, all sites participating

- Name       Title of this Talk

- November  14-17, 2000

## Accomplishments contd.

- August/September 2002
- 3 week dc1 production run using GRAT
- Generated 200,000 events, using ~ 30,000 CPU hours, 2000 files, 100 GB storage
- October/November 2002
- Prepare demos for SC2002
- Deployed VO server at BNL Tier I facility
- Deployed new VDT 1.1.5 on testbed
- Test iVDGL packages Worldgrid, Sciencegrid
- Interoperability tests with EDG
- December 2002
- Developed software evolution plan during meeting with Condor/VDT team at UW-Madison
- Generated 75k SUSY and Higgs events for DC1
- Total DC1 files generated and stored > 500 GB, total CPU used >1000 CPU days in 4 weeks

- Name       Title of this Talk

- November  14-17, 2000

## Lessons Learned

- Globus, Magda and Pacman make grid production easy!
- On the grid - submit anywhere, run anywhere, store data anywhere - really works!
- Error reporting, recovery &  cleanup very important - will always lose/hang some jobs
- Found many unexpected limitations, hangs, software problems - next time, need larger team to quantify these problems and provide feedback to Globus, Condor, and other middleware teams
- Large pool of hardware resources available on testbed: BNL Tier 1, LBNL (pdsf), IU & BU prototype Tier 2 sites, UTA (new $1.35M NSF-MRI), OU & UNM CS supercomputing clusters...
- Testbed production effort suffering from severe shortage of human resources.  Need people to debug middleware problems and provide feedback to middleware developers

- Name       Title of this Talk

- November  14-17, 2000

## DC1 Phase II

- Provide data for high level trigger studies
- Data analysis from production at BNL
- Unified set of grid tools for international ATLAS
- Magda, PACMAN, cook-book database, VO management
- There may still be divergences among grids (US/EU/Nordu grid)

- Name       Title of this Talk

- November  14-17, 2000

## ATLAS DC1 Phase 2

- Pile up production:
- Ongoing!  Both grid & non-grid based
- Add min. bias events to DC1 Phase 1 sample
- Simulation in the U.S.
- Higgs re-simulation - 50k events
- SUSY simulation - 50k events
- Athena reconstruction
- of complete DC1 Phase 2 sample
- Analysis / user access to data
- Magda already provides access to ~30k  catalogued dc1 files from/to many grid locations (need ATLAS VO to make this universal)
- Need higher level (web based?) tools to provide easy access to physicist
- DIAL being developed at BNL http://www.usatlas.bnl.gov/~dladams/dial/

- Name       Title of this Talk

- November  14-17, 2000

## DC2 Specifics

- From new WBS (we will review the situation in early 2003)
- Test LCG-1 (software and hardware) - in particular POOL
- ATLAS tests
- simulation
- status of G4 (validation, hits, digits)
- pile-up in Athena
- relative role of G3/G4
- calibration and alignment
- detector description and EDM
- distributed analysis

- Name       Title of this Talk

- November  14-17, 2000

## Integration

- Coordination with other grid efforts and software developers - very difficult task!
- Project centric:
- Intl. ATLAS – K. De
- GriPhyN/iVDGL - Rob Gardner (J. Schopf, CS contact)
- PPDG – John Huth
- LCG – John Huth (POB), V. White (GDB), L. Bauerdick (SC2)
- EDG - Ed May, Jerry Gieraltowski
- ATLAS/LHCb - Rich Baker
- ATLAS/CMS - Kaushik De
- ATLAS/D0 - Jae Yu
- Fabric/Middleware centric:
- Afs Software installations - Alex Undrus, Shane Canon, Iwona Sakrejda
- Networking - Shawn McKee, Rob Gardner
- Virtual and Real Data Management - Wendsheng Deng, Sasha Vaniachin, Pavel Nevski, David Malon, Rob Gardner, Dan Engh, Mike Wilde
- Security/Site AA/VO - Rich Baker, Dantong Yu

- Name       Title of this Talk

- November  14-17, 2000

## NLV Analysis Tool: Plots Time vs. Event Name

**Menu bar**

**Scale for load-line/**
**points**

**Events**

**Legend**

**Zoom window**
**controls**

**Zoom box**

**Playback controls**

**Window size**

**Max window size**

**Zoom-box actions**

**Playback speed**

**Summary**
**line**

**Time axis**

**You are**
**here**

**Title**

- Name       Title of this Talk

- November  14-17, 2000

## Grid Planning

- For Data Challenge 1, Phase 2, generation of High Level Trigger – using a unified format for International ATLAS computing in a grid configuration.
- For R+D purposes: increased integration into iVDGL/GriPhyN tools:  VDT, Chimera, working with tools in common with US CMS
- Issue of divergences with EDG software, configurations

- Name       Title of this Talk

- November  14-17, 2000

## External Groups

- Trillium (GriPhyN,iVDGL,PPDG)
- EU initiatives
- LCG
- Intl. ATLAS
- U.S. CMS (large ITR pre-proposal)
- Other groups/experiments (HEP/LIGO/SDSS...)
- Funding agencies

- Name       Title of this Talk

- November  14-17, 2000

## Trillium Group

- PPDG  (J. Huth)
- Metadata catalog (MAGDA)  (Wensheng Deng, BNL)
- Interoperability, middleware evaluation (Jerry Gieraltowski, ANL)
- Virtual organization, monitoring in grid environment (Dantong Yu, BNL)
- Distributed analysis (David Adams, BNL)
- iVDGL (R. Gardner)
- Package deployment/installation (PACMAN)( S. Youssef, BU) –adopted by VDT and CMS
- Incorporation of VDT, Chimera in next prototype round
- Hardware support – prototype Tier 2’s (Indiana, Boston University)
- NB A tremendous amount of support comes from base efforts at Labs and Universities (netlogger – LBNL, grat- De UTA, support – H. Severini, Oklahoma, S. McKee, Michigan, May, ANL, BNL, LBNL, BU)

- Name       Title of this Talk

- November  14-17, 2000

## EU Initiatives

- Linkages via
- International ATLAS (using EDG tools/testbed)
- LCG
- ITR initiative
- Middleware representatives (Foster/Kesselman/Livny/Gagliardi)
- Issues:
- EDG mandate is larger than the LHC
- Divergences in middleware, approaches
- “best in show” concept – David Foster (LCG)
- Interoperability of testbeds

- Name       Title of this Talk

- November  14-17, 2000

## LCG

- More than just grids – applications group is very large
- Linkages via
- Intl. Experiments
- Middleware representatives
- Architects’ forum
- Applications (T. Wenaus)
- Representation: L Bauerdick (SC2), V. White (GDB), J. Huth (POB)
- Issues:
- Communications about LCG-1
- What are requirements for Tier 1’s?
- Divergences in site requirements?
- Linage of Intl. Experiments with LCG
- DC1Phase2 versus LCG-1

- Name       Title of this Talk

- November  14-17, 2000

## International ATLAS

- Main focus: Data Challenges
- G. Poulard (DC coordinator), A. Putzer (NCB chair)
- Linkages
- National Computing Board (resources, planning) – J. Huth
- Grid planning – R. Gardner
- Technical Board – K. De
- Issues
- Divergence in US-EU grid services an issue
- Commonality vs. interoperability
- Probably the tightest coupling (nothing focuses the mind like data)
- Middleware selection
- Facilities planning

- Name       Title of this Talk

- November  14-17, 2000

## US CMS

- Commonality in many tools (PACMAN, VDT, Chimera)
- Discussions in common forums (LCG, Trillium, CS groups)
- Interoperability tests
- Large ITR proposal:
- Coordination with EDG, LCG, US CMS, CS community
- Use existing management structure of US ATLAS+ US CMS
- Coordinating group
- Physics-based goal of “private grids” for analysis groups
- Coordination among funding agencies, LCG, experiments
- Further funding initiatives coordinated (EGEE, DOE)

- Name       Title of this Talk

- November  14-17, 2000

## Funding Agencies

- Recent initiative to pull together funding agencies involved in grids and HEP applications.
- Nov. 22nd meeting of NSF, DOE, Computing, HEP, EU and CERN representatives
- Follow-up date proposed: Feb. 7th
- November ITR workshop, with US CMS, US ATLAS, computer scientists and funding agency representatives
- Follow-up: M. Kasemann
- Goal: Coordinated funding of EU/US/CERN efforts on the experiments and grid middleware.
- Beyond the large ITR:
- Medium ITR’s
- DOE initiatives
- EU funding
- Question: UK, other funding

- Name       Title of this Talk

- November  14-17, 2000

## Other groups

- Experiments
- Direct linkage via GriPhyN, iVDGL: SDSS, LIGO
- Contacts, discussions with D0, RHIC experiments
- Grid organizations
- PPARC
- CrossGrid
- HICB
- Interoperability initiatives

- Name       Title of this Talk

- November  14-17, 2000

## US ATLAS Grid Organization

- Present status: WBS projection onto grid topics of software and facilities
- Efforts have focused on SC2002 demos and ATLAS production – near term milestones
- I have held off until now on reorganizing the management structure because of the rapid evolution
- Gain experience with the US ATLAS testbed
- Creation of new organizations
- Shake out of the LHC schedule
- We are now discussing the necessary changes to the  management structure to coordinate the efforts

- Name       Title of this Talk

- November  14-17, 2000

## Proposed new structure

- Creation of  level-2 management slot for distributed computing applications
- Interaction with above list of groups (or delegate liaison)
- Three sub-tasks
- Architecture –
- Series of packages created for ATLAS production or prototyping
- Components
- Testing and support of grid deliverables (e.g. high level ATLAS-specific interfaces)
- Grid deliverables associated with IT initiatives
- Production
- Running Intl. ATLAS production on the US ATLAS fabric
- Contributions to Intl. ATLAS production

- Name       Title of this Talk

- November  14-17, 2000

## Slide 27

- Name       Title of this Talk

- November  14-17, 2000

## Next Steps in Organization

- Bring this new structure to the US ATLAS Computing Coordination Board, seek advice on level-2 and level-3 managers.
- Coordinate with “liaison” groups

- Name       Title of this Talk

- November  14-17, 2000

## Comments/Summary

- US ATLAS has made tremendous progress last year in testing grid production
- SC2002 Demo
- Interoperability tests
- Data Challenges
- Management of diverse sources of effort is challenging!!
- Mechanisms begun to coordinate further activities
- Goal: without creating divergences, or proliferation of new groups
- New US ATLAS management structure under discussion

- Name       Title of this Talk

- November  14-17, 2000