# Presentation: 324686

## US ATLAS Distributed IT Infrastructure

- Rob Gardner
- Indiana University
- October 26, 2000
- http://atlassw1.phy.bnl.gov/rwg-Planning/dit.html

## Distributed IT Infrastructure

- Software
- “Grid” toolkits: PPDG, GriPhyN, DataGrid
- ATLAS extensions and adaptors
- Tier 2 regional centers
- Networks

- US ATLAS Distributed IT

- 10/26/00

## GriPhyN

- 4 physics experiments + leading computer scientists in distributed computing
- ITR R&D project: 11.9M/5y
- ATLAS resources:
- 1 postdoc (physicist-computation)
- 2 grad students
- Significant matches from IU and BU
- ITP2 (both);  0.5FTE of several IU IT personnel (IU)

- US ATLAS Distributed IT

- 10/26/00

## Typical Tier 2 Regional Center(1 of 5)

- CPU: 50K SpecInt95                                (t1: 209K)
- Commodity Pentium/Linux
- Estimated 144 Dual Processor Nodes     (t1: 640) Online Storage: 70 TB Disk                    (t1: 365)
- High Performance Storage Area Network
- Baseline: Fiber Channel Raid Array

- US ATLAS Distributed IT

- 10/26/00

## Tertiary Storage Capability

- Exploit existing mass store infrastructure at 2 of the 5 Tier 2 centers
- Assume existing HPSS or equivalent liscense, tape silo, robot
- Augment with drives, media, mover nodes, and disk cache
- Each site contributes 0.3-0.5 PB store
- Reprocessed ESDs, User AODs,

- US ATLAS Distributed IT

- 10/26/00

## Timeline (follows Tier 1) 

- R&D Tier 2’s – FY ‘01 & FY ‘02
- Initial Development & Test, 1% to 2% scale
- Start Grid testbeds:  ATLAS-GriPhyN, PPDG, DataGrid
- Data Challenges – FY ‘03 & FY ‘04
- Production Tier 2’s  – FY ‘04 & FY ‘05
- Operation – FY ‘05, FY ‘06 & beyond
- Full Scale System Operation, 20% (‘05) to 100% (‘06)

- US ATLAS Distributed IT

- 10/26/00

## Slide 7

- US ATLAS Distributed IT

- 10/26/00

## Slide 8

- US ATLAS Distributed IT

- 10/26/00

## Slide 9

- US ATLAS Distributed IT

- 10/26/00

## Tier 2 Costs

- Facilities and labor

- $K

- US ATLAS Distributed IT

- 10/26/00

## ATLAS Grid Related Activities

## ATLAS Grid Workshops

- June 14-15 at Indiana University
- ATLAS-GriPhyN Testbed
- ATLAS requirements for grid software.
- Identify APIs between ATLAS and grid services.
- Specify grid related milestones.
- Identify deliverables for MOU, WBS documents.
- July 17-18 at CERN
- First ATLAS-wide grid workshop
- Talks from most major grid efforts in HEP
- September 29-30 at CERN
- Focus attention on ATLAS requirements

- http://atlasinfo.cern.ch/Atlas/GROUPS/SOFTWARE/OO/grid/

- US ATLAS Distributed IT

- 10/26/00

## Participants at June Testbed Workshop

- Lab:
-  Larry Price (ANL)
-  Ed May (ANL)
-  Rich Baker (BNL)
-  Stu Loken (LBL)
-  David Malon (ANL)
-  Craig Tull (LBL)
-  Bruce Gibbard (BNL)
-  Torre Weanus (BNL)
-  David Quarrie (LBL)
-  Bill Allcock (ANL)

- University:
-  John Huth (HU)
-  Rob Gardner (IU)
-  Fred Luehring (IU)
-  Shawn McKee (UM)
-  Jim Shank (BU)
-  Steve Wallace  (IU)
-  Leigh Grundhoefer (IU)
-  Thom Sulanke (IU)
-  Mary Papakian (IU)
-  Jane Liu (IU)

- US ATLAS Distributed IT

- 10/26/00

## ATLAS GriPhyN Goals

- Provide linkage between Athena, database, simulation framework, and the grid toolkits
- Feedback to software developers in both communities (ATLAS core developers and grid toolkit developers)
- Develop an ATLAS-GriPhyN Test-bed
- Validate distributed computing model for LHC computing
- Provide input to new models by testing tools and distributed functionality of ATLAS software
- Provide input to planning for facilities development (at each Tier) and networks

- US ATLAS Distributed IT

- 10/26/00

## Who’s doing what in the US?

- Participants and projects
- Argonne:
- PPDG activities, Database-grid
- GriPhyN-DataGrid Integration
- Berkeley Lab
- Athena grid interfaces, DOE sciences grid
- Brookhaven Lab
- Tier 1 development, file replication, grid requirements document
- Boston U
- Globus evaluations, applns
- Harvard U
- US ATLAS computing management, planning
- U of Michigan
- Globus interfaces, QoS authentication
- Indiana U
- Athena grid interfaces, Testbed coordination, GIS

-  UTA: D0 cluster, SAM
-  SUNY Albany  - test bed

- US ATLAS Distributed IT

- 10/26/00

## Gaudi (Athena) Control Framework

**Converter**

**Algorithm**

**Event Data**
**Service**

**Persistency**
**Service**

**Algorithm**

**Algorithm**

**Transient Event Store**

**Detec. Data**
**Service**

**Persistency**
**Service**

**Transient Detector **
**Store**

**Message**
**Service**

**JobOptions**
**Service**

**Particle Prop.**
**Service**

**Other**
**Services**

**Histogram**
**Service**

**Persistency**
**Service**

**Transient**
**Histogram Store**

**Application**
**Manager**

**Converter**

**Converter**

- US ATLAS Distributed IT

- 10/26/00

## Grid vs. Gaudi Services

- Craig Tull
- NERSC/LBL

- Athena and Grids:  http://documents.cern.ch/age?a00380

- US ATLAS Distributed IT

- 10/26/00

## ATLAS-GriPhyN Testbed

- Platform for testing grid concepts, computing models
- Provide input back to grid developers
- Expose weaknesses to better plan for infrastructure upgrades
- Identify and specify application-grid services interfaces
- Developers, administrators, users need grid experience
- Perform realistic test cases and make available as a test-suite
- Prepare infrastructure for ATLAS data challenges
- Distributed Monte Carlo production for TDRs

- US ATLAS Distributed IT

- 10/26/00

## Initial Participants

**UC Berkeley**
**LBNL-NERSC**

**Brookhaven**
**National**
**Laboratory**

**Indiana **
**University**

**Boston **
**University**

**Argonne**
**National**
**Laboratory**

- HPSS sites

**U Michigan**

**University of**
**Texas at**
**Arlington**

**SUNY**
**Albany**

- US ATLAS Distributed IT

- 10/26/00

## ATLAS Applications

- Tile Cal testbeam data (expand on Ed May’s PPDG work)
- TRT System Test Data (Module Production –Fred Luehring)
- Monte Carlo Production
- Evaluation of Data Distribution Models
- Data Cataloging, Metadata, Bookkeeping

- US ATLAS Distributed IT

- 10/26/00

## ATLAS Grid Schedules

## ATLAS Data Challenges “DC”

- DC 1 first half 2002
- first project releases:
- of DATAGRID (PM12) is January 2002
- of GriPhyN **VDT-1** October 2001
- ATLAS DC Milestones
- were written on the safe side, but we will try to be more ambitious
-  involve more sites and more complexity
- further define DC1 in next months getting involved GRID people and sites
- Computing TDR ready by end 2002: all experimentation needed before TDR to be planned for DC1

- US ATLAS Distributed IT

- 10/26/00

## ATLAS Schedule Summary

- Significant milestones
-  **11/01 – 12/01**  Data Challenge 0
-  **01/02 – 07/02**  Data Challenge 1
-  Provide input to computing model
-  **05/02 – 11/02**  Computing TDR
-  **01/03 – 09/03**  Data Challenge 2
- Major tests of grid enabled software
-  **01/04 – 06/06**  Physics Readiness 				  Report

- US ATLAS Distributed IT

- 10/26/00

## ATLAS GriPhyN Milestones (1)

- Year 1
- Perform tests using Globus and Particle Physics Data Grid tool kits providing basic file replication, transport, and security infrastructure.  Data samples of order (1 TB) from test beam data recorded at CERN will be used in these tests.
- Participate in development of information models for data types and elements.  Develop metadata file catalogs, and tools to manipulate them to organize existing test beam and Monte Carlo data sets.
- Begin implementing an ATLAS-GriPhyN test bed involving CERN and several US ATLAS institutions, including,  Brookhaven, Argonne, and Berkeley Labs; Boston University, Indiana University, University of Michigan,  UT Arlington,  and SUNY-Albany
- Begin identification and specification of APIs for the ATLAS analysis framework (Athena) and Monte Carlo programs.  Evaluate and adopt basic grid services from the **VDT-1** toolkit as they become available.
- Provide feedback to CS researchers designing and planning VDT-2 services.

- ’00-’01

- US ATLAS Distributed IT

- 10/26/00

## ATLAS GriPhyN Milestones (2) 

- Year 2
- Deploy and evaluate **VDT-2** (Centralized Virtual Data Services) on existing ATLAS-GriPhyN testbed focusing on support of the Athena analysis framework and Monte Carlo simulation programs.
- Provide support to physicists located at many US sites and CERN       requiring access to distributed data sets using metadata file      catalogs, high speed multi-site file replication and network       caching services.  Testbeam data and Monte Carlo data data will be used, ranging in size of 2 to 10 TB.
- Incorporate request planning and execution services into Athena framework.  Gather policy and resource information for      participating nodes on the grid.
- Continue to expand US ATLAS testbed.
- Tests of distributed object databases across several sites in the ATLAS-GriPhyN testbed.

- ’01-’02

- DC1

- DC0

- US ATLAS Distributed IT

- 10/26/00

## ATLAS GriPhyN Milestones (3) 

- Year 3
- Athena framework becomes fully distributed: incorporate **VDT-3** toolkit (distributed virtual data services).
- Perform  tests of the hierarchy in a production mode.  This will involve the CERN (as the Tier 0 site), Brookhaven Lab (as the US ATLAS Tier 1 site), and Tier 2 regional data centers
- Begin multi-site distributed simulations to generate O(100) TB Monte  Carlo data samples for physics studies and Data Challenges.
- Continue to provide feedback especially with regard to fault tolerance  in practical, realistic planning and request executions.

- ’02-’03

- DC2

- TDR

- US ATLAS Distributed IT

- 10/26/00

## ATLAS GriPhyN Milestones (4) 

- Year 4
- Scaled versions of previous Mock Data Challenges using **VDT-4** (Scalable Virtual Data Services) using existing testbed between CERN, the Tier 1 center at BNL, Tier 2 regional centers, and of order 20 Tier 4 university groups with physicists doing Monte Carlo studies and analysis of testbeam data.
- The goal is to approach realistic data samples >O(500 TB) and to involve thousands of computers.
- Full integration of ATLAS analysis tools such as Athena and the Monte Carlo production control framework with virtual data services.

- ’03-’04

- DC3

- US ATLAS Distributed IT

- 10/26/00

## ATLAS GriPhyN Milestones (5) 

- Year 5
- Continue to build on experience from data challengs in previous years.
- Build a production quality offline distributed data analysis for the ATLAS GRID using GriPhyN tools.

- ’04-’05

- Physics
- Readiness
- Report

- US ATLAS Distributed IT

- 10/26/00

## ATLAS Grid Requirements

- EU DataGrid
- To produce a grid requirements document June 1, 2001
- ATLAS:
- A group has been formed led by Rich Baker (BNL) and Larry Price (ANL) to produce an ATLAS grid requirements document by November 1, 2000 (draft)

- US ATLAS Distributed IT

- 10/26/00

## ATLAS – CS Collaborative Projects

## Possible CS-Physics Projects

- Athena grid interface
- Toolkit installation & distribution kit (a la INFN)
- Collaboration w/ DataGrid WP8 through grid forum, etc.
- Grid information services: user registration interface

- US ATLAS Distributed IT

- 10/26/00

## Summary of Grid Activities

- ATLAS fully supports grid efforts and is redesigning its computing model in this new context
- Significant involvement of ATLAS personnel in PPDG, DataGrid, GriPhyN
- This involvement will grow as more US ATLAS institutions become involved
- GriPhyN needs to develop a clear and open policy for their contributions

- US ATLAS Distributed IT

- 10/26/00