# Presentation: 894420

## Particle Physics Data Grid

- Richard P. Mount
- SLAC
- DoE NGI PI Program PI Meeting
- Washington DC,  October 4, 1999

## Particle Physics Data GridUniversities, DoE Accelerator Labs, DoE Computer Science

- Particle Physics: a Network-Hungry Collaborative Application
- Petabytes of compressed experimental data;
- Nationwide and worldwide university-dominated collaborations analyze the data;
- Close DoE-NSF collaboration on construction and operation of most experiments;
- The PPDG lays the foundation for lifting the network constraint from particle-physics research.
- Short-Term Targets:
- High-speed site-to-site replication of newly acquired particle-physics data (> 100 Mbytes/s);
- Multi-site cached file-access to thousands of ~10 Gbyte files.

## Slide 3

## PPDG Collaborators

-                                              Particle          Accelerator           Computer                                              Physics           Laboratory             Science
- ANL	X		X
- LBNL                            	X		X
- BNL	X	X	x
- Caltech	X		X
- Fermilab	X	X	x
- Jefferson Lab	X	X	x
- SLAC	X	X	x
- SDSC			X
- Wisconsin			X

## Particle Physics Computing Challenges

-  Geographical dispersion: of people and resources
-  Complexity: the detector and the data
-  Scale: 	     Petabytes per year of data per experiment

**1750  Physicists**
**  ****150  Institutes**
**    ****32  Countries**

**Major challenges associated with:**
**Communication and collaboration at a distance**
**Distributed computing resources **
**Remote software development and physics analysis**

- Example: CMS Experiment

## Complex Detectors

- Complex Collision Products

## Particle Physics Data Models

**Particle physics data models are complex!**
**Rich hierarchy of hundreds of complex data  types (classes)**
**Many relations between them**
**Different access patterns ****	****(Multiple Viewpoints)**

**Event**

**TrackList**

**Tracker**

**Calorimeter**

**Track**

**Track**

**Track**

**Track**

**Track**

**HitList**

- Hit

- Hit

- Hit

- Hit

**Hit**

## Data Volumes

- Quantum Physics yields predictions of probabilities;
- Understanding physics means measuring probabilities;
- Precise measurements of new physics require analysis of hundreds of millions of collisions (each recorded collision yields ~1Mbyte of compressed data)

## Access Patterns

- Raw Data ~1000 Tbytes

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- Reco-V1 ~1000 Tbytes

- Reco-V2 ~1000 Tbytes

- ESD-V1.1 ~100 Tbytes

- ESD-V1.2 ~100 Tbytes

- ESD-V2.1 ~100 Tbytes

- ESD-V2.2 ~100 Tbytes

**Access Rates ****(aggregate, average)**
- 100 Mbytes/s (2-5 physicists)
- 1000 Mbytes/s (10-20 physicists)
- 2000 Mbytes/s (~100 physicists)
- 4000 Mbytes/s (~300 physicists)

- Typical particle physics experiment in 2000-2005:On year of acquisition and analysis of data

## Data Grid Hierarchy Regional Centers Concept

_**LHC Grid Hierarchy Example**_
** ****Tier0: CERN**
** ****Tier1: National ****	****“Regional” Center**
** ****Tier2: Regional ****	****Center**
** ****Tier3: Institute ****  Workgroup Server**
** ****Tier4: Individual ****	****Desktop**
_**Total 5 Levels**_

## PPDG as an NGI Problem

- 	**PPDG Goals**
- 	***The ability to query and partially retrieve hundreds of terabytes across Wide Area Networks within seconds, ***
***Making effective data analysis from ten to one hundred US universities possible.***
- 	_**PPDG is taking advantage of NGI services in three areas:**_
**Differentiated Services:**** to allow particle-physics bulk data transport to coexist with interactive and real-time remote collaboration sessions, and other network traffic.**
**Distributed caching:**** to allow for rapid data delivery in response  to multiple “interleaved” requests**
**“****Robustness”:  Matchmaking and Request/Resource                         co-scheduling:**** to manage workflow and use computing and net resources efficiently; to achieve high throughput**

## First Year PPDG Deliverables 

- 	**Implement and Run two services in support of the major physics experiments at BNL, FNAL, JLAB, SLAC:**
**“****High-Speed Site-to-Site File Replication Service”;****                      ****	****Data replication up to 100 Mbytes/s**
**“****Multi-Site Cached File Access Service”: ****Based on deployment of file-cataloging, and transparent cache-management and data movement middleware **
**First Year:**** Optimized cached read access to file in the range of 1-10 Gbytes, from a total data set of order ****One Petabyte**
**	****Using middleware components already developed                         by the Proponents**

## PPDG Site-to-Site Replication Service

-  Network Protocols Tuned for High Throughput
-  Use of DiffServ  **for **	(1) Predictable high priority delivery of high - bandwidth 		 data streams	(2) Reliable background transfers
-   Use of integrated instrumentation  	to detect/diagnose/correct problems in long-lived high speed 	transfers [NetLogger + DoE/NGI developments]
-   Coordinated reservaton/allocation techniques 		for storage-to-storage performance

**SECONDARY SITE**
**CPU, Disk, **
**Tape Robot**

**PRIMARY SITE**
**Data Acquisition,**
**CPU, Disk, **
**Tape Robot**

## Typical HENP Primary Site ~Today (SLAC)

- 15 Tbytes disk cache
- 800 Tbytes robotic tape capacity
- 10,000 Specfp/Specint 95
- Tens of Gbit Ethernet connections
- Hundreds of 100 Mbit/s Ethernet connections
- Gigabit WAN access.

## Slide 15

## PPDG Multi-site Cached File Access System

**University**
**CPU, Disk, **
**Users**

**PRIMARY SITE**
**Data Acquisition,**
**Tape, CPU, Disk, Robot**

**Satellite Site**
**Tape, CPU, Disk, Robot**

**Satellite Site**
**Tape, CPU, Disk, Robot**

**University**
**CPU, Disk, **
**Users**

**University**
**CPU, Disk, **
**Users**

**Satellite Site**
**Tape, CPU, Disk, Robot**

## PPDG Middleware Components 

## First Year PPDG “System” Components 

_**Middleware Components (Initial Choice): See PPDG Proposal Page 15**_
**   ****	****Object and File-Based ****	****Objectivity/DB (SLAC enhanced)  Application Services ****	****GC Query Object, Event Iterator,****	****   ****	****	****	****     ****	**** Query Monitor**
** ****	****	****	****	****FNAL SAM System**
**    ****Resource Management****	****Start with Human Intervention****	****	****(but begin to deploy resource discovery & mgmnt tools)**
**    ****File Access Service ****	****Components of OOFS (SLAC)**
**    ****Cache Manager****	****GC Cache Manager (LBNL)**
**    ****Mass Storage Manager****	****HPSS, Enstore, OSM (Site-dependent)**
**    ****Matchmaking Service              ****	****Condor (U. Wisconsin)**
**    ****File Replication Index              ****	****MCAT (SDSC)**
**    ****Transfer Cost Estimation Service****	****Globus (ANL)**
**    ****File Fetching Service****	****Components of OOFS**
**    ****File Movers(s)   ****	****SRB (SDSC); Site specific**
**    ****End-to-end Network Services****	****Globus tools for QoS reservation**
**    ****Security and authentication****	****Globus (ANL) **

## Slide 19

## PPDG First Year Milestones

- Project Start	August, 1999
- Decision on existing middleware to be 	October, 1999	integrated into the first-year Data Grid;
- First demonstration of high-speed 	January, 2000 site-to-site data replication;
- First demonstration of multi-site 	February, 1999 cached file access (3 sites);
- Deployment of high-speed site-to-site 	July, 2000 data replication in support of two particle-physics experiments;
- Deployment of multi-site cached file 	August, 2000 access in partial support of at least two particle-physics experiments.

## PPDG Longer-Term Goals

- Agent Computing
- on
- Virtual Data

## Why Agent Computing?

_**LHC Grid Hierarchy Example**_
** ****Tier0: CERN**
** ****Tier1: National ****	****“Regional” Center**
** ****Tier2: Regional ****	****Center**
** ****Tier3: Institute ****  Workgroup Server**
** ****Tier4: Individual ****	****Desktop**
_**Total 5 Levels**_

## Why Virtual Data?

- Raw Data ~1000 Tbytes

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- AOD ~10 TB

- Reco-V1 ~1000 Tbytes

- Reco-V2 ~1000 Tbytes

- ESD-V1.1 ~100 Tbytes

- ESD-V1.2 ~100 Tbytes

- ESD-V2.1 ~100 Tbytes

- ESD-V2.2 ~100 Tbytes

**Access Rates ****(aggregate, average)**
- 100 Mbytes/s (2-5 physicists)
- 1000 Mbytes/s (10-20 physicists)
- 2000 Mbytes/s (~100 physicists)
- 4000 Mbytes/s (~300 physicists)

- Typical particle physics experiment in 2000-2005:On year of acquisition and analysis of data

## Existing Achievements

- SLAC-LBNL memory-to-memory transfer at 57 Mbytes/s over NTON (RIP1);
- Caltech tests of writing into Objectivity DB at 175 Mbytes/s
- 1) Not really moribund, but many decayed connections.

## Cold Reality(Writing into the BaBar Object Database at SLAC)

- 60 days ago:  ~2.5 Mbytes/s

- 3 days ago:
- ~15 Mbytes/s

## Testbed Requirements

- Site-to-Site Replication Service
- 100 Mbyte/s goal possible through the resurrection of NTON (SLAC-LLNL-Caltech-LBNL are working on this).
- Multi-site Cached File Access System
- Will use OC12, OC3, (even T3) as available
- (even 20 Mits/s international links)
- Need “Bulk Transfer” service:
- Latency unimportant
- Tbytes/day throughput important (Need prioritzed service to achieve this on international links)
- Coexistence with other network users important. (This is the main PPDG need for differentiated services on ESnet)

## How the PPDG Intends to Maximize its Impact and Relevance

- Develop and evolve a long-term plan for particle physics data management based on full exploitation of and contribution to relevant CS:
- APOGEE draft proposal to DoE HENP     (A Physics-Optimized Grid Environment for Experiments)
- Proposed common project management for APOGEE and PPDG

## APOGEE in Context

- Construction and Operation of HENP Data Management and Data Analysis Systems at DoE Laboratories
- Tiers 0/1

- GriPhyN
- HENP Data Manage-ment at Major University Centers
- Tier 2

- OO Databases and Analysis Tools

- Resource Management Tools

- Metadata Catalogs

- WAN Data Movers

- Mass Storage Management Systems

- Matchmaking

- PPDG
- Particle Physics Data Grid
- NGI Project

- Unified Project Management

- Optimization and Evaluation

- Instrumentation

- Modeling and Simulation

- A new level of rigor as the foundation for future progress

- APOGEE