# Presentation: 591678

## A Reality of “Grid” Computing–SamGrid–

**Adam Lyon  ****(Fermilab Computing Division and DØ Experiment)**
*GridKa School’04*
*September, 2004*

- _Outline_
- Introduction
- Use Cases
- Deployment & Usage
- Implementation
- Operations, Monitoring, & Testing
- The Future

**Notes:**
Christian Zeitnitz, Ruediger Berlich, Gunter Quast
I’m going to talk about Grid computing for the Tevatron Run II experiments, in particular the SamGrid system used by CDF and DØ. I put quotes around grid because there’s some debate as to whether or not we’re a true grid system. I’ll let you decide that for yourselves. 

## Data at an HEP Experiment

- Detector (DØ)

- Tape Storage

- Compute Farm

- Collect data
- Reconstruct
- Skim

- Analyze
- Re-reconstruct
- Produce Monte Carlo

**Notes:**
HEP experiments produce and use many times of data
Skims are selected and condensed analysis data files production from production files.

DØ has nearly 600,000 raw files with 1.3 billion events for 250 TB
DØ stores reconstructed data with two formats... 
  Format with all information (DST) amounts to 1,083,742 files for 1.7B events and 400 TB
 Summary format (thumbnail) [collider data] is 221,363 files for 4.913B events for 121 TB [but this includes skimming]	
220K MC thumbnail files with 111M events for 3.6 TB

## Then and now

- For Run I at DØ [1991–1997]:
- Collected about 200 pb-1 of data
- Amounted to 60 TB total (all forms of data)
- “Thumbnail” version of entire data lived on disk
- Almost all processing was done at Fermilab

- For Run II at DØ [2000-]:
-  We have collected 470 pb-1 so far (hope to get 4-8 fb-1 by the end of the run)
- We collect ~1 TB of raw data per day
- We have saved 0.75 Petabytes to tape (expect 10-20+ PB)
- Need to do re-reconstruction and analyses at remote locations
- DØ reads the equivalent of Run 1 data every 11 days and writes Run 1 every 2 months

## What do we need?

- Don’t want to know the details [where files sit, where jobs run] (transparent)
- Find data easily (query tools)
- Solution...
- An integrated data handling and job management system
- A GRID
- SamGrid
- SamGrid = SAM + JIM

- Enormous amounts of data need to be transferred for different activities (scalable)
- ... sometimes over large distances and with non-fault tolerant hardware (robust)
- Knowledge of what we are doing and what we did (monitoring and bookkeeping)
- Use our limited resources effectively both at home and away (efficient)

**Notes:**
Find data easily - cataloging, query tools (useful to the end user)

## What can SamGrid do?

- SAMGrid manages file storage (replica catalogs)
- Data files are stored in tape systems at Fermilab and elsewhere. Files are cached around the world for fast access
- SAMGrid manages file delivery
- Users at Fermilab and remote sites retrieve files out of file storage. SAMGrid handles caching for efficiency
- You don't care about file locations
- SAMGrid manages file metadata cataloging
- SAMGrid DB holds metadata for each file. You don't need to know the file names to get data
- SAMGrid manages analysis bookkeeping
- SAMGrid remembers what files you ran over, what files you processed successfully, what applications you ran, when you ran them and where
- SAMGrid manages jobs
- Choose execution site, deliver job and its needed data, store output

## SamGrid Buzzword Glossary

**Dataset**: metadata description which is resolved through a catalog query to file list. Datasets are named.Examples: (syntax not exact)
- data_type physics  and run_number 78904 and data_tier raw
- request_id 5879 and data_tier thumbnail
**Snapshot**: The list of files that satisfy the Dataset query at a particular time (*e.g. *start of the project)
**Process**: User application (one or many exe instances)Examples: script to copy files; reconstruction job

*A **project** runs on a **station** and requests delivery of a **dataset **snapshot** to one or more **processes** on that station. *
**Project**: Run an application over data
**Station**:
- Has processing power
- Has disk cache
- Can connect to outside world (for file transfers and DB access)
- Examples: Linux analysis cluster at DØ, GridKa’s farm

## Sample Use Cases

- Add Raw Detector Data to SamGrid
- Process Unskimmed Collider Data
- Process Skimmed Collider Data
- Process Missed/New Data
- Monte Carlo Production
- Process Simulated Data

## I. Add Raw Detector Data to SamGrid 

- Raw data collected into files by online detector DAQ
- Online system creates metadata for files
- Run #
- Start time/end time
- Event catalog (triggers)
- Luminosity info
- Online SamGrid station system submits files to SamGrid
- SamGrid stores files onto permanent storage and saves metadata to database

## II. Process Unskimmed Collider Data

- Reconstruct raw data (production)
- Process the direct output of production
- Skimming
- Re-reconstruction
- User defines **dataset** by describing files of interest (not listing file names) using SamGrid command-line or GUI
- data_tier thumbnail and version p14.06.01 and run_type physics and run_qual_group MUO and run_quality GOOD
- User submits project to SamGrid station (two ways)
- User selects station and submits with experiment’s tools
- User submits to SamGrid, SamGrid job management chooses station (execution site) and manages project

## III. Process Skimmed Collider Data

- Someone (a Physics group, the Common Skimming Group, or an individual) has produced skimmed files
- They created a **dataset **that describes these files
- You...
- Submit project using their dataset name OR
- Create a new dataset based on theirs and adding additional constraints
- __set__ DiElectronSkim and run_number 168339
- Submission is same

**Notes:**
Output of a skim is individualistic and cannot use meta-data
Application name / version

## IV. Process Missed/New Data

- The set of files that satisfy the dataset query at a given time is a **snapshot** and is remembered with the SamGrid project information
- One can make new datasets with:
- Files that satisfy a dataset but are newer than the snapshot (new since the last project ran)
- Files that **should** have been processed by the original project but were not consumed
- 	__set__ myDataSet **minus** 	(project_name myProject and  	 consumed_status consumed and consumer lyon)

## V. Monte Carlo Production

- Physics group submits a SamGrid *Request* for MC production, giving parameters. SamGrid assigns a *Request Id. *
- SamGrid chooses execution site
- Workflow manager (*Runjob*) oversees production (event generator, simulator, reconstruction)
- SamGrid launches job to merge output files and submit them into SamGrid catalog and storage

**Notes:**
Runjob will produce metadata for these files. Some go into SamGrid, some do not.

## VI. Process Simulated Data 

- Look up simulation request with parameters of interest
- e.g. Request **5874** has top Monte Carlo generated using Pythia with *m**t** = *174 GeV/*c**2*
- Define **dataset **(via command-line or GUI):
- request_id 5874 and data_tier thumbnail
- Submit project

## SamGrid Deployment

- DØ
- SamGrid is*** THE*** data handling system. Has been in **production **for five years. 45 active SamGrid stations deployed worldwide (including GridKa)
- Moving to SamGrid’s automated job management system(10 execution sites so far)
- CDF
- Completing testing and migration to SamGrid for data handling in production
- Large analysis station at FNAL, 8 major remote stations (Italy, GridKa, Taiwan, Toronto, ...)
- MINOS
- Initial deployment underway
- US-CMS
- Using SamGrid metadata catalog components for proof-of-principle

**Notes:**
MINOS (Neutrino experiment)

## SamGrid Statistics (8/2003-8/2004)

- File delivery and consumption
- DØ (production):
- CDF (testing and initial production):
- Total: 1.5 PB, 12B events
- GridKa largest  offsite SAM consumer
- Can reach peak of 25 TB/day at FNAL

|  | # files (K) | Terabytes | # Events (B) |
| --- | --- | --- | --- |
| Total | 4000 | 2000 | 48.0 |
| Remote | 500 | 142 | 3.8 |
| GridKa | 100 | 47 | 1.6 |

**Notes:**
For CDF (repeat SamGrid to avoid difference between dCache)
CDF GridKa has consumed 50 TB per month - 3-7 TB per day
Remove average per day 

Gridka DØ is 100K Files, 1.6B events, 46.8 TB for the year

Avg per month for all is 390K files, 4.8B events, 193TB
Avg per month remote is 44K files, 316M events, 11TB
Avg per month Gridka is 9.1K files, 143M events, 4.25TB

## DØ SamGrid File Delivery  (Files delivered by month)

**Notes:**
Avg per month for all is 390K files, 4.8B events, 193TB
Avg per month remote is 44K files, 316M events, 11TB
Avg per month Gridka is 9.1K files, 143M events, 4.25TB

## DØ Monte Carlo Production (all remote)

**Notes:**
Top producers

## DØ Past Re-reprocessing

## Implementation of SamGrid

_**Overview**_
- Metadata
- Metadata is the conceptual glue for SamGrid
- Tight coupling
- Database
- Repository of metadata
- DBServers provide easy access
- Services
- Stations, stagers, workers, storage servers, submission sites, execution sites
- Client Side
- The user experience

## The Glue: Metadata

- “SamGrid is a collection of services each of which is described by metadata.” Metadata are **interrelated**.

## SamGrid Database

- DØ, CDF, and MINOS use the _same_** **DB Schema shown here
- Relational
- Matches metadata
- Monolithic
- Interrelated information are close by
- Flexible
- Schema updates are allowed, but are carefully controlled
- Successful!
- In production use at DØ for five years. *It may look scary, but it is well understood and it works!*

**Notes:**
This database schema is PRODUCTION for DØ and CDF and MINOS

## Data Files Metadata

- Data Files: The heart of SamGrid
- Fixed metadata
- File name, size, crc
- Production group
- Data Tier (Raw, Reconstructed, Thumbnail)
- Application
- Locations
- Detector Runs
- Event info
- Project/Process
- Luminosity
- Stream/Trigger
- Connection to free metadata (Params) ...

## Params (Free file metadata)

- Fixed metadata allows easy and performant querying
- Free metadata for application specific items
- Categories group parameters (pythia, isajet, ...)
- Types are the keywords(decayfile, topmass, ...)
- Values
- Queries are more difficult

## Project Metadata

- Projects run on a dataset Snapshot with nodes from a SAMGrid station
- A Project has one or more Consumers (usually one)
- A Consumer has one or more Processes
- A Process is a job on a node. Keeps track of consumed files

**Notes:**
Chuck

## Database Details

- Centralized Oracle Database at FNAL
- Three tier system ensures DB integrity (for all DBs at Fermilab)
- Development - Newest schema with artificial or special data. Used for testing
- Integration - Test new schema with replica of production data
- Production - The real thing

**Notes:**
DB tier -- all DBs at Fermilab do this

## Central vs. Distributed DB Design

- Pros of Central
- Database software easier to write, manage, and control
- DB queries are simpler and more performant
- Cons of Central
- Single point of failure - all data handling can stop
- Hardware and network outages
- Need to apply updates (DØ mitigates with monthly down day)
- Perhaps too monolithic (station must access DB to discover its cache disks)
- Future directions
- Information servers to remotely cache DB information
- Initiative with a small business to produce software to transparently query distributed databases
- But I doubt we’ll split off much of the metadata

**Notes:**
SamGrid is robust against outages!

## DB Servers (Middleware)

- Clients do not connect directly to Oracle but instead go through DB Server middleware
- Use a CORBA Infrastructure (standardize DB access)
- Server written in Python
- Client interfaces with Python and C++
- DBServer Improvements
- Multithreading
- Revamped CORBA Infrastructure

**Notes:**
Remove license limits
No Generator

## DB Server Deployment

- Oracle DB

- User
- Clients
- dbserver

- Remote
- Stations
- dbserver

- FNAL
- Analysis
- Stations

- FNAL
- Reco
- Farm

- Remote
- Stations

- Reco
- Farm

- Analysis
- Stations

*Remote*

*Fermilab*

## SamGrid Data Handling Services

- Head Node

**Station Master**

- Worker node 1

- Worker node 2

- Cache

- Cache

- Stager

- Stager

- pmaster

- DB

- Many station configurations are possible

**Notes:**
Station Master - runs on head node, one instance, persistent, robust; Coordinates file deliveries to the compute farm. Accesses outside world and DB
Pmaster (Project master) - coordinates file deliveries to running processes, tracks file delivery
Stager - Runs on node with cache to manage those disks, Clears old files if room is needed, Initiates file transfers (use sam_cp, wrapper for rcp, grid-ftp)

## SamGrid Data Handling Services

- Station Master
- Runs on head node, one instance, persistent, robust
- Coordinates file deliveries to compute farm
- Accesses the DB server
- Project Master
- Runs on head node (future distributed), one per project
- Coordinates file deliveries to running processes, tracks file consumption
- Stager
- Runs on node with cache to manage those disks
- Clears old files if room is needed
- Initiates file transfers (use s*am_cp*, wrapper for rcp, grid-ftp)

**Notes:**
Robust means the station will continue where it left off if station master is restarted

Station manager manages cache disks

## Project can manage parallel processes
Multiple processes (batch jobs) can pull files from the project’s dataset
Files spread among processes evenly
If a process dies, others pick up the slack

File delivery both optimized and throttled for performance
SamGrid tries to deliver files before the jobs need them (prefetching)
File delivery can start before the processes start
File delivery continues while processes are executing
On FNAL analysis farm, 40% of time process did not need to wait for file
Can set limits on simultaneous transfers
Avoids overloading network

Files may come from multiple sources and different transports
Sources are tape systems (FNAL enstore), other stations, other worker cache disks
Transfers via grid-ftp, kerberized rcp, AFS, ... (wrap with sam_cp)

- Multiple processes (batch jobs) can pull files from the project’s dataset
- Files spread among processes evenly
- If a process dies, others pick up the slack
- File delivery both optimized and throttled for performance
- SamGrid tries to deliver files **before**** **the jobs need them (prefetching)
- File delivery can start before the processes start
- File delivery continues while processes are executing
- On FNAL analysis farm, 40% of time process did not need to wait for file
- Can set limits on simultaneous transfers
- Avoids overloading network
- Files may come from multiple sources and different transports
- Sources are tape systems (FNAL enstore), other stations, other worker cache disks
- Transfers via grid-ftp, kerberized rcp, AFS, ... (wrap with *sam_cp*)

- Special features of Data Handling

**Notes:**
Ability to prefetch is limited by how fast the process executes and how busy the other stations or tape systems are

## Job Information & Management

- Client “site”:
- User writes JDL and submits job to SamGrid
- User closes laptop (laptop only needs submission client software) and gets on plane
- Submission Site:
- Submission site calls on broker to determine execution site (criteria: load, files in cache, ...)
- (Execution sites advertise classads, and connect with SamGrid catalog)
- Submission site transfers job to execution site, job(s) enter local batch system

**Notes:**
User provides JDL, gets transformed by Submission

Sandbox is only a piece -- user sandbox is extra configurations, or a piece of code that you want plugged -- bulk of code is in SamGrid

## Job Information & Management

- Execution site:
- Submission site transfers bootstrap sandbox, is unpacked on head node
- Jobs awaken, SamGrid transfers needed software to node (samClient allows for SamGrid use on vanilla nodes)
- Jobs request data files from SamGrid and run
- Result files stored back into SamGrid. Log files sent back to submission site
- Client
- User lands, opens laptop, retrieves logs from submission site, gets result files out of SamGrid, discovers something new!

## Job Management Details

- Grid (sites talking to each other)
- Control, monitor, and transfer of information between sites
- Uses standard grid tools (Globus: gridftp, gram, mds) and Condor-g
- Fabric (collection of services and resources on site)
- Turned out managing the fabric was the real work for DØ
- Sandboxing, job driving, workflow, setting up application
- SamGrid uses a thick interface to weave the fabric (needs knowledge of application, batch system, ...)
- Thick interface can determine job status, even if job is sleeping - useful for monitoring
- Perhaps this should have been experiment’s responsibility, but...

## Monte Carlo Production via SamGrid Automated Job Management

## User Experience

- Command line tools to query SamGrid services
- sam translate constraints --dim=“data_tier thumbnail”
- Dimension language to shield users from SQL
- Extensible, Improving
- Web interfaces
- DB queries
- Dataset creation
- Command line administrative tools

## Operations, Monitoring & Testing

- SamGrid shifters watch the system and respond to users’ questions/requests
- Cover 18 hours per day
- Shifters in US, Canada, Europe, India, Brazil
- SamGrid experts at Fermilab rotate pager
- Local site SamGrid admins too
- Many tailored tools for monitoring
- Shifters and close monitoring beget much good will from users

**Notes:**
Have about 30 shifters who get control room shift credit

## Sam-At-A-Glance

## SamTV (DØ)

- Quickly check health of projects on FNAL stations
- Can discover if a station is having delivery problems
- Users can check on the status of their projects

## SamTV History

**Notes:**
Station is clued0

## Job Management Monitoring

- XMLDB
- Users can check on job progress

## Future of Monitoring

- Current SamTV parses log files
- Fragile, hard to maintain
- 2nd generation monitoring in the works
- Monitoring and Information Service (MIS)
- MIS server receives events from SamGrid services via Corba (new project, open new file, delete file from cache) or can pull information from service
- MIS Backends process events: store in local DB, send alert e-mail, update real time displays, export to other monioring systems (MonaLisa)

## SamGrid + MonaLisa

## Test Harness

- Unit testing of services is not enough
- Must mimic loads of a production system
- Performance and stress testing
- Discover problems, optimize performance
- Use a dedicated farm with SamGrid Test Harness to load the system
- Automatic tests with pass fail reports
- Check configuration of new installations
- Stress the system and use monitoring for results

## Future of SamGrid

- Continuously refining our system
- Adapting to needs of other experiments
- Minos has two detectors
- Refactoring and improving the implementation
- Adapting further to standard Grid tools
- Writing SamGrid SRM interfaces to access grid storage elements
- Interface to standard monitoring tools (but we need our own specific ones too)

- Moving to use of standard VO authorization
- Open problems
- More advanced brokering algorithms and scheduling
- VO Management - assign roles and attributes to users; finer grained security, temporary special privileges
- Automatically resubmit failed jobs (must be careful)

**Notes:**
Manpower shifting to Grid

## Summary

- SamGrid is a large scale distributed system integrating data delivery and job management for the many Petabyte data size era
- Successfully being used at DØ and CDF, initial deployment for MINOS. US-CMS investigating
- SamGrid continues to move into the Grid era

## EXTRA SLIDES

- Extra slides go here

## V. Re-reconstruction

- Reprocessing group submits projects to SamGrid. SamGrid chooses execution site and launches job(s)
- Jobs are run using *RunJob*, a work flow management system (CMS & DØ)
- Code arrives to job(s) via SamGrid
- Data arrives to job(s) via SamGrid
- Output files are sent back to FNAL for merging and storage back into SamGrid (future - will do on remote site)

**Notes:**
DØ code comes from SamGrid
Sandbox gets bootstrapped
Get rid of left side

## Process Execution Times

## Failures 

- On linux nodes, ~1% files are not sucessfully consumed
- Application crashes (pilot error)
- IDE disk problems (must check CRC after each file transfer)
- Hardware failures
- Temporary no access to certain tapes
- On SMP machine, failure rate is 0.1%
- Hardware and disks are much more robust
- People tend to run standard applications

**Notes:**
Perhaps chuck

## SamTV History

**Notes:**
Station is Clued0

## Process Wait Times

- Time between *Request Next File* and*Open File*
- For CAB and CABSRV1
- 50% of enstore transfers occur within 10 minutes.
- 75% within 20 minutes
- 95% within 1 hour
- For CENTRAL-ANALYSIS and CLUED0
- 95% of enstore transfers within 10 minutes

| Station | CAB | CABSRV1 |
| --- | --- | --- |
| % no wait | 30% | 40% |

**Notes:**
Influenced by:
Transfer throttle on stager
Transfer throttle on station
Network speed & load
Enstore queue length, tape transfer time, network access to enstore
Influenced by:
Transfer throttle on stager
Transfer throttle on station
Network speed & load
Enstore queue length, tape transfer time, network access to enstore

## SAMGrid Statistics - Usage Data

***9000 Projects!***

***233 Different Users!***

**Data from early January 6 until February 24 at DØ**

## SAMGrid Statistics - Usage Data

***~500K Files!***

***~1%***

## SAMGrid Statistics - Usage Data

*Raw*

*Thumbnails + ...*

***256 TB!***

***8.3 Billion Events!***

**Data from early January 6 until February 24 at DØ**

## SAMGrid Statistics - Operations Data

**Notes:**
Note that d0mino use of its cache disk is not counted as a file transfer, so it does not show up at all on the plot (should be local)

## SAMGrid Statistics - Operations Data

## Stress Testing

- There are many station parameters to tune
- Maximum parallel transfers
- Maximum concurrent enstore requests
- Configuration of cache disks
- ...
- We're moving away from d0mino to Linux
- How robust are these linux machines?
- How many projects can they run?
- How many concurrent file transfers can they handle?
- Running test harness on a small cluster to explore SAMGrid parameter space

## SAMGrid Stress Testing

- max transfers =5

- max transfers =1

## SAMGrid Stress Testing

- max transfers =5

- max transfers =1

## SAMGrid Stress Testing

- max transfers =5

- max transfers =1

## ENSTORE Statistics

**0.6 Petabytes** in tape storage!

- Only 5 files unrecoverable (5 GB total;  8ppm loss)  !!!
- One of them was RAW file

## Top Users (Jan 6, 2004 - Feb 24, 2004)

- Top users by # of projects

- Top users by consumed files

## SAMGrid Statistics                 What are people doing?

- Not accurate since
- users must fill in
- application manually
- (and most don't)

## SAMGrid Statistics                      Process wait times

- File Source

## Some SAMGrid buzzwords

- Dataset Definition
- A set of requirements to obtain a particular set of files
*e.g.* data_tier thumbnail and run_number 181933
- Datasets can change over time
- More files that satisfy the dataset may be added to SAMGrid
- Snapshot
- The files that satisfy a dataset at a particular time (e.g. when you start an analysis job)
- Snapshots are static
- Project
- The running of an executable over files in SAMGrid
- Consists of the dataset definition, the snapshot from that dataset definition, and application information
- Bookkeeping data is kept - how many files did you successfully process, where did your job run, how long did it take

## SAM-GRID Projects    

**Active Subprojects:** C++ API, DBServer, JIM, H Stream Reco for CDF, Caching, Chains&Links, CDF DFC, Test Harness, Linux deploy of DBServers, Config Man
**Planned Subprojects:** Request system, Autodest, Further monitoring (MIS)
**Related Subprojects:** d0tools, SBIR II, Condor mods, workflow packages for CDF & D0, Authorization & Accounting
**Recently completed Subprojects:**  Python API, V5.1 Schema Design, Batch Adapter, D0 Online dcache TDP, 1st Gen Monitoring Tools, Data Dimensions Grammar

## DB Servers