# Presentation: 093149

## Computing at FermilabFermilab Onsite ReviewStephen WolbersAugust 7, 2002

## Outline

- Introduction
- Run 2 computing and computing facilities
- US CMS Computing
- GRID Computing
- Lattice QCD Computing
- Accelerator Simulations
- Conclusion/Future

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

**Notes:**
Yet clearly, all of us in HEP communication are in this together. When there’s trouble at one lab, it means trouble for everyone. Two current examples bring that home all too painfully. The LHC’s cost and schedule problems at CERN and the Tevatron’s luminosity struggle at Fermilab have implications for every lab, for DOE and NSF.

On the plus side, we all share in the credit for scientific advances. In this day and age, a particle physics discovery at one lab inevitably incorporates previous advances that have occurred at other labs. I cite a few examples.

Yet, we don’t always do so well at sharing the credit. Let me give you an example.

## Introduction

- Computing at Fermilab:
- Serves the Scientific Program of the laboratory
- Participates in R&D in computing
- Prepares for future activities
- Contributes to nationwide scientific efforts
- Consists of an extremely capable computing facility
- With a very
- 	strong staff

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Introduction (2)

- Fermilab Computing’s emphasis has changed to include more participation in non-FNAL and non-HEP efforts.
- Fermilab plans to leverage efforts in computing and expand the “outlook” and involvement with computer scientists, universities and other laboratories.
- Fermilab is in an excellent position to do this because of its involvement in Run 2 (CDF and D0), CMS, lattice QCD, SDSS, and other programs; and because of its strong computing facilities.

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing/Facilities

- Run 2 Computing is a major activity of Fermilab Computing Division
- The Computing Division also provides support for and facilities for:
- Fixed-Target experiments
- miniBooNE and MINOS
- SDSS and Auger
- US CMS
- R&D efforts for future
- 	experiments and accelerators
- Lattice QCD/theory
- Data Acquisition

**Central Farms Use**

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing/Facilities

- CD also supports and maintains facilities for the laboratory as a whole:
- Networks
- Mass Storage/tape management
- System administration
- Windows 2000
- PREP
- Operations
- Computer security
- Electronics Engineering
- Web
- Help for Beams Division

**Network Architecture**

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing/Facilities

- The activities map into facilities at the laboratory:
- Networks
- Mass Storage robotics/tapedrives
- Large computing farms
- Databases
- Operations
- Support

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing/Facilities

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing/Facilities

- Run 2 computing is a huge effort. Fermilab CD uses Run 2 joint projects to maximize the effectiveness of the CD resources (especially people):
- Enstore storage management system
- Reconstruction farms procurement, management and job scheduling
- Oracle database management
- C++ code development
- Compilers and debuggers and code build systems
- ROOT for analysis
- SAM data handling system
- Analysis hardware purchases

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## D0 computing systems

**D0 computing systems**

**12-20 MBps**

**12-20 MBps**

**100+ MBps**

**400+ MBps**

**“****data-logger”**

**“****farm”**

**“****central-analysis”**

- Enstore
- Mass Storage System

**“****linux-analysis**
**-clusters”**

**“****linux-build-cluster”**

**“****clueD0” ~100**
**desktops **

**“****d0-test” and**
**“****sam-cluster”**

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing/Futures

**Data Volume doubles every 2.4 years**

**Lots of Data!**

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Status of Run 2 computing

- Total datasets ~ 500 TB /yr /expt    (including raw, reconstructed,derived and simulated data)
**Both experiments**** are logging data reliably and moving data in and out of mass storage on a scale well beyond Run I capability (several TB’s / day) **
**Both experiments**** are reconstructing data approximately in real time with reasonable output for start-up analyses **
**Both experiments**** are providing analysis CPU to 150-300 users/day**** **

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing

- D0 has a concept of Regional Analysis Centers (RAC’s):
- Distributed analysis around the globe

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Run 2 Computing

**Off-Site**
**ESNET**
**Traffic**

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## US CMS Computing

- Fermilab is the host lab of U.S. CMS
- Fermilab hosts the project management for the U.S. CMS Software and Computing Program in DOE
- L1 project manager (Lothar Bauerdick)
- L2 projects
- User Facilities (Tier 1 and Tier 2 centers)
- Core Application Software

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## US CMS Computing

- Fermilab and US CMS has been extremely successful and influential in CMS software and computing
- Influence on CMS software
- Framework
- Persistency
- CMS distributed production environment
- Tier 1 and Tier 2 regional centers
- Tier 1 at Fermilab
- Tier 2 in a small set of universities

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## US CMS Computing

- Tier 1/Tier 2 Centers and Prototypes

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## US CMS Computing

- Contribution to CMS is substantial
- Spring production 2002
- 8.4 million events fully simulated
- 50% in US
- 29 TB processed
- 14 TB in US

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## US CMS Computing

- The US CMS Software and Computing project has been reviewed and “baselined”
- Funding has been lower than hoped for and this has led to:
- Slower ramp-up of staff
- Smaller hardware purchases
- Fewer projects being completed
- It is hoped that funding will improve so that the necessarily people and facilities can be hired/built to properly prepare for CMS data-taking and physics analysis

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## GRID Computing

- Fermilab is participating in many GRID initiatives:
- ppdg (DOE SciDAC)
- Particle physics data grid
- Fermilab, SLAC, ANL, BNL,
- 	JLAB, Caltech, UCSD,
- 	Wisconsin, SDSC
- Griphyn
- Grid physics network
- iVDGL
- International virtual grid laboratory
- GRID activities have been a very natural outgrowth of distributed computing of the large collaborations.

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## GRID Computing

- Working on GRID computing projects has many benefits for Fermilab computing
- Connection with a leading technology in computing
- Connection with computer scientists in universities and labs in the US and around the world
- Interaction and participation in GRID initiatives and software development.
- Experiment participation in GRID testbeds and production facilities
- D0, CMS, SDSS, CDF

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## GRID Computing

- SciDAC funding for Fermilab is devoted to:
- D0 GRID work
- CMS production activities
- Authentication Issues
- Mass Storage access across the GRID
- SDSS is working on projects associated with
- SciDAC
- Griphyn
- iVDGL

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## GRID Computing

- Fermilab CD is involved in many projects:
- File Replication, transfers and data management
- Scheduling of processing and analysis applications
- Monitoring
- Authentication
- Application and application support
- HEP experiments provide excellent testbeds for the GRID middleware:
- HEP experiments have data
- HEP experiments have people all over the world that want to access and analyze that data

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Lattice QCD

- SciDAC-funded Lattice QCD facility at Fermilab
- Part of a nationwide coordinated effort to provide facilities and software for lattice QCD calculations (Computational Infrastructure for Lattice Gauge Theory)
- Fermilab
- Jefferson Lab
- Columbia/RIKEN/BNL (QCD on a chip, QCDOC)
- Fermilab is pursuing PC-based solutions along with software from the MILC collaboration.

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Lattice QCD

**Improvements from**
**Lattice QCD **
**Calculations**
**(B. Sugar)**

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Lattice QCD

- Fermilab’s contributions
- Large PC clusters
- Acquisition (128 nodes now)
- Operation and maintenance
- Interconnect technologies
- Commercial/Myrinet
- Ethernet R&D
- Software infrastructure
- Joint effort with MILC collaboration

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Lattice QCD

**Muon Lab**

**Room To Expand**

**Complex Interconnects**

**128 Nodes**

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Lattice QCD

- Goals
- Larger clusters
- 128 more nodes within 6 months (0.5 Tflops)
- Substantially larger within years (>6 Tflops)
- Software framework and support
- Facility for research on Lattice problems

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Accelerator Simulations

- Fermilab is collaborating on a SciDAC program of accelerator simulations
- Other collaborators:
- BNL, LANL, LBNL, SLAC, SNL
- Stanford, UCLA, UCD, USC, Maryland
- Plan to work on many topics
- Designing next-generation accelerators
- Optimizing existing accelerators
- Developing new accelerator technologies

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Accelerator Simulations

- Incorporate Ionization Cooling modules in Beam Dynamics Codes
- Develop code with 3D space-charge capabilities for circular accelerators
-   (in collaboration with BD theory)
-  Model & study the FNAL Booster.
- Beam data comparison with
-     simulation: almost unique in
-     this business!
- (in collaboration with Booster group)

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002

## Summary

- Fermilab Computing is in an excellent position to support the physics program of the laboratory and to participate in and foster partnerships with our collaborators, computer scientists and other institutions.
- By participating in these programs and by building working relationships we are in a position to better extend our capabilities in:
- Distributed and GRID computing (important for most of our collaborations)
- Lattice QCD
- Beam simulations

- FERMILAB ONSITE REVIEW

- AUGUST 7, 2002