# Presentation: 725861

## Computing at CDF

_**Computing at CDF**_

** ****Introduction**
** ****Computing requirements**
** ****Central Analysis Farm**
** ****Conclusions**

**Frank Wurthwein**
*MIT/FNAL-CD*
- for the CDF Collaboration

## CDF in a Nutshell

_**CDF in a Nutshell**_

***Frank Wurthwein/MIT***

***LCCWS '02***

** ****CDF + D0 experiments analyze pp collisions from Tevatron at Fermilab**
** ****Tevatron highest energy collider in world (           ****TeV****) until LHC**
** ****Run I (1992-1996) huge success ******** 200+ papers (t quark discovery, ...)**
** ****Run II (March 2001-) upgrades for luminosity (********10) + ****energy (~10%********)**
-  expect integrated luminosity 20 (Run IIa) and 150 (Run IIb) of Run I

** ****Run II physics goals:**
**Search for Higgs boson**
**Top quark properties** (mt, tot, ...)
**Electroweak** (mW, W, ZZ, ...)
**Search for new physics** (e.g. SUSY)
**QCD at large Q****2** (jets, s, ...)
**CKM tests in *****b***** hadron decays**

## CDF RunII Collaboration

_**CDF RunII Collaboration**_

***LCCWS'02***

**Goal**: 	**Provide computing resources for 200+ collaborators**
- 		s**imultaneously doing analysis per day!**

***Frank Wurthwein/MIT***

## CDF DAQ/Analysis Flow

***Frank Wurthwein/MIT***

***LCCWS'02***

**CDF**

**Level 3 Trigger**

**Production Farm**

**Central Analysis Facility**

**(CAF)**

**User**

**Desktops**

**Robotic**

**Tape Storage**

_**CDF DAQ/Analysis Flow**_

**Data**

**Analysis**

**7MHz beam Xing**

**0.75 Million channels**

**300 Hz**

**L1**
- 
**L2**
- 

**75 Hz**

**20 MB/s**

**Recon**

**Read/write**

**Data**

**MC**

## Reconstruction Farms

_**Reconstruction Farms**_

***Frank Wurthwein/MIT***

***LCCWS'02***

**Data reconstruction + validation, Monte Carlo generation**
**	****154 dual P3's (equivalent to 244 1 Ghz machines)**
**	****Job management:**
**Batch system ********FBSNG developed at FNAL**
**Single executable, validated offline**

**150 Million events**

## Data Handling

_**Data Handling**_

***Frank Wurthwein/MIT***

***LCCWS'02***

**Data archived using STK 9940 drives and tape robot**
**Enstore: ****N****etwork-attached tape system developed at FNAL**
**	********provides interface layer for staging data from tape**

**5 TB/day**

**100 TB**

***Today: 176TB on tape***

## Database Usage at CDF

_**Database Usage at CDF**_

***Frank Wurthwein/MIT***

***LCCWS'02***

**Oracle DB: Metadata + Calibrations**
**   ****DB Hardware:**
**2 Sun E4500 Duals**
**1 Linux Quad**
**Presently evaluating:**
**MySQL**
**Replication to remote sites**
**Oracle9 streams, failover, load balance**

## Data/Software Characteristics

_**Data/Software Characteristics**_

***LCCWS'02***

**Data Characteristics:**
**Root I/O sequential for raw data: ****~250 kB/event**
**Root I/O multi-branch for reco data: ****50-100 kB/event**
**'Standard' ntuple: ****5-10 kB/event**
**Typical RunIIa secondary dataset size: ****10****7**** events**** **
**Analysis Software:**
**Typical analysis jobs run @ **** 5 Hz**** on 1 GHz P3**
**	****	******** ****few MB/sec**
**CPU rather than I/O bound (FastEthernet)**

## Computing Requirements

_**Computing Requirements**_

***LCCWS'02***

**Requirements set by goal:**
**	****200 simultaneous users **t**o analyze secondary data set (10****7**** evts) in a day**
**Need ~****700 TB**** of disk and ~****5 THz**** of CPU by end of FY'05:**
**	********need lots of disk******** ****need cheap disk**** ********IDE Raid**
**	********need lots of CPU********commodity CPU**** ********dual Intel/AMD**** **

## Past CAF Computing Model

_**Past CAF Computing Model**_

***LCCWS'02***

**Very expensive to expand and maintain**
**Bottom line: **
**	****Not enough 'bang for the buck'**

## Design Challenges

_**Design Challenges**_

***LCCWS'02***

**	****develop/debug interactively @ remote desktop**
******code management & rootd**
**	****Send binary & 'sandbox' for execution on CAF**
******kerberized gatekeeper**
**	****no user accounts on cluster **
**BUT**
**user access to scratch space with quotas**
******creative use of kerberos**

## CAF Architecture

_**CAF Architecture**_

***LCCWS'02***

**Users are able to:**
**submit jobs**
**monitor job progress**
**retrieve output**
**from 'any desktop' in the world**

## CAF Milestones

_**CAF Milestones**_

**11/01**
**2/25/02**
**3/6/02**
**4/25/02**
**5/30/02**

** ****Start of CAF design**
** ****CAF prototype (protoCAF)**
**assembled**
** ****Fully-functional prototype system (>99% job success)**
** ****ProtoCAF integrated into Stage1 system**
** ****Production Stage1 CAF for**
**collaboration**

**Stage1**

**ProtoCAF**

**Design ********Production system in 6 months!**

***LCCWS'02***

## CAF Stage 1 Hardware

_**CAF Stage 1 Hardware**_

***LCCWS'02***

**Worker Nodes**

**File Servers**

**Code Server**

**Linux 8-ways**
**(interactive)**

## Stage 1 Hardware: Workers

_**Stage 1 Hardware: Workers**_

***LCCWS'02***

**Workers**** (****132**** ****CPUs****, 1U+2U rackmount):**
**	****16 2U Dual Athelon 1.6GHz / 512MB RAM**
**	****50 1U/2U Dual P3 1.26GHz / 2GB RAM**
**	****FE (11 MB/s) / 80GB job scratch each**

## Stage 1 Hardware: Servers

_**Stage 1 Hardware: Servers**_

***LCCWS'02***

**Servers ****(****35TB total****, ****16 4U rackmount):**
**	****2.2TB useable IDE RAID50 hot-swap**
**	****Dual P3 1.4GHz / 2GB RAM**
**	****SysKonnect 9843 Gigabt Ethernet card**

## File Server Performance

_**File Server Performance**_

**Server/Client Performance****: ****Up to ****200MB/s local reads, 70 MB/s NFS**
**Data Integrity tests****: md5sum of local reads/writes under heavy load **
**	****BER read/write =  1.1+- 0.8 ********10****-15  ****/ 1.0+- 0.3 ********10****-13**
**Cooling tests****: Temp profile of disks w/ IR gun after extended disk thrashing**

***LCCWS'02***

**200 MB/s**

**60 MB/s**

**70 MB/s**

## Stage2 Hardware

_**Stage2 Hardware**_

***LCCWS'02***

**Worker**** nodes:**
**	****	****238 Dual Athlon MP2000+, 1U rackmount**
**1 THz of CPU power**
**File servers:**
**	****	****76 systems, 4U rackmount, dual red. Power supply**
**	****	****14 WD180GB in 2 RAID5 on 3ware 7500-8**
**	****	****2 WD40GB in RAID1 on 3ware 7000-2**
**	****	****1 GigE Syskonnect 9843**
**	****	****Dual P3 1.4GHz**
**150 TB disk cache**

## Stage1 Data Access

_**Stage1 Data Access**_

***LCCWS'02***

**	****	****Static files on disk:**
**	****	****	****NFS mounted to worker nodes**
**	****	****	****remote file access via rootd**
**	****	****Dynamic disk cache:**
**	****	****	****dCache in front of Enstore robot****	**

## Problems & Issues

_**Problems & Issues**_

***LCCWS'02***

**Resource overloading: **
**DB meltdown ********dedicated replica, startup delays**
**Rcp overload ********replaced with fcp**
**Rootd overload ********replaced with NFS,dCache**
**File server overload ********scatter data randomly **
**System issues:**
**Memory problems ********improved burn-in for next time**
**Bit error during rcp ********checksum after copy**
**dCache filesystem issues ********xfs & direct I/O**

## Lessons Learned

_**Lessons Learned**_

***LCCWS'02***

**Expertise in FNAL-CD is essential.**
**Well organized code management is crucial.**
**Independent commissioning of data handling and job processing ********3 ways of getting data to application.**
**	**

## CAF: User Perspective

_**CAF: User Perspective**_

**Job Related:**
**Submit jobs**
**Check progress of job**
**Kill a job**
**Remote file system access:**
**'ls' in job's 'relative path'**
**'ls' in a CAF node's absolute path**
**tail' of any file in job's 'relative path'**

***LCCWS'02***

## CAF Software

_**CAF Software**_

***LCCWS'02***

## CAF User Interface

_**CAF User Interface**_

** ****Compile, build, debug analysis job on 'desktop'**

** ****Fill in appropriate fields & submit job **

** ****Retrieve output using kerberized FTP tools**
**	****... or write output directly to  'desktop'!**

- section integer range

- user exe+tcl directory

- output destination

***LCCWS'02***

## Web Monitoring of User Queues

**Each user a different ****queue**
**Process type**** for job length**
**	****test****:****	****	****5 mins**
**	****short****:****	****	****2 hrs**
**	****medium****:****	****6 hrs**
**	****long****:****	****	****2 days**
**This example:**
**	****1 job ********11 sections **
**(+ 1 additional section automatic for job cleanup)**

_**Web Monitoring of User Queues**_

***LCCWS'02***

## Monitoring jobs in your queue

_**Monitoring jobs in your queue**_

***LCCWS'02***

## Monitoring sections of your job

_**Monitoring sections of your job**_

***LCCWS'02***

## CAF Utilization

***LCCWS'02***

**CAF in active use by CDF collaboration**
**300 CAF Users (queues) to date**
**Several dozen simultaneous users in **
**	****a typical 24 hr period**

_**CAF Utilization**_

## CAF System Monitoring

_**CAF System Monitoring**_

***LCCWS'02***

**Round-robin**
**Database**
**(RRD)**

## CPU Utilization

***LCCWS'02***

_**CPU Utilization**_

**CAF utilization steadily rising **
**  ****since opened to collaboration**
**Provided 10-fold increase in **
**  ****analysis resources for last**
**  ****summer physics conferences**
**Need for more CPU for winter**

***Week***

***3 months***

***Day***

## Data Processing

***LCCWS'02***

**File Server**

**Worker Node**

**Aggregate I/O**
**     ****4-8TB/day**

**Aggregate I/O **

_**Data Processing**_

**Average I/O**
**1-2MB/sec @ **
**~80%CPU util.**

**3 months**

**1 week**

## Work in Progress

_**Work in Progress**_

***LCCWS'02***

**Stage2 upgrade: 1THz CPU & 150TB disk**
**SAM ******** framework for global data handling/distribution**
**''DCAF'' ******** remote ''replicas'' of CAF**
**Central login pool @ FNAL**

## CAF Summary

_**CAF Summary**_

***LCCWS'02***

**Distributed Desk-to-Farm Computing Model**
**Production system under heavy use:**
**Single farm at FNAL**
**	****	****4-8TB/day processed by user applications**
**	****	****Average CPU utilization of 80%**
**Many users all over the world**
**	****	****300 total users**
**	****	****typical: 30 users per day share 130 CPU's**
**	****	****Regularly several 1000 jobs queued**
**Connected to tape via large cache**
**Currently updating to 1THz & 150TB**

## CDF Summary

_**CDF Summary**_

***LCCWS'02***

**Variety of computing systems deployed:**
**Single app. Farms: Online & Offline**
**Multiple app. Farm: user analysis farm**
**Expecting 1.7Petabyte tape archive by FY05**
**Expecting 700TB disk cache by FY05**
**Expecting 5THz of CPU by FY05**
**Oracle DB cluster with loadavg & failover for metadata.**