# Presentation: 792172

## D0 Computing Model and Plans 

**Amber Boehnlein**
**DO International Financial Committee**
**April 04, 2003**

## Computing Status

**DO has a highly successful computing structure in place**
**Nov Status****                                                                April status**
**Sequential Access by MetaData (SAM) catalogs and manages data access-**_**Operations are stabilizing, schema evolution, users inserting skimmed data sets.**_
**Robotic storage with reliable drives and media—**_**transition to 9940bs underway**_
**Domino provides high I/O capacity and user access to large amounts of data-**_**Reduced to 128 processors**_
**Commissioning the commodity backend **_**In production**_
**Basic software infrastructure in place—**_**In need of attention**_
**Upgrading processing farm:**_**Completed**_
**Fruitful collaboration with the Computing Division on joint projects—**_**Deployment of JIM-SAM-GRID in beta this month**_
**MC generation performed at collaborating institutions-**_**Now doing analysis and able to do reprocessing as well **_
**DORECO has basic functionality—**_**Improved tracking, use of calibration constants from data starting**_
**Basic Filtering at L3—**_**In need of attention**_
**Online output rate is at design.**
**                    **

## Farm Production

**DØ Reconstruction Farm—20M event/week capacity- operates at 78% efficiency—events processed within days of collection.  400M events processed in Run II. **
**DØ Monte Carlo Farms—0.6M event/week capacity-globally distributed resources. Running Full Geant and Reconstruction and trigger simulation **
**Successful effort to start data processing at National Partnership Advanced Computing Infrastructure resources at University of Michigan.****          ****                                                           **

**FNAL farm CPU utilization for the past year**

**March 02**

**Feb 03**

**Commissioning**
**New nodes**

**Full farm in production in Nov, 02**

**# Processors**

- Integrated Farm Production

## Data Storage & Access

| Station | 
Projects
 | Analyzed  | Transfer
(in/out) |
| --- | --- | --- | --- |
| DOmino | 3789 | 35 TB | 12/5 TB |
| CAB | 1626 | 35 TB | 8/0.5 TB |
| CLuEDO | 939 | 3.5 TB | 1.6/0.5 TB |
| Karlsrule | 259 | 1.4 TB | 0.5 TB* |

- Analysis Activity, March

| Library |  Stored | Eff. |
| --- | --- | --- |
| STK | 206 TB | 0.97 |
| LTO | 77 TB | 0.88 |

- Data to tape, April 2, 3 GB Lost

**4TB in**

**3TB out**

**Plot shows daily transfers in/out**
**Of SAM stations**
**Colors represent stations-**
**Blue is to/from tape**
**Red is to/from farm**

**March, plotted daily**

## MC Production

- Integrated Monte Carlo Production for the past two years

- 60M
- events

- 3/21/2001

- 3/21/2003

## Central Analysis Systems

- Domino

- FEB

- March

- 50% cpu

- CLuED0 batch usage

- 14K Jobs in March—over half non-SAM

- No impact from reduction in processors

- Delivery Effectiveness (includes Reco and delivery losses)
- W/Z skim sample/Raw data available =98.2%

## Computing Architecture

- Central Data Storage

- dØmino

- ...

- Central Analysis System

- Central
- Farm

- Remote
- Farm

- Remote
- Analysis

- Linux Desktops
- (ClueDØ)

- DØ

## Data Flow	

- Data Flow

- Data Handling System

- Robotic Storage

- Raw Data

- User Data

## Analysis Model

- Analysis CPUs to be provided by:

-  Central Analysis System (CAS) at FCC:
-   A PC/Linux dØmino back-end supplied and administrated by the computing division
-  Remote Analysis Centers (RAC):
-   Institutions with CPU, disk  and personnel resources to serve collaborators
***Emphasis on Remote Analysis with formal agreements***
***FNAL pursuing improved offsite connectivity ***
***Analysis needs must be better understood and modeled***
***http://www-hep.uta.edu/~d0race/d0rac-wg/d0rac-final.pdf***

## The Virtual Center

**For cost basis, determine the cost of the full computing system to met all needs as if the center were located at FNAL, plus equipment required to support offsite**
**Presented today as draft**
**Disk and servers and cpu for analysis**
**Production activities such as MC generation, processing and reprocessing.**
**In preparation-using older estimates**
**Infrastructure such as gateway machines and code servers **
**database machines and servers**
**Mass storage**
**Cache machines and drives to support extensive data export**
**Wide Area Networking**

## Analysis Patterns

**Current access and analysis patterns**
**Physics group coordinated efforts**
**Derived data sets by skimming through data sets (DST or TMB)**
**Picked event samples of raw data for re-reco studies**
**Specialized reprocessing of small data sets**
**Physics topic analysis includes generation of test samples, trigger simulation, background studies, efficiency studies**
**User level analysis primarily takes place on skimmed data samples on high level tier**
***Beginning to acquire more information in order to size the analysis systems and tape plant***
**  **

## Estimated Analysis Cost

- Includes more realistic estimate of analysis CPU needs based on Moriond
- Usage
- Assumes full DST storage and 10 copies of TMB sized objects

##         Possible ReReco Scenarios 

**Resources needed will vary as a function of **
**Amount of data to process**
**How quickly it needs to done**
**Speed of Reco**
**Constrained by release cycle, analysis scales**

## MC and Primary Production

**Possible that 50% MC to data rate is skimpy-assume 75%, with 25% full chain**
**Effectively the same cost as processing for Reco 50GHz-sec event.**
**Speed and memory consumption big unknown.**
**Legacy systems considered for Reco**

## Estimated Reprocessing Costs

- A test case for checking the validity of the calculations is reprocessing of one years’ worth of data in 6 months
- Cross checks
-  current farm (1 THz) is sized to keep up with data collection with 30% over capacity for reco average 10 GHz-sec/event
- Comparable in rate to online data collection

## Cost Estimate-April 2003

- Infrastructure includes database machines, networking,
- web servers, code servers and build machines, and dCache
- machines
- Re-reconstruction is current cost driver
- —selective reprocessing and speeding up Reco

- DRAFT-DRAFT-DRAFT

## Towards a Global Model

**Effectively integrating hardware resources will require some structural changes.**
**Introduction of a strategic planning board charged with focusing on global issues.  Putting in place now**
**Support model in which developer and operations effort is supplied in conjunction with the hardware to make the global model a success—Support also covered by MOU.**
**Increased focus on aligned activities within GRID projects.**
**Track available resources and determine the deployment as best meets strategic and tactical needs**

## Survey of Resources

- Beginning the process of surveying and tracking resources—this particular example targeted a fall reprocessing—Very rough- first pass to show direction—NOT NECESSARILY CORRECT
- Using for networking, drive and buffering estimates

- DRAFT

## FNAL Equipment Budget

**The guidance to the experiments is $1.5M/year**
**	****Pinching pennies—evaluating expenditures**
**Use the FNAL equipment budget to provide basic level of functionality**
**Database and other infrastructure**
**Reconstruction farm**
**Robotic storage and tape drives**
**Disk cache**
**Basic analysis computing**
**Support for data access to enable offsite computing**

## Institution Contributions

**All Monte Carlo production takes place at regional centers**
**Secondary reprocessing at Michigan National Partnership for Advanced Computing Initiative Center—targeting reprocessing for 15% of data set, goal for March**
**	****Additional sites identified**
**Analysis at home institutions**
**Contributions at FNAL to project disk and to CLuED0**
**Offsite Analysis—See Next talk**

## RAC Cost Estimate

**Assume 10% dsts**
**Assume 0.1% RAW and 0.1% RAW/DST**
**Assume 2 copies data thumbnails**
**Assume 20% each PMCS and MC thumbnails**
**Assume disk purchased as needed**
**Assume 3 Medium jobs and 50 short jobs—probably an over estimate compared to total needs**

## RAC Cost Estimate

## Conclusions

**The D0 computing model is successful**
** ****SAM, an integrated data handling system, enables flexibility in the allocation of resources and effective use disk cache and robotic storage and path into the GRID era**
**Use FNAL Computing budget to provide base for infrastructure, robotic storage, reconstruction and analysis**
**DO is shifting our thinking towards a more global model—and making structural changes accordingly.   **