# Presentation: 812098

## D0 Production Status and Plans

- Run II Computing Review 2004
- Michael Diesburg

- Michael Diesburg RunII Review 2004

## Reconstruction Farm Status

- Current Configuration:
- 8 processor SGI Origin, 4GB memory, 2GB disk
- Used as NFS server, output staging area
- 19 dual processor PIII nodes
- Used as input stagers for production nodes
- 368 dual processor worker nodes (PIII, Athlon, Xeon)
- 1225 GHz total capacity (PIII equivalent)
- Software components:
- FBSNG – Fermilab farm batch system
- Dfarm – distributed disk cache (used for code distribution)
- FCP – farm copy (controls network copy load)
- SAM – Used for all data I/O
- Local control scripts

- Michael Diesburg RunII Review 2004

## Reconstruction Farm Status

- Current Operating Performance:
- Have demonstrated operating efficiency > 80%
- i.e. can utilize > 80% of rated capacity for periods of months
- Current demand on resources (since Jan ’04) has been ~60% of capacity.
- This is nearly ideal match of demand and available resources
- Reconstructed data available in SAM in ~48 hours
- Special requests can be satisfied in < 24 hours
- Capacity available for new tasks (TMB fixing)
- Can catch up from down periods in a few days
- Can reconstruct ~3.5M events/day if average execution time is ~25GHz-secs/event.

- Michael Diesburg RunII Review 2004

## Reconstruction Farm Status

- Operating Performance Projections:
- Production capacity is strongly dependent on luminosity of the data being processed
- Average reconstruction time is ~25 secs/event at initial luminosity of ~35E30,  ~65 secs/event at 70E30, (~110 secs/event at 90 E30)
- To produce 3.5M events/day at L > 35E30 we will be dependent on duty cycle mismatch between farm and accelerator
- With current configuration we can keep up with data collection to ~75E30 assuming accelerator duty cycle of ~1/3.
- But, will lose advantages of excess capacity noted above.
- Operational efficiency will also suffer

- Michael Diesburg RunII Review 2004

## Reconstruction Farm Status

- Farm Expansion Plans:
- Will add 80   3GHz nodes to farm this year
- Will bring total capacity to ~1500 GHz (PIII)
- With this addition we should be able to keep up with data collection to ~85E30
- Need HDCF ready by ~October in order to have these nodes available at end of current shutdown
- Will also move nodes currently in NML to HDCF
- Will upgrade input staging node connections to GB
- Will move some functionality off IRIX to Linux
- FBS, Dfarm server
- SAM station

- Michael Diesburg RunII Review 2004

## Reconstruction Farm Status

- Farm Expansion Plans:
- Operational software needs updating
- Current software is very robust and allows high efficiency operation
- Not very flexible (tailored to specific executables)
- Not portable (very specific to FNAL installation)
- Will shift to non-process specific, non-site specific system
- Plan on moving to samGRID as soon as possible
- Need to maintain efficiency of operation as cut over is made
- Will allow us to more easily shift operation between reconstruction, Monte Carlo, post-processing
- Also  will make us interoperable with remote sites.

- Michael Diesburg RunII Review 2004

## Monte Carlo Production Status

- MC Production:
- All MC production has been done off-site
-  Utilized 13 sites in 8 countries over the last year
- Brazil	Czech Republic	Denmark	France
- Germany	India		UK		US
- Expect additional sites to come online in next year in
- Canada	China	Korea	US
- Difficult to characterize the potential capacity since resources are not all dedicated
- Generated 37M events last year (Max 1.5M/week)
- Usually very little delay between request submission and start
- Generation time ~1-2 weeks (depends on site)
- Demand varies significantly during year
- Currently have idle capacity

- Michael Diesburg RunII Review 2004

## Reprocessing Status

- Reprocessing, Round I
- Planning for reprocessing run began June 2003
- Required ~5 1⁄2 months preparation until startup
- Most time spent on making p14 version of reco sufficiently robust to proceed
- Significant changes in reco capabilities lead to rethinking entire processing chain and priorities
- Began reprocessing ~15 Nov 2003, finished ~5 Jan 2004
- ~100M events processed at remote sites
- ~25TB data transferred
- Final merging, storage of TMBs done at FNAL to reduce load on remote sites, processing done from DSTs (no Db access)
- ~ 1⁄2 FTE required at each remote site for duration of processing phase

- Michael Diesburg RunII Review 2004

## Reprocessing Status

- Reprocessing, Round II
- Scheduled to begin ~Dec 2004
- Scale of task ~10x first round of reprocessing
- 1000M events  (100M first round)
- 250 TB data (25 TB first round)
- Remote sites will do merging and final storage to SAM
- Processing from raw data rather than DST (DB access required, data unfiltered)
- Expect to require 6-8 months for completion
- Expect to use samGRID at all sites for Round II
- Biggest problem likely to be manpower
- ~1 FTE/site for 6-8 months is significant burden

- Michael Diesburg RunII Review 2004

## Production Status

- Conclusions:
- Local production going well, but will have difficulties next year as luminosity increases
- Monte Carlo production in good shape but subject to non-uniform demand
- First round of remote reprocessing was major success, second round to start soon.   Manpower demands will be significant issue

- Michael Diesburg RunII Review 2004