# Presentation: 725231

## BaBar Clusters

- Charles Young, SLAC

## Supported Platforms

- Initially – AIX, HP and OSF.
- Added Solaris & Linux. Dropped HP & AIX.
- Now...
- Linux – rapid growth.
- OSF – a little bit.
- Solaris – only one for online.
- Future?
- Good to have more than one vendor.
- Not too many.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## BaBar Clusters

- Online.
- Offline.
- Multiple reconstruction farms.
- Prompt and Non-Prompt (or reprocessing).
- Physics skims.
- Monte Carlo production.
- At SLAC and other sites, e.g. 1⁄2 in U.K.
- Analysis.
- At SLAC and other sites, e.g. CC-IN2P3.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Reconstruction Cluster

- Prompt Reconstruction in almost real time.
- Feedback to data taking.
- Calibration constants.
- Output for downstream physics analyses.
- Good uptime and reliability.
- Controlled environment, organized activity.
- Equipment manageability and density issues.
- Initially (and still) all Solaris.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Reconstruction Cluster – Configuration 

- Up to 200 farm CPUs.
- Solaris only, due to online related code.
- Sun T1 – 1 RU, 1 CPU, 440 MHz, 256 MB.
- Several (Objectivity) file servers.
- All Solaris.
- Typically Sun 420R – 4 CPUs.
- Typically 2 x T3 500-GB disk arrays.
- Gigabit Ethernet.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Reconstruction Cluster – Experience 

- Hardware generally reliable.
- Customized job control instead of batch.
- More focused on specific needs.
- Increased maintenance.
- Scaling issues.
- Tightly coupled system.
- Serialization or choke points.
- Reliability concerns when scaling up.
- (Weak) connection to acceleration schedule.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Reconstruction Cluster – Plans 

- Aggressively pursuing Linux farm nodes.
- VA Linux 1220 – 1 RU, 2 CPUs, 866 MHz, 1 GB.
- Higher density, faster CPUs and lower cost.
- Reduced networking per unit CPU power.
- Fewer CPUs. One link for 2 CPUs.
- Increase file server capacity.
- Linux compatibility issues.
- Some online related code.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Monte Carlo Cluster 

- MC production.
- Physics simulation.
- Mix in real beam background.
- Reconstruct with standard code.
- Capacity related to volume of beam data.
- Controlled environment, organized activity.
- Very similar to Reconstruction.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Monte Carlo Cluster – Configuration 

- ~100 farm CPUs. Mix of Solaris and Linux.
- Sun T1 – 1 RU, 1 CPU, 440 MHz, 256 MB.
- VA Linux 1220 – 1 RU, 2 CPUs, 866 MHz, 1 GB.
- Several (Objectivity) file servers.
- All Solaris.
- Typically Sun E4500 – 4 CPUs.
- Typically 1 x A3500 500-GB disk array.
- Gigabit Ethernet.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Monte Carlo Cluster – Experience 

- Hardware generally reliable.
- Batch system (LSF) generally adequate.
- Scaling issues.
- Little schedule interaction with accelerator.
- Want to be up 24 x 365 .
- No major platform issues.
- Code runs on all BaBar platforms.
- External code, e.g. Objectivity, Geant4.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Monte Carlo Cluster – Plans 

- Capacity to scale with *integrated* luminosity.
- Many production sites.
- Increase compute power.
- More and faster (Linux) nodes.
- Increase file server capacity.
- More flexibility in solving scaling issues.
- Very few choke points.
- Embarrassingly parallel.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Analysis Cluster 

- Data stored (primarily) in Objectivity format.
- Disk cache with HPSS back end.
- Different levels of detail.
- Tag, Micro, Mini, Reco, Raw, etc.
- Varied access pattern.
- High rate and many parallel jobs to Micro.
- Lower rate and fewer jobs to Raw.
- Uncontrolled environment and activities.
- Activities coupled to conferences.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Analysis Cluster – Configuration 

- ~200 farm CPUs.
- Sun T1 – 1 RU, 1 CPU, 440 MHz, 256 MB.
- VA Linux 1220 – 1 RU, 2 CPUs, 866 MHz, 1 GB.
- Several (Objectivity) file servers.
- All Solaris.
- Typically Sun E4500 – 4 CPUs.
- Typically 2 x T3 500-GB disk arrays.
- Gigabit Ethernet.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Analysis Cluster – Experience 

- Hardware generally reliable.
- No major platform issues.
- Code runs on all BaBar platforms.
- External code, e.g. Objectivity, Geant4.
- Batch system (LSF) generally adequate.
- Minor schedule interaction with accelerator.
- Major schedule interaction with conferences.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Analysis Cluster – Plans 

- Increase computing power.
- More and faster (Linux) nodes.
- Increase file server capacity.
- Scales with *integrated* luminosity.
- Flexibility in solving scaling issues.
- Reduce # choke points.
- Embarrassingly parallel.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Offline Clusters – Equipment 

- Similar servers.
- ~40 4-CPU Sun machines & ~2 x 500 GB disks.
- Gigabit Ethernet.
- Two kinds of farm nodes.
- ~900 Sun T1 – 1 RU, 1 CPU, 440 MHz, 256 MB.
- ~60 VA Linux 1220 – 1 RU, 2 CPUs, 866 MHz, 1 GB.
- Straight forward to reassign servers & clients.
- Especially batch nodes.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Batch System (LSF) Issues

- Licensing costs.
- Significant compared with H/W.
- Scaling concerns.
- Many jobs.
- Complex priority algorithm.
- Failed to dispatch fast enough.
- Idle nodes .

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Offline Clusters – Schedule 

- Few outage opportunities.
- Prompt Reconstruction tied to accelerator.
- Peak analysis load before conferences.
- Vendor schedule.
- H/W and S/W availability.
- BaBar code readiness.
- Extensive testing required.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## Offline Cluster – Plans 

- Cost concerns.
- Choice among multiple platforms/vendors.
- Schedule concerns.
- Must support multiple OS levels.
- Essential to have capacity headroom.
- Scalability concerns.
- Must reduce # choke points.
- Should be embarrassingly parallel.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## BaBar Computing Centers

- Divided into Tier A, B and C.
- Union of all Tier A’s => all data.
- Two Tier-A centers currently.
- SLAC.
- CC-IN2P3 fully operational by end of 01.
- Potentially more in the future.
- (Almost) all usual activities at all Tier-A sites.
- Simulation, reprocessing, analysis.
- Prompt reconstruction only at SLAC.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## The G Word

- Nirvana.
*Automatic* discovery of resources.
*Automatic* job configuration and execution.
- Results at desktop *instantly*.
- Foreseeable future.
*Assisted* discovery of resources.
*Assisted* job configuration and submission.
- Results at desktop *before retirement*.
- Rapid evolution and changes.

- Charles Young, SLAC

- LCCWS, May 22, 2001

## GRID and Future of BaBar Clusters

- Offline clusters more likely GRID adaptable.
- Distributed resources.
- Distributed management.
- Distributed users.
- Cautious awareness and participation.
- Be involved but not be seduced.

- Charles Young, SLAC

- LCCWS, May 22, 2001