# Presentation: 664931

## Design & Management of the JLAB Farms

- Ian Bird, Jefferson Lab
- May 24, 2001
- FNAL LCCWS

## Overview

- JLAB clusters
- Aims
- Description
- Environment
- Batch software
- Management
- Configuration
- Maintenance
- Monitoring
- Performance monitoring
- Comments

## Clusters at JLAB - 1

- Farm
- Support experiments – reconstruction, analysis
- 250 ( 320) Intel Linux CPU ( + 8 Sun Solaris)
- 6400  8000 SPECint95
- Goals:
- Provide 2 passes of 1st level reconstruction at average incoming data rate (10 MB/s)
- (More recently) provide analysis, simulation, and general batch facility
- Systems
- First phase (1997) was 5 dual Ultra2 + 5 dual IBM 43p
- 10 dual Linux (PII 300) acquired in 1998
- Currently 165 dual PII/III (300, 400, 450, 500, 750, 1GHz)
- ASUS motherboards, 256 MB, ~40 GB SCSI, IDE, 100 Mbit
- First 75 systems towers, 50 2u rackmount, 40 1u (1⁄2u?)
- Interactive front-ends
- Sun E450’s, 4-proc Intel Xeon, (2 each), 2GB RAM, Gb Ethernet

## Intel Linux Farm

_**Intel Linux Farm**_

## Clusters at JLAB - 2

- Lattice QCD cluster(s)
- Existing clusters – in collaboration with MIT, at JLAB:
- Compaq Alpha
- 16 XP1000 (500 MHz 21264), 256 or 512 MB, 100 Mbit
- 12 Dual UP2000 (667 MHz 21264), 256 MB, 100 Mbit
- All have Myrinet interconnect
- Front-end (login) machine has GB Ethernet, 400 GB fileserver for data staging and transfers MIT  JLAB
- Anticipated (funded)
- 128 cpu (June 2001), Alpha or P4(?) in 1u
- 128 cpu (Dec/Jan ?) – identical to 1st 128
- Myrinet

## LQCD Clusters

## Environment

- JLAB has central computing environment (CUE)
- NetApp fileservers – NFS & CIFS
- Home directories, group (software) areas, etc.
- Centrally provided software apps
- Available in
- General computing environment
- Farms and clusters
- Managed desktops
- Compatibility between all environments – home and group areas available in farm, library compatibility, etc.
- Locally written software provides access to farm (and mass storage) from any JLAB system
- Campus network backbone is Gigabit Ethernet, with 100 Mbit to physicist desktops, OC-3 to ESnet

## Jefferson Lab

- Mass Storage and Farm Systems
- 2001

*Work File Servers*
*10 TB – RAID 5*

*Farm Cache File Servers*
*4 x 400GB*

*DST/Cache File Servers*
*15 TB – RAID 0*

*Batch and Interactive Farm*

*DB Server*

*Tape Servers*

- From CLAS DAQ

- From Hall A,C DAQ

- 100 Mbit/s

- 1000 Mbit/s

- FCAL

- SCSI

## Slide 9

## Batch Software

- Farm
- Use LSF (v 4.0.1)
- Pricing now acceptable
- Manage resource allocation with
- Job queues
- Production (reconstruction, etc)
- Low-priority (for simulations), High-priority (short jobs)
- Idle (pre-emptable)
- User + group allocations (shares)
- Make full use of hierarchical shares  - allows single undivided cluster to be used efficiently by many groups
- E.g.

## Batch software - 2

- Users do not use LSF directly, use Java client (jsub), that:
- Is available from any machine (does not need LSF)
- Provides missing functionality, e.g.
- Submit 1000 jobs in 1 command
- Fetches files from tape, pre-stages before job queued for execution (don’t block farm with jobs waiting for data),
- Ensures efficient retrieval of files from tape  - e.g. sort 1000 files by tape and by file no. on tape.
- Web interface (via servlet) to monitor job status and progress (as well as host, queue, etc.)

## View job status

## View host status

## Batch software - 3

- LQCD clusters use PBS
- JLAB written scheduler
- 7 stages – mimic LSF hierarchical behaviour
- Users access PBS commands directly
- Web interface (portal) – authorization based on certificates
- Used to submit jobs between JLAB & MIT clusters

## Slide 15

## Batch software - 4

- Future
- Combine jsub & LQCD portal features to wrap both LSF and PBS
- XML-based description language
- Provide web-interface toolkit to experiments to enable them to generate jobs based on expt. run data
- In context of PPDG

## Cluster management

- Configuration
- Initial configuration
- Kickstart, 2 post-install scripts for configuration, sw install (LSF etc), driven by a floppy
- Looking at PXE – DHCP (available on newer motherboards)
- Avoids need for floppy – just power on
- System working (last week)
- Software: PXE standard bootprom (_[www.nilo.org/docs/pxe.html](http://www.nilo.org/docs/pxe.html)_) – talks to DHCP,
- bpbatch – pre-boot shell (_[www.bpbatch.org](http://www.bpbatch.org/)_) - downloads vmlinux, kickstart etc
- Alphas configured “by hand + kickstart”
- Updates etc.
- Autorpm (especially for patches)
- New kernels – by hand with scripts
- OS upgrades
- Rolling upgrades – use queues to manage transition
- Missing piece:
- Remote, network-accessible console screen access
- Have used serial console, KVM switches, monitor on a cart ...
- Linux Networks Alphas have remote power management – don’t use!

## System monitoring

- Farm systems
- LM78 to monitor temp + fans via /proc
- This was our largest failure mode for Pentiums
- Mon (_[www.kernel.org/software/mon](http://www.kernel.org/software/mon)_)
- Used extensively for all our systems – page “on-call”
- For batch farm checks mostly – fan, temp, ping
- Mprime (prime number search) has checks on memory and arithmetic integrity
- Used in initial system burn-in

## Monitoring

## Performance monitoring

- Use variety of mechanisms
- Publish weekly tables and graphs based on LSF statistics
- Graphs from mrtg/rrd
- Network performance, #jobs, utilization, etc

## Slide 21

## Slide 22

## Comments & Issues

- Space – very limited
- Installing a new STK silo, moved all sys admins out
- Now have no admins in same building as machine room
- Plans to build a new Computer Center ...
- Have always been lights-out

## Future

- Accelerator and experiment upgrades
- Expect first data in 2006, full rate 2007
- 100 MB/s data acquisition
- 1 – 3 PB/year (1 PB raw, > 1 PB simulated)
- Compute clusters:
- Level 3 triggers
- Reconstruction
- Simulation
- Analysis – PWA can be parallelized, but needs access to very large reconstructed and simulated datasets
- Expansion of LQCD clusters
- 10 Tflops by 2005