# Presentation: 332094

## CLRC-RAL Site Report

- John Gordon
- CLRC eScience Centre

- Tim Adye

- 12th September 2002

- General PP Facilities
- New UK Supercomputer
- BaBar TierA Centre
- Networking

## UK GridPP Tier1/A Centre at CLRC

**Computing Farm**
- This years new hardware consists of 4 racks holding 156 dual cpu PCs, a total of 312 1.4GHz Pentium III Tualatin cpus. Each box has 1GB of memory, a 40GB internal disk and 100Mb ethernet.

**Inside The Tape Robot**
- The tape robot  was upgraded last year and now uses 60GB STK 9940 tapes. It currently holds 45TB but could hold 330TB when full.

_**UK GridPP Tier1/A Centre at CLRC**_

**40TByte disk-based Disk Farm**
- The new mass storage unit can store 40Tb or raw data after a RAID 5 overhead. The PCs are clustered on network switches with up to 8x1000Mbit ethernet out of each rack.

**Prototype Tier 1 centre for CERN LHC and FNAL experiments**
**Tier A centre for SLAC BaBar experiment**
**Testbed for EU DataGrid project**

**Notes:**
GridPP
The Grid for UK Particle Physics
Slogan: “From Web to Grid... exploiting the next IT revolution”

First idea for a poster
Background of data stream? Standard background for all GridPP posters.
Standard logo and slogan.(or partsof slogon?)
Poster emaulates web page?
This one should illustrate central role of UK in worldwide collaboration.
Emphasise middleware in handout/description?
Other posters are:
BaBar
LHCb
Query Optimisation
Tier 1 - RAL
Tier 2 – ScotGRID 
...others?

- Free and out
- of robot

- Free

- 36TB HEP Data

## HPCx

**UK SuperComputer for next 6 years**
**Collaboration of CLRC Daresbury Laboratory, Edinburgh EPCC. IBM**
**Sited at CLRC-DL**
**http://www.hpcx.ac.uk**
**Double in performance every 2 years ie 2 upgrades**
**Capability computing**
**Target to get 50% of the jobs using 50% of the machine**
**Hardware**
**40x32 IBM pSeries 690 Regatta-H nodes (Power4 CPUs) **
**1280 1.3GHz cpus – estimated peak performance 6.6TeraFLOPS**
**IBM Colony switch connects blocks of 8 cpus (ie looks like 160x8, not 40x32)**
**1280 GB of Memory**
**2x32 already in place as a migration aid.**
**Service testing mid November, service December.**

## HPCx

**Software**
**Capability computing on around 1000 high performance CPUs**
**Terascale Applications team**
**Parallelising applications for 1000’s of CPUs**
**Different architecture compared to T3E etc**
**HPCx is a cluster of 32 Processor machines compared to MPP style of T3E**
**Some MPI operations now very slow (eg barriers, all-to-all communications)**

## RAL Tier A

- RAL is TierA Centre for BaBar
- Like CC-IN2P3 but concentrating on different data.
- Shared resource with LHC and other experiments
- Use

## Hardware

- 104 “noma”-like machines allocated to BaBar
- 156+old farm shared with other experiments
- 6 BaBar Suns (4-6 CPUs each)
- 20 TB disk for BaBar
- Also using ~10 TB of pool disk for data transfers
- All disk servers on Gigabit ethernet
- Pretty good server performance
- ... as well as existing RAL facilities
- 622 Mbits/s network to SLAC and elsewhere
- AFS cell
- 100TB Tape robot
- Many years’ experience running BaBar software

## Problems

- Disk problems tracked down to a bad batch of drives
- All drives are now being replaced by the manufacturer
- our disks should be done in ~1 month
- By using spare servers, replacement shouldn’t interrupt service
- Initially suffered from lack of support staff and out-of-hours support (for US hours)
- Two new system managers now in post
- Two more being recruited (one just for BaBar)
- Additional staff have been able to help with problems at weekends
- Discussing more formal arrangements

## Slide 12

## RAL Batch CPU Use

**RAL Batch CPU Use**

## RAL Batch Users(running at least one non-trivial job each week)

- A total of 113 new BaBar users registered since December

**Notes:**
Non-UK Users CPU Hours   Jobs        Non-UK Users CPU Hours   Jobs
            (P450 equivalent)                    (P450 equivalent)
sunilj         79361.5   5066        roethel           62.2     81
thiessen       34941.6   1752        jschubi           22.3     52
eich           30501.4    731        jack               1.0    194
yumiceva       25339.9   3209        sdu                0.0     19
sekula         24001.7  49517        dndong             0.0      2
croat          23955.8   4451
dickopp        21733.4   1315        UK Users     CPU Hours   Jobs
dattam         14666.4  10459                    (P450 equivalent)
edward         12359.3   2854        kiwi           99964.8  24817
marcp           9326.6    604        fwilson        52153.9   7271
otto            6877.9   5124        rjb            36454.3   5608
joe             5743.2   1457        babarmc        27950.4   8265
behera          5465.5   1661        djpayne        21819.1   7660
mwilson         4610.4    425        egede          17779.0   2854
hearty          3994.3   1236        gopal          16157.0  22289
swaban          3834.0   2115        sloane         14321.8   7417
petzold         1879.7   1529        xella           7165.7   7666
ppenn           1561.0     72        bhimji          6472.3   1692
asarti          1488.2     84        djk             5177.0   5334
tbrandt         1128.5    122        jback           4681.7    439
aagaard          963.8    105        gptaylor        3361.0    632
sandrel          669.8   1777        jamieg          2092.1    188
dubitzky         632.9    189        morton          1585.9   1128
plaszczy         626.6    143        markhod          818.9   8307
best             520.8    305        olaiya           122.6    324
achim            453.6     80        adye               1.2      2
wilden           233.8    274        salvator           0.2     10
aldi             176.9    244        dhs1               0.0      1
maly             125.8     79

## Data at RAL

- All data in Kanga format is at RAL
- 19 TB currently on disk
- Series-8 + series-10 + reskimmed series-10
- AllEvents + streams
- data + signal+generic MC
- New data copied from SLAC within 1-2 days
- RAL is now the primary Kanga analysis site
- New data is archived to tape at SLAC and then deleted from disk

## Changes since July

- Two new RedHat 6 front-end machines
- Dedicated to BaBar use
- Login to **babar.gridpp.rl.ac.uk**
- Trial RedHat 7.2 service
- One front-end and (currently) 5 batch workers
- Once we are happy with the configuration, many/all of the rest of the batch workers will be rapidly upgraded
- ssh AFS token passing installed on front-ends
- So, your local (eg. SLAC) token is available when you log in
- Trial Grid Gatekeeper available (EDG 1.2)
- Allows job submission from the Grid
- Improved new user registration procedures

## Plans

- Upgrade full farm to RedHat 7.2
- Leave RedHat 6 front-end for use with older releases
- Upgrade Suns to Solaris 8 and integrate into PBS queues
- Install data dedicated import-export machines
- Fast (Gigabit) network connection
- Special firewall rules to allow scp, bbftp, bbcp, etc.
- AFS authentication improvements
- PBS token passing and renewal
- integrated login (AFS token on login, like SLAC)

## Plans

- Objectivity support
- Works now for private federations, but no data import
- Support Grid “generic accounts”, so special RAL user registration is no longer necessary
- Procure next batch of hardware
- Delivery probably early 2003

## Network

**Network**

- Tier1 internal networking will be a hybrid of
- 100Mb to nodes of cpu farms with 1Gb up from switches
- 1Gb to disk servers
- 1Gb to tape servers

- UK academic network SuperJANET4
- 2.5Gbit backbone upgrading to 10Gb in 2002
- RAL 622Mb into SJ4 upgraded to 2.5Gb June 02
- SJ4 has 2.5Gb interconnect to Geant
- 2.5Gb links to ESnet and Abilene just for research users
- UK involved in networking development
- internal with Cisco on QoS
- external with DataTAG
- Lamda CERN -> Starlight
- Private connections