# Presentation: 718296

## The Status of Clusters at LLNL

- Bringing Tera-Scale Computing to a Wide Audience at LLNL and the Tri-Laboratory Community
- Mark Seager
- Fourth Workshop on Distributed Supercomputers
- March 9, 2000

## Overview

**Architecture/Status of Blue-Pacific and White**
**SST Hardware Architecture**
**SST Software Architecture (Troutbeck)**
**MuSST Software Architecture (Mohonk)**
**Code development environment**
**Technology integrations strategy**
**Architecture of Compaq clusters**
**Compass/Forest**
**TeraCluster98**
**TeraCluster2000**
**Linux cluster**

## SST Hyper-Cluster Architecture

**SST Hyper-Cluster Architecture**

**Sector S**

**Sector Y**

**Sector K**

**24**

**24**

**24**

**Each SP sector comprised of**
** ****488 Silver nodes**
** ****24xTB3 links to 6xHPGN**

**System Parameters**
** ****3.89 TFLOP/s Peak**
** ****2.6 TB Memory**
** ****62.5 TB Global disk**

**2.5 GB/node Memory**
**24.5 TB Global Disk**
**8.3 TB Local Disk**

**1.5 GB/node Memory**
**20.5 TB Global Disk**
**4.4 TB Local Disk**

**1.5 GB/node Memory**
**20.5 TB Global Disk**
**4.4 TB Local Disk**

**FDDI**

**6**

**6**

**12**

**High-speed external connections**
** ****6xTB3 @ 150 MB/s bi-dir**
** ****12xHIPPI-800 @ 100 MB/s bi-dir**
** ****6xFDDI @ 12.5 MB/s bi-dir**

## I/O Hardware Architecture of SST

- System Data and Control Networks

- 488 Node IBM SP Sector

**56 GPFS**
**Servers**

**432 Thin Silver Nodes**

**Each SST Sector**
- Has local and two global I/O file systems
- 2.2 GB/s delivered global I/O performance
- 3.66 GB/s delivered local I/O performance
- Separate SP first level switches
- Independent command and control

**Full system mode**
- Application launch over full 1,464 Silver nodes
- 1,048 MPI/us tasks, 2,048 MPI/IP tasks
- High speed, low latency communication between all nodes
- Single STDIO interface

**24 SP Links to **
**Second Level**
**Switch**

## LoadLeveler Pool Layout Geared Toward Servicing Large Parallel Jobs

- HPGN

***K***

***S***

***Y***

**Each sector independently scheduled**
**Cross sector runs accomplished by dedicating nodes to the user/job**
**Normal production limited to size constraints of single PBATCH partition.  Can only support THREE simultaneous 256 jobs!**
**S = 425 = ****256**** + ****128**** + 41**
**K = 425 = ****256**** + ****128**** + 41**
**Y = 393 = ****256**** + ****128**** + 9**

## I/O Hardware Architecture of MuSST (PERF)

**I/O Hardware Architecture of MuSST (PERF)**

- 512 NH-2 Node IBM SP

**16 GPFS**
**Servers**

**484 NH-2 PBATCH Nodes**

**MuSST (PERF) System**
- 4 Login/Network nodes w/16 GB SDRAM
- 8 PDEBUG nodes w/16 GB SDRAM
- 258 w/16GB, 226 w/8GB PBATCH nodes
- 12.8 GB/s delivered global I/O performance
- 5.12 GB/s delivered local I/O performance
- 24 Gb Ethernet External Network

**Programming/Usage Model**
- Application launch over ~492 NH-2 nodes
- 16-way MuSPPA, Shared Memory, 32b MPI
- 4,096 MPI/US tasks, 8,192 MPI/IP tasks
- Likely usage is 4 MPI tasks/node with 4 threads/MPI task
- Single STDIO interface

**8 NH-2 PDEBUG Nodes**

- System Data and Control Networks

**Notes:**
This new baseline reflects the new baseline I/O configuration as of 2/29/2000
Each Login/Network node has six Gb Enets.  Two Gb Enets are in the PCI slots on the NH-2 node and do not support transfers with jumbo frames (login traffic and slower FTP).  Four other Gb Enets are installed in two RIO drawers and will be dedicated to FTP traffic over sub-nets at LLNL that support jumbo frames.  This is specifically targeted at the SGI visualization services and also the archive.

## TeraClusterSystem Architecture

- QSW, Gb EtherNet, 100BaseT EtherNet

- 128x4 Node 0.683 TF Sierra System

**~12 CFS**
**Servers**

**~114 Regatta Compute Nodes**

**System I/O Requirements**
- <10 s, 200 MB/s MPI latency and Bandwidth over QSW
- Support 64 MB/s transfers to Archive over Gb-Enet and QSW links
- 19 MB/s POSIX serial I/O to any file system
- except to local OS and swap
- Over 7.0 TB of global disk in RAID5 with hot spares
- 0.002 B/s/F/s = ~1.2 GB/s delivered parallel I/O performance
- MPI I/O based performance with a large sweet spot
- 64 < MPI tasks < 242
- Separate QSW, Gb and 100BaseT EtherNet networks
- GFE Gb EtherNet switches
- Consolidated consoles

**~2 Login nodes**
** ****with Gb-Enet**

**CFS**

**Final Config**
**August2000**

## Phase1 TeraClusterSystem HW/SW Architecture

**Phase1 TeraCluster****System HW/SW Architecture**

- 1x128 QSW & 4x32 CFS Node Sierra System

**~30 Regatta Compute Nodes in each CFS partition**

**System Architecture**
- ES40 compute nodes have 4 EV67 @ 667 MHz and 2 GB memory
- ES40 login nodes have 4 EV67 @ 667MHz and 8 GB memory
- Support 64 MB/s transfers to Archive over Gb-ENet
- 19 MB/s POSIX serial I/O to local file system
- CFS **ONLY** used for system functions.  NFS home directories
- Three 18.2 GB SCSI local disks for system images, swap, /tmp and /var/tmp
- Consolidated consoles
- JURA Kit 48

**1 CFS**
**Server**

**RMS/QSW and CFS Partitions**
- Switch is 128-way
- Single RMS partition for capability (and capacity)
- Running with three partitions (64, 42, 14)
- Current CFS only scales to 32-way partition

## Smaller Clusters at LLNL

**IBM GA clusters**
**Blue - 442 Silver node (1,768xPPC604@332MHz) TB3 switch system on Open Network**
**Open/Classified HPSS servers**
**IBM Technology Integration, Support & Prototype Clusters**
**Baby – 8 Silver Wide – Problem isolation and SW eval**
**ER – 24 Silver Thin & 4 Silver Wide -  hot spares, workload simulators**
**16 THIN2 nodes – System admin**
**Snow – 16 NH-1/Colony/Mohonk (128xPower3@210MHz) prototype**
**Compaq GA**
**TeraCluste98 – 24 DS40 (**_**[96xEV56@500MHz](mailto:96xEV56@500MHz)**_**) - Open network**
**Compass – 8 DS8400 (**_**[80xEV5@440MHz](mailto:80xEV5@440MHz)**_**) - Open network**
**Forest – 6 DS8400 (**_**[60xEV56@500MHz](mailto:60xEV56@500MHz)**_**) - SCF**
**SierraCluster – 38 ES40 (**_**[152xEV67@667MHz](mailto:152xEV67@500MHz)**_**) - SCF**
**Compaq Technology Integration, Support and Linux Clusters**
**SandBox – 8 ES40 (**_**[EV6@500MHz](mailto:EV6@500MHz)**_**) - Problem isolation and SW eval**
**LinuxCluster – 8 ES40 (**_**[EV6@500MHz](mailto:EV6@500MHz)**_**) - Linux development**