# Presentation: 097238

## Computational Plant (CplantTM)NPACI All-Hands MeetingDeveloping a Cluster Strategy for NPACI

**Computational Plant (Cplant****TM****)****NPACI All-Hands Meeting****Developing a Cluster Strategy for NPACI**

**Rolf Riesen (rolf@cs.sandia.gov)**
**Sandia National Laboratories**
**Scalable Computing Systems Department**

- http://www.cs.sandia.gov/cplant

- February 11, 2000

## FY97 - Phase I Prototype (Hawaii)

**128 Digital PWS 433a (Miata)**
**433 MHz 21164 Alpha CPU**
**2 MB L3 Cache**
**128 MB ECC SDRAM**
**24 Myrinet dual 8-port SAN switches**
**32-bit, 33 MHz LANai-4 NIC**
**Two 8-port serial cards per SSS-0 for console access**
**I/O - Six 9 GB disks**
**Compile server - 1 DEC PWS 433a**
**Integrated by SNL**

## FY98 - Phase II Production (Alaska)

**400 Digital PWS 500a (Miata)**
**500 MHz Alpha 21164 CPU**
**2 MB L3 Cache, 192 MB RAM**
**16-port Myrinet switch**
**32-bit, 33 MHz LANai-4 NIC**
**6 DEC AS1200, 12 RAID (.75 Tbyte) || file server**
**1 DEC AS4100 compile & user file server**
**Integrated by Compaq**
**125.2 GFLOPS on MPLINPACK (350 nodes)**
**would place 53rd on June 1999 Top 500**

## FY99 - Phase III Production (Siberia)

**624 Compaq XP1000 (Monet)**
**500 MHz Alpha 21264 CPU**
**4 MB L3 Cache**
**256 MB ECC SDRAM**
**16-port Myrinet switch**
**64-bit, 33 MHz LANai-7 NIC **
**1.73 TB disk I/O**
**Integrated by Compaq and Abba Technologies**
**247.6 GFLOPS on MPLINPACK (572 nodes)**
**would place 40th on Nov 1999 Top 500**

## FY00 - Phase IV Production

**Purchase in progress**
**~1024 more nodes attached to Siberia**
** ****--> ~1600-node Linux cluster**
**Likely to be again XP1000 or slates**
**In production use this Summer**

- ?

## Distributed & Parallel Systems

- Distributed
- systems
- hetero-
- geneous

- Massively
- parallel
- systems
- homo-
- geneous

- Legion\Globus

- Beowulf

- Berkley NOW

- Cplant

- Internet

- ASCI Red
- Tflops

**Gather (unused) resources**
**Steal cycles**
**System SW manages resources**
**System SW adds value**
**10% - 20% overhead is OK**
**Resources drive applications**
**Time to completion is not critical**
**Time-shared**

**Bounded set of resources **
**Apps grow to consume all cycles**
**Application manages resources**
**System SW gets in the way**
**5% overhead is maximum**
**Apps drive purchase of equipment**
**Real-time constraints**
**Space-shared**

- Tech Report SAND98-2221

## Cluster Pros & Cons

**Advantages**
**Additional resources to MPPs; e.g ASCI Red (Tflops)**
**Cheaper**
**Vendor independence**
**More control (e.g. architecture, topology, SW, etc.)**
**Disadvantages:**
**Performance is not scalable**
**Management is not scalable**
**Limited user environment**
**Not production quality**

## Challenges

**New purchase and operations model**
**Options and products change as evaluation is in progress.**
**No customer service engineer on call 7x24**
**Staff has to be able to diagnose to the node level**
**Reliability: node doesn’t boot; next time it is another node -> keep a log.**
**Being compared to ASCI Red**
**Community has little experience with large scale clusters (******** 256 nodes)**

## More Challenges

**Beowulf software doesn’t cut it for ******** 128 nodes**
**Non-scalable applications**
**Newcomers are drawn to clusters, but everything has to be scalable, including the application, to gain performance**
**PC mentality**
**Linux lets me do this on my desktop, why can’t I do it on 1000 nodes?**
**System software (including diagnostics and admin) is not there yet **

- 

## Recommendations

**If you can live with a cluster ******** 128, go for it (no glory, but practical and cost-effective)**
****** ****512 is a major challenge (lots of potential for glory or chopped heads)**
**Assign system software people to support users and diagnose problems (in addition to application support people who are already there)**
**Assign system administrators with hardware experience**
**Work with vendors**
**Buy SMPs of size 1**

## http://www.cs.sandia.gov/cplant