# Presentation: 714890

## Ron Brightwell / Rolf Riesen
Sandia National Laboratories
Barney Maccabe
The University of New Mexico

**Sandia National Laboratories**
**Barney Maccabe**
**The University of New Mexico**

**http://www.cs.sandia.gov/cplant*******
*******The blueprint for Chiba City**

**Extreme Linux Developers Forum**
**February 24, 2000**

**Computational Plant (Cplant****TM****)**

## Why Can't TCP/IP Just be Fast ;-) ? 

**Because it is a wide area network protocol.**
**Fast != scalable anyway.**
**Low latency and high bandwidth are necessary, but not sufficient for scalability.**
**Use mismatch:**
**client/server (authentication, validation, etc.)**
**Scientific codes are based on tag bit matching, not streams**

## OS Bypass: Solution or Bandaid?

**It is a bandaid if it is done to support TCP/IP.**
**No need to make TCP/IP fast!**
**OS Bypass: push OS policies into the NIC**
**current OS Bypass (VIA and ST) emphasize connections (e.g., TCP) and miss the critical point**
**Application Bypass: push application policies (e.g., matching) into the NIC**

## Difference Between WAN and Cluster?

**Cold phone call to a large corporation**
**Find the phone number**
**Get a dial tone**
**Dial the number**
**Announce your presence (ring the bell) **
**Find and get connected to the right person**
**Identify yourself**
**State why you are calling**
**Provide a credit card number**
**Talk**

**Pillow Talk**
**Talk**

**Reliability, routing, flow control, you name it...**

## Does "The Grid" Scale Down?

**When did it scale up?**
**No (if you care about performance)**
**We need environments that are tuned for tightly integrated clusters (single point of ownership)**
**Each cluster needs to be a single (large) machine**
**now we have clusters, we need MPPs**

## Tera-Scale Scalability Issues (1 of 371)

**Non-Networking or message passing:**
**Administration**
**Install and update new sys SW**
**(Re)Configure sys (compute, service, I/O) **
**Manage user accounts, file systems, ...**
**Handling 6000 floppies is not scalable**
**Reliability**
**Deal with node, network, and switch failures:**
**detect, locate, disable, replace, re-enable**
**No single point of failure**

## Tera-Scale Scalability Issues (2 of 371)

**Using components that were not designed for a scalable system**
**Out of 6000 floppy drives to boot a system, many will fail**
**5% floppy failures on ANL cluster**
**5% on 6000-node system **
**= 300 nodes that don't boot**
**= 300 floppies someone has to touch**

## Tera-Scale Scalability Issues (3 of 371)

**Message-passing system software**
**Something that supports MPI**
**Something that supports I/O**
**Something that supports the runtime environment**
****** ****Portals 3.0**** (see Sandia Techreport)**
**Networking HW**
**Arbitrary topology**
**On the memory bus (probably not Infiniband)**
**Source routed**
**Connectionless**
**Programmable NICs**

## http://www.cs.sandia.gov/cplant

## Tera-Scale Scalability Issues (4 of 371)

**Something that supports MPI:**
**low latency, high bandwidth, low host processor overhead**
**progress rule**
**real asynchronous communication**
**one sided completion semantics**
**no buffers for unexpected messages for every process in the job**
**application bypass**
**no user-level progressing of the network**
**thread support**

## Tera-Scale Scalability Issues (5 of 371)

**Something that supports I/O:**
**real asynchronous communication**
**sender-managed communication - sender determines offset**
**Something that supports the runtime environment**