# Presentation: 364948

## High-Performance Communication Protocols that Scale

- Sandia National Laboratories
- Student Seminar Series - 2001
- Rolf Riesen
- July 26, 2001

## Cplant

- Largest scientific Linux cluster in the world
- Will reach almost 1 Teraflops this fall
- Scalable Computing Systems group at Sandia responsible for design and implementation of Cplant
- ~1920 network interfaces, 448 16-port switching elements

**http://www.cs.sandia.gov/cplant**

- Sandia Summer Student Seminar Series

- July 2, 2001

## Distributed and Parallel Systems

**Distributed**
**systems**
- hetero-
- geneous

**Massively**
**parallel**
**systems**
- homo-
- geneous

- Legion\Globus

- Beowulf

- Berkley NOW

- Cplant

- Internet

- ASCI Red
-       Tflops

- Gather (unused) resources
- Steal cycles
- System SW manages resources
- System SW adds value
- 10% - 20% overhead is OK
- Resources drive applications
- Time to completion is not critical
- Time-shared

- Bounded set of resources
- Apps grow to consume all cycles
- Application manages resources
- System SW gets in the way
- 5% overhead is maximum
- Apps drive purchase of equipment
- Real-time constraints
- Space-shared

- SETI@home

- Sandia Summer Student Seminar Series

- July 2, 2001

## Outline

- Network architectures
- Characteristics of MPP networks
- TCP/IP
- RMPP
- Congestion and Flow Control
- OS and application bypass

- Sandia Summer Student Seminar Series

- July 2, 2001

## MPP Network Architecture

- CPU

- CPU

- Memory

- Memory Bus

- FIFOs

- Network

- Memory bus to network ratio  1:1
- memcpy() unacceptable

- Sandia Summer Student Seminar Series

- July 2, 2001

## COTS Network Architecture

**COTS Network Architecture**

- CPU

- CPU

- Memory

- Memory Bus

- Network

- PCI Bridge

- PCI Bus

- Bus to network ratio  100:1
- memcpy() are no problem

- Sandia Summer Student Seminar Series

- July 2, 2001

## Myrinet Network Architecture

**Myrinet Network Architecture**

- CPU

- CPU

- Memory

- Memory Bus

- Network

- PCI Bridge

- PCI Bus

- Memory

- DMA

- DMA

- DMA

- DMA

- NIC

- PCI bus to network ratio  1:1
- memcpy() unacceptable

- Mem to PCI bus ratio  10:1
- memcpy() OK

- CPU

- Sandia Summer Student Seminar Series

- July 2, 2001

## General Network Characteristics

- Completely reliable
- Occasional bit errors
- Frequent bit errors
- Truncated packets
- Misrouted packets
- Delayed packets
- Reordered packets
- Lost packets
- Duplicated packets
- Replay of messages
- Performance

- None? (ASCI Red)
- Myrinet
- Ethernet
- Myrinet switch resets
- IP guards against that
- due to congestion, long path
- multiple paths
- IP routers when flooded
- Retransmission
- Due to delayed packets (protocol)
- “Mysterious” slowdowns

- Level of reliability

- Examples

- Sandia Summer Student Seminar Series

- July 2, 2001

## Characteristics of MPP Networks

- Examples (which we have had here at Sandia)
- nCUBE 2, Intel Paragon, DOE ASCI Red, Myrinet
- Source routed
- Non-adaptive
- Misrouting possible (IP disallows that) [unlikely]
- No packet reordering
- No packet duplication
- No packet loss (except for Myrinet with timeouts)
- No maximum packet length (except for Myrinet)
- Low bit error rate (BER) (compared to Ethernet)

- Sandia Summer Student Seminar Series

- July 2, 2001

## Detect and correct bit errors
Resend lost, damaged, and misrouted packets
Ignore duplicated packets & messages
Guarantees packet & message order
Fragmentation/reassembly
Multiplexing/demultiplexing
Flow control
Congestion control (Jacobson)
Stream oriented

- Resend lost, damaged, and misrouted packets
- Ignore duplicated packets & messages
- Guarantees packet & message order
- Fragmentation/reassembly
- Multiplexing/demultiplexing
- Flow control
- Congestion control (Jacobson)
- Stream oriented

**What TCP/IP Does**

- Sandia Summer Student Seminar Series

- July 2, 2001

## What is Needed

**What is Needed**

- Correct bit errors

- Resend packets

- Avoid duplication

- Guarantee order

- Reassembly

- Multiplexing

- Flow control

- Congestion control

- Stream oriented

- Ethernet

- 

- 

- 

- 

- 

- 

- 

- 

- 

- Sandia Summer Student Seminar Series

- July 2, 2001

## What is Needed

**What is Needed**

- Correct bit errors

- Resend packets

- Avoid duplication

- Guarantee order

- Reassembly

- Multiplexing

- Flow control

- Congestion control

- Stream oriented

- -

- -

- -

- -

- -

- -

- maybe

- -

- -

- Ethernet

- 

- 

- 

- 

- 

- 

- 

- 

- 

- Perfect
- Network

- Sandia Summer Student Seminar Series

- July 2, 2001

## What is Needed

**What is Needed**

- Correct bit errors

- Resend packets

- Avoid duplication

- Guarantee order

- Reassembly

- Multiplexing

- Flow control

- Congestion control

- Stream oriented

- Perfect
- Network

- Almost
- perfect

- -

- -

- -

- -

- -

- -

- maybe

- -

- -

- 

- -

- -

- -

- -

- -

- maybe

- ?

- -

- Ethernet

- 

- 

- 

- 

- 

- 

- 

- 

- 

- Sandia Summer Student Seminar Series

- July 2, 2001

## What is Needed

**What is Needed**

- Correct bit errors

- Resend packets

- Avoid duplication

- Guarantee order

- Reassembly

- Multiplexing

- Flow control

- Congestion control

- Stream oriented

- Perfect
- Network

- Almost
- perfect

- -

- -

- -

- -

- -

- -

- maybe

- -

- -

- 

- -

- -

- -

- -

- -

- maybe

- ?

- -

- Ethernet

- 

- 

- 

- 

- 

- 

- 

- 

- 

- Myrinet

- 

- 

- -

- -

- -

- -

- maybe

- ?

- -

- Sandia Summer Student Seminar Series

- July 2, 2001

## The Reliable Message Passing Protocol

**The Reliable Message Passing Protocol**

- Correct bit errors

- Resend packets

- Avoid duplication

- Guarantee order

- Reassembly

- Multiplexing

- Flow control

- Congestion control

- Stream oriented

- Perfect
- Network

- Almost
- perfect

- -

- -

- -

- -

- -

- -

- maybe

- -

- -

- 

- -

- -

- -

- -

- -

- maybe

- ?

- -

- Ethernet

- 

- 

- 

- 

- 

- 

- 

- 

- 

- Myrinet

- 

- 

- -

- -

- -

- -

- maybe

- ?

- -

- RMPP

- 

- 

- 

- 

- -

- 

- 

- No

- -

- † Due to resends

- ‡ Due to packetization

- †

- †

- ‡

- Sandia Summer Student Seminar Series

- July 2, 2001

## Why not TCP/IP?

- It works for the Internet! Cplant is smaller than that!! Why don’t you use TCP/IP?
- Performance:
- Protocol overhead
- Store and forward switches‡
- Scalability:
- Number of simultaneously open connections on a given node
- Processing speed on a slow NIC
- Switches limit topology and, therefore, bysection bandwidth (a 2-D mesh contains loops ) ‡

- ‡Characteristics of Ethernet, not the protocol

- Sandia Summer Student Seminar Series

- July 2, 2001

## Can we Ignore Congestion Control?

- Sandia scientific applications running on sufficiently rich network topologies create little congestion...
- Myrinet switch timeouts cause a problem...

- Sandia Summer Student Seminar Series

- July 2, 2001

## Can we Ignore Flow Control?

- Flow control at the physical layer is handled by hardware
- If we are using OS and Application bypass, protocol layer flow control may not be needed

- Sandia Summer Student Seminar Series

- July 2, 2001

## OS Bypass

- Bulk data flows directly into user space
- No interrupts to OS to handle incoming messages
- Address validation requires interaction with OS and is done before message starts arriving

- Sandia Summer Student Seminar Series

- July 2, 2001

## Application Bypass

- No need of user-space code to do anything to let an incoming message progress:
- Long messages
- Acknowledgements
- Examples: non-blocking MPI receive of a large message while application is in BLAS library
- Data must continue to flow; e.g. subsequent messages or large messages

- Sandia Summer Student Seminar Series

- July 2, 2001

## http://cs.sandia.gov/cplant

- Sandia Summer Student Seminar Series

- July 2, 2001