# Presentation: 028566

## A Hybrid MPI Design using SCTP and iWARP

- Distributed Systems
- Group

**Mike Tsai, ****Brad Penoff****, and Alan Wagner**
- Department of Computer Science
- University of British Columbia
- Vancouver, Canada

- April 14, 2008

## A Hybrid Message Passing Interface Design using theStream Control Transmission Protocol and the Internet Wide Area Remote Direct Memory Access Protocol

- Distributed Systems
- Group

**Mike Tsai, ****Brad Penoff****, and Alan Wagner**
- Department of Computer Science
- University of British Columbia
- Vancouver, Canada

- April 14, 2008

## Research Background

- SCTP – Stream Control Transmission Protocol
- IETF standardized transport protocol for IP
- Can be used anywhere TCP or UDP are used
- Additional features
- SCTP and MPI middleware
- LAM (unreleased)‏
- MPICH2 (1.0.5 and on) ch3:sctp
- Open MPI SCTP BTL (in v1.3 trunk)‏

## State-of-the-Art Networking

- Hardware acceleration techniques for IP
- Protocol offload
- OS bypass
- Zero copy
- RDMA
- 10 GigE
- How would these look for SCTP?
- Are there benefits here for using SCTP?

## Story/motivation

- iWARP - Internet Wide Area RDMA protocol
- IETF standard for RDMA over IP
- Use RDMA, point-to-point, or a mix?
- “Why Compromise?” (G. Shainer @ HPCWire.com)‏
- Depending on the application, use whichever is best.
- For MPI middleware, who decides what’s best?

- The programmer!

## Contribution

- Hybrid MPI with functional decomposition lets the programmer decide:
- Let RMA use RDMA
- Let other communications use point-to-point
- Explore SCTP’s use within iWARP
- Extended OSC userspace software iWARP, making many internal OSC changes

## iWARP : DDP & LLP

**D****irect ****D****ata ****P****lacement**
- Fragments messages
- Reassembles segments
- Segments self-contained
- Data delivery and placement separation
- Out-of-order delivery
- Requires LLP to:
- Keep segment boundaries
- Be reliable
- Take a strong checksum

## iWARP : LLP = MPA over TCP

**M****essage ****P****DU ****A****ligned**
- Message framing
- DDP segment vs. TCP stream
- Markers for out-of-order
- For middlebox fragmentation
- Stronger checksum
- ... is a complex layer (majority of OSC code)!
- ... can lead to non-compliant TCP stacks.

- LLP

## SCTP is a better LLP

- LLP’s needs built-in to SCTP:
- Reliable, message-based
- CRC32c checksum
- Out-of-order support:
- MSG_UNORDERED
- Multistreaming
- Multihoming
- Unmodified stack supports:
- Path failover
- Multirail data striping

- LLP

## In the beginning, there was ch3:sctp

## OSC iWARP was modified and incorporated in as a thread....

## RMA done by modified OSC iWARP

## OSC iWARP changes to support MPI

- Running in a thread
- Use SCTP
- Making all OSC ops non-blocking
- Locks around shared data

## Connection Management Design 

- Connection establishment:
- Separate one-to-many socket for new QPs
- SCTP “peeloff” feature
- New QP sends request from one-to-many socket
- Request/ACK received, then QP socket peeled-off
- For conflicts, MPI rank resolves who sends ACK

## Progress Engine

## Performance

-    What we tested...
- Compared our new ch3:hybrid  to the original ch3:sctp
- Two 3.2 GHz Intel boxes (GigE + switch)‏
- OSU latency tests (MPI_Put & MPI_Get)‏
- Homemade synthetic benchmark
- Combination of RMA and MPI-1 calls

## OSU One-sided Latency Tests

- ch3:hybrid  adds 2-8% overhead

## Synthetic Application

- ch3:hybrid was faster than ch3:sctp
-  3.8 seconds vs. 4.5 seconds
- Extra thread helps in some cases

## Conclusions

- RDMA versus point-to-point for MPI
- Why choose?
- Functional decomposition lets programmer decide
- SCTP is a good match for iWARP
- Implementation of iWARP using SCTP shown.
- SCTP has its place in the state-of-the-art.
- It’d be more exciting to have SCTP-based devices...

## Thank you!

**Thank you!**

## Connection Management Design 

- Connection Management Design