# Presentation: 667534

## An Evaluation of Current 

- High-Performance Networks

- Christian Bell, Dan Bonachea, Yannick Cote, Jason Duell, Paul Hargrove, Parry Husbands, Costin Iancu, Michael Welcome, Kathy Yelick
- Lawrence Berkeley National Lab
- &
- U.C. Berkeley

- http://upc.lbl.gov

## Motivation

- Benchmark a variety of current high-speed Networks
- Measure Latency and Software Overhead, not just Bandwidth
- One-sided communication provides advantages vs. 2-sided MPI?
- Global Address Space (GAS) Languages
- UPC, Titanium (Java), Co-Array Fortran
- Small message performance (8 bytes)
- Support sparse/irregular/adaptive programs
- Programming model: incremental optimization
- Overlapping messages can hide the latency

## Systems Evaluated

| System | Network | Bus
(per sec) | 1-sided hardware | APIs |
| --- | --- | --- | --- | --- |
| Cray T3E | Custom | Custom
(330 MB) |   | SHMEM, 
E-registers |
| IBM SP | SP switch 2 | GXX bus
(2 GB) |  | LAPI |
| HP AlphaServer | Quadrics | PCI 64/66
(532 MB) |   | SHMEM |
| IBM Netfinity | Myrinet | PCI 32/66
(266 MB) |   | GM |
| PC cluster | GigE | PCI 64/66
(532 MB) |  | VIPL |

## Modified LogGP Model

**LogGP: no overlap**

**Observed: overheads can overlap:  L can be negative**

- EEL: 	end to end latency (instead of transport latency L)
- g:  	minimum time between small message sends
- G:  	additional gap per byte for larger messages

## Microbenchmarks

- P0

**o****send**

**gap**

- P0

**o****send**

**gap**

**cpu**

- P0

**o****send**

**gap**

**cpu**

- Ping-pong test:  measures EEL (end-to-end latency)
- Flood test: measures gap (g/G)
- CPU overlap test: measures software overheads

- Flood Test

- CPU Test 1

- CPU Test 2

## Latencies for 8 byte ‘puts’

## 8-byte ‘put’ Latencies with Software Overheads

**Notes:**
Zero receive overhead for some: significant if 2-way communication
MPI cost: not just higher latency, but also less potential for hiding latency

## Gap varies with msg clustering

- Clustering messages can both use idle cycles, and reduce the number of idle cycles that need to be filled

**Notes:**
Compiler can automate this, hopefully.
How did we measure this? (CPU overlap, with different q)

## Potential for CPU overlap during clustered message sends

- Hardware support for 1-way communication provides more opportunity for computational overlap

**Notes:**
Confusing:  show best ‘g’, but overhead at q=1.
Advantage of 1-way even more pronounced if simultaneously sending/recving msgs.

## Fixed message cost (g), vs. per-byte cost (G)

**Notes:**
Gives rough idea of the importance of packing messages

## “Large” Messages

- Factor of 6 between minimum sizes needed for “large” message (large = bandwidth dominates fixed message cost)

## Small message performance over time

- Software send overhead for 8-byte messages over time.
- Not improving much over time (even in absolute terms)

## Conclusion

- Latency and software overhead of messages varies widely among today’s HPC networks
- Affects ability to effectively mask communication latency, with large effect on GAS language viability
- especially software overhead--latency can be hidden
- These parameters have historically been overlooked in benchmarks and vendor evaluations
- Hopefully this will change
- Recent discussions with vendors promising
- Incorporation into standard benchmarks would be nice...
- http://upc.lbl.gov