# Presentation: 806136

## Interconnects: Which one is right for you?

**Brett M. Bode**
**Scalable Computing Laboratory**

- www.scl.ameslab.gov

## Motivation

- In the past 5 years the speed of the interconnects commonly found in clusters has improved by a factor of 20!
- While 1-3 years ago the PCs commonly available were unable to make full use of the available bandwidth, today’s systems are demonstrating some impressive performance.
- There are now several interconnect technologies that each offer advantages in certain situations.

## The Past

- 10 Mbps Ethernet
- ATM
- Fast Ethernet
- Others

## ALICE - 1997

- 64 dual 200 MHz Pentium Pro CPUs
- Fast Ethernet
- 256MB RAM per node
- 2GB disk per node
- 8 additional nodes dedicated to development
- Large Fast Ethernet based switch
- Theoretically capable of 100Mbits/sec (12 Mbytes/sec) full duplex between any pair of nodes (most systems can utilize 90% of that peak
- Switch has a 55 Gbit/sec backplane and can take modules with Fast Ethernet, Gigabit Ethernet, or several other technologies.
- Servers connected via Gigabit Ethernet

## Do we need faster networks?

- In general the compute/communication ratio in a cluster needs to remain fairly constant.
- So as the computational power increases the network speed must also be increased.
- This graph illustrates that balance working. It compares single CPU PII nodes with Fast Ethernet, versus dual CPU IBM workstations using Gigabit. (roughly factor of 6 in CPU, 10 in network)

## Lightweight Communications

- Some applications, like the GAMESS SCF test in this example have very low communications requirements and scale very well on almost any interconnect.
- The test compares running a calculation over Fast Ethernet and Gigabit Ethernet and with 1 or 2 CPUs active per node.
- Here the network makes virtually no difference!

## Communication Intensive Applications

- In this test the GAMESS parallel MP2 code is used. It uses a psuedo global shared memory approach leading to a high communication requirement.
- Here Fast Ethernet is NOT good enough.
- Gigabit Ethernet works better, but is still not fully adequate for two CPUs.

## The Present

- Gigabit Ethernet
- Myrinet
- SCI - Dolphin Wulfkit
- Quadrics

## Gigabit Ethernet

- Early Gigabit Ethernet boards obtained fairly meager performance (often 300-400 Mbps or below), at 10 times the cost of Fast Ethernet.
- One solution was to extend the Ethernet specification for MTU size to 9000 bytes.
- Doubles the performance in some cases (up to 940Mbps)
- Limits the choice of NICs and switches (Alteon, 3Com, SysKonnect, Extreme, Force10...)
- Have tended to be more expensive ($300-$800/NIC, $500 or more / switch port)

## Gigabit Ethernet

- A new generation of inexpensive (or even integrated) NICs offers very good performance for very low cost   (< $100/NIC).
- Similarly 4-24 port 1000BaseT switches are available at <$100/port.
- Larger Switches are still quite expensive (~$500/port for 32 ports, ~$1300/port for up to 336 ports).
- Switch aggregation really doesn’t work.

## Myrinet

- Early versions of Myrinet had a few hardware (mostly cabling) and software issues that affected stability
- About one year ago new software and the change to all fiber interfaces resulted in a dramatic improvement in reliability.
- Runs on a wide range of platforms.
- Scales to a large number of nodes via switch aggregation in Fat Tree arrangement.

## Myrinet

- Performance is quite good on the latest fast PCI 64bit-66MHz buses.
- Latency is low 7-10 μsec.
- Peak bandwidth 1800+Mbps.
- Performance in IP mode is also quite good, 1600+Mbps.
- Per port costs are ~$1500 for up to 128 ports, and up to ~$2200 for 1024 ports.

## SCI: Dolphin Wulfkit

- SCI offers 2 or three 500MByte per second external links
- Nodes are configured in a 2D or 3D wrapped mesh without switches.
- Pass through messages are handled entirely on the NIC at a very low additional latency.
- Scalability is limited by aggregate bandwidth on the link, a mesh dimension of 8-10 is a reasonable limit.
- Cable lengths are very short making setup a challenge.
- Per port costs are fixed at ~$1600 for 2D, ~$2100 for 3D.

## SCI: Dolphin Wulfkit

- Performance is quite good with very low latency (~4 μsec) and high peak bandwidth (1800 Mbps).
- Performance in the 1KB-100KB range is particularly impressive.
- IP mode is much less impressive.
- The mesh topology is particularly well suited to applications that divide work up on a rectangular grid. If these apps are mapped to the physical grid layout the scalability would be very good.

## Quadrics - QsNet

- Currently seen mostly on AlphaServer SC systems, but now also sold in an HP McKinley cluster (PNNL).
- Switched architecture.
- Hardware is capable of 2-5 μsec latencies
- Hardware link bandwidth is 340 MB/sec
- Configurations up to 1024 nodes with switches in a fat tree arrangement.
- Cost ??? > Myrinet/SCI

## Quadrics performance

- Application  performance on the SC has been a bit mixed.
- Overall scaling is quite good as the number of nodes in increased.
- But a closer look at the scaling within a box reveals a significant problem (probably a shared lock).

## The Future

- Infiniband
- Initial reference boards available now
- Initial performance looks promising                        (over 1300 Mbps in IP mode!)
- Software has a ways to go. (no MPI yet)
- Will it scale to large numbers of nodes?
- In order to really achieve its promise it must be tied in closer to the processor/memory bus (no PCI).
- IF Infiniband takes off as a standard storage, etc interconnect the price should plummet.

## The Future

- Higher performance products will be appearing from the current vendors.
- Higher performance will require moving to PCI-X buses in PCs (just beginning to happen).
- Vendors are moving to higher chip integration on NICs.
- Larger switches and more links per NIC.
- Eventually will see higher link speeds.

## Conclusions: So which one is the right one?

- Small Clusters

- Price is obviously a concern so Gigabit Ethernet is an obvious choice up to 32 nodes.
- For increased bandwidth on an SMP consider two NICs per node and two switches.

## Conclusions: So which one is the right one?

- Medium  sized Clusters  33-xxx nodes

- Prices for Gigabit Ethernet, SCI, and Myrinet are generally in the same range, but SCI and Myrinet offer performance up to a factor of two higher.
- Per port pricing increases quite a bit in this range from a low around $1400/port to ~$2200 per port.
- Performance of SCI and Myrinet are similar, if IP is needed Myrinet is a better choice. SCI performance is significantly better in the 1KB - 100KB message range.

## Conclusions: So which one is the right one?

- Large Clusters  > xxx nodes

- SCI, Myrinet and Quadrics all scale to 1000 nodes (in theory).
- Myrinet appears to be the only one to have 1024 nodes and larger systems installed.
- SCI will likely begin to show bottlenecks in the mesh by 103.
- Myrinet appears to be the best choice at the moment.

## Thank you

**$$$’s**
**U.S. Department of Energy**
**NSF**
**IBM**
**People**
**Alex Habeger**
**Troy Benjegerdes**

## Questions?

- ?