# Presentation: 265569

## The Quadrics Network (QsNET): High-Performance Clustering Technology

- Fabrizio Petrini, Wu-chun Feng,  Adolfy Hoisie, Salvador Coll and Eitan Frachtenberg
- CCS Group
- Los Alamos National Laboratory

## Resources

- More information can be found at
- http://www.c3.lanl.gov/~fabrizio
- Quadrics web site
- http://www.quadrics.com
- Or sending an e-mail to
- fabrizio@lanl.gov

- Hot Interconnects 2001

## Outline

- Introduction
- Quadrics network design
- Elan
- Elite
- Fat-trees, topological properties, routing algorithm, flow control
- Communication/Programming libraries
- Performance Analysis
- Experiment description
- Results

- Hot Interconnects 2001

## Network Overview

- QsNET provides an abstraction of distributed virtual shared memory
- Each process can map a portion of its address space into the global memory
- These address spaces constitute the virtual shared memory
- This shared memory is fully integrated with the native operating system

- Hot Interconnects 2001

## Building Blocks

- The QsNET is based on two building blocks
- A network interface card called Elan
- A crossbar switch called Elite

- Hot Interconnects 2001

## Elan

- 400 MB/sec
- Bidirectional
- 200 Mhz/10 bits

- 2 Virtual
- Channels

- TLB Synchronized
- With Host

- Thread Processor
- Runs Communication Protocols
- 32-bit SPARC-based

- Performance Counters
- 51 Events
- 8 32-bit Registers

- 66 Mhz/64-bit
- PCI Interface

- Hot Interconnects 2001

- Hot Interconnects 2001

## Elite

- 8 bidirectional links with 2 virtual channels in each direction
- An internal 16x8 full crossbar switch
- 400 MB/sec on each link direction
- Packet error detection and recovery, with routing and data transactions CRC protected
- 2 priority levels
- Hardware support for broadcast
- Adaptive routing

- Hot Interconnects 2001

## A Quaternary Fat Tree

- Hot Interconnects 2001

## Fat Tree Recursive Topology

- Hot Interconnects 2001

## Adaptive Routing 

**Adaptive Routing**

- Hot Interconnects 2001

- Hot Interconnects 2001

## Packet Format

- Transaction Type

- Memory  Address

- Data (up to 64 Bytes)

- Context Value

- CRC

- Route

- One or More Transactions

- EOP

- 320 Bytes Data Payload
- 74-80 Bytes Overhead

- Hot Interconnects 2001

## Wormhole Flow-Control

- Hot Interconnects 2001

## Virtual Channels

- Hot Interconnects 2001

## Programming Libraries

- Elan3lib
- event notification
- memory mapping and allocation
- remote DMA
- Elanlib and Tports
- MPI

- Hot Interconnects 2001

- tport

- tports

- Hot Interconnects 2001

## Execution of a Remote DMA

- μproc

- Event

- Source

- DMA descriptor

- Event

- MMU

- Table Walk
- Engine

- Inputter

- Command
- proc

- DMA
- Engine

- μproc

- Event

- Destination

- Event

- Inputter

- DMA
- Engine

- MMU

- Table Walk
- Engine

- Command
- proc

- Elan SDRAM

- Hot Interconnects 2001

## Experimental Framework

- Cluster of 16 dual-processor SMPs
- Intel 733 Mhz Pentium III
- Motherboard: based on ServerWorks HE chipset which provides 2-64 bit 66Mhz PCI slots
- Each SMP is equipped with one Elan
- One 16-port Elite Switch is use for interconnection

- Hot Interconnects 2001

## Unidirectional

- Hot Interconnects 2001

## Latency Unidirectional

- Hot Interconnects 2001

## Unidirectional bandwidth on the ES40

- Hot Interconnects 2001

## Bidirectional

- Hot Interconnects 2001

## Latency Bidirectional

- Hot Interconnects 2001

## Hotspot

- Hot Interconnects 2001

## Hardware Barrier

- Hot Interconnects 2001

## Conclusion

- QsNet provides the abstraction of a distributed  virtual shared memory with protected and fault-tolerant communication
-  Latency is as low as 2s and bandwidth as high as 307 MB/s
- Network Bandwidth between Elan buffers is 335 MB/s
- Performance degradation in the presence of bi-directional traffic, due to the current PCI bus implementations

- Hot Interconnects 2001