# Presentation: 801323

## Network Interface Cards (NICs) as First-Class Citizens

- Wu Feng*†
- feng@lanl.gov
- Ronald G. Minnich* and Fabrizio Petrini*

- * Los Alamos National Laboratory
-   Los Alamos, NM  87545

- Purdue University†
- W. Lafayette, IN  47907

**Notes:**
First Class:  Moving NIC onto memory bus.
Idea is fairly new,
e.g., ASCI Blue Pacific (SGI & AGP also).

However, making it a TRUE first-class citizen ...
Means more than MOVING the NIC on memory bus.
Means TREATING the NIC as a first-class citizen.
This is 1 of 2 things that I suspect that ASCI Blue Pacific failed to do (the other is implementing an efficient OS-bypass mechanism).

## The ASCI Target (or ASCI Curves)

- 1011

- 1012

- ‘96

- ‘97

- ‘00

- ‘03

- Year

- 1013

- 1014

- Computing Speed (FLOPS)

- 0.05

- 0.5

- 5

- 50

- Memory (TB)

- 0.13

- 1.3

- 13

- 130

- Archival
- Storage (PB)

- Parallel I/O (Gbps)

- 5

- 50

- 500

- 5000

- 0.13

- 1.3

- 13

- 130

- Network Speed (Gbps)

- ... a good start but ...

**Notes:**
We need to be more concerned with the bottlenecks between these components!

## Recent Solutions BetweenProcessor & Network

- HiPPI-6400 NIC (beta prototype)
- NIC processor to free CPU from network operations.
- Hardware capabilities
- IP checksum
- Error detection and re-transmission
- Flow control
- Low-level messaging operations for OS-bypass protocols.
- OS-Bypass Protocol
- Orders-of-magnitude reduction in app-to-network latency.
- Problem
- Application-to-network (vice versa) still a bottleneck!

**Notes:**
HiPPI-6400
Runs at 6400 Mb/s = 6.4 Gb/s
Developed by our group at LANL (in conjunction with SGI).
Commercial:  Super HiPPI Access Controller (SHAC 2.0 by SGI).
Low-level messaging operations such as “Clear-to-Send” (specific to ST).

PROBLEM
Application-to-network interface STILL A BOTTLENECK!

## Current PC Technology

- Goal
-  Alleviate app/network
-       bottleneck.

- (Example) Benefits
-  Enable QoS in middleware.
-  WWW  World Wide Wait
-  Remote Viz (FY01)
- 	- 80 GB/s = 640 Gb/s.
-  High-speed bulk data transfer.

- Future for I/O Bus and Memory Bus, respectively ...
-  *PCI-X:*  4.3 Gb/s (1Q00) & *RAMBUS:*  9.6 Gb/s, 28.8 Gb/s, 86.4 Gb/s (“now”).

- SGI XIO?  6.4 Gb/s max, 0.8 Gb/s delivered.
- Problems:  Directory-based cache coherency (ccNUMA) & 10:1 CPU:NIC ratio.

**Notes:**
Look at PC due to the emergence of PC clusters.
RAMBUS: < half the pins as 100 MHz DRAM
DEC Alpha 21264
600 MHz * 6 instr/cycle = 3.6 Gips
3.6 Gips * 64 bits/instr = 230 Gbps

Component		"Latency"	"Bandwidth”
CPU			1-2 ns	3.6 Gips (1.68)
DRAM access time	60-120 ns	6.4 Gbps (> 4)
Network link		< 1 s	6.4 Gbps	
Appl-to-net interface	5-10 s	600-850 Mbps
RAID I/O BW		50 Gbps

Trends:	CPU speed doubling every 18 months.
		Memory bus speed increasing.  100 MHz.
			RAMBUS!

## NICs as First-Class Citizens

- CPU

- $

- Main
- Memory

- Memory Bus

- I/O Bus

- I/O
- Bridge

- NIC

- CPU

- $

- Main
- Memory

- I/O
- Bridge

- NIC

- I/O Bus

- Memory Bus

- Network

- Note:  Each node could contain multiple CPUs.

-  Move NIC to memory bus.
- 	-  What’s new?
-  Integrate NIC into memory
-     subsystem.
-  Treat NIC as a peer CPU.

- $

- Goals
-  Alleviate application/network
-     bottleneck.

**Notes:**
APPROACH = SUBGOALS

Current NICs:  Sit on I/O bus (i.e., second-class)!
Future NICs:  Sit on memory bus (i.e., first-class)!

Making a NIC a true first-class citizen involves MORE than just MOVING NIC to memory bus, it involves TREATING it as first-class citizen.
This is where ASCI BLUE PACIFIC fails, and this is where IBM & Compaq will fail.

Their Solution
Move NIC onto the memory bus.
Treat the NIC as a peripheral device on the memory bus. 

Problems
DMA transfer
Too much overhead.  Bad for fine-grain communication.
Interrupts
Corrupts the cache.  Adversely affects cache hit rate.
ASCI Blue Pacific
600 Mbps with NIC attached to memory bus.
Application-to-application latency?

## NIC Access Memory Access

## Status

- Internal interface:  Memory access.
- External interface:  Myrinet.
- Problems:	Myrinet performance degrades under heavy load.
-  		Nearly all other technologies are PCI I/O-based.
- Solution:	HiPPI-6400 when commercially available.
-  		Use Myrinet to prototype for now.
- Implementation of Intel x86-based simulator underway.
- Re-evaluation of FPGAs to implement our NIC processor:  SLAAC instead of RCA-2 & Pamette.

**Notes:**
1.	Pamette FPGA card (COMPAQ) as in ALVIN
Altera chips
Non-standard non-HDL environment 
Partially open -- details available under NDA 
RCA FPGA card performs better.
2.	RCA-2 (LANL)
Altera chips (large, 250K gate chips) 
Standard VHDL environment 
Wide open with local designers supporting us 
HIGH performance via external connectors (1 Gb/s)
But not to VME. Its interface: only 16 bits data 
Supports add-on Myrinet card
3.	SLAAC (DARPA)
Xilinx chips 
Standard VHDL environments 
Wide open design -- source, hardware specs available on the web 
Supports add-on myrinet card 
Not a high performance design 

VeriBest CAD on Windows NT.

## Future Work

- Finalization of internal/external interface design. (12/99)
- Simulator
- Issues to address in presence of a cache-coherent NIC.
- Bus-based cache-coherency protocol. (5/00)
- Scalability of system bus. (10/00)
- NIC integration within CPU?
- Hardware Prototype
- Identification of existing CAD environment to support work. (11/99)
- Completion of initial prototype. (10/00)
- Simulation results guide evolution of prototype. (Ongoing ...)
- Continuing discussions with Intel.

**Notes:**
Bus-based cache-coherency protocol.
False sharing?
Scalability of system bus.
When does the bus saturate?  Depends in part on cache coherency.

## DOE NGI Program Structure

- “Regular” NGI PI meetings
- To address interface issues between NGI technologies.
- To stay abreast of the work of fellow NGI colleagues.

- Hardware

- Architecture

- Network Software

- Middleware

- Apps

- M
- e
- a
- s
- u
- r
- e
- m
- e
- n
- t

- Network software implies
-  High-speed protocols
-  QoS protocols
-  OS/network interface

## DOE NGI Program Structure

- More tightly-coupled collaboration across academia, government labs, and industry.
- Why?
- Advance research & development at a faster rate.
- Enhance visibility of the overall DOE NGI program.
- Ensure that our basic research has practical application.
- Examples
- HiPPI-6400 (LANL and SGI)
- NICs as First-Class Citizens (LANL, Purdue, and Intel)