SC08

Booth Display & Demonstration @ #3003
MaSTER-1: 5-port 10GbE Testbed
Performance Optimization of TCP/IPGRAPE-DR
Katsuyuki Hasebe
The University of Tokyo&National Institute of Information and Communications Technology

Booth Display & Demonstration• MaSTER-1– 5-port 10GbE Testbed• Stream Harmonizer– Optimizing parallel TCP stream/performance• Performance optimization of TCP/IP• High speed TCP communication experiments• CosmoGrid• GRAPE-DR processor chip/system

2

MaSTER-1
5-port 10GbE Testbed
MaSTER-1 Overview•10GbE (LAN PHY) experimental testbed•5 XFP ports connected to FPGAs through MACs
•All FPGAs are connected all-to-all at the speed of 10 Gbps
•FPGAs can be communicated with a control PC thorough a USB port

MaSTER-1 Advantages•Packets input from a port can be processed at the speed of 10 Gbps
•Packets can be output from any ports•Each port has powerful configurable FPGA
•Each port has large memories to store packets

MaSTER-1 (12 Layers PCB)

MaSTER-1 Applications
・Programmable 10GbE switch•Currently running•Packet Filters•Packet Logger•Pseudo Long Fat-pipe Networks•Maximum 400 ms delay

MaSTER-1
improves
theperformance of Parallel TCPStreams•Multiple ports allow MaSTER-1 tohandle multiple connections
without switches•MaSTER-1 can observe directly
the packets transmitted by endhosts•MaSTER-1 will clarify theproblems with the method for
dropping and merging packets in10GbE switches• MaSTER-1 is a good tool forverifying the performance ofparallel TCP streams on Long Fatpipe Networks (LFNs)

3

MaSTER-1
5-port 10GbE Testbed
MaSTER-1 in a Rack

MaSTER-1 Specification
Ports
Processors
Interconnection
MemoryI/ODimension5 XFPs (10GBASE-SR/LR)
5 FPGAs (Xilinx XC5VFX70T-1FF1136) – 1 for each port
All-to-all 10 Gbps – Xilinx Rocket IO
DDR2 SDRAM 2.56 GiB – 512 MiB for each processor
USB 2.0430 mm x 430 mm x 50mm (WDH)

MaSTER-1 Block Diagram
10GBASE-SR/LR

MaSTER-1 FPGA Interconnections

XFP
MAC
Serdes
10GBASE-SR/LR
10GBASE-SR/LR
FPGA

MAC
Serdes

FPGA

GTX
IF

FPGA

MAC
Serdes

DDR2512MB

DDR2512MB

FPGA

FPGA

CPLD

USB

SPI
IF

SPI
IF

GTX
IF

FPGA Controller
10GBASE-SR/LR

MAC
Serdes

MAC
SerdesGTXIF
FIFO

10GBASE-SR/LRSPIIF
DDR2
512MB

4

Stream Harmonizer
optimizing parallel TCP stream/performanceinter-packet gap(se c)

0.07• Inter-cluster data transfer using parallel TCP streams• Throughput unbalance among streams• Fairness lost because of slow recovery of flow on LFN
Data transfer speed limited by slow streams
cluster

Senderhost 3234567

8

Flow balancing effect of the proposed scheduler

9 10 11 12 13 14 15 16 17 18 1 9 20RTT

Evaluating the scheduler using experimental network

Senderhost 3

Packet scheduler to synchronize loss timingflow(Mbits/sec)

600
400

1222

3 334441 1

23 4

123

Packet queue
for each stream

Rate control
for each stream

stream 1

Throughput of each stream (average of 300ms sections)• Different packet loss timing at starting phase• Generating throughput unbalance among streams• Slow feedback to shrink the throughput difference• Throughput unbalance preserved

stream 2………
host 1
………

………

………
host kLAN

flo w(Mbits/se c)

150time(se c)

18 0

2 10240270

30 0totalstream 0
stream 1
stream 2
stream 3

80 0
60 040 0Stream Harmonizer

DDR-SDRAM
2GBPacket scheduling FPGA logic on TGNLE-1

8• Packet transmission order preserved among streams• stream 0, 2, 1, 3
• Last stream experiences packet loss first• switch buffer crowded by the other streams• packet loss order : stream 3, 1, 2, 0

DDR-SDRAM 34Gbps, 2GB

DDR-SDRAM
2GBDDR-SDRAM interface

CH0 FPGA

CH1 FPGA

20 0010GbE MAC
Rx

1000
total
stream 0
stream 1
stream 2
stream 3

800
600
400
200
04

4.5

5

5.5time (sec)

66.5

7

Throughput of each stream(average of 5ms sections)• Synchronized packet
transmission• Fairness among streams

10 00

7time(sec)

120

1200

• Packet scheduling hardware
at sender gateway• Per-stream queue• Transmission timing control
for each stream

WAN

Block diagram of network testbed TGNLE-1

690

Throughput of each stream (average of 300ms sections)• Synchronized packet loss timing• Less throughput unbalance among streams

4

Throughput of each stream (average of 10ms sections)
12 00

0.305

60

stream N
Sender cluster

0.30 430

2time

190time (sec)

0.303

400

• Throughput unbalance because of different loss timing
Synchronizing packet loss timing by shuffling them0time (se c)

stream 0
stream 1
stream 2
stream 3
total

600

200timestream 0
stream 1
stream 2
stream 3
total

800

0 .302

Receiverhost 3

10GbE

800

0
1 1

Twait
Twait
Twait

200

0. 301

Packetscheduler

0
RX

1000

0 .3

Receiverhost 2

1GbE

1000

Receiverhost 3TX

stream 0
stream 1
stream 2
stream 3

SwitchForce10
E600

1200

1200

Packet transmission timing at first RTT

delay emulator
150ms/direction

Senderhost 210GbE

60Receiverhost 0Receiverhost 1switchMG8Receiver
host 2

Experimental network to model LFN• 10 gigabit Ethernet (10GbE)• Delay emulator• 150*2 = 300ms round trip time (RTT)30

1GbE bottleneck

Senderhost 1Receiver
host 1Switch
Force10
E600

1GbE

0

Measurement
point

Senderhost 0flow(Mbits/sec)

Senderhost 2

0.01

Receiverhost 01GbE bottleneckdelay emulator
150ms/direction

0.02

flow(Mbits/sec)

switch
MG8

DATA
ACK

0.03

Packet transmission timing difference
between stream 0 and 3 in each RTT• Difference increases gradually• Increased packet gap by the 1GbE bottleneck• ACK timing preserved in the next RTT• Initial transmission order preservedwith increased inter-packet gap

Throughput unbalance among TCP streams

Senderhost 10.04

1cluster

Senderhost 00.05

0Long Fat-pipe Network (LFN)

Measurement
point

0.06

LAN
side
MAC

RX
buffer

10GbE MAC

Tx
Tx
Rx
USB
10GbE
10GbE
I/F
300pin MSA
300pin MSA
10GBASE-LR
10GBASE-LR
Port 0Port 1
Controller PC

enqueue

FIFO
statusStreamclassifier

Packetcounterdequeuescheduler

TX
buffer

WAN
sideMACFlow allocation

FPGA5

Performance Optimization of TCP/IP
Rapid expansion of Long Fat pipe Network (LFN) all over the world.
10Gbps network interfaces are available for commodity PCs with reasonable price.
Our objective is to fully utilize 10Gbps connection with TCP/IP communication.
External bus speed such as 10GbE becomes comparable with internal.
There exist many causes of performance decrease of LFN TCP communication.
Complex of these causes makes situation more complicated.

Mysterious phenomena
Early version of 10GbE NIC uses PCI-X internal bus, less than 8Gbps.
Version up of NIC with PCI-express (max speed is full10Gbs) causes terrible performancedecrease.

PCI-X NIC

Utilizing hardware support for TCP communication also brings strange phenomena, such as
slow-down of scaling speed.
PCI-express NIC

Possible Causes of Performane Decrease

Networknetwork

PackedBursty ACKs

Rx QueueTx QueueRx QueueTx QueueTx QueueRx QueueTx QueueRx Queue

SwitchOverflow

Bottleneck

Switch6Performance Optimization of TCP/IP
Clarification of Issue byPrecise analysis 1 ms scale

TAPEE• Analysis with fine time granularity– 100 nsthroughput

TGNLE-1[Sugawara ‘05]

1 sec scalethroughput

Optical tap

Programmed withour TAPEE design

USB

Packet lossoccurred

Control host

10GbE logging hostHDD

10GbE logging hostHDD• Compared and evaluated methods

Applicationwrite(2)– X Application level– X Limiting window size
– O Inter packet gap (IPG) control [Chelsio]• Raw behavior of packets on a network

TGNLE-1

Packet Pacing (1)• IPG control is essential for WAN PHY

TCP Stackcwnd
IF QueueBus– 700 octet NG, 720 octet OKNetwork I/F• Without pacing, TCP cannot achieve
congestion avoidance

Scheduling,IPG
Networkcwnd
TCP StackNetwork I/F
IPG length

Comparison of Pseudo andReal LFN using TAPEETransmission point

Virtual delay
by network simulator Anue

Pseudo-LFN
500ms

Interval between
short packets
are shorten

Receiver

Sender
TAPEE

TAPEE

After passed
pseudo-LFNRound the world circuit(Longer than 30000km)

Real-LFN
508ms

Receiver

Sender
TAPEE

TAPEE

Dynamic Adjusted IPG Control

Pseudo vs. Real

After passed
real-LFN

Successfully Probed the Bottleneck
Without Specifying Limit
Packet loss
occurred atthe very
speed ofbottleneck
1 sec scale
throughput
is suppressed

7Internet2 Land Speed Record•

99% of physical bandwidth for 5 hours on 522ms RTT network8

Thank you9High-speed TCP communication experiments
・ GoalEfficient TCP communication on Long Fat pipe Networks- Single and multiple stream TCP
Destination: Faculty of Science, the University of Tokyo- Adaptive inter-layer cooperation- Balancing parallel TCP streams- Austin(SC08) ⇔ the University of Tokyo

・ Network- WAN PHY 10 Gbps network- 9.2 Gbps maximum payload performance
・ System used
- Intel IA32 servers with Chelsio S310E network adaptor- MaSTER TCP stream stabilizer- TAPEE network instrumentation device

IEEAF
Univ. of Tokyo

ChicagoStarLight

Pacific
Wave

WIDETokyo

SeattleJGN2plus

L.A.

Network paths

NLR
AustinSC08

Network connection point
Source and destination

10

CosmoGrid
・ Using 30 processors on Cray XT4 of National Astronomical Observatory of Japan・ 30 processors of Huygens(IBM Power6） cluster of the University of Amsterdam.
・ Cosmological N-body calculation with 256^3 particles
・ The size of the simulation box is 60Mpc (megaparsec), with comoving coordinates and periodic boundary・ The specialized calculation code was developed to reduce the required communication bandwidth between two computers
and to allow for large communication latency.

Cray XT4 at Tokyo

Huygens (Power6) at SARA

Simulation Results (Test run)
National AstronomicalObservatory
SINET3Tokyo

JGN2plus

NSF
IRNC
ChicagoBlue part on Cray, Gray part on IBM
AmsterdamSARA

11

GRAPE-DR Processor Chip/System
GRAPE-DR Processor chip

GRAPE-DR supercomputing system(2008)

The University of Tokyo
National Astronomical Observatory of JapanGRAPE-DR compliersSpecification (2009)
Specification

Pipelined program execution

Technology:90nm CMOS
Number of PEs
512 PEs
Peak performance
512Gflops(single)
256Gflops(double)Size:
18mm X 18mm
Number of Tr
about 400M Tr
Clock Freq.
500Mz
Power Consumption
MAX60W
Idle time30W•

Pipeline of three execution phases

Phase 1

Phase 2

Data distribution to PEs

Computationin PEsInterconnect
OS
Phase 3
Memory on a hostCollection ofresults•2Pflops
4096 chips
512
400KW
40 Racks

Peak Performance:
Number of SING chips
Number of servers
Power Consumption
Size>10 Gbps
Linux 2.6.x

Optimizing compiler for GRAPE-DR––

Automatic parallelization by global analysis
Special purpose optimization for GDR
architecture○ Compiler Ver.1 flat-C compiler（2005)
・Parallel constructs, parallel statements
・Explicit description of parallelism○ Compiler Ver.2 Optimizing C compiler○ Currently, a prototype compiler is working
・ Generate native GRAPE-DR codes

GRAPE-DR
Dense Chip Multi-Processor

Application fields◎Highly efficient application fields for GRAPE-DR––––

Layout of a PE

ODP module

ModuleName

On chip shared memory Register

Color○Application fields with Effective acceleration by
GRAPE-DR

red

Flt. MulorangeFlt. Add

green

Int ALU

blueLocalMemo.

magentaothersyellow––

PLL & DFT module

Processor block(32 PEs）

N-body simulation in Astronomy,
Molecular dynamics（MD)、CFD（SPH method, Global model）etc.
Linpack, linear systems

Simulation in nano-technology
Simulation in bio-technology (FMO etc)

×Application fields with wide memory accesses––

Classical CFD, FFT
Application software optimized for vector
processors

GRAPE-DR system image>10 Gbps network cards
GRAPE-DR cards
PCserverInfiniband Switch

Floor plan of the chip

IP interface for
Parallel systems
GRAPE-DRcompiler

PC server
PC server

Total system
conductor

PC server

Distributed shared file system

IP storage switch

×Application fields with network bottlenecks–••

SIMD like architecture
512x64 bit arithmetic units + shared memory+ broadcast/reduction network–Elimination of inter-PE interconnection–Dedicated Reduction network (witharithmetic operations)

Our roadmap of supercomputer

Processing Element

FLOPS

•

512 PE in a chip27

shared memory

Control signals

Main memory on
A host server

External sharedmemory

KEISOKU
Computer
10+PFLOPS

To shared memory

EarthSimulator
40TFLOPS

1E
1P

From shared memory

Reduction

Top-end Parallel system

1Z

Shared memory Shared memory

On chip network

メモリ

Grape DR
2 PFLOPS

10LocalmemoryProcessingelement)(512)

Hierarchy of GRAP-DR30

101Y- Integer operation- Floating point
operation-Conditional
execution
byBroadcastingmask register
memoryaccesses

IP storage drives (iSCSI disk drives)

QCD

GRAPE-DR covers about half of importantscientific applications

Processor chip architecture

512 PEs/Chip

2K PEs / card

4 K PEs / server

512 GFlops /Chip

2TFLOPS/card

4 TFLOPS/server

Local memory
Registers
Mask register・・・Mask write operation of memory by
setting mask registers

PCI-express
16 laneI/O bus

1T1G

Processor chip
SIGMA-1

Core2Exreme
3GHz2M PEs / system

1M708090

2000 20102020 2030 2040 2050 Year

2PFLOPS/system

This research is partially supported by the Special Coordination Fund for Promoting Science andTechnology from Ministry of Education, Culture, Sports, Science and Technology, Japan.

12For more information
Visit
The University of Tokyo booth#3003

You’reHere13