Network research group topics
・Scalable, high-performance and reliable cluster network
・Low-power and high-performance network link

Taisuke Boku

Center for Computational Sciences, Univ. of Tsukuba

Scalable, high-performance and reliable clusternetwork
 Background

Most of inexpensive PC clusters still use Ethernet (mainly GbE)
for cost/effectiveness, but
Scalability is limited by its tree-configured network topology
 Performance is limited by relatively poor peak performance Reliability is limited by intermittent failure on switches (and cables)How to provide scalability, performance and reliability oncommodity Ethernet without much cost ?

Solutions
Multi-link aggregation both for performance and reliability inthe same manner with RAID in HDD
 Multi-path utilization to configure fat-tree topology underconditions of Ethernet
CCS ExtReview 2007
22007/10/30

Center for Computational Sciences, Univ. of Tsukuba

VFREC-Net: Scalable and high-performance network
Simple tree network topology largely
degrades the total bandwidth on
Ethernet with a large number of PC
cluster nodes
 SAN (e.g. Inifiniband) supports Fat
Tree to distribute the traffic load on
multiple switches on upper-level intree Fat Tree is not available with Ethernet
because only “tree configuration” isallowed, but no cyclic graph Solution: If we can make “logicallyisolated” network on each logical tree
⇒ VLAN technology

Loop

CCS ExtReview 20073

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

VLAN routing method (by Kudo@AIST)

Each VLAN configures isolated Tree



Loop

Each node belongs to one or more VLAN andLinux driver determines the VLAN-ID (tag)Eliminating the logical loop

Flat distribution of VLAN-ID distributes theload on upper-level
Possible to make Fat-Tree configuration

Original method by Kudo depends on Linuxvirtual NIC⇒ fixed and limited configuration, several
detailed technical difficulty

application

virtual virtual virtual
NIC 1 NIC 2 NIC 3
VID=1 VID=2 VID=3NIC
CCS ExtReview 20074

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

VFREC-Net
application

VLAN-based Flexible, Reliable,
Extendable Cluter Network
 Based on VLAN routing technology
 Developed new driver to handle
multiple VLAN-ID by software on
virtual NIC (VFN driver)Virtual NIC
VID=1 VID=2 VID=3

NIC

Table-based VLAN-ID decision
Can be referred as ordinary Ethernet
device to support any protocol andcommunication library withoutmodification

VBFT (VLAN-Based Fat Tree) issupported
CCS ExtReview 20075

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

NAS Parallel Benchmark with Xeon cluster with 32
nodes
Relative Performance (Flat=1)

Why not improved ?
1.2
1Flat
Tree (L=2)
VBFT (L=2, U=2)
VBFT (L=2, U=4)
Tree (L=4)
VBFT (L=4, U=2)
VBFT (L=4, U=4)0.8
0.60.40.20EPFTISKernels

MGCG

CCS ExtReview 2007
62007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Traffic distribution on upper-level switch
With imbalanced traffic

Ideal caseFlatly distributed on upper-level-> high performance

NPBCG has
this problem
VID=2
VID=3 VID=4

VID=1

Concentrated traffic on some switches-> low performance

VID=1

Congestion
Switchon route

Switch

VID=2

VID=3Switch

VID=4

Switch01234567012345674567012345670123CCS ExtReview 2007
72007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Traffic pattern in NPB-CG

Matrix transposing is dominantTraffic congestion by default (flat) VID
distribution

Congestion
通信組合せ

VID

1–4

2

2–8

36 －9

33 －124

7 – 13

411 – 14

40

4

8

12

1

12

3

4

Switch

VID=1

5

913

67

Switch

VID=2

VID=3

SwitchVID=4

Switch8910 11

261412 13 14 15
3711

Dynamic routing on VBFT
We modified the VBF driver (virtual NIC) for dynamic
modification of VLAN-ID assignment table⇒ It corresponds to “dynamic routing”
 User API is provided for modification of VLAN-ID table,
which is available even in MPI programming
 “How to optimize the routing according to the traffic” isstill open question, but we provide a generic platform foruser/system level dynamic routing Applying it, NPB-CG performance is improved

CCS ExtReview 2007
92007/10/30

Center for Computational Sciences, Univ. of Tsukuba

NPB Kernel CG result (with dynamic routing)
1.2
2x performance

相対性能1

Flat0.8

Tree0.6

VBFT0.4

Pre-optimized VBFT0.2

Dynamic VBFT0CCS ExtReview 2007102007/10/30

Center for Computational Sciences, Univ. of Tsukuba

VFREC-Net with dynamic routing
Enables very cost-effective high-performance networkwith any size of PC clusters
 Today’s most of inexpensive L2 Ethernet switches areequipped with IEEE802.1Q VLAN technology, so it can beapplied very widely Just adding a virtual NIC driver and several daemons Provides user-flexibility to tune the network routing
according to the applications behavior
 Optimization is still open question, but we provide theway to apply it

CCS ExtReview 200711

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

RI2N: Reliable & high-performance network
RI2N: Redundant Interconnection with Inexpensive
Network⇔ RAID: Redundant Array of Inexpensive Disks
 Multiple (parallel) network links are used for

Multiplying the bandwidth (without failure) Fault recovery network with live links (on link or switch failure)The network must be user-transparent, so we
implemented it as a pseudo network driver onGbEhernet

CCS ExtReview 200712

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Data sending
ApplicationData transmission
Pseudo device is added
 Device transmit function iscalled according to IP space Rewrite the protocol ID forcorrect receiving on receivenode Select real-NIC in round-robinmanner, then push the packet
TCP
IP
Pseudo dev.（RI2N/DRV）Real NIC

Real NIC

Real NIC

Real NIC
・・・

Recv. Queue in kernel
CCS ExtReview 200713

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Data receiving
・・・

RI2N/DRV
Real NIC

Data receiving
 Add the protocol handler

Real NIC

Handling pseudo protocol (0x100)

Rewrite the protocol ID andmerge the split packets in
multiple channels
 Re-push the packet in to kernel
Real NIC

Real NIC

Recv. queue in kernel
0x1000（RI2N/DRV）Real protocol handling is
performed on higher layer

IP
TCP
Application

CCS ExtReview 200714

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Throughput on failure

Restartingwith 1 link
Restartingwith 2 links

Failure!

TCP/IP communication
CCS ExtReview 200715

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Comparison with Linux “channel bonding”
TCP/IP ping-pong with various message size
250
singlebonding
RI2N

throughput [MB/s]

200

150

100

500
110100

1K 10K 100K
message size [byte]

1M

10M100M

Low-power & high-performance link
Study just started in this year Preliminary study on next generation MPP systemCPU power consumption has been reduced while the network
power consumption is increasing For the future systems with millions of processors, “global”
network is impossible⇒ “local” network (something like “mesh”)
 Network link power is mainly used for driving for long distancecommunication (e.g. 100m for Ethernet)⇒ reducing the communication range much contributes tosave the power for network link “DVFS” + “Parallelization” ⇒ new generation network link Reliability is also required for large scale network

CCS ExtReview 200717

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Reducing the network power on node
Short-range & high-frequency network link
 Link-aggregation with relatively low performance (lowpower) rather than a single high performance link

Reducing power Redundancy for fault tolerant on link disconnection or switchfailureDynamic frequency changing according to theapplication requirement
Low data rate is enough for CPU-bound applications
 Real-time application requires “slow” mode or “rush” mode Dynamic controlling is necessary for total power reductionCCS ExtReview 200718

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Our solutionUsing PCI-Express Gen2 network as the platform

Development of new network link device and controlling chip






Basic serial link can be used not just for I/O device connection but also fornode-to-node connection (interconnect)
2.5Gbps/5.0Gbps mode change by device control
Multiple lanes for various aggregated data rate
Covering all features of PCI-E gen2
Dynamic lane# controlling to increase/decrease data rate
Link malfunction is automatically covered by othre links
Routing function to be an interconnection router on parallel processing
systemWithout big PHY for long distance communication

Prototype development with Renesas Technology
CCS ExtReview 2007
192007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Block diagram of SiP to develop
4
PCIE2Communicator(新規作成)

PCI-bus

CPUSH-4A

PCI-E link(4-lane x 4)
Memory-bus

FLASHMemory (?MB)

DDR-SDRAM
128MB

CCS ExtReview 200720

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

“Communicator” Block Diagram
Implement as “Communicator” Chip
PCI-E
Gen2 Link(2.5～5.0Gbps/link)

PCI Interface
Data Trans. to main CPU (SH-4)

Routing
PHY

Buffer+ Controller

PHY

Buffer+ Controller

PHY
PHY

Command/Response
CPU (controller)
M32R (?)

Buffer Comm./Res.
+ Controller
Buffer+ Controller

Data Trans. Comm.
ROM
or FLASH
SDRAM

DMA
Controller
CCS ExtReview 200721

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Status
In JST/CREST project, the platform is developed as aprototype of high-performance embedded sysem
 FPGA version of logic will be delivered on March 2008
 With FPGA board which corresponds to a single channel(4 lanes), system software on controlling processor(M32R) will be developed Communicator chip with 4-lanes x 4 channels will beshipped on March 2009
 SiP module with a compact module for embedded HPC
will be available on March 2010 ?
CCS ExtReview 200722

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba