PACS-CS
A massively parallel cluster
for computational sciences
Taisuke Boku
Center for Computational Sciences /Graduate School of Systems and Information Engineering
University of Tsukuba

Center for Computational Sciences, Univ. of Tsukuba

PACS-CS Project Overview
Purpose
Creating a new CCS computational facility with tens of TFLOPSand Performing various target applications with large scaleparallelism
 Not just procuring a machine but developing unique one for
CCS application requirementsPeriod

Apr. 2005 – Mar. 2008 (FY 2005-2007)

TeamCollaboration with computer scientists and computationalscientists in CCS
 Collaboration with university and vendor: U. Tsukuba, Hitachi
and FujitsuCCS, ExtReview
2

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

HPC resources required at CCS
Target applications and computation characteristics
full QCD （small size of complex matrix calculation +nearest neighboring communication) Nano material science （CG method）
 Astrophysics （hybrid computing for particles & field）
 Environment, Biology （parameter search）


Shift from MPP
Previous machine: CP-PACS with 2048 CPU & 614 Gflops
 We need replacement of MPP with Commodity Technology
 Main target applications require the bandwidth both onmemory and network
CCS, ExtReview
3

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Previous main resource: CP-PACS#1 in TOP500 on Nov.’96
Peak perf. 614 GFLOPS
Linpack: 368 GFLOPS
Dropped off from TOP500
list on Nov. ‘03

Operation stopped on Sep. ‘05

CCS, ExtReview
4

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Concept of PACS-CS
Making an MPP-like system based on commodity
technology
 Cluster is OK, but we keep the balance among
CPU ： memory : network performances
 To reduce the cost, considering our target
applications
PACS-CS
Parallel Array Computer System for Computational
Sciences
CCS, ExtReview

5

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Real Space Approach in Computational ScienceReal Space modeling
MPP code based on real-spacedescritization⇒ Large scale parallelism with
NNM comm. + collective comm.
 Reducing Comp. Time (not reducingComp. Amount)Traditional methods：FFT, spectrum, …
⇒Indirect computation, not enlarging modelspace Real Space method：⇒Simulating real model on real spaceDirect mapping fromreal-spacedescritization toprocess space

We need re-program the traditional
codes
CCS, ExtReview

62007/10/30

Center for Computational Sciences, Univ. of Tsukuba

General HPC clusters (as in yr. 2004)
Intel-compatible CPU (Xeon, Opteron, Itanium2, ...)
 Dual CPU SMP






So much CPU frequencyTo reduce the space and the number of network interface keeping total
system peak performance
Rack of memory bandwidth (memory wall problem)(but very fast for Linpack !)Low sustained performance on network bound applications
For very high peak performance

SAN (System Area Network)

MyrinetXP: dual connection for 500MB/s -> 10GbpsInfiniband: x4 spec. for 1GB/s
Gb Ethernet is still OK for non-network bound applications(10GbE will come soon, but still expensive)
CCS, ExtReview

7

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

CPU / Memory performance balanceSingle CPU / node (not SMP)
 Memory wall problem ⇒ appropriate CPU speed(not too high frequency)
No chasing for the peak speed and frequency to CPU
 We don’t make “Linpack-aware machine”
High density implementation
Same as traditional HPC clusters in SMP: 2CPU/1U
 Keeping the same total memory size per chassis (1U)CCS, ExtReview
8

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Balance between node and network performance

Our target application’s parallelizing models
Nearest Neighboring Communication in n-dimension
 Collective Communication (Broadcast/Reduction)
 Not much random trafficUsing commodity technology (= Ethernet)
Trunked links make wider bandwidth Relatively large message size (non latency-sensitive) Multi-dimensional implementation for much morebandwidth No I/O-bus (PCI) bottleneckHyper-Crossbar Network with trunked GbECCS, ExtReview
92007/10/30

Center for Computational Sciences, Univ. of Tsukuba

3-dimensional Hyper-Crossbar (3D-HXB) Network・・・・・・

Nz・・・・・・

Ny・・・・・・・・・・・・
Nx・・・・・・・・・・・・

X-dim switch
Y-dim switch
Z-dim switch
Computation node (CPU)Route through a single
dimension of switch
Route through multiple
dimensions of switch(intermediate CPU isrequired)¾Dynamic network used in CP-PACS¾Small size of crossbar switches¾Small degree (# of links)¾High random transfer rate
（30% of peak)HXB based on trunked GbE

Effective use of low price commodity technology
NIC chip Switch （L2, 10～20 ports）


Network bandwidth
A few links are trunked ⇒ Software solution is enough Simultaneous transfer on multiple dimensions
bandwidth ＝ （link bandwidth）×(# of trunks) × (# of dim.) Distributing I/O load from single I/O point (ex. PCI) to multipleones

Routing is required for transfer on multiple dimensions
3-D configuration is enough for up to 4096 nodes (16 CPUs /dimension) Software trunking + routing

CCS, ExtReview
11

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Logical connection among nodes (CPUs)(3-D HXB network) 2560 nodes

・・・

・・・

Z=10・・・・・・

Y=16・・・・・・・・・

・・・

Computation node (CPU)・・・・・・

X-dim switch
Y-dim switch
Z-dim switch

・・・
・・・
X=16

in the figure

real connection(dual link trunking)Special network driver for GbE-trunk-HXBPM-Ethernet/HXB enables
Direct inter-node communication on single dimension
 Multiple GbE links are trunked to multiply bandwidth Up to 3-D simultaneous sending/receiving


250 MByte/sec (dual-link GbE) x 3 = 750 MByte/sec

Routing for a message requiring 1 or 2 hops oftransfer on intermediate nodes
 Fault tolerant operation for single link failure (futureplan) For more information ⇒ another paper on ICS’06
CCS, ExtReview
13

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Mother board & Chasis
SATA
RAID-1CPU
LV-Xeon
2.8GHz160GB
HDD x 2

Front
HDD#1

South Bridge

PXH

memory

HDD#3

HDD#2

HDD#4

North Bridge

PXH

133MHz/64bit
PCI-X x 4
Gigabit
Ethernet× 8 ports

Power Unit

GbE GbE GbE GbE

MotherBoard#1

MotherBoard#2

RearOperation
Managementx0 x1 y0 y1 z0 z1
NetworkNetwork
Data Network

2 mother boards in 1-U chassis

CCS, ExtReview
14

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Specially designed motherboard
CPUChipset

PCI bridge

GbE NICs
VGA
USBx2

Memory8 portsof RJ-45

CCS, ExtReview
15

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Unit chassis (19inch x 1U)

Motherboard
HDD x 2
Power Unit

Motherboard

HDD x 2

CCS, ExtReview
16

2007/10/30

Center for Computational Sciences, Univ. of TsukubaSummary of PACS-CS spec.
# of nodes

2560 (16 x 16 x 10)

peak performance

14.3 Tflops

node configuration

single CPU / node

CPUIntel LV Xeon EM64T, 2.8GHz, 1MB L2 cachememory2GB/node (5.12 TB/system), DDR2 interleaved

network for parallel
processing

3-dimensional Hyper-Crossbar Network

link bandwidth

one-sided: 250MB/s/dim.
one-sided: 750MB/s (3-D simultaneous trans.)

local HDD

160 GB/node（RAID-1）

total system size

59 rack

power consumption

550 kW

CCS, ExtReview
172007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Real machine (just finished to construct!)

Node Rack

Switch Rack

CCS, ExtReview
18

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Real machine (cont’d)

CCS, ExtReview
19

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Full system of PACS-CS (2560 nodes)

CCS, ExtReview
20

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Whole system image and rack floor plan

Whole system image

Node racks and network
swtich racks are separated

Node rack

Operation switch rack

X & Y dim. switch rack

File server & RAID

Z dim. switch rackCCS, ExtReview
21

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

4 categories of interconnection network
Parallel Processing Network (for Data)

General Purpose Network

Generic tree network with link aggregation (LACP)
Generic UNIX network processing : NFS, NIS, DNS, rsh, ...

Operation & Maintenance Network

3-D HXB network based on dual-GbE trunking
For high speed parallel processing on applications

Tree network
Remote operation (power on/off, reboot, individual/broadcast console access) toeach/all nodeSurveillance Network

Watching a large number of (about 380) switches
All switches are managed and monitored by SNMP

CCS, ExtReview
22

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

General Purpose Network・Basic fabric: 48 port GbE switch (with LACP)・For a single file-server （multiplied by the number of file servers）

External Connection
External Connection

fibrechannel

Large Capacity
RAID

Management Nodefile-server 1
2 linkOther fileservers

single switch for Stage-3
82 link trunk

240

240

40

four switches for Stage-2

2 link trunk……

22 link trunk

2 link trunk

240

2……

………

40

……

64 switches for Stage-1 （40 CPU×64=2560 CPU）

Access to the file server“All nodes access the file server simultaneously” is impossible
 Each node is equipped with a local HDD (160GB) - data have to becopiedfrom file server before calculation
 Result file are transferred to the file server after computation
 Well-scheduled file transfer to keep a reasonable bandwidth Several special nodes can NFS mount to the file server⇒ Dynamic parameter changing
Computation...

Parameter for Exec.

...

Execution binary, initial data NFS mount for master node only
(direct file access via NFS）
rcp & scrun（well-scheduled and serializedoperation）

Data exchange by MPI...

Operation & Maintenance Network・Basic fabric: 48 ports GbE switch

Remote console feature by
serial over LAN on IPMI
2.0

External network

management
servera switch for Stage-3
4
1

1

40
1

1

40

40

……

40

two switches for Stage-2
1

……

………

40

……

64 switches for Stage-1 （40 CPU×64=2560 CPU）
CCS, ExtReview
25

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

SoftwareLinux + SCore
SCore: Cluster middleware to support partitioning and job
management for thousands of nodes PM/Ethernet-HXB driver Checkpoint/Restart

Scalable health-care monitor with GUI
 Batch/Queue （OpenPBS）
 Parallel programming in MPI Languages: Fortran90, C, C++ Math Libraries
CCS, ExtReview
26

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

P2P performance on 1-dim communication
Ping-pong performance in MPICH (single dimension)
250

Bandwidth (MByte/s)Without routing
1-dim max.223MB/s (89%)
N1/2=60KB

200

2-dim
3-dim

150

Perf. down by routing
For 128KB message
2-dim=20%
3-dim=38%

100

Perf. down by routing
For 1MB messages
2-dim=6%3-dim=9%50
0512

2K8K32K

128K

512K

Message Size (Byte)
CCS, ExtReview
27

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Aggregated bisection bandwidth
Bisection bandwidth in MPICH (Simultaneous ping-pong on a single dimension)
8x8x8=512 node, 1MB message on 8x8x8 system)
60

Total bandwitdh [GB/s]

50

Ideal full-bisection
bandwidth by 3-D HXB

4030
20100050100150200

250

300

# of node pairs
CCS, ExtReview
28

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Simultaneous communication


Simultaneous 3-D comm. (burst data transfer of long message)
Bandwidth/node [MB/s] (% to peak)

256 node

512 node

average586.8 (78.2%)

582.0 (77.6%)

max.

619.3 (82.6%)

629.6 (84.0%)

min.

559.2 (74.6%)

434.0 (57.9%)

Almost 80% of performance is achieved on simultaneous 3-D comm.


Simultaneous 1-D. comm. with routing (diagonal comm., burst transfer)
Bandwidth/node [MB/s] (% to peak)

256 node

512 node

average

186.2 (74.5%)

184.1 (73.6%)

max.

123.5 (49.4%)

232.5 (93.0%)

min.

122.7 (49.1%)

230.9 (92.4%)

Almost 70% of performance is achieved with routing function
CCS, ExtReview
29

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Linpack performance
10.35 TFLOPS with 2560 nodes(#34 at 2006/Jun. TOP500 list, #2 as Made-in-Japan machine) Performance differs by 2-D array configuration on HPLP×Q

NRmax (Tflops)

Efficiency (%)

3D-HXB routing

16×160706560

10.33

72.05

No

32×80

722944

10.35

72.20

Yes

6.7 hours of running time No error on 10 times of running We performed node aging test, complete check of 5000 of HDD,
and system level check before full installation
CCS, ExtReview
30

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Installation
System design and implementation
 Hardware + basic software : Hitachi
 Network software: Fujitsu
 “Multi-Vendor” solution is available for such a commodity-basedwork
 System operation started from July 2006
 Mostly used with 256 or 512 nodes of groups(1.4 – 2.8 TFLOPS/partition)
 Performance tuning for 3-D simultaneous communication is still
under going ⇒new PMvX API & library PM/Ethernet-HXB on SCore middleware was released under GPL


CCS, ExtReview
31

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Partitions

Mixing and scheduling them according to user’s request and applications
CCS, ExtReview
32

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Operation
Currently under “Interdisciplinary Computational Science
Promotion Program” with 13 of research groups
 Most of jobs are running with 256 or 512 node group,
some with 128 node group
 Almost 100% of resource is always used (except regular& irregular maintenance) Very stable with a couple of troubles on CPU,
motherboard under regular maintenance (every month) A couple of local HDD failure per month, but covered by
RAID-1 system and no effect on application runningCCS, ExtReview
33

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Applications running now
Particle Physics: Lattice QCD
 Material Science:

RS-DFT (Real Space Density Function Theory) QM/MM, MD (AMBER), CPMD
Nuclear Physics & Material Science: RT-DFT (Real-TimeDFT)
 Geoenvironment: NICAM & WRF (climate simulation)
 Biology: Tree-Puzzle ….
(details will be presented in each division/group report)
CCS, ExtReview
34

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba

Conclusions

PACS-CS is an MPP based on commodity technology
Balance among CPU : Memory : Network performancesis the essential
High cost-effectiveness network suitable for a certainclass of applications : GbE trunked 3D-HXB
Same density with traditional dual CPU SMP node2560 CPU, 14.3 TFLOPS system is running since July
2006

CCS, ExtReview
35

2007/10/30

Center for Computational Sciences, Univ. of Tsukuba