Briefing:High Performance Computing System Division
Taisuke Boku, Division Leader
Center for Computational Sciences /Graduate School of Systems and Information Science

Center for Computational Sciences, Univ. of Tsukuba

Organization
High Performance Computing System Division

System Architecture Research Group
Taisuke Boku (Professor)
 Daisuke Takahashi (Associate Professor)
Grid Computing Research Group
Mitsuhisa Sato (Professor)
 Osamu Tatebe (Associate Professor)


PD




Toshihiro Hanawa

Since system architecture and Grid computing related very
closely, all faculty members are working in borderless manner

CCS ExtReview 200722007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Research field

HPC system research & development


HPC system architecture
HPC processor architecture including memory hierarchy
 Scalable & high-performance interconnection network Total system design and solution

System software
Compiler HPC math. Library Network drivers for PC cluster
 Model and language for HPC
Grid computing
Grid RPC system
 Distributed file system for Data GridCCS ExtReview 20073

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Collaboration with application fields
Advising application people from system viewpointLarge scale parallelization (MPI programming, performancebottleneck checking, …)
 Performance tuning for processor architecture (FPacceleration, memory hierarchy, …)
 Numerical solutions (development of math. library) System design (file system, new project support, …)
System operation support
PACS-CS FIRSTCCS ExtReview 20074

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Education
4 professors share a collaborative laboratory named“High Performance Computing System Lab.” inDepartment of Computer Science, Graduate School ofSystems and Information Engineering StudentsDoctoral Course: 5 Master Course: 7 Undergraduate: 4


CCS ExtReview 20075

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Research topics in HPCS Division


HPC System ArchitectureSoftware

HPC processor architecture and memory system: SCIMA
Large scale parallel processing network: HXB
Power-aware computing: Optimized DVFS on parallel processing
Large scale cluster computing: VFREC-Net, RI2N
OpenMP compiler: Omni OpenMP
New language model for large scale parallel processing: OpenMPD
High performance and scalable math. Library: FFT, orthogonalization

HPC Grid

Grid RPC: OmniRPC
Data Grid on distributed file system: Gfarm
Grid interoperability
CCS ExtReview 2007

62007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: PACS-CS development
Total system design
 3D-HXB/Ethernet network driverdevelopment
 Special communication library for
low-level high-performancecommunication
 After installation, mainly supporting
the performance tuning, parallelcode development and code porting




・・・
・・・・・・・・・・・・・・・・・・・・・・・・・・・

Z=10

Y=16・・・・・・
X=16

Lattice QCD
RS-DFT
Tree-Puzzle
WRF
CCS ExtReview 20077

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: FIRST development
Basic concept design ofHMCS (HeterogeneousMulti-Computer System) Conceptual design ofBlade-GRAPE solution
 Cluster configuration
 Gfarm installation andoperation for shared filesystem

CCS ExtReview 20078

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: Power-aware parallel processing
MegaProto (JST/CREST, finished on Oct. 2006)
Prototype R&D for very small and low-power CPU module withembedded technology
 Using Efficion processor (Transmeta, 3W TDP) 16 of CPU modules + 1 controlling processor module On-board dual-link Gigabit Ethernet + 24 port switch x 2(with 16 ports of up-link) MegaProto/E prototype with 16 computation nodes and duallink GbE network in 1U form factor⇒ 32 GFLOPS peak performance (2.7 times faster than dualXeon SMP solution) with same power consumption
CCS ExtReview 2007
92007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: Power-aware computing (cont’d)

130mm

65mm

CPU module of MegaProto/E

Cluster Unit of MegaProto/E with 16 CPUs

CCS ExtReview 2007
102007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Power-aware computing (cont’d)Power optimization on parallel program

DVFS: Dynamic Voltage and Frequency Scaling
Changing CPU “gear” for optimal one utilizing “slack-time” of parallel
execution with imbalanced parallel application

Program executed at standard gear

Power optimized program

CCS ExtReview 200711

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: HPC math. libraries


FFT-E (FFT East)
Included in HPCC benchmark suite Multi-platform highly optimized FFT library Well-tuned with cache awareness for large scale PC cluster
 Combining multicolumn FFT and data transposing to reduce
cache miss-hit on each node Better performance than well-known FFT-W library

CCS ExtReview 2007
122007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Performance of parallel 3-D FFTs
（dual Xeon PC SMP cluster, N1xN2xN3 = 2^24xP)

5000
MFLOPS

4000
3000
2000
1000
0124816
Number of Nodes

FFTE (SSE2,1CPU)FFTE (SSE2,2CPUs)
FFTE (x87,1CPU)FFTE (x87,2CPUs)
FFTW(1CPU)FFTW(2CPUs)

CCS ExtReview 2007
132007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: math. library (cont’d)Cache-aware orthogonalization library
Especially required for RS-DFT (Real Space Density FunctionTheory) on large number of atoms
 On classical Gram-Shcmidt Orthogonalization does not work
well on cache architecture Modifying the algorithm to fit Level-2 or Level-3 BLASCCS ExtReview 200714

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Performance on Xeon 3GHz (1CPU)
7

NaiveColumn-wise CGS

Recursive CGS

6G FL O P S

5432
100

2000

4000Matrix Size

6000

8000

CCS ExtReview 200715

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Performance on 32 node 3GHz Xeon PC Cluster
140

Naive
Column-wise CGS
Recursive CGS

120

GFLO PS

100
8060
40
200010000

20000
Matrix Size

30000

40000

CCS ExtReview 200716

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: Gfarm distributed file system for Data Grid
Commodity-based distributed file system that federates storageof each site
 It can be mounted from all cluster nodes and clients
 It provides scalable I/O performance wrt the number of parallel
processes and users
 It supports fault tolerance and avoids access concentration byautomatic replica selection


Globalnamespace /gfarm
ggfaist
file1

jp
gtrc
file2

file1
file3

Gfarm File Systemmapping
file2

file4

File replica creation

Topic: Gfarm (cont’d)Files can be shared among all nodes and clients
 Physically, it may be replicated and stored on any file systemnode Applications can access it regardless of its location
 File system nodes can be distributed


GridFTP, samba, NFS GridFTP, samba, NFS server
servermetadata
Compute & fs node Gfarm metadata server
Compute & fs nodeCompute & fs nodeFile A

Compute & fs nodeCompute & fs nodeFile File
B C

Compute & fs nodeCompute & fs nodeFile A

Compute & fs nodeCompute & fs nodeCompute & fs nodeCompute & fs node

USFile B

Japan

File C/gfarm

ClientPC

File A
File C
File B
Gfarm
file system

NotePC…

Gfarm for particle physics data analysis
O. Tatebe et al, “High Performance Data Analysis for Particle
Physics using the Gfarm File System”, SC06 HPC Storage
Challenge, Winner – Large Systems, 2006
• Construct 26 TB of Gfarm FS
using 1112 nodes• Store all 24.6 TB of Belleexperiment data
• 52.0GB/s in parallel read
→ 3,024 times speedup• 24.0GB/s in skimming
process for b → s γ decays
using 704 nodes→ 3 weeks to 30 minutes

I/O bandwidth [GB/sec]

52.0 GB/sec

60
50
40
30
20
1000200

400
600
800
number of client nodes

1000

12001112nodes

CCS ExtReview 200719

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Topic: HPC interconnection for PC cluster
High-performance, scalable and reliable commodity
network for inexpensive PC clusters: VFREC-Net & RI2N
Utilizing multi-link Gigabit Ethernet
 High-performance: aggregated bandwidth of multiple links
 Scalability: VLAN-based Fat-Tree is enable even with
inexpensive Layer-2 Ethernet switch
 Reliability: Multiple links work as redundant connection on thefailure of switches and links
 Everything is implemented as a special network driver

CCS ExtReview 200720

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Relative Performance (Flat=1)

NAS Parallel Benchmark with Xeon cluster with 32nodes
1.21

Flat
Tree (L=2)
VBFT (L=2, U=2)
VBFT (L=2, U=4)
Tree (L=4)
VBFT (L=4, U=2)
VBFT (L=4, U=4)

0.8
0.6
0.4
0.20EPFTISKernels

MGCG

CCS ExtReview 200721

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Throughput on failure
All links normal

Single link failure

All links normal

Restartingwith 1 link
Restartingwith 2 links

Failure!

TCP/IP communication
CCS ExtReview 200722

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Comparison with Linux “channel bonding”
TCP/IP ping-pong with various message size250singlebonding
RI2N

throughput [MB/s]

200

150

100

500
110100

1K 10K 100K
message size [byte]

1M

10M

100M

Major Research Collaboration
Large scale cluster computing and Grid

Low-power & High-performance processor architecture




RIKEN

HPC Grid computing

U. Tokyo

Next Generation Supercomputer System

T2K Alliance (U. Tokyo & Kyoto U.)
AIST

INRIA
PRAGMA (Asia-Pacific Grid middleware community)

HPC Cluster network system


AISTNIICCS ExtReview 200724

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba

Activities outside CCSContribution to RIKEN’s Next Generation Supercomputer
Development ProjectSato and Boku are the members of system architecture
working group as visiting researchers of RIKEN All members will contribute the performance tuning andevaluation on large scale QCD, RS-DFT and FFT under researchcontract with RIKENSocial works
All members have been playing important roles on HPC society
such as symposium and workshop organization, PC chairs, etc.
 IPSJ SIGHPC Chairs: Sato (1998-2001) & Boku (2006-)
 Recognized as very active research group in HPC community inJapan, Asia and the world
CCS ExtReview 200725

2007/10/31

Center for Computational Sciences, Univ. of Tsukuba