HPCI System Development Team
Software Development TeamHPCI System Development TeamHPCI : High Performance Computing InfrastructureOne-Stop service for naƟonwide super computer resourcesSupport from project proposal to user report.
The K computer and other major super computers in japan,
large scale MPP, GPU Cluster, PC Cluster, and Vector machines
are available.
The HPCI helpdesk is available for HPCI users, and provides
consulta on before applying project proposal, technicalsupport for each computer resources, and guidance of user
report a er finishing the project.
HPCI Shared Storage
Provide 22PBytes single view file space for all the HPCIusers.
Available from all the HPCI super computer resources.
HPCI single sign on authen ca on, if a user wants to transfer
the computa onal results to the HPCI shared storage a er thecomputa onal session, the user does not have to log in again
to the HPCI shared storage.
Automa c file replica on, HPCI shared storage adopts 2 file
replicas in default. Secure network communica on, HPCI user
can specify data encryp on method such as “gsi” in~/.gfarm2fs.

Hokkaido University

University of TsTsukubaTukuba
The Institute ofStatistical Mathematics

TohokuTo
Thoku University

Kyoto University
Osaka University

JAMSTEC

Tokyo
University off T
okyo
HPCI EAS
ST HUB
EASTHUB

Nagoya University
Riken AICSTokyoTokyo Institute of
HPCI WEST HUB
KyushuKyush
hu University
TechnologyT
Technology

 BBBBBBBBBBBBBBBBBSingle sign On AuthenƟcaƟon
Allow eﬃcient use of na on wide super computer resources.
Users of the HPCI can u lize all the granted HPC resources seamlessly a er signing on one of the resources.

Software Development Team
Collaborate between the system and the application to improve system and usability

16.0
[MW]
14.0

Limit ofPower
Consump on

Received power
CGS1
CGS2

12.0
10.0

HPCG Performance Tuning on the K computer

Recently jobs with large power consumption haveincreased and total power consumption exceeded thepower consumption limit regulated by contract withprovider.
If the total power consumption exceed the power limit,
we have to pay additional charge as a penalty.

8.0

Evaluate Original HPCG on the K
5.23%
ComputeSPMV_ref

1,00010010186.0
Power[MW/sys]

6.0

The original HPCG code evaluation on the Kcomputer gave these informations.

10,000GFLOPS (log)

Improvement of Operation Eﬃciency and Eﬀort forPower Consumption Reduction on the K computer

64512 4096 32768
Processes (log)

Weak-scagaling performance

5.0

Linear scalability was obtained, soparallelization tuning is not ncessary.
Single CPU performance was low
since SYMGS is not multi-thread.

93.09%
ComputeSYMGS_ref(No multi-thread)

Therefore we have aimed the. single CPU.

Cost components in CG

4.0

HPCG start

Tune: Coloring for SYMGS

4.00.0In the original code, SYMGS is notable to be multi-thread since there
are data dependencies between rows.

3.0

2.0

To avoid data dependencies, we emp lo y ed t h e c o lo red b lo c kin g t h a t
divide the mesh into some blocks anddo the coloring to blocks.

Day/Time

Analysis of correlation with power consumption
and application performance Standby power and memory throughput has agreater impact on the power consumption ofthe K computer.

1.0

0.0

Memory Throuput

L2 Throuput

L1D Throuput

Integer Calc.

Power Mesurement

Simple kernel loops

PowerPower

n/n=1

R.1

CPUcoreR.4

R.3

NodeInput meshfor a process

R.2

R.3

・ ・R.4・ ・CPUcore

CPU
core

・ ・ ・ ・R.8

Divide intosome blocks
and coloring

R.5

Each block has
almost samenodes

CPU
core

Memory serialization for matrix
Data access ordering improvement for SYMGS
Loop optimization for SPMV, SYMGS

Node × Time and Power Saving RaƟo800

16.0%

Speed UpX19400R.5200
Original

We obtained 19 times improvement.

18.0%

600

0Reﬁnement miscellaneous routinesRaƟo [%]
20.0%

 The performance of each big userʼs applications has
estimated how much diﬀers compared to the ﬁeld average.
 If both the CPU performance and the memory performance
improved to ﬁeld average, NODE×TIME can be reduced asbelow.
 When we improve performance about the 10(50)major user application, we can use more 25M(66M)
NODExTIME.
 The power consumption is When we improve performance about 10(50) majoruser application, we can achieve energy savings of
2.7GWh(7.0GWh).

There are no data dependencies between samecolored blocks, so the blocks are able to beprocessed by multi-threading.

We tried these additional ways for improving.

Parameter adjustment

Time

Nodes in same colored blocks are reordered tobe continuous in memory space.

Signiﬁcant Improving Obtained

A trial of improving of power consumption andsystem operation
 A evaluation model of power consumptionof each performance tuned applications.

1/n
TimeR.2

Floang 26
Calc．
Integer
1 2B/F
3 Change
4 5 6 7 8 9Throuput
10 11 12Change
13 14 15 16 STREAM
17 18 19DGEMM
20 21 22 23
24 25
27 28 29 30 31
32 33 Calc．
34 35 36 37

Floa ng Calc.

δPower[MW/sys] = 8.8128  1.3659 × Floating Calc.[%]+ 4.3906  Memory Throughput[%] + 0.0857 × L2 Throughput[%]+ 2.3299 × L1D Throughput[%] + 0.2429 × Integer Calc.[%]×nR.1

Time [s]2.0TunedComparing CG run time for 5 CG calls

Result for SC15

14.0%
12.0%
10.0%
8.0%
6.0%

Node × Time

4.0%

Power saving

2.0%
0.0%
0204060
The number of applicaƟon

80

100

Rank

Computer

12345

Tianhe-2
K computer
Titan
TrinityMira

HPL
PFLOPS
33.86
10.51
17.59
8.10
8.59

HPCG
PFLOPS
0.580
0.461
0.322
0.1830.167

Ratio to
HPL %
1.7%
4.4%
1.8%
2.3%
1.9%

http://www.hpcg-benchmark.org/custom/index.html?lid=155&slid=282

Using V2.4, We obtained the HPCG score 0.461
PFLOPS using 82944 processors.This score wasranked 2nd on the list at SC15.
Under the measurement rule of
the V3.0, the performance ofcomputational kernel is same toV2.4, but the HPCG score will bedecreased.