hpcg-benchmark.org

HPCG UPDATE: ISC’16
Jack Dongarra
Michael Heroux
Piotr Luszczek

1

hpcg-benchmark.org

2

HPCG Snapshot• High Performance Conjugate Gradients (HPCG).
• Solves Ax=b, A large, sparse, b known, x computed.
• An optimized implementation of PCG contains essential computationaland communication patterns that are prevalent in a variety of methods
for discretization and numerical solution of PDEs

• Patterns:• Dense and sparse computations.
• Dense and sparse collectives.
• Multi-scale execution of kernels via MG (truncated) V cycle.
• Data-driven parallelism (unstructured sparse triangular solves).
• Strong verification (via spectral properties of PCG).

hpcg-benchmark.org

Model Problem Description• Synthetic discretized 3D PDE (FEM, FVM, FDM).
• Zero Dirichlet BCs, Synthetic RHS s.t. solution = 1.
• Local domain:• Process layout:• Global domain:(nx × ny × nz )(npx × npy × npz )(nx * npx ) × (ny * npy ) × (nz * npz )• Sparse matrix:• 27 nonzeros/row interior.
• 8 – 18 on boundary.
• Symmetric positive definite.

hpcg-benchmark.org

Merits of HPCG

• Includes major communication/computational patterns.
• Represents a minimal collection of the major patterns.
• Rewards investment in:• High-performance collective ops.
• Local memory system performance.
• Low latency cooperative threading.
• Detects/measures variances from bitwise reproducibility.
• Executes kernels at several (tunable) granularities:• nx = ny = nz = 104 gives• nlocal = 1,124,864; 140,608; 17,576; 2,197• ComputeSymGS with multicoloring adds one more level:• 8 colors.
• Average size of color = 275.
• Size ratio (largest:smallest): 4096
• Provide a “natural” incentive to run a big problem.
44hpcg-benchmark.org

5

HPL vs. HPCG: Bookends• Some see HPL and HPCG as “bookends” of a spectrum.
• Applications teams know where their codes lie on the spectrum.
• Can gauge performance on a system using both HPL and HPCG
numbers.

hpcg-benchmark.org

6

HPCG Statushpcg-benchmark.org

7

HPCG 3.0 Release, Nov 11, 2015
• Available on GitHub.com• Using GitHub issues, pull requests, Wiki.
• Optimized 3.0 version:• Vendor or site developed.
• Used for all results (AFAWK).
• Intel, Nvidia, IBM: Available to their customers.
• All future results require HPCG 3.0 use.
• Quick Path option makes this easier.

hpcg-benchmark.org

8Main HPCG 3.0 Features
See http://www.hpcg-benchmark.org/software/index.htmlfor full discussion• Problem generation is timed.
• Memory usage counting and reporting.
• Memory bandwidth measurement and reporting• "Quick Path" option to make obtaining results onproduction systems easier.
• Provides 2.4 rating and 3.0 rating in output.
• Command line option (--rt=) to specify the run time.

hpcg-benchmark.org

9Other Items• Reference version on GitHub:• https://github.com/hpcg-benchmark/hpcg• Website: hpcg-benchark.org.
• Mail list hpcg.benchmark@gmail.com• HPCG & Student Cluster Competitions.
• Used in SC15/16, ASC• SC15: HPCG replaced HPL, ranking matched overall cluster ranking.

• HPCG-optimized kernels going into vendor libraries.
• Next event: SC’16:• 80 entries ISC16• 61 – SC15, 42 – ISC15, 25 – SC14, 15 – ISC14

hpcg-benchmark.org

Summary• HPCG is• Addressing original goals.
• Rewarding vendor investment in features we care about.
• HPCG has traction.
• Original goal of top 50 systems is reachable, and more.
• Version 3.X is the final planned major version.
• HPL and HPCG make a nice set of bookends.
• Anyone got a (wood) router?

10hpcg-benchmark.org

HPCG RANKINGS
JUNE 201611hpcg-benchmark.org

12

hpcg-benchmark.org

13

hpcg-benchmark.org

14

hpcg-benchmark.org

15

Rank(HPL) Site

1-10

1 (2) NSCC / Guangzhou

Computer
Tianhe-2 NUDT, Xeon 12C
2.2GHz + Intel Xeon Phi 57C+ Custom

RIKEN Advanced Institute for2 (5) Computational Science
National Supercomputing Center in
3 (1) Wuxi

K computer, SPARC64 VIIIfx
2.0GHz, Tofu interconnect
Sunway TaihuLight -SW26010, Sunway

4 (4) DOE/NNSA/LLNL

Sequoia - IBM BlueGene/Q
Titan - Cray XK7 , Opteron6274 16C 2.200GHz, Cray
Gemini interconnect, NVIDIA
K20x
Trinity - Cray XC40, Intel E52698v3, Aries custom

5 (3) DOE/SC/Oak Ridge Nat Lab
6 (7) DOE/NNSA/LANL/SNL

Mira - BlueGene/Q, Power7 (6) DOE/SC/Argonne National Laboratory BQC 16C 1.60GHz, CustomPangea -- Intel Xeon E58 (11) TOTAL
2670, Infiniband FDR
Pleiades - SGI ICE X, Intel
E5-2680, E5-2680V2, E52680V3, Infiniband FDR
9 (15) NASA / Mountain View
10 (9) HLRS/University of StuttgartHazel Hen - Cray XC40, Intel
E5-2680v3, Cray Aries

Cores Rmax

HPCG

HPCG/HPL% of Peak

3,120,000

33.8630.58001.7%

1.1%705,024

10.5100.55445.3%

4.9%

10,649,600

93.0150.3712

0.4%

0.3%

1,572,864

17.1730.33041.9%

1.6%

560,640

17.5900.32231.8%

1.2%

301,0568.101

0.18262.3%

1.6%

786,4328.587

0.16701.9%

1.7%2185925.283

0.16273.1%

2.4%

185,3444.089

0.15553.8%

3.1%

185,088

5.640

0.13802.4%

1.9%

Bookends: Peak, HPL, and HPCG
1000

100

Peak
1

HPL Rmax (Pflop/s)
HPCG (Pflop/s)

0.1

0.01

0.001
13
57911131523
273138
41
48
50
5766
80
108
127
158
253
279
303
338
349

Pflop/s10hpcg-benchmark.org

HPCG Highlights• 80 Systems:• Up from 61 at SC’15, 42 at ISC’15, 25 at SC’14 and 15 at ISC’14.
• All of the top 10, most of the top 20.
• Notable new entries are:
• New #3: Sunway TaihuLight, HPL #1• New #4: Sequoia, HPL #4.
• Expect upgraded Trinity result (with KNL) for SC’16.
• HPCG has diverse adoption:• Vendor/site versions complete.
• Cluster competitions.
• Industry analysis.

Rank Site

11-20

115,984

6.271

0.12462.0%

1.6%

12 KAUST / Jeddah

ComputerPiz Daint - Cray XC30, Xeon
E5-2670 8C 2.600GHz, Aries
interconnect , NVIDIA K20x
Shaheen II - Cray XC40, IntelHaswell 2.3 GHz 16C, CrayAries

196,608

5.537

0.11392.1%

1.6%

13 Japan Aerospace eXploration Agency

SORA-MA -- SPARC64 XIfx

103,680

3.157

0.1102

3.5%

3.2%

Stampede - PowerEdge
C8220, Xeon E5-2680 8C
Texas Advanced Computing Center/Univ. of 2.700GHz, Infiniband FDR,
14 Texas
Intel Xeon Phi SE10P
15 Forschungszentrum JülichJUQUEEN - BlueGene/Q

522,080
458,752

5.168
5.009

0.09680.09551.9%
1.9%

1.0%
1.6%

Swiss National Supercomputing Centre
11 (CSCS)

Cores Rmax

HPCG

HPCG/HPL % of Peak

Information Technology Center, Nagoya
16 University

ITC, Nagoya - Fujitsu
PRIMEHPC FX100, SPARC64
Xifx, Tofu interconnect 2

92,1602.910

0.08653.0%

2.7%

17 Leibniz Rechenzentrum
18 DOE/NNSA/LLNLSuperMUC - iDataPlex
DX360M4, Xeon E5-2680 8C
2.70GHz, Infiniband FDR
Vulcan - IBM BlueGene/Q

147,456
393,216

2.897
4.293

0.08330.0809

2.9%
1.9%

2.6%
1.6%

118,080

1.643

0.08084.9%

3.2%

133,8241.655

0.0786

4.8%

3.1%

19 EPSRC/University of Edinburgh
20 DOE/SC/LBNL/NERSCARCHER - Cray XC30, IntelXeon E5 v2 12C 2.700GHz,
Aries interconnect
Edison - Cray XC30, Intel Xeon
E5-2695v2 12C 2.4GHz, Aries
interconnect

Rank Site

21-30

21 National Institute for Fusion Science

Computer
Plasma Simulator - Fujitsu
PRIMEHPC FX100, SPARC64
Xifx, Tofu Interconnect 282,944

2.376

0.07323.1%

2.8%

22 GSIC Center, Tokyo Institute of Technology

TSUBAME 2.5 - Cluster Platform SL390s
G7, Xeon X5670 6C 2.93GHz, Infiniband
QDR, NVIDIA K20x

76,0322.785

0.0725

2.6%

1.3%

23 Forschungszentrum Jülich

JURECA - T-Platform V-Class
Cluster, Xeon E5-2680v3 12C
2.5GHz, Infiniband EDR,NVIDIA Tesla K80/K40

49,4761.425

0.06834.8%

3.8%

94,6562.763

0.06612.4%

1.7%

65,3201.283

0.0615

4.8%

4.2%

Earth Simulator - NEC SX-ACE

8,192

0.487

0.0578

11.9%

11.0%

76,8001.043

0.0565

5.4%

5.0%

8,192

0.487

0.0547

11.2%

10.4%

29 CEA/TGCC-GENCI

Oakleaf-FX -- SPARC64 Ixfx
Earth Simulator -- NEC SXACE
Curie thin nodes - Bullx B510,
Xeon E5-2680 8C 2.700GHz,
Infiniband QDR

77,1841.359

0.0510

3.8%

3.1%

30 Exploration & Production - Eni S.p.A.

HPC2 - iDataPlex DX360M4, Intel
Xeon E5-2680v2 10C 2.8GHz,
Infiniband FDR, NVIDIA K20x

62,6403.003

0.04891.6%

1.2%

24 HLRS/Universitaet Stuttgart
25 Max-Planck-Gesellschaft MPI/IPP
26 CEIST / JAMSTEC
Information Technology Center, The
27 University of Tokyo
28 CEIST / JAMSTEC

Hornet - Cray XC40, Xeon E52680 v3 2.5 GHz, Cray Aries
iDataPlex DX360M4, Intel Xeon
E5-2680v2 10C 2.800GHz,
Infiniband FDR

Cores Rmax

HPCG

HPCG/HPL % of Peak

Rank Site

31-40

ComputerGrand Equipement National de Calcul
Occigen Bullx B720, Xeon E5Intensif - Centre Informatique National de2690v3 12C 2.600GHz,
31 l'Enseignement Superieur (GENCI-CINES) InfiniBand FDR
International Fusion Energy Research Centre Helios Bullx B510, Xeon E5(IFERC), EU(F4E) - Japan Broader Approach 2680 8C 2.700GHz, Infiniband
QDR
32 collaboration
33Cyfronet
Lvliang/National University of Defense
34 TechnologyMoscow State University / Research35 Computing Center
36DKRZ - Deutsches Klimarechenzentrum

Prometheus - HP ProLiant Intel
E5-2680v3, Infiniband FDR
Tianhe-2 Lvliang - Intel Xeon
E5-2692v2 12C, TH Express-2,
Intel Xeon Phi 31S1P
Lomonosov 2 - Intel Xeon E52680V2, Infiniband FDR,
NVIDIA K40
Mistral -- Intel Xeon E5-2695v4,
Infiniband FDR

Cores Rmax

HPCG

HPCG/HPL % of Peak

50,5441.629

0.04552.8%

2.2%

70,5601.237

0.0426

3.4%

2.8%

55,7281.670

0.03992.4%

1.7%

174,7202.071

0.03761.8%

1.2%

37,1201.849

0.0315

1.7%

1.2%

19,2001.371

0.0283

2.1%

1.7%

4,096

0.246

0.0279

11.3%

10.7%

38Stanford University / Palo Alto
39CINECA

Cyberscience Center, Tohoku
University -- NEC SX-ACE
Xstream - Dual Intel E52680V2, 8-way NVIDIA K80,
Infiniband FDR
Fermi - IBM BlueGene/Q

237,120
163,840

0.781
1.789

0.02300.0216

2.9%
1.2%

2.3%
1.0%

40SURFsara, Amsterdam

Cartesius2 bullx B720, dual socket Intel
Xeon E5-2690 v3, Infiniband FDR

25,9200.848

0.01952.3%

1.8%

37Cyberscience Center, Tohoku University

Rank Site

41-50

ComputerCores Rmax

41 Cyberscience Center / Tohoku University

NEC SX-ACE 4C+IXS

2,048

0.123

0.0150

12.2%

11.4%

42 Cybermedia Center, Osaka University

Osaka U ACE -- NEC SX-ACE

2,048

0.123

0.0142

11.5%

10.8%

43 SGI

SGI ICE X -- Intel Xeon E52690v4, Infiniband EDR

16,1280.602

0.01222.0%

1.8%

44 LNCC

Santos Dumont, Bullx Intel E52695v2, Infiniband FDR

17,6160.321

0.0121

3.8%

3.5%

45 IntelEndeavor - Intel Cluster, DualIntel Xeon E5-2697v3 14C
2.700GHz, Infiniband FDR,
Intel Xeon Phi 7120P

51,3920.759

0.01121.5%

1.2%

46 Meteo FranceBeaufix - Bullx DLC B710Blades, Intel Xeon E5-2697v212C 2.7GHz, Infiniband FDR

24,1920.469

0.01102.3%

2.1%

47 Saint Petersburg Polytechnic University

Polytechnic - RSC TornadoIntel E52697v3, Infiniband FDR

17,4440.658

0.01081.6%

1.3%

48 Meteo FranceProlix - Bullx DLC B710 Blades,Intel Xeon E5-2697v2 12C
2.7GHz, Infiniband FDR

23,7600.465

0.01002.1%

1.9%

Manny Bullx B720, Xeon E5-2690v3
12C 2.600GHz, InfiniBand FDR

12,9600.430

0.00972.3%

1.8%

bwForCluster - Intel E5-2630v3,
Infiniband QDR

7,552

0.241

0.00933.9%

3.2%

49 Bull Angers
University Heidelberg and University
50 Mannheim

HPCG

HPCG/HPL % of Peak

Rank Site

51-60

Computer

51Michigan State University

Laconia -- Intel Xeon E5-2680v4,
Infiniband EDR FDR

52University of Duisburg-Essen

Cores Rmax

HPCG

HPCG/HPL % of Peak

1,008,760

0.536

0.00911.7%

1.2%

magnitUDE -- Intel Xeon E5-2650v4,Intel OmniPath120.437

0.00902.1%

1.9%

53CALMIP / University of Toulouse

EOS - Bullx DLC B710 Blades, Intel
Xeon E5-2680v2 10C 2.8GHz,
Infiniband FDR

12,2400.255

0.00732.8%

2.6%

54Christian-Albrechts-Universitaet zu Kiel

NEC SX-ACE -- NEC SX-ACE

1,024

0.062

0.0068

11.1%

10.5%

2,720

0.273

0.00682.5%

1.6%

TSUBAME-KFC/DL -- Intel Xeon E5-55GSIC Center, Tokyo Institute of Technology 2620-V2, Infiniband FDR
56University of Tuebingen

BinAC -- Intel Xeon E5-2680v4,
Infiniband FDR

4,800

0.209

0.00633.0%

2.2%

The Institute of Atmospheric Physics,
57 Chinese Academy of Sciences

Earth System Numerical Simulator-1- Intel E5-2680-V3, Infiniband FDR

24,9120.738

0.00630.8%

0.6%

58Joint Supercomputer Center RAS

MVS-10P - Intel E5-2690, Infiniband
FDR, Xeon Phi SE10X

2,992

0.376

0.00491.3%

0.9%

59University of Rijeka

Bura - Bullx Intel E5-2690v3,Infiniband FDR

5,952

0.234

0.00472.0%

1.6%

60CINECA

Galileo - Dual Intel E5-2630 v3 2.4
GHz, Infiniband QDR, Dual NVIDIA
K80

2,7200.0046

1.9%

Rank Site

61-70ComputerCores Rmax

61NSC / Linkoping

Bifrost - ASUS, Intel Xeon E5-2640v3 8C
2.6GHz, Intel Truescale Infiniband QDR

10,2560.326

0.00451.4%

0.8%

62Shanghai Supercomputer Center

Magic Cube II - Intel E5-2680-V3, Infiniband EDR

9,960

0.296

0.00441.5%

1.1%

63Max-Planck-Institut für Mikrostrukturphysik

Cruncher - Intel E5-2680-V3, Intel Truescale
Infiniband QDR

120.1120.0040

3.6%

2.8%

64Cambridge University

Wilkes - Dell T620 Cluster, Intel Xeon E5-2630v26C 2.600GHz, Infiniband FDR, NVIDIA K205,120

0.240

0.00391.6%

1.0%

65Chelyabinsk

RSC Tornado SUSU, Intel X5680, Infiniband
QDR, Xeon Phi SE10X

4,032

0.288

0.00361.2%

0.8%

66CINECA

Galileo - Dual Intel E5-2630 v3 2.4 GHz,
Infiniband QDR, Dual Intel Xeon Phi 7120P

13,60067Atos Angers

Sid - Bullx Intel E5-2680v3, InfiniBand FDR

4,224

0.129

0.00322.5%

2.0%

68St. Petersburg Polytechnic University

Polytechnic RSC PetaStream - Intel E5-2650 v2,
Infiniband FDR, Xeon Phi 5120D

232

0.170

0.0031

1.8%

1.2%

Supercomputing Center of Chinese Academy Era-2 - Intel E5-2680-V3, Infiniband FDR, Xeon
69 of Sciences
Phi + NVIDIA K20

13560

0.407

0.00300.7%

0.6%

Cartesius - Bullx B515 cluster, Intel Xeon E52450v2 8C 2.5GHz, InfiniBand 4x FDR, Nvidia
K40m

3,036

0.154

0.0025

1.7%

1.2%

70SURFsara

HPCG

HPCG/HPL % of Peak0.0034

1.5%

Rank Site

71-80

ComputerCores Rmax

71CINECA

Galileo - Dual Intel E5-2630 v3 2.4
GHz, Infiniband QDR

6,400

Moscow State University / Research
72 Computing Center

Lomonosov - Intel Xeon
X5570/X5670/E5630 2.93/2.53 GHz,
PowerXCell 8i Infiniband QDR, Dual
NVIDIA Fermi 2070

78,6600.617

0.00170.3%

0.2%

73IT Services Provider

Aquarius - Intel Xeon E5-2640-V3,
Infiniband QDR

80.0340.00144.0%

3.2%

74Joint Supercomputer Center RAS

RSC PetaStream - Intel E5-2667 v2,
Infiniband FDR, Intel Xeon Phi
7120D

3,904

0.054

0.0012

2.2%

1.5%

75Yaqingjie Street 30

hbemc_2016A -- Intel E5-2680v3,
Infiniband FDR

2,304

YUJING -- Intel Xeon E5-2680v3,

76Hefei City,Anhui Province
customNo.180 Wusidong Road. Baoding City, Hebei KunYu -- Intel Xeon E5-2680v3,
77 Province,P.R.C
Infiniband FDRhongguancun Software Park II, No. 10 West
Dongbeiwang Road, Haidian District, Beijing CSRC -- Intel Xeon E5-2680v3,
78 100193, China
Infiniband FDR
18, Xueyuan Road, Haidian District,geo -- Intel Xeon E5-2680v3,
79 Beijing,China
Infiniband FDR
80CINECA

Pico - Dual Intel Xeon E5-2670v2 2.5
GHz, Gigabit Ethernet

HPCG

HPCG/HPL % of Peak0.00200.0009

1,440

0.001

0.0008960

0.001

0.0006528

0.000

0.0004120.000

0.0003

1,200

1.6%

0.0003

1.1%

HPCG Lists Over Time
HPCG Performance (Pflop/s)

1.00000.1000

Jun'16
Nov'150.0100Jun '15
Nov '14
Jun '140.00100.000180 77 74 71 68 65 62 59 56 53 50 47 44 41 38 35 32 29 26 23 20 17 14 11 8 5 2
HPCG Rank

Performance by Region
Europe26%

Americas
29%

Asia
45%

Performance by Country
BrazilChina
USA

Croatia
FranceUK
The Netherlands
Switzerland
Sweeden
Saudi Arabia
RussiaPolandGermany

Italy

Japan

Performance by Network TypeInfiniband

Ethernet

CustomPerformance by Network Details
Tofu

Blue GeneTianhe Express

Cray

SunwayGigabit Ethernet
Infiniband EDR
NEC
Inifiniband OmniPath
Infiniband QDRInfiniband FDR

Performance by Processor TypeVector
NVIDIA GPU

Intel Phi

CPU

Performance by Processor Detail
SunwayIBM Power

Sparc FXIntel Phi

NVIDIA

NEC SXIntel Xeon

Performance by Segment
VendorAcademic

Classified
Government
Industry

Research

Performance by Integrator
Sugon ASUS
SGI T-Platforms

Bull

RSC OtherGroup

NUDT

Cray

NRCPC
Dawning
DellNEC
MegawareIntel
LenovoFujitsu

IBMHP