新しい科学を開拓する
Oakforest-PACS中島 研吾
東京大学情報基盤センター
最先端共同HPC基盤施設（研究開発部門長）
2017年10月26日
サイエンティフィックシステム研究会
科学技術計算分科会2017年度会合
拡がるHPC ～新たな鼓動～

2

Oakforest-PACS （OFP）• 2016年12月1日稼働開始• 8,208 Intel Xeon/Phi (KNL)，ピーク性能25PFLOPS– 富士通が構築• TOP 500 7位（国内1位），HPCG 5位（国内2位）（2017年6月）• 最先端共同HPC 基盤施設(JCAHPC: Joint Center
for Advanced High Performance Computing)– 筑波大学計算科学研究センター– 東京大学情報基盤センター• 東京大学柏キャンパスの東京大学情報基盤センター内に、両機関の教職員が中心となって設計するスーパーコンピュータシステムを設置し，最先端の大規模高性能計算基盤を構築・運営するための組織– http://jcahpc.jp

最先端共同HPC基盤施設 JCAHPC 平成 年 月、筑波大学と東京大学は「計算科学・工学
及びその推進のための計算機科学・工学の発展に資するための連携・協力推進に関する協定」を締結
 本協定の下、筑波大学計算科学研究センターと東京大を設置
学情報基盤センターが
平成23年のプレスリリース

3

2センターのミッション• 筑波大学計算科学研究センターのミッション：• 計算機科学と計算科学の協働:学際的な高性能計算機開発
PACSシリーズの開発：CP-PACS@1996 TOP1
• 先端学際科学共同研究拠点: 最先端の計算科学研究推進• これからの計算科学に必要な学際性を持つ人材を育成• 東京大学情報基盤センターのミッション：• 学際大規模情報基盤共同利用・共同研究拠点（８大学の情報基盤センター群からなるネットワーク型） の中核拠点：
大規模情報基盤を活用し学際研究を発展• HPCI資源提供機関：最先端スパコンの共同設計開発及び運用、Capability資源および共用ストレージ資源の提供• 人材育成：計算科学の新機軸を創造できる人材の育成
4

HPCI: High Performance Computing Infrastructure
日本全体におけるスパコンインフラ

大学（北大、東北大、筑波大、東大、東工大、名大、京大、阪大、九大）の情報基盤センター
海洋開発研究機構、統数研＋3

Plan of 9 Tier-2 Supercomputer Centers(May 2017）
Fiscal Year
Hokkaido

2014

2015

2016

2017

2018

HITACHI SR16000/M1（172TF, 22TB）
Cloud System BS2000 （44TF, 14TB）

Tsukuba

2020

2021

2022

2023

2024

2025

30 PF （UCC +CFL-M） 2MW

3.2 PF (UCC + CFL/M) 0.96MW
0.3 PF (Cloud) 0.36MW

Data Science Cloud / Storage HA8000 / WOS7000（10TF, 1.96PB）

Tohoku2019

NEC SX~30PF, ~30PB/s Mem BW (CFL-D/CFL-M)
SX-ACE(707TF,160TB, 655TB/s)
9他
LX406e(31TF), Storage(4PB), 3D Vis, 2MW~3MW(60TF)
HA-PACS (1166 TF)
PACS-X 10PF (TPF) 2MW
COMA (PACS-IX) (1001 TF)

Tokyo

Oakforest-PACS 25 PF(UCC + TPF) 3.2 MW

Fujitsu FX10 (1PFlops, 150TiB, 408 TB/s),
Reedbush 1.93 PF 0.7 MW

Reedbush-L 1.40 PF 0.18 MW

Hitachi SR16K/M1 (54.9 TF, 10.9 TiB, 28.7 TB/s)

Tokyo Tech.

Nagoya

Kyoto

Osaka

Kyushu100+ PF 4.5MW(UCC + TPF)
200+ PF
(FAC)
6.5MW

BDEC: 50+ PF (FAC) 3.5MW

TSUBAME 2.5 (3～4 PF, extended)

TSUBAME 2.5 (5.7 PF,
110+ TB, 1160 TB/s),
1.4MW

TSUBAME 3.0 (20 PF, 4~6PB/s) 2.0MW
TSUBAME 4.0 (100+ PF,(3.5, 40PF at 2018 if upgradable)
c10PB/s, ~2.0MW)
FX10(90TF)
Fujitsu FX100 (2.9PF, 81 TiB)
100+ PF
50+ PF (FAC/UCC + CFL-M)
Fujitsu CX400 (774TF, 71TiB)(FAC/UCC+CFLup to 4MW M)up to 4MW
SGI UV2000 (24TF, 20TiB)
2MW in total

Cray: XE6 + GB8K +XC30 (983TF)
Cray XC30 (584TF)

7-8 PF(FAC/TPF + UCC)
1.5 MW

NEC SX-ACE NEC Express5800(423TF) (22.4TF)
HA8000 (712TF, 242 TB)
SR16000 (8.2TF, 6TB)
FX10 (272.4TF, 36 TB)2.0MW
CX400 (966.2 TF, 183TB)

3.2PB/s, 5-10Pflop/s, 1.0-1.5MW (CFL-M)

0.7-1PF (UCC)

15-20 PF (UCC/TPF) 2.6MW
FX10(90.8TFLOPS)

50-100+ PF(FAC/TPF + UCC) 1.8-2.4 MW
25.6 PB/s, 50100Pflop/s,1.52.0MW

100-150 PF(FAC/TPF + UCC/TPF)3MW

Power consumption indicates maximum of power supply(includes cooling facility)
6

フラグシップとの両輪として第１階層（理研：京、ポスト京）と第２階層（JCAHPC：Oakforest-PAC ）が両輪となって xascale Computingへ

1000

100

FutureExascale

Post K Computer
理研 AICS

Oakforest-PACS

10

JCAHPC（東大と筑波大）

1

Tokyo Tech.
TSUBAME2.0

T2K

U. of Tsukuba
U. of Tokyo
Kyoto U.

2008

2010

2012

2014

2016

2018

2020

Oakforest-PACSの由来• Oakforest：東大– 2012年4月 東大情報基盤センター柏拠点開所– Oakleaf-FX（Fujitsu PRIMEHPC FX10，1+PF）運用開始– 1F：Oakleaf-XXX，2F：Oakforest-YYY• PACS：筑波大– PACS： Parallel Advanced System for Computational
Science– 9世代にわたって継承• 両センターの思いの詰まった名称– 必ず「オークフォレストパックス」と呼んでください（OFPでも可）。
Year
1978
1980
1983
1984
1989
1996
2006
2012
2014

System
PACS-9 (PACS I)
PACS-32 (PACS II)
PAX-128 (PACS III)
PAX-32J (PACS IV)
QCDPAX (PACS V)
CP-PACS (PACS VI)
PACS-CS (PACS VII)
HA-PACS (PACS VIII)COMA (PACS IX)

Performance
7 KFLOPS
500 KFLOPS
4 MFLOPS
3 MFLOPS
14 GFLOPS
614 GFLOPS
14.3 TFLOPS
1.166 PFLOPS
1.001 PFLOPS

FY
08

09 10 11 12 13 14 15 16 17 18 19 20 21 22

Hitachi SR11K/J2
IBM Power-5+

Yayoi: Hitachi SR16000/M1
IBM Power-7

18.8 FLOP , 16.4 B

34.

FLOP , 11.2 B

Hitachi HA8000 (T2K)
AMD Opteron

Oakforest-PACS
Fujitsu, Intel KNL

140 FLOP , 31.3 B

25PFLOPS, 919.3TB

Oakleaf-FX: Fujitsu PRIMEHPC
FX10, SPARC64 IXfx
1.13 PFLOP , 130 B

Post 2K
JCAHPC:
筑波大・
東大

Post FX10
30+ PFLOP (?)

Oakbridge-FX
136.2 FLOP , 18.4 B
データ解析・シミュレーション
融合スーパーコンピュータ

Peta

K

K computer

Reedbush, SGI
Broadwell + Pascal
1.80-1. 3 PFLOP

Post-K

8

Oakforest-PACSの特徴• 計算ノード– 1ノード 68コア，3TFLOPS×8,208ノード＝ 25 PFLOPS– MCDRAM（３D積層・高速，16GB）＋DDR4（低速，96GB）• 多様なメモリモード，クラスタリングモードの選択可能• ノード間通信– フルバイセクションバンド幅を持つFat-Treeネットワーク– 全系運用時のアプリケーション性能に効果を発揮，多ジョブ運用– Intel Omni-Path Architecture• ファイルI/O– 並列ファイルシステム– 高速ファイルキャッシュシステム（DDN IME）
• 計算科学・ビッグデータ解析・機械学習にも貢献• 消費電力– Green 500でも世界6位（2016年11月）– Linpack：4,986 MFLOPS/W（OFP），830 MFLOPS/W（京）9Memory Modes for KNL

10

〔塙他 SC16 BoF,
HPC-157，2016〕

Results (OFP: 32 nodes)
Parallel FEM (GeoFEM), HB 16x4
• C: Cache, F: Flat
• Q: Quadrant, N4: SNC-4
1.6 -02• (m) : core binding to avoid “busy” core
• (m) case are generally betterwith (m)w/o (m)

1.3 -02

1.4 -02

1.3 -02

1.2 -02

1.1 -02

1.0 -02
Original

C+Q

F+Q

C+ 4

F+ 4

C+Q(m)

F+Q(m)

C+ 4(m)F+ 4(m)11

Results (OFP: 32 nodes): (m)Parallel FEM, HB 16x4 (16t x 4p)• C: Cache, F: Flat
• Q: Quadrant, N4: SNC-4• (m) : core binding to avoid “busy” core
• Best case: F+Q. dynamic 500 (m)1.3 -02

1.0 -02

3.0 -03

0.0 +00
Original〔塙他 SC16 BoF,
HPC-157，2016〕

Overlap

C+Q(m)30

100

F+Q(m)

200

C+ 4(m)

300

400

300

F+ 4(m)12

Comparisons at 8,192 cores
OFP: 128 nodes, Parallel FEM
Flat MPI is much faster than HB 16x4, Alg.2 is the best
3.3

Down is Good ■ HB 16x4 ■ Flat MP3lapsed ime

2.3

2

1.3

1

0.3

0

〔塙他,
Alg.1
HPC-157，2016〕

Alg.2

Alg.3-AR

Hybrid(ppn4) CMRCM(10)Alg.3- AR

Alg.4-AR

Flat MP CMRCM(10)Alg.4- AR
13

Oakforest-PACS の仕様
総ピーク演算性能
ノード数
Product
計算
ノード
プロセッサ

23 PFLOP
8,208

富士通 PR M RGY CX600 M1 (2U) +CX1640 M1 x 8node
ntel® Xeon Phi™ 230（開発コード: Knights Landing）
68 コア、1.4 GHz
メモリ 高バンド幅 16 GB, MCDRAM, 実効 4 0 GB/sec
低バンド幅 6 GB, DDR4-2400, ピーク 113.2 GB/sec
ntel® Omni-Path Architecture
相互結 Product
合網
100 Gbps
リンク速度
トポロジ

フルバイセクションバンド幅Fat-tree網
14計算ノードとシャーシ

Chassis with 8 nodes, 2U size

Computation node (Fujitsu next generation PR M RGY)
with single chip ntel Xeon Phi (Knights Landing, 3+ FLOP )and ntel Omni-Path Architecture card (100Gbps)

13フルバイセクションバンド幅を用いた
網
68 port Director witch
12台( ource by ntel)

22Uplink: 24

48 port dge witch
362 台
Downlink: 24
1

...

24

23

...

48

4

...

2

コストはかかるがフルバイセクションバンド幅を維持
 システム全系使用時にも高い並列性能を実現
 柔軟な運用：ジョブに対する計算ノード割り当ての自由度が高い

16

Oakforest-PACS の仕様（続き）
並列ファイ
ルシステム

高速ファイ
ルキャッ
シュシステ
ム

ype
総容量
Product
総バンド幅
ype
総容量
Product
総バンド幅

総消費電力
総ラック数

Lustre File ystem
26.2 PB
DataDirect etworks FA14K
300 GB/sec
Burst Buffer, nfinite Memory ngine(by DD )40 B ( VMe D, パリティを含む)
DataDirect etworks M 14K
1,360 GB/sec
4.2MW（冷却を含む）
102

1

各種ベンチマーク• TOP 500（Linpack，HPL（High Performance Linpack））– 連立一次方程式ソルバー（直接法），計算速度（FLOPS値）– 規則的な密行列：連続メモリアクセス– 計算性能• HPCG– 連立一次方程式ソルバー（反復法），計算速度（FLOPS値）– 有限要素法から得られる疎行列（ゼロが多い）• 不連続メモリアクセス• 実アプリケーションに近い– メモリアクセス性能，通信性能• Green 500– HPL（TOP500）実行時の消費電力量から求めたFLOPS/W値18

49th TOP500 List (June, 2017)
Site

1

Computer/Year Vendor

National Supercomputing
Center in Wuxi, China

Sunway TaihuLight , Sunway MPP,
Sunway SW26010 260C 1.45GHz,
2016 NRCPC

Tianhe-2, Intel Xeon E5-2692, TH
Express-2, Xeon Phi, 2013 NUDT
Piz Daint
3
Cray XC30/NVIDIA P100, 2013 Cray
Titan
4
Cray XK7/NVIDIA K20x, 2012 Cray
Sequoia5
BlueGene/Q, 2011 IBM
Cori, Cray XC40, Intel Xeon Phi 7250
6
68C 1.4GHz, Cray Aries, 2016 Cray
Oakforest-PACS, PRIMERGY CX600
Joint Center for Advanced
M1, Intel Xeon Phi Processor 7250 68C
7 High Performance
1.4GHz, Intel Omni-Path,
Computing, Japan
2016 Fujitsu
K computer, SPARC64 VIIIfx , 2011
8 RIKEN AICS, Japan
FujitsuArgonne National Laboratory, Mira
9
BlueGene/Q, 2012 IBM
USA
Trinity, Cray XC40, Xeon E5-2698v3
10 DOE/NNSA/LANL/SNL, USA
16C 2.3GHz, 2016 Cray
2

National Supercomputing
Center in Tianjin, China
Swiss Natl. Supercomputer
Center, Switzerland
Oak Ridge National
Laboratory, USA
Lawrence Livermore National
Laboratory, USA
DOE/SC/LBNL/NERSC
USA

Cores

Rmax
Rpeak
Power
(TFLOPS) (TFLOPS) (kW)10,649,600

93,015(= 93.0 PF)

125,436

15,371

3,120,000

33,863(= 33.9 PF)

54,902

17,808361,760

19,59033,8632,272

560,640

17,59027,1138,209

1,572,864

17,173

20,133

7,890

632,400

14,01527,881

3,939

557,056

13,55524,914

2,719 919 TB

705,024

10,51011,28012,660 1,410TB

786,432

8,587

10,066

3,945

301,056

8,101

11,0794,233

Rmax: Performance of Linpack (TFLOPS)
Rpeak: Peak Performance (TFLOPS), Power: kW

http://www.top500.org/

19

HPCG Ranking (June, 2017)
Computer
1 K computer
2 Tianhe-2 (MilkyWay-2)
3 Sunway TaihuLight
4 Piz Daint

HPL Rmax TOP500(Pflop/s)
Rank
705,024
10.5108
3,120,000
33.863
2
10,649,600
93.015
1
Cores

HPCG
Peak
(Pflop/s)
0.60275.3%
0.58011.1%
0.48080.4%

361,760

19.5903

0.4767

1.9%

Oakforest-PACS
Cori
Sequoia
Titan
Trinity

557,056
632,400
1,572,864
560,640
301,056

13.55513.832
17.17317.590
8.101

7
6
5
4
10

0.38550.35540.33040.32230.1826

1.5%
1.3%
1.6%
1.2%
1.6%

10 Pleiades – NASA/SGI

243,0085.952

150.1752

2.5%

5
6
7
8
9

http://www.hpcg-benchmark.org/
20

Green 500 Ranking (November, 2016)
Site

ComputerCPU

NVIDIA DGX-1, Xeon E5-2698v4 20C
2.2GHz, Infiniband EDR, NVIDIA
Tesla P100
Swiss National
Cray XC50, Xeon E5-2690v3 12C
2 Supercomputing Piz Daint
2.6GHz, Aries interconnect , NVIDIA
Centre (CSCS)
Tesla P100
3 RIKEN ACCS
ShoubuZettaScaler-1.6 etc.
National SC
SunwaySunway MPP, Sunway SW26010
4
Center in Wuxi
TaihuLight 260C 1.45GHz, Sunway
SFB/TR55 at
PRIMERGY CX1640 M1, Intel Xeon
5 Fujitsu Tech.
QPACE3Phi 7210 64C 1.3GHz, Intel OmniSolutions GmbHPath
PRIMERGY CX1640 M1, Intel Xeon
Oakforest6 JCAHPCPhi 7250 68C 1.4GHz, Intel OmniPACSPath
DOE/SC/Argonne
Cray XC40, Intel Xeon Phi 7230 64C
7
Theta
National Lab.
1.3GHz, Aries interconnect
Cray CS-Storm, Intel Xeon E5-2680v2
Stanford Research
8
XStream
10C 2.8GHz, Infiniband FDR, Nvidia
Computing Center
K80
ACCMS, Kyoto
Cray XC40, Intel Xeon Phi 7250 68C
9Camphor 2
University
1.4GHz, Aries interconnect
Jefferson Natl.
KOI Cluster, Intel Xeon Phi 7230 64C
10
SciPhi XVI
Accel. Facility
1.3GHz, Intel Omni-Path
NVIDIA
1
Corporation

DGX
SATURNV

HPL
TOP500Rmax
Rank(Pflop/s)

Power
(MW)

GFLOPS/W

3.307

28

0.350

9.462

9.779

8

1.312

7.454

1.001

116

0.150

6.674

93.01

1

15.37

6.051

0.447

375

0.077

5.806

1.355

6

2.719

4.986

5.096

18

1.087

4.688

0.781

162

0.190

4.112

3.057

33

0.748

4.087

0.426

397

0.111

3.837

http://www.top500.org/
21

Green 500 Ranking (June, 2017)
Site

ComputerCPU

HPL Rmax TOP500(Pflop/s)
Rank

Power
(MW)

GFLOPS/W

1Tokyo Tech.

TSUBAME3.0

SGI ICE XA, IP139-SXM2, Xeon E52680v4, NVIDIA Tesla P100 SXM2, HPE

1,998.0

61

142

14.1102Yahoo Japan

kukai

ZettaScaler-1.6, Xeon E5-2650Lv4,,
NVIDIA Tesla P100 , Exascalar

460.7

465

33

14.046

3

AIST, JapanAIST AI
Cloud

NEC 4U-8GPU Server, Xeon E5-2630Lv4,
NVIDIA Tesla P100 SXM2 , NEC

961.0

148

76

12.681

4

CAIP, RIKEN,JAPANRAIDEN GPU
subsystem -NVIDIA DGX-1, Xeon E5-2698v4, NVIDIA
Tesla P100 , Fujitsu

635.1

305

60

10.603

5

Univ.
Cambridge, UK

Wilkes-2 -Dell C4130, Xeon E5-2650v4, NVIDIA
Tesla P100 , Dell

1,193.0100

114

10.428

6

Swiss Natl. SC.
Center (CSCS)

Piz Daint

Cray XC50, Xeon E5-2690v3, NVIDIA
Tesla P100 , Cray Inc.

19,590.0

32,272

10.398

7

JAMSTEC,Japan

Gyoukou,

ZettaScaler-2.0 HPC system, Xeon D1571, PEZY-SC2 , ExaScalar

1,677.1

69

164

10.226

8

Inst. for Env.
Studies, JapanGOSAT-2(RCF2)

SGI Rackable C1104-GP1, Xeon E52650v4, NVIDIA Tesla P100 ,NSSOL/HPE

770.4

220

79

9.797

9

Facebook, USA

Penguin
RelionXeon E5-2698v4/E5-2650v4, NVIDIA
Tesla P100 , Acer Group

3,307.0

31

350

9.462

10

NVIDIA, USA

DGX SaturnV

Xeon E5-2698v4, NVIDIA Tesla P100 ,Nvidia3,307.0

32

350

9.462

11

ITC, U.Tokyo,
Japan

Reedbush-H

SGI Rackable C1102-GP8, Xeon E52695v4, NVIDIA Tesla P100 SXM2 , HPE

802.4

203

94

8.575

http://www.top500.org/

22

23

Now operating 4 (or 6)systems !!
• Oakleaf-FX (Fujitsu PRIMEHPC FX10)– 1.135 PF, Commercial Version of K, Apr.2012 – Mar.2018• Oakbridge-FX (Fujitsu PRIMEHPC FX10)– 136.2 TF, for long-time use (up to 168 hr), Apr.2014 – Mar.2018• Reedbush (HPE, Intel BDW + NVIDIA P100 (Pascal))– Integrated Supercomputer System for Data Analyses &Scientific Simulations• Jul.2016-Jun.2020– Our first GPU System (Mar.2017), DDN IME (Burst Buffer)– Reedbush-U: CPU only, 420 nodes, 508 TF (Jul.2016)– Reedbush-H: 120 nodes, 2 GPUs/node: 1.42 PF (Mar.2017)– Reedbush-L: 64 nodes, 4 GPUs/node: 1.43 PF (Oct.2017)• Oakforest-PACS (OFP) (Fujitsu, Intel Xeon Phi (KNL))– JCAHPC (U.Tsukuba & U.Tokyo)– 25 PF, #7 in 49th TOP 500 (June.2017) (#1 in Japan)– Omni-Path Architecture, DDN IME (Burst Buffer)

24

JPY (=Watt)/GFLOPS Rate
Smaller is better (efficient)
SystemOakleaf/Oakbridge-FX (Fujitsu)(Fujitsu PRIMEHPC FX10)
Reedbush-U (SGI)(Intel BDW)
Reedbush-H (SGI)(Intel BDW+NVIDIA P100)
Oakforest-PACS (Fujitsu)(Intel Xeon Phi/Knights Landing)

JPY/GFLOPS
125
62.0
17.1
16.5

運用
• 計算資源は全系を共用（パーティション分けはしない）
全8,208ノード（25PF）を常に全系で運用できるようにしておき，国内最大の計算資源を有効に活用する
Flat/Quadrant，Cache/Quadrantで半々• 様々な利用形態
各大学独自の利用コース
グループ利用（最大2,048ノード）
パーソナル利用（東大のみ）
企業利用（東大のみ）
国際共同利用（グループ利用の一部）

HPCI
全資源の20%を「JCAHPC」として拠出，企業利用可能

JHPCN（学際大規模情報基盤共同利用共同研究拠点）
全資源の5%程度：企業共同研究，国際共同研究も含む（東大のみ）

教育（講義，講習会）
大規模HPCチャレンジ：全ノード占有：公募中（非ユーザも可）23

Oakforest-PACS のソフトウェア• OS: Red Hat Enterprise Linux (ログインノード)，CentOS および
McKernel (計算ノード，切替可能)– McKernel: 理研AICSで開発中のメニーコア向けOS

McKernel• Linuxに比べ軽量，ユーザプログラムに与える影響なし• ポスト京にも搭載される予定。

• コンパイラ：GCC, Intel Compiler, XcalableMP– XcalableMP: 理研AICS・筑波大で開発中の並列プログラミング言語• ライブラリ・アプリケーション： オープンソースソフトウェア– ppOpen-HPC,
OpenFOAM,
ABINIT-MP, PHASE
system,FrontFlow/blue，
LAPACK,
ScaLAPACK,PETSc, METIS,SuperLU etc.

26

ppOpen-HPC: Overview• Application framework with automatic tuning (AT)“pp” : post-peta-scale• Five-year project (FY.2011-2015) (since April 2011)
P.I.: Kengo Nakajima (ITC, The University of Tokyo)
Part of “Development of System Software Technologies for
Post-Peta Scale High Performance Computing” funded by
JST/CREST (Supervisor: Prof. Mitsuhisa Sato, RIKEN AICS)• Target: Oakforest-PACS (Original Schedule: FY.2015)
could be extended to various types of platforms• Team with 7 institutes, >50 people (5PDs) from various fields: Co-Design• Open Source Software
http://ppopenhpc.cc.u-tokyo.ac.jp/
English Documents, MIT License

2

Framework
Appl. Dev.
Math
Libraries
Automatic
Tuning (AT)
SystemSoftware28

29

ppOpen-HPC covers …

29

30

Featured Developments• ppOpen-AT: AT Language for Loop Optimization• HACApK library for H-matrix comp. in ppOpenAPPL/BEM (OpenMP/MPI Hybrid Version)– First Open Source Library by OpenMP/MPI Hybrid• ppOpen-MATH/MP (Coupler for Multiphysics
Simulations, Loose Coupling of FEM & FDM)
• Sparse Linear Solvers

31

Oakforest-PACS：代表的アプリケーション• SALMON/ARTED– 電子動力学• Lattice QCD– 格子量子色力学• NICAM-COCO– 全地球大気・海洋連成• GHYDRA– 三次元地盤震動（FEM）• Seism3D/OpenSWPC– 三次元広域波動伝搬（FDM）

Research Area based on CPU Hours
Oakforest-PACS in FY.2017(2017.4~2017.9M)
NICAM-COCO
GHYDRA
Seism3D

SALMON/ARTED
Lattice QCD

32

ngineering
arth/ pace
Material
nergy/Physics
nformation ci.
ducationndustryBioocial ci. & conomics
Data

33

Research Area based on CPU Hours
FX10 in FY.2015 (2015.4~2016.3E)
ngineering
arth/ pace
Material
nergy/Physics
nformation ci.
ducationndustryBioconomics

Oakleaf-FX + Oakbridge-FX

34

Research Area based on CPU Hours
FX10 in FY.2016 (2016.4~2017.3E)
ngineering
arth/ pace
Material
nergy/Physics
nformation ci.
ducationndustryBioocial ci. & conomics

Oakleaf-FX + Oakbridge-FX

35

Research Area based on CPU Hours
Reedbush-U in FY.2016(2016.7~2017.3E)
ngineering
arth/ pace
Material
nergy/Physics
nformation ci.
ducationndustryBioocial ci. & conomics

第1回OFP利活用報告会（5th JCAHPCセミナー）
2017年10月12日（木）13:30-18:00•

13:30 - 13:45 中島研吾（JCAHPC/東京大学情報基盤センター）– 開会 システム概要•

13:45 - 14:15 斉藤圭亮（東京大学）– 光合成蛋白質内の水分子のシミュレーション•

14:15 - 14:45 佐々木勝一（東北大学）– 物理点格子QCDシミュレーションによる核子構造の研究•

14:45 - 15:15 廣川祐太（筑波大学）– 電子動力学シミュレーションARTEDのOakforest-PACSでの全系性能評価•

15:30 - 16:00 井戸村泰宏（日本原子力研究開発機構）– Oakforest-PACSにおける大規模CFDコードの計算技術開発•

16:00 - 16:30 八代 尚（理化学研究所計算科学研究機構）– 『超』並列時代の気象・気候シミュレーション：メニーコアは怖くないぞ！

•

16:30 - 17:00 渡邊啓正（HPCシステムズ株式会社）– Oakforest-PACS上でのFMO計算プログラムABINIT-MPの性能評価•

17:10 - 17:40 講演者，センター教員– 質疑応答•17:40 - 17:45 朴泰祐（JCAHPC/筑波大学計算科学研究センター）– 閉会 （終了後希望者のみでOakforest-PACSを見学）

36

各アプリ最適化• SALMON，Lattice QCD，GAMERA/GHYDRA– OFP全系での稼働実績あり，ピーク性能の12-16%を達成済# Middle layer processes
1
2
4
8
16 32

• z-Pares

Speed-up– 筑波大学 櫻井鉄也教授– 積分型疎行列固有値解析エンジン– 1万原子系の複素バンド構造計算– 3300万次元の非線形固有値問題– OFPの2048ノードまでを使用，高いスケーラビリティを確認– SC17のPaperに採択

32

Ideal
Solve linear equations
Total

16
8

# Top
layer processes: 16# Bottom layer processes: 64

4
2
1
64

128 256 512 1024 2048
# Processors
37

Atmosphere-Ocean Coupling38

on OFP by NICAM/COCO/ppOpen-MATH/MP•

High-resolution global atmosphere-ocean coupledsimulation by CAM and COCO (Ocean imulation) throughppOpen-MA H/MP on the K computer is achieved.
– ppOpen-MA H/MP is a coupling software for the models
employing various discretization method.

•

An O(km)-mesh CAM-COCO coupled simulation isplanned on the Oakforest-PAC system (3.3km-0.10deg.,
3+B Meshes).
– A big challenge for optimization of the codes on new ntel
Xeon Phi processor– ew insights for understanding of global climate dynamics[C/O M. atoh (AOR /U okyo)@ C16]

NICAM-COCO連成
東京大学(東大)は9月20日、雲の生成・消滅を詳細に計算できる全球大気モデルNICAMに、全球気候モデルMIROCの海洋部分を連結させた気象-海洋モデル「NICAM-COCO(NICOCO)」を開発し、熱帯域を東進する巨大な雲群マッデン＝ジュリアン
振動(MJ0)と、東太平洋の海面温度が通常より高くなるエルニーニョ現象との相互作用の再現を可能にしたと発表した。
NICAM-COCOのカプラー：
ppOpen-MATH/MG• 現状：全球大気14 km・全球海洋0.25-1.00°
メッシュ（京）• OFP：大気3.5km・海洋0.10°（従来の20倍超，
39
50億メッシュ）
201 年 月20日 日経Earthquake Simulations
Prof. Ichimura (ERI, U.Tokyo)• GOJIRA/GAMERA
FEM with Tetrahedral
Elements (2nd Order)
Nonlinear/Linear,
Dynamic/Static Solid
Mechanics
Mixed Precision, EBEbased Multigrid
SC14, SC15: Gordon Bell
Finalist
SC16: Best Poster• GHYDRA
Time-Parallel Algorithm
Oakforest-PACS (on-going)

Simulation example: Earthquake simulation of
10.25 km x 9.25 km area of central Tokyo
using full K computer. Response of 328
thousand buildings are evaluated using threedimensional ground data and building data.
Analyzed using a 133 billion degrees-offreedom nonlinear finite-element model.

40

41

Earthquake Simulations
Prof. Ichimura (ERI, U.Tokyo)• GAMERA– multi-Grid method Adaptive conjugate gradient method,
Multi-precision arithmetic, Element-by-element method,pRedictor with Adams-bashworth method• GHYDRA– Great-HYDRA– HYbird tempo-spatial-arithmetic multi-griD solveR withconcentrated computAtionGAMERA• Poisson 3D
ICCG for FVM (7-pt)• GeoFEM/Cube Matrix Assembly for FEM• GeoFEM/Heat
CG with SELL-C-σ• Dynamic Loop Scheduling of Parallel FEM[ICPP 2017]

42

43

Poisson3D-OMP• Finite Volume Method, Poisson Equations (1283 cells)– FDM-type mesh (7-pt. Stencil), Unstructured data structure– SPD matrix– ICCG: Data Dependency for Incomplete Cholesky Fact.

• Fortran 90 + OpenMP• Thread Parallelization by OpenMP: Reordering
needed– CM-RCM + Coalesced/Sequential• Storage of Matrix– CRS, ELL (Ellpack-Itpack)• Outer Loops

z

∆Z

NZ
y

NY

– Row-Wise, Column-Wise
x

NX

∆X

∆Y

44

Code Name
Architecture
Frequency(GHz)
Core # (MaxThread #)Peak
Performance(GFLOPS)

BDW
KNC
KNL-0
Intel Xeon Phi Intel Xeon Phi Intel Xeon E57210
2695 v4
5110P(Knights(Knights(BroadwellCorner)
Landing)
EP)
1.053

1.30

2.10

60 (240)64 (256)

18 (18)1,010.9

2,662.4

604.8

8

MCDRAM: 16
DDR4: 96

128

MemoryBandwidth(GB/sec., Stream
Triad)159

MCDRAM:454
DDR4: 72.5

65.5

Out-of-Order

N

Y

Y

OFP-mini

Reedbush-U

Memory (GB)SystemComp. Time for ICCG (Best ELL)

45

Effects of synchronization overhead are significant on
KNL & KNC, if number of colors is larger
Generally, optimum number of color is 10 for KNL/KNC
Down is Good !! • KNC/KNL6.00

KNC
BDW
KNL-DDR4
KNL-MCDRAM

5.00

sec.

4.00
3.00
2.00
1.00
0.00
1

10

Color#〔中島他 HPC-15７，2016〕

100

MCDRAM比は妥当• メモリバンド幅を考慮するとKNC，
KNL-MCDRAMは
この倍の性能が出ても良い（現在ピークの1%）• BDW，KNL-DDR4
はTriad並にメモリ
バンド幅使い切り• 7-ptステンシルなので性能は出にくいComp. Time for ICCG (Best ELL)

46

Effects of synchronization overhead are significant on
KNL & KNC, if number of colors is larger
Generally, optimum number of color is 10 for KNL/KNC
Down is Good !!
8.00

6.00

sec.

• FX10は60GB/sec
位出ている@20色

KNC
BDW
KNL-0:DDR4
KNL-0:MCDRAM
FX10

4.00

2.00

0.00
1

10

Color#

100〔中島他 HPC-15７，2016〕

47

Code Name
Architecture
Frequency(GHz)
Core # (MaxThread #)Peak
Performance(GFLOPS)
Memory (GB)MemoryBandwidth(GB/sec., Stream
Triad)P100

KNCIntel XeonPhi 5110P(Knights
Corner)KNL-0Intel Xeon
Phi 7210(Knights
Landing)

1.053

1.30

1.328

60 (240)64 (256)

1,792

1,010.9

2,662.4

4,759

8

MCDRAM:
16
DDR4: 96

16

159

MCDRAM:454
DDR4: 72.5

530

NVIDIA Tesla
P100(Pascal)48

Comp. Time for ICCG (Best ELL)
Xeon Phi (KNL) and P100 are competitive
Down is Good !!
6.00

KNL-0:DDR4:BR-1
KNL-0:MCDRAM:BR-1
NVIDIA P100: AR-1

5.00

sec.

4.00
3.00
2.00
1.00
0.00
1

10

Color#

100

[Kreutzer, Hager, Wellein 2014] 49

50

Effects of SELL-C-σ on KNL-0-MC/D
Improvement over Original Best ELL Cases (2-10 colors)
SELL-C-σ is rather slower on KNC
10.00

Performance Gain (%)

K C

K L-0-MCDRAM

5.00

0.00
2

3

4

5

6

(5.00)(10.00)Colors

7

8

9

10

51

Further Reduction of OpenMP
Overhead on KNL
星野他[Hoshino et al. 2017]
HPC-158
2017• NO Wait for SpMV• Call $OMP PARALLEL only once at Each Iteration• Remove $OMP DO• Remove REDUCTION: Dot products are calculated
at each thread• Performance Improvement on SELL-8-1– KNL: 16.7 %, KNC: 25.0 %• Finally, KNL-MCDRAM is 3.6x faster than KNLDDR4– Room for further performance engineering– 1% of peak (e.g. HPCG (27-pt stencil): 1.5% of peak)– Throughput: 70+%

• Poisson 3D
ICCG for FVM (7-pt)• GeoFEM/Cube Matrix Assembly for FEM• GeoFEM/Heat
CG with SELL-C-σ• Dynamic Loop Scheduling of Parallel FEM[ICPP 2017]
•

52

53

Matrix Assembly in FEM• Integration on each Element: Element Matrix (Dense)• Accumulation of Elem. Mat: Global Matrix (Sparse)• Integration/Matrix: Element
13
14
15
16
• Matrix Computation: Node789
– Components of each Node:
Accumulation by Neighboring
Elements.
– Data dependency may happen– Coloring needed: Multicoloring

9

10

4
5

5
6

1
1

11

6
7

2
2

12

8

3
3

4

• Hardware– Intel Xeon Phi (KNC/KNL)– NVIDIA Tesla K40/P100• Atomic OP’s supported by H/W[KN, Naruse et al. HPC-152, 2015]

54

GeoFEM/Cube• Parallel FEM Code (& Benchmarks)• 3D-Static-Elastic-Linear (Solid Mechanics)• Performance of Parallel Preconditioned Iterativez
Solvers– 3D Tri-linear Hexa. Elements
U =0 @ y=Y– SPD matrices: CG solver– Fortran90+MPI+OpenMP(N -1) elements– Distributed Data Structure
N nodes– Reordering for IC/SGS Precond.
yUniform Distributed Force inz-direction @ z=Zmax

min

Ux=0 @ x=Xminzz• MC, RCM, CM-RCM– MPI，OpenMP，OpenMP/MPI Hybridx(Ny-1) elements
Ny nodesy
Uz=0 @ z=Zmin(Nx-1) elements
Nx nodes• only OpenMP case

• Focusing on Matrix Assembling in this Study: 1283 cubes

55

Type-ABLKSIZ: Elem.# in Blocks, NBLK: Block #, icel: Elem. ID!$omp parallel (…)
do color= 1, COLORtot!$omp dodo ip= 1, THREAD_total
NBLK: calculated by (col_index, color, thread#)
do ib= 1, NBLK
do blk= 1, BLKSIZ
icel: calculated by (col_index, ib, blk)!$omp simd
do ie= 1, 8; do je= 1, 8＜② Address in Sparse Matrix＞
enddo; enddo
enddo
do blk= 1, BLKSIZ
icel: calculated by (col_index, ib, blk)＜① Jacobian & Grad. Shape Fn’s＞!$omp simd
do ie= 1, 8; do je= 1, 8＜③ Element Matrices ＞
enddo; enddo
enddo
do blk= 1, BLKSIZ
icel: calculated by (col_index, ib, blk)!$omp simd
do ie= 1, 8; do je= 1, 8＜④ Global Matrices＞
enddo; enddo
enddo
enddo
enddo
enddo
!$omp end parallel

56

Code Name
Architecture
Frequency(GHz)
Core # (MaxThread #)Peak
Performance(GFLOPS)
Memory (GB)MemoryBandwidth(GB/sec., Stream
Triad)K40

P100

NVIDIA Tesla
K40

NVIDIA Tesla
P100(Pascal)

1.40

0.745

1.328

60 (240)68 (272)

2,800

1,792

1,010.9

3,046.4

1,430

4,759

8

MCDRAM:
16
DDR4: 96

12

16

159

MCDRAM:490
DDR4: 84.5

218

530

KNC
Intel XeonPhi 5110P(Knights
Corner)

KNL-1
Intel Xeon
Phi 7250(Knights
Landing)

1.053

Real OFP

57

Results (Fastest Case)sec. (GFLOPS)
Coloring +OpenMP/OpenACC

Atomic+OpenACCno SIMD: 1.339 (48.5)
SIMD: 0.690 (94.0)
no SIMD: 0.379 (171.1)
SIMD: 0.184 (352.5)

0.576 (112.7)
No Coloring
0.185 (351.5)
No coloring

K40

0.675 (96.2)

0.507 (128.1)

0.362 (179.4)

P100

0.210 (308.9)

0.190 (341.4)

0.154 (412.2)

KNC
KNL

Atomic +CUDA

Based on 〔KN, Naruse et al. HPC-152, 2015〕
Computation on KNL by Dr. Horikoshi (Intel)

• Poisson 3D
ICCG for FVM (7-pt)• GeoFEM/Cube Matrix Assembly for FEM• GeoFEM/Heat
CG with SELL-C-σ• Dynamic Loop Scheduling of Parallel FEM[ICPP 2017]

58

59

Storing Formats for Sparse Matrices• ELL/Sliced ELL: Excellent Performance by Prefetching• SELL-C-σ: Good for Vector/SIMD (by FAU/Erlangen)– This work is the first example where SELL-C-σ is applied to
IC/ILU preconditioning.
C
σ

CRS

ELL

Sliced ELL

SELL-C-σ(SELL-2-8)

60

SELL-C-σ for SpMV in GeoFEM/Heat
27-pt. Stencil，HPCGと似ている（1.5%@8,192nodes）• KNC (240 threads)– CRS
1.699 sec
– SELL-128-1 1.063 sec• 20GFLOPS• KNL-1 (192 threads)– CRS
0.421 sec
– SELL-128-1 0.384 sec• 55GFLOPS– read 350/400 GB/sec.

• miniFEM (SNL): 42GF

• Generally, difference
between CRS & ELL issmaller on KNL (BDW)– Out-of-Order!$omp parallel do private (j,k)
do j= 1, NWW(j,Q)= D(j)*WW(j,P)
enddo!$omp parallel do private (j,k)
do j= 1, N
do k= INL(j-1)+1, INL(j)WW(j,Q)= WW(j,Q) + AL(k)*WW(IAL(k),P)
enddo
enddo

!$omp parallel dodo i= 1, N/2
WW(2*(i-1)+1,Q)= D(2*(i-1)+1) * WW(2*(i-1)+1,P)
WW(2*(i-1)+2,Q)= D(2*(i-1)+2) * WW(2*(i-1)+2,P)
enddo!$omp parallel dodo i= 1, Ndiv
do j= 1, NCOL!$omp simd
do k= 1, 128
WW(128*(i-1)+k,Q)= WW(128*(i-1)+k,Q) +&AL(sell_CS(i-1)+(j-1)*128+k)&* WW(IAL(sell_CS(i-1)+(j-1)*128+k),P)
enddo
enddo
enddo

〔中島他 HPC-151，2015〕に基づく

&

Intro

• Poisson 3D
ICCG for FVM (7-pt)• GeoFEM/Cube Matrix Assembly for FEM• GeoFEM/Heat
CG with SELL-C-σ• Dynamic Loop Scheduling of Parallel FEM[ICPP 2017]

61

62

Comm.-Comp. Overlapping(CC-Overlapping): Static
Pure Internal Meshes
External (HALO) Meshes
Internal Meshes on
Boundary’s(Boundary Meshes)call MPI_Isend
call MPI_Irecv
do i= 1, Ninn
(calculations)enddo
call MPI_Waitall

Good for Stencil
Not so Effective SpMV

do i= Ninn+1, Nall(calculationas)
enddoCC-Overlapping

63

Comm.-Comp. Overlapping+ Dynamic Loop Scheduling: Dynamic
Pure Internal Meshes
External (HALO) Meshes
Internal Meshes on
Boundary’s(Boundary Meshes)call MPI_Isend
call MPI_Irecv
call MPI_Waitall

Masterdo i= 1, Ninn
(calculations)enddo

Dynamic

do i= Ninn+1, Nall(calculationas)
enddo

Static64

Dynamic Loop Scheduling• “dynamic”• “!$omp master～!$omp end master”!$omp parallel private (neib,j,k,i,X1,X2,X3,WVAL1,WVAL2,WVAL3)!$omp&
private (istart,inum,ii,ierr)!$omp masterCommunication is done by the master thread (#0)!C
!C– Send & Recv.
(…)
call MPI_WAITALL (2*NEIBPETOT, req1, sta1, ierr)!$omp end master!C!C-- Pure Internal Nodes

The master thread can join computing of internal
nodes after the completion of communication!$omp do schedule (dynamic,200)
Chunk Size= 200
do j= 1, Ninn
(…)
enddo
!C
!C-- Boundary Nodes
Computing for boundary nodes are by all threads!$omp dodo j= Ninn+1, N
(…)
enddo

default: !$omp do schedule (static)!$omp end parallelIna, T., Asahi, Y., Idomura, Y., Development of optimization of
stencil calculation on Tera-flops many-core architecture, IPSJ SIG
Technical Reports 2015-HPC-152-10, 2015 (in Japanese)

65

GeoFEM/Cube• Parallel FEM Code (&U =0 @ y=Y
Benchmarks)• 3D-Static-Elastic-Linear (Solid
Mechanics)(N -1) elements
N nodes• Performance of Parallel
Preconditioned Iterative Solversyz
Uniform Distributed Force inz-direction @ z=Zmax

min

Ux=0 @ x=Xminzz– 3D Tri-linear Hexahedral
Elements– Block Diagonal LU + CG
– Fortran90+MPI+OpenMP i– Distributed Data Structure– MPI，OpenMP，
OpenMP/MPI Hybrid– Block CRS Format(Ny-1) elements
Ny nodesy
Uz=0 @ z=Zmin(Nx-1) elements
Nx nodesxj(− 1,+1,+1) 8(− 1,−1,+1) 5

7 (+ 1,+1,+1)
6 (+ 1,−1,+1)(− 1,+1,−1)
4
1(ξ ,η , ζ ) = (− 1,−1,−1)

ai j
 e e 11
 a ie je 21a ie je 31

a ie je12
ai je e 22
aie je 32

2

3 (+ 1,+1,−1)(+ 1,−1,−1)

aie je 13 aii j  (ie , je = 1K8)
e e 23
aie je 33 

66

Code Name
Architecture
Frequency(GHz)
Core # (MaxThread #)Peak
Performance(GFLOPS)

BDW
FX10
KNL
Intel Xeon Phi Intel Xeon E52695 v4
7250
SPARC IX fx(Knights(BroadwellLanding)
EP)
1.40

2.10

1.848

68 (272)

18 (18)16 (16)

3,046.4

604.8

236.5

Memory (GB)MCDRAM: 16
DDR4: 96

128

32

MemoryBandwidth(GB/sec., Stream
Triad)MCDRAM:490
DDR4: 80.1

65.5

64.7

Y

N

Reedbush-U

Oakleaf-FX

Out-of-Order
SystemY
OakforestPACS

67

Code Name

BDW

FX10

Intel Xeon E52695 v4(Broadwell-EP)

SPARC IX fx

1.40

2.10

1.848

68 (272)

18 (18)16 (16)Peak
Performance(GFLOPS)/core

44.8

33.6

14.8

MemoryBandwidth(GB/sec., Stream
Triad)/core

MCDRAM:
7.21
DDR4: 1.24

3.64

4.04

Y
Omni-Path
Architecture

Y
Mellanox EDR
Infiniband

NTofu
6D Torus

Architecture

Frequency (GHz)Core # (MaxThread #)Out-of-Order
Network

KNLIntel Xeon Phi
7250(Knights
Landing)

FX10: 240 nodes, 368,640,000 DOF

68

HB 16x1, Performance Analysis by Fujitsu’s Profiler(single node)
Original Static

Dynamic:
Chunk
Size=100

Dynamic:
Chunk
Size=500

GFLOPS/node
Memory Throughput(GB/sec)
L2 Throughput(GB/sec)

12.59

13.33

14.47

13.82

61.61

64.86

69.44

68.07

71.13

75.03

84.15

79.03

sec./(100 iterations)

2.21

2.10

1.93

2.00

.229

.122

.073

.061

.655

.657

.540

.614

Synchronous waitingtime between threads(sec) Averaged
L2 waiting for FP
Load (sec) Averaged

69

3,840 cores, PA Profiler
FX10: 240 nodes, 368,640,000 DOF“Original”: 2.21 sec.
■ Synchronization Waiting, ■ L2 Load
No instruction commit due to memory access for an integer load instruction
No instruction commit because SP(store port) is full
No instruction commit due to L2 cache for a floating-point load instruction
No instruction commit waiting for a floating-point instruction to be completed
No instruction commit waiting for an instruction to be fetched
No instruction commit waiting for a micro-operation to be completed[sec]
2.5E+00

No instruction commit due to memory access for a floating-point load instruction
No instruction commit due to L2 cache access for an integer load instruction
No instruction commit waiting for an integer instruction to be completed
No instruction commit waiting for a branch instruction to be completed
Synchronous waiting time between threads
No instruction commit for other reasons

2.0E+00

1.5E+00

1.0E+00

5.0E-01

0.0E+00
Thread 0-5.0E-01

Thread 1

Thread 2

Thread 3

Thread 4

Thread 5

Thread 6

Thread 7

Thread 8

Thread 9

Thread 10

Thread 11

Thread 12

Thread 13

Thread 14

Thread 15

703,840 cores, PA Profiler
FX10: 240 nodes, 368,640,000 DOF“Static”: 2.10 sec.
■ Synchronization Waiting, ■ L2 Load
No instruction commit due to memory access for an integer load instruction
No instruction commit because SP(store port) is full
No instruction commit due to L2 cache for a floating-point load instruction
No instruction commit waiting for a floating-point instruction to be completed
No instruction commit waiting for an instruction to be fetched
No instruction commit waiting for a micro-operation to be completed[sec]
2.5E+00

No instruction commit due to memory access for a floating-point load instruction
No instruction commit due to L2 cache access for an integer load instruction
No instruction commit waiting for an integer instruction to be completed
No instruction commit waiting for a branch instruction to be completed
Synchronous waiting time between threads
No instruction commit for other reasons

2.0E+00

1.5E+00

1.0E+00

5.0E-01

0.0E+00
Thread 0-5.0E-01

Thread 1

Thread 2

Thread 3

Thread 4

Thread 5

Thread 6

Thread 7

Thread 8

Thread 9

Thread 10

Thread 11

Thread 12

Thread 13

Thread 14

Thread 15

713,840 cores, PA Profiler
FX10: 240 nodes, 368,640,000 DOF“Dynamic, Csz=100”: 1.93 sec.
■ Synchronization Waiting, ■ L2 Load[sec]
2.5E+00

2.0E+00

No instruction commit due to memory access for an integer load instruction
No instruction commit because SP(store port) is full
No instruction commit due to L2 cache for a floating-point load instruction
No instruction commit waiting for a floating-point instruction to be completed
No instruction commit waiting for an instruction to be fetched
No instruction commit waiting for a micro-operation to be completed
One instruction commit
Two or three instructions commit for other reasons

No instruction commit due to memory access for a floating-point load instruction
No instruction commit due to L2 cache access for an integer load instruction
No instruction commit waiting for an integer instruction to be completed
No instruction commit waiting for a branch instruction to be completed
Synchronous waiting time between threads
No instruction commit for other reasons
Two or three instructions commit due to the number of GPR write ports
Four instructions commit

1.5E+00

1.0E+00

5.0E-01

0.0E+00
Thread 0-5.0E-01

Thread 1

Thread 2

Thread 3

Thread 4

Thread 5

Thread 6

Thread 7

Thread 8

Thread 9

Thread 10

Thread 11

Thread 12

Thread 13

Thread 14

Thread 15

723,840 cores, PA Profiler
FX10: 240 nodes, 368,640,000 DOF“Dynamic, Csz=500”: 2.00 sec.
■ Synchronization Waiting, ■ L2 Load
No instruction commit due to memory access for an integer load instruction
No instruction commit because SP(store port) is full
No instruction commit due to L2 cache for a floating-point load instruction
No instruction commit waiting for a floating-point instruction to be completed
No instruction commit waiting for an instruction to be fetched
No instruction commit waiting for a micro-operation to be completed[sec]
2.5E+00

No instruction commit due to memory access for a floating-point load instruction
No instruction commit due to L2 cache access for an integer load instruction
No instruction commit waiting for an integer instruction to be completed
No instruction commit waiting for a branch instruction to be completed
Synchronous waiting time between threads
No instruction commit for other reasons

2.0E+00

1.5E+00

1.0E+00

5.0E-01

0.0E+00
Thread 0-5.0E-01

Thread 1

Thread 2

Thread 3

Thread 4

Thread 5

Thread 6

Thread 7

Thread 8

Thread 9

Thread 10

Thread 11

Thread 12

Thread 13

Thread 14

Thread 15

Preliminary Results: Best Cases
3,840 cores, 368,640,000 DOF
Improvement of CG from Original Cases
FX10: HB 16x1

BDW: HB 8x2

KNL: HB 64x2 (2T)25.00

Speed-Up (%)

20.00
15.00
10.00
5.00
0.00
-5.00
-10.0073Preliminary Results: Original Cases
3,840 cores, 368,640,000 DOF
Communication Overhead by Collective/Point-to-Point
Communications
Rest

Send/Recv

Allreduce

FX10: HB 16x1

BDW: HB 8x2

KNL: HB 64x2
(2T)

7.5%

4.0%

14.6%

100%
80%
60%
40%
20%
0%

74

75

Features
Effect of Optimum
Dynamic
Chunk
Size
Scheduling
FX10
BDW

KNL

Medium
Small

Large

Notes

100

Memory Throughput

500+

Low Comm. Overhead
Small number of threads.

Effects are significant for
HB 64x2, 128x1, whereloss of performance by
300-500
communications onmaster thread is rather
smaller.

HB 2x8: Csz=500
HB 16x1: Csz=500

HB 4x4: Csz=100

FX10

100.0

Parallel Performance (%)

Parallel Performance(%)
BEST case for each
HB MxN
50,331,648 DOF
Computation Time of Flat
MPI at Min.# cores:
100%

120.0

80.0
60.0
40.0
20.0
0.0

32
120.0

Parallel Performance (%)

Strong Scaling

Flat MPI: Original
HB 8x2: Csz=100

64

128

Flat MPI: Static
HB 8x2: Csz=100

256

512

1024

2048

4096

HB Total
2x8: Csz=500
Core #
HB 16x1: Csz=500

8192

16384

HB 4x4: Csz=500

100.0
80.0
60.0
40.0

BDW

20.0
0.0

32

64

Flat MPI: Original
HB 16x4: Csz=500

128

256

512

1024

2048

Total Core#
HB 2x32: Csz=500
HB 4x16:
Csz=100
HB 32x2: Csz=500
HB 64x1: Csz=500

4096

8192

HB 8x8: Csz=100

120.0

Parallel Performance (%)

This difference between
BDW and KNL might bebecause difference ofperformance between
Infiniband EDR and OmniPath Architecture.

KNL

100.0
80.0
60.0
40.0
20.0
0.0

256

512

1024

2048
4096
Total Core #8192

16384 76

HB 16x1: Original

Strong Scaling

Parallel Performance (%)

Parallel Performance(%)
Effect of Dynamic Loop
Scheduling
50,331,648 DOF
Computation Time of Flat
MPI at Min.# cores:
100%

80.0
60.0
40.0
20.0

FX10: HB 16x1
32

64

128

256

HB 8x2: Original

512
1024
2048
4096
Total Core #Static
Csz=100

8192

16384

Csz=500

Parallel Performance (%)

120.0
100.0
80.0
60.0
40.0
20.0

BDW: HB 8x2

0.0

32

64

128

256

512
1024
2048
Total Core #Static
Csz=100

4096

8192

Csz=500

120.0

Parallel Performance (%)

IdomuraCsz=500

100.0

HB 64x1/1T: Original

FX10: 20%-40%
BDW: 6%-10%
KNL with HB 8×8 (1T): 20%-30%
KNL with HB 64×1 (1T): 40%-50%

Csz=100

0.0

Effect of Dynamic Loop Schedulingwith more than 8,192 cores••••

Static

120.0

100.0
80.0
60.0
40.0
20.0

KNL: HB 64x1 (1T)
77

0.0

256

512

1024

2048

4096

8192

16384

まとめ
• Oakforest-PACSの概要• Oakforest-PACSを使用したアプリケーション事例・最適化
事例• 今後– 更なる最適化– SC18 Gordon Bellへのチャレンジ– Deep Learning等への適用• SC17 Technical Paper, Cori/LBNLを使用した事例, IntelCaffe• Deep Learning at 15PF: Supervised and Semi-Supervised
Classification for Scientific Data

おまけ78

SIAM Conference on

Thanks for Your
Contributions !!

Parallel Processing for
Scientific Computing (PP18)
March 7-10, 2018
Waseda University, Tokyo, Japan

Record Breaking 126 MS Proposals, 100+ CP
700+ Participants ? Please book your Hotel ASAP !
• Organizing Committee Co-Chairs’s– Satoshi Matsuoka (Tokyo Institute of Technology, Japan)– Kengo Nakajima (The University of Tokyo, Japan)– Olaf Schenk (Universita della Svizzera italiana, Switzerland)• Contact– Kengo Nakajima, nakajima(at)cc.u-tokyo.ac.jp– http://www.siam.org/meetings/pp18/– http://siampp18.jsiam.org/