2018/7/10

KNL

1

102  
 
KNL






  
hanawa  cc.u-tokyo.ac.jp

2018/7/10

KNL2• (H2018 710EF 13H30 - 18H00• H7C8D 4) 413&*"
• %H • %#>B4@?H• 13H00 - 13H30

• 13H30 - 14H30 Oakforest-PACSB43C+Oakforest-PACS56:?!

(14H30 - 14H45 )• 14H45 - 16H15 KNL$+ KNL0,-2OpenMP E%"G#F(16H15 - 16H30 )• 16H30 - 18H00 Oakforest-PACS.1MPI +Oakforest-PACS.1<3=A9; /' (%"+#)2018/7/10

3

KNL

Oakforest-PACS
Oakforest-PACS

2018/7/10

KNL
•






 : t00271
 gt00
8/10 17:00



4

2018/7/10




KNL

52018/7/10

;KNL 

6 FY 2 6
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Yayoi: Hitachi SR16000/M1
IBM Power-7

2-8'&
*8.8'6/38+
9#! "$!;7 :

54.9 TFLOPS, 11.2 TB

Oakforest-PACS
Fujitsu, Intel KNL

T2K Tokyo

140TF, 31.3TB

Big Data &Extreme Computing

25PFLOPS, 919.3TB

Oakleaf-FX: Fujitsu PRIMEHPC
FX10, SPARC64 IXfx

BDEC System

50+ PFLOPS (?)

1.13 PFLOPS, 150 TB

Oakbridge-FX

Oakbridge-II

136.2 TFLOPS, 18.4 TB,8+ 7(1358(46*8.8'6/38+
 )40 
%*8.8'6/38+

5-10 PFLOPS

Reedbush, HPE
Broadwell + Pascal
1.93 PFLOPS

Reedbush-L
HPE
1.43 PFLOPS 
*8.8'6/38+

2018/7/10

KNL

2 4 • Oakleaf-FX (  PRIMEHPC FX10)• 1.135 PF, &6/38*, 2012 4  2018 3
• Oakbridge-FX (  PRIMEHPC FX10)• 136.2 TF, ! :168!;, 2014 4  2018 3
• Reedbush (HPE, Intel BDW + NVIDIA P100 (Pascal))• -8*7'0358'46(8.8&6/38*• 2016-Jun.20167<2020 6• )"GPU'(,1• Reedbush-U: CPU only, 420 nodes, 508 TF (2016 7)• Reedbush-H: 120 nodes, 2 GPUs/node: 1.42 PF (2017 3)• Reedbush-L: 64 nodes, 4 GPUs/node: 1.43 PF (2017 10)• Oakforest-PACS (OFP) ( , Intel Xeon Phi (KNL))• JCAHPC ( CCS9 ITC)
• 25 PF, 9 (2017 11) (2)• Omni-Path #8$,%+2, DDN IME (Burst Buffer)7

2018/7/10

 

KNL • $%#"• • •  
!

8



•



http://www.cc.u-tokyo.ac.jp/system/ofp/
http://www.cc.u-tokyo.ac.jp/system/reedbush/2018/7/10

KNL

Oakforest-PACS

9

2018/7/10

10

10

KNL

Oakforest-PACS*(JCAHPC: Joint Center for
Advanced High Performance Computing)• !HPC
• 
• "


:?;A
)# :?;A

• :?;A/2.!35143%&•  7>?<93(-')#6$@+

0

:?;A

• 2016121,
• 8,208 Intel Xeon/Phi (KNL)• =A8'25PFLOPS• TOP 500

12B 2CDHPCG 7BC B20186C

2

2018/7/10

KNL

Oakforest-PACS (1 / 2)
•  &2$• 1&2$ 686
3TFLOPS8,208&2$7 25
PFLOPS• +,.3MCDRAM36
16GB45DDR436
96GB44• &2$
• (/'!-1'1$
Fat-Tree%"#02•  
*.2-1 6 -)
• Intel Omni-Path Architecture1111

2018/7/10

!KNL

12

Oakforest-PACS 2 / 2• -"#1I/O• -"#1&'*.:
Lustre 26PB
• -"#1$/)&0&'*.
4DDN IME56
1TB/sec!
 ,
1PB
• 2,)%+3(
2

• 
• Green 5006
• Linpack6 2.72 MW• 4,986 MFLOPS/W4OFP5•830 MFLOPS/W45

 120 
 

12

2018/7/10

KNL

13

Oakforest-PACS 
'2  %2$
Product

%2$
)0!"

25 PFLOPS
8,208

 PRIMERGY CX600 M1 (2U) +CX1640 M1 x 8node
Intel® Xeon Phi™ 7250
3 2$: Knights Landing468 1.4 GHz+ &1$ 16 GB, MCDRAM,  490 GB/sec, &1$ 96 GB, DDR4-2400, '2 115.2.
GB/secIntel® Omni-Path Architecture Product
100 Gbps.1
#*0(/&!-1&1$Fat-tree2018/7/10

KNL

14

Oakforest-PACS 
 Type%

Lustre File System
26.2 PB
Product
DataDirect Networks SFA14KE
& 500 GB/sec
Burst Buffer, Infinite Memory Engine  Type(by DDN)
%!
"

940 TB (NVMe SSD, $)Product
DataDirect Networks IME14K
& 1,560 GB/sec#

4.2MW'(102

2018/7/10

KNL

15

Oakforest-PACS 
• OS: Red Hat Enterprise Linux (dJFeUgS)+

CentOS /A= McKernel (!UgS+ )• McKernel: AICS8(<^TgLE

2OS• Linux;>$'+`gMZdJb];.B):5• [ORLeW_gQ;?%4CB,• LeVFbhGCC, Intel Compiler, XcalableMP• XcalableMP: AICS9 8 (<ZdJb\eJ #• C@Fortran8"&4C7LgS;D .B398+
<*-EZcKgNaeD;(6B390
81B,• bFYbcfEZcKgNaeh IgZePgOPXRGH
E• ppOpen-HPC, OpenFOAM, ABINIT-MP, PHASE system,

FrontFlow/blue，LAPACK, ScaLAPACK, PETSc, METIS, SuperLU etc.

2018/7/10

16

KNL

Oakforest-PACS: Intel Omni-Path Architecture      Fat-tree
768 port Director
Switch12(Source by Intel)

22Uplink: 24

48 port Edge Switch
362 
Downlink: 24
1

...

24

25

...

48

49

...

72

5>4
;1+#

,/3%(7<6*0+-:=6=4 )
• -/29#& )• "?.:8# (5>4' !$2018/7/10

17

 KNL

Knights Landing: Next Intel® Xeon Phi™ Processor

Oakforest-PACS 
Intel® Many-Core Processor targeted for HPC and SupercomputingFirst self-boot Intel® Xeon Phi™ processor that is binary
compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance• Intel XeonIntegration
Phi of(Knights
Landing)Memory on package:innovative memoryarchitecture for high bandwidth and high capacity• 11
Integration of Fabric on package
Three products• MCDRAM: KNL Self-Boot

KNL Self-Boot w/ Fabric

KNL Card(Fabric Integrated)(PCIe-Card)16GB+ DDR4(Baseline)

HotChips27
Knights Landing
Overview
KNL 
MCDRAMMCDRAM

2 x16
1 x4

X4
DMI

PCIe
Gen 3

D
M
I

MCDRAMMCDRAM

Potential future options subject to change without notice.
All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

EDC


:
16GB 696GB

3

Core

C
H
A
N
N
E
L
S

2 VPU
1MB
L2

MCDRAM

4

EDC

3
D
D
R
4

36 Tilesconnected by
2D MeshInterconnect

DDR MC

EDC

Core

EDC

Tile

D
D
R
4

MCDRAM: 490GB/ DDR4: 115.2 GB/=(8Byte 2400MHz 6 channel)
2 VPU

EDC

EDC

MCDRAM

Omni-path not shown

misc

Package

DDR MCEDC

MCDRAM

C
H
A
N
N
E
L
S

EDC

MCDRAM2018/7/10

KNL

18
• TOP 500CLinpack, HPLD
• & 8?9BCDE"% CFLOPSD• !+ F&;<>1265• "
• HPCG• & 8?9BCDE"% CFLOPSD• ' ). .0/C7@*(D• &;<>1265•
1:>3B4=A,#(• ;<>1265E$
• Green 500• HPLCTOP500D -FLOPS/W2018/7/10

19

KNL

48th TOP500 List (November, 2016)
Site

Computer/Year Vendor

Sunway TaihuLight , Sunway MPP,
1
Sunway SW26010 260C 1.45GHz,
2016 NRCPC
Tianhe-2, Intel Xeon E5-2692, TH
National Supercomputing
2
Express-2, Xeon Phi, 2013 NUDT
Center in Tianjin, China
Titan
Oak Ridge National
3
Cray XK7/NVIDIA K20x, 2012 Cray
Laboratory, USA
Lawrence Livermore National Sequoia
4
BlueGene/Q, 2011 IBM
Laboratory, USA
DOE/SC/LBNL/NERSC
Cori, Cray XC40, Intel Xeon Phi 7250
5USA
68C 1.4GHz, Cray Aries, 2016 Cray
Oakforest-PACS, PRIMERGY CX600
Joint Center for Advanced
M1, Intel Xeon Phi Processor 7250 68C
6 High Performance
1.4GHz, Intel Omni-Path,
Computing, Japan
2016 Fujitsu
K computer, SPARC64 VIIIfx , 2011
7 RIKEN AICS, Japan
Fujitsu
Piz Daint
Swiss Natl. Supercomputer
8
Cray XC30/NVIDIA P100, 2013 Cray
Center, Switzerland
Argonne National Laboratory, Mira
9
BlueGene/Q, 2012 IBM
USA
Trinity, Cray XC40, Xeon E5-2698v3
10 DOE/NNSA/LANL/SNL, USA
16C 2.3GHz, 2016 Cray
National Supercomputing
Center in Wuxi, China

Cores

Rmax
Rpeak
Power
(TFLOPS) (TFLOPS) (kW)10,649,600

93,015(= 93.0 PF)

125,436

15,371

3,120,000

33,863(= 33.9 PF)

54,902

17,808560,640

17,59027,1138,209

1,572,864

17,173

20,133

7,890

632,400

14,01527,881

3,939

557,056

13,55524,9142,719

705,024

10,51011,28012,660206,720

9,779

15,9881,312

786,432

8,587

10,066

3,945

301,056

8,101

11,0794,233

Rmax: Performance of Linpack (TFLOPS)
Rpeak: Peak Performance (TFLOPS), Power: kW

http://www.top500.org/

2018/7/10

20

KNL

51st TOP500 List (ISC18, June 2018)
Site

Computer/Year Vendor

Cores

Rpeak
Power
Rmax
(PFLOPS) (PFLOPS) (MW)1

Oak Ridge National
Laboratory, USA

Summit,IBM P9 22C 3.07GHz, Mellanox
EDR, NVIDIA GV100, 2018 IBM

2,282,544

122.3

187.7

8.8

2

National Supercomputing
Center in Wuxi, China

Sunway TaihuLight , Sunway MPP,
Sunway SW26010 260C 1.45GHz, 2016
NRCPC

10,649,600

93.0

125.4

15.4

Sierra,IBM P9 22C 3.1GHz, Mellanox
1,572,480
71.6
119.1
EDR, NVIDIA GV100, 2018 IBM
Tianhe-2A, Intel Xeon E5-2692v2, TH
National Supercomputing
4
4,981,760
61.4
100.6
Express-2, Matrix-2000, 2018 NUDT
Center in Tianjin, China
AI Bridging Cloud Infrastructure
National Institute of
5 Advanced Industrial Science (ABCI) , Intel Xeon Gold 20C 2.4GHz, IB391,680
19.9
32.6
EDR, NVIDIA V100, 2018 Fujitsu
and Technology, Japan
Piz Daint, Cray XC50, Intel
Swiss National
6 Supercomputing Centre
Xeon E5 12C 2.6GHz, Aries, NVIDIA
361,760
19.6
25.3
Tesla P100, 2017 Cray
(CSCS) , Switzerland
Titan
Oak Ridge National
Oakforest-PACS:
12 27,1
7560,640
17.6
Cray XK7/NVIDIA K20x, 2012 Cray
Laboratory, USA 16
Sequoia
Lawrence Livermore
8
1,572,864
17.2
20,1
BlueGene/Q, 2011 IBM
National Laboratory, USA
Los Alamos NL / Sandia NL, Trinity, Cray XC40, , Intel Xeon Phi 7250
979,9689
14.1
43.9
68C 1.4GHz, Cray Aries, 2017 Cray
USA
DOE/SC/LBNL/NERSC
Cori, Cray XC40, Intel Xeon Phi 7250
632,400
10
14.0
27.9
USA
68C 1.4GHz, Cray Aries, 2016 Cray
3

Lawrence Livermore
National Laboratory, USA

Rmax: Performance of Linpack (PFLOPS)
Rpeak: Peak Performance (PFLOPS), Power: MW

http://www.top500.org/

18.5
1.65

2.27
8.21
7.89
3.84
3.94

2018/7/10

KNL

21 
• OFP@'7H</GXeon Phi 7250ASWL$I#0G,• MJ[ 68MJ
• MJ:FAAVX-512 UQNO[ 2• AVX-512UQNO:FA Y! Z[ 8•  (Fused Multiply Add: FMA) : 2 @
• LVNL : 1.40 GHz• RWP:FSWL$[

68 * 2 * 8 * 2 * 1.40 = 3046.4 GFLOPS• 828+AVX-512UQNOB B1.40GHz=B8?/
YDF/ Z
• SWL@(/$3EHGB9ACAYOFP=A Z• DGEMM (! A%Z: 2200 GFLOPS (SWL[72%)• HPL: 2000 GFLOPS (SWL[ 66%)
• Top500@15GOFPA*BSWLA54.4%• -SWL$.A "I848<146>3)&XX• CPUTWKW@D;< "AC?G2018/7/10

22

KNLHPCG Ranking (SC16, November, 2016)
Site
1 RIKEN AICS, Japan
2

NSCC / Guangzhou,
China

3 JCAHPC, Japan

Computer

TOP500Rank

HPCG(Pflop/s)

HPCG/
HPL (%)

705,024

10.51070.6027

5.73

Tianhe-2

3,120,000

33.86320.58001.71

557,056

13.555

6

0.3855

2.84

10,649,600

93.0151

0.3712.399

632,400

13.83250.3554

2.57

1,572,864

17.17340.3304

1.92

Titan

560,640

17.5903

0.3223

1.83

Trinity

301,0568.101

100.1826

2.25

Pleiades: SGI ICE X

243,0085.952

130.1752

2.94

Mira: IBM BlueGene/Q,

786,4328.587

90.16701.94

Oakforest-PACS

6 DOE/NNSA/LLNL, USA Sequoia

http://www.hpcg-benchmark.org/

HPL Rmax(Pflop/s)

K computer

National
4 Supercomputing Center Sunway TaihuLightin Wuxi, China
DOE/SC/LBNL/NERSC
Cori
5
USA
DOE/SC/
7 Oak Ridge National
Laboratory, USA
DOE/NNSA/
8 LANL/SNL,USA
NASA / Mountain View,
9
USA
DOE/SC/
10 Argonne National
Laboratory, USA

Cores

2018/7/10

KNL23

HPCG Ranking (June, 2018)
ComputerCores

HPL Rmax(Pflop/s)

TOP500Rank

HPCG HPL(Pflop/s) ratio

1 Summit

2,282,544

122.300

1

2.92582.4%

2 Sierra

1,572,480

71.6103

1.79572.5%

3 K computer

705,024

10.510160.6027

5.3%

4 Trinity

979,968

14.137

9

0.546

3.9%

5 Piz Daint

361,760

19.590

6

0.486

2.5%

6 Sunway TaihuLight

10,649,600

93.01520.48080.5%

7 Oakforest-PACS

557,056

13.555120.3855

2.8%

8 Cori

632,400

13.832100.3554

2.5%

9 Tera-1000-2

561,408

11.966140.33382.8%

1,572,864

17.17380.3304

2.8%

10 Sequoia
http://www.hpcg-benchmark.org/

2018/7/10

24

KNLGreen 500 Ranking (SC16, November, 2016)
Site

ComputerCPU

NVIDIA DGX-1, Xeon E5-2698v4 20C
2.2GHz, Infiniband EDR, NVIDIA Tesla
P100
Swiss National
Cray XC50, Xeon E5-2690v3 12C
2 Supercomputing Piz Daint
2.6GHz, Aries interconnect , NVIDIA
Centre (CSCS)
Tesla P100
3 RIKEN ACCS
ShoubuZettaScaler-1.6 etc.
National SC
SunwaySunway MPP, Sunway SW26010
4
Center in Wuxi TaihuLight 260C 1.45GHz, Sunway
SFB/TR55 at
PRIMERGY CX1640 M1, Intel Xeon
5 Fujitsu Tech.
QPACE3Phi 7210 64C 1.3GHz, Intel OmniSolutions GmbHPath
PRIMERGY CX1640 M1, Intel Xeon
Oakforest6 JCAHPCPhi 7250 68C 1.4GHz, Intel OmniPACSPath
DOE/SC/Argonne
Cray XC40, Intel Xeon Phi 7230 64C
7
Theta
National Lab.
1.3GHz, Aries interconnect
Cray CS-Storm, Intel Xeon E5-2680v2
Stanford Research
8
XStream
10C 2.8GHz, Infiniband FDR, Nvidia
Computing Center
K80
ACCMS, Kyoto
Cray XC40, Intel Xeon Phi 7250 68C
9Camphor 2
University
1.4GHz, Aries interconnect
Jefferson Natl.
KOI Cluster, Intel Xeon Phi 7230
10
SciPhi XVI
Accel. Facility
64C 1.3GHz, Intel Omni-Path
NVIDIA
1
Corporation

http://www.top500.org/

DGX
SATURNV

HPL
TOP500Rmax
Rank(Pflop/s)

Power
(MW)

GFLOPS/W

3.307

28

0.350

9.462

9.779

8

1.312

7.454

1.001

116

0.150

6.674

93.01

1

15.37

6.051

0.447

375

0.077

5.806

1.355

6

2.719

4.986

5.096

18

1.087

4.688

0.781

162

0.190

4.112

3.057

33

0.748

4.087

0.426

397

0.111

3.837

2018/7/10

KNL

25

Green 500 Ranking (June, 2018)
Site

ComputerCPU

HPL Rmax TOP500(Tflop/s)
Rank

Power
(kW)

GFLOPS/W

1

RIKEN, Japan

Shoubusystem B

ZettaScaler-2.2 HPC system, Xeon D1571, PEZY-SC2 , ExaScalar

857.6

359

47

18.4042

KEK, Japan

Suiren2

ZettaScaler-2.2 HPC system, Xeon D1571, PEZY-SC2 , ExaScalar

798.0

419

47

16.8353

PEZY, Japan

SakuraZettaScaler-2.2 HPC system, Xeon E52618Lv3, PEZY-SC2 , ExaScalar

824.7

385

50

16.657

4

NVIDIA, USA

DGX Saturn
V Volta

Xeon E5-2698v4, NVIDIA Tesla V100 ,Nvidia1,070.0

227

97

15.113

5

ORNL, USA

SummitIBM P9 22c, NVIDIA Tesla V100, IBM

122,300.0

1

8,806

13.889

6

Tokyo Tech.

TSUBAME3.
0

SGI ICE XA, IP139-SXM2, Xeon E52680v4, NVIDIA Tesla P100 SXM2, HPE

8,125.013792

13.704

7

AIST, JapanAIST AI
Cloud

NEC 4U-8GPU Server, Xeon E52630Lv4, NVIDIA Tesla P100 SXM2 ,
NEC

961.0

148

76

12.681

19,880.0

5

1,649

12.054

8

AIST, JapanABCI

Fujitsu PRIMERGY CX2550 M4, Xeon
Gold 6148, NVIDIA Tesla V100 SXM2,
Fujitsu

9

BSC, Spain

MareNostru
m P9 CTE

IBM P9 22c, NVIDIA Tesla V100, IBM

1,018.0

25586

11.865

10

AIP, RIKEN,Japan

RAIDENGPU
Subsystem

NVIDIA DGX-1 Volta36, Xeon E5-2698v4,
NVIDIA Tesla V100, Fujitsu

1,213.0

171

107

11.363

http://www.top500.org/

2018/7/10

KNL26

IO 500 Ranking (June, 2018)
Site

ComputerFile system

Client nodes IO500 Score

BW(GiB/s)MD
kIOP/s)

1

JCAHPC,Japan

OakforestPACS

DDN IME

2048

137.78560.1033.89

2KAUST, Saudi

Shaheen2

Cray
DataWarp

1024

77.37

496.81

12.05

3

KAUST, Saudi

Shaheen2

Lustre

1000

41.00

54.17

31.03

4

JSC, Germany

JURON

BeeGFS

8

35.77

14.24

89.81

5

DKRZ,
Germany

Mistral

Lustre2

100

32.15

22.77

45.39

6

IBM, USA

SonasadSpectrum
Scale

10

24.24

4.57

128.61

7

Fraunhofer,
Germany

Seislab

BeeGFS

24

16.96

5.13

56.14

8

DKRZ,
Germany

Mistral

Lustre1

100

15.47

12.68

18.88

9

Joint Institute for
NuclearResearch

Govorun

Lustre

24

12.08

3.34

43.65

EMSL
CascadeLustre

126

11.12

4.88

25.33

10 PNNL, USA
http://www.io500.org/

2018/7/10

27

KNL• 4,6$J-P'`Z_WQS\^AJBG=a• 8,208Y_X`25PFaPH-E7'E@ON>HBD?@bI4,6$P H#'CO:'
&2I'R_U• HPCI•• 6$I20%P;JCAHPC<FBDb!'1• JHPCN`93"( ' )+%a•6$I5%*c! )+b9 )+ML`IKa• 0`5.b5/a• 3"HPCV[]^TcY_Xb1248*/27

2018/7/10

KNL

28 Oakforest-PACS
 !
• DI?BG<I>J$K• 100,000 L 1 8CIA( )&512CIA3-• 3 3• ;GIE<I>• 400,000 ( 480,000) L 1
CIA3• 2&'@I:H

8CIAJ

K&2048(-#• +"4CIA%360%24$1'@I:H(*)687• DI?BG<I>22CIACIA3-2&@I:H *1.0•
CIA9!)7.&!),2& *2.00/7• 1FI=2Reedbush.1@I:H5 •2018/7/10

KNL29

JPY (=Watt)/GFLOPS Rate
Smaller is better (efficient)
SystemOakleaf/Oakbridge-FX (Fujitsu)(Fujitsu PRIMEHPC FX10)
Reedbush-U (SGI)(Intel BDW)
Reedbush-H (SGI)(Intel BDW+NVIDIA P100)
Oakforest-PACS (Fujitsu)(Intel Xeon Phi/Knights Landing)

JPY/GFLOPS
125
62.0
17.1
16.5

2018/7/10

KNL30 

• ?PcRe@Oakleaf/Oakbridge-FX, Reedbush-U/H, Oakforest-PACS

NOT^;9G6'VaIHb`eO7:EB6!VaIHb`eO7;8FC<5• HJU]SK(• ZeQXbMeO4Lbe\MeO@• %(• ZeQXbMeOfhd

f1d l3d g

lid gfFX10: 24YeW496YeW,
RB-U: 16YeW, RB-H: 2YeW4OFP: 16YeW, 64Ye
Wg"0,@ 0;34#'• Lbe\MeO• 'VaIHb`eOkfhd lid gk'fFX10:1,440YeW4
RB-U: 128YeW4RB-H: 32YeW4OFP: 2048YeWg• !VaIHb`eOkfhd l2*jd g4!f/*1&A'= )g• OeZeMc[_eR(1$-# @#;.f2g•
@MeO=D?4+>( @;.

2018/7/10

KNL

31   2018/7/10

KNL

 

32



• "?OUGRQW&T XD)Y?D89)T / Y >5C2;*TT / • -:@*• '@):0C.<,.@)?

WETHSIQX:

01• ETHSIQ)>
• #?6A?FVNPKM?
• #7/B&• LVJ !&(43/$,:0=+%?2018/7/10

KNL

33   
• (2/03*5 88,128-3, x 8(&/-3,• Oakforest-PACS : 8,208-3, x 68(&/-3,• Sunway TaihuLight: 40,960-3, x 260(&/-3,• Summit : 4,600-3, x 6 GPU/-3,

x (80 or 5120 (&)/GPU• 1(& # ).(2 !$• ).(2 4% '1+'%$=> (& %"2018/7/10

34

KNL

Weak ScalingStrong Scaling
@-+=PQTX"L017I
• Weak Scaling: 9J:JAØ A(OMR/Z@5=[01?I

Ø $ANYVWSUB*CH• Strong Scaling: AØ(OMRB ,8L*3I

(OMR/

@

KG?+.)EE7I(OMRL ,8@L*3I
5=41?I

Ø $ANYVWSUB@01?I

Weak Scaling

Strong ScalingD%!

9JC>#2?.<6 (">)&@LIZFH'5+[

;"A

(/#2I

2018/7/10

KNL

• <A41B57• OpenMP• ?D0#=>@& → *%$&OK
.

• <A4;C21• MPI EMessage Passing
InterfaceF• ?D0#6D3.

)

35

• 8D7#)(•8D7&,;C21)=>@+)"→(!#<A4;C21'<A41B57+

→9/:@57MPI/OpenMP2018/7/10

36

KNL   •MIMD*1"/+2"-(01. SPMD4Single Program Multiple Data5• 6*1"/,

*1&'#





• MPI4)3$.265-(02. Master / Worker4Master / Slave5• 6*1&%4Master5

4Worker5!4

*1&%
5 2018/7/10

!KNL 

37 
• 
• 
•   $ ! 
• #"

SIMD
 



æ 1 2 3 ö æ 9 8 7 ö æç 1* 9 + 2 * 6 + 3 * 3 1* 8 + 2 * 5 + 3 * 2 1* 7 + 2 * 4 + 3 *1 ö÷÷ ç
÷ " ç 4 * 9 + 5 * 6 + 6 * 3 4 * 8 + 5 * 5 + 6 * 2 4 * 7 + 5 * 4 + 6 *1÷
çç 4 5 6 ÷ ç 6 5 4 ÷ ç 7 * 9 + 8 * 6 + 9 * 3 7 * 8 + 8 * 5 + 9 * 2 7 * 7 + 8 * 4 + 9 *1÷
øç7 8 9÷ ç 3 2 1÷ è
ø è
ø
è


2 3ö
÷
ç
CPU ç 4 5 6 ÷
CPU ç 7 8 9 ÷øèCPU æ 1

CPU

æ9 8 7öæ 1* 9 + 2 * 6 + 3 * 3 1* 8 + 2 * 5 + 3 * 2 1* 7 + 2 * 4 + 3 *1 ö
÷ " çç 4 * 9 + 5 * 6 + 6 * 3 4 * 8 + 5 * 5 + 6 * 2 4 * 7 + 5 * 4 + 6 *1÷÷ç
ç 6 5 4÷
ç 7 * 9 + 8 * 6 + 9 * 3 7 * 8 + 8 * 5 + 9 * 2 7 * 7 + 8 * 4 + 9 *1÷
øè
ç3 2 1÷ø !   è2018/7/10

38

 KNL 
• 0/-
• 0/-8.329* ("!(• 170$ 8@9%#(')#
• 0/- $?,57*(• :?*(• ;?*(• <? *(• =?6*) &• >?,574+*)(



:
;
<

=

>



2018/7/10




KNL

39

2018/7/10

40

KNL 

•  S P = TS / TP (0 £ S p )
• P
• TS
P3  $PP/3 TP
• P'. S P = P30)$1NidealO • P'. S P > P30)$=MEKD9L=GMC9@H• 1

4$256$AM?9;>=( +8.$:I@<JF@B( ,7*0257#• 
• P E = S / P ´ 100 (0 £ E ) [%]PPp• " •  3!
• Saturation, %+-7&2018/7/10

KNL

41
• "%G K =5F)

6@-8(
• 3@=1(0<1F G α =5F)
A@D-?>F)S P = K /( Ka / P + K (1 - a ))= 1 /(a / P + (1 - a )) = 1 /(a (1 / P - 1) + 1)
• !@/E(7=.&@@MPJLIG9;

CSP→∞T( A('* 1/(1-α) <+F)SHNKQO@ T• @90%0

<17=4;C(&@@MPJLIG:/9;C( 1 / (1 - 0.9) = 10  ?4/>E>,R→'G#5F7B?A(4<C
G2F
G5F3=0=;C$ <+F2018/7/10

42

!KNL 

  



"88.8%





9/3=3



 

9/2=4.5 ≠ 62018/7/10

KNL

43
>VDUFR23/6)'/'.…

KNL1 1C>1'$Oakforest-PACS
1 1 LZJ=7 <*,18:
0-'. 5([• +11! (• >VDUFR=) http://www.cc.u-tokyo.ac.jp/support/kosyu/
4,'• 10/31,11/1 OpenMP/OpenACC09;PVHC>YSK?C>OWBT

QXB"

• +1
• 11/7, 8%#/M@NUIJOWBTQXB&• 3/1, 2
MPI • 
MPI • 
GPUATEG!()  


2018/7/10

KNL44

KNL (Intel Xeon Phi,
Knights Landing)

2018/7/10

45

KNLIntel Xeon Phi (KNL: Knights Landing)l® Xeon Phi™ Processor

rgeted for HPC and Supercomputingthat is binary
ard OS.

• Atom (Silvermont)

ctor performance+ AVX512
TILE x2
2 VPU

ative memory
apacity

Knights Landing Overview• 2  1 

notice.
o change without further notification.

MCDRAMEDC

MCDRAM

EDC

3

X4
DMI

PCIe
Gen 3

D
M
I

MCDRAMEDC

MCDRAM

EDC

EDC

EDC

misc

2 VPU

1MB
L2

Core

Chip: 36 Tiles interconnected by 2D MeshTile: 2 Cores + 2 VPU/core + 1 MB L2• 64, 68

32, 34 •Memory:
MCDRAM: High BW
MCDRAM:
16 GB on-package;
KNL✝ コア 16GB+
DDR4: 6 channels @ 2400 up to 384GB
4-way ハイパースレッディング(最大 4 × 72 = 288 論理プロセ
DDR4
IO:
36 lanes PCIe Gen3. 4 lanes of DMI for chipset
DDR
436 Tilesconnected by
2D MeshInterconnect

DDR MC

CHA

3

Tile

D
D
R
4
C
H
A
N
N
E
L
S

2 x16
1 x4

Core
 4 
(HyperThreading)• 

KNL Card(PCIe-Card)

DDR MC

EDC

C
H
A
N
N
E
L
S

EDC

L2 キャッシュはタイル上の 2 つのコアで共有される

Node: 1-Socket only
タイル コアアウト
アウト
コア Omni-Path
Fabric:on-package(not shown)オブオブ
命令キャッシュ命令キャッシュ

オーダー

オーダー

デコード
+ リタイヤデコード+ リタイヤVector Peak Perf: 3+TF DP and 6+TF SP Flops
1MiB
ベクトル
Scalar
Perf: ~3xover Knights
Corner ベクトル
ベクトル
ベクトル
L2
ALU
ALU
ALU Triad (GB/s):データ
ALU
Streams
MCDRAM:
400+;
DDR: 90+
レガシー
レガシー
キャッシュ

MCDRAM

4

MCDRAM

Omni-path not shown

Package

MCDRAMMCDRAM

Source Intel: All products, computer systems, dates and figures specified are preliminary based on current expectations,
HW + SWbased on current
L1 without
データキャッシュ
32KiBexpectations
L1 データキャッシュ
are subject 32KiB
to change
notice. KNL data are preliminary
and are subject to change
プリフェッチwithout notice. 1Binary Compatible with Intel Xeon processors
using Haswell Instruction Set (except TSX). 2Bandwidth
numbers are based on STREAM-like memory access pattern when MCDRAM used as flat memory. Results have been
HW + SWestimated based on internal Intel analysis
and are provided for informational
purposes only. Any difference in system
メッシュへ
hardware or software design or configuration may affect actual プリフェッチ
performance.

HotChips 27 KNL
, Colfax (colfaxresearch.com/knl-webinar )KNL コアの機能✝

© Colfax I2018/7/10

KNL

46

KNL• BCICL@O3#• Flat:MCDRAM-DDR4 ()*5@J=• Cache: MCDRAM1DDR4BCI07D?;E-)+
• Hybrid• 8H=>IK9CL@: 5#• (All-to-all: 5@J= (/… " )• Quadrant, Hemisphere: !,5@J= (4(2*12)/
MFL:'31&.%N
• SNC-4, SNC-2: NUMA@B6K(/4 (or 2)/

CL@0 /1 ( => ,1$ CL@(Flat, Cache)0<GA
7EL4 (regular-flat / regular-cache)2018/7/10

 KNL

47
• Cache• MCDRAML3

• Flat• MCDRAM
MCDRAM
16 GB

MCDRAM
16 GB

DDR4
96 GB

DDR4
96 GB






2018/7/10

KNL

48

Flat NUMA• $ numactl -H

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6… 271
node 0 size: 98164 MB
node 0 free: 92027 MB
node 1 cpus: 
node 1 size: 16384 MB
node 1 free: 15832 MB
node distances:
node 0 1
0: 10 31
1: 31 10

Node 0
96 GB
CPU

DDR4

Node 116 GB
MCDRAM2018/7/10

49

 KNL  "
ClusterOFP!Quadrant
Mode: Sub-NUMA
C• Quadrant: .31()&#!3801459$• !4
MCDRAM
*60-7;/+<%• SNC-4: 4OPIO
MCDRAMOPIO
MCDRAM

PCIe

EDC

EDC

IIO

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

iMC

Tile

Tile

OPIO
MCDRAM

EDC

EDC

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

3
1

4

2
iMC

DDR

NUMA2
4':$• 4.,01$OPIO
MCDRAM





DDR
Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

EDC

EDC

Misc

EDC

EDC

OPIO
MCDRAMOPIO
MCDRAMOPIO
MCDRAMOPIO
MCDRAM

HotChips27
1) L2 miss, 2) Directory access, 3) Memory
access, 4) DataKNL
return
16

2018/7/10

50

KNL

Stream Triad
DDR4• DDR4: micprun -D -k stream• MCDRAM: micprun -k stream• unset KMP_AFFINITY• export KMP_HW_SUBSET=66c@2,1t 0  

Cache

Flat

346.5 GB/sec
84.8
GB/sec

600
Cache

500

 (GB/sec)

Flat MCDRAM
400

Flat DDR4

300
200
100
0
1

11

21

31



41

51

MCDRAM

61

494.8
GB/sec2018/7/10

KNL

51

FlatMCDRAM(1)(1) numactl.36
• MCDRAM0NUMA=B<1,'*&7*!6• $ numactl –H

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 … () … 268 269 270 271
node 0 size: 98164 MB
node 0 free: 85707 MB
node 1 cpus:
node 1 size: 16384 MB
node 1 free: 15496 MB
node distances:• ;B<$"8%)! .0
node01  /#-!%0+$-!
0: 10 31
1: 31 10

• >A:@?.8• $ numactl --membind 1 ./a.out(MCDRAM 16GB.5$4-!,9@BC• $ numactl --preferred 1 ./a.out(MCDRAM. 15 4-!DDR42(6C• $ numactl --interleave 1,0 ./a.out(MCDRAM,DDR48.(6C2018/7/10

KNL

52

FlatMCDRAM(2)(2)  (,.3Intel MPI/• I_MPI_HBW_POLICY • export I_MPI_HBW_POLICY=hbw_bind• export I_MPI_HBW_POLICY=hbw_preferred• export I_MPI_HBW_POLICY=hbw_interleave• numactl/+)&• 021!/"$(* ) '"*%!' Intel MPI)&!#  ('-'2018/7/10

KNL

53

FlatMCDRAM(3)(3) memkind:17:;+-/
• C• #include <hbwmalloc.h> 0• ,. 89;, +”hbw_”0(&/• malloc() => hbw_malloc()• posix_memalign() => hbw_posix_memalign()• free() => hbw_free()• ;=3+ -lmemkind 0

• Fortran• ,#*!&$ • Fastmem,0 (allocatable)• !dir$ attributes fastmem :: array+'#)%*!
• ;=3+ -lmemkind 0
• hbw_set_policy())."• HBW_POLICY_BIND• HBW_POLICY_PREFERRED >462<5?
• HBW_POLICY_INTERLEAVE

2018/7/10

54

&KNL

  • AVX-512

%512;687@5/(",(*
7@5<=> (#$. &• 512;68 = 64:18))09?3(0243!.-(%*

•)0243('.

• A(double)/8(=512;68)+!

 
 !#"

$ %!#"=
01

2

364

4

5

6

7

0

1

2

3

4

5

6

7

0

1

2

3

4

2018/7/10

KNL    • C• __attribute__((aligned(64))) "(64 byte = 512 bit)• , double A[1024] __attribute__((aligned(64)))•  posix_memalign() "
• 

,
#include <stdlib.h>double *A;
posix_memalign(&A, 64, 8192);• Fortran• %+'#*  -align array64byte $(&)+"• attribute  !()real*8, dimension(:), allocatable :: A
!dir$ attributes align:64 :: A

55

2018/7/10

KNL

OpenMP

56

2018/7/10

57

+KNL 

OpenMP
• OpenMP





PE

PE

PE

PE

OpenMP 

OpenMP 

OpenMP 

OpenMP
 






A[ ]

$ %PE$'(*)⇒!$ & #"%" #2018/7/10

KNL

58

OpenMP• OpenMP (OpenMP Application Program Interface)16$

JKN
95T

"4HOAMI>,=.


2. M@GMN
3. >!+.:50,%
1.

•

•

LQC)$HOAMI5 *-=.95>(=:50,%BPF@M4;=06&<8-?%
JKN RMPI32S47/$EQD
55#)'$) 0,%

2018/7/10

KNL

OpenMP59

• J_MO

C"/T`G\VaGYN^• U^LHD-XPbHD% ;*• <J_MO"9,.

C4B;=-T`G\VaG<0#
1.
2.

XEaXY]-FZMI[+<NbK')! 0 !;>.
OpenMP9C91:.T`G\W;:68.Bc(d• Oakforest-PACS (Intel Xeon Phi 7250, Knights Landing)• 68HD-272&HD !
• T`G\W<0# => J_MOCA342:.5?;=MPI+OpenMPRES]MO"@$<7• QbO+< =OpenMP9=91:.
• QbO+< =MPIC.B

2018/7/10

KNL

OpenMP • C
• #pragma omp







• Fortran
• !$omp







60

2018/7/10

KNL61

OpenMP
• )4.'1)4.'2(/*04OpenMP
(/*04&$• 6Intel Fortran90)4.'1

ifort -O3 -qopenmp foo.f90• 6Intel C)4.'1
icc -O3 -qopenmp foo.c• • OpenMP25/ • )4.'1"#"$+3,-
$ $ !$• OpenMP $ OpenMP"$+3,- % )4.'1"$
• 6Intel Fortran90)4.'1

ifort -O3 -qparallel -qopenmp foo.f90

2018/7/10

KNL

62

OpenMP • OpenMP7IPAML=BQG?N.2.1#!

H>?N7#8)07H>?N=/<-43#+• CODE=) OMP_NUM_THREADS3
• TOpenMP6;<#!H>?N,a.out7$ export OMP_NUM_THREADS=16$ ./a.out
918$ env OMP_NUM_THREADS=16 ./a.out• • %BQG?N7IPAML4)OpenMP6;<IPAML7#&,)OMP_NUM_THREADS=16.2:)$T

5<-4,*<S• -7 8)OpenMP6;<7S@RFRJDET• (CODE#3)-7@RFRJDE6;<&,'"
• IPAMKQA73 !

2018/7/10

KNL

OpenMP

63

2018/7/10

64

'KNL

OpenMP C
OpenMP!(#pragma omp parallel
,
!)
!*
 +
OMP_NUM_THREADS!(  %#" $

 &

!)

!)

 

!*

 +-1…

!)2018/7/10

65

'KNL

OpenMP Fortran
OpenMP
!(!$omp parallel!)!$omp end parallel!* +
OMP_NUM_THREADS!(  %#" $

 &

!)

!)

 

!*

 +-1…

!)2018/7/10

KNL

66

Work sharing
• parallel27#12<@=>-*8


1$",OpenMP-:*8B?A=;BC
2: (parallel region). 4•  : ),<@=>-*8:
*8OpenMP2:Work sharing. 4• Work sharing3 &'(,2D%!81.



-*862• forBdoC• sections
• single (master)0/2.

parallel.59+862• parallel for  (parallel do)• parallel sections0/2018/7/10

KNL

67

2018/7/10

68

-KNL Fortran

Fordo#pragma omp parallel forfor (i=0; i<100; i++){a[i] = a[i] * b[i];}!$omp parallel do!$omp end parallel do


%)&'



%)&'+

%)&',

%)&'2

%)&'3

for (i=0; i<25; i++){a[i] = a[i] * b[i];}for (i=25; i<50; i++){a[i] = a[i] * b[i];}for (i=50; i<75; i++){a[i] = a[i] * b[i];}for (i=75; i<100; i++){a[i] = a[i] * b[i];}%)&' #!
 "#(*$"







2018/7/10

8KNL 

69For   
for (i=0; i<100; i++) { •453 &1)
)#+1
a[i] = a[i] +1;
6a[i-1]#%2'+
b[i] = a[i-1]+a[i+1];
#17}
for (i=0; i<100; i++) {a[i] = a[ ind[i] ];}•ind[i]- ,/0453($1"*!".1•a[ind[i]]#(%21 #+,453($12018/7/10

70

(KNL 

Private
#pragma omp parallel for private(c)for (i=0; i<100; i++){a[i] = a[i] + c * b[i];}) "%#$! →  


"%#$&
for (i=0; i<25; i++){a[i] = a[i] + c0*b[i];}"%#$'for (i=25; i<50; i++){a[i] = a[i] + c1*b[i];}"%#$

"%#$2

"%#$3

for (i=50; i<75; i++){a[i] = a[i] + c2*b[i];}for (i=75; i<100; i++){a[i] = a[i] + c3* b[i];}"%#$

2018/7/10

71

>KNL 

Private #pragma omp parallel for private( j )for (i=0; i<100; i++) {for (j=0; j<100; j++) {a[ i ] = a[ i ] + amat[ i ][ j ]* b[ j ];}•:=8? 4;57%)0!$  /.
•private( j ) ' 4;57% j )31<60 (#$!*"+100):=8 (','
→&'-29=&'.

2018/7/10

>KNL 

72

Private  Fortran
!$omp parallel do private( j )do i=1, 100
do j=1, 100a( i ) = a( i ) + amat( i , j ) * b( j )
enddo
enddo
!$omp end parallel do•:=8? 4;57%)0!$  /.
•private( j ) ' 4;57% j )31<60 (#$!*"+100):=8 (','
→&'-29=&'.

2018/7/10

KNL


C

73



• 76(DHEF9A#/-<(K39

A2)

80>

• "9#/-<:DHEF8'

87.?>• reduction!*)5(ddot: 7>87>2=(
 4%95 @7,7,7>#pragma omp parallel for reduction(+:ddot )for (i=1; i<=100; i++) {ddot += a[ i ] * b[ i ]}ddot9

:DCG9<"$

I&:"$4+;1BJ

2018/7/10

KNL

74



Fortran• 65'CGDE8@".,;'J28

@1(

7/=

• !8".,;9CGDE7&

76->=• reduction )(4'ddot9 76=1<'
3$84 ?6+6+6=!$omp parallel do reduction(+:ddot )do i=1, 100
ddot = ddot + a(i) * b(i)enddo!$omp end parallel do
ddot8

9CBF

8;!#

H%9!#3*:0AI

2018/7/10

KNL

75
• reduction/*!H7(F+<.NMLGD7

-<9• =GW`Z\O:7-<&?9• GK:F7ddot'G4 O)?C2%D +@L"<6
3EJ8LdAB?75VQX7]c\RSPe!$omp parallel do private ( i )W`Z\ G_c^Of#gW`Z\ 'do j=0, p-1
W`Z\DPUYW@LQb[ZUW,O F0
do i=istart( j ), iend( j )ddot_t( j ) = ddot_t( j ) + a(i) * b(i)W`Z\D'9L7acT_Eddot'G
enddo
4 ddot_t()O)?70F $?C;=
enddo!$omp end parallel doddot = 0.0d0
do j=0, p-1
2%D1?>Iddot = ddot + ddot_t( j )enddo2018/7/10

76

!KNL

FortranSingle• Single!

!$omp single/!$omp end single
%+-)'& 
%. (,)*# $•  (,)*# "%$! • nowait  &%# $
#pragma omp parallel for$A#pragma omp single{ B }…
} 

A

A
"



#
…

A

2018/7/10

77

KNL

Master• .single)"• %&!master( !%@/*--;>96BA.
#<78?7=9:,3'4• - 2+
• $-%0,13

54

2018/7/10

KNL

78

OpenMP2018/7/10

 KNL

79  
•  • integer (Fortranl Fortran90)

omp_get_num_threads()int (C)l C use omp_lib
Integer :: nthreads#include <omp.h>
int nthreads;

nthreads = omp_get_num_threads()

nthreads = omp_get_num_threads();

2018/7/10

 KNL

80

 omp_get_thread_num()• integer (Fortran) int (C)
l C l Fortran90 •use omp_lib
Integer :: myid#include <omp.h>
int myid;

myid = omp_get_thread_num()

myid = omp_get_thread_num();2018/7/10

81

KNL

 omp_get_wtime()  • double precision (Fortran ) double (C•l Fortran90



l C

)



use omp_lib
real(8) :: dts, dte#include <omp.h>
double dts, dte;

dts = omp_get_wtime()
dte = omp_get_wtime()
print *, “Elapse time [sec.] =”,dte-dts

dts = omp_get_wtime();dte = omp_get_wtime();printf(“Elapse time [sec.] = %lf ¥n”,
dte-dts);

2018/7/10

KNL

OpenMP


822018/7/10

83

 KNL 

• Parallel for/do9>+&HJG=

K/?MO
N=)4LC+ !<DIEF<K("6A@.<L58+C6A,
HJG=B
1

 0

 1

 2

 3

 4

n K

*L

} 3=:1+

DIEF957HJG<6A%
'#09;-:+DIEF$= 
1  0
 1
 2
 3
 4 n
02;A

%'#

2018/7/10



84

KNL } :,813#7'-#
1

,86&17.!":)+#.946%$
n



} 

0#7 .!"HBEF?@<A/65I3#
DGC=>;/# /081 ,8$} 27 .:&( *9.%8$

2018/7/10

 KNL




85



• schedule (static, n)• ;@8)19>-.*0/<250&!A/<25C/<25D???%!:+>5=6> $B (%!
'(n!19>-.*0) (• Schedule  ) "37,;4#
static19>-.*0#;@8//<25 
1
 0

 1

 2

 3

2018/7/10

 KNL




86



• schedule(dynamic, n)• (+&#'* "  !)$%   n#'* "

1
 0

 1

 2

 3

2018/7/10




 KNL

87

• schedule(guided, n)• HKF9@GJ<=:?.(0@GJ<=:?9'&(/$5$(*>IAD#
5!+.96-7n0@GJ<=:?9 .%7
• @GJ<=:?1

$11619>IAD.
,*""4)1$@GJ<1=:?0/7
• @GJ<=:?2 1 0 #,-0'&/7
• @GJ<=:?0 1 46 %! k 9 (*@GJ< =:
?20 k 3.'&/7$1@GJ<2 k 46'
&/7
$ 7
1
• @GJ<=:?$
'8-!/!
BE;HC2 1 0/7
 0
 1
 2
 3

2018/7/10

 KNL






l Fortran90

88





!$omp parallel do private( j, k ) schedule(dynamic,10)
j-461$ 
#') %*$! do i=1, n
do j=indj(i), indj (i+1)-1
i-461$
y( i ) = amat( j ) * x( indx( j ) )!* enddo
 #$enddo +(" &
dynamic0./2635-, !$omp end parallel do


l C



#pragma omp parallel for private( j, k )schedule(dynamic,10)for (i=0; i<n; i++) {for ( j=indj(i); j<indj (i+1); j++) {y[ i ] = amat[ j ] * x[ indx[ j ]];}}2018/7/10

KNL

89   • dynamic2guidedG[gnRVPZH$F9;/• [gnRVPZ8<>:MC,&ainYH%;EM8 .2
!AGQoad\_89;EM3• 2[gnRVPZ89>:C,&ainY8;EM .2
!AGQoad\_8<;EM3• *G#G^lo_Qb84M3• 'G[gnRVPZG[ho`nS80B2[ho`nS
UY^87M3• staticGKB1-)8B9Mp84Mq• dynamicEDG'YTXhojnSH2WY]fGQoao
d\_8M82staticHQoaod\_HpJCODq53•  F,&8(CEMkoc"N+I?B2
staticYTXhojnSN6C2L
8%5 $84M3• ?@=2cmSienSGUY^H>M2018/7/10

KNL

OpenMP 



90

2018/7/10

KNL

OpenMP

91 

• OpenMP)

parallel.#&for7:4'&+ % • &OpenMP)48/659/012+($OpenMP(48/659/(• parallel-,+'*+)private  (!
.!&%3/"+;2018/7/10

92

KNLPrivate • OpenMP15%

-,3 +:5%+70
Hshared variableI32:• C "4%Fortran90 "4module5%.4881532:• BD=CG@3*/&

5%Threadprivate'• parallel1#6

FG>E3

**0&: %.4#1*0&:9%32:• .48815%

1 *2&• );<$(35%4?GA4'• !4FG>E<3*/#6*<:• !4FG>E<3*0%Threadprivate+:2018/7/10

93

KNL

Parallel    • Paralleldo

!$omp parallel!$omp do private(j,tmp)
do i=1, 100
do j=1, 100
tmp = b(  ) + c(  )a( i ) = a( i ) + tmp
enddo
enddo
!$omp end do!$omp end parallel

Parallel 
 parallel do








!$omp parallel do private(j,tmp)
do i=1, 100
do j=1, 100
tmp = b(  ) + c(  )a( i ) = a( i ) + tmp
enddo
enddo
!$omp end parallel do2018/7/10

94

 KNL

Parallel    • Parallel,do(#'(".
• 342+/#%,#%

!*).
• %&#•

342/(".,$+!
342*140 !-(")

do i=1, n!$omp parallel dodo j=1, n
<
>
enddo
!$omp end parallel do
enddo

!

!$omp parallel
do i=1, n!$omp dodo j=1, n
<
>
enddo
!$omp end do
enddo!$omp end parallel2018/7/10

KNL

95

OpenMPSIMD2018/7/10

KNL

96

OpenMP+SIMD• SIMD (Single Instruction - Multiple Data)• 1 >'P\OI• AVX512 `1 >#8I• KNL>BAVX512 I -9@7HC/1%4FH@
1• SIMD"BOpenMP 4.03FNU\Q• M[SKX4&TLQZE9=6HG4_

I9;4• M[SKX4*<;!2I:8?E0GA>()]$.

D^• M\RA52I(?:GE]9C9C^0G• VWYAJXK[V[Q4,(]+^2018/7/10

KNLOMP ForDo+SIMD• C

#pragma omp parallel for simdfor (i=0; i < num; i++) {sum = sum + a[i];}• Fortran:!$omp parallel do simd
do i=1, num
sum = sum + a(i)!$omp end parallel97

2018/7/10

KNL98

OMP SIMD• C
#pragma omp parallel forfor (i=0; i < num; i++) {#pragma omp simdfor (j=0; j < num; j++) {sum[i] = sum[i] + a[i][j];}}• Fortran:!$omp parallel dodo i=1, num!$omp simd
do j=1, numsum(i) = sum(i) + a(j,i)
enddo!$omp end simd
enddo!$omp end parallel2018/7/10

KNL

99

Declare SIMD
•  SIMD•OMP SIMDSIMD • #pragma omp declare simd notinbranchfloat min (float a, float b){ return a < b ? a : b; }void minner (float *a, float *b, float *c) {#pragma omp parallel for simdfor (i=0; i<N; i++)c[i] = min(a[i], b[i]);}2018/7/10

KNLKNL OpenMP 

100

2018/7/10

101

 KNLKNL(OFP)HyperThreading1/: 681/• HyperThreading(HT) : 11/=>  2722<46
•• AVX-512:745')$*+ 4 HT1/+2(=136

)• 2"&AVX-512:745(VPU).

-( 1 HT$'• AVX-512:7451#%-( 2 HT!

• 89;/032., -(
2 VPU
2 VPU
3~4 HT*
Core

1MB
L2

1 

=2

Core

2018/7/10

102

 KNL

OFP  • ZX^SP => OS

L/G (CBDJMUWVE2O

:I<5• 15SP85MA82O:M• 'DVQY

L/GF 1ms$DI>?9M• OFP@F3SP0EGVQY

L/GO: MJ6D.

_NM,H4L`
• SP0A3VQ]@L2R[WT\O+#;?5MSP1H2O

:M• HTSPH&2O:M• SP) 0,1, 68,69, 136,137, 204, 205

F-7K;="855(85)• !%,8•<M*C"%F0

6
0
8

1
3
6

6
1
9

2
0
4

2 VPU
Core0

1
3
7

2 VPU
1MB
L2

1 

Core1

=2

2
0
5

2018/7/10

KNL

103

KNL  

Xeon Phi,Xeon)( +DH;62CPU*-&• 62+5J<5"
• Reedbush+Broadwell (2.1GHz)*-& 1.6

• EFG2598I3=L7"• ;<B@<>KM5"
ÞOpenMP+* %1#
25
PARALLEL• N+4MA

C<? (EPCC syncbench)• 862!01662' #)"1/* 1
• 262,:3H
• 4~6462,62+.
• 128~25662,HT

Overhead (usec)

4MAC<?" #$)1

20

FOR
PARALLEL FOR

15

BARRIERSINGLE

10

REDUCTION

5
0
2

4

8

16

32

# Core

64

128

256

2018/7/10

104

KNL

OpenMP
• 0;4&Atomic7KR>=

(Atomic)2@UKNGI(EPCC syncbench)9)
Xeon Broadwell 32C>-3
Atomic: 0.1us• Parallel for/do0/!"01
Critical, Lock/Unlock: 0.4us
61KR>3 5<,$;=> OPR>AFES?HTD2•   %&Parallel region=;

Parallel for/do+*! %0
Parallel region 2-omp for/do=:)• MPI+OpenMPJ?LRGI1

8:!ESGI=';• >MRBUDQT1
• #.-(&);

Overhead (usec)• 

3
CRITICAL

2.5

LOCK/UNLOCK2ATOMIC

1.5
1
0.5
0
2

4

8

16
# Core

32

64

128

256

2018/7/10

KNL(1KNL)

105

2018/7/10

106KNLIntel Compiler 2018 update 1+ 
(OFP$5;-D6Iintel/2018.1.163, 2018
• C0E:,B: icc• Fortran0E:,B: ifort

7)• 2018 update2, 2018 update3 ',E26FD&• @E48E2$3,?E/"#9F1AE*%
• ( GJ2018 update3H• 0>E7B,EI1A<2.C=6
module switch intel intel/2018.3.222•

  *!)
G"'#9/')'H2018/7/10

KNL

107 
'NXQ\a• -xMIC-AVX512• KNL 1PbV• Xeon;D4H>G -axMIC-

AVX512• -O3• $ `Y_• -qopenmp• OpenMP• -align array64byte• (Fortran@Bd%I64byte?J]
Kac"d37.++• -qopt-streaming-stores=
always / never / auto
• always: OZSQ[?!5>+

I,)OZSQ[@&.
+ ?
• neverA6@#• TWM_UAauto• _bX@ %2<?D (Aalways)• #pragma vector nontemporal (inC/C++)• !DIR$ vector nontemporal (in F)• -qopt-prefetch={0~5}• X^WLSRI=@0G+(H• -no-vec• SIMD I*/GCHe9E:<8
1F0>H .*H

2018/7/10

KNL

108
• 681-# 641-,", 
• 376,'#1- 0,1, 68,69, 136,137, 204, 205,
• 89<-054()# +$681-!%
• >
1. 16.=)11-#&321-• /:72;  ,*
2. 11-1HT641-
3. 11-2HT1281-
4. 11-3HT1921-
5. 11-4HT2561-2018/7/10

109

KNL

32
• ' 21&3(*• !.($/ +4*  
%0,&"1-)
#PJM --omp thread=32 
5 &3(* export OMP_NUM_THREADS=32
# 6
export KMP_AFFINITY=proclist=[2,4,6,8,10,12,14,16,18,20,
22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,
56,58,60,62,64,66],explicit,verbose
2 VPU

OS

Core

2 VPU

3

Core

1MB
L2

1MB
L2

2 VPU

2 VPU

Core

Core

0

1MB
L2

2 VPU

2 VPU

Core

Core

1

2 VPU
Core



1MB
L2

2 VPU

2 VPU

Core

Core

2

1MB
L2

2 VPU
Core

2018/7/10

110KNL

64• 

1
• KMP_HW_SUBSET• c@,
• 64c@2,1t: 64t2 1

export KMP_HW_SUBSET=64c@2,1t
export KMP_AFFINITY=scatter,verbose
2 VPU
Core
OS
2 VPU

6

Core

1MB
L2

1MB
L2

2 VPU
Core

2 VPU

0

Core

1MB
L2

2 VPU

2 VPU

1

Core

Core

2 VPU

7

Core



2

1MB
L2

2 VPU

3

Core

2 VPU

4

Core

1MB
L2

2 VPU

5

Core

2018/7/10

111

KNL

128• 

2#
• KMP_HW_SUBSET $64c@2,2t• KMP_AFFINITYcompact
• scatter#!" 

064

#

165…

export KMP_HW_SUBSET=64c@2,2t
export KMP_AFFINITY=compact,verbose
2 VPU
Core

OS

2 VPU

12 13
Core

1MB
L2

1MB
L2

2 VPU

2 VPU

Core

0Core
1

2 VPU



14 15
Core

1MB
L2

2 VPU

2 VPU

23

4Core5

Core

1MB
L2

2 VPU

2 VPU

67

8Core
9

Core

1MB
L2

2 VPU

10Core11

2018/7/10

KNL

112

192, 256
192
• 3 

256
• 4 

export KMP_HW_SUBSET=64c@2,3t
export KMP_AFFINITY=compact,
verboseexport KMP_HW_SUBSET=64c@2,
4t
export KMP_AFFINITY=compact,
verbose2018/7/10

KNL

113
.'&2$*
C 1,+()+#)%3&2D
• KMP_AFFINITY ,verbose3• 7?BCe+ -=46@D,A>B;
• 2/".-0 +C-D• 3285-: 9A:<0!852, 9A:<1!854, 9A:<2!856…,1 )OMP: Info #242: KMP_AFFINITY: pid X tid XX thread 0 bound to OS proc set {2}OMP: Info #242: KMP_AFFINITY: pid X tid XY thread 1 bound to OS proc set {4}OMP: Info #242: KMP_AFFINITY: pid X tid XZ thread 2 bound to OS proc set {6}…

2018/7/10



KNL

114

   • ) & '$*
• -qopt-report=5• !&.optrpt • *& +• Intel Advisor XE,• #&'

• Intel VTune XE• "(!%)

2018/7/10

KNLKNL/OFPMPI+OpenMP

115



2018/7/10

116

KNLMPI+OpenMP • MPIDF4'

&BCE,3G?A:<,


•

•*

('-%+0$.">G<HOpenMP>G<HMPI
Reedbush'-&285:;#NUMA1

"

.)85:;: OpenMP
85:;+>G<: MPI• @D:;MPI,!-/%
B=G624D79#(#,%2018/7/10

117

KNLMPI
• i^\qYb^XpR$ElOelm-Eu@• i^\qYb^XpREj`nA7J• TpbOlE-6"E]daPQMHlOelmK?GEAFC

8r• ijm•.(-.(:*A,D;• ufo\^VD9<JijmVO[HdLOnVO[E)K&*E 8XZ_hsMassively Parallel Processing(MPP)XZ_htK$8J,D ;• fo\^V• ufo\^V(A+C,3E.(K6%3A#*• ':

• APIsApplication Programming InterfacetE!

• ZSqlcm_N6*:58• 1#KkqW:/0?J=BDIJMnUm[hE2:*• foRlgpR:4>8s:58t

2018/7/10

KNL

118

MPI
• MPI0,6327http://www.mpi-forum.org/8• 1994 5 1.07MPI-18• 1995 6 1.1• 1997 7 1.2 (' 2.07MPI-28• 2008 5 1.3 2008 6 2.1
• 2009 9 2.2•  http://www.pccluster.org/ja/mpi.html• MPI-2 !&)9• I/O• C++Fortran 90*5/60+6• 14.- /• $#"%2018/7/10

KNL

119

MPI MPI-3.1• MPI-3.0 2012 9• MPI-3.1 2015 6• *0$/'",+.&!
• http://mpi-forum.org/docs/docs.html• http://meetings.mpi-forum.org• http://meetings.mpi-forum.org/mpi31-impl-status-Nov15.pdf•  • (./)-%".#

1MPI_IALLREDUCE2•   1RMARemote Memory
Access)• Fortran2008  2018/7/10

KNL

120

MPI MPI-4.0• -KV<*UC7NMTB1
• http://meetings.mpi-

forum.org/MPI_4.0_main_page.php• %0)"/
• D4HQ@CIS8PLT8.• MPI3IQ9V;OT-  WFault Tolerance,
FTX• "$(#-34A31
• Active Messages (M@>V<-ISB:R)••••+-6VEP@I-1"' 
"6VEVJ@C!F4IP4T
E@G2QT8,&*!4T?PIBDTCP*$• Stream Messaging• ISG24RU4T?VG5V=

2018/7/10

121

KNLMPI
• MPICHO:HMEBAP• 8J=KC'• MVAPICH (:HL7EBA)• ;D9;

 . !MPICH6GN>• InfiniBand(35+ • OpenMPI• ;NFK?N>• GK@MPI• !305&'GN>21,-%4
: "#!FX103MPI: Open-MPIGN>Intel MPI: MPICH!MVAPICHGN>• :IN<

'1*5-%4)/'$4

2018/7/10

KNLMPI• /.

(A• . -
1. /"4/
2. .)*$52/0,'.#5&
3. .)*$52//!
4. .)*$52//
5. ?6 .5 /@?:8@• MPI+0A
1. /ID"%31"4/ID
2. ;>:/7<=93. ;>:
4. ;>:
5. :8

122

2018/7/10

KNL

MPI
• +,.1 • !%5 "#!5 "$(5 !(5

• 44  
• 02-)3*• !5'5

• /302-)3*• $!5 #'5• 4 • $%
•








• &5• • % #&5###51232018/7/10

KNL

124
• MPI_COMM_WORLD2#>JMG=PC/63:9

; *9 • >JMG=PC2#;%1HOBE?;59• .2#QSnumprocs –R4.1HOBE?
&#R,1>JMG=PC08-7:9• (1&#!MPI_COMM_WORLD"• HOBE?;)+$#MPI_Comm_split;• KEBP@;#1HOBE?0*9/'0• “IND<LAF”.

2018/7/10

KNL

125

MPI• MPI-=A;:,4

.'=A;:-C-D=A;<9C/&#-85D+)1(03.'• ?B6CRankD
MPI=A;:,  ,$*• MPI)-MPI_Comm_rank)%32 
C9B=@=A7?>)-myidD+FHPEEG,"2• ,,MPI=A;:42+-
MPI_Comm_size4!
C9B=@=A7?>)-numprocs +$,"2D•

2018/7/10

KNL

126

MPI
• 

 
 = SPMD (Single Program Multiple Data)mpiexec.hydra -n 4 ./a.out

a.out

a.out

a.out

a.out

2018/7/10

KNLMPI



127

2018/7/10

C
FortranKNL








 ierr ierr = MPI_Xxxx(….);• Fortran   ierrcall MPI_XXXX(…., ierr)• 
•

• 

MPI_Status istatus;• Fortraninteger istatus(MPI_STATUS_SIZE)128

2018/7/10

C
Fortran• MPI!$

KNL




129






• C
MPI_CHAR ( )  MPI_INT ()
MPI_FLOAT ( )MPI_DOUBLE( )• Fortran
MPI_CHARACTER ( ) MPI_INTEGER()MPI_REAL ( )
MPI_DOUBLE_PRECISION(  ) 
MPI_COMPLEX( )• %# "$ 2018/7/10

130KNLMPI―MPI_Recv 1/2
• ierr = MPI_Recv(recvbuf, icount, idatatype, isource,

itag, icomm, istatus);• recvbuf :"& %
• icount : ",0*& %
• idatatype : ",0*"& %
• MPI_CHAR (MPI_FLOAT () MPI_INT ()
) MPI_DOUBLE(• isource :
./'& %• "PE$

)-+)0(&%PE"

!#MPI_ANY_SOURCE &

%

2018/7/10

KNL

131

MPI―MPI_Recv 2/2
• itag : *

25,PLJTI:,6,AKF;C*• ;KF;PLJTIC

25,8/<)MPI_ANY_TAG C*• icomm : *PE(

C!#3A

7+AHOQNGTKC*• %7<MPI_COMM_WORLD C3B=>,*• istatus : MPI_StatusU ;& V* :'3A
.A*-9?4;  C25& C3A08*• .MPI_STATUS_SIZE;&. 1BA*•
25PLJTI;$;RSE. istatus[MPI_SOURCE])KF. istatus[MPI_TAG] :1BA*• C "W MPI_Status istatus;• Fortran "W integer istatus(MPI_STATUS_SIZE)• ierr(@) : 

*DRTHTM.A*2018/7/10

KNL

132

MPI―MPI_Send• ierr = MPI_Send(sendbuf, icount, idatatype, idest,

itag, icomm);• sendbuf : % *• icount :   %5?3*
• idatatype :   %5?3% *• idest :   "PE%icomm#%<>,*• itag :   ":42?1$&)"3-%*• icomm : 8=240? *!(#(/9;7.?3*
• ierr (') :  +<?/?6(2018/7/10

KNL

133

Send Recv  
PE0

PE

PE

PE

MPI_Send  

MPI_Recv

2018/7/10

KNL

134
• IJ+/'(IJ2

D@:67>BE#%0
• H


=6;ADKE
E• @:67>B-

0→85?DF

*2• Dcollective communication

operationE*.10• ,&,!)
3B9<4C8"
$0

G,

*$

2018/7/10

135KNL
•PE • MPI_Reduce• 

PE

PE0
PE0
PE1




PE2

PE0

• MPI_Allreduce• 


PE0

PE1

PE
PE0


PE2

PE1

PE2

2018/7/10

KNL

136

MPI―MPI_Reduce• ierr = MPI_Reduce(sendbuf, recvbuf, icount,

idatatype, iop, iroot, icomm);• sendbuf : %':(• recvbuf :A2>*
A2>*iroot 713PE:

':(<7.#<-90?>*%'8 ';) 7+56;9=9,*29@4)9>&A19/6;9=9,*• icount : *%':CDB" A2>*• idatatype : *%':CDB:A2>*• EFortranFHG8!IA$2A2> ;)MPI_2INTEGER())MPI_2REAL( ))MPI_2DOUBLE_PRECISION() )A2>*

2018/7/10

KNL

137

MPI―MPI_Reduce• iop :
 $0"2)1$• MPI_SUM ( )# MPI_PROD ()# MPI_MAX ()#MPI_MIN ()# MPI_MAXLOC ( -*0)# MPI_MINLOC (-*0) /.$• iroot : $2'1PE0icomm ,0<=42)1$+0icomm 0PE
, (2)1&%1$
• icomm : $PE! 2 )1,%16:;95>72)1$• ierr :  $ 3<>6>8&1$

2018/7/10

138

 KNLMPI_Reduce
PE0

PE

MPI_Reduce()

MPI_Reduce()

PE

PE

MPI_Reduce()

MPI_Reduce()

iroot






iop 

2018/7/10

139

KNLMPI―MPI_Allreduce• ierr = MPI_Allreduce(sendbuf, recvbuf, icount,

idatatype, iop, icomm);• sendbuf : %':(• recvbuf :B2?*
B2?*iroot 713PE:

':(<7.#<-90@?*%'8 ';) 7+56;9>9,*29A4)9?&B19/6;9>9,*• icount : *%':DEC" B2?*• idatatype : *%':DEC:B2?*• =8!B$2B2?))MPI_2FLOAT ( ))MPI_2DOUBLE() B2?*;)MPI_2INT(2018/7/10

KNL

140

MPI―MPI_Allreduce• iop :

"$ #• MPI_SUM () MPI_PROD ()MPI_MAX ( )MPI_MIN ()
MPI_MAXLOC ( ) MPI_MINLOC() ! • icomm :PE$##'+,*&.($ #• ierr : %-.'.)#2018/7/10

141 KNLMPI_AllreducePE0
MPI_Allreduce()




PE

PE

PE

MPI_Allreduce()

MPI_Allreduce()

MPI_Allreduce()









iop

 

2018/7/10

KNL

142


1.

=@80A2•

D&;8<. %>86C4,D&;8<. %/165B "+)" ( *$

••;8<. &9C7& -
MPI_Send, MPI_Bcast$#2.

:A=@80A2• D&;8<.

%

(

• ;8<.

• +&9C7-!

&9C7& -!
&'?C3&2018/7/10

143

&KNL  • 0  
send   

"#! 0



"#! 1



"#! $



"#! %



…

send



send

recv



send


recv

…



 


recv



 




 

…

 
 

2018/7/10

KNL

144

 

• ierr = MPI_Isend(sendbuf, icount, datatype,

idest, itag, icomm, irequest);• sendbuf :  • icount : '+%

• datatype : '+%• idest :PEicomm )*!
• itag :(&$+#
%"



2018/7/10

KNL

145

 

• icomm : PE*(!(-463,80*
(•  !$MPI_COMM_WORLD *)%&
• irequest : MPI_Request9 #:*51/8." ') (• ierr :  +78-82(2018/7/10

KNL

146


• ierr = MPI_Wait(irequest, istatus);• irequest : MPI_RequestDE#7(+?=;C:/,%46+#• istatus :MPI_Status DE#/!)5 $5#• $MPI_STATUS_SIZE0

7(-)5#• (+?=;C:00@A8$
istatus[MPI_SOURCE] "<9$istatus[MPI_TAG] /'65#• >C<7

1*

2&.

)5B

>C<73)/2018/7/10

 

• &'#"0%($!

send $()%'#( 

&'"!0



send send

&'"!1



recv

&'"!*



&'"!+



…

147

,KNL

send

…


recv

!MPI_Wait 



 


 


recv



 



…








2018/7/10

KNL

148


•   %• MPI_Send • MPI_Wait 

&• MPI_Isend • MPI_Wait &•  !$#" &2018/7/10

KNL

149


1.
2.

3.
4.
5.

MPI-3$1.4$!P.+)#%  8   -3$1.4$" MPI!
 !
 '4(
6 http://accc.riken.jp/HPC/training/text.html 7
Message Passing Interface Forum
6 http://www.mpi-forum.org/ 7
MPI-J/524$2&*
6 http://phase.hpcc.jp/phase/mpi-j/ml/ 7
%4,05(! !69;;:7

2018/7/10

9KNL 

150 C
double t0, t1, t2, t_w;..
ierr = MPI_Barrier(MPI_COMM_WORLD);
t1 = MPI_Wtime();
:!!&

15,
+""#26-43+;

t2 = MPI_Wtime();t0 = t2 - t1;
ierr = MPI_Reduce(&t0, &t_w, 1,
MPI_DOUBLE,MPI_MAX, 0,
MPI_COMM_WORLD);

26/0.7$<0'(
%*
!' ())''
+26/0.8 *

2018/7/10

151

9KNL  Fortranreal(8) :: t0, t1, t2, t_w..
call MPI_BARRIER(MPI_COMM_WORLD, ierr)
t1 = MPI_WTIME()
:!!&

15,
+""#26-43+;

t2 = MPI_WTIME()
t0 = t2 - t1
call MPI_REDUCE(t0, t_w, 1, MPI_REAL8, &MPI_MAX, 0, MPI_COMM_WORLD, ierr)26/0.7$<0'(%*
!' ())''+26/0.8 *2018/7/10

KNL

OFPMPI

152

2018/7/10

153

KNL• 47)(1.8':#2$.+$;"

!

• /9- <47)( "
#PJM -L node=/9#PJM --mpi proc=47)(• (&64,• MPI

:25*,MPI;• /usr/local/bin/mpi_core_setting.sh• MPI+ OpenMP0%36*• /usr/local/bin/hybrid_core_setting.sh• (&64,



2018/7/10

154

 KNLMPI• &$

'%##PJM --mpi proc=source /usr/local/bin/mpi_core_setting.sh• 1~33%)!
2 VPU
Core
OS

1MB

L2

2 VPU

2 VPU

Core

Core

• 34~66%)!
2 VPU
Core
OS

1MB

L2

• 67%)!

*1"("(
0

1MB

L2

2 VPU

2 VPU

Core

Core

1

1MB

L2

2 VPU

2 VPU

Core

Core

2

1MB

L2

2 VPU
Core

*1"(

2 VPU

2 VPU

Core

Core

0

1MB

L2

2 VPU

2 VPU

1

Core

Core

2

1MB

L2

2 VPU

2 VPU

3

Core

Core

4

1MB

L2

2 VPU

5

Core

*

• 68%)!

*
• 69~272%)! *  
2 VPU
Core

OS

1MB

L2

2 VPU

2 VPU

Core

0Core
1

1MB

L2

2 VPU

2 VPU

23

Core

Core

45

1MB

L2

2 VPU

2 VPU

67

8Core
9

Core

1MB

L2

2 VPU
Core
10
11

2018/7/10

KNL

155

MPI
• 69"



#!$%



export
I_MPI_PIN_PROCESSOR_EXCLUDE_LIST=0,1,68,69,13
6,137,204,205• "

export
I_MPI_PIN_PROCESSOR_EXCLUDE_LIST=0,68,136,204

2018/7/10

KNL

156

MPI+OpenMP• )/"!1~68!.$&

+(!,)% #PJM --mpi proc=)/"!
#PJM --omp thread='0&!.$&source /usr/local/bin/hybrid_core_setting.sh *0&• *0&
1: 1#- #-
333!.$&
2: 1#- 
366!.$&
3: 1 
367!.$&
4: 368!.$&1 2•

2018/7/10

157

KNLMPI+OpenMP• 69(0*+





• 3128(0*+/.1)((1,2+1.1)()




#PJM -L node=16#PJM --mpi process=16#PJM --omp thread=64 #68(0*+"source /usr/local/bin/hybrid_core_setting.sh 2
export OMP_NUM_THREADS=128 
export KMP_HW_SUBSET=2t'%!(0*+
mpiexec.hydra -n ${PJM_MPI_PROC}
-$&/

2018/7/10

158

KNLMPI(+OpenMP)• MCDRAM  ((• I_MPI_HBW_POLICY 
export I_MPI_HBW_POLICY=hbw_bind
export I_MPI_HBW_POLICY=hbw_preferred
export I_MPI_HBW_POLICY=hbw_interleave• %'"!
export I_MPI_DEBUG=5• !&#$



export KMP_AFFINITY+=,verbose[40] MPI startup(): shm and tmi data transfer modes…
[0] MPI startup(): 192
269565
c3850.ofp {2,70,138,206}

2018/7/10

MPI+OpenMP • MPI
• 3

159

%KNL 

-4810923 !+%

$5;3: Master only, Funneled, Multiple• Multiple!%%#%!"'*0.;8"• Master only!%Parallel7;/6:)",&"

•#"$!,&Funneled!$(Master only

Funneled

Multiple#pragma omp parallel{....
}
MPI_Send( ... );#pragma omp parallel
{
....
}

#pragma omp parallel{....
#pragma omp master
MPI_Send( ... );....
}#pragma omp parallel{....
MPI_Send( ... );....
}Master only
parallel 

"#$ !
11 2018/7/10

160

KNL MPI• MPI_Init()

MPI_Init_thread()• C:

int provided;
MPI_Init_thread(&argc,&argv,MPI_THREAD_FUNNELED,&provided);• Fortran

integer provided, required
required=MPI_THREAD_FUNNELED
call MPI_Init_thread(required, provided, ierr)2018/7/10

KNL

• “dynamic/runtime”• “!$omp master!$omp end master”

161#   #!"  

!$omp parallel private (neib,j,k,i,X1,X2,X3,WVAL1,WVAL2,WVAL3)!$omp&
private (istart,inum,ii,ierr)!$omp masterCommunication is done by the master thread (#0)!C
!C– Send & Recv.
(…)
call MPI_WAITALL (2*NEIBPETOT, req1, sta1, ierr)!$omp end master!C!C-- Pure Inner Nodes

The master thread can join computing of internal
nodes after the completion of communication!$omp do schedule (runtime)
export OMP_SCHEDULE="dynamic,[chunksize]"
do j= 1, Ninn
(…)
enddo
!C
!C-- Boundary Nodes
Computing for boundary nodes are by all threads!$omp dodo j= Ninn+1, N
(…)
enddo

default: !$omp do schedule (static)!$omp end parallel

Idomura, Y. et al., Communication-overlap techniques for improved strong scaling of gyrokinetic Eulerian codebeyond 100k cores on the K-computer, Int. J. HPC Appl. 28, 73-86, 2014

2018/7/10

162

KNL

Speedup by communication overlapping(OFP: 32 nodes)• Memory model: C: Cache, F: Flat
• Sub NUMA: Q: Quadrant, N4: SNC-4

C+Q

F+Q

C+N4

F+N4

• (m) : core binding to avoid “busy”
core
C+Q(m)

F+Q(m)

C+N4(m)F+N4(m)

200

300

400

500

16%
14%
12%
10%
8%
6%
4%
2%
0%
-2%
-4%

Overlap

50

100

2018/7/10

KNL

163


• MPI







2M!"

Fortran (&Fortran# "2018
%!DIR$ ATTRIBUTES ALIGN : 2097152 :: arrayMPI
real(8),allocatable :: arrayMPI(:,:,:,:)C $"!# "%arrayMPI = (double *)__mm_malloc(array_size,
2*1024*1024);

2018/7/10

164

KNL (DDN: Infinite Memory
Engine)• SSD  /E925H508E7• I960 TBH ;E,  1.5TB/sec• IO500Top• ,A)'D6:B5(%• erasure coding#• Lustre<&)C025> +?40@$
• /cache +?40@! ',32F8E7 "G• 25E1D-F25E1)DH25E1'*6 G#PJM -x STGIN_LIST=<%B26<&)C>#PJM -x STGOUT_LIST=<%B26<&)C>• 25E1)D%• prestage-wait .=D7

2018/7/10

KNL165

IME
1. POSIX:/work/gt00/t002
xx 
/cache/gt00/t00
2xx 
Intel MPI  

2. MPI IO:
a. MPIIntel MPI

MVAPICH2
module switch impi mvapich2b.  ime:// 
MPI_File_open(MPI_COMM_
WORLD,”ime:///work/gt00/t002xx…”,… );2018/7/10

166

KNL

IME
• NAS Parallel benchmark, BT-IO • class=C, 64 (16 x 4)
• Lustre

Simple  • class=D, 256(16x16)• Lustre
Simple  , IME

Simple



90000
80000

class=C

Mop/s/total

70000

class=D

60000
50000
40000
30000
20000
10000
0
LustreIME (POSIX)
SimpleSimpleIME

LustreIME (POSIX)
Full
Full

IME

2018/7/10

167

KNLMcKernel• 92@)%6>.0*OS
http://www.pccluster.org/ja/mckernel/index.html• '@3=• Linux 100%

DLinux $6>(;8)?4&=
McKernel" • OS#$:@+6>(;8 (OS Jitter)/>• 7-1-4)?"• OFPC!"• OFP

2 VPU
Core

1MB

L2

$
&9@,D 4)% Linux(OS*@5-)+ 64)% McKernel A:@+@%6<B

2 VPU Linux2 VPU
Core

Core

1MB

L2

2 VPU

2 VPU

Core

Core

1MB

L2

2 VPU
Core

McKernel 2 VPU



Core

1MB

L2

2 VPU
Core

2018/7/10

KNL   168

2018/7/10

KNL

169

• 1

.Fortran90 (!#+0Samples-knl.tar.gz• tar%-)+1
Fortran90&",$'* • C/ : C• F/ : Fortran90

• (!#+
/work/gt00/z30105 (/home

/2018/7/10

KNL


1.
1.

2.
3.
4.
5.

170

 (1/2)

cd &+"  Lustre$)'
$ cd /work/gt00/t0044x/work/gt00/z30105   Samples-knl.tar.gz  *!(#,
$ cp /work/gt00/z30105/Samples-knl.tar.gz ./Samples-knl.tar.gz  $ tar xvfz Samples-knl.tar.gz
Samples $)
$ cd Samples-knl
%($)
$ cd streamC- $ cd C
Fortran90- $ cd F2018/7/10

171KNL (2/2)

6. make$ make7. (stream)

$ ls







2018/7/10

 
1.
2.
3.

4.
5.

KNL

172

stream2/61"+ )$ pjsub stream.bash
% * 043+)$ pjstat)#%2,-6*)
stream.bash.eXXXXXX
stream.bash.oXXXXXX 8XXXXXX& 9
% 2,-6%+!')$ cat stream.bash.oXXXXXX.57 &%2,-6$)%" % ($ cat stream.bash.eXXXXXX