2018/10/29

MPI

1

106  


MPI






  
hanawa  cc.u-tokyo.ac.jp

2018/10/29

MPI

2
• 1[

20181029YZ 10[00 - 17[45• [ $AVBX 42 413.3(• ,[  Y@XJX?VKQXE:V=%&/0W Z• ,)MU=RP[• 1029YZ• 9[30 - 10[00• 10[00 - 12[00 Oakforest-PACSU=;V4MPI+4
Oakforest-PACS659MPI*8'4IVLUD<V=-4!)• 13[30 - 14[45#FXB 4MPI-IO4!)• 15[00 - 16[15 ?OQH>XB4NSC@TDG7MultipleEndpoint4!)• 16[30 - 17[45 "-4!)2018/10/29

<MPI

3  
FY
11

2 6
12 13 14 15 16 17 18 19 20 21 22 23 24 25
Yayoi: Hitachi SR16000/M1
IBM Power-7

3.9(' +9/9(7049,:$"!#%"< 8

54.9 TFLOPS, 11.2 TB;

Oakforest-PACS
Fujitsu, Intel KNL

T2K Tokyo
140TF, 31.3TB

Big Data &Extreme Computing

25PFLOPS, 919.3TB

Oakleaf-FX: Fujitsu PRIMEHPC
FX10, SPARC64 IXfx

BDEC System

50+ PFLOPS (?)

1.13 PFLOPS, 150 TB

Oakbridge-II

Oakbridge-FXIntel/AMD/P9/ARM CPU only

136.2 TFLOPS, 18.4 TB-9, 8)2469)57
+9/9(7049,
*51 
&+9/9(7049,

5-10 PFLOPS

Reedbush, HPE
Broadwell + Pascal
1.93 PFLOPS

Reedbush-L
HPE
1.43 PFLOPS+9/9(7049,

2018/10/29

MPI

4

Oakforest-PACS (OFP)• 2016121&6
• 8,208 Intel Xeon/Phi (KNL)9PVH025PFLOPS• 4:+• TOP 500 6W 1XYHPCG 3W 2

XW201611X• ( HPC #3(JCAHPC: Joint Center
for Advanced High Performance Computing)• )"2*%$'LTMV•  #LTMV• GRTOKC #LTMVB9!7C/ :@A>?32<EKVOVITPSVMJKNQF3.;Y(C1 802* #F+U5 <E=DC,• http://jcahpc.jp

2018/10/29

5

 MPI

Knights Landing: Next Intel® Xeon Phi™ Processor

Oakforest-PACS 
Intel® Many-Core Processor targeted for HPC and SupercomputingFirst self-boot Intel® Xeon Phi™ processor that is binary
compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance• Intel XeonIntegration
Phi of(Knights
Landing)Memory on package:innovative memoryarchitecture for high bandwidth and high capacity• 1#1,68
Integration of Fabric onpackageThree products• MCDRAM: "#KNL Self-Boot"
+ DDR4 !
   (Baseline)

HotChips27
Knights Landing
Overview
KNL
 

KNL Self-Boot w/ Fabric

KNL Card(Fabric Integrated)(PCIe-Card)

!16GB


MCDRAMMCDRAM

2 x16
1 x4

X4
DMI

PCIe
Gen 3

D
M
I

MCDRAMMCDRAM

Potential future options subject to change without notice.
All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

EDC
3

Core

C
H
A
N
N
E
L
S

2 VPU
1MB
L2

MCDRAM

4

EDC

3
D
D
R
4

36 Tilesconnected by
2D MeshInterconnect

DDR MC

EDC

Core

EDC

Tile

D
D
R
4

MCDRAM: 490GB/ $
%DDR4: 115.2 GB/=(8Byte2400MHz6 channel)
2 VPU

EDC

EDC

MCDRAM

Omni-path not shown

misc

Package

DDR MCEDC

MCDRAM

C
H
A
N
N
E
L
S

EDC

MCDRAM2018/10/29

6

MPI

Oakforest-PACS
Oakforest-PACS

2018/10/29

MPI
•

 : t00631
 gt00
11/29 9:00



7

2018/10/29




MPI

8

2018/10/29

9MPI

2 3,4 • Oakleaf-FX (  PRIMEHPC FX10)• 1.135 PF, &6/38*, 2012 4  2018

• Oakbridge-FX (• 136.2 TF, !

3 PRIMEHPC FX10)
:168!;, 20144  20183

• Reedbush (HPE, Intel BDW + NVIDIA P100 (Pascal))• -8*7'0358'46(8.8&6/38*• 2016-Jun.20167<2020 6• )"GPU'(,1• Reedbush-U: CPU only, 420 nodes, 508 TF (2016 7)• Reedbush-H: 120 nodes, 2 GPUs/node: 1.42 PF (2017 3)• Reedbush-L: 64 nodes, 4 GPUs/node: 1.43 PF (2017 10)• Oakforest-PACS (OFP) ( , Intel Xeon Phi (KNL))• JCAHPC ( CCS9 ITC)
• 25 PF, 12 (2018 6) (2)• Omni-Path #8$,%+2, DDN IME (Burst Buffer)2018/10/29

ReedbushTop500:

RB-L 291@Nov. 2017
RB-H 203@Jun. 2017
RB-U 361@Nov. 2016
Green500: RB-L 11@Nov. 2017
RB-H 11@Jun. 2017

10

MPI

Reedbush-U

201671
201691


 

201731
201743


 

2017102
2017111


 

Reedbush-H
Reedbush-L2018/10/29

外部接続ルータ 1Gigabit/10Gigabit Ethernet Network
x6

管理サーバ群SGI Rackable C1110-GP2
9nodes

E5-2680v4 2.4GHz
14core,256GiB Mem

E5-2680v4 2.4GHz
14core,128GiB Mem

64 nodes, 256GPUs, 1.434PFLOPS

Reebush-U
x420

Reedbush-H
x240 (FDRx2/node)

Reedbush-L
x128( EDR x2/node)x2(for PBS)

InterConnect( 4x EDR InﬁniBand)x4(MDS:x 2)

x24(OSS(VM):x 12 x2)

Lustre Filesystem
DDN SFA14KE x3set
5.04PBx10(Ctrl:8,MDS:2)

InterConnect( 4x EDR InﬁniBand)x12

x36(IME:6x6)

高速キャッシュ
DDN IME14K x6set
209TB
x12

x16(IME:8x2)

x2

Management port
管理コンソール
Mac Pro
電力管理サーバ
電力計器

高速キャッシュ
DDN IME240 x8set
153.6 TB
x8

管理用補助
サーバ
SGI Rackable
C1110-GP2 x2
x2

ラ
イ
フ
／
管
理
ネ
ッ
ト
ワ
ー
ク
1Gigabit/10Gigabit Ethernet Network

x2

x64

・CPU : E5-2695v4 2.1GHz 18core
・GPU : NVIDIA Tesla P100 SXM2 x4/node

x4

x4

x120

Reedbush-L
SGI Rackable C1102-GP8

NAS Storage
24TB
x9

Reedbush-H
SGI Rackable C1102-GP8

120 nodes, 240GPUs, 1.418PFLOPS

GbE SWx6

x420

・CPU : E5-2695v4 2.1GHz 18core
・GPU : NVIDIA Tesla P100 SXM2 x2/nodex9

NFS Filesystem
16TB

Reedbush-U
SGI Rackable C2112-4GP3

420 nodes, 508.03TFLOPS・CPU : E5-2695v4 2.1GHz 18core

x9

ログインノード群
SGI Rackable C1110-GP2
6nodesx6

11

MPI

2018/10/29

12

MPI

Reedbush-H 
76.8GB/s
QPIIntel Xeon
E5-2695 v4(BroadwellEP)Intel Xeon
E5-2695 v4(BroadwellEP)

QPI
76.8GB/s

15.7 GB/s

15.7 GB/s

x1
676.8GB/s

NVLinK
20 GB/s
EDR switch

NVIDIA
Pascal6x1

NVLinK

3

3



128G
B

G

20 GB/s

G3
x16

G3
x16

NVIDIA
Pascal

DDR4
DDR4
DDR4
DDR4

PCIe sw

PCIe sw

IB FDR
HCA

G3
x16

G3
x16

76.8GB/s

G



128G
B

DDR4
DDR4
DDR4
DDR4

IB FDR
HCA

EDR

2018/10/29

MPI

13

Reedbush
Reedbush-U

Reedbush-H

Reedbush-L

Intel Xeon E5-2695v4 (Broadwell-EP, 2.1GHz, 18core) x 2
sockets (1.210 TF), 256 GiB (153.6GB/sec)

CPU/nodeGPU-

NVIDIA Tesla P100 (Pascal, 5.3TF,
720GB/sec, 16GiB)

Infiniband

EDR



420

120

GPU

-

240 (=120 2)

256 (=64

 (TFLOPS)

509

1,417
(145 + 1,272)

1,433(76.8 + 1,358)

64.5

191.2
(18.4+172.8)

194.2(9.83+184.3)

2016.072017.032017.10 (TB/sec)




FDR

2ch

EDR 2ch64
4)2018/10/29

MPI

14

Oakforest-PACS OFP
• 2016121&6
• 8,208 Intel Xeon/Phi (KNL)YPVH025PFLOPS• 4:+• TOP 500 12W 2XYHPCG 7W 2

X
W20186X• ( HPC #3(JCAHPC: Joint Center
for Advanced High Performance Computing)• )"2*%$'LTMV•  #LTMV• GRTOKC #LTMVB9!7C/ :@A>?32<EKVOVITPSVMJKNQF3.;Y
(C1 802* #F+U5 <E=DC,• http://jcahpc.jp

2018/10/29

MPI

Oakforest-PACSの特徴 (1/2)•  &2$• 1&2$ 686
3TFLOPS8,208&2$7 25
PFLOPS• +,.3MCDRAM36
16GB45DDR436
96GB44• &2$
• (/'!-1'1$
Fat-Tree%"#02•  
*.2-1
 6 -)
• Intel Omni-Path Architecture15

15

2018/10/29

MPI

16

Oakforest-PACS 
'2  %2$
Product

%2$
)0!"

+,
.

 PRIMERGY CX600 M1 (2U)
+ CX1640 M1 x 8node
Intel® Xeon Phi™ 7250
3 2$: Knights Landing468 1.4 GHz

&1$

16 GB, MCDRAM,  490 GB/sec

&1$


96 GB, DDR4-2400, '2 115.2
GB/secIntel® Omni-Path Architecture
100 Gbps

 Product

.1
#*0

25 PFLOPS
8,208(/&!-1&1$Fat-tree

16

2018/10/29

"MPI

17

Oakforest-PACS 2 / 2• -"#1I/O• -"#1&'*.:
Lustre 26PB
• -"#1$/)&0&'*.
4DDN IME56
1TB/sec!
 ,
1PB
• 2,)%+3(




 2

• 
• Green 5006(2016.11)• Linpack6 2.72 MW• 4,986 MFLOPS/W4OFP5•830 MFLOPS/W45
120!
 

17

2018/10/29

MPI

18

Oakforest-PACS 
 Type%

Lustre File System
26.2 PB
Product
DataDirect Networks SFA14KE
& 500 GB/sec
Burst Buffer, Infinite Memory Engine  Type(by DDN)
%!
"

940 TB (NVMe SSD, $)Product
DataDirect Networks IME14K
& 1,560 GB/sec#

4.2MW'(102

18

2018/10/29

19

 MPI

Knights Landing: Next Intel® Xeon Phi™ Processor

Oakforest-PACS 
Intel® Many-Core Processor targeted for HPC and SupercomputingFirst self-boot Intel® Xeon Phi™ processor that is binary
compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance• Intel XeonIntegration
Phi of(Knights
Landing)Memory on package:innovative memoryarchitecture for high bandwidth and high capacity• 11
Integration of Fabric on package

Three products• MCDRAM: KNL Self-Boot

KNL Self-Boot w/ Fabric

KNL Card(Fabric Integrated)(PCIe-Card)16GB+ DDR4(Baseline)

HotChips27
Knights Landing
Overview
KNL 
MCDRAMMCDRAM

2 x16
1 x4

X4
DMI

PCIe
Gen 3

D
M
I

MCDRAMMCDRAM

Potential future options subject to change without notice.
All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

EDC

EDC

3



16GB 6 96GB DD

2 VPU
Core

C
H
A
N
N
E
L
S

2 VPU
1MB
L2

MCDRAM

4

3
D
D
R
4

36 Tilesconnected by
2D MeshInterconnect

DDR MC

EDC

Core

EDC

Tile

R
4

MCDRAM: 490GB/ DDR4: 115.2 GB/=(8Byte 2400MHz 6 channel)

EDC

EDC

MCDRAM

Omni-path not shown

misc

Package

DDR MCEDC

MCDRAM

C
H
A
N
N
E
L
S

EDC

MCDRAM2018/10/29

20

MPI

Oakforest-PACS: Intel Omni-Path Architecture      Fat-tree
768 port Director
Switch12(Source by Intel)

22Uplink: 24

48 port Edge Switch
362 

Downlink: 24
1

...

24

25

...

48

49

...

72

5>4
;1+#


,/3%(7<6*0+-:=6=4 )
• -/29#& )• "?.:8# (5>4' !$2018/10/29

MPI

21: Performance of Linpack (TFLOPS)
51th TOP500 List (June, 2018) RRmax
peak: Peak Performance (TFLOPS),Power: kW

Site

Computer/Year Vendor

Cores

http://www.top500.org/

Rpeak
Power
Rmax
(TFLOPS) (TFLOPS) (kW)1

Summit, 2018, USA
DOE/SC/Oak Ridge National Laboratory

IBM Power System AC922, IBM POWER9 22C
3.07GHz, NVIDIA Volta GV100, Dual-rail Mellanox
EDR Infiniband

2,282,544

122,300(= 122.3 PF)

187,6598,806

2

Sunway TaihuLight, 2016, China
National Supercomputing Center in Wuxi

Sunway MPP, Sunway SW26010 260C 1.45GHz,
Sunway10,649,600

93,015

125,436

15,371

3

Sieera, 2018, USA
DOE/NNSA/LLNLIBM Power System S922LC, IBM POWER9 22C
3.1GHz, NVIDIA Volta GV100, Dual-rail Mellanox
EDR Infiniband

1,572,480

71,610119,194

TH-IVB-FEP Cluster, Intel Xeon E5-2692v2 12C
2.2GHz, TH Express-2, Matrix-2000

4,981,760

61,445

100,679

18,482PRIMERGY CX2550 M4, Xeon Gold 6148 20C
2.4GHz, NVIDIA Tesla V100 SXM2, Infiniband EDR

391,680

19,88032,577

1,649

Cray XC50, Xeon E5-2690v3 12C 2.6GHz, Aries
interconnect , NVIDIA Tesla P100

361,760

19,59025,3262,272

560,640

17,59027,113

8,209

4
5
6

Tianhe-2A, 2018, China
National Super Computer Center in
Guangzhou
ABCI (AI Bridging Cloud Infrastructure),
2018, Japan
National Institute of Advanced Industrial
Science and Technology (AIST)
Piz Daint, 2017, Switzerland
Swiss National Supercomputing Centre(CSCS)

7

Titan, 2012, USA
DOE/SC/Oak Ridge National Laboratory

Cray XK7, Opteron 6274 16C 2.200GHz, Cray
Gemini interconnect, NVIDIA K20x8

Sequoia, 2011, USA
DOE/NNSA/LLNLBlueGene/Q, Power BQC 16C 1.60 GHz, Custom

1,572,864

17,173

20,133

7,890

9

Trinity, 2017, USA
DOE/NNSA/LANL/SNL

Cray XC40, Intel Xeon Phi 7250 68C 1.4GHz, Aries
interconnect

979,968

14,13743,903

3,844

Cori, 2016, Japan

Cray XC40, Intel Xeon Phi 7250 68C 1.4GHz, Aries
interconnect

622,336

14,01627,881

3,939

556,104

13,556

24,913

2,719

10 DOE/SC/LBNL/NERSC
Oakforest-PACS, 2016, Japan

PRIMERGY CX1640 M1, Intel Xeon Phi 7250 68C

12 Joint Center for Advanced High Performance 1.4GHz, Intel Omni-Path
Computing2018/10/29

MPI

22

HPCG Ranking (June, 2018)
Computer

HPL Rmax TOP500(Pflop/s)
Rank
2,392,000
122.300
1
835,584
71.6103705,024
10.510
16
Cores

HPCG
(Pflop/s)
2.926
1.796
0.603

Peak

1
2
3

SummitSierraK computer

4

Trinity

979,072

14.137

9

0.546

1.8%

5

361,760

19.590

6

0.486

1.9%

10,649,600

93.015

2

0.481

0.4%

7

Piz Daint
SunwayTaihuLight
Oakforest-PACS

557,056

13.555

12

0.385

1.5%

8

Cori

632,400

13.832

10

0.355

1.3%

9

Tera-1000-2

522,240

11.965

14

0.334

1.4%

10

Sequoia

1,572,864

17.173

8

0.330

1.6%

6

http://www.hpcg-benchmark.org/

1.5%
1.5%
5.3%

2018/10/29

23

MPI

Green 500 Ranking (SC16, November, 2016)
Site

ComputerCPU

NVIDIA DGX-1, Xeon E5-2698v4 20C
2.2GHz, Infiniband EDR, NVIDIA Tesla
P100
Swiss National
Cray XC50, Xeon E5-2690v3 12C
2 Supercomputing Piz Daint
2.6GHz, Aries interconnect , NVIDIA
Centre (CSCS)
Tesla P100
3 RIKEN ACCS
ShoubuZettaScaler-1.6 etc.
National SC
SunwaySunway MPP, Sunway SW26010
4
Center in Wuxi TaihuLight 260C 1.45GHz, Sunway
SFB/TR55 at
PRIMERGY CX1640 M1, Intel Xeon
5 Fujitsu Tech.
QPACE3Phi 7210 64C 1.3GHz, Intel OmniSolutions GmbHPath
PRIMERGY CX1640 M1, Intel Xeon
Oakforest6 JCAHPCPhi 7250 68C 1.4GHz, Intel OmniPACSPath
DOE/SC/Argonne
Cray XC40, Intel Xeon Phi 7230 64C
7
Theta
National Lab.
1.3GHz, Aries interconnect
Cray CS-Storm, Intel Xeon E5-2680v2
Stanford Research
8
XStream
10C 2.8GHz, Infiniband FDR, Nvidia
Computing Center
K80
ACCMS, Kyoto
Cray XC40, Intel Xeon Phi 7250 68C
9Camphor 2
University
1.4GHz, Aries interconnect
Jefferson Natl.
KOI Cluster, Intel Xeon Phi 7230
10
SciPhi XVI
Accel. Facility
64C 1.3GHz, Intel Omni-Path
1

NVIDIA
Corporation

DGX
SATURNVhttp://www.top500.org/

HPL
TOP500Rmax
Rank(Pflop/s)

Power
(MW)

GFLOPS/W

3.307

28

0.350

9.462

9.779

8

1.312

7.454

1.001

116

0.150

6.674

93.01

1

15.37

6.051

0.447

375

0.077

5.806

1.355

6

2.719

4.986

5.096

18

1.087

4.688

0.781

162

0.190

4.112

3.057

33

0.748

4.087

0.426

397

0.111

3.837

2018/10/29

MPI

24

Green 500 Ranking (June, 2018)
TOP 500
Rank

SystemCores

http://www.top500.org/

HPL Rmax(Pflop/s)

Power
(MW)

GFLOPS/W

1

359

Shoubu system B, Japan

794,400858.

47

18.4042

419

Suiren2, Japan

762,624798.

47

16.835

3

385

Sakura, Japan

794,400

825.

50

16.6574227

DGX SaturnV Volta, USA

22,4401,070.

97

15.1135

1Summit, USA

2,282,544

122,300.

8,806

13.889

6

19

TSUBAME3.0, Japan

135,828

8,125.

792

13.704

7

287

AIST AI Cloud, Japan

23,400961.

7612.681

8

5

ABCI, Japan

391,680

19,880.

1,649

12.054

9255

MareNostrum P9 CTE, Spain

19,4401,018.

86

11.86510

171

RAIDEN GPU, Japan

35,3601,213.

107

11.36313

411

Reedbush-L, U.Tokyo, Japan

16,640806.

7910.16719414

Reedbush-H, U.Tokyo, Japan

17,760

802.

94

8.575

2018/10/29

MPI

25

IO 500 Ranking (June, 2018)
BW

MD(GiB/s)kIOP/s)

137.78560.10

33.89

1024

77.37

496.81

12.05

Lustre

1000

41.00

54.17

31.03

JURON

BeeGFS

8

35.77

14.24

89.81

DKRZ, Germany

Mistral

Lustre2

100

32.15

22.77

45.39

6

IBM, USA

SonasadSpectrum Scale

10

24.24

4.57

128.61

7

Fraunhofer,
Germany

Seislab

BeeGFS

24

16.96

5.13

56.14

8

DKRZ, Germany

Mistral

Lustre1

100

15.47

12.68

18.88

9

Joint Institute for
Nuclear Research,
RussiaGovorun

Lustre

24

12.08

3.34

43.65

10

PNNL, USAEMSL Cascade

Lustre

126

11.12

4.88

25.33

Site

ComputerFile system

Client nodes IO500 Score

1

JCAHPC, Japan

Oakforest-PACS

DDN IME

2048

2

KAUST, Saudi

Shaheen2

Cray DataWarp

3

KAUST, Saudi

Shaheen2

4

JSC, Germany5

http://www.io500.org/

2018/10/29

26

MPI Oakforest-PACS
 
201841 • 9>47<1>3• 100,000 A 1 88>6( ) 3 +%
5>/=A28>6 x 24 x 360?1 @20488>6+%• 0<>:1>3• 400,000 ( 480,000)

A1

88>6?

8>6+%
5>/=A88>6 x 24 x 360 ?1• *

@

2048

@

5>/=!%•8>6+%*5>/=#1.0•

8>6."-&"$*#2.0(');>2*Reedbush&)5>/=, •2018/10/29

MPI

27

 Reedbush
 
201841 • HMCFK@MB• 150,000 PRB-U: 4GMENO&128GME60
RB-H: 1GMEN&5U42.5O& 32GME60
RB-L: 1GMEN&5U44O& 16GME60• ?KMI@MB• 300,000PRB-U 1
4GMENO&128GME60&
RB-H: 1GMEN&5U42.5O& 32GME60
RB-L: 1GMEN&5U44O& 16GME60•  RB-U47 360,000 P 1
4GMENO&128GME60•  RB-H47 216,000 P 1
1GMENO&32GME60•  RB-L47 360,000 P 1
1GMENO&16GME60• 5&'DM>L (0#• -"7GME%360%24$4'DM>L(+*9<;• GME605&DM>L +1.0 (H5.42.5, L54)• GME=!*;1&!*/ 5& +,93232;• 4JMA5Oakforest-PACS14DM>L8 
• GME8): (Reedbush-U, L)2018/10/29

MPI

28

 

• ?PcRe@Reedbush-U/H/L, Oakforest-PACSNOT^;9G6'VaIHb`eO7:EB6!VaIHb`eO7;8FC<5• HJU]SK(

gfRB-U: 16YeW, RB-H: 4YeW4
RB-L: 4YeW4OFP: 16YeWg• Lbe\MeOf1l9d g fRB-U: 128YeW4RB-H: 32Ye
W4RB-L: 16YeW4OFP: 2048YeWg• ZeQXbMeOf1l3d• %(

gfRB-U: 16YeW, RB-H: 4Ye
W4RB-L: 4YeW4OFP: 16YeWg"0,@ 0;34#'• Lbe\MeO• 'VaIHb`eOkfhd lid gk'fRB-U: 128
YeW4RB-H: 32YeW4OFP: 2048YeWg• !VaIHb`eOkfhd l2*jd g4!f/*1&
A'= )g• OeZeMc[_eR(1$-# @#;.f2g•
@MeO=D?4+>( @;.
• ZeQXbMeOfhlid

2018/10/29

MPI

29

 
 • $%#" •• • •  
!

https://www.cc.u-tokyo.ac.jp/guide/2018/10/29

MPIMPI (Message Passing
Interface)

30

2018/10/29

31

MPI

MPI
• i^\qYb^XpR$ElOelm-Eu@• i^\qYb^XpREj`nA7J• TpbOlE-6"E]daPQMHlOelmK?GEAFC

8r• ijm•.(-.(:*A,D;• ufo\^VD9<JijmVO[HdLOnVO[E)K&*E 8XZ_hsMassively Parallel Processing(MPP)XZ_htK$8J,D ;• fo\^V• ufo\^V(A+C,3E.(K6%3A#*• ':

• APIsApplication Programming InterfacetE!

• ZSqlcm_N6*:58• 1#KkqW:/0?J=BDIJMnUm[hE2:*• foRlgpR:4>8s:58t

2018/10/29

MPI

32

MPI
• MPI0,6327http://www.mpi-forum.org/8• 1994• 1995• 1997• 2008• 20095 1.07MPI-18
6 1.1
7 1.2 (' 2.07MPI-28
5 1.3 2008 6 2.19 2.2•  http://www.pccluster.org/ja/mpi.html• MPI-2 !&)9• I/O• C++Fortran 90*5/60+6• 14.- /• $#"%2018/10/29

MPI

33

MPI MPI-3.1• MPI-3.0 2012 9• MPI-3.1 2015 6• *0$/'",+.&!
• http://mpi-forum.org/docs/docs.html• http://meetings.mpi-forum.org• http://meetings.mpi-forum.org/mpi31-impl-status-Nov15.pdf•  • (./)-%".#

1MPI_IALLREDUCE2•   1RMARemote Memory
Access)• Fortran2008  2018/10/29

MPI

34

MPI MPI-4.0• -KV<*UC7NMTB1
• https://www.mpi-forum.org/mpi-40/• %0)"/
• D4HQ@CIS8PLT8.• Multiple Endpoint• MPI3IQ9V;OT-  WFault Tolerance,

FTX• "$(#-34A31

• Active Messages (M@>V<-ISB:R)••••+-6VEP@I-1"' 
"6VEVJ@C!F4IP4T
E@G2QT8,&*!4T?PIBDTCP*$• Stream Messaging•
ISG24RU4T?VG5V=

2018/10/29

35

MPI

MPI
• MPICHO:HMEBAP• 8J=KC'• MVAPICH (:HL7EBA)• ;D9;  . !MPICH6GN>• InfiniBand(35+ • OpenMPI• ;NFK?N>• GK@MPI• !305&'GN>21,-%4
: "#!FX103MPI: Open-MPIGN>Intel MPI, SGI MPE: MPICH!MVAPICHGN>• :IN<

'1*5-%4)/'$4

2018/10/29

MPI

MPI• /.

(A• . -
1. /"4/
2. .)*$52/0,'.#5&
3. .)*$52//!
4. .)*$52//
5. ?6 .5 /@?:8@• MPI+0A
1. /ID"%31"4/ID
2. ;>:/7<=93. ;>:
4. ;>:
5. :8

36

2018/10/29

MPI

37

MPI• MPI-@D<;,5.&@D<;-G

-H@D<>:G0%"-96H+)2(14.&

• CE7GRankH
MPI@D<;, ,#*• MPI)-MPI_Comm_rank) $43 +0J@D<;-1 ,!3• 9AB?8F=,MPI@D<;53'/+I
MPI_Comm_size5 •2018/10/29

MPI

38
• MPI_COMM_WORLD<*IUXRH[N8A=EDF3D • IUXRH[N<*F".#;SZMPJ!F?D• 7<*\ ^numprocs –] >7;SZMPJ/*]5;IUXRH[N: C6BED• 1;

/*(MPI_COMM_WORLD)• SZMPJ!F24-*MPI_Comm_split 'F• VPM[KF*&;SZMPJ!:

%3D80:
• “TYOGWLQ”7• :@+9/,D => $2018/10/29

MPI

39

MPI
/1.'%&ON• p p
• fl_UmW• /1.63;B:q /1.62;9Iq• dmfl_UmW• /1.6.F;B:q /1.6.E;9Iq• ao^• /1.64JD;6• 
• /1.6*97FG (/1.62;:H9;q /1.6)@@E;:H9;q/1.6*7EE>;Eq• WkogLYijcXo^• /1.6+CAA6:HD ( /1.6+CAA6FD@>G (• gl]\bhl[• /1.6+7EG69E;7G; (• OP• /1.6.B>Gq /1.6+CAA6E7B?q/1.6+CAA6F>K;q /1.6,>B7@>K;q•• /1.65G>A;• .B<CTf[SVb• gl]\ n• 
• /1.61HG ( /1.6-;G (• Rm^eSo\• eQRk.0 !/1.$.0"• /1.6,>@;6CD;B#• ,CEGE7BM+O8>B:>B=• `okZhob

2018/10/29

MPIMPI



40

2018/10/29

C
FortranMPI








 ierr ierr = MPI_Xxxx(….);• Fortran   ierrcall MPI_XXXX(…., ierr)• 
•

• 

MPI_Status istatus;• Fortraninteger istatus(MPI_STATUS_SIZE)

41

2018/10/29

C
Fortran• MPI!$

MPI




42






• C
MPI_CHAR ( )  MPI_INT ()
MPI_FLOAT ( )MPI_DOUBLE( )• Fortran
MPI_CHARACTER ( ) MPI_INTEGER()MPI_REAL ( )
MPI_DOUBLE_PRECISION(  ) 
MPI_COMPLEX( )• %# "$ 2018/10/29

43

MPIMPI―MPI_Recv• ierr = MPI_Recv(recvbuf, count, datatype, source,tag, comm, status);• void * recvbuf (OUT):

 +2 #0
• int count (IN): +=D:2 #0
• int datatype (IN): +=D:+ 2 #0
• MPI_CHAR () MPI_INT ( )
MPI_FLOAT (  ) MPI_DOUBLE(



))(

• int source (IN):"$@;9D72#0PE+
BC32 #0

• +PE/"$'!,MPI_ANY_SOURCE 2

• int tag (IN): "$@;9D7*%0:4+2• +:4+@;9D72"$'!,MPI_ANY_TAG 2

• MPI_Comm comm (IN): 6?A>5D:2
• &,MPI_COMM_WORLD 2#1-.• MPI_Status status (OUT): 8<D:8

2018/10/29

44

MPIMPI―MPI_Send• ierr = MPI_Send(sendbuf, count, datatype, dest,tag, comm, status);• void * sendbuf (IN): $+• int count (IN):  $4;2+• int datatype (IN):  $4;2$+
• int dest (IN):  $9:,+• $PE("%MPI_ANY_SOURCE +

• int tag (IN): 731;0#• )2-$+

$2-$731;0+"%MPI_ANY_TAG +

• MPI_Comm comm (IN): /685.;2+• !%MPI_COMM_WORLD +

*&'

2018/10/29

MPI

45

Send Recv  
PE0

PE

PE

PE

MPI_Send 

MPI_Recv

2018/10/29

46

MPIMPI―MPI_Bcast• ierr = MPI_Bcast(sendbuf, count, datatype,root, comm);• void * sendbuf (IN/OUT) : (root)(root

)



• int count (IN) : %-$• MPI_Datatype datatype (IN) :  %-$• int root (IN): '+#"*,
• MPI_Comm comm (IN): !()& -$*,


!!

2018/10/29

 MPI

47

MPI_Bcast
PE0

PE

MPI_Bcast()

MPI_Bcast()

root

PE

PE

MPI_Bcast()

MPI_Bcast()

2018/10/29

MPI

48


• IJ+/'(IJ2

D@:67>BE#%0
• H


=6;ADKE
E• @:67>B-

0→85?DF

*2• Dcollective communication

operationE*.10• ,&,!)
3B9<4C8"
$0

G,

*$

2018/10/29

49

MPI
•PE• MPI_Reduce• 

PE

PE0
PE0
PE1




PE2

PE0

• MPI_Allreduce• 


PE0

PE1

PE
PE0


PE2

PE1

PE2

2018/10/29

MPI

50

MPI―MPI_Reduce• ierr = MPI_Reduce(sendbuf, recvbuf, count,datatype, op, root, comm);• void * sendbuf (IN) : 1 • void * recvbuf (OUT) :  1 • . 2" -$+,2030%#(05*"046'0&,2030%#• int count (IN) : 1=E<• MPI_Datatype datatype (IN) :  1=E<
• MPI_Op op (IN): 1!
• MPI_SUM ()" MPI_PROD ()" MPI_MAX ()"MPI_MIN ()" MPI_MAXLOC (.)1)" MPI_MINLOC (.)1) 0/#• int root (IN): ?C;:1BD7
• MPI_Comm comm (IN): 9@A>8E<

2018/10/29

51

 MPIMPI_Reduce
PE0

PE

MPI_Reduce()

MPI_Reduce()

PE

PE

MPI_Reduce()

MPI_Reduce()

iroot




iop





2018/10/29

52

MPI

MPI_Reduce 

MPI_2DOUBLE_PRECISION  MPI_MAXLOC

PE0

PE

MPI_Reduce()

MPI_Reduce()

PE

PE

MPI_Reduce()

MPI_Reduce()

iroot
3.1

4.1

5.9

2.6

2.0

5.0

9.0

13.0

MPI_MAXLOC
5.9
9.0

LU





2018/10/29

MPI

53

MPI―MPI_Allreduce• ierr = MPI_Allreduce(sendbuf, recvbuf, count,datatype, op, comm);• void * sendbuf (IN) : 0
• void * recvbuf (OUT) :  0• - 1!,#*+1/2/$"'/4)!/35&/%+1/2/$"• int count (IN) : 09=8
• MPI_Datatype datatype (IN) :  09=8
• MPI_Op op (IN): 0
• MPI_SUM ()! MPI_PROD ()! MPI_MAX ( )!MPI_MIN ()! MPI_MAXLOC ( -(0)! MPI_MINLOC (-(0) /."• MPI_Comm comm (IN): 7;<:6=8

2018/10/29

54

 MPIMPI_AllreducePE0
MPI_Allreduce()




PE

PE

PE

MPI_Allreduce()

MPI_Allreduce()

MPI_Allreduce()iop















2018/10/29

MPI

55

MPI―MPI_Gather• ierr = MPI_Gather ( sendbuf, sendcount, sendtype,recvbuf, recvcount, recvtype, root, comm);• void * sendbuf (IN) : *
• int sendcount (IN) :*7?6
• MPI_Datatype sendtype (IN) : *7?6• void * recvbuf (OUT) :*• (!&(+ '%&+)-)")/$).0!) &+)-)

• int recvcount (IN):*7?6
• MPI_Datatype recvtype (IN) : *7?6• root '!#PE*, (recvbuf, recvcount, recvtype)• int root (IN):9=54*<>1
• MPI_Comm comm (IN): 3:;82?6
sendcount * size = recvcount2018/10/29

MPI

56

MPI―MPI_Scatter• ierr = MPI_Scatter ( sendbuf, sendcount, sendtype,recvbuf, recvcount, recvtype, root, comm);• void * sendbuf (IN) : *
• int sendcount (IN) :*7?6
• MPI_Datatype sendtype (IN) : *7?6
root '!#PE*, (sendbuf, sendcount, sendtype)• void * recvbuf (OUT) :*• (!&(+ '%&+)-)")/$).0!) &+)-)

• int recvcount (IN):*7?6
• MPI_Datatype recvtype (IN) : *7?6• int root (IN): 9=54*<>1
• MPI_Comm comm (IN): 3:;82?6
sendcount = recvcount * size

2018/10/29

MPI

57


1.

=@80A2•

D&;8<. %>86C4,D&;8<. %/165B "+)" ( *$

••;8<. &9C7&MPI_Send, MPI_Bcast$#

2.

-

:A=@80A2• D&;8<.

%

(

• ;8<.

• +&9C7-!

&9C7& -!
&'?C3&2018/10/29

58

&MPI  • 0  
send   

"#! 0



"#! 1



"#! $



"#! %



…

send



send

recv



send


recv

…



 


recv










 

…

 
 

2018/10/29

MPI

59

 

• ierr = MPI_Isend(sendbuf, icount, datatype,

idest, itag, icomm, irequest);• sendbuf :  • icount : '+%

• datatype : '+%• idest :PEicomm )*!
• itag :(&$+#
%"



2018/10/29

MPI

60

 

• icomm : PE*(!(-463,80*
(•  !$MPI_COMM_WORLD *)%&
• irequest : MPI_Request9 #:*51/8." ') (• ierr :  +78-82(2018/10/29

MPI

61
• ierr = MPI_Wait(irequest, istatus);• irequest : MPI_RequestDE#7(+?=;C:/,%46+#• istatus :MPI_Status DE#/!)5 $5#• $MPI_STATUS_SIZE0

7(-)5#• (+?=;C:00@A8$
istatus[MPI_SOURCE] "<9$istatus[MPI_TAG] /'65#• >C<7

1*

2&.

)5B

>C<73)/2018/10/29

 

• &'#"0%($!

send $()%'#( 

&'"!0



send send

&'"!1



recv

&'"!*



&'"!+



…

62

,MPI

send

…


recv

!MPI_Wait 





recv



 








…







2018/10/29

63

MPI•   %• MPI_Send • MPI_Wait 

&• MPI_Isend • MPI_Wait

&
• 
&



!$#"2018/10/29

MPI

64


1.
2.
3.

4.
5.

Message Passing Interface Forum
3 http://www.mpi-forum.org/ 4
MPI*0!.,1!P.)' "  5   
*0!.,1! MPI   %1&
3 http://accc.riken.jp/HPC/training/text.html 4
MPI-J-2/1!/$(3 http://phase.hpcc.jp/phase/mpi-j/ml/ 4+2# (RIST)
http://www.hpci-office.jp/pages/seminar_text

2018/10/29

65

MPI

Intel MPI
• Intel Parallel Studio XE cluster edition&"?

C, Fortran.=5,;$&• MPICH, MVAPICH2+6>0&*")?

(

!")
•

'2019• (,=02><*)'• OFP#%4>/:=• 2018 update 1 / 2 / 3• • Web7>1<=>3-98=2
• https://software.intel.com/en-us/mpi-developer-guide-linux• https://software.intel.com/en-us/mpi-developer-reference-linux• https://software.intel.com/en-us/intel-software-technicaldocumentation?field_software_product_tid[]=20827

2018/10/29

66

MPIMVAPICH2• ,0+, $ Prof. D.K. Panda'-572
• Argonne National Lab.$!)#(MPICH37/(MPICH3)• 8&9InfiniBand:*"• $' ; 2.3• OFP$%17.46: 2.3a•  • http://mvapich.cse.ohio-state.edu• http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-

2.3-userguide.html

2018/10/29

MPI

67

Open MPI• LAM/MPI*• LAM: Ohio State Univ. => Univ. of Nortre Dam => Indiana Univ.

( 3.1.2•• 2.1.5• 3.0.2•( 4.0 '

• OFP"(2348&(/;35<:!%• "1;6/:(•  • https://www.open-mpi.org/doc/
• (

+

!,"()-*%

• https://www.open-mpi.org/faq/• MCA0729;'

%$(*.%#…

2018/10/29

68

MPI

Oakforest-PACSMPI• 



!



3

• Intel MPI 2018 Update1• OFP 2018/10
• Intel 2018u1 (intel/2018.1.163 impi/2018.1.163)• intelIntel MPI 2018 Update 3• Intel 2018u3 (intel/2018.3.222 impi/2018.3.222)• module switch intel/2018.1.163 intel/2018.3.222• MVAPICH2 2.3a
• Intel 2018u1  (intel/2018.1.163 mvapich2/2.3a)• module switch impi mvapich2

2018/10/29

 moduleMPI

69



• #.;H?9EJMPI

.-74$5*10;AH>• ?=3
.-.! %/ '65
• <D@ /2;H?9G,(module7load)5&,7Kmodule avail• 0B<CIG7"Kmodule list
• B<CIG0load: module load B<CIG• B<CIG0unload: module unload B<CIG• B<CIG04$Kmodule switch B<CIG B<CIG• B<CIG7+:F8: module purge• .B<CIG0

2018/10/29

70

MPI • MPI•

8HNBTIMRPWF+59#<6(/LWJ@A?4

V.3(=>-496(• Oakforest-PACS8LWJ@A?(OPA)978OUIDTEFGC• export I_MPI_FABRICS_LIST={tmi,ofi}
export I_MPI_FABRICS=KWJ:KWJ!
• shm: PQSXKWJ8:Y• tmi: Intel2018.1 49Omni Path8*00;• ofi: 

8'Intel2018.3"• 2018.1HNBTI49 shm:tmi '%$9 tmi:tmi 7.1+&• 2018.3HNBTI49 shm:ofi (tmi9)6,621Y'$ 49ofi:ofi+(((?)

2018/10/29

MPI

71

EagerRendezvous• EagerOTKBS• 503%(,<• 5• PJI,<84&!=6
• PJI*2)=6VLJN@3 => LJN@74BMU&• QJGUECAF

40'3*."<• RendezvousOTKBS• 43V&"<%1#%?*.%:<=> HTBMU/& 
• QJGUECAF 40'3*."<
•  VQJGUECAF3+.3;><• ;$5RUD39-.8• export I_MPI_EAGER_THRESHOLD=QJGUE (IntelMPI)• export MV2_IBA_EAGER_THRESHOLD=QJGUE (MVAPICH2)

2018/10/29

72

MPI  !
• #MPI_Allreduce
•

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.





Recursive doubling
Rabenseifner’s
Reduce + Bcast
Topology aware Reduce + Bcast
Binomial gather + scatter
Topology aware binominal gather + scatter
Shumilin's ring
Ring
KnomialTopology aware SHM-based flat
Topology aware SHM-based Knomial
Topology aware SHM-based Knary• export I_MPI_ADJUST_ALLREDUCE=8:1024-4096@16-32;…
• 1024~4096 Byte 16~32"Ring! 2018/10/29

MPI

73

 

• MPI-3.0•  



MPI_Ixxx


MPI_IN_PLACE

Intercomm









One-toAll

MPI_Bcast

MPI_Ibcast

MPI_Scatter{,v}

MPI_Iscatter{,v}

recvbuf@root✔

All-toOne

MPI_Gather{,v}

MPI_Igather{,v}

sendbuf@root✔

MPI_Reduce

MPI_Ireduce

sendbuf@root✔

All-to-All

MPI_Allgather{,v}

MPI_Iallgather{,v}

sendbuf✔

MPI_Alltoall{,v,w}

MPI_Ialltoall{,v,w}

sendbuf✔

MPI_Allreduce

MPI_Iallreduce

sendbuf✔

sendbuf✔

MPI_Reduce_scatter MPI_Ireduce_scatter{,_block}{,_block}
OthersMPI_Barrier

MPI_Ibarrier

MPI_Scan, Exscan

MPI_Iscan, Iexscan✔

✔

sendbuf2018/10/29

MPI

74


•  ' "1!$• collective' + "1sendbuf, recvbuf+#/(256+*/=> 0.+MPI_IN_PLACE1• +-%&7sendbufrecvbuf+8463,1• ,1!$• MPI_STATUS_IGNORE• MPI_Recv*)'7MPI_Status ,.*• MPI_STATUSES_IGNORE• MPI_Waitall*)' MPI_Status [],.

• MPI_ERRCODES_IGNORE• Fortran'ierr,.

*

*



2018/10/29

MPI• MPI





75

2018/10/29

Allreduce• 16!


10000

• Intel MPI• tmi:tmi• 1000 (usec)• 1024

76

MPI

default

1

2

3

4

5

6

7

8

9

10

11

12

100

10

1
4

64

1024
16384
! (Byte)

2621442018/10/29

MPI

Allreduce (1677x64)•  
• export I_MPI_ADJUST_ALLREDUCE=“10:1-32;12:3316384;5:16385-32768;10:32769-65536;2:65537-524288;8:5242891048576”
10000
default
5
10
OPT



(usec)

1000

2
8
12

100

10

1
4

32

256
2048
16384


131072 1048576

2018/10/29

MPI

 
 

78

2018/10/29

79MPI 

• ierr = MPI_Type_vector(count, blocklength, stride, oldtype,newtype)• int count (IN): $stride%• int blocklength (IN):!" $#• int stride (IN):!"
• MPI_Type oldtype (IN): #
• MPI_Type * newtype (OUT):  #% MPI_Type_commit!!

•

blocklength
0

1

2

3

4

5

stride

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22 …

2018/10/29

MPI 

• ierr = MPI_Type_indexed(count, array_of_blocklengths,

array_of_displacements, oldtype, newtype);• int count (IN):• int array_of_blocklengths[] (IN):• int array_of_displacements[] (IN):• MPI_Datatype oldtype (IN):• MPI_Datatype *newtype (OUT):



 


802018/10/29

MPI 

(2)81• ierr = MPI_Type_create_hindexed(count, array_of_blocklengths,

array_of_displacements, oldtype, newtype);• int count (IN):!#
• int array_of_blocklengths[] (IN):!#
• MPI_Aint array_of_displacements[] (IN):!#
% "&• MPI_Datatype oldtype (IN):$
• MPI_Datatype *newtype (OUT):
$• MPI_Type_indexedarray_of_displacement

• MPI-2.0MPI_Type_hindexed2018/10/29

MPI  

82



• ierr = MPI_Type_create_struct(count, array_of_blocklengths,

array_of_displacements, array_of_types, newtype);• int count (IN):• int array_of_blocklengths[] (IN):
• MPI_Aint array_of_displacements[] (IN): 
• MPI_Datatype array_of_types[] (IN): 
• MPI_Datatype *newtype (OUT):• MPI-2.0MPI_Type_struct2018/10/29

MPI 
MPI-IO83

2018/10/29

MPI

84

Oakforest-PACS 
PATH

+



/home/[IG\NFS[IG\R^Q0C>@  !
(16[IG\=$<A>] &TFG
Z<;,)>A>8435:/work/IZ^V/[IG\(Lustre)[IG\R^Q0CA%R^Q0CA
 !`"<'=/D
SNMKYU>?550C#.

/cache/IZ^V/[IG\
ime://work/IZ^V/[IG\ _MPI-IO)

S^L
PSN
TF
(IME)

Lustre>HWNJX>B.= [IG\R^Q0C? 92<>9LO^K\I1$

Ramdisk*=(16E7<-/tmp/dev/shm2018/10/29

MPI85

Oakforest-PACS  • Lustre>45C78<A•>45C "B:=E:/+!
->45C78<A• =E:/3>45C',.
()• 6E?D9E8 (2*%0#-@D;E&)*$1•  FEFS: Lustre>45C78<A/F
Reedbush

2018/10/29

86 MPI

Lustre

• & "(  MDS/MDT (Meta Data Server/Target)• $') *OSS/OST"(  
• "(  OSS/OST (Object Storage Server/Target)• $'"( • OFP
• MDT+3
 (DNE: Distributed Namespace)• OSS+RAID6 (8D2P)  41 + 10 % 10!# => 26 PB
D:
P:







…




2018/10/29

87

%MPI

Lustre
• AL@/• MD9:H/AL@81,)/OST.'6#/OST.
&*'6#0GL=$ +%6• AD;HB+0MD9:H(51,)/OST.D9:H
+'6OST$2476• lfs getstripe / lfs setstripe<FJC+K  • EI?>#4/D9:H.'68&(!

.0/OST8"3"-










$




$$1OST  !#











OST  !#" "

2018/10/29



Lustre

88

!MPI



• lfs setstripe [OPTIONS] <:1B4;@|<02A>• *3=6>CE-s size -c count• 7;?2=528 size +count ,OST+'(:D9/$-) ! +#&,<02A/$•  :1B4;@+ #(!)%, +-<02A+".• count+-1/ $-)OST/
• 

4
1 6 .
.
09 9/ 9/ 9$.. 0 ./ =/ 6 60

  

1 6 .
1 6 .
.

.
9 $

• lfs getstripe <:1B4;@|<02A>•   /$• lfs df• MDT/OST/$-

-6 5

2018/10/29

MPI

89 (DDN: Infinite Memory
Engine)• SSD•!1)25250(1'-0&*.$  • %+/&3 9D+1P • ,1"

export IM_CLIENT_PGEOM=<D>+<P>• %1#2*.$ export IM_CLIENT_PGEOM=10+0

2018/10/29

MPI

90

MPI-IO •nghqijloeMPIY",\T&)XQ]Y,\(Ob• Db+7eQPQ39eObLUTAu(.H$XObLUWJv =W1H (/XWbuLUH!Mcbv• w (.ZnghqYmtkH> YVLX>-McS#NEGRKe→ MPI-IOX`aB[U]S6\8fTMPI:T >>-CM
cS@;X 'H0dcbAG^*cWE•  API Z C24TY2_

FortranT^Y?H(e5ObHA(TIb)WY<EWVZpngrsje%YLU• C++2Z^FdcSEWEAC2e (Ob•2018/10/29

91

MPIMPI-IO
I/O1







•

• 1DJ=<01(I/O5&$(MPI)!/23@L>5
K#+4• MPI_Gather + fwrite• fread + MPI_Scatter

•



File

• C68I/24.3*• 1)0 • • <9LGBH?70 "•   5:ELA*C68I;<?F05',.%memorymemorymemoryppp2018/10/29

92

MPIMPI-IO
I/O2• 
• =C76%I/O, memoryp• • <-0B569?&

4>D:,#+• 63D@;A9/ 
• 
• 

File

&

• &<-0B.176• <-0B%)+

$• =C2@?E8D6F('/

*"
%!



memorymemorypp2018/10/29

93MPIMPI-IO  • "




• $
• #

I/O

! 
%$&

memorymemorymemory

p

p

p



2018/10/29

MPI

94
• MPI-IO'A*-54(#GIED7=?

FBCH:8
+?
• 9



A

7"*-5A

85 8%A5.(!72)39-1@>6)•
FBCH7/?80;(8!<,0)79lfs setstripe8&<$2018/10/29

MPIMPI-IO
• 

•



Pe
l
Ip
Pe
l

fn
oF
m

,* ) 11,* (//3. 2 1 . 1.
1,* * 22,* ) 1 23,* ) 1 /,* ) 1 23,* ) 11 p
Mts,* ) 1 / 1 p

re

i

_

l
h a

M

s
e

C
lcna

e

lcna

1

95

r
dn

I

2018/10/29

  •

• _at read/write• MPI_File_seek 

)*
*
)*
/
) *(//
//
,

 



  
F

P
MO

FOa

df
"'-(*,-%&.

• MPI_File_set_view 

)*
*
)*
/
) *(//)*)*/



"'-(*,



*
F

P
MO

FOa

h
/

F
e

,
)* /

96

0MPI

c

_
I D

+/)

O
! +/)

$#

2018/10/29

MPI



97• "view#=$*&6+(-8!/IO>SCNQ%• JPG?Q@R• MPI_File_write• JPG?Q@R• MPI_File_iwrite / MPI_Wait• JPG?Q@R• MPI_File_write_all• JPG?Q@R• MPI_File_write_all_begin / MPI_File_write_all_end• 0 <),$(:• JPG?Q@T6&%<9/',82=-&:•  T BLMHASF2KPED%<14/9/$ 5.7,<;:*76&232018/10/29

98

,MPI MPI_File_set_view/write• 

MPI
 



= ; 9 SM TNP3
= ; I[
3
U L W3
U LI I3U
PI N SMUITMFG 4 LI I* LI 3

= ; ;U " I O & I O 3= ; 6 TT IUR "= ; 6 == E A 7&

1)OFP 16%'! *PM L[TW LI I* LI))))))) )))* )))) )))+ )))) ))), )))) ))) )))))))))*) ))). )))) )))/ )))) )))0 )))) )))1 )))))))))+) )))2 )))) )))I )))) ))) )))) ))) ))))))))),) )))L )))) )))M )))) )))N )))) ))*) ))))))))) )IUR 3

L W 4 _M N" U $ IUR3 (( tprmlewfuixvz
LI I 4 * IUR3 (( wfuhgdoan
zb= ; sk *c= ; 9 SM= ; ; 9

WMU"= ; 6 == E A 7& N SMUITM& = ; = 78 A7EA = ; = 78 6A85C8&
D & TNP 3= ; 9 SM M
M]"TNP& L W& = ; ; C8 8A& = ; ; C8 8A& UI= ; 9 SM ] M"TNP& LI I& *& = ; ; C8 8A&
3= ; 9 SM S M" TNP 3= ; 9 UIS _M" 3

M &= ; ; 9D• #("  • +$&

32018/10/29

MPI 

99



• MPI_File_iwrite / MPI_Wait
) ;
; E
) ),
,1 "
) ; ;E ;
)
;
"

;

) ;
) ),
) ;

;

) ), &(&) ), &(&) ), &(&• MPI_File_write_all
; E
,1 "
E;

;

"

) ), &(&) ), &(&) ), &(&;"• MPI_File_write_all_begin / MPI_File_write_all_end
) ;
) ),
) ;
) ;

; E
,1 "
E;
E;

;

;

;

E;

) ), &(&;) ), &(& ""• MPI_File_write_at)) ), &(&;) ), &(&"

  MPI_File_open
MPI_File_close   

2018/10/29

MPI

100

  • MPI_File_set_viewUdatarep$TXZerd4*TXOP#]GKZFRCQDZ• “native”: jkoRIQU4*sW"HS@t• “internal”: IMPI5],JZRD;<CS@/U"• “external32”: MPI],JZ:Y;<CS@XAT"• LUU^hamqVMPI(&])• >View?] \S@ +V# C 7G[S@• MPI_File_open → MPI_File_wite_at → MPI_File_close V689C
SE0NC=# C@•-S• Oakforest-PACSR`qglrdQIhp_niscrb`rft] @M@• nativeQV`qglrdROakforest-PACSU2'C3HS@• external32QV2'C3JZs`qglrdU2'T!At• Oakforest-PACSQWMPI+1]%JZR2'C\ZBW.

[S@

2018/10/29

101

MPI • G>?KI?MF= .;,291+;• MPI_File_read/write_shared1 G>?KI?MF=(0• 8+*HLDC58+49".;$%@NE
K&5* -<;• HLDC1#4O #4PQG>?K=8+.;:)3 4)&
2•• RLBG>?K75

,G>?KI?MF=
(/! 9';,

• MPI_File_read/write_ordered• #*-<;• JMA#4 -<;•  163(

)
FF
&

,

)

2
CD

,

& ,,

#I

,

,

#,

,

)
#

,

)

(
E

F

#&

2# ,

; CF

#
)

E
2#

F

#

2018/10/29

MPI

102




non-collective

collective





blocking

MPI_FILE_{READ,WRITE}_AT

MPI_FILE_{READ,WRITE}_AT_ALL

nonblocking

MPI_FILE_I{READ,WRITE}_AT

MPI_FILE_I{READ,WRITE}_AT_ALL

split collective

N/A

MPI_FILE_{READ,WRITE}_AT_ALL_{BEGIN,END}



! 
"

blocking

MPI_FILE_{READ,WRITE}MPI_FILE_{READ,WRITE}_ALL

nonblocking

MPI_FILE_I{READ,WRITE}MPI_FILE_I{READ,WRITE}_ALL

split collective

N/A

MPI_FILE_{READ,WRITE}_ALL_{B
EGIN,END}



! 
"

blocking

MPI_FILE_{READ,WRITE}_SHA
RED

MPI_FILE_{READ,WRITE}_ORDER
ED

nonblocking

MPI_FILE_I{READ,WRITE}_SH
ARED

N/A

split collective

N/A

MPI_FILE_{READE,WRITE}_ORDE
RED_{BEGIN,END}

OFP MPI_FILE_WRITE_SHARED  %#$

"2018/10/29

103

 MPI

Collective IO• MPI  IO =>Collective IOIO 
rank0

rank 1

rank2

noncollective

rank3


IO


collective

rank0

rank 1

rank2

rank3





IO

2018/10/29

104

+MPI  

• /work  /cache #"
+"'*&)

(,*%• 2(,/work/group1/user/file1/cache/group1/user/file1!($%
 #
 

(,*% +(, "&)'.$

$%


• "&.!-!( #


• 

2018/10/29

105

MPI  



*&

!#/
.

*&! )/+ %-(")!#/.
• !#/ .,!$!#/ $,!$%-
:
input_file.txt, output_file.txt•*&0%-!#'%-(") !#'•*& : %-(")!#' %-!#'a.out
inp1.txt
inp2.txt
INP_DIR

a.out
inp1.txt
inp2.txt
INP_DIR



*&


!#/
$
a.out
inp1.txt
inp2.txt
INP_DIR

a.out
inp1.txt
inp2.txt
INP_DIR

out.txt

out.txt

%!#'(/work/group1/user)%-(")!#'(/cache/group1/
user)

2018/10/29

106MPI





#!/bin/bash#PJM –L rscgrp=regular-cache#PJM –L node=16#PJM –g group1#PJM –x STGIN_LIST=input_files.txt#PJM –x STGOUT_LIST=output_files.txt
cd $PJM_O_CACHEDIR
prestage-wait
mpiexec.hydra –n ${PJM_MPI_PROC} ./a.out/cache300 

2018/10/29

•

"MPI

107   








$ cat input_file.txt# comment/work/group1/userA/filename1 # Filename2# 
file?.*# ! (*, ?)output_dir/#  

2018/10/29

MPI

108

  !
• +-2*'(. $!
*0/"+-2*'(.%
#!

• +-2*'(.&3/1,)4$ sync-wait -f output_files.txt• +-2*'(.
&$ sync-list -f output_files.txt -v
StatusClean
Dirty Pending
Unsync0 1,048,576
0/work/group1/username/filename.1
Sync
1,048,576
00/work/group1/username/filename.2

2018/10/29

109

&MPI(%. /cache &• >BP=6O

+&-!

"%!$• )8JA<K)(1,/work *R

59@>1-!Q#$• >BP=57C%(E46N+ /work &+;6? 0 *E46N'(/-!
• [] E46N8JA<K<>
BI2 #
E46N2!0)+ime-rm:H
OD20/-!
• >BP=57C#E46N.

&!
• =LF>9MGC /cache *
E46N+ rm %./"3##2018/10/29

MPI

110
• E>F?N<SG;S@* 03&HTD• 2* L9<QUsingle shared fileV• KTJ85-1"+CATQU '+V• $HTD%• $HTD8 2CGTBS@• $L9<Q• CGTB:=I/.#2!(• CACHE_TMPDIR8• )24 /cache/tmp/BOM/ H;R?IP+,76

• W export TMPDIR=$CACHE_TMPDIR

2018/10/29

MPI

111

  • KPGAB?F3 %&R1MBS• Fortran.3 94+<$C.3write, read2 ;1MB/(-&,'!
• J=?OB?F; %&R1GBS• J=?O1*: I/O2.67!
• 16TB;":J=?O2
3 8#)5J=?OCE
HM2EIN?L$.*• Lustre201J=?O316TB4.

EIN?L(16TB1(0!/EHQD>@I.%0&094*2018/10/29

MPI• 

IO• MPIIO• File Per Process (FPP)• Single Shared File (SSF)• block• stripe or cyclic

• IME



112

2018/10/29

MPI



 



113

2018/10/29

MPI

114


• MPI*"'A+*)0MPI:>65+

1

%.+
•
-+4;<93@7• MPI_COMM_WORLD : MPI

+:>65• MPI_COMM_SELF: +:>65• 4;<93@7+
• 2?8=4;<93@7 (Intra-communicator)• "*#A• !+,$&/

#

• 2?7@4;<93@7 (Inter-communicator)• )02?8=4;<93@7*%0:>65(+*

02018/10/29

1.

  • MPI_Comm_dup, MPI_Comm_idup :• MPI_Comm_split :

2.

115

MPI






• MPI_Comm_create•  • MPI_Comm_free





2018/10/29

MPI

116

  • 9=573/$

MPI_Comm_split /• ierr = MPI_Comm_split (comm, color, key, newcomm);• MPI_Comm comm (IN) :)(.2:;81?6• int color (IN):!color/%<>0 !2:;8
1?6).
• int key (IN):key* 9=54,@0,)<>0/-&.
• MPI_Comm* newcomm (OUT):  2:;81?6• comm,@!color/%*2:;81?6)
• newcomm)+9=54*+(• comm*9=54# ".'2018/10/29

117

MPIMPI_Comm_split


0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15

color

0

0

0

0

1

1

1

1

2

2

2

2

3

3

3

3

key

7

8

3

2

1

5

4

2

9

1

2

3

0

0

4

2

MPI_Comm_split
newcomm
newcomm(comm0)
2

3

1

comm1
0

0

3

2

comm2
1

3

0

1

comm3
2

0

1

3

2

1

0

3

2

118

2018/10/29

MPI  (1)• 



• ierr = MPI_Comm_group(comm, group);• MPI_Comm comm (IN): 
• MPI_Group* group (OUT):  



119

2018/10/29

MPI  (2)&.0*$ 3$
• ierr = MPI_Group_incl(group, n, ranks, newgroup);•• MPI_Group group (IN):• int n (IN):!&.0*
&.0*"!+/)• int ranks[] (IN):&.0*"!,/%-'(12• MPI_Group* newgroup (OUT): "&.0*• &.0*$!
##!
• MPI_Group_union, MPI_Group_intersection, MPI_Group_difference• MPI_Group_excl, MPI_Group_range_{incl,excl},…

120

2018/10/29

MPI  (3)• ",.)$*+(#.'!
• ierr = MPI_Comm_create(comm, group, newcomm);• MPI_Comm comm (IN):$*+(#.'• MPI_Group group (IN):",.)/comm• MPI_Comm* newcomm (OUT):  $*+(#.'• comm)-&%





2018/10/29

MPI

121

 SUMMA
• SUMMAScalable Universal Matrix

Multiplication Algorithm
Ø

R.Van de GeijinØ


1997

2018/10/29

 MPI122

SUMMA 
• 

A

B
 !/#•• 

A

B

2018/10/29

123

MPI

SUMMA•• 0:@62A3CEMPI_BcastD& %/'! "(. *)(/
• SUMMA) #/<?52=48+*F FCEMPI_IsendD& %/$'&'*1B9>6;C  D• *G476;0,-) 

A

B

A

B
…

2018/10/29

124MPI• MPI360/(• ('#%360/4127.$+"'$+&• )(360/

*%

,
• ! 




57-(

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

2018/10/29

MPI

125


 %$,/)( %'-.+&0*%# • ierr = MPI_Dims_create(nnodes, ndims, dims);•• int nnodes (IN):,/)(• int ndims (IN):! • int dims[] (INOUT):  !,/)(•!,/)(

"$2018/10/29

126MPI   • ierr = MPI_Cart_create(comm_old, ndims, dims,

periods,reorder, comm_cart);• MPI_Comm comm_old (IN): %.39:72>6• int ndims (IN):&• int dims[] (IN):& (ndims)• int periods[] (IN):$(ndims)• int reorder (IN):;=10*.$• MPI_Comm *comm_cart (OUT):39:72>6• comm_old% .8<54) .
• dims%'MPI_Dims_create()# -/",&0!&++ (2018/10/29

MPI

127


• ierr = MPI_Cart_coords(comm, rank, maxdims, coords);• MPI_Comm comm (IN):• int rank (IN):• int maxdims (IN):• int coords[] (OUT):.120-5/34, (comm%6
(%)• comm&MPI_Cart_create$.120-5/• 34,

&(create$reorder=1#()" *)"$ => MPI_Comm_rank!' 34, +)2018/10/29



128

MPI• ierr = MPI_Cart_shift(comm, direction, displ, source, dest);• MPI_Comm comm (IN):• int direction (IN):• int displ (IN):• int * source (OUT):• int * dest (OUT):

0563/91
%
("%
($# !%)78.
78.

• %

%)$-'492%78. *-,

0

1N

• MPI_Cart_shift(comm, 0, 1, &north, &south);45• MPI_Cart_shift(comm, 1, 1, &west, &east);• %+&Fortran

2

3

6

7

8

9 S 10

11

12

13

15

W

E

14

2018/10/29

HybridMPI

129

2018/10/29

130

MPI

MPI  •(+!/"*.1#-&'• MPI_THREAD_SINGLE• ),%#-&'• MPI_THREAD_FUNNELED• )#$/#-&'MPI• MPI_THREAD_SERIALIZED•MPI0

• MPI_THREAD_MULTIPLE• ),%#-&'





2018/10/29



• SingleParallel!

• Funneledmaster• Serialized, Mutiple 
• 

131

MPI

Single#pragma omp parallel{....
}
MPI_Send( ... );#pragma omp parallel
{
....
}



Serialized,
Multiple

Funneled#pragma omp parallel{....
#pragma omp master
MPI_Send( ... );....
}Master only
parallel #pragma omp parallel{....
MPI_Send( ... );....
}2018/10/29

132

MPI MPI• MPI_Init()

MPI_Init_thread()• C:

int provided;
MPI_Init_thread(&argc,&argv,MPI_THREAD_FUNNELED,&provided);• Fortran

integer required, provided
required=MPI_THREAD_FUNNELED
call MPI_Init_thread(required, provided, ierr)2018/10/29

MPI

133

Probe / Iprobe
$,$=D9-:1;2 6<D9
6%(-• ierr = MPI_Probe (source, tag, comm, status)• ierr = MPI_Iprobe (source, tag, comm, flag, status)•• int source (IN), int tag (IN), MPI_Comm comm (IN), MPI_Status*status (OUT): MPI_Recv"• int *flag (OUT):
flag=TRUE%"status-• A;8D54/7-?;@.4/7-

• nonblocking%
•$* &EMPI_Wait,'!+#
• MPI_Probe$* ?;23B0C>%)2018/10/29

MPI

134

MPI_ProbeMPI_Status status;
int count;int *rbuf;
MPI_Probe( MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status );MPI_Get_count( &status, MPI_INT, &count );// status%*/3.-
MPI_Alloc_mem( count*sizeof(int), MPI_INFO_NULL, rbuf ); 012 
MPI_Recv( rbuf, count, MPI_INT, status.MPI_SOURCE, status.MPI_TAG,
MPI_COMM_WORLD, MPI_STATUS_IGNORE );  (  !#!&),+' "##$"   "##$" 

2018/10/29

135

MPI probe• MPI_Probe, MPI_Iprobe!

• MPI_Mprobe, MPI_Improbe
MPI_Imrecv



 MPI_Mrecv,• ierr = MPI_Mprobe(source, tag, comm, message, status)• ierr = MPI_Improbe(source, tag, comm, flag, message, status)• ierr = MPI_Mrecv (buf, count, datatype, message, status)• ierr = MPI_Imrecv(buf, count, datatype, message, request)•

MPI_Message *message : Mrecv/Imrecv






2018/10/29

MPI

136

MPI_THREAD_MULTIPLE• 7</@8%3=B8"$' +• ;@9 ( ((#- : Open MPI• MPI_Init_thread ) provided%

.#)"• (*'• 4A69% (50,2>?:1B5.
-!&%.#- -•2018/10/29

137

MPI

Multiple Endpoint
• Intel"%& '&!
• 2018.3  (?)



• export I_MPI_THREAD_SPLIT=1• export I_MPI_THREAD_RUNTIME=openmp• export PSM2_MULTI_EP=1• export I_MPI_THREAD_MAX=<"
Multi-EP• ,0(.-# $ )/*+>

Rule: Thread N in rank A has to communicate with Thread N in rank B in thesame communicator.
Thread0Communicator 0

Thread0

Thread1

Thread1

Communicator 1

Thread0

Thread1

2018/10/29

MPI

138

 /One sided communication

2018/10/29

139

MPI•  "23+*"$ •  " !#!%• "• 04/'(& Remote DMA (RDMA)• -4,)14"• )*."

"2018/10/29


 A

140

 MPI

 B

vs


 A

 B

MPI_Send

MPI_Get

MPI_Recv

2018/10/29

MPI 
1.
2.
3.
4.
5.
6.

Window  
 Window 



141

2018/10/29

142

MPI

Window 
• A?D7>?A'-42Window&



• 

$
• ierr = MPI_Win_create(base, size, disp_unit, info, comm, win);• void *base (IN):*(C8* >?A!'8B2• MPI_Aint size (IN): 0)3E:)7F• int disp_unit (IN):Put, Get
%$6D5!0)3• MPI_Info info (IN): ;C7 E F• MPI_Comm comm (IN):/=@9.D5• MPI_Win *win (OUT): #%$Window,<1+-7• /=@9.D5!>C:D"2018/10/29

Window 

• KINAHIK17><)Window0• &*0%.

143MPI• ierr = MPI_Win_allocate(size, disp_unit, info, comm, baseptr, win);• MPI_Aint size (IN): :3=OD3AP• int disp_unit (IN):Put, Get' $/.@N?+:3=• MPI_Info info (IN): EMA O P
• MPI_Comm comm (IN):
9GJC8N?
• void *baseptr (OUT):
42MB4*).HIK+1BL<
Fortran+ type(C_PTR)') ( ")O+0P• MPI_Win *win (OUT): -/.Window6F;57A• 9GJC8N?+HMDN!

,#(2018/10/29

144

MPI

Window 
• ierr = MPI_Win_free(win);• MPI_Win *win (INOUT): Window2018/10/29

MPI

145

MPI_Put:  
• ierr = MPI_Put(origin_addr, origin_count, origin_datatype,target_rank, target_disp, target_count, target_datatype, win );• void *origin_addr (IN):(+!  &* $-"• int origin_count (IN):(+! $-"
• MPI_Datatype origin_datatype (IN): (+! $-"
• int target_rank (IN):"-#%),
• MPI_Aint target_disp (IN): "-#%,&
• int target_count (IN):"-#% $-"
• MPI_Datatype target_datatype (IN):"-#%$-"
• MPI_Win win (IN):,&'%• target_addr = window_base+target_dispdisp_unit (Window)2018/10/29

MPI

146

MPI_Get:  
• ierr = MPI_Get(origin_addr, origin_count, origin_datatype,target_rank, target_disp, target_count, target_datatype, win );• void *origin_addr (OUT):-0&%+/%3)2'• int origin_count (IN):-0&%)2'
• MPI_Datatype origin_datatype (IN): -0&%)2'• int target_rank (IN):'2#(*.1"• MPI_Aint target_disp (IN): '2#(*1+• int target_count (IN):'2#(* )2'
• MPI_Datatype target_datatype (IN):'2#(*)2'
• MPI_Win win (IN):1+!,$ "*• target_addr = window_base+target_dispdisp_unit (Window)2018/10/29


• MPI_Accumurate• MPI_Get_accumurate•
• MPI_Compare_and_swap• MPI_Fetch_and_op• AtomicMPI

147

2018/10/29

MPI

148


• -'

5./0


%$

•



•

(+&3!')5

1.
2.



,1%$

MPI_Win_fence
MPI_Win_post, MPI_Win_start, MPI_Win_complete,
MPI_Win_wait (PSCW)• *'"+&3!')50#2,1%$4&3!')
• MPI_Win_lock/MPI_Win_unlock, MPI_Win_flush2018/10/29

MPI

149

Fence
• .1#

4win !02+*
• "4 !
• ierr = MPI_Win_fence( assert, win);• int assert (IN): 0• MPI_Win win (IN): %$3-%'/)&(,
MPI_Win_fence(0, win);if(rank == 0) MPI_Put( … , win);
MPI_Win_fence(0,win);


444

2018/10/29

150

MPIPSCW
•    • post-wait:  start-complete:



CHAPTER 11. ONE-SIDED COMMUNICATIONSPROCESS 0

start(1,2)
put(1)put(2)
complete()

PROCESS 1

PROCESS 2post(0)
.
.
.
.
.
.
.
.
.
.
.
wait()

post(0,3)
.
.
.
.
.
.
.
.
.
.
.
wait()PROCESS 3

start(2)put(2)

complete()2018/10/29

MPI151

Post, Start, Complete, Wait
• ierr = MPI_Win_post(group, assert, win);• ierr = MPI_Win_start(group, assert, win);• ierr = MPI_Win_complete(win);• ierr = MPI_Win_wait(win);• MPI_Group group (IN): &($

• int assert (IN):0• MPI_Win win (IN): '!#• wingroup%'"(





2018/10/29

MPI

Lock/Unlock• 





• ierr = MPI_Win_lock(lock_type, rank, assert, win);• int lock_type (IN): MPI_LOCK_EXCLUSIVE or
MPI_LOCK_SHARED• int rank (IN):
• int assert (IN):0• MPI_Win win (IN):  • ierr = MPI_Win_unlock(rank, win);152

2018/10/29

153

MPI

Flush• "#'

$
• 
• ierr = MPI_Win_flush(rank, win);• int rank (IN):' %& • MPI_Win win (IN): &!#2018/10/29

154

MPI   (Fence)
 

 

MPI_Win_fence





MPI_Win_fence

MPI_Put

MPI_Win_fence

MPI_Win_fence

2018/10/29

155

MPI   (PSCW) 
MPI_Win_Post

MPI_Win_Start

 

MPI_Put

MPI_Win_complete
MPI_Win_wait2018/10/29

156

MPI (Lock)




MPI_Win_lock
Lock



MPI_Put

MPI_Win_flush

MPI_Win_unlock

/


Unlock2018/10/29

MPI


•2





• Lock• Fence• PSCW2N• N
•• MPI-advanced/onesided/157