2018/2/14

KNL

1

94  
 
KNL






  
hanawa  cc.u-tokyo.ac.jp

2018/2/14

KNL2• +K2018 214H I 10K00 - 18K00• K":F;G 4, 413)-$
• 'K • '%AE7CBK
9K30 - 10K00
• 10K00 - 12K00 Oakforest-PACSE76F.Oakforest-PACS89=B#
.KNL&• (12K00 - 13K30 @()• 13K30 - 15K00 KNL3/05* H'$J!%I• (15K00 - 15K15 )• 15K15 - 18K00 Oakforest-PACS14?6@D<> 2*('$+!%)•

2018/2/14

3

KNL

Oakforest-PACS
Oakforest-PACS

2018/2/14

KNL
• 

 : t00441
 gt00
3/14 17:00



4

2018/2/14




KNL



5



6

 FY 2 6
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Yayoi: Hitachi SR16000/M1
IBM Power-7.)4#"
&4*4#2+/4'
5 7 3 6

54.9 TFLOPS, 11.2 TB

Oakforest-PACS
Fujitsu, Intel KNL

T2K Tokyo

140TF, 31.3TB

Big Data &Extreme Computing

25PFLOPS, 919.3TB

Oakleaf-FX: Fujitsu PRIMEHPC
FX10, SPARC64 IXfx

BDEC System

50+ PFLOPS (?)

1.13 PFLOPS, 150 TB

Oakbridge-FX

Oakbridge-II

136.2 TFLOPS, 18.4 TB(4' 3$-/14$02
&4*4#2+/4' %0,
!&4*4#2+/4'

5-10 PFLOPS

Reedbush, HPE
Broadwell + Pascal
1.93 PFLOPS

Reedbush-L
HPE
1.43 PFLOPS 
&4*4#2+/4'2018/2/14

7

KNL

4• Oakleaf-FX ( PRIMEHPC FX10)• 1.135 PF, #2+/4&, 2012

• Oakbridge-FX (• 136.2 TF, 4 2018 3

 PRIMEHPC FX10)

6168 7, 2014 4

 20183• Reedbush (SGI, Intel BDW + NVIDIA P100 (Pascal))• )4&3$,/14$02%4*4#2+/4&• 3.3 PF, 2016 7  2020 6• ITCGPU$%(- (2017 3 ), DDN IME
(Burst Buffer)• Oakforest-PACS (OFP) ( Intel Xeon Phi (KNL))• JCAHPC (CCS5ITC)• 25 PF, TOP 5006 (2016 11 ) ( 1)• Omni-Path4!("'., DDN IME (Burst Buffer)2018/2/14

8

KNL  
1/3

Fujitsu PRIMEHPC FX10 FX10   

Total Peak performance: 1.13 PFLOPS
Total number of nodes: 4,800
Total memory: 150TB
Peak performance per node : 236.5 GFLOPSMain memory per node
: 32 GBDisk capacity: 2.1 PBSPARC64 IXfx 1.848GHz



20127~2018330

Oakbridge-FX
  FX1024576
  168




8

2018/2/14

KNL

9  
2/3-1

Reedbush-U

SGI Rackable   

Total Peak performance: 508 TFLOPS
Total number of nodes: 420
Total memory: 105 TB
Peak performance per node : 1209.6 GFLOPSMain memory per node
: 256 GBDisk capacity: 5.04 PB
File Cache system (SSD): 230 TBIntel Xeon E5-2695v4 2.1GHz 18 core x2 socket201671201691 92018/2/14

KNL

10  
2/3-2

Reedbush-H

SGI Rackable   

Total Peak performance: 145 TFLOPS+ 1272 TFLOPS
Total number of nodes: 120
Total memory: 30 TB + 8 TB
Peak performance per node : 1209.6 GFLOPS+ 10.6 TFLOPSMain memory per node
: 256 GB+ 32 GBDisk capacity (shared w/ U) : 5.04 PB
File Cache system (SSD, (shared w/ U): 230 TBIntel Xeon E5-2695v4 2.1GHz 18 core x2 socket+ NVIDIA Tesla P100 with NVLink x 2

201731201743 102018/2/14

11

KNL  
1682/3-3

Reedbush-L SGI Rackable

 

 

Total Peak performance: 1.43 PFLOPS(77.4 TFLOPS + 1356 TFLOPS)
Total number of nodes: 64
Total memory: 16TB + 4 TB
Peak performance per node : 1209.6 GFLOPS+ 21.2 TFLOPSMain memory per node
: 256 GB+ 64 GBDisk capacity (shared w/ U,H) : 5.04 PB
File Cache system (SSD, separately): 153.6 TB
Intel Xeon E5-2695v4 2.1GHz 18 core x2 socket+ NVIDIA Tesla P100 with NVLink x 4201710

2





201711

1





Reedbush-U+H+L = 3.36 PFLOPS
11

2018/2/14

12

12

KNL    
 (3/3)
Oakforest-PACS Fujitsu PRIMERGY CX600
Total Peak performance: 25 PFLOPS
Total number of nodes: 8,208
Total memory: 897.7 TB
Peak performance per node : 3.046 TFLOPSMain memory per node
: 96 GB (DDR4)+ 16 GB(MCDRAM)
Disk capacity: 26.2 PB
File Cache system (SSD): 960 TBIntel Xeon Phi 7250 1.4 GHz 68 core x1 socket

2016121

2017432018/2/14

13

 KNL

Knights Landing: Next Intel® Xeon Phi™ Processor

Oakforest-PACS 
Intel® Many-Core Processor targeted for HPC and SupercomputingFirst self-boot Intel® Xeon Phi™ processor that is binary
compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance• Intel XeonIntegration
Phi of(Knights
Landing)Memory on package:innovative memoryarchitecture for high bandwidth and high capacity• 11
Integration of Fabric on package
Three products• MCDRAM: KNL Self-Boot

KNL Self-Boot w/ Fabric

KNL Card(Fabric Integrated)(PCIe-Card)16GB+ DDR4(Baseline)

HotChips27
Knights Landing
Overview
KNL 
MCDRAMMCDRAM

2 x16
1 x4

X4
DMI

PCIe
Gen 3

D
M
I

MCDRAMMCDRAM

Potential future options subject to change without notice.
All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

EDC


:
16GB 696GB

3

Core

C
H
A
N
N
E
L
S

2 VPU
1MB
L2

MCDRAM

4

EDC

3
D
D
R
4

36 Tilesconnected by
2D MeshInterconnect

DDR MC

EDC

Core

EDC

Tile

D
D
R
4

MCDRAM: 490GB/ DDR4: 115.2 GB/=(8Byte 2400MHz 6 channel)
2 VPU

EDC

EDC

MCDRAM

Omni-path not shown

misc

Package

DDR MCEDC

MCDRAM

C
H
A
N
N
E
L
S

EDC

MCDRAM2018/2/14

14

KNL

Oakforest-PACS: Intel Omni-Path Architecture      Fat-tree
768 port Director
Switch12(Source by Intel)

22Uplink: 24

48 port Edge Switch
362 
Downlink: 24
1

...

24

25

...

48

49

...

72

5>4
;1+#

,/3%(7<6*0+-:=6=4 )
• -/29#& )• "?.:8# (5>4' !$2018/2/14

KNL

15 Oakforest-PACS
 
• BG=@E:G<H"I• :G<JL 100,000 L 8AG?( )$16AG?1+• :G<KL 200,000 L 16AG?( )$64AG?1+• 9EGC:G<• 400,000 ( 480,000) L 1
AG?1+• 0$%>G8F
• )

8AG?H

I$2048&+!

2AG?#360#24"/%>G8F&('465•
AG?1+0$>G8F(1.0•AG?7'5,$'*0$(2.0.-5• /DG;0FX10$Reedbush,/>G8F3 

2018/2/14

KNL

16 Oakforest-PACS
 !
• DI?BG<I>J$K• 100,000 L 1 8CIA( )&512CIA3-• 3 3• ;GIE<I>• 400,000 ( 480,000) L 1
CIA3• 2&'@I:H

8CIAJ

K&2048(-#• +"4CIA%360%24$1'@I:H(*)687• DI?BG<I>22CIACIA3-2&@I:H *1.0•
CIA9!)7.&!),2& *2.00/7• 1FI=2Reedbush.1@I:H5 •2018/2/14

KNL17

JPY (=Watt)/GFLOPS Rate
Smaller is better (efficient)
SystemOakleaf/Oakbridge-FX (Fujitsu)(Fujitsu PRIMEHPC FX10)
Reedbush-U (SGI)(Intel BDW)
Reedbush-H (SGI)(Intel BDW+NVIDIA P100)
Oakforest-PACS (Fujitsu)(Intel Xeon Phi/Knights Landing)

JPY/GFLOPS
125
62.0
17.1
16.5

2018/2/14

 

KNL • $%#"• • •  
!

18



•



http://www.cc.u-tokyo.ac.jp/system/ofp/
http://www.cc.u-tokyo.ac.jp/system/reedbush/2018/2/14

19

KNL 

• ?PcRe@Oakleaf/Oakbridge-FX, Reedbush-U/H, Oakforest-PACS

NOT^;9G6'VaIHb`eO7:EB6!VaIHb`eO7;8FC<5• HJU]SK(• ZeQXbMeO4Lbe\MeO@• %(• ZeQXbMeOfhd

f1d l3dg

lid gfFX10: 24YeW496YeW,
RB-U: 16YeW, RB-H: 2YeW4OFP: 16YeW, 64Ye
Wg"0,@ 0;34#'• Lbe\MeO• 'VaIHb`eOkfhd lid gk'fFX10:1,440YeW4
RB-U: 128YeW4RB-H: 32YeW4OFP: 2048YeWg• !VaIHb`eOkfhd l2*jd g4!f/*1&A'= )g• OeZeMc[_eR(1$-# @#;.f2g•
@MeO=D?4+>( @;.

2018/2/14

KNL

20   2018/2/14

KNL

 

21



• "?OUGRQW&T XD)YD89)T / Y >5C2;*

T

T / 
• -:@*• '@):0C.<,.@)?

WETHSIQX:

01• ETHSIQ)>

(43/$,:0=+%?

• #?6A?FVNPKM?
• #7/B&• LVJ !&?   

 

22

  • 9>28@0/• MPI BMessage Passing
InterfaceC• <A.!4A1,'

8@0/$/?35(•:;=,#"+ %
•&+8@0/):;=*•&+/?35#* (:;=,#"+• 9>2/?35• OpenMP• <A.!,

9>28@0/$9>2/?35)

→6-7=35MPI/OpenMP    

23

 

   •MIMD)0!.*1!,'/1. SPMD3Single Program Multiple Data4• 5)0!.+)0%&"

• MPI3(2#-154,'/

2. Master / Worker3Master / Slave4• 5)0%$3Master4

3Worker4)0%$3 42018/2/14

!KNL 

24 
• 
• 
•   $ ! 
• #"

SIMD
 



æ 1 2 3 ö æ 9 8 7 ö æç 1* 9 + 2 * 6 + 3 * 3 1* 8 + 2 * 5 + 3 * 2 1* 7 + 2 * 4 + 3 *1 ö÷ç÷ ç÷ " ç 4 * 9 + 5 * 6 + 6 * 3 4 * 8 + 5 * 5 + 6 * 2 4 * 7 + 5 * 4 + 6 *1÷
ç 4 5 6 ÷ ç 6 5 4 ÷ ç 7 * 9 + 8 * 6 + 9 * 3 7 * 8 + 8 * 5 + 9 * 2 7 * 7 + 8 * 4 + 9 *1÷
øç7 8 9÷ ç 3 2 1÷ è
è
ø è
ø

CPU æ 1

2 3ö
ç÷CPU ç 4 5 6 ÷
CPU ç 7 8 9 ÷èøCPU

æ9 8 7öæ 1* 9 + 2 * 6 + 3 * 3 1* 8 + 2 * 5 + 3 * 2 1* 7 + 2 * 4 + 3 *1 öç÷ " çç 4 * 9 + 5 * 6 + 6 * 3 4 * 8 + 5 * 5 + 6 * 2 4 * 7 + 5 * 4 + 6 *1÷÷
ç 6 5 4÷
ç 7 * 9 + 8 * 6 + 9 * 3 7 * 8 + 8 * 5 + 9 * 2 7 * 7 + 8 * 4 + 9 *1÷
èø
ç3 2 1÷èø !   

2018/2/14

25

 KNL 
• 0/-
• 0/-8.329* ("!(• 170$ 8@9%#(')#
• 0/- $?,57*(• :?*(• ;?*(• <? *(• =?6*) &• >?,574+*)(



:
;
<

=

>



2018/2/14




KNL

26

2018/2/14

27

KNL 

•  S P = TS / TP (0 £ S p )
• P
• TS
P3  $PP/3 TP
• P'. S P = P30)$1NidealO • P'. S P > P30)$=MEKD9L=GMC9@H• 1

4$256$AM?9;>=( +8.$:I@<JF@B( ,7*0257#• 
• P E P = S P / P ´ 100 (0 £ E p ) [%]• " •  3!
• Saturation, %+-7&2018/2/14

KNL

28
• "%G K =5F)

6@-8(
• 3@=1(0<1F G α =5F)
A@D-?>F)S P = K /(Ka / P + K (1 - a ))= 1 /(a / P + (1 - a )) = 1 /(a (1 / P - 1) + 1)

• !@/E(7=.&@@MPJLIG9;

CSP→∞T( A('* 1/(1-α) <+F)SHNKQO@ T• @90%0

<17=4;C(&@@MPJLIG:/9;C( 1 / (1 - 0.9) = 10  ?4/>E>,R→'G#5F7B?A(4<C
G2F
G5F3=0=;C$ <+F2018/2/14

29

!KNL 

   "88.8%

 

9/3=3



9/2=4.5 ≠ 6 

2018/2/14

KNL

30
>VDUFR4518,)1)0…

KNL3 3C>3)&Oakforest-PACS
3 3"LZJ=9 <-0&  +*= 7*[• .33# () http://www.cc.u-tokyo.ac.jp/support/kosyu/• >VDUFR= 6/)• 10/31,11/1 OpenMP/OpenACC2:;PVHC>YSK?C>OWBT

QXB$
• .3
• 11/7, 8'%!1M@NUIJOWBTQXB(• 3/1, 2
MPI • 
MPI • 
GPUATEG#()  


2018/2/14

KNL

Oakforest-PACS

31

2018/2/14

32

32

KNL

Oakforest-PACS*(JCAHPC: Joint Center for
Advanced High Performance Computing)• !HPC
• 


:?;A
• " )# :?;A
• :?;A/2.!35143%&•  7>?<93(-')#6$@+ 0

:?;A

• 2016121,
• 8,208 Intel Xeon/Phi (KNL)• =A8'25PFLOPS• TOP 500

7B 1CDHPCG 5BC B20176C

2

2018/2/14

KNL

Oakforest-PACS (1 / 2)
•  &2$• 1&2$ 686
3TFLOPS8,208&2$7 25
PFLOPS• +,.3MCDRAM36
16GB45DDR436
96GB44• &2$
• (/'!-1'1$
Fat-Tree%"#02•  
*.2-1 6 -)
• Intel Omni-Path Architecture33

33

2018/2/14

!KNL

34

Oakforest-PACS 2 / 2• -"#1I/O• -"#1&'*.:
Lustre 26PB
• -"#1$/)&0&'*.
4DDN IME56
1TB/sec!
 ,
1PB
• 2,)%+3(
2

• 
• Green 5006
• Linpack6 2.72 MW• 4,986 MFLOPS/W4OFP5•830 MFLOPS/W45

 120 
 

34

2018/2/14

KNL

35

Oakforest-PACS 
'2 



25 PFLOPS

%2$

8,208

Product%2$

 PRIMERGY CX600 M1 (2U) +CX1640 M1 x 8node)0!"

Intel® Xeon Phi™ 7250
3 2$: Knights Landing468 1.4 GHz+ &1$ 16 GB, MCDRAM,  490 GB/sec, &1$ 96 GB, DDR4-2400, '2 115.2.
GB/sec Product

.1
#*0

Intel® Omni-Path Architecture
100 Gbps(/&!-1&1$Fat-tree2018/2/14

KNL

36

Oakforest-PACS 
 Type%
Product

Lustre File System
26.2 PB
DataDirect Networks SFA14KE

& 500 GB/sec
Burst Buffer, Infinite Memory Engine  Type(by DDN)
%!
"

940 TB (NVMe SSD, $)Product
DataDirect Networks IME14K
& 1,560 GB/sec#

4.2MW'(102

2018/2/14

37

KNL

Oakforest-PACS 
• OS: Red Hat Enterprise Linux (dJFeUgS)+

CentOS /A= McKernel (!UgS+
• McKernel: AICS8(<^TgLE

)

2OS

• Linux;>$'+`gMZdJb];.B):5• [ORLeW_gQ;?%4CB,• LeVFbhGCC, Intel Compiler, XcalableMP• XcalableMP: AICS9 8 (<ZdJb\eJ #• C@Fortran8"&4C7LgS;D .B398+
<*-EZcKgNaeD;(6B390
81B,• bFYbcfEZcKgNaeh IgZePgOPXRGH
E• ppOpen-HPC, OpenFOAM, ABINIT-MP, PHASE system,

FrontFlow/blue，LAPACK, ScaLAPACK, PETSc, METIS, SuperLU etc.

2018/2/14

KNL

38
• TOP 500CLinpack, HPLD
• & 8?9BCDE"% CFLOPSD• !+ F&;<>1265• "
• HPCG• & 8?9BCDE"% CFLOPSD• ' ). .0/C7@*(D• &;<>1265•
1:>3B4=A,#(• ;<>1265E$
• Green 500• HPLCTOP500D -FLOPS/W

48th TOP500 List (November, 2016)
Site

1

Computer/Year Vendor

National Supercomputing
Center in Wuxi, China

Sunway TaihuLight , Sunway MPP,
Sunway SW26010 260C 1.45GHz,
2016 NRCPC

Tianhe-2, Intel Xeon E5-2692, TH
Express-2, Xeon Phi, 2013 NUDT
Titan
Cray XK7/NVIDIA K20x, 2012 Cray
Sequoia
BlueGene/Q, 2011 IBM
Cori, Cray XC40, Intel Xeon Phi 7250
68C 1.4GHz, Cray Aries, 2016 Cray
Oakforest-PACS, PRIMERGY CX600
Joint Center for Advanced
M1, Intel Xeon Phi Processor 7250 68C
6 High Performance
1.4GHz, Intel Omni-Path,
Computing, Japan
2016 Fujitsu
K computer, SPARC64 VIIIfx , 2011
7 RIKEN AICS, Japan
Fujitsu
Piz Daint
Swiss Natl. Supercomputer
8
Cray XC30/NVIDIA P100, 2013 Cray
Center, Switzerland
Argonne National Laboratory, Mira
9
BlueGene/Q, 2012 IBM
USA
Trinity, Cray XC40, Xeon E5-2698v3
10 DOE/NNSA/LANL/SNL, USA
16C 2.3GHz, 2016 Cray
National Supercomputing
Center in Tianjin, China
Oak Ridge National
3
Laboratory, USA
Lawrence Livermore National
4
Laboratory, USA
DOE/SC/LBNL/NERSC
5
USA
2

2018/2/14

Cores

Rmax
Rpeak
Power
(TFLOPS) (TFLOPS) (kW)10,649,600

93,015(= 93.0 PF)

125,436

15,371

3,120,000

33,863(= 33.9 PF)

54,902

17,808560,640

17,59027,1138,209

1,572,864

17,173

20,133

7,890

632,400

14,01527,881

3,939

557,056

13,55524,9142,719

705,024

10,51011,28012,660206,720

9,779

15,9881,312

786,432

8,587

10,066

3,945

301,056

8,101

11,0794,233

Rmax: Performance of KNL
Linpack (TFLOPS)
Rpeak: Peak Performance (TFLOPS), Power: kW

http://www.top500.org/
3949th TOP500 List (June, 2017)
Site

Computer/Year Vendor

National Supercomputing
Center in Wuxi, China

Sunway TaihuLight , Sunway MPP,
Sunway SW26010 260C 1.45GHz,
2016 NRCPCNational Supercomputing
Center in Tianjin, China
Swiss Natl. Supercomputer
3
Center, Switzerland

Tianhe-2, Intel Xeon E5-2692, TH
Express-2, Xeon Phi, 2013 NUDT
Piz Daint
Cray XC30/NVIDIA P100, 2013 Cray

1
2

Titan
Cray XK7/NVIDIA K20x, 2012 Cray
Sequoia
BlueGene/Q, 2011 IBM
Cori, Cray XC40, Intel Xeon Phi 7250
68C 1.4GHz, Cray Aries, 2016 Cray
Oakforest-PACS, PRIMERGY CX600
Joint Center for Advanced
M1, Intel Xeon Phi Processor 7250 68C
7 High Performance
1.4GHz, Intel Omni-Path,
Computing, Japan
2016 Fujitsu
K computer, SPARC64 VIIIfx , 2011
8 RIKEN AICS, Japan
FujitsuArgonne National Laboratory, Mira
9
BlueGene/Q, 2012 IBM
USA
Oak Ridge National
Laboratory, USA
Lawrence Livermore National
5
Laboratory, USA
DOE/SC/LBNL/NERSC
6USA
410 DOE/NNSA/LANL/SNL, USA

Trinity, Cray XC40, Xeon E5-2698v3
16C 2.3GHz, 2016 Cray

Cores

10,649,600

93,015(= 93.0 PF)

125,436

15,371

3,120,000

33,863(= 33.9 PF)

54,902

17,808361,760

19,59033,8632,272

560,640

17,59027,1138,209

1,572,864

17,173

20,133

7,890

632,400

14,01527,881

3,939

557,056

13,55524,9142,719

705,024

10,51011,28012,660786,432

8,587

10,066

3,945

301,056

8,101

11,0794,233

Rmax: Performance of Linpack (TFLOPS)
Rpeak: Peak Performance(TFLOPS), Power: kW
KNL
2018/2/14

Rmax
Rpeak
Power
(TFLOPS) (TFLOPS) (kW)http://www.top500.org/

4050th TOP500 List (Nov., 2017)
Site

Computer/Year Vendor

National Supercomputing
Center in Wuxi, China

Sunway TaihuLight , Sunway MPP,
Sunway SW26010 260C 1.45GHz,
2016 NRCPCNational Supercomputing
Center in Tianjin, China
Swiss Natl. Supercomputer
3
Center, Switzerland

Tianhe-2, Intel Xeon E5-2692, TH
Express-2, Xeon Phi, 2013 NUDT
Piz Daint
Cray XC30/NVIDIA P100, 2013 Cray

4 JAMSTEC

Gyoukou
ZettaScaler-2.2, PEZY-SC2

Oak Ridge National
Laboratory, USA
Lawrence Livermore National
6
Laboratory, USA

Titan
Cray XK7/NVIDIA K20x, 2012 Cray
Sequoia
BlueGene/Q, 2011 IBM

7 DOE/NNSA/LANL/SNL, USA

Trinity, Cray XC40, Intel Xeon Phi
7250 68C 1.4GHz, 2017 Cray

1
2

5

8

DOE/SC/LBNL/NERSC
USA

Joint Center for Advanced
9 High Performance
Computing, Japan
10 RIKEN AICS, Japan

Cori, Cray XC40, Intel Xeon Phi 7250
68C 1.4GHz, Cray Aries, 2016 Cray
Oakforest-PACS, PRIMERGY CX600
M1, Intel Xeon Phi Processor 7250 68C
1.4GHz, Intel Omni-Path,2016 Fujitsu
K computer, SPARC64 VIIIfx , 2011
FujitsuCores

10,649,600

93,015(= 93.0 PF)

125,436

15,371

3,120,000

33,863(= 33.9 PF)

54,902

17,808361,760

19,59033,863

2,272

19,860,000

19,135.8

28,192

1,350

560,640

17,59027,1138,209

1,572,864

17,173

20,133

7,890

979,968

14,13743,902

3,844

632,400

14,01527,881

3,939

557,056

13,55524,9142,719

705,024

10,51011,28012,660

Rmax: Performance of Linpack (TFLOPS)
Rpeak: Peak Performance(TFLOPS), Power: kW
KNL
2018/2/14

Rmax
Rpeak
Power
(TFLOPS) (TFLOPS) (kW)http://www.top500.org/

41

HPCG Ranking (SC16, November, 2016)
Site
1 RIKEN AICS, Japan
2

NSCC / Guangzhou,
China

3 JCAHPC, Japan

Computer

TOP500Rank

HPCG(Pflop/s)

HPCG/
HPL (%)

705,024

10.51070.6027

5.73

Tianhe-2

3,120,000

33.86320.58001.71

557,056

13.555

6

0.3855

2.84

10,649,600

93.0151

0.3712.399

632,400

13.83250.3554

2.57

1,572,864

17.17340.3304

1.92

Titan

560,640

17.5903

0.3223

1.83

Trinity

301,0568.101

100.1826

2.25

Pleiades: SGI ICE X

243,0085.952

130.1752

2.94

Mira: IBM BlueGene/Q,

786,4328.587

90.16701.94

Oakforest-PACS

6 DOE/NNSA/LLNL, USA Sequoia

http://www.hpcg-benchmark.org/
2018/2/14

HPL Rmax(Pflop/s)

K computer

National
4 Supercomputing Center Sunway TaihuLightin Wuxi, China
DOE/SC/LBNL/NERSC
5
Cori
USA
DOE/SC/
7 Oak Ridge National
Laboratory, USA
DOE/NNSA/
8 LANL/SNL,USA
NASA / Mountain View,
9
USA
DOE/SC/
10 Argonne National
Laboratory, USA

Cores

KNL
42

HPCG Ranking (June, 2017)
Computer
1 K computer
2 Tianhe-2 (MilkyWay-2)
3 Sunway TaihuLight
4 Piz Daint

HPL Rmax TOP500(Pflop/s)
Rank
705,024
10.5108
3,120,000
33.863
2
10,649,600
93.015
1
Cores

HPCG
Peak
(Pflop/s)
0.60275.3%
0.58011.1%
0.48080.4%

361,760

19.5903

0.47671.9%

557,056
632,400
1,572,864
560,640

13.55513.832
17.17317.59076
5
40.38550.35540.33040.3223

1.5%
1.3%
1.6%
1.2%

9 Trinity

301,0568.101

100.1826

1.6%

10 Pleiades – NASA/SGI

243,0085.952

150.1752

2.5%

5
6
7
8

Oakforest-PACS
Cori
Sequoia
Titan

http://www.hpcg-benchmark.org/
2018/2/14

KNL43

HPCG Ranking (Nov, 2017)
Computer
1 K computer
2 Tianhe-2 (MilkyWay-2)
3 Trinity
4 Piz Daint

HPL Rmax TOP500(Pflop/s)
Rank
705,024
10.51010
3,120,000
33.863
2
979,072
14.137
7
Cores

HPCG
Peak
(Pflop/s)
0.60275.3%
0.5801
1.1%
0.546
1.8%

361,760

19.590

3

0.486

1.9%

5 Sunway TaihuLight

10,649,600

93.01510.4808

0.4%

6
7
8
9

557,056
632,400
1,572,864
560,640

13.55513.832
17.17317.590986
50.38550.35540.33040.3223

1.5%
1.3%
1.6%
1.2%

136,080

8.125

13

0.189

1.6%

Oakforest-PACS
Cori
Sequoia
Titan

10 TSUBAME3.0

http://www.hpcg-benchmark.org/
2018/2/14

KNL
44

2018/2/14

45

KNLGreen 500 Ranking (SC16, November, 2016)
Site

ComputerCPU

NVIDIA DGX-1, Xeon E5-2698v4 20C
2.2GHz, Infiniband EDR, NVIDIA Tesla
P100
Swiss National
Cray XC50, Xeon E5-2690v3 12C
2 Supercomputing Piz Daint
2.6GHz, Aries interconnect , NVIDIA
Centre (CSCS)
Tesla P100
3 RIKEN ACCS
ShoubuZettaScaler-1.6 etc.
National SC
SunwaySunway MPP, Sunway SW26010
4
Center in Wuxi TaihuLight 260C 1.45GHz, Sunway
SFB/TR55 at
PRIMERGY CX1640 M1, Intel Xeon
5 Fujitsu Tech.
QPACE3Phi 7210 64C 1.3GHz, Intel OmniSolutions GmbHPath
PRIMERGY CX1640 M1, Intel Xeon
Oakforest6 JCAHPCPhi 7250 68C 1.4GHz, Intel OmniPACSPath
DOE/SC/Argonne
Cray XC40, Intel Xeon Phi 7230 64C
7
Theta
National Lab.
1.3GHz, Aries interconnect
Cray CS-Storm, Intel Xeon E5-2680v2
Stanford Research
8
XStream
10C 2.8GHz, Infiniband FDR, Nvidia
Computing Center
K80
ACCMS, Kyoto
Cray XC40, Intel Xeon Phi 7250 68C
9Camphor 2
University
1.4GHz, Aries interconnect
Jefferson Natl.
KOI Cluster, Intel Xeon Phi 7230
10
SciPhi XVI
Accel. Facility
64C 1.3GHz, Intel Omni-Path
NVIDIA
1
Corporation

http://www.top500.org/

DGX
SATURNV

HPL
TOP500Rmax
Rank(Pflop/s)

Power
(MW)

GFLOPS/W

3.307

28

0.350

9.462

9.779

8

1.312

7.454

1.001

116

0.150

6.674

93.01

1

15.37

6.051

0.447

375

0.077

5.806

1.355

6

2.719

4.986

5.096

18

1.087

4.688

0.781

162

0.190

4.112

3.057

33

0.748

4.087

0.426

397

0.111

3.837

2018/2/14

KNL46

Green 500 Ranking (Nov., 2017)
Site

ComputerCPU

HPL Rmax TOP500(Pflop/s)
Rank

Power
(kW)

GFLOPS/W

1

RIKEN, Japan

Shoubusystem B

ZettaScaler-2.2 HPC system, Xeon D1571, PEZY-SC2 , ExaScalar

842.0

259

50

17.0092

KEK, Japan

Suiren2

ZettaScaler-2.2 HPC system, Xeon D1571, PEZY-SC2 , ExaScalar

788.2

307

47

16.7593

PEZY, Japan

SakuraZettaScaler-2.2 HPC system, Xeon E52618Lv3, PEZY-SC2 , ExaScalar

824.7

276

50

16.657

4

NVIDIA, USA

DGX Saturn
V Volta

Xeon E5-2698v4, NVIDIA Tesla V100 ,Nvidia1,070.0

149

97

15.113

5

JAMSTEC,Japan

Gyoukou

ZettaScaler-2.2 HPC system, Xeon D1571, PEZY-SC2 , ExaScalar

19,135.8

41,350

14.173

6

Tokyo Tech.

TSUBAME3.0

SGI ICE XA, IP139-SXM2, Xeon E52680v4, NVIDIA Tesla P100 SXM2, HPE

8,125.013792

13.704

7

AIST, JapanAIST AI
Cloud

NEC 4U-8GPU Server, Xeon E5-2630Lv4,
NVIDIA Tesla P100 SXM2 , NEC

961.0

148

76

12.681

8

CAIP, RIKEN,JAPANRAIDEN GPU
subsystem -NVIDIA DGX-1, Xeon E5-2698v4, NVIDIA
Tesla P100 , Fujitsu

635.1

305

60

10.6039

Univ.
Cambridge, UK

Wilkes-2 -Dell C4130, Xeon E5-2650v4, NVIDIA
Tesla P100 , Dell

1,193.0100

114

10.428

10

Swiss Natl. SC.
Center (CSCS)

Piz Daint

Cray XC50, Xeon E5-2690v3, NVIDIA
Tesla P100 , Cray Inc.

19,590.0

32,272

10.39811

ITC, U.Tokyo,
Japan

Reedbush-L

SGI Rackable C1102-GP8, Xeon E52695v4, NVIDIA Tesla P100 SXM2 , HPE

805.6

291

79

10.167

http://www.top500.org/

2018/2/14

KNL47

IO 500 Ranking (Nov., 2017)
Site

ComputerFile system

Client nodes IO500 Score

BW(GiB/s)MD
kIOP/s)

1

JCAHPC,Japan

OakforestPACS

DDN IME204

101.48471.2521.85

2

KAUST, Saudi

Shaheen2

Cray
DataWarp

300

70.90

151.5333.17

3KAUST, Saudi

Shaheen2

Lustre

1000

41.00

54.17

31.03

4

JSC, Germany

JURON

BeeGFS

8

35.77

14.24

89.83

5

DKRZ,
Germany

Mistral

Lustre

100

32.15

22.77

45.39

6

IBM, USA

SonasadSpectrum
Scale

10

21.63

4.57

102.38

7

Fraunhofer,
Germany

Seislab

BeeGFS

24

18.75

5.13

68.58

8

PNNL, USA

EMSL
CascadeLustre

126

11.17

4.88

25.57

9

SNL, USA

SerranoSpectrum
Scale

16

4.25

0.65

27.98

http://www.io500.org/

2018/2/14

48

KNL• 20173"%UOT-lL)$A4FZm• =5?+T6\/lfkc]_hj KTLQGm• 8,208ekdl25PFm\R6O@/OJ[YHRLNIJn
!S=5?+\# R*/M[

C/• .;S/^ka• HPCI• ?+S20%\DJCAHPCEPLNn'/:

• JHPCNlB<(0 /13,m• & ?+S5%2o'13nB13XWl&S
Vm• 9l>7n>8m• <(HPCbgij`oekd#n124A2/"48

2018/2/14

KNL

49

 
• OFP@'7H</GXeon Phi 7250ASWL$I#0G,• MJ[ 68MJ
• MJ:FAAVX-512 UQNO[ 2• AVX-512UQNO:FA Y! Z[ 8•  (Fused Multiply Add: FMA) : 2 @
• LVNL : 1.40 GHz• RWP:FSWL$[

68 * 2 * 8 * 2 * 1.40 = 3046.4 GFLOPS• 828+AVX-512UQNOB B1.40GHz=B8?/
YDF/ Z
• SWL@(/$3EHGB9ACAYOFP=A Z• DGEMM (! A%Z: 2200 GFLOPS (SWL[72%)• HPL: 2000 GFLOPS (SWL[ 66%)
• Top500@15GOFPA*BSWLA54.4%• -SWL$.A "I848<146>3)&XX• CPUTWKW@D;< "AC?G2018/2/14

50

KNLIntel Xeon Phi (KNL: Knights Landing)l® Xeon Phi™ Processor

rgeted for HPC and Supercomputingthat is binary
ard OS.

• Atom (Silvermont)

ctor performance+ AVX512
TILE x2
2 VPU

ative memory
apacity

Knights Landing Overview• 2  1 

notice.
o change without further notification.

MCDRAMEDC

MCDRAM

EDC

3

X4
DMI

PCIe
Gen 3

D
M
I

MCDRAMEDC

MCDRAM

EDC

EDC

EDC

misc

2 VPU

1MB
L2

Core

Chip: 36 Tiles interconnected by 2D MeshTile: 2 Cores + 2 VPU/core + 1 MB L2• 64, 68

32, 34 •Memory:
MCDRAM: High BW
MCDRAM:
16 GB on-package;
KNL✝ コア 16GB+
DDR4: 6 channels @ 2400 up to 384GB
4-way ハイパースレッディング(最大 4 × 72 = 288 論理プロセ
DDR4
IO:
36 lanes PCIe Gen3. 4 lanes of DMI for chipset
DDR
436 Tilesconnected by
2D MeshInterconnect

DDR MC

CHA

3

Tile

D
D
R
4
C
H
A
N
N
E
L
S

2 x16
1 x4

Core
 4 
(HyperThreading)• 

KNL Card(PCIe-Card)

DDR MC

EDC

C
H
A
N
N
E
L
S

EDC

L2 キャッシュはタイル上の 2 つのコアで共有される

Node: 1-Socket only
タイル コアアウト
アウト
コア Omni-Path
Fabric:on-package(not shown)オブオブ
命令キャッシュ命令キャッシュ

オーダー

オーダー

デコード
+ リタイヤデコード+ リタイヤVector Peak Perf: 3+TF DP and 6+TF SP Flops
1MiB
ベクトル
Scalar
Perf: ~3xover Knights
Corner ベクトル
ベクトル
ベクトル
L2
ALU
ALU
ALU Triad (GB/s):データ
ALU
Streams
MCDRAM:
400+;
DDR: 90+
レガシー
レガシー
キャッシュ

MCDRAM

4

MCDRAM

Omni-path not shown

Package

MCDRAMMCDRAM

Source Intel: All products, computer systems, dates and figures specified are preliminary based on current expectations,
HW + SWbased on current
L1 without
データキャッシュ
32KiBexpectations
L1 データキャッシュ
are subject 32KiB
to change
notice. KNL data are preliminary
and are subject to change
プリフェッチwithout notice. 1Binary Compatible with Intel Xeon processors
using Haswell Instruction Set (except TSX). 2Bandwidth
numbers are based on STREAM-like memory access pattern when MCDRAM used as flat memory. Results have been
HW + SWestimated based on internal Intel analysis
and are provided for informational
purposes only. Any difference in system
メッシュへ
hardware or software design or configuration may affect actual プリフェッチ
performance.

HotChips 27 KNL
, Colfax (colfaxresearch.com/knl-webinar )KNL コアの機能✝

© Colfax I2018/2/14

KNL

51

KNL
• ABHBK?N3"• Flat:MCDRAM,DDR4 '()4?I<• Cache: MCDRAM0DDR4ABH/6C>:D,(*• Hybrid• 7G<=HJ8BK?: 5"• (All-to-all: 4?I< '.… ! )• Quadrant, Hemisphere:  +4?I< '4(1)02).
LEK9&20%-$M
• SNC-4, SNC-2: NUMA?A5J'.4 (or 2).

BK?/ .0' => +0# BK?(Flat, Cache)/;F@
6DK3

2018/2/14

 KNL

52
• Cache• MCDRAML3

• Flat• MCDRAM
MCDRAM
16 GB

MCDRAM
16 GB

DDR4
96 GB

DDR4
96 GB






2018/2/14

KNL

53

Flat NUMA• $ numactl -H

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6… 271
node 0 size: 98164 MB
node 0 free: 92027 MB
node 1 cpus: 
node 1 size: 16384 MB
node 1 free: 15832 MB
node distances:
node 0 1
0: 10 31
1: 31 10

Node 0
96 GB
CPU

DDR4

Node 116 GB
MCDRAM2018/2/14

54

 KNL  "
ClusterOFP!Quadrant
Mode: Sub-NUMA
C• Quadrant: .31()&#!3801459$• !4
MCDRAM
*60-7;/+<%• SNC-4: 4OPIO
MCDRAMOPIO
MCDRAM

PCIe

EDC

EDC

IIO

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

iMC

Tile

Tile

OPIO
MCDRAM

EDC

EDC

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

3
1

4

2
iMC

DDR

NUMA2
4':$• 4.,01$OPIO
MCDRAM





DDR
Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

Tile

EDC

EDC

Misc

EDC

EDC

OPIO
MCDRAMOPIO
MCDRAMOPIO
MCDRAMOPIO
MCDRAM

HotChips27
1) L2 miss, 2) Directory access, 3) Memory
access, 4) DataKNL
return
16

2018/2/14

55

KNL

Stream Triad
DDR4• DDR4: micprun -D -k stream• MCDRAM: micprun -k stream• unset KMP_AFFINITY• export KMP_HW_SUBSET=66c@2,1t 0  

Cache

Flat

346.5 GB/sec
84.8
GB/sec

600
Cache

500

 (GB/sec)

Flat MCDRAM
400

Flat DDR4

300
200
100
0
1

11

21

31



41

51

MCDRAM

61

494.8
GB/sec2018/2/14

KNL

56

FlatMCDRAM(1)(1) numactl.36
• MCDRAM0NUMA=B<1,'*&7*!6• $ numactl –H

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 … () … 268 269 270 271
node 0 size: 98164 MB
node 0 free: 85707 MB
node 1 cpus:
node 1 size: 16384 MB
node 1 free: 15496 MB
node distances:• ;B<$"8%)! .0
node01  /#-!%0+$-!
0: 10 31
1: 31 10

• >A:@?.8• $ numactl --membind 1 ./a.out(MCDRAM 16GB.5$4-!,9@BC• $ numactl --preferred 1 ./a.out(MCDRAM. 15 4-!DDR42(6C• $ numactl --interleave 1,0 ./a.out(MCDRAM,DDR48.(6C2018/2/14

KNL

57

FlatMCDRAM(2)(2)  (,.3Intel MPI/• I_MPI_HBW_POLICY • export I_MPI_HBW_POLICY=hbw_bind• export I_MPI_HBW_POLICY=hbw_preferred• export I_MPI_HBW_POLICY=hbw_interleave• numactl/+)&• 021!/"$(* ) '"*%!' Intel MPI)&!#  ('-'2018/2/14

KNL

58

FlatMCDRAM(3)(3) memkind:17:;+-/
• C• #include <hbwmalloc.h> 0• ,. 89;, +”hbw_”0(&/• malloc() => hbw_malloc()• posix_memalign() => hbw_posix_memalign()• free() => hbw_free()• ;=3+ -lmemkind 0

• Fortran• ,#*!&$ • Fastmem,0 (allocatable)• !dir$ attributes fastmem :: array+'#)%*!
• ;=3+ -lmemkind 0
• hbw_set_policy())."• HBW_POLICY_BIND• HBW_POLICY_PREFERRED >462<5?
• HBW_POLICY_INTERLEAVE

2018/2/14

59

&KNL

  • AVX-512

%512;687@5/(",(*
7@5<=> (#$. &• 512;68 = 64:18))09?3(0243!.-(%*

•)0243('.

• A(double)/8(=512;68)+!

 
 !#"

$ %!#"=
01

2

364

4

5

6

7

0

1

2

3

4

5

6

7

0

1

2

3

4

2018/2/14

KNL    • C• __attribute__((aligned(64))) "(64 byte = 512 bit)• , double A[1024] __attribute__((aligned(64)))•  posix_memalign() "
• 

,
#include <stdlib.h>double *A;
posix_memalign(&A, 64, 8192);• Fortran• %+'#*  -align array64byte $(&)+"• attribute  !()real*8, dimension(:), allocatable :: A
!dir$ attributes align:64 :: A

60

2018/2/14

KNL

OpenMP

61

2018/2/14

62

+KNL 

OpenMP
• OpenMP





PE

PE

PE

PE

OpenMP 

OpenMP 

OpenMP 

OpenMP
 






A[ ]

$ %PE$'(*)⇒!$ & #"%" #2018/2/14

KNL

63

OpenMP• OpenMP (OpenMP Application Program Interface)16$

JKN
95T

"4HOAMI>,=.


2. M@GMN
3. >!+.:50,%
1.

•

•

LQC)$HOAMI5 *-=.95>(=:50,%BPF@M4;=06&<8-?%
JKN RMPI32S47/$EQD
55#)'$) 0,%

2018/2/14

KNL

OpenMP

64 • :M=?3&BN7JDO7G>L• CL<84$F@P84.!
•/:M=? ,#%3)2.0$
BN7JDO7/'
1.
2.

F5OFGK-6H=9I"/>P; '.1%OpenMP,3,(-%BN7JE.-*+%2QR• AP?"/0OpenMP,0,(-%• AP?"/0MPI3%2

2018/2/14

65

KNL

OpenMP • )7,/
• 16)7,/=0:/• T2K$:39)1'9;AMD Quad Core Opteron(Barcelona) 4*&,.<FX10):1'925:+()-4;Sparc64 IXfx<• 32>128)7,/=0:/• HITACHI SR16000 (IBM Power7)• 32'#64>128'#;SMT<• Reedbush (Intel Xeon E5-2695 v4, Broadwell-EP)• 36'#• 60>272)7,/=0:/• Intel Xeon Phi (Intel MIC(Many Integrated Core) Knights Conner)• 60'#120>240'#;HT<• Oakforest-PACS (Intel MIC, Knights Landing)• 68'#272'#• 100)7,/"OpenMP•  38%64 

!



2018/2/14

KNL

OpenMP • C
• #pragma omp







• Fortran
• !$omp







66

2018/2/14

KNL

67

OpenMP
• )4.'1)4.'2(/*04OpenMP
(/*04&$• 6Intel Fotran90)4.'1

ifort -O3 -qopenmp foo.f• 6Intel C)4.'1
icc -O3 -qopenmp foo.c• • OpenMP25/ • )4.'1"#"$+3,-
$ $ !$• OpenMP $ OpenMP"$+3,- % )4.'1"$
• 6Intel Fortran90)4.'1

ifort -O3 -qparallel -qopenmp foo.f2018/2/14

KNL

68

OpenMP • OpenMP7IPAML=BQG?N.2.1#!

H>?N7#8)07H>?N=/<-43#+• CODE=) OMP_NUM_THREADS3
• TOpenMP6;<#!H>?N,a.out7$ export OMP_NUM_THREADS=16$ ./a.out
918$ env OMP_NUM_THREADS=16 ./a.out• • %BQG?N7IPAML4)OpenMP6;<IPAML7#&,)OMP_NUM_THREADS=16.2:) 5<-4,*<S$T• -7 8)OpenMP6;<7S@RFRJDET• (CODE#3)-7@RFRJDE6;<&,'"
• IPAMKQA73 !

2018/2/14

KNL

OpenMP

69

2018/2/14

70

'KNL

OpenMP C
OpenMP!(#pragma omp parallel
,
!)
!*
 +
OMP_NUM_THREADS!(  %#" $

 &

!)

!)

 

!*

 +-1…

!)2018/2/14

71

'KNL

OpenMP Fortran
OpenMP
!(!$omp parallel!)!$omp end parallel!* +
OMP_NUM_THREADS!(  %#" $

 &

!)

!)

 

!*

 +-1…

!)2018/2/14

KNL

72

Work sharing
• parallel27#12<@=>-*8


1$",OpenMP-:*8B?A=;BC
2: (parallel region). 4•  : ),<@=>-*8:
*8OpenMP2:Work sharing. 4• Work sharing3 &'(,2D%!81.



-*862• forBdoC• sections
• single (master)0/2.

parallel.59+862• parallel for  (parallel do)• parallel sections0/2018/2/14

KNL

73

2018/2/14

74

-KNL Fortran

Fordo#pragma omp parallel forfor (i=0; i<100; i++){a[i] = a[i] * b[i];}!$omp parallel do!$omp end parallel do

 
%)&'



%)&'+

%)&',

%)&'2

%)&'3

for (i=0; i<25; i++){a[i] = a[i] * b[i];}for (i=25; i<50; i++){a[i] = a[i] * b[i];}for (i=50; i<75; i++){a[i] = a[i] * b[i];}for (i=75; i<100; i++){a[i] = a[i] * b[i];}%)&' #!
 "#(*$"

 



2018/2/14

8KNL 

75For   
for (i=0; i<100; i++) { •453 &1)
)#+1
a[i] = a[i] +1;
6a[i-1]#%2'+
b[i] = a[i-1]+a[i+1];
#17}
for (i=0; i<100; i++) {a[i] = a[ ind[i] ];}•ind[i]- ,/0453($1"*!".1•a[ind[i]]#(%21 #+,453($12018/2/14

Fortran

Sections#pragma omp sections{#pragma omp section
sub1();#pragma omp section
sub2();#pragma omp section
sub3();#pragma omp section
sub4();}!$omp sections
!$omp end sections
l 






sub1();



 





sub1();2sub2();

sub3();

sub4();l 

76

KNL







 


sub2();2
sub3();

3
sub4();

2018/2/14

77

 KNLFortran

Critical

!$omp critical!$omp end critical#pragma omp critical{s = s + x;
}

•



1 

 

 



 2


 3

s= s+xs= s+xs= s+xs= s+x2018/2/14

78

(KNL 

Private
#pragma omp parallel for private(c)for (i=0; i<100; i++){a[i] = a[i] + c * b[i];}) "%#$! →  


"%#$&
for (i=0; i<25; i++){a[i] = a[i] + c0*b[i];}"%#$'for (i=25; i<50; i++){a[i] = a[i] + c1*b[i];}"%#$

"%#$2

"%#$3

for (i=50; i<75; i++){a[i] = a[i] + c2*b[i];}for (i=75; i<100; i++){a[i] = a[i] + c3* b[i];}"%#$

2018/2/14

79

>KNL 

Private #pragma omp parallel for private( j )for (i=0; i<100; i++) {for (j=0; j<100; j++) {a[ i ] = a[ i ] + amat[ i ][ j ]* b[ j ];}•:=8? 4;57%)0!$  /.
•private( j ) ' 4;57% j )31<60 (#$!*"+100):=8 (','
→&'-29=&'.

2018/2/14

>KNL 

80

Private  Fortran
!$omp parallel do private( j )do i=1, 100
do j=1, 100a( i ) = a( i ) + amat( i , j ) * b( j )
enddo
enddo
!$omp end parallel do•:=8? 4;57%)0!$  /.
•private( j ) ' 4;57% j )31<60 (#$!*"+100):=8 (','
→&'-29=&'.

2018/2/14

KNL


C

81



• 76(DHEF9A#/-<(K39

A2)

80>

• "9#/-<:DHEF8'

87.?>• reduction!*)5(ddot: 7>87>2=(
 4%95 @7,7,7>#pragma omp parallel for reduction(+:ddot )for (i=1; i<=100; i++) {ddot += a[ i ] * b[ i ]}ddot9

:DCG9<"$ I&:"$4+;1BJ

2018/2/14

KNL

82



Fortran• 65'CGDE8@".,;'J28

@1(

7/=

• !8".,;9CGDE7&

76->=• reduction )(4'ddot9 76=1<'
3$84 ?6+6+6=!$omp parallel do reduction(+:ddot )do i=1, 100
ddot = ddot + a(i) * b(i)enddo!$omp end parallel do
ddot8

9CBF 8;!# H%9!#3*:0AI

2018/2/14

KNL

83
• reduction/*!H7(F+<.NMLGD7

-<9• =GW`Z\O:7-<&?9• GK:F7ddot'G4 O)?C2%D +@L"<6
3EJ8LdAB?75VQX7]c\RSPe!$omp parallel do private ( i )W`Z\ G_c^Of#gW`Z\ 'do j=0, p-1
W`Z\DPUYW@LQb[ZUW,O F0
do i=istart( j ), iend( j )ddot_t( j ) = ddot_t( j ) + a(i) * b(i)W`Z\D'9L7acT_Eddot'G
enddo
4 ddot_t()O)?70F $?C;=
enddo!$omp end parallel doddot = 0.0d0
do j=0, p-1
2%D1?>Iddot = ddot + ddot_t( j )enddo2018/2/14

84

!KNL

FortranSingle!

!$omp single/!$omp end single• Single   %+-)'&
%. (,)*# $•  (,)*# "%$! • nowait  &%# $
#pragma omp parallel for$A#pragma omp single{ B }
…
}











A

A
"



#
…

A

2018/2/14

85

KNL

Master• .single)"• %&!master( !%@/*--;>96BA.
#<78?7=9:,3'4• - 2+
• $-%0,13

54

2018/2/14

86

KNL

Flush
• &'^_`LO4W

U• Flush#KDVJ;UOS9GOK4W
U:GVO !OP9^_`OLO4P%;:
b$-."PaYZ[NDVUHA:^_`N2-."W ?6X
K;M;c• IRT9flush1 W =M;L9Za\]7KN5EBX
H.">90CLN*MU:• barrier1 9critical1 O 9parallel#O 9for9sections9single#O KP98+NflushDVJ;U:• FlushW<L/P@MU:K?UHA(;M;:#pragma omp flush (3LMUOQ),)FUL9JO>32018/2/14

KNL

87

Threadprivate
• -6/1"$24+370

 ) (• -6/1"$#()'$(-6/1!*,.-!(% $• "&-6/1"$#(572%

"%

#include <omp.h>…
int myid, nthreds, istart, iend;void main() {#pragma omp threadprivate(istart, iend)…
…
#pragma omp parallel private (myid,
void kernel() {nthreds, istart, iend) {int i;
nthreds = omp_num_threds();for (i=istart; i<iend; i++) {myid = omp_get_thread_num();for (j=0; j<n; j++) {istart = myid * (n/nthreads);a[ i ] = a[ i ] + amat[ i ][ j ] * b[ j ];iend = (myid+1)*(n/nthreads);}
if (myid == (nthreads-1)) {}nend = n;-6/1$#()}} )parallel…
kernel();! (}2018/2/14

KNL

88

OpenMP2018/2/14

 KNL

89  
•  • integer (Fortranl Fortran90)

omp_get_num_threads()int (C)l C use omp_lib
Integer nthreads#include <omp.h>
int nthreads;

nthreads = omp_get_num_threads()

nthreads = omp_get_num_threads();

2018/2/14

 KNL

90

 omp_get_thread_num() • integer (Fortran) int (C)
l C l Fortran90 •

use omp_lib
Integer myid#include <omp.h>
int myid;

myid = omp_get_thread_num()

myid = omp_get_thread_num();2018/2/14

91

KNL

 omp_get_wtime()  • double precision (Fortran ) double (C•l Fortran90



l C

)



use omp_lib
double precision dts, dte#include <omp.h>
double dts, dte;

dts = omp_get_wtime()
dte = omp_get_wtime()
print *, “Elapse time [sec.] =”,dte-dts

dts = omp_get_wtime();dte = omp_get_wtime();printf(“Elapse time [sec.] = %lf ¥n”,
dte-dts);

2018/2/14

KNL

OpenMP


922018/2/14

 KNL93 

• Parallel for/do9>+&HJG=

K/?MO
N=)4LC+ !<DIEF<K("6A@.<L58+C6A,
HJG=B
1

 0

 1

 2

 3

 4

n K *L

} 3=:1+

DIEF957HJG<6A%
'#09;-:+DIEF$= 
1  0
 1
 2
 3
 4 n
02;A

%'#

2018/2/14



94

KNL } :,813#7'-#
1

,86&17.!":)+#.946%$
n



} 

0#7 .!"HBEF?@<A/65I
3#DGC=>;/# /081
,8$} 27 .:&( *9.%

2018/2/14

 KNL




95



• schedule (static, n)• ;@8)19>-.*0/<250&!A/<25C/<25D???%!:+>5=6> $B (%!
'(n!19>-.*0) (• Schedule  ) "37,;4#
static19>-.*0#;@8//<25 
1
 0

 1

 2

 3

2018/2/14

 KNL




96



• schedule(dynamic, n)• (+&#'* "  !)$%   n#'* "

1
 0

 1

 2

 3

2018/2/14




 KNL

97

• schedule(guided, n)• HKF9@GJ<=:?.(0@GJ<=:?9'&(/$5$(*>IAD#
5!+.96-7n0@GJ<=:?9 .%7
• @GJ<=:?1

$11619>IAD.
,*""4)1$@GJ<1=:?0/7
• @GJ<=:?2 1 0 #,-0'&/7
• @GJ<=:?0 1 46 %! k 9 (*@GJ< =:
?20 k 3.'&/7$1@GJ<2 k 46'
&/7
$ 7
1
• @GJ<=:?$
'8-!/!
BE;HC2 1 0/7
 0
 1
 2
 3

2018/2/14

 KNL






l Fortran90

98





!$omp parallel do private( j, k ) schedule(dynamic,10)
j-461$ 
#') %*$! do i=1, n
do j=indj(i), indj (i+1)-1
i-461$
y( i ) = amat( j ) * x( indx( j ) )!* enddo
 #$enddo +(" &
dynamic0./2635-, !$omp end parallel do


l C



#pragma omp parallel for private( j, k )schedule(dynamic,10)for (i=0; i<n; i++) {for ( j=indj(i); j<indj (i+1); j++) {y[ i ] = amat[ j ] * x[ indx[ j ]];}}2018/2/14

KNL

99   • dynamic2guidedG[gnRVPZH$F9;/• [gnRVPZ8<>:MC,&ainYH%;EM8 .2
!AGQoad\_89;EM3• 2[gnRVPZ89>:C,&ainY8;EM .2
!AGQoad\_8<;EM3• *G#G^lo_Qb84M3• 'G[gnRVPZG[ho`nS80B2[ho`nS
UY^87M3• staticGKB1-)8B9Mp84Mq• dynamicEDG'YTXhojnSH2WY]fGQoao
d\_8M82staticHQoaod\_HpJCODq53•  F,&8(CEMkoc"N+I?B2
staticYTXhojnSN6C2L
8%5 $84M3• ?@=2cmSienSGUY^H>M2018/2/14

100

KNL

Static

• $-SMPTE-8;W(!D+X
NUOQCTVR
WNUOQ7@CTVR!$omp parallel do private(S,J_PTR,I)
DO K=1,NUM_SMP
DO I=KBORDER(K-1)+1,KBORDER(K)LI;GB&X
S=0.0D0
B)F>'8>21;/
DO J_PTR=IRP(I),IRP(I+1)-1*#4%@AI
S=S+VAL(J_PTR)*X(ICOL(J_PTR))NUOQ7@CTVR
END DO
Y(I)=SW NUOQD/,"8>1
I4/
END DO
$B/ NUOQ49ITVR
B=1>/
%ATVRL'X
END DO,"9IH>?/3=/:J?*#4%9I!$omp end parallel do.B-?5I0
$B*#4 BK<>16
A1

D-?5

2018/2/14

KNL

OpenMP 



101

2018/2/14

KNL

OpenMP

102

 

• OpenMP)

parallel'&+.# &for7:4% • &OpenMP)48/659/012+($OpenMP(48/659/(• parallel-,+'*+)private(!.!&%3/"+;2018/2/14

103

KNL

Private • OpenMP-2#./5private

1DFA 2#-)/%"4#,  0/5$• <@8D=1l 

2#:E;>!- 0)*-/%

!$omp parallel do /)0AC7BF= .),(6512#'1i-DFA 13
do i=1, 100'1j-DFA 2#private/)-2  0/5
do j=1, 100←:E;>!-% +- ← 0?9
tmp = b(i) + c(i)a( i ) = a( i ) + tmp
enddo'1 tmp2#private/)-2  0/5←:E;>!-% +-& ← 0?9
enddo!$omp end parallel do2018/2/14

KNLPrivate 



104



• Private%"B(*:KMI:<F/)+M- 9/3=/;C-CMH:@/- E9,49/PYQS C M
!:K7@40Kl   !$omp parallel dodo i=1, 100
call foo(i,arg1,arg2,arg3,
arg4,arg5, ….., arg100)
enddo!$omp end parallel do-D#BUXNWZRB
AK<F/private%"B(*:KM?5K←939/- E9COZTVQ
S4:K←PYQS$B21>G/- E
9COZTZVQS4&?5A6
AJ/  4.8LK
'[?59>M2018/2/14

KNL

105

Private  • OpenMP15%

-,3 +:5%+70
Hshared variableI32:• C "4%Fortran90 "4common%
module5%.4881532:• BD=CG@3*/&

5%Threadprivate'• parallel1#6

FG>E3

**0&: %.4#1*0&:9%32:• .48815%

1 *2&• );<$(35%4?GA4'• !4FG>E<3*/#6*<:• !4FG>E<3*0%Threadprivate +:2018/2/14

KNL

Parallel    

106



• Parallel"do  $• /,.*!$do  !
 ,.* fork$'.(% $'-)&+
#$$!$omp parallel!$omp do private(j,tmp)
do i=1, 100
do j=1, 100
tmp = b(  ) + c(  )a( i ) = a( i ) + tmp
enddo
enddo
!$omp end do!$omp end parallel

Parallel parallel do!$omp parallel do private(j,tmp)
do i=1, 100
do j=1, 100
tmp = b(  ) + c(  )a( i ) = a( i ) + tmp
enddo
enddo
!$omp end parallel do2018/2/14

107

 KNL

Parallel    • Parallel,do(#'(".
• 342+/#%,#%

!*).
• %&#•

342/(".,$+!
342*140 !-(")

do i=1, n!$omp parallel dodo j=1, n
<
>
enddo
!$omp end parallel do
enddo

!

!$omp parallel
do i=1, n!$omp dodo j=1, n
<
>
enddo
!$omp end do
enddo!$omp end parallel2018/2/14

108

KNL   • , 1.@ALGFBD;26!3@• , <IEML-0?>-DKFH$<EAJLC 7*8"1#2- 48 +/3@
1.@
• (;=,+56/@l• OpenMP<=-GME)<'=2:/• GME)<';=-critical%:9<1&



  

!$omp parallel do private( j )do i=1, n
j = indx( i )a( j ) = a( j ) + 1
enddo!$omp end parallel do!$omp parallel do private( j )do i=1, n
j = indx( i )!$omp criticala( j ) = a( j ) + 1!$omp end criticalenddo!$omp end parallel do2018/2/14

KNL109

OpenMP    • OpenMPR)Otvn\FYCMPA• #^nscvfquP=BY.8OtvnR9GQSSLROpenMP P<K<O<CM?;Y:
1. private- &P>ZYQ?+POY• tvn>XOpenMP
FY96LJK<YQ?<CM?;Y• privatesgkP\@ZKU9dum_rPWY`rvRO<:w
Q2RpveP;YHTx• ,FYM9h_oubPE0(* ?3!M$OY:
NC?74JK<Y>[>XO<QL9jlia?POY:• /"'ydum_rPWJKR95 \ FYCM
?L@Y:GQ>X9IV]Mprivate DZK<Y>%
1FY:

2018/2/14

KNL

110

OpenMP    :\n`a-I,@H= J_jqbpZ@9
1. 5JdqaVWUKhilUY^\J,@('I=
2. mqeCJNJICSLG @H=rmqe7@)=s• 1#BRIK;Um[l]gJ;.J;@0
IHQ;OpenMPJ %F<RHeoZkfpZTH>
3. /8H\n`aeoZkfpZIK?H=•+H2*JXqcmmqeT;parallel for!F34
BR6F"@PSE=R• /8H&K;task;MDKPthreadHGJnativeH\n`aAPI;MDKTBB, CilkHGJJ$F AL>@
OQOB=
2.

2018/2/14

KNL

111

OpenMPSIMD2018/2/14

KNL

112

OpenMP+SIMD• OpenMP 4.0'/4;0• .:2,73-09&!)(>





•

#!.:2,7

• .;1$

**$" *#(&<%%=(• 568$+7,:5:0<=2018/2/14

KNLOMP ForDo+SIMD• C

#pragma omp parallel for simdfor (i=0; i < num; i++) {sum = sum + a[i];}• Fortran:!$omp parallel do simd
do i=1, num
sum = sum + a(i)!$omp end parallel113

2018/2/14

KNL114

OMP SIMD• C
#pragma omp parallel forfor (i=0; i < num; i++) {#pragma omp simdfor (j=0; j < num; j++) {sum[i] = sum[i] + a[i][j];}}• Fortran:!$omp parallel dodo i=1, num!$omp simd
do j=1, numsum(i) = sum(i) + a(j,i)
enddo!$omp end simd
enddo!$omp end parallel2018/2/14

KNL

115

Declare SIMD
•  SIMD•OMP SIMDSIMD • #pragma omp declare simd notinbranchfloat min (float a, float b){ return a < b ? a : b; }void minner (float *a, float *b, float *c) {#pragma omp parallel for simdfor (i=0; i<N; i++)c[i] = min(a[i], b[i]);}2018/2/14

KNLKNL OpenMP 

116

2018/2/14

117

 KNLKNL(OFP)HyperThreading1/: 681/• HyperThreading(HT) : 11/ + 4 HT=>  2722<46
•• AVX-512:745')$* 1/ +2
• 2"&AVX-512:745(VPU). -( 1 HT$'• AVX-512:7451#%-( 2 HT!

• 89;/032., -(3~4 HT*
2 VPU
Core

2 VPU
1MB
L2

1 

=2

Core

2018/2/14

118

 KNL

OFP  • ZX^SP => OS

L/G(CBDJMUWVE2O

:I<5• 15SP85MA82O:M• 'DVQY

L/GF 1ms$DI>?9M• OFP@F3SP0EGVQY

L/GO: MJ6D.

_NM,H4L`
• SP0A3VQ]@L2R[WT\O+#;?5MSP1H2O

:M• HTSPH&2O:M• SP) 0,1, 68,69, 136,137, 204, 205

F-7K;="855(85)• !%,8•<M*C"%F0

6
0
8

1
3
6

6
1
9

2
0
4

2 VPU
Core0

1
3
7

2 VPU
1MB
L2

1 

Core1

=2

2
0
5

2018/2/14



119

KNL

(1)Xeon Phi,Xeon)( +EI<73CPU*-&• 73+6K=6"
• Reedbush+Broadwell (2.1GHz)*-& 1.6

• FGH36:9J4>M8/• <=CA=?LN6"
ÞOpenMP+* %2#25
PARALLEL• O+5NB

D=@ (EPCC syncbench)• 873!11673' #)"20* 2
• 273,;4I• 4~6473,73+.
• 128~25673,HT

Overhead (usec)

5NBD=@" #$)2

20

FOR
PARALLEL FOR

15

BARRIERSINGLE

10

REDUCTION

5
0
2

4

8

16

32

# Core

64

128

256

2018/2/14


•

120

KNLZ

(2)(Atomic)6DYORKM (EPCC syncbench)• Xeon Broadwell 32GB17 Atomic: 0.1us

Critical, Lock/Unlock: 0.4us=> STVBEJIWCLXH6
$• Parallel for/do43%&45:5OVB7

9@0(?

• #)*Parallel regionA?
•  Parallel for/do.?6174*%
Parallel 61for/do/+A>".
• MPI+OpenMPNCPVKM 5<>%IWKMA,?
• BQVFYHUX5.?
• '21!-*CRITICAL

2.5

LOCK/UNLOCK
Overhead (usec)• 4?8*Atomic;OVBA=.

3

2

ATOMIC

1.5
1
0.5
0
2

4

8

16
# Core

32

64

128

256

2018/2/14

KNL(1KNL)

121

2018/2/14

KNL

122


Intel Compiler 2017 update 4? (OFP9JNCVK)• CFXMAT: icc• FortranFXMAT: ifort
'DOGSX• -xMIC-AVX512• KNL/FYL

• -O3• $ WPV• -qopenmp• OpenMP• -align array64byte• (Fortran9;[%?64byte8@TAXZ"[14-**• -qopt-streaming-stores=
always / never / auto
• always: EQIGR8!27*

?+)EQIGR9&-* 8
• never:39#• JNCVK:auto• VYO9 %058<(:always)• #pragma vector nontemporal (inC/C++)• !DIR$ vector nontemporal (in F)• -qopt-prefetch={0~5}• OUNBIH?69.=*(>,2018/2/14

KNL

123
• 68"64"
• $&% " 0,1, 68,69, 136,137, 204, 205
1. 1% )1"32"• !'&#(  
2. 1"1HT64"
3. 1"2HT128"
4. 1"3HT192"
5. 1"4HT256"2018/2/14

KNL

124

32
• #*1!+$%• ($ )&-% 
•',"! ./export OMP_NUM_THREADS=32
export KMP_AFFINITY=proclist=[2,4,6,8,10,12,14,16,18,20,
22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,
56,58,60,62,64,66],explicit,verbose
2 VPU

0

Core

2 VPU

2

Core

1MB
L2

1MB
L2

2 VPU

2 VPU

Core

Core

2 VPU

2 VPU

Core

Core

1

3

1MB
L2

1MB
L2

2 VPU
Core

2 VPU
Core





2018/2/14

KNL

12564• 

1
• KMP_HW_SUBSET • c@,
• 64c@2,1t: 64t2 1

export OMP_NUM_THREADS=64
export KMP_HW_SUBSET=64c@2,1t
export KMP_AFFINITY=scatter,verbose
2 VPU

0

Core

2 VPU

4

Core

1MB
L2

1MB
L2

2 VPU

2 VPU

Core

1

Core

2 VPU

2 VPU

5

Core

Core

2

6

1MB
L2

1MB
L2

2 VPU

3

Core

2 VPU

7

Core





2018/2/14

126

KNL

128• 

2#
• KMP_HW_SUBSET $64c@2,2t• KMP_AFFINITYcompact
• scatter#!" 

064

#

export OMP_NUM_THREADS=128
export KMP_HW_SUBSET=64c@2,2t
export KMP_AFFINITY=compact,verbose
2 VPU

01

Core

2 VPU
Core

89

1MB
L2

1MB
L2

2 VPU

2 VPU

Core

23

4Core5

2 VPU

2 VPU

10Core11

12 13
Core

1MB
L2

1MB
L2

2 VPU

67

Core

2 VPU

14 15
Core





165…

2018/2/14

KNL

127

192, 256
192
• 3



256
• 4 

export OMP_NUM_THREADS=192
export KMP_HW_SUBSET=64c@2,3t
export KMP_AFFINITY=compact,
verboseexport OMP_NUM_THREADS=256
export KMP_HW_SUBSET=64c@2,
4t
export KMP_AFFINITY=compact,
verbose2018/2/14

KNL

128
.'&2$*
C 1,+()+#)%3&2D
• KMP_AFFINITY ,verbose3• 7?BCe+ -=46@D,A>B;
• 2/".-0 +C-D• 3285-: 9A:<0!852, 9A:<1!854, 9A:<2!856…,1 )OMP: Info #242: KMP_AFFINITY: pid X tid XX thread 0 bound to OS proc set {2}OMP: Info #242: KMP_AFFINITY: pid X tid XY thread 1 bound to OS proc set {4}OMP: Info #242: KMP_AFFINITY: pid X tid XZ thread 2 bound to OS proc set {6}…

2018/2/14



KNL

129

   • ) & '$*
• -qopt-report=5• !&.optrpt • *& +• Intel Advisor XE,• #&'

• Intel VTune XE• "(!%)

2018/2/14

KNL

130

2018/2/14

KNLKNL/OFPMPI+OpenMP

131



2018/2/14

KNLMPI+OpenMP • MPIDF4'132

&BCE,3G?A:<,


•

•*

('-%+0$.">G<HOpenMP>G<HMPI
Reedbush'-&285:;#NUMA1 "

.)85:;: OpenMP
85:;+>G<: MPI• @D:;MPI,!-/%
B=G624D79#(#,%2018/2/14

KNL

133

MPI
• i^\qYb^XpR$ElOelm-Eu@• i^\qYb^XpREj`nA7J• TpbOlE-6"E]daPQMHlOelmK?GEAFC

8r• ijm•.( A,D ;-.(:*• ufo\^VD9<JijmVO[HdLOnVO[E)K&*E 8XZ_hsMassively Parallel Processing(MPP)XZ_htK$8J,D ;• fo\^V• ufo\^V(A+C,3E.(K6%3A#*• ':

• APIsApplication Programming InterfacetE!

• ZSqlcm_N6*:58• 1#KkqW:/0?J=BDIJMnUm[hE2:*• foRlgpR:4>8s:58t

2018/2/14

KNL

134

MPI
• MPI0,6327http://www.mpi-forum.org/8• 1994 5 1.07MPI-18• 1995 6 1.1• 1997 7 1.2 (' 2.07MPI-28• 2008 5 1.3 2008 6 2.1
• 2009 9 2.2•  http://www.pccluster.org/ja/mpi.html• MPI-2 !&)9• I/O• C++Fortran 90*5/60+6• 14.- /• $#"%2018/2/14

KNL

135

MPI MPI-3.1• MPI-3.0 2012 9• MPI-3.1 2015 6• *0$/'",+.&!
• http://mpi-forum.org/docs/docs.html• http://meetings.mpi-forum.org• http://meetings.mpi-forum.org/mpi31-impl-status-Nov15.pdf•  • (./)-%".#

1MPI_IALLREDUCE2•   1RMARemote Memory
Access)• Fortran2008  2018/2/14

KNL

136

MPI MPI-4.0• -KV<*UC7NMTB1
• http://meetings.mpi-

forum.org/MPI_4.0_main_page.php• %0)"/
• D4HQ@CIS8PLT8.• MPI3IQ9V;OT-  WFault Tolerance,
FTX• "$(#-34A31
• Active Messages (M@>V<-ISB:R)••••+-6VEP@I-1"' 
"6VEVJ@C!F4IP4T
E@G2QT8,&*!4T?PIBDTCP*$• Stream Messaging• ISG24RU4T?VG5V=

2018/2/14

137

KNLMPI
• MPICHO:HMEBAP• 8J=KC'• MVAPICH (:HL7EBA)• ;D9;

 . !MPICH6GN>• InfiniBand(35+ • OpenMPI• ;NFK?N>• GK@MPI• !305&'GN>21,-%4
: "#!FX103MPI: Open-MPIGN>Intel MPI: MPICH!MVAPICHGN>• :IN<

'1*5-%4)/'$4

2018/2/14

KNLMPI• /.

(A• . -
1. /"4/
2. .)*$52/0,'.#5&
3. .)*$52//!
4. .)*$52//
5. ?6 .5 /@?:8@• MPI+0A
1. /ID"%31"4/ID
2. ;>:/7<=93. ;>:
4. ;>:
5. :8

138

2018/2/14

KNL

MPI
• +,.1 • !%5 "#!5 "$(5 !(5

• 44  
• 02-)3*• !5'5

• /302-)3*• $!5 #'5• 4 • $%
•









• &5• • % #&5###5139

2018/2/14

KNL

140
• MPI_COMM_WORLD2#>JMG=PC/63:9

; *9 • >JMG=PC2#;%1HOBE?;59• .2#QSnumprocs –R4.1HOBE?
&#R,1>JMG=PC08-7:9• (1&#!MPI_COMM_WORLD"• HOBE?;)+$#MPI_Comm_split;• KEBP@;#1HOBE?0*9/'0• “IND<LAF”.

2018/2/14

KNL

141

MPI• MPI;'MQKJ(#:!C*<2&MQKJ;S!

;T'MQKLI(S=1.;%HET97 ?6>
A<2&
• %'MPIMQKJ(8.:;"*:7%//7;PE
SProcesser Elements:T8-<2&• 451816'PE(;;)<?BA6*<3D&• ORFSRankT•'MPIMQKJ(:'
 (:/8&• !MPI7;%MPI_Comm_rank$70A@SI

RMPMQGON7;myidT9%VX PEUW :,@• :: MPIMQKJC@9;%MPI_Comm_size$
C+&SIRMPMQGON7;%numprocs 9%/:,@T2018/2/14

KNLMPI



142

2018/2/14

C
FortranKNL








 ierr ierr = MPI_Xxxx(….);• Fortran   ierrcall MPI_XXXX(…., ierr)• 
•

• 

MPI_Status istatus;• Fortraninteger istatus(MPI_STATUS_SIZE)

143

2018/2/14

C
Fortran• MPI!$

KNL




144






• C
MPI_CHAR ( )  MPI_INT ()
MPI_FLOAT ( )MPI_DOUBLE( )• Fortran
MPI_CHARACTER ( ) MPI_INTEGER()MPI_REAL ( )
MPI_DOUBLE_PRECISION(  ) 
MPI_COMPLEX( )• %# "$ 2018/2/14

145

KNLMPI―MPI_Recv 1/2
• ierr = MPI_Recv(recvbuf, icount, idatatype, isource,

itag, icomm, istatus);• recvbuf :"& %
• icount : ",0*& %
• idatatype : ",0*"& %
• MPI_CHAR (MPI_FLOAT () MPI_INT ()
) MPI_DOUBLE()• isource :-+)0(&%PE"

• 

!#MPI_ANY_SOURCE & %


./'& %"PE$

2018/2/14

KNL

146

MPI―MPI_Recv 2/2
• itag : * 25,PLJTI:,6,AKF;C*• ;KF;PLJTIC

25,8/<)MPI_ANY_TAG C*• icomm : *PE(

C!#3A 7+AHOQNGTKC*• %7<MPI_COMM_WORLD C3B=>,*• istatus : MPI_StatusU ;& V* :'3A
.A*-9?4;  C25& C3A08*• .MPI_STATUS_SIZE;&. 1BA*•
25PLJTI;$;RSE. istatus[MPI_SOURCE])KF. istatus[MPI_TAG] :1BA*• C "W MPI_Status istatus;• Fortran "W integer istatus(MPI_STATUS_SIZE)• ierr(@) : 

*DRTHTM.A*2018/2/14

KNL

147

MPI―MPI_Send• ierr = MPI_Send(sendbuf, icount, idatatype, idest,

itag, icomm);• sendbuf : % *• icount :   %5?3*
• idatatype :   %5?3% *• idest :   "PE%icomm#%<>,*• itag :   ":42?1$&)"3-%*• icomm : 8=240? *!(#(/9;7.?3*
• ierr (') :  +<?/?6(2018/2/14

KNL

148

Send Recv  
PE0

PE

PE

PE

MPI_Send  

MPI_Recv

2018/2/14

KNLMPI―MPI_Bcast• ierr = MPI_Bcast(sendbuf, icount, idatatype,iroot, icomm);• sendbuf :(' & + #*
• icount :   &3:1+ #*
• idatatype :   &3:1& + #*
• iroot :  !$720:/ *PE&+ #*PE%"+ #* *
• icomm :  PE +#*%*.685-:1+ #*
• ierr ()) :  ,9:.:4 *

1492018/2/14

 KNL

150

MPI_Bcast
PE0

PE

MPI_Bcast()

MPI_Bcast()

iroot

PE

PE

MPI_Bcast()

MPI_Bcast()

PE







!!

2018/2/14

KNL

151
• IJ+/'(IJ2

D@:67>BE#%0
• H


=6;ADKE
E• @:67>B-

0→85?DF

*2• Dcollective communication

operationE*.10• ,&,!)
3B9<4C8"
$0

G,

*$

2018/2/14

152

KNL
•PE• MPI_Reduce• 

PE

PE0
PE0
PE1




PE2

PE0

• MPI_Allreduce• 


PE0

PE1

PE
PE0


PE2

PE1

PE2

2018/2/14

KNL

153

MPI―MPI_Reduce• ierr = MPI_Reduce(sendbuf, recvbuf, icount,

idatatype, iop, iroot, icomm);• sendbuf : %':(A2>*• recvbuf :':( A2>*iroot 713PE:<7.#<-90?>*%'8 ';) 7+56;9=9,*29@4)9>&A19/6;9=9,*• icount : *%':CDB" A2>*• idatatype : *%':CDB:A2>*• EFortranFHG8!IA$2A2> ;)MPI_2INTEGER())MPI_2REAL( ))MPI_2DOUBLE_PRECISION() )A2>*

2018/2/14

KNL

154

MPI―MPI_Reduce• iop :
 $0"2)1$• MPI_SUM ( )# MPI_PROD ()# MPI_MAX ()#MPI_MIN ()# MPI_MAXLOC ( -*0)# MPI_MINLOC (-*0) /.$• iroot : $2'1PE0icomm ,0<=42)1$+0icomm 0PE
, (2)1&%1$
• icomm : $PE! 2 )1,%16:;95>72)1$• ierr :  $ 3<>6>8&1$

2018/2/14

155

 KNLMPI_Reduce
PE0

PE

MPI_Reduce()

MPI_Reduce()

PE

PE

MPI_Reduce()

MPI_Reduce()

iroot




iop





2018/2/14

156

KNLMPI_Reduce 

MPI_2DOUBLE_PRECISION  MPI_MAXLOC

PE0

PE

MPI_Reduce()

MPI_Reduce()

PE

PE

MPI_Reduce()

MPI_Reduce()

iroot
3.1

4.1

5.9

2.6

2.0

5.0

9.0

13.0

MPI_MAXLOC
5.9
9.0

LU





2018/2/14

157

KNLMPI―MPI_Allreduce• ierr = MPI_Allreduce(sendbuf, recvbuf, icount,

idatatype, iop, icomm);• sendbuf : %':(B2?*• recvbuf :':( B2?*iroot 713PE:<7.#<-90@?*%'8 ';) 7+56;9>9,*29A4)9?&B19/6;9>9,*• icount : *%':DEC" B2?*• idatatype : *%':DEC:B2?*• =8!B$2B2?))MPI_2FLOAT ( ))MPI_2DOUBLE() B2?*;)MPI_2INT(2018/2/14

KNL

158

MPI―MPI_Allreduce• iop :

"$ #• MPI_SUM () MPI_PROD ()MPI_MAX ( )MPI_MIN ()
MPI_MAXLOC ( ) MPI_MINLOC() ! • icomm :PE$##'+,*&.($ #• ierr : %-.'.)#2018/2/14

159

 KNLMPI_AllreducePE0
MPI_Allreduce()




PE

PE

PE

MPI_Allreduce()

MPI_Allreduce()

MPI_Allreduce()iop















2018/2/14

KNL

160
• 
• ,'$&*.

00

!


• (-%+) !/• MPI_Allreduce
MPI_Reduce  !
• MPI_Allreduce   "• "!MPI_Reduce #

2018/2/14

161

KNLMPI―MPI_Gather• ierr = MPI_Gather (sendbuf, isendcount, isendtype,recvbuf, irecvcount, irecvtype, iroot, icomm);• sendbuf : 6! >.;#• isendcount:# 6@A?>.;#• isendtype :# 6@A?6>.;#• recvbuf : 6! >.;#iroot 3,/PE
683(8'5+<;#• 5&4,2"4  7" 3$127595%#.5=0"5;>,5)27595%#• irecvcount:

#



6@A?>.;#• *67"BPE/:6@A?>.;*4#• MPI_Gather 37

3(5%63"PE35;6@A?>->.;*4#

.;*47

2018/2/14

KNL

162

MPI―MPI_Gather• irecvtype :





!)0( #"
• iroot : )0(#"PE!
icomm  !./%#"• !icomm !PE#"
"
• icomm : PE #""',-+&0(#"• ierr : $.0'0*"2018/2/14

 KNL

163

MPI_Gather
PE0

PE

PE

PE

MPI_Gather()

MPI_Gather()

MPI_Gather()

MPI_Gather()

C

D

iroot
A

B


A
B
C
D

2018/2/14

KNL

164

MPI―MPI_Scatter• ierr = MPI_Scatter ( sendbuf, isendcount, isendtype,recvbuf, irecvcount, irecvtype, iroot, icomm);• sendbuf : 5!=,:#• isendcount:# 5?@>=,:#• )56"APE-948;:?@>=,:)2#• MPI_Scatter 163%51"

PE13:5?@>=,:)261'+=,:)2 #• isendtype :# 5?@>5=,:#iroot 1*-PE57 23:#• recvbuf : 5!=,:#• 3&

2*0" 2  6" 1$/06383%#,3<."3:=*3(06383%#• irecvcount:

#



5?@>=,:#2018/2/14

KNL

165

MPI―MPI_Scatter• irecvtype :





!)0( #"
• iroot : )0(#"PE!
icomm  !./%#"• !icomm !PE#""• icomm : PE #""',-+&0(#"• ierr : $.0'0*"2018/2/14

 KNL

166

MPI_Scatter
PE0

PE

PE

PE

MPI_Scatter()

MPI_Scatter()

MPI_Scatter()

MPI_Scatter()

iroot
A
B
C
D


A

B

C

D

2018/2/14

 KNL

167

MPI_Scatter
PE0

PE

PE

PE

MPI_Scatter()

MPI_Scatter()

MPI_Scatter()

MPI_Scatter()

iroot
A
B
C
D


A

B

C

D

2018/2/14

KNL

168
1.

=@80A2•

D&;8<. %>86C4,D&;8<. %/165B "+)" ( *$

••;8<. &9C7& -
MPI_Send, MPI_Bcast$#2.

:A=@80A2• D&;8<.

%

(

• ;8<.

• +&9C7-!

&9C7& -!
&'?C3&2018/2/14

169

&KNL  • 0  
send   

"#! 0



"#! 1



"#! $



"#! %



…

send



send

recv



send


recv

…




recv













 

…

 
 

2018/2/14

KNL

170

 

• ierr = MPI_Isend(sendbuf, icount, datatype,

idest, itag, icomm, irequest);• sendbuf :  • icount : '+%

• datatype : '+%• idest :PEicomm )*!
• itag :(&$+#
%"



2018/2/14

KNL

171

 

• icomm : PE*(!(-463,80*
(•  !$MPI_COMM_WORLD *)%&
• irequest : MPI_Request9 #:*51/8." ') (• ierr :  +78-82(2018/2/14

KNL

172
• ierr = MPI_Wait(irequest, istatus);• irequest : MPI_RequestDE#7(+?=;C:/,%46+#• istatus :MPI_Status DE#/!)5 $5#• $MPI_STATUS_SIZE0

7(-)5#•
(+?=;C:00@A8$
istatus[MPI_SOURCE] "<9$istatus[MPI_TAG] /'65#• >C<71*)5B >C<73)/2&.

2018/2/14

 

• &'#"0%($!

send $()%'#( 

&'"!0



send send

&'"!1



recv

&'"!*



&'"!+



…

173

,KNL

send

…


recv





recv

…

!MPI_Wait 


 









 
 

2018/2/14

KNL

174•   %• MPI_Send • MPI_Wait 

&• MPI_Isend • MPI_Wait

&
• 
&

 !$#" 2018/2/14

KNL

175


MPI-3$1.4$!P.+)#%  8   2. -3$1.4$" MPI!
 !
 '4(
6 http://accc.riken.jp/HPC/training/text.html 7
3. Message Passing Interface Forum
6 http://www.mpi-forum.org/ 7
4. MPI-J/524$2&*
6 http://phase.hpcc.jp/phase/mpi-j/ml/ 71.

5.

%4,05(:7! !69;;

2018/2/14

9KNL 

176 C
double t0, t1, t2, t_w;..
ierr = MPI_Barrier(MPI_COMM_WORLD);
t1 = MPI_Wtime();15,+":!!& "#26-43+;
t2 = MPI_Wtime();t0 = t2 - t1;
ierr = MPI_Reduce(&t0, &t_w, 1,
MPI_DOUBLE,MPI_MAX, 0,
MPI_COMM_WORLD);

26/0.7$<0'(
%*
!' ())''
+26/0.8 *

2018/2/14

177

9KNL  Fortrandouble precision t0, t1, t2, t_w
double precision MPI_WTIME..
call MPI_BARRIER(MPI_COMM_WORLD, ierr)
t1 = MPI_WTIME(ierr)
:!!&

15,
+""#26-43+;

t2 = MPI_WTIME(ierr)
t0 = t2 - t1
call MPI_REDUCE(t0, t_w, 1,&
MPI_DOUBLE_PRECISION,&MPI_MAX, 0, MPI_COMM_WORLD, ierr)26/0.7$<0'(%*
!' ())''+26/0.8 *2018/2/14

178

KNL

Omni Path
CH?9G:<=8• I_MPI_FABRICS=AI@
:AI@• shm: DEFJAI@13K• tmi: Omni Path1%))4• ofi:
1
• >B7G?,2 shm:tmi+'" 2
tmi:tmi 0(*'!

(*'$$• HFI_NO_CPUAFFINITY
=1
•

'6
'#6L

-/$

• AI@CH;:

/.2
&5




(*'$$

2018/2/14

MPI+OpenMP • MPI
• 3

179

%KNL 

-4810923 !+%

$5;3: Master only, Funneled, Multiple• Multiple!%%#%!"'*0.;8"• Master only!%Parallel7;/6:)",&"

•#"$!,&Funneled!$(Master only

Funneled

Multiple#pragma omp parallel{....
}
MPI_Send( ... );#pragma omp parallel
{
....
}

#pragma omp parallel{....
#pragma omp master
MPI_Send( ... );....
}#pragma omp parallel{....
MPI_Send( ... );....
}Master only
parallel 

"#$ !
11 2018/2/14

180

KNL MPI• MPI_Init()

MPI_Init_thread()• C:

int provided;
MPI_Init_thread(&argc,&argv,MPI_THREAD_FUNNELED,&provided);• Fortran

integer provided, required
required=MPI_THREAD_FUNNELED
call MPI_Init_thread(required, provided, ierr)2018/2/14

KNL

MPI• 1
• HT OpenMP  • (KMP_AFFINITY, KMP_HW_SUBSET)

export OMP_NUM_THREADS=64 # 1HT
export I_MPI_DEBUG=5 #  
export I_MPI_PIN_DOMAIN=256
export I_MPI_PIN_PROCESSOR_EXCLUDE_LIST=0,1,68,69,
136,137,204,205 # 
export KMP_AFFINITY=scatter
export KMP_HW_SUBSET=1t181

2018/2/14

KNLMPI  
• PPN (Process Per Node)= 2 

• I_MPI_PERHOST=2

• mpiexec.hydra“-ppn 2”



• I_MPI_PIN_DOMAIN

 
 

182

: 2

export OMP_NUM_THREADS=32# 1HT
export I_MPI_DEBUG=5#  
export I_MPI_PIN_DOMAIN=128
export I_MPI_PERHOST=2
export
I_MPI_PIN_PROCESSOR_EXCLUDE_
LIST=0,1,68,69,136,137,204,2
05 # 

export KMP_AFFINITY=scatter
export KMP_HW_SUBSET=1t

2018/2/14

KNLMPI  
• PPN (Process Per Node)= 4 

• I_MPI_PERHOST=4

• mpiexec.hydra“-ppn 4”



• I_MPI_PIN_DOMAIN

 
 

183

: 4

export OMP_NUM_THREADS=16# 1HT
export I_MPI_DEBUG=5#  
export I_MPI_PIN_DOMAIN=64
export I_MPI_PERHOST=4
export
I_MPI_PIN_PROCESSOR_EXCLUDE_
LIST=0,1,68,69,136,137,204,2
05 # 

export KMP_AFFINITY=scatter
export KMP_HW_SUBSET=1t

2018/2/14

KNLMPI



184

: 8• 8)&• I_MPI_PERHOST'$I_MPI_PIN_DOMAIN''
256&%!", +%$1/.&2310!
"(• I_MPI_DEBUG=5&,$2310&+



"*-

/.

*-,'# +,$[40] MPI startup(): shm and tmi data transfer modes…
[0] MPI startup(): 192
269565
c3850.ofp {2,70,138,206}

2018/2/14

KNL

185


• core-setting.sh• 3

( '%+• HTX=1 (-,!).1/0 2• VERBOSE=1 (
*!
+
2
• DEBUG=1 (-,) "%+2• source core-setting.sh• *&#!$+• 3

Setting:
export I_MPI_PIN_DOMAIN=64
export I_MPI_PERHOST=4
export KMP_HW_SUBSET=1t
export KMP_AFFINITY=verbose,compact
export HFI_NO_CPUAFFINITY=1
export
I_MPI_PIN_PROCESSOR_EXCLUDE_LI
ST=0,1,68,69,136,137,204,205[0] MPI startup(): Multi-threaded optimized
library…

2018/2/14

?KNL 

186 C MPI+OpenMP
double t0, t1, t2, t_w;..$&)parallel:=28<)-'
#pragma omp master{
ierr = MPI_Barrier(MPI_COMM_WORLD);}#pragma omp barrier
t1 = omp_get_wtime();@"") #%6;097. A#pragma omp reduction(max: t0){t2 = omp_get_wtime();t0 = t2 - t1;}
#pragma omp master{ierr = MPI_Reduce(&t0, &t_w, 1,
MPI_DOUBLE,MPI_MAX, 0,
MPI_COMM_WORLD);

5:/ 
.#6;341=&B0*+(-"* +,,**
.6;341>!-

2018/2/14

?KNL 

187 Fortran
MPI+OpenMP
double precision t0, t1, t2, t_w..$&)parallel:=28<)-'$!omp mastercall MPI_BARRIER(MPI_COMM_WORLD, ierr)$!omp end master$!omp barrier
t1 = omp_get_wtime()@"") #%6;097. A$!omp reduction (max:t0)
t2 = omp_get_wtime()
t0 = t2 - t1$!omp end reduction$!omp mastercall MPI_REDUCE(t0, t_w, 1,&
MPI_DOUBLE_PRECISION,&MPI_MAX, 0, MPI_COMM_WORLD, ierr)$!omp end master5:/ 
.#6;341&B0*+(-"* +,,**.6;341>
!-2018/2/14

KNL

• “dynamic/runtime”• “!$omp master!$omp end master”

188#   #!"  

!$omp parallel private (neib,j,k,i,X1,X2,X3,WVAL1,WVAL2,WVAL3)!$omp&
private (istart,inum,ii,ierr)!$omp masterCommunication is done by the master thread (#0)!C
!C– Send & Recv.
(…)
call MPI_WAITALL (2*NEIBPETOT, req1, sta1, ierr)!$omp end master!C!C-- Pure Inner Nodes

The master thread can join computing of internal
nodes after the completion of communication!$omp do schedule (runtime)
export OMP_SCHEDULE="dynamic,[chunksize]"
do j= 1, Ninn
(…)
enddo
!C
!C-- Boundary Nodes
Computing for boundary nodes are by all threads!$omp dodo j= Ninn+1, N
(…)
enddo

default: !$omp do schedule (static)!$omp end parallel

Idomura, Y. et al., Communication-overlap techniques for improved strong scaling of gyrokinetic Eulerian codebeyond 100k cores on the K-computer, Int. J. HPC Appl. 28, 73-86, 2014

2018/2/14

189

KNL

Speedup by communication overlapping(OFP: 32 nodes)• Memory model: C: Cache, F: Flat
• Sub NUMA: Q: Quadrant, N4: SNC-4

C+Q

F+Q

C+N4

F+N4

• (m) : core binding to avoid “busy”
core
C+Q(m)

F+Q(m)

C+N4(m)F+N4(m)

200

300

400

500

16%
14%
12%
10%
8%
6%
4%
2%
0%
-2%
-4%

Overlap

50

100

2018/2/14

KNL
• Intel VTune 2017  

190

2018/2/14

KNL

191
• MPI







2M!"

Fortran (&Fortran# "2018
%!DIR$ ATTRIBUTES ALIGN : 2097152 :: arrayMPI
real(8),allocatable :: arrayMPI(:,:,:,:)C $"!# "%arrayMPI = (double *)__mm_malloc(array_size,
2*1024*1024);

2018/2/14

192

KNL

Intel Advisor
• (%*/!+#&
• module load advisor • $." 0advixe-cl)-&• advixe-cl --collect survey --project-dir ./advi --no-auto-finalize -./a.out• MCDRAM0numactl --membind=1 advixe-cl --collect survey --project-dir ./advi--no-auto-finalize -- ./a.out• MPI0
mpiexec.hydra -n ${PJM_MPI_PROC} advixe-cl --collect survey--project-dir ./advi --no-auto-finalize -- ./a.out• Tips:• --no-auto-finalize : KNLfinalize
,-'.&

/



2018/2/14

KNL

193

Intel Advisor(1)• VNC$.( • ofp01$ vncserver :0 -geometry 1280x1024/VNC )%,.' 0• ofp01$ export DISPLAY=:0/ :0 0•

VNC*.& • note$ ssh -N -L 55901:localhost:5900 z30105@ofp01.jcahpc.jp &• VNC$.( :0  5900, :1  5901, …
• VNC#+"!-& localhost:55901

2018/2/14

194

KNL

Intel Advisor(2)•source, binary 

2018/2/14

195

KNL

Intel VTune
• +,-)/' 1-$0#
• module load vtune• &0% 2amplxe-cl "*/'
• amplxe-cl -collect hpc-performance --no-auto-finalize -r vt000hp -./a.out• MCDRAM2numactl -m 1 amplxe-cl -collect hpc-performance --no-auto-finalize-r vt000hp -- ./a.out• MPI2
mpiexec.hydra -n ${PJM_MPI_PROC} amplxe-cl -collect hpcperformance --no-auto-finalize -r vt000hp -- ./a.out•  knob
• Tips:• --no-auto-finalize : KNLfinalize/(0'

1.!

2018/2/14

KNL

Intel VTune(1)
• VNC

• Advisor• CUI finalize• $ amplxe-cl -finalize -search-dir . -r vt000hp

1962018/2/14

KNL

Intel VTune(2)197

2018/2/14

198

KNL (DDN: Infinite Memory
Engine)• SSD  /E925H508E7• I960 TBH ;E,  1.5TB/sec• IO500Top• ,A)'D6:B5(%• erasure coding#• Lustre<&)C025> +?40@$
• /cache +?40@! ',32F8E7 "G• 25E1D-F25E1)DH25E1'*6 G#PJM -x STGIN_LIST=<%B26<&)C>#PJM -x STGOUT_LIST=<%B26<&)C>• 25E1)D%• prestage-wait .=D7

2018/2/14

KNL199

IME
1. POSIX:/work/gt00/t004
4x 
/cache/gt00/t00
44x 
Intel MPI  

2. MPI IO:
a. MPIIntel MPI

MVAPICH2
module switch impi mvapich2b.  ime:// 
MPI_File_open(MPI_COMM_
WORLD,”ime:///work/gt00/t0044x…”,… );2018/2/14

200

KNL

IME
• NAS Parallel benchmark, BT-IO • class=C, 64 (16 x 4)
• Lustre

Simple 

• class=D, 256(16x16)• Lustre
Simple  , IME

Simple



90000
80000

class=C

Mop/s/total

70000

class=D

60000
50000
40000
30000
20000
10000
0
LustreIME (POSIX)
SimpleIME

LustreIME (POSIX)
Full

IME

2018/2/14

KNL   201

2018/2/14

KNL

202

• 1

.Fortran90 (!#+0Samples-knl.tar.gz• tar%-)+1
Fortran90&",$'* • C/ : C• F/ : Fortran90

• (!#+
/work/gt00/z30105 (/home

/2018/2/14

KNL


1.
1.

2.
3.
4.
5.

203

 (1/2)

cd &+"  Lustre$)'
$ cd /work/gt00/t0044x/work/gt00/z30105   Samples-knl.tar.gz  *!(#,
$ cp /work/gt00/z30105/Samples-knl.tar.gz ./Samples-knl.tar.gz  $ tar xvfz Samples-knl.tar.gz
Samples $)
$ cd Samples-knl
%($)
$ cd streamC- $ cd C
Fortran90- $ cd F2018/2/14

204

KNL (2/2)

6. make$ make7. (stream) 

$ ls





2018/2/14

 
1.
2.
3.

4.
5.

KNL

205

stream2/61"+ )$ pjsub stream.bash
% * 043+)$ pjstat)#%2,-6*)
stream.bash.eXXXXXX
stream.bash.oXXXXXX 8XXXXXX& 9
% 2,-6%+!')$ cat stream.bash.oXXXXXX.57 &%2,-6$)%" % ($ cat stream.bash.eXXXXXX