&EKAI#>D!ABCI GN
8*<;5OL:6 =47 P/'0IJ2.1) 8*FH?-IJ9)A@BIJ% )A@B#IJ!+
3,(C5% "%# #$ M#+2AI
A

.

.

•AI

.

2018

619I•

•

3
17A.

3!  AI Bridging Cloud Infrastructuren@EK/ONS:+#*^adF[X21H28PQZH0_[Xne- EAVW7^a<5I`%,"&(,%) !$'n

6 MUS;T?/ONSY\^a<5>R
3.>cbG
49CLD]JfB=8Univ. Tokyo / Kashiwa II Campus 
550 PFLOPS ()37 PFLOPS ( )
19.88 PFLOPS(TOP5005,  1)
14.423 GFLOPS/W (GREEN5004# 2)508.85 TFLOPS(HPCG5 2)" (): 2.3 MW!PUE: 1.1 ( )201881

4

 #""! for everyone
#'&(Expert•••

ABCI,CG:5BFG0YxS^JSr\S_o Nj[f]yc ZABCIX/17@ [H24QRk|
ABCI845BFG0)>G84 W )>G.$'G1`-G719UABCI D=!FG1"*7+5BKJiw

Advanced & Intermediate•••[H512<:gqD31 bWt
n 3=9%&"84269YeA8E k|
P gqmT pW uxz~UOVl a}
LIV `Beginner••

8#>C;G,dYMn WebUI?1UO{v UOk|
B2B2CVl>C69=(@bWt

AI8)

%657.&;/[hAIz~ sn AI

1= :<-*$,4& 2+3&
19 :<(#2+>0 ;<!%(4C6 >B1("K@E 'Top500 (Nov 2018) LinpackHPCG (Nov 2018) 

Green500 (Nov 2018) Rank

SystemRmax

Rank

System

HPCG

Rank

System

GF/W

1

Summit ( )143.5PF1

Summit ( )

2.9PF

1

Shoubu system B ()

17.6

2

Sierra ( )94.6PF

2Sierra ( )

1.8PF

2

DGX SaturnV Volta ( )

15.1

3

Sunway TaihuLight ()93.0PF3

K computer ()

603TF

3

Summit ( )

14.7

4

Tianhe-2A ()61.4PF

4Trinity ( )

546TF

4

ABCI ()14.4

5

Piz Daint ( )21.2PF

5

ABCI ()509TF

5TSUBAME3.0 ()

13.7

6

Trinity ( )20.2PF

6Piz Daint ( )497TF

6Sierra ( )12.7

7ABCI ()19.9PF

7

Sunway TaihuLight ()481TF

7AIST AI Cloud ()

12.7

8

SuperMUC-NG ()19.5PF

8Nurion ()391TF

8MareNostrum P9 ( )

11.9

9

Titan ( )17.6PF9

Oakforest-PACS ()385TF

9PreE ()

11.4

10

Sequoia ( )17.2PF

10Cori ( )

355PF

10

Taiwania 2 ()11.3

?L!%.9HABCI2,DOESummitSierra
48D>#8D;<!%*&)G07/
%+AIIJ3-F5@E+$( ;<=A:/'ImageNet Large Scale Visual Recognition Challenge (ILSVRC)• ImageNet%"!#&CLS (Top-5error) (CLS)ClassificationResults• WordNet<W . %"3[RT($-2• Ce%"120Fh$&%"10Fh0.3• S.7c(CLS)g?
GAIb

%*.'2• 20124H`%*.'2+8> EU• 20154MSRAResNet5;M_N=Top5 error5.1%] )10• H`HW/SW6@HPCVZDdAVZ
iO Ce\?YX:Ba9

Classification Error• ILSVRC.7c(CLS)JQ^<(LOC)
K@M_(DET)Lf"0/ ,PjY

0.25

0.28
0.26

Ensemble0.2AlexNet

0.15

0.16

0.12

0.1SENetResNet

ZFNet

GooLeNet

16.7% ↓ 23.3% ↓0.07

0.0502010

2011

2012

2013

2014

0.036

0.030.023

2015

2016

2017

http://imagenet.org/challenges/talks_2017/
ILSVRC2017_overview.pdf

5DX 5@&30$'(*/1 4#
!#"" 2 ( 1KE-1)

ImageNet / ResNet-50 (Relative speedup & Accuracy)

224 sec
Tesla V100 x2176 0.765
0.76

6.6 minTesla P40 x2048

29 hours
Tesla P100 x80.755
0.75

15 min30 min Tesla P100 x1024
Full TPU Pod1 hour
Tesla P100 x256

0.745 8:S?8DB

4<L  $OT1R793M, %)0;,H(WEB), 2018.11.15 97> H30&.91>
H30&.LN5@'=(/ UIWQ2>AJ+27>
P'=(/FBV6CG*

0.74

MSRA (2015)

Facebook(2017)

Google Brain(2017)

PreferredNetworks(2017)

Relative speedup

Tencent(2018)

Sony + ABCI(2018)

Accuracy

"Google  */62ABCI GoogleTencent)-.% !5,+-= O(seconds)= O(hours or days)= O(weeks or months)

model.py
import tensorflow as tf# code@protonn.parameters #decorate
def do_something():
parameter_1 = 42
print(parameter_1)# codeAnnotated Python CodeABCI

Hyper-Optimizer

control

Metadata Assimilation
Logging

LauncherMonitor & Visualize
Drive protoNN(Command Line Interface)

optimizeevaluate
profilescale…..

Frontend

Online(Parallel Instances)
InvokesubmitJob

OfflineSpawn

resultReportdesignC P
OS H N
DSubmit

Repeat Submitting
Jobs Until it Works!

-:

Launch

Traditional DL• Time for inference• Time for training• Time for designing

Empowered DL

500
450400350
300
250200150
100
500

Request On-demand

Scheduler(Virtual Elastic Queue)

BatchScheduler

BackendprotoNN Component
protoNN Sub-component

ABCI

External Component

monitor & visualize

https://github.com/undertherain/protoNN

https://github.com/undertherain/protoNN

7

ABCI

ABCI
550 PFlops(FP16), 37.2 PFlops(FP64)
476 TiB, 1.74 PB NVMe SSD(GPU22 PB GPFS
DDN SFA14K (w/ SS8462 Enclosure x 10) x 3) x 1088

• 12TB 7.2Krpm NL-SAS HDD x 2400
• 3.84TB SAS SSD x 216• NSD Server x 12

GPU

NVIDIA Tesla V100 SXM2 x 4CPUIntel Xeon Gold 6148 (2.4GHz/20cores) x 2Memory384GiB

Local Storage

Intel SSD DC P4600 (NVMe) 1.6TB x 1Interconnect

InfiniBand EDR x 2(GPU) x 10

• Intel Xeon Gold 6132 (2.6GHz/14cores) x 2• 768GiB Memory, 3.8TB NVMe SSD

2018x48128x 15(Infiniband EDR)

• Mellanox CS7500 x 2• Mellanox SB7890 x 2298

• Nexsus 3232C x2• FortiGate 1500D x2• FortiAnalyzer 400E x1(10GbE)

100Gbps

SINET513 
FUJITSU PRIMERGY Server (2 servers in 2U)
CPUXeon Gold 6148 (27.5M Cache, 2.40 GHz, 20 Core) x2

GPU

NVIDIA Tesla V100 (SXM2) x4Memory384GiB DDR4 2666MHz RDIMM

Local Storage

1.6TB NVMe SSD (Intel SSD DC P4600 u.2) x1

Interconnect

InfiniBand EDR x2

SkylakeSkylake
DDR4-2666
32GB x 6
IB HCA (100Gbps)

128GB/s

Xeon Gold 10.4GT/s x3 Xeon Gold6148
6148
UPI x3

DDR4-2666
32GB x 6

PCIe gen3 x16

PCIe gen3 x16
PCIe gen3 x16

128GB/s

PCIe gen3 x16x48 switch

x64 switch
NVLink2 x2

Tesla V100 SXM2

Tesla V100 SXM2

Tesla V100 SXM2

Tesla V100 SXM2IB HCA (100Gbps)
NVMe

14   
SPINE#1
CS7500SPINE#2CS7500n  34 Tesla V100 136
•   ) $'
1.16 PFLOPS (FP64)17 PFLOPS (FP16)
c.f. Google TPU 2.0 Pod (11.2PF), 3.0 Pod (>100PF)•  *(&67.33 kW

1/3 Oversubscription BW
IB-EDR x 24

LEAF#1 LEAF#2 LEAF#3 LEAF#4
SB7890 SB7890 SB7890 SB7890

CX400 CX400 CX400
#1#2
#3

CX400
#17

Rack #1

CX2570#3
CX2570#4

CX400
#17

CX2570#3
CX2570#4

CX400 CX400 CX400
#1
#2
#3

CX2570#1
CX2570#2

CX2570#33
CX2570#34

LEAF#1 LEAF#2 LEAF#3 LEAF#4SB7890 SB7890 SB7890 SB7890

CX2570#5
CX2570#6

Full bisection BW
IB-EDR x 72

CX2570#1
CX2570#2

Full bisection BW
IB-EDR x 72

CX2570#33
CX2570#34

FBB#1 FBB#2 FBB#3
SB7890 SB7890 SB7890

CX2570#5
CX2570#6

FBB#1 FBB#2 FBB#3
SB7890 SB7890 SB7890

Rack #2n    •   ,7200Gbps!
"  +%•  #1/3   +%InfiniBand EDR x4
InfiniBand EDR x6
InfiniBand EDR x1

ABCI%*4>%F“1#70kW=% ") 84 ”
$%!

: 2! 3 10;&)=
/$%0.
'A-0+6 (3.25MW)

6: 19m x 24m x 6m

n 9B; 71

UPS?, >:*18#UPS n ED/2%/m2 %$#&$%! # 2<+5
#
C@G<8,:200kW<8,:3MW

72#

n #.

• (#: 90 (ABCI 41+-!)• ": 144n 0(3;• 3.25 MW (ABCI "2.3MW-!)?,>:*n C@4(• 3.2 MW (97 " 5)•  '<60kW1<10kW
Data Center Image

15

ABCI

034

Cold Water
Circuit32℃

16Water
40℃ Hot
Circuit

Fan Coil Unit
Coolability 10kW

C20CDU
Coolability 60kW

Front side

35℃

AirWater

Air40℃
Hot Aisle CappingComputing
ServerAir 35℃

Air 40℃

Cold Aisle

35℃

Water Block(CPU or/and Accelerator, etc.)F

Hot Aisle
19 or 23 inch
Rack (48U)

40℃

D17

ABCI Software Stack (1)ABCIHPC#'%3 *,5 .246)"*&GPU+($0/ .2 " ! -1

Operating System

CentOS
RedHat Enterprise LinuxJob Scheduler

Univa Grid Engine

Container Engine

DockerSingularity

MPI

OpenMPI
MVAPICH2-GDR
MVAPICH2Intel MPI

Development Tools

Intel Parallel Studio XE Cluster EditionPGI Professional EditionNVIDIA CUDA SDK
GCC, Python, Ruby, R, Java, Scala, Lua, Perl

Deep Learning(available)

Caffe, Caffe2, TensorFlow, Theano, Torch, PyTorch, CNTK, MXnet, Chainer, Keras
NVIDIA GPU Cloud (NGC)Big Data Processing

Hadoop, Spark

18

ABCI Software Stack (1)’[user@es1 ~] $ qrsh –g group_name –l rt_F=4 –l h_rt=1:00:00

(#!$"[user@g0001 ~] $ module load python/3.6/3.6.5[user@g0001 ~] $ module load cuda/9.0/9.0.176.4[user@g0001 ~] $ module load cudnn/7.4/7.4.2[user@g0001 ~] $ module load openmpi/2.1.5

ABCI*/

[user@g0001 ~] $ python3 -m venv horovod[user@g0001 ~] $ source horovod/bin/activate(horovod) [user@g0001 ~] $ pip3 install tensorflow-gpu keras torch(horovod) [user@g0001 ~] $ pip3 install horovod(horovod) [user@g0001 ~] $ mpirun -n 16 -map-by ppr:4:node python3 keras_mnist.py qsub$"

Python1.-,)'+&!

MPI$"

%0

19ABCI Software Stack (2)n 2  ABCI• Docker: R<+8,C• Singularity: @?B,5(?DLdhSMI !HPCHKUPX
n    %"• )@5>*9;4/$MI !RJT]" ABCIE`• ABCIMI *B/3")@5>*9;4/$qWYln ABCI #'! 
$&• ABCIRkO !ChainerMNNNabla"  Hp0$7=4B)6?:A("*B/3OtC^j0."\0$7=4B)_PQ
V a[*B/3mesgOfcFiDockerSingularity*B/3"8, -61&'#rnbo%B,1>NGZUP"a[ 92>&'#20

ABCI Cloud ServicesABCI #$,.3 5 )6!"(7-+4/CPU / GPU /  (GB) /  / )40 / 404/4

360 / 384

1.4 / 1.6

G.large

20 / 404/4

240 / 384

0.7 / 1.6

G.small

5 / 401/4

60 / 384

0.175 / 1.6

C.large

20 / 400/4

120 / 384

0.7 / 1.6

C.small

5 / 400/4

30 / 384

0.175 / 1.6

F( 50 2*&%1G.large
C.largeGPU

C.small'%-+GPU0CPU04/
G.small

GPU0CPU0GPU1GPU1GPU2CPU1GPU2
CPU1GPU3GPU321ABCI Cloud ServicesABCI\I!1+$&M5H]U ($#i?NiF7)!O
:YD=RK)!IDEZj!&2 )!Th
Spot

On-demand

Reserved
IDE ()

Group Storage

Secure Object Storage ()
/ ($#F7i /*)!
PJ @A;amSc8LVJ@Alc8

1 / 512

3"0%*F7i /*)!
PJ @A;amSc8LVJ@Alc8

1 / 324mSGoRK)!
RK@A[ c81 / 32

WebUI,!_9E<gbnB

1+CXW !&2 )!
^q200GB-.fd6`QB r> 1+fd PJ]UN/AspH?[ * &!&2 )!
Amazon S3kt 0'!&2 PJe[ c8N/A

22ABCI     ABCIr(L) KtsweN"(*,2;f_R#8GAI~Z klmap*,2; *@',M¢H{wU wKtswPS
aocj`n
IqU41861
0fxMg,2%:C/| "(*,2;
H i\g[Ty^£pgRr
u\Y¡XQ f¦i\dW©JV ?)
X¤ABCI&AG50=DG+b

¥ABCIr3.
"(*,2;

$>B2:F!497
_X§H?)
X

¨B2B2C
Yv8E&A;
YzCªe]}[i\g«AIh3.G.ON<3CZ

AB– SP/,uNpT /,LuZO/,9'Ekf!3?
/,Q7/</,RIdZOABCI^ /, %(*.6} +"8;.^mm
KDs/,^IC
– !3?|/,Aa/,tq Xn+?,jo /,
xHE_BM{ZOz@e9'IM{@ry/,]%?4wlZOV3>$:6Jh  \F SINET5

100Gbps

100Gbps

ABCI#:1*0=)(*)

X+?,
(*) H31.4W~&2*Ig

100Gbps x20cvZO

+?,
ABCI[i&2*YbQ`9'Ekf/,RI
UGM{@e/,5<M{ )ooo. 4(bgorsoibdsdis

4EAG IG I GB. 4odo
4 .GC BCEB
DFosuaso stoooedrooorcioic. 4. 4fcscsicsi dcscob. 4. 4ooib. 4soibiocbgbs . 4ossifsdesegsdssgsiooio ebocobs. 4baicucooo( ioos) ibd . 4oisb. 4oo. 4soocid . 4obiobboi

cibo. 4. 4ssido. 4. 4oub. 4. 4obso

939o. 4dnocoiis. 4obiibfo, ibbo. 4io. 4i

版数： Version 1.0
発行日： 2018/12/20国立研究開発法人産業技術総合研究所
情報・人間工学領域

A•–C CCIABCI のセキュリティ
ホワイトペーパー
Version 1.0

2018 年 12 月

国立研究開発法人 産業技術総合研究所
情報・人間工学領域

1/19

C fe
BBaf

MPS _Ib

C feUC



dc

B

0127 07 0

A/ /00/.

B

0127 07 0/

A/ /00/ 0:://- 97:0 / GGFF





.

w
j _ r
9 : A 1 B A,c an02 rbi N o e Ttm

FV– N
F j _ rVUF v
IpG lreF 9CC B

7: :DsP.
•chN
PVC

D/3. .

PC 9C

27

ABCI Grand Challenge•••{9vd_F!"g_#;]\_>DLVNmTzK%Uj#AIkt'2&,ABCI(27,+157* /6(20%Hc
ABCI(27,+157*ABCI
L9Wd-,Q #1,088-,4,352GPU%L924@A1ns(4/waSI  #Jxq+157*/6(20eMCP`oP ?Z$#zKCnsmT^[% l "

2019:G 3EHcXB
6=/9=/12=

•R1ERFP4=XB

n VN8rSIb<3.)4%pOYnih

#12018430

2018530

20187

#22018831

2018930

201810#320181130

2018122220191 2018:G8(4/fxmT

uyHc$28 HPCIHigh Performance Computing Infrastructure•HH
P
C SINET5I •

HPCI 
HPCI ABCI

HPCI•

31

H


ABCIbmiI

HPCIdurlA
onIA

Cmiwvgmi e
mifc_ a c
mif_ a c

B

AC

onI.
.
.
.

td. /. /. /. /::. . /

sp.
.

jh.
. .
. .

.

.

//