ビッグデータ・AI時代のクラスタ型スーパコン
ピュータ：東工大TSUBAME3.0から産総研ABCI (AI Bridging Cloud Infrastructure)へ
松岡 聡
教授：東工大 GSIC & 特定フェロー：産総研AIRC
2016年12月16日
PCクラスタコンソシウム講演@秋葉原

JST‐CREST “Extreme Big Data” Project (2013‐2018)
Future Non-Silo Extreme Big Data Scientific Apps

Given a top‐classsupercomputer,how fast can we
accelerate next
generation big
data c.f. Clouds?

Ultra Large Scale
Graphs and Social
Infrastructures

Large Scale
Metagenomics

Co-Design
EBD Bag

Massive Sensors andData Assimilation in
Weather Prediction

Co-Design Co-Design

EBD System Softwareincl. EBD Object System

KVS

KVSGraph Store

Cloud IDC
Very low BW & Efficiency
Highly available, resilient

Cartesian Plane

NVM/Fla
NVM/Flas
2Tbps HBM
NVM/Flashh
4~6HBM Channels NVM/Flas
NVM/Fla
NVM/Flasshh
1.5TB/s DRAM & hsh
DRAM
DRAM
NVM BW
DRAM
DRAM
DRAM
DRAM
Low
Low
High Powered
30PB/sI/O BW Possible
Main CPUPowerPower
1
Yottabyte/YearCPUCPU
TSV Interposer

KVS

EBD KVS

Exascale Big Data HPC

Issues regading
Architectural,
algorithmic, andsystem software
evolution?

PCB

Convergent Architecture (Phases 1~4)
Large Capacity NVM, High-Bisection NW

Use of GPUs?

Supercomputers
Compute&Batch-Oriented
More fragileExtreme Big Data (EBD) Team
Co-Design EHPC and EDB Apps
• Satoshi Matsuoka (PI), Toshio • Yutaka Akiyama, Ken Kurokawa (TokyoEndo, Hitoshi Sato (Tokyo Tech.) Tech) (EBD App1 Genome)(EBD Software System)•• Osamu Tatebe (Univ. Tsukuba) • Takemasa Miyoshi (Riken AICS)(EBD App2 Weather, data assim.)(EBD-I/O)• Michihiro Koibuchi (NII)(EBD Network)••

Toyotaro Suzumura (IBM Watson /Columbia U)(EBD App3 Social Simulation)(now merged into Matsuoka Team)EBD Recent Awards & Accolades• IEEE Sidney Fernbach Award (2014, Matsuoka)• Rakuten Technology Award (2014, Matsuoka)• HPCWire 2015 Readers Choice Awards Outstanding Leadership inHPC (2015/11)• Satoshi Matsuoka, co‐award with Prof. Jack Dongarra@U‐Tennessee• World No.1 in Graph500 Benchmark on K computer(2016/11)• 4 consecutive wins: 2015/06, 2015/11, 2016/06, 2016/11• IPSJ Computer Science Research Award for Young Scientists(2016/10)• Keita Iwabuchi, “Towards a Distributed Large‐Scale Dynamic Graph DataStore”, IPSJ SIG Technical Report Vol. 2015‐HPC‐153, 2015/03• 2015年度情報処理学会長尾真記念特別賞(2016/06)• 鯉渕 道紘 「ラックスケールコンピュータ・ネットワークの設計に関する先駆的な研究」

Open Source Release of EBD System
Software (install on T3/Amazon/ABCI)• mrCUDA• rCUDA extension enabling remoteto-local GPU migration• https://github.com/EBDCREST/mrCUDA• GPU 3.0
• Co-Funded by NVIDIA• CBB• I/O Burst Buffer for Inter CloudEnvironment• https://github.com/EBDCREST/cbb• Apache License 2.0• Co-funded by Amazon• ScaleGraph Python• Python Extension for ScaleGraph
X10-based Distributed Graph Library• https://github.com/EBDCREST/scalegraphpython• Eclipse Public License v1.0

• GPUSort• GPU-based Large-scale Sort
• https://github.com/EBDCREST/gpusort• MIT License• Others in development…

Elapsed Time (ms)

The Graph500 – 2015~2016 – 4 Consecutive world #1
K Computer #1 Tokyo Tech[EBD CREST] Univ. Kyushu [Fujisawa
Graph CREST], Riken AICS, Fujitsu 88,000 nodes,
1500

Communi…
Computati…

73% total exec
time wait incommunication

660,000 CPU Cores
1.3 Petabyte mem
20GB/s Tofu NW

1000
500065536 nodes(Scale 30)(Scale 40)

Effective x13
performance c.f.
Linpack≫

64 nodesListRank

GTEPS

November 2013

45524.12

LLNL-IBM Sequoia TaihuLight*Problem size is
1.6 million CPUs
weak scaling
10 million CPUs
1.6Petabytemem
1.3 Petabyte mem“Brain-class” graph
Top-down only

June 2014

117977.05

Efficient hybrid

November 2014

2June, Nov 2015
June Nov 2016

1Implementation

Efficient hybrid

38621.4

Hybrid + NodeCompression

K‐computer No.1 on Graph500: 4th Consecutive Time

Performance (GTEPS)• What is Graph500 Benchmark?
• Supercomputer benchmark for data intensive applications.
• Rank supercomputers by the performance of Breadth‐First Search for very hugegraph data.
45000
K computer (Japan)
This is achieved by a combination
40000
Sequoia (U.S.A.)
of high machine performance and35000
Sunway TaihuLight (China)
our software optimization.
30000
25000
20000
15000

No.1

10000
5000
0
Jun 2012•

Nov Jun 2013 Nov Jun 2014 Nov
2012
2013
2014

Jul 2015

Nov Jun 2016
2015

• Efficient Sparse Matrix Representation withBitmap• Vertex Reordering for Bitmap Optimization• Optimizing Inter‐Node Communications• Load Balancing
etc.

Koji Ueno, Toyotaro Suzumura, Naoya Maruyama, Katsuki Fujisawa, and Satoshi Matsuoka, "Efficient Breadth‐First Search on
Massively Parallel and Distributed Memory Machines", in proceedings of 2016 IEEE International Conference on Big Data (IEEEBigData 2016), Washington D.C., Dec. 5‐8, 2016 (to appear)Towards a Distributed Large-Scale Dynamic Graph Data Store
Goal: to develop the data store for large-scale

Dynamic Graph Construction (on-memory)

dynamic graph analysis on supercomputers

Against STINGER (single-node)

Strea m ing ed g es

Com p .
N od eCom p .
N od eD yna m ic Gra p h A p p lica tion

La rg e-sca le D yna m ic Gra p h D a ta Storesh a re .sa n d ia .g o v

m m ap
N VRA M

mmap
N VRA M

mmap
N VRA M

Node Level Dynamic Graph Data Store
Follows an adjacency-list format and leverages anopen address hashing to construct its tables

STINGER• A state-of-the-art dynamic graph processing
framework developed at Georgia Tech
Baseline model•
A naïve implementation using Boost library (C++) andthe MPI communication framework
Baseline
DegAwareRHH
212x

Speed Up

Com p .
N od e2000612Parallels

24

Extend for multi-processes using an async
MPI communication framework

Inserted Billion Edges/sec

Multi-node Experiment

2 billion
insertions/s

Number of Nodes (24 processes per node)
K. Iwabuchi, S. Sallinen, R. Pearce, B. V. Essen, M. Gokhale, and S. Matsuoka, Towards a distributed large‐scale dynamic graph data store. In 2016
IEEE Interna‐ tional Parallel and Distributed Processing Symposium Workshops (IPDPSW)Large-scale Graph Colouring (vertex coloring)

SC’16• Color each vertices with the minimal #colours so that no two adjacentvertices have the same colour• Compare our dynamic graph colouring algorithm on DegAwareRHH against:
1. two static algorithms including GraphLab
2. an another graph store implementation with same dynamic algorithm (Dynamic-MAP)

Static colouring

Our DegAwareRHH

Baseline

Scott Sallinen, Keita Iwabuchi, Roger Pearce, Maya Gokhale, Matei Ripeanu, “Graph Coloring as a Challenge Problem for Dynamic
Graph Processing on Distributed Systems”, SC’169

Incremental Graph Community Detection• Background• Community detection for large‐scale time‐evolving and dynamicgraphs has been one of important research problems in graphcomputing.
• It is time‐wasting to compute communities entire graphs every timefrom scratch.

101.0xfaster• Proposal• An incremental community detection algorithm based on core
procedures in a state‐of‐the‐art community detection algorithm
named DEMON.

101.5xfaster• Ego Minus Ego, Label Propagation and Merge

69.2x
fasterHiroki Kanezashi and Toyotaro Suzumura, An Incremental Local‐First Community Detection
Method for Dynamic Graphs, Third International Workshop on High Performance Big Graph
Data Management, Analysis, and Mining (BigGraphs 2016), to appear

GPU‐based Distributed Sorting

EBD Algorithm Kernels[Shamoto, IEEE BigData 2014, IEEE Trans. Big Data 2015]• Sorting: Kernel algorithm for various EBD processing• Fast sorting methods– Distributed Sorting: Sorting for distributed system• Splitter‐based parallel sort
• Radix sort
• Merge sort

– Sorting on heterogeneous architectures• Many sorting algorithms are accelerated by many cores and high memory bandwidth.

• Sorting for large‐scale heterogeneous systems remains unclear• We develop and evaluate bandwidth and latency reducing GPU‐based HykSort onTSUBAME2.5 via latency hiding– Now preparing to release the sorting library•

Weak scaling performance (GrandChallenge on TSUBAME2.5)– 1 ~ 1024 nodes (2 ~ 2048 GPUs)– 2 processes per node
– Each node has 2GB 64bit integer•

C.f. Yahoo/Hadoop Terasort:
0.02[TB/s]– Including I/O

Performance prediction

HykSort 1thread
HykSort 6threads
HykSort GPU + 6threads

30

x1.4

0.25
[TB/s]

Keys/second(billions)

GPU implementation of splitter‐based sorting (HykSort)

20

x3.61

10x389
00500

1000

1500

2000

# of proccesses (2 proccesses per node)•x2.2 speedup compared toCPU‐based
implementation when the# of PCI bandwidth
increase to 50GB/s

PCIe_#: #GB/s bandwidthof interconnect between
CPU and GPU
8.8% reduction of overallruntime when theaccelerators work 4 times
faster than K20x

Xtr2sort: Out-of-core Sorting Acceleration
using GPU and Flash NVM [IEEE BigData2016]
How to combine deepening memory layers for future
HPC/Big Data workloads, targeting Post Moore Era?
•Sample-sort-based Out-of-core Sorting Approach for Deep Memory Hierarchy
Systems w/ GPU and Flash NVM– I/O chunking to fit device memory capacity of GPU– Pipeline-based Latency hiding to overlap data transfers between NVM, CPU, and GPU
using asynchronous data transfers,e.g., cudaMemCpyAsync(), libaio
GPU

RD

R2H

H2DEX

D2H

H2W

RD

R2H

H2D

EX

D2H

H2W

WR

WR

chunk&i+6
chunk&i+5
chunk&i+4

RD

R2H

H2DEX

D2H

H2W

RD

R2H

H2D

EX

D2H

H2W

WR

RD

R2H

H2DEX

D2H

H2W

WR

RD

R2H

H2DEX

D2H

H2W

WR

R2H

H2DEX

D2H

H2W

WR

chunk&i+3
chunk&i+2
chunk&i+1
chunk&ic chunks

RDWR

GPU + CPU + NVMtimex4.39

CPU + NVMDeep Learning Shock
NVIDA GTC2015 March 2015

Elon Musk@TESLA

Jeff Dean@Google

Andrew Ng@Baidu

Deep Learning Shock
GTC2015 March 2015

Elon Musk@TESLA
Deep Learning Rules!
HPC important basis

Jeff Dean@Google
Deep Learning Rules!
HPC important basis

Andrew Ng@Baidu
Deep Learning Rules!
HPC important basis

Deep Learning ShockNVIDIA GTC2015 March 2015
Comment from Anonymous TSUBAME user
We have thousands of GPUs
and I can use them at will;
But I know nothing about DL
Despite being vendor talks
it seems to have wide‐ranging apps,
making it mainstream HPC
So how are we going to evensurvive?
2011 ACM Gordon Bell Prize Winner & First Author

Just Putting in a few Deep
Learning Libraries on CustomMade Supercomputers will not
Infrastructural Proliferation…
So how? Well we already know
how to make things go viral onthe Internet!

Background

Deep Learning (DL) A machine learning technique using “Deep” Neural Network
 DL is achieving state-of-the-art in large machine learning area Training DNN with huge dataset requires large scale computation
 eg. 15-layer CNN training takes 8.2 days on 16 nodes (48 GPUs) of TSUBAME2.5
 Researchers have to train DNN for several times to optimize DNN structure and hyperparameters by hand…

Barn swallow = 0.95
Police dog= 0.03
Water beetle = 0.01
Output(Classes)

Imput(Image)(Reference: http://image-net.org/)

Neural Network18

Estimated Compute Resource Requirements for Deep Learning[Source: Preferred Network Japan Inc.]
To complete the learning phase in one day

Image/VideoRecognition
10P（Image) 〜 10E（Video） Flops

100P 〜 1E Flops

学習データ：1億枚の画像 10000クラス分類
数千ノードで6ヶ⽉ [Google 2015]

Image Recognition

10P〜 Flops
1万⼈の5000時間分の⾳声データ
⼈⼯的に⽣成された10万時間の⾳声データを基に学習 [Baidu 2015]

P:Peta
E:Exa
F:Flops

Bio / Healthcare

⼀⼈あたりゲノム解析で約10M個のSNPs
100万⼈で100PFlops、1億⼈で1EFlops

Auto Driving

1E〜100E Flops

⾃動運転⾞１台あたり1⽇ 1TB
10台〜1000台, 100⽇分の⾛⾏データの学習

Robots / Drones

1E〜100E Flops
1台あたり年間1TB
100万台〜1億台から得られたデータで学習する場合

機械学習、深層学習は学習データが大きいほど高精度になる
現在は人が生み出したデータが対象だが、今後は機械が生み出すデータが対象となる各種推定値は1GBの学習データに対して1日で学習するためには1TFlops必要だとして計算

10PF

100PF

2015

2020

1EF

10EF
2025

100EF
2030

Performance Modeling of a Large Scale Asynchronous Deep Learning System under Realistic SGD
Settings – Detailed Performance Modeling and Optimization of Scalable DNN
Yosuke Oyama1, Akihiro Nomura1, Ikuro Sato2, Hiroki Nishimura3, Yukimasa Tamatsu3, and Satoshi Matsuoka1
1Tokyo Institute of Technology 2DENSO IT LABORATORY, INC. 3DENSO CORPORATION[To Appear IEEE Big Data 2016]

Proposal and Evaluation• We propose a empirical performance model for an ASGD
training system on GPU supercomputers, which predicts
CNN computation time and time to sweep entire dataset– Considering “effective mini‐batch size”, time‐averaged mini‐batch size as a criterion for training quality

Our model achieves 8% prediction error for these metrics
in average on a given platform, and steadily choose thefastest configuration on two different supercomputerswhich nearly meets a target effective mini‐batch size

•0.93548 GPUs

30

1 GPU

2520
15
10

Worse than 1 GPU training

50.8
0.7
0.6
0.5
0.4

Model A

0.3

Model B

0.2

0100

200300Epoch400

500

Validation Error of ILSVRC 2012
Classification Task on Two Platforms:
Trained 11 layer CNN with ASGD method

600

01234567891011

Number of samples processed in one itera onMeasured Time (Solid) and Predicted Time (Dashed)of CNN Computation of Three 15‐17 Layer Models

5e+02 sec
1e+03 sec
2e+03 sec
5e+03 sec
1e+04 sec
2e+04 sec
5e+04 sec
1e+05 sec

The best configurationto achieve the shortest
epoch time

Model C

0.10

Number of samples
processed in one iteration
2468
1040
Computa on me [sec]Top‐5 valida on error [%]BetterBackground•
Deep Convolutional Neural Networks (DCNNs) haveachieved stage‐of‐the‐art performance in various
machine learning tasks such as image recognition•Asynchronous Stochastic Gradient Descent (SGD)
method has been proposed to accelerate DNN training– It may cause unrealistic training settings anddegrade recognition accuracy on large scalesystems, due to large non‐trivial mini‐batch size

1020
Number of nodes

3040

Predicted Epoch Time of ILSVRC 2012 Classification Task:
Shaded area indicate the effective mini‐batch size
is in 138±25%

BackgroundMini-batch Size and Staleness Staleness # of updates done within one gradient computation

 Existing researches showed that the error is increased by larger mini-batch size andstaleness There was no way of knowing these statistics in advance
Staleness=0E-ηΣi ∇Ei

W(t)W(t+1)

Twice updates
within gradientcomputation

W(t+3)

W(t+1)
W(t+2)-ηΣi ∇Ei

Staleness=221

Approach and Contribution
 Approach: Proposing a performance model for an ASGD deep learning system

SPRINT which considers probability distribution of mini-batch size and staleness Takes CNN structure and machine specifications as input

 Predicts time to sweep entire dataset (epoch time) and the distribution of the statistics

 Contribution
 Our model predicts epoch time, average mini-batch size and staleness with 5%, 9%, 19%error in average respectively on several supercomputers Our model steadily choose the fastest machine configuration that nearly meets a targetmini-batch size

22

Predicting Statistics of Asynchronous SGD Parameters for a Large‐Scale
Distributed Deep Learning System on GPU Supercomputers
BackgroundProposal• In large‐scale Asynchronous Stochastic Gradient Descent • We propose a empirical performance model for an ASGD
(ASGD), mini‐batch size and gradient staleness tend to be
deep learning system SPRINT which considers probability
large and unpredictable, which increase the error of trained distribution of mini‐batch size and staleness
DNN

Mini‐batch size

100W(t+2)•400

0.4500600

NSubbatch = 110246810

NSubbatch = 110.4

0.10

Predicted

0Staleness=2-ηΣi ∇Ei
DNN parameters space

300

0.0

W(t+1)

200

0.00

W(t+3)

Probability

W(t+1)

Twice asynchronous
updates within
gradient computation

Predicted
Measured

0.00.00-ηΣi ∇Ei

W(t)Staleness=0

NSubbatch = 10.8

Mini‐batch size

0.10

Probability

NSubbatch = 14 nodes
8 nodes
16 nodes

0.8

Objective function E

Staleness

100200300

40050060002468N
Measured(NSubbatch: # of samples per one GPU iteration)
NMinibatch

Staleness

Yosuke Oyama, Akihiro Nomura, Ikuro Sato, Hiroki Nishimura, Yukimasa Tamatsu, and Satoshi Matsuoka, "Predicting Statistics ofAsynchronous SGD Parameters for a Large‐Scale Distributed Deep Learning System on GPU Supercomputers", in proceedings of2016 IEEE International Conference on Big Data (IEEE BigData 2016), Washington D.C., Dec. 5‐8, 2016 (to appear)

10SWoPP2016

16/08/08

Performance Prediction of Future HW for CNN
 以下の2点を⽤いた想定で最適なパラメータを予測 FP16: 半精度浮動⼩数点数を⽤いた計算・データ保持性能の向上
 EDR IB: 4xEDR InfiniBand (12.5GB/s)を⽤いたノード間通信性能の向上→ Not only # of nodes, but also fast interconnect is important for scalability
TSUBAME-KFC/DLでのILSVRC2012データセットの学習における最適なパラメータの予測 (平均ミニバッチサイズ138±25%)
N_NodeN_Subbatch

Epoch時間平均ミニバッチサイズ(現在のHW)

881779165.1

FP167221462

170.1

EDR IB

12111245

166.6

FP16 + EDR IB

8151128171.5

24

Algorithms and Tools for Vector Space Models of Big Data Computational Linguistics
isba...

...

Distributedimplementation
with MPI libraryon

Integration with SLEPC highperformance distributed sparselinear algebra library for
dimensionality reductiontnext symbol

bCorpus: large
collection of texts

Efficient and compact
intra-node data structures
based on ternary trees

Co-occurrence matrix inCompressed Sparse Row
format stored to parallel storage
in hdf5

The whole pipeline orchestrated from Python

Various machinelearning algorithms

http://vsm.blackbird.pw

Publications:zA. Drozd, A. Gladkova, and S. Matsuoka, “Word embeddings, analogies, and machine learning: beyond king - man + woman = queen,” accepted for COLING 2016.
z
A. Gladkova, A. Drozd, and S. Matsuoka, “Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.,” inProceedings of the NAACL-HLT SRW, San Diego, California, June 12-17, 2016, 2016, pp. 47–54.
z
A. Gladkova and A. Drozd, “Intrinsic evaluations of word embeddngs: what can we do better?,” in Proceedings of The 1st Workshop on Evaluating Vector Space
Representations for NLP, Berlin, Germany, 2016, pp. 36–42.
z
A. Drozd, A. Gladkova, and S. Matsuoka, “Discovering Aspectual Classes of Russian Verbs in Untagged Large Corpora”, Proceedings of 2015 IEEE International Conferenceon Data Science and Data Intensive Systems (DSDIS), 2015, pp. 61–68.
z
A. Drozd, A. Gladkova, and S. Matsuoka, “Python, Performance, and Natural Language Processing,” in Proceedings of the 5th Workshop on Python for High-Performanceand Scientific Computing, New York, NY, USA, 2015, p. 1:1–1:10.

GPU‐Based Fast Signal Processing for Large Amounts of Snore Sound Data

• BackgroundSnore sound (SnS) data carry very important information for diagnosis andevaluation of Primary Snoring and Obstructive Sleep Apnea (OSA). With
the increasing number of collected SnS data from subjects, how to handle
such large amount of data is a big challenge. In this study, we utilize theGraphics Processing Unit (GPU) to process a large amount of SnS datacollected from two hospitals in China and Germany to accelerate thefeatures extraction of biomedical signal.

• Acoustic features of SnS data
we extract 11 acoustic features from a large amount of SnS data, which can bevisualized to help doctors and specialists to diagnose, research, and remedy
the diseases efficiently.

Snore sound data information
Subjects

Total Time(hours)

57
187.75(China +Germany)

Data Size(GB)Dataformat

Sampling Rate

31.10

WAV

16 kHz, Mono

Results of GPU and CPU based systems for processing SnS data

• Result
We set 1 CPU (with Python2.7, numpy 1.10.4 and scipy 0.17 packages) forprocessing 1 subject’s data as our baseline. Result show that the GPU based
system is almost 4.6×faster than the CPU implementation. However, thespeed-up decreases when increasing the data size. We think that this result
should be caused by the fact that, the transmission of data is not hidden by othercomputations, as will be a real-world application.

* Jian Guo, Kun Qian, Huijie Xu, Christoph Janott, Bjorn Schuller, Satoshi Matsuoka, “GPU-Based Fast Signal Processing for Large Amounts of Snore Sound Data”, In proceedings of 5th IEEE Global Conference onConsumer Electronics (GCCE 2016), October 11-14, 2016.