TSUBAME Grand Challenge program
Adopted projects on TSUBAME3.0

This program is only chance to use all nodes of TSUBAME3.0 exclusively, because TSUBAME3.0 is shared by thousands of users. There are two categories:
Category A The large scale application aims high peak-performance.
All of TSUBAME3.0 nodes are available.
Category B The large scale application aims scientifically meaningful results.
Table Number of Adopted Projects in the TSUBAME Grand-Challenge Program
2018

2017

2016

2015

2014

2013

2012

2011

Total

A

1223314

7

23

B

21

144202

16

Total

3337734939We started this program since FY2011, and keep on carrying out
twice in each year.
Under this program, we have adopted total 39 fruitful projects, someof which were awarded such as AMC Gordon Bell Prize in SC11.

Projects in Fall 2017
Deep Learning for Fusion Simulation Using
the Fusion Recurrent Neural Network
FRNN (Fusion Recurrent Neural Net) is the deep learning framework developed for fusionenergy applications. It is implementing a distributed data-parallel approach to train deep neuralnetworks (in particular, stacked LSTMs). With this approach, a replica of the model is kept oneach worker, processing different mini-batches of the training dataset in parallel. The model parameters from each worker are collected using MPI, and synchronized via parameter averaging
with learning rate adjusted after each epoch to improve convergence. This produces a global setof parameters, which are then broadcasted to each model replica. The stochastic gradient descent (SGD) method is used for large-scale optimization with parallelization via mini-batch training
to reduce communication costs. In FRNN, we integrate Keras (TensorFlow and Theano backends) and MPI to enable training across multiple GPU nodes using high-speed interconnects.
The starting phase of this proposed 2017 TSUBAME 3.0 Grand Challenge project will be to initiate scaling studies on the powerful TSUBAME 3.0 supercomputer featuring thousands of Pascal
P-100 GPU’ s will begin with the goal of replicating the excellent FRNN scaling observed on“Titan” as well as Pascal P-100 systems at NVIDIA and Princeton University. We look forward toexploring challenges associated with engaging the full capability of TSUBAME
3.0. In particular, since the FRNN software uses the steepest gradient descent(SGD) with mini-batch training to reduce
communication costs, it will be very interesting to observe the behavior of this DLsoftware with scaling studies on TSUBAME 3.0 to examine if the convergence
rate saturates/decreases with increasing
mini-batch size to thousands of P-100
GPU’ s.
Fusion Simulation Using the Fusion Recurrent Neural Network

Realizing Extreme Large-Scale Matrix Computations witha Memory Management Library Utilizing Fast Flash SSDs

The objective of our project is to achieve extremely high-speed and large-scale matrix computation on TSUBAME3.0. Here we focus on large scale Cholesky decomposition, which is an important kernel in solving semi-definite programming (SDP) problem. In order to support even largermatrices than aggregated host memory capacity of available nodes, we harness large capacity of
NVMe SSDs, which is 8 times larger than DRAM capacity in TSUBAME3.0. This should beachieved with less development effort; thus we have developed a global address space runtime library, named vGASNet. On vGASNet, the aggregated capacity of SSDs distributed among compute nodes are virtually visible as a single address space. For high-performance computing, caching mechanism using DRAM is necessary. Moreover, for scalable data movement, it supports “cooperative caching mechanism” to avoid bottleneck in access congestion. On top of vGASNet, we
also have implemented the parallel Cholesky decomposition algorithm. While it is based on datadriven execution and tile-based
dynamic scheduling to suppress
global synchronization, the implementation is relatively simple
and clean owing to the global
address space view. Insideeach task, we use CUBLAS andMAGMA library in order to harness high performance of
NVIDIA P100 GPUs. Taking thisco-design approach, we aim toachieve peta-scale matrix computation.
Global Address Space Model of vGASNetmSMS: A New DSM System for HPC
Hiroko Midorikawa (Seikei University) midori@st.seikei.ac.jp

Key Features

mSMS realizes a highly productive parallel programming environment.
・ Seamless virtual shared memory which is fully accessible by all threads in a cluster.
・ C-compatible address pointer-based programming for shared data is available formulti-node parallel processing.
・ Global shared data are transparently distributed over nodes in a cluster by C-based
programming (MpC). Directive-based API (SMS-Mint) is also available in conjunction
with OpenMP and OpenACC for multi-core & multi-node computing.
・ Remote data preload API is available for sub-array-data prefetch before thecomputing of the sub-array to achieve higher performance in data-parallel comp.
shared double A[NZ][NY][NX]::[NPROCS][1][1] (0, NPROCS); // Global 3D- Arrays, A and B, distributed
shared double B[NZ][NY][NX]::[NPROCS][1][1] (0, NPROCS); // in z-dimension to multiple nodes in a clustermain(){ double (*src)[NY][NX]; double (*dst)[NY][NX]; double (*tmp)[NY][NX]; // pointers to a global 3D-arraysms_startup(&argc, &argv); // mSMS is started up
Ex. ofGlobal 3DNX:// A & B arrays Initialization
NPROCS=4 NY
src = A; dst = B; // set pointers to A and B arrays
Threadfor (t = 0; t < NT; t++) {// time step loopparallel
NodeNode-0:#pragma Mint parallel for // Node parallel processing in a cluster
parallelNode-1:#pragma omp parallel for // Thread parallel processing in one nodeNZ
Node-2Threadfor (z =1; z < NZ-1; z++)Node-3parallelfor (y = 1; y < NY-1; y++)for (x = 1; x < NX-1; x++) {// 7-point Stencil Calculations
dst[z][y][x]=0.4*src[z][y][x]+0.1*(src[z-1][y][x]+src[z+1][y][x]+src[z][y-1][x]+src[z][y+1][x]+src[z][y][x-1]+src[z][y][x+1]);}
sms_sync_drop(); // Execution & memory consistency Sync. (Each node discards or reﬂects cached pages)
tmp = dst; dst = src; src = tmp;// switch A and B arrays ( swap src & dst pointers )} // time step loop end
sms_shutdown(); // mSMS is ﬁnalized
The mSMS program for 7-point stencil comp.
}・ High performance: mSMS incorporates dedicated sms-system-threads for efficient
communication to achieve the comparable performance of MPI programs.

Results
・ mSMS realized
30TiB-addressspace and 23TiBglobal-arrays byusing 180 nodes(9900 threads) inTsubame3.0.
・ It achieved MPI
comparable perf.

10000 mSMS vs. MPI in Performance (GFlops)
9007
9007
Simple 27p stencil computation
9000
8023 8023
6436
8000
6934
6436
MPI
7000
5890
4880 4880
SMS segv 16M-page
6000
3658
4632
3658
3276
5000
SMS preload
3565
3276
4000
3209
1656 1669
3000
839 860 1656217
4272000 113
438 839218
114
4271000 113
217
0GFlopsTSUBAME Grand Chalenge Summary

248

16326472
Num of nodes

96128

160

180

Training ImageNet on 2048 GPUs
Rio Yokota, Kazuki Osawa, Hiroki Naganuma, Hiroyuki Otomo,Mohamed Wahib, Alexandr Drozd, Yosuke Oyama, Yohei Tsuji,Keita Yashima, Hitoshi Sato (Tokyo Institute of Technology)

Second Order Optimization for Deep Learning
As deep neural networks increase in size
the amount of data and time to train
them become prohibitively large tohandle on a single compute node. Distributed deep learning on thousands ofGPUs forces the batch stochastic descent methods to operate in a regime
where the increasing batch size starts tohave detrimental effect on the convergence and generalization. We investigate the possibility of using second orderoptimization methods with proper regularization as an alternative to conventional stochastic gradient decent methods.

Hybrid data and model parallelism

Second order optimization methods require the communication of the Hessianmatrix. The present method reduces thecommunication significantly through Kronecker factorization and the use of reduce-scatter and allgather collectives toswtich between data parallel and modelparallel execution. The optimal implementation of allreduce operations has thesame communication pattern, so the present method simply inserts the computationof the Hessian between the two collectiveoperations.

http://www.gsic.titech.ac.jp/sc18