TSUBAME Grand Challenge program
Adopted projects in the spring 2017 program
TSUBAME Grand Challenge Summary
This program is only chance to use all nodes of TSUBAME2.5/3.0 exclusively, because TSUBAME2.5/3.0 is shared by thousands of users. There are two categories:
Category A The large scale application aims high peak-performance.
All of TSUBAME2.5/3.0 nodes are available.
Category B The large scale application aims scientifically meaningful results.
A large portion (1/3) of TSUBAME2.5 is available.
Table Number of Adopted Projects in the TSUBAME Grand-Challenge Program
FY2017Category

FY2016FY2013
FY2014 FY2012
FY2014 FY2015FY2013

FY2011
Total
Fall Spr. FallFall
Spr.Spr.
Fall Spr. Fall
Fall Spr.
Spr.
Fall Fall
Spr. Spr.Fall
FallSpr. Fall Spr.

We started this program since FY2011, and keep on carrying out
twice in each year.
Under this program, we have adopted total 36 fruitful projects, someof which were awarded prizes as below.

FY2012FY2011

TotalA

201 112 1201122 0 2 13 2 42318422B010 212 1312120 1 0 120 -0

212-14Total211 324 25132

42 1 2 25 2 42530436

2011 ACM Gordon Bell Prize: Special Achievements
in Scalability and Time-to-Solution“Peta-scale Phase-Field Simulation for Dendritic
Solidification on the TSUBAME 2.0 Supercomputer”

2011 ACM Gordon Bell Prize:
Honorable Mention
Large scale bioﬂuidics
simulations on TSUBAME2

2011 Graph500 Challenge
on TSUBAME 2.0

The First Adopted Projects on TSUBAME3.0 in Fall 2017
Deep Learning for Fusion Simulation Using
the Fusion Recurrent Neural Network

Realizing Extreme Large-Scale Matrix Computations witha Memory Management Library Utilizing Fast Flash SSDs

The objective of our project is to achieve extremely high-speed and large-scale matrix computaFRNN (Fusion Recurrent Neural Net) is the deep learning framework developed for fusion energy applications. It is implementing a distributed data-parallel approach to train deep neural networks (in par- tion on TSUBAME3.0. Here we focus on large scale Cholesky decomposition, which is an importticular, stacked LSTMs). With this approach, a replica of the model is kept on each worker, processing ant kernel in solving semi-definite programming (SDP) problem. In order to support even larger
different mini-batches of the training dataset in parallel. The model parameters from each worker are matrices than aggregated host memory capacity of available nodes, we harness large capacity ofcollected using MPI, and synchronized via parameter averaging with learning rate adjusted after each NVMe SSDs, which is 8 times larger than DRAM capacity in TSUBAME3.0. This should be
epoch to improve convergence. This produces a global set of parameters, which are then broadcasted achieved with less development effort; thus we have developed a global address space runtime lito each model replica. The stochastic gradient descent (SGD) method is used for large-scale optimiza- brary, named vGASNet. On vGASNet, the aggregated capacity of SSDs distributed among comtion with parallelization via mini-batch training to reduce communication costs. In FRNN, we integrate pute nodes are virtually visible as a single address space. For high-performance computing, cachKeras (TensorFlow and Theano backends) and MPI to enable training across multiple GPU nodes ing mechanism using DRAM is necessary. Moreover, for scalable data movement, it supports “cooperative caching mechanism” to avoid bottleneck in access congestion. On top of vGASNet, weusing high-speed interconnects.
The starting phase of this proposed 2017 TSUBAME 3.0 Grand Challenge project will be to initiate also have implemented the parallel Cholesky decomposition algorithm. While it is based on datascaling studies on the powerful TSUBAME 3.0 supercomputer featuring thousands of Pascal P-100 driven execution and tile-based
GPU’s will begin with the goal of replicating the excellent FRNN scaling observed on “Titan” as well as dynamic scheduling to suppress
Pascal P-100 systems at NVIDIA and Princeton University. We look forward to exploring challenges
global synchronization, the imassociated with engaging the full capability
plementation is relatively simpleof TSUBAME 3.0. In particular, since theand clean owing to the global
FRNN software uses the steepest gradientaddress space view. Inside eachdescent (SGD) with mini-batch training totask, we use CUBLAS andreduce communication costs, it will be very
MAGMA library in order to harinteresting to observe the behavior of thisness high performance ofDL software with scaling studies on TSUBNVIDIA P100 GPUs. Taking thisAME 3.0 to examine if the convergence rateco-design approach, we aim tosaturates/decreases
withincreasingachieve peta-scale matrix commini-batch size to thousands of P-100putation.
Global Address Space Model of vGASNet
GPU’s.
Fusion Simulation Using the Fusion Recurrent Neural Network

DC-DFTB-metadynamics for primary proton transfer of bacteriorhodopsin
Minori Imai1, Junichi Ono1, Yoshifumi Nishimura2, Hiromi Nakai1-4(1Grad. Sch. of Adv. Sci. & Eng., Waseda Univ., 2RISE, Waseda Univ., 3JST-CREST, 4ESICB, Kyoto Univ.)

Bacteriorhodopsin: light-driven proton pumpMultiple pathways of primary proton transfer

Bacteriorhodopsin (BR) is one of the most famous photoreceptive
proteins, which actively translocates proton across the membrane withthe aid of light (Fig. 1a) [1]. The resulting proton gradient can produce
ATP, i.e., energy resources in biological systems. Thus, elucidation ofmicroscopic mechanism of the proton transfers in BR is important forunderstanding the biological function of the light-energy conversion.

In 2016, successive snapshots of BR along the photocycle captured
with X-ray free electron laser are reported (Fig. 1b) [2]. Due to thelimitation of the experiment, however, only heavy-atom positions aredetermined. Therefore, no proton transfer has been observed, and thepathway and the timing of the proton transfers have still been unclear.

(a)

(a)

Path A(b)

Path B(b)
W452

Figure 2. Multiple pathways of theprimary proton transfer from theprotonated Schiff base (PSB) toAsp85 on the photocycle of BR:(a) path A via water (W452) andThr89, and (b) path B only via Thr89.

W452

H+H+Thr89

Thr89

PSB

PSB

Asp85

Asp85

W400
W401

Figure 1. (a) Proton transfers of BR and (b) heavy-atom snapshots of BR captured
with X-ray free electron laser along the photocycle [2].
[1] O. P. Ernst, et al., Chem. Rev., 114, 126 (2014).
[2] E. Nango et al., Science, 345, 1552 (2016).

W401
W400

In the present study, divide-and-conquer density-functional tightbinding (DC-DFTB) metadynamics simulations were performed for theprimary proton transfer, starting from the snapshots. It is found thatthe proton transer can occur via two pathways: path A and B(Fig. 2). Moreover, the former two L-state structures (760 ns & 2 μsin Fig. 1b) are found to be relevant for the primary proton transfer.

http://www.gsic.titech.ac.jp/sc17