TSUBAME Grand Challenge program
Adopted projects in the spring 2016 program
TSUBAME Grand Chalenge Summary
This program is only chance to use all nodes of TSUBAME2.5 exclusively,
because TSUBAME2.5 is shared by thousands of users. There are two categories:
Category A The large scale application aims high peak-performance.
All of TSUBAME2.5 nodes are available.
Category B The large scale application aims scientifically meaningful results.
A large portion (1/3) of TSUBAME2.5 is available.

We started this program since FY2011, and keep on carrying out
twice in each year.
Under this program, we have adopted total 30 fruitful projects,
some of which were awarded prizes as below.

Table Number of Adopted Projects in the TSUBAME Grand-Challenge Program

FY2016FY2015FY2014FY2014FY2013

FY2013
FY2012FY2012FY2011

Fall Spr.

Fall Spr.
Spr.
Fall

Fall
Fall Spr.
Spr.

Fall Spr.
FallA

1111002B

0121232121

101020-0

122-12Total

1232

45

3142

122252 4230
5433

Category

12221

12

Fall
FallSpr.

FY2011
Total
Fall Spr.

Total

32 421832042011 ACM Gordon Bell Prize: Special Achievements
in Scalability and Time-to-Solution“Peta-scale Phase-Field Simulation for Dendritic
Solidifcation on the TSUBAME 2.0 Supercomputer”

2011 ACM Gordon Bell Prize:
Honorable Mention
Large scale bioﬂuidics
simulations on TSUBAME2

2011 Graph500 Challenge
on TSUBAME 2.0

Daino: A High-level Framework for Parallel and Efficient AMR on GPUs
Mohamed Wahib (AICS, RIKEN)

Motivation
Adaptive Mesh Refinement (AMR) is a model for reducing
computation by adapting the resolution of a stencil meshes locally.
However, producing efficient AMR code is hard, especially for GPUs.
As a result, AMR frameworks require the user to write his own
optimized code for the target architecture.
Our approach to achieve target- A compiler-based framework for producing efficient AMR code (forGPUs)- Architecture-independent interface provided to the user
- A performance model for quantifying the efficiency
Key results
Our framework generates code comparable in speedup andscalability to hand-written optimized GPU AMR implementations using
up to 3,600 GPUs.
GPUCPU

Copy Initial Arrays

1. Initialize
2. Exchange Ghost Layers

CPU MemoryLoop

3. Compute StencilOctree(AMR Metadata)

GPU0 Memory

InvokeInvokeStencil Kernel
Correction Kernel

5. Evaluate Error

Decision Kernel

Copy
Invoke<Refine>Consolidate

InvokeApplications- Hydrodynamics Solver: We model a hydrodynamics application
using Euler equations extending the GAMER implementation.
- Shallow-water Solver: We model shallow water simulations bydepth-averaging the Navier–Stokes equations.
- Phase-field Simulation: We evaluate an AMR version of aphase-field simulation for modeling 3D dendritic growth.

Figure Weak scaling of uniform mesh, hand-written and automated AMR (GAMER-generated AMR included in hydrodynamic)

Copy Ghost Layers

4. Post-Stencil (Correction)

InvokeCome see our paper at SC16: Wed. 16th 16:30-17:00 at 355-D (Best Paper Finalist)

Refine Kernel
Consolidate Kernel

Octants(Data Arrays)
GPU1 Memory

Octants(Data Arrays)
GPU2 Memory

6. Update & Balance Octree
7. Finalize

Copy Final Arrays

Octants(Data Arrays)

Figure Conceptual overview of the proposed model. All operations touching the data arrays are done by GPU kernels.
Accordingly, CPU specializes in operations applied on the octree while GPU specializes in operations applied on the data arrays.
Figure Strong scaling of uniform mesh, hand-written and automated AMR (GAMER-generated AMR included in hydrodynamic)

Particle-In-Cell Simulation of Fusion Plasmas
Gyrokinetic Particle-In-Cell Simulation
The Gyrokinetic Toroidal Code at
Princeton (GTC-P) is a highly scalable and portable particle-in- cell(PIC) code. It solves the 5D
Vlasov-Poisson equation featuring efficient utilization of modern parallel
computer architectures at the petascale and beyond. Motivated by thegoal of developing a modern code capable of dealing with the physics challenge of increasing problem size with
sufficient resolution, new thread-level
optimizations have been introduced
as well as a key additional domain decomposition. GTC-P’ s multiple levelsof parallelism, including inter-node 2D
domain decomposition and particle
decomposition, as well as intra-node shared memory partition and vectorization have enabled pushing the scalability of the PIC method toextreme computational scales. New discovery science capabilities in
the magnetic fusion energy application domain are enabled, includinginvestigations of Ion-Temperature-Gradient (ITG) driven turbulencesimulations with unprecedented spatial resolution and long temporal
duration.

Rio YOKOTA (GSIC, Tokyo Tech)

Comparison with Other GPU Supercomputers
The TSUBAME GrandChallenge Category A wasused to perform a full systemrun. GTC-P has been benchmarked on other leadershipclass GPU systems such asPiz Daint (CSCS) and Titan(ORNL). These performancecomparisons using a realistic
discovery- science-capable
domain application code provide valuable insights on optimization techniques across one of the broadest sets of current
high-end computing platforms worldwide. A strong scalability plot isshown above for 256, 512, and 1024 MPI processes. Piz Daint is thenewest machine while Titan is slightly older and TSUBAME2.5 is justabout to be upgraded to TSUBAME3.0 so the difference in the absolute performance is somewhat expected. Though, we are currently
investigating exactly what is causing the large difference in the performance between these machines. These investigations beneficial
to not only the current cross-machine studies, but also to the eventual deployment of GTC-P as a meaningful science-capable code on
TSUBAME3.0.

http://www.gsic.titech.ac.jp/sc16