Applications on TSUBAME3.0
Fast Algorithms and Large Scale CFD
Fast Algorithms for HPC and Deep Learning
Structured Low-Rank Matrices

Error Correction for TensorCores

Structured low-rank matrices canTensorCores multiply two 16bit
reduce the complexity of dense
matrices and accumulate into amatrix multiplication and32 bit matrix. This conversion tofactorization from O(N3) to O(N).
16bit results in a significant loss
There are many variants ofin precision. The use of an
structured low-rank matrices, andauxiliary 16bit matrix to store
we are investigating the trade-offs
the mantissa loss is know tobetween computational work,
partially correct this error.
parallelism, communication, load
However, the round-to-zero inimbalance to achieve the best
TensorCore accumulation isperformance. There are very few
another source of error that has
high performance implementations
not been accounted for inof these type of matrices, whichprevious work. We develop a novel method that also corrects for thisresult in batched, small,error, which allows us to exactly match an GEMM in single precision,
rectangular matrix operations.
while exploiting the compute capability of TensorCores.

Memory Efficient Deep Learning

Training ImageNet in 2 minutes on 2048 GPUs

The size of deep neural networks
Data-parallel training of deep
continues to grow rapidly, whereneural networks suffers from themodels like GPT-3 andlarge-batch problem, where theMegatron-LM exceed the memory
generalization gap increases as
capacity of a single GPU. The trend
the mini-batch size increases
in computer architecture where theproportional to the number ofarithmetic throughput is growingGPUs. Using the Kronecker
faster than memory capacity,
Factorization mention on the right,
suggests that memory consumption
we are able to use a second order
is a critical issue in deep learning.
optimization method with minimum
We reduce the memory usage ofoverhead, which allows us to retain the convergence even for extremely
deep neural networks by scattering, large batch sizes. We trained ImageNet in 2 minutes on 2048 GPUs,
recomputing, and offloading theusing a batch size of 131,072.
model parameters and activations.

Large-scale Mesh-based and Particle-based Simulations
A Large-scale Foam Simulation
using a Multi-phase Field LBM and AMR

Large-scale Granular Simulation

AMR for Multi-phase Flows

Dynamic Load Balancing using A Space-filling
Curve

Discrete element method(DEM) is often used toFoam formation withsimulate granulara stable thin liquid
dynamics and its simple
film is very difficult toalgorithm with the contact
simulate using
interaction is suitable forconventionalGPU computing.
methods due to theHowever so many
limitation of meshparticles are included andresolution that canthe particle distributions
be used. We
arechanging
intimeandaddressed this issue by using a Multi-phase Field Lattice Boltzmann
space.
Adynamicdomain
Method and Adaptive Mesh Refinement. Using the Multi-Phase Field
LBM, we can prevent a “numerical coalesce” phenomenon that leads to decomposition has to be introduced for multiple-node computing. In a
bunker shot, the sand wedge does not hit to the golf ball directly andbubble break-up. Adaptive Mesh Refinement has been introduced sotransferring the force through the sand to the ball in order to reduce thethat the thin liquid film can be simulated efficiently. Herein, weimpact. In this simulation, 16.7 millions of DEM particles are used todemonstrate a simulation of foam formed from 200 air bubbles using
represent the dynamics of the sands with 256 GPUs.
our proposed method.
Simulations for multi-phase flows
require high-resolution grids tocapture phenomena at the interface.
By using the adaptive meshrefinement (AMR) method, which
dynamically adapts high-resolution
grids to interfaces, computational costand memory usage are educed. The
spatial distribution of a computationalload change in time; therefore,
dynamic domain partitioning using aspace-filling curve is introduced for
multi-GPU computing to assign anequal number of grid points to eachGPU. The figures show thelarge-scale free-surface flow simulations for the dam-breaking process
and corresponding domain decomposition for 64 GPUs.

For large-scaleparticle-based
simulation and Adaptive
Mesh Refinement(AMR), it is a criticalissue to achieve
computational loadbalance and equal
memory usage on
multiple computenodes. A domain
partitioning in terms of a space-filling curve(SFC) is one of promising
candidates and it is recognized that a 1-dimensional mapping of
3-dimensional space by cutting the equal length. Due to low cost of
SFC domain partitioning, it is suitable for frequent re-partitioning in thesimulations of unsteady phenomena.

A Hilbert Curve

https://www.gsic.titech.ac.jp/sc21