Applications on TSUBAME3.0
Fast Algorithms and Large Scale CFD
Fast Algorithms for HPC and Deep Learning
PresenThu 4 tation:Room 00pm
C140-142
Structured low-rank

Structured Low-Rank Matrices

matrices can reduce the complexityof dense matrix multiplication andfactorization from O(N3) to O(N).
There are many variants ofstructured low-rank matrices, andwe are investigating the trade-offs
between computational work,
parallelism, communication, load
imbalance to achieve the bestperformance. We have developedan O(N) factorization method thathas no dependency on trailingsubmatrices. We have a technicalpaper on Thursday that describes
this approach.

Memory Efficient Deep Learning

Error Correction for TensorCores

PosterAvaila
C1-2-3 ble

TensorCores multiply two 16bit
matrices and accumulate into a
32 bit matrix. This conversion to
16bit results in a significant loss
in precision. The use of anauxiliary 16bit matrix to store
the mantissa loss is known topartially correct this error.
However, the round-to-zero inTensorCore accumulation isanother source of error that has
not been accounted for inprevious work. We develop a novel method that also corrects for thiserror, which allows us to exactly match an GEMM in single precision,
while exploiting the compute capability of TensorCores.

Training ImageNet in 2 minutes on 2048 GPUs

The size of deep neural networks
Data-parallel training of deep
continues to grow rapidly, whereneural networks suffers from themodels like GPT-3 andlarge-batch problem, where theMegatron-LM exceed the memory
generalization gap increases as
capacity of a single GPU. The trend
the mini-batch size increases
in computer architecture where theproportional to the number ofarithmetic throughput is growing
GPUs. Using the Kronecker
faster than memory capacity,
Factorization mention on the right,
suggests that memory consumption
we are able to use a second order
is a critical issue in deep learning.
optimization method with minimum
We reduce the memory usage ofoverhead, which allows us to retain the convergence even for extremely
deep neural networks by scattering, large batch sizes. We trained ImageNet in 2 minutes on 2048 GPUs,
recomputing, and offloading theusing a batch size of 131,072.
model parameters and activations.

Large-scale Mesh-based and Particle-based Simulations
A Large-scale Foam Simulation
using a Multi-phase Field LBM and AMR

Large-scale Granular Simulation

AMR for Multi-phase Flows

Dynamic Load Balancing using A Space-filling
Curve

Discrete element method(DEM) is often used toFoam formation withsimulate granulara stable thin liquid
dynamics and its simple
film is very difficult toalgorithm with the contact
simulate using
interaction is suitable forconventionalGPU computing.
methods due to theHowever so many
limitation of meshparticles are included andresolution that canthe particle distributions
be used. Wearechanging
intimeandaddressed this issue by using a Multi-phase Field Lattice Boltzmann
space.
Adynamicdomain
Method and Adaptive Mesh Refinement. Using the Multi-Phase Field
LBM, we can prevent a “numerical coalesce” phenomenon that leads to decomposition has to be introduced for multiple-node computing. In abunker shot, the sand wedge does not hit to the golf ball directly andbubble break-up. Adaptive Mesh Refinement has been introduced sotransferring the force through the sand to the ball in order to reduce thethat the thin liquid film can be simulated efficiently. Herein, weimpact. In this simulation, 16.7 millions of DEM particles are used todemonstrate a simulation of foam formed from 200 air bubbles using
represent the dynamics of the sands with 256 GPUs.
our proposed method.
Simulations for multi-phase flows
require high-resolution grids tocapture phenomena at the interface.
By using the adaptive meshrefinement (AMR) method, which
dynamically adapts high-resolution
grids to interfaces, computational costand memory usage are reduced. Thespatial distribution of a computationalload change in time; therefore,
dynamic domain partitioning using aspace-filling curve is introduced for
multi-GPU computing to assign anequal number of grid points to eachGPU. The figures show thelarge-scale free-surface flow simulations for the dam-breaking process
and corresponding domain decomposition for 64 GPUs.

For large-scaleparticle-based
simulation and Adaptive
Mesh Refinement(AMR), it is a criticalissue to achieve
computational loadbalance and equal
memory usage on
multiple computenodes. A domain
partitioning in terms of a space-filling curve(SFC) is one of promising
candidates and it is recognized that a 1-dimensional mapping of3-dimensional space by cutting the equal length. Due to low cost of
SFC domain partitioning, it is suitable for frequent re-partitioning in thesimulations of unsteady phenomena.

A Hilbert Curve

https://www.gsic.titech.ac.jp/sc22