JST CREST Project on
Scalable Deep Learning
Scalable Deep Learning for Large Image and Video Data
About the project

The overreaching aim of the project is to establish a high-performance real-time deep learning algorithm basis for detecting
objects and anomalies from a large amount of high definition
videos recorded by drive recorders, surveillance cameras,
and the like. Computer science researchers specializing infour different levels from architectures to applications including GPU fast computation, parallel computation, machinelearning, and data mining, collaborate to realize 1000 times
faster processing with 0.1% the amount of memory compared
to conventional platforms.

The Case for Strong Scaling in Deep Learning:
Training Large 3D CNNs with Hybrid Parallelism

We present a newcapability for spatiallypartitioning the trainingof 3D convolutional
neural networks. Our
approach enables
training on very large data cubes that would otherwise beinfeasible due to memory limitations. Due to the huge size of thedata, we also develop a new data ingestion pipeline thatleverages
parallelI/O
as
well
as
anin-memory
distributeddata
Creating Deep Learning Software Ecosystemstore in order to reduce I/O overheads. We study the impact offor Fugaku
data size and present comprehensive performance and scaling
Fugaku has 6 TFLOPS
results of two different huge 3D CNNs, the CosmoFlow network
FP32 performance andand 3D U-Net, including training a single model with up to 2048
1TB/s memory bandV100 GPUs. Furthermore, we demonstrate an order-ofwidth per chip. Developmagnitude improvement in the prediction quality of theing a deep learning softCosmoFlow network.
ware stack on this architecture requires porting
DNNLtosupport
Fully-Attentional System For Noisy Speech
SVE/A64FX optimizaFor the front-end (beforetion with some Fugaku-specific optimizations like memory speech recognition), use a fully
management subsystems.
attentional network for theimprovement of speech
Understanding Deep Learning Performance diarization and speech
separation. This in turn canBenchmarking
lead to better speechmachine learningenhancement from using
algorithms and thespeaker IDs. We represent ourhardware it runs on
task of diarization in a way
is a non-trivial task.
which utilizes axial- attention. We can also try various numbers ofDifferent models favor different hardware (e.g. transformer
concurrent windows of MFCC spectrograms (previous, current,models are highly GPU organized, while smaller kernels tendand lookahead) on which the attention model will focus.
to underutilize GPU parallelism). For this reason, detailed
Additionally, since speech networks often use 1D CNNs, we canbench- marking is needed. We measured different models,
combine those vectors into a larger 2D vector.
examining the per- formance of their parts (transformers,convolutional layers, dense layers etc.) and compared themacross different hardware (TPU and A64FX measurements
A Systematic Performance Analysis of
are in progress).

Second-order Optimization using K-FAC

We perform second order
Fast Approximations of Betweenness
optimization on deep neuralCentrality with Graph Neural Networks
networks using Kronecker
Major key points of ourfactored approximatefeature aggregation
curvature (K-FAC). Various
schemes are: 1) Feature
performance optimizations
aggregation is done
were performedd including theseparately for incoming
use of fp16 gradients and fp21
and outgoing paths. 2)
Fisher matrices, hierarchicalNode’ s own features are not aggregated. Hence at each layer ring AllReduce communication,
node gets unique feature information corresponding toand the use of stale Fisher
neighborhood structure. 3)Nodes with no shortest paths arematrices. We achieved 2
identified and corresponding rows in A and AT are set to zero. minutes training of ResNet-50 / ImageNet-1K with 2048 GPUs
by Distributed K-FAC.
This helps features flow along possible shortest paths.

http://www.gsic.titech.ac.jp/sc20