AI-for-Scienceを加速しその先へと導く初のエ
クサスケールスーパーコンピューター(The First Exascale Supercomputer Accelerating
AI-for-Science and Beyond)モハマド ワヒブ (Mohamed WAHIB)
理研 計算科学研究センター (RIKEN-CCS)
⾼性能⼈⼯知能システム研究チームリーダー

27-10-2022
HPCIコンソーシアム

PreludeØ AI/ML is not magicØImportant to distinguish hype from real prospectØ There is prospect for AI/ML in HPC

Ø Optimizing
Ø ScalingAI

HPC
Ø Improving
Ø Automating

2

Fugaku Supercomputer
Kobe
Keisan Kagaku Center StationTokyo

RIKEN R-CCS

423 km (263 miles)
west of Tokyo
Research
Building

Chillers
Substation Supply

Computerbuilding

Computer room 50 m x 60 m = 3,000 m2
Electric power up to 37 MW
Water cooling system
Gas-turbine co-generation 5 MW x 2
3

Fugaku Supercomputer4

Fugaku Research Highlights (FY21)
Achieved world's first five titles(4 consecutive terms + new ML Perf HPC 1st place)

Gordon Bell Special PrizeFight against COVID-19

In four HPC performance rankings (Top500, HPCG, HPLAI, Graph500), Fugaku won four titles consecutively from
June 2020. In November 2021, also awarded first place in
ML Perf HPC, a new overall performance evaluation of AI
processing.
Fugaku's high overall
performance in a wide range offields, as well as its ability tomake a significant contribution
to the realization of Society
5.0/SDGs.

Successfully developed a detailed and quantitative COVID19 droplet and aerosol dispersion model using "Fugaku"
for the first time in the digital transformation of infectious
disease epidemiology. Visualizing arised awareness of
the importance of understanding droplet and aerosol
infection changing behaviour not only in Japan, but also
around the world.
ITU-AJ Special
Achievement Award

Data Assimilation Research“GENESIS” new version released(Prediction of Sudden downpours, COVID-19 infection)

Using big data from weather radar, a real-time of ultra-fast
precipitation forecasting, was conducted in the Tokyo area
using Fugaku during the Tokyo Olympic and Paralympic
Games.
Data assimilation methods developed innumerical weather forecasting wereapplied to the forecasting of COVID-19
infections

Molecular dynamics (MD) simulations

A new version of GENESIS, optimized for 'Fugaku' by co-design,
more than 125 times faster and with many new features,has been released as free software in 2020. Work on the
dynamic
structure of spike proteins on thesurface of COVID-19 has analized
successfully. RIKEN EIHO Award(RIKEN Significant Achievement
Award)

5

AI on Fugaku
Ø World 2nd fastest Supercomputer (Top500: June 2022)Ø

1st on HPCG and Graph500

Ø Fujitsu A64FX Arm v8.2-A +SVE CPUs (32GB HBM memory)Ø Creating DL ecosystem for

Fugaku:https://github.com/dl4fugaku
6

Scaling AI/ML on Fugaku§§

Past and on-going effort on scaling ML on Fugaku goes into benchmarks
MLPerf (https://mlcommons.org/en/)§Subcategory: MLPerf Training HPC

7

Scaling AI/ML on Fugaku

Farrell et al, “MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems“, WS col-located with SC’218

AI-for-Science on Fugaku*by R-CCS Research Teams & Priority application programs(collected by Kento Sato)* To Appear: Book Chapter “The First Exascale Supercomputer Accelerating AI-for- Science and Beyond ”
Satoshi Matsuoka, Mohamed Wahib, Aleksandr Drozd, Kento Sato9

R-CCS

Nonlinear Dimensionality Reduction for Three-dimensional Flow Field Around Circular Cylinder
with Distributed Parallel Machine Learning on Fugaku (Kazuto Ando et al. [1])lMode decomposition is an approach used to decompose flow fields into physically important flow structures
known as modes, POD (Proper Orthogonal Decomposition) is a powerful methodl

MD-CNN-AE [1] extended to support 3-D flow
MD-CNN-AE [2]: A neural network for a mode decomposition method for 2-D flow field around circularcylinder using a convolutional neural networkll

A hybrid parallelism method combining the distribution of network structure (model parallelism) and the training
data (data parallelism) using up to 10,500 nodes on Fugaku was employed for learning.
Schematics of 3D-extended version of MD-CNN-AEℱ!"#$ 𝑥 = 𝝋$𝑥

Unﬂa6en+

Decoder(CNN)・・・

Unflatten

・・・

𝝋$% 𝒙
⋮
ℱ!"# 𝒙 =
%
𝝋& 𝒙

FCN

・・・

Encoder(CNN)

Output

・・・

・・・

#

3D flow fieldsnapshots

・・・

Unflatten

・・・

Flatten

Input

"

・・・

!

・・・

3D flow fieldsnapshots

・・・

LatentvectorFull-order model!
"#

Streamwis
e velocity𝒖on X-Y pla
ne

POD (26 modes)

Neural network (26 modes)Verticalvelocity 𝒗
on X-Y planeℱ!"#' 𝑥 = 𝝋% 𝑥

Spanwisevelocity 𝒘
on X-Z plane[1] Ando, K., Onishi, K., Bale, R., Tsubokura, M., Kuroda, A., Minami, K. (2021). Nonlinear Mode Decomposition and Reduced-Order Modeling for Three-Dimensional Cylinder Flow by D
istributed Learning on Fugaku. In: Jagode, H., Anzt, H., Ltaief, H., Luszczek, P. (eds) High Performance Computing. ISC High Performance 2021. Lecture Notes in Computer Science(),vol 12761. Springer, Cham. https://doi.org/10.1007/978-3-030-90539-2_8[2] T. Murata, K. Fukami, and K. Fukagata, “Nonlinear mode decomposition with convolutional neural networks for fluid dynamics,” J. Fluid Mech., vol. 882, A13., 2020, doi:10.1017/jfm.

10

Priority Application

Development of NN for High-resolution, Real-Time Tsunami Flood Prediction(Fumihiko Imamura group [1])ll

Tsunami simulations to generate training datalTraining Input data: Tsunami waveform in offshore areaslTraining Output data: Flooding conditions in coastal areas

Training an AI model to predict flooding condition in coastal areas from Tsunami wave format in offshoreà This approach makes it possible to accurately and rapidly obtain detailed flooding forecast before landfall of Tsunami

Fig. 1 Overview of tsunami prediction with AI

Fig 2. Comparison between anticipated flooding (tsunami source model created by
Cabinet Office of Japan with tripled wave heights) of Nankai Trough Megathrust
Earthquake and prediction results of newly developed AI[1] (Press release) International Research Institute of Disaster Science, Tohoku University, Earthquake Research Institute, The University of Tokyo, Fujit
su Laboratories Ltd.Fujitsu Leverages World’s Fastest Supercomputer ‘Fugaku’ and AI to Deliver Real-Time Tsunami Prediction in Joint Project11

Priority Application

Elucidation of the Cause and Diversity of Cancer using Large-scale Data Analysis and AI Tech. [1]lEven if a patient receives a targeted cancer drug therapy, the appearance of drug-resistant cancer cells
represents an ongoing threat to full remissionll

Fujitsu implemented parallel conditional and causal algorithmslll

The mechanism for how certain cancers become drug resistant remains unclear
Utilizing Fujitsuʼs “Wide Learning” AI technology to extract combinations of potential genes relating to
the emergence of drug resistance based on statistical information
Also, maximizing computational performance with the supercomputer Fugaku

Fugaku, Fujitsu and TMDU were able to search the entire human genome for conditions and causality
within a single day and determine the genes that cause resistanceto drugs used to treat lung cancer[1] (Press release) Fujitsu Limited, Tokyo Medical and Dental University,” Fujitsu and Tokyo Medical and Dental University leverage world’s fastest supercomputer and AI technology for scientific discovery to shed light on drug resistance in cancer treatment, March 7, 2022 (https://www.fujitsu.com/global/
about/resources/news/press-releases/2022/0307-01.html)

12Priority Application

Forecasting the Expansion of Supernova Shells Using Deep Learning toward High-Resolution
Galaxy Simulations (Kaiya Hirashima et al. [1])lGoal: achieve the highest-resolution galaxy simulation which can resolve individual stars
on the supercomputer Fugaku.

l

One of the bottlenecks is the calculation including small scall phenomena (e.g.,supernovae)lJust using massively parallel computers cannot solve it.

lAnd algorithm that uses deep learning to resolve the bottlenecks.

BAD

à The deep learning model to forecast the expansion of supernova shells
GOOD

v They applied the framework for video predictions toforecasting the results of simulations• Prediction of spatial features: CNN• Prediction of temporal changes: LSTM
v Training data
• Input: 1 image, Density distribution just before theexplosion• Output: 19 images, Spatiotemporal changes in density
after the explosion
v Training & Prediction• The training takes several days using an NVIDIA GeForce
RTX 3090 GPU.
• The estimation itself takes only a few second.

Strong Scaling of GADGET-4(Based on [5] Figure 63)[1] K. Hirashima and K. Moriwaki and M. S. Fujii and Y. Hirai and T. Saitoh and J. Makin, “Predicting the Expansion of Supernova Shells for Hig
h-Resolution Galaxy Simulations Using Deep Learning”, Journal of Physics: Conference Series, March, 2022
[2] Springel et al. (2021)

13

Other AI-based Scientific Achievements(Fugaku is not used)by R-CCS research team & Priority application programs

14

R-CCS

Remote Sensing by Deep Learning From Simulated Data for Flood and Debris-Flow
Mapping for flood disaster damage estimation (Naoto Yokoya et al., 2020 [1])This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

l

l

l

Yokoyama et al. proposed a framework that estimates the inundation depth (maximum water level) anddebris-flow-induced topographic deformationl

From remote sensing imagery by integrating deep learning and numerical simulation.

l

A water and debris-flow simulator generates training data for various artificial disaster
scenarios.

10

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

They showed that regression models based on Attention U-Net and LinkNet architectures trained on such
synthetic data can predict the maximum water level and topographic deformation from a remote sensingderived change detection
anda digital
This article hasmap
been accepted
for inclusion
in a futureelevation
issue of this journal.model
Content is final as presented, with the exception of pagination.
YOKOYA et al.: BREAKING LIMITS OF REMOTE SENSING BY DEEP LEARNING FROM SIMULATED DATA
The proposed
framework has an inpainting capability, thus mitigating the false negatives that are3 inevitable inremote sensing image analysis qualitatively

Fig. 8. (a), (c), and (e) Maximum water level and (b), (d), and (f) topographic deformation of (a) and (b) reference simulation, (c) and (d) average simulation,and (e) and (f) our method with (g) reference map of (blue) flood and (brown) debris-flow extent and (h) binary change detection map. Mask for clouds andno data is shown in gray.

successfully improved the detection of disaster extent areas.

high accuracy has been achieved for flood and debris-

TheRemotefact that ourSensing
IoU scores are[1] Yokoya, Naoto & Yamanoi, Kazuki & He, Wei & Baier, Gerald & Adriano, Bruno & Miura, Hiroyuki & Oishi, Satoru. (2020). Breaking Limits ofbycomparable to or even flow detection. The RMSE and LSHI of our results are
better than those of the reference simulation suggests that better than those of the average simulation with large
Fig. 2. Overview concept of the proposed framework.
Deep Learning From Simulated Data for Flood and Debris-Flow Mapping. IEEE Transactions on Geoscience and Remote Sensing. PP. 1-15. 10.1109/TGRS.2020.3035469.

15

trained by PredNet can predict the next RGB image frame
R-CCS
for the inference phase. PredNet accepts both floating-point
Compressionof Time
Data through Predictive Deep Neural Networks
and integer
values for RGB
valuesEvolutionary
and predicts theImage
next RGB
image in floating-point. For the training phase,
PredNet(RupakRoyiset al., 2021 [1])designed
RGB values
as the training
data, and thenl Roy toet receive
al. proposed
new AI-drive
data compressor(TEZIP) for time evolutionary data
produce a trained model that can learn the hidden trends of thel
We train PredNet to learn how pixels move and how fast by inputting a number of time evolutional frames
pixel movement and predict future frames from base frames.
lWhencompressing
frames,ofwe
predictfor
futureframes from original base data, compute delta values and apply
We leverage
thisprediction engine
PredNeteffective
seriesencoding
compression
of of
time
evolutionary data.
à We only store (1) base frame data and (2) compressed data

III. TEZIP: (D E )C OMPRESSION OF T IME E VOLUTIONARY
I MAGEF RAMES ration compared to existing video encoder (Zstd, HFYU, FFV1, x.265 )l They achieved higher
compression
16

Original frames (or decompressed image frames)

Time evolutionary data
(Training data)Timeevolutionary
image frames

2
7

1

B1

B2

3

3

P1

P2

8

8

D1

D2

4

4

Trained model

9: Model training: Compression workflow: Decompression workflow

5

B0

6

C1

Predicted frames 3

6

Pn
8

Delta frames

DnCompressedframes

C2

4

STD

V1

X26

T

IP

14

Bn

6

Cn

Fig. 1. Workflows of TEZIP (de)compression

Compression Ratio

B0

aseline

12
10
8
6
4
2
0

Fig. 8. Compression ratio with lossless compressors.

[1] Rupak Roy, Kento Sato, Subhadeep Bhattacharya, Xingang Fang, Yasumasa Joti, Takaki Hatsui, Toshiyuki Hiraki, Jian Guo and Weikuan Yu,“Compression of Time Evolutionary Image Data through Predictive Deep Neural Networks”, In the proceedings of the 21 IEEE/ACM International
In our evaluation, we select the lossless option of x265. All
Symposium on Cluster, Cloud and Internet Computing (CCGrid 2021), May, 2021

We explain how to compress time evolutionary image

and dewhile b16

<latexit sha1_base64="HQB0SEEhfiHXoSjNURCxHoqkGBQ=">AAAChHichVG7SgNBFD1Z3/GRqI1gEwyKhYSJRg0WErCx1MREQUPYXcc4ZLO77E6CMfgDgq0prBQsxN4fsPEHLPIJYhnBxsKbzYJoUO8wM2fO3HPnDFezDeFKxpoBpae3r39gcCg4PDI6FgqPT+Rcq+LoPKtbhuXsaarLDWHyrBTS4Hu2w9WyZvBdrbTRvt+tcscVlrkjazbPl9WiKY6ErkqiMieFZCEcZTHmRaQbxH0QhR9bVvgBBziEBR0VlMFhQhI2oMKlsY84GGzi8qgT5xAS3j3HGYKkrVAWpwyV2BKtRTrt+6xJ53ZN11Pr9IpB0yFlBLPsmd2xFnti9+yFffxaq+7VaHup0a51tNwuhM6nMu//qsq0Sxx/qf70LHGEpOdVkHfbY9q/0Dv66mmjlVlLz9bn2A17Jf/XrMke6Qdm9U2/3ebpqz/8aOTljNoT/9mMbpBbjMVXYontRDSV9Bs1iGnMYJ66sYoUNrGFLFUv4gKXaCj9yoKypCx3UpWAr5nEt1DWPwFVipCE</latexit>x8 x9

Priority Application<latexit sha1_base64="rrpDUCCC4K14aRmiLz0cBZaY0XY=">AAAChHichVG7SgNBFD2u72eiNoJNMCgWEiYaNVqIYGNpjHlADGF3ncQhm91ldxKMIT8g2GphpWAh9v6AjT9g4SeIpYKNhTebBdGg3mFmzpy5584ZrmYbwpWMPXUp3T29ff0Dg0PDI6NjgeD4RNq1qo7OU7plWE5WU11uCJOnpJAGz9oOVyuawTNaeat1n6lxxxWWuSfrNs9X1JIpikJXJVHJo8JaIRhmEeZFqBNEfRCGHztW8A77OIAFHVVUwGFCEjagwqWRQxQMNnF5NIhzCAnvnqOJIdJWKYtThkpsmdYSnXI+a9K5VdP11Dq9YtB0SBnCLHtkN+yVPbBb9sw+fq3V8Gq0vNRp19pabhcCJ1PJ939VFdolDr9Uf3qWKCLueRXk3faY1i/0tr52fP6aXN+dbcyxK/ZC/i/ZE7unH5i1N/06wXcv/vCjkZcmtSf6sxmdIL0Yia5EYolYeDPuN2oA05jBPHVjFZvYxg5SVL2EU5zhXOlTFpQlZbmdqnT5mkl8C2XjE1eokIU=</latexit>

64(f) divided
Automatic Quantum Circuit Encoding of a given Arbitrary Quantum 64
State
1
2
3
(Tomonori Shirakawa et al. [1])<latexit sha1_base64="0nk6GhmSIeyqwbUYia6mcFVo3hQ=">AAACgnichVE9SwNBEH05o8b4kaiNYBMMEbEIexpUxEKwscyH+YAYwt25iUcud8fdJaDBP6CtYmGlYCH2/gEb/4BFfoJYRrCxcHI5lKDGWXZ35u282TeMbGqq7TDW9glD/uGR0cBYcHxicioUnp7J2UbDUnhWMTTDKsiSzTVV51lHdTReMC0u1WWN5+XaTvc93+SWrRr6nnNk8lJdqupqRVUkh6CUWA5HxThzLfLlsHiiH4nCs6QRfsA+DmBAQQN1cOhwyNcgwaZVhAgGk7ASWoRZ5KnuO8cJgsRtUBanDInQGp1ViooeqlPcrWm7bIV+0WhbxIwgxp7ZHeuwJ3bPXtjHn7Vabo2uliO65R6Xm+XQ6Vzm/V9WnW4Hh9+sgZodVLDhalVJu+ki3S6UHr95fNnJbKZjrUV2w15J/zVrs0fqQG++Kbcpnr4aoEcmLSe/jueHk1uJi2vxRCoR3d7wBhXAPBawRNNYxzZ2kUSWqnOc4RwXgl9YFkRhtZcq+DzOLPpM2PoEP16PmA==</latexit><latexit sha1_base64="+tBPoQq+hz3hNS6RFtqvBj3+ESE=">AAACgnichVFNSwJRFD1OVmYfWm2CNpIY0UKeJiXRQmjT0o/8ABOZmZ41OM4MM6Ng4h+obdGiVUGLaN8faNMfaOFPiJYGbVp0HaWQyu7jvXfveffcdy5XMlTFshnruIQx9/jEpGfKOz0zO+fzzy/kLL1uyjwr66puFiTR4qqi8ayt2CovGCYXa5LK81J1t/eeb3DTUnRt324avFQTjzSlosiiTVAqWvYHI2HmWODLYeHYMBLEwJK6/wEHOIQOGXXUwKHBJl+FCItWEREwGISV0CLMJE9x3jna8BK3TlmcMkRCq3QeUVQcoBrFvZqWw5bpF5W2ScwAQuyZ3bEue2L37IV9/Fmr5dToaWnSLfW53Cj7Tpcy7/+yanTbOP5mjdRso4K4o1Uh7YaD9LqQ+/zGyWU3s50OtVbZDXsl/deswx6pA63xJt+mePpqhB6JtLR/Hc8PJxcNRzbDsVQsmIgPBuXBMlawRtPYQgJ7SCJL1TnOcI4LwS2sCxFho58quAacRQyZsPMJQXyPmQ==</latexit>

5

l

13
<latexit sha1_base64="G2vI2serqx5X1fldvCHzD9rurQw=">AAACg3ichVFNSwJRFD1OWWYfWm2CNpIYQSBPk5IgENq09CM/wERmpmcNjjPDzCiY+AeibbVoVdAi2vcH2vQHWvgToqVBmxZdRymksvt479173j33ncuVDFWxbMY6LmFs3D0x6ZnyTs/Mzvn88ws5S6+bMs/KuqqbBUm0uKpoPGsrtsoLhsnFmqTyvFTd7b3nG9y0FF3bt5sGL9XEI02pKLJoE5SObJT9wUiYORb4clg4NowEMbCk7n/AAQ6hQ0YdNXBosMlXIcKiVUQEDAZhJbQIM8lTnHeONrzErVMWpwyR0CqdRxQVB6hGca+m5bBl+kWlbRIzgBB7Znesy57YPXthH3/Wajk1elqadEt9LjfKvtOlzPu/rBrdNo6/WSM126gg7mhVSLvhIL0u5D6/cXLZzWynQ61VdsNeSf8167BH6kBrvMm3KZ6+GqFHIi3tX8fzw8lFw5HNcCwVCybig0F5sIwVrNE0tpDAHpLIUvUKznCOC8EtrAtRIdZPFVwDziKGTNj5BNEHj9U=</latexit>

7<latexit sha1_base64="4kVirno3ehZho0PprT7Npw2i94g=">AAACgnichVFNSwJRFD1OVmYfWm2CNpIY0UKeJSbRQmjT0o/8ABOZmZ41OM4MM6Ng4h+obdGiVUGLaN8faNMfaOFPiJYGbVp0HaWQyu7jvXfveffcdy5XMlTFshnruIQx9/jEpGfKOz0zO+fzzy/kLL1uyjwr66puFiTR4qqi8ayt2CovGCYXa5LK81J1t/eeb3DTUnRt324avFQTjzSlosiiTVAqVvYHI2HmWODLYeHoMBLEwJK6/wEHOIQOGXXUwKHBJl+FCItWEREwGISV0CLMJE9x3jna8BK3TlmcMkRCq3QeUVQcoBrFvZqWw5bpF5W2ScwAQuyZ3bEue2L37IV9/Fmr5dToaWnSLfW53Cj7Tpcy7/+yanTbOP5mjdRso4K4o1Uh7YaD9LqQ+/zGyWU3s50OtVbZDXsl/deswx6pA63xJt+mePpqhB6JtLR/Hc8PJ7cRjsTC0VQ0mIgPBuXBMlawRtPYQgJ7SCJL1TnOcI4LwS2sCxFhs58quAacRQyZsPMJSfSPnQ==</latexit>

4

AQCE

8

<latexit sha1_base64="k/T3Fa5cT8mGWsrLLzSrQ958BCU=">AAACgnichVFNSwJRFD1OVmYfWm2CNpIY0UKeJSnRQmjT0o/8ABOZmZ41OM4MM6Ng4h+obdGiVUGLaN8faNMfaOFPiJYGbVp0HaWQyu7jvXfveffcdy5XMlTFshnruIQx9/jEpGfKOz0zO+fzzy/kLL1uyjwr66puFiTR4qqi8ayt2CovGCYXa5LK81J1t/eeb3DTUnRt324avFQTjzSlosiiTVAqVvYHI2HmWODLYeHoMBLEwJK6/wEHOIQOGXXUwKHBJl+FCItWEREwGISV0CLMJE9x3jna8BK3TlmcMkRCq3QeUVQcoBrFvZqWw5bpF5W2ScwAQuyZ3bEue2L37IV9/Fmr5dToaWnSLfW53Cj7Tpcy7/+yanTbOP5mjdRso4K4o1Uh7YaD9LqQ+/zGyWU3s50OtVbZDXsl/deswx6pA63xJt+mePpqhB6JtLR/Hc8PJ7cRjmyFo6loMBEfDMqDZaxgjaYRQwJ7SCJL1TnOcI4LwS2sCxFhs58quAacRQyZsPMJTBKPng==</latexit><latexit sha1_base64="7NP1rsa0pZtDkbHw7pGcNMUQh/g=">AAACgnichVFNSwJRFD1OVmYfWm2CNpIY0UKeJSXRQmjT0o/8ABOZmZ42OM4MM6Ng4h+obdGiVUGLaN8faNMfaOFPiJYGbVp0HaWQyu7jvXfveffcdy5XMlTFshnruIQx9/jEpGfKOz0zO+fzzy9kLb1uyjwj66pu5iXR4qqi8Yyt2CrPGyYXa5LKc1J1r/eea3DTUnTtwG4avFgTK5pSVmTRJigZK/mDkTBzLPDlsHB0GAliYAnd/4BDHEGHjDpq4NBgk69ChEWrgAgYDMKKaBFmkqc47xxteIlbpyxOGSKhVTorFBUGqEZxr6blsGX6RaVtEjOAEHtmd6zLntg9e2Eff9ZqOTV6Wpp0S30uN0q+06X0+7+sGt02jr9ZIzXbKCPmaFVIu+EgvS7kPr9xctlN76RCrVV2w15J/zXrsEfqQGu8ybdJnroaoUciLe1fx/PDyW6EI1vhaDIajMcGg/JgGStYo2lsI459JJCh6hxnOMeF4BbWhYiw2U8VXAPOIoZM2P0ETjCPnw==</latexit><latexit sha1_base64="8IQgS5/ZFvrpEK4uGprxss/krf4=">AAACg3ichVE9SwNBEH2e3/EjURvBRgwRQQhzIagIgmBjmUSjgga5OzfxyH1xtwlo8A+IrVpYKViIvX/Axj9gkZ8glgo2Fk4uQRE1zrK7M2/nzb5hdM8yA0lU71A6u7p7evv6IwODQ8PR2MjoRuBWfEPkDddy/S1dC4RlOiIvTWmJLc8Xmq1bYlMvrzTeN6vCD0zXWZcHnijYWskxi6ahSYZyKu3G4mqSQpv8dCiZ/o7E0bKMG7vDDvbgwkAFNgQcSPYtaAh4bUMFwWOsgBpjPntm+C5whAhzK5wlOENjtMxniaPtFupw3KgZhGyDf7F4+8ycRIIe6YZe6IFu6Yne/6xVC2s0tBzwrTe5wtuNHo+vvf3LsvmW2P9itdUsUcRCqNVk7V6INLowmvzq4fnL2mIuUZumK3pm/ZdUp3vuwKm+GtdZkbtoo0dnLUe/jueHs5FKqnPJdDYdX15oDaoPE5jCDE9jHstYRQZ5rl7ECU5xpnQrs0pKSTdTlY4WZwzfTFn6AMqtj9I=</latexit><latexit sha1_base64="SsFWc2ykKcWmu8QIhTIfI8FOA9I=">AAACg3ichVFNS8NAEH2NX7V+Vb0IXsSiCELYSNAiCAUvHttqq6AiSdxqaJqEZFuopX9AvKoHTwoexLt/wIt/wEN/gnis4MWDkzQo4tcsuzvzdt7sG0Z3LdMXjLViUld3T29fvD8xMDg0PJIcHSv6TtUzeMFwLMfb0jWfW6bNC8IUFt9yPa5VdItv6uXV4H2zxj3fdOwNUXf5bkU7sM2SaWiCoLyi7CVTisxCm/pwmKx+RVKILOsk77CDfTgwUEUFHDYE+RY0+LS2oYDBJWwXDcI88szwnaOJBHGrlMUpQyO0TOcBRdsRalMc1PRDtkG/WLQ9Yk5hhj2yG9ZmD+yWPbG3X2s1whqBljrdeofL3b2R44n1139ZFboFDj9Zf2oWKCEdajVJuxsiQRdGh187Om+vL+dnGrPsij2T/kvWYvfUgV17Ma5zPH/xhx6dtDR/HM83p7ggK4uymlNTmXQ0qDgmMY05msYSMlhDFgWqXsIJTnEm9Ujz0oKkdlKlWMQZxxeTVt4BzMuP0w==</latexit>

14<latexit sha1_base64="2jVcDkhu7bztkVw/fo8pGYX7wFE=">AAACg3ichVFNS8NAEH2NVut31YvgRSwVQQgbCSqCUPDiUau1hVpKErc1NE1Csi1o6R8Qr+rBk4IH8e4f8OIf8OBPEI8KXjw4SYtS1DrL7s68nTf7htFdy/QFY08Rqac32tcfGxgcGh4ZHYuPT+z6Ts0zeMZwLMfL6ZrPLdPmGWEKi+dcj2tV3eJZvbIevGfr3PNNx94Rhy4vVLWybZZMQxMEpRW1GE8oMgtt5sthstqJJNC2TSd+hz3sw4GBGqrgsCHIt6DBp5WHAgaXsAIahHnkmeE7RxODxK1RFqcMjdAKnWWK8m3Upjio6Ydsg36xaHvEnEGSPbIb9soe2C17Zh9/1mqENQIth3TrLS53i2PHU9vv/7KqdAscfLO6ahYoYSXUapJ2N0SCLowWv350/rq9mk425tgVeyH9l+yJ3VMHdv3NuN7i6YsuenTS0vx1PD+c3UVZWZLVLTWRWmkPKoZpzGKeprGMFDawiQxVL+EEpziTotKCtCiprVQp0uZMosOktU/TJY/W</latexit>

15<latexit sha1_base64="Dn5JEdZ7asP8jcIvTznSrMuR7PU=">AAACg3ichVFNSwJRFD1OWWYfWm2CNpIYQSBPsZIgENq09CM/wERmpmcNjjPDzCiY+AeibbVoVdAi2vcH2vQHWvgToqVBmxZdRymksvt479173j33ncuVDFWxbMY6LmFs3D0x6ZnyTs/Mzvn88ws5S6+bMs/KuqqbBUm0uKpoPGsrtsoLhsnFmqTyvFTd7b3nG9y0FF3bt5sGL9XEI02pKLJoE5SObJT9wUiYORb4clg4NowEMbCk7n/AAQ6hQ0YdNXBosMlXIcKiVUQEDAZhJbQIM8lTnHeONrzErVMWpwyR0CqdRxQVB6hGca+m5bBl+kWlbRIzgBB7Znesy57YPXthH3/Wajk1elqadEt9LjfKvtOlzPu/rBrdNo6/WSM126gg7mhVSLvhIL0u5D6/cXLZzWynQ61VdsNeSf8167BH6kBrvMm3KZ6+GqFHIi3tX8fzw8lFw5HNcCwVCybig0F5sIwVrNE0tpDAHpLIUvUKznCOC8EtrAtRIdZPFVwDziKGTNj5BNVDj9c=</latexit><latexit sha1_base64="KsIa6FF43m7yeqQQO+1toq2WKpc=">AAACg3ichVFNSwJRFD1OWWYfWm2CNpIYQSBPkZIgENq09CM/wERmpqcNjjPDzCiY+AeibbVoVdAi2vcH2vQHWvgToqVBmxZdRymksvt479173j33ncuVDFWxbMa6LmFi0j017Znxzs7NL/j8i0s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb67/nm9y0FF07sFsGL9XFqqZUFFm0CUpHomV/MBJmjgW+HBaOjSJBDC2p+x9wiCPokNFAHRwabPJViLBoFREBg0FYCW3CTPIU552jAy9xG5TFKUMktEZnlaLiENUo7te0HLZMv6i0TWIGEGLP7I712BO7Zy/s489abadGX0uLbmnA5UbZd7qSef+XVafbxvE3a6xmGxXEHa0KaTccpN+FPOA3Ty57mZ10qL3Obtgr6b9mXfZIHWjNN/k2xdNXY/RIpKXz63h+OLloOLIVjqViwUR8OCgPVrGGDZrGNhLYRxJZql7BGc5xIbiFTSEqxAapgmvIWcaICbufzumP1A==</latexit>

16<latexit sha1_base64="Rqawdi1t1q7XUC8L7Dx100dcdTs=">AAACg3ichVFNSwJRFD1OWWYfWm2CNpIYQSBPEZMgENq09CM/wERmpmcNjjPDzCiY+AeibbVoVdAi2vcH2vQHWvgToqVBmxZdRymksvt479173j33ncuVDFWxbMa6LmFi0j017Znxzs7NL/j8i0t5S2+YMs/JuqqbRUm0uKpoPGcrtsqLhsnFuqTyglTb7b8Xmty0FF3bt1sGL9fFI02pKrJoE5SJxCv+YCTMHAt8OSwcG0WCGFpK9z/gAIfQIaOBOjg02OSrEGHRKiECBoOwMtqEmeQpzjtHB17iNiiLU4ZIaI3OI4pKQ1SjuF/Tctgy/aLSNokZQIg9szvWY0/snr2wjz9rtZ0afS0tuqUBlxsV3+lK9v1fVp1uG8ffrLGabVSRcLQqpN1wkH4X8oDfPLnsZbczofY6u2GvpP+addkjdaA13+TbNM9cjdEjkZbOr+P54eSj4Ug8HEvHgsnEcFAerGINGzSNLSSxhxRyVL2KM5zjQnALm0JUiA1SBdeQs4wRE3Y+Addhj9g=</latexit>

Inspired by a tensor network method, they have proposed a quantum-classical hybrid algorithm to construct an
FIG. 10.by
Quantum
encoding of encoding
a gray scale picture known as “P
optimal quantum circuit for classical data that is represented as a quantum state
the circuit
amplitude
data (with 8 ⇥ 8 pixels, as an example). (b) Original picture with 256⇥

ˆ on L = 16 qubits with the di↵erent numbe
the quantum circuit states C|0i
The proposed algorithm employs as an objective function the absolute value 520.
of (f)
fidelity𝐹 = divided
0 𝐶' &into
Ψ 16, pieces
which
is1, 2, . . . , 16 indicat
Original picture(ms =(g)-(i)
Picturesreconstructedbydecoding
each
quantum
circuit state Cˆ(maximized iteratively to construct an optimal quantum circuit 𝐶' with controlled
accuracy.
of two-qubit unitary operators, M = 24, 42, and 450.

They proposed a method to construct a quantum circuit Application of this method(composed of up to two-qubit gates) representing a given
Amplitude Encoding of classical data
quantum state approximately with controlled accuracy.

10 2|h0|Cˆ† | c i|1/L ,(m )|h0|(Cˆ(ms ) )† | c s i|1/Ll<latexit sha1_base64="9dN+o108HorGsFMmyr4Cynkl1pc=">AAACfnichVFLSwJRFP6crMweWqugjSRGtJBrSIkroU1LH/kAE5mZrjY4L2ZGwcQ/UNuiRauCFtG+P9CmP9DCnxAtDdq06DhKIZWdy733nO+e79zvcCRTVWyHsZ5HmPJOz8z65vzzC4tLgaB/uWAbTUvmedlQDaskiTZXFZ3nHcVRecm0uKhJKi9Kjb3Be7HFLVsx9AOnbfKKJtZ1pabIokNQJl4NhmNR5lroy2HR+DgSxsjSRvABhziCARlNaODQ4ZCvQoRNq4wYGEzCKugQZpGnuO8cXfiJ26QsThkioQ066xSVR6hO8aCm7bJl+kWlbREzhAh7Znesz57YPXthH3/W6rg1BlradEtDLjergdPV3Pu/LI1uB8ffrImaHdSQcLUqpN10kUEX8pDfOrns55LZSGeD3bBX0n/NeuyROtBbb/JthmevJuiRSEv31/H8cArb0dhONB5OJUZj8mEN69ikWewihX2kkafaHGc4x4XgFbaE2DBR8IwYKxgzIfkJ0mGO1g==</latexit>

Quantum computer can in principle treat an exponentially large amount of data, however, a major challenge in
9
10
11
12
practice to represent classical data such as a classical image into a quantum circuit<latexit sha1_base64="e4s8WvPHOKo93MFbFrWxviOSxEw=">AAACgnichVE9TwJBEH2cqIgfoDYmNkSCMRZkUaJoLEhsLPmQjwQJuTtXvHDcXe4OEiT8AW01FlaaWBh7/4CNf8CCn2AsMbGxcDiIhqg4m92deTtv9k1GMlTFshlru4QR9+jYuGfCOzk1PePzz85lLb1myjwj66pu5iXR4qqi8Yyt2CrPGyYXq5LKc1Jlt/ueq3PTUnRt324YvFgVy5pypMiiTVByq+QPRsLMscCXw8LRQSSIviV0/wMOcAgdMmqogkODTb4KERatAiJgMAgrokmYSZ7ivHO04CVujbI4ZYiEVugsU1TooxrF3ZqWw5bpF5W2ScwAQuyZ3bEOe2L37IV9/Fmr6dToamnQLfW43Cj5ThfS7/+yqnTbOP5mDdVs4wgxR6tC2g0H6XYh9/j1k8tOejsVai6zG/ZK+q9Zmz1SB1r9Tb5N8tTVED0SaWn9Op4fTnYtHNkIR5PRYDzWH5QHi1jCCk1jE3HsIYEMVec4wzkuBLewKkSE9V6q4Opz5jFgws4nUE6PoA==</latexit>l<latexit sha1_base64="40oW74jKRRbGdgDhmo2NqozC3PA=">AAACgnichVFNSwJRFD1OVmYfWm2CNpIY0UKeKSXRQmjT0o/8ABOZmZ41OM4MM6Ng4h+obdGiVUGLaN8faNMfaOFPiJYGbVp0HaWQyu7jvXfveffcdy5XMlTFshnruIQx9/jEpGfKOz0zO+fzzy/kLL1uyjwr66puFiTR4qqi8ayt2CovGCYXa5LK81J1t/eeb3DTUnRt324avFQTjzSlosiiTVAqWvYHI2HmWODLYeHYMBLEwJK6/wEHOIQOGXXUwKHBJl+FCItWEREwGISV0CLMJE9x3jna8BK3TlmcMkRCq3QeUVQcoBrFvZqWw5bpF5W2ScwAQuyZ3bEue2L37IV9/Fmr5dToaWnSLfW53Cj7Tpcy7/+yanTbOP5mjdRso4K4o1Uh7YaD9LqQ+/zGyWU3s50OtVbZDXsl/deswx6pA63xJt+mePpqhB6JtLR/Hc8PJ7cRjmyGY6lYMBEfDMqDZaxgjaaxhQT2kESWqnOc4RwXgltYFyJCtJ8quAacRQyZsPMJQ5qPmg==</latexit>

6<latexit sha1_base64="6sRJsNazBnpMZ1+q3Td67VCojpo=">AAACgnichVE9TwJBEH2cqIgfoDYmNkSCMRZkUVRiLEhsLPmQjwQJuTtXvHDcXe4OEiT8AW01FlaaWBh7/4CNf8CCn2AsMbGxcDiIhqg4m92deTtv9k1GMlTFshlru4QR9+jYuGfCOzk1PePzz85lLb1myjwj66pu5iXR4qqi8Yyt2CrPGyYXq5LKc1Jlt/ueq3PTUnRt324YvFgVy5pypMiiTVByo+QPRsLMscCXw8LRQSSIviV0/wMOcAgdMmqogkODTb4KERatAiJgMAgrokmYSZ7ivHO04CVujbI4ZYiEVugsU1TooxrF3ZqWw5bpF5W2ScwAQuyZ3bEOe2L37IV9/Fmr6dToamnQLfW43Cj5ThfS7/+yqnTbOP5mDdVs4wgxR6tC2g0H6XYh9/j1k8tOejsVai6zG/ZK+q9Zmz1SB1r9Tb5N8tTVED0SaWn9Op4fTnYtHNkMR5PRYDzWH5QHi1jCCk1jC3HsIYEMVec4wzkuBLewKkSE9V6q4Opz5jFgws4nR9aPnA==</latexit>(g) 24 unitary op

256 ⇥ 256 (16 qubits)
64 ⇥ 64 (12 qubits)

10 3

vector,ms = 1
the whtum storiginaFig 1: Quantum circuit encoding of a gray scale. (a) Labeling of two-dimensional classical data (with 8 × 8 pixels,
as an example). (b) Original picture with 256×256 pixels. (c)-(e) Pictures reconstructed by decoding the quantum
circuit states. (f ) Original picture divided into 16 pieces. (g)-(i) Pictures reconstructed by decoding each quantum c
ircuit state1

Arbitrary
quantum
state

1

10 4
10 5
0

200

(C
|0i) indicated by red triangles (green crosses). Here,
ms = 1, 2, . . . , 16 and thus 16 di↵erent results are shown forthe case where
the picture is divided into 16 pieces.
[1] Tomonori Shirakawa, Hiroshi Ueda, Seiji Yunoki, “Automatic Quantum Circuit Encoding of a given Arbitrary QuantumState”, arXiv:2112.14524, Dec, 2021

where

400
600
800
M
Fig 2: Fidelity per site between the quantum state qubits
representing
theperoriginal
picturethein quantum
Fig. 1(b)state[the |msths
FIG.
11. Fidelitysite betweenci(ms )(| egmenti) on= original
16 (L = picture12) qubits
representingof Lthein Fig.1(f)]andthetheorigquantu
c
with
inal
picture statein Fig.
10(b) [thesegment (greenof the origis thtriangles
m circuitindicatedby mredcrosses)P212
ˆ
nal
n=0
. picture in Fig. 10(f)] and the quantum circuit state C|0iˆ(ms )

the baTherefthe se17

Priority Application

Other AI researches by Priority Application Projects(Fugaku is not used)lComprehensive gene network analysis by XAI for uncovering molecular mechanism of disease (Heewon Park etal., 2020 [1])llAlthough various approaches were developed to gene network analysis, comprehensive interpretation of the massive
networks remains a challenge
They conducted comprehensive analysis of the massive gene networks with the use of explainable artificial intelligence(XAI) approaches-ll

DeepTensor and Tensor Reconstruction-based Interpretable Prediction (TRIP), proposed by Maruhashi et al. 2018; 2020

The use of the XAI enables us to overcome limitation of existing gene network analysis, i.e., narrow angle in the largescale gene network analysis, and this leads to a better understanding of molecular interplay involved to disease

Machine learning by neural network for quantum many-body solver and experimental data analysis (Yusuke
Nomura et al., 2021 [2])lQuantum many-body problems are known to show NP hard difficultyl

We have developed a solver by using Boltzmann machines to approximate solutions accurately and efficientlyl

It was applied to a challenging problem of a quantum spin model and established the existence of quantum spin liquid
phase, which bears a long-ranged quantum entanglement. On another application of machine learning, we haveextracted electron self-energy, which is hidden in the photoemission experimental data. It has established the existenceof prominent resonance peaks which are responsible for the high-temperature superconductivity.

[7] Park, Heewon Maruhashi, Koji, Yamaguchi, Rui, Imoto, Seiya, Miyano, Satoru, “Global gene network exploration based on explainable artifi
cial intelligence approach”, Nov., 2020
[8] Nomura, Yusuke and Imada, Masatoshi, “Dirac-Type Nodal Spin Liquid Revealed by Refined Quantum Many-Body Solver Using Neural-Net
work Wave Function, Correlation Ratio, and Level Spectroscopy”, American Physical Society, 10.1103/PhysRevX.11.031034, Nov., 2021

18

Priority Application

Other AI researches by Priority Application Projects(Fugaku is not used)l

Facilitating ab initio configurational sampling of multicomponent solids using an on-lattice neural network model and active learning(Kasamatsu et al., 2020 [1])lKasamatsu et al. proposed a scheme for ab initio configurational sampling in multicomponent crystalline solids using Behler-Parinello typeneural network potentials (NNPs)l

The NNPs are trained to predict the energies of relaxed structures from the perfect lattice with configurational disorder instead of the usual
way of training to predict energies as functions of continuous atom coordinates.
lllTraining set bias is avoided through an active learning scheme
This enables bypassing of the structural relaxation procedure which is necessary when applying conventional NNP approaches to the latticeconfiguration problemlThis idea is demonstrated on the calculation of the temperature dependence of the degree of A/B site inversion in MgAl2O4, which is amultivalent system requiring careful handling of long-range interactionsl

The present scheme may serve as an alternative to cluster expansion for ʻdifficultʼ systems, e.g., complex bulk or interface systems withmany components and sublattices that are relevant to many technological applications today

Density functional theory from supervised learning (Ryo Nagai et al. 2021 [2])lDensity functional theory (DFT) is the standard electronic structure theory and is widely used as a basis for materials designl

DFT is based on the Hohenberg-Kohn theorem that there is a one-to-one correspondence between particle density and energy, a
relationship that should be machine learnable, but the lack of this relationship makes the accuracy limitedlThey recently proposed a method to establish this relationship using neural network methodsllThey found that the DFT developed in this study can improve the accuracy not only for molecular systems, for which the training set can beobtained by a very accurate quantum chemical method, but also for solids, for which the training set is unavailable but theoretically derived
physical conditions work alternatively
This paves a systematic way for further accuracy and will serve as a tool for developing the ultimate material database[1] Kasamatsu, Shusuke and Motoyama, Yuichi and Yoshimi, Kazuyoshi and Matsumoto, Ushio and Kuwabara, Akihide and Ogawa, Takafumi, “Facil
itating {¥it ab initio} configurational sampling of multicomponent solids using an on-lattice neural network model and active learning, 10.48550/ARXI
V.2008.02572”[2] Nagai, Ryo and Akashi, Ryosuke and Sugino, Osamu, “Machine-Learning-Based Exchange-Correlation Functional with Physical Asymptotic Const

19

Highlight of Efforts in Bridging AI <-> HPC
in R-CCS (Fugaku)by R-CCS High Performance Artificial Intelligence Systems Research Team20

R-CCS HPAIS TeamØ PI: Mohamed WahibØ Senior Research Scientists: Jun Igarashi
Ø Research Scientists: Aleksandr Drozd, Emil Vatai
Ø Visiting Scientists: Rio Yokota (TokyoTech), Balazs Gerofi (Intel)
Ø Interns: Soyturk (Koc U., Turkey), Zhang (Hokkaido U), Pucceti (SNS)MohamedWahib

Jun
Igarashi

Aleksandr
Drozd

Emil
Vatai

Rio
YokotaBalaszGerofiMuhammet

Soyturk

Enzhi
Zhang

Giovanni
Pucceti21

Bottlenecks in Scaling AI

HPC à AI

Unlike traditional HPC applications, AI workloads have multiple bottlenecks
Memory capacity
High

Applications, Models, Paradigms
Sci. Data’21 CiSE’21 ICS’21 SC’20 IPDPS’20 SC’19 (2) SC’14

I/O
Throughput
Latency
Network

Level of Abstraction(Nature)(ACM)(Elsevier)(IEEE)(ACM/IEEE)(ACM/IEEE)(ACM/IEEE)

Evostar’17 SC’16(Species)(Elsevier)(ACM/IEEE)(IEEE)(IEEE)(IEEE)

Memory BW(ACM/IEEE)

HPDC’15(ACM/IEEE)

Frameworks, Runtimes, Middleware
PARCO’21 SC’20 TPDS’22 IPDPS’19 CEC’11 (2)

(ACM)

EC’13

(Springer)

CEC’11 (1)(IEEE)

Networking and I/O
IPCCC’19

CCPE’20

Hardware
IPDPS’21

IPCCC’19(IEEE)Low(ACM)

Algorithms
HPDC’21 SoftwareX’21 SC’19 (1)(IEEE)

Compute(IEEE)(Wiley)IPDPS’22(IEEE)(IEEE)

Overview of previous research: perspective of
AI+HPC computing stack

22

ParDNN: An Oracle for Characterizing and Guiding
Large-Scale Training of Deep Neural Networks

HPC à AI(a) Sequential implementation on a single PEØDifferent forms of model parallelism are emerging
ØWe provide a model-driven analysis and a utility tohelp in detecting the limitations and bottlenecks ofdifferent parallelism approaches at scale(b) Data parallelism(c) Spatial parallelism splits the input G and output ~ on either width (asshown in this �gure), height or both dimensions(d) Layer parallelism (partition the model vertically with pipeline implementations)

Overview of
ParDNN(e) Filter parallelism (partition the model horizontally)(f) Channel parallelism (partition the model horizontally)

ACM HPDC’21 A. Kahira, T. Nguyen,, L. Gomez, Ryousei Takano, R. Badia, M. Wahib: ParDNN: An Oracle for
Characterizing and Guiding Large-Scale Training of Deep Neural Networks, ACM Proceedings of theInternational Symposium on High-Performance Parallel and Distributed Computing 2021 (ACM HPDC’21)(g) Hybrid parallelism (example of �lter on top of data parallelism)

23

Figure 1: Di�erent strategies for distributed training of CNNs. Red

HPC à AI

GTOPX Space Mission BenchmarksØLandscape Analysis for MLØImportant to understand importance ofparametersØSpace mission benchmarks
(a)

(a)

(b)

(c)

(d)

X1

X2

X3

X4

X5

X6

X7

X8

X9

X10

X11

X12

X13

X14

X15

X16

X17

(b)

X18

X19 X20

X21

X22

X1

X2

X3

X4

X5 X6

X7

X8

X9

X10

X11

X12 X13X14 X15 X16 X17 X18

(e)

(c)

Elsevier
M. Schlueter, using
M. Neshat,
M. Wahib,(a)M.Messenger
Munetomo, M.
Figure 5: Landscape analysis
of SoftwareX
GTOPX benchmarks
grid search.
Wagner:GTOPX
SpaceMissionBenchmarks
Elsevier
SoftwareX
Volume 14,(full), (b) Rosetta, (c) Messenger (reduced)
June 2021

24

HPC à AI

Scaling Distributed Deep Learning Workloads beyond
the Memory Capacity with KARMA

Performance on V100. For all figures, only the firstreported mini-batch size (x-axis) fits in memory
ResNet-50

VGG16

ResNet-200

300

Out-of-core

100

200

Performance [samples/s]

250

Out-of-core

90
80

80

70

200
60
60

50

150

40
40

100

30

50

20

0

0

20
10

128

256

384

512

BatchSize

640

0
32

768

WRN-28-10Out-of-core

64

96

128

BatchSize

160

ResNet-1001

4

8

12

16

BatchSize

20

24

U-Net

18

Out-of-core200

Out-of-core
16

120

Performance [samples/s]ØConcurrency-driven out-of-CoreØCapacity-based interleaved with recomputeØ1.52x over state-of-the-art (single GPU)ØFirst out-of-core to support multi-GPUØHeterogeneity and careful orchestrationØOutperforming DP+MD with out-of-coreØExperiments with up to 2,048 GPUs

Out-of-core

14
100

150

12

80

10
100

60

8
6

40
50

4

20

2

0

in-core

0

0
256

512

768

102 4

BatchSize

128 0

SuperNeurons

ACM/IEEE SC’20 Mohamed Wahib, Haoyu Zhang, Truong Thao Nguyen, Aleksandr Drozd, Jens Domke, Lingqi
Zhang, Ryousei Takano, Satoshi Matsuoka, Scaling Distributed Deep Learning Workloads beyond the Memory
Capacity with KARMA, ACM/IEEE Proceedings of Supercomputing 2020 (ACM/IEEE SC’20)

64

128

192

256

320

8

BatchSize
CheckmatevDNN++

KARMA

16

24

32

BatchSize

40

KARMA (w/ re-computation)

25

Optimizing Network Architecture for Deep Learning

HPC à AI

Problem: Number of involved GPUs (P)becomes too big, 1000s GPUs.
Ø Ring-based algorithm O(P)𝛼 : latency factor
increase.
Ø Halving-Doubleling algorithms O(logP)𝛼 :network congestion of communication.

Proposal: Distributed Loop Network topology
Pipelined Halving – Doubling Algorithm

CCGRID’21 Truong Thao Nguyen, Mohamed Wahib: An Allreduce Algorithm and Network Co-design for Large-Scale
Training of Distributed Deep Learning, In proceedings of the IEEE/ACM International Symposium on Cluster, Cloud andInternet Computing 2021

26

AI à HPC

AI for Scientific Codes
Use AI for code auto generation to aid scientific programmers
in producing HPC programs.
Compiler Technologies

High-level Code-Transformation

Code auto-generation

High-level context

High-level context lost

Not scalable

Constrained time budget

Problem-specificThis Proposal
Ø C o m p ile r t e c h .
Ø H ig h - le v e l t r a n s .
Ø A u t o - t u n in g

+ Machine Learning

Scalable exploration; general
High-level context
Code auto-generation
Elastic time budget

Auto-tuning
Scalable exploration; generalG eneral and expandable approach

No code auto-generation27

Fugaku is driving innovation in bridging HPC <-> AI, and willcontinue to do!

Thank you.