Integration of
Simulation/Data/Learning and Beyond
Kengo Nakajima
Information Technology Center
The University of Tokyo
International Workshop on “Integration of Simulation/Data/Learning and Beyond”
45th ASE Seminar (Advanced Supercomputing Environment)
November 29, 2023, Kashiwa, Japan & Online

Acknowledgements• JSPS Grant-in-Aid for Scientific Research (S)(19H05662)• New Energy & Industrial Technology Development
Organization (NEDO): Cross-ministerial Strategic
Innovation Promotion Program (SIP): Big-Data and AIEnabled Cyberspace Technologies• Joint Usage/Research Center for Interdisciplinary
Large-scale Information Infrastructures (JHPCN)– jh210022-MDH, jh220029, jh230017, jh230018• Information Technology Center, The University of Tokyo

2

3• Integration of (Simulation/Data/Learning)– Wisteria/BDEC-01– h3-Open-BDEC• Applications on Wisteria/BDEC-01 with h3-Open-BDEC– Seismic Wave Propagation– Global Atmosphere– International/Domestic Collaborations• Integration of (Simulation/Data/Learning) and Beyond• Summary4

2001-2005
Hitachi SR8000
1,024 GF

2006-20102011-20152016-2020

Hitachi SR11000
J1, J2

Hitachi SR16K/M1
Yayoi

5.35 TF, 18.8 TF

54.9 TF

Hitachi
SR2201

Hitachi
SR8000/MPP

307.2GF

2,073.6 GF

2021-2025

2026-2030

OBCX(Fujitsu)
6.61 PF

Hitachi HA8000
T2K Todai

OakforestPACS (Fujitsu)

140 TF25.0 PF

Fujitsu FX10
Oakleaf-FX

Supercomputers@ITC/U.Tokyo
2,600+ Users
55+% outside of U.Tokyo

OFP-II
75+ PF

Fujitsu
33.1 PF

1.13 PFBDEC02250+ PF

ReedbushU/H/L (SGI-HPE)
3.36 PF

Ipomoea-01 25PB

Ipomoea03

Ipomoea-02

5

2001-2005
Hitachi SR8000
1,024 GF

SR80002006-2010
Hitachi SR11000
J1, J2
5.35
18.8 TF
IBMTF,
Power5+

Hitachi
SR2201

Hitachi
SR8000/MPP

307.2GF

2,073.6 GF

HARP-1E

2011-20152016-2020

Hitachi SR16K/M1
Yayoi
54.9
TF
IBM Power76.61 PF

Hitachi HA8000
T2K Todai

OakforestPACS (Fujitsu)

140 TF25.0 PFAMD Opteron

OFP-II
75+ PF

AcceleratorsIntel Xeon Phi
Fujitsu FX10
Oakleaf-FX

BDEC02

Fujitsu
33.1 PF

1.13 PFSPACR64 IXfx

Multicore CPU

2026-2030

Intel CLX
OBCX(Fujitsu)

SR8000Pseudo Vector

2021-2025

ReedbushU/H/L (SGI-HPE)

250+ PFA64FX,
Intel Icelake+
NVIDIA A100

Accelerators

3.36 PFIntel BDW +NVIDIA P100

Ipomoea-01 25PB

Ipomoea03

GPU, Accelerators
Ipomoea-02

6

Research Area based on Machine Hours (FY.2022)■CPU, ■GPUBio
Informatics
BioPhysics

Energy/Physics

Materials

Engineering

Earth/Space

OBCX
CascadeLake

Engineering
Earth/Space
Material
Engineering
Energy/Physics
Info. Sci. : System
Info. Sci. : Algorithms
Earth/
Energy/Info. Sci. : AI
Space
Physics
Education
Industry
Bio
Bioinformatics
Materials
Social Sci. & Economics
Data

Odyssey
A64FX

Data

工学・
Engineering
Analytics ものつくり
Earth/Space
BioInformatics
Material
地球科学・
Energy/Physics
エネルギー・
宇宙科学
Engineering
Info.
Sci. : System
物理学
AI
Info.
Sci. : Algorithms
Info. Sci. : AI
材料科学
Education
Earth/
Algorithm
Space
Industry
Bio
Materials
Bioinformatics
Social Sci. & Economics
Data

Aquarius
A100

Engineering
Earth/Space
Material
Energy/Physics
Info. Sci. : System
Info. Sci. : Algorithms
Info. Sci. : AI
Education
Industry
Bio
Bioinformatics
Social Sci. & Economics
DataSupercomputing is changing• Various Types of Workloads– Computational Science & Engineering: Simulations– Big Data Analytics– AI, Machine Learning …

• Integration of (Simulation+Data+ Learning) (S+D+L) is
important towards Society 5.0, Human-Centered Society
proposed by Japanese Gov.
– By Integration of Cyber & Physical SpaceBDEC (Big Data &• BDEC (Big Data & Extreme Computing)
Extreme Computing)– Platform for Integration of (S+D+L)– Focusing on S (Simulation)• AI for HPC, AI for Science– Planning started in 2015

SDL
+

+

7

8

Wisteria/BDEC-01

The 1st BDEC System (Big Data &Extreme Computing)• Operation starts on May 14, 2021 Platform for Integration of (S+D+L)• 33.1 PF, 8.38 PB/sec by Fujitsu– ~4.5 MVA with Cooling, ~360m2• 2 Types of Node Groups– Hierarchical, Hybrid, Heterogeneous (h3)– Simulation Nodes: Odyssey• Fujitsu PRIMEHPC FX1000 (A64FX), 25.9 PF– 7,680 nodes (368,640 cores), Tofu-D– General Purpose CPU + HBM– Commercial Version of “Fugaku”– Data/Learning Nodes: Aquarius• Data Analytics & AI/Machine Learning• Intel Xeon Ice Lake + NVIDIA A100, 7.2PF– 45 nodes (90x Ice Lake, 360x A100), IB-HDR• Some of the DL nodes are connected to external resources directly• File Systems: SFS (Shared/Large) + FFS (Fast/Small)9

Wisteria/BDEC-01

The 1st BDEC System (Big Data &Extreme Computing)• Operation starts on May 14, 2021 Platform for Integration of (S+D+L)• 33.1 PF, 8.38 PB/sec by Fujitsu– ~4.5 MVA with Cooling, ~360m2• 2 Types of Node Groups– Hierarchical, Hybrid, Heterogeneous (h3)– Simulation Nodes: Odyssey• Fujitsu PRIMEHPC FX1000 (A64FX), 25.9 PF– 7,680 nodes (368,640 cores), Tofu-D– General Purpose CPU + HBM– Commercial Version of “Fugaku”– Data/Learning Nodes: Aquarius• Data Analytics & AI/Machine Learning• Intel Xeon Ice Lake + NVIDIA A100, 7.2PF– 45 nodes (90x Ice Lake, 360x A100), IB-HDR• Some of the DL nodes are connected to external resources directly• File Systems: SFS (Shared/Large) + FFS (Fast/Small)10

Wisteria/BDEC-01

The 1st BDEC System (Big Data &Extreme Computing)• Operation starts on May 14, 2021 Platform for Integration of (S+D+L)• 33.1 PF, 8.38 PB/sec by Fujitsu– ~4.5 MVA with Cooling, ~360m2• 2 Types of Node Groups– Hierarchical, Hybrid, Heterogeneous (h3)– Simulation Nodes: Odyssey• Fujitsu PRIMEHPC FX1000 (A64FX), 25.9 PF– 7,680 nodes (368,640 cores), Tofu-D– General Purpose CPU + HBM– Commercial Version of “Fugaku”– Data/Learning Nodes: Aquarius• Data Analytics & AI/Machine Learning• Intel Xeon Ice Lake + NVIDIA A100, 7.2PF– 45 nodes (90x Ice Lake, 360x A100), IB-HDR• Some of the DL nodes are connected to external resources directly• File Systems: SFS (Shared/Large) + FFS (Fast/Small)Simulation Nodes

Odyssey
25.9 PF, 7.8 PB/s

Fast File
System（FFS）
1.0 PB,
1.0 TB/s

Shared File
System
（SFS）
25.8 PB,
0.50 TB/s

Data/Learning Nodes

Aquarius
7.20 PF, 578.2 TB/s

11

Simulation Nodes

Odyssey
25.9 PF, 7.8 PB/s

Fast File
System（FFS）
1.0 PB,
1.0 TB/s

Shared File
System
（SFS）
25.8 PB,
0.50 TB/s

Data/Learning Nodes

Aquarius
7.20 PF, 578.2 TB/s

12

Optimization of Models/Parameters for
Simulations by Data Analytics & Machine
Learning (S+D+L)

13

Research Area based on Machine Hours (FY.2022)■CPU, ■GPUBio
Informatics
BioPhysics

Energy/Physics

Engineering

Materials

OBCX

Earth/Space

Engineering
Earth/Space
Material
Engineering
Energy/Physics
Info. Sci. : System
Info. Sci. : Algorithms
Earth/
Energy/Info. Sci. : AI
Space
Physics
Education
Industry
Bio
Bioinformatics
Materials
Social Sci. & Economics
Data

Odyssey

Data

工学・Engineering
Analytics ものつくり
Earth/Space
BioInformatics
Material
地球科学・
Energy/Physics
エネルギー・
宇宙科学
Engineering
Info.
Sci. : System
物理学
AI
Info.
Sci. : Algorithms
Info. Sci. : AI
材料科学
Education
Earth/
Algorithm
Space
Industry
Bio
Materials
Bioinformatics
Social Sci. & Economics
Data

Aquarius

Engineering
Earth/Space
Material
Energy/Physics
Info. Sci. : System
Info. Sci. : Algorithms
Info. Sci. : AI
Education
Industry
Bio
Bioinformatics
Social Sci. & Economics
Data

14

Research Area based on Machine Hours (FY.2023)■CPU, ■GPU (April-September)
Bio
Informatics
Bio
Physics
Engineering

Energy/Physics

Materials

Earth/Space

OBCX
Retired in
September 2023

Engineering
Earth/Space
Material
Engineering
Energy/Physics
Info. Sci. : System
Energy/Info. Sci. : Algorithms
Physics
Info. Sci. : AI
Earth/
Education
Space
Industry
Bio
Materials
Bioinformatics
Social Sci. & Economics
Data

Data
Engineering
Analytics
Earth/
Earth/SpaceSpaceEngiBio Material
neering
Informatics
Energy/Physics
Info. Sci. : System
BioInfo. Sci. : Algorithms
Physics
Info. Sci. : AI
Materials
Education
AI
Industry
Bio
LLM
Bioinformatics
Social Sci. & Economics
Data

Odyssey

Aquarius

Engineering
Earth/Space
Material
Energy/Physics
Info. Sci. : System
Info. Sci. : Algorithms
Info. Sci. : AI
Education
Industry
Bio
Bioinformatics
Social Sci. & Economics
Data

15

• Integration of (Simulation/Data/Learning)– Wisteria/BDEC-01– h3-Open-BDEC• Applications on Wisteria/BDEC-01 with h3-Open-BDEC– Seismic Wave Propagation– Global Atmosphere– International/Domestic Collaborations• Integration of (Simulation/Data/Learning) and Beyond• Summary

16h3-Open-BDEC Innovative Software Platform
for Integration of (S+D+L) on the BDEC
System, such as Wisteria/BDEC-01• 5-year project supported by
Japanese Government (JSPS)
since 2019
– FY.2023 is the final year
– Today is the WS of this Project• Leading-PI: Kengo Nakajima (The
University of Tokyo)• Total Budget: 1.41M USD

17Members (Co-PI’s) of h3-Open-BDEC ProjectComputer Science, Computational Science, Numerical Algorithms,
Data Science, Machine Learning• Kengo Nakajima (ITC/U.Tokyo, RIKEN), Leading-PI• Takeshi Iwashita (Hokkaido U), Co-PI, Algorithms• Hisashi Yashiro (NIES), Co-PI, Coupling, Utility• Hiromichi Nagao (ERI/U.Tokyo), Co-PI, Data Assimilation• Takashi Shimokawabe (ITC/U.Tokyo), Co-PI, ML/hDDA• Takeshi Ogita (Waseda U.), Co-PI, Accuracy Verification• Takahiro Katagiri (Nagoya U), Co-PI, Appropriate
Computing• Hiroya Matsuba (ITC/U.Tokyo, Hitachi), Co-PI, Container

18

Former Pos-Doc’s• Masatoshi Kawai (ITC/U.Tokyo -> Nagoya U.)• Hayato Shiba (ITC/U.Tokyo -> U.Hyogo)

19

Contributors/Collaborators• Information Technology Center, • Hokkaido University– T. Fukaya
The University of Tokyo– S. Sumimoto, T. Arakawa• Nagoya University– T. Suzumura, M. Hanai– T. Hanawa– T. Hoshino• Kyushu University• Earthquake Research Institute,– S. Oshima, K. Inoue
The University of Tokyo• RIKEN R-CCS– T. Furumura, H. Tsuruoka– T. Ichimura, K. Fujita, S. Ito• Tokyo Institute of Technology– R. Yokota, R. Sakamoto– M. Nakao, T. Imamura• Fujitsu– Y. Sakaguchi, Y. Kasai, D. Obinata• My Former Students– Y.C. Chen (KIT), R. Yoda (BWU)– A.T. Magro (Aitia)

20(Part of) International Collaborators• Osni Marques (Lawrence Berkeley National Laboratory, USA)
• Richard Vuduc (Georgia Institute of Technology, USA)
• Edmond Chow (Georgia Institute of Technology, USA)• Weichung Wang (National Taiwan University, Taiwan)• Feng-Nan Hwang (National Central University, Taiwan)• Gerhard Wellein (FAU Erlangen & Nuremberg, Germany)• Mathias Bolten (University of Wuppertal, Germany)• Serge Petiton (University of Liles/CNRS, France)• Xing Cai (Simula Research Laboratory, Norway)• Estela Suarez (Jülich Supercomputing Cetner/Univ. Bonn, Germany)• France Boilllod-Cerneux (CEA, France)

21h3-Open-BDEC Innovative Software Platform
for Integration of (S+D+L) on the BDEC
System, such as Wisteria/BDEC-01• “Three” Innovations– New Principles for Numerical
Analysis by Adaptive Precision,
Automatic Tuning & Accuracy
Verification– Integration of (S+D+L) by
13:30-13:45
Iwashita(Hokkaido U.)
HierarchicalTakeshi
Data Driven
Approach
13:45-14:00
Takahiro Katagiri (Nagoya U.)(hDDA)
15:35-15:50
Ogita (Waseda U.)– Software & Takeshi
Utilities for
15:50-16:05
Masatoshi
Kawai(NagoyaU.)
Heterogenous
Environment,suchasWisteria/BDEC-01

22

Approximate Computing with
Low/Adaptive/Trans Precision• Mostly, scientific computing has been conducted using FP64 (double
precision, DP)– Sometimes, problems can be solved by FP32 (single precision, SP) or lower
precision• Lower precision may save time, energy and memory

23

Approximate Computing with
Low/Adaptive/Trans Precision• Mostly, scientific computing has been conducted using FP64 (double
precision, DP)– Sometimes, problems can be solved by FP32 (single precision, SP) or lower
precision• Lower precision may save time, energy and memory• Approximate Computing– Originally for image recognition etc. where accuracy is not necessarily
required– Also applied to numerical computations• Computations by lower precision and by mixed precision may
provide results with less accuracy

P3D: Steady State 3D Heat
Conduction by FVM (1/2) NZ∇ ⋅ ( λ ∇φ ) + f = 0
z

∆Z
∆Y

y

∆X

NX

x

φ = 0@ z = zmax

λ1
NZ

• 7-point Stencil• Heterogenous Material Property

NY

λ2
λ1
24

P3D: Steady State 3D Heat
Conduction by FVM (1/2) NZ∇ ⋅ ( λ ∇φ ) + f = 0
z

∆Z
∆Y

y

• 7-point Stencil• Heterogenous Material Property

NY
NX

x

φ = 0@ z = zmax

– λ1/λ2 is proportional to the condition
number of coefficient matrices

λ1• ICCG Solver• Fortran 90 + OpenMP• CM-RCM Reordering

NZ• Coefficient Matrix– Sparse, SPD

∆X

λ2
λ1
25

P3D: Steady State 3D Heat
Conduction by FVM (2/2)• Various Configurations– FP64 (Double), FP32 (Single), FP16 (Half) (just for preconditioning)– Matrix Storage Format (CRS, ELL, SELL-C-σ etc.)
C
σ

CRS

ELL

Sliced ELL

SELL-C-σ26

27

Ratio of FP32(SP)/FP64(DP): CRS

λ1

λ2
λ1
∇ ⋅ ( λ∇φ ) + f = 0
Intel Xeon BDW
1 Node: 18 cores x 2 soc’s

Ratio ofRatio
FP32/FP64

Iterations● & Time△ for ICCG 3.00
λ1/λ2 , 1283 DOF, CRS
Ratio<1 ⇒ FP32 is faster
2.50

Iterations
sec.
●Iterations
△Time

2.00

1.50

1.00

0.50

0.00
1.E+00

1.E+01

1.E+02

1.E+03

1.E+04Lambda-1/Lambda-2
Ratioof λ1/ λ2

1.E+05

1.E+06[KN et al. 2018]

28

Ratio of FP32(SP)/FP64(DP) : CRS

λ1

λ2
λ1
∇ ⋅ ( λ∇φ ) + f = 0
Intel Xeon BDW
1 Node: 18 cores x 2 soc’s

Ratio ofRatio
FP32/FP64

Iterations● & Time△ for ICCG 3.00
λ1/λ2 , 1283 DOF, CRS
2.50
Ratio<1 ⇒ FP32 is faster

Iterations
sec.
●Iterations
△Time

2.00

FP32: Slower

1.50

1.00

0.50

FP32: faster
0.00
1.E+00

1.E+01

1.E+02

1.E+03

1.E+04Lambda-1/Lambda-2
Ratioof λ1/ λ2

1.E+05

1.E+06[KN et al. 2018]

29

Ratio of FP32(SP)/FP64(DP) : CRS

λ1

λ2
λ1
∇ ⋅ ( λ∇φ ) + f = 0
Intel Xeon BDW
1 Node: 18 cores x 2 soc’s

Ratio ofRatio
FP32/FP64

Iterations● & Time△ for ICCG 3.00
λ1/λ2 , 1283 DOF, CRS
2.50
Ratio<1 ⇒ FP32 is faster

Iterations
sec.
●Iterations
△Time

2.00

+20% iterations
by FP32

1.50

1.00

0.50

-40~-45% Time by FP32
0.00
1.E+00

1.E+01

1.E+02

1.E+03

1.E+04
1.E.+04

Lambda-1/Lambda-2
Ratioof λ1/ λ2

1.E+05

1.E+06[KN et al. 2018]

30

Ratio of FP32(SP)/FP64(DP) : CRS

λ1

λ2
λ1
∇ ⋅ ( λ∇φ ) + f = 0
Intel Xeon BDW
1 Node: 18 cores x 2 soc’s

Ratio ofRatio
FP32/FP64

Iterations● & Time△ for ICCG 3.00
λ1/λ2 , 1283 DOF, CRS
2.50
Ratio<1 ⇒ FP32 is faster

Iterations
sec.
●Iterations
△Time

2.00

+20% iterations
by FP32

1.50

1.00

0.50

-40~-45% Time by FP32
0.00
1.E+00

1.E+01

1.E+02

1.E+03

1.E+04
1.E.+04

Lambda-1/Lambda-2
Ratioof λ1/ λ2

1.E+05

1.E+06[KN et al. 2018]31[Sakamoto et al. 2020]
Results on Intel Xeon BDW λ1= λ2
N=1283, ■: CPU，■: Memory，●：Time
400.00Power: Watt

1.80

700.00

Energy: Joule

1.80

1.60

1.60

600.00
1.40

0.80
0.60

100.000.40

0.00
SingleFP32
SingleDoubleFP64
Double1.20
400.00

1.00

sec.

1.00
200.001.40
500.00

Energy Consumption (J)

1.20

sec.

Power Consumption (W)300.00300.000.80
0.60

200.000.40

0.20

100.00

0.00

0.00

0.20

FP32
SingleSingle0.00

FP64
DoubleDouble32

Approximate Computing with
Low/Adaptive/Trans Precision• Accuracy verification is important, especially for computation withlower/mixed precision.
• A lot of methods for accuracy verification have been developed forproblems with dense matrices– But very few examples for13:30-13:45
sparse matrices& H-matrices
Takeshi
Iwashita (Hokkaido U.)• Generally speaking, processes
for accuracy
verificationis very U.)
13:45-14:00
Takahiro
Katagiri (Nagoya
expensive
15:35-15:50 Takeshi Ogita (Waseda U.)– Sophisticated Method needed– Automatic Selection of Optimum Precision by Technology of AT (Auto Tuning)• Accuracy Verification of Sparse Linear Solvers [Ogita, Nakajima 2019]

Adaptive Precision Computing with FP21/FP42

33

In recent years, the usefulness of low-precision floating-point
representation has been studied in various fields such as machinelearning. Low accuracy can be expected to have effects such as
shortening calculation time and reducing power consumption. For
example, in an application with a memory bandwidth bottleneck, theeffect of reducing the calculation time by reducing the amount ofmemory transfer is significant. However, in fields such as iterative
methods, it is common to use FP64 because the calculation accuracy
strongly affects the convergence, and there are few application
examples of low-precision arithmetic. This study investigates theHeat Conduction withapplicability of low-precision representation to the Krylov subspace
Heterogeneous Material Property
and stationary iterative methods. In this research, we focus on the FP32,
120
30
FP64-FP64 FP42-FP64
FP64-FP64
FP16, and FP42, FP21, which are not standardized by IEEE754.
FP32-FP32 FP21-FP32
FP42-FP64
100
25
FP32-FP32
Developed method has been evaluated for ICCG solver, which solves
FP21-FP32
20
80
linear equations derived from 3D FVM code for steady-state head
15
60
conduction with heterogeneous material property (λ1=100, λ2=100~109).
1040
Generally, computation with lower precision (e.g. FP32-FP32, FP215
20FP32) becomes unstable, if condition number of the coefficient matrix
00
1.E+00
1.E+01
1.E+02
1.E+03
1.E+04
1.E+00
1.E+03
1.E+06
1.E+09is larger (λ2 is larger), FP21-FP32 provides the best performance if λ2
λ2
λ2
is up to 104. (“FP21-FP32” means “matrices are in FP21, and vectors
Computation Time for ICCG Solver
Various Types of Precisions on Intel Xeon Cascadelake are in FP32)
Computational time [s]Computational time [s]Masatoshi Kawai (kawai@cc.u-tokyo.ac.jp)

34h3-Open-BDEC Innovative Software Platform
for Integration of (S+D+L) on the BDEC
System, such as Wisteria/BDEC-01• “Three” Innovations

14:00-14:15 Hiromichi Nagao (U.Tokyo)– New Principles for Numerical
14:15-14:30 Takashi Shimokawabe (U.Tokyo)
Analysis by Adaptive Precision,
16:35-16:50 Hayato Shiba (U.Hyogo)
Automatic Tuning & Accuracy
Verification– Integration of (S+D+L) by
Hierarchical Data Driven Approach(hDDA)– Software & Utilities for
Heterogenous Environment, such as
Wisteria/BDEC-01

Prediction of CFD Simulation by ML/CNN

35

Takashi Shimokawabe (shimokawabe@cc.u-tokyo.ac.jp)
Simulations: LBM

CNN Predictions

Computational fluid dynamics (CFD) is widely used in science andengineering. However, since CFD simulations requires a large
number of grid points and particles for these calculations, these kindsof simulations demand a large amount of computational resources
such as supercomputers. Recently, deep learning has attracted
attention as a surrogate method for obtaining calculation results by
CFD simulation approximately at high speed. We are working on a
project to develop a parallelization method to make it possible toapply the surrogate method based on the deep learning to large scale
geometry. Unlike the model parallel computing, the method we are
currently developing predicts large-scale steady flow simulation
results by dividing the input geometry into multiple parts andapplying a single small neural network to each part in parallel. Thismethod is developed based on considering the characteristics of CFD
simulation and the consistency of the boundary condition of eachdivided subdomain. By using the physical values on the adjacentsubdomains as boundary conditions, applying deep learning to each
subdomain can predict simulation results consistently in the entire
computational domain. It is possible to predict the simulation resultsin about 36.9 seconds by the developed method, compared to about
286.4 seconds by the conventional numerical method. In addition tothis, we are also attempting to develop a method for fast predictionof time evolution calculations using deep learning.

Machine learning slow molecular dynamics
Our proposal ー BOnd Targeting Network (BOTAN)INPUTOUTPUTnodes = particle typenodes = particle motion
Graph
NeuralNetworks

edges= relative positions

edges = relative motion

H. Shiba, M. Hanai, T. Suzumura, and T. Shimokawabe, arXiv:2206.14024 (2022)

37

h3-Open-BDEC Innovative Software Platform
for Integration of (S+D+L) on the BDEC
System, such as Wisteria/BDEC-01• “Three” Innovations– New Principles for Numerical
Analysis by Adaptive Precision,
Automatic Tuning & Accuracy
Verification– Integration of (S+D+L) by
16:05-16:20 Hisashi Yashiro (NIES)
Hierarchical Data Driven Approach
16:20-16:35 Shinji Sumimoto (U.Tokyo)(hDDA)– Software & Utilities for
Heterogenous Environment, such as
Wisteria/BDEC-01

38

Wisteria/BDEC-01: The First “Really
Heterogenous” System in the World

39

h3-Open-SYS/WaiIO-Socket• Wisteria/BDEC-01– Aquarius (GPU: NVIDIA A100)– Odyssey (CPU: A64FX)• Combining Odyssey-Aquarius– Single MPI Job over O-A is
impossible• Connection between OdysseyAquarius– IB-EDR with 2TB/sec.
– Fast File System– h3-Open-SYS/WaitIO-Socket• Library for Inter-Process
Communication through IBEDR with MPI-like interface

WaitIO-Socket

40• Integration of (Simulation/Data/Learning)– Wisteria/BDEC-01– h3-Open-BDEC• Applications on Wisteria/BDEC-01 with h3-OpenBDEC– Seismic Wave Propagation– Global Atmosphere– International/Domestic Collaborations• Integration of (Simulation/Data/Learning) and Beyond• Summary41

3D Earthquake Simulation
with Real-Time Data
Observation/Assimilation
Simulation of Strong Motion (Wave
Propagation) by 3D FDM

JDXnetJDXnet[c/o Prof. T.Furumura(ERI/U.Tokyo)]42

Earthquake simulation is always withuncertainty• Subsurface/Underground Structure– Heterogenous, Random, Stochastic– Fluctuations• Traditional Simulations– Forward Simulations• Integration of Simulation/Observation isessential• New Types of Methods for Simulations
combined with Data Assimilation/RealTime Observation is under development– Forecast by Simulations, Correction by[c/o Prof. T. Furumura,
Data Assimilation
ERI/U.Tokyo]

43

Simulations of Long-Period Ground Motion[Furumura et al.]• 3D Equation of Motions solved by
FDM (Finite-Difference Method)
1  ∂σ xp+ ρ  ∂xn −1/2

n
p

v =v

n −1
p

+

n −1/2∂σ yp∂y∂σ zpn −1/2 
+
 ∆t
∂z 

• Seism3D– Staggered Discretization in Space/Time– 4th order in Space– 2nd order in Time (Explicit Time Marching)– OpenMP + MPI, Fortran( p = x, y , z )44

3D Earthquake Simulation
with Real-Time Data
Observation/Assimilation
Simulation of Strong Motion (Wave
Propagation) by 3D FDM

JDXnetJDXnet[c/o Prof. T.Furumura(ERI/U.Tokyo)]

Real-Time Sharing of Seismic Observation ispossible in Japan by JDXnet with SINET
Japan Data eXchange network• Seismic Observation Data (100Hz/3-dir’s/O(103) observation points)
by JDXnet is available through SINET in Real Time– O(102) GB/day: available at Website of NIED– O(105) pts in future including stations operated by industry[c/o Prof. H.Tsuruoka(ERI/U.Tokyo)]45

Real-Time Assimilation of “Observation+Computation” inSeismic Wave Propagation [c/o Oba & Furumura]・Data Assimilation of Wave Propagation
by “Optimal Interpolation Technique”
Residual
Obs.
Comp.

Assim. Comp.
Comp.

46

（A）Pure S （B）A+S: Time Step
: Weighting Matrix

10 sec.

Assim.

（A) Pure Simulation: Wave Propagation
simulation（B）Assimilation+Sim.

：Obs. Pts. (No info for Epicenter needed)

20 sec.

30 sec.
Epicenter

46

46

Real-Time Assimilation of “Observation+Computation” inSeismic Wave Propagation [c/o Oba & Furumura]・Data Assimilation of Wave Propagation
by “Optimal Interpolation Technique”
Residual
Obs.
Comp.

Assim. Comp.
Comp.

47

（A）Pure S （B）A+S: Time Step
: Weighting Matrix

10 sec.

Assim.

（A) Pure Simulation: Wave Propagation
simulation（B）Assimilation+Sim.

：Obs. Pts. (No info for Epicenter needed)

20 sec.

30 sec.
Epicenter

47

47

Starting from (A+S:
Assim+Sim.) to (Pure
S: Pure Simulation)[c/o Prof. T. Furumura, ERI/U.Tokyo]

Residual
Obs.
Comp.

Assim. Comp.
Comp.

Assim.
: Wave Propagation
simulation: Time Step48

: Weighting Matrix

3D Earthquake Simulation with Real-Time Data
Observation/Assimilation on Wisteria/BDEC-01
Output&Movie

JDXnetObserved Data

Data/Learning
Nodes: Aquarius

FilterVisualizer

Intel Ice Lake + NVIDIA A100

7.20 PF, 578.2 TB/s

h3-Open-SYS/WaitIO-Socket
Obs. Data

External Server
for Observed
Data

2.0 TB/s

Simulation Nodes:
Odyssey
Fujitsu/Arm A64FX
25.9PF, 7.8 PB/s

Seism3D/
OpenSWPC-DAF
Mesh DataUG Model49

Communications by WaitIO-Socket
Aquarius: SEND

[Kasai et al. 2021]

Odyssey: RECV

program dmy_filter<省略: 型宣言等>
call mpi_init (ierr)call mpi_comm_size (MPI_COMM_WORLD, nprocs, ierr)call mpi_comm_rank (MPI_COMM_WORLD, myrank, ierr)call WAITIO_CREATE_UNIVERSE (WAITIO_COMM_UNIVERSE, ierr)
if (myrank==0) thenopen(100,file='./obsfile_list.txt’, form=‘formatted’, status=‘old’, iostat=ierr)
do i=1,300<省略: obsデータ読み込み処理>
print *,"Send obs data ...... "call WAITIO_MPI_ISEND (NTMAX1_o, 1,
WAITIO_MPI_INTEGER, 2,1, WAITIO_COMM_UNIVERSE,req(1,1), ierr)call WAITIO_MPI_ISEND (DT_o,1,
WAITIO_MPI_FLOAT,
2,2, WAITIO_COMM_UNIVERSE,req(1,2), ierr)call WAITIO_MPI_ISEND (NST_o,1,
WAITIO_MPI_INTEGER, 2,3, WAITIO_COMM_UNIVERSE,req(1,3), ierr)call WAITIO_MPI_ISEND (AT_o,1,
WAITIO_MPI_FLOAT,
2,4, WAITIO_COMM_UNIVERSE,req(1,4), ierr)call WAITIO_MPI_ISEND (T0_o,
1,
WAITIO_MPI_FLOAT,
2,5, WAITIO_COMM_UNIVERSE,req(1,5), ierr)call WAITIO_MPI_ISEND (ISO_X_o, NSMAX,WAITIO_MPI_INTEGER, 2,6, WAITIO_COMM_UNIVERSE,req(1,6), ierr)call WAITIO_MPI_ISEND (ISO_Y_o, NSMAX,WAITIO_MPI_INTEGER, 2,7, WAITIO_COMM_UNIVERSE,req(1,7), ierr)call WAITIO_MPI_ISEND (ISO_Z_o, NSMAX,WAITIO_MPI_INTEGER, 2,8, WAITIO_COMM_UNIVERSE,req(1,8), ierr)call WAITIO_MPI_ISEND (ISTX_o,NST,WAITIO_MPI_INTEGER, 2,9, WAITIO_COMM_UNIVERSE,req(1,9), ierr)call WAITIO_MPI_ISEND (ISTY_o,NST,WAITIO_MPI_INTEGER, 2,10,WAITIO_COMM_UNIVERSE,req(1,10),ierr)call WAITIO_MPI_ISEND (ISTZ_o,NST,WAITIO_MPI_INTEGER, 2,11,WAITIO_COMM_UNIVERSE,req(1,11),ierr)call WAITIO_MPI_ISEND (STC_o,
6*NST,WAITIO_MPI_CHAR,
2,12,WAITIO_COMM_UNIVERSE,req(1,12),ierr)call WAITIO_MPI_ISEND (VxAll_obs,NST*NOBS_LEN,WAITIO_MPI_FLOAT,
2,13,WAITIO_COMM_UNIVERSE,req(1,13),ierr)call WAITIO_MPI_ISEND (VyAll_obs,NST*NOBS_LEN,WAITIO_MPI_FLOAT,
2,14,WAITIO_COMM_UNIVERSE,req(1,14),ierr)call WAITIO_MPI_ISEND (VzAll_obs,NST*NOBS_LEN,WAITIO_MPI_FLOAT,
2,15,WAITIO_COMM_UNIVERSE,req(1,15),ierr)call WAITIO_MPI_WAITALL (15,req, status, ierr)call sleep(1)enddoclose (100)endifcall WAITIO_FINALIZE (ierr)call mpi_finalize (ierr)
end

50

51

Example: Off Niigata 2007 Mw6.6 Earthquake• Observed Data: Stored in External Server• Aquarius receives observed data, and apply filtering• “Data Assimilation + Simulation (A+S)”, and “Forecast by Simulation(Pure S)” are separated codes, while same number of computing
nodes were used on Odyssey• Movies were created after simulations (O(10) sec.)• Seism3D/OpenSWPC-DAF– 3D FDM + Optimal Interpolation Technique
n −1/2n −1/2∂σ yp∂σ zpn −1/2 
1  ∂σ xp
for Data Assimilationnn −1
vp = vp + ++ ∆t ( p = x, y, z )∂y∂z ρ  ∂x– Each Mesh: 240m×240m×240m: Time Step
Residual
Assim. Comp.
Obs.
Comp.
: Weighting Matrix– 1,920×1,920×240 meshes（8.85×108）
Comp.
Assim.
– 460.8 km×460.8 km×57.6 km: Wave Propagation
simulation

52

Off Niigata 2007 Mw6.6 Earthquake

Epicenter■ Hi-net (Short Period) 349 pts● F-net (Broadband)18 pts[c/o Prof. T. Furumura,
ERI/U.Tokyo]

53

Off Niigata 2007 Mw6.6 Earthquake

Epicenter■ Hi-net (Short Period) 349 pts● F-net (Broadband)18 pts[c/o Prof. T. Furumura,
ERI/U.Tokyo]

Results at Kotoh ▲ (N.KOTH)

Data Assimilation + Pure Simulation/Forecast N 35°37.0’
E 139°46.9’

482 K-NET, KiK-net Observation

Assimilation by 70s forecast(A+S)⇒(Pure S)@30s(A+S) (Pure S)

Epicenter

50s

70s

Input
P

S

Point at Kotoh(Tokyo)
▲

90s

110s

Obs

70s for Assimilation
54

Future Directions towards
Integration of (S+D+L)• Accurate Prediction of Seismic Wave
Propagation with Real-Time Data
Observation/Assimilation– Emergency Info. for Safer Evacuation– 10x faster than real phenomena with
O(103) nodes of supercomputers

55

(A+S) for 60-sec and (Pure S) for 240-sec
on Odyssey, ①②：without/with scalar tuning
1,920 nodes needed for the Pure-S in < 24-sec.
(A+S) 60 sec., (Pure S) 240 sec.
■ Pure S■ A+S■ Preconditioning

16-nodes64-proc’s

32
128

64
256

128
512

256
1,024

512
2,048

1,024
4,096

1,536
6,144

1,920
7,680

56

Future Directions towards
Integration of (S+D+L)• Accurate Prediction of Seismic Wave
Propagation with Real-Time Data
Observation/Assimilation– Emergency Info. for Safer Evacuation– 10x faster than real phenomena with
O(103) nodes of supercomputers• 3D Underground Model– Heterogeneous, Observation is difficult– Inversion analyses of seismic waves areimportant for prediction of structure ofunderground model– ML may be utilized for acceleration of thisprediction based on analyses of small
earthquakes in normal time (e.q. Mw < 3.0)
– More sophisticated DA method (e.g. 4DVar)

57

58

Construction of 3D Underground Modelby Data Assimilation/Machine Learning• Local models with smaller meshes should be used
Replica Exchange
Monte Carlo
Nagao et al.

2nd Order Adjoint
Nagao et al.

Large-Scale ML
Ichimura, Fujita
SC22 GB Finalists

59Web-based Simulation
System for Outreach
Activities• Web-based simulation system forenlightenment of disasterprevention/mitigation awarenessusing “3D Earthquake Simulation
with Real-Time Data
Observation/Assimilation”• Users including general citizens
and high-school/junior-high-school
students, access the web-serveron the mdx system, andmanipulate simulations on the
Wistera/BDEC-01.
• The framework can be utilized invarious types of applications.

60

Web-based Simulation System for Outreach
Activities

61

Simulation (cm/sec)

Observation (cm/sec)

Web-based Simulation
System for Outreach
Activities (Prototype)

62• Integration of (Simulation/Data/Learning)– Wisteria/BDEC-01– h3-Open-BDEC• Applications on Wisteria/BDEC-01 with h3-OpenBDEC– Seismic Wave Propagation– Global Atmosphere– International/Domestic Collaborations• Integration of (Simulation/Data/Learning) and Beyond• Summary

63

Multiphysics Coupler• Traditional Coupler: ppOpen-MATH/MP– Weak-Coupling of Multiple (usually two) Applications• Each application does a single computation• Ocean-Atmosphere• Fluid-Structure

64

Atmosphere-Ocean Coupling by
ppOpen-MATH/MP (Previous Project)• High-resolution global atmosphere-ocean coupled simulation by NICAM(Atmosphere) and COCO (Ocean) through ppOpen-MATH/MP on the K
computer is achieved.
– ppOpen-MATH/MP is a coupling software for the models employing various
discretization method.

[C/O M. Satoh (AORI/UTokyo)@SC16]

65

El Niño Simulations: NICAM-COCO Coupling
Mechanism of the Abrupt Terminate of Super El Niño in1997/1998 has been revealed by Atmosphere-Ocean
Coupling Simulations for the Entire Earth using ppOpenHPC on the K computer
El NinoNormalNormalEl Ninoh3-Open-UTIL/MP (h3o-U/MP)
Extended Multiphysics Coupler66

h3-Open-UTIL/MP +h3-Open-SYS/WaitIO-Socket
Available in June 2022
Aquarius

Odyssey

IB-EDRMay 2021: MPI Only

June 2022: Coupler＋WaitIO
67

Atmosphere-ML Coupling[Yashiro (NIES), Arakawa (ClimTech/U.Tokyo)] Motivation of this experiment Tow types of Atmospheric models: Cloud resolving VS Cloudparameterizing
 Could resolving model is difficult to use for climate simulation
 Parameterized model has many assumptions
 Replacing low-resolution cloud processes calculation with ML!

Odyssey

Aquarius

ML App(Python)Input

h3-Open-UTIL/MP
Replacing
Coupling without
Physical process Coupling with
Grid Remapping
Grid Remapping
Output(Coupler)
this part+
Low Resolution Atmospheric Model with AI
High Resolution Atmospheric Model(Convection-Parameterization Mode)(Convection-Resolving Mode)h3-Open-SYS/WaitIOSocket Coupling Phase 2
Coupling Phase 1
75%
25%
~0%
Replacing Physical Process
Training with high-resolution
NICAM data

in Low-Resolution NICAM
with Machine Learning

Diagram of applying ML to an atmospheric modelEnsemble-Based Data Assimilation
 Ensemble Data Assimilation of Atmospheric model Data Assimilation + Ensemble Calculation
 Effective technique to enhance forecast accuracy

 Problems with high-resolution ensemble
 High-resolution ensemble requires significant resources (time and/or CPU)
 Potential for poor reproducibility of large-scale fields

14km ensemble DA

640 process x 64 member

Observation
Assimilation
EnsembleEnsemble Coupling Low-resolution ensemble + high-resolution calculation
 224km ensemble (40 process x 64 ensemble) + 14km simulation( >= 640 process) Fewer computational resources More accurate reproduction of large-scale fields
224km ensemble DA

14km filtered nudging

Coupling
data exchange andgrid remappingby the coupler

Observation
Assimilation
Ensemble

40 process x 64 member>= 640 process

71

How to run the workloads• Total Number of Nodes– Odyssey: 7,680 nodes: not so crowded– Aquarius: 45 nodes, 360 GPUs, very
crowded• One node of Aquarius is reserved
for this type of workload on the
integration of (S+D+L)• 2 separate jobs (Odyssey, Aquarius)
should be submitted• If both jobs “grab” resources,
execution starts.
• More flexible (& complicated) policy
needed72• Integration of (Simulation/Data/Learning)– Wisteria/BDEC-01– h3-Open-BDEC• Applications on Wisteria/BDEC-01 with h3-OpenBDEC– Seismic Wave Propagation– Global Atmosphere– International/Domestic Collaborations• Integration of (Simulation/Data/Learning) and Beyond• Summary

73

Uncertainty Quantification of Extreme
Weather Prediction
Y. Sawada (U.Tokyo)

Prediction of Marine “Ecosystem”
Prediction of the position and size of fish schools by predicting physical, chemical, andmicrobial fields in the coastal ocean (RIKEN, U.Tokyo, Osaka U., Tohoku U.)

Continuous automatic obs./
Environmental omics analysis ofwater sampling data

Machine Learning（RIKEN, U.Tokyo）

環境DNA測定

Simulations

Sato e al. (2018)（RIKEN, Osaka U.）
Ocean/Weather/Rivers

Ocean Simulation- Wide-Area- High-Resolution- Long-TermPrevious Work: Accurate Predictionof Red Tide using Sim/Obs/ML
Density of Chlorophyll a (ug/l)Observation（RIKEN, Tohoku U.）“Big Data of Coastal
Areas” for ML
Consideration ofSurface/Spatial Structure

Simulation/Data Assimilation by Advanced
Bio-Chemical Models

Observation

Prediction

Development of the System
for Prediction ofFish Schools

FY.2023-2025, JHPCN Project
Innovative Computational Science by Integration of
Simulation/Data/Learning under Heterogeneous Computing Env.

 Jülich Supercomputing Centre（JSC）
 Rudjer Boskovic Institute, Centre for Informatics andComputing, Croatia
 Friedrich-Alexander-Universität Erlangen-Nürnberg（FAU）
 French Atomic Energy Commission (CEA)75

76

Target Applications• JSC– Terrestrial Systems Modeling Platform (TSMP)• Coupling: Groundwater Flow & Atmosphere• https://www.terrsysmp.org/– Chebyshev Accelerated Subspace Eigensolver (ChASE)• Quantum Chemistry, Heterogeneous Environment• https://github.com/ChASE-library– Brain Aneurysm Simulations• Multiscale，Multiphysics• CFD Codes (m-AIA) at JSC– https://www.hpccoe.eu/2021/06/04/m-aia/• CEA– Selection of inhibitors of the SARSCoV-2 Main Protease– BigDFT + Polaris/Gromacs

77

Collaboration since FY.2018
High resolution simulation of cardiac
electrophysiology on realistic whole-heart geometries• Coordinated electrical activities are vital for the heart• Computer-enabled “in-silico” experimentation is important for studying
the physiological mechanism and the cause of diseases• Need biophysically accurate simulations of cardiac electrophysiology– Extremely fine spatial and temporal resolutions ⟶ huge computations– Realistic 3D geometry of the heart ⟶ unstructured computational mesh

78

• Integration of (Simulation/Data/Learning)– Wisteria/BDEC-01– h3-Open-BDEC• Applications on Wisteria/BDEC-01 with h3-Open-BDEC– Seismic Wave Propagation– Global Atmosphere– International/Domestic Collaborations• Integration of (Simulation/Data/Learning) and
Beyond• Summary

79

Anything is possible with WaitIO
WaitIO over Internet/cloud is possible
CPU

A64FX
Arm

GPU

NVIDIA
X86

Intel
AMD
Arm

AMD

Intel

h3-Open-SYS/WaitIO

80

2001-2005
Hitachi SR8000
1,024 GF

SR80002006-2010
Hitachi SR11000
J1, J2
5.35
18.8 TF
IBMTF,
Power5+

Hitachi
SR2201

Hitachi
SR8000/MPP

307.2GF

2,073.6 GF

HARP-1E

2011-20152016-2020

Hitachi SR16K/M1
Yayoi
54.9
TF
IBM Power7

2021-2025

2026-2030

Pseudo Vector
Multicore CPUIntel CLX
OBCX(Fujitsu)

GPU, Accelerators

6.61 PF

SR8000

Hitachi HA8000
T2K Todai

OakforestPACS (Fujitsu)

140 TF25.0 PFAMD OpteronSupercomputers@ITC/U.Tokyo
2,600+ Users
55+% outside of U.Tokyo

OFP-II
75+ PF

AcceleratorsIntel Xeon Phi

Fujitsu FX10 A64FX,
Oakleaf-FX Intel Icelake+
NVIDIA A100
1.13 PFSPACR64 IXfx
ReedbushU/H/L (SGI-HPE)
3.36 PFIntel BDW +NVIDIA P100

BDEC02

Fujitsu
33.1 PF

250+ PF

Accelerators
Mercury
Accelerators
Ipomoea-01 25PB

Ipomoea03

Ipomoea-02

OFP-II (1/2)
Bid opened on November 9th, Fujitsu awarded• Group-A: CPU Only: Intel Xeon Max 9480 (SPR)– Node: Intel Xeon Max 9480 (1.9GHz, 56c) x 2• 6.8 TF, 128 GiB, 3,200 GB/sec (HBM2e only)– Total• 190 nodes, 1.3 PF, IB-NDR 200• 372 TB/sec for STREAM Triad (Peak: 608 TB/sec)• Group-B: CPU+GPU: NVIDIA GH200– Node: NVIDIA GH200 Grace-Hopper Superchip• Grace: 72c, 2.9 TF, 111.7 GiB, 512 GB/sec (LPDDR5X)• H100: 66.9 TF DP-Tensor Core, 89.4 GiB, 4,022 GB/sec (HBM3)• NVMe SSD for each GPU: 1.9TB, 8.0GB/sec– Total (Aggregated Performance: CPU+GPU)• 1,120 nodes, 78.2 PF, 5.07 PB/sec, IB-NDR 200
81

OFP-II (2/2)
Bid opened on November 9th, Fujitsu awarded• File System: DDN EXA Scalar, Lustre FS– 10.3 PB (NVMe SSD) 1.0TB/sec, “Ipomoea-01” with 26 PB is also available• All nodes in Group-A/B are connected with Full Bisection Bandwidth– (400Gbps/8)×(32×20＋16×1) = 32.8 TB/sec• Operation starts in January 2025, h3-Open-SYS/WaitoIO will be
adopted for communication between Group-A and Group-B
IB-NDR（400Gbps）
IB-HDR（200）
IB-NDR200（200）Group-AIntel Xeon Max(HBM2e) 2 x 190
1.3 PF, 608 TB/sec

Group-B
NVIDIA GH200 1,120
78.2 PF, 5.07 PB/sec

Ipomoea-01

Common Shared Storage
26 PB

File System
DDN EXA Scaler
10.3 PB, 1.0TB/sec82

83

Anything is possible with WaitIO
WaitIO over Internet/cloud is possible
CPU

A64FX
Arm

GPU

OthersNVIDIA
X86

Intel
AMD
Arm

AMD

Intel

h3-Open-SYS/WaitIO

Sambanova
CerebrasGraphcoreetc.

Quantum

84

System SW for QC-HPC Hybrid Environment (1/2)• Quantum Computer = Accelerator of Supercomputers: QC-HPC Hybrid• System SW for Efficient & Smooth Operation of QC (Quantum Computer,including simulators on supercomputers)-HPC Hybrid Environment– QHscheduler: A job scheduler that can simultaneously use multiple computerresources distributed in remote locations– h3-Open-BDEC/QH: Coupling to efficiently implement and integrate communicationand data transfer between QC-HPC on-line and in real time
– Collaboration with RIKEN R-CCS, funded by Japanese Government• Target Application– AI for HPC, combined workload• Simulations in Computational Science• Quantum Machine Learning– Quantum Simulations, Error Correction

QHscheduler
HPC(1)

HPC HPC
(2)
(3)

QC
(a)

QC
(b)

h3-Open-BDEC/QH

QC
On
HPC

85

System SW for QC-HPC Hybrid Environment (2/2)• Innovations– This is the world's first attempt
to link multiple
supercomputers and quantum
computers installed at
different sites in real time.
– In particular, by using multiple
QCs simultaneously, it ispossible to form a virtual QC
with higher processing
capacity.
• Many people are thinking aboutsame thing all over the world– This idea can be extended to
any types of systems

QC
On
HPC

HPC
(2)

HPC
(1)

QC
(b)

QHscheduler
h3-Open-BDEC/QH

HPC
(3)

QC
(a)
AI

86

• Integration of (Simulation/Data/Learning)– Wisteria/BDEC-01– h3-Open-BDEC• Applications on Wisteria/BDEC-01 with h3-Open-BDEC– Seismic Wave Propagation– Global Atmosphere– International/Domestic Collaborations• Integration of (Simulation/Data/Learning) and Beyond• Summary87

Summary• Integration of (Simulation/Data/Learning) at ITC/U.Tokyo• Wisteria/BDEC-01• h3-Open-BDEC• Domestic/International Collaborations• Challenges towards Quantum Computing

88

Activities• Publications– Peer Reviewed Journal Papers/Proceedings– Presentations (International/Domestic)– Invited Talks (International/Domestic)– Book58
89
33
1• Awards– Shinji Sumimoto, Takashi Arakawa, Yoshio Sakaguchi, Hiroya Matsuba, Hisashi
Yashiro, Toshihiro Hanawa, and Kengo Nakajima, A System-Wide
Communication to Couple Multiple MPI Programs for Heterogeneous
Computing, The 23rd International Conference on Parallel and Distributed
Computing, Applications and Technologies(PDCAT’ 22), Best Paper Award– Kengo Nakajima, SCA (Supercomputing Asia) HPC Pioneer & Achievement
Award (Japan), 2023

89

Invited/Keynote Talks by KN after April 2023
• HPC Workshop for Nuclear Explosion Monitoring 2023, CTBTO (Preparatory
Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization), Vienna,
Austria, May 2023
• 2023 Japan Geoscience Union Meeting, Makuhari, Japan, May 2023
• The 6th International Workshop on Nonhydrostatic Models (NHM-WS 2023),
Sapporo, Japan, August 2023
• Accelerated Computing Summit 2023 (OACS), Online, October 2023
• The Initiative for Design Evolution, AIST, Online, October 2023
• 14th Workshop on Latest Advances in Scalable Algorithms for Large-Scale
Heterogeneous Systems (ScalAH23) in conjunction with SC23, Denver, CO, USA,
November 2023
• 24th Northeast Asia Symposium 2023, Guangzhou, China, November 2023
• WCCM 2024 (16th World Congress on Computational Mechanics)/PANCAM 2024
(4th Pan American Congress on Computational Mechanics), Vancouver, Canada,
July 2024 (planned)

90Final Goal stated in the Proposal (Nov. 2018)• We aim to reduce the amount of computations and power
consumption by more than 10 times while maintaining the sameaccuracy as conventional methods in multi-level simulations that
integrate (S+D+L).
– Mixed Precision/Adaptive Precision– Machine Learning, Hierarchical Data Driven Approach– Heterogeneous Computing• Self evaluation so far … although we still have 4 months left
– ○○

91

Kengo Nakajima (The University of Tokyo)
Integration of Simulation/Data/Learning and Beyond
Takeshi Iwashita (Online) (Hokkaido University, Kyoto University)
Research activities of Hokkaido University group for next-generation linear
13:30 - 13:45
solvers -mixed precision computing, accelerators, subspace correction
techniquesTakahiro Katagiri (Nagoya University)
13:45 - 14:00
Exploring AI for Auto-tuning through Sparse Matrix Image Information
Hiromichi Nagao (The University of Tokyo)
14:00 - 14:15 Deep Learning to Extract Earthquakes and Low-Frequency Tremors inContinuous Seismic Waveforms
Takashi Shimokawabe (The University of Tokyo)
14:15 - 14:30 Fast Prediction Methods for Fluid Simulation Results Using Deep Neural
Networks
Richard Vuduc (Georgia Institute of Technology, USA)
14:30 - 15:00(Keynote Talk) Data-movement accelerators for scientific computing problems
15:00 - 15:20 (Discussion)
15:20 - 15:35 (Break)
13:00 - 13:30

92

Takeshi Ogita (Online) (Waseda University)
15:35 - 15:50 Verified Solutions of Large Sparse Linear Systems Arising from 3D Poisson
Equation
Masatoshi Kawai (Online) (Nagoya University)
15:50 - 16:05
Effectiveness of low/adaptive precision with ICCG method
Hisashi Yashiro (Online) (National Institute for Environmental Studies)
16:05 - 16:20 Algorithmic transformation from physical models to data-driven models using
the coupling library: a case of a climate model
Shinji Sumimoto (The University of Tokyo)
16:20 - 16:35 h3-Open-SYS/WaitIO: A System-wide Heterogeneous Communication Library to
Couple Multiple MPI programs
Hayato Shiba (Online) (University of Hyogo)
16:35 - 16:50
Deep learning of simulated glassy dynamics
Xing Cai (Simula Research Laboratory, Norway)
16:50 - 17:20(Keynote Talk) Towards high-performance unstructured-mesh computations
17:20 - 17:40 (Discussion)
17:40 - 17:45 (Closing)

93

Invited Speakers• Richard Vuduc– Professor– School of Computational Science and Engineering,
Georgia Institute of Technology, USA• Xing Cai– Professor/Chief Research Scientist/Head of Department– High Performance Computing Department, Simula
Research Laboratory, Norway