GPU-accelerated Multiphysics-based Seismic Wave Propagation
Simulation and its Surrogate Model with Machine Learning
Ryota Kusakabe1, Tsuyoshi Ichimura1, Kohei Fujita1, Muneo Hori2, Lalith Wijerathne1

CG-based method to solve the equation• Utilizing domain specific knowledge, a surrogate modelof complex soil liquefaction simulation was successfully
constructed with relatively a small neural network• Training data were generated such that its statistical𝑛information is similar to that of the target soil model,
leading to efficient training with a relatively small number of training data

GPU-accelerated method• We developed a high-performance method forseismic wave propagation simulation with finite
element method (FEM) considering a complex
multiphysics phenomenon, soil liquefaction [1]• Load balancing scheme that considers GPU
architecture and characteristics of soil liquefaction
simulation was adopted• A 10.7-fold speed up over CPU-based
implementation was achieved

Preconditioner• Adaptive CG method is used𝛿𝒖 ⇐ 𝛿𝒖0 , 𝒓 ⇐ 𝒃 − 𝑨𝛿𝒖Solve 𝑨𝒛 = 𝒓• In preconditioner, preconditioning
do while (| 𝒓 |/| 𝒃 | < 𝜀tol ) with PCG on
equation 𝑨𝒛 = 𝒓 is solved withSolve 𝑨𝒛 = 𝒓
a coarse meshthe preconditioned CG method
use 𝒛 to update 𝛿𝒖
Use as initial solution• Multigrid method and mixedprecision are used in preconditioner 𝒓 ⇐ 𝒃 − 𝑨𝛿𝒖
Solve 𝑨𝒛 = 𝒓
end whilewith PCG on
to reduce computational cost
the original meshwithout compromising the solution[In double precision]accuracy[In single precision]• Whole simulation is parallelized with MPI and OpenACC

Load Balancing
Conventionalpartitioning

Neural network based surrogate model• HQC (high quality computing) consideringuncertainty of information (e.g., soil properties,ground structure) is important→ Many cases of 3D simulation are required• Not realistic to perform 3D simulation for
hundreds times, even with the developed method• We constructed a neural network (NN) basedsurrogate model to enable faster judgement of soilliquefaction with over 90% accuracy

Multiphysics-based Simulation
Multiphysics Problem• Complex dynamic multiphysics 𝑧

1D spring𝑧′

problem in which soil behaves
highly nonlinearly as it gets𝑥′ 𝜓 𝑦′𝜙liquefied and phase transition𝑦from solid to liquid occurs
𝜃𝑥• Strain space multiple mechanism
model (Iai, et al 1992, Iai 1993): 3D constitutive
law that is expressed as a superposition of 300 of
1D springs• Computation can be unstable, but it is overcome
with a stabilization method (Kusakabe et al. 2021)

Governing Equation
Motion equation of soil

𝜕 2 𝒖 𝜕𝝈
𝜌
−=𝒇𝜕𝒕
𝜕𝒙

GPU #0GPU #1GPU #2𝑨𝛿𝒖 = 𝒃solved over 10 K time steps using conjugate gradient(CG)-based method (𝑨 is updated every time step)

GPU #0

Revised partitioning

Input: thickness of each layer 𝑥0 , 𝑥1 , 𝑥2 , 𝑥3Training

GPU #1#0

GPU#0 GPU#1 GPU#2

#1

#2

#0

Revised domain partitioning• Load balance is improved → high parallel efficiency• More sequential memory access → suitable for GPU computing• Computation is accelerated by GPU;sign exponent fraction
communication is notSingle precision
1 bit 8 bits
23 bits→ Communication can be a bottleneck(32 bits)• FP21 variables are used in MPI
FP21 (21 bits) 1 bit 8 bits
12 bits
communication in preconditioner• 3 x FP21 variables (63 bits) are packed to a double precision variable (64 bits) andcommunicated among GPUs
• Comparison on a compute node on AI
Bridging Cloud Infrastructure [2] (ABCI)

Solve equation

constitutive law

CPU based
CPUx9.9GPU basedGPU

4,854,570 DOFs, 100 time steps
2 CPUs (Intel Xeon Gold 6148)
4 GPUs (NVIDIA Tesla V100 SXM2)
10.7-fold speedup by using GPUsother

10 models, 200m×200m×~60m
0.4 million elements, 1 million DOFs

Target soil model
Soil models for training NN• Loss function: cross entropy loss • Number of epochs: 50• Optimizer: Adam (𝛽1 = 0.9, 𝛽2 = 0.999, learning rate = 10−4 )# ofhidden
layers• Accuracy of more than 90%

Physics-based
Estimation byDifference
Simulation surrogate model(*)
0(not liquefied)1(liquefied)

-1

0

1

# ofunits inhidden
layers

259

2

102

207

3

5

3

105

320

4

53

4

10

4

Total
Liquefied
Not liquefied

201
85%

90%

95%

100%

Accuracy for target soil model

x10.7

GPU +
based+
GPU
FP21
0

10

20

30

40

Computation time[s]50

Oakforest-PACS
128 computing nodes × 14 h 37 min(1 Intel Xeon Phi 7250 CPU/node)

89,146,716 DOFs
30,000 time steps

90% reduction incomputational costof simulation(*) # of hidden layers = 4, # of units in hidden layers = 5• Comparison of large-scale simulation

Wharf

Liquefiablelayer
1m(*): an index of soil liquefaction
0: not liquefied; 1: completely liquefied

Estimation results

MPI communication with a 21-bit data type

Performance measurement

42,867,001 elements
94,286,505 DOFs

#2

Rearrangement of elements• Judge to be liquefied
when the max. ofexcess pore waterpressure ratio (*) inthe orange region islarger than 0.9

Latin hypercube
sampling oflayer depthGPU #2#1

Output: whether soil gets liquefied

𝑥0• Sampled at 1 m intervals
Non-liquefiable𝑥1• Based on earthquakelayer
engineering knowledge
𝑥2
that thickness and depthof liquefiable layer have Liquefiable layer
influence on likelihood of𝑥3
Bedrocksoil liquefaction

Reordered data alignment

discretized with FEM and Newmark β method
Sparse matrix equation with 1 million – 1 billion DOFs

Conventional data alignment

Input• is expected to contribute to estimation andmitigation of earthquake damage• entails high computational cost especially in 3Dsimulation

outputMultiphysics-based earthquake simulation

AffineNeural Network based surrogate model
ReLU

GPU-accelerated Simulation

AffineIntroduction

Softmax

1: The University of Tokyo, 2: Japan Agency for Marine-Earth Science and Technology

ABCI
13 computing nodes × 3 h 33 min(4 NVIDIA Tesla V100 SXM2 GPUs/node)

Without high performance computing: 282K DOFs × 40K time steps → 1 CPU × 1 month

Developed method enables faster simulation with smaller computation environment

Conclusion• This poster presents an example method to realize HQC on complex nonlinear
problems by efficiently combining capability and capacity computing and NN• With GPU-accelerated Multiphysics-based simulation and surrogate models
constructed from its results, it is expected that more advanced evaluation ofearthquake response can be performed with smaller computation cost and time
Acknowledgements: Computational resource of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute ofAdvanced Industrial Science and Technology (AIST) was used. This work was supported by MEXT as “Program for Promoting
Researches on the Supercomputer Fugaku” (Large-scale numerical simulation of earthquake generation, wave propagation and soilamplification: hp200126, hp210171) and JSPS KAKENHI Grant Numbers JP20J22348, JP18H05239, JP18H03795, JP17K14719.

References[1] Ryota Kusakabe, Kohei Fujita, Tsuyoshi Ichimura, Takuma Yamaguchi, Muneo Hori, and Lalith Wijerathne. 2021. Developmentof regional simulation of seismic ground-motion and induced liquefaction enhanced by GPU computing. Earthquake Engineering andStructural Dynamics 50 (2021), 197–213.
[2] National Institute of Advanced Industrial Science and Technology. 2021. About ABCI. https://abci.ai/en/about_abci/