HPCI magazine vol.9

Computing
Crustal Deformation withUncertainty Quantification– Large-Scale Earthquake Simulation
beyond Common Sense“In an ideal simulation, I would like to do this calculation, but the number of calculations is huge, so
it is impossible.” Overcoming problems like this is a showcase for simulation researchers. Prof.
Ichimura, who studies earthquake simulation, has released many computational technologies thatenable ideal calculations with existing computational resources. “Five or 10 years from now, when
computational resources such as the supercomputer Fugaku become available to more people, wewill be able to make improvements based on our achievements for their own purposes. We arepursuing methods that can be used with modifications that suit each purpose based on ourresults,” Prof. Ichimura said, introducing his research.

Tsuyoshi Ichimura
Professor/Head of Center, Research Center for Computational Earth Science,
Earthquake Research Institute, The University of Tokyo

Large-Scale Earthquake
Simulation– A Pressing Need for Society
To allow the government to decide how toprepare for future major earthquakes, such asa direct hit on the Tokyo Metropolitan Area ora Nankai Trough earthquake, more accurate
earthquake assessments are required. It is
not possible to predict when an earthquake
will strike, but better assessments will enabledisaster prevention professionals to makebetter measurements and decisions. To
construct an integrated prediction systemfor quantitative assessments of earthquakes,
in April 2020, one of the Programs for
Promoting Research on the supercomputer
Fugaku was launched. The name of thestudy is “Large-scale numerical simulation01

of earthquake generation, wave propagation a n d s o i l a m p l i f i c a t i o n ” (PrincipalInvestigator: Dr. Takane Hori, Japan
Agency for Marine-Earth Science andTechnology).
Prof. Ichimura is creating a series of innovative earthquake simulation methods inthis program. Here we introduce two representative simulations from his results.

Fig. 1

A comprehensive simulation fromthe occurrence of an earthquake tothe shaking of structures on theentire Kanto Plain

A Comprehensive
Simulation fromthe Fault to the Structures,
Which Has Beena Long-Standing Issue *1
In an earthquake simulation, researchers
calculate how the waves generated at thefault propagate through the earth’s crustand soil, until they meet structures such

They simulated a model consisting of the crust and ground in
an area including the Kanto Plain (256 km east-west x 205 km
north-south x 100 km depth) and a highly concentrated city
around a terminal train station. The size of the lattice ranges
from about 0.125 to 64m, and the scale of calculation is 320iii) Seismic response of cityCity

as buildings and houses and make themshake. However, to accurately simulate
wave propagation, the crust and soil must
be calculated on a fine grid. Furthermore,
simulating the shaking of structures due
to transmitted waves requires a huge
number of calculations. That makes it
difficult to analyze everything from thefault to the structures at the same time,targeting a wide area.

billion degrees of freedom. The calculated area consists of soft
soil layers on heterogeneous crust, as well as above-groundand underground structures. The complex geometry of thestructure is modeled in detail with a very fine mesh of minimum
0.125m.

=15 s

Buildings

2mWhole domain

Underground passage
with mesh visualized=20 sUnderground structures=30 s

Underground structure

Surface soilii) Wave amplification

Bedrock
Domain ofapplication
simulationi)-b Wave propagationi)-a Fault slips and releases strain

=25 s

Crust

Close-up view of the city

0.2 kmTsuyoshi Ichimura, Kohei Fujita, Kentaro Koyama, Ryota Kusakabe, Yuma Kikuchi, Takane Hori, Muneo Hori, Lalith Maddegedara, Noriyuki Ohi, Tatsuo Nishiki, Hikaru Inoue, Kazuo Minami, Seiya Nishizawa, Miwako Tsuji, and Naonori Ueda. 2022. 152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquakesimulation. In International Conference on High Performance Computing in Asia-Pacific Region (HPCAsia2022). Association for Computing Machinery, New York, NY, USA, 18 -29. https://doi.org/10.1145/3492805.3492814. Fig. 1 and 2.

vol.9 - Computing Crustal Deformation with Uncertainty Quantification – Large-Scale Earthquake Simulation beyond Common Sense02

“Therefore, we usually divide thed o m a i n from the fault to the structure
into several stages and perform calculations,” Prof. Ichimura says. “Is it possible to obtain correct calculation resultswith this method? Aren’t there any ways
to calculate all of them at once continuously?”
Using Fugaku, Prof. Ichimura and hiscolleagues took on the challenge andsucceeded in conducting a comprehensive
simulation from the fault to the structures,targeting an area including the Kanto Plainaround Tokyo (Fig. 1). It took a lot of ingenuity to make full use of the characteristicsof Fugaku as a computer suitable for thefusion of data science and simulation.
Prof. Ichimura had previously developed
a learning method *2 using artificial intelligence (AI), and this time he developed
this into a revolutionary data learning
method. By accumulating the calculation
results during the simulation and learning
the data in real time, the initial value ofthe subsequent calculation called theiterative method can be estimated withhigh accuracy. This reduces the amountof computation and speeds up the simulation. As a result, he and his teamachieved a calculation speed 1,070 times

faster than the maximum performance of
RIKEN’s K computer.

A Large-Scale Simulation
with 32 Trillion Degrees ofFreedom that Surprised
the World *3
Prof. Ichimura and others also took up thechallenge of simulating the Nankai Trough.
This is a large-scale simulation of theground surface deformation that occurs
immediately after a large earthquake
caused by the slip of the fault. The featureof this simulation is that it handles a wide
area and includes “ambiguity.” Ordinarily,

vol.9 - Computing Crustal Deformation with Uncertainty Quantification – Large-Scale Earthquake Simulation beyond Common Sense

this is such a huge problem they would noteven attempt it because the number ofcalculations required is enormous.
What do they mean by “ambiguity ”? Thehardness of the earth's crust is not actually
directly measured by digging, so its hardness cannot be determined as a singlevalue. It is more reliable to express it as“this kind of crustal deformation occurs withthis probability” with a certain distribution.
When simulating uncertain events whileconsidering probability like this, researchers usually use a method called the “MonteCarlo simulation.” This calculation method03

0.05
0.04

Closeup view

Pacific plateEurasian plate604030 km

20-0.20

Very complex geometry discretizedusing 375 m unstructured secondorder tetrahedral elements

2496 kma) Target area
Mean response

x-direction
y-direction

km400

Response for +3km400b) Finite-element modelStandard deviation-0.05-0.02d) Stochastic response at
two points on surface
km
400

-0.04
-0.05

500 km0.00

Displacement in z-direction (m)Mean response

km400-0.100.00

Response for +3

500 km/ of displacement in z-directione) Stochastic
response shown in 3Dkm
400200200200

200200200000

000-200

km
400

km
400

km
400

km
400

-200
km
km
400

200

200

200

200

200

200

0

0

0

0

00-200

-200

0

200

400

km

-200

-200

0

200

400

km

-200

-200

0

200

400

km

-200

-200

0

200

400

-200

0

200

400

km

-200

km
400

km
400

km
400

-200
km
km
400

200

200

200

200

200

200

0

0

0

0

0

0

-200

-200

-200

0

0

200

200

400

400

km

km

-200

-200

-200

-200

0

0

200

200

Coseismic slip

400

400

km

km

-200

-200

-200

-200

00

200

200

400

400

km

km

-200

-200

-200

-200

0

0

200

200

400

400

-200
kmc) Stochastic response at surface (displacement (m))

Standard deviation-2000200

400km-2000200

400km-2000200

400km

km
400km
400-200z-direction

2496 km-0.15

probability

Philippine seaplatekm
4000.0280-200

0200

400km

km
400-200

0200

400km

-200

Afterslip©2023 IEEE. Reprinted, with permission, from T. Ichimura, et al., ‘‘Extreme Scale Earthquake Simulation with Uncertainty Quantification,’’ in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, Dallas, TX, US, 2022 pp. 37-47.

Fig. 2

Simulation ofNankai Trough earthquake
considering ambiguity

The team simulated an area of 2,496 km x 2,496 km x 1,101km centered on the Nankai Trough (which includes the entire
Japanese archipelago). Maps in lower half represent themagnitude of deformation in the x, y, and z directions. Simula-

vol.9 - Computing Crustal Deformation with Uncertainty Quantification – Large-Scale Earthquake Simulation beyond Common Sense

tion results of “mean,” “mean + 3σ (σ is the standard deviation;
the probability of falling within this range is 99.7%)” and“standard deviation” are shown. This broad range of results
enhances the quality of the simulation.

04

repeatedly selects one numerical value
from the possible range of values andperform calculations with it, but in order toobtain highly accurate results, it is necessary to repeat the calculation more than
10,000 times. Therefore, doing a wide-area
earthquake simulation is beyond even
Fugaku’s abilities.
Instead, Prof. Ichimura and his teamperformed a simulation using a calculation
method called the stochastic finite element
method. With this method, results withdistribution can be obtained in a single
calculation. The calculated area covers theentire Japanese archipelago, which measures 2,496 km east-west, 2,496 km
north-south, and 1,101 km deep, with ascale of approximately 32 trillion degrees offreedom. The finite element method, whichis the basis of the stochastic finite element
method, is thought to be unsuitable forlarge-scale calculation. Even 10 million
degrees of freedom was considered large,
and only a billion could be calculated at
most. So, researchers around the world
were amazed at the 32 trillion degrees offreedom of the stochastic finite element
method. What’s more, Prof. Ichimura’s
calculation is 224 times faster than theprevious record. The reason the team
could perform such large-scale calculations
so quickly was that they studied a method
to execute calculations using Fugaku
skillfully from the aspect of computational
science and computer science.
However, the code to perform thisc a l c u l a tion contains thousands-lines
complex parts that would be prone to errors
if written manually. Therefore, Prof. Ichimuraand his colleagues created software tohave a computer automatically write acomplicated part of the code.
This achievement of a large-scale, precise
simulation that considers the “quantification of uncertainties” has attracted attention
from all over the world. Regarding thiscalculation method, Prof. Ichimura said,“People who had given up on considering
ambiguity in large-scale problems would
wonder, ‘Is there such a solution?’ If we canobtain information on the slip of the fault,
we can use this research to understand thedeformation of the ground surface. The
value of the deformation can be used asdata for post-earthquake predictions. Since
this method can handle hard objects, itcan be used in applications other thanearthquakes.”

vol.9 - Computing Crustal Deformation with Uncertainty Quantification – Large-Scale Earthquake Simulation beyond Common Sense

05

Prof. Ichimura continued, “In this simulation,
we calculated the deformation immediately after the earthquake, but I would liketo solve deformation over time and more
complex nonlinear deformation.” He isexcited at the prospect of tackling problems
that are even more complex, with the computation amount increasing exponentially.
He and his team will continue to create
innovative methods to get closer to the“ideal simulation.” The team members havehigh hopes for the next move.

About theResearcher

From novels to science texts, Prof. Ichimura as a child read books whenever he had freetime. He liked to think, “What’s going to happen next?” When he entered the University ofTokyo, he chose to study science, but at the end of his sophomore year, he seriously
considered going into the Faculty of Letters or the Faculty of Economics; he ended up
choosing the path of Civil Engineering, where he could come into contact with a widerange of academic subjects. He began to devote himself to earthquake simulation
research, which he started with his graduation thesis.

The members of the lab: from left in the back row, Associate Professor Kohei Fujita, Prof. Ichimura, and Project Academic
Support Specialist Yumiko Nagasaki. From left in the front row, Ryota Kusakabe, a third-year doctoral student, and Souta
Murakami, a second-year doctoral student.
*1: HPC Asia 2022 Best Paper Award research results*2: Workshop on Accelerator Programming Using Directives(WACCPD 2021) Honorable Mention award-winningresearch results*3: 2022 ACM Gordon Bell Prize finalist research results

Associated Research Projects
Large-scale numerical simulation of earthquake generation, wave propagation and soil amplification (hp200126/hp210171/hp220171)
Principal Investigator: Takane Hori, Japan Agency for Marine-Earth Science and Technology

vol.9 - Computing Crustal Deformation with Uncertainty Quantification – Large-Scale Earthquake Simulation beyond Common Sense06

vol.9HPCI Portal site

HPCI PR site

https://www.hpci-office.jp/enhttps://fugaku100kei.jp/e/