HPCI magazine vol.2

Fugaku's
Record-breaking
Computations Enable
More Accurate
Weather Forecasting
Although weather forecasts are steadily improving, it is still difficult to accurately predictthe paths of tropical cyclones and where heavy rains will occur. More accurate weather
forecasts require detailed simulations on supercomputers with more processing power,
and also applications designed to utilize that power. To help address this issue, Hisashi
Yashiro, a member of a research project*1 under the “Program for Promoting Researchon the Supercomputer Fugaku”, carried out the largest-ever meteorological computations
for weather forecasting use.

Hisashi Yashiro

Senior Researcher
Satellite Remote Sensing Research Section, Earth System Division,
National Institute for Environmental Studies

Better Weather Forecasts
Require Massive Computations
Modern weather forecasting is based onsupercomputer simulations. As supercomputers
have become more powerful, the forecasts
have become more accurate over the last twodecades, but nevertheless sometimes fallshort. Moreover, it is quite difficult to accurately
predict the paths of tropical cyclones and whereh e a v y r a i n s m a y o c c u r. I m p r o v i n g t h e s esimulations will require processing colossal
amounts of meteorological data points known
as “state values” in the weather model.
Looking back on 2014, when the development
work on Fugaku first got underway, Yashiro
says, “I wanted to make use of the computing
power of Fugaku to help contribute to moreaccurate weather predictions.” It was decided01

that Fugaku would undergo a “co-design”
approach, where the hardware and softwaredevelopers would collaborate in determining
specifications. Yashiro contributed to thisco-design effort, overseeing the weather
and climate software perspective. Theco-design team began work on a project, “todevelop an application that could address
the challenges of massive-scale calculations
at a practical level when a supercomputerwith the power of Fugaku became available
to national meteorological organizations inthe future.”

Better Forecasts Start withBetter Replication ofCurrent Weather Conditions
Weather patterns, such as clouds, rain,and snow, are the result of atmosphericconditions, such as temperature, humidity,pressure, and wind. To model theseconditions for the entire planet, scientists
divide the globe into a 3-dimensional gridand compute the changes expected at eachgrid point (or box) according to the laws ofphysics (Fig.1a). The challenge is choosing
initial values for each grid point that
accurately reflect the atmospheric conditions
there. Weather pattern developments are
sensitive to initial conditions. If the initialvalues used in the computations don't

vol.2 - Fugaku's Record-breaking Computations Enable More Accurate Weather Forecasting

match the actual state of the atmosphere,
the predictions will suffer. But, there are
far more grid points in the model thanactual observation points measuring
atmospheric conditions. So to begin runninga simulation, researchers need a way tocalculate a suitable starting value for eachgrid point where no direct measurement data
is available. Many meteorologists havebeen working on ways to compensate forthis incomplete initial data.

Fig. 1aAtmospheric Simulations
and Data Assimilation

In this model, the Earth's atmosphere is divided into a grid.
Changes to the atmospheric conditions at each grid point
are computed according to the laws of physics. In thisexample, the grid is divided horizontally by latitude andlongitude, and vertically by altitude with corresponding
barometric pressure. The finer the mesh of the grid, themore accurate the simulation becomes, but the longer thecomputations and data processing take. This model isused both for weather forecasting and for data assimilation.
The lower left diagram shows the physical processesmodeled for each grid point, including how atmosphericheat and water interact over land and sea.

Yashiro and his colleagues used an
approach called “ensemble-based dataassimilation”. First, they assign a temporary
initial value for each grid point. Next, they
run weather forecast simulations for awhile to see how those values changeover time. They then check those values
against values acquired from actual
observations, and update the value forSource: National Oceanic and Atmospheric Administration

02

Yashiro's team for this project used about
80% of Fugaku's total node count toperform computations with 1024-member
ensembles, and a mesh size of approximately
3.5 kilometers. They were able to executelarge-scale computations, about 500
times larger than those performed bymeteorological agencies, with run times
fast enough for use in weather forecasting
systems.

Fig. 1b

Atmospheric Simulations andData Assimilation
Observational data is assimilated into theresults of atmospheric simulations thatbegan with approximated initial values. Eachgrid point is updated to more realistic values
based on statistical estimates.
Graphics based on JMA data

each grid point based on statistics andprobability theory (Fig.1b). This is “dataassimilation”. “Ensemble” means running
multiple meteorological simulations at thesame time with slightly different states.
Like real weather, atmospheric simulations
evolve non-linearly over time. If the weathermodel is incomplete, the longer thesimulation runs, the more the results deviate
from real-world observations. To address
this problem, a number of simulations arerun in parallel with different initial values.
Comparing the results of multiple simulations
tells the researchers which particular grid
point locations are more sensitive to theinitial values, and therefore more likely tocontribute to a forecasting error. Those
more sensitive grid points (areas) are thenvol.2 - Fugaku's Record-breaking Computations Enable More Accurate Weather Forecasting

weighted with greater importance when
the observation data is assimilated.
“How well data assimilation can reproduce
the real atmospheric conditions will greatly
affect the accuracy of the prediction
results,” explains Yashiro. For this reason,
meteorological agencies worldwide areattempting to assimilate data using
ensembles with more members, and gridswith finer meshes. Application performanceand supercomputer processing power havetypically allowed only dozens to hundredsof members per ensemble, with each grid
point representing 10 to 50 kilometers ofthe Earth's surface.
The ensemble data assimilation done by

Developing more
Computationally Efficient
Applications that Make Useof Fugaku's Performance
The application that Yashiro and hiscolleagues ran for atmospheric simulations
featured optimizations by RIKEN researchteams. Team Leader Hirofumi Tomita
oversaw
optimizations
using
the
“Non-hydrostatic Icosahedral AtmosphericModel” (NICAM), a very high-resolution
global atmospheric model. Team Leader
Takemasa Miyoshi directed optimizationsof the data assimilation system using a“Local Ensemble Transform Kalman Filter”(LETKF). In 2013, Tomita's team, whichthen included Yashiro, ran the world's
highest-resolution model simulation of theglobal atmosphere, with a horizontal mesh03

Fig. 2

Atmospheric Simulation withNICAM

The 6-hour later forecast produced by a global atmospheresimulation with a grid mesh size of 870 meters and withobservation data from midnight August 25, 2012, as the initialvalues. Using one-quarter of the K computer's 80,000+
nodes allowed accurate reproductions of atmospheric flow
and typhoon development.

Joint research by RIKEN, the Japan Agency for Marine-Earth Science andTechnology, and the University of Tokyo's Atmosphere and Ocean
Research Institute (HPCI Strategic Programs for Innovative Research,Area 3).
Visualization: Ryuji Yoshidasize of approximately 870 meters, using
the K computer (Fig. 2).
This time, the mesh was less dense, withgrid points every 3.5 kilometers, but with
1024 parallel NICAM calculations for each,and data assimilation throughout, making
these historically intense computations.
For these calculations to be practical in
actual weather forecasting, however, they
have to run faster than time elapses in reallife. Even with the power of Fugaku, that is
no simple task. Yashiro and his associates
had to devise a number of modifications tomake the application run faster.

Yashiro says, “Fugaku is a world-classsupercomputer that inherits the best
features of the K computer. And because it
was co-designed by hardware and software engineers, we could come up withways to make full use of its performance.
I think co-design will improve the results fora number of scientific fields, not only ours.”
There are plans for future calculations toIn particular, a large amount of data needsto be transferred whenever shifting fromthe atmospheric simulation process to thedata assimilation process. They therefore
modified the program to minimize the timerequired for data transfers. They also tried
to make maximum use of the solid-statedrives (SSD) attached to Fugaku's
computation nodes to speed up datareading and writing. SSDs have very highperformance for reading and writing files.
For this project, the team designed thesoftware to take advantage of Fugaku's
hardware, which was designed to include
SSDs.
Explaining the significance of co-design,

vol.2 - Fugaku's Record-breaking Computations Enable More Accurate Weather Forecasting04

Fig. 3

Increasing the number ofensemble members toimprove weather forecasting.
ResultsfromFugaku

calculations

21-Member Ensemble

usingobservation data from torrential rains in July
2020 by a research project team under the“Program for Promoting Research on theSupercomputer Fugaku”. After data assimilation,
the team ran ongoing ensemble calculations
to produce a 12-hour later forecast predicting
the likelihood of heavy rains (more than 50mm of rain in 3 hours). The closer to red, thehigher the probability. The 21-member ensemble

100-Member Ensemble

used by the JMA's meso-ensemble prediction
system did not clearly identify which locations

assess the accuracy of weather forecasts
based on these ensemble data assimilation
results. Yashiro, however, is confident.
“Seeing the results (Fig. 3) of other teams
working under the 'Program for Promoting
Research on the Supercomputer Fugaku',
we expect the accuracy will improveconsiderably.” As meteorological agencies
around the world develop and use software incorporating our techniques overthe next ten to twenty years, they will
surely be able to more accurately forecast
the paths of tropical cyclones and whereheavy rains will occur, which will help
minimize the damage they cause.

could expect heavy rains. The research team's
1,000-member ensemble, however, was able
to identify those locations, making theprediction results a strong contender for use
in providing emergency sheltering advisories.
The areas that the 1,000-member ensemble
predicted were very likely to, did actuallyexperience heavy rains.

1,000-Member Ensemble

Le Duc et al. (2021):
Forecasts of the July 2020 Kyushu heavy rain using a
1000-member ensemble Kalman filter, SOLA

vol.2 - Fugaku's Record-breaking Computations Enable More Accurate Weather Forecasting05

About theThe“LargeEnsemble

AtmosphericandEnvironmental Prediction for Disaster Prevention
andMitigation”

research

project

receivedenhancement support from RIST.
The long execution time for the parallel
pre-processing

programfor

theJMA'snon-hydrostatic model (NHM) was madesignificantly faster with enhancement support
from RIST. This made it possible to executecomputations including pre-processing at higher
speeds with up to 10,656 Fugaku nodes,
calculations not possible on the K computer.

*1 Research project name: “Large Ensemble Atmospheric andEnvironmental Prediction for Disaster Prevention and Mitigation”(Principal Investigator: Professor Masaki Satoh, Atmosphere andOcean Research Institute, The University of Tokyo)• Theme 1: “Short-range Regional Prediction”(Principal Investigator: Takuya Kawabata, Head, MeteorologicalResearch Institute)• Theme 2: “Global-scale Prediction”(Principal Investigator: Associate Professor Tomoki Miyakawa,
Atmosphere and Ocean Research Institute, The University of Tokyo)• Theme 3: “Advanced Technology of Data Assimilation”(Principal Investigator: Hisashi Yashiro)

vol.2 - Fugaku's Record-breaking Computations Enable More Accurate Weather Forecasting

Researcher
Hisashi Yashiro did not originally intend to
become a researcher. While still a master's
student, he spoke passionately about his
research during a job interview. The interviewer
asked him why he wasn't continuing hisresearch, which put the idea in his head of
becoming a researcher. As a student, hespecialized in monitoring greenhouse gases,
working with simulations only as a user ofthe resulting data. His research shifted tosimulation-based studies and atmosphericmodeling, and then to application development.
In July 2019, he moved from RIKEN to theNational Institute for Environmental Studies(NIES), where he is involved in research that
integrates data received from the Ibuki greenhouse gas monitoring satellite into model
simulations. A choral singer since his schooldays, he's a member of the institute's choral
singing club.

Associated Research Projects• “Large Ensemble Atmospheric and Environmental Prediction for Disaster Prevention and Mitigation” (hp200128)
Principal Investigator: Masaki Satoh, Atmosphere and Ocean Research Institute, The University of Tokyo
A paper reporting on the findings of this project was selected as a 2020 ACM Gordon Bell Prize finalist.

06

HPCI magazine FUGAKU HYAKKEI vol.2

Registered Institution for Facilities Use Promotion /The Representative for HPCI Operation

Research Organization
for Information Science and Technology
1-5-2, Minatojimaminami-machi, Chuo-ku, Kobe, Hyogo, 650-0047 JAPAN
Tel: +81-78-599-9511

HPCI Portal Site

HPCI PR site

https://www.hpci-office.jp/folders/english

https://fugaku100kei.jp/e/Issued: March 2021
© 2022 Research Organization for Information Science and Technology