Algorithmic transformation from physical models todata-driven models using the coupling library:a case of a climate model
J-cup

h3OU/MPJ-cup

Hisashi YASHIRO (National Institute for Environmental Studies)
withTakashi Arakawa(ClimTech Inc.), Shinji Sumimoto, Kengo Nakajima (U-Tokyo), and h3-Open-BDEC team
45th ASE Seminar (Advanced Supercomputing Environment):
International Workshop on "Integration of Simulation/Data/Learning and Beyond"2023/11/29

Demands on HPC resource for weather/climate research
Resolution

km-scale is required formore accurate predictions

Climate simulation ismore than 100 yearsDurationWe need morecomputational power!

Complexity
we want to use more sophisticatedmodel components

100 members is not enoughEnsembleWorkshop on Integration of S/D/L and Beyond, 2023.11.29

Why we need high-resolution atmospheric simulations?
• When the grid spacing is O(100km), we have to parameterize the clouds andHigh resolu)on: Explicit clouds,convection processnonhydrosta)c dynamics• Coarse-grained models cause large uncertainty

Clouds are ubiquitous
Deep convec)ve clouds are • However, high resolution simulation takes large computational cost
responsible for a signiﬁcant amountof severe weather (high winds, hail,• 1/10 horizontal grid spacing -> x1000 problem size
ﬂooding, lightning, etc.)
The largest uncertain)es inatmospheric weather and climate
models involve clouds.

NASA composite image (bluemarblewest)

NASA photo, Florida squall line (view to E-SE)

Workshop on Integration of S/D/L and Beyond, 2023.11.29

Simulated precipitation diurnal cycle over the Amazon region• Observations show lateafternoon peak

• Too early peak is simulatedin O(100km) resolution• The simulation in O(10km)

resolution shows midnightpeak• Super high-resolution(O(1km)) reproduce thetiming wellWorkshop on Integration of S/D/L and Beyond, 2023.11.29

…But big obstacles stand in our way
The trend of the HPC is changing and changing• Weather/climate models are huge and very hard to port
• The memory throughput mainly determines simulation speeds, and its performance will
not improve faster as we expected• No large hot-spots of computation: we have to optimize everywhere in the application

ACM Gordon Bell prize for climate (2023~)• This prize encourages collaboration and further efforts between climate science andcomputational science• The rst winner was the very big team: They’ve rewritten the atmospheric model with C++

fi

Workshop on Integration of S/D/L and Beyond, 2023.11.29

Development of the coupling library h3-Open-UTIL/MP

Two practical cases for using our coupling library• The reduction of the computational cost of high-resolution weather data
assimilation• The reduction of the computation through the making of data-driven surrogatedmodelsWorkshop on Integration of S/D/L and Beyond, 2023.11.29

The reduction of the computational cost of high-resolution weather
data assimilation
Ensemble Karman ltering

Ensemble Mean

Observations

Large computatio

nal cost

Realistic atmospheric state

Hi-res. Ensemble simulations

Ensemble data assimilation (DA)• Total computational cost = 1-member simulation x # of ensemble members• high-resolution x large ensemble is costfulfiWorkshop on Integration of S/D/L and Beyond, 2023.11.29

The reduction of the computational cost of high-resolution weather
data assimilation
Ensemble Karman lteringh3-Open-UTIL/MP
Coupling

Observations
Realistic atmospheric state• Large scale: nudged by Low-res. DA

Low-res. Ensemble simulations• Small scale: explicitly simulated

Online coupling of Low-Res. Ensemble DA + Filtered Nudging• DA result and high-res. prediction is compared in the low-resolution➡ Difference is applied as a nudging tendency (=Newtonian relaxation)fiWorkshop on Integration of S/D/L and Beyond, 2023.11.29

Ensemble + Coupling by h3-Open-UTIL/MP• Coupling Layer : data exchange between different models• grid remapping• MxN (local to local) send/recv• Ensemble Layer : data exchange between same models• Gather and Scatter on the same area of different ensemble• Average calculation of ensemble data

Workshop on Integration of S/D/L and Beyond, 2023.11.29

Estimation of the computational cost
• 14km mesh, 64 members• 640 MPI procs per member➡ 0.5 simulation day per wall-clock hour (SDPH) with 1280node of wisteria-oHigh-res. Ensemble DAEnsemble DA part+•

224km mesh, 64 members•

40 MPI procs per member➡ 181 SDPH with 640node of wisteria-oFiltered Nudging part•

14km mesh, 1 member•

2560 MPI procs per member➡ 9.1 SDPH with 640node of wisteria-oTotal• Two simulations are almost
overwrapped➡ 9.1 SDPH with 1280node of wisteria-o➡ x18 faster than high-res. ensemble DA!

Low-res. Ensemble DA + One Hires. Nudging

Workshop on Integration of S/D/L and Beyond, 2023.11.29

The reduction of the computation through the making of datadriven surrogated models

AI weather models!
• GraphCast, Pangu Weather, FourCastNet, etc…
• Hot topics in the world• It surrogates simulation results ~= full model

But we are climate/weather scientist• We want understand the process of the phenomena in the atmosphere• We will not throw away the physics models from now and future• We want to apply new technics step by step in the huge-codebase software➡ Component-level surrogation by AI technics to make a “booster”
Workshop on Integration of S/D/L and Beyond, 2023.11.29

Training system with coupling libraryh3-Open-UTIL/MP
J-cup

Physics-based model(in full app.)

Coupling

J-cup

Data-driven model“Algorithmic transformation”• Based on our knowledge• Fast because less computation• Relatively slow, difficult tospeedup anymore• Dense computation fits GPUs

• Provide teaching data on-the- y• Lots of GPU-ready libraries• Plug-in to target component ofthe application

flWorkshop on Integration of S/D/L and Beyond, 2023.11.29

Training system with coupling library

Coupling with a legacy fortran application and a modern python ML library
By using h3-Open-SYS/WaitIO together, users will be able to perform calculations using
heterogeneous supercomputers easily
Workshop on Integration of S/D/L and Beyond, 2023.11.29

The case of the cloud microphysics schemeStates• Input: air density, temperature, water vapor amount• Output: tendency of the inputs

Temperature

Water vapor

Reference (simulation result)• 3-layers MLP• For more accurate
reproducibility:• More input variables• Inputs from the multiple
atmospheric layers
ML output

Workshop on Integration of S/D/L and Beyond, 2023.11.29

Summary

Weather/climate models still need huge amount of thecomputational resources• However, accelerating weather/climate models will be more dif cult in the next decade• Performance portability, memory wall, algorithms that cannot increase computational
intensity…

Solving some issues by using coupling library• h3-Open-UTIL/MP is powerful tool for bridging models across the scale and algorithm

fiWorkshop on Integration of S/D/L and Beyond, 2023.11.29