ClimSim: A large multi-scale dataset for hybrid physics-ML climate emulation
Authors: Sungduk Yu
Walter Hannah
Liran Peng
Jerry Lin
Mohamed Aziz Bhouri
Ritwik Gupta
Björn Lütjens
Justus Christopher Will
Gunnar Behrens
Julius Busecke
Nora Loose
Charles I Stern
Tom Beucler
Bryce Harrop
Benjamin R Hillman
Andrea Jenney
Savannah Ferretti
Nana Liu
Anima Anandkumar
Noah D Brenowitz
Veronika Eyring
Nicholas Geneva
Pierre Gentine
Stephan Mandt
Jaideep Pathak
Akshay Subramaniam
Carl Vondrick
Rose Yu
Laure Zanna
Tian Zheng
Ryan Abernathey
Fiaz Ahmed
David C Bader
Pierre Baldi
Elizabeth Barnes
Christopher Bretherton
Peter Caldwell
Wayne Chuang
Yilun Han
YU HUANG
Fernando Iglesias-Suarez
Sanket Jantre
Karthik Kashinath
Marat Khairoutdinov
Thorsten Kurth
Nicholas Lutsko
Po-Lun Ma
Griffin Mooers
J. David Neelin
David Randall
Sara Shamekh
Mark A Taylor
Nathan Urban
Janni Yuval
Guang Zhang
Michael Pritchard
Venue: NeurIPS 2023 Datasets and Benchmarks Oral (2023)
Keywords: climate
climate modeling
benchmark
dataset
baseline
emulation
superparameterization
multi-scale modeling framework
physics-informed machine learning
TL;DR: ClimSim is the largest dataset for multi-scale, physics-informed machine learning climate simulations along with a wide range of baselines for the modeling of atmospheric convection and radiation.

Abstract: Modern climate projections lack adequate spatial and temporal resolution due to computational constraints. A consequence is inaccurate and imprecise predictions of critical processes such as storms. Hybrid methods that combine physics with machine learning (ML) have introduced a new generation of higher fidelity climate simulators that can sidestep Moore's Law by outsourcing compute-hungry, short, high-resolution simulations to ML emulators. However, this hybrid ML-physics simulation approach requires domain-specific treatment and has been inaccessible to ML experts because of lack of training data and relevant, easy-to-use workflows. We present ClimSim, the largest-ever dataset designed for hybrid ML-physics research. It comprises multi-scale climate simulations, developed by a consortium of climate scientists and ML researchers. It consists of 5.7 billion pairs of multivariate input and output vectors that isolate the influence of locally-nested, high-resolution, high-fidelity physics on a host climate simulator's macro-scale physical state.

The dataset is global in coverage, spans multiple years at high sampling frequency, and is designed such that resulting emulators are compatible with downstream coupling into operational climate simulators. We implement a range of deterministic and stochastic regression baselines to highlight the ML challenges and their scoring. The data (https://huggingface.co/datasets/LEAP/ClimSim_high-res) and code (https://leap-stc.github.io/ClimSim) are released openly to support the development of hybrid ML-physics and high-fidelity climate simulations for the benefit of science and society.

ClimSim: A large multi-scale dataset for hybrid physics-ML climate emulation
Sungduk Yu', Walter M. Hannah', Liran Peng', Jerry Lin', Mohamed Aziz Bhouri
Ritwik Gupta', Björn Lütjens,, Justus C. Will', Gunnar Behrens Julius J. M. Busecke
Nora Loose", Charles Stern Tom Beucler Bryce E. Harrop", Benjamin R. Hillman
Andrea M. Jenneyy Savannah L.. Ferretti', Nana Liu', Anima Anandkumar
Noah D. Brenowitz Veronika Eyring", Nicholas Geneva 2 Pierre Gentine
Stephan Mandt', Jaideep Pathak Akshay Subramaniam Carl Vondrick', Rose Yu
Laure Zanna 14, Tian Zheng', Ryan P. Abernathey Fiaz Ahmed David C. Bader?,
Pierre Baldi', Elizabeth A. Barnes Christopher S. Bretherton 17 Peter M. Caldwell",
Wayne Chuang', Yilun Han 18 Yu Huang", Fernando Iglesias-Suarez" Sanket Jantre
Karthik Kashinath 12, Marat Khairoutdinov ,Thosteen Kurth Nicholas J. Lutsko Po-Lun Ma Griffin Mooers', J. David Neelin 15, David A. Randall Sara Shamekh
Mark A. Taylor' Nathan M. Urban '9 Janni Yuval', Guang J. Zhang
Michael S. Pritchard
UCI, LINII, CColumbia, *UCB, *MIT, DDRR Princeton, SUNIL, 9NNNI, SNI, HOSU, NVIDIA, UCCD, NNNY,, UULLA, 1CCSU 17 AI, Tsinghuu,, 19BNL, SUNY
Abstract
Modern climate projections lack adequate spatial and temporal resolution due to computational constraints. A consequence is inaccurate and imprecise predictions of critical processes such as storms. Hybrid methods that combine physics with machine learning (ML) have introduced a new generation of higher fidelity climate simulators that can sidestep Moore's Law by outsourcing compute-hungry short, hig--resolution simulations to ML emulators. However, this hybrid ML-physics simulation approach requires to ML experts because of lack training data and relevant, easy-to-use workflows. We present ClimSim, the largest-ever dataset designed for hybrid MI--physics search. It comprises multi-scale climate simulations, developed by consortium climate scientists and ML. researchers. It consists of 5.7 billion pairs of multivariate input and output vectors that isolate the influence of locally-nested, high-resolution, high-fidelity physics on a host climate simulator's macro-scale physical state.
treatment and has been inaccessible
The dataset is global in coverage, spans multiple years at high sampling frequency, and designed such that resulting emulators are compatible with downstream cou- pling into operational climate simulators. We implement a range of deterministic and stochastic regression baselines highlight the ML challenges and their scoring. co/ ww//ddaatttssaeee LLEA /Cl immiimmmm..... iisSimmmiihhhi www.hhihhrss res))
The data thttps://hugging and code (https: /ttta::://eap-seee stc. .gw.guitth.uu... io/Clllcii Cmimmimm are released openly to sup- port the development of hybrid ML-physics and high-fidelity climate simulations for the benefit of science and society.
Corresponding author: sungduk @uci.edu
2 Also available low-resolution version http::::/wuugingaac.....aatttsts lou-res) and aquaplanct version hutppp///wnggnggaaaeeecc/sssssts aqua-plane sluaet.anet).
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
1.1 Overview
Predictions from numerical physical simulations are the primary tool informing policy on climate change. However, current climate simulators poorly represent cloud and extreme rainfall physics [1, 2] despite stretching the limits of the world's most powerful supercomputers. The complexity of the Earth system imposes significant restrictions on the spatial resolution simulations [3]. Physics occurring on scales smaller than the temporal and/or spatial resolutions of climate simulations commonly represented using empirical mathematical representations called "parameterizations" Unfortunately, assumptions in these parameterizations often lead to errors that can grow into inaccuracies in the future predicted climate.
use in these
Machine learning (ML) is an attractive approach to emulate the complex nonlinear sub-resolution physics-processes occurring on scales smaller than the resolution of the climate simulator-ai lower computational complexity. Their implementation has the exciting possibility of resulting in climate simulations that are both cheaper and more accurate than they currently are [4, SJ. Current climate simulators have typical smallest resolvable scale 80-200 km, equivalent to the size of a typical U.S. county. However, accurately representing cloud formation requires resolution of 100 m or finer, demanding six orders of magnitude increase in computational intensity. Exploiting ML remains a conceivable solution to sidestep the limitations of classical computing [5]: resulting hybrid- ML. climate simulators combine traditional numerical methods- which solve the equations governing large-scale fluid motions of Earth's ' atmosphere with ML emulators of the macro-scale effects of small-scale physics. Instead of relying on heuristic assumptions about these small-scale processes, the emulators learn directly from data generated by short-duration, high-resolution simulations [4, 6-18]. The task is essentially a regression problem: in the climate simulation, an ML parameterization emulator returns the large-scale autputs-changer in wind, moisture, or temperature- that occur due to unresolved small-scale (sub-resolution) poattiioonpchyssss physics, given large-scale resolved inputs (e.g., temperature, wind velocity; see Section 4).
While several proofs concept have emerged recent years, hybrid-ML climate simulators have yet to be advanced to operational use. Obtaining sufficient training data is a major challenge impeding interest from the ML. community. This data must contain all macro-scale variables that regulate the behavior of sub-resolution physics and be compatible with downstream hybrid ML-climate simulations. Addressing this using training data from uniformly high-resolution simulations has proven be very expensive and can lead issues when coupled to host climate simulation.
promising solution is to utilize multi-scale climate simulation methods to generate training data. Crucially, these provide clean interface between the emulated high-resolution physics and the host climate simulator's ' planetary-scal dynamics [19]. In theory, this makes downstream hybrid coupled simulation approachable and tractable. In practice, the full potential of multi-scale methods remains largely untapped due to a scarcity of existing datasets, exacerbated by the combination of operational simulation code complexity and the need for domain expertise in choosing variables.
We introduce ClimSim, the largest and most physically comprehensive dataset for training ML emulators of atmospheric storms, clouds, turbulence, rainfall, and radiation for use in hybrid-ML climate simulations. ClimSim a comprehensive collection of inputs and outputs from physical climate simulations using the multi-scale method. ClimSim was prepared by atmospheric scientists and climate simulator developers to lower the barriers to entry for ML experts on this important problem. Our benchmark dataset serves a foundation for developing robust frameworks that emulate parameterizations for cloud and extreme rainfall physics, and their interaction with other sub-resolution processes. These frameworks enable online coupling within the host coarse-resolution climate simulator, ultimately improving the performance and accuracy of climate simulators used for long-ten projections.
1.2 Concepts and Terminology from Earth Science
Convective Parameterization: In atmospheric science, "convection" refers to storm cloud and rain development, as well as the associated turbulent air motions. Convective parameterizations represent the integrated effects of these processes, such as the vertical transport of heat, moisture, and momentum within the atmosphere, and condensational heating and drying, on the temporal and spatial

---

scale of the host climate simulator [20-22], Stochastic parameterizations represent sub-resolution ("sub-grid scale" in the terminology of Earth science) effects as stochastic processes, dependent on grid-scale variable inputs [23, 24] to capture variations arising from sub-grid scale dynamics.
Multi-Scale Climate Simulators: Multi-scale climate simulation is technique that represents convection without convective parameterization, by deploying smaller-scale, high-resolution cloud-resolving simulator nested within each host grid column of climate simulator [25-29]. The smaller-scale simulator explicitly resolves the detailed behavior of clouds and their turbulent motions both a higher spatial and temporal resolution (but with smaller domain) than the host simulator. This improves the accuracy of the host simulations, but comes at a high computational cost [30, 31]. The time-integen integraatd and horizontally averaged influence of the resolved convection is fed upscale to the host climate simulator, and is the target of hybrid ML-climate simulation approaches.
Significance of Precipitation Processes for Climate Impacts: In climate simulations, changes in precipitation with warming is a particularly important issue. The frequency of extreme precipitation events increases with warming [32-34], with corresponding societal impacts [35]. Current elimate simulators agree on the direction of this change, but exhibit large spread in the quantitative rate of increase with warming [36, 37].
2 Related Work
There have been several recent efforts to produce hybrid-ML emulators using multi-scale climate simulations, analogous to ClimSim [4, 10-16, 38]. Most of these focused on simple aquaplanets [4, 10-13, 16, 38] and those that included real geography [14, 15] did not include enough variables for complete land-surface coupling. to our knowledge. Most examine simple multi-layer perceptrons except for [12, 15]. who used a ResNet architecture, and [39] who used a variational encoder-decoder that accounts for stochasticity. Although downstream hybrid testing in gcog-ggogyyphy settings is error-prone, [15] demonstrates some hybrid stability. Compressing input data avoid causal confounders may improve downstream accuracy [16], and methods have been proven to enforce physical constraints [40, 41].
Compared to the training data used above, ClimSim's 's comprehensive variable coverage is unprece- dented, including all variables needed couple to and from land system simulator and enforce physical constraints. Its availability across coarse-resolution, high-resolution, aquaplanet and real- geography cases is also to the community. Successful ML innovations with ClimSim can have a downstream impact since it is based on a state-of-the-ar multi-scale climate simulator that is actively supported by mission agency (U.S. Department of Energy).
In non-multi-scall settings, an important body of related work [6-9] has made exciting progress on using analogous hybrid ML. approaches to reduce biases in uniform resolution climate simulations, including in an operational climate code with land coupling and downstream hybrid stability [17, 18] (see Supplementary Information; SI), Other related work includes full model emulation (FME) for short-term weather prediction [42-44]. Whether this approach is possible for climate simulation using the high-frequency output of its state variables remains open question. For instance, it has recently been shown that incorporating spherical geometry and resolution invariance through spherical Fourier neural operators leads to stability of long rollouts [43]. While ClimSim is focused on hybrid-ML climate simulation and we do not demonstrate FME baselines, ClimSim contains full atmospheric state variable sampling well suited for the task.
3 ClimSim Dataset Construction
ClimSim presents a regression problem with mapping from a multivariate input vector, with inputs x Rd. of size d; = 124 and targets y d of size d, =128 (Figure 1). The input represents the local vertical structure (in horizontal location and time) of macro-scale state variables
Experiment Outline:
multi-scale physical climate simulator before any adjustments from sub-gru ab-grrid scale convection and radiation are made. The input also includes concatenated scalars containing boundary conditions of incoming radiation at the top of the atmospheric column, and land surface model constraints at its base. The target vector contains the tendencies of the same state variables representing the redistribution of mass and water, mierophysical water species conversions, and radiative heating feedbacks associated with explicitly resolved convection. This brackets the change
3

---

MA
Figure 1: The spatially-local version of ClimSim that our baselines are scored on. A spatially-globb version the problem that expands to the full list of variables would be useful to try.
in atmospheric state after tens of thousands of computationally intensive, spatially nested simulators integration. The ultimate goal is to
of explicit cloud physics have completed a temporally-neste outsource these physics ML by mapping inputs to targets comparable fidelity. The target vector includes scalar fields and fluxes from the bottom of the atmospheric column expected by the land surface model component that must couple to; atndhhhoopphere coupling is important predicting regional water cycle dynamics [45, 46]. Importantly, ClimSim also includes the option for expanded inputs x E Rd: of size d, 617 and targets Rdd of size d, 368, which we demonstrate one of our experiments.
Locality vs. Nonlocality: A spatially-global version of the problem could be of practical use for improving ML via helpful spatial context [47, 48]. In such a case, the problem becomes 2D - 2D regression, and would encompass inputs Rd: of maximum size d, 617 21,600 (grid columns) and targets, E gd, of maximum size d, = 368 x 21,600. Here the second dimension represents the unstructured "cube-sphere" computational mesh used by the climate model, which is list of grid cell locations that span the surface of the sphere [49]. In contrast to typical image-to-imagee translation or spatio-temporal prediction problems ML that involve data on structured grid (i.e. rectilinear), the task at hand is of lower dimensionality. Further details about the climate simulator configuration, simulations, and data, including complete variable lists, can found in SI.
Dataset Collection: We ran the E3SM-MMF multi-scale climate simulator [28, 29, 49, 50), using multiple NVIDIA A100 GPUs for total of 9,800 GPU-hourS. We saved global instantancous values of the atmospheric state before and after high-resolution calculations occurred, isolating state updates due explicitly-resolve moist convection, boundary layer turbulence, and radiation; details of the climate simulator configuration can be found in SI. These data were saved at 20-minute intervals (i.e. the time step of the climate model) for 10 simulated years, resulting in 5.7 billion samples for the high-resolution simulation that uses an unstructured "cube-sphere" horizontal grid with 21,600 grid columns spanning the globe. This grid yields an approximate horizontal grid spacing 1.5%, but unlike traditional climate model that maps points across the sphere using two dimensions aligned with cardinal north/south and east/west directions, unstructured grids use single dimension to organize the horizontal location of points. The atmospheric columns at each location and time

---

treated as independent samples. Thus, the total number of samples can be understood by considering that atmospheric columns each location and time are treated as independent samples, such that 5.7 billion x 21,600 horizontal locations per time step x 72-time steps per simulated day x 3,650 simulated days).
important to note that each sample retains a 1D structure corresponding to the vertical variation across 60 levels. We also ran two additional simulations with approximately ten times less horizontal resolution, with only 384 grid columns spanning the globe, resulting in 100 million samples for each simulation. These low-resolution options allow for fast prototyping of ML models, due to smaller training data volumes and less geographic complexity. One low- resolution simulation uses "aquaplanet" configuration, i.e., lower boundary condition of specified sea surface temperature, invariant in the longitudinal dimension with no seasonal cycle. This is the simplest prototyping dataset, removing variance associated with continents and time-varying boundary conditions. The total data volume is 41,2TB for the high-resolution dataset and 744GB for each of the low-resolution datasets.
Dataset Interface: Raw model outputs emerge from the climate simulator as standard NetCDF files which can be easily parsed in any language. Each timestep yields files containing input and target vectors separately, resulting in a total of 525,600 files for each of the three datasets. To prevent redundancy, variable metadata and grid information was saved separately.
The raw tensors from the climate simulations are initially either 2D or 3D, depending on the variable. For 2D tensors, the dimensions represent time and horizontal location. While these variables actually depend on three physical dimensions (time and 2D space), since each location on the sphere is indexed along single axis due to the climate model's unstructured horizontal grid, the apparent dimensionality is lower. Such variables include solar insolation, snow depth over land, surface energy fluxes, and surface precipitation rate. 3D tensors include the additional dimension representing altitude relative to the Earth's s surface, for height-varying state variables like temperature, humidity, and wind vector components. Separate files are used to store each timestep and variable. ClimSim includes a total of 24 2D variables and 10 3D variables (see Table SI).
Dataset Split: The 10-year datasets are divided into: (a) training and validation spanning the first 8 years (0001-02 0009-01; YYYY-MM), excluding the first simulated month for numerical spin-up, and (b) a test set spanning the remaining two years (i.c., 0009-03 to 001 0-12- A one-month gap is intentionally introduced between the two sets to prevent test set contamination via temporal correlation. Both sets are stored separately in our data repositories.
Energy use: The computing and energy costs of generating ClimSim could be viewed wasteful and having a negative consequence for society through associated emissions. We emphasize that while it can appear large, the compute used is actually orders of magnitude less than what is consumed by operational climate prediction. Associated emissions are minimized given that our integrations were performed on energy-efficiees GPU hardware. The cost must also be weighed against the potential social benefit of mitigating future energy consumption by eliminating end users' need for costly physics-based based MMF simulations. Meanwhile, a large consortium of interested parties have helped agree on this dataset, to help ensure it is not wasted.
Experiments
To guide ML practitioners using ClimSim, we provide an example ML workflow using the low- resolution, real-geggraphy dataset for the task described in Section 1. All but one of our baselines focuses on emulating the subset of total available input and target variables illustrated in Figure 1, with the following inputs & E Rdd, of size dN = 124, and targets y E Rdd of size d, = 128 (Figure 1, Table 1), chosen for its similarity to recent attempts in the literature.
Split: We divide the 8 y-ar training/validatinn set into the first 7 years (i.c., Training/Validatinn 01 in the raw filenames" "year-month" notation) for training and the subsequent I 0001-02 to 0008-01 year (0008-02 to 0009-01) for validation.
Our preprocessing steps were (1) downsample in time by using every 7th sample, (2) collapse horizontal location and time into a single sample dimension, (3) normalize variables by subtracting the mean and dividing by the range, with these statistics calculated separately at each of the 60 vertical levels for the four variables with vertical dependence, and (4) concatenate variables into multi-varian input and output vectors for each sample (Figure 1). The heating tendency
Preprocessing Workflow:
5

---

Input
Size
Target
Size
Temperature [K]
60
Heating tendency, d'/ [K/s]
60
Specific humidity [kg/kg]
60
Moistening tendency, dq/ dt [kg//g/s] Net surface shortwave flux, NETSW [W/m]]
60
Surface pressure [Pa]
Insolation [W/m']
Downward surface longwave flux, FLWDS [W/m]] Snow rate, PRECSC [m/s]
Surface latent heat flux [Wm*]]
Surface sensible heat flux [W/m')
Rain rate, PRECC [m/s] Visible direct solar flux, SOLS [W/m*]
Near-IR direct solar flux, SOLL [W/m]]
Visible diffused solar flux, SOLSD [W/m*] Near-IR diffused solar flux, SOLLD [Wmm]]
The subset of input and target variables used in most of our experiments (Figure 1). Dimension length 60 corresponds to the total number vertical levels (discretized altitudes) of the climate simulator.
Table I:
target dT/dtt (i.e., time rate of temperature 7) was calculated from the raw climate simulator output as (Tafter - / ) At, where At = 1200 s) is the climate simulator's known macro-scale timestep. Likewise, the moisture tendency was calculated via taking the difference of humidity state variables recorded before versus after the convection and radiation calculations. This target variable transformation is done so that we can compare the performance of our baseline models to that of previously published models that reported errors of emulated tendencies [14, 39]. Additionally, this transformation implicitly normalizes the target variables leading to better convergence properties for ML algorithms. Given the domain-specific nature of the preprocessing workflow, we provide scripts the GitHub repository for workflow reproduction.
4.1 Baseline Architectures
Six baseline models used in our experiment are briefly described here. Refer to SI for further details.
Convolutional Neural Network (CNN) uses a ID ResNet-style network. Each ResNet block contains two 1D convolutional layers and skip connection. CNNs learn spatial structure and have outperformed MLP and graph-based networks [51]. The inputs and outputs for the CNN are stacked in the channel dimensions, such that the mapping is 60 variables have been repeated along the vertical dimension.
60 10. Accordingly, global
Encoder-Decoder (ED) consists of an Encoder and Decoder with 6 fully-connected hidden layers each [39]. The Encoder of ED condenses the original dimensionality of the input variables down to only 5 nodes inside the latent space. This enhances the interpretability of ED and makes the model beneficial for advanced postprocessing multivariate climate data [39].
Heteroskedastic Regression (HSR) [52] predicts a separate mean and standard deviation for each output variable, using a regularized MLP.
Multi-layer Perceptron (MLP) is a fully connected, feed-forward neural network. The MLP architecture used for our experiments optimized via extensive hyperparameter search with 8,257 trials.
Randomized Prior Network (RPN) is an ensemble model [53]. Each member of the RPN is built as the sum of a trainable and a non-train non-trainable (so-called "prior") surrogate model; we used MLP for simplicity. Multiple replicas of the networks are constructed by independent and random sampling of both trainable and non- non-traine ao--trainable parameters [54, 55]. RPNs also resort to data bootstrapping (e.g., subsampling and randomization) in order mitigate the uncertainty collapse of the ensemble method when tested beyond the training data points [55].
Conditional Variational Autoencoder (cVAE) uses amortized variational inference to fit deep generative model that conditioned on the input and can produce samples from a complex predictive distribution.

---

MAE [W/m']
R²
Variable
CNN
ED
HSR
MLP
RPN
eVAE
CNN
ED
HSR
MLP
RPN
eVAB
dd/
2.585
2.864
2.845
2.683
2.685
2.732
0.627
0.542
0,568
0.589
0.617
0.590
dgf dt
4,401
4.673
4.784
4.495
4.592
4.680
NETSW
18.85
14.968
19.82
13.36
18.88
19.73
0.944
0,980
0.959
0.983
0.968
0.957
FLWDS
8.598
6.894
6.267
5.224
6.018
6.588
0.828
0.802
0.904
0.924
0.912
0.883
PRECSC
3.364
3,046
3,511
2.684
3,328
3.322
PRECC
37.83
37.250
42.38
34.33
37.46
38.81
0.077
-17.909
-68.35
-38.69
-67.94
-0,926
SOLS
10.83
8.554
11.31
7.971
10.36
10,94
0.927
0.960
0.929
0.961
0,943
0,929
SOLL
13.15
10.924
13.60
10,30
12.96
13.46
0.916
0.945
0.916
0.948
0.928
0.915
SOLSD
5,817
5,075
6.331
4.533
5,846
6.159
0.927
0,951
0.923
0.956
0,940
0.921
SOLLD
5.679
5.136
6.215
4.806
5.702
6,066
0.813
0.857
0.797
0.866
0.837
0.796
Table 2: MAE and R² for target variables averaged globally and temporally (from 0009-03 to 001 122).. Variables include heating tendency (dT) /dt), moistening tendency (dq/ dt). net surface shortwave flux (NETSW), downward surface longwave flux (FLWDS), snow rate (PRECSC), rain rate (PRECC), visible direct solar flux (SOLSS), near-IR direct solar flux (SOLL), visible diffused solar flux (SOLSD), and near-IR IR diffused solar flux (SOLLD). Units of non-energy flux variables are converted to a common energy unit, W/m?. Best model performance for each variable is bolded.
4.2 Skill Boost from Expanding Features and Targets
We performed an ablation of our best performing MLP baseline to demonstrate the added value of the expanded inputs and targets available in ClimSim, i.c. using inputs x of size d, 617 and targets y E Rd of size de = 368; see Table I in SI for the full list of variables. We use the same transformation described in our preprocessing workflow to compute and add condensate (cloud liquid and cloud ice) and momentum (zonal and meridional winds) tendencies to the target vector. We conducted this ablation study with both the low-resolution and the high-resolution datasets (see Section 3.1 in SI for further details regarding these MLP variants). For common elements of the target vector, using all available variables leads to uniform improvement in prediction accuracy, especially for precipitation, in both resolutions (Figures S17, S18 and Table S14). The larger errors (e.g.. MAE and RMSE) observed in the high-resolution emulators are anticipated due to the increased variance of nigher-resolution data. Nevertheless, the similarity of their R values to those of the corresponding low-resolution emulators confirms their adequate performance.
4.3 Evaluation Metries
Our evaluation metrics are computed separately for each variable in the output vector. Mean absolute error (MAE) and the coefficient of determination (R2) are calculated independently at each horizontal and vertical location, and then averaged horizontally and vertically to produce the summary statistics in Figure 2. For the vertically-varying fields, we first form masswwiggh and then convert moistening and heating tendencies into common energy units in Watts per square meter as in [56]. We also report continuous ranked probability scores (CRPS) for all considered models in SI.
4.4 Baseline Model Results
Figure 2 summarizes the error characteristies. Whereas heating and moistening rates have comparable global mean MAE, behind a common background vertical structure (Figure 2 b.c) the coefficient of determination R² (d,e) reveals that certain architectures (RPN, HSR, cVAE, CNN) consistently perform better in the upper atmosphere (model level < 30) whereas the highly optimized MI.P model outperforms in the lower atmosphere (model level 30) and therefore the global mean (Table 2). For the global mean MAE we see the largest averaged errors for PRECC and NETSW (mean MAE > 15 W/m?, Figure 2 and Table 2), where MLP clearly has the best the best skill compared to all other benchmark models. For the other variables, the global mean MAE is considerably smaller and the skill of the benchmarks model appears to be more similar in absolute numbers. While for the global mean R we find the lowest measurable performance for d'T/dt and PRECC (mean R² < 0.7) and in these cases, CNN gives the most skillful predictions. The other variables have larger R² of order 0.8 or higher, which suggests that these quantities are easier to deep-learn (Table 2). For dq/dt and PRECSC global mean R² is not an ideal evaluation metric due to negligible variability in dq/dt the upper atmosphere and for PRECSC in the tropics in the dataset (Table 2).
7

---

Additional tables and figures that reveal the geographic and vertical structure of these errors, fit quality, and analysis of stochastic metries, are included in SI (Sections 4.3, 8.1, and 8.2 SI).
40
(a) MAE
MLP
RPN
W/m2
HSR
CVAE
20
0
dT//tt
da/dt
NETSW
FLWDS
PRECSC
PRECC
SOLS
SOLSD
SOLLD
(b) MAE, dT/dt
(c) MAE, dq/dt
W/m2
4
Wmm
2
5
1.0
(d) R2, dT/dt
1.0
(e) R2, dq/dt
0.5
0.5
0.0
0.0
10
20
30
40
50
60
Level index
Level index
Figure 2: (a) Summary, where dt/dt and dg/dt are the tendencies of temperature and specific humidity, respectively, and were vertically integrated with mass weighting. (b,c) retain the vertical structure of MAE and (d.e) R2. Error bars and grey shadings show the the 5- to 95-percentile range of MLP. Refer to Table for variable definitions.
4.5 Physics-Informen Guidance to Improve Generalizability and Coupled Performance
Physical Constraints: Mass and energy conservation are important criteria for Earth system simula- tion. these terms are not conserved, errors estimating sea level rise or temperature change over time may become as large the signals we hope measure. Enforcing conservation on emulated results helps constrain results be physically plausible and reduce the potential for errors accumu- lating over long time scales. We discuss how to do this and enforce additional constraints, such as non-negativity for precipitation, condensate, and moisture variables in the Supporting Information.
Stochasticity and Memory: The results of the embedded convection calculations regulating do are chaotic, and thus worthy of stochastic architectures, as in our RPN, HSR, and cVAE baselines. These solutions are likewise sensitive to sub-grid initial state variables from an interior nested spatial dimension that has been included in our data.
Temporal Locality: Incorporating the previous timesteps' target or feature in the input vector inflation could be beneficial it captures some information about this convective memory and utilizes temporal autocorrelations present in atmospheric data.
Causal Pruning: A systematic and quantitative pruning of the input vector based objectively assessed causal relationships to subsets of the target vector has been proposed as an attractive preprocessing strategy, as it helps remove spurious correlations due confounding variables and optimize the ML algorithm [16].
Normalization that goes beyond removing vertical structure could be strategic, such as removing the geographic mean (e.g., latitudinal, land/sea structure) or composite seasonal variances (e.g., local smoothed annual cycle) present in the data. For variables exhibiting exponential variation and approaching zero the highest level (e.g., metrics of moisture), log-normalization might be beneficial.
Normalization:

---

Expanded Resolution and Complete Inputs and Outputs: Our baseline models have focused on the low-resolution dataset, for ease of data volume, and using only a subset of the available imputs and outputs. This illustrates the essence of the MI challenge. However, we show in our ablation study, using MIPPs, that including all input variables yields generally an improved reproduction of the target variables in both the low-resolution and the high-resolution dataset (Figures SI7 and SI8 and Table SI4). Accordingly, we encourage users who discover competitive fits in this approachable limit to expand to all inputs/outputs in the high-resolution, real-geography dataset, for which successful fits become operationally relevant.
Further ML Approaches: Recent methods to capture multi-scale processes using neural operators that learn in a iiisertizatiooninvaaaan manner and can predict at higher resolutions than available during training time [57] may be attractive. Their performance can be further enhanced by incorpo- rating physics-informed losses at a higher resolution than available training data [58]. Ideas on MI. modeling for sub-grid closures from adjacent fields like turbulent flow physics and reactive flows can also be leveraged for developing architectures with an inductive bias for known priors [59], easing prediction of stiff non-linear behavior [60-62], generative modeling with physical constraints [63, 64] and for interpretability of the final trained models [60].
5 Limitations and Other Applications
Idealizations: A limitation of the multi-scale climate simulator used to produce ClimSim (E3SM- MMF) is that it assumes scale separation, i.c., that convection can be represented as laterally periodic within the grid size of the host simulator, and neglects sub-grid scale representations of topographic and land-surface variability. Despite these simplifications, the data adequately captures many essential aspects of the ML problem, such as stochasticity, and interactions across radiation, microphysics, and turbulence.
Hybrid testing: Inclusion of natural path for downstream testing of learned physics emulators as fully coupled components of a hybrid-ML climate simulator is vital. However, such a workflow not yet included in ClimSim, since there is no easy way for the ML community to run many hybridized variants of the E3SM-MME in a distributed begh-performance GPU computing infrastructure via a lightweight API. It is our eventual goal to tackle the software engineering needed to enable such protocol, since, in the long term, it is in this downstream environment where ML researchers should expect to have their maximum impact on the field of hybrid-ML climate simulation. Meanwhile, ClimSim provides the first step.
One open problem that the dataset may allow assessing is understanding the role of stochasticity in hybrid-ML simulation. While primarily used as a dataset for regression, it would be also interesting to assess and understand the degree to which different variables are better modeled as stochastic or deterministic, or if the dataset gives rise to heavy-tailed or even multi-modal conditional distributions that are important to capture. To date, these questions have been raised based on physical conjectures [e.g.. 65] but remain to be addressed in the ML-based parameterization literature. For instance, precipitation distributions have long tails that are projected to lengthen under global warming [34, 66] and will thus tend to generate -a--ple extremes. ClimSim could help construct optimal architectures to capture precipitation tails and other impactful climate variables such as surface temperature.
Stochasticity:
This dataset could also be utilized to discover physically interpretable models for atmospheric convection, radiation, and microphysics. A possible workflow would apply dimensional- ity reduction techniques to identify dominant vertical variations, followed by symbolic regression to recover analytic expressions [67, 68].
Interpretabilliy:
Although the impacts of global warming and inter-annual variability are absent in this initial version of ClimSim, important questions surrounding climate-convecttoon interactions can begin to be addressed. One strategy would involve partitioning the data such that the emulator is trained on cold columns, but validated on warm columns, where warmth could be measured by surface temperatures, as in [56]. However, the results from this approach may also reflect the dependence of convection on the geographical distribution of surface temperatures the current climate and should be interpreted with caution. To optimally engage ML. researchers in solving the climate generalization problem, a multi-climate extension of ClimSim should be developed that includes physical simulations that samples future climate states and more internal variability.
Generalizability:

---

Relevance determination and active learning: While the climate simulator code offers data generation flexibility, guidance on ideal regimes target for improved learning would benefit the domain scientists able to run it. This question can be addressed with the current data and metrics of interest provided.
Conclusion and Future Work
We introduce ClimSim, the most physically comprchensive dataset yet published for training ML emulators of atmospheric storms, clouds, turbulence, rainfall, and radiation for use in hybrid-ML climate simulation. It contains all inputs and outputs necessary for downstream coupling in full- complexity multi-scale climate simulator. We conduct a series of experiments on a subset these variables that demonstrate the degree which climate data scientists have been able to fit their deterministic and stochastic components.
We hope ML community engagement in ClimSim will advance fundamental ML methodology and clarify the path to producing increasingly skillful sub-grid physics emulators that can be reliably used for operational climate simulation. To facilitate two-way commications between ML practitioners and climate scientists, we incorporate many desired characteristics for an ideal benchmark dataset suggeted in [69]. Such interdisciplinary collaboration will open up exciting future in which the computational limits that currently constrain climate simulation can be reconsidered.
We plan to soon extend ClimSim to include, first, sampling of multiple future climate states. Second, we aim to provide a protocol for downstream hybrid simulation testing. We hope lessons learned in our chosen limit of multi-scale atmospheric simulation will have applicability in other sub-fields of Earth System Science where computational constraints are currently a barrier to including explicit representations of more systems of nested complexity.
10

---

Acknowledgements
This work is broadly supported across countries and agencies. Primary support is by the National Science Foundation (NSF) Science and Technology Center (STC) Learning the Earth with Artificial Intelligence and Physics (LEAP), Award # 2019625-STE and the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy (DOE) Office of Science (SC), the National Nuclear Security Administration, and the Energy Exascale Earth System Model project, funded by DOE grant DE-SC0022331 M.S.P, S.Y., L.P., A.M.J., J.I.., N.L.,, and G.M. further acknowledge support from the DOE (DE- DE--SC0033368 and NSF (AGS-1912134), R.Y. S.M, P.G. M.P. acknowledge funding from the DOE Advanced Scientific Computing Research (ASCR) program (DE-SC0022255), V.E., P.G., G.B., and F.I.-S. acknow funding from the European Research Council Synergy Grant (Agreement No. 855187) under the Horizon 2020 Research and Innovation Programme. E.A.B. was supported, in part, by NSF grant AGS-22 10068. S.J. acknowledges funding from DOE ASRC under an Amalie Emmy Noether Fellowship Award in Applied Mathematics (B&R #KJ04400100. M.A.B acknowledges NSF funding from an AGS-PRF Fellowship Award (AGS- 2218197). R.G. acknowledges funding from the NSF (DGE-21 212911) and the U.S. Department of Defense (DOD). S.M. acknowledges support from an NSF CAREER Award and NSF grant 1IS-2007719. I.Z.. and N.L., received MILInES research funding by the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a DOE SC User Facility operated under Contract No. DE-ACO2-05CHII 1231. The Pacific Northwest National Laboratory is operated by Battelle for the DOE under Contract DE-ACO5-76RLOD the auspices of the DOE by Lawrence Livermore National Laboratory under Contract DE-AC52- 07NA27344. This work used Bridges2 at the Pittsburgh Supercomputing Center through allocation ATM190002 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF grants #2 22138259, #2138286, #21 #2138307, #2137603, and #2138296. This work also utilized the DOD High Performance Computing Modernization Program (HPCMP).
This work was performed under
References
[1] IPCC, Climate Change 2021: The Physical Science Basis. Contribution of Working Group to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. 2021.
[2] S. Sherwood, J. Webb, J. D. Anman, K. C. Armour, P. M. Forster, J. C. Hargreaves, G. Hegerl, S. A. Klein, K. D. Marvel, E. Rohling, et al., "An assessment of earth's 's climate sensitivity using multiple lines of evidence," Rev. Geophys., vol. 58, no. 4, p. 22019RG00067
2020.
[3] T. Schneider, J. Teixeira, C. Bretherton, F. Brient, K. G. Pressel, C. Schär, and A. P. Siehesma, "Climate goals and computing the future of clouds," " Nat. Clim. Change, vol. 7, no. 1, pp. 3-5, 2017.
[4] P. Gentine, M. Pritchard, S. Rasp, G. Reinaudi, and G. Yacalis, "Could machine learning break the convection parameterization deadlock?," Geophys. Res. Lett., vol. 45, no. 11, pp. 5742-5751, 2018.
[5] V. Eyring, V. Mishra, G. P. Griffith, L. Chen, T. Keenan, M. R. Turetsky, S. Brown, F. Jotzo, C. Moore, and van der Linden, "Reflections and projections on decade of climate science," Nat. Clim. Change, vol. 11, no. 4, pp. 279-285, 2021.
[6] C. S. Bretherton, B. Henn, A. Kwa, N. D. Brenowitz, O. Watt-Meyer, J. McGibbon, W. A. Perkins, S. K. Clark, and L. Harris, "Correcting coarse-grid weather and climate models by machine learning from global storm-resolving simulations," J. Adv. Model. Earth Syst., vol. 14, no. 2, p. 2202IMS00279,, 2022.
[7] K. Clark, N. D. Brenowitz, B. Henn, A. Kwa, McGibbon, W. A. Perkins, O. Watt-Meyer C. S. Bretherton, and I.. M. Harris, "Correcting 200 km resolution climate model in multiple climates by machine learning from 25 km resolution simulations," Journal of Advances in Modeling Earth Systems, vol. 14, no. 9, p. c2022M5003219, 2022.

---

[8] A. Kwa, S. K. Clark, B. Henn, N. D. Brenowitz, J. McGibbon, O. Watt-Meyer, W. A. Perkins, L. Harris, and C. S. Bretherton, "Machine-learned climate model corrections from global storm-resolving model: Performance across the annual cycle," J. Adv. Model. Earth Syst., vol. 15, no. 5, p. e2022M5003400, 2023.
[9] C. H. Sanford, A. Kwa, O. Watt-Meyer, S. K. Clark, N. D. Brenowitz, J. McGibbon, and C. S. Bretherton, "Improving the reliability of ml-corrected climate models with novelty detection," Authorea Preprints, 2023.
[10] S. Rasp, M. S. Pritchard, and P. Gentine, "Deep learning to represent subgrid processes in climate models," Proc. Natl. Acad. Sei. USA, vol. 115, no. 39, pp. 9684-9689, 2018.
[11] N. D. Brenowitz, T. Beucler, M. Pritchard, and C. S. Bretherton, "Interpreting and stabilizing machine-learning parameterizations of convection," J. Atmos. Sci., vol. 77, no. 12, pp. 4357- 4375, 2020.
[12] Y. Han, G. Zhang, X. Huang. and Y. Wang, "A moist physies parameterization based on deep learning," Adv. Model. Earth Syst., vol. 12, no. 9, p. e2020MS002076, 2020.
[13] Ott, M. Pritchard, N. Best, E. Linstead, M. Curcie, and P. Baldi, "A fortran-keras deep learning bridge for scientific computing," 2020. arxiv:2004. 10652.
[14] G. Mooers, M. Pritchard, T. Beucler, J. Ott, G. Yacalis, P. Baldi, and P. Gentine, "Assessing the potential deep learning for emulating cloud superparameterization in climate models with real- geography boundary conditions," J. Adv. Model. Earth Syst., vol. 13, no. 5, p. c2020MS002385, 2021.
[15] X. Wang. Y. Han, W. Xue, G. Yang, and G. J. Zhang, "Stable climate simulations using realistic general circulation model with neural network parameterizations for atmospheric moist physics and radiation processes," Geosci. Model Dev., vol. 15, no. 9, pp. 3923-3940, 2022.
[16] Suurez, Gentine, B. Solino-Fernandez, T. Beucler, M. Pritchard, Runge, and V. Eyring, "Causally-ifformed deep learning to improve climate models and projections," " 2023. arxiv:2304.12952.
[17] Yuval and P. A. O'Gorman, "Stable machine-leamnin parameterization of subgrid processes for climate modeling a range of resolutions," Nature Comm., vol. 11, no. 1, p. 3295, 2020.
[18] Yuval, P. A. O'Gorman, and C. N. Hill, "Use of neural networks for stable, accurate and physically consistent parameterization of subgrid atmospheric processes with good performance at reduced precision," Geophys. Res. Lett., vol. 48, no. 6, p. e2020GL091363, 2021.
[19] S. Rasp, "Coupled online learning way to tackle instabilities and biases in neural network parameterizations: general algorithms and lorenz 96 case study (v1. 0)." Geosci. Model Dev., vol. 13, no. 5, pp. 2185-2196, 2020.
[20] K. A. Emanuel, Ammosperic convection. 1994.
[21] D. Randall, Atmosphere, clouds, and climate, vol. 6. 2012.
[22] A. P. Siebesma, S. Bony, C. Jakob, and B. Stevens, Clouds and climate: Climate science's ss greatest challenge. 2020.
[23] J. W.-B. Lin and D. Neelin, "Influence of a stochastic moist convective parameterization on tropical climate variability," Geophys, Res. Lett., vol. 27, no. 22, pp. 3691-3694, 2000.
[24] J.D. Neelin, O. Peters, B. Lin, K. Hales, and C. E. Holloway, "Rethinking convective quasi- equilibrium: observational constraints for stochastic convective schemes in climate models," Phil. Trans. Royal Soc. A, vol. 366, no. 1875, pp. 2581-2604, 2008
[25] W. W. Grabowski and P. K. Smolarkiewiez, "Crep: A cloud resolving convection parameteriza- tion for modeling the tropical convecting atmosphere," Phys. D: Nonlinear Phenom., vol. 133, no. 1-4, pp. 171-178. 1999.
12

---

[26] J. Benedict and D. A. Randall, "Structure of the madden-juilll oscillation in the superparam- eterized cam," Atmos. Sci., vol. 66, no. 11, pp. 3277-3296, 2009.
[27] D. A. Randall, "Beyond deadlock," R Geophys. Res. Lett., vol. 40, no. 22, pp. 5970-5976, 2013.
[28] W. M. Hannah, C. Jones, B. R. Hillman, M. R. Norman, D. C. Bader, M. A. Taylor, I.. Leung, M. S. Pritchard, M. D. Branson, G. Lin, et al., "Initial results from the super-parameterized
e3sm," " Journal of Advances in Modeling Earth Systems, vol. 12, no. I, p. c2019MS001863, 2020.
[29] M. R. Norman, D. C. Bader, C. Eldred, W. M. Hannah, B. R. Hillman, C. R. Jones, J. M. Lee, I.. Leung, I. Lyngaas, K. G. Pressel, el al., "Unprecedented cloud resolution in gpu-enabled full-physics atmospheric climate simulation on olcf's summit supercomputer," Int. J. High Perform. Compu. Appl., vol. 36, no. pp. 93-105, 2022.
[30] D. Randall, M. Khairoutdinov, A. Arakawa, and W. Grabowski, "Breaking the cloud parameter- ization deadlock," " Bull. Am. Meteorol. Soc., vol. 84, no. 11, pp. 1547-1564, 2003.
[31] M. Khairoutdinov, C. DeMott, and D. Randall, "Evaluation of the simulated interannual and sub- seasonal variability in amip-style simulation using the multiscale modeling framework," 2008
J. Clim,, vol. 21, no. 3, pp.
[32] P. Pall, M. R. Allen, and D. A. Stone, "Testing the clausius - clapeyron constraint on changes in extreme precipitation under co2 warming," Clim. Dyn., vol. 28, no. 4, pp. 351-363, 2007.
[33] S. B. Guerreiro, H. J. Fowler, R. Barbero, S. Westra, G. Lenderink, S. Blenkinsop, E. Lewis, and X. F. Li, "Detection of continental-scale intensification of hourly rainfall extremes," Nut. Clim. Change, vol. 8, no. 9, pp. 803 807, 2018.
[34] J. D. Neelin, C. Vartinez-Villllobos,, S. N. Stechmann, F. Ahmed, G. Chen, J. M. Norris, Y.-H. Kuo, and G. Lenderink, "Precipitation extremes and water vapor: Relationships in current climate and implications for climate change," Current Clim. Change Rep., vol. 8, no. 1, pp. 17-
33, 2022.
[35] V. Davenport, M. Burke, and N. S. Diffenbaugh, "Contribution of historical precipitation change to us flood damages," Proc. Natl. Acad. Sci. USA, vol. 1 118, no. 4, p. e20175241 18, 2021.
[36] A. G. Pendergrass and D. L. Hartmann, "Two modes of change of the distribution of rain," J. Clim., vol. 27, no. 22, pp. 8357-8371, 2014.
[37] C. Martinez-Villalobo and J. D. Neelin, "Regionally high risk increase for precipitation extreme events under global warming," Sei. Rep., vol. 13, p. 5579, 2023.
[38] J. Lin, S. Yu, T. Beucler, P. Gentine, D. Walling, and M. Pritchard, "Systematic sampling and validation of machine Saaaming-Paaameteriiaatte in climate models," " Sept. 2023.
[39] G. Behrens, T. Beucler, P. Gentine, F. AgglesiasSuurrzz M. Pritchard, and V. Eyring, "Non-linear dimensionality reduction with a variational encoder decoder to understand convective processes in climate models," J. Adv. Model. Earth Syst., vol. 14,
p. e2022MS003130, 2022.
[40] T. Beucler, M. Pritchard, S. Rasp. J. Ott, P. Baldi, and P. Gentine, "Enforcing analytic constraints in neural networks emulating physical systems," Phys. Rev. Lett., vol. 126, no. 9, p. 098302, 2021.
[41] C. Reed, R. Gupta, S. Li, S. Brockman, C. Funk, B. Clipp, K. Keutzer, S. Candido, M. Uytten- dacle, and T. Darrell, "Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning," 2023. 1iii:2212.115333
[42] J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, P. Hassanzadeh, K. Kashinath, and A. Anandkumar, "Four- castnet: A global data-driven high-resolution weather model using adaptive fourier neural operators," 2022. arxiv:2202.12214
[43] B. Bonev, Kurth, C. Hundt, J. Pathak, M. Baust, K. Kashinath, and A. Anandkumar, "Spherical fourier neural operators: Learning stable dynamics on the sphere," in Proc. ICLR, 2023.
13

---

[44] R. Lam, A.
M. Willson, P. Wirnsberger, M. Fortunato, A. Pritzel, S. Ravuri, T. Ewalds, F. Alet, Z. Eaton-Rosen, W. Hu, A. Merose, S. Hoyer, Holland, Stott, O. Vinyals, S. Mohamed, and P. Battaglia, "Grapheast: Learning skillful medium-range global weather forecasting." " 2022. arxiv:2212.12794.
[45] E. M. Fischer, S. I. Seneviratne, P. L. Vidale, D. Lüthi, and C. Schär. "Soil moisture atmooshhrrr interactions during the 2003 european summer heat wave," J. Clim., vol. 20, no. 20, pp. 5081 5099, 2007.
[46] S. I. Seneviratne, T. Corti, E. L. Davin, M. Hirschi, E. B. Jaeger, I. Lehner, B. Orlowsky, and A. J. Teuling, "Investigating soil moisture-climant interactions in changing climate: A review," Earth-Sci. Rev. vol. 99, no. 3-4, pp. 125-161, 2010.
[47] P. Wang, J. Yuval, and P. A. O' Gorman, "Non-local parameterization of atmospheric subgrid processes with neural networks," " Adv. Model. Earth Syst., vol. 14, no. 10, p. e2022MS002984, 2022.
[48] B. Lütjens, C. H. Crawford, C. D. Watson, C. Hill, and D. Newman, "Multiscale neural operator: Learning fast and grid-independent pde solvers," 2022. arxiv:2207114417.
[49] W. M. Hannah, K. G. Pressel, M. Ovchinnikov, and G. S. Elsaesser, "Checkerboard patterns in e3smv2 and e3sm-mmfv2," Geosci. Model Dev., vol. 15, no. 9, pp. 6243-6257, 2022.
[50] W. M. Hannah, A. M. Bradley, O. Guba, Q. Tang, J.-C. Golaz, and Wolfe, "Separating physies and dynamics grids for improved computational efficiency in spectral element earth system models," " Adv. Model. Earth Syst., vol. 13, no. 7, p. c2020MS002419, 2021.
[51] R. Cachay, Ramesh, J. N. Cole, H. Barker, and Rolnick, "Climart: A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models," 2021. arxiv:2111.14671
[52] E. Woi, A. Boyd, V. Fortuin, and S. Mandt, "Understanding pathologies of deep het- eroskedastic regression," 2023. arxiv:2306.16717.
[53] Osband, Aslanides, and A. Cassirer, "Randomized prior functions for deep reinforcement learning," 2018. 1rxiv:1806.033
[54] Y. Yang, G. Kissas, and P. Perdikaris, "Scalable uncertainty quantification for deep operator networks using randomized priors," Comput. Methods Appl. Mech. Eng., vol. 399, p. 1 115399, 2022.
[55] M. A. Bhouri, M. Joly, R. Yu, S. Sarkar, and P. Perdikaris, "Scalable bayesian optimization with high-dimensional outputs using randomized prior networks," 2023. arxi:2302.002660.
[56] T. Beucler, M. Pritchard, J. Yuval, A. Gupta, L. Peng, S. Rasp, F. Ahmed, P. A. O'Gorman, J. D. Neelin, N. J. Lutsko, and P. Gentine, "Climate-invariant machine learning," 2021. arxiv:2112.084400
[57] Z. Li, N. Kovachki, K. Azizzadeneshelii B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, "Fourier neural operator for parametric partial differential equations,"" 2021. arxiv:2010.08895.
[58] Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadeneshhll, and A. Anand- kumar, "Physics-ifformed neural operator for learning partial differential equations," 2023. arxiv:2111.03794,
[59] J. Ling, A. Kurzawski, and J. Templeton, "Reynolds averaged turbulence modelling using deep neural networks with embedded invariance," J. Fluid Mech., vol. 807, pp. 155-166, 2016.
[60] F. MacArt, J. Sirignano, and J. B. Freund, "Embedded training of turbulence models," Phys. Rev. Fluids, vol. 6, no. 5, p. 050502, 2021.
subgrid-scar
[61] V. Xing, C. Lapeyre, T. Jaravel, and T. Poinsot, "Generalization capability of convolutional neu- ral networks for progress variable variance and reaction rate subgrid-sean scale modeling," Energies, vol. 14, no. 16, p. 5096, 2021.
14

---

[62] M. P. Brenner, J. D. Eldredge, and J. B. Freund, "Perspective on machine learning for advancing fluid mechanies," Phys. Rev. Fluids, vol. 4, p. 100501, 2019.
[63] A. Subramaniam, M. L. Wong, R. D. Borker, Nimmagadda, and S. K. Lele, "Turbulence enrichment using physics-informed generative adversarial networks," 2020. arxiv:2003.01907.
[64] B. Kim, V. C. Azevedo, N. Thuerey, T. Kim, M. Gross, and B. Solenthaler, "Deep fluids: A generative network for parameterized fluid simulations," Comput. Graph. Forum, vol. 38, no. 2, pp. 50-70, 2019.
[65] W.-B. Lin and J. Neelin, "Toward stochastic moist convective parameterization in general circulation models," Geophys. Res. Lett., vol. 30 (4), 1162, 2003.
[66] P. A. O' O'Gommnn, "Precipitation extremes under climate change," Current Clim. Change Rep.. vol. 1, pp. 49-59, 2015.
[67] L. Zanna and T. Bolton, "Data-driven equation discovery ocean mesoscale closures," Geophys. Res. Letr., vol. 47, 17, p. c2020GL088376, 2020.
[68] A. Grundner, T. Beucler, P. Gentine, and V. Eyring, "Data-driven equation discovery of cloud cover parameterization." 2023. arxiv:2304.0003.
[69] I. Ebert-Uphoff, D. R. Thompson, I. Demir, Y. R. Gel, A. Karpatne, M. Guereque, V. Ku- mar, Cabral-Cam and P. Smyth, "A vision for the development of benchmarks to bridge geoscience and data science," in 17th International Workshop on Climate Informatics, 2017.
15


Reviews:
[{"review_id": "0igAk2PMsD", "reviewer": "Reviewer_HQ4J", "summary": "", "questions": "", "limitations": "Yes, the authors have included a comprehensive discussion on limitations.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This climate dataset is the largest and most physically comprehensive for hybrid-ML research. Also, a benchmark is provided to serve as a foundation for future research. \n\n- This paper is well-written and very detailed in the context of dataset and benchmark.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "byvy3iEjTD", "reviewer": "Reviewer_pLrq", "summary": "", "questions": "", "limitations": "Yes, the manuscript clearly addresses these, far more so than most papers in my stack.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "I see several clear strengths of the paper. My comments on the strengths are brief because there is not much to say. \n\n### Public well-done runs from a simulator\n\nGetting credible simulators to work properly requires expertise that is usually beyond the ML community. I do not work in climate science, but do work in another area with similar characteristic (astrophysics). Having this data available at the given scale and with the given format is incredibly helpful  and a great opportunity for the community.\n\n### Extensive benchmarking\n\nThe authors have tried a variety of flavors of methods for solving the problem and have the results in the paper. These results will be helpful for the community. \n\n### Good analysis and many extra potential directions pointed to\n\nThe paper does a good job of pointing to issues and potential opportunities for researchers. \n- While there are some additional analyses that would improve the work, the current analysis is more than adequate for a paper. \n- There are plenty of ideas for how to extend the the proposed baselines. Sometimes it is claimed that the onus falls on the researchers producing the dataset to run these extensions themselves. I'd preemptively push-back on this point. Simply having a basic version of methods is extremely helpful and often enough to get the community started.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VBkXPPWQMc", "reviewer": "Reviewer_uz8R", "summary": "", "questions": "", "limitations": "Limitations of the given approach and data set are discussed in section 5.\n", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* An impressive 40 TB of data, split into training and test sets\n* 3 scenarios at different scales or complexity\n* Baseline results for several ML approaches (MLP, RPN, CNN, HSR, cVAE)\n* A set of Python scripts that were used to produce the baseline results, which can serve as a template for new methods\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9pE6vzzrLH", "reviewer": "Reviewer_ujho", "summary": "", "questions": "", "limitations": "The authors addressed some of the limitations of their work. However, I'm very surprised there was no mention of the societal impact due to the processing power required. There could even be positive ones,  for example:\n* a positive could be the fact that the simulation was done on a very streamlined/efficient system\n* a large consortium of interested parties agreed on a dataset to ensure it isn't wasted.\n\nA brief statement might be worth adding.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The scale of the dataset is massive. They address the biggest problem with training extensive ML methods is the fact that they are data hungry. This is an excellent start to training large-scale models.\n* They target a wide range of relevant variables, which could be helpful for different people involved in climate science applications. There could be an overlap between the variables used and the case studies one wishes to address.\n* There are detailed preprocessing steps (which people may want to modify).\n* The data is hosted on the HuggingFace platform, which is excellent for visibility and ease of access. This is the first climate-based data.\nFrom an ML perspective, it answers some key questions:  1) The problem we are trying to solve, 2) The subsequent toy problem/reduced problem we wish to solve, 3) The training data (and structure) provided, 4) Results comparison.\n* There are some solid benchmark models which span a wide range of assumptions were included. They were evaluated fairly and consistently.\n* The authors include an excellent detailed overview of improving the proposed models based on physics-informed considerations.\n* Lastly, there was a good overview of some of the limitations and potential applications for which this dataset can be used.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ptEF6kPcGv", "reviewer": "Reviewer_PWUd", "summary": "", "questions": "", "limitations": "The limitations are sufficiently described in the paper. The limitations are ones that should be expected at this level of development of the dataset and do not detract from the contribution.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "There are clear advantages to using a hybrid climate simulation, if the ML models can provide sufficient accuracy for the high-resolution data. This work will facilitate the development of such research. The dataset is comprehensive and the multiple levels of resolution allow researchers to develop ML models on a reasonably-sized dataset and then test their model with the  high-resolution dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper proposes a climate simulation dataset and benchmark. A climate dataset with varying resolutions of climate simulated data is produced. ML models for climate simulation are likely to provide substantial gains. Past work in this area has been on simplified simulations, with limited baselines, and evaluations. The proposed work produces a dataset from a large simulation as well as multiple baselines that are evaluated for their ability to predict the simulation. The authors additionally provide baseline results for five methods and exemplary Python worksheets to get started.

All reviewers, while not climate experts, see a potential for this benchmark,  due to its size, to enable the advancement of machine learning research for climate model surrogates.

All reviewers agree for the quality of the paper. Its a well written paper, and a well-organised benchmark dataset, with associated code, documentation, and Python workbooks to compare ML surrogate model performance for climate research. The work is original and stands apart the existing available datasets related to climate research, due to its big size, varying resolutions and comprehensive evaluation setup.

Key strengths include:
- An impressive 40 TB of data, split into training and test sets
- Three scenarios at different scales or complexity
- Baseline results for several ML approaches (MLP, RPN, CNN, HSR, cVAE)
- Data is hosted on the HuggingFace platform, which is excellent for visibility and ease of access. 
-  A set of Python scripts that were used to produce the baseline results, which can serve as a template for new methods

Reviewers did not identify major concerns, and several suggestions for improvements have been a result of in-depth interactions between authors and reviewers, during the review process. Authors further improved this work based on the suggestions of the reviewers.
Decision: Accept (Oral)