Multiscale Neural Operator: Learning Fast and Grid-independent PDE Solvers
Authors: Björn Lütjens
Catherine H. Crawford
Campbell D Watson
Christopher Hill
Dava Newman
Venue: Submitted to ICLR 2023 (2023)
Keywords: physics-informed machine learning
pinns
scientific machine learning
neural ODEs
neural operators
machine learning
neural networks
Matryoshka
multiphysics
multiscale
parametrizations
closure
subgrid
superstructures
partial differential equations
PDEs
differential equations
numerical solvers
physics
hpc
surrogate
reduced order modeling
model reduction
uncertainty quantification
climate
fluid dynamics
physics
computational physics

Abstract: Numerical simulations in climate, chemistry, or astrophysics are computationally too expensive for uncertainty quantification or parameter-exploration at high-resolution. Reduced-order or surrogate models are multiple orders of magnitude faster, but traditional surrogates are inflexible or inaccurate and pure machine learning (ML)-based surrogates too data-hungry. We propose a hybrid, flexible surrogate model that exploits known physics for simulating large-scale dynamics and limits learning to the hard-to-model term, which is called parametrization or closure and captures the effect of fine- onto large-scale dynamics. Leveraging neural operators, we are the first to learn grid-independent, non-local, and flexible parametrizations. Our \textit{multiscale neural operator} is motivated by a rich literature in multiscale modeling, has quasilinear runtime complexity, is more accurate or flexible than state-of-the-art parametrizations and demonstrated on the chaotic equation multiscale Lorenz96.

Under review as a conference paper at ICLR 2023
MULTISCALE NEURAL OPERATOR: LEARNING FAST AND GRID-INDEPENDENT PDE SOLVERS
Anonymous authors Paper under double-blind review
ABSTRACT
Numerical simulations in climate, chemistry, or astrophysics are computation- ally too expensive for uncertainty quantification parameter-exploration at high- resolution. Reduced-order or surrogate models are multiple orders of magnitude faster, but traditional surrogates inflexible inaccurate and pure machine learning (ML)-based surrogates too data-hungry. We propose a hybrid, flexible surrogate model that exploits known physics for simulating large-scale dynamics and limits learning to the hard-to-model term, which called parametrization or closure and captures the effect of fine- onto large-scale dynamics. Leveraging neural operators, we are the first to learn grid-independent, non-local, and flexible parametrizations. Our multiscale neural operator is motivated by a rich litera- ture in multiscale modeling, has quasilinear runtime complexity, is more accurate or flexible than state-of-the-a parametrizations and demonstrated on the chaotic equation multiscale Lorenz96.
INTRODUCTION
Climate change increases the likelihood of storms, floods, wildfires, heat waves, biodiversity loss and air pollution (ICCC) 2018). Decision-makers rely on climate models to understand and plan for changes in climate, but current climate models computationally too expensive:
result, they are hard to access, cannot predict local changes (< 10km). fail to resolve local extremes (e.g., rainfall), and do not reliably quantifyy uncertainties (Palmer al. 2019). For example, running global climate model at 1km resolution can take ten days on a 4858 x GPU node supercomputer, consuming the same electricity as coal power plants generates in one hour (Fuhrer et al. 2018).
Similarly, in molecular dynamics (Batzner et al., 2022). chemistry (Behler 2011). biology Yazddan al., 2020), energy (Zhang et 2019), astrophysics or fluids (Duraisamy al., 2201)). scientific progress is hindered by the computational cost of solving partial differential equations (PDEs) at regoltoso (Karniadakis et al. 202T). We are proposing the first PDE surrogate that quickly computes approximate solutions via correcting known large-scale simulations with learned, grid- independent, non-local parametrizations.
Multiscale Neural Operator
Grid-independent Neural Operator
Modelled Saalee--allleeeee Dynamics
Hard Model Small-Scale Scale Dynamics
Figure 1: Multiscale neural operator (MNO). Explicitly modeling scales Earth's weather too expen- sive for traditional and learning-based solvers (Palmer et al. 2019). MNO dramatically reduces the compu- tational cost by modeling the large-scale explicitly and learning the effect of fine- onto large-scale dynamics; such turbulence slowing down river stream. We embed indeeddedeennenntt neural operator in the large-scale physical simulations as "parametrization" conceptually similar stacking dolls (Snagglebit 2022).

---

Under review as a conference paper at ICLR 2023
Surrogate models are fast, reduced-order, and lightweight copies of numerical simulations (Quar-
teroni & Rozza 2014 and of significant interest physics-informed machine learning (Kashinath
et al. 2021: Reichstein
| et al. 2019; Karpatne et al. 2019; Ganguly al. 2014). Machine learning
(ML)) surrogates have simulated PDEs up to 3 order of magnitude faster than traditional
numerical solvers and more flexible and accurate than traditional surrogate models (Karniadakis
ct al. 2021). However, pure ML-based surrogates are too data-hungry (Rasp et al., (020)): so, hybrid Mh-physics models are created, for example, via incorporating known symmetries (Bronstein al.
2021 Batzner et al., 2022) or equations (Willard et al. 2022). Most hybrid models represent the so-
lution at the highest possible resolution which becomes computationally infeasible multiscale or very high-resolution physics; even at optimal runtime (Pavliotis & Stuart, 2008
Peng et al., 2021).
As depicted in Figs. and we simulate multiscale physies by running easy-to-acces large-scale models and focusing learning on the challenging task: How can we model the influence of fine- onto large-scale dynamics, iee, what is the subgrid parametrization term? The lack accuracy in current subgrid parametrizations,, also called closure or residual terms, is one of the major sources of uncertainty in multiscale systems, such as turbulence or climate (Palmer et al., 2019 Gentine be combined with incorporating equations
eet al. 2018). Learning subgrid parametrizations as soft (Raissi et al.] (2019) or hard (Beucler et al.] 2021a) constraints. Various works learn subgrid parametrizations, but are either inaccurate, hard to share inflexible because they local (Gentine et al., 2018), grid-dependent (Lapeyre et al. 2019), or domain-specific (Behler J 22077), respectively as detailed in Section 2 We are the first to formulate the parametrization problem as learning
neural operators (Anandkumar et al. 2020) to represent non-local, flexible, and grid-independent parametrizations.
We propose, multiscale neural operator (MNO), a novel learning-based PDE surrogate for multi- scale physics with the key contributions:
A corming-based multiscale PDE surrogate that has quasilinear runtime complexity, lever- ages known large-scale physics, is grid-indeppndent, flexible, and does not require autodif- ferentiable solvers.
The first surrogate to approximate grid-independent, non-local parametrizations via neural operators.
Demonstration of the surrogate on the chaotic, coupled, multiscale PDE: multiscale Lorenz96.
2 RELATED WORKS
We embed our work in the broader field of physics-informet machine learning and surrogate mod- eling. We propose the first surrogate that corrects a coarse- grasseegriied simulation via learned, grid- independent, non-local parameterizations.
Direct numerical simulation. Despite significant progress in simulating physics numerically it remains prohibitively expensive to repeatedly solve high-dimensional partial differential equations (PDEs) (Karniadakis et al., 2021). For example, finite difference, element, volume, and (pseudo- ) spectral methods have to be re-run for every choice of initial or boundary condition, grid, or parameters [Farlow| 1993 Boyd 2013). The issue arises if the chosen method does not have optimal runtime. i.c., does not scale linearly with the number of grid points, which renders it infeasibly expensive for calculating ensembles (Boyd) (013). Select methods have optimal or close-to-optimal runtime, e.g., quasi-linear O(N log N). and outperform machine learning- based methods in runtime and accuracy, but their implementation often requires significant problem-specific adaptations; for example multigrid (Briggs et al. 2000) or spectral methods (Boyd 2013). We acknowledge the existence impressive resarch directions towards optimal and flexible non-ML solvers, such as the spectral solver called Dedalus (Burns et al. 2020), but advocate to simultancously explore easy-to- adapt ML methods to create fast, accurate, and flexible surrogate models.
Surrogate modeling. Surrogate models are approximations, lightweight copies, or reduced- order models of PDE solutions, often fit to data, and used for parameter exploration un- certainty quantificiation (Smith 2013; Quarteroni & Rozza 2014). Surrogate models via SVD/POD (Chatterjee 2000), FigendecoompsittoosskLLE (Fukunaga & Koontz. 1970). Koopman
2

---

Under review as a conference paper at ICLR 2023
operators/DMD (Williams al. 2013), take simplying assumptions to the dynamics, e.g.. lineariz- ing the equations, which can break down in high- digh-dimensional or nonlinear regimes (Quarteroni & Rozza 2014). Instead, our work leverages the expressiveness of neural operators as universal ap- proximations (Chen & Chen 1995) to learn fast nigh-dimensional surrogates that are accurate in nonlinear regimes (Lütjens et al., 2022]] Yuval et al. 202T Costa Nogueira et al. 2020 Nogueira Jr.
ct al. 2021). Pure ML-baaed surrogate models have shown impressive sucess in approximating
dynamical systems from ground-truth simulation data for example with neural ODEs (Rackauckas
et al. 2020; Chen et al. 2018: Hasani et al. (021). GNNs (Brandstetter et al. [022 Cachay et al.
2021a), CNNs (Stachenfeld et al. (022), neural operators eLi |00|]]022aa] Anandkumar et al. 2020
Pathak et al., [022; Lu et al. [02] Jiang et al. 2021). RNNs (Kani & Elsheikh [2017; Rasp et al.
2020), GPs (Chakraborty et al. 2021 reservoir computing (Pathak et al. 2018 Nogueira Jr. et al.
2021 or transformers (Chattopadhyay et al. 2020a) - but, without incorporating physical knowlege
become data-hungry and poor at generalization KKrniadakis et al. 2021 Beucler et al. 2021b).
Physics-informed machine learning, Two main approaches of incorporating physical knowledge
into ML systems is via known symmetries (Bronstein et al.) (02T) or equations (Karniadakis al.)
2021].
Our approach leverages known equations for computing a coarse-grid prior; which is com-
plementary to using known equations as soft (Raissi et al. 2019 Lee & Carlberg, 2020) et al,
202
Wu et al., 2020 Zhang et al. 2018 Yazdani | 2020) or hard constraints (rryydanus al.
2019
Lutter et al. 2019, Beucler et al. 202 Donti et al. 2021; Beucler et al. 2019 Jin et al.
2020
as these methods can still be used to constrain the learned parametrization. In terms of sym-
metry, our approach exploits translational equivariance via Fourier transformations (Li et al. 2021a),
but can be extended other frameworks that exploit in- or equivariance of PDEs (Olver 1986) to
rotational (Fuchs et al. 2020 Thomas et al.] 2018), Galilean (Wu et al.) 2018 Prakash et al.
2021
scale (Beucler al. [202Tb), translational (Subel et al. 202T). reflectional (Cohen & Welling 2017)
(Coeeen
or permutational (Zhou et al. 2020) transformations.
The field of physics-informed machine learning is very broad, as reviewed most recently in (Willard itt al. 2022) and (Karniadakis et al. [02T] Carleo et al. 2019 Karpatne et al. 2017). We focus on the task of learning fast and accurate surrogate models of fine-scale models when fast and approximate
coarse-grained simulation is availabe. This task differs from other interesting research areas in
equation discovery or symbolic regression (Brunton et al., 2016; Long al. 2018b] 2019 Liu
et al. [021 Qian et al. 2022), downscaling or superresolution (Xie et al. 20118 Bode et al. 2021
Kurinchi- Vendhan et al. 2022 Stengel et al. [2020] Vandal et al. Groenke et al. 2020) design
space exploration or data synthesis CCen & Ahmed/ Chan & Elsheikh 2019). controls (Bieker et al.
2020) or interpretability (Toms et al. 2020 McGGaww
Barnes 2018). Our work complementary
to data assimilation parameter calibration (Jia 2019; 2021| Karpatne al. 2017 Zhang
jtt al. 2019 Bonavita & Laloyaux|2022
which fit to observational data instead of models and differs
from inverse modeling and parameter estimation (Parish & Duraisamy 2016 Hammlton al. 2017
Yin et al. 2021 Long et al. 2018a) which usually fit parametrizations that are independent of the previous state.
Correcting coarse-grid simulations via parametrizations. Problems with large domains are of- ten solved via multiscale methods (Pavhotis & Stuari, 2008). Multiscale methods simulate the dy- namics on a coarse-grid and capture the effects of small-scale dynamics that occur within a grid cell via additive terms, called subgrid parametrizations, closures, residuals (Pavliotis & Stuart 2008 McGuffie & Seunderson-Sellery 2005). Existing subgrid parametrizations for many equations are still inaccurate (Webb al. 2015) and ML outperformed them by learning parametrizations directly from high-resolution simul iimulations; for example in turbulence (Duraisamy et al.) 2019). climate (Gen- tine et al. 2018), chemistry (Hansen et al. 2013). biology (Peng et (021). materials (Liu al., 2022). or hydrology (Bennett & Nijssen 2002)) The majority of ML- based parametrizations, how- ever, is local (Gentine et 2018 O O'Gorman & Dwyer 2018
Brenowitz & Bretherton] 2018
Brenowitz et 2020 Bretherton et al. 2022; Yuval et al.) 2021 Cachay et [202 Bennett & Nijssen 2020 Hansen et al.) 2013 Liu et al. 2022; Prakash et al. 2021 Ling et al.) 2016)) Parish & Duraisamy, 2016 Wu et al.) 2018 8 Rasp 2020). i.e., the in- and output are variables of single
grid points, which assumes perfect scale separation, for example, in isotropic homogeneous turbu- lent flows (P., 2006). However, local parametrizations are inaccurate; for example in the case of anisotropic nonhomogeneous dynamics (P. 2006 Wang et al.) 2022). for correcting global error of coarse spectral discretizations (Boyd 2013). large-scale climate models (Dueben & Bauer
3

---

Under review as a conference paper at ICLR 2023
Multieale Nound Operabor
Neural
Operator
Ke
a At)
Lange-seale
N(ū))
model
N
000
Figure 2: Left: Model Architecture. A physics- based model, .N,, can quickly propagate the state, üt, at large- scale, but will accumulatc the error, h N(u) Wu. A neural operator, KGs wraps the computational and implementation complexities unmodeled fine-scale dynamics into a non-local and grid-independent term, h, that iteratively corrects the large-scale model. Right: Multiscale Lorenz96. We demonstrate multiscale neural operator (MNO) on the multiscale Loren296 equation, model chaotic atmospheric dynamics. Im-
age: Rasp 2020
2018 Pathak et al. 2018). More recent works propose non-local parametrizations, but their for-
mulations either rely on fixed-resolution grid (Wang al. 2022 Blakseth al. 2022 Lapeyre
ct al. 2019 Chattopadhyay et al., 2020b). an autodifferentiaale solver (Um et al. 2020: Sirignano
al. 2020 Frezat al. (022). or formulated for specific domain (Behler (007). A single
work proposes non-local and grid- frid-inndepedeet parametrizations (Pathak et al. (020), but requires
the explicit representation of a high-resolution state which computationally infeasible for large domains, such as in climate modeling. We are the first to propose grid-independent and non-local parametrizations via neural operators to create fast accurate surrogate models fine-scale sim- ulations.
Neural operators for grid-independent,, non-local parametrizations. Most current learning- based non-local parametrizations rely FCNNs, CNNs (Lapeyre al. 2019), RNNs (Chat- topadhyay et al. (020b),. which are mappings between ditee-dimensional spaces and thus grid- dependent. In comparison, neural operators learn mappings in between infinie-diimensional func- tion spaces (Kovachki et al., 202T) such as the Laplacian, Hessian, gradient, or Jacobian. Typi- cally, neural operators lift the input into rrddiindeeendett state such as Fourier (Li al. (2021)), Eigen- (Bhattacharya et al. (2020), graph kernel (Li et al. [2020] AAnandkumar et al.) 2020) or other latent (Lu et al.] [2021) modes and learn weights in the lifted domain. We are the first formulate neural operators for learning parametrizations.
3 APPROACH
We propose multiscale neural operator (MNO): a surrogate model with quasilinear runtime com- plexity that exploits know coarse-grained simulations and learns grid-independent, non-local parametrization. As detailed in the following MNO propagates the dynamics according to:
du
N(ū
Öt
Corrected Largo-seale Dyn.
Large-scale Dyn.
Parametrization
3.1 MULTISCALE NEURAL OPERATOR
Partial differential equations. We focus on partial differential equations (PDEs) that can be writ- ten initial value problem (IVP) via the method of lines (William 1991). The PDEs in focus have one temporal dimension, t E [0, =: D,, and (multiple) spatial dimensions, x = [r1,..,, E Dr, and can be written in the iterative, explicit, symbolic form (Farlow] 1993):
Ou
Nu)) with x E [0, D
Øt
u(0, x) = = (tu x) = with x Dry
(1, x
In our case, the (non-)linear operator, N, encodes the known physical equations: for example a combination of Laplacian, integral, differential, etc. operators. Further, D, Dr D,, is the
4

---

Under review as a conference paper at ICLR 2023
solution to the initial values, u² D= - Du. and Dirichlet, Bp[u] = u u-b or Neumann boundary conditions, Bx [u] = n 022 - bys with outward facing normal on the boundary. n.
Scale separation. We transfer a concept from the rich and mathematical literature in multiscale modeling (Pavliotis & Stuart 2008) to consider filter kernel operator, Gv, that creates the large- scale solution, u(z) x(x) + u (z), where u' are the small-scale deviations and denotes the JDD ((zz,=")((z"ddz))) Assuming the kernel, G, 1) preserves constant fields, a a. 2) commutes with differentiation, [g*, L with 8=z, L, and 3) is linear,
filtered variable, (xx) = G
x+ レ =
P. 2006). we can rewrite 2 to:
du dū
g
9 (u() =N(ū) (1+
Öt Øt
where (Gu) G (((u Nig
u) is the filter subgrid parametrization, closure term, or commutation emror, i.c., the error introduced through propagating the coarse-grained solution.
Approximations of the subgrid parametrization an operator that acts domain expertise and are derived on problem-speciffe basis. In the case of isotropic homogeneous turbulence, for example, the subgrid parametrization can be approximated as the spatial derivative of
require significant
the subgrid stress tensor, [G+ [
murhulence
Bre
0u
P.
2006). Many works approximate
the subgrid stress tensor with deyysics-informer ML Prakash et al. 2021 Ling et al. 2016 Parish &
Duraisamy/ 2016 Wu et al., 2018). but are domain- specitic, local, or require a differentiable solver or fixed-grid. We propose general purpose method to approximating the subgrid parametrization, independent the grid, domain, isotropy, and underlying solver.
domain-speeific,
Multiscale neural operator. We aim to approximate the parametrization filter commutation ror, N] se; h, via learning a neural operator on high-resolution training data. Let K.g be a neural operator with the mapping:
[G te Ko :((,, ((D;;
where 9 are the learned parameters and Ü, H. are separable Banach spaces of all continuous func- tions taking values, Rdd. defined the bounded, open set, D. Rdd,, with norm lIflle || = (f(r) We embed the neural operator as an autoregressive model with fixed sime-discretization, At, such that the final multiscale neural operator (MNO) model is:
Ou 02u
ū(l + At) [(L, ú, Dr dx ..... (T)
(5)
where ft,, ú, Ba 8y2
du 8'
f(+ (())dr is the known large-scale tendency, i.e. one-step solution.
MNO is fit using MSE with the loss function:
L E,EEajutt)-pl (a(K)) ((x())(((tt))
(6)
where the ground-truth data, u(t) ~ p((). is generated by integrating high-resolution simulation with varying parameters, initial boundary conditions and uniformly sampling time snippets cording to the distribution p(t). Similar to problems in superresolution. there exist multiple realiza- tions of the learned commutation error, (T), for given ground-truth, ( using MSE
will learn a smooth average and future work will explore adversarial losses (Goodfellow et al. 2014)
or an intersection between neural operators and normalizing flows (Rezende Mohamed| (015) or
diffusion- models (Sohl- Dick-Dictttin et al. (015)) to account for the stochasticity (Wilks) 2005). During training, the model input is generated via (t) = G (u(l)) and the target via
Murget (u (w).
During inference MNO initialized with a large-scale state and integrates the dynamics in time via coupling the neural operator and large-scale simulation.
5

---

Under review as a conference paper at ICLR 2023
Our approach does not need access to the high-resolution simulator or equations; it only requires precomputed high-resolution dataset, which are increasingly available (Hersbach et al. [0200] Burns et al. 2022), and allows the user to incorporate existing casy-to-accees solvers of large-scale equa- tions. There is no requirement for the large-scale solver to be autodifferentiable which significantly simplifies the implementation for large-scale models, such in climate. If desired, loss func-
tion can easily be augmented with a physics-informed loss (Raissi et al. 2019) on the large-scale dynamics or parametrization term.
Choice of neural operator. Our formulation is general enough to allow the use of many opera- (021a). PCA-based (Bhattacharya et (0.0),, low-rank (Khoo &
tors, such as Fourier (Li et al.
Ying 2019). Graph (Li et al.,
2020) operators, or DeepOnet (Wang et al. 2001 Lu et al. (021) 2021) focuses on interpolation and assumes fixed-grid sensor data,
Because DeepONet (Lu et al.
we decided to modity Fourier Neural Operator (FNO) (Li et al. 2021a) for our purpose. FNO
universal approximator of nonlinear operators (Kovachki et al. 2021; Chen & Chen 1995). grid-independent and can be formulated autoregressive model (Li et al.
2021a). As there ex- ists significant knowledge on symmetries and conservation properties of the commutation error (P. 2006), MNO's explicit formulation increases interpretability and case of incorporating symmetries and constraints. With FNO, we exploit approximate translational symmetries in the data and leave novel opportunities for neural operators that exploit the full range of known equi- and invariances of the subgrid parametrization term, such as Galilean invariance (Prakash et al. 2021), for future work.
3.2 ILLUSTRATION OF MNO VIA MULTISCALE LORENZ96
We illustrate the idea of MNO on a canonical model of atmospheric dynamics, the multiscale Lorenz96 equation Lorenz (2066) Thornes et al. (2017). This PDE is multiscale, chaotic, time- continuous, space-discretized, 2D (smace+time), nonlinear, displayed Fig. right and detailed in Appendix A.3. Most importantly, the large- and small-scale solutions, X R, Y R V E (0. J}. 1e {0, .... K). demonstrate the curse of dimensionality: the number of the small-scale states grows exponentially with scale and explicit modeling becomes computationally expensive, for example, quadratic for two-scales: O(N2) = O(JK). The PDE writes:
0Xk
- (X - +F haC
X
Σ (X))
Øt
==0
(8)
Large scale Dyn::
0% "DT"
Parametnzzaiion h
0Y
hac Xx.
-Y,
0l
where F the forcing, h2 the coupling strength, b the relative magnitude of scales, and the evolution speed. With the multiscale framework from Section 3.1] we define:
u(x) [oo, Yo,0, Yı.0.... .... YJ,0, X1, Yo.1... K(J+ 1)}
.K...., Yy, VxeD
(8xx
if-==([+1) (kE(O..... K)
(())(z) =
8Y &t
otherwise.
G(a,z)) = if 4(J + Vk E (0....,K)
0 otherwise,
with the solution, u, operator, N,, and kernel, G.
MNO learns the parametrization term via neural operator, Ko h se h, and then models:
a& ax + (Xx(xxx))()))
(9)
Øt ōt
where the known large-scale dynamics are approximated with truth parametrization is h(x) h- Σ5= (Xx, (axxx)ifa if (O. .0..... K} and otherwise). See Appendix A.4 for all terms.
ax. Oxe
and
ground-
82
85
k(J + 1)
Vk.

---

Under review as a conference paper at ICLR 2023
Runtime Complexity (log-log)
Numerical Solver
24
O(N*)
Runtime in seconds
2
MNO (ours)
log
2-2
2-3
2
2-11
2-14
2
2
2
2
2
Grid size
Figure 3: MNO faster than direct numerical simulation. Our proposed multiscale neural operator (orange) can propagate multiscale PDE dynamics in quasilinear complexity, O(N log N). For a grid with K 215. MNO is ~ 1000-times faster than direct numerical simulation (black) which has quadratic complexity, O(N2)
The parametrization, Kg., accepts inputs that are sampled anywhere inside the spatial domain, which differs from previous local (Rasp 2020) grid-dependent (Chattopadhyay et al. 2020b) Lorenz96
parametrizatioos.
We create the ground-truth data via randomly sampled initial conditions, periodic boundary con- ditions, and integrating the coupled equation with 4th-order Runge-Kutta solver. After Lya- рunov timescale the state is independent of initial conditions and extract 4K snippets with T/ At = 400steps length, corresponding to 10 Earth days, for I 1-step training. During test the model autoregressively 1K samples from a different initial condition, detailed in Appendix A.3.
4 RESULTS
Our results demonstrate that multiscale neural operator (MNO) is faster than direct numerical sim- ulation, generates stable solutions, and is more accurate than current parametrizations. We now proceed to discussing each of these in more detail.
4.1 RUNTIME COMPLEXITY: MNO IS FASTER THAN TRADITIONAL PDE SOLVERS
MNO (orange in Fig. 3) has quasilinear, O(N log N), runtime complexity in the number of large- scale grid points, - in the multiscale Lorenz96 equation. The runtime is dominated by a lifting operation, here a fast Fourier transform (FFT), which is necessary to learn spatial correlations in grid-indeppndent space. In comparison, the direct numerical simulation (black) has quadratic runtime complexity, O(N?), because of the explicit representation of > small-scale states. Both models are linear in time, (T)). Local parametrizations can achieve optimal runtime, O(N), but it is open question if there exists decomposition that replaces FFT to yield an optimal, non-local, grid-indeee inddinddeeeneent model.
We ran MNO up to resolution of K = 224, which would equal 75cm/pz in global 1D (space) climate model and only took Rs on single CPU. MNO three orders magnitude (1000-times) faster than DNS, at resolution of K = 215 or 200m /pr. For 2D or 3D simulations the gains of using MNO vs. DNS are even higher with O(V2 log N) vs. O(N+) and O(N3 log N) vs. O(N6), respectively Khairouudinov et al. 2005).
The runtimes have been calculated by choosing the best of 1-100k runs depending on grid size on single-threaded Intel Xeon Gold 6248 CPU@2.500C update which, for DNS, is the calculation of ( and for MNO the calculation of ( i.e., the sum of large-scale step and pass through the neural operator.
with 164Gb RAM. We time step
In Fig. the runtime of MNO and DNS plateaus at low-resolution (K < 29). because runtime measurement is dominated by grid-indeppndent operations. DNS plateaus lower runtime, be-

---

Under review as a conference paper at ICLR 2023
Sample Predictions
HNO lours)
Method
RMSE
HL Param.
30
Climaologyy
Param
Climatology Traditional parametrizations
6.902
2.326
MI.-hased parametrization
(Rasp et al. 2018)
2.053
-10
MNO (ours)
0.5067
Forecast
Figure 4: Left:: MNO is more accurate than traditional parametrizations. sample plot shows, that our proposed multiscale neural operator (yellow/orange-dottte))) can accurately forecast the large-scale physics (black-solid), Xx-o(t). In comparison, ML-bbased bluc-dotted) and traditional (red-dotted) parametrizations quickly start diverge. Note that the system is chaotic and small deviations rapidly amplified; even inserting the exact parametrizations in float32 instead of float64 quickly diverges. Right: Accuracy. MNO more accurate than traditional parametrizations as measured by the root mean-square error (RMSE).
cause MNO contains several fixed-cost matrix transformations. The runtime of DNS has slight discontinuity at K x 2 due to extending from cache to RAM memory. We focus on a runtime comparison, but MNO also has significant savings in memory: representing the state K 247 in double precision occupies 64GB RAM for DNS and 0.5MB for MNO.
4.2
MNO IS MORE ACCURATE THAN TRADITIONAL PARAMETRIZATIONS
Figure left shows a forecasted trajectory of a sample at the left boundary, & = 0. where MNO (orange-dotted) accurately forecasts the large-scale dynamics, (t(t). (black-solid) while current ML-based (blue-dotted) (Gentiiee et al.) 2018) and traditional parametrizations (red-dotted) quickly diverge. The quantitive comparison of RMSE and mean/std plot Fig. 4 over 1K samples and 200steps or 10days (At = 0.005 36min) confirms that MNO is the most accurate in compari- son to ML-based parametrizations, traditional parametrizations, and mean forecast (climatology). Note, the difficulty of the task: when forecasting chaotic dynamics even numerical errors rapidly amplify (P. 2006).
ML-based parametrizations is a stte-offthhear (SoA) model in learning parametrizations and trains a ResNet to forecast a local, grid-independent parametrization, h& NN(Xx), similar to (Gentine et al. 2018). The traditional parametrizations (trad. param.) are often used in practice and use linear regression to learn local, grid-independent parametrization (MeGuffie Henderson- Sellers 2005). It was suggested that multiscale Lorenz96 is too easy as a test-case for comparing offline models because traditional parametrizations already perform well (Rasp) 2019), but the sig- nificant difference between MNO and Trad. Params. shows that online evaluation still interesting. The climatology forecasts the mean of the training dataset, Xx 1 Et I/N > (t). The full list of hyperparameters and model parameters can be found in Appendix A.5.2. For fair- ness, we only compare against grid- iideinndeeendent methods that do require an autodifferentiable solver. Models with soft or hard constraints, e.g., PINNs (Raissi et al.) 2019) or DC3 (Donti et al., 2021), are complementary to MNO. Further, note that our implementation of MNO uses an a priori loss function and could likely be improved by implementing loss functions that propagates the loss over multiple time steps similar (Frezat al. 2022) which requires an autodifferentiable solver or (Brandstetter et al. 2022) which does not.
a posteriori loss functions, i.e.,
4.3 MNO IS STABLE
Figure shows that predicting large-scale dynamics with MNO is stable. We first plot randomly selected sample of the first large-scale state, Xt= (left-black), to illustrate that the prediction is bounded. The MNO prediction (left-yellow) follows the ground-truth up to an approximate horizon of, 1.8 or 9 days, then diverges from the ground-truth solution, but stays within the bounds of the ground-truth prediction and does not diverge infinity. The RMSE over time in Figure shows that MNO (yellow) is approximately more accurate than current ML-based (blue) and traditional (red) parametrizations for ~ 100%-longer time, measuring the time intersect with climatology. Despite the difficulty in predicting chaotic dynamics, the RMSE of MNO reaches a plateau, which slightly above the optimal plateau given by the climatology (black).
8

---

Under review as a conference paper at ICLR 2023
RMST
10
NM parsm
Climsooloyy
RAGE
Forecastrange,
(a) Long-term sample forecast, (t())
(b) Error over time
Figure 5: MNO stable. MNO can propagate a sample state, Xt)o(t), over a long time horizon without diverging to infinity (left). The right plot shows that the RMSE MNO plateaus long-term forecasts, further confirming stability. Further, MNO (yellow) maintains accuracy longer than MI--based parametrizations (blue) and a climatology (black).
The RMSE over time is calculated as:
1 RMSE(() = K
Σ Σ
(8)=(1) xxx((1)3).
(10)
k= t=
5 LIMITATIONS AND FUTURE WORK
We demonstrated the accuracy, speed, and stability of MNO on the chaotic multiscale Lorenz96 equation. Future work, can extend MNO towards higher-dimensional or time-irregular systems and further integrate symmetries or constraints:
The results show promise extend MNO to higher-dimensional, chaotic, multiscale, multiphysics problems. We demonstrate the first steps towards
turbulence and Rayleigh- Benard convection Appendix [A.]] [ and aim towards integration in large-scale global atmospheric models,
e.g., as approximation of cloud processes (Wang et al., 2022 Palmer et al., 2019). Reducing the cost of climate models could dramatically improve uncertainties (Lüttens al. 2021) or decision-
exploration (Rooney- Varga et al. 2020).
MNO is grid-independent in space but not in time which could alleviated via integrations with Neural ODEs (Chen al. 2018). MNO is myopic model which might suffice for chaotic dy- namics Li et al. 202 2021b). lb), but could combined with LSTMs (Mohan et al. 2019) or reservoir computing Pathak al., 2018) to contain memory. Further, leveraged global Fourier de- compositions to exploit grdd-indeennden periodic spatial correlations, but future work could also capture local discontinuities, e.g., along coastlines (Jiang et al. 2021) with multiwavelets (Guptal et al. 2021), or incorporate non-periodic boundaries via Chebyshev polynomials.
Lastly, MNO can be combined with Geometric deep learning, PINNs, hard constraint models. This avenue of research is particularly exciting with MNO as there exist many known symmetries for various paramtrization terms (Prakash et al. 2021).
CONCLUSION
We proposed hybrid physics-MI. surrogate multiscale PDEs that is quasilinear, accurate, and stable. The surrogate limits learning to the influence of fine- onto large-scale dynamics and is the first to use neural operators for grid-independent, non-local corrective term of large-scale sim- ulations. We demonstrated that multiscale neural operator (MNO) is faster than direct numerical simulation (O(N log N) vs. O(N2)). and more accurate (=s 100% longer prediction horizon) than state-of-the. afrtt------rttt parametrizations the chaotic, multiscale equations multiscale Lorenz96. With the dramatic reduction in runtime MNO could enable rapid parameter exploration and robust uncertainty quantification in complex climate models.

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
Climate change is defining challenge of our time and environmental disasters will become more frequent: from storms, floods, wildfires and heat waves biodiversity loss and air pollution (IPCC) 2018). The impacts of these environmental disasters will likely be unjustly distributed: island states, minority populations, and the Global South are already facing the most severe consequences of climate change, while the Global North is responsible for the most emissions since the industrial revolution (Althor et al. 2016). Decision- require more accurate, accessible, and local tools to understand and limit the economic and human impact of changing climate (Palmer al. 2019). We propose multiscale neural operator (MNO) to improve the parametrizations in climate models, thus leading to more accurate predictions. Related techniques MNO, specifically neural operator- based surrogate models, could help reduce computational complexity of large-scale weather and climate models. The reduced computational complexity would make them more accessible low- resource countries or allow for higher resolution predictions. Unfortunately, discoveries for faster differential equations solvers can and likely will be leveraged in ethically questionable fields, such as missile development or oil discovery. We acknowledge the possible negative impacts and hope that our targeted discussion and application to equations from climate modeling can steer the our work towards positive impact.
REPRODUCIBILITY STATEMENT
The code will be submitted in comment directed to the reviewers and area chairs with a link to an anonymous repository once the discussion forum is opened. The code will be made publicly available upon acceptance along with an open-source license, data, and instructions to reproduce the main results. A list of hyperparameters per model is detailed in Appendix [A.6.2] data splits are explained in results; all simulation details in Appendix[A.4 and background of the neural op- erator model in Appendix A.3] Conducting the study from ideation to publication used a total of approximately 10K CPU hours on an internal cluster.
REFERENCES
Glenn Althor, James E. M. Watson, and Richard A. Fuller. Global mismatch between greenhouse gas emissions and the burden of climate change. Scientific Reports, 6, 2016.
Anima Anandkumar, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differ- ential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Korn- bluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural net- works for data-efficient and accurate interatomic potentials. Nature Communicutions, 13, 2022.
Jörg Behler. Neural network potential-energy surfaces chemistry: tool for large-scale simula- tions. Phys. Chem. Chem. Phys., 13, 2011.
Parrinello M. Behler Generalized neural-network representation of high-dimensional potential- energy surfaces. Phys Rev Lett., 14, 2007.
Andrew Bennett and Bart Nijssen, Deep learned process parameterizations provide better represen- tations of turbulent heat fluxes in hydrologic models. Earth and Space Science Open Archive, pp- 20, 2020.
Tom Beucler, Stephan Rasp. Michael Pritchard, and Pierre Gentine. Achieving Conservation of Energy in Neural Network Emulators for Climate Modeling. jun 2019.
Tom Beucler, Michael Pritchard, Stephan Rasp, Jordan Ott. Pierre Baldi, and Pierre Gentine. En- forcing analytic constraints in neural networks emulating physical systems. Phys. Rev. Lett., 126: 098302, Mar 2021a.
10

---

Under review as a conference paper at ICLR 2023
Tom Beucler, Michael S. Pritchard, Janni Yuval, Ankitesh Gupta, Liran Peng, Stephan Rasp, Fiaz Ahmed, Paul A. O'Gorman, David Neelin, Nicholas J. Lutsko, and Pierre Gentine. Climate- invariant machine learning. CoRR, 2021b.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model re- duction and neural networks for parametric pdes, 2020.
Katharina Bieker, Sebastian Peitz, Steven L. Brunton, J. Nathan Kutz, and Michael Dellnitz. Deep model predictive flow control with limited sensor data and online learning, 2020.
Sindre Stenen Blakseth, Adil Rasheed, Trond Kvamsdal, and Omer San. Deep neural network enabled corrective source term approach to hybrid analysis and modeling. Neural Networks, 146: 181-199, 2022.
Mathis Bode, Michael Gauding, Zeyu Lian, Dominik Denker, Marco Davidovic, Konstantin Klein- heinz, Jenia Jitsev, and Heinz Pitsch. Using physics-informa enhanced super-resolution gener- ative adversarial networks for subfilter modeling in turbulent reactive flows. Proceedings of the Combustion Institute, 38(2):2617-2625, 2021.
Massimo Bonavita and Patrick Laloyaux. Machine learning for model error inference and correction. Journal of Advances in Modeling Earth Systems, 12(12), 2020.
J.P. Boyd. Chebyshev and Fourier Spectral Methods: Second Revised Edition. Dover Books on Mathematics. Dover Publications, 2013.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations (ICLR), 2022.
N. D. Brenowitz and C. S. Bretherton. Prognostic validation of neural network unified physics parameterization. Geophysical Research Letters, 45(12):6289-6228, 2018
Noah D. Brenowitz, Tom Beucler, Michael Pritchard, and Christopher Bretherton. Interpreting and stabilizing machine-learning parametrizations of convection. Journal of the Atmospheric Sciences, 77(12):4357 4375, 2020.
Christopher S. Bretherton, Brian Henn, Anna Kwa, Noah D. Brenowitz, Oliver Watt-Meyer, Jeremy McGibbon, W. Andre Perkins, Spencer K. Clark, and Lucas Harris. Correcting coarse-gu grid weather and climate models by machine learning from global storm-resolving simulations. Jour- nal of Advances in Modeling Earth Systems, 14(2), 2022.
William L. Briggs, Van Emden Henson, and Steve F. McCormick. A Multigrid Tutorial (2nd Ed.). Society for Industrial and Applied Mathematics, USA, 2000. ISBN 08987146211
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning:
Grids, groups, graphs, geodesies, and gauges. CoRR, 2021.
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):333--399) 3937, 2016.
Keaton J. Burns, Geoffrey M. Vasil, Jeffrey S. Oishi, Daniel Lecoanet, and Benjamin Brown. Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research, 2(2), April 2020.
Randal Burns, Gregory Eyink, Charles Meneveau, Alex Szalay, Tamer Zaki, Ethan Vishniac, Ak- shat Gupta, Mengze Wang, Yue Hao, Zhao Wu, and Gerard Lemson. Johns hopkins turbulence database, 2022. last accessed May, 2022.
Salva Rühling Cachay, Emma Erickson. Arthur Fender C. Bucker, Emnest Pokropek, Willa Potosnak, Suyash Bire. Salomey Osei, and Björ Lütjens. The world as a graph: Improving el niño forecasts with graph neural networks, 2021a.
11

---

Under review as a conference paper at ICLR 2023
Salva Rühling Cachay, Venkatesh Ramesh, Jason N. S. Cole, Howard Barker, and David Rolnick. ClimART: A benchmark dataset for emulating atmospheric radiative transfer in weather and cli- mate models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
J. Campin, C. Hill, H. Jones, and J. Marshall. Super-paraamettrization in ocean modeling: Applica- tion to deep convection. Ocean Modelling, 36:90-101, 2011.
Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer. Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Rev. Mod. Phys., 91, Dec 2019.
S. Chakraborty, S. Adhikari, and R. Ganguli. The role of surrogate models in the development of digital twins of dynamic systems. Applied Mathematical Modelling, 90:662-681, 2021.
Shing Chan and Ahmed H. Elsheikh. Parametric generation of conditional geological realizations using generative neural networks, 2019.
Anindya Chatterjee. An introduction to the proper orthogonal decomposition. Current Science, 78 (7):808 817, 2000.
Ashesh Chattopadhyay, Mustafa Mustafa, Pedram Hassanzadeh, and Karthik Kashinath. Deep spa- tial transformers for autoregressive data-driven forecasting of geophysical turbulence. In Pro- ceedings of the 10th International Conference on Climate Informatics, C12020, pp. 106-112, New York, NY, USA, 2020a. Association for Computing Machinery.
Ashesh Chattopadhyay, Adam Subel, and Pedram Hassanzadeh. Data-driven super- paper-pprameterzzation using deep learning: Experimentation with multiscale lorenz 96 systems and transfer learning. Journal of Advances in Modeling Earth Systems, 12(11). 2020b.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. Advances in Neural Information Processing Systems 31, pp. 6571-6583. Curran Associates, Inc., 2018.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917, 1995.
Wei Chen and Faez Ahmed. Padgan: Learning to generate high-quality novel designs. Journal of Mechanical Design, 143(3).
Taco S. Cohen and Max Welling. Steerable cnns. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 2017.
Alberto Costa Nogueira, João Lucas de Sousa Almeida, Guillaume Auger, and Campbell D. Watson. Reduced order modeling of dynamical systems using artificial neural networks applied to water circulation. In Heike Jagode, Hartwig Anzt, Guido Juckeland, and Hatem Ltaief (eds.), High Performance Computing, pp. 1 116-136, Cham, 2020. Springer International Publishing.
Priya Donti, David Rolnick, and J Zico Kolter. DC3: learning method for optimization with hard constraints. In International Conference on Learning Representations (ICLR), 2021.
P. D. Dueben and P. Bauer. Challenges and design choices for global weather and climate models based on machine learning. Geoscientific Model Development, 11(10):3999 4009, 2018.
Karthik Duraisamy, Gianluca laccarino, and Heng Xiao. Turbulence modeling in the age of data. Annual Review of Fluid Mechanics, 51(1))357-377, 2019.
A. Dutt and V. Rokhlin. Fast fourier transforms for nonequispaced data. SIAM Journal on Scientific Computing, 14(6):1368-1399 1993.
S.J. Farlow. Partial Differential Equations for Scientists and Engineers. Dover books on advanced mathematies. Dover Publications, 1993.
12

---

Under review as a conference paper at ICLR 2023
Hugo Frezat, Julien Le Sommer, Ronan Fablet, Guillaume Balarac, and Redouane Lguensat. A posteriori learning for geasttggost turbulence parametrization, 2022.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se(3)-transformmrs: 3d roto- translation equivariant attention networks. In Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33. Curran Asso- ciates, Inc., 2020.
O. Fuhrer, T. Chadha, T. Hoefler, G. Kwasniewski, X. Lapillonne, D. Leutwyler, D. Lüthi, C. Osuna, C. Schär, T. C. Schulthess, and H. Vogt. Near-global climate simulation at km resolution: establishing performance baseline on 4888 gpus with cosmo 5.0. Geosci. Model Dev., 11:1665
-1681, 2018.
K. Fukunaga and W.L.G. Koontz. Application of the karhunen-loève expansion to feature selection and ordering. IEEE Transactions on Computers, C-19(4)3311-318, 1970.
A. R. Ganguly, E. A. Kodra, A. Agrawal, A. Banerjee, S. Boriah, Sn. Chatterjee, So. Chatter- jee, A. Choudhary, D. Das, J. Faghmous, P. Ganguli, S. Ghosh, K. Hayhoe, C. Hays, W. Hen- drix, Q. Fu, J. Kawale, D. Kumar, V. Kumar, W. Liao, S. Liess, R. Mawalagedara, V. Mithal, R. Oglesby, K. Salvi, P. K. Snyder. K. Steinhaeuser, D. Wang, and D. Wuebbles. Toward enhanced understanding and projections of climate extremes using physics-guided data mining techniques. Nonlinear Processes in Geophysics, 21(4):777-795, 2014.
P. Gentine, M. Pritchard, S. Rasp, G. Reinaudi, and G. Yacalis. Could machine learning break the convection parameterization deadlock? Geophysical Research Letters, 45(11):5742-5751, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, volume 27. Curran Associates, Inc., 2014.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 15379-15389. Curran Associates, Inc., 2019.
Brian Groenke, Luke Madaus, and Claire Monteleoni. Climalign: Unsupervised statistical down- scaling of climate variables via normalizing flows. In Proceedings of the 10th International Con- ference Climate Informatics, CI2020, pp. 60 60-66, New York, NY, USA, 2020. Association for Computing Machinery.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- operator learning for differen- tial equations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NeurIPS). 2021.
Franz Hamilton, Alun L. Lloyd, and Kevin B. Flores. Hybrid modeling and prediction of dynamical systems. PLOS Computational Biology, 13(7):1-20, 07 2017.
Katja Hansen, Grégoire Montavon, Franziska Biegler, Siamac Fazli, Matthias Rupp, Matthias Schef- fler, O. Anatole von Lilienfeld, Alexandre Tkatchenko, and validation of machine learning methods for predicting molecular atomization energies. Journal of Chemical Theory and Computation, 9())3444-3441, 2013.
Müller. Assessment and
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time- constant networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9):6657- 7666, May 2021.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz- Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bid- lot, Massimo Bonavita, Giovanna De Chiara, Per Dahlgren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haim- berger, Sean Healy, Robin Hogan, Elías Hólm, Marta Janisková, Sarah Keeley, Patrick Laloy- aux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia de Rosnay, Iryna Rozum, Freja Vam- borg, Sebastien Villaume, and Jean-Noël Thépaut. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730); 1999-2049, 2020.
13

---

Under review as a conference paper at ICLR 2023
IPCC.
Global warming of 1.5c. ipec special report the impacts of global warming of 1.5c above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, sustainable development, and efforts to eradicate poverty, 2018.
Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan Read, Jacob Zwart, Michael S Steinbach, and Vipin Kumar. Physics guided rmns for modeling dynamical systems: A case study in simulating lake temperature profiles. In SIAM International Conference on Data Mining, SDM 2019, SIAM International Conference on Data Mining, SDM 2019, pp. 558-566. Society for Industrial and Applied Mathematics Publications, 2019.
Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan S. Read, Jacob A. Zwart, Michael Steinbach, and Vipin Kumar. Physics- -uuded machine learning for scientific discovery: An application in simulating lake temperature profiles. ACM/IMS Trans. Data Sci., 2(3), 2021.
Peishi Jiang, Nis Meinert, Helga Jordão, Constantin Weisser, Simon Holgate, Alexander Lavin, Björn Lüfjens, Dava Newman, Haruko Wainwright, Catherine Walker, and Patrick Barnard. Dig- ital Twin Earth Coasts: Developing fast and physics surrogate model for coastal floods via neural operators. 2021 NeurIPS Workshop on Machine Learning for the Physical Sci- ences (ML4PS), 2021.
Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrin- sic structure-presevvin symplectic networks for identifying hamiltonian systems. Neural Net- works, 132, 12 2020.
J. Nagoor Kani and Ahmed H. Elsheikh. DR-RNN: A deep residual recurrent neural network for model reduction. CoRR, Abs/1709.009333 2017.
George Em Karniadakis, Ioannis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3:422-440, June 2021.
A. Karpatne, Ehert-Uphoff, S. Ravela, H. A. Babaie, and V. Kumar. Machine learning for the geo- sciences: Challenges and opportunities. IEEE Transactions Knowledge and Data Engineering, 3148)114441554, Aug 2019.
Anuj Karpatne, Gowtham Atluri, James H. Faghmous, Michael Steinbach, Arindam Banerjee, Au- roop Ganguly, Shashi Shekhar, Nagiza Samatova, and Vipin Kumar. Theory-guided data science: A new paradigm for scientific discovery from data. IEEE Transactions on Knowledge and Data Engineering, (9(10)233182333 2017.
Anuj Karpatne, William Watkins, Jordan Read, and Vipin Kumar. Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling. arXiv e-prints, pp. arXiv:1710. 14311, October 2017.
K. Kashinath, M. Mustafa, A. Albert, J-L. Wu, C. Jiang, S. Esmaeilzadeh, K. Azizzadenesheli, R. Wang, A. Chattopadhyay, A. Singh, A. Manepalli, D. Chirila, R. Yu, R. Walters, B. White, H. Xiao, H. A. Tchelepi, P. Marcus, A. Anandkumar, P. Hassanzadeh, and null Prabhat. Physics- informed machine learning: case studies for weather and climate modelling. Philosophical Trans- actions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(21 2021.
Marat Khairoutdinov, David Randall, and Charlotte DeMott. Simulations of the atmospheric gen- eral circulation using cloud glool-resoliing model as a superparameteriizttoo of physical processes. Journal of the Amospheric Sciences, 62(7 D):236-2154 jul 2005.
Yuehaw Khoo and Lexing Ying. Switchnet: A neural network model for forward and inverse scat- tering problems. SIAM Journal on Scientific Computing, 41(5), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
14

---

Under review as a conference paper at ICLR 2023
Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadeneshelii Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Neural operator learning maps between function spaces. CoRR, foss/2188008481,, 2021.
Rupa Kurinchi-Vendhan, Bjöm Lütjens, Ritwik Gupta, Lucien Werner, and Dava Newman. Wiso- super. Benchmarking super-resolution methods on wind and solar data. Conference on Neural Information Processing Systems (NeurIPS) Workshop on Tackling Climate Change with Machine Learning (CCML.) 2021.
Corentin J. Lapeyre, Antony Misdariis, Nicolas Cazard, Denis Veynante, and Thierry Poinsot. Train- ing convolutional neural networks to estimate turbulent sub-grid scale reaction rates. Combustion and Flame, 203:255-2666 2019.
Kookjin Lee and Kevin Carlberg. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020. ISSN 0021-9991.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. In Advances in Neural Information Processing Systems, volume 33, pp-
Curran Associates, Inc., 2020.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. ICML, 2021a.
Zongyi Li, Nikola B. Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew M. Stuart, and Anima Anandkumar. Markov neural operators for learning chaotic systems. CoRR, abs/2106.066 2021b.
Julia Ling, Andrew Kurzawski, and Jeremy Templeton. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155- 166,166, 2016.
Burigede Liu, Nikola Kovachki, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, An- drew M. Stuart, and Kaushik Bhattacharya. A learning- based multiscale method and its applica- tion to inelastic impact problems. Journal of the Mechanics and Physics of Solids, 158:104668, 2022.
Ziming Liu, Bohan Wang, Qi Meng, Wei Chen, Max Tegmark, and Tie- Yan Liu. Machine-learnine nonconservative dynamics for new-physics detection. Phys. Rev. E, 104:055302, Nov 2021.
Yun Long, Xueyuan She, and Saibal Mukhopadhyay, Hybridnet: Integrating model-based and data- driven learning to predict evolution of dynamical systems. Proceedings of The 2nd Conference on Robot Learning. volume 87 of Proceedings of Machine Learning Research, pp. 551-560. PMLR, 29-31 Oct 2018a.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDEs from data. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pp. 3208-3216. PMLR, 10-15 Jul 2018b.
Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric- symbolic hybrid deep network. Journal of Computational Physics, 399:108925, 2019.
Edward Lorenz. Predictability- a problem partly solved. In Tim Palmer and Renate Hagedorn (eds.), Predictability of Weather and Climate. Cambridge University Press, Cambridge, 2006.
Edward N. Lorenz and Kerry A. Emanuel. Optimal sites for supplementary weather observations: Simulation with a small model. Journal of the Atmospheric Sciences, 55(3):399 414, 1998.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Kariadakis.. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3:218-229 2021.
15

---

Under review as a conference paper at ICLR 2023
Bjöm Lütjens, Catherine H. Crawford, Mark Veillette, and Dava Newman. Pce-pinns: Physics- informed neural networks for uncertainty propagation in ocean modeling. International Con- ference Learning Representations (ICLR) Workshop Al for Modeling Oceans and Climate Change, May 2021.
Björn Lürjens, Catherine H Crawford, Mark Veillette, and Dava Newman. Spectral PINNs: Fast un- certainty propagation with physics-informed neural networks. In Advances in Neural Information Processing Systems (NeurIPS) Workshop on The Symbiosis of Deep Learning and Differential Equations (DLDE), 2021.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning, In International Conference on Learning Representations (ICLR), 2019.
Andrew Majda and Xiaoming Wang. Nonlinear Dynamics and Statistical Theories for Basic Geo- physical Flows. Cambridge University Press, 2006.
Marie C. McGraw and Elizabeth A. Barnes. Memory matters: A case for granger causality in climate variability studies. Journal of Climate, 3 3188)3289- 3300, 2018.
Kendal McGuffie and Ann Henderson-Sellers. A Climate Modeling Primer, Third Edition. John
Wiley and Sons, Ltd., Jan. 2005.
Arvind Mohan, Don Daniel, Michael Chertkov, and Daniel Livescu. Compressed convolutional Istm: An efficient deep learning framework model high fidelity 3d turbulence, 2019.
Alberto C. Nogueira Jr., Felipe C. T. Carvalho, João Lucas S. Almeida, Andres Codas, Eloisa Ben- tivegna, and Campbell D. Watson. Reservoir computing in reduced order modeling for chaotic dynamical systems. In Heike Jagode, Hartwig Anzt, Hatem Ltaief, and Piotr Luszezek (eds.), Hlgh Performance Computing, pp. 56-72, Cham, 2021. Springer International Publishing.
Paul A. O'Gorman and John G. Dwyer. Using machine learning to parameterize moist convection: Potential for modeling of climate, climate change, and extreme events. Journal of Advances in Modeling Earth Systems, 10(10):2548-2) 2563,, 2018.
Peter J. Olver. Symmetry Groups of Differential Equations, pp. 77-185. Springer New York, New York, NY, 1986.
Sagaut P. Large Eddy Simulation for Incompressible Flows: An Introduction. Scientific Computa- tion. Verlng Berlin Heidelberg, 3 edition, 2006.
Tim Palmer, Bjorn Stevens, and Peter Bauer. We need an international center for climate mod- eling, 2019. URL httppssr/hbiggg....e.eeietteeiaa we-need- an- htt.::/aaeeeettaaaaaoooooo-o--ea 04/13/20.
wtt../////.........ooooooooo..ot last accessed
Ambrish Pandey, Janet D. Scheel, and Jörg Schumacher. Turbulent superstructures in rayleigh- bénard convection. Nature Communications, 9, 2018.
Eric J. Parish and Karthik Duraisamy. paradigm for data-driven predictive modeling using field inversion and machine learning. Journal of Computational Physics, 305:758-774, 2016. ISSN 0021-9991.
Jaideep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott. Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach. Phys. Rev. Lett., 120, Jan 2018.
Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten Kurth, and Marcus Day. Using machine learning to augment coarse-grid computational fluid dynamics sim- ulations, 2020.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth. David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. FourCastNet: A Global Data- driven High-resolution Weather Model using Adaptive Fourier Neural Operators. February 2022.
16

---

Under review as a conference paper at ICLR 2023
Greg Pavliotis and Andrew Stuart. Multiscale Methods Averaging and Homogenization, volume 53 of Texts in Applied Mathematics. Springer- Verlag New York, edition, 2008. ISBN 978-0-387- 73829-1.
Grace C. Y. Peng, Mark Alber, Adrian Buganza Tepole, William R. Cannon, Suvranu De. Savador Dura-Bernal, Krishna Garikipati, George Karniadakis, William W. Lytton, Paris Perdikaris, Linda Petzold, and Ellen Kuhl. Multiscale modeling meets machine learning: What can we learn?
Archives of Computational Methods in Engineering, 18:1017- 2021.
Aviral Prakash, Kenneth E. Jansen, and John A. Evans. Invariant Data-Driver Subgrid Stress Mod- eling in the Strain-Rate Eigenframe for Large Eddy Simulation. arXiv e-prints, June 2021.
Zhaozhi Qian, Krzysztof Kacprzyk, and Mihaela van der Schaar. D-CODE: Discovering closed- form ODEs from observed trajectories. In International Conference on Learning Representations, 2022.
Alfio Quarteroni and Gianluigi Rozza, Reduced order methods for modeling and computational reduction, 2014.
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, and Ali Ramadhan. Universal differential equations for scientific machine learning, ArXiv, abs/2001.04385. 2020.
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686 2019.
S. Rasp. Coupled online learning as a way to tackle instabilities and biases in neural network parameterizations: general algorithms and lorenz 96 case study (v1.0). Geoscientific Model De- velopment, 13(5):2185-2166 2020.
Stephan Rasp. Lorenz 96 is too easy!
machine learning research needs more re- wwrrassstepaan.ggtthaeet io/blog/
alistic toy model, July 2019, URL lorenz 10-996-81 is too easy/ last accessed May 2022.
Stephan Rasp, Michael S. Pritchard, and Pierre Gentine. Deep learning to represent subgrid pro- cesses in climate models. Proceedings of the National Academy of Sciences, 11863)):984--669, 2018.
Stephan Rasp, Peter D. Dueben, Sebastian Scher, Jonathan A. Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12(11). 2020.
Markus Reichstein, Gustau Valps-Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Car- valhais, and Prabhat. Deep learning and process understanding for data-driven earth system sci- ence. Nature, 566: 195 - 204, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, ICML' pp. 1530-1538. JMLR.org, 2015.
Juliette N. Rooney- Varga, Florian Kapmeier, John D. Sterman, Andrew P. Jones, Michele Putko, and Kenneth Rath. The climate action simulation. Simulation & Gaming, 51(2):114-140, 2020.
URL https: //en: roads c wwt.mmmteeittteratttteeeeee
Justin Sirignano, Jonathan F. MacArt, and Jonathan B. Freund. Dpm: A deep learning pde augmen- tation method with application to large-eddy simulation. Journal of Computational Physics, 423, 2020.
Ralph C. Smith. Uncertainty quantification: Theory, implementation, and applications. In Compu- fational science and engineering, pp. 382. SIAM, 2013.
17

---

Under review as a conference paper at ICLR 2023
Snagglebit.
Sloth
nesting dolls
hand painted modern russian matryoshka doll www.coo//////.e heam//eelissingggg 690540535/ last accessed 01/22.
set. 2022. URL
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceed- ings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.
Kim Stachenfeld, Drummond Buschman Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Learned
Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro simulators for turbulence. In International Conference on Learning Representations, 2022.
Karen Stengel, Andrew Glaws, Dylan Hettinger, and Ryan N. King. Adversarial super-resolution of climatological wind and solar data. Proceedings of the National Academy of Sciences, 117(29): 16805-16815, 2020.
Adam Subel, Ashesh Chattopadhyay, Yifei Guan, and Pedram Hassanzadeh. Data-driven subgrid- scale modeling of forced burgers turbulence using deep learning with generalization to higher reynolds numbers via transfer learning. Physics of Fluids, 33(3):031702, 2021.
Nathaniel Thomas, Tess H. Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and -raslaaion-equiivarian neural networks for 3d point clouds. CoRR, abs/1802.0822 2018.
Tobias Thomess, Peter Duben, and Tim Palmer. On the use of scale-dependent precision in earth system modelling. Quarterly Journal of the Royal Meteorological Society, 143(703):897-908, 2017.
Benjamin A. Toms, Elizabeth A. Barnes, and Imme Ebert-Uphoff, Physically interpretable neural networks for the geosciences: Applications to earth system variability. Journal of Advances in Modeling Earth Systems, 12(9), 2020.
Kiwon Um Robert Brand, Yun Fei, Philipp Holl, and Nils Thuerey. Solver-i-thhe-Lopp: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers. Advances in Neural Informa- fion Processing Systems, 2020.
Thomas Vandal, Evan Kodra, Sangram Ganguly, Andrew Michaelis, Ramakrishna Nemani, and Au- roop R. Ganguly. Deepsd: Generating high resolution climate change projections through single image super-resolution. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pp. 1663-1672, New York, NY, USA, 2017. Association for Computing Machinery.
Peidong Wang, Janni Yuval, and Paul A. O'Gorman. Non-local parameterization of atmospheric subgrid processes with neural networks, 2022.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances, 7(40), 2021.
Mark J. Webb, Adrian P. Lock, Christopher S. Bretherton, Sandrine Bony, Jason N.S. Cole, Ab- derrahmane Idelkadi, Sarah M. Kang, Tsuyoshi Koshiro, Hideaki Kawai, Tomoo Ogura, Romain Roehrig, Yechul Shin, Thorsten Mauritsen, Steven C. Sherwood, Jessica Vial, Masahiro Watan- abe, Matthew D. Woelfle, and Ming Zhao. The impact of parametrized convection on cloud feedback. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engi- neering Sciences, 373(2054), nov 2015.
Daniel S. Wilks. Effects of stochastic parametrizations in the lorenz '96 system. Quarterly Journal of the Royal Meteorological Society, 131 (606))33394447 2005.
Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating scien- tific knowledge with machine learning for engineering and environmental systems. ACM Comput. Surv., jan 2022.
18

---

Under review as a conference paper at ICLR 2023
Schiesser William. The Numerical Method of Lines: Integration of Partial Differential Equations. Elsevier, 1991.
Matthew O. Williams, Ioannis G. Kevrekidis, and Clarence W. Rowley, A data-driven approxima- tion of the Koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25:1432-1467, 2015.
Jin-Long Wu, Heng Xiao, and Eric Paterson. Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework. Phys. Rev. Fluids, 3:074602, Jul 2018.
Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Prabhat, and Heng Xiao. Enforcing statistical constraints generative adversarial networks for modeling chaotic dynamical systems. Journal of Computational Physics, 406, 2020.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. Tempogan: temporally coherent, volumetric gan for super-resolution fluid flow. ACM Trans. Graph., 37(4), jul 2018.
Alireza Yazdani, Lu Lu, Maziar Raissi, and George Em Karniadakis. Systems biology informed deep learning for inferring parameters and hidden dynamies. PLOS Computational Biology, 16 (11):1-19, 11 2020.
Yuan Yin, Vincent Le Guen, Jérémic Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):12401,, dec 2021.
Janni Yuval, Paul A. O'Gorman, and Chris N. Hill. Use of neural networks for stable, accurate and physically consistent parameterization subgrid atmospheric processes with good performance at reduced precision. Geophysical Research Letter, 48:e2020G1.0013633, 2021.
Yang Zeng, Jin- Jin-Long Wu, and Heng Xiao. Enforcing imprecise constraints on generative adversarial networks for emulating physical systems. Communications in Computational Physics, 30(3):635- 665, 2021.
Liang Zhang, Gang Wang, and Georgios B. Giannakis. Real-time power system state estimation and forecasting via deep unrolled neural networks. IEEE Transactions on Signal Processing, 67(1)): 4069-4077, 2019.
Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and Weinan E. Deep potential molecular dynamics: A scalable model with the accuracy of quantum mechanics. Phys. Rev. Lett., 120, Apr 2018.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changeheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. Al Open, 1:57-81, 2020.
19

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1
QUASI-GEOSTROPHIC TURBULENCE
3
3
32
24
2
2
16
1
1
0
8
-1
-1
16
-2
-2
24
32
-3
-3
-2
2
-2
0
2
w
w
(a) Ground-truth
Figure 6: Quasi-Geostrophic Turbulence (QGT). We demonstrate the first steps to extend MNO to 2D QGT. This plot shows our ground-truth training data with the direct numerical simulation, and subgrid parametriza- tion, w'
We are planning to demonstrate multiscale neural operator a high-dimensional system, specifically the one-layer quasi-gen guasi-geoottophic (QG) turbulence depicted in Fig. 6 QG turbulence is derivative of the Navier-Stoke equations and good model for atmospheric turbulence at the equator. The equations are derived from the incompressible, i.e., V O, Navier-Stokes equations by 1) taking the curl of velocity field, w V x and 2) assuming the beta-plane approximation. fo + By, and hydrostatic, By dz pg. and geostrophic, fv Bp. fu By balances (Majda & Wang Øz
8p
2006). The resulting equations, called guasi-geostrophie turbulence, are given by:
d.w + (wb,w) = v pن 300 F
(11)
w w V
where wi is the vorticity, = [u, 1- JJ7 is the velocity vector, & is the streamfunction, J[b, w) 0,y 8, 2D the nonlinear Jacobian operator. Further, the parameters are the turbulent viscosity, v, linear drag coefficient, μ, Rossby parameter, 33, and source term, F. Vorticity can be computed with & x u d,v- 0,u
Filtering the equation with kernel results in the parametrized large-scale equation given by:
0,5 + J(b, = vー μشر 30. + F (v.2) J(bhw
(12)
Paramstrization: ((().w)
We then aim approximate the subgrid-scale (SGS) parametrization, Ko h, with the neural operator, such that the final model is:
aw(r, v) @w(r,y) Ko (x, y)
(13)
(v v) 0 v) + Ko w) y)
Öt
Ot
The QG turbulence equation is solved with a pseudospectral solver in space and RK4 explicit time integration. We choose the parameters, N₂ N, 512, At 480s, = 1.25 x -11 352m² /s. 3 = 0, and Reynolds number, Re = 22 x 10 The variables are non-dimensionalized with T4 1.2 10% i.e., Alaover Al/Ta and La 504 10 /Tmm i.e., Attwover 2n / /N,. The
20

---

Under review as a conference paper at ICLR 2023
reduced system is with scale 5 4, such that N₂ = N, 128. The forcing initiates turbulent mixing and simulates slowly varying wind stress according to the solution of, F (y(t) 4y + sin(1.4t)) cos(4x + и sin(1.5t))], and 0.5| To generate turbulent chaotic dynamics that are decoupled from the initial state, the simulation is initialized with some large-scale Fourier states and warmed up for 1300days. After warm-up generate 18000 iterations (618000 at the fine-scale) from which we independently sample training and validation snippets.
3 with enstrophy injection rate, Cr(t).
A.2 BAYLEIGH-BÉNARD CONVECTION
The proposed multiscale neural operator can also leveraged for systems without explicit multi- scale formulation. We demonstrate this by formulating the MNO equations for Rayleigh-Bénard Convection equations, as displayed in Fig.
= 15.257
buoyancy 0.0
-0.8
-0.4
0.4
0.8
vorticity
-20
20
(a) Ground-truth
Figure 7: Rayleigh-Bénard Convection. We depicted a sample plot for ground- training data of the 2D RBC.
A.2.1 DETAILS AND INTERPRETATION
Rayleigh-Bénn Beneig Convection (RBC) challenging set of equations for turbulent, chaotic, and convection-domintted flows. The equation finds applications in fluid dynamics, atmospheric dy- namics, radiation, phase changes, magnetic fields, and more (Pandey et al. 2018).
So far, we have generated ground-truth dataset that we implemented with the 2D turbulent Rayleigh-Bénard Convenction equations with Dedalus spectral solver (Burns al. 2020) similar to (Pandey et al. 2018):
0u
+
u
Vu
2
Vp+b
01
Ra 1
0T
(14)
u VT
2T
Øt
/ RRaPr
V - 0
21

---

Under review as a conference paper at ICLR 2023
with temperatture/uuyynneyy T. Rayleigh number, Ra go (oOATH//Vx)) thermal expansion efficient, a, Prandtl number, Pr = V/K. momentum diffusivity or kinematic viscosity, V, thermal diffusivity, VRaFt' acceleration due to gravity, 9, temperature difference, AT, unit vector, e,
Pr Reynolds number, Re- - v. and full volume- pressure, p, Nusselt number, Nu - Ra
time average, v.),,, cell length, Lg. The equations have been non-dimensionalized with the free-fall velocity. U VgaaH, and cell height, H. In the horizontal direction, I, we use periodic bound- ary conditions and in the vertical direction, 2, we use no-slip boundary conditions for the velocity, u(z 0) = u(= = - O, and fixed temperatures, ( 0) = Lz, T( L2) = 0. The inital conditions are sampled randomly, b(z, 1 0) = L__ + + 2 (I.g 2)4,, with w ~ No,, 10-3.
We chose: Ra 2 x 10 Pr 1, L 4, H =
A.3 FOURIER NEURAL OPERATOR
Our neural operator for learning subgrid parametrizations based on Fourier neural operators (Li al. 2021a). Intuitively, the neural operator learns a to-osutt-t--solution mapping by learning global convolution kernel. In detail, it learns the operator to transforms the current large-scale state,
X (Zo.K, 1) E &xxx to the subgrid parametrization, f (fo:k, 1) = Xo.K E R xdx with number of grid points, K, and input dimensionality, dx. according to the following equations:
Eo Xook Pd + 11K1bbp
114 ( u,WT+ / no(2, x (')oo((')dd
(15)
2 2 (2, + +1 bw + +F Fe,))
f2.0 = Enn QT + 1Ix1bb
First, MNO lifts the input via a linear transform with matrix, P E Rnoxdx bias, by E R R Ixn, vector of ones, 1xx1 and number of channels, Re.. The linear transform is local in space, i.e. the same transform is applied to each grid point.
Second, multiple nonlinear "Fourier layers" applied the encoded/lifted state. The coded/lifted state's, E R KKx,, spatial dimension is transformed into the Fourier domain via fast Fourier transform. We implement the FFT as multiplication with the pre-built forward and inverse Type-1 DST matrix, J E C Clmsse X and JF E CK Xkmax respectively, returning the vector, Fg, E Chimax Xn.
The dynamics are learned via convoluting the encoded state with a weight matrix. In Fourier space, convolution is multiplication, hence each frequency multiplied with complex weight matrix across the channels, such that R E 5 Xn, Xn. In parallel to the convolution with R, the en- coded state is multiplied with the linear transform, W E R xn, and bias, bw E R RIx... From a representation pertringgpepppeieeeee the Fourier decomposition as fast and interpretable feature extraction method that extracts smooth, periodic, and global features. The linear transform can be interpreted residual term concisely capturing nonlinear residuals.
So far, we have only applied linear transformations. To introduce nonlinearities, we apply a non- linear activation function, a, at the end of each Fourier layer. While the non-smoothness of the activation function ReLu, (22) = max(0, 2). could introduce unwanted discontinuities in the solu- tion, we choose it resulted in more accurate models than smoother activation functions such as tanh or sigmoid.
Finally, the transformed state, Un is projected back onto solution space via another linear transform, Q E Rdx xn, and bias, bi.
The values of all trainable parameters, P, R. W, b., are found by using nonlinear optimization algorithm, such as stochastic gradient descent or, here, Adam Kingma & Ba (2015). We have used MSE between the predicted, fz. and ground-truth, fa, subgrid parametrizations loss. The neural operator is implemented in pytorch, but does not require an autodifferentiable PDE solver to generate training data. During implementation, we used the DFT which assumes a uniformly spaced grids,
22

---

Under review as a conference paper at ICLR 2023
but can be exchanged with non-uniform DFTs (NUDFT) transform non-uniform grids (Dutt & Rokhlin 1993).
A.4 MULTISCALE LORENZ96
A.4.1 DETAILS AND INTERPRETATION
The equation contains K variables, X E IR, and JK small-scale variables, Y5,8 E R that represent large-scale or small-scale atmospheric dynamics such as the movement of storms or formation of clouds, respectively. At every time-step each large-scale variable, Xx. influences and is influenced by- small-scale variables, Yo.J.k- The coupling could be interpreted as X causing static instability and , causing drag from turbulence or latent heat fluxes from cloud formation. The indices k. are both interpreted as latitude, while & E {0, K- K-1} indexes boxes of latitude and j E [0-....J11 indexes elements inside the box. Illustrated on a 1D Earth with cireumference of 360 that discretized with K- = 36, J = 10, one spatial step in k, would equal 10", 1 respectively Lorenz (2006;; we choose K = J = 4. A time step with At 0.005 would equal 36 minutes Lorenz 2006).
We choose a large forcing, F > 10, for which the equation becomes chaotic. The last terms each equation capture the interaction between small- and large-scale, fa,k = - hc Ej= (X,, The scale interaction defined by the parameters where = 0.5 is the coupling strength between spatial scales (with no coupling if h would be zero), b = 10 is the relative magnitude, and the evolution speed of X Y. The linear, -Xa, and quadratic terms, X model dissipative and advective (e.g., moving) dynamics, respectively.
The equation assumes perfect "scale separation" which means that small-scale variables of different grid boxes, are independent of each other given timestep, Yj (Y),Y (t) Vt. f1. k + k2. The separation of small- and large-scale variables can be along the same or different domain and the discretized variables would then be y E [0, Ar] or y E [yo, Bend]. respectively. The equation wraps around the full large- or small-scale domain by using periodic boundaries, X_
Y=+ Note that having periodic boundary conditions in the small-scale domanin allows for superparametrizatioon, i.e., independent simulation of the small-scale dynamics Campin et al. (201T) and differs from the three-tier Lorenz96 where variables at the bor- ders of the small-scale domain depend small-scale variables of the neighbouring k Thornes et al. (2017).
Xk+E:=X1,
A.4.2 SIMULATION
The initial conditions are sampled uniformly from a set of integers, ((oo ~ UC-5, -4..,, 6), as mean-zero unit-variance Gaussian Y (to) (N 1). and lower scale Gaussian Z(to) 0.05.((O, 1). The train and test set contains 4k and 1k samples, respectively. Each sample is model time unit (MTU) or 200 (=T/At) time-steps long, which corresponds to Earth days (= T/At 36min with At = 0.005) Lorenz (2006). Hence, our results test the generaliza- tion towards different initial conditions, but not robustness to extrapolation or different choices of parameters, b, h F. The sampling starts after T = 10. warmup time. The dataset uses double precision.
We solve the equation by fourth order Runge- Kutta in time with step size At 0.005, similar to (Lorenz& & Emanuel 1998). For PDE that is discretized with fixed time step, At, the ground- truth train and test data, u (t), is constructed by integrating the coupled large- and small-scale dynamics.
Note, that the neural operator only takes in the current state of the large-scale dynamics. Hence, ie., it uses the full large-scale spatial domain as input, which exploits spatial correlations and learns parametrizations that are independent of the large-scale spatial discretization.
Our method can be queried for infinite time-steps into the future as it does not use time as input.
We are incorporating the prior knowledge from physics by calculating the large-scale dynamics, dXX i.t.k.0.- Note that the small-scale physics do not need to be known. Hence, MNO could be applied to any fixed time-step dataset for which an approximate model is known.
23

---

Under review as a conference paper at ICLR 2023
Mean Predictions
10
x
2/
-2
-4
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Forecast range,
Figure 8: Mean accuracy. MNO (orange) forecasts the (solid) of the ground-truth DNS (blue) accurately in comparison to ML-based parametrizations (green) climatology (red). The standard deviations plotted as dotted lines.
A.5 APPENDIX TO ILLUSTRATION MNO VIA MULTISCALE LORENZ96
The other large-scale (LS) and fine-scale (FS) terms are
0Xx ेt
if k(J= Ak (O....,K) K}
filtered FS dynamics, (z))(x)
{0
otherwise
0& 8t
if = (J+1) Ak E (0................. .o.....K] K}
LS dynamics, (N (.)
otherwise
(16)
0Xx
with abbreviation,
X X Xx-2) - Xx + F
Ot
LS state, ū(x:) G u(x) (Xo. 0, O,. Xx]
0,
A.6 APPENDIX TO RESULTS
A.6.1 ACCURACY
Figure 8 shows that the predicted mean and standard deviation of MNO (orange) closely follows the ground-truti (blue). The ML-based parametrization (green) follows the ground-truth only for a few time steps (until ~ f 0.125). The climatology (red) depicts the average prediction in the training dataset.
A.6.2 MODEL CONFIGURATION
Multiscale Lorenz96: MNO As hyperparameters chose the number of channels, n. 64,
number of retained modes, kimax = 3, number of Fourier layers, nı 3, and no batch norm layer. The time-series modeling task uses history of only one time step learn chaotic dynamics (Li et al. 2021b). We are using ADAM optimizer with learning rate, = 0.001, step size, 20, number of epochs, n4e 2, and an exponential learning rate scheduler with gamma, y 0.9 (Kingma & Ba 2015). Training took 1 50min on a single core Intel 17-7500U CPU@2.70GHz.
Multiscale Lorenz96: ML-based parametrization The ML-based parametrizations uses
ResNet with Players = 2 residual layers that contain a fully connected network with Tluiits 32 units. The model is optimized with Adam (Kingma & Ba 2015) with learning rate 0.01, / = (0.9, 0.999), 20.
1 10 8
trained for 20Mopochs
Multiscale Lorenz96: Traditional parametrization The traditional parametrization uses least- squares to find the best linear fit. The weight matrix is computed with A x where X and are the concatenation of input large-scale features and target parametrizations, respectively. Inference conducted with ý Az.
24

---

Under review as a conference paper at ICLR 2023
A.7 NEURAL NETWORKS VS. NEURAL OPERATORS
Most work in physics-informed machine learning relies on fully-connected neural networks (FC- NNs) convolutional neural networks (Kariaaakii al. 2021 FCNNs however are mappings between finite-dimensional spaces and learn mappings for single equation instances rather than learning the PDE solver. In our case FCNNs only learn mappings on fixed spatial grids. We lever- age the recently formulated neural operators to extend the formulation to arbitrary grids. The key distinction is that the FCNN learns parameter-dependent set of weights, Pay, that has be re- trained for every new parameter setting. The neural operator is a learned function mapping with parameter-indeeendeet weights, θ, that takes parameter settings as input and returns a function over the spatial domain, Ge(ay). In comparison, the forcing term is approximated by an FCNN as fu, d (Ix: ay) Mbay (Ix) and by a neural operator fz,e( ay) Go (xx) The mappings given by:
FCNN: goug D2 + R.x..
(17)
NO: Ge Ha, (Da; + Hx (x;;
Ho, is a function space (Banach) of PDE parameter functions, ag. that map the spatial domain, Dy. onto da, dimensional parameters, such ICs, BCs, parameters, or forcing terms. Hx is the function space of residuals that map the spatial domain, Dz, onto the space of dx-dimensional residuals, Rdx.
25


Reviews:
[{"review_id": "kt33_8GPVHI", "reviewer": "Reviewer_ALWF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about a new method named MNO (Multiscale Neural Operator), which is related to the Fourier Neural Operator. It is used to learn a closure term for a chaotic system, a parameterization for the small-scale dynamics of the multi-scale Lorenz system. The dynamics can then be propagated forward by using the solver for the large scales and the surrogate for the small scales. The method is quasilinear (much faster than the full simulation), grid-independent and non-local (via FNO), and doesn't require having a differentiable solver. ", "summary_of_the_review": "Overall, I think that this is a novel approach that might be quite effective. It heavily relies on the prior work of the Fourier Neural Operator, but incorporates some other ideas and uses it for a different task. It would have been nice to see more extensive empirical results. I have quite a few clarifying questions above, but I think that the authors will be able to address them. ", "strength_and_weaknesses": "To my knowledge, it is novel to use FNO for a closure problem, and it seems like a nice way to handle multi-scale physics, creating a hybrid model. I appreciate that due to the FNO approach, this method is grid-independent. I don't often see runtime complexity plots in papers like this, and I appreciate it. I also thought that the authors did a nice job of mentioning how other research could be complementary to this approach and incorporated in future work. The results are mostly impressive compared to other parameterizations, although I have questions below about the results in Figure 8. I have some questions and comments below to improve the clarity of the writing and to check on some concerns related to quality/correctness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X9haUYcs7k", "reviewer": "Reviewer_PmjN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies a neural operator-based surrogate model, called multi-scale neural operator. It keeps the known physics knowledge, but injects machine learning to learn the closure term. The paper studies the Lorenz 96 system.", "summary_of_the_review": "While the paper brings some interesting ideas, the experiment can be more complete. I think this work is marginally below the threshold for acceptance.", "strength_and_weaknesses": "Strength: the paper raises a good point to only learn the closure using machine learning. The hybrid model usually inherits the stability of well-studied numerical solver, but faster. Further, the paper considers scale separation, potentially making the model more efficient.\n\nWeakness: it seems to the reviewer that there already exists a huge body of work studying hybrid models of the form ut = N(u) + K(u). For example, this paper (https://arxiv.org/pdf/2107.06658.pdf) takes a very comprehensive study of chaotic systems including Lorenz 96. Similarly, the hybrid framework that learns ML-closure has been studied on LES (https://arxiv.org/pdf/2010.10491.pdf) and climate simulation (https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2022MS003105). The reviewer is not very sure about the novelty of the proposed framework.\n\nBesides, the reviewer feels the numerical experiment is relatively weak. It will be more interesting to compare against standard FNO mode, or some ML-enhanced models, for example, these correcting coarse-grid simulations as discussed in the related works section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UiX9Am5GOU_", "reviewer": "Reviewer_HHZo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a multiscale PDE surrogate, where a coarse-grained numerical model and a neural operator are added. The neural operator corrects for the errors which the numerical model makes.\nThe paper promises grid independent solutions, and quasi-linear runtime complexity. \n", "summary_of_the_review": "The most pressing issues of the paper are (i) over-ambitious claims, (ii) lack of experiments, and (iii) expensive need for training data. These are discussed in detail in \"Strength and Weaknesses\". \n", "strength_and_weaknesses": "Learning multiscale PDE surrogates is an important direction, and this paper presents an interesting approach. However, at the current stage the paper is far away from publication. I want to address the three major issues of this paper, i.e. (i) over-ambitious claims, (ii) lack of experiments, and (iii) expensive need for training data.\nAdd i) In e.g. [1], Fourier Neural Operators are ablated on various operator learning problems. For the claims “grid-independent”, “flexible”, “non-local parametrizations” which are repeated mantra-like over and over again, I would have expected discussions of similar problems in this paper. Or even better, have experiments in that direction. Fourier Neural Operators in their current implementation are not really grid-independent, although claimed heavily. They can interpolate between various regular (!) grids, and I am expecting the same is true for MNO? Why is non-local parametrization so important after all? U-Nets e.g. do very well in computer vision.\nAdd ii) I honestly don’t think that 1d problems are really the place where multiscale PDE surrogate are going to shine. I would have expected large scale Navier Stokes problems, shallow water equations, or thereof. But ok, even if staying in 1 dimension there are many more equations, such as Kuramoto Sivashinsky or Korteweg-de Vries which are very interesting, and why not compare against the methods discussed in the related work, e.g. [2]. One experiment on one dataset is pretty scarce.\nAdd iii) The solver might be quasi-linear on the test set, but it still requires many many trajectories which are generated by the very same slow classical solvers the new solver should replace. That is a common chicken-egg problem in neural PDE surrogates, but especially when talking about multiscale and high resolution, I would have expected that this is addressed. That is after all a fundamental limitation.\n\n1)\tLu, Lu, et al. \"A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data.\" Computer Methods in Applied Mechanics and Engineering 393 (2022): 114778. \n2)\tUm, Kiwon, et al. \"Solver-in-the-loop: Learning from differentiable physics to interact with iterative pde-solvers.\" Advances in Neural Information Processing Systems 33 (2020): 6111-6122.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R1ygNHLq5r", "reviewer": "Reviewer_rbkc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the Multi-scale Neural Operator for ML-based closure/parameterization of PDEs with multi-scale properties. In the framework of spatial coarse-graining of PDEs, the multiscale neural operator relies on a fast/easy/simple solve of a coarse-grained problem and estimates/computes the commutator between the coarse-graining and the spatial differential operator by learning from data. Obtaining this commutator accurately allows one to run a differential equation solve entirely in coarse-grained space, while incorporating the relevant information from the smaller subgrid scales.The context of weather simulations is chosen as an example application, the improvement of which can have real-world impact in combating climate change. It is shown on the Lorenz96 model that the proposed solver is significantly faster than direct solves at high resolution, and significantly more accurate than selected competing methods.", "summary_of_the_review": "This paper introduces the Multiscale Neural Operator for closure/parameterization problems. It proposes to learn the closure commutator.\n\nWhile the stated goals of improving climate science and combating climate change are noble, and the numerical results look convincing, many required comparisons are missing, which may better situate the performance of the proposed method. It is further suggested to analyze and visualize the behavior of the method on a larger set of toy problems.\nFinally, it is pointed out that the term Multiscale Neural Operator is not very descriptive of the procedure.\n\nI do hope the authors are willing and able to address some of these comments and am open to changing my assessment if that is the case.", "strength_and_weaknesses": "Strengths: \n- The proposed method clearly shows a significant improvement in speed and accuracy in the bench marks- There is an extensive related work section- Advancing climate science is a very important goal\n\nWeaknesses:\nWork on improving climate simulations is very important and I am all for pushing this forward at full power. Alas, there are a number of weaknesses to this contribution that give me pause concerning its utility and fitness for publication at ICLR.\n\n- It is very important to note that the Fourier Neural Operator is not actually grid-independent, or alternatively to note that it is precisely as grid independent as all convolutional methods based on stencils in signal space. The only difference is that they rely on slightly different interpolation kernels. This is a very important point, and of course the original FNO paper is not up for discussion here, but this misconception should be avoided in papers that reference the FNO paper in order to avoid further confusion.\n\n- The reason the above note is very important, is that it removes a number of the perceived differences between the FNO and other convolutional approaches, for example that in LaPeyre 2019, or Bar-Sinai 2020 (not cited). The main difference that remains to other convolutional approaches is the possibility of having filters that are larger in size than the usual 3x3 or 5x5 stencils. A number of other convolutional methods must hence be compared to assess which type of model is best for closure modeling. Further, the UNet approach in Lapeyre 2019, is actually substantially non-local and actually as global as required. Similarly, for speed, it would be great to have reference benchmarks of classical methods that use non-learned closure models.\n\n- It is entirely unclear to what extent the inductive bias of the FNO, which is a convnet that can allow large-scale convolutional filters, is actually required in the application tested here. It would be great to see tests on this by progressively restricting the possible convolutional filter size and see its impact on different problem settings.\n\n- It is not explained how the Fourier Neural Operator is adapted to the current problem (was it modified? If so, how?). \n\n- In general, it would be great to have a detailed section with toy examples. Beyond the Lorenz 96 system, it would be great to see a) transport equation. what happens to a transported dirac? b) wave equation. It is possible to build a wave equation to which the solution is a superposition of two waves, one high-frequency, one low-frequency (use a forcing term or involve the zeroth derivative in the equation). Make the coarse-graining a strict low-pass allowing only one wave through. How does the closure model behave in this non-interacting setting? c) A simple phase-amplitude coupling model should be explored (phase of a low-frequency oscillation dictates amplitude of a high-frequency oscillation). d) Simple diagnostics on Burgers' (shocks are multi-frequency) and KS (what happens at different smoothing scales?). For all of these examples, it would be great to see the contribution of the coarse-grained model and the correction from the learned part.\n\n- Finally, it would be great to see this method in action on some real, relevant data. This can be either from climate science or astrophysics or other hydrodynamic applications with turbulence and multi-scale effects. It may be interesting to compare bottom-up 2D turbulence to top-down 3D turbulence. A comparison to Stachenfeld 2022 on the data set that they use would be adequate.\n\n- (minor) The proposed method is not multi-scale, but at best two-scale. The Russian doll figure seems to insinuate that there are multiple scales involved or that the method can be applied recursively, but that does not seem to be the case here. This makes the naming quite misleading. It would be much better to name it something related to closure, or make it actually multi-scale, since the current discrepancy will cause confusion. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject