Discretization-invariance? On the Discretization Mismatch Errors in Neural Operators
Authors: Wenhan Gao
Ruichen Xu
Yuefan Deng
Yi Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Neural Operators
Operator Learning
Discretization Mismatch Errors
Discretization Invariance
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: A study of resolution invariance and discretization mismatch errors in neural operators

Abstract: In recent years, neural operators have emerged as a prominent approach for learning mappings between function spaces, such as the solution operators of parametric PDEs. A notable example is the Fourier Neural Operator (FNO), which models the integral kernel as a convolution operator and uses the Convolution Theorem to learn the kernel directly in the frequency domain. The parameters are decoupled from the resolution of the data, allowing the FNO to take inputs of different resolutions.
However, training at a lower resolution and inferring at a finer resolution does not guarantee consistent performance, nor can fine details, present only in fine-scale data, be learned solely from coarse data. In this work, we address this misconception by defining and examining the discretization mismatch error: the discrepancy between the outputs of the neural operator when using different discretizations of the input data. We demonstrate that neural operators may suffer from discretization mismatch errors that hinder their effectiveness when inferred on data with resolutions different from that of the training data or when trained on data with varying resolutions. As neural operators underpin many critical cross-resolution scientific tasks, such as climate modeling and fluid dynamics, understanding discretization mismatch errors is essential. Based on our findings, we propose a Cross-Resolution Operator-learning Pipeline that is free of aliasing and discretization mismatch errors, enabling efficient cross-resolution and multi-spatial-scale learning, and resulting in superior performance.

Published as a conference paper at ICLR 2025
DISCRETIZATTIONIIVAAIINN ON THE DISCRETI- ZATION MISMATCH ERRORS IN NEURAL OPERATORS
Wenhan Gao Ruichen Xu Yuefan Deng' Liu
1 Department of Applied Mathematics and Statistics, Stony Brook University 2 Department of Computer Science, Stony Brook University
ABSTRACT
In recent years, neural operators have emerged as a prominent approach for learn- ing mappings between function spaces, such as the solution operators of paramet- ric PDEs. A notable example is the Fourier Neural Operator (FNO). which models the integral kernel as convolution operator and uses the Convolution Theorem to learn the kernel directly in the frequency domain. The parameters are decou- pled from the resolution of the data, allowing the FNO to take inputs of different resolutions. However, training at a lower resolution and inferring at a finer res- olution does not guarantee consistent performance, nor can fine details, present only in fine-scale data, be learned solely from coarse data. In this work, we ad- dress this misconception by defining and examining the discretization mismatch error: the discrepancy between the outputs of the neural operator when using dif- ferent discretizations of the input data. We demonstrate that neural operators may suffer from discretization mismatch errors that hinder their effectiveness when in- ferred on data with resolutions different from that of the training data or when trained on data with varying resolutions. As neural operators underpin many criti- cal cross- reosuueesoltionn scientific tasks, such as climate modeling and fluid dynamics, understanding discretization mismatch errors is essential. Based on our findings, we propose cooss-eesolttion Aperator-learning Pipeline that is free of aliasing and discretization mismatch errors, enabling efficient cross-resolution and multi- spatial-scale learning, and resulting superior performance. The code publicly available at https: // com cotm://ee.baanaaal1111R3 CROP
INTRODUCTION
In diverse fields across science and engineering, researchers aim to investigate the behavior of phys- ical systems under different parameters, such as different initial conditions forcing functions. Traditional numerical approaches often prove to be excessively time-consuming for simulating para- metric physical systems. Data-driven surrogate models, known as neural operators (NOs), present an efficient alternative (Anandkumar et al. 2019 Raonic et al. 2023 Lu et al. 2021). Neural op- erators learn the mapping from parameter function space to the solution function space. After the training phase, performing inference only necessitates a forward pass of the network, which can be several orders of magnitude faster than traditional numerical methods. A particular example is the Fourier Neural Operator (FNO) (Li et al.) 202I), which models the integral kernel as convolution operator and employs the Convolution Theorem to learn the kernel directly in the frequency domain to decouple the parameters from the resolution of the data.
However, there is a long-standing misconception in the operator learning community: widely believed that training an FNO on one resolution allows inference on another without degrading its performance, since FNO operates and parameterizes the kernel the Fourier space. reality, it still exhibits a strong bias toward the training resolution. We attribute this inconsistency across different resolutions to discretization error. Traditionally, in numerical methods, discretization error refers to the discrepancy between continuous mathematical models and their discrete approximations, which arises due to the finite resolution of computational systems. In the context neural operators, the continuous mathematical model is represented when the neural operator ((: 9), parameterized by network parameters 8, takes a continuous function a as input: G(a; #). contrast, the discrete
"Correspondence yi.liu.4 stonybrook.edn

---

Published as a conference paper at ICLR 2025
Continuous Discretization
Discrerization
Difference between the Inputs
Neural Operator
Neural Operator
Neural Operator
Difference between the Output Predictions Caused by Discretization Mismatch Error
Discretization Mismatch Error
Figure 1: Neural operators map between function spaces and can handle different discretizations. However, their architectures still introduce discretization errors. In computational math, researchers analyze the discrepancies between continuous processes and their discrete approximations. the case of FNOs, we never have the continuous model that is trained with continuous data. In- stead, we define discrepancies between two discrete approximations as discretization mismatch er- rors (DMEs) in Sec. 4.1 Minimizing DMEs is crucial in cross resolutin tasks.
approximations occur when the neural operator takes a discretization of the continuous function as input: ((al,,;)), 0), where 4/2, denotes a discretization of J observation points of the function optimized discretizations of functions; therefore,
In practice, network parameters argue that for neural operators, is essential to minimize the discrepancy between different discrete representations. We will refer this type of error as the discretization mismatch error (DME).
DMEs are caused by the network architectures; a simple example is Convolutional Neural Networks (CNNs), which generally considered discretization-iinaaian models. When the resolution of the input data converges to infinity, discrete convolution, whose kernel remains fixed in size, con- verges to point-wise operation Lchu-Schaaiiii et al., 2024). In this work, we analyze the DME in neural operator. This work is prototypi-
FNO, which is the most popular diicretizationninvarrt cal analysis of DMEs neural operators, especially for mesh-based neural operators. We refer to tasks where the inference resolutions differ from the training resolution cross-resolution tasks for the reasons given Sec. 4.. and the analysis of DMEs is pivotal for such tasks. As an integral neural operator, the mathematical formulation of FNO operates in the continuum, representing groundbreaking approach to function space mappings via neural networks. However, the realization of FNO is still at discrete level as the inputs and intermediate outputs gue that FNO suffers from DMEs. Our work does not seek to criticize FNO in any way but rather builds upon this revolutionary framework while highlighting the issue of DMBs for the community's consideration to advance the designs of neural operators.
discrete; thus,
In this work, study and locate the sources of DMEs. Both theoretical analysis and empirical evaluation are provided to convey our findings. Moreover, neural operators may suffer from alias- ing errors that stem from inadequate sampling high-frequency contents introduced by the neural operator (Raonic et al.) 2023). To mitigate these issues, we propose the coss-Resolutton Operator- learning Pipeline (CROP); CROP is free of both DMEs and aliasing errors. We summarize and compare several popular neural operators in Table [ Our contributions can be summarized in four parts: (1) We address this important, yet often overlooked, issue of the accumulation of DMEs in neural operators.
(2) We provide mathematical analysis and empirical evaluation to identify the sources and accumulation of DMEs. (3) design CROP components that capture global patterns and are robust in cross-resolution tasks. (4) CROP represents general pipeline in which the in- termediate neural operator can be freely chosen to map the feature functions between band-limited function spaces, such as CNN-based networks for fine-grained feature extraction.
2 RELATED WORK
Over the past few years, neural PDE solvers have emerged as promising alternative to conventional numerical methods for solving PDE problems across various domains in practical engineering and
2

---

Published as a conference paper at ICLR 2025
Table 1: A comparison of various popular architectures for learning function space mappings.
Discretization Invariance
Free of
Free of DME
Architecture
Efficiency
Aliasing Error
FNO
X
U-Net
X
X
CNO
/
DeepONet
x!
CROP (ours)
In Sec. [4.1 the notions of discretization invariance and discrctization mismatch error are introduced.
The notion of aliasing error (continuous-disce
equivalence) is introduced in Raonic et al. (2033.
Interpolation is employed for resolution independence and being free of discretization mismatch errors. Interpolation operations in up and down samplings for activation functions can be computationally costly.
DeepONet is only resolution independent on the output.
life sciences (Sirignano et al. 2020 Pathak et al. 2022 Zhang et al., 2022 Azizzadenesheli et al,
2024 Zhang et al. 2023). Traditionally, solving PDE involves seeking smooth function satis- rying the derivative constraints imposed by the equations. From this perspective, Physics-Informer Neural Networks (PINNs) (Raissi al., 2019) have been developed to approximate solutions of PDEs individually. An alternative perspective views differential operators as mappings between function spaces, where the solution operators are the inverses of these mappings. Building upon this viewpoint, neural operators have been introduced as a method for solving family of parametric
PDEs (Anandkumar et al., 2019] Lu et al., 202T] [Brandstttter et al. 2022 Fanaskov & Oseledets) 2024 Raonic et al. 2023). Neural operators parameterize solution operators with neural networks to map the input parameter functions to their respective solutions directly. More generally, beyond the solution operators of PDEs, neural operators can approximate other function-to-functio mappings, including derivative operators, integral operators, etc..
Integral neural operator is a general framework that parameterizes the operator as a sequence of ker- nel integrals and nonlinear activations (Kovachki et al. 2023; Anandkumar et al. 2019). FNO is a type of integral neural operator in which the kernel integral is imposed to be convolution (Li et al. 2021]. Global convolution is performed via Fourier layers in the frequency domain. By leverag- ing low-frequency modes and truncating nigh-frequency modes, FNO captures global information while incurring low computational costs and achieving discretization invariance. Although FNO is discretization-ineat -nvvariann,, its inconsistent performance super-resolution tasks has been noted in sev- eral works (Raonic et al. [2023] Li et al. 2022). Mesh-based neural operators including FNO, GNO (Anandkumar a 2019), CNO (Raonic et al., 2023), and DeepONet (Lu al., 2021) either are not completely discretization invariant or suffer from DMEs. While function- basis models, such as
SNO (Fanaskov & Oseledets 2022), may not suffer from DMEs, they might be inferior in capturing spatial information and dependencies. In this work, we specifically analyze the DMEs in FNO, model that is not intuitively expected to encounter such issucs.
Relations with Prior Work. In Li et al. (2022), the inconsistent super-resolution performance has been noted and the authors propose to mitigate this issue by using physics-informed constraints to fine-tune the neural operator. However, this method is limiting, different
informed constraints (losses) can be difficult to optimize and balance (McClenny & BBaaa--etto 2020 Gao & Wang 2023 Wang et al. 2023). and in practical applications,
might have only data available, not the governing PDEs. In Raonic et al. (202)); Bartolucci et al. (2023), aliasing errors in neural op- equivalence (CDE) is formalized to study whether there is an equivalence between their continuous and discrete representations in neural operators, i.e., whether the outputs of the neural operator reside in the band-limited function space of interest. Based on CDE, Convolutional Neural Operators are proposed to be an operator adaptation of the widely used U-Net architecture. Their views on aliasing errors neural operators are constructive and critical. However, while aliasing error is a very important perspective on the design of neural operators, we note that anti-alias activation functions do not overcome the afore- mentioned issues; this will be discussed further in Sec. 5.1 We provide different explanation as to why FNO is not robust across different resolutions. In this work. DMES are defined and analyzed; propose the Cross- Resolution Operator-learning Pipeline that minimizes DMEs to achieve robust performance across different resolutions. Our CROP components can be seamlessly used FNO or CNN-based architectures such as CNO to enable efficient learning of both global features and local details while decoupling the network parameters from discretizations of the input data.
erators have been investigated. The notion of continuous-diseree
3

---

Published as a conference paper at ICLR 2025
3 BACKGROUND
3.1 OPERATOR LEARNING
Operator learning refers to the use of neural networks to approximate or learn an operator, G, map- ping between function spaces (Kovachki et al. 2023). Neural operators are particularly relevant in the field of scientific computing, where complex mathematical operators are often involved in solv- ing partial differential equations (PDEs). A particular and commonly seen example of operators is for the solution operator of parametric PDEs. Consider a parametric PDE of the form:
N (aa, (x) [(x), x EEΩ
u(a)) = 0, I E aΩ
where Ω C Rd is a bounded open set, ./ is possibly non-linear, differential operator, is the parametric input function. f is a given fixed function in an appropriate function space determined by the structure of N, and is the PDE solution. The PDE solution operator is defined as G(a) u.
In practice, for appropriate function spaces A and U, we are interested in learning an operator G: A:A+ + U with neural operator (-; () through a finite collection of observations of input-output pairs {aj; /z) where @ is set of network parameters and aj ll are i.i.d. samples drawn from some probability measure j supported on A. We aim to control the (A:2() Bochner norm of the approximation on average with respect j.:
min [|G(-) - G(: (AM)) En Euwpl(G(a) () AE6
= |IG(a) ((a 0) du((a)
Nau
|0y Go Ga (ag))liz
Nd
Practically, each aj and Uy are functions that come with a discretization (function values at some sensor points), could seek to approximately solve the empirical-risk minimization problem di- rectly at those sensor points.
3.2 FOURIER NEURAL OPERATORS
Inspired by the kernel method for PDEs, each integral neural operator layer consists of a fixed non- linearity and kernel integral operator I. modeled by network parameters. defined as (KCv) (lr,))) All operations in integral neural operators are defined on functions; thus, integral neural operators are understood as function space architectures.
As natural choice inspired by CNNs and the perspective of fundamental solutions, FNO imposes the integral kernel to be translation invariant, s(x,, y) л(x y). Thus, the kernel integral operator becomes convolution operator, and FNO performs global convolution in the frequency domain. Each Fourier layer FNO consists of a fixed non-linearity and convolution operator IC modeled by network parameters:
(Kv)(z) / <(х y)v(y)dy. RRd Convolution can be efficiently carried out as element- multiplication in the frequency domain:
(KC)(())
(Fr
(2)
where F and FF1 are the Fourier transform and its inverse, respectively. FNO directly learns Fx in the frequency domain instead of learning the kernel и in physical space. The Fourier transform captures global information effectively and efficiently, leading superior performance for FNO.
4 PROSS-RESOLUUTE
OPERATOR-LEARNE
PIPELINE
In this section, we delve into the following questions: (1) Do neural operators exhibit robustness across different resolutions? (2) not, what factors contribute this instability?
(3) Based on the analysis, how can we improve the resilience of neural operators across various resolutions and potentially achieve additional benefits? We first define DMES in Sec. 4.1 we answer the first two questions in Sec. 4.2 and the last question in Sec. 4.3 We will use FNO as an example for the analysis of DMEs; however, the concepts and analysis can be extended to other mesh-basce neural operators, such as the Wavelet neural operator (Gupta et al. (202
4

---

Published as a conference paper at ICLR 2025
4.1 DISCRETIZATION ERROR AND DISCRETIZATION MISMATCH ERROR
For simplicity, we consider H (TT) and H' (T*) as the input-output function spaces, unit torus Q = T = [o. 1)d as the domain, and equidistant (uniform) grids as the form of our discretization. The FNO architecture is usually applied to such discretizations to harness the power of the Fast Fourier Transform (FFT). The operations in FNO are mainly defined on general geometries and discretization. The FNO architecture is not limited rectangular domains, periodic functions, or uniform grids. Likewise, our analysis is not limited by these constraints; however, the derivation can become considerably more tedious; thus, we adopt this simplification.
Definition 4.1. A uniform discrete refinement (ft)) J=1 of the domain Ω = T is a sequence of nested sets of equidistant grids 92 C S2p C C Ω with [Stl] = Jd for any J E N such that, for any h >> 0, there exists number I E N such that
Ω S
U (VERR: ly 1111-112-10 < h}. reQ,
Any member S2, is called a discretization of the domain Ω. Note that, for an -valuee function + R defined on Ω2 we call the pointwise evaluation of the function on fhy, alo, = {aa (2y) Iy E S2y}. discretization of the function a(r). Notice that cla, can be viewed as vector in R Therefore, an FNO that takes discrete inputs can be viewed as a mapping 9 R contrast to its continuous counterpart G
- U, in : A - U. Here, å indicates a pseudo-FNO that handles discrete data inputs. For simplicity, we will omit the notation as the context will make it clear.
As we assume uniform grids, storing the sensor point coordinates, Zj. is unnecessary; thus, als can be directly used as input the Fourier neural operator. While including coordinates input to the neural network may sometimes improve performance, as noted in Kovachki et al. (2023). Nevertheless, our analysis still applies if we consider coordinate inputs as the identity function (the function's 's output at point Ij is z, itself). For the purposes of discussion in this work, we will treat alo, as a vector in Rddd, For discretization of size N, we will view it as a function reconstructed by the N Fourier coefficients. This view is natural from the perspective of the orthogonal Fourier projection operators.
is well known in the field of operator learning that FNO regarded as a petpettizaatinnnnvvireen model. We herein present the definition of discretization invariance as given in Kovachki et al. 2023 with slight modifications related to our notations and conventions.
Definition 4.2 (Kovachki al. (2023)) ). Let + C RRP be finiteddimensional parameter space and 9 A x θ + a map representing a parametric class of operators with parameters A E e. Given discrete refinement (1t))== of the domain Ω Rdd. say is discretivation-inaaria if there
exists a sequence of maps G1, G....., where فJ Jd d x θ + Ul such that
lim sup ||g (ale, 0) G(a; 0) 0, J aeK
for any 0 E θ and any compact set K C A.
Generally speaking, discretization invariance defined in Kovachki et al. (2021) refers to the property that neural operator taking discrete data as input converges to the continuum operator taking con- tinuous functions as input as the discretization becomes finer. However, this does not necessarily imply that the neural operator's performance is consistent regardless of the input data's discretiza- tion, which is a common misconception within the operator learning community. We note that this property implies convergence to the continuum, rather than the ability to achieve cross-resolution tasks. Instead, throughout this paper, we will refer to discretization invariance as the property that the neural operator's 's parameters remain decoupled from the resolution of the data and that the neu- ral operator can take inputs of different discretizations directly without sacrificing its performance. Many in the operator learning community interpret this as the meaning of discretization invariance, often conducting zero-shot super-resolution tests in their research. To avoid confusion with the discretization (or resolution) invariance as defined in Kovachki et al. (202I). we will use the term "cross-resolution"" To this end, we introduce DMEs, key concept for cross-resolution pipelines.
Definition 4.3. Let θ C R be a finite dimensional parameter space and G A x e + Ui a map representing parametric class of Fourier neural operators with parameters 0 E e. Given uniform
5

---

Published as a conference paper at ICLR 2025
discrete refinement (Rt)) of the domain Ω RR, the discretization mismatch enror between the Fourier neural operator taking two different discretizations of the function a is defined as
EM == || (alaxt - Go (alaw)llz
4.2 THE ACCUMULATION OF DISCRETIZATION MISMATCH ERROR
To provide a detailed analysis of discretization errors in FNOs, we formally and mathematically outline the full FNO architecture in Appendix A. and present Proposition 4.4
Proposition 4,4. Let ④ S TRP be finite dimensional parameter space, A = Ha (T4: H (TT: and G A - U a map representing a parametric class of Fourier neural operators with parameters function that is (globally) w- w-Lipschitz continuous for all layers. Given a discretization a alax then the discretization mismatch ervor between the Fourier neural operator taking a alay and another dis- cretization al given by EMN || (,)) - Go (alay) II with M N, will increase M increases. Additionally, the discretization mismatch error might increase as L and w increase.
θ with L Fourier layers. Consider (zx) as the activation
Proposition 4.4 indicates that DMEs propagate and accumulate across the intermediate Fourier lay- ers, and as straightforward extension from the continuity property of G. the DMEs also propagate over time in autoregressive tasks. The training resolution can M or N, so this bound suggests that long as the inference resolution differs from that of the training, it will suffer from DMEs; nevertheless, it is a super-resolution or sub-resolution (lower resolution) task. We demonstrate this empirically in Sec. [5. The proof for this proposition and formal statements the upper bound given in Appendix that neural operators may struggle perform cross-resolution tasks. Therefore, we propose the Cross-Resolution Operator-learning Pipeline to address this issue.
We experimentally verify our findings in Sec. [.1] These findings suggest
4.3 CROP: MINIMIZING THE DISCRETIZATION MISMATCH ERROR
As shown in Sec. 4.2 the DMEs arise in neural operators, limiting their ability to perform cross- resolution tasks. To address this, we propose the CROP lifting and projection layers:
PCROP Ha (T ( Rd))) + Bw (T" Rd4) and QCROP B (ToR H" (T")
The CROP lifting layer, PCROP, maps the input function to band-limited, usually high-dimensional, latent feature function through a global convolution with a band-limited convolution kernel C(x- y). Hence, (2) fTu C(x ()) (x)ddd. and the resulting latent feature function (z(x) will be band- limited as well. As we consider band -imited function space, we can fix the discretization of the latent feature function in accordance with the band-limit we choose. Since the discretization is fixed for the latent feature functions, any intermediate neural operator architecture L can be chosen to learn the mapping between latent feature functions, including those that are mesh-dependent:
C B. R4) + B (T4
Moreover, because the latent discretizations are fixed, CROP does not suffer from the DME, and we interpret the intermediate neural operator an operation on these discretizations. As result, it will not introduce high-frequenge components beyond the established band- that the discretiza- tion can accommodate. Therefore, the intermediate neural operator satisfies the Continuous-Disereee Equivalence (Raonic et al.) 2023).
Similarly, the CROP projection layer, QCROP, consists of a global combined with a fully connected point- point-wise network to map the band-limited latent feature function to an output function with frequencies that can exceed the band-limit in the appropriate function space. The high-frequency information beyond the band-limit is learned in the CROP projection layer through a fully connected point- wise network, potentially augmented by a residual connection to the input, as tolerated by the refined discretization.
kernel convolution
Remark 4.5. Let v(x) B. (T Rd be band-limited function, and let analytic non-linear function o(-) be the fixed activation function in the fully connected posnt-wise network. Then the resulting function may not limd-iimited anymore. Additionally, the high frequencies of the re- sulting function is determined by both the fully connected point- boint-wise network and the learned latent feature function v(x).
The proof of this remark is given in Appendix C This remark suggests that the high-frequency com- ponents beyond the band-limit are determined by both the fully connected point-wise network and

---

Published as a conference paper at ICLR 2025
Intermediaale Lalent Band limited Feature Funclions
Inpurt Punction with Any Discreizztion
Output Punction with Any Discreization
CROP Lifting
Intermediate Noural perrloor
CROP Projection
Figure 2: Tross-Resolution Operator learning Pipeline: Firstly, a CROP lifting layer maps the input parametric function to latent feature function band-limited function space with a fixed dis- cretization. Consequently, spaces is applied to process the latent feature function. This intermediate neural operator does not necessarily have to be discretization invariant, such as the U-Net. Finally, a CROP projection layer will map the latent feature function to final output function in the appropriate function space. A Fully connected, point-wise network is incor- porated in the CROP projection layers to learn higher frequency information beyond the band-limits.
intermediate neural operator that operates band-limited function
can choose models of local features
The intermediate neural operator is tasked to learn an appropriate latent feature function v to capture fehhfreeuency components in conjunction with the fully connected point-wise network. The key ideas of CROP is to have the latent feature functions to be a Sand-limited function space so that they can be processed by an intermediate neural operator that can be more freely chosen; for exam- ple, we can use mesh-dependent architectures such as CNNs. This process, however, will "CROP" out high frequencies, we utilize the CROP projection layer to recover such frequencies. We provide a spectral comparison between FNO and CROP in Appendix(C.I The CROP lifting and projection layers are able to capture global information efficiently; thus, we can select an intermediate neural operator the local nature, such as U-Nets, without sacrificing the ability to learn global features. Overall, this framework is more robust for cross-resolution tasks by having operator that maps between band-limited function spaces and we can choose appropriate intermedi- ate neural operators efficiently learn the nulti-spatio-scale nature of many problems in physical modeling (Gupta & Brandstetter 2022).
intermediate neural
5 EXPERIMENTAL RESULTS
In this section, we conduct experiments to substantiate our claims and demonstrate the effectiveness of our proposed CROP components. In Sec. 5. we conduct experiments on the incompressible Navier-Stokes equation showcase the accumulation of DMEs and verify our CROP components' ability to perform cross-resolution tasks. In Sec. 5.2 we conduct experiments to demonstrate the efficient callt-sspati-scaale learning capability of our CROP components.
5.1 CROSS-RESOLUTION TESTS
In this example, we demonstrate the DME in neural operators as discussed in Sec. 4.2 and then explore the ability of CROP components for cross-resolution tasks.
Description. We consider the 2D Navier-Stokes equation for viscous, incompressible fluid in vorticity form, described in Li et al. (2021):
@zw(x, t) + u(z,t) Vw(x, t) = vAw(I, (I, f(x),
T (0,1)2,t (0,7) x E (0,1),.t [0,
V u(xx t) = 0,
(3)
w(I, 0) wo(xx)
x x (0,
where C (10, T); Her ())1)); for any r 0 is the velocity field, the viscosity coefficient, and e L2 ((0, 1);;R) R) is the forcing function. We are interested in learning the operator that maps the vorticity up to time 10 to the vorticity up to a later time
x is the
vorticity, wp E 12 (10, 1)2; the initial vorticity, v = le
in an auto-regressive manner for all the intermediate time steps with a step size of 1 following Li et al. (202I). More details on the experimental setups can found Appendix D.I.I and D.1.2

---

Published as a conference paper at ICLR 2025
5.1.1 THE ACCUMULATION OF DISCRETIZATION MISMATCH ERROR
First, we recognize that neural operators may not be robust across different resolutions. In Figure 3 (a), we present results from a well-trained FNO on 200 testing input parametric functions and can clearly observe the impacts of the DME. Even at the very first time step when the error has not propagated through time yet, inferences 128 128 (0.66%) three times high those at the training resolution (0.22%), and inferences at other resolutions are even worse.
plot the average relative lg error under different resolutions;
Relative Discrettration Mismatch Enar Number Foureer
0.14
Comparative Plot Inference &g Error under Different Resolutions
0.00
Standard Dey abin
Mean fakarine Discetization Narratch Broor
Averaage intereres Reletteed Ertor
24 64 mhaniing Readlttrr)
0.12
128
0.03
0.06
0.04
0.02
0.01
6.00
10
12
Number Toutter Layerr
Figure 3: (a): The relative &a error during inference for the Navier-Stokes equation across different input resolutions. We can clearly observe the effects of DME on performance, even at the very first time step. Unsurprisingly, this effect propagates through the auto-regressive time steps. (b): The growth in relative DME with respect to increasing numbers of layers. This highlights observation that DMEs propagate through the Fourier layers.
As shown Figure 3 (b), there is a clear trend: as the network stacks more layers, the DME in- creases, As significant discretization errors exist within the network architecture, their impact on the overall performance of super-resolution tasks profound. Therefore, based on both experimental observations and mathematical amalysis, conclude that FNO is not suited for cross-resolution tasks (inference on resolutions different from that of the training). However, we emphasize again that the formulation of FNO is revolutionary and lays the foundation for function space mappings. Philosophically, since the training data lacks high-resolution information, it remains unclear whether neural operators can learn what they have not seen (high-resolution information) shot super-resolution without the inclusion of specific designs, such as transfer learning, physics- informed loss (Li et al. 2022 Wang al. 2021), conservation laws (Liu al., 2024). symmetries (Helwig et al. 2023 Gao et al. 2024), etc.. However, we believe that FNO offers great potential to
achieve zero-
achieve lata-driven super-resolution tasks; we will discuss this further in Appendix [
5.1.2 CROP: MINIMIZING THE DISCRETIZATION MISMATCH ERROR
We demonstrate that CROP can achieve robust performance across differentt resolutions of inference. We compare the CROP-enhanced FNO model with the plain FNO and the FNO with anti-alias activation functions (Raonic et al. 2023).
Table 2:
The mean and standard deviation of relative error over 10 runs during inference are re- ported for the Navier-Stokes Equation with varying resolutions of input parametric functions. CROP consistent across all resolutions. Baseline models continue show a strong bias toward the train- ing resolution. The DME is obviously shown when inferred on data with different resolutions.
256 x 256
128 x 128
64 x 64
32 x 32
FNO Alias-free FNO (2x)
4.65(+1.47)
3.35(+00.77
0.58(+0.04)
9.45(+2.92)
6.91(+2.73)
6.18(+3.01)
0.61(+0.04)
39.74(+14.11)
Alias-free FNO (Fixed)
6.95(+22660)
7.40(43.49)
0.61(+0.04)
42.08(+15.03)
CNO (with interpolation)
4.17(++1.46)
2.95(+11.6)
1.98(+0.14)
7.69(41.69)
CROP (ours)
0.54(+0.05)
0.54(+0.05)
0.54(+0.05)
0.54(+0.05)
Training resolution 64
64.
We present the results in Table 5.1.2 as percentages. is clear that FNOs exhibit a strong bias toward the training resolution, regardless of whether upsampling and downsampling used to
8

---

Published as a conference paper at ICLR 2025
mitigate aliasing issues [Raonic et al. 2023 We additionally include the cross- ross-uesso results for CNO (Raonic et al. 2023) with downsampling and bicubic interpolation to illustrate that relying on explicit up/downsamplin ling may degrade performance. Therefore, assert that alias-free activa- tion functions cannot fully enable FNO to succeed in cross-resolution tasks. The major challenge need to solve is the discretization error inherent in the realization of neural operators a discrete level. Clearly, CROP minimizes the DME and achieves the best results on cross-resolution tasks. Moreover, CROP exhibits superior performance compared other baselines even when inferring at the same resolution of the training data. This can be attributed to the fact that CROP is continuous- discrete-equivalen FNO truncates some high-frequnncy modes in each layer for efficiency and re- duction of the number of parameters, but this truncation will break continuous-discrete -ispaassssseeeeeeeeallleeeeeee equivalence. Moreover, such truncations are necessary if we aim to train at a lower resolution and infer at a different resolution. CROP strictly adheres to the band-limits in accordance with the truncation; thus, CROP improves the overall results. Additionally, as CROP does not suffer from DMEs in any autoregressive time step, CROP is stable for long-term roll-outs in terms of cross-resolution abilities.
5.2 LEARNING CAPABILITY TEST
In this example, explore the learning capabilities of CROP components. We demonstrate that CROP achieves superior performance compared to the baselines. This performance gain can be attributed to factors such as exceptional multi-spatio-scale abilities while being free from aliasing and DMEs, as outlined in Sec. 4.3 Additionally, we include the results of cross-resolution tests in Appendix[D3] showcase CROP's cross-resolution capabilities.
5.2.1 NAVIER-STORER EQUATION WITH HIGH REYNOLDS NUMBER
Description. We examine the same Navier-Stokes equation discussed in Section 5.1 but with an increased Reynolds number. High Reynolds numbers pose significant learning challenges due to the turbulent characteristies and small-scale features of the flow (Liu-Schiaffini et al. [024). We conduct experiments at Reynolds numbers of 5, 000 and 10, 000, respectively. As higher Reynolds numbers lead to more complex problems, we utilize a larger training dataset for Re = 10, 000. More details can be found in Appendix D.1.3
Table 3: Results on the Navier-Stoker equation with high Reynolds numbers are presented. The mean and standard deviation of the relative & error over 10 inference runs are provided. Timing results are also included reference. Clearly, CROP outperforms all baselines.
Rc = 5000
Re = 10000
#Par. (M)
BPS
IT
Test (%)
Test (%)
FNO
2.27
18.47
3.39
7.25(=0.38)
8.35(=0.55)
CNO
2.61
1.64
9.73
5.77(=0.19)
4.66(+10.07
U-Net
7.76
34.54
1.56
8.96(=0.06)
8.75(+0.22)
ResNet
23.50
2.01
6.73
37.79(+2.111
53.01
DeepONet 4.89
50.35
0.89
16.93(+1..0))
17.60
CROP (ours) 4.03*
12.70
4.69
3.84(=0.09)
4.06 4.06(10.22)
BPS (Batch per Second): The number batches processed per second during training. IT (Inference Time): The time, in milliseconds, it takes to infer on batch of 16 samples.
Each complex parameter (which twice the size of a real parameter) is counted as 2.
The results clearly demonstrate that CROP exhibits exceptional learning capabilities, effectively capturing both fine-scale and global features. While Fourier Neural Operator (FNO) performs well at low Reynolds numbers, as shown in Sec. 5.I it tends to over-smooth fine-scale details, leading to degraded performance. Although the Convolutional Neural Operator (CNO) can learn global features through its U-path. it excels primarily in local contexts due to its convolutional nature, making it less effective capturing long- rangee dependencies compared to our CROP components. Similarly, U-Net and ResNet also struggle with learning long-range dependencies. We recognize that DeepONet is groundbreaking framework that performs well in various applications; however, like FNO, it may also over-smooth fine-scale details. Overall, CROP's ability effectively learn complex patterns results in superior performance compared to the baselines. An interesting phenomenon is that FNO performs worse while CNO performs better under higher Reynolds numbers. Our hypothesis is that as the Reynolds number increases, fine-scale details become more crucial to learn due to the increased chaos. CROP consistently performs well under both Reynolds numbers due to the multi-spatio-scall learning capabilities it offers.
Results and Discussion.

---

Published as a conference paper at ICLR 2025
5.2.2 DARCY FLOW, HELMHOLTZ, AND POISSON EQUATIONS
Description: Darey Flow. We consider the steady-state of the 2D Darcy flow equation from Li
et al. ( Hasani & Ward (2024). We learn both the nonlinear operator mapping the diffusion coeffcient to the solution function, a + u, following the same setup and data as in et al. (202T),
and the operator mapping the forcing function to the solution function, f + following the same generation setup in Hasani & Ward 2024).
Description:
Helmholtz. We consider the 2D Helmholtz equation from De Hoop et al. 2022). We learn the nonlinear operator mapping the wave-speed field to the solution function, e + u, following the same setup and data as in De Hoop et al. (2022) with normalization the data.
Description:
Poisson. We consider the 2D Poisson equation from Raonic et al. (2023) and learn the operator mapping the forcing function to the solution function, f + following the same setup and data as in Raonic et al. (2023). More details about these equations, data generation, and setups provided in Appendix D
Table 4: Results for the Darcy flow, Helmholtz, and Poisson equations are presented. Nonlinear
Darcy denotes the non-linear mapping a + u. It is evident that CROP achieves competitive perfor-
mance with multi-spatio-learminn capabilities while maintaining the cross-resolution property.
Nonlinear Darcy
Darcy Flow
Helmholtz
Poisson
FNO CNO
(0.00)
1.58(#0.14)
0.68(=0.01)
7.21(#0.74)
(.50
1.32 (0.33)
0.87(=0.02)
1.67(=0.56)
U-Net
1.34(++000)
2.68(+0017)
1.51(=0.06)
1.40(++0.2)
ResNet
7.26(++0.59)
9.85(+0.68)
14.86(+0.53)
5.64(+0.34)
DeepONet
2.99 +0)
1.36(=0.16)
3.03(=0.08)
12.26(+1.48)
CROP (ours)
0.51
1.11(10.08)
0.80(+0.13) 0.52(+0.02)
The findings clearly indicate that CROP demonstrates outstanding learn- ing abilities because we can freely choose an intermediate operator while maintaining properties such as resolution-ivvariann bility enables us to select a suitable intermediate neural operator that focuses on local details. For the Darcy tasks, this is relatively easy learning task, all models, except ResNet, perform well: how- ever, CROP still shows improvement over all baselines. We additionally include the cross-resolution performance for CROP and FNO on nonlinear Darcy in Appendix D.2 A similar trend is observed for the Helmholtz equation, where all models perform well except ResNet, and CROP outperforms all baselines. For the Poisson equation, we note that architectures with local characteristics, includ- ing CNO, U-Net, and ResNet, all perform better than FNO. CROP, as a paradigm with both local and global learning capabilities, outperforms all baselines by attending to global patterns while focusing on local structures.
Results and Discussion.
discrete-continuovt equivalence, and freedom from DMEs. This flexi-
learning
CONCLUSION
In this work, we have explored the challenges posed by discretization mismatch errors (DMEs) in neural operators. Despite FNO's advantages in formulating the kernels in the Fourier domain to capture global features and learn dynamic physical phenomena, its performance is not consistent across different resolutions. Moreover, FNO tends to over-smooth fine-scale details. We identified the sources of these discrepancies and introduced a Cross-Resolution Operator-learning Pipeline (CROP) designed to mitigate DMEs. Our proposed method not only succeeds cross-resolution tasks but also outperforms baselines in learning complex dynamics, as it upholds continuous-discrete cquivalence and efficiently learns both global and local features through CROP designs.
Limitations and Future Work. First, like FNO, CNO, and other CNN-based architectures, our CROP components perform optimally with rectangular domains and uniform grids. We acknowl- edge that this limitation may affect the applicability of CROP. Efforts have been made to address these drawbacks in FNO and CNN-based architectures (Liu et al. 2003] Gao et al., (021)). While these concepts may be adapted for CROP, we plan to explore this the future. Second, although our method can capture high-frequency information using both the learned feature function and the fully-connected pointwise network, further investigation into the expressive power and universality of these components would enhance our understanding and inform better designs for intermediate neural operators and activation functions in operator learning.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
This work has used the computational equipment supported by the US Army Research Office under the award N991NNF20-ND 10159. Y. Liu and W. Gao would like thank Dr. Xiaolin Li Stony Brook University for providing this computational equipment.
ETHICS STATEMENT
This research focuses the development and evaluation of neural operators, which are mathemati- cal and deep learning tools designed to model complex systems. The study does not involve human subjects, personal data, or sensitive information that could raise privacy, security, or fairness con- cerns. No potential conflicts of interest, legal compliance issues, or harmful applications have been identified in this work.
REPRODUCIBILITY
All baseline models used in this work were adopted with minimal or no modifications from their original implementations. The datasets utilized are either open-source or generated using pub- licly available code, as described in detail in the appendix. We provide the source code. datasets, pre-trained models, and configuration necessary replicate the key experiments at hhtps: 7/github. com wttmm//eeaangao///ccaeeo 1tt.oo////I225CCROO0 evaluation will also be included. Additionally, all theoretical claims are accompanied by proofs in the appendix, which have been empirically verified through experiments.
Comprehensive instructions for training and
REFERENCES
Anima Anandkumar, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki. Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differ- ential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2019. URL /attps://openreeee
n Forummidd fd22FF2FX-.
Kamyar Azizzadenesheli, Nikola Kovachki, Zongyi Li, Miguel Liu-Schiaffini, Jean Kossaifi, and Anima Anandkumar. Neural operators for accelerating scientific simulations and design. Nature Reviews Physics, Apr 2024. ISSN 2522-5820. doi: 10. 1..10//42254-02-00772--- URL httpst:::
//doi. 0
Francesca Bartolucci, Emmanuel de Bezenac, Bogdan Raonic, Roberto Molinaro, Siddhartha Mishra, and Rima Alaifari. Representation equivalent neural operators: framework for alias- free operator learning, In Thirty-seventth Conference on Neural Information Processing Systems, 2023. htt.l/tppee//eeeeeee..eeeeeeeeee http://wwww..tttoooooooooooo//ee
Johannes Brandstetter, Daniel E. Worrall, and Max Welling, Message passing neural PDE solvers. In International Conference Learning Representations, 2022. URL httpts/t///peeeeieee
net http://oummiiiiem 2tte//oommmmdd1v53333333
Maarten De Hoop, Daniel Zhengyu Huang. Elizabeth Qian, and Andrew M Stuart. The cost- accuracy trade-off in operator learning with neural networks. arXiv preprint 1rXiv:2203.13188 2022.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 38(2): 295-307, 2016.
doi: 10. 10.1109/PAMM.22155223000
V. Fanaskov and I. Oseledets. Spectral neural operators, 2024. URL https://arxiv.oo 0oss/2205.100
Vladimir Fanaskov and I. Oseledets. Spectral neural operatoos. ArXiv, wwwpa222...00....200000 2022. URL https://api. www.ntiissttttnee ar. art.org/corpuss1:22888888

---

Published as a conference paper at ICLR 2025
Han Gao, Luning Sun, and Jin Wang. Phygeonet: Physics-infommt informed geometry-
convolutional neural networks for solving parameterized steady-state pdes on irregular domain.
Journal of Computational Physics, 428:110079: 2021. ISSN 0021-9991.
doi: https://doi.
1100:1100111..ee.2000101000 10079. URL https://www/ http::://www.c://////// sc enceiencediee rect. cww. com.science//// ence/
articclep11//00021190 0tt.://///000019900031333300
Wenhan Gao and Chummei Wang. Active learning based sampling for high-dimensional non- linear partial differential equations. Journal of Computational Physics, 475:111848, 2023.
ISSN 0021-9991. doi: http:://oooong//10.0000111g22222
URL https://wwwt
sci scieenedi com feam/ccien http:///c.........tatotaaaaaa//.
Wenhan Gao, Ruichen Xu, Hong Wang, and Yi Liu. Coordinate transform fourier neural operators for symmetries in physical modelings. Transactions on Machine Learning Research, 2024. ISSN
2835-8856. URL htttps://oppeeeeeette
net httpee/www.ee..eee/fooommooooopp 7d
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- operator learning for differen- tial equations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. Associates, Inc., 2021. URL/attps.///rrrceediiiii wwwp:/iinnnnnneuuuuueuu..... cc/paper files/
Cur-
Jayesh K Gupta and Johannes Brandstetter. Towards multi- multi-spptiotempprra-sccllel generalized pde modeling. arXiv preprint arXiv:2209. 2022.
Erisa Hasani and Rachel A. Ward. Generating synthetic data for neural operators, 2024. URL httgg//b//24420 2401. 02398
https://arxiv org/ abs
Jacob Helwig, Xuan Zhang, Cong Fu, Jerry Kurtin, Stephan Wojtowytsch, and Shuiwang Ji. Group equivariant Fourier neural operators for partial differential equations. In Proceedings of the 40th International Conference on Machine Learning. 2023.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. The Journal of Machine Learning Research, 22(1):13237- 13312, 2021.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli. Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Learming maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1 1-97, 2023. URL http://3mi. or.. papers wwtpppper...paa.oooepeooopp...re 424//22 22244/111550 m
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URI.
net forum?id=cco (errummlidocPPPNNv..n
Zongyi Li, Hongkai Zheng, Nikola Borislavov Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Andrew Stuart, Kamyar Azizzadenesheli, and Anima Anandkumar, Physics-informed neural op- erator for learning partial differential equations, 2022. URL httpps://cpenrevieew.coo
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. arXiv preprint arXiv:2108. 10X252200..1027, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Ning Liu, Siavash Jafarzadeh, and Yue Yu. Domain agnostic fourier neural operators. In
Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
/ //openzeview.o wwwp:reviiew.nt/.oo....e wwt.tortttooooooeeoe umriddmiieee wiwpiiddduduapaddaea
Ning Liu, Yiming Fan, Xianyi Zeng, Milan Klöwer, Lu Zhang, and Yue Yu. Hamessing the power of neural operators with automatically encoded conservation laws, 2024. URL https: pecttps://arx.. //arxii. brg/abs/2311 2312. 11176
12

---

Published as a conference paper at ICLR 2025
Miguel Liu-Schiaffini, Julius Berner, Boris Bonev, Thorsten Kurth, Kamyar Azizzadenesheli, and Anima Anandkumar. Neural operators with localized integral and differential kernels. In Forty- first International Conference Machine Learning. 2024. URL httl::ttp/////t https://openzeviert
net/ net/fooum:s forum?
d=vl d 9GB3 fbht.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence. 3(3):218-229, March 2021. ISSN 2522-5839.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using soft attention mechanism. arXiv preprint (tti::2000004554,, 2020.
Shaowu Pan, Steven L. Brunton, and Nathan Kutz. Neural implicit flow: a mesh-agnostic dimen- sionality reduction paradigm of spatio-temporal data. Journal of Machine Learning Research, 24
(41):1-60, 2023. WRLhhtpp:// /mmlr. jmlr org/ wwwpp//ggppp//p/ottottpt. pwpprs/v2sa v24/22 0365. html
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data- driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint 220.::2202.12214.. 2022.
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 378:686-707, 2019, ISSN 0021-9991. doi: https://doi.
10.045. URL https: //wp:///wwwww. scwencedirect. com/ sommm/ccienee/
3tracle/s11/5002199111117
Bogdan Raonic, Roberto Molinaro, Tobias Rohner, Siddhartha Mishra, and Emmanuel de Bezenac. Convolutional neural operators. In /CLR 2023 Workshop Physics for Machine Learning, 2023.
URL httpss:/
nnt. wwt./trruteeeeee//eom forum? SoammiddGT@p00....
Justin Sirignano, Jonathan F. MacArt, and Jonathan B. Freund. DPM: A deep learning PDE aug- mentation method with application to large-eddy simulation. Journal of Computational Physics, 423:109811, 1, 2020. ISSN 0021-9991.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning, 2024.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. The Eleventh International Conference on Learning Representations, 2023. URL
net Forum ?i 1MP 4PP
Sifan Wang. Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric par- tial differential equations with physics-infommed deeponets. Science Advances, 7(40)xabi88605, 2021. doi: 10. v URL http://wwww..ceeeeeeeeeee/////// abi 8605.
Sifan Wang, Shyam Sankaran, Hanwen Wang. and Paris Perdikaris. An expert's guide to training physics-ifformed neural networks, 2023, URL /attp:://arxi....
abs /200/ 08 688
Min Wei and Xuesong Zhang. Super-resolution neural operator. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18247-18256, June 2023.
Mingrui Zhang, Jianhong Wang. James TIhomole, and Matthew Piggott. Learning to estimate and refine fluid motion with physical dynamics. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Xuan Zhang, Limei Wang. Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan. Keir Adams, Maurice Weiler, Xiner Li, Tianfan Fu, Yucheng Wang, Haiyang Yu. YuQing Xie, Xiang Fu, Alex Strasser, Shenglong Xu, Yi Liu, Yuanqi Du, Alexamdra Saxton, Hongyi Ling, Hannah Lawrence, Hannes Stärk, Shurui Gui, Carl Edwards, Nicholas Gao,
13

---

Published as a conference paper at ICLR 2025
Adriana Ladera, Tailin Wu, Elyssa F. Hofgard, Aria Mansouri Tehrani, Rui Wang, Ameya Daiga- vane, Montgomery Bohde, Jerry Kurtin, Qian Huang, Tuong Phung, Minkai Xu, Chaitanya K. Joshi, Simon V. Mathis, Kamyar Azizzadenesheli, Ada Fang, Alán Aspuru-Guzik, Erik Bekkers, Michael Bronstein, Marinka Zitnik, Anima Anandkumar, Stefano Ermon, Pietro Liò, Rose Yu, Stephan Günnemann, Jure Leskovec, Heng Ji, Jimeng Sun, Regina Barzilay, Tommi Jaakkola, Connor W. Coley, Xiaoning Qian, Xiaofeng Qian, Tess Smidt, and Shuiwang Ji. Artificial intelli- gence for science in quantum, atomistic, and continuum systems, 2023,
14

---

Published as a conference paper at ICLR 2025
A ACCUMULATION OF DISCRETIZATION ERROR IN FNO
A.1 MATHEMATICAL FORMULATION OF THE FNO ARCHITECHTURE
To give a detailed analysis of discretization errors in FNOs, first formally and mathematically lay out the most common FNO architecture. Let A (Ω, IRd))) and U (22, R4.) be two appropriate
function spaces defined on bounded domain f C R The Fourier neural operator 9 : A - l is
defined as a compositional mapping between function spaces:
((a): Q o L2 LL-1 o... L1 P(a).
(4)
The input function a E A is lifted to the latent space of R -valued (usually deg > d,) functions through a lifting layer acting locally:
:: {a Ω Rd + {vo Ω Rd00
Lifting layer usually is implemented as a linear layer represented by a matrix P E Rd.xda or as a point-wise multi- layer perceptron (1 by 1 convolution layers) with activation function
Then the result goes through L Fourier layers:
(x) = a (We((z) + be(x) Kev(a)) f= 1.2.....44
(5)
where, is non-linear activation function, We acts locally and can be represented by matrix IR du xdue, be(x)= E U (22, is the bias function (usually constant function for easy imple- mentation), and
- - -F()(())
is linear but non-local convolution operator carried out in the Fourier space with Pj 2d Cd, being the Fourier coefficients of the convolution kernels. We will denote the output of the l-th Fourier layer by ez.
The output function is obtained from the projection layer acting locally, similar to the lifting layer:
Q: : P2 : + Rd
} + uu: Ω + Rd...
For simplicity, will assume that all the inputs and outputs the Fourier layers have the same channel dimension, i.e., den, d. = =... do
de.
In practice, the FNO, well as all the intermediate layers, takes discretizations of functions as inputs and produces discretizations of functions as outputs. Therefore, this discrete implementation termed the Pseudo Fourier Neural Operators: readers are referred to Kovachki al. (202I) for more details.
B PROOF TO PROPOSITION 4.4
We first present the following lemma for the discretization error of Fourier layers and with only single hidden channel. We will later extend this result to the full FNO architecture and with de hidden channels in general. In this section, all norms are E norms unless specified otherwise.
Lemma B.1. Let θ C Rp be finite-iimnssion parameter space, and let 2 be an integen. Consider L: : H² (T: R) x θ H" (T: KT) being the composition of parametric Fourier layers with parameters 0 E θ and hidden channel dimension of one. Assume the activation function R R.. is w-Lipschitz continuous. Given discretization u ulstgs the discretization mismatch error between the Fourier layers L taking vlay and another discretization u/a is bounded by L (oloxx)l N-1
ENM = (uleww)
2x N
M
N Σ
I wh
CI
Σ
v(
12 $
+
Σ N=
v(
12m -22m 3
NM
(6)
where without the loss of generality assume M > N, C. [|eell Crap with ||Well being the operator norm of the linear operator W; in the l-th layer, and Cze is a constant depending on the l-th layer's kernel Ket-
15

---

Published as a conference paper at ICLR 2025
Proof. We aim to establish bound the discretization mismatch error EMN between the outputs of C L1 o - .... L1 applied to the input function at different discretizations Qn and QMs where each layer L3 is defined (x(f (We + bx(2) + (x(f))))). We view un and UM as functions through their Fourier coefficients that are approximated numerically.
Applying L1 to uN and uM, have:
(v)w)) (Wiun(a) + + Ki(w))()))
(I( - (MWe))))))) (Wi) (
Let the difference between the outputs be A1(x) = [i(ww)(2) - (z)) By the Lipschitz continuity of a, have:
A - w (WieN (x) + (i(eN)(z)) - (Wieu - ( = w (Wi (ow (x) uM (x))) + (K)(v))(()) - (=))))
Assuming that Wi a bounded linear operator with operator norm || |, we have:
Mi (UN(2) - vM = [xx()) - uar (z)).
Moreover, the convolution difference (a(u))(z) - KG (uM)((( / k1(x 1) (vN(V) uas (3)) dy-
The difference after the l-th layer is:
(8)
(8)
UN
UM
Using the Lipschitz continuity of a with constant w and the boundedness of We and Kee, we have:
(0)
I w (I]Mvll Caa) Cnd) ( "
LA(T)*
L2(T)
where Cxe frr (K(()) By induction over layers, follows that:
(I [[ Cnn)) Cr..)
||00
A
S wt
(2(T)
We now estimate
A [0)
Let the difference between the Fourier coefficients be
L2(T))
N
M Σ
((E) = 7x) (E)
Σ
w(xn) (e 12 12 - 12
u(xn)e 12 Re
x=1
n=
where e -12n #5
12n + can be approximated using a Taylor expansion:
1
N M
12 -2 -i2mn
12x
Ee 12n RC
-i2mn
Çe
12x #5.
M
NM
follows that
IN M
M Σ
((E)
<
2T
Σ
ol )n
12x $2
v(z,)e
12 fr5
NM
N+=+1
Using Parseval's identity, we can sum over the relevant frequencies:
A 10)
Σ
A +
(((E))
L2(T)
5=0
16

---

Published as a conference paper at ICLR 2025
Substituting the estimate
((E)
we obtain:
N
M
A (0)
2m
Σ ( )
125 &
Σ
v ( 35
(P(T)
NM
5=0
R=N+1
Therefore, combining the above estimates and simplify, we have:
ENAr = 11
C (vlam))l
II
Ce
N-1
Σ
2x N M
Σ m=1
v( Ie 12x G5
+
Σ
c(xx)c
12 $
NM
N+1
(8)
where without the loss of generality we assume M > N, C. [|Well + Cru, with [[We]] being the operator norm of the linear operator We in the E-th layer.
Intepretation of this bound: When N .,, clearly this bound will be O. Without loss of general- ity, we assume M N, when is fixed, [N- gradually increases to 1, and as M increases, this bound increases. Moreover, Σ -5=0 Em=N+1 ((x,)e 12 + behaves as term that quantifies the difference between the input functions. The errors will propagate through the layers.
NM
Now, we are ready to prove Proposition 4.44 which has been written to help readers better under- stand our results. We begin proving the following lemma concerning the error bound of the discretization error for the full FNO, which implies Proposition 4.4
Lemma B.2. Let & S Iº be a finite- mnnite-immensionl parameter space, A = H (T² u Hd (T²,R4)) and A θ U map representing parametric class of Fourier neural operators with parameters E & with L Fourier layers and de hidden channels. Consider o as the activation function that (globally) w-Liscchitz continuous. Given discretization alay- then the discretization mismatch error between the Fourier neural operator taking alsx and another discretization a Bag given by
ENM = ||9 (alax) - (alax) (2(TT)))
2
MI NM
S CwL
II Ct
Σ
Ine
12x #5
Σ n=N+1
a(xn) 120 12 12
where both C and C2 are constants independent of N, M, and L.
Proof. We aim to establish the bound equation 9 on the discretization mismatch error EaSN by
extending Lemma B.I to the composite mapping G Q L with de hidden channels. Without the loss of generality, we assume = min
Let aN alpy and am alpas denote the discretizations the input function Applying the lifting layer P to a, and aM, we obtain
ag P(ax),
a = P(ax),
where the superscript denotes the number of Fourier layers that the latent feature functions have gone through.
By the linearity of P, Q, and the channel mixing of intermediate layers, and a E H= Rda), we have:
ExM = ||g (alex) ) 9 (ala (iess))ll2ee )1
< || ||||| ||c (a*) C (al) LP(T)
(10)
C- || (a) L' (a) (A(T))R4)))
where is constant depends on all the linear channel mixings and independent of N, M, and L, and L is the induced single channel operator of Fourier layers.
17

---

Published as a conference paper at ICLR 2025
To estimate the error propagation through L, utilize Lemma B.1 which provides an error bound for a single-channel case on T. We extend this result to the multi-channel, de-dimensional setting by considering each channel separately and summing the contributions. The error in each channel propagates similarly due to the Lipschitz continuity of a and the boundedness of the operators W2 and Ke. We bound the error after T. layers by:
ENM= 19 (alax)
9 (alaw)
)l LA(T")
<
Cww"
II
L
Ce
N-1
2x N M
Σ n=1
N
a(In
ne
12
12
+
M Σ
a(zn)e
12
NM
E=
n==++
(11)
This completes the proof.
Now, Proposition 4.4 follows immediately. The interpretation of this bound is similar that of Lemma B.I Note that we can fix N or M to be the training resolution; thus, this bound suggests that long as the inference resolution deviates from that of the training, DMEs arise. In the main text, we use plain language to ensure easier interpretation for the readers.
C FULLY CONNECTED POINT-WISE NETWORK AND BAND-LIMITS
For the definition of band-limited function spaces, refer readers to Raonic et al. (2023. latent feature function is obtained, a fully connected point-wise network (implemented as a 1 x convolution with non-linear activations) is applied to capture information beyond the band-limit. Remark 4.5. Let v(x) E _ (T xd) be a band-limited function, and let an analytic non-linear function (-) be the fixed activation function in the fully connected point-wise network. Then the resulting function may not be hund-limited anymore. Additionally, the high frequencies of the re- sulting function determined by both the fully connected point-wise network and the learned latent feature function uzz)
). Once the
Proof. As point-wise linear transforms will only result in linear transform of the feature func- tion. Thus, does not introduce any high frequency information nor does it involve any interaction between different frequency modes. Without loss of generality, consider only layer of fully connected point-wise network, and we analyze the non-linear activations. Let e({) = ((v()))(EE) denote the Fourier transform of v(v), defined by
O(E) = / u(I)e ((z))e-2nt 2miie dx.
If (xx is a smooth function of x, we can expand a as follows:
1 "0)a²² +
o(x) + + 2!
3." 00)x² +....
Thus, we can express v(z) as:
o o v() = o(0) + (0)(((())))))))))))))))))))))))) + " = +....
Now, performing the Fourier transformation, we obtain:
((oo(()))( ( + ((0)((())))))))))))))))))))))))) 1 000 [
dEu
(En) 6 (E Eo) dEo
21 jn 2m
///
dEp dei
31 (0()))
i (So) 8 (51
(o) 0 (g - {1) dE +....
2n 2m
18

---

Published as a conference paper at ICLR 2025
We observe that, without loss of generality, if there exists > such that (()) 0 a o(x) may introduce frequencies beyond the band-limit of v(r).
It is important note that, based on the formulation, the high-frre nigh-frequency components beyond the band-limit are determined by both the fully connected point-wise network and the latent feature function learned by the intermediate neural operator. The intermediate neural operator is tasked to learn an appropriate latent feature function to capture high-frequeney components together with the fully connected point-wise network. Whether or not this structure can be proven have universality is of interest to future exploration.
C. SPECTRAL COMPARISON: FNO AND CROP
In Figure we present a spectral comparison between FNO and CROP on the Burgers example, adapted directly from Li et al. (202I). Both FNO and CROP demonstrate the ability to learn high- frequency information beyond their truncation or band-limit when performing inference at the same resolution training. However, under super-resolution inference, high frequencies show slight deviation from the ground truth. Notably, CROP consistently performs better at higher frequencies in both scenarios. It should be mentioned that this plot is in semilog scale, which visually amplifies the error for high frequencies.
Spectrum Comperison: No Resoltuion
Spectrum Comparison: Resoltuion
Ground Truhh
Ground Turh
FNO
10"1
FNO
Tuncation Modie/ssndimit
Tuncatior MocelBsndimit
10-2
Hagnhtce
Mayshure
20
10-1
10
(a)
(b)
Figure 4: (a): Illustration the magnitude spectra of CRNO and FNO for inference the same resolution as training. (b): Illustration of the magnitude spectra of CRNO and FNO for super- resolution inference.
Most importantly, the issue with DMEs is not not just on the ability to recover unseen high frequencies. Instead, DMEs cause significant errors in the low-frequency range. We demonstrate this with the spectral error plot for low frequencies. shown in Fig. [S] for the results in Table [.1.2 (the Navier- Stokes equation). It is evident that DMEs cause high errors in low frequencies. At the training res- olution, both FNO and CROP exhibit minimal errors in low frequencies; however, FNO introduces substantial errors in low frequeeciess during super-resolution inferences, whereas CROP remains consistent.
D EXPERIMENTAL DETAILS
All the experimental results, especially the timing results, are recorded on NVIDIA RTX A6000
with 48 GB GDDR6.
19

---

Published as a conference paper at ICLR 2025
0685
3400
0.005
Mumoer
CRNO: viferenee 239
CINO Widerenoo
Figure 5: Illustration of the magnitude spectra of the error for FNO and CRNO at a training resolu- tion of 64 x 64. is evident that DMEs cause significant inaccuracies in the low-frequency range.
D.1 THE NAVIER-STOKES EQUATION
We consider the 2D Navier-Stokes equation for viscous, incompressible fluid in vorticity form from Li et al. (202I):
(,w(z, t( Vw(x, t) vAw(2, f()).
IE (0,1)2,t (0, x (0,1), [0, I x 10,
V uu((,,) t) 0,
(12)
w(x, 0) wo(x),
where C (10, per ((0, R)) for any 0 is the velocity field, V x is the vorticity, wh E T ((0, 1)2;R) is the initial vorticity, v the viscosity coefficient, and I2 ((0, ()),))2:R) RR) is the forcing function.
D.1.1 THE ACCUMULATION OF DISCRETIZATION ERRORS IN SEC. 5.1
For Sec. 5.I] // is set to le We are interested learning the operator mapping the vorticity up to time 10 to the vorticity up to later time T 50 in auto-regressive manner for all the intermediate time steps with a step size of as in Li et al. (202T). This dataset is generated using the data generation scripts provided by the authors of Li al. (202T) in their GitHub repository. The PDE is solved using numerical solvers on 256 x 256 grid. The training dataset contains 1,000 trajectories (input-output function pairs), while the testing dataset includes 200 trajectories. Following the setups in Li et al. (202T), we downsample the training dataset to 64 x 64; the FNO model fully trained on this data and evaluated on the testing data at different resolutions, namely 256 x 256, 128 x 128, 64 x 64, and 32 32, also obtained through downsampling.
For Figure 3 (a), the FNO architecture is followed exactly provided by the authors Li et al. 2021 in their GitHub repository (4 Fourier Layers and hidden channel dimension of 20) except that the teacher foreing training strategy from Tran al. (2023) is adopted. The FNO model is trained on 64 x 64 data and we make inferences on different resolutions mentioned above.
The relative & error given by
Uu
relative & error
||u||2
20

---

Published as a conference paper at ICLR 2025
where ug is the prediction and u is the label produced by numerical solvers. Since the obtained results are discrete, the 2-norm is calculated as the 2-norm of vectors. We report the relative & error for all results in this paper unless otherwise specified. The experiment is repeated 10 times, and we record the mean.
In Figure 3 (b), we consider the same Navier-Stokes Equation and generated data. Due to the tremendous computational resources it might take, we are only interested learning the mapping G u10 + u1 without auto-regressive time steps, where ur denotes the vorticity at time t. The FNO is trained on data at 64 x 64, and we compare the difference between the FNO output when taking input data of resolution 64 64 and that of 256 256. The relative DME is given by
| (210l Daa) - Go (210lame))l
|| ll2
where the numerator is the DME and the denominator || llz is applied to help the readers to compare the DME relative to the label solution to which the neural operator is trained to map. As
Go (alma)) and Go (alozs) practically have different resolutions, the 2-norm is calculated using the
(2x)" Parsevals identity: |u|]2 := 2 ΣnEza [0x] ? for mapping T Rd where Tn (k)
denotes the k-th Fourier coefficient of for any & E Zd.
The experiment repeated 20 times; the mean and standard deviation are reported in Table 5 In addition, observe that for different datasets and different settings of FNO, the DME might fluctuate or not increase much after a few layers (potentially due to residual connections). As we are interested in the upper bound, we report one set of results that clearly reflects our findings. However, for all the experiments we have conducted, DME clearly exists, even with only 2 to 3 Fourier layers.
Table 5: The growth in relative discretization mismatch error (in percentage) with respect to increas- ing numbers of layers. This highlights our observation that discretization mismatch errors propagate through Fourier layers.
Number of Layers
1 0.28(+0.09)
4
5 0.43(+0.13)
Mean (Std)
0.46(=0.11) 0.43(+0.07) 0.45(+0.09)
Number of Layers Mean (Std)
6
9
10 1.5.
0.46(+1.12)
(.
0.65(+0.26)
(.9.())
Number of Layers
11
12
13
14
15 4.19(++062)
Mean (Std)
1.90(++0.6)
2.73(++0.77
3.30(+0.88)
0.97(+
1.1.2 CROSS RESOLUTION TEST IN SEC. 5.1
For the cross-resolution test of different models Sec. 5.1 the experiment repeated 10 times and the mean and standard deviation of the average relative & error over 40 auto-regressive time steps are plotted and also given in Table g in percentage.
We implemented two versions of the alias-free activation functions. For the 2x upsampling im- plementation, which is directly taken from Raonic et al. (2023), we increase the spatial grid size by factor of two by bicubic spline interpolation, e.g. 64 x 64 to 128 x 128, apply the activation functions on higher-resolution grids, then downsample to the original grid size by, again, bicubic spline interpolation. For the fixed upsampling implementation, inputs of different grid sizes all interpolated to the same size (e.g. 128 x 128) to apply the activation functions, and then interpolated back to their original grid sizes. The main difference between these two is that, during inference, the grid sizes also increase by factor of two in the 2 upsampling implementation while they are fixed in the fixed upsampling implementation.
1.1.3 THE NAVIER-STOKES EQUATION WITH HIGH REYNOLDS NUMBERS
In Sec. 1.2.1 is set to 5e 4 and 1e 4, respectively for Reynolds numbers 5, 000 and 10, 000. We are interested in learning the operator mapping the vorticity from time 0 to the vorticity at time T 10 and T 5 for Reynolds numbers 5, 000 and 10, 000, respectively. This dataset
21

---

Published as a conference paper at ICLR 2025
Table Average &g error during inference for the Navier-Stokes Equation under different resolutions of input parametric functions. CROP result with more significant digits. We can rarely observe any difference between inferences at different resolutions.
256 x 256
128 x 128
64 x 64
32 x 32
CROP 0.54137
0.54140(#0.0527) 0.54154(20.052)) 0.54142(10.0525)
Training resolution is 64 64.
is generated using the data generation scripts provided by the authors of Li et al. (2021) in their GitHub repository. The PDE is solved using numerical solvers on 256 x 256 grid. For Reynolds numbers 000, the dataset contains 1, 024 trajectories and follows 768/ 128 split. As higher Reynolds number leads to a more difficult learning task, for Reynolds numbers 10, 000, the dataset contains 2, 048 trajectories and follows 1792/ 1122/22//11 128 split.
D.2 THE DARCY FLOW EQUATION
We consider the steady-state of the 2D Darey Flow equation given by:
-V -V- (a(=)) = (x) u(x) = 0 x E 000,
(0, 1)2
where a(x) is the coefficient function and f(x) is the forcing functiio. We learn the non-linear operator mapping +> following the exact same setup and data as in Li al. (2021) and the operator mapping f + following the generation setup in Hasani & Ward (2024), where the diffusion coefficient, a (z), that accounts for anisotropy the flow is taken as
27 It + D2
sin (21
].
a(x) =
42
where x, and Ø2 are the two dimensions of x. We are interested in learning the solution operator, G: f + u, mapping the source term ((x to the solution u(x). 1000 input-output pairs of resolution 256 x 256 are generated based on Hasani & Ward (202)) and the program provided in their GitHub repository. The data is downsampled 800/100/100 split.
64 training, and the train/val/test data follows
RrssstResolution Results on the Non-linear Mapping a u. In Table 7 we present the com- parison of cross-resolution tasks between CROP and FNO. Clearly, CROP remains consistent under different resolutions of inference, while FNO suffers from DMEs. This is not an autoregressive task, so the DMEs do not propagate through time; however, still observe significant downgrade in performance for FNO.
The mean and standard deviation of relative & eтror over 10 runs during inference are reported for the nonlinear operator - u on the Darcy flow equation with varying resolutions of input parametric functions. CROP is consistent across all resolutions. FNO shows strong bias toward the training resolution. The DME is obviously shown when inferred on data with different resolutions.
Table 7:
421 421
141 X 141
85 85
43 43
FNO
0.87
0.75(+0.003 0.51(10.01)
0.68(=00.3) 0.51(10.01)
0.97(=0.03)
CROP (ours)
0.52 +0.01)
0.57 +0.01)
Training resolution is 85 85.
D.3 THE POISSON EQUATION
We consider a prototypical 2D Poisson equation with classical Dirichlet boundary condition from Raonic al. 2023 given by
-du(z) = f(x),
x E (0, 1)2 x E 0(0, 1)² u(z),
(13)
22

---

Published as a conference paper at ICLR 2025
The solution operator, G 1 u, maps the source term f(x) the solution u(z). The source term given by
K
f(z, = π
(2) "iin(iii)) sin( v(r,v) D.
Gij
K
JJ=1
with -0.5, the corresponding exact solution can analytically computed by
K
uxx, y)
Eas 6,
anj
(2) sin(Tri)) sin(.)y)).
TK²
The data is generated by fixing K = 16 and choosing d; to be i.idd uniformly distributed from
[-1,1]. 1]. I We directly use the data provided by Raonic et al. (2023) and follow their setups.
D.4 THE HELMHOLTZ EQUATION
We consider the 2D Helmholtz equation from
De Hoop et al. (2022) given by
w2
-
c²
(x)
u
=
0
in Ω
Ou
0
on 00,, 091,
(14)
On
Ou
un on 8Q3,
On
where ! [0, 1]2 and {$4) are the four edges of the square domain. is set to 103, c::: Ω - R the wave-speed field, and u - is the excitation field that solves the equations. The Neumann boundary condition imposed on 0Ω is non-zero only the top edge Ω23 Throughout the
experiments presented this work, uN fixed at 1 (0.355=50.05) following De Hoop et al. (2022) The wave-speed field c(x) is assumed to be
c(x) = 20 + tanh(((z)),
where č is a centered Gaussian
غ N(O, C) and C= (-
Here - denotes the Laplacian on D. subject to homogeneous Neumann boundary conditions on the space of spatial-mean zero functions with 2 and 5 3. We are interested the operator mapping the wave-speed field c to the solution u. The dataset is provided by De Hoop et al. (2022) directly, where this PDE is solved using a finite element method on 100 x 100 grid. We take the first 800 data samples as the training data and the next 200 data samples as the testing. Additionally, the data normalized to [0, 1] based on the training data.
D.5 ADDITIONAL
RESULTS
Table 8: The Relative f2 error for varying resolutions of input parametric functions. In the Navier- Stokes example, the 256 x 256 data does not contain more high-freyuency information. We can clearly see that CRNO performs consistently.
Navier-Stokes (R, 10, 000)
Darcy Flow
256 x 256
64 6
256 256
64 64
FNO
8.903623
5.789168
3.636376
1.281445
CROP
2.466607
2.466613
2.804997
1.096139
We provide the cross-resolution results for FNO and CROP in Table D.5 Other models either cannot directly perform cross coss-resso tasks (c.g.. DeepONet) or exhibit dramatically poor performance (e.g., CNN-based models), which leads omit their comparison.
23

---

Published as a conference paper at ICLR 2025
We test the cross-resolution performance for both models using the first batch of data from the test dataset. In the Navier-Stole Stokes example, the 256 x 256 data does not contain more figh-freuuency information than the 64 x 64 data. We can clearly that CRNO performs consistently during inference at different resolutions. In the Darey flow (linear operator f + u) example, however, the 256 x 256 data does contain more high-frequency information, resulting in degradation for both models. This again demonstrates our concerns about zero-shot super-resolution tasks, which we will discuss further in Appendix E
D.6 IMPLEMENTATION DETAILS
Model details:
FNO (Li et al. 202I): Adopted from the code provided by the authors in their GitHub
repository with no change to their architecture except that we increase the modes to 24 with width of 16.
CNO (Raonic et al. 2023): Adopted from the code provided by the authors in their GitHub repository with no change to their architecture.
U-Net: Adopted from the code provided by PDEBench (Takamoto al. 2024 with no
change to their architecture at all.
ResNet: Adopted from the well-known ResNet50 architecture, we made minor adjustments to adapt for our function mapping tasks instead of classification tasks. Specifically, removed the final pooling and linear layers, and we used a padding of 1 and stride to ensure that all channels maintain the same spatial size the input. Finally, we employed
1 X 1 convolutional layer to project the feature maps into the final output.
DeepONet: Adopted from the code provided by the authors of Raonic et al.) (2023) in their GitHub repository, we made some corrections make it runnable on end. Specifially, a CNN-based network is used as the branch network and an MLP is used as the trunk network.
CROP (ours): For the Navier Stokes equation with Reynold number of 1, 000, we apply the FNO as the intermediate neural operator as the pattern is rather global. For all other ex- periments, we use U-Net or FNO with local 3 x 3 convolution kernels as the intermediate neural operator to capture fine details. Additionally, we find that for some problems, such as the nonlinear Darcy problem and the Navier Stokes equation with low Reynolds num- ber, applying learnable convolution kernels the CROP lifting and projection layers prone to overfitting; band-limited identity kernels (i.e. Fourier projection and interpolation operators) are applied.
All models trained using the hyperparameters in Table 9 except the ones that the original work provides the hyperparameter settings, such as FNO with the Darch flow and the Navier Stokes equation with low Reynolds number, with early stopping implemented if the validation error does not improve over a specified number of epochs. One exception in which we do not apply early stopping is the nonlinear Darcy example, the Helmholtz example, and the Navier Stokes equation with low Reynolds number; we follow the original setups from Li et al. (2021), and there is no validation set. However, for baseline methods, grid search over the hyperparameter space still conducted based on test error, which solely benefits the baselines.
Table 9: Training paraee-parameters
Hyper-parameter
Value
Learning Rate Weight Decay
0.001; 0.0005 for ResNet; 0.0001 for DeepONet
le -6
Scheduler Step
10
Scheduler Gamma
0.98
Epochs
1000; 2000 for DeepONet
Batch Size
16
Patience for Early Stopping
100; 500 for DeepONet
24

---

Published as a conference paper at ICLR 2025
E SUPER-RESOLUTION AND ZERO-SHOT SUPER-RESOLUTION
Zero-shot super-resolution refers to performing super-resolution without prior training on high- resolution target data. For FNO (Li al. 202T), this concept suggests that the model can be trained on lower-resolution data (e.g., e. 64 x 64) and applied to higher-resolution inference (e.g., 256 x 256). However, as demonstrated this work, FNO fails achieve zero-shot super-resolution due dis- cretization mismatch errors. Philosophically. without high-resolution information in the training set, it is questionable whether neural operators can infer unseen nigh-resolution features without leveraging techniques like transfer learning, physics-informet losses, conservation laws, or Hamil- tonian principles. Therefore, we encourage the community to shift focus from super-resolution tests for neural operators toward fundamentally improving their design, incorporating physics and prior knowledge about the operator. Despite these challenges, we believe FNO offers strong potential for data-driven super-resolution tasks.
Super-resolution, on the other hand, typically relies on training a model on a dataset that contains high-resolution target data, learning to predict high-resolution details from lower-resolution inputs (Dong et al. 2016 Lim et al.] 2017] Liang et al. 2021). The main difference between zero-shot super-resolution and data-driven super-resolution is that high-resolution images are actually seen by the model during training of the model. Interestingly, neural operators recently have been in- vestigated and applied for such tasks; for example, Wei & Zhang (2023) employs deep operator learning framework for super-resolution tasks in computer vision, specifically targeting natural im- age applications, and has achieved notable results.
A key task in physical models is reconstruction with limited sensors, which can be framed as a super- resolution problem. The objective is infer missing information and enhance resolution or fidelity by leveraging underlying physical principles or learned patterns. Specifically, this task involves reconstructing high-resolution or fine-scale field f TR' from coarse coarse-grained or limited sensor measurements E R', where < h, and often / < h. The goal is to learn a mapping F RR + g from data, such that F(s) RS f. Neural operators extend this approach by mapping from fixed-size vectors to continuous functions, i.e., F RR U for some appropriate function space 4. This task aligns well with the DeepONet paradigm eLu et al. 202I). Indeed, Neural Implicit Flow (Pan et al.) 2023). which differs from but shares similarities with DeepONet, has been proposed in this context. We believe that FNO holds strong potential for such tasks as well, especially for applications with symmetries such as modeling climate dynamics spherical geometries. Although some innovation may be needed to adapt FNO for these applications, it is very interesting future work to explore.
25


Reviews:
[{"review_id": "kDNoRQEBj7", "reviewer": "Reviewer_w3hk", "summary": "This paper examines \"discretization mismatch errors\" in neural operators, like the Fourier Neural Operator (FNO), which learn mappings between different types of data. The authors find that training these models at low resolutions and applying them at high resolutions can lead to performance issues. They propose a new Cross-Resolution Operator-learning Pipeline to avoid these errors, improving accuracy in tasks requiring precise predictions across multiple resolutions, such as climate modeling and fluid dynamics.", "questions": "Please refer to the questions in the weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The phenomenon of \"discretization mismatch errors\" is a significant issue in operator learning. This paper proposes the CROP framework to address (reduce) this error.\n\n2. The numerical examples in this paper show that CROP achieves improved performance over classic neural operator architectures, with the Fourier Neural Operator (FNO) used as an example.", "weaknesses": "1. Compared to approaches like \"physics-informed constraints\" and \"continuous-discrete equivalence (CDE),\" the CROP framework takes a more practice-oriented perspective. That is, the CROP framework is not inherently discretization-invariant. Specifically, it can only operate on fixed scales chosen *a priori*. This raises a question: how can the CROP framework be adapted to handle infinitely fine resolutions? Is there a formulation of CROP that accommodates an infinite range of scales?\n\n2. The theoretical treatment of \"discretization mismatch errors,\" particularly the results in Proposition 4.4, warrants closer examination:\n   1. The authors only provide an upper bound for the error $E_{MN}$. Mathematically, this does not necessarily imply that the \"discretization mismatch errors\" will vary in line with the trends of the upper bound.\n   2. Given the upper bounds in Lemma B.2, how should we interpret the conclusion that \"discretization mismatch errors\" increase as $M$ grows? Interestingly, it appears that the upper bound does not depend on $M$.\n   3. The derivation of Lemma B.2 is relatively straightforward, especially for the terms $\\omega^L$ and $\\prod_{\\ell=1}^L C_\\ell$, making the conclusion that \"discretization mismatch errors\" may increase with $L$ and $\\omega$ unsurprising. This could likely be generalized to other neural operator models as well.\n\n3. The numerical results in Table 2 could be expanded. Specifically, it would be beneficial to include results from training resolutions of $127 \\times 128$ or even $256 \\times 256$. Many recent studies on neural operators (such as papers from ICLR 2024) include similar numerical tests, but their results appear more favorable than those shown in Table 2 of this manuscript.\n\n4. I have concerns about the numerical tests in Table 4, as the operators used in the Darcy and Poisson problems are both linear, while the neural operators being tested (FNO, CNO, U-Net, ResNet, DeepONet) are nonlinear. Moreover, the results for the Poisson problem in Table 7 do not demonstrate a significant advantage of CROP over the standard FNO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wuETFTIZYf", "reviewer": "Reviewer_FHqG", "summary": "This paper first pointed out a common misconception that FNO does not depend on the resolution of input. The authors proved that resolution DOES affect FNO and defined a metric, discretization mismatch error (DME), to quantify the effect of resolution changing. Then they proposed a solution to mitigate the issue that ensures FNO's high performance across different resolutions.\n\nThe author provided estimation of DME by mathematical analysis. \nLemma B.1. provides the estimation of difference between input functions of FNO layers at different resolution $N$ and $M$, which is bounded by $o(\\frac{1}{N^{s-1}})$ assuming $N<M$ and $s\\geq2$. Lemma B.2. extended the analysis of Lemma B.1. to neural operators with lifting and projection layers.\n\nThey proposed a solution, cross-resolution operating learning pipeline (CROP), which relies on 1x1 convolution as lifting and projection layers.\nThen the authors conducted experiments to show two superior aspects of CROP: cross-resolution tasks and learning capability.", "questions": "N.A.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality** Though the development of neural operators has been accelerating recently, some fundamental concepts need clarification. I really appreciate the contribution of this paper that addresses the particular misconception of cross-resolution ability of FNO and puts neural operators on solid ground. The solution proposed here is based on band-limited function spaces and implemented as 1x1 convolution.\n\n**Quality** The mathematical analysis is solid and experiment is persuasive. \n\n**Clarity** I find the paper clear to read and carefully composed.\n\n**Significance** is relatively high. Cross-resolution application is a favorable feature of neural operators like FNO. The price to pay for such feature should be reminded for researchers in this field.", "weaknesses": "Since your work is to some degree a direct challenge to part of the arguments in [1], can you provide some insights to their evidence of \"discretization invariance\" that contradicts your findings? Namely, in Section 7 of [1], several experiments support the argument \"the error of FNO is independent of resolution or any specific discretization\", see Fig. 8 and Table 2&3. How would you explain such results?\n\n\n[1] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew\nStuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces\nwith applications to pdes. Journal of Machine Learning Research, 24(89):1–97, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M3rm9AoQR2", "reviewer": "Reviewer_z3xH", "summary": "This paper addresses the challenge of discretization mismatch errors (DMEs) in neural operators, particularly FNOs. The authors identify that while FNOs are theoretically designed to be discretization-invariant, they still exhibit performance inconsistencies across different resolutions due to DMEs. I like that they also provide a thorough mathematical analysis of how these errors accumulate through the layers of the neural network. To address this issue, they propose a novel Cross-Resolution Operator-learning Pipeline (CROP). CROP introduces lifting and projection layers that map input and output functions to and from a band-limited function space, allowing the use of fixed discretization's in the latent space. This approach effectively minimizes DMEs and enables consistent performance across different resolutions. The authors demonstrate CROP's effectiveness through extensive experiments on various partial differential equations, including the Navier-Stokes equation at high Reynolds numbers. Results show that CROP not only achieves robust cross-resolution performance but also outperforms baseline models in learning complex dynamics. Overall, this paper contributes significantly to the field of neural operators by addressing a critical limitation and proposing a solution that maintains the advantages of FNOs while overcoming their resolution-dependent weaknesses. As a person who works primarily on Neural Operators, I figured someday someone would finally do this work: ) and talk about it. I do have certain questions and concerns but overall, i really enjoyed the presentation of this paper.", "questions": "I have several questions so please bare with me:\n1  - So, is CROP still a universal approximation for operators? It seems like its not but I may be mistaken.\n2 - I do understand the concept of band limiting and then learning a NO in the intermediate representation, but to be more fair in comparison to FNO for high Reynolds tasks, you should use higher number of modes to capture those details or use something like an Incremental Fourier Neural Operator [1]. It would be great to do some ablation studies on that if you have the time. \n3 - How sensitive is CROP's performance to the choice of band-limit in the lifting and projection layers? Is there a systematic way to determine optimal band-limits for different types of problems?\n4 - How does the computational complexity of CROP compare to standard FNO, especially for very high-resolution inputs?\n5 - How does the choice of intermediate neural operator in CROP affect its performance and computational efficiency? Are there certain types of problems where specific architectures are preferable?\n6 - Also I do get you have additional experiments on deeper FNO's but it would be great to actually also use skip connections or other techniques to actually train them well. They do suffer from convergence issues.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Analysis: The paper  identifies and provides a rigorous theoretical analysis of discretization mismatch errors (DMEs) in neural operators, particularly FNOs, addressing a critical gap in the field. The results do make sense to me as it makes sense that although FNO does have the discretization invariance property, just training on low res - isnt guaranteed enough to get the high level features in high -res images.\n2. Unique solution: The proposed CROP method effectively mitigates DMEs through a clever use of lifting and projection layers, allowing for consistent cross-resolution performance and flexibility in choosing intermediate neural operators. \n3. Strong empirical validation: Extensive experiments across multiple PDEs, including challenging cases like high Reynolds number Navier-Stokes equations, demonstrate CROP's superior performance compared to state-of-the-art baselines.\n4. Clear exposition and reproducibility: I like the fact that the paper is well-structured, clearly written, and provides detailed information on experimental setups, enhancing understanding and reproducibility of the results as well as they don't just bash FNO's, they actually justify their reasons well.", "weaknesses": "Some weaknesses\n1. Variance in some results: Some empirical results, particularly in Figure 3b, show high variance. Maybe some explanation of the high variance?\n2. There's minimal discussion on how sensitive CROP is to hyperparameter choices, which is important for understanding its robustness and ease of application to new PDE problems\n3. Long-term stability in time-dependent problems: For time-dependent PDEs like Navier-Stokes, there's limited discussion on the long-term stability of CROP predictions over extended time horizons.\n4. Limited exploration of high-frequency information: While CROP addresses the issue of DMEs, the paper doesn't deeply explore how effectively it captures high-frequency information, especially in comparison to methods specifically designed for super-resolution tasks.\n5. It would be also great to have pseudocode or the full architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wvqbOmUsEg", "reviewer": "Reviewer_WGpV", "summary": "This paper emphasizes that the colloquial definition of discretization invariance does not align with the formal definition. They also demonstrate both formally and empirically that popular operator learning architectures (such as FNO) do not have constant performance across resolutions for many problems. The authors propose the CROP method to address these issues with bandlimited lifting and projection operators (as well as some pointwise operators to capture high frequency details), and they show improved performance compared to popular neural operator architectures.", "questions": "1. In the introduction, the authors claim that “It is widely believed that training an FNO on one resolution allows inference on another without degrading its performance, since FNO operates and parameterizes the kernel on the Fourier space.” Do you have references for this claim?\n2. What is L in the equation in Definition 4.1?\n3. In Proposition 4.4, it is stated that the discretization mismatch error increases with M, but M does not seem to appear in Lemma B.2. I may be missing something, but how does Proposition 4.4 follow from Lemma B.2?\n4. How much were the hyperparameters tuned for the baselines, particularly in the Navier-Stokes setting?\n5. What architectures were tested for the intermediate neural operator in CROP?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written, especially for those who may not be experts in neural operator theory. I appreciate the formal definition of discretization invariance from Kovachki et al., 2021 and the discussion of the differences between this definition and the colloquial definition. The experimental results are also quite compelling and suggest that the proposed CROP method vastly outperforms other methods.", "weaknesses": "Although the paper has some strengths, there are still some areas of improvement. My first concern is with the novelty of the insights about discretization invariance. Previous works have included similar discussions, such as [1] (See Figure 1 and Section 2), which uses the term “discretization convergence” to refer to the formal definition by Kovachki et al., 2021. Prior works have also demonstrated empirically that FNO performs well in zero-shot super-resolution tasks on smooth problems. Given these results, it seems to me that the primary contribution of this work is some theoretical analysis of the discretization mismatch error (the Lemmas in the appendix) and the CROP method.\n\nIn regards to the proposed CROP method, it is unclear to me whether high frequencies are adequately being captured by the proposed pointwise neural network in the projection layer. For some problems, looking at the L2 error may not be sufficient to ensure that high frequencies are captured. As such, I would recommend the authors include additional evaluation metrics, such as plotting the spectrum of each and seeing how CROP vs. FNO perform on the higher frequencies (for reference, see Figure 1 in [1]).\n\nI also have some concerns about the statement of Proposition 4.4. I understand that it is meant to be written informally for a broad audience. However, I believe some aspects of it are potentially misleading: for instance, it is not necessarily always true that the discretization mismatch error must increase with the higher resolution, right? If the operator learns a constant function, for instance, then this would not strictly increase.\n\n**References:**\n1. “Neural operators for accelerating scientific simulations and design” (2024).\n\n**Minor notes and typos:**\n1. The authors write that “The FNO architecture is not limited to rectangular domains, periodic functions, or uniform grids.” I think this might be a bit misleading, since typical instantiations of FNO are limited by these constraints: the FFT can only be applied on uniform grids, without Fourier continuation, the output of the FNO is periodic, etc. It would be ideal if the authors can add some context to this claim.\n2. Typo in line 234 “aN FNO.”\n3. I see that the appendix contains proofs of rigorous statements. I would clearly denote that Proposition 4.4 is informal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)