Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning
Authors: Francesca Bartolucci
Emmanuel de Bezenac
Bogdan Raonic
Roberto Molinaro
Siddhartha Mishra
Rima Alaifari
Venue: NeurIPS 2023 poster (2023)
Keywords: Operator Learning
Neural Operators
PDEs
Frame theory
Sampling theory
TL;DR: A unifying mathematical framework for learning operators is introduced, ensuring consistency at the continuous and discrete levels

Abstract: Recently, operator learning, or learning mappings between infinite-dimensional function spaces, has garnered significant attention, notably in relation to learning partial differential equations from data. Conceptually clear when outlined on paper, neural operators necessitate discretization in the transition to computer implementations. This step can compromise their integrity, often causing them to deviate from the underlying operators. This research offers a fresh take on neural operators with a framework Representation equivalent Neural Operators (ReNO) designed to address these issues. At its core is the concept of operator aliasing, which measures inconsistency between neural operators and their discrete representations. We explore this for widely-used operator learning techniques. Our findings detail how aliasing introduces errors when handling different discretizations and grids and loss of crucial continuous structures. More generally, this framework not only sheds light on existing challenges but, given its constructive and broad nature, also potentially offers tools for developing new neural operators.

Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning
Francesca Bartolucei' Emmanuel de Bézenac Bogdan Raonic2.3
Roberto Molinaro Siddhartha Mishra
Rima Alaifari2.3
Delft University of Technology, Netherlands Seminar for Applied Mathematics, ETH. Zurich, Switzerland ETH Al Center, Zurich, Switzerland
Abstract
Recently, operator learning, learning mappings between infinite-dimensional function spaces, has garnered significant attention, notably in relation to learning partial differential equations from data. Conceptually clear when outlined on paper, neural operators necessitate discretization in the transition to computer implemen- tations. This step can compromise their integrity, often causing them to deviate from the underlying operators. This research offers a fresh take on neural operators with a framework Representation equivalent Neural Operators (ReNO) designed address these issues. At its core is the concept of operator aliasing, which measures inconsistency between neural operators and their discrete representations. We explore this for widely-used operator learning techniques. Our findings detail how aliasing introduces errors when handling different discretizations and grids and loss of crucial continuous structures. More generally, this framework not only sheds light on existing challenges but, given its constructive and broad nature, also potentially offers tools for developing new neural operators.
Introduction
Operators are mappings between infinite-dimee dimensional function spaces. Prominent examples are solution operators for ordinary and partial differential equations (PDEs) p which map function space inputs such as initial and boundary data to the function space valued PDE solution. They are also the natural mathematical framework for inverse problems, both in the context of PDEs [2] and in (medical) imaging [3), where the object of interest is the inverse operator which maps observables to the underlying material property/image that needs to be inferred or reconstructed.
Given the prohibitive cost of traditional physics based algorithms for approximating operators, particularly those associated with PDEs, increasing attention is being paid in recent years to machine- learning based operator approximation. Such techniques for learning operators from data fall under the genre of operator learning. (by no means comprehensive) list of examples for operator learning architectures include operator networks (6), DeepONets (20), Graph neural operators (18), multipole
neural operators 19), PCA-Nets (4, Fourier neural operators (FNO) [17, VIDON 21 spectral
neural operators 10, LOCA 13, NOMAD (24), Continuous Generative Neural Networks 1 and
transformer based operator learning models [5].
However, there is still lack of clarity on what constitutes operator learning? Clearly, an operator learning framework, neural operator in the nomenclature of (15), should be able process functions as inputs and oulputs. On the other hand, functions are continuous objects and in practice,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

one may not have access to functions either at the input or output level. Instead, one can only access functions and perform computations with them on digital computers through their discrete representations such as point values on grid, cell averages or in general, coefficients of an underlying basis. Hence, practice, neural operators have to map discrete inputs to discrete outputs. This leads to dichotomy, i.e., neural operators are designed to process functions as inputs/outputs but only have access to their discrete representations. Consequently, at any finite resolution, possible mismatches between the discretizations and the continuous versions of neural operators can lead to inconsistencies in the underlying function spaces, These inconsistency errors can propagate through the networks and may mar the performance of these algorithms. Moreover, important structural properties of the underlying operator, such as symmetries and conservation laws, hold at the continuous level. Inconsistent discretizations may not preserve these structural properties of the operator, leading to symmetry breaking etc. with attendant adverse consequences for operator approximation.
Addressing this possible inconsistency between neural operators and their discretizations is the central point of this article. To this end, we revisit and adapt existing notions of the relationship between continuous and discrete representations in signal processing. applied harmonic analysis and numerical analysis, with respect to the questions of sampling and interpolation. Roughly speaking, we aim to find mathematical framework in which the continuous objects (functions) can be completely (uniquely and stably) recovered from their discrete representations (point evaluations, basis coefficients, etc.) at any resolution. Consequently, working with discrete values is tantamount to accessing the underlying continuous object. This equivalence between the continuous and the discrete is leveraged to define class of neural operators. for which the discrete input-output representations or their continuous function space realizations are equivalent. On the other hand, lack of this equivalence leads to aliasing errors, which can propagate through the neural operator layers adversely affect operator approximation. More concretely, our contributions are
We provide novel, very general unifying mathematical formalism to characterize class of neural operators such that there is an equivalence between their continuous and discrete representations. These neural operators termed as Representation equivalent Neural Operators or ReNOs. Our definition results in an automatically consistent function space formulation for ReNOs.
To define ReNOs, we provide novel and precise quantification of the notion of aliasing error for operators. Consequently, ReNOs are neural operators with zero aliasing error.
We analyze existing operator learning architectures to find whether they are ReNOs not.
Synthetic numerical experiments are presented to illustrate ReNOs learn operators without aliasing and to also point out the practical consequences of aliasing errors particularly, with respect to evaluations of the neural operators on different grid resolutions.
2 A short background on Frame Theory and Aliasing
We start with a short discussion, revisiting concepts on the relationship between functions and their discrete representations. This the very essence of frame theory, widely used in signal processing and applied harmonic analysis. We refer to fMrr detailed introduction to frame theory.
Equivalence between functions and their point samples. For simplicity of exposition, we start with univariate functions f E (R) and the following question: Can a function be uniquely and stably recovered from its values, sampled from equispaced grid points {f("') heez? ? The classical Whittaker Shaeer--Shaaannnnne nikov (WSK) sampling theorem (25). which lies at the heart of digital-to- analog conversion, answers this question in the affirmative when the underlying function f E Ba, i.e., it belongs to the Faley-Wienee space of band-limited functions Be { (B): supp/ C [-22, [2],, for some Ω > 0. if the Fourier transform of f and sampling rate 1 1/T 2 20, with 20 termed as the Nyquist rate. The corresponding reconstruction formula is,
f(x) 2TΩ Σ (olT))sinc(((4)) nI))), sinc(a)) sin(TT)
(2.1)
2

---

We note that the sequence of functions (6)(2) ) = sinc(20ix ml) constitutes an orthonormal basis for BR and denote by PBa+ L (R) + Bp the orthogonal projection operator onto Ba,
PBnf Σ (f EE
Σ nEZ
12
On.
(2.2)
20
where the last equality is a consequence of Bq being a reproducing kernel Hilbert space with kernel (2 y) sinc(29 - y)). Hence, J corresponds to the right-hand side of (2.T) for T = 1/20 and formula (2.1) is exact if and only if f E Ba, ic.. f = PB,f + f E Bn.
What happens when we sample a function at sampling rate below the Nyquist rate, i.c. when 1/T < 292? Alternatively, what is the effect of approximating a function f E Ba by PB. Again, sampling theory provides an answer in the form of the aliasing error:
Definition 2.1. Aliasing for bandlimited functions 2 $51 The aliasing error function e(f) and the corresponding aliasing error of f E (TR for sampling at the rate 2n are given by
s(f) f-
and
| || fl2.
If the aliasing errors sff) is zero, i.e. if E Ba, we say that there is continuous-discrete equivalence (CDE) between f and its samples (f(n(2))
Continuous- Discrete Equivalence in Hilbert spaces. Next, we generalize the above described concept of continuous-discres and norm || ||- A countable sequence of vectors (fi)ier in H is a frame for H if there exist constants
equivalence to any separable Hilbert space H. with inner product (::))
A, B > 0 such that for all f H
A|S11 S DIVADPS S BBI/1I
(2.3)
SEI
Clearly, an orthonormal basis for H. is an example of a frame with A = B = 1. We will now define maps that will allow us to make the link between functions and their discrete representations, which we will extensively use in the following sections. The bounded operator 7: (T( - H. T((cheer)
Eier C, Ji: is called synthesis operator and its adjoint T": H - (T), T (V. (f..e) which discretizes the function by extracting its frame coefficients, is called analysis operator. By composing T' and T we obtain the frame operator S ;= TT* which is an invertible, self-adjoint and positive operator. Furthermore, the pseudo-inverse of the synthesis operator is given by T': H - (()): f = {( S (t) hir, which will be used in the following to reconstruct the function from its discrete representation, i.e. its frame coefficients.
With these concepts. one can introduce the most prominent result in frame theory M. the frame decomposition theorem, which states that every element in H can be uniquely and stably reconstructed from its frame coefficients by means of the reconstruction formula
ʃ S=TT'S UU,
MJ Σ (U.
(2.4)
SE
YEl
where the series converge unconditionally. Formula (2.4) is clearly generalization of reconstruction formula (2.1. However, it is worth pointing out that, general, the coefficients in (2.4) are not necessarily point samples of the underlying function f, but more general frame coefficients.
In general, it may not be possible access all the frame coefficients to reconstruct function in Hilbert space. Instead, just like in the case of reconstructing functions from point samples, one will need to consider approximations to this idealized situation. This is best encapsulated by the notion of a frame sequence, i.e., a countable sequence {ui E H. which is a frame for its closed linear span, i.e. for span(u; : E 13. With this notion, we are position to generalize aliasing errors and the CDE, to arbitrary Hilbert spaces. Let H be a separable Hilbert space and let {u:)ie/ be a frame sequence for H with = span (o; : i E I} and frame operator S: V + V. Then, the orthogonal projection of H onto V is given by Pvf = Eier (f. S Us Thus, formula (2.4) holds and the function f can be uniquely and stably recovered from its frame coefficients if and only if f E V. If E V. reconstructing f from the corresponding frame coefficients results in an aliasing error: Definition 2.2. Aliasing for functions in arbitrary separable Hilbert spaces. The aliasing error function {(f) and the resulting aliasing error [[s(S)] of f E H for the frume sequence (uiliet C V are given by
= = -Pff ||=(f]] Ilf Puffll
3

---

(1)
u(v.0)
(()
T
Tv
H
K
H
V
K
Tv
T4
H
U
K
T 26
(1) e (K)
Tip'
Th.
(1) u
(I)
e (K)
(b) ReNO: an alias-frce u can be constructed by discretizing the op- erator U for different discretiza- tions given V, d.
(c) ReNO: discrete representations u, for different discretizations are equiva- lent.
(a) An alias-free operator's diagram commutes.
Figure I: Alias-free framework. U is the underlying operator, u its discrete implementation. The synthesis operators and their pseudo-inverses make the link between function and discrete space.
If the aliasing error e(f) is zero, i.e if E V. say that there is continuous-discrete equivalence (CDE) between f and its frame coefficients {(f, 1z) hee.
3 Alias-Free Framework for Operator Learning
In this section, we will extend the concept aliasing functions operators. We demonstrate that this notion of operator aliasing is fundamenial to understanding how the neural operator performs across various discretizations, thereby addressing the shortcomings possible lack of continuous- discrete equivalence. We will then define Representation equivalent Neural operators (ReNOs), whose discrete representations are equivalent across discretizations.
Assume we posit a (neural) operator U, mapping between infinite diinnte-dimensional function spaces. As mentioned before, this operator is never computed practice, instead a discrete mapping is used. The difficulty in formalizing this is that we should be able to compute this discrete mapping for any discretizations of the input and output function. We formalize this notion below in a practical manner.
Setting. Let U: Dom U C H + K. be operator between two separable Hilbert spaces, and let v = { her and = [ōu]ken be frame sequences for H and IC, respectively, with synthesis operators Tи and Ts. We denote their closed linear spans by My := span( : / and M4 span k K). We note that by classical frame theory (7), the pseudo-inverses T and Ty. initially defined on My and Mr, respectively, can fact be extended to the entire Hilbert spaces, i.c. T 7 (1) and T I. (K).
3.1 Operator Aliasing and Representation Equivalence
Once the discretization is chosen determined by input and output frame sequences (V, +) connecting the continuous operator U with its discrete counterpart is the notion of operator aliasing. Given any mapping u: (T) (KK), definition clearly depends on the choices the frame sequences that we make on the continuous level. In other words, any mapping continuous operator, which in general, may differ from the operator U. that is of interest here. Hence, define the aliasing error of U relative the discrete
build the operator Te
T.. 7 - X, whose
be interpreted
discrete representation of underlying
in analogy Definitions 2.T and 2.2 representation as,
Definition 3.1. Operator aliasing. The aliasing error operator =(U, u. V, D): Dom C H K given by
<(U, u. V. 1) U- and the corresponding scalar error is |l= u, V [. with denoting the operator norm.
An aliasing error of zero implies that the operator U can be perfectly represented by first discretizing the function with TJ. applying u, and then reconstructing with T4, equivalently, that the diagram in Figure Ta commutes, i.e. the black and the blue directed paths in the diagram lead the same result. If the aliasing error is zero, say that (U, u, v, Φ) satisfies continuous discrret equivalence

---

(CDE). implying that accessing the discrete representation u is exactly the same as accessing the underlying continuous operator U.
Example 3.2. Aliasing with the magnitude squared operator. Consider the operator U(f) If as an operator from Bn into Bzn- The choice to discretize inputs and outputs on the same grid 20 nEZ corresponds choosing V/ @b {sinc(20er n)) and to defining the discrete mapping (Z) + (Z) by u(v) T.j U o Tw()) = @ ह, where @ denotes the entrywise product. Then, for every f E Bg such that (f) E B2n Ba, we have
a(U, U, V, () = U() To To (U()) = U() PB, (p,(((()) #0,
and we encounter aliasing. This occurs because U introduces new frequencies that exceed the bandwidth f2 However,
rectify this by sampling the output functions on a grid with twice the resolution of the input grid. This corresponds to choosing 4 = {sinc (49tx - n) mee and to defining To U To, which simply maps samples from grid points { 202 J REz into squared samples from the double resolution grid { ( This effectively removes aliasing since the equality
U = Ta u Th is satisfied. Furthermore, sampling the input and output functions with arbitrarily higher sampling rate, i.e. representing the functions with respect the system (sine(20tr - -n) ne) with 9 > 292, yields no aliasing error since {sinc(202r n)) mE constitutes a frame for B2n 2 Ba-
In practice the discrete representation of the operator U depends on the choice of the frame sequences. This means that as mentioned at the beginning of the section - there is one map for every input/output frame sequence V 4. The consistency between operations u, u' with respect to different frame sequences can be evaluated using the following error.
Definition 3.3. Representation equivalence error. Suppose that u, w are discrete maps with associated frame sequences V & and V/', Φ, respectively. Then, the representation equivalence error given by the function T (u, u') : (T) - =²((K), defined as:
T(U, u') u T o Tar o ud o T o Tw
and the corresponding scalar error is [[ el)l
Intuitively, this amounts to computing each mapping on their given discretization, and comparing them by expressing u' in the frames associated u. the following section we leverage the notion of operator aliasing in the context of operator learning, as well as explore its practical implications with respect to representation equivalence at the discrete level.
3.2 Representation equivalent Neural Operators (ReNO)
Equipped with the above notion of comtinuous-disrred equivalence, we can now introduce the concept of Representation equivalent Neural Operator (ReNO). To this end, for any pair (v, 4) of frame sequences for H and KC, we consider a mapping at the discrete level w(V,, Φ) Ran T + Ran T which handles discrete representations of the functions. Notice how this map is indexed by the frame sequences: this is normal as when the discretization changes, the definition of the function should also change. In order to alleviate the notation, when this is clear from the context, we will refer to u(V v, 2) simply u. See SM D.D for an explanation of the condition u( u 2) Ran T + Ran T.. Definition 3.4. Representation equivalent Neural Operators (ReNO). We say that (U, u) is a ReNO for every pair (M, 2) of frame sequences that satisfy Dom U C Ma and Ran C MM there is no aliasing, i.e. the aliasing error operator is identical to zero:
eU, u, V +)) 0.
(3.1)
We will write this property in short as EU, u)
In other words, the diagram in Figure la commutes for every considered pair (V, b). In this case, the discrete representations u( V 4) are all equivalent, meaning that they uniquely determine the same caatinuous-disceet equivalence property holds at the level of the
underlying operator U, whenever function spaces. The domain and range conditions in Definition simply imply that the frames can adequately represent input and output functions of U.
5

---

Remark 3.5. If the aliasing error e(U, u, W, V) is zero (as required in Definition 3.4), then the assumption that u( V, +) maps Ran T C (T) into Ran T C (K) implies that
V ②) = T o U oT..
(3.2)
We observe that this definition of u(V, ④) is such that the diagram in Figure lb commutes. In other words, once we fix the discrete representations V, associated to the input and output functions, there exists unique way define discretization u( +) that is consistent with the continuous operator U and this is given by (3.)). D. In practice, we may have access to different discrete representations of the input and output functions, which in the theory amounts to change of reference systems in the function spaces. Note that to avoid any aliasing error, the discrete representation U has to depend on the chosen frame sequences, i.e. inevitably, must depend v and Φ, and hence, must be discretization dependent. See See A.2 for the proof of Remark 3.5
In particular, Remark as
3.5 3.5 directly implies formula to go from one discrete representation another,
u (VV (') T To al V ②) Ti o Tw
(3.3)
whenever the pairs of frame sequences (V: +) and (V, Φ') satisfy the conditions in Definition 3.4. In other words, the diagram in commutes.
Formula (3.3) immediately implies Proposition 3.6 which establishes a link between aliasing and rep- resentation equivalence. This highlights the contrast to discretization invariance, discussed length in [1]] while this concept establishes asymptotic consistency, representation equivalence includes the direct comparison between any two given discretizations and guarantees their equivalence.
Proposition 3.6. Equivalence of ReNO discrete representations. Let (U, u) be a ReNO. For any two frame sequence pairs (V, 4 and (V,, D²) satisfying conditions in Definition 3.4 we have that
T(u, u') O,
where, by slight abuse of notation, u' denotes (V' $²).
Hence, under the assumption that the discrete map each discretization is consistent with the underlying continuous operator, we have a unique way to express the operator at each discretization. Moreover, formula (3.3) closely resembles analogous formulas presented in [17 13 23 when evaluating single shot super resolution. However, in Section we offer nuanced perspective, indicating variability across different scenarios.
3.3 Liyer-wise Instantiation
As shown in Remark 3.5 we can compute the outputs of the ReNO on the computer by first discretizing the continuous operator U. Typically, if U is neural operator composed of multiple layers, as we will show in this section, it possible to discretize each layer, while remaining consistent with the underlying operator.
Consider neural operator U with layers, each a mapping between separable Hilbert spaces:
U= = oUL.10.. U1,
Ug: He Het1,
8= 1= T.,
(3.4)
we denote by V, a frame sequence for He. The choice frame sequences for each Hr corresponds to the choice of accessible discrete representations of functions in the underlying function space He.
Proposition 3.7. Stability of ReNO under composition. Consider the composition U ULo...ol ......u,, as in eq. 3.4 as well as a discrete mapping u = 4y o... o U1 If each layer (Ue, ue) is a ReNO, the composition (U, u) also is ReNO.
As the proof of Proposition 3.7 presented in SM SMA.1 also shows, if each hidden layer in the operator 3.4 has an aliasing error then these errors may propagate through the network and increase with increasing number of layers.

---

3,4 C-ReNOs
In practice, we may not need to set the aliasing error to zero. But it essential be able to control and make it as small as desired. The notion of ReNOs, Definition3..4 can very simply be extended to the case where we allow for small, controlled amount of aliasing. Indeed we can introduce ReNOs, which satisfy [|s(U, u)ll S (for every pair of admissible frame sequences) instead of
E(U, u) =0
Proposition 3.8. Let (U, u) be ReNO. For any two frame sequence pairs (V, b) and (V' ②()
satisfying conditions in Definition 3.4 And such that Mer S MM, we have
[u, u')ll S 2cvBy
VA+
where, by slight abuse of notation, u' denotes u(W, ②²).
Examples
Equipped with our definition of Representation equivalent Neural Operators (ReNOs), analyze some existing operator learning architectures to ascertain whether they are neural operators or not.
Convolutional Neural Networks (CNN). Classical convolutional neural networks are based on the convolutional layer k, involving a discrete kernel f E IR2++1 and a discrete input c:
k(c)[m] (f c) [m] = Σ clm i]]
We can then analyze this layer using our framework to ask whether this operation can be associated to some underlying continuous operator. Intuitively, if this is the case, the computations conducted on different discretizations effectively representing the input should be consistent; in the contrary case, no associated continuous operator exists and the convolutional operation is not a ReNO.
Consider the case where the discrete input corresponds to pointwise evaluation on grid of some E Z with associated orthonormal basis V {sinc(2ftz: n)) nEZ. Consider now an alternate representation of n)
underlying bandlimited function f Bn, for example cnn] =f ( (2n)),
point samples of grid twice as fine: d/n] f (m)) with V (sinc(492x n))nez as basis. Clearly, even though discrete inputs agree, i.c. cinj d[2n]. this is no longer true for the outputs, (k [n] A (k * d) [2]] This in turn implies that there exist frame sequences V, y such that:
Tv (c(c) o T.j + Twp ky (d) T
(4.1)
thereby defying the representation equivalence property of ReNOs, in the sense of Definition 3.4 This fact also corroborated with the experimental analysis, Section 5
Fourier Neural Operators (FNO). FNOs are defined in [17 in terms of layers, which are either lifting or projection layers or Fourier layers. As lifting and projection layers do not change the underlying spatial structure of the input function, but only act on the channel width, these linear layers will satisfy Definition of ReNO here. Hence, we focus on the Fourier layer of the form,
(++1 (Aqve(x) + Be(x) Kue(x)),
(4.2)
with the Fourier operator given by
Kv FROF
Here, F,F-1 are the Fourier and Inverse Fourier transforms.
For simplicity of exposition, we set A; = B, = and focus on investigating whether the Fourier layer (4.2) satisfies the requirements of a ReNO. Following [17], the discrete form of the Fourier layer given by a (kr), with kv = F-1 ④) F(u)), where F, F denote the discrete Fourier transform (DFT) and its inverse.
7

---

In B. we show that the convolution in Fourier space operation for the FNO layer (.2) sat- isfies the requirements of ReNO, However the pointwise activation function o of)). applied a bandlimited input f E B2 will not necessarily respect the bandlimiiss, i.e., aff) & Bp- In fact, with popular choices of activation functions such as ReL.U, (f) $ Bw. for any w > (see SM for numerical illustrations). Thus, the Fourier layer operator (4.2) may not respect the continuous-disrrtte equivalence and can lead to aliasing errors, a fact already identified in [10]. Hence, FNO may not be ReNO in the sense of Definition 3.4
Convolutional Neural Operators (CNO). Introduced in operator consist of three elementary operations
23 the layers of a convolutional neural
v++1 = Pi o En o (it)) 01-
(4.3)
where K2 is a convolution operator, Σn is non-linear operator whose definition depends the choice of an activation function a: IR + R. and Pi is a projection operator. We show in SM CNO layers respect equation (3.2) and consequently CNOs are Representation equivalent Neural Operators (ReNOs) in the sense of Definition 3.4. This is the result of the fact that the activation layer defined in way respect the band-limits of the underlying function space.
5 Empirical Analysis
5.1 Assessing Representation Equivalence
In the previous section, we have studied existing neural operator architectures from a theoretical perspective. In this section, for the same architectures, we corroborate these findings from an empirical viewpoint. Aliasing a quantity that cannot be computed in practice, as we cannot access the underlying operator U on a computer, we can nonetheless compute representation equivalent errors introduced in Definition 3.3, which is related to aliasing by Propositions 3.7 and 3.8 being a quantity of interest in itself).
(as well as
Experimental Setting, We wish to learn an unknown target operator Q using a neural operator. In this experiment, all neural operators (CNN, FNO and SNO) take as input pointwise evaluations on the grid, and are able deal with varying input resolutions. A simple way of constructing the target operator Q H -> H, where H is the space of periodic and K 30-bandlimited functions, by sampling input and output pairs a random fashion. Sampling of function H can be realized as follows: As we know that Vx == {dx(- == K...., K constitutes a frame for H, dx being the Dirichlet kernel of order K and 2µ 2k+ any function f E H can be written as f(x) = E (x zτ). Thus, the discrete representation of f simply corresponds to its 2K +1 61 point-wise evaluations on grid, i.e. (=x))))==-K..
Note that for simplicity we have used and will use the same frame sequences for both input and output spaces, as these are the same.
Training and Evaluation. Once the data is generated, we train neural operators u ( Kx) on discretizations associated to the frame sequences, which are simply the point-wise evaluations of the input-target functions in the data. In other words, we regress the frame coefficients of the target function, with coefficients of the input function as input. Once training is over, we evaluate how the different neural operators behave when dealing with changing input and output frame sequences. The frame sequences here are V M for different testing frames, with associated 2M + 1 sized grids, with associated operator uM : R2M++ + R2M+1
The results of this experiment are presented in Figure 2 we also provide results for additional architectures in Figure 4 The results in Figure [ clearly show that as predicted by our theory, neither CNN nor FNO are representation equivalent, in the sense of Definition 3.4 changing the resolution, which amounts to change of frame, does not keep the operator invariant, causing aliasing errors that materialize themselves as representation equivalence errors (as in Definition 3.3 here. On the other hand, as predicted by the theory, CNO more likely to be ReNO - as long as the frames selected satisfy conditions of Definition 3.4 (i.e. "Representation Equivalence" zone). When these conditions no longer hold ("No Equivalence" zone), CNO also generates aliasing errors. Thus, this experiment clearly demonstrates the practical implications of the ReNO framework. In contrast, the
8

---

200%
No Equivalence
Representation Equivalence
Discrete Aliasing Error (%6
173%
CNN
100%
F9%
FNO
CNO
Resolution
Figure 2: Representation equivalence is computed for three classical architectures, CNN, FNO and CNO trained resolution of 61 (yellow dot). The "Resolution Equivalence" zone, located on the right-hand side, denotes the region where discrete representations have an associated frame, while the left-hand side represents the area where this is longer the case (loss of input/output information). As predicted by the theory. CNN and FNO are not representation equivalent, while CNO error-free in the equivalence region, but failing do so outside of it.
discretization invariance [3] condition only ensures that in the infinite resolution limit, the neural operator converges, while not making any predictions about the behavior of the neural operator at finite resolutions.
Minimizing representation equivalence error. Instead of directly tackling aliasing errors from theoretical perspective as is done in the previous sections, we select an architecture that may have aliasing, i.e. FNO, and try to minimize the representation equivalence enror during training. We observe and detail our findings in SM
5.2 Assessing Structure Preservation
Representation Equivalence Error (%)
10.38
20.24
42.97
98.55 104.44
Translation Equivariance Error (%)
10.54
20.54
45.79
Table Representation equivalence error vs translation equivariance error. observe a direct link between aliasing and preserving continuous structures.
Our methodology aims to maintain the structure the underlying operator, as defined at the contin- level. Fundamental structural characteristics of the underlying operator, like symmetries and conservation principles, are maintained at the discrete level. Discretizations that are not aligned may fail uphold these intrinsic properties of the operator, resulting in deviations like symmetry breaking which can negatively influence the results.
As example of this, consider FNO. Similarly what is done in [11. the primary structure we aim to uphold is translation equivariance, as defined at the operator level. We look at different trained FNOs all of which have different representation equivalence error and assess whether they are translation equivariant or not. More specifically, these FNOs are trained by minimizing both the regression loss and the representation equivalence error simultaneously, introducing more less large multiplier À on the latter, just like SM
As we vary the parameter X, we observe in Table 1 there is distinct relationship between the Discrete Aliasing Error (DAE) and the Translation Equivariance Error. Both errors show similar trend with nearly perfect positive linear association.

---

6 Discussion
Summary. Although a variety of architectures for operator learning are available since the last couple of years, there is still a lack of clarity about what exactly constitutes operator learning. Everyone would agree that an operator learning architecture should have functions as inputs and outputs. However, in practice, one does not necessarily have access to functions, either at the input or output level. Rather the access limited to some form of discrete representations of the underlying functions, for instance, point values, cell averages, coefficients with respect to basis etc. Moreover, one can only perform computations with discrete objects on digital computers. Hence, given the necessity of having to work with discrete representations of functions, it is essential to enforce some relationship between the continuous and discrete representations of the underlying functions or in other words, demand consistency in function space.
Unlike in 15 where the authors advanced a form of asymptotic consistency in terms of discretization parameters, we go further to require a stronger form of structure prreeerinn equivalence of the continuous and discrete representations of operators. To this end, we leverage the equivalence between continuous and discrete representations of functions in Hilbert spaces in terms of frame theory. The main point about this equivalence is the fact that functions, belonging to suitable function spaces, can be uniquely and stably reconstructed, from their frame coefficients. A failure to enforce this equivalence results in the so-called aliasing errors that quantitatively measure function space inconsistencies.
We extend this notion of aliasing error to operators here and use it to define Representation equivalent Neural Operators (ReNOs), see Definition 3.4 Our framework automatically implies consistency in function spaces and provides a recipe for deriving ReNOs in terms of changing the underlying frames. We also employ our framework to analyze whether or not existing operator learning architectures are ReNOs and also corroborate our results through experiments.
Related Work. Our current paper relies heavily on structure preservation from classical numerical analysis, 22 and references therein, as well as concepts from signal processing and applied harmonic analysis 25 and references therein. Among the emerging literature on operator learning, [15] was
one the first papers to attempt a unifying framework that encompasses many operator learning architectures and codify them through particular definition of neural operators. We take analogous route here but with our main point of departure from [5] being that unlike their notion asymptotic consistency, require systematic consistency in function space by enforcing the representation cquivalence for the underlying operator at each layer. Another relevant work is [10] where the authors flag the issue of possible aliasing errors with specific operator learning architectures. We significantly expand on the approach of [IO] by providing a rigorous and very general definition for aliasing errors. Finally, errors for operators, is analogous to similar approach for computing with functions, rather than discrete representations, which is the cornerstone of the Chebfun project [8] and references therein.
definition of Representation equivalent Neural Operators, relying aliasing
Limitations and Extensions. Our aim in this paper was to tackle a fundamental question of what defines neural operator? We have addressed this question here and shown that enforcing some form of equivalence between continuous and discrete representations needed for the architecture to genuinely learn the underlying operator rather than just discrete representation of it. What we have not addressed here are quantitative measures of the error associated with a Representation equivalent Neural Operator, introduced in Definition 3.4, in approximating the underlying operator. This is a much broader question than what is addressed here, since sources of error, other than aliasing errors, such as approximation, training and generalization errors also contribute the total error (see
[16 14 15]).
One interesting direction for further analysis would involve exploring operators and their discretized counterparts that form c-ReNOs and also enforce the ReNO property approximately. Characterizing the implications of this concept within operator learning is topic for future work.
10

---

References
[1] G. Alberti, M. Santacesaria, and S. Sciutto. Continuous generative neural networks. arXiv preprint arXiv:2205.14627, 2022.
[2] J. Benedetto. Irregular sampling and frames. Wavelets: A Tutorial in Theory and Applications, 2:445-500 1992.
[3] M. Bertero, P. Bocacci, and C. De Mol. Introduction to inverse problems in imaging. CRC press, 2021.
[4] K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021.
[5] Cao. Choose transformer: Fourier or galerkin. In 35th conference on neural information processing systems, 2021.
[6] T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917, 1995.
[7] O. Christensen. Frames and bases: An introductory course. Springer Science & Business Media, 2008.
[8] T. A. Driscoll, N. Hale, and L. N. Trefethen. Chebfun Guide. Pafnuty Publishers, Oxford, 2014.
[9] L. C. Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
[10] V. Fanaskov and I. Oseledets. Spectral neural operators. arXiv preprint arXiv:2205.1057.
2022.
[11] N. Gruver, M. A. Finzi, M. Goldblum, and A. G. Wilson. The lie derivative for measuring learned equivariance. In The Eleventh International Conference on Learning Representations, 2023.
[12] V. Isakov. Inverse Problems for Partial Differential Equations. Springer, 2017.
[13] G. Kissas, J. H. Seidman, L. F. Guilhoto, V. M. Preciado, G. J. Pappas, and P. Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1- 63, 2022.
[14] N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for Fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021.
[15] N. Kovachki, Li, Liu, K. Azizzadenesheli, K. Bhattacharya, A. M. Stuart, and A. Anand- kumar. Neural operator: Learning maps between function spaces. CoRR, abs/2108. 0.8881 2021.
[16] S. Lanthaler, S. Mishra, and G. E. Kariaadaki.. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1 6(1):nac0011, 2022.
[17] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anand- kumar. Fourier neural operator for parametric partial differential equations. In International Conference Learning Representations, 2021.
[18] 7. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anand- kumar. Neural operator: Graph kernel network for partial differential equations. CoRR, abs/2003.03453
2020.
[19] Z. Li, N. B. Kovachki, K. Azizzadeneshelii B. Liu, A. M. Stuart, K. Bhattacharya, and A. Anand- kumar. Multipole graph neural operator for parametric partial differential equations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6755-6766. Curran Associates, Inc., 2020.

---

[20] Lu, P. Jin. G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 333):218-22, 2021.
[21] M. Prasthofer, T. De Ryck, and S. Mishra. Variable input deep operator networks, arXiv preprint arXiv:2205.11404, 2022.
[22] A. Quarteroni and A. Valli. Numerical approximation of Partial differential equations, vol- ume 23. Springer, 1994.
[23] B. Raonić, R. Molinaro, T. De Ryck, T. Rohner, Bartolucci, R. Alaifari, S. Mishra, and E. de Bézenac. Convolutional neural operators for robust and accurate learning of PDEs. To appear in 37th conference on neural information processing systems, http://aw...rrr//bb2222020011 2023.
[24] J. H. Seidman, G. Kissas, P. Perdikaris, and G. J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. arXiv preprint arXiv:2206. 03551, 2022.
[25] M. Unser. Sampling-50 years after Shannon. Proceedings of the IEEE, 88(4):569-55) 2000.
[26] M. Vetterli, J. Kovacevic, and V. Goyal. Foundations of Signal Processing. Cambridge University Press, 2014.
12


Reviews:
[{"review_id": "myeIhmkIIG", "reviewer": "Reviewer_nALh", "summary": "The concept of “representation equivalence” is introduced in the context of operator learning. The definition amounts to requiring zero aliasing error from the mode. It is shown whether several popular architectures satisfy the new definition.", "questions": "I do not understand the details of the numerical example. After having read the paper and the Appendix, I feel that I am missing something. I will describe the set-up to the best of my knowledge. I will denote by f the input function and by u the output i.e. Q(f)=u. For simplicity, consider only two resolutions and denote by x_1,...,x_m the first discretization and by y_1,...,y_n the second with the assumption that they are nested so that {x_j} is a proper subset of {y_j}. Data is generated by randomly (from a Gaussian) sampling the point values f(y_j) and u(y_j). Three neural network based methods are then trained to regress on samples f(x_j) and u(x_j). They are all then tested on fitting u(y_j) from f(y_j) (I assume with samples from a test set). It is shown that the SNO achieves the same error on regressing u(x_j) from f(x_j) as it does on regressing u(y_j) from f(y_j) while the other two methods do not. I do not understand how this is possible. The SNO approximates Q in the form Q(f)(x) = sum_{k=-d/2}^{d/2-1} g_k ({<f, e^{i .}>, …, <f, e^{id .}>}) e^{ikx} where g is a neural network mapping C^d to C^d with d some fixed positive integer. By passing from f(x_j) to f(y_j), the input to g changes since the inner products now see f(y_j),  but the periodic bases stay the same. The values of u(y_j) were picked at random and independently from those of f(y_j) so how can this new information about f possibly have an effect on the prediction of u? In general, this does not seem possible since whenever Q transforms all modes of f whatever method that regresses Q must extrapolate this transformation to new modes. While this may happen sometimes, it cannot be guaranteed. But this looks completely impossible in the current experiment since the SNO must extrapolate randomness. There must be something in this experiment that I am missing; it would be helpful for the authors to clarify. \n\nWhile the authors do not propose the SNO, it is the only method which they show satisfies ReNO, so I do wonder about it. It is well known that, for approximation in L^2, an optimal linear subspace is given by PCA. This is the motivation behind PCA-Net [3] which is precisely like the SNO but, instead of picking Fourier bases, computes bases for the inputs and outputs by doing PCA on the data. What is the motivation behind sticking with Fourier bases? In fact, since any method which is ReNO is a linear method of approximation, is PCA-Net not the optimal ReNO method (barring optimality of the finite-dimensional neural network)? Generally, if the ReNO definition is to be adopted as a standard way of doing operator learning, how do the authors envision new architectural designs? Remark 3.5 fully characterizes ReNO architectures, so it is hard for me to see a way forward.\n\nThe first example of Section 4 shows that discrete convolution layers are not ReNO, yet the work https://arxiv.org/pdf/2302.01178.pdf (CNO, Proposition 2.1) claims that they are. As far as I can tell, the way that ReNO is proved for CNO is by showing that discrete convolution maps w-bandlimited functions to w-bandlimited functions and then constructing a pointwise nonlinearity that does the same. I am not convinced by the CNO proof since nonlinearities used in practice (like ReLU or, more generally, learned MLPs) will not map bandlimited functions to bandlimited functions of any order and, so, even going through the upsampling-nonlinearity-downsampling procedure, will not give an aliasing error that is exactly zero (but it will mitigate the aliasing error as was first proposed in https://arxiv.org/pdf/2106.12423.pdf). Putting nonlinearities aside, however, one still needs that the discrete convolution is ReNO, so where do the differences lie? \n\nHow do the authors envision a similar theory for Banach spaces when the $\\ell_2$ isomorphism breaks down?", "limitations": "The authors adequately address potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow. The math is made accessible and good intuition is given. Claims are proven rigorously with detailed proofs.", "weaknesses": "While I like the general direction of this paper, I do not appreciate catchy titles that obfuscate mathematical ideas for the purposes of a “wow” factor. Are Neural Operators Really Neural Operators? Yes they are because that’s how they were defined in [14]. Putting titles aside, I think that two ideas about what should constitute “operator learning” are being mixed. The point of [14] is to define families of architectures whose parameters are not tied to the discretization of the input or output functions. This makes the cost of a model (with cost defined as the size of the model) independent of discretization. The analogy is to explicit vs. implicit methods for solving time-dependent PDEs. For an explicit method the cost (defined here as the inverse size of the time step) increases as the spatial discretization increases due to CFL. On the other hand, for implicit methods, the time step and the spatial discretization are independent. From this point of view, neural operators are analogous to implicit methods while, say CNNs, are analogous to explicit methods. This makes CNNs a perfectly reasonable method for operator learning, however, the challenge which remains is how does one increase the size of the CNN with the resolution in order to guarantee consistent results. This can then map into a proper cost-accuracy trade-off analysis between methods just as one can do for explicit vs implicit methods. Note that for this to be done, architectures need to be re-trained at each testing resolution. Indeed what is shown in [14] is if one re-trains the same model with higher and higher resolutions, the answer remains consistent. A consequence of the neural operator definition (parameters not tied to grid points) is that a trained architecture can be used to predict at multiple resolutions, however, one cannot, in general, have guarantees that the answer will stay consistent. Indeed if the true operator transforms all modes of an input function in some way then how can one guarantee that a surrogate trained only on some of the modes will generalize to unseen modes? One can hope that the surrogate extrapolates correctly and that has empirically been shown on some problems (some in [14] but also many others), but it cannot be guaranteed any more than generalization abilities of deep neural networks can be guaranteed. From this point of view, neural operators do precisely what they are meant to do, \nand I strongly think that the authors should be much more clear and explicit about the problem that their work tries to address because it is ultimately a different one. \n\nI like the idea of quantifying and mitigating the aliasing error. It is a very important direction that has been almost unexplored in deep learning with many practical consequences. However, I do think that Definition 3.4 is much too strong to be useful. Indeed, in Remark 3.5, the authors show that, as a consequence of this definition, there is only one recipe for constructing ReNO architectures, mainly eq. (3.6). Note the similarity of this equation to eq. (2.3), upon combining with eq. (2.2), in https://arxiv.org/pdf/2005.03180.pdf. I point this out because constructions of this kind yield only linear methods of approximation. It is well known in function approximation that linear methods are suboptimal (for example, https://people.math.sc.edu/devore/publications/NLACTA.pdf). While this has certainly not been studied nearly as deeply in the operator setting, similar empirical and theoretical results are beginning to emerge (for example, https://arxiv.org/pdf/2210.01074.pdf). I therefore struggle to see the practicality of such a strong definition. For what kinds of problems would this be useful?\n\nLastly, I think the numerical experiments are quite insufficient in demonstrating the practical usability of ReNO. Is aliasing error actually prevalent or is the main source of error, the approximation error? If the latter, should architecture design not be focused on reducing approximation as opposed to aliasing error? What is the trade-off between approximation and aliasing, and can an architecture which is not ReNO, still be modified to reduce aliasing error while retaining the same approximation power?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and remarks, which we address below:\n\n1. We strongly acknowledge the significance of neural operators, as defined in [14] as a concept, and this is exactly why we intend to draw inspiration from it and further develop upon it. Although the reviewer may have interpreted our approach as a challenge to the Neural Operator (NO) concept, this was never our intention. To the contrary, our research aims to advance better understanding of various aspects of NOs. For this reason and to avoid any possible misunderstanding by intended readers, we propose to change the title to a more informative \"Representation equivalent Neural Operators: Frame Theory Meets Operator Learning\".\n2. With this context, one of our goals was to analyze under what conditions do NOs actually correspond to the mathematical notion of operators. Indeed, NOs satisfying just the discretization invariance property may not necessarily correspond to operators, leading to possible negative implications such as loss of *structures* like symmetries and locality, only defined at the continuous level, leading to possibly poor generalization as well as inconsistencies across resolutions. Our aim here was to develop a theoretical framework that effectively addresses these aspects. We demonstrate that when seeking to establish a relationship between continuous and discrete or between different resolutions, the concept of aliasing is the crux and as acknowledged by the reviewer, one key and highly non-trivial contribution of our paper was a rigorous definition of aliasing errors for operators (Eqn (3.1)) which is the first instance of this formula in the literature. Moreover, we believe that aliasing complements the consistency errors for NOs in [14] very well. We will highlight this contribution more prominently in the CRV, if accepted.\n3. Regarding the *rigidity and utility of Def 3.4*, as the reviewer suggests, it automatically leads to the possibility of not necessarily setting the aliasing error to zero, but to control it and make it as small as desired. This is a natural corollary of def 3.4 and we will discuss it at greater length in the CRV (see also pt below). At the same time, we politely disagree with the reviewer's contention that Def3.4 (Rem 3.5) is too restrictive. It leaves the room to choose frames at each layer, as well as the neural net model itself, i.e., nonlinear layer $G_\\ell$. Choosing the frames $\\Psi_\\ell$ and $\\Psi\\_{\\ell+1}$ for the input and output spaces, only specifies how the corresponding discrete operator $g_\\ell(\\Psi_\\ell,\\Psi\\_{\\ell+1})$ has to be chosen. This just states that moving from continuous to discrete representations has to happen in a precise and controlled manner, but does not specify the architecture. The neural operators $G_\\ell$ are to be learned and are not fixed by the ReNO framework and there is considerable room for innovative architectural designs here. \n4. Regarding *linear vs. nonlinear models in operator learning*, we start with the contention that frames based on multiresolution, such as wavelets, allow in fact to go beyond linear approximation of signals to *nonlinear approximation* (in the sense of DeVore, Acta Num 1998). With such frames, one can extract signal-adapted approximations that can zoom in where necessary, instead of using a uniform discretization. Explicitly implementing such constructions are topics for future research, but our framework of considering general frames is, in our opinion, the right setup to look for such architectures. Moreover in this context, the reviewer defines the cost as model size. We believe that the overall computational complexity is more pertinent. This is a subtle but important point. In the case where model size $\\ll$ resolution of the data, the computational bottleneck is data resolution, which we have no control over, as it is given to us a-priori. In this case, using a linear method does not necessarily increase complexity, putting into question the utility of so-called *non-linear reconstruction* operators of Lanthaler et al. Clearly, more discussion on this point is necessary and will be included in a CRV, if accepted. \n5. We apologize for the possible lack of clarity in our description of the numerical experiment in Sec. 5. We have added a detailed description in the general response to all reviewers above and request the reviewer to examine it and ask us further questions, if needed.\n6. The reviewer's excellent questions about *trade-off between approximation and aliasing errors* and *can a non-ReNO NO be made to reduce aliasing* inspired us to perform the follow-up experiment described in the general response. As shown in figure 2 of the attached pdf, we were able to reduce the aliasing error of FNO significantly by minimizing the discrete aliasing error (Eqn 5.1) (Fig.2 Left) while keeping training errors small (Fig. 2 Right). In this example, there appears to be no trade-off between aliasing and approximation errors!\n7. SNO is simply one example of a ReNO and we have analyzed when DeepONet is a ReNO (SM E, lines 480-489). Following the reviewer's suggestion, we can show that PCANet is also a ReNO, see Fig 3 (attached pdf) for the commutative diagram and Fig. 4 for experimental verification. However, we also observe that the resolution invariance of PCAnet is very sensitive to slight perturbations like randomly removing even one data sample while evaluating on a different resolution (See plot for PCAnetJitter in Fig. 4). In the CRV, we will investigate this issue carefully, particularly in the context of the question about *optimality* of PCAnet.  \n8. It is indeed possible, although technical, to derive a ReNO framework in Banach spaces using Banach frames (Casazza et. al, 1999). We will add a discussion on this topic in the CRV.\n\nWe sincerely hope to have addressed your concerns satisfactorily and would kindly request the reviewer to update their assessment accordingly.\n"}, {"review_id": "MGwGfEotc7", "reviewer": "Reviewer_9wa4", "summary": "This paper investigate the aliasing error of operator learning. The work points out that many existing neural operators have aliasing error, so the error is higher on super-resolution problem. To deal with the aliasing error, the paper proposed Representation equivalent Neural Operators (ReNO), which have no aliasing error. Numerical experiments show ReNO model perform better with super-resolution task.", "questions": "The CNO work [1] describes a method that makes neural operator model using activation layers also aliasing-free. Specifically it apply upsampling and downsampling before and after band-limited activation layers. As claimed in Proposition 2.1, CNO are representation equivalent.\n\nI wonder if other works such as FNO enjoy a similar properties. Since the IFFT is equivalent to upsampling and FFT is equivalent to downsampling. FNO can restrict the number of Fourier modes of its representation space too if equipped with a band-limited activation function. On the Fourier space FNO truncates the latent function to N Fourier modes, and the band-limited activation function introduce at most M new Fourier modes, so the highest number of Fourier modes is N*M. If N*M < the resolution, there will be no aliasing error.\n\nIf it is the case, does FNO also belong to ReNO?\n\n\n[1] Raonić, Bogdan, et al. \"Convolutional Neural Operators.\" arXiv preprint arXiv:2302.01178 (2023).", "limitations": "If will be better to discuss the relationship between the aliasing error and approximation power.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper investigates a very interesting problem -- the concept of resolution invariance in operator learning. It introduces frame theory to study the aliasing error in operator learning and it proposes a neural class of model called Representation equivalent Neural Operators (ReNO). The paper helps the community to better understand the key concept of neural operator. ", "weaknesses": "I think it's very important to distinguish the two different concepts:\n\n### Discretization invariance vs representation equivalence\n- Previous work define neural operator as asymptotic discretization invariant model that converge as the grid refines [1]. In a sense that the neural operator model will converge to the underlying solution operator in continuum. In the case of Fourier representation, discretization invariant FNO model has the capacity to learning infinite number of Fourier basis as created by the non-linear activation layer, which in the end, will cause the aliasing error.\n\n- In this work, the authors the representation equivalent model that has no aliasing error. It is equivalent to say these models are prescribed in a fixed linear subspace of the underlying infinite-dimensinal function space. In the case of Fourier representation, the representation equivalent SNO model has a fixed number of Fourier basis, and a consequential irreducible approximation error.\n\nWhile the newly proposed representation equivalent models have no aliasing error, they have a limited expressive power, which prevents the model extrapolates to unseen models and higher frequencies. Since the ReNO models cannot use non-linear activation layers after the inverse spectral transform, the output space is restricted to the spectral space, and the model is of the class of linear reconstruction operator [2], which may require exponentially more number of parameter compared to non-linear reconstruction operator like FNO.\n\nIn practice, the ReNO models probably perform better at super-resolution, while previous asymptotic discretization invariant models perform better at fixed resolution or mixed resolution training.\n\n### Writing\nBesides, the overall writing is very clear. The reviewer kindly suggests to use a more informative title.\n\n\n[1] Kovachki, Nikola, et al. \"Neural operator: Learning maps between function spaces with applications to PDEs.\" Journal of Machine Learning Research 24.89 (2023): 1-97.\n[2] Lanthaler, Samuel, et al. \"Nonlinear reconstruction for operator learning of pdes with discontinuities.\" arXiv preprint arXiv:2210.01074 (2022).\n\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for their appreciation of the merits of our paper and their welcome suggestions to improve it. We address their detailed concerns below.\n\n1. Following the reviewer's welcome suggestion, we propose to change the title to the hopefully more informative *Representation equivalent Neural Operators: Frame Theory Meets Operator Learning*.\n\n2. On the reviewer's question about *link between discretization invariance (DI) and representation equivalence (RE)*, we start by pointing out that these concepts are indeed different and to some extent, complementary. This difference is precisely what motivated the current paper. We believe that, in practice, the main utility of the  notion of representation equivalence is that it precisely describes what is happening when changing resolutions, when compared to discretization invariance which corresponds to a notion of asymptotic consistency in the infinite resolution limit. Using the notion of RE, we can interpret varying resolution plots, such as those in Kovachki et al or even in Fig. 4 of our paper, while the asymptotic consistency requirements of DI do not explain why the observations of these figures hold true as one cannot rule out large errors between resolutions when consistency is only required in the infinite resolution limit. Given this context, we would provide a detailed discussion of DI vs. RE, emphasizing their complementary roles in a CRV, if accepted. \n\n3. Regarding the reviewer's questions about *linear vs non-linear methods in the context of operator learning*, we would like to start by clarifying that the results of Lanthaler et. al. were specific to the case of scalar 1-D transport of a single discontinuity and a related 1-D Burgers example where they claimed that FNO is better than DeepONet as the approximation error of a linear reconstruction operator can be bounded below by the rate of decay of eigenvalues that may converge slowly for some transport equations. This does not imply that methods with a linear reconstruction cannot be efficient or have limited expressive power for a very large class of PDEs, say elliptic and parabolic PDEs, which have a regularizing effect resulting in fast spectral decay. Moreover, the result of  Lanthaler et. al. only pertains to size efficiency vis a vis approximation error. In the case where model size $\\ll$ resolution of the data, the computational bottleneck is data resolution, which we have no control over, as it is given to us a-priori. In this case, using a linear method does not necessarily increase complexity as the data resolution dominates. Moreover, other sources of error, such as aliasing error, discussed extensively in our paper can also affect the performance. Finally, as the reviewer has pointed out the work of Raonic et. al. on CNO -- Table 1 in Raonic et. al. shows that a ReNO like CNO can outperform FNO even on benchmarks with discontinuities (Transport Eqn.) and Shocks (Compressible Euler Equations), attesting to the abilities of methods based on linear reconstruction for approximating operators with slow spectral decay.\n\n4. In reply to the reviewer's excellent question, it is indeed possible to modify the FNO architecture in a similar spirit as it is done in CNO of Raonic et al. , to respect the continuous-discrete equivalence. We will mention this fact in the CRV, if accepted. Here, we have explored another option for making FNO into a ReNO, which we have described in some detail in our general response to all reviewers above. Our approach consists of training FNO to simultaneously minimize the regression error as well as the discrete aliasing error (Eqn. 5.1 in our paper). The results in Figure 2 of the attached pdf show that the aliasing error is indeed minimized by this approach while the training error remains as small. This experiment shows that working directly to minimize the aliasing error is also a viable approach for making a neural operator into a ReNO.\n\n5. Regarding your comment about *link between approximation and aliasing errors*, we would like to point out that there is no intuition to believe that there is a possible link (or even trade-off) between aliasing error and approximation power. Moreover, the experiment on FNO (see pt. 4 above) reported in Figure 2 in the uploaded pdf for this rebuttal further strengthens the argument that such a link does not exist. It is rather the case that aliasing error is an artifact that can be eliminated without necessarily losing approximation power. Following your suggestion, we will include a detailed discussion of this issue in the CRV, if accepted.\n\nWe sincerely hope to have addressed your concerns, particularly about the differences between discretization invariance and representation equivalence, satisfactorily and would kindly request the reviewer to update their assessment accordingly.\n\n"}, {"review_id": "776X464rDk", "reviewer": "Reviewer_3pPU", "summary": "In this paper, the authors investigate the concept of neural operators in the context of operator learning architectures. They address the fundamental question of what defines a neural operator and propose the notion of Representation equivalent Neural Operators (ReNOs) that satisfy a systematic consistency between continuous and discrete representations, referred to as continuous-discrete equivalence (CDE).\n\nThe authors present a framework for analyzing existing operator learning architectures and determining whether they qualify as ReNOs. They emphasize the importance of enforcing CDE at each layer of the approximating operator to ensure genuine learning of the underlying operator, rather than just a discrete representation. Experimental analysis is conducted to validate the theoretical claims and demonstrate the practical implications of respecting representation equivalence.", "questions": "1. Is SNO only suitable for periodic signals? If one uses a chebyshev basis instead of fourier basis to implement SNO, then does the analysis in the paper still hold?", "limitations": "The authors provide a through and insightful discussion about limitations and extensions in Section 6.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Originality: The paper introduces a novel perspective on neural operators in the context of operator learning architectures by focusing on the concept of continuous-discrete equivalence. The notion of Representation equivalent Neural Operators (ReNOs) provides a fresh and rigorous definition that emphasizes the importance of maintaining consistency between continuous and discrete representations. This original approach extends the understanding of neural operators and adds a new dimension to the analysis of operator learning architectures.\n\n2. Quality and Clarity:  the framework is well-structured, and the definitions and mathematical formulations seem correct and rigorous.\n\n3. Significance: The paper holds significant importance for the theoretical analysis of operator learning architectures.", "weaknesses": "1. Experimental Validation: Although the paper includes experimental analysis to support the theoretical claims, there is room for further elaboration and validation, e.g. incorporating more SOTA architectures and more benchmark datasets.  \n2. Practical Significance: Lack of a principled architecture instance or design. The paper does not propose a new architecture satisfying the ReNO definition, and does not provide a systematic way to design new instances of ReNO.\n3. Readability: Lack of intuition and hints between mathematical definition and derivation, especially in section 3.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for their appreciation of the merits of our paper and their welcome suggestions to improve it. We address their detailed concerns below.\n\n1. Regarding the reviewer's concerns about *experimental limitations, lack of SOTA architectures, benchmarks*, we start by saying that the main point of our paper was to provide a novel theoretical framework of representation equivalence, based on a non-trivial and novel definition of aliasing errors for general operators. In our view, this is the crucial contribution of this paper and we apply it to analyze whether and when existing neural operator architectures are ReNOs. In the paper, we have already analyzed CNN, FNO (not ReNOs in general) and SNO, DeepONet (in SM Sec. E) which are ReNOs. Following your and other reviewer's suggestions, we have also analyzed the PCA-net architecture of Bhattacharya et al as a ReNO (see general response to all reviewers as well as the attached 1-page pdf). We would also like to mention that recently, CNO (Arxiv:2302.01178) was also shown to be a ReNO. The authors of CNO show that this ReNO is state-of-the-art on learning operators corresponding to a variety of PDEs, already demonstrating the practical utility of the concept of ReNO. We will add further discussion about these architectures in a CRV, if accepted. Moreover, we have reported additional experiments in the 1-page pdf and described them in the general response to all reviewers. In particular, we show how FNO can be made into a ReNO (approximately) by minimizing the differentiable aliasing error (Eqn 5.1 of the main paper). \n\n2. Regarding the reviewer's comment about a *systematic way to design ReNOs*, in addition to the reply in pt 1, we would like to emphasize that one of our key contributions is Eqn (3.6), which specifies how ReNOs act under a change of frame sequences. This is novel and allows for a principled way in which resolution can be varied for a ReNO and it has been utilized in the design/evaluation of CNO for instance. Moreover, we have added a possible algorithm on how non-ReNO can be made into a ReNO by minimizing aliasing error, without any significant loss of expressive power -- please check the general response to all reviewers as well Figure 2 of the attached 1-page pdf. \n\n3. Following the reviewer's excellent suggestion,  we will include more intuition and hints regarding the mathematical aspects of Sections 2 and 3 in the CRV, if accepted. As an example, we plan to add a cartoon depiction of the ReNO framework  that we have created in Figure 1 in the uploaded pdf for this rebuttal. Hopefully, this will make our underlying concepts clearer to a reader. \n\n4. The work on SNO (Fanaskov et al) indeed already considers both choices of basis functions: Chebyshev and trigonometric polynomials. Thus, SNO is a ReNO with respect to a Chebyshev basis too. \n\nWe sincerely hope to have addressed your concerns, particularly about the practical utility and realization of ReNOs, satisfactorily and would kindly request the reviewer to update their assessment accordingly."}, {"review_id": "VVJh4v3HqX", "reviewer": "Reviewer_2w9i", "summary": "Many recent studies have emerged in the field of operator learning. However, many models attempt to learn operators using the discretized values of functions rather than sending functions as operators to other functions. In this paper, the authors interpret the relationship between infinite-dimensional functions and their discretized values using the framework of frame theory. The authors analyze various operator learning models to assess whether they effectively capture this relationship in an equivalent manner, and they propose a framework called ReNO for this purpose.", "questions": "* Is there a specific reason why not all the various models mentioned in line 30 were examined under ReNO? Why was the analysis limited to only CNN, FNO, and SNO?\n\n* What are the definitions of $f(nT)$ and $f(n/2\\Omega)$ mentioned in line 70? Additionally, why do the sinc functions in line 75 form an orthonormal basis? Understanding these concepts seems crucial to grasp the difference between Definition 2.1's $f$ and $P_B f$.\n\n* I'm curious to understand why the adjoint $T^*$ in line 93 is well-defined uniquely for all $f_i$. Furthermore, what makes the frame operator $S$ invertible and well-defined?\n\n* In line 102, when referring to \"point samples of the underlying function $f$,\" which specific $f_i$ does this pertain to? (Similarly, for line 221, when does $T^\\dagger_\\Psi$ precisely become a discretization representation given $\\Psi$?)\n\n* I'm interested in understanding how Definitions 2.1 and 2.2 naturally lead to the discussion of aliasing error in Definition 3.1. Explaining the differences and similarities between these concepts would be helpful.\n\n* The diagrams between lines 156 and 157, the diagram mentioned below line 265, and the captions and explanations for Figures 1, 2, and 3 are significantly lacking. They need to be added with essential details. What are the differences between the three figures, and how are they interconnected?\n\n* What is the relevance of Equation 4.3 in the analysis of SNO, considering the diagram? While A and B were assumed to be zero in FNO, does this equation account for the bias term b?\n\n* DeepONet is known as a successful operator learning model mentioned in line 271. What does it mean that DeepONet is also equivalent to SNO? Further elaboration is needed.\n\n* In Section 5, it would be beneficial to summarize the experimental setups (Line 506 F.2) , even briefly in the main text, rather than relegating them to the appendix. It would be helpful to explain what type of regression was performed, whether it involved the solution operator of PDE or not, or if it was a different type of experiment.", "limitations": "* The explanation of frame theory is challenging to understand, and the extension of frame theory to operators is equally difficult to grasp. There are intricate aspects in the theoretical part that make it hard to follow, such as meticulous definitions of notations, the reasons for orthogonality, the existence of inverses, whether $T^*$ is necessary for operator extension, and the well-definedness of $T^\\dagger$. It would be beneficial to summarize the essential concepts needed to extend these aspects to operators and provide a slightly more accessible explanation. Ultimately, it would be helpful to clearly articulate why frame theory is crucial for explaining operators, rather than presenting it as a separate theory.\n\n* There seems to be a lack of experimental evidence supporting the theoretical claims. In Section 5, which focuses on experiments and analysis related to resolution, additional experiments that better illustrate the theory could enhance the persuasiveness of the need for this theory.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Investigating whether models that engage in operator learning, a mapping process between infinite-dimensional function spaces, effectively handle and address this infinite dimensionality is a pivotal and highly relevant topic within the field. The endeavor to analyze and comprehend this specific aspect of operator learning appears to be a novel and significant contribution. Leveraging the definitions associated with frame theory, as established in Hilbert spaces, and extending their application to operators, enables a profound exploration of the extent to which various operator models align with the ReNO framework. This comprehensive analysis sheds light on the capacity of these models to capture and interpret the intricate relationship between infinite-dimensional functions and their discretized counterparts. Such insights provide valuable guidance for the advancement of operator learning methodologies and contribute to the broader understanding of this field.", "weaknesses": "The explanations of frame theory discussed in Sections 2 and 3, as well as the summaries of applying frame theory to operators, are written in a manner that is not easily comprehensible. Many readers, including myself, who may lack a strong background in signal processing and functional analysis, find these sections challenging to understand (See Questions). It would be beneficial to provide clearer explanations and examples regarding aspects such as the invertibility of the frame operator S and the well-definedness of the pseudo-inverse, clarifying their significance.\n\nFurthermore, in Section 4, which aims to verify whether the defined operator learning models conform to ReNO, it is intuitive to consider that CNN and FNO may not fall under the purview of ReNO, as they simply discretize functions into image-like forms. On the other hand, SNO, which predefines a basis and solely learns the mapping between the coefficients, appears to naturally align with ReNO. It would be intriguing to explore additional insights derived from these theories based on this intuition. Additionally, there is a well-known operator learning model called DeepONet, which could be described in more detail. While it is briefly mentioned on line 271 that DeepONet also belongs to SNO, further elaboration is necessary to understand how and what differentiates it from other models. Is there a specific reason for focusing solely on CNN, FNO, and SNO among the numerous operator learning models?\n\nMoreover, in Section 5, the experiments conducted, particularly those related to super-resolution discussed in the FNO paper, and the results presented in Figure 4, raise questions regarding whether they truly represent the best experiments to validate the aforementioned theories. Specifically, what are the specific definitions of the frames $\\Phi$ and $\\Psi$ used in the experiment where the resolution is altered in the predictions (as explained in Section 4 of the FNO paper)? It is unclear whether these definitions are explicitly provided or not. Are there alternative approaches, apart from altering resolution, to demonstrate the equivalence of arbitrary frames $\\Phi$ and $\\Psi\"? It would be valuable to explore additional experimental setups that further support the underlying theories, as relying solely on the resolution experiment appears inadequate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for their welcome suggestions to improve our paper. We address their detailed concerns below.\n\n1. We apologize for the lack of clarity in the explanation of frame theory. To address this shortcoming and following the excellent suggestion of the reviewer, we suggest to add a new section  *Frame Theory* in the SM dedicated to the mathematical background on frame theory, including examples of frames for some classical Hilbert spaces and explanations of properties such as the well-definedness and invertibility of the frame operator and the well-definedness of the pseudoinverse. We note that the frame operator $S$ is invertible because it is bounded and one can show that the frame property implies $\\\\|Id-S\\\\|<1$.\n2. To further improve on our exposition, we will add a cartoon depiction of the ReNO framework as a visual guide in the CRV, if accepted. We have created such a scheme in Fig 1 in the uploaded pdf for this rebuttal.\n3. We agree that DeepONet is an important operator learning architecture. It was already discussed and highlighted in Appendix E, lines 480--496.\n4. Regarding the reviewer’s concern about lack of experimental details, we kindly refer to our answer in the general response to all reviewers. With this we hope to clarify the questions about the experiments and we would add such an explanation in Section 5 of a\nCRV. As for experiments with other frames, where coefficients do not correspond to point samples: we thank the reviewer for this useful feedback. Preliminary experiments have confirmed that similar results hold in other settings such as wavelet frames and we would add such experiments in the appendix of a CRV, if accepted. \n5. Fig 2 in the uploaded pdf for this rebuttal reports on an additional experiment we have conducted. It highlights that aliasing error can in fact be eliminated without compromising the approximation power of the network. This further supports the necessity of constructing architectures that respect the representation equivalence. Moreover, we have also explored PCANet of Bhattacharya et al and CNO of Raonic et al arXiv:2302.01178 as additional ReNO architectures (please see General Response to all reviewers and Figs 3 and 4 of attached pdf)\n6. While our experiments do not involve the solution of a PDE, CNO, which is a ReNO, is demonstrated to be the SOTA architecture for a wide range of operators for benchmark PDEs.\n7. The expressions $f(nT)$ and $f(n/2\\Omega)$ denote evaluation of the function $f$ at the specified points. For example, if $\\Omega=1$ and $T=1/2$, then the expressions become $f(nT)=f(n/2\\Omega)=f(n/2)$ for $n\\in \\mathbb{Z},$ resulting in the sequence $(\\dots,f(-5/2),f(-3/2),f(-1/2),f(1/2),f(3/2),\\dots)$.\n8. Intuition behind why the $\\mbox{sinc}$ functions form a basis for bandlimited functions: Bandlimited functions have Fourier transforms in $L^2([-\\Omega,\\Omega])$. Consequently, these Fourier transforms admit a Fourier series expansion. The appearing complex exponentials in the Fourier series are related to the translated $\\mbox{sinc}$ functions via the Fourier transform.  For a rigorous proof, we refer to the WSK sampling theorem Unser Proc. IEEE, 2000.\n9. The adjoint $T^*$ is defined as the bounded linear operator such that \n$\\langle h,T(\\\\{c_i\\\\}\\_{i \\in I}) \\rangle_{\\mathcal{H}} = \\langle T^* h,  \\\\{c_i\\\\}\\_{i \\in I} \\rangle_{\\ell^2(I)},$\nfor all $h \\in \\mathcal{H}$ and all $\\\\{c_i\\\\}_{i \\in I} \\in \\ell^2(I)$. The fact that $T^*$ takes the form in line 93 follows from Riesz' representation theorem (see Christensen 2008, Lemma 3.1.1). \n10. In line 102, the general frame coefficients of a function $f$ for a frame $\\\\{f_i\\\\}_{i \\in I}$ refer to the expressions $\\langle f, f_i \\rangle$. In the specific case of $\\\\{f_i\\\\}\\_{i \\in I}$ the orthonormal basis of sinc functions, the frame coefficients become point evaluations of $f$, i.e. $\\langle f, f_i \\rangle = f(i/2\\Omega)$ and $I=\\mathbb{Z}$ (cf. also Eqn. (2.2)).\n11. In line 221, since $\\Psi$ is a frame for Dom $U$ (cf. Eqn (3.2)) and $u_i \\in$ Dom $U$, we obtain that $T_\\Psi^\\dagger u_i$ is a discrete representation of $u_i$. This means that $u_i$ is obtained by applying $T_\\Psi$ to $T_\\Psi^\\dagger u_i$, i.e. $u_i = T_\\Psi T_\\Psi^\\dagger u_i$ (cf. Eqn (2.4)).\n12. Definition 2.2 describes the continuous-discrete equivalence between functions in a separable Hilbert space $\\mathcal{H}$ and their discrete representation given by frame coefficients w.r.t. a frame sequence $\\Psi$ for $\\mathcal{H}$. The aliasing error function $\\epsilon(f)$ defined therein quantifies the loss of information in representing $f$ by its frame coefficients and can be equivalently expressed as $\\epsilon(f)=\\operatorname{Id}(f)-T_\\Psi\\circ \\operatorname{id}\\circ T_{\\Psi}^\\dagger(f)$ (cfr. formula in Def. 2.2), where $\\operatorname{Id}$ and $ \\operatorname{id}$ denote resp. the identity operators of $\\mathcal{H}$ and $\\ell^2(I)$, with $I$ the index set of $\\Psi$. It therefore results natural to generalize this formula by replacing the identity operator $\\operatorname{Id}$ with an arbitrary operator between separable Hilbert spaces and $\\operatorname{id}$ by a discrete mapping, which is Definition 3.1. \n13. In a CRV, we will ensure that all figures and diagrams have captions and references within the main text with the necessary explanations.\n14. The relevant aspect of SNO in terms of the representation equivalence is not so much the specific form of the neural network in Eqn 4.3, but rather the fact that SNO first performs an analysis step by extracting the Fourier coefficients, then the neural network acts only on the coefficients and then, the resulting coefficient vector is synthesized back to a signal that is periodic and bandlimited.\n15. Yes, Eqn 4.3 accounts for a bias term.\n\nWe sincerely hope to have addressed your concerns, particularly about the mathematical details, satisfactorily and would kindly request the reviewer to update their assessment accordingly."}, {"review_id": "ZGTjXyvqPW", "reviewer": "Reviewer_ydDu", "summary": "The paper proposes a new paradigm to look at operator learning via the lens of frame theory termed as Representation Equivalent Neural Operator (ReNO). It talks about an important missing piece in the literature, the one regarding the balance of continuous nature of operators and the discrete nature of data these are trained on. Aliasing error is used as the quantifier for determining the equivalence. ", "questions": "Of all the three architectures considered in the paper, none of them perform on resolutions lower than the one used for training. Can the authors discuss this a bit?\n\nWhile the paper is written in general for frames, can the authors comment on some pros/cons of specific choice for example wavelet frames? That would be very interesting.", "limitations": "I don't see any direct potential negative social impact.\n\nAuthors have mentioned technical limitations which is very much appreciated.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is very well motivated and suitably placed in the literature.\n\nThe use of frames as a tool for representation is presented in an interesting way.\n\nThe paper address a common issue in a very mathematically rigorous manner. \n\nThe experimental evidence is convincing.\n\n\n", "weaknesses": "The overall presentation of the paper can be improved significantly. \n\nFew pointers: Figures appearing the paper have no caption, Figure 1,2 and 3 have been referred within the text, others not. In a paper like these, visual guides are very much helpful, otherwise it is easy to not follow. Additionally there is over use of inline equations which are hard to parse. If space is the issue, some details can be moved out to the appendix, for example section 2 can be compressed, although it reads well, it is somewhat standard.\n\nNotations, need to made clear around when introduced. For example: that $\\mathfrak{g}$ is dicretization of $\\mathcal{G}$ is mentioned around line 211 and introduced much earlier. This makes it harder to read through.\n\nExperimental section should have more details, the current form is not at all clear in the main paper. I see lot of blank space in that page anyway.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for their appreciation of the merits of our paper and their welcome suggestions to improve it. We address their detailed concerns below.\n\n1.  In a CRV, if accepted, we will ensure that all figures have captions and are referenced within the main text.\n\n2. The reviewer's point about *visual guides being important* is excellent and we follow this suggestion. As a visual guide, in the CRV, we will add a cartoon depiction of the ReNO framework  that we have created in Figure 1 in the uploaded pdf for this rebuttal.\n\n3. We will also reduce the number of inline equations and move parts of Section 2 to the appendix if necessary.\n\n4. The definition of $\\mathfrak{g}$ is stated in line 191, just shortly after the mentioned instance in line 211, and its definition is also a numbered equation that we refer to in line 211.\n\n5. Regarding the reviewer's concern about lack of experimental details, we kindly refer to our answer in the general response to all the reviewers. With this, we hope to clarify the questions about the experiments and we would add such explanation in a CRV.\n\n6. When testing below training resolution in our experiment, indeed, any architecture will yield poor results. The input and output space in these experiments are spaces of bandlimited periodic function of bandwidth $K$. If we then test with resolution $M<K$, we are possibly undersampling the functions: if a function has bandwidth $K$, we will need at least $2K+1$ coefficients to represent it. At resolution $M$, only $2M+1<2K+1$ point samples of the function are taken. This automatically introduces errors as it means an attempt to represent a $2K+1$-dimensional vector space by only $2M+1<2K+1$ elements. We hope that this clarifies the reviewer's question and we will make it point to add this explanation to a CRV, if accepted. \n\n7. Wavelet systems can be constructed to yield frames for $L^2$ and in the context of ReNOs, this setup would enjoy the  property that applications of nonlinear activation functions naturally remain in the space spanned by the frame. In many applications, however, the measurements are given as point evaluations rather than wavelet coefficients, so that some transformation would have to be performed initially. Also, an architecture based on a multiresolution analysis such as wavelets, curvelets, shearlets, etc. is in general more technical (e.g. separate channel for each scale). These difficulties can of course be overcome so that we believe wavelets to be an interesting family of frames for the ReNO framework that we plan to study in future work on designing ReNO architectures.\n\nWe sincerely hope to have addressed your concerns, particularly about the presentation of the paper, satisfactorily and would kindly request the reviewer to update their assessment accordingly.\n\n"}]

Author rebuttal:
At the outset, we would like to thank all five reviewers for their thorough and patient reading of our article. Their criticism and constructive suggestions will enable us to improve the quality of our article. If our paper is accepted, we will incorporate all the changes that we outline below in a camera-ready version ( CRV) of our article. As many of the reviewers had questions about the experimental analysis (Sec 5), we provide a very detailed clarification below in this response. Individual comments of each reviewer is answered in the following. We have also added a 1-page pdf to supplement our argument.

Yours sincerely,

the Authors

### Clarification of Experimental Analysis Section

We wish to learn an unknown target operator $Q$ using neural operators. In this experiment, all neural operators (CNN, FNO and SNO) take as input pointwise evaluations on a grid, and are able evaluate inputs at varying grid resolutions. The goal here is to check whether their outputs at the discrete level are consistent when varying grid resolution, and verify our theory, which--informally-- tells us the outputs are consistent if and only if the operator is a ReNO. 

**Construction of the target operator:** $Q: H \to H$ with $H$ being the space of periodic and $K=30$-bandlimited functions on $[-1,1]$. A simple way of constructing such a mapping is by sampling input and output pairs in a random fashion. How to sample a function? As we know that $\Psi_K := \\{ d_K(. - x_k)\\}\_{k=-K, ..., K}$ constitutes a frame for $H$, $d_K$ being the Dirichlet kernel of order $K$ and $x_k=\frac{k}{2K+1}$, 
 any function $f \in H$ can be written as $f(x) = \sum_{k=-K}^{K} f(x_k) d_K(x-x_k)$. Thus, the discrete representation of $f$ simply corresponds to its point-wise evaluations on a grid, i.e. $\\{f(x_k)\\}_{k=-K, \dots, K}$. Note that for simplicity we have used and will use the same frame sequences for both the  input and output space $H$. After sampling a finite number of such input-output function pairs, we wish to learn the mapping between them using our neural operators. 

**Training and Evaluation:** During training, we simply learn the neural operator $u_K: \mathbb{R}^{2K+1} \to \mathbb{R}^{2K+1}$ between frame coefficients associated to $\Psi_K$, which are the point-wise evaluations of the input-target functions that we had sampled in the previous step.  In other words, we regress to the frame coefficients of the target function, with coefficients of the input function as input to the neural operator. Once training is over, we evaluate how the different neural operators behave when dealing with changing input and output frame sequences. The frame sequences here are $\Psi_M$ for different testing frames, with associated $2M+1$ sized grids, with associated discrete operator $u_M: \mathbb{R}^{2M+1} \to \mathbb{R}^{2M+1}$. 

On the continuous level, SNO maps from $H$ into $H$. Thus, at the discrete level, $u_M$ corresponds to the following: it takes in $2M+1$ point values, synthesizes these to a function in $H$ which is then sampled on the training grid. Then, $u_K$ is applied to this input vector of length $2K+1$. Finally, the output vector is synthesized to a function in $H$ and then sampled on the evaluation grid of $2M+1$ points. On the other hand, evaluation of both CNN and FNO on different grids is straightforward. 

Having defined training and evaluation frame sequences, the resulting aliasing errors (See Eqn (5.1)) can now be computed with the discrete aliasing map $$\epsilon(u_K, u_M): \mathbb{R}^{2K+1} \to \mathbb{R}^{2K+1},  \epsilon(u_K, u_M) = u_K - T_{\Phi_K}^\dagger \circ T_{\Phi_M} \circ u_M \circ T_{\Psi_M}^\dagger \circ T_{\Psi_K}$$  

Intuitively, $T_{\Psi_M}^\dagger \circ T_{\Psi_K}: \mathbb{R}^{2K+1} \to \mathbb{R}^{2M+1}$ first transforms input data from the training $2K+1$ grid to the evaluation $2M+1$ grid, then the output is computed using the neural operator at resolution $M$, and finally $T_{\Phi_K}^\dagger \circ T_{\Phi_M}: \mathbb{R}^{2M+1} \to \mathbb{R}^{2K+1}$ transforms outputs back from evaluation grid to training grid, so that even though $M \neq K$, discrete outputs are of a different size, these can be readily compared. This is precisely the procedure that led to the aliasing error plot of Fig. 4 of the main paper. 

**Additional Experimental Follow-up**: Following the suggestions of some reviewers about adding experiments as well questions about whether FNO can be made into a ReNO, we describe a simple method to do so. We repeat the same experiment as above with FNO but we add a term to the loss function that corresponds to the discrete aliasing error $\epsilon(u_K, u_M)$, defined above. Thus, we minimize both the regression error and the aliasing error simultaneously, with a parameter $\lambda$ weighing the aliasing error. To evaluate $\epsilon(u_K, u_M)$ during training, we randomly sample $M \in [K,2K]$, evaluate the differentiable aliasing error as above and backpropagate. The results are shown in Fig. 2 of the attached pdf. They demonstrate that as $\lambda$ increases, the aliasing error is indeed minimized (Fig. 2 (Left)) and is almost negligible for $\lambda =1$, not just in the range $M \in [K,2K]$, used to evaluate aliasing error during training, but it also generalizes very well to $M > 2K$, not seen during training. Moreover, the training error (Fig. 2 (Right)) continues to be very small even when $\lambda$ is increased, showing that there is no trade-off between approximation error and aliasing error, at least for this experiment and both can be made small simultaneously for FNO.

**Are any other neural operators ReNOs ?**: Following suggestions from reviewers, we investigate PCAnet of Bhattacharya et al and were able to show it is a ReNO with the commutative diagram (Fig. 3 of attached pdf) and in Fig. 4, we also verify the ReNO property empirically in the above experiment for PCAnet and the recently introduced CNO of arXiv:2302.01178.


Meta-review: The paper proposes an interesting perspective and discussion on neural operators. I found the extensive discussions between the authors and the reviewers very interesting and these really add to the paper. I believe the discussion in the paper with the agreed changes will be pertinent for the community. Given this, I recommend the paper for acceptance, conditioned on the authors making the promised changes (updated title, additional details about the experimental setting, summary figure, background on frame theory, incorporating the discussion in the rebuttal including super-resolution), as these strongly improve the paper.
Decision: Accept (poster)