Title: Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Operator Learning

Neural Operators

PDEs

Frame theory

Sampling theory
Abstract: Recently, operator learning, or learning mappings between infinite-dimensional function spaces, has garnered significant attention, notably in relation to learning partial differential equations from data. Conceptually clear when outlined on paper, neural operators necessitate discretization in the transition to computer implementations. This step can compromise their integrity, often causing them to deviate from the underlying operators. This research offers a fresh take on neural operators with a framework Representation equivalent Neural Operators (ReNO) designed to address these issues. At its core is the concept of operator aliasing, which measures inconsistency between neural operators and their discrete representations. We explore this for widely-used operator learning techniques. Our findings detail how aliasing introduces errors when handling different discretizations and grids and loss of crucial continuous structures. More generally, this framework not only sheds light on existing challenges but, given its constructive and broad nature, also potentially offers tools for developing new neural operators.

Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning

Abstract

Recently, operator learning , or learning mappings between infinite-dimensional function spaces, has garnered significant attention, notably in relation to learning partial differential equations from data. Conceptually clear when outlined on paper, neural operators necessitate discretization in the transition to computer implementations. This step can compromise their integrity, often causing them to deviate from the underlying operators. This research offers a fresh take on neural operators with a framework Representation equivalent Neural Operators (ReNO) designed to address these issues. At its core is the concept of operator aliasing, which measures inconsistency between neural operators and their discrete representations. We explore this for widely-used operator learning techniques. Our findings detail how aliasing introduces errors when handling different discretizations and grids and loss of crucial continuous structures. More generally, this framework not only sheds light on existing challenges but, given its constructive and broad nature, also potentially offers tools for developing new neural operators.

1 Introduction

Operators are mappings between infinite-dimensional function spaces. Prominent examples are solution operators for ordinary and partial differential equations (PDEs) [9] which map function space inputs such as initial and boundary data to the function space valued PDE solution. They are also the natural mathematical framework for inverse problems, both in the context of PDEs [12] and in (medical) imaging [3], where the object of interest is the inverse operator which maps observables to the underlying material property/image that needs to be inferred or reconstructed.

Given the prohibitive cost of traditional physics based algorithms for approximating operators, particularly those associated with PDEs, increasing attention is being paid in recent years to machinelearning based operator approximation . Such techniques for learning operators from data fall under the genre of operator learning . A(by no means comprehensive) list of examples for operator learning architectures include operator networks [6], DeepONets [20], Graph neural operators [18], multipole neural operators [19], PCA-Nets [4], Fourier neural operators (FNO) [17], VIDON [21], spectral neural operators [10], LOCA [13], NOMAD [24], Continuous Generative Neural Networks [1] and transformer based operator learning models [5].

However, there is still a lack of clarity on what constitutes operator learning? Clearly, an operator learning framework, or a neural operator in the nomenclature of [15], should be able to process functions as inputs and outputs . On the other hand, functions are continuous objects and in practice,

one may not have access to functions either at the input or output level. Instead, one can only access functions and perform computations with them on digital computers through their discrete representations such as point values on a grid, cell averages or in general, coefficients of an underlying basis. Hence, in practice, neural operators have to map discrete inputs to discrete outputs. This leads to a dichotomy, i.e., neural operators are designed to process functions as inputs/outputs but only have access to their discrete representations. Consequently, at any finite resolution, possible mismatches between the discretizations and the continuous versions of neural operators can lead to inconsistencies in the underlying function spaces. These inconsistency errors can propagate through the networks and may mar the performance of these algorithms. Moreover, important structural properties of the underlying operator, such as symmetries and conservation laws, hold at the continuous level. Inconsistent discretizations may not preserve these structural properties of the operator, leading to symmetry breaking etc. with attendant adverse consequences for operator approximation.

Addressing this possible inconsistency between neural operators and their discretizations is the central point of this article. To this end, we revisit and adapt existing notions of the relationship between continuous and discrete representations in signal processing, applied harmonic analysis and numerical analysis, with respect to the questions of sampling and interpolation. Roughly speaking, we aim to find a mathematical framework in which the continuous objects (functions) can be completely (uniquely and stably) recovered from their discrete representations (point evaluations, basis coefficients, etc.) at any resolution. Consequently, working with discrete values is tantamount to accessing the underlying continuous object. This equivalence between the continuous and the discrete is leveraged to define a class of neural operators , for which the discrete input-output representations or their continuous function space realizations are equivalent . On the other hand, lack of this equivalence leads to aliasing errors , which can propagate through the neural operator layers to adversely affect operator approximation. More concretely, our contributions are

We provide a novel, very general unifying mathematical formalism to characterize a class of neural operators such that there is an equivalence between their continuous and discrete representations. These neural operators are termed as Representation equivalent Neural Operators or ReNOs. Our definition results in an automatically consistent function space formulation for ReNOs.

To define ReNOs, we provide a novel and precise quantification of the notion of aliasing error for operators . Consequently, ReNOs are neural operators with zero aliasing error.

We analyze existing operator learning architectures to find whether they are ReNOs or not.

Synthetic numerical experiments are presented to illustrate ReNOs learn operators without aliasing and to also point out the practical consequences of aliasing errors particularly, with respect to evaluations of the neural operators on different grid resolutions.

2 A short background on Frame Theory and Aliasing

We start with a short discussion, revisiting concepts on the relationship between functions and their discrete representations. This is the very essence of frame theory, widely used in signal processing and applied harmonic analysis. We refer to SM D for a detailed introduction to frame theory.

Equivalence between functions and their point samples. For simplicity of exposition, we start with univariate functions f ∈ L 2 ( R ) and the following question: Can a function be uniquely and stably recovered from its values, sampled from equispaced grid points { f ( nT ) } n ∈ Z ? The classical Whittaker-Shannon-Kotel'nikov (WSK) sampling theorem [25], which lies at the heart of digital-toanalog conversion, answers this question in the affirmative when the underlying function f ∈ B Ω , i.e., it belongs to the Paley-Wiener space of band-limited functions B Ω = { f ∈ L 2 ( R ) : supp ˆ f ⊆ [ -Ω , Ω] } , for some Ω > 0 , ˆ f the Fourier transform of f and sampling rate 1 /T ≥ 2Ω , with 2Ω termed as the Nyquist rate . The corresponding reconstruction formula is,

We note that the sequence of functions { ϕ n ( x ) = sinc (2Ω x -n ) } n ∈ Z constitutes an orthonormal basis for B Ω and denote by P B Ω : L 2 ( R ) →B Ω the orthogonal projection operator onto B Ω ,

where the last equality is a consequence of B Ω being a reproducing kernel Hilbert space with kernel K Ω ( x, y ) = sinc (2Ω( x -y )) . Hence, P B Ω f corresponds to the right-hand side of (2.1) for T = 1 / 2Ω and formula (2.1) is exact if and only if f ∈ B Ω , i.e., f = P B Ω f ⇐⇒ f ∈ B Ω .

What happens when we sample a function at a sampling rate below the Nyquist rate, i.e. when 1 /T < 2Ω ? Alternatively, what is the effect of approximating a function f ̸∈ B Ω by P B Ω f ? Again, sampling theory provides an answer in the form of the aliasing error:

Definition 2.1. Aliasing for bandlimited functions [2, §5] The aliasing error function ε ( f ) and the corresponding aliasing error of f ∈ L 2 ( R ) for sampling at the rate 2Ω are given by

If the aliasing error ε ( f ) is zero, i.e. if f ∈ B Ω , we say that there is a continuous-discrete equivalence (CDE) between f and its samples { f ( n/ 2Ω) } n ∈ Z .

Continuous-Discrete Equivalence in Hilbert spaces. Next, we generalize the above described concept of continuous-discrete equivalence to any separable Hilbert space H , with inner product ⟨· , ·⟩ and norm ∥ · ∥ . A countable sequence of vectors { f i } i ∈ I in H is a frame for H if there exist constants A,B > 0 such that for all f ∈ H

Clearly, an orthonormal basis for H is an example of a frame with A = B = 1 . We will now define maps that will allow us to make the link between functions and their discrete representations, which we will extensively use in the following sections. The bounded operator T : ℓ 2 ( I ) →H , T ( { c i } i ∈ I ) = ∑ i ∈ I c i f i , is called synthesis operator and its adjoint T ∗ : H → ℓ 2 ( I ) , T ∗ f = {⟨ f, f i ⟩} i ∈ I , which discretizes the function by extracting its frame coefficients, is called analysis operator . By composing T and T ∗ , we obtain the frame operator S := TT ∗ , which is an invertible, self-adjoint and positive operator. Furthermore, the pseudo-inverse of the synthesis operator is given by T † : H → ℓ 2 ( I ) , T † f = {⟨ f, S -1 f i ⟩} i ∈ I , which will be used in the following to reconstruct the function from its discrete representation, i.e. its frame coefficients.

With these concepts, one can introduce the most prominent result in frame theory [7], the frame decomposition theorem , which states that every element in H can be uniquely and stably reconstructed from its frame coefficients by means of the reconstruction formula

where the series converge unconditionally. Formula (2.4) is clearly a generalization of reconstruction formula (2.1). However, it is worth pointing out that, in general, the coefficients in (2.4) are not necessarily point samples of the underlying function f , but more general frame coefficients.

In general, it may not be possible to access all the frame coefficients to reconstruct a function in a Hilbert space. Instead, just like in the case of reconstructing functions from point samples, one will need to consider approximations to this idealized situation. This is best encapsulated by the notion of a frame sequence , i.e., a countable sequence { v i } i ∈ I ∈ H which is a frame for its closed linear span, i.e. for span { v i : i ∈ I } . With this notion, we are in a position to generalize aliasing errors and the CDE, to arbitrary Hilbert spaces. Let H be a separable Hilbert space and let { v i } i ∈ I be a frame sequence for H with V = span { v i : i ∈ I } and frame operator S : V → V . Then, the orthogonal projection of H onto V is given by P V f = ∑ i ∈ I ⟨ f, v i ⟩ S -1 v i . Thus, formula (2.4) holds and the function f can be uniquely and stably recovered from its frame coefficients if and only if f ∈ V . If f ̸∈ V , reconstructing f from the corresponding frame coefficients results in an aliasing error:

Definition 2.2. Aliasing for functions in arbitrary separable Hilbert spaces. The aliasing error function ε ( f ) and the resulting aliasing error ∥ ε ( f ) ∥ of f ∈ H for the frame sequence { v i } i ∈ I ⊆ V are given by

(a) An alias-free operator's diagram commutes.

H

K

ℓ

2

(

I

)

ℓ

2

(

K

)

U

T

†

Φ

T

Ψ

u

ℓ

2

(

I

)

ℓ

2

(

K

)

H

K

ℓ

2

(

I

′

)

ℓ

2

(

K

′

)

u

(Ψ

,

Φ)

T

Φ

T

†

Ψ

U

T

†

Φ

′

u

(Ψ

′

,

Φ

′

)

T

Ψ

′

(c) ReNO : discrete representations u, u ′ for different discretizations are equivalent.

If the aliasing error ε ( f ) is zero, i.e. if f ∈ V , we say that there is a continuous-discrete equivalence (CDE) between f and its frame coefficients {⟨ f, v i ⟩} i ∈ I .

3 Alias-Free Framework for Operator Learning

In this section, we will extend the concept of aliasing of functions to operators. We demonstrate that this notion of operator aliasing is fundamental to understanding how the neural operator performs across various discretizations , thereby addressing the shortcomings of possible lack of continuousdiscrete equivalence. We will then define Representation equivalent Neural operators (ReNOs), whose discrete representations are equivalent across discretizations.

Assume we posit a (neural) operator U , mapping between infinite-dimensional function spaces. As mentioned before, this operator is never computed in practice, instead a discrete mapping u is used. The difficulty in formalizing this is that we should be able to compute this discrete mapping for any discretizations of the input and output function. We formalize this notion below in a practical manner.

Setting. Let U : Dom U ⊆ H → K be an operator between two separable Hilbert spaces, and let Ψ = { ψ i } i ∈ I and Φ = { ϕ k } k ∈ K be frame sequences for H and K , respectively, with synthesis operators T Ψ and T Φ . We denote their closed linear spans by M Ψ := span { ψ i : i ∈ I } and M Φ := span { ϕ k : k ∈ K } . We note that by classical frame theory [7], the pseudo-inverses T † Ψ and T † Φ , initially defined on M Ψ and M Φ , respectively, can in fact be extended to the entire Hilbert spaces, i.e. T † Ψ : H → ℓ 2 ( I ) and T † Φ : K → ℓ 2 ( K ) .

3.1 Operator Aliasing and Representation Equivalence

Once the discretization is chosen - determined by input and output frame sequences (Ψ , Φ) -connecting the continuous operator U with its discrete counterpart u is the notion of operator aliasing. Given any mapping u : ℓ 2 ( I ) → ℓ 2 ( K ) , we can build the operator T Φ ◦ u ◦ T † Ψ : H → K , whose definition clearly depends on the choices of the frame sequences that we make on the continuous level. In other words, any mapping u can be interpreted as a discrete representation of an underlying continuous operator, which in general, may differ from the operator U , that is of interest here. Hence, in analogy to Definitions 2.1 and 2.2, we can define the aliasing error of U relative to the discrete representation u as,

Definition 3.1. Operator aliasing. The aliasing error operator ε ( U, u, Ψ , Φ): Dom U ⊆ H → K is given by

and the corresponding scalar error is ∥ ε ( U, u, Ψ , Φ) ∥ , with ∥ · ∥ denoting the operator norm.

An aliasing error of zero implies that the operator U can be perfectly represented by first discretizing the function with T † Ψ , applying u , and then reconstructing with T Φ , or equivalently, that the diagram in Figure 1a commutes, i.e. the black and the blue directed paths in the diagram lead to the same result. If the aliasing error is zero, we say that ( U, u, Ψ , Φ) satisfies a continuous-discrete equivalence

(CDE), implying that accessing the discrete representation u is exactly the same as accessing the underlying continuous operator U .

Example 3.2. Aliasing with the magnitude squared operator. Consider the operator U ( f ) = | f | 2 as an operator from B Ω into B 2Ω . The choice to discretize inputs and outputs on the same grid { n 2Ω } n ∈ Z corresponds to choosing Ψ = Φ = { sinc (2Ω x -n ) } n ∈ Z , and to defining the discrete mapping u : ℓ 2 ( Z ) → ℓ 2 ( Z ) by u ( v ) = T † Ψ ◦ U ◦ T Ψ ( v ) = v ⊙ v , where ⊙ denotes the entrywise product. Then, for every f ∈ B Ω such that U ( f ) ∈ B 2Ω \ B Ω , we have

̸

and we encounter aliasing. This occurs because U introduces new frequencies that exceed the bandwidth Ω . However, we can rectify this by sampling the output functions on a grid with twice the resolution of the input grid. This corresponds to choosing Φ = { sinc (4Ω x -n ) } n ∈ Z and to defining u = T † Φ ◦ U ◦ T Ψ , which simply maps samples from grid points { n 2Ω } n ∈ Z into squared samples from the double resolution grid { n 4Ω } n ∈ Z . This effectively removes aliasing since the equality U = T Φ ◦ u ◦ T † Ψ is satisfied. Furthermore, sampling the input and output functions with arbitrarily higher sampling rate, i.e. representing the functions with respect to the system { sinc (2Ω x -n ) } n ∈ Z with Ω > 2Ω , yields no aliasing error since { sinc (2Ω x -n ) } n ∈ Z constitutes a frame for B 2Ω ⊇ B Ω .

In practice, the discrete representation u of the operator U depends on the choice of the frame sequences. This means that - as mentioned at the beginning of the section - there is one map for every input/output frame sequence Ψ , Φ . The consistency between operations u, u ′ with respect to different frame sequences can be evaluated using the following error.

Definition 3.3. Representation equivalence error. Suppose that u, u ′ are discrete maps with associated frame sequences Ψ , Φ and Ψ ′ , Φ ′ , respectively. Then, the representation equivalence error is given by the function τ ( u, u ′ ) : ℓ 2 ( I ) → ℓ 2 ( K ) , defined as:

and the corresponding scalar error is ∥ τ ( u, u ′ ) ∥ .

Intuitively, this amounts to computing each mapping on their given discretization, and comparing them by expressing u ′ in the frames associated to u . In the following section we leverage the notion of operator aliasing in the context of operator learning, as well as explore its practical implications with respect to representation equivalence at the discrete level.

3.2 Representation equivalent Neural Operators (ReNO)

Equipped with the above notion of continuous-discrete equivalence, we can now introduce the concept of Representation equivalent Neural Operator (ReNO) . To this end, for any pair (Ψ , Φ) of frame sequences for H and K , we consider a mapping at the discrete level u (Ψ , Φ): Ran T † Ψ → Ran T † Φ , which handles discrete representations of the functions. Notice how this map is indexed by the frame sequences: this is normal as when the discretization changes, the definition of the function should also change. In order to alleviate the notation, when this is clear from the context, we will refer to u (Ψ , Φ) simply as u . See SM D.1 for an explanation of the condition u (Ψ , Φ): Ran T † → Ran T † .

Ψ Φ

Definition 3.4. Representation equivalent Neural Operators (ReNO). We say that ( U, u ) is a ReNO if for every pair (Ψ , Φ) of frame sequences that satisfy Dom U ⊆ M Ψ and Ran U ⊆ M Φ there is no aliasing, i.e. the aliasing error operator is identical to zero:

We will write this property in short as ε ( U, u ) = 0 .

In other words, the diagram in Figure 1a commutes for every considered pair (Ψ , Φ) . In this case, the discrete representations u (Ψ , Φ) are all equivalent, meaning that they uniquely determine the same underlying operator U , whenever a continuous-discrete equivalence property holds at the level of the function spaces. The domain and range conditions in Definition 3.4 simply imply that the frames can adequately represent input and output functions of U .

Remark 3.5. If the aliasing error ε ( U, u, Ψ , Φ) is zero (as required in Definition 3.4), then the assumption that u (Ψ , Φ) maps Ran T † Ψ ⊆ ℓ 2 ( I ) into Ran T † Φ ⊆ ℓ 2 ( K ) implies that

We observe that this definition of u (Ψ , Φ) is such that the diagram in Figure 1b commutes. In other words, once we fix the discrete representations Ψ , Φ associated to the input and output functions, there exists a unique way to define a discretization u (Ψ , Φ) that is consistent with the continuous operator U and this is given by (3.2). In practice, we may have access to different discrete representations of the input and output functions, which in the theory amounts to a change of reference systems in the function spaces. Note that to avoid any aliasing error, the discrete representation of U has to depend on the chosen frame sequences, i.e. inevitably, u must depend on Ψ and Φ , and hence, must be discretization dependent. See A.2 for the proof of Remark 3.5.

In particular, Remark 3.5 directly implies a formula to go from one discrete representation to another, as

whenever the pairs of frame sequences (Ψ , Φ) and (Ψ ′ , Φ ′ ) satisfy the conditions in Definition 3.4. In other words, the diagram in 1c commutes.

Formula (3.3) immediately implies Proposition 3.6, which establishes a link between aliasing and representation equivalence. This highlights the contrast to discretization invariance, discussed at length in [15]: while this concept establishes an asymptotic consistency, representation equivalence includes the direct comparison between any two given discretizations and guarantees their equivalence.

Proposition 3.6. Equivalence of ReNO discrete representations. Let ( U, u ) be a ReNO. For any two frame sequence pairs (Ψ , Φ) and (Ψ ′ , Φ ′ ) satisfying conditions in Definition 3.4, we have that

where, by a slight abuse of notation, u ′ denotes u (Ψ ′ , Φ ′ ) .

Hence, under the assumption that the discrete map at each discretization is consistent with the underlying continuous operator, we have a unique way to express the operator at each discretization. Moreover, formula (3.3) closely resembles analogous formulas presented in [17, 15, 23] when evaluating single shot super resolution . However, in Section 5, we offer a nuanced perspective, indicating variability across different scenarios.

3.3 Layer-wise Instantiation

As shown in Remark 3.5, we can compute the outputs of the ReNO on the computer by first discretizing the continuous operator U . Typically, if U is a neural operator composed of multiple layers, as we will show in this section, it is possible to discretize each layer, while remaining consistent with the underlying operator.

Consider a neural operator U with L layers, each a mapping between separable Hilbert spaces:

we denote by Ψ ℓ a frame sequence for H ℓ . The choice of frame sequences for each H ℓ corresponds to the choice of accessible discrete representations of functions in the underlying function space H ℓ .

Proposition 3.7. Stability of ReNO under composition. Consider the composition U = U L ◦ . . . ◦ U 1 as in eq. 3.4, as well as a discrete mapping u = u L ◦ . . . ◦ u 1 . If each layer ( U ℓ , u ℓ ) is a ReNO, the composition ( U, u ) also is a ReNO.

As the proof of Proposition 3.7, presented in SM A.1, also shows, if each hidden layer in the operator (3.4) has an aliasing error (2.2), then these errors may propagate through the network and increase with increasing number of layers.

3.4 ϵ -ReNOs

In practice, we may not need to set the aliasing error to zero. But it is essential to be able to control it and make it as small as desired. The notion of ReNOs, Definition 3.4, can very simply be extended to the case where we allow for a small, controlled amount of aliasing. Indeed we can introduce ϵ -ReNOs, which satisfy ∥ ε ( U, u ) ∥ ≤ ϵ (for every pair of admissible frame sequences) instead of ε ( U, u ) = 0 .

Proposition 3.8. Let ( U, u ) be an ϵ -ReNO. For any two frame sequence pairs (Ψ , Φ) and (Ψ ′ , Φ ′ ) satisfying conditions in Definition 3.4 and such that M Φ ′ ⊆ M Φ , we have

where, by a slight abuse of notation, u ′ denotes u (Ψ ′ , Φ ′ ) .

4 Examples

Equipped with our definition of Representation equivalent Neural Operators (ReNOs), we analyze some existing operator learning architectures to ascertain whether they are neural operators or not.

Convolutional Neural Networks (CNN). Classical convolutional neural networks are based on the convolutional layer k , involving a discrete kernel f ∈ R 2 s +1 and a discrete input c :

We can then analyze this layer using our framework to ask whether this operation can be associated to some underlying continuous operator. Intuitively, if this is the case, the computations conducted on different discretizations effectively representing the input should be consistent; in the contrary case, no associated continuous operator exists and the convolutional operation is not a ReNO.

̸

Consider the case where the discrete input c corresponds to pointwise evaluation on a grid of some underlying bandlimited function f ∈ B Ω , for example c [ n ] = f ( n 2Ω ) , n ∈ Z with associated orthonormal basis Ψ = { sinc (2Ω x -n ) } n ∈ Z . Consider now an alternate representation of f , point samples of a grid twice as fine: d [ n ] = f ( n 4Ω ) , with Ψ ′ = { sinc (4Ω x -n ) } n ∈ Z as basis. Clearly, even though discrete inputs agree, i.e. c [ n ] = d [2 n ] , this is no longer true for the outputs, ( k ∗ c )[ n ] = ( k ∗ d )[2 n ] . This in turn implies that there exist frame sequences Ψ , Ψ ′ such that:

̸

thereby defying the representation equivalence property of ReNOs, in the sense of Definition 3.4. This fact is also corroborated with the experimental analysis, Section 5.

Fourier Neural Operators (FNO). FNOs are defined in [17] in terms of layers, which are either lifting or projection layers or Fourier layers . As lifting and projection layers do not change the underlying spatial structure of the input function, but only act on the channel width, these linear layers will satisfy Definition 3.4 of ReNO here. Hence, we focus on the Fourier layer of the form,

with the Fourier operator given by

Here, F , F -1 are the Fourier and Inverse Fourier transforms.

For simplicity of exposition, we set A ℓ = B ℓ ≡ 0 and focus on investigating whether the Fourier layer (4.2) satisfies the requirements of a ReNO. Following [17], the discrete form of the Fourier layer is given by σ ( kv ) , with kv = F -1 ( R ⊙ F ( v )) , where F, F -1 denote the discrete Fourier transform (DFT) and its inverse.

In SM B.1, we show that the convolution in Fourier space operation for the FNO layer (4.2) satisfies the requirements of a ReNO. However the pointwise activation function σ ( f ) , applied to a bandlimited input f ∈ B Ω will not necessarily respect the bandlimits, i.e., σ ( f ) / ∈ B Ω . In fact, with popular choices of activation functions such as ReLU, σ ( f ) / ∈ B ω , for any ω > 0 (see SM C for numerical illustrations). Thus, the Fourier layer operator (4.2) may not respect the continuous-discrete equivalence and can lead to aliasing errors, a fact already identified in [10]. Hence, FNO may not be a ReNO in the sense of Definition 3.4 .

Convolutional Neural Operators (CNO). Introduced in [23], the layers of a convolutional neural operator consist of three elementary operations

where K l is a convolution operator, Σ l is a non-linear operator whose definition depends on the choice of an activation function σ : R → R , and P l is a projection operator. We show in SM B.2 that CNO layers respect equation (3.2) and consequently CNOs are Representation equivalent Neural Operators (ReNOs) in the sense of Definition 3.4. This is the result of the fact that the activation layer is defined in a way to respect the band-limits of the underlying function space.

5 Empirical Analysis

5.1 Assessing Representation Equivalence

In the previous section, we have studied existing neural operator architectures from a theoretical perspective. In this section, for the same architectures, we corroborate these findings from an empirical viewpoint. Aliasing is a quantity that cannot be computed in practice, as we cannot access the underlying operator U on a computer, we can nonetheless compute representation equivalent errors introduced in Definition 3.3, which is related to aliasing by Propositions 3.7 and 3.8 (as well as being a quantity of interest in itself).

Experimental Setting. We wish to learn an unknown target operator Q using a neural operator. In this experiment, all neural operators (CNN, FNO and SNO) take as input pointwise evaluations on the grid, and are able to deal with varying input resolutions. A simple way of constructing the target operator Q : H → H , where H is the space of periodic and K = 30 -bandlimited functions, is by sampling input and output pairs in a random fashion. Sampling of a function in H can be realized as follows: As we know that Ψ K := { d K ( . -x k ) } k = -K,...,K constitutes a frame for H , d K being the Dirichlet kernel of order K and x k = k 2 K +1 , any function f ∈ H can be written as f ( x ) = ∑ K k = -K f ( x k ) d K ( x -x k ) . Thus, the discrete representation of f simply corresponds to its 2 K +1 = 61 point-wise evaluations on a grid, i.e. { f ( x k ) } k = -K,...,K . Note that for simplicity we have used and will use the same frame sequences for both input and output spaces, as these are the same.

Training and Evaluation. Once the data is generated, we train neural operators u (Ψ K , Ψ K ) on discretizations associated to the frame sequences, which are simply the point-wise evaluations of the input-target functions in the data. In other words, we regress to the frame coefficients of the target function, with coefficients of the input function as input. Once training is over, we evaluate how the different neural operators behave when dealing with changing input and output frame sequences. The frame sequences here are Ψ M for different testing frames, with associated 2 M +1 sized grids, with associated operator u M : R 2 M +1 → R 2 M +1 .

The results of this experiment are presented in Figure 2; we also provide results for additional architectures in Figure 4. The results in Figure 2 clearly show that as predicted by our theory, neither CNN nor FNO are representation equivalent, in the sense of Definition 3.4 and changing the resolution, which amounts to a change of frame, does not keep the operator invariant, causing aliasing errors that materialize themselves as representation equivalence errors (as in Definition 3.3) here. On the other hand, as predicted by the theory, CNO is more likely to be a ReNO - as long as the frames selected satisfy conditions of Definition 3.4 (i.e. 'Representation Equivalence" zone). When these conditions no longer hold ('No Equivalence" zone), CNO also generates aliasing errors. Thus, this experiment clearly demonstrates the practical implications of the ReNO framework. In contrast, the

0

25

50

75

100

125

150

175

200

Resolution

0%

25%

50%

75%

100%

125%

150%

175%

200%

Discrete Aliasing Error (%)

CNN

FNO

CNO

No Equivalence

Representation Equivalence

discretization invariance [15] condition only ensures that in the infinite resolution limit, the neural operator converges, while not making any predictions about the behavior of the neural operator at finite resolutions.

Minimizing representation equivalence error. Instead of directly tackling aliasing errors from a theoretical perspective as is done in the previous sections, we select an architecture that may have aliasing, i.e. FNO, and we try to minimize the representation equivalence error during training. We observe and detail our findings in SM C.3.

5.2 Assessing Structure Preservation

Our methodology aims to maintain the structure of the underlying operator, as defined at the continuous level. Fundamental structural characteristics of the underlying operator, like symmetries and conservation principles, are maintained at the discrete level. Discretizations that are not aligned may fail to uphold these intrinsic properties of the operator, resulting in deviations like symmetry breaking which can negatively influence the results.

As an example of this, we consider FNO. Similarly to what is done in [11], the primary structure we aim to uphold is translation equivariance, as defined at the operator level. We look at different trained FNOs - all of which have a different representation equivalence error - and assess whether they are translation equivariant or not. More specifically, these FNOs are trained by minimizing both the regression loss and the representation equivalence error simultaneously, introducing a more or less large multiplier λ on the latter, just like in SM C.3.

As we vary the parameter λ , we observe in Table 1, there is a distinct relationship between the Discrete Aliasing Error (DAE) and the Translation Equivariance Error. Both errors show a similar trend with a nearly perfect positive linear association.

6 Discussion

Summary. Although a variety of architectures for operator learning are available since the last couple of years, there is still a lack of clarity about what exactly constitutes operator learning . Everyone would agree that an operator learning architecture should have functions as inputs and outputs. However, in practice, one does not necessarily have access to functions, either at the input or output level. Rather the access is limited to some form of discrete representations of the underlying functions, for instance, point values, cell averages, coefficients with respect to a basis etc. Moreover, one can only perform computations with discrete objects on digital computers. Hence, given the necessity of having to work with discrete representations of functions, it is essential to enforce some relationship between the continuous and discrete representations of the underlying functions or in other words, demand consistency in function space.

Unlike in [15], where the authors advanced a form of asymptotic consistency in terms of discretization parameters, we go further to require a stronger form of structure-preserving equivalence of the continuous and discrete representations of operators. To this end, we leverage the equivalence between continuous and discrete representations of functions in Hilbert spaces in terms of frame theory. The main point about this equivalence is the fact that functions, belonging to suitable function spaces, can be uniquely and stably reconstructed, from their frame coefficients. A failure to enforce this equivalence results in the so-called aliasing errors that quantitatively measure function space inconsistencies.

We extend this notion of aliasing error to operators here and use it to define Representation equivalent Neural Operators (ReNOs) , see Definition 3.4. Our framework automatically implies consistency in function spaces and provides a recipe for deriving ReNOs in terms of changing the underlying frames. We also employ our framework to analyze whether or not existing operator learning architectures are ReNOs and also corroborate our results through experiments.

Related Work. Our current paper relies heavily on structure preservation from classical numerical analysis, [22] and references therein, as well as concepts from signal processing and applied harmonic analysis [25] and references therein. Among the emerging literature on operator learning, [15] was one of the first papers to attempt a unifying framework that encompasses many operator learning architectures and codify them through a particular definition of neural operators. We take an analogous route here but with our main point of departure from [15] being that unlike their notion of asymptotic consistency, we require systematic consistency in function space by enforcing the representation equivalence for the underlying operator at each layer. Another relevant work is [10] where the authors flag the issue of possible aliasing errors with specific operator learning architectures. We significantly expand on the approach of [10] by providing a rigorous and very general definition for aliasing errors. Finally, our definition of Representation equivalent Neural Operators, relying on aliasing errors for operators, is analogous to a similar approach for computing with functions , rather than discrete representations, which is the cornerstone of the Chebfun project [8] and references therein.

Limitations and Extensions. Our aim in this paper was to tackle a fundamental question of what defines a neural operator? We have addressed this question here and shown that enforcing some form of equivalence between continuous and discrete representations is needed for the architecture to genuinely learn the underlying operator rather than just a discrete representation of it. What we have not addressed here are quantitative measures of the error associated with a Representation equivalent Neural Operator, introduced in Definition 3.4, in approximating the underlying operator. This is a much broader question than what is addressed here, since sources of error, other than aliasing errors, such as approximation, training and generalization errors also contribute to the total error (see [16, 14, 15]).

One interesting direction for further analysis would involve exploring operators and their discretized counterparts that form ϵ -ReNOs and also enforce the ReNO property approximately. Characterizing the implications of this concept within operator learning is a topic for future work.

References

G. S. Alberti, M. Santacesaria, and S. Sciutto. Continuous generative neural networks. arXiv preprint arXiv:2205.14627 , 2022.

J. J. Benedetto. Irregular sampling and frames. Wavelets: A Tutorial in Theory and Applications , 2:445-507, 1992.

M. Bertero, P. Bocacci, and C. De Mol. Introduction to inverse problems in imaging . CRC press, 2021.

K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics , 7:121-157, 2021.

S. Cao. Choose a transformer: Fourier or galerkin. In 35th conference on neural information processing systems , 2021.

T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks , 6(4):911-917, 1995.

O. Christensen. Frames and bases: An introductory course . Springer Science & Business Media, 2008.

T. A. Driscoll, N. Hale, and L. N. Trefethen. Chebfun Guide . Pafnuty Publishers, Oxford, 2014.

L. C. Evans. Partial differential equations , volume 19. American Mathematical Soc., 2010.

V. Fanaskov and I. Oseledets. Spectral neural operators. arXiv preprint arXiv:2205.10573 , 2022.

N. Gruver, M. A. Finzi, M. Goldblum, and A. G. Wilson. The lie derivative for measuring learned equivariance. In The Eleventh International Conference on Learning Representations , 2023.

V. Isakov. Inverse Problems for Partial Differential Equations . Springer, 2017.

G. Kissas, J. H. Seidman, L. F. Guilhoto, V. M. Preciado, G. J. Pappas, and P. Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research , 23(215):163, 2022.

N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for Fourier neural operators. Journal of Machine Learning Research , 22:Art-No, 2021.

N. B. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces. CoRR , abs/2108.08481, 2021.

S. Lanthaler, S. Mishra, and G. E. Karniadakis. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications , 6(1):tnac001, 2022.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2021.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural operator: Graph kernel network for partial differential equations. CoRR , abs/2003.03485, 2020.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, A. M. Stuart, K. Bhattacharya, and A. Anandkumar. Multipole graph neural operator for parametric partial differential equations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 6755-6766. Curran Associates, Inc., 2020.

L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3(3):218-229, 2021.

M. Prasthofer, T. De Ryck, and S. Mishra. Variable input deep operator networks. arXiv preprint arXiv:2205.11404 , 2022.

A. Quarteroni and A. Valli. Numerical approximation of Partial differential equations , volume 23. Springer, 1994.

B. Raoni´ c, R. Molinaro, T. De Ryck, T. Rohner, F. Bartolucci, R. Alaifari, S. Mishra, and E. de Bézenac. Convolutional neural operators for robust and accurate learning of PDEs. To appear in 37th conference on neural information processing systems, https://arxiv.org/abs/2302.01178 , 2023.

J. H. Seidman, G. Kissas, P. Perdikaris, and G. J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. arXiv preprint arXiv:2206.03551 , 2022.

M. Unser. Sampling-50 years after Shannon. Proceedings of the IEEE , 88(4):569-587, 2000.

M. Vetterli, J. Kovacevic, and V. Goyal. Foundations of Signal Processing . Cambridge University Press, 2014.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new paradigm to look at operator learning via the lens of frame theory termed as Representation Equivalent Neural Operator (ReNO). It talks about an important missing piece in the literature, the one regarding the balance of continuous nature of operators and the discrete nature of data these are trained on. Aliasing error is used as the quantifier for determining the equivalence.\n\nSTRENGTHS:\nThe paper is very well motivated and suitably placed in the literature.\n\nThe use of frames as a tool for representation is presented in an interesting way.\n\nThe paper address a common issue in a very mathematically rigorous manner. \n\nThe experimental evidence is convincing.\n\nWEAKNESSES:\nThe overall presentation of the paper can be improved significantly. \n\nFew pointers: Figures appearing the paper have no caption, Figure 1,2 and 3 have been referred within the text, others not. In a paper like these, visual guides are very much helpful, otherwise it is easy to not follow. Additionally there is over use of inline equations which are hard to parse. If space is the issue, some details can be moved out to the appendix, for example section 2 can be compressed, although it reads well, it is somewhat standard.\n\nNotations, need to made clear around when introduced. For example: that $\\mathfrak{g}$ is dicretization of $\\mathcal{G}$ is mentioned around line 211 and introduced much earlier. This makes it harder to read through.\n\nExperimental section should have more details, the current form is not at all clear in the main paper. I see lot of blank space in that page anyway.\n\nQUESTIONS:\nOf all the three architectures considered in the paper, none of them perform on resolutions lower than the one used for training. Can the authors discuss this a bit?\n\nWhile the paper is written in general for frames, can the authors comment on some pros/cons of specific choice for example wavelet frames? That would be very interesting.\n\nLIMITATIONS:\nI don't see any direct potential negative social impact.\n\nAuthors have mentioned technical limitations which is very much appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMany recent studies have emerged in the field of operator learning. However, many models attempt to learn operators using the discretized values of functions rather than sending functions as operators to other functions. In this paper, the authors interpret the relationship between infinite-dimensional functions and their discretized values using the framework of frame theory. The authors analyze various operator learning models to assess whether they effectively capture this relationship in an equivalent manner, and they propose a framework called ReNO for this purpose.\n\nSTRENGTHS:\nInvestigating whether models that engage in operator learning, a mapping process between infinite-dimensional function spaces, effectively handle and address this infinite dimensionality is a pivotal and highly relevant topic within the field. The endeavor to analyze and comprehend this specific aspect of operator learning appears to be a novel and significant contribution. Leveraging the definitions associated with frame theory, as established in Hilbert spaces, and extending their application to operators, enables a profound exploration of the extent to which various operator models align with the ReNO framework. This comprehensive analysis sheds light on the capacity of these models to capture and interpret the intricate relationship between infinite-dimensional functions and their discretized counterparts. Such insights provide valuable guidance for the advancement of operator learning methodologies and contribute to the broader understanding of this field.\n\nWEAKNESSES:\nThe explanations of frame theory discussed in Sections 2 and 3, as well as the summaries of applying frame theory to operators, are written in a manner that is not easily comprehensible. Many readers, including myself, who may lack a strong background in signal processing and functional analysis, find these sections challenging to understand (See Questions). It would be beneficial to provide clearer explanations and examples regarding aspects such as the invertibility of the frame operator S and the well-definedness of the pseudo-inverse, clarifying their significance.\n\nFurthermore, in Section 4, which aims to verify whether the defined operator learning models conform to ReNO, it is intuitive to consider that CNN and FNO may not fall under the purview of ReNO, as they simply discretize functions into image-like forms. On the other hand, SNO, which predefines a basis and solely learns the mapping between the coefficients, appears to naturally align with ReNO. It would be intriguing to explore additional insights derived from these theories based on this intuition. Additionally, there is a well-known operator learning model called DeepONet, which could be described in more detail. While it is briefly mentioned on line 271 that DeepONet also belongs to SNO, further elaboration is necessary to understand how and what differentiates it from other models. Is there a specific reason for focusing solely on CNN, FNO, and SNO among the numerous operator learning models?\n\nMoreover, in Section 5, the experiments conducted, particularly those related to super-resolution discussed in the FNO paper, and the results presented in Figure 4, raise questions regarding whether they truly represent the best experiments to validate the aforementioned theories. Specifically, what are the specific definitions of the frames $\\Phi$ and $\\Psi$ used in the experiment where the resolution is altered in the predictions (as explained in Section 4 of the FNO paper)? It is unclear whether these definitions are explicitly provided or not. Are there alternative approaches, apart from altering resolution, to demonstrate the equivalence of arbitrary frames $\\Phi$ and $\\Psi\"? It would be valuable to explore additional experimental setups that further support the underlying theories, as relying solely on the resolution experiment appears inadequate.\n\nQUESTIONS:\n* Is there a specific reason why not all the various models mentioned in line 30 were examined under ReNO? Why was the analysis limited to only CNN, FNO, and SNO?\n\n* What are the definitions of $f(nT)$ and $f(n/2\\Omega)$ mentioned in line 70? Additionally, why do the sinc functions in line 75 form an orthonormal basis? Understanding these concepts seems crucial to grasp the difference between Definition 2.1's $f$ and $P_B f$.\n\n* I'm curious to understand why the adjoint $T^*$ in line 93 is well-defined uniquely for all $f_i$. Furthermore, what makes the frame operator $S$ invertible and well-defined?\n\n* In line 102, when referring to \"point samples of the underlying function $f$,\" which specific $f_i$ does this pertain to? (Similarly, for line 221, when does $T^\\dagger_\\Psi$ precisely become a discretization representation given $\\Psi$?)\n\n* I'm interested in understanding how Definitions 2.1 and 2.2 naturally lead to the discussion of aliasing error in Definition 3.1. Explaining the differences and similarities between these concepts would be helpful.\n\n* The diagrams between lines 156 and 157, the diagram mentioned below line 265, and the captions and explanations for Figures 1, 2, and 3 are significantly lacking. They need to be added with essential details. What are the differences between the three figures, and how are they interconnected?\n\n* What is the relevance of Equation 4.3 in the analysis of SNO, considering the diagram? While A and B were assumed to be zero in FNO, does this equation account for the bias term b?\n\n* DeepONet is known as a successful operator learning model mentioned in line 271. What does it mean that DeepONet is also equivalent to SNO? Further elaboration is needed.\n\n* In Section 5, it would be beneficial to summarize the experimental setups (Line 506 F.2) , even briefly in the main text, rather than relegating them to the appendix. It would be helpful to explain what type of regression was performed, whether it involved the solution operator of PDE or not, or if it was a different type of experiment.\n\nLIMITATIONS:\n* The explanation of frame theory is challenging to understand, and the extension of frame theory to operators is equally difficult to grasp. There are intricate aspects in the theoretical part that make it hard to follow, such as meticulous definitions of notations, the reasons for orthogonality, the existence of inverses, whether $T^*$ is necessary for operator extension, and the well-definedness of $T^\\dagger$. It would be beneficial to summarize the essential concepts needed to extend these aspects to operators and provide a slightly more accessible explanation. Ultimately, it would be helpful to clearly articulate why frame theory is crucial for explaining operators, rather than presenting it as a separate theory.\n\n* There seems to be a lack of experimental evidence supporting the theoretical claims. In Section 5, which focuses on experiments and analysis related to resolution, additional experiments that better illustrate the theory could enhance the persuasiveness of the need for this theory.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors investigate the concept of neural operators in the context of operator learning architectures. They address the fundamental question of what defines a neural operator and propose the notion of Representation equivalent Neural Operators (ReNOs) that satisfy a systematic consistency between continuous and discrete representations, referred to as continuous-discrete equivalence (CDE).\n\nThe authors present a framework for analyzing existing operator learning architectures and determining whether they qualify as ReNOs. They emphasize the importance of enforcing CDE at each layer of the approximating operator to ensure genuine learning of the underlying operator, rather than just a discrete representation. Experimental analysis is conducted to validate the theoretical claims and demonstrate the practical implications of respecting representation equivalence.\n\nSTRENGTHS:\n1. Originality: The paper introduces a novel perspective on neural operators in the context of operator learning architectures by focusing on the concept of continuous-discrete equivalence. The notion of Representation equivalent Neural Operators (ReNOs) provides a fresh and rigorous definition that emphasizes the importance of maintaining consistency between continuous and discrete representations. This original approach extends the understanding of neural operators and adds a new dimension to the analysis of operator learning architectures.\n\n2. Quality and Clarity:  the framework is well-structured, and the definitions and mathematical formulations seem correct and rigorous.\n\n3. Significance: The paper holds significant importance for the theoretical analysis of operator learning architectures.\n\nWEAKNESSES:\n1. Experimental Validation: Although the paper includes experimental analysis to support the theoretical claims, there is room for further elaboration and validation, e.g. incorporating more SOTA architectures and more benchmark datasets.  \n2. Practical Significance: Lack of a principled architecture instance or design. The paper does not propose a new architecture satisfying the ReNO definition, and does not provide a systematic way to design new instances of ReNO.\n3. Readability: Lack of intuition and hints between mathematical definition and derivation, especially in section 3.\n\nQUESTIONS:\n1. Is SNO only suitable for periodic signals? If one uses a chebyshev basis instead of fourier basis to implement SNO, then does the analysis in the paper still hold?\n\nLIMITATIONS:\nThe authors provide a through and insightful discussion about limitations and extensions in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigate the aliasing error of operator learning. The work points out that many existing neural operators have aliasing error, so the error is higher on super-resolution problem. To deal with the aliasing error, the paper proposed Representation equivalent Neural Operators (ReNO), which have no aliasing error. Numerical experiments show ReNO model perform better with super-resolution task.\n\nSTRENGTHS:\nThe paper investigates a very interesting problem -- the concept of resolution invariance in operator learning. It introduces frame theory to study the aliasing error in operator learning and it proposes a neural class of model called Representation equivalent Neural Operators (ReNO). The paper helps the community to better understand the key concept of neural operator.\n\nWEAKNESSES:\nI think it's very important to distinguish the two different concepts:\n\n### Discretization invariance vs representation equivalence\n- Previous work define neural operator as asymptotic discretization invariant model that converge as the grid refines [1]. In a sense that the neural operator model will converge to the underlying solution operator in continuum. In the case of Fourier representation, discretization invariant FNO model has the capacity to learning infinite number of Fourier basis as created by the non-linear activation layer, which in the end, will cause the aliasing error.\n\n- In this work, the authors the representation equivalent model that has no aliasing error. It is equivalent to say these models are prescribed in a fixed linear subspace of the underlying infinite-dimensinal function space. In the case of Fourier representation, the representation equivalent SNO model has a fixed number of Fourier basis, and a consequential irreducible approximation error.\n\nWhile the newly proposed representation equivalent models have no aliasing error, they have a limited expressive power, which prevents the model extrapolates to unseen models and higher frequencies. Since the ReNO models cannot use non-linear activation layers after the inverse spectral transform, the output space is restricted to the spectral space, and the model is of the class of linear reconstruction operator [2], which may require exponentially more number of parameter compared to non-linear reconstruction operator like FNO.\n\nIn practice, the ReNO models probably perform better at super-resolution, while previous asymptotic discretization invariant models perform better at fixed resolution or mixed resolution training.\n\n### Writing\nBesides, the overall writing is very clear. The reviewer kindly suggests to use a more informative title.\n\n\n[1] Kovachki, Nikola, et al. \"Neural operator: Learning maps between function spaces with applications to PDEs.\" Journal of Machine Learning Research 24.89 (2023): 1-97.\n[2] Lanthaler, Samuel, et al. \"Nonlinear reconstruction for operator learning of pdes with discontinuities.\" arXiv preprint arXiv:2210.01074 (2022).\n\nQUESTIONS:\nThe CNO work [1] describes a method that makes neural operator model using activation layers also aliasing-free. Specifically it apply upsampling and downsampling before and after band-limited activation layers. As claimed in Proposition 2.1, CNO are representation equivalent.\n\nI wonder if other works such as FNO enjoy a similar properties. Since the IFFT is equivalent to upsampling and FFT is equivalent to downsampling. FNO can restrict the number of Fourier modes of its representation space too if equipped with a band-limited activation function. On the Fourier space FNO truncates the latent function to N Fourier modes, and the band-limited activation function introduce at most M new Fourier modes, so the highest number of Fourier modes is N*M. If N*M < the resolution, there will be no aliasing error.\n\nIf it is the case, does FNO also belong to ReNO?\n\n\n[1] Raonić, Bogdan, et al. \"Convolutional Neural Operators.\" arXiv preprint arXiv:2302.01178 (2023).\n\nLIMITATIONS:\nIf will be better to discuss the relationship between the aliasing error and approximation power.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe concept of “representation equivalence” is introduced in the context of operator learning. The definition amounts to requiring zero aliasing error from the mode. It is shown whether several popular architectures satisfy the new definition.\n\nSTRENGTHS:\nThe paper is well written and easy to follow. The math is made accessible and good intuition is given. Claims are proven rigorously with detailed proofs.\n\nWEAKNESSES:\nWhile I like the general direction of this paper, I do not appreciate catchy titles that obfuscate mathematical ideas for the purposes of a “wow” factor. Are Neural Operators Really Neural Operators? Yes they are because that’s how they were defined in [14]. Putting titles aside, I think that two ideas about what should constitute “operator learning” are being mixed. The point of [14] is to define families of architectures whose parameters are not tied to the discretization of the input or output functions. This makes the cost of a model (with cost defined as the size of the model) independent of discretization. The analogy is to explicit vs. implicit methods for solving time-dependent PDEs. For an explicit method the cost (defined here as the inverse size of the time step) increases as the spatial discretization increases due to CFL. On the other hand, for implicit methods, the time step and the spatial discretization are independent. From this point of view, neural operators are analogous to implicit methods while, say CNNs, are analogous to explicit methods. This makes CNNs a perfectly reasonable method for operator learning, however, the challenge which remains is how does one increase the size of the CNN with the resolution in order to guarantee consistent results. This can then map into a proper cost-accuracy trade-off analysis between methods just as one can do for explicit vs implicit methods. Note that for this to be done, architectures need to be re-trained at each testing resolution. Indeed what is shown in [14] is if one re-trains the same model with higher and higher resolutions, the answer remains consistent. A consequence of the neural operator definition (parameters not tied to grid points) is that a trained architecture can be used to predict at multiple resolutions, however, one cannot, in general, have guarantees that the answer will stay consistent. Indeed if the true operator transforms all modes of an input function in some way then how can one guarantee that a surrogate trained only on some of the modes will generalize to unseen modes? One can hope that the surrogate extrapolates correctly and that has empirically been shown on some problems (some in [14] but also many others), but it cannot be guaranteed any more than generalization abilities of deep neural networks can be guaranteed. From this point of view, neural operators do precisely what they are meant to do, \nand I strongly think that the authors should be much more clear and explicit about the problem that their work tries to address because it is ultimately a different one. \n\nI like the idea of quantifying and mitigating the aliasing error. It is a very important direction that has been almost unexplored in deep learning with many practical consequences. However, I do think that Definition 3.4 is much too strong to be useful. Indeed, in Remark 3.5, the authors show that, as a consequence of this definition, there is only one recipe for constructing ReNO architectures, mainly eq. (3.6). Note the similarity of this equation to eq. (2.3), upon combining with eq. (2.2), in https://arxiv.org/pdf/2005.03180.pdf. I point this out because constructions of this kind yield only linear methods of approximation. It is well known in function approximation that linear methods are suboptimal (for example, https://people.math.sc.edu/devore/publications/NLACTA.pdf). While this has certainly not been studied nearly as deeply in the operator setting, similar empirical and theoretical results are beginning to emerge (for example, https://arxiv.org/pdf/2210.01074.pdf). I therefore struggle to see the practicality of such a strong definition. For what kinds of problems would this be useful?\n\nLastly, I think the numerical experiments are quite insufficient in demonstrating the practical usability of ReNO. Is aliasing error actually prevalent or is the main source of error, the approximation error? If the latter, should architecture design not be focused on reducing approximation as opposed to aliasing error? What is the trade-off between approximation and aliasing, and can an architecture which is not ReNO, still be modified to reduce aliasing error while retaining the same approximation power?\n\nQUESTIONS:\nI do not understand the details of the numerical example. After having read the paper and the Appendix, I feel that I am missing something. I will describe the set-up to the best of my knowledge. I will denote by f the input function and by u the output i.e. Q(f)=u. For simplicity, consider only two resolutions and denote by x_1,...,x_m the first discretization and by y_1,...,y_n the second with the assumption that they are nested so that {x_j} is a proper subset of {y_j}. Data is generated by randomly (from a Gaussian) sampling the point values f(y_j) and u(y_j). Three neural network based methods are then trained to regress on samples f(x_j) and u(x_j). They are all then tested on fitting u(y_j) from f(y_j) (I assume with samples from a test set). It is shown that the SNO achieves the same error on regressing u(x_j) from f(x_j) as it does on regressing u(y_j) from f(y_j) while the other two methods do not. I do not understand how this is possible. The SNO approximates Q in the form Q(f)(x) = sum_{k=-d/2}^{d/2-1} g_k ({<f, e^{i .}>, …, <f, e^{id .}>}) e^{ikx} where g is a neural network mapping C^d to C^d with d some fixed positive integer. By passing from f(x_j) to f(y_j), the input to g changes since the inner products now see f(y_j),  but the periodic bases stay the same. The values of u(y_j) were picked at random and independently from those of f(y_j) so how can this new information about f possibly have an effect on the prediction of u? In general, this does not seem possible since whenever Q transforms all modes of f whatever method that regresses Q must extrapolate this transformation to new modes. While this may happen sometimes, it cannot be guaranteed. But this looks completely impossible in the current experiment since the SNO must extrapolate randomness. There must be something in this experiment that I am missing; it would be helpful for the authors to clarify. \n\nWhile the authors do not propose the SNO, it is the only method which they show satisfies ReNO, so I do wonder about it. It is well known that, for approximation in L^2, an optimal linear subspace is given by PCA. This is the motivation behind PCA-Net [3] which is precisely like the SNO but, instead of picking Fourier bases, computes bases for the inputs and outputs by doing PCA on the data. What is the motivation behind sticking with Fourier bases? In fact, since any method which is ReNO is a linear method of approximation, is PCA-Net not the optimal ReNO method (barring optimality of the finite-dimensional neural network)? Generally, if the ReNO definition is to be adopted as a standard way of doing operator learning, how do the authors envision new architectural designs? Remark 3.5 fully characterizes ReNO architectures, so it is hard for me to see a way forward.\n\nThe first example of Section 4 shows that discrete convolution layers are not ReNO, yet the work https://arxiv.org/pdf/2302.01178.pdf (CNO, Proposition 2.1) claims that they are. As far as I can tell, the way that ReNO is proved for CNO is by showing that discrete convolution maps w-bandlimited functions to w-bandlimited functions and then constructing a pointwise nonlinearity that does the same. I am not convinced by the CNO proof since nonlinearities used in practice (like ReLU or, more generally, learned MLPs) will not map bandlimited functions to bandlimited functions of any order and, so, even going through the upsampling-nonlinearity-downsampling procedure, will not give an aliasing error that is exactly zero (but it will mitigate the aliasing error as was first proposed in https://arxiv.org/pdf/2106.12423.pdf). Putting nonlinearities aside, however, one still needs that the discrete convolution is ReNO, so where do the differences lie? \n\nHow do the authors envision a similar theory for Banach spaces when the $\\ell_2$ isomorphism breaks down?\n\nLIMITATIONS:\nThe authors adequately address potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}