Title: Convolutional Neural Operators for robust and accurate learning of PDEs
Venue: NeurIPS 2023
Accepted: yes
Keywords: PDEs

Neural Operators

Scientific Machine Learning

Convolutional Neural Networks
Abstract: Although very successfully used in conventional machine learning, convolution based neural network architectures -- believed to be inconsistent in function space -- have been largely ignored in the context of learning solution operators of PDEs. Here, we present novel adaptations for convolutional neural networks to demonstrate that they are indeed able to process functions as inputs and outputs. The resulting architecture, termed as convolutional neural operators (CNOs), is designed specifically to preserve its underlying continuous nature, even when implemented in a discretized form on a computer. We prove a universality theorem to show that CNOs can approximate operators arising in PDEs to desired accuracy. CNOs are tested on a novel suite of benchmarks, encompassing a diverse set of PDEs with multi-scale solutions and are observed to significantly outperform baselines, paving the way for an alternative framework for robust and accurate operator learning.

Convolutional Neural Operators for robust and accurate learning of PDEs

Abstract

Although very successfully used in conventional machine learning, convolution based neural network architectures - believed to be inconsistent in function space have been largely ignored in the context of learning solution operators of PDEs. Here, we present novel adaptations for convolutional neural networks to demonstrate that they are indeed able to process functions as inputs and outputs. The resulting architecture, termed as convolutional neural operators (CNOs), is designed specifically to preserve its underlying continuous nature, even when implemented in a discretized form on a computer. We prove a universality theorem to show that CNOs can approximate operators arising in PDEs to desired accuracy. CNOs are tested on a novel suite of benchmarks, encompassing a diverse set of PDEs with possibly multi-scale solutions and are observed to significantly outperform baselines, paving the way for an alternative framework for robust and accurate operator learning.

1 Introduction.

Partial Differential Equations (PDEs) [13] are ubiquitous as mathematical models in the sciences and engineering. Solving a PDE amounts to (approximately) computing the so-called solution operator that maps function space inputs such as initial and boundary conditions, coefficients, source terms etc, to the PDE solution which also belongs to a suitable function space. Well-established numerical methods such as finite differences, finite elements, finite volumes and spectral methods (see [48]) have been very successfully used for many decades to approximate PDE solution operators. However, the prohibitive computational cost of these methods, particularly in high dimensions and for many query problems such as UQ, inverse problems, PDE-constrained control and optimization, necessitates the design of fast, robust and accurate surrogates. This provides the rationale for the use of data-driven machine learning methods for solving PDEs [21].

As operators are the objects of interest in solving PDEs, learning such operators from data, which is loosely termed as operator learning , has emerged as a dominant paradigm in recent years for the applications of machine learning to PDEs. A very partial list of architectures for operator learning include operator networks [9], DeepONets [39] and its variants [41, 7], PCA-net [5] , neural operators [25] such as graph neural operator [34], Multipole neural operator [35] and the very popular Fourier Neural Operator [33] and its variants [36, 45], VIDON [47], spectral neural operator [14], LOCA [23], NOMAD [52] and transformer based operator learning architectures [8].

Despite the considerable success of the recently proposed operator learning architectures, several pressing issues remain to be addressed. These include, but are by no means restricted to, limited expressivity for some of these algorithms [28] to aliasing errors for others [14] to the very fundamental issue of possible lack of consistency in function spaces for many of them. As argued in a recent paper [2], a structure-preserving operator learning algorithm or representation equivalent neural operator has to respect some form of continuous-discrete equivalence (CDE) in order to learn the underlying operator, rather than just a discrete representation of it. Failure to respect such a CDE can lead to the so-called aliasing errors [2] and affect model performance at multiple discrete resolutions.

Despite many attempts, see [1, 16] and references therein, the absence of a suitable CDE, resulting in aliasing errors, has also plagued the naive use of convolutional neural networks (CNNs) in the context of operator learning, see [63, 33, 2] on how using CNNs for operator learning leads to results that heavily rely on the underlying grid resolution. This very limited use of Convolution (in physical space) based architectures for operator learning stands in complete contrast to the fact that CNNs [30] and their variants are widely used architectures for image classification and generation and in other contexts in machine learning [29, 37, 61]. Moreover, CNNs can be thought of as natural generalizations of the foundational finite difference methods for discretizing PDEs [17, 38]. Given their innate locality, computational and data efficiency, ability to process multi-scale inputs and outputs and the availability of a wide variety of successful CNN architectures in other fields, it could be very advantageous to bring CNN-based algorithms back into the reckoning for operator learning. This is precisely the central point of the current paper where we make the following contributions,

Wepropose novel modifications to CNNs in order to enforce structure-preserving continuousdiscrete equivalence and enable the genuine, alias-free, learning of operators. The resulting architecture, termed as Convolutional Neural Operator (CNO), is instantiated as a novel operator adaptation of the widely used U-Net architecture.

In addition to showing that CNO is a representation equivalent neural operator in the sense of [2], we also prove a universality result to rigorously demonstrate that CNOs can approximate the operators, corresponding to a large class of PDEs, to desired accuracy.

We test CNO on a novel set of benchmarks, that we term as Representative PDE Benchmarks (RPB), that span across a variety of PDEs ranging from linear elliptic and hyperbolic to nonlinear parabolic and hyperbolic PDEs, with possibly multiscale solutions . We find that CNO is either on-par or outperforms the tested baselines on all the benchmarks, both when testing in-distribution as well as in out-of-distribution testing.

Thus, we present a new CNN-based operator learning model, with desirable theoretical properties and excellent empirical performance, with the potential to be widely used for learning PDEs.

2 Convolutional Neural Operators.

Scales:

Concat

D Block

K

D

→Du

P

u

G(u)

UBlock

7

K

u

→ulo

Concat

I Block

K

→Io

HH

S

Concat

R Block

K

KRu

:Activation::

u

D

→E

Layer

Setting. For simplicity of the exposition, we will focus here on the two-dimensional case by specifying the underlying domain as D = T 2 , being the 2 -d torus. Let X = H r ( D, R d X ) ⊂ Z and Y = H s ( D, R d Y ) be the underlying function spaces, where H r,s ( D, · ) are Sobolev spaces of order r and s . Without loss of generality, we set r = s hereafter. Our aim would be to approximate continuous operators G † : X → Y from data pairs ( u i , G † ( u i ) ) M i =1 ∈ X × Y . We further assume that there exists a modulus of continuity for the operator i.e.,

with ω : R + → R + being a monotonically increasing function with lim y → 0 ω ( y ) = 0 . The underlying operator G † can correspond to solution operators for PDEs (see Section 3 for the exact setting) but is more general than that and encompasses examples such as those arising in inverse problems, for instance in imaging [4].

Bandlimited Approximation. As argued in a recent paper [2], Sobolev spaces such as H r are, in a sense, too large to allow for any form of continuous-discrete equivalence (CDE), i.e., equivalence between the underlying operator and its discrete representations, which is necessary for robust operator learning. Consequently, one has to consider smaller subspaces of H r which allow for such CDEs. In this respect, we choose the space of bandlimited functions [57] defined by,

for some w > 0 and with ̂ f denoting the Fourier transform of f . It is straightforward to show using (2.1) (see SM A.1) that for any ε > 0 , there exists a w , large enough depending on r , and a continuous operator G ∗ : B w ( D ) →B w ( D ) , such that ∥ ∥ G † -G ∗ ∥ ∥ < ε , with ∥ · ∥ denoting the corresponding operator norm. In other words, the underlying operator G † can be approximated to arbitrary accuracy by the operator G ∗ that maps between band-limited spaces. Consequently, as shown in SM A.2, one can readily define discrete versions of G ∗ using the underlying sinc basis for bandlimited functions and establish a continuous-discrete equivalance for it.

Definition of CNO. Given the above context, our goal will be to approximate the operator G ∗ in a structure-preserving manner i.e., as the underlying operator maps between spaces of bandlimited functions, we will construct our operator approximation architecture to also map bandlimited functions to bandlimited functions, thus respecting the continuous-discrete equivalence. To this end, we denote the operator G : B w ( D ) →B w ( D ) as a convolutional neural operator (CNO) which we define as a compositional mapping between functions as

From (2.3), we see that first, the input function u ∈ B w ( D ) is lifted to the latent space of bandlimited functions through a lifting layer :

Here, d 0 > d X is the number of channels in the lifted, latent space. The lifting operation is performed by a convolution operator which will be defined below.

Then, the lifted function is processed through the composition of a series of mappings between functions (layers), with each layer consisting of three elementary mappings, i.e., P l is either the upsampling or downsampling operator, K l is the convolution operator and Σ l is the activation operator. These elementary operators are defined below and are inspired by the modifications of CNNs for image generation in [22]. Finally, the last output function in the iterative procedure v L is projected to the output space with a projection operator Q , defined as

The projection operation is also performed by a convolution operator defined below.

where

Convolution Operator. For simplicity of exposition, we will present the single-channel version of the convolution operator K l here. See SM A.3 for the multi-channel version for this and other operators considered below. Convolution operations are performed with discrete kernels

defined on the s × s uniform grid on D with grid size ≤ 1 / 2 w , in-order to satisfy the requirements of the Whittaker-Shannon-Kotelnikov sampling theorem [56], and z ij being the resulting grid points, k ∈ N being the kernel size and δ x denoting the Dirac measure at point x ∈ D . The convolution operator for a single-channel K w : B w ( D ) →B w ( D ) is defined by

where the last identity arises from the fact that f ∈ B w . Thus, our convolution operator is directly parametrized in physical space, in contrast to the Fourier space parametrization of a convolution in the FNO architecture of [33]. Hence, our parametrization is of a local nature.

Upsampling and Downsampling Operators. For some w > w , we can upsample a function f ∈ B w to the higher band B w by simply setting,

On the other hand, for some w < w , we can downsample a function f ∈ B w to the lower band B w by setting D w,w : B w ( D ) →B w ( D ) , defined by

where ⋆ is the convolution operation on functions defined above and h w is the so-called interpolation sinc filter :

Activation Layer. Naively, one can apply the activation function pointwise to any function. However, it is well-known that such an application will no longer respect the band-limits of the underlying function space and generate aliasing errors [22, 14, 2]. In particular, nonlinear activations can generate features at arbitrarily high frequencies. As our aim is to respect the underlying CDE, we will modulate the application of the activation function so that the resulting outputs fall within desired band limits. To this end, we first upsample the input function f ∈ B w to a higher bandlimit w > w , then apply the activation and finally downsample the result back to the original bandlimit w (See Figure 1). Implicitly assuming that w is large enough such that σ ( B w ) ⊂ B w , we define the activation layer in (2.3) as,

Instantiation through an Operator U-Net architecture. The above ingredients are assembled together in the form of an Operator U-Net architecture that has bandlimited functions as inputs and outputs. In addition to the blocks that have been defined above, we also need additional ingredients, namely incorporate skip connections through ResNet blocks of the form, R w,w : B w ( D, R d ) → B w ( D, R d ) such that

We also need the so-called Invariant blocks of the form, I w,w : B w ( D, R d ) →B w ( D, R d ) such that

Finally, all these ingredients are assembled together in a modified Operator U-Net architecture which is graphically depicted in Figure 1. As seen from this figure, the input function, say u ∈ B w ( D, R d X ) is first lifted and then processed through a series of layers. Four types of blocks are used i.e.,

downsampling (D) block corresponding to using the downsampling operator D as the P in (2.4), upsampling (U) block corresponding to using the upsampling operator U as the P in (2.4), ResNet (R) block corresponding to (2.7) and Invariant (I) block corresponding to (2.8). Each block takes a band-limited function as input and returns another band-limited function (with the same band) as the output. Finally, U-Net style patching operators, which concatenate outputs for different layers as additional channels are also used. As these operations act only in the channel width and leave the spatial resolution unchanged, they conform to the underlying bandlimits. Thus, CNO takes a function input and passes it through a set of encoders, where the input is downsampled in space but expanded in channel width and then processed through a set of decoders, where the channel width is reduced but the space resolution is increased. At the same time, encoder and decoder layers (at the same spatial resolution or band limit) are connected through additional ResNet blocks. Thus, this architectural choice allows for transferring high frequency content via the skip connections, before filtering them out with the sinc filter as we go deeper into the encoder. Hence, the high frequency content is not just recreated with the activation function, but also modified through the intermediate networks. Consequently, we build a genuinely multiscale operator learning architecture .

Continuous-Discrete Equivalence for CNO. We have defined CNO (2.3) as an operator that maps bandlimited functions to bandlimited functions. In practice, like any computational algorithm, CNO has to be implemented in a discrete manner, with discretized versions of each of the abovedefined elementary operations being specified in SM A.4. Given how each of the elementary blocks (convolution, up- and downsampling, activation, ResNets etc) are constructed, we prove the following proposition (in SM A.5):

Proposition 2.1. Convolutional Neural Operator G : B w ( D, R d X ) → B w ( D, R d Y ) (2.3) is a Representation equivalent neural operator or ReNO , in the sense of [2], Definition 3.4.

Further details about the notion of ReNOs is provided in SM A.4 and we refer the reader to [2], where this concept is presented in great detail and the representation equivalence of CNO is discussed. In particular, following [2], representation equivalence implies that CNO satisfies a form of resolution invariance , allowing it to be evaluated on multiple grid resolutions without aliasing errors.

3 Universal Approximation by CNOs.

We want to prove that a large class of operators, stemming from PDEs, can be approximated to desired accuracy by CNOs. To this end, we consider the following abstract PDE in the domain D = T 2 ,

with L being a differential operator and B a boundary operator. We assume that the differential operator L only depends on the coordinate x through a coefficient function a ∈ H r ( D ) . The corresponding solution operator is denoted by G † : X ∗ ⊂ H r ( D ) → H r ( D ) : a ↦→ u , with u being the solution of the PDE (3.1). We assume that G † is continuous. Moreover, we also assume the following modulus of continuity,

for some p ∈ { 2 , ∞} and 0 ≤ σ ≤ r -1 , and where ω : [0 , ∞ ) → [0 , ∞ ) is a monotonously increasing function with lim y → 0 ω ( y ) = 0 . (3.2) is automatically satisfied if X ∗ is compact and G † is continuous. Under these assumptions, we have the following universality theorem for CNOs (2.3),

Theorem 3.1. Let σ ∈ N 0 and p ∈ { 2 , ∞} as in (3.2) , r > max { σ, 2 /p } and B > 0 . For any ε > 0 and any operator G † , as defined above, there exists a CNO G such that for every a ∈ X ∗ with ∥ a ∥ H r ( D ) ≤ B it holds,

In fact, we will prove a more general version of this theorem in SM B, where we also include additional source terms in the PDE (3.1).

4 Experiments.

Training Details and Baselines. We provide a detailed description of the implementation of CNO and the training (and test) protocol for CNO as well as all the baselines in SM C.1. To ensure a level playing field among all the tested models for each benchmark, we follow an ensemble training procedure by specifying a range for the underlying hyperparameters for each model and randomly selecting a subset of the hyperparameter space. For each such hyperparameter configuration, the corresponding models are trained on the benchmark and the configuration with smallest validation error is selected and the resulting test errors are reported, allowing us to identify and compare the best performing version of each model for every benchmark. We compare CNO with the following baselines: two very popular operator learning architectures, namely DeepONet (DON) [39] and FNO [33], a transformer based operator-learning architecture, i.e., Galerkin Transformer (GT) [8], feedforward neural network with with residual connections (FFNN) [18] and the very widely-used ResNet [18] and U-Net [50] architectures. 1

Representative PDE Benchmarks (RPB). Given the lack of consensus on a standard set of benchmarks for machine learning of PDEs, we propose a new suite of benchmarks here. Our aims in this regard are to ensure i) sufficient diversity among the types of PDE considered, ii) access to training and test data is readily available for rapid prototyping and reproducibility and iii) intrinsic computational complexity of problem to make sure that it is worthwhile to design fast surrogates to classical PDE solvers for a particular problem. In other words, we will only consider PDEs where classical PDE solvers can only resolve the underlying operator on fine enough grids. To meet these requirements, we will not consider PDEs in one space dimension as traditional numerical methods are already quite fast for them. On the other hand, it is hard to obtain and store data for problems in three dimensions, due to computational expense of traditional methods. The sweet spot is achieved by considering PDEs in two space dimensions. We further restrict to Cartesian domains here as all models can be readily evaluated in this setting. In addition to including a diverse set of PDEs, we only consider problems with sufficiently many spatial and temporal scales . Otherwise, traditional numerical solvers can approximate the underlying PDE on very coarse grids and it is not worthwhile to design surrogates (see SM C.3.8 for a discussion in this context on a widely used Navier-Stokes benchmark). With these considerations in mind, we present the following subset of Representative PDE Benchmarks or RPB ,

Poisson Equation. This prototypical linear elliptic PDE is given by,

The solution operator G † : f ↦→ u , maps the source term f to the solution u . With source term,

with r = -0 . 5 , the corresponding exact solution can be analytically computed (see SM C.3.1) and represents K - spatial scales. For training the models, we fix K = 16 in (4.2) and choose a ij to be i.i.d. uniformly distributed from [ -1 , 1] (See SM D for a representation of the inputs and outputs of G † ). This multiscale solution needs fine enough grid size to be approximated accurately by finite element methods, fitting our complexity criterion for benchmarks. In addition to in-distribution testing , we also consider an out-of-distribution testing task by setting K = 20 in (4.2). This will enable us to evaluate the ability of the models to generalize to inputs (and outputs) with frequencies higher than those encountered during training.

Wave Equation. This prototypical linear hyperbolic PDE is given by

with a constant propagation speed c = 0 . 1 . The underlying operator G † : f ↦→ u ( ., T ) maps the initial condition f into the solution at the final time. If we consider initial conditions to be given by (4.2) with r = 1 , then one can explicitly compute the exact solution (see SM C.3.2) to represent a multiscale standing wave with periodic pulsations (depending on K ) in time. The training and in-distribution test samples are generated by setting T = 5 , K = 24 and a ij to be i.i.d. uniformly distributed from [ -1 , 1] (See SM D for input and output samples). For out-of-distribution testing, we change the exponent of decay of the modes in (4.2) to r = 0 . 85 and K = 32 , in order to test the ability of the models to generalize to learn the effect of higher frequencies, than those present in the training data.

Transport Equation. The transport of scalar quantities of interest is modeled by PDE,

with a given velocity field and initial data f . The underlying operator G † : f ↦→ u ( ., T = 1) maps the initial condition f into the solution at the final time. We set a constant velocity field v = ( v x , v y ) = (0 . 2 , 0 . 2) leading to solution u ( x, y, t ) = f ( x -v x t, y -v y t ) . Two different types of training data are considered, i.e., smooth initial data which takes the form of a radially symmetric Gaussian, with centers randomly and uniformly drawn from (0 . 2 , 0 . 4) 2 and corresponding variance drawn uniformly from (0 . 003 , 0 . 009) and a discontinuous initial data in the form of the indicator function of radial disk with centers, uniformly drawn from (0 . 2 , 0 . 4) 2 and radii uniformly drawn from (0 . 1 , 0 . 2) (See SM C.3.3 for details and SM D for illustrations). For out-of-distribution testing in the smooth case, the centers of the Gaussian inputs are sampled uniformly from (0 . 4 , 0 . 6) 2 and in the discontinuous case, the centers of the disk are drawn uniformly from (0 . 4 , 0 . 6) 2 , while keeping the variance and the radii, respectively, the same as that of in-distribution testing . This out-of-distribution task tests the model's ability to cope with input translation-equivariance.

Allen-Cahn Equation. It is a prototype for nonlinear parabolic PDEs ,

with a reaction rate of ε = 220 and underlying operator G † : f ↦→ u ( ., T ) , mapping initial conditions f to the solution u at a final time T = 0 . 0002 . The initial conditions for training and in-distribution testing are of the form (4.2), with r = 1 and K = 24 and coefficients a ij drawn uniformly from [ -1 , 1] . For out-of-distribution testing, we set K = 16 and randomly select the initial decay r , uniformly from the range [0 . 85 , 1 . 15] of the modes in (4.2), which allows us to test the ability of the model to generalize to different dynamics of the system. Both training and test data are generated by using a finite difference scheme [62] on a grid at 64 2 resolution (see SM D for illustrations).

Navier-Stokes Eqns. These PDEs model the motion of incompressible fluids by,

in the torus D = T 2 with periodic boundary conditions and viscosity ν = 4 × 10 -4 , only applied to high-enough Fourier modes (those with amplitude ≥ 12 ) to model fluid flow at very high Reynoldsnumber . The solution operator G † : f ↦→ u ( ., T ) , maps the initial conditions f : D → R 2 to the solution at final time T = 1 . We consider initial conditions representing the well-known thin shear layer problem [3, 27] (See SM C.3.5 for details), where the shear layer evolves via vortex shedding to a complex distribution of vortices (see SM D for samples). The training and in-distribution testing samples are generated, with a spectral viscosity method [27], from an initial sinusoidal perturbation of the shear layer [27], with layer thickness ρ = 0 . 1 and 10 perturbation modes, each sampled uniformly from [ -1 , 1] . For out-of-distribution testing, the layer thickness is reduced to ρ = 0 . 09 and the layers are shifted up in the domain to test the ability of the models to generalize to a flow regime with an increased number and different locations of the shed vortices.

Darcy flow. The steady-state Darcy flow is described by the second order linear elliptic PDE,

where a is the diffusion coefficient and f is the forcing term. We set the forcing term to f = 1 . The solution operator is G † : a ↦→ u , where the input is the diffusion coefficient a ∼ ψ # µ , with µ being a Gaussian Process with zero mean and squared exponential kernel

Wechose the length scale l = 0 . 1 for the in-distribution testing and l = 0 . 05 for the out-of-distribution testing. The mapping ψ : R → R takes the value 12 on the positive part of the real line and 3 on the negative part. The push-forward measure is defined pointwise. The experimental setup is the same as the one presented in [33].

Flow past airfoils. We model this flow by the compressible Euler equations,

with density ρ , velocity v , pressure p and total Energy E related by an ideal gas equation of state. The airfoils we consider are described by perturbing the shape of a well-known RAE2822 airfoil [40] by Hicks-Henne Bump functions [42] (see SM C.3.7). Freestream boundary conditions are imposed and the solution operator maps the shape function onto the steady state density distribution (see SM D for samples) and training data are obtained with a compressible flow solver (NUWTUN) with shapes corresponding to 20 bump functions, with coefficients sampled uniformly from [0 , 1] . Out-of-distribution testing is performed with 30 bump functions.

Results. The test errors, for both in-distribution and out-of-distribution testing for all the models on the RPB benchmarks are shown in Table 1. Starting with the in-distribution results, we see that among the baselines, FNO clearly outperforms both FFNN and DeepONet on all the RPB benchmarks as well as the Galerkin Transformer on all except the Poisson test case. On the other hand, the convolution-based U-Net and ResNet models are quite competitive vis-a-vis FNO, with comparable performances on most benchmarks, while outperforming FNO by a factor of 7 -9 for the Poisson test case. This already indicates that convolution-based architectures can perform very well. Moreover, we observe from Table 1 that CNO is the best performing architecture on every task except Allen-Cahn. It readily outperforms FNO, for instance by almost a factor of 20 on the Poisson test case but more moderately but significantly on other tasks. It also outperforms U-Net and ResNet on all tasks considered here, emerging as the best-performing model over these benchmarks.

Out-of-Distribution Testing. This trend is further reinforced when we consider out-of-distribution testing. CNO generalizes well to unseen data in a zero-shot mode, with test errors increasing by approximately a factor of 2 , at most (with Allen-Cahn being an outlier) and still outperforms the baselines significantly in all cases other than Allen-Cahn, where FNO generalizes the best. FNO shows decent generalization for most problems but generalizes poorly on the transport problems. This can be attributed to its lack of translation equivariance, in contrast to U-Net, ResNet and CNO. Finally, the lack of translation invariance severely limits the out-of-distribution generalization performance of DeepONet, FFNN and Galerkin Transformer models on transport problems.

32

Ground Truth

32

CNO

32

FNO

32

UNet

(%

Test Error (in

15

UNet

Training res.

tError(in

(%

4.5

4.0

FNO

Training res.

OND

10

16

16

16

16

4

3.5

0

0

-5

5

Test

3.0

-16

-16

-16

-16

-6

-32

-16

0

16

32

-32

-16

0

16

ZE

-32

16

0

16

32

-32

-16

0

16

0

32

64

96

128

2.5

32

64

96

128

Testing Resolution

Testing Resolution

Resolution Invariance. We select three of the best-performing models (U-Net, FNO and CNO) and highlight further differences between them for the Navier-Stokes test case (see also SM C). To this end, we start with Figure 2 (left), where we present the averaged (log) spectra for the ground truth (reference solution computed with the spectral viscosity method) and those computed with CNO, FNO and U-Net. We observe from this figure that i) the spectrum of the exact solution is very rich with representations of many frequencies, attesting to the multi-scale nature of the underlying problem and ii) there are significant differences in how CNO and FNO approximate the underlying solution in Fourier space. In particular, the decay in CNO's spectrum is more accurate. On the other hand, the FNO spectrum is amplified along the horizontal axis, possibly on account of aliasing errors that add incorrect frequency content. The U-Net spectra are similar to that of CNO but with high-frequency modes being amplified, which leads to a higher test error. Next, in Figure 2 (right), we compare CNO, FNO and U-Net vis-a-vis the metric of how the test error varies across resolutions , see SM C.4 for details, which is an important aspect for robust operator learning that been highlighted in [33, 25], see also [2] for a discussion with respect to representation equivalent neural operators or ReNOs. We find from Figure 2 (right) that for the Navier-Stokes benchmark, the FNO error is not invariant with respect to resolution, with an increase in error of up to 25% on lower-resolutions as well as a more modest but noticeable increase of 10% on resolutions, higher than the training resolution of 64 2 , implying that FNO is neither able to perform alias-error free super- or sub-resolution in this case, see also [2] for a detailed discussion on the resolution-invariance of FNO. Similarly, the increase of U-Net test error with respect to varying resolutions is even more pronounced, with a maximum increase of a factor of 3 , indicating neither FNO nor U-Net are resolution (representation) equivalent in this case. In contrast, CNO error is invariant with respect to test resolution, verifying that it respects continuous-discrete equivalence . Further ablation studies for CNO are presented in SM C.5.

Efficiency. In order to further compare CNO with FNO, which is the most widely used neural operator these days, we illustrate not just the performance in terms of errors but also terms in computational efficiency. To this end, in Figure 3 (left), we plot the size (total number of parameters) vs validation error for the Navier-Stokes benchmark for all FNO and CNO models that were considered in the model selection procedure to observe that for the same model size, CNO models led to significantly smaller validation errors. This resulted in smaller errors for the same per-epoch training time (Figure 3 (center)) for CNO vis-a-vis FNO. As observed in Figure 3 (center), this also implies that one of the best-performing CNO models, with a significantly smaller error than the best-performing FNO model is also almost twice as fast to train, thus showcasing the computational efficiency of CNOs.

Scaling laws. A very important aspect of modern deep learning is to evaluate how the performance of models scales with respect to data (and model size). To investigate this, we focus on the NavierStokes benchmark and present test error vs. (log of) number of training samples for GT, FNO and CNO in Figure 3 (right) to observe that the errors decrease consistently with number of samples. To quantify the rates of decrease, we fit power laws of the form E = ( N 0 /N ) -r , with E being the test error, N number of samples and N 0 normalized to be the number of samples required to reach errors of 1% to obtain that CNO attains a much faster convergence rate ( r = 0 . 37 ) compared to FNO ( r = 0 . 28 ) and GT ( r = 0 . 27 ). This implies that CNO will require N 0 ≈ 14 . 3 K samples to attain 1% error which is 4 -times less than FNO ( N 0 ≈ 60 . 1 K ) and almost 10 -times less than GT ( N 0 ≈ 133 . 2 K ), highlighting the data efficiency of CNO. Finally, from Figure 3 (right), we also observe that a smaller CNO model (with 0 . 82 Mparameters) scales worse (with r = 0 . 3 ) compared to the best performing CNO model with 7 . 8 M parameters, illustrating that model size could be a bottleneck for data scaling and larger models are required to scale with data.

5.00

5.00

u!)

Error

4.50

4.25

4.00

4.75

CNO

FNO

Error

三

4.00

4.75

4.50

4.25

CNO

FNO

%

Error

4.5

3.5

4.0

CNO, 7.28M

GT,10.01M

FNO, 42.06M

CNO, 0.82M

Validation

3.75

Validation

3.75

3.0

3.50

3.50

Testing

2.5

3.25

3.25

2.0

3.00

3.00

106

107

3.0

3.5

4.0

4.5

5.0

5.5

6.0

6.5

1.5

Time per epoch [s]

29

210

211

212

213

Number of training samples

5 Discussion.

Summary. We propose CNO, a novel convolution-based architecture for learning operators. The basic design principle was to enforce a form of continuous-discrete equivalence in order to genuinely learn the underlying operators, rather than discrete representations of them. To this end, we modified the elementary operators of convolution, up-and downsampling and particularly nonlinear activations to realize CNO as a representation equivalent neural operator or ReNO in the sense of [2]. We also prove a universality theorem to show that CNO can approximate a large class of operators arising in PDEs to desired accuracy. A novel suite of experiments, termed as representative PDE benchmarks ( RPB ), encompassing a wide variety of PDEs, with multiple scales in the corresponding solutions, which are hard to resolve with traditional numerical methods, is also proposed and the model tested on them. We demonstrate that CNO outperforms the baselines, including FNO, significantly on most benchmarks. This also holds for the considered out-of-distribution testing tasks which ascertain the ability of the models to generalize to unseen data in a zero-shot manner.

Comparison to Related Work. We emphasize that our construction of CNO follows the theoretical prescription of recent paper [2] on enforcing structure preserving continuous-discrete equivalence . CNO is a representation equivalent neural operator , with respect to spaces of bandlimited functions, in the sense of [2]. Another motivating work for us is [22], see also [58], where the authors modify CNNs to eliminate (or reduce) aliasing errors in the context of image generation. We adapt the construction of [22] to our setting and deploy the resulting architecture in a very different context from that of [22], namely that of operator learning for PDEs rather than image generation. Moreover, we also instantiate CNO with a very different operator UNet architecture than that proposed in [22]. We would also like to mention related work on using CNNs for solving PDEs such as [1, 16] and emphasize that in contrast to CNO, they lack suitable notions of continuous-discrete equivalence. Finally comparing CNO to the widely used FNO model, we observe that unlike FNO which can fail to enforce CDE (see [14, 2] and Figure 2(right)), CNO preserves continuous-discrete equivalence. Moreover, the convolution operator in CNO is local in space, in contrast to convolution in Fourier space for FNO. The detailed empirical comparison presented here demonstrates CNO can outperform FNO and other baselines on many different metrics namely performance, computational efficiency, resolution invariance, out-of-distribution generalization as well as data scaling, paving the way for its widespread applications in science and engineering.

Limitations and Future Work. We have presented CNO for operators on an underlying twodimensional Cartesian domain. The extension to three-space dimensions is conceptually straightforward but computationally demanding. Similarly, extending to non-Cartesian domains will require some form of transformation maps between domains, for instance reworking those suggested for FNO in [32, 55] can be readily considered. Adapting CNO to approximate trajectories (in time) of time-dependent PDEs, for instance by employing it in an auto-regressive manner, is another possible extension of this paper. At the level of theoretical results, we believe that the generic framework of [11] can be adapted to show that not only does CNO approximate a large class of PDEs universally, it does so without incurring any curse of dimensionality, as shown for DeepONets in [26] and FNOs in [24]. Finally, adapting and testing CNO for learning operators, beyond the forward solution operator of PDEs is also interesting. One such direction lies in efficiently approximating PDE inverse problems , for instance those considered in [44].

References

I. Ayed, E. de Bézenac, A. Pajot, J. Brajard, and P. Gallinari. Learning dynamical systems from partial observations. CoRR , abs/1902.11136, 2019.

F. Bartolucci, E. de Bézenac, B. Raonic, R. Molinaro, S. Mishra, and R. Alaifari. Are neural operators really neural operators ? frame theory meets operator learning. Technical Report 2023-21, Seminar for Applied Mathematics, ETH Zürich, Switzerland, 2023.

J. Bell, P. Collela, and H. M. Glaz. A second-order projection method for the incompressible Navier-Stokes equations. J. Comput. Phys. , 85:257-283, 1989.

M. Bertero, P. Bocacci, and C. De Mol. Introduction to inverse problems in imaging . CRC press, 2021.

K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics , 7:121-157, 2021.

N. Boullé, Y. Nakatsukasa, and A. Townsend. Rational neural networks. Advances in Neural Information Processing Systems , 33:14243-14253, 2020.

S. Cai, Z. Wang, L. Lu, T. A. Zaki, and G. E. Karniadakis. DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics , 436:110296, 2021.

S. Cao. Choose a transformer: Fourier or galerkin. In 35th conference on neural information processing systems , 2021.

T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks , 6(4):911-917, 1995.

T. De Ryck, S. Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks , 2021.

T. De Ryck and S. Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Q. Delfosse, P. Schramowski, M. Mundt, A. Molina, and K. Kersting. Adaptive rational activations to boost deep reinforcement learning. arXiv preprint arXiv:2102.09407 , 2021.

L. C. Evans. Partial differential equations , volume 19. American Mathematical Soc., 2010.

V. Fanaskov and I. Oseledets. Spectral neural operators. arXiv preprint arXiv:2205.10573v1 , 2022.

I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning , volume 1. MIT Press, 2016.

J. K. Gupta and J. Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling, 2022.

E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse problems , 34, 2018.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv: 1512.03385 , 2015.

J. S. Hesthaven, S. Gottlieb, and D. Gottlieb. Spectral methods for time-dependent problems , volume 21. Cambridge University Press, 2007.

P. Isola, J.-Y . Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition , 2017.

G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics informed machine learning. Nature Reviews Physics , pages 1-19, may 2021.

T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Aliasfree generative adversarial networks. Advances in Neural Information Processing Systems , 34:852-863, 2021.

G. Kissas, J. H. Seidman, L. F. Guilhoto, V. M. Preciado, G. J. Pappas, and P. Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research , 23(215):163, 2022.

N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research , 22:Art-No, 2021.

N. Kovachki, Z. Li, B. Liu, K. Azizzadensheli, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481v3 , 2021.

S. Lanthaler, S. Mishra, and G. E. Karniadakis. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications , 6(1):tnac001, 2022.

S. Lanthaler, S. Mishra, and C. Parés-Pulido. Statistical solutions of the incompressible euler equations. Mathematical Models and Methods in Applied Sciences , 31(02):223-292, Feb 2021.

S. Lanthaler, R. Molinaro, P. Hadorn, and S. Mishra. Nonlinear reconstruction for operator learning of pdes with discontinuities. In International Conference on Learning Representations , 2023.

Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks , 6(6):861867, 1993.

Z. Li, D. Z. Huang, B. Liu, and A. Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries, 2022.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2021.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural operator: Graph kernel network for partial differential equations. CoRR , abs/2003.03485, 2020.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, A. M. Stuart, K. Bhattacharya, and A. Anandkumar. Multipole graph neural operator for parametric partial differential equations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 6755-6766. Curran Associates, Inc., 2020.

Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794 , 2021.

Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 , Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 11966-11976. IEEE Computer Society, 2022.

Z. Long, Y. Lu, X. Ma, and B. Dong. Pde-net: Learning pdes from data. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 3214-3222. PMLR, 2018.

L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3(3):218-229, 2021.

K. O. Lye, S. Mishra, D. Ray, and P. Chandrashekar. Iterative surrogate model optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networks. Computer Methods in Applied Mechanics and Engineering , 374:113575, 2021.

Z. Mao, L. Lu, O. Marxen, T. Zaki, and G. E. Karniadakis. DeepMandMnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neuralnetwork approximation of operators. Preprint, available from arXiv:2011.03349v1, 2020.

D. A. Masters, N. J. Taylor, T. Rendall, C. B. Allen, and D. J. Poole. Geometric comparison of aerofoil shape parameterization methods. AIAA Journal , pages 1575-1589, 2017.

A. Molina, P. Schramowski, and K. Kersting. Pad \ 'e activation units: End-to-end learning of flexible activation functions in deep networks. arXiv preprint arXiv:1907.06732 , 2019.

R. Molinaro, y. Yang, E. Engquist, and S. Mishra. Neural inverse operators for solving pde inverse problems. arXiv:2301.11167 , 2023.

J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, p. Hassanzadeh, K. Kashinath, and A. Anandkumar. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214 , 2022.

P. Petersen and F. V oigtlaender. Equivalence of approximation by convolutional neural networks and fully-connected networks. Proceedings of the American Mathematical Society , 148(4):15671581, 2020.

M. Prasthofer, T. De Ryck, and S. Mishra. Variable input deep operator networks. arXiv preprint arXiv:2205.11404 , 2022.

A. Quarteroni and A. Valli. Numerical approximation of Partial differential equations , volume 23. Springer, 1994.

M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

T. Schanze. Sinc interpolation of discrete periodic signals. IEEE Transactions on Signal Processing , 43(6):1502-1503, 1995.

J. H. Seidman, G. Kissas, P. Perdikaris, and G. J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. arXiv preprint arXiv:2206.03551 , 2022.

M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

M. Telgarsky. Neural networks and rational functions. In International Conference on Machine Learning , pages 3387-3393. PMLR, 2017.

A. Tran, A. Mathews, L. Xie, and C. S. Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations , 2023.

M. Unser. Sampling-50 years after shannon. Proceedings of the IEEE , 88(4):569-587, 2000.

M. Vetterli, J. Kovacevic, and V. Goyal. Foundations of Signal Processing . Cambridge University Press, 2014.

S. Wang, S. Suo, W. Ma, A. Pokrovsky, and R. Urtasun. Deep parametric continuous convolutional neural networks. CoRR , abs/2101.06742, 2021.

S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets. arXiv preprint arXiv:2103.10974 , 2021.

S. E. Wei. Aliasing-free nonlinear signal processing using implicitly defined functions. IEEE Access , 10:76281-76295, 2022.

R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. CoRR , abs/2110.00476, 2021.

J. Yang, Q. Du, and W. Zhang. Uniform l p-bound of the allen-cahn equation and its numerical discretization. International Journal of Numerical Analysis & Modeling , 15, 2018.

Y. Zhu and N. Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics , 336:415-447, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a UNet-like architecture for learning solution operators of PDEs. The architecture is assembled from building blocks analogous to the classical UNet architecture, but individual components respect a continuous-discrete equivalence. This makes the proposed architecture a representation equivalent neural operator in the sense of a recent paper (Bartolucci et al., 2023). The authors show that under some continuity assumptions the proposed method can learn the solution operators to a variety of PDEs. They also compare the in- and out-of-distribution performance of their method to a variety of baselines for a suite of different types of PDEs.\n\nSTRENGTHS:\n- There is a large body of work on methods and architectures for operator learning which this paper builds up on. Having architectures that respect the continuous-discrete equivalence seems very significant to me.\n- The presentation is clear and the paper overall well written.\n- The proposed CNO architecture improves the overall performance for a comprehensive suite of problems and appropriate baselines.\n- Many evaluations and ablation studies are included in the appendix.\n\nWEAKNESSES:\n- As discussed in the related work section, the building blocks that make up the CNO architecture have been introduced in previous papers (Karras et al. 2021, Alias-free generative adversial networks). This slightly weakens the novelty of the paper.\n- Many definitions are adopted from a relatively recent paper (Bartolucci et al. 2023), which to my knowledge is only available as a preprint at the moment.\n\n- The modifications to the UNet architecture introduce additional up-sampling and down-sampling layers, which increase the training time significantly compared to other methods. This needs to be kept in mind. Also, the implementation of CNOs is much more complicated than of e.g. FNOs due to the windowed-sinc filters.\n\n- I found the evaluation across resolutions somewhat unclear and partially misleading. The appendix goes into more detail here, but the results in the main paper show a single NS case that resulted from a downsampled solution (hence no high-frequency details exist in the targets). In this case the CNO seems to show a constant error, which is dubious for meaningful real world cases where higher resolutions naturally would exhibit structures that aren't resolved with lower resolutions. I think it will be important for future versions to clarify this, and replace figure 2 with one of the other two cases from the appendix.\n\nQUESTIONS:\n- The practical implementation of the interpolation filters uses windowed-sinc. In theory, doesn't this break the continuous-discrete equivalence?\n\n- Regarding \"operator\" networks, I was wondering why the authors didn't compare to a fully convolutional ResNet (along the lines of Solver-in-the-loop Um'21 or the accelerated CFD from Kochkov'21); that architecture should more naturally extend to different resolutions than a Unet. Can the authors comment on this omission?\n\n- Can the authors provide details on the performance, especially how much slower the CNO is compared to the regular Unet?\n\nLIMITATIONS:\nLimitations are only discussed as a future work discussion. I don't think this is sufficient. E.g., a clear discussion of the performance impact is completely missing as far as I can tell.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
In this paper, the authors propose a new operator learning framework called Convolution Neural Operator which although works on discrete space, satisfies the property of Continuous-Discrete Equivalence (CDE) property. They define CNO over a UNet architecture and provide some universal approximation proofs. Further, they come up with a Benchmark of representative PDEs and compare all the existing methods and present superior performance of CNO over other methods.

STRENGTHS:
+ Novel idea of CNOs
+ Theory provided for soundness
+ Representative Benchmark Dataset and making it public in zenodo is also good.
+ Very neat and clean code.

WEAKNESSES:
- Some notations are messed up or confusing to follow. e.g. r in line 82.
- While the performance of these models is compared... the computation requirements like memory or training time are not compared. I think if we are talking about benchmarking, it is appropriate to give the complete story including the computational requirements.
- Some ablation studies on whether using CNOs reduces the architecture/compute requirements compared to just using a CNN or FNO is missing.

QUESTIONS:
See above in weaknesses*

LIMITATIONS:
Yes. The authors have accurately identified the limitations. There are no negative societal impact for their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper under review proposes a CNN architecture as a neural operator for solving PDEs via neural networks.  The goal is to preserve the underlying continuous structure while being implemented with discrete convolutional architecture, based on U-Net.  The idea is to consider the space bandlimited functions in Sobolev spaces.  Defining a convolution layer involves a discrete convolution, upsampling/downsampling that requires sinc interpolation to avoid aliasing, and an activation layer that upsamples, activates and then downsamples to maintain the original band-limitation.  The authors prove in supplementary material that their approach can approximate the solution to a PDE with arbitrary accuracy with the CNN architecture based on their neural operator.  Experiments are done on solving several PDEs, with experiments favorable to the authors' method against state of the art for most PDEs considered.\n\nSTRENGTHS:\n- The architecture based on CNNs potentially simplifies e.g., FNO operators where operations are done in the Fourier domain without the use of down-sampling - which can potentially be more expensive than the authors' proposed method.\n- Rigorous mathematical treatment of the interplay between continuous and discrete operators.\n- Experiments show higher accuracy compared to SOA in PDE solving with NN.\n\nWEAKNESSES:\n- No evaluation of efficiency; is there a speed advantage of the proposed method (e.g., compared to FNO)?\n- Evaluation of parameter sizes is in supplementary and there isn't a clear trend; I would have thought this method would require fewer parameters than e.g., FNO; but in many cases this isn't so.\n- I would think efficiency in the sense above is a key differentiator compared to SOA, but this isn't the case at least looking at supplementary.\n- The authors make use of sinc interpolation, which is the ideal interpolator for sequences of infinite length.  In practice, you are operating on finite length data, for which the sinc interpolater is not the right one.  How applicable is the theory presented to the case of finite length data?\n- Prop 2.1 talks about a representation equivalent operator and the definition is cited in a reference.  Please specify the definition in the paper.\n- Theory is poorly presented in the paper; everything is in supplementary.  At least the key ideas should be in the main paper.\n- In general many of the key parts that should be in the paper are in supplementary.\n- I find the authors' proposed idea to set a standard for benchmarking misguided.   There is a laundry list of PDEs to approximate in the benchmark, however, I doubt any one architecture would be good for all PDEs.  Each PDE has particular properties that one would want to aim to preserve (e.g., conservation laws in some PDEs) that may not be relevant to others.  So the implication that one architecture should do well on all PDEs seems mis-guided.  I do not think this should be an accepted standard benchmarking.\n\nQUESTIONS:\n- Bandlimited: many solutions to PDEs will have shocks, discontinuities, etc.  How relevant is this assumption?\n\nLIMITATIONS:\nYes, discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper re-introduces convolution neural networks to the operator learning setting. It uses interpolation including up-sampling and downsampling layer to make sure the convolution layer is well-defined in the function space. The paper also discuss the space of band-limited functions and the trade off between continuous-discrete equivalence and representation power of the infinite dimension function space. The paper proves an approximation theorem for the CNO model and compared it across many partial differential equations.\n\nSTRENGTHS:\nThe convolution neural operator (CNO) takes the advantages of the efficient of the conventional CNNs and UNets methods, meanwhile it also satisfies the resolution-invariant and representation-equivalent properties. The work shows an approximation theorem that the CNO models can approximate any continuous solution operator, and the numerical experiments show it has a comparative performance compared to other machine learning methods such as  UNet, DeepONet, and FNO.\n\nWEAKNESSES:\nI have a few questions and concerns, mainly in the trade-off between the band-limited functions and Lp/Hp space.\n1. The paper claim the CNO models can only learn band-limited functions, however, the target PDE system are intrinsically infinitely dimensional system (in Lp/Hp). It means, given a fixed number of parameters (size of the model), as the number of training sampling and the qualify(resolution) of training dataset increase, the truncation error will dominate and the bandlimited class of operator will underperform the band-unlimited models.\n2. The band-limited operator is, by definition, linear-reconstruction of the chosen basis, meaning it can only learn the coefficient but unable to learn the basis. The linear-reconstruction are less efficient compared to non-linear reconstruction model, as discussed in [1].\n3. The experiments (Table 1) are designed in a manner that CNO dominates all other methods. It shows the potential of CNO, but the results might be biased. It's certainly reasonable that the band-limited CNO outperform band-unlimited model on super-resolution tasks. However, for in-distribution problems, if the resolution is fixed and the dataset is sufficient, then Unet should be equivalent to CNO, right? And for these non-smooth problems, the band-unlimited could be more expressive than CNO. These cases might not be sufficiently considered in the experiments design. It is great to introduce these new experiments, but it would be better to include as least one of the previous standard benchmark, for example, the Burgers or Darcy equation dataset from [2].\n4. While the perspective is new and interesting, it's hard to say the proposed model is very novel. The architecture is highly similar to the previous UNet model + StyleGAN3 de-aliasing trick.\n\n[1] Lanthaler, Samuel, et al. \"Nonlinear reconstruction for operator learning of pdes with discontinuities.\" arXiv preprint arXiv:2210.01074 (2022).\n\n[2] Li, Zongyi, et al. \"Fourier neural operator for parametric partial differential equations.\" arXiv preprint arXiv:2010.08895 (2020).\n\nQUESTIONS:\nAs discussed in page 4, CNO uses upsampling and downsampling to reduce the aliasing error, but it's also claimed that the activation layer will introduce the aliasing error. This is overcome by \"Implicitly assuming that $\\bar{w}$ is large enough\". Can the author give some examples of the activation such that this is satisfied? Speaking of frequencies expansion, for ReLU or leaky ReLU, $\\sigma(x)$  is non-smooth. Even if $\\sigma$ is a k-frequency function, $\\sigma \\circ \\sigma$ can be $k^2$-frequency, right? So after several layers it easily goes unbounded.\n\nFigure 2 is also a bit surprising. Can CNO achieve constant super-resolution error curve for problem like Navier-Stokes? The FNO error also seems a bit too high. How many modes are used in FNO? If FNO is designed with a reasonably small number of modes and no padding (padding may cause error in resolution), FNO may also get flatten curve.\n\nLIMITATIONS:\nThe author discussed some limitations, but it's also good to discuss the trade-off between using band-limited functions and Lp/Hp space. I think both side have their advantages and disadvantages.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}