Convolutional Neural Operators for robust and accurate learning of PDEs
Authors: Bogdan Raonic
Roberto Molinaro
Tim De Ryck
Tobias Rohner
Francesca Bartolucci
Rima Alaifari
Siddhartha Mishra
Emmanuel de Bezenac
Venue: NeurIPS 2023 poster (2023)
Keywords: PDEs
Neural Operators
Scientific Machine Learning
Convolutional Neural Networks
TL;DR: A new convolution based neural operator architecture is proposed for accurate and robust learning of PDEs.

Abstract: Although very successfully used in conventional machine learning, convolution based neural network architectures -- believed to be inconsistent in function space -- have been largely ignored in the context of learning solution operators of PDEs. Here, we present novel adaptations for convolutional neural networks to demonstrate that they are indeed able to process functions as inputs and outputs. The resulting architecture, termed as convolutional neural operators (CNOs), is designed specifically to preserve its underlying continuous nature, even when implemented in a discretized form on a computer. We prove a universality theorem to show that CNOs can approximate operators arising in PDEs to desired accuracy. CNOs are tested on a novel suite of benchmarks, encompassing a diverse set of PDEs with multi-scale solutions and are observed to significantly outperform baselines, paving the way for an alternative framework for robust and accurate operator learning.

Convolutional Neural Operators for robust and accurate learning of PDEs
Bogdan Raonić
Roberto Molinaro'
Tim De Ryck'
Tobias Rohner!
Francesea Bartolucci Rima Alaifari* Siddhartha Mishra Emmanuel Bézenac"
Seminar for Applied Mathematics, ETH, Zurich, Switzerland ETH AI Center, Zurich, Switzerland
Delft University of Technology, Netherlands
Abstract
Although very successfully used in conventional machine learning, convolution based neural network architectures believed to be inconsistent in function space- have been largely ignored in the context of learning solution operators of PDEs. Here, we present novel adaptations for convolutional neural networks to demon- strate that they are indeed able to process functions as inputs and outputs. The resulting architecture, termed as convolutional neural operators (CNOs), is designed specifically to preserve its underlying continuous nature, even when implemented in discretized form on computer. We prove universality theorem to show that CNOs can approximate operators arising in PDEs desired accuracy. CNOs are tested on novel suite of benchmarks, encompassing a diverse set of PDEs with possibly multi-scale solutions and are observed to significantly outperform baselines, paving the way for alternative framework for robust and accurate operator learning.
Introduction.
ubiquitous mathematical models in the sciences and engineering. Solving a PDE amounts to (approximately) computing the so-called solution operator that maps function space inputs such initial and boundary conditions, coefficients, source terms etc, to the PDE solution which also belongs to a suitable function space. Well-established numerical methods such finite differences, finite elements, finite volumes and spectral methods (see 48) ) have been very successfully used for many decades to approximate PDE solution operators. However, the prohibitive computational cost of these methods, particularly in high dimensions and for many query problems such as UQ, inverse problems, PDE-constrained control and optimization, necessitates the design of fast, robust and accurate surrogates. This provides the rationale for the use of data-driven machine learning methods for solving PDFs 21
Partial Differential Equations (PDEs) 1K
As operators are the objects interest in solving PDEs, learning such operators from data, which is loosely termed as operator learning, has emerged as a dominant paradigm in recent years for the applications of machine learning PDEs. A very partial list of architectures for operator learning
include operator networks [9], DeepONets 39 and its variants [41 7]. PCA-net (5] neural operators
25 such as graph neural operator 34 Multipole neural operator 35 and the very popular Fourier
Neural Operator (33) and its variants [36
45). VIDON [47). spectral neural operator [4], LOCA
23 NOMAD [52] and transformer based operator learning architectures
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Despite the considerable success of the recently proposed operator learning architectures, several pressing issues remain be addressed. These include, but are by no means restricted to, limited expressivity for some of these algorithms [28] to aliasing errors for others [14] to the very fundamental issue of possible lack of consistency in function spaces for many of them. As argued in recent paper operator learning algorithm or representation equivalent neural operator has to respect some form of continuous-dsserer equivalence (CDE) in order to learn the underlying operator, rather than just a discrete representation of it. Failure to respect such CDE can lead to the so-called aliasing errors and affect model performance at multiple discrete resolutions.
(2) gructtre-preservvng
Despite many attempts, see [6] and references therein, the absence of suitable CDE, resulting in aliasing errors, has also plagued the naive use of convolutional neural networks (CNNs) in the context of operator learning, see 663 33 2 on how using CNNs for operator learning leads to results that heavily rely on the underlying grid resolution. This very limited use of Convolution (in physical space) based architectures for operator learning stands in complete contrast to the fact that CNNs 30 and their variants are widely used architectures for image classification and generation and
in other contexts in machine learning [29 37 61.. Moreover, CNNs can be thought of as natural generalizations of the foundational finite difference methods for discretizing PDEs 17 38 Given their innate locality, computational and data efficiency, ability to process multi-scale inputs and outputs and the availability of a wide variety of successful CNN architectures in other fields, it could be very advantageous to bring This is precisely the central point of the current paper where we make the following contributions,
algorithms back into the reckoning for operator learning.
We propose novel modifications to CNNs order to enforce structure- pre.errringg continuous- discrete equivalence and enable the genuine, alias-free, learning of operators. The resulting architecture, termed Convolutional Neural Operator (CNO), is instantiated novel operator adaptation of the widely used U-Net architecture.
In addition to showing that CNO is a representation equivalent neural operator in the sense of [2, we also prove a universality result to rigorously demonstrate that CNOs can approximate the operators, corresponding to large class of PDEs, to desired accuracy.
We test CNO on novel set of benchmarks, that we term as Representative PDE Benchmarks (RPB), that span across a variety of PDEs ranging from linear elliptic and hyperbolic to nonlinear parabolic and hyperbolic PDEs, with possibly multiscale solutions. We find that CNO is either on-par or outperforms the tested baselines on all the benchmarks, both when testing in-distribution as well as in out-o--distribution testing.
Thus, we present a new CNN-based operator learning model, with desirable theoretical properties and excellent empirical performance, with the potential to be widely used for learning PDEs.
2 Convolutional Neural Operators.
Scallee:
Groat
P Block
Pk
E
De
(uu)
Blokk
ww.
S-Rr
Actintion
D
22
Laysr
Figure 1: Schematic representation of CNO (..3 as modified U-Net with sequence of layers (each identified with the relevant operators on the right, see Section (2) mapping between bandlimited functions. Rectangles represent multi-channe signals. Larger the height, larger is the resolution. Wider the rectangles, more channels are present.

---

Setting. For simplicity of the exposition, we will focus here on the two-dimensional case by specifying the underlying domain D T being the 2-d torus. Let H" H" (D, C Z and y (D, Rdd) be the underlying function spaces, where H (D,) are Sobolev spaces of order r and s. Without loss of generality, we set r = s hereafter. Our aim would be to approximate
continuous operators G X + y from data pairs (u, G (u.)) = & x. that there exists a modulus of continuity for the operator i.c.,
We further assume
|| (u) (v)lly Sw (1]u-ollz) VU,EE,
(2.1)
with 1 R being monotonically increasing function with limy-so w(y) The underlying operator G can correspond solution operators for PDEs (see Section for the exact setting) but is more general than that and encompasses examples such those arising in inverse problems, for instance in imaging
Bandlimited Approximation. As argued in a recent paper 2 Sobolev spaces such as H" are, in sense, too large to allow for any form of saniinoou-ddscrelt equivalence (CDE), i.e. equivalence between the underlying operator and its discrete representations, which is necessary for robust operator learning. Consequently, one has to consider smaller subspaces of H" which allow for such CDEs. In this respect, we choose the space of bandlimited functions [57 defined by,
Bar (m((D) = (f L2(D)) suppf S [[w,, w/ w)}}
(2.2)
for some w > 0 and with f denoting the Fourier transform of f. It is straightforward to show using (.. (see SMA.T) that for any e >> O, there exists a w, large enough depending on r, and a continuous operator G Be (D) > B., such that g 9 < E, with || denoting the corresponding operator norm. In other words, the underlying operator G can be approximated arbitrary accuracy by the operator g" that maps between sand-limited spaces. Consequently, as shown SMA.2, one can readily define discrete versions of G using the underlying sinc basis for bandlimited functions and establish a continuous-discrete equivalance for it.
Definition of CNO. Given the above context, our goal will be to approximate the operator g in a structure- preserving manner i.e., as the underlying operator maps between spaces of bandlimited functions, we will construct our operator approximation architecture to also map bandlimited functions bandlimited functions, thus respecting the continuous-discrete equivalence. To this end, denote the operator G (D) - (D) as convolutional neural operator (CNO) which we define as a compositional mapping between functions
g P(u) to ou an..... H Q(u)) = T.
(2.3)
where
U1+ Pi E o Ki(),), 1 1 --1-
(2.4)
we see that first, the input function u E (D) is lifted to the latent space of bandlimited functions through lifting layer:
From 2.3)
P: fu BT (D, + (vo E B (D,
Here, do > dx is the number of channels in the lifted, latent space. The lifting operation is performed by a convolution operator which will be defined below.
Then, the lifted function is processed through the composition of a series of mappings between functions (layers), with each layer consisting of three elementary mappings, i.e., Pi is either the upsampling or downsampling operator, IG₃ is the convolution operator and E, is the activation operator. These elementary operators are defined below and inspired by the modifications of CNNs for image generation in (22). Finally, the last output function in the iterative procedure et, is projected to the output space with projection operator Q, defined
Q: {uL E (D, + B,
The projection operation is also performed by convolution operator defined below.
3

---

Convolution Operator. For simplicity of exposition, we will present the single-channel version of the convolution operator X, here. See SM3 for the multi-channel version for this and other operators considered below. Convolution operations are performed with discrete kernels
K Kig Ōzyy
2,1=1
defined the x uniform grid D with grid size < 1/2ur, in-order satisfy the requirements of the Saitaker-ShanonnKoteeein
sampling theorem [56]. and ba being the resulting grid points, N being the kernel size and 82 denoting the Dirac measure at point æ E D. The convolution operator for a single-channe Kw Bu (D) + Bw (D) is defined by
&
fw) = (Kw fa()) (w - v)f(yddy = E
kzff2 - (zj)), Vx E D,
where the last identity arises from the fact that f Bw Thus, convolution operator is directly parametrized in physical space, in contrast to the Fourier space parametrization a convolution in the FNO architecture of [33.. Hence, our parametrization is of local nature.
Upsampling and Downsampling Operators. For some w >> w, we can upsample function B to the higher band B by simply setting,
Uw, Bw (D) - Be(D), Uw,=f(2) = fxx), Vr D. On the other hand, for by setting Dw.w: :Bu(D) - (D)),, defined by
w,
downsample function B to the lower band B
10 2
11' 2 1 - () Vz E DD
D f
(h. * f)())
w
w
where is the convolution operation on functions defined above and leg is the so-called interpolation sine filter:
Aiw ( .i) sine(2waro) sinn(2w.r)), (æo, 21) E R
(2.5)
Naively, one apply the activation function pointwise any function. How- ever, is well-known that such an application will no longer respect the band-limits of the underlying function space and generate aliasing errors [22] [4 2]. In particular, nonlinear activations can generate features at arbitrarily high frequencies. As our aim is to respect the underlying CDE, we will modulate the application of the activation function so that the resulting outputs fall within desired band limits. To this end, we first upsample the input function f E B,, to a higher bandlimit w > w, then apply the activation and finally downsample the result back the original bandlimit (See Figure I). Implicitly assuming that w is large enough such that (Bw) C Bw, we define the activation layer in (2.3 as,
Activation Layer.
Ev.m - (D),,, Ew,m((x) Dm, (a o (n, Vr E D.
(2.6)
Instantiation through an Operator U-Net architecture. The above ingredients are assembled together in the form of an Operator U-Net architecture that has bandlimited functions as inputs and outputs. In addition to the blocks that have been defined above, we also need additional ingredients, namely incorporate skip connections through ResNet blocks of the form, Rw,, B,, (D, (D) B., (D, (Rd) such that
(w) = u KC Ew, (w), VU B (w,
(2.7)
We also need the so-called Invariant blocks of the form, Inn B,, (D, Rd) + (., such that
(v) = Ew,w (w), Ye E (D,,
(2.8)
Finally, all these ingredients are assembled together in modified Operator U-Net architecture which graphically depicted in Figure 1. As seen from this figure, the input function, say u E B,, Rdx) is first lifted and then processed through series of layers. Four types of blocks are used i.c.

---

downsampling (D) block corresponding to using the downsampling operator D as the P in (2.4). upsampling (U) block corresponding using the upsampling operator Li as the in (2.4). ResNet (R) block corresponding to (2.7) and Invariant (I) block corresponding to (2.8) Each block takes band-limited function as input and returns another band-limited function (with the same band) as the output. Finally, U-Net style patching operators, which concatenate outputs for different layers as additional channels are also used. As these operations act only in the channel width and leave the spatial resolution unchanged, they conform to the underlying bandlimits. Thus, CNO takes function input and passes it through a set encoders, where the input downsampled in space but expanded in channel width and then processed through a set of decoders, where the channel width is reduced but the space resolution is increased. At the same time, encoder and decoder layers (at the same spatial resolution or band limit) are connected through additional ResNet blocks. Thus, this architectural choice allows for transferring high frequency content via the skip connections, before filtering them out with the sinc filter content is not just recreated with the activation function, but also modified through the intermediate networks. Consequently, we build genuinely multiscale operator learning architecture.
go deeper into the encoder. Hence, the high frequency
Continuous-Discred Equivalence for CNO. We have defined CNO (2.3) as an operator that maps bandlimited functions to bandlimited functions. In practice, like any computational algorithm, CNO has to be implemented in a discrete manner, with discretized versions of each of the above- defined elementary operations being specified in A.4 Given how each of the elementary blocks (convolution, up- and downsampling, activation, ResNets etc) are constructed, we prove the following proposition (in AM
(D, Rdx) + B (D, 2.3 is a Representation equivalent neural operator ReNO, in the sense of (21, Definition 3.4.
Proposition 2.1. Convolutional Neural Operator G
Further details about the notion of ReNOs is provided in SM and refer the reader 22. where this concept is presented in great detail and the representation equivalence of CNO is discussed. In particular, following (2), representation equivalence implies that CNO satisfies form of resolution invariance, allowing it to be evaluated on multiple grid resolutions without aliasing errors.
3 Universal Approximation by CNOs.
We want to prove that a large class of operators, stemming from PDEs, can be approximated to desired accuracy by CNOs. To this end, we consider the following abstract PDE in the domain D = TT,
((w) 0, 0, B(u)= = 0,
(3.1)
with L being differential operator and B boundary operator. We assume that the differential operator L only depends on the coordinate x through a coefficient function a E H" (D).. The
corresponding solution operator denoted by G : X C H" + HD) : a + u, with being the solution of the PDE (3.1). We assume that Gi is continuous. Moreover, also assume the following modulus of continuity,
[|g -a) 9' S (Ila a'll==((T)))
(3.2)
for some p (2, and 0 c 1, and where 10, [0, oo) is a monotonously
increasing function with limy w()) 0. (3.2) is automatically satisfied if X is compact and Gt is continuous. Under these assumptions, we have the following universality theorem for CNOs (2.3).
Theorem 3.1. Let a E No and p E {2, bo) as in (.2). > max(o, 2/p} and B > 0. For any E > O and any operator gt, as defined above, there exists a CNO 9 such that for every a E X with ||a|| (D)) S B it holds,
(C)(()) - G(a)lle<()) <.
(3.3)
In fact, we will prove more general version of this theorem in SM where we also include additional source terms in the PDE 3.1
5

---

Experiments.
Training Details and Baselines. We provide a detailed description of the implementation CNO and the training (and test) protocol for CNO well as all the baselines in SM SM To ensure level playing field among all the tested models for each benchmark, we follow an ensemble training procedure by specifying range for the underlying hyperparameters for each model and randomly selecting a subset of the hyperparameter space. For each such hyperparameter configuration, the corresponding models are trained the benchmark and the configuration with smallest validation error is selected and the resulting test errors are reported, allowing us to identify and compare the best performing version of each model for every benchmark. We compare CNO with the following baselines: two very popular operator learning architectures, namely DeepONet (DON) (39 and
FNO 33 transformer based operator- learnnnn- architecture, i.e., Galerkin Transformer (GT)
feedforward neural network with with residual connections (FFNN) 18] and the very widely-used ResNet and U-Net [50] architectures.
Table 1: Relative median L test errors, for both in- and out-of-distribution testing, for different benchmarks and models.
In/ut
FFNN
GT
UNet
ResNet
DON
FNO
CNO
Poisson
In 5.74%
2.77%
0.71%
0.43%
12.92%
4.98%
0.21 %
Equation
Out
5.35%
2.84%
1.27%
1.10%
9.15%
7.05%
0.27%
Wave
In 2.51%
1.44%
1.51%
0.79%
2.26%
1.02%
0.63%
Equation Out 3.01%
1.79%
2.03%
1.36%
2.83%
1.77%
1.17%
Smooth
7.09%
0.98%
0.49%
0.39%
1.14%
0.28%
0.24%
Transport Out 650.6%
875.4%
1.28%
0.96%
157.2%
3.90%
0.46%
Discontinuous In 13.0%
1.55%
1.31%
1.01%
5.78%
1.15%
1.01 %
Transport Out 257.3%
22691.19 1.1 %
1.35%
1.16%
117.1%
2.89%
1.09%
Allen-Cahn
In 18.27%
0.77%
0.82%
1.40%
13.63%
0.28 %
0.54%
Equation Out 46.93%
2.90%
2.18%
3.74%
19.86%
1.10%
2.23%
Navier-Stokes
8.05%
4.14%
3.54%
3.69%
11.64%
3.57%
2.76%
Equations Out 16.12%
11.09%
10.93%
9.68%
15.05%
9.58%
7.04%
Darcy
2.14%
0.86%
0.54%
0.42%
1.13%
0.80%
0.38%
Flow
Out 2.23%
1.17%
0.64%
0.60%
1.61%
1.11%
0.50 %
Compressible
In 0.78%
2.09%
0.38%
1.70%
1.93%
0.44%
0.35 %
Euler
Out
1.34%
2.94%
0.76%
2.06%
2.88%
0.69%
0.59 %
Representative PDE Benchmarks (RPB). Given the lack of consensus on standard set of benchmarks for machine learning PDEs, we propose new suite of benchmarks here. Our aims in this regard are to ensure i) sufficient diversity among the types of PDE considered, ii) access to training and test data readily available for rapid prototyping and reproducibility and iii) intrinsic computational complexity of problem to make sure that it is worthwhile to design fast surrogates to classical PDE solvers for a particular problem. In other words, we will only consider PDEs where classical PDE solvers only resolve the underlying operator fine enough grids. To meet these requirements, we will not consider PDEs in one space dimension as traditional numerical methods already quite fast for them. On the other hand, it is hard to obtain and store data for problems in three dimensions, due to computational expense of traditional methods. The sweet spot is achieved by considering PDEs in space dimensions. We further restrict to Cartesian domains here as all models can be readily evaluated in this setting. In addition to including a diverse set of PDEs, we only consider problems with sufficiently many spatial and temporal scales. Otherwise, traditional numerical solvers can approximate the underlying PDE on very coarse grids and is not worthwhile design surrogates (see SM C.3.8 for discussion in this context on a widely used Navier-Stokes benchmark). With these considerations in mind, we present the following subset of Representative PDE Benchmarks RPB.
'The code can be found at http://hhh...oooaaoooo////aaaaae

---

Poisson Equation.
This prototypical linear elliptic PDE is given by,
f. in ulap = O.
(4.1)
The solution operator Gt f + u, maps the source term f to the solution u. With source term,
f(z,8)
Σ K JJ=1
oij (12 + sin(rix) sin(Tjy), V(a.y) e D,
(4.2)
with r = -055, the corresponding exact solution can be analytically computed (see SM C.3.I and represents K- spatial scales. For training the models, we fix K = 16 in (42 and choose dij to be Li.d. uniformly distributed from [-1, 1] (See SMD for a representation of the inputs and outputs of Gt). This multiscale solution needs fine enough grid size to be approximated accurately by finite element methods, fitting our complexity criterion for benchmarks. In addition to in-distribution testing, we also consider an out-of-distribution testing task by setting K = 20 in (4.2). This will enable us to evaluate the ability of the models to generalize to inputs (and outputs) with frequencies higher than those encountered during training.
Wave Equation. This prototypical linear hyperbolic PDE is given by
Veee A² = 0, in D x (0,T), ua(a, y) = f(x, v),
(4.3)
with a constant propagation speed 0.1. The underlying operator G : ʃ + ul., 7)) maps the initial condition f into the solution at the final time. If we consider initial conditions to be given by (4.) with = 1, then one can explicitly compute the exact solution (see SM C.3.2) to represent a multiscale standing wave with periodic pulsations (depending on K) in time. The training and in-distributiio test samples are generated by setting T = 5, K 24 and dij to be i.i.d. uniformly distributed from [-1, 1] (See SMD for input and output samples). For out-of-distribution testing, change the exponent of decay of the modes in (.2) to 0.85 and K = 32, in order to test the ability of the models to generalize to learn the effect of higher frequencies, than those present in the training data.
Transport Equation.
The transport of scalar quantities of interest is modeled by PDE,
Ue + v Vu 0, u(t = 0) f.
(4.4)
with a given velocity field and initial data f. The underlying operator G f + u(., = 1) maps the initial condition f into the solution at the final time. We set a constant veloeity field v = (Va, My) = (0.2, 0.2) leading to solution u(x, y. f) = f(x - v2t,y vyt). Two different types of training data are considered, i.e., smooth initial data which takes the form of radially symmetric Gaussian, with centers randomly and uniformly drawn from (0.2, 0.4)² and corresponding variance drawn uniformly from (0.003, 0.009) and a discontinuous initial data in the form of the indicator function of radial disk with centers, uniformly drawn from (0.2, 0.4)² and radii uniformly drawn from 0.21 (See SM CM3.333 for details and SM for illustrations). For out-of- distriibiitiitti testing the smooth case, the centers of the Gaussian inputs are sampled uniformly from (0.4, 0.6)² and in the discontinuous case, the centers of the disk are drawn uniformly from (0.4, 0.6) while keeping the variance and the radii, respectively, the same as that of in in-distribution testing. This dut-of-distribution task tests the model's ability to cope with input translation eetiiiiniiivviinnc......
Allen-Cahi Equation.
It is prototype for nonlinear parabolic PDEs,
4 = Au - - -1),
(4.5)
with reaction rate of = 220 and underlying operator gt f u(., T), mapping initial conditions ff the solution at a final time = 0.0002. The initial conditions for training and in in-distribution testing are of the form (.2). with = 1 and K = 24 and coefficients a,y drawn uniformly from [-,, For out-of-distribution testing, we set K = 16 and randomly select the initial decay r, uniformly from the range (0.85, 1.15] of the modes in (4.2). which allows us to test the ability of the model generalize to different dynamics of the system. Both training and test data are generated by using a finite difference scheme [62] on a grid at 642 resolution (see D illustrations).
7

---

Navier-Stokes Eqns. These PDEs model the motion of incompressible fluids by,
Uc u (u. V)u+ + Vp= = v'u, div 0,
(4.6)
in the torus D T with periodic boundary conditions and viscosity 4 x 10-2, only applied to high-enough Fourier modes (those with amplitude 2 12) to model fluid flow at very high Reynolds- number. The solution operator G f u(.,, 2), maps the initial conditions f D 1222 to the solution at final time T = 1. We consider initial conditions representing the well-known thin shear layer problem 3 27 (See SM C.3.5 for details), where the shear layer evolves via vortex shedding to complex distribution of vortices (see SMD for samples). The training and in-distribution testing samples are generated, with a spectral viscosity method (7]. from an initial sinusoidal perturbation of the shear layer [27]. with layer thickness p = 0.1 and 10 perturbation modes, each sampled uniformly from |-1, 1]. For out-of-distribution testing, the layer thickness reduced to p = 0.09 and the layers are shifted up in the domain to test the ability of the models to generalize to a flow regime with an increased number and different locations of the shed vortices.
Darcy flow. The steady-state Darey flow is described by the second order linear elliptic PDE,
-V (aVu) f. in D, u|aa 0,
(4.7)
where a is the diffusion coefficient and f is the forcing term. We set the forcing term to f = 1. The solution operator is G a u, where the input is the diffusion coefficient #4 with being a Gaussian Process with zero mean and squared exponential kernel
y/ 12
(xx 8) = 0² exp
0² = =01.
(4.8)
We chose the length scale ( = 0.1 for the in in-distribution testing and I 0.05 for the uut-of-distribution testing. The mapping &b : IR + R takes the value 12 on the positive part of the real line and 3 on the negative part. The push-forward measure is defined pointwise. The experimental setup is the same as the one presented in 33
Flow past airfoils.
We model this flow by the compressible Euler equations,
ue + div F(u) 0, u= = [p, pv, E F lpv. pv @ - pI, (E p)lu]+ (4.9) with density p. velocity v. pressure p and total Energy E related by an ideal gas equation of state. The airfoils consider are described by perturbing the shape of well-known RAE2822 airfoil [40] by Hicks-Henne Bump functions [42] (see SM SM C.3.7). Freestream boundary conditions are imposed and the solution operator maps the shape function onto the steady state density distribution (see for samples) and training data are obtained with a compressible flow solver (NUWTUN) with shapes corresponding to 20 bump functions, with coefficients sampled uniformly from [o, 1].
Out-of- distributtio testing is performed with 30 bump functions.
The test errors, for both in-distribution and out-of diitriibtitttiitiion testing for all the models on the RPB benchmarks are shown in Table D Starting with the in-distribution results, we see that among the baselines, FNO clearly outperforms both FFNN and DeepONet on all the RPB benchmarks as well as the Galerkin Transformer on all except the Poisson test case. On the other hand, the cempllution-basede U-Net and ResNet models are quite competitive vis-a-vis FNO, with comparable performances on most benchmarks, while outperforming FNO by a factor of 7 - 9 for the Poisson test case. This already indicates that convolution-base architectures can perform very well. Moreover, we observe from Table I that CNO is the best performing architecture on every task except Allen-Cahi test case but more moderately but significantly on other tasks. It also outperforms U-Net and ResNet on all tasks considered here, emerging as the best- best-performing model over these benchmarks.
Results.
It readily outperforms FNO, for instance by almost factor of 20 on the Poisson
Distriiiiitiittonnn Testing. This trend is further reinforced when we consider dist---dittiibuutin testing. CNO generalizes well to unseen data in a zero-shot mode, with test errors increasing by approximately a factor of 2, at most (with Allen-Cahn being an outlier) and still outperforms the baselines significantly in all cases other than Allen-Cahn, where FNO generalizes the best. FNO shows decent generalization for most problems but generalizes poorly on the transport problems. This can be attributed to its lack of translation equivariance, in contrast to U-Net, ResNet and CNO. Finally, the lack of translation invariance severely limits the ofttt--disti generalization performance of DeepONet, FFNN and Galerkin Transformer models on transport problems.
8

---

4.5
Ground Truth
CNC
PNO
Emror (in
10
m
Tes Emer (mn
-.0
3.5
3000
32
64
96
128
2.5
57
64
96
128
Testing Resolution
Testing Resolution
Figure 2: Thin Shear Layer Left: Averaged logarithmic amplitude spectra comparing Ground Truth,
CNO, FNO and UNet. Right: Test error vs. Resolution for UNet, FNO and CNO.
Resolution Invariance. We select three of the best- models (U-Net, FNO and CNO) and highlight further differences between them for the Navier-Stokes test case (see also SM(). To this end, we start with (left) where we present the averaged (log) spectra for the ground truth (reference solution computed with the spectral viscosity method) and those computed with CNO, FNO and U-Net. We observe from this figure that i) the spectrum of the exact solution is very rich with representations of many frequencies, attesting to the multi-scale nature of the underlying problem and ii) there significant differences in how CNO and FNO approximate the underlying solution in Fourier space. In particular, the decay in CNO's s spectrum is more accurate. On the other hand, the FNO spectrum is amplified along the horizontal axis, possibly on account of aliasing errors that add incorrect frequency content. The U-Net spectra are similar to that of CNO but with high-frequency modes being amplified, which leads to higher test error. Next, in Figure 2 (right). we compare CNO, FNO and U-Net vis-a-vis the metric of how the test error varies across resolutions, see SMC 4 for details, which is an important aspect for robust operator learning that been highlighted in [33 25).
see also [2) for a discussion with respect to representation equivalent neural operators or ReNOs.
We find from Figure (right) that for the Navier-Stokes benchmark, the FNO error is not invariant with respect to resolution, with an increase in error of up to 25% on lower-resolutions as well as a more modest but noticeable increase of 10% on resolutions, higher than the training resolution of 64", implying that FNO is neither able to perform alias-error error free super- or sub-resolution in this case, see also (2] for a detailed discussion on the resolution-invariance of FNO. Similarly, the increase of U-Net test error with respect to varying resolutions is even more pronounced, with maximum increase of a factor of 3, indicating neither FNO nor U-Net are resolution (representation) equivalent in this case. In contrast, CNO error is invariant with respect to test resolution, verifying that it respects continuous-discrtte equivalence. Further ablation studies for CNO are presented in SM C.5
Efficiency. In order to further compare CNO with FNO, which is the most widely used neural opera- tor these days, we illustrate not just the performance in terms of errors but also terms in computational efficiency. To this end, in Figure 3 (left), we plot the size (total number of parameters) error for the Navier-Stoker benchmark for all FNO and CNO models that were considered in the model selection procedure to observe that for the same model size, CNO models led to significantly smaller validation errors. This resulted in smaller errors for the same per-epoch training time (Figure 3 (center)) for CNO vis-a-vis FNO. As observed in Figure 3 (center). this also implies that one of the best-performing CNO models, with a significantly smaller error than the best-performing FNO model is also almost twice as fast train, thus showcasing the computational efficiency of CNOs.
validation
Scaling laws. A very important aspect of modern deep learning is to evaluate how the performance of models scales with respect to data (and model size). To investigate this, we focus on the Navier- Stokes benchmark and present test error vs. (log of) number of training samples for GT, FNO and CNO in Figure 3 (right) observe that the errors decrease consistently with number of samples. To quantify the rates of decrease, we fit power laws of the form E' = (No/N)-r, with E' being the test error, N number of samples and No normalized be the number of samples required to reach errors of 1% to obtain that CNO attains a much faster convergence rate (r = 00.77) compared to FNO (r = 0.28) and GT (r = 0.27). This implies that CNO will require N, z 14.3K samples to attain 1% error which is 4-times less than FNO (No ~ 60.1K) and almost 10-times less than GT (Ng As 133.2K), highlighting the data efficiency of CNO. Finally, from Figure 3 (right), we also observe that smaller CNO model (with 0.82 M parameters) scales worse (with = 0.3) compared to the best performing CNO model with 7.8M parameters, illustrating that model size could be a bottleneck for data scaling and larger models are required to scale with data.

---

CNO
mho
45
4.0
010 0.0M
16AGM
25
Mlddet
211
Figure 3: Navier-Stokes Benchmark: Test Error (Y-axis) vs. Model size (Left) and per-epoch training time (Center) for all the FNO and CNO models tested. The best-performing models are highlighted as is a scall-scale yet efficient CNO model. Right: Log of Error Log of Number of training samples for GT, FNO, CNO and a small-scale CNO model.
5 Discussion.
Summary. We propose CNO, novel convolution-base architecture for learning operators. The basic design principle was to enforce a form continuous- equivalence in order to genuinely learn the underlying operators, rather than discrete representations of them. To this end, modified the elementary operators of convolution, up-and downsampling and particularly nonlinear activations to realize CNO representation equivalent neural operator or ReNO in the sense of [2 We also prove a universality theorem to show that CNO can approximate large class of operators arising in PDEs to desired accuracy. A novel suite of experiments, termed as representative PDE benchmarks (RPB), encompassing a wide variety of PDEs, with multiple scales in the corresponding solutions, which are hard resolve with traditional numerical methods, is also proposed and the model tested on them. We demonstrate that CNO outperforms the baselines, including FNO, significantly on most benchmarks. This also holds for the considered out-of- out-of-distributionn testing tasks which ascertain the ability of the models to generalize to unseen data in zero-shot manner.
Comparison to Related Work. We emphasize that our construction of CNO follows the theoretical prescription of recent paper (2) on enforcing structure preserving continuous-disceel equivalence. CNO is a representation equivalent neural operator, with respect spaces of bandlimited functions, in the sense of [2). Another motivating work for us is [22). see also [58), where the authors modify CNNs to eliminate (or reduce) aliasing errors in the context of image generation. We adapt the construction of 22 to our setting and deploy the resulting architecture in very different context from that of 22 namely that of operator learning for PDEs rather than image generation. Moreover, we also instantiate CNO with a very different operator UNet architecture than that proposed in [22) We would also like to mention related work on using CNNs for solving PDEs such as [ 16 and emphasize that in contrast to CNO, they lack suitable notions of mantinuous-discred equivalence. Finally comparing CNO to the widely used FNO model, we observe that unlike FNO which can fail to enforce CDE (see Moreover, the convolution operator in CNO is local in space, in contrast to convolution in Fourier space for FNO. The detailed empirical comparison presented here demonstrates CNO can outperform FNO and other baselines on many different metrics namely performance, computational efficiency, resolution invariance, out-of-distribution generalization well as data scaling, paving the way for its widespread applications in science and engineering.
[1] 2 and Figure 2 right)), CNO preserves continuous-discres equivalence.
Limitations and Future Work. We have presented CNO for operators an underlying two- dimensional Cartesian domain. The extension to three-space dimensions is conceptually straightfor- ward but computationally demanding. Similarly, extending to Con-Cartesian domains will require some form of transformation maps hetween domains, for instance reworking those suggested for be readily considered. Adapting CNO to approximate trajectories (in time) of
FNO in 32 55] time-dependent PDEs, for instance by employing it in an auto-regressive manner, is another possible extension of this paper. At the level of theoretical results, we believe that the generic framework of || can be adapted to show that not only does CNO approximate large class of PDEs universally, it does so without incurring any curse of dimensionality. as shown for DeepONets in (26 and FNOs in 24 Finally, adapting and testing CNO for learning operators, beyond the forward solution operator of PDEs is also interesting. One such direction lies in efficiently approximating PDE inverse problems, for instance those considered in 44.
10

---

References
[1] I. Ayed, E. de Bézenac, A. Pajot, J. Brajard, and P. Gallinari. Learning dynamical systems from partial observations. CoRR, abs/1902. 1136, 2019.
[2] F. Bartolucci, F.. de Bézenac, B. Raonic, R. Molinaro, S. Mishra, and R. Alaifari. Are neural operators really neural operators ? frame theory meets operator learning. Technical Report 2023-21, Seminar for Applied Mathematics, ETH Zürich, Switzerland, 2023.
[3] Bell, Collela, and M. Glaz. A second-order projection method for the incompressible Navier-Stokes equations. J. Comput. Phys., 85:257-283, 1989.
[4] M. Bertero, P. Bocacci, and C. De Mol. Introduction to inverse problems in imaging. CRC press, 2021.
[5] K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021.
[6] N. Boullé, Y. Nakatsukasa, and A. Townsend. Rational neural networks. Advances in Neural Information Processing Systems, 33: 33:14243-14253, 2020.
[7] S. Cai, Z. Wang, Lu, T. A. Zaki, and G. Karniadakis. D Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics, 436: 11029,, 2021.
[8] Cao. Choose transformer: Fourier or galerkin. In 35th conference on neural information processing systems, 2021.
[9] T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-911 197, 1995.
[10] T. De Ryck, Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 2021.
[11] T. De Ryck and S. Mishra. Generic bounds the approximation error for physics-informed (and) operator learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[12] Q Delfosse, P. Schramowski, M. Mundt, A. Molina, and K. Kersting, Adaptive rational activations to boost deep reinforcement learning. arXiv preprint arXiv:2102.09400) 2021.
[13] L. C. Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
[14] V. Fanaskov and I. Oseledets. Spectral neural operators. arXiv preprint arXiv:2205.10573v/
2022.
[15] Goodfellow, Bengio, A. Courville, and Bengio. Deep learning, volume 1. MIT Press, 2016.
[16] J. K. Gupta and J. Brandstetter. Towards multi-spatiotemporal-scall generalized pde modeling, 2022.
[17] E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34, 2018.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv: 1512.03385, 2015.
[19] J. S. Hesthaven, S. Gottlieb, and D. Gottlieb. Spectral methods for time-dependent problems, volume 21. Cambridge University Press, 2007.
[20] Isola, J.-Y. Zhu, Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In IEEE Conference Computer Vision and Pattern Recognition, 2017.

---

[21] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics informed machine learning. Nature Reviews Physics, pages 1-19, may 2021.
[22] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias- free generative adversarial networks. Advances in Neural Information Processing Systems, 14:852-863, 2021.
[23] G. Kissas, J. H. Seidman, L. F. Guilhoto, V. M. Preciado, G. J. Pappas, and P. Perdikaris. Learning operators with coupled attention. Journal of Muchine Leurning Research, 23(215):1 63, 2022.
[24] N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021.
[25] N. Kovachki, Z. Li, B. Liu, K. Azizzadensheli, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint 108.4:2100..848113,, 2021.
[26] S. Lanthaler, S. Mishra, and G. Karniadakis. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(I):tnac001, 2022.
[27] S. Lanthaler, S. Mishra, and C. Parés-Pulido. Statistical solutions of the incompressible culer equations. Mathematical Models and Methods in Applied Sciences, (00):2233-2 Feb 2021.
[28] S. Lanthaler, R. Molinaro, P. Hadorn, and S. Mishra. Nonlinear reconstruction for operator learning of pdes with discontinuities. In International Conference on Learning Representations, 2023.
[29] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, (2153)))3 2015,
[30] Y. Lecun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, (6(11):2278-2322 1998.
[31] M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural networks, 6(6):861- 867, 1993.
[32] Z. Li, D. Z. Huang, B. Liu, and A. Anandkumar. Fourier neural operator with learned deforma- tions for pdes on general geometries, 2022.
[33] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anand- kumar. Fourier neural operator for parametric partial differential equations. In International Conference Learning Representations, 2021.
[34] Z. Li, B. Kovachki, Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anand- kumar. Neural operator: Graph kernel network for partial differential equations. CoRR, Abs/2003.03485. 2020.
[35] Z. N. B. Kovachki, K. Azizzadenesheli, B. Liu, A. M. Stuart, K. Bhattacharya, and A. Anand- kumar. Multipole graph neural operator for parametric partial differential equations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6755 6766. Curran Associates. Inc., 2020.
[36] Li. H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar. Physics-informe neural operator for learning partial differential equations. arXiv preprint 1:Xiv:2/11.0370 2021.
[37] Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, Darrell, and S. Xie. convnet for the 2020s. In Proceedings- 2022 IEFF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages
IEEE Computer Society, 2022.
12

---

[38] Z. Long, Y. Lu, X. Ma, and B. Dong. Pde-net: Learning pdes from data. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceed- ings of Machine Learning Research, pages 3214 3222. PMLR, 2018.
[39] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-229 2021.
[40] K. O. Lye. S. Mishra, D. Ray, and P. Chandrashekar, Iterative surrogate model optimization (ISMO); An active learning algorithm for PDE constrained optimization with deep neural networks. Computer Methods in Applied Mechanics and Engineering, 374:113575, 2021.
[41] Z. Mao, L. Lu. O. Marxen, Zaki, and G. E. Karniadakis. DeepMandMnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind normal shock using neural- network approximation of operators. Preprint, available from arXiv:201 1. 1.03349v1, 2020.
[42] D. A. Masters, J. Taylor, T. Rendall, C. B. Allen, and J. Poole. Geometric comparison of aerofoil shape parameterization methods. AIAA Journal, pages 1 1575-1589, 2017.
[43] A. Molina, Schramowski, and K. Kersting. Pad' ee activation units: End-to-end learning of flexible activation functions in deep networks. arXiv preprint urXiv:1907.0672: 2019.
[44] R. Molinaro, y. Yang. Engquist, and S. Mishra. Neural inverse operators for solving pde
inverse problems. 1..iv:2331.111666 2023.
[45] J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, Hassanzadeh, K. Kashinath, and A. Anandkumar. Four- castnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:220..12214 2022.
[46] P. Petersen and F. Voigtlaender. Equivalence of approximation by convolutional neural networks and fully-connected networks. Proceedings of the American Mathematical Society, 148(4):1567- 1581, 2020.
[47] M. Prasthofer, T. De Ryck, and S. Mishra, Variable input deep operator networks, arXiv preprint UrXiv:2205.11404 2022.
[48] A. Quarteroni and A. Valli. Numerical approximation of Partial differential equations, vol- ume 23. Springer, 1994.
[49] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informi neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[50] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted InterventionMIICAA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part 111 18, pages 234-241. Springer, 2015.
[51] T. Schanze. Sine interpolation of discrete periodic signals. IEEE Transactions on Signal Processing, 1306):150- 1995.
[52] J. H. Seidman, G. Kissas, P. Perdikaris, and G. J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. arXiv preprint arXiv:2206.085 2tt.:22006.05511. 03551. 2022.
[53] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich Keil,, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[54] M. Telgarsky. Neural networks and rational functions. International Conference on Machine Learning, pages 3387-3393. PMLR, 2017.
13

---

[55] A. Tran, A. Mathews, L. Xie, and C. S. Ong. Factorized fourier neural operators, In The Eleventh International Conference on Learning Representations, 2023.
[56] M. Unser. Sampling-Se years after shannon. Proceedings of the IEEE, 88(4):569-587, 2000.
[57] M. Vetterli, J. Kovacevic, and V. Goyal. Foundations of Signal Processing. Cambridge University Press, 2014.
[58] S. Wang, S. Suo, W. Ma, A. Pokrovsky, and R. Urtasun. Deep parametric continuous convolu- tional neural networks. CoRR, abs/2101 2021.
[59] S. Wang, H. Wang. and P. Perdikaris. Learning the solution operator of parametric partial differential equations with thysics-informer DeepOnets. arXiv preprint arXiv:2103.10974, 2021.
[60] S. E. Wei. Aliasing-free nonlinear signal processing using implicitly defined functions. IEEE Access, 10:76281-76299 2022.
[61] R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. CoRR, abs/2110.004 2021.
[62] J. Yang, Q. Du, and W. Zhang. Uniform I p-hound of the allen-cahn equation and its numerical discretization. International Journal of Numerical Analysis & Modeling, 15, 2018.
[63] Y. Zhu and N. Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physies, 336:415 447, 2018.
14


Reviews:
[{"review_id": "py307ERD24", "reviewer": "Reviewer_KkQX", "summary": "The paper re-introduces convolution neural networks to the operator learning setting. It uses interpolation including up-sampling and downsampling layer to make sure the convolution layer is well-defined in the function space. The paper also discuss the space of band-limited functions and the trade off between continuous-discrete equivalence and representation power of the infinite dimension function space. The paper proves an approximation theorem for the CNO model and compared it across many partial differential equations. ", "questions": "As discussed in page 4, CNO uses upsampling and downsampling to reduce the aliasing error, but it's also claimed that the activation layer will introduce the aliasing error. This is overcome by \"Implicitly assuming that $\\bar{w}$ is large enough\". Can the author give some examples of the activation such that this is satisfied? Speaking of frequencies expansion, for ReLU or leaky ReLU, $\\sigma(x)$  is non-smooth. Even if $\\sigma$ is a k-frequency function, $\\sigma \\circ \\sigma$ can be $k^2$-frequency, right? So after several layers it easily goes unbounded.\n\nFigure 2 is also a bit surprising. Can CNO achieve constant super-resolution error curve for problem like Navier-Stokes? The FNO error also seems a bit too high. How many modes are used in FNO? If FNO is designed with a reasonably small number of modes and no padding (padding may cause error in resolution), FNO may also get flatten curve.", "limitations": "The author discussed some limitations, but it's also good to discuss the trade-off between using band-limited functions and Lp/Hp space. I think both side have their advantages and disadvantages.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The convolution neural operator (CNO) takes the advantages of the efficient of the conventional CNNs and UNets methods, meanwhile it also satisfies the resolution-invariant and representation-equivalent properties. The work shows an approximation theorem that the CNO models can approximate any continuous solution operator, and the numerical experiments show it has a comparative performance compared to other machine learning methods such as  UNet, DeepONet, and FNO.  ", "weaknesses": "I have a few questions and concerns, mainly in the trade-off between the band-limited functions and Lp/Hp space.\n1. The paper claim the CNO models can only learn band-limited functions, however, the target PDE system are intrinsically infinitely dimensional system (in Lp/Hp). It means, given a fixed number of parameters (size of the model), as the number of training sampling and the qualify(resolution) of training dataset increase, the truncation error will dominate and the bandlimited class of operator will underperform the band-unlimited models.\n2. The band-limited operator is, by definition, linear-reconstruction of the chosen basis, meaning it can only learn the coefficient but unable to learn the basis. The linear-reconstruction are less efficient compared to non-linear reconstruction model, as discussed in [1].\n3. The experiments (Table 1) are designed in a manner that CNO dominates all other methods. It shows the potential of CNO, but the results might be biased. It's certainly reasonable that the band-limited CNO outperform band-unlimited model on super-resolution tasks. However, for in-distribution problems, if the resolution is fixed and the dataset is sufficient, then Unet should be equivalent to CNO, right? And for these non-smooth problems, the band-unlimited could be more expressive than CNO. These cases might not be sufficiently considered in the experiments design. It is great to introduce these new experiments, but it would be better to include as least one of the previous standard benchmark, for example, the Burgers or Darcy equation dataset from [2].\n4. While the perspective is new and interesting, it's hard to say the proposed model is very novel. The architecture is highly similar to the previous UNet model + StyleGAN3 de-aliasing trick.\n\n[1] Lanthaler, Samuel, et al. \"Nonlinear reconstruction for operator learning of pdes with discontinuities.\" arXiv preprint arXiv:2210.01074 (2022).\n\n[2] Li, Zongyi, et al. \"Fourier neural operator for parametric partial differential equations.\" arXiv preprint arXiv:2010.08895 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for your comments which we address below.\n\n1. Regarding the reviewer's question about \"trade-off between $L^p-H^p$ functions and bandlimited functions\" and concern that \"CNO only learns bandlimited functions\", we start by pointing out that in Theorem 3.1 of our paper, we prove an universal approximation theorem by showing that CNOs can approximate any continuous operator (with a modulus of continuity), mapping between two ($H^p$) Sobolev spaces, to desired accuracy. Most PDE solution operators satisfy this assumption. As was shown in SM A.1, our approximation is performed in two steps: first, using the modulus of continuity of the underlying operator, it can be approximated to desired accuracy by an operator mapping between band-limited spaces  and then this operator is in turn, approximated by CNOs. Thus, the \"truncation error\" can be made arbitrarily small by choosing the right bands. In practice, the bands are given by the resolutions of the underlying input and output function, which also determine the associated truncation error. It is unclear to us why the size of the training dataset matters in this context, in fact, in SM Figure 25, we show empirically how the test error of CNO decreases with increasing sample size. Also, we would like to emphasize that \"band-unlimited\" models are subject to discretization errors as the data is only available at finite resolutions, which automatically determines the size of their discretization error. Given this very valid question raised by the reviewer, we will discuss this issue further in a CRV, if accepted.\n2. We would like to clarify that the results of [1] were specific to the case of scalar 1-D transport of a single discontinuity and a related 1-D Burgers example where they claimed that FNO is better than DeepONet as the approximation error of a linear reconstruction operator can be bounded below by the rate of decay of eigenvalues that may converge slowly for some transport equations. This does not imply that methods with a linear reconstruction cannot be efficient for a very large class of PDEs which have a regularizing effect resulting in fast spectral decay. Moreover, the result of [1] only pertains to size efficiency vis a vis approximation error. Even if a model is size efficient, the overall error will be dominated by the finite resolution of given data. Another pertinent source of error is the aliasing error (as discussed in our paper) and a nonlinear reconstruction might lead to aliasing errors which can affect the test error whereas, CNO is based on reducing this important source of error. Thus, a priori, it is not possible to rule out that a method with a fixed basis is necessarily less accurate than a nonlinear reconstruction method. To further investigate this question,  we have extensively tested our method on transport problem with discontinuities as well as on a compressible Euler problem with shocks and contacts and in both cases (see Table 1), despite the presence of discontinuities, CNO was comparable to FNO in-distribution and significantly better out-of-distribution due to translation invariance, demonstrating that method like CNO can also be very competitive in approximating transport-dominated problems. We will include a discussion on this question in the CRV.\n3. Regarding reviewer's concern about the choice of benchmarks, we would like to state that the benchmarks were explicitly chosen, not to favor CNO. On the contrary, they include problems with discontinuities (Transport, Euler), sharp gradients (Allen-Cahn) and problems with explicit solutions in Fourier space (Poisson, Wave) which we argue favor FNO a priori. However, the results clearly show that even on this dataset, CNO is competitive with FNO (and many times is performing better) (Table 1). Similarly we had already presented the Navier-Stokes vorticity benchmark from [2] in SM C.3.7, providing a clear rationale on why did not include this benchmark in the main paper. Even here (Line 1177), CNO is marginally better than FNO. Following the reviewer's excellent suggestion, we will also include the Darcy benchmark from [2] in the CRV. The results shown in uploaded pdf (Table 1) again show that CNO performs very well on this benchmark too. Finally, apart from the filtering operators, CNO is also different from the traditional U-Net architecture of Ref. [49] in using ResNets connecting encoder and decoder as well as in using a different pooling operation.\n4. We differentiate our work from StyleGAN paper in lines 314-318 in related work section and the difference with U-Nets are described above. \n5. A detailed discussion on activation functions and aliasing is provided in SM Remark B.4 where polynomial and rational activation functions are given as examples of exactly alias-free activations. Moreover, the choice of $\\bar{\\omega}$ is also extensively discussed is remark SM Section C1.3 and Remark C.1, see Table 2 for a comparative study on how the choice of $\\bar{\\omega}$ affects performance.\n6. The CNO discretization invariance results in Figure 2 can be explained by the fact that almost all the frequency information in the problem is already contained at the training resolution of $64^2$) (See spectrogram for Ground Truth Fig.2 Left) and there is very little high-frequency content in finer discretizations. On the other hand for the Poisson problem, Spectogram (SM Fig. 6) and discretization dependence (SM fig 24) show that missing high-frequency information can cause small test errors as the test resolution is increased towards $128^2$, corroborating our explanation about the reviewer's question on Figure 2. Regarding the discretization dependence of FNO, we agree with the reviewer that padding might be a factor in causing in some aliasing error and will investigate the effect of padding on FNO errors in a CRV.\n\nWe hope to have addressed all the reviewer's concerns and kindly request you to update your assessment accordingly"}, {"review_id": "YRxekBh0mc", "reviewer": "Reviewer_MYhB", "summary": "The paper under review proposes a CNN architecture as a neural operator for solving PDEs via neural networks.  The goal is to preserve the underlying continuous structure while being implemented with discrete convolutional architecture, based on U-Net.  The idea is to consider the space bandlimited functions in Sobolev spaces.  Defining a convolution layer involves a discrete convolution, upsampling/downsampling that requires sinc interpolation to avoid aliasing, and an activation layer that upsamples, activates and then downsamples to maintain the original band-limitation.  The authors prove in supplementary material that their approach can approximate the solution to a PDE with arbitrary accuracy with the CNN architecture based on their neural operator.  Experiments are done on solving several PDEs, with experiments favorable to the authors' method against state of the art for most PDEs considered.", "questions": "- Bandlimited: many solutions to PDEs will have shocks, discontinuities, etc.  How relevant is this assumption?", "limitations": "Yes, discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The architecture based on CNNs potentially simplifies e.g., FNO operators where operations are done in the Fourier domain without the use of down-sampling - which can potentially be more expensive than the authors' proposed method.\n- Rigorous mathematical treatment of the interplay between continuous and discrete operators.\n- Experiments show higher accuracy compared to SOA in PDE solving with NN.", "weaknesses": "- No evaluation of efficiency; is there a speed advantage of the proposed method (e.g., compared to FNO)?\n- Evaluation of parameter sizes is in supplementary and there isn't a clear trend; I would have thought this method would require fewer parameters than e.g., FNO; but in many cases this isn't so.\n- I would think efficiency in the sense above is a key differentiator compared to SOA, but this isn't the case at least looking at supplementary.\n- The authors make use of sinc interpolation, which is the ideal interpolator for sequences of infinite length.  In practice, you are operating on finite length data, for which the sinc interpolater is not the right one.  How applicable is the theory presented to the case of finite length data?\n- Prop 2.1 talks about a representation equivalent operator and the definition is cited in a reference.  Please specify the definition in the paper.\n- Theory is poorly presented in the paper; everything is in supplementary.  At least the key ideas should be in the main paper.\n- In general many of the key parts that should be in the paper are in supplementary.\n- I find the authors' proposed idea to set a standard for benchmarking misguided.   There is a laundry list of PDEs to approximate in the benchmark, however, I doubt any one architecture would be good for all PDEs.  Each PDE has particular properties that one would want to aim to preserve (e.g., conservation laws in some PDEs) that may not be relevant to others.  So the implication that one architecture should do well on all PDEs seems mis-guided.  I do not think this should be an accepted standard benchmarking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for your appreciation of the merits of our paper and your welcome suggestions to improve it. We address your detailed concerns below.\n1. Our focus in the empirical part of the paper was on accuracy and during model selection, we chose the hyperparameters for the model with the lowest validation error. We agree fully with the reviewer that efficiency is also a very important aspect and following your excellent suggestion, we showcase the comparative efficiency of CNO vis-a-vis FNO for the Navier-Stokes benchmark in Figure 1 in the uploaded pdf for this rebuttal. In this figure, we plot the validation error (Y-axis) for all the CNO and FNO models we have tested (details in SM C.2) with respect to  Model size (Left panel) and per-epoch training time (Right panel). We see from this figure that 1) for fixed model size: CNO is clearly more accurate than FNO 2) For fixed training time: CNO is also clearly more accurate than FNO 3) Instead of selecting the CNO with the smallest error (as in the paper), if we select a more efficient CNO model (highlighted with dark Red in Figure 1), we observe that this model has almost two orders of magnitude less parameters than the best performing FNO while also being more than 50% faster (2.8 secs vs 4.4 secs) and at the same time, with a significantly smaller validation error (3.5%) compared to FNO (4%). This figure demonstrates that CNO is not only more accurate but clearly more efficient than FNO for this problem. Needless to say, we will include a detailed efficiency study along these lines for each of the benchmarks and other models in the CRV, if accepted and thank the reviewer for asking us to explore this very important issue that we believe further strengthens our message.\n2. The reviewer raises an important point about infinite length sinc filters. We have already discussed this issue in some detail in SM C.1.3 about the use of windowed sinc filters, where we have also shown the effect of choosing different window lengths.\n3. Given the 9-page limit for the main paper, we had to make some difficult choices about what to include in the main paper and what to include in the SM. In particular, the definition of Representation equivalent neural operator (ReNO) from Bartolucci et. al. is quite technical and we had not included it. We had tried to motivate the heuristics behind this definition in lines 137-141 but we agree with the reviewer's point and will include the definition of ReNO in the CRV, if accepted. This will also enable us to make the paper more self-contained. \n4. Our objective was to motivate the theory in the main paper while presenting the very detailed and technical arguments of the proofs in the SM, particularly SM A.1, A.2, A.5 and B. The fact that the motivations were very terse is partly due to the 9-page limit. A CRV, if accepted, allows us to add an additional page and we will focus on providing more motivation about the theory in this additional space. In particular, we will add more details about the definition of ReNO and how and under what conditions CNO is a ReNO and what happens when these conditions are satisfied only approximately. We will also provide an outline of the proof of theorem 3.1, mentioning the main steps while postponing the details to the SM. We again thank the reviewer for highlighting this issue and acknowledge that it is difficult to balance the main text/SM delineation within a strict page limit. We will also use the additional space to discuss computational efficiency (see Point 1).\n5. We apologize for possible misunderstanding in our write-up about benchmarks (line 177-192), Clearly, we agree with the reviewer that PDEs are too vast in scope to allow a single set of benchmarks. This was not our aim. Rather, our starting point was to propose a set of criteria for choosing possible benchmarks and these are outlined in lines 179-182. We believe that the reviewer will agree that these are necessary criteria for good benchmarks while we acknowledge that these are not sufficient and other issues such as conservation, symmetries etc should also be considered. We also use this criteria to critically examine some existing benchmarks (see SM C3.7). We can include a further discussion on these criteria in a CRV. Moreover, we have not claimed that one model will do well on all chosen problems -- in fact, Table 1 already shows a diversity of best-performing models and we will make that more explicit in a CRV, if accepted.\n6. Consider a PDE with discontinuities such as linear advection Eqn with discontinuous initial data. The underlying solution operator is readily approximated (in $L^2$) to desired accuracy by mollifying the initial data and this approximate operator, which maps between smooth functions, can be approximated further by an operator mapping between band-limited functions (SM A.1). Similarly, consider inviscid Burgers' Eqn with sine-wave initial data -- the solution operator will contain shocks but we can approximate it to desired accuracy (in $L^1$) by the solution operator of viscous Burgers' (with small viscosity) which maps between smooth functions and can, in turn, be approximated by operators mapping between band limited functions (SM A.1). Again, the underlying operator can be approximated to desired accuracy by an operator mapping between bandlimited spaces. This consideration is also corroborated by the section on numerical experiments where we consider a 2D advection equation with moving discontinuities (SM C3.3) as well as a compressible Euler equations with shocks (SM C3.6). In both cases (see Table 1), CNO which maps between band limited functions, performs very well for both in-distribution as well as for out-of-distribution testing, supplementing our arguments above. \n\nWe sincerely hope to have addressed  your concerns, particularly about the computational efficiency of CNOs and would kindly request the reviewer to update their assessment accordingly. "}, {"review_id": "8AcOUGKzHL", "reviewer": "Reviewer_ZTbu", "summary": "In this paper, the authors propose a new operator learning framework called Convolution Neural Operator which although works on discrete space, satisfies the property of Continuous-Discrete Equivalence (CDE) property. They define CNO over a UNet architecture and provide some universal approximation proofs. Further, they come up with a Benchmark of representative PDEs and compare all the existing methods and present superior performance of CNO over other methods.", "questions": "See above in weaknesses*", "limitations": "Yes. The authors have accurately identified the limitations. There are no negative societal impact for their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ Novel idea of CNOs\n+ Theory provided for soundness\n+ Representative Benchmark Dataset and making it public in zenodo is also good.\n+ Very neat and clean code. ", "weaknesses": "- Some notations are messed up or confusing to follow. e.g. r in line 82.\n- While the performance of these models is compared... the computation requirements like memory or training time are not compared. I think if we are talking about benchmarking, it is appropriate to give the complete story including the computational requirements.\n- Some ablation studies on whether using CNOs reduces the architecture/compute requirements compared to just using a CNN or FNO is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for your appreciation of the merits of our paper and your welcome suggestions to improve it. We address your detailed concerns below.\n\n1. We apologize about the possible lack of clarity in notation for $r$ on line 82 -- it refers to the smoothness index $r$ of the Sobolev space $H^r$, introduced in line 67 where the underlying operator maps from this Sobolev space. We will make this $r$ more explicit in a CRV, if accepted and also comb through the paper for any further notational ambiguities. \n\n2. Our focus in the empirical part of the paper was on accuracy of the tested architectures and during model selection, we chose the hyperparameters for the model with the lowest validation error. We agree fully with the reviewer's point that efficiency, particularly in terms of model size and training time, is also a very important aspect and following your excellent suggestion, we showcase the comparative efficiency of CNO vis-a-vis FNO for the Navier-Stokes benchmark (see detailed description in SM C3.5) in Figure 1 in the uploaded pdf for this rebuttal. In this figure, we plot the validation error (Y-axis) for all the CNO and FNO models we have tested (details in SM C.2) with respect to  Model size (Left panel) and per-epoch training time (Right panel). We see from this figure that 1) for fixed model size: CNO is clearly more accurate than FNO 2) For fixed training time: CNO is also clearly more accurate than FNO 3) Instead of selecting the CNO with the smallest error (as in the paper), if we select a more efficient CNO model (highlighted with dark Red in Figure 1), we observe that this model has almost two orders of magnitude less parameters (model size) than the best performing FNO while also being more than 50% faster (2.8 secs vs 4.4 secs) in terms of training time and at the same time, with a significantly smaller validation error (3.5%) compared to FNO (4%). This figure demonstrates that CNO is not only more accurate but clearly more efficient than FNO for this problem. Needless to say, we will include a detailed efficiency study along these lines for each of the benchmarks and also compared with other models in the CRV, if accepted and thank the reviewer for asking us to explore this very important issue of computational efficiency that we believe further strengthens our message.\n\n3. The ablation study that the reviewer refers to has been addressed in some detail in point 2 of our rebuttal to your comments. In figure 1 of the uploaded rebuttal pdf, we have only compared with FNO due to time constraints as well as the fact that FNO is acknowledged as a state of the art neural operator. We will also compare with other models, include CNNs (UNets) and present the results in a CRV, if accepted. We would also like to refer the reviewer to SM C.5. where we have performed a detailed ablation study for different building blocks of CNO in order to investigate which building blocks are crucial to its functioning. \n\nWe sincerely hope to have addressed  your concerns, particularly about the computational efficiency of CNOs and would thank the reviewer again for pointing out these concerns.  \n"}, {"review_id": "3atcBoGhkG", "reviewer": "Reviewer_uN3N", "summary": "The authors propose a UNet-like architecture for learning solution operators of PDEs. The architecture is assembled from building blocks analogous to the classical UNet architecture, but individual components respect a continuous-discrete equivalence. This makes the proposed architecture a representation equivalent neural operator in the sense of a recent paper (Bartolucci et al., 2023). The authors show that under some continuity assumptions the proposed method can learn the solution operators to a variety of PDEs. They also compare the in- and out-of-distribution performance of their method to a variety of baselines for a suite of different types of PDEs.\n", "questions": "- The practical implementation of the interpolation filters uses windowed-sinc. In theory, doesn't this break the continuous-discrete equivalence?\n\n- Regarding \"operator\" networks, I was wondering why the authors didn't compare to a fully convolutional ResNet (along the lines of Solver-in-the-loop Um'21 or the accelerated CFD from Kochkov'21); that architecture should more naturally extend to different resolutions than a Unet. Can the authors comment on this omission?\n\n- Can the authors provide details on the performance, especially how much slower the CNO is compared to the regular Unet?\n", "limitations": "Limitations are only discussed as a future work discussion. I don't think this is sufficient. E.g., a clear discussion of the performance impact is completely missing as far as I can tell. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- There is a large body of work on methods and architectures for operator learning which this paper builds up on. Having architectures that respect the continuous-discrete equivalence seems very significant to me.\n- The presentation is clear and the paper overall well written.\n- The proposed CNO architecture improves the overall performance for a comprehensive suite of problems and appropriate baselines.\n- Many evaluations and ablation studies are included in the appendix.\n", "weaknesses": "- As discussed in the related work section, the building blocks that make up the CNO architecture have been introduced in previous papers (Karras et al. 2021, Alias-free generative adversial networks). This slightly weakens the novelty of the paper.\n- Many definitions are adopted from a relatively recent paper (Bartolucci et al. 2023), which to my knowledge is only available as a preprint at the moment.\n\n- The modifications to the UNet architecture introduce additional up-sampling and down-sampling layers, which increase the training time significantly compared to other methods. This needs to be kept in mind. Also, the implementation of CNOs is much more complicated than of e.g. FNOs due to the windowed-sinc filters.\n\n- I found the evaluation across resolutions somewhat unclear and partially misleading. The appendix goes into more detail here, but the results in the main paper show a single NS case that resulted from a downsampled solution (hence no high-frequency details exist in the targets). In this case the CNO seems to show a constant error, which is dubious for meaningful real world cases where higher resolutions naturally would exhibit structures that aren't resolved with lower resolutions. I think it will be important for future versions to clarify this, and replace figure 2 with one of the other two cases from the appendix.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We start by thanking the reviewer for your appreciation of the merits of our paper and your welcome suggestions to improve it. We address your detailed concerns below.\n1. Regarding the reviewer's point about novelty vis-a-vis Karras et al 2021, while acknowledging that we have used building blocks, particularly the dealiasing trick from there, we would also like to highlight the novelty of our contributions with respect to this paper. They are 1) unlike in Karras et al, we had to rewrite their discrete operators in a continuous setting in order to realize CNO as a neural operator 2) Karras et al considered a very different setting (image generation) and a significant part of the novelty of our paper lies in realizing these ideas in the setting of approximating PDE operators where these ideas have not been previously proposed 3) we instantiate CNO with a different UNet type model than the construction of Karras et al 4) We prove that CNO is a ReNO and can approximate any \ncontinuous operator mapping between Sobolev spaces and 5) provide considerable amount of empirical testing of CNO on many PDE benchmarks. Some of these points are already mentioned in lines 312-318 of the paper and this differentiation from Karras et al will be sharpened and strengthened in the main text of a CRV, if accepted. \n2. Indeed, the key definition of ReNO is adopted from Bartolucci et al. Given the 9-page limit, we did not have the space to include this definition in the main paper. But with an additional page in the CRV, if accepted, we would include the definition and make the paper as self-contained as possible. \n3. The reviewer's point about the computational efficiency of CNO, vis-a-vis other models is certainly valid. To address your question, we have included a figure (Figure 1) in the uploaded pdf with this rebuttal where we we showcase the comparative efficiency of CNO vis-a-vis FNO for the Navier-Stokes benchmark (see detailed description in SM C3.5) in Figure 1 in the uploaded pdf for this rebuttal. In this figure, we plot the validation error (Y-axis) for all the CNO and FNO models we have tested (details in SM C.2) with respect to  Model size (Left panel) and per-epoch training time (Right panel). We see from this figure that 1) for fixed model size: CNO is clearly more accurate than FNO 2) For fixed training time: CNO is also clearly more accurate than FNO 3) Instead of selecting the CNO with the smallest error (as in the paper), if we select a more efficient CNO model (highlighted with dark Red in Figure 1), we observe that this model has almost two orders of magnitude less parameters (model size) than the best performing FNO while also being more than 50% faster (2.8 secs vs 4.4 secs) in terms of training time and at the same time, with a significantly smaller validation error (3.5%) compared to FNO (4%). This figure demonstrates that CNO is not only more accurate but clearly more efficient than FNO for this problem. Needless to say, we will include a detailed efficiency study along these lines for each of the benchmarks and also compared with other models in the CRV, if accepted and thank the reviewer for asking us to explore this very important issue of computational efficiency that we believe further strengthens our message.\n4. The ground truth for the Navier-Stokes experiment is computed at a resolution of $128^2$. As the reviewer alludes to, the CNO discretization invariance results in Figure 2 can be explained by the fact that almost all the frequency information in the problem is already contained at the training resolution of $64^2$) (See spectrogram for Ground Truth Fig.2 Left) and there is very little high-frequency content in finer discretizations. On the other hand, and as spotted by the reviewer, for the Poisson problem, Spectogram (SM Fig. 6) and discretization dependence (SM fig 24) show that missing high-frequency information can cause small test errors as the test resolution is increased towards $128^2$, corroborating our explanation about the reviewer's question on Figure 2. We agree with the reviewer's contention that SM Figure 24 is a better visual representation of this issue and we will include it (and SM Figure 6) for further context on this issue in a CRV, if accepted.\n5. Indeed windowed sinc is an approximation to the ideal sinc filter and can induce some aliasing error. This issue has been mentioned and discussed in some detail in SM C.1.3 and the effects of windowing are quantified there. \n6. We thank the reviewer for their excellent suggestion of testing out fully-convolutional ResNets as baselines. As the solver-in-loop and Kochkov et al ResNets are in-house, we use the widely used CycleGAN implementation of convolutional ResNets (link in Table 2 caption). Given the time constraints for the rebuttal, we could only perform the tests with some benchmarks and the results for both in- and out-of-distribution testing are presented in Table 2 of the uploaded pdf. They show that ResNets are a competitive baseline on some problems but do not change the main conclusions of Table 1 of our paper. We will include convolutional ResNets as an additional baseline in the CRV, if accepted.\n7. As the reviewer points out, adding additional filtering operations will increase the cost of CNO. This issue has already been addressed in an ablation study shown in SM C.5 where the CNO w/o filters model is similar to an UNet. We report there (see lines 1280-1285) that removing filters is between 1.5-2 times faster than including them and UNet has a similar training time. However, we introduce very large aliasing errors by doing so (Fig 2 and SM fig 24). Hence, we argue that the additional cost is justified to retain CDE and a fairer comparison in efficiency is wrt FNO (see pt 3 for this comparison). \n8. We will further discuss the computational efficiency of CNO (see pts 3 and 7) in a CRV, if accepted.\n\nWe sincerely hope to have addressed your concerns to your satisfaction.\n "}]

Author rebuttal:
At the outset, we would like to thank all four reviewers for their thorough and patient reading of our article. Their criticism and constructive suggestions will enable us to improve the quality of our article. If our paper is accepted, we will incorporate all the changes that we outline below in a camera-ready version ( CRV) of our article. As allowed by the conference, we are uploading a one page pdf that contains figures and tables on numerical experiments which support our arguments below. With this context, We proceed to answer the points raised by each of the reviewers individually, below. 

Yours sincerely,

Authors of "Convolutional neural operators for robust and accurate learning of PDEs".

Meta-review: Overall the reviewers all appreciated the work and the use of convolution based neural operators may be of interest to the community. There were some good questions by reviewers and the authors should add these discussions and promised changes to the paper.
Decision: Accept (poster)