SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Representative PDE Benchmarks (RPB). Given the lack of consensus on a standard set of benchmarks for machine learning of PDEs, we propose a new suite of benchmarks here. Our aims in this regard are to ensure i) sufficient diversity among the types of PDE considered, ii) access to training and test data is readily available for rapid prototyping and reproducibility and iii) intrinsic computational complexity of problem to make sure that it is worthwhile to design fast surrogates to classical PDE solvers for a particular problem. In other words, we will only consider PDEs where classical PDE solvers can only resolve the underlying operator on fine enough grids. To meet these requirements, we will not consider PDEs in one space dimension as traditional numerical methods are already quite fast for them. On the other hand, it is hard to obtain and store data for problems in three dimensions, due to computational expense of traditional methods. The sweet spot is achieved by considering PDEs in two space dimensions. We further restrict to Cartesian domains here as all models can be readily evaluated in this setting. In addition to including a diverse set of PDEs, we only consider problems with sufficiently many spatial and temporal scales. Otherwise, traditional numerical solvers can approximate the underlying PDE on very coarse grids and it is not worthwhile to design surrogates (see SM C.3.8 for a discussion in this context on a widely used Navier-Stokes benchmark). With these considerations in mind, we present the following subset of Representative PDE Benchmarks or RPB,
<sup>1</sup>The code can be found at https://github.com/bogdanraonic3/ConvolutionalNeuralOperator
Poisson Equation. This prototypical linear elliptic PDE is given by,
$$
- \Delta u = f, \text {i n} D, \quad u | _ {\partial D} = 0. \tag {4.1}
$$
The solution operator  $\mathcal{G}^{\dagger}:f\mapsto u$  , maps the source term  $f$  to the solution  $u$  . With source term,
$$
f (x, y) = \frac {\pi}{K ^ {2}} \sum_ {i, j = 1} ^ {K} a _ {i j} \cdot \left(i ^ {2} + j ^ {2}\right) ^ {- r} \sin \left(\pi i x\right) \sin \left(\pi j y\right), \quad \forall (x, y) \in D, \tag {4.2}
$$
with  $r = -0.5$ , the corresponding exact solution can be analytically computed (see SM C.3.1) and represents  $K$ -spatial scales. For training the models, we fix  $K = 16$  in (4.2) and choose  $a_{ij}$  to be i.i.d. uniformly distributed from  $[-1,1]$  (See SM D for a representation of the inputs and outputs of  $\mathcal{G}^{\dagger}$ ). This multiscale solution needs fine enough grid size to be approximated accurately by finite element methods, fitting our complexity criterion for benchmarks. In addition to in-distribution testing, we also consider an out-of-distribution testing task by setting  $K = 20$  in (4.2). This will enable us to evaluate the ability of the models to generalize to inputs (and outputs) with frequencies higher than those encountered during training.
Wave Equation. This prototypical linear hyperbolic PDE is given by
$$
u _ {t t} - c ^ {2} \Delta u = 0, \text {i n} D \times (0, T), \quad u _ {0} (x, y) = f (x, y), \tag {4.3}
$$
with a constant propagation speed  $c = 0.1$ . The underlying operator  $\mathcal{G}^{\dagger}: f \mapsto u(.,T)$  maps the initial condition  $f$  into the solution at the final time. If we consider initial conditions to be given by (4.2) with  $r = 1$ , then one can explicitly compute the exact solution (see SM C.3.2) to represent a multiscale standing wave with periodic pulsations (depending on  $K$ ) in time. The training and in-distribution test samples are generated by setting  $T = 5$ ,  $K = 24$  and  $a_{ij}$  to be i.i.d. uniformly distributed from  $[-1,1]$  (See SM D for input and output samples). For out-of-distribution testing, we change the exponent of decay of the modes in (4.2) to  $r = 0.85$  and  $K = 32$ , in order to test the ability of the models to generalize to learn the effect of higher frequencies, than those present in the training data.
Transport Equation. The transport of scalar quantities of interest is modeled by PDE,
$$
u _ {t} + v \cdot \nabla u = 0, \quad u (t = 0) = f, \tag {4.4}
$$
with a given velocity field and initial data  $f$ . The underlying operator  $\mathcal{G}^{\dagger}: f \mapsto u(.,T = 1)$  maps the initial condition  $f$  into the solution at the final time. We set a constant velocity field  $v = (v_{x},v_{y}) = (0.2,0.2)$  leading to solution  $u(x,y,t) = f(x - v_{x}t,y - v_{y}t)$ . Two different types of training data are considered, i.e., smooth initial data which takes the form of a radially symmetric Gaussian, with centers randomly and uniformly drawn from  $(0.2,0.4)^2$  and corresponding variance drawn uniformly from  $(0.003,0.009)$  and a discontinuous initial data in the form of the indicator function of radial disk with centers, uniformly drawn from  $(0.2,0.4)^2$  and radii uniformly drawn from  $(0.1,0.2)$  (See SM C.3.3 for details and SM D for illustrations). For out-of-distribution testing in the smooth case, the centers of the Gaussian inputs are sampled uniformly from  $(0.4,0.6)^2$  and in the discontinuous case, the centers of the disk are drawn uniformly from  $(0.4,0.6)^2$ , while keeping the variance and the radii, respectively, the same as that of in-distribution testing. This out-of-distribution task tests the model's ability to cope with input translation-equivariance.
Allen-Cahn Equation. It is a prototype for nonlinear parabolic PDEs,
$$
u _ {t} = \Delta u - \varepsilon^ {2} u \left(u ^ {2} - 1\right), \tag {4.5}
$$
with a reaction rate of  $\varepsilon = 220$  and underlying operator  $\mathcal{G}^{\dagger}:f\mapsto u(.T)$ , mapping initial conditions  $f$  to the solution  $u$  at a final time  $T = 0.0002$ . The initial conditions for training and in-distribution testing are of the form (4.2), with  $r = 1$  and  $K = 24$  and coefficients  $a_{ij}$  drawn uniformly from  $[-1,1]$ . For out-of-distribution testing, we set  $K = 16$  and randomly select the initial decay  $r$ , uniformly from the range [0.85, 1.15] of the modes in (4.2), which allows us to test the ability of the model to generalize to different dynamics of the system. Both training and test data are generated by using a finite difference scheme [62] on a grid at  $64^{2}$  resolution (see SM D for illustrations).
Navier-Stokes Eqns. These PDEs model the motion of incompressible fluids by,
$$
u _ {t} + (u \cdot \nabla) u + \nabla p = \nu \Delta u, \quad \operatorname {d i v} u = 0, \tag {4.6}
$$
in the torus  $D = \mathbb{T}^2$  with periodic boundary conditions and viscosity  $\nu = 4 \times 10^{-4}$ , only applied to high-enough Fourier modes (those with amplitude  $\geq 12$ ) to model fluid flow at very high Reynolds number. The solution operator  $\mathcal{G}^\dagger : f \mapsto u(., T)$ , maps the initial conditions  $f: D \to \mathbb{R}^2$  to the solution at final time  $T = 1$ . We consider initial conditions representing the well-known thin shear layer problem [3, 27] (See SM C.3.5 for details), where the shear layer evolves via vortex shedding to a complex distribution of vortices (see SM D for samples). The training and in-distribution testing samples are generated, with a spectral viscosity method [27], from an initial sinusoidal perturbation of the shear layer [27], with layer thickness  $\rho = 0.1$  and 10 perturbation modes, each sampled uniformly from  $[-1, 1]$ . For out-of-distribution testing, the layer thickness is reduced to  $\rho = 0.09$  and the layers are shifted up in the domain to test the ability of the models to generalize to a flow regime with an increased number and different locations of the shed vortices.
Darcy flow. The steady-state Darcy flow is described by the second order linear elliptic PDE,
$$
- \nabla \cdot (a \nabla u) = f, \text {i n} D, \quad u | _ {\partial D} = 0, \tag {4.7}
$$
where  $a$  is the diffusion coefficient and  $f$  is the forcing term. We set the forcing term to  $f = 1$ . The solution operator is  $\mathcal{G}^{\dagger}: a \mapsto u$ , where the input is the diffusion coefficient  $a \sim \psi \# \mu$ , with  $\mu$  being a Gaussian Process with zero mean and squared exponential kernel
$$
k (x, y) = \sigma^ {2} \exp \left(\frac {\left| x - y \right| ^ {2}}{l ^ {2}}\right), \quad \sigma^ {2} = 0. 1. \tag {4.8}
$$
We chose the length scale  $l = 0.1$  for the in-distribution testing and  $l = 0.05$  for the out-of-distribution testing. The mapping  $\psi : \mathbb{R} \to \mathbb{R}$  takes the value 12 on the positive part of the real line and 3 on the negative part. The push-forward measure is defined pointwise. The experimental setup is the same as the one presented in [33].
Flow past airfoils. We model this flow by the compressible Euler equations,
$$
u _ {t} + \operatorname {d i v} F (u) = 0, u = [ \rho , \rho v, E ] ^ {\perp}, F = [ \rho v, \rho v \otimes v + p \mathbf {I}, (E + p) ] v ] ^ {\perp}, \tag {4.9}
$$
with density  $\rho$ , velocity  $v$ , pressure  $p$  and total Energy  $E$  related by an ideal gas equation of state. The airfoils we consider are described by perturbing the shape of a well-known RAE2822 airfoil [40] by Hicks-Henne Bump functions [42] (see SM C.3.7). Freestream boundary conditions are imposed and the solution operator maps the shape function onto the steady state density distribution (see SM D for samples) and training data are obtained with a compressible flow solver (NUWTUN) with shapes corresponding to 20 bump functions, with coefficients sampled uniformly from  $[0,1]$ . Out-of-distribution testing is performed with 30 bump functions.
Results. The test errors, for both in-distribution and out-of-distribution testing for all the models on the RPB benchmarks are shown in Table 1. Starting with the in-distribution results, we see that among the baselines, FNO clearly outperforms both FFNN and DeepONet on all the RPB benchmarks as well as the Galerkin Transformer on all except the Poisson test case. On the other hand, the convolution-based U-Net and ResNet models are quite competitive vis-a-vis FNO, with comparable performances on most benchmarks, while outperforming FNO by a factor of  $7 - 9$  for the Poisson test case. This already indicates that convolution-based architectures can perform very well. Moreover, we observe from Table 1 that CNO is the best performing architecture on every task except Allen-Cahn. It readily outperforms FNO, for instance by almost a factor of 20 on the Poisson test case but more moderately but significantly on other tasks. It also outperforms U-Net and ResNet on all tasks considered here, emerging as the best-performing model over these benchmarks.
Out-of-Distribution Testing. This trend is further reinforced when we consider out-of-distribution testing. CNO generalizes well to unseen data in a zero-shot mode, with test errors increasing by approximately a factor of 2, at most (with Allen-Cahn being an outlier) and still outperforms the baselines significantly in all cases other than Allen-Cahn, where FNO generalizes the best. FNO shows decent generalization for most problems but generalizes poorly on the transport problems. This can be attributed to its lack of translation equivariance, in contrast to U-Net, ResNet and CNO. Finally, the lack of translation invariance severely limits the out-of-distribution generalization performance of DeepONet, FFNN and Galerkin Transformer models on transport problems.
Figure 2: Thin Shear Layer Left: Averaged logarithmic amplitude spectra comparing Ground Truth, CNO, FNO and UNet. Right: Test error vs. Resolution for UNet, FNO and CNO.
Resolution Invariance. We select three of the best-performing models (U-Net, FNO and CNO) and highlight further differences between them for the Navier-Stokes test case (see also SM C). To this end, we start with Figure 2 (left), where we present the averaged (log) spectra for the ground truth (reference solution computed with the spectral viscosity method) and those computed with CNO, FNO and U-Net. We observe from this figure that i) the spectrum of the exact solution is very rich with representations of many frequencies, attesting to the multi-scale nature of the underlying problem and ii) there are significant differences in how CNO and FNO approximate the underlying solution in Fourier space. In particular, the decay in CNO's spectrum is more accurate. On the other hand, the FNO spectrum is amplified along the horizontal axis, possibly on account of aliasing errors that add incorrect frequency content. The U-Net spectra are similar to that of CNO but with high-frequency modes being amplified, which leads to a higher test error. Next, in Figure 2 (right), we compare CNO, FNO and U-Net vis-a-vis the metric of how the test error varies across resolutions, see SMC.4 for details, which is an important aspect for robust operator learning that been highlighted in [33, 25], see also [2] for a discussion with respect to representation equivalent neural operators or ReNOs. We find from Figure 2 (right) that for the Navier-Stokes benchmark, the FNO error is not invariant with respect to resolution, with an increase in error of up to  $25\%$  on lower-resolutions as well as a more modest but noticeable increase of  $10\%$  on resolutions, higher than the training resolution of  $64^2$ , implying that FNO is neither able to perform alias-error free super- or sub-resolution in this case, see also [2] for a detailed discussion on the resolution-invariance of FNO. Similarly, the increase of U-Net test error with respect to varying resolutions is even more pronounced, with a maximum increase of a factor of 3, indicating neither FNO nor U-Net are resolution (representation) equivalent in this case. In contrast, CNO error is invariant with respect to test resolution, verifying that it respects continuous-discrete equivalence. Further ablation studies for CNO are presented in SM C.5.
Efficiency. In order to further compare CNO with FNO, which is the most widely used neural operator these days, we illustrate not just the performance in terms of errors but also terms in computational efficiency. To this end, in Figure 3 (left), we plot the size (total number of parameters) vs validation error for the Navier-Stokes benchmark for all FNO and CNO models that were considered in the model selection procedure to observe that for the same model size, CNO models led to significantly smaller validation errors. This resulted in smaller errors for the same per-epoch training time (Figure 3 (center)) for CNO vis-a-vis FNO. As observed in Figure 3 (center), this also implies that one of the best-performing CNO models, with a significantly smaller error than the best-performing FNO model is also almost twice as fast to train, thus showcasing the computational efficiency of CNOs.
Scaling laws. A very important aspect of modern deep learning is to evaluate how the performance of models scales with respect to data (and model size). To investigate this, we focus on the Navier-Stokes benchmark and present test error vs. (log of) number of training samples for GT, FNO and CNO in Figure 3 (right) to observe that the errors decrease consistently with number of samples. To quantify the rates of decrease, we fit power laws of the form  $E = (N_0 / N)^{-r}$ , with  $E$  being the test error,  $N$  number of samples and  $N_0$  normalized to be the number of samples required to reach errors of  $1\%$  to obtain that CNO attains a much faster convergence rate ( $r = 0.37$ ) compared to FNO ( $r = 0.28$ ) and GT ( $r = 0.27$ ). This implies that CNO will require  $N_0 \approx 14.3K$  samples to attain  $1\%$  error which is 4-times less than FNO ( $N_0 \approx 60.1K$ ) and almost 10-times less than GT ( $N_0 \approx 133.2K$ ), highlighting the data efficiency of CNO. Finally, from Figure 3 (right), we also observe that a smaller CNO model (with  $0.82\mathrm{M}$  parameters) scales worse (with  $r = 0.3$ ) compared to the best performing CNO model with  $7.8M$  parameters, illustrating that model size could be a bottleneck for data scaling and larger models are required to scale with data.
Figure 3: Navier-Stokes Benchmark: Test Error (Y-axis) vs. Model size (Left) and per-epoch training time (Center) for all the FNO and CNO models tested. The best-performing models are highlighted as is a small-scale yet efficient CNO model. Right: Log of Error vs. Log of Number of training samples for GT, FNO, CNO and a small-scale CNO model.
5 Discussion.
Summary. We propose CNO, a novel convolution-based architecture for learning operators. The basic design principle was to enforce a form of continuous-discrete equivalence in order to genuinely learn the underlying operators, rather than discrete representations of them. To this end, we modified the elementary operators of convolution, up-and downsampling and particularly nonlinear activations to realize CNO as a representation equivalent neural operator or ReNO in the sense of [2]. We also prove a universality theorem to show that CNO can approximate a large class of operators arising in PDEs to desired accuracy. A novel suite of experiments, termed as representative PDE benchmarks (RPB), encompassing a wide variety of PDEs, with multiple scales in the corresponding solutions, which are hard to resolve with traditional numerical methods, is also proposed and the model tested on them. We demonstrate that CNO outperforms the baselines, including FNO, significantly on most benchmarks. This also holds for the considered out-of-distribution testing tasks which ascertain the ability of the models to generalize to unseen data in a zero-shot manner.
Comparison to Related Work. We emphasize that our construction of CNO follows the theoretical prescription of recent paper [2] on enforcing structure preserving continuous-discrete equivalence. CNO is a representation equivalent neural operator, with respect to spaces of bandlimited functions, in the sense of [2]. Another motivating work for us is [22], see also [58], where the authors modify CNNs to eliminate (or reduce) aliasing errors in the context of image generation. We adapt the construction of [22] to our setting and deploy the resulting architecture in a very different context from that of [22], namely that of operator learning for PDEs rather than image generation. Moreover, we also instantiate CNO with a very different operator UNet architecture than that proposed in [22]. We would also like to mention related work on using CNNs for solving PDEs such as [1, 16] and emphasize that in contrast to CNO, they lack suitable notions of continuous-discrete equivalence. Finally comparing CNO to the widely used FNO model, we observe that unlike FNO which can fail to enforce CDE (see [14, 2] and Figure 2(right)), CNO preserves continuous-discrete equivalence. Moreover, the convolution operator in CNO is local in space, in contrast to convolution in Fourier space for FNO. The detailed empirical comparison presented here demonstrates CNO can outperform FNO and other baselines on many different metrics namely performance, computational efficiency, resolution invariance, out-of-distribution generalization as well as data scaling, paving the way for its widespread applications in science and engineering.
Limitations and Future Work. We have presented CNO for operators on an underlying two-dimensional Cartesian domain. The extension to three-space dimensions is conceptually straightforward but computationally demanding. Similarly, extending to non-Cartesian domains will require some form of transformation maps between domains, for instance reworking those suggested for FNO in [32, 55] can be readily considered. Adapting CNO to approximate trajectories (in time) of time-dependent PDEs, for instance by employing it in an auto-regressive manner, is another possible extension of this paper. At the level of theoretical results, we believe that the generic framework of [11] can be adapted to show that not only does CNO approximate a large class of PDEs universally, it does so without incurring any curse of dimensionality, as shown for DeepONets in [26] and FNOs in [24]. Finally, adapting and testing CNO for learning operators, beyond the forward solution operator of PDEs is also interesting. One such direction lies in efficiently approximating  $PDE$  inverse problems, for instance those considered in [44].
[1] I. Ayed, E. de Bézenac, A. Pajot, J. Brajard, and P. Gallinari. Learning dynamical systems from partial observations. CoRR, abs/1902.11136, 2019.
[2] F. Bartolucci, E. de Bezenac, B. Raonic, R. Molinaro, S. Mishra, and R. Alaifari. Are neural operators really neural operators? frame theory meets operator learning. Technical Report 2023-21, Seminar for Applied Mathematics, ETH Zürich, Switzerland, 2023.
[3] J. Bell, P. Collela, and H. M. Glaz. A second-order projection method for the incompressible Navier-Stokes equations. J. Comput. Phys., 85:257-283, 1989.
[4] M. Bertero, P. Bocacci, and C. De Mol. Introduction to inverse problems in imaging. CRC press, 2021.
[5] K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021.
[6] N. Boullé, Y. Nakatsukasa, and A. Townsend. Rational neural networks. Advances in Neural Information Processing Systems, 33:14243-14253, 2020.
[7] S. Cai, Z. Wang, L. Lu, T. A. Zaki, and G. E. Karniadakis. DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics, 436:110296, 2021.
[8] S. Cao. Choose a transformer: Fourier or galerkin. In 35th conference on neural information processing systems, 2021.
[9] T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917, 1995.
[10] T. De Ryck, S. Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 2021.
[11] T. De Ryck and S. Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[12] Q. Delfosse, P. Schramowski, M. Mundt, A. Molina, and K. Kersting. Adaptive rational activations to boost deep reinforcement learning. arXiv preprint arXiv:2102.09407, 2021.
[13] L. C. Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
[14] V. Fanaskov and I. Oseledets. Spectral neural operators. arXiv preprint arXiv:2205.10573v1, 2022.
[15] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning, volume 1. MIT Press, 2016.
[16] J. K. Gupta and J. Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling, 2022.
[17] E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34, 2018.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv: 1512.03385, 2015.
[19] J. S. Hesthaven, S. Gottlieb, and D. Gottlieb. Spectral methods for time-dependent problems, volume 21. Cambridge University Press, 2007.
[20] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.