SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bogdan Raonic $^{1,2}$  Roberto Molinaro $^{1}$  Tim De Ryck $^{1}$  Tobias Rohner $^{1}$
Francesca Bartolucci<sup>3</sup> Rima Alaifari<sup>1</sup> Siddhartha Mishra<sup>1,2</sup> Emmanuel de Bezenac<sup>1</sup>
1 Seminar for Applied Mathematics, ETH, Zurich, Switzerland
2 ETH AI Center, Zurich, Switzerland
3 Delft University of Technology, Netherlands
Although very successfully used in conventional machine learning, convolution based neural network architectures – believed to be inconsistent in function space – have been largely ignored in the context of learning solution operators of PDEs. Here, we present novel adaptations for convolutional neural networks to demonstrate that they are indeed able to process functions as inputs and outputs. The resulting architecture, termed as convolutional neural operators (CNOs), is designed specifically to preserve its underlying continuous nature, even when implemented in a discretized form on a computer. We prove a universality theorem to show that CNOs can approximate operators arising in PDEs to desired accuracy. CNOs are tested on a novel suite of benchmarks, encompassing a diverse set of PDEs with possibly multi-scale solutions and are observed to significantly outperform baselines, paving the way for an alternative framework for robust and accurate operator learning.
1 Introduction.
Partial Differential Equations (PDEs) [13] are ubiquitous as mathematical models in the sciences and engineering. Solving a PDE amounts to (approximately) computing the so-called solution operator that maps function space inputs such as initial and boundary conditions, coefficients, source terms etc, to the PDE solution which also belongs to a suitable function space. Well-established numerical methods such as finite differences, finite elements, finite volumes and spectral methods (see [48]) have been very successfully used for many decades to approximate PDE solution operators. However, the prohibitive computational cost of these methods, particularly in high dimensions and for many query problems such as UQ, inverse problems, PDE-constrained control and optimization, necessitates the design of fast, robust and accurate surrogates. This provides the rationale for the use of data-driven machine learning methods for solving PDEs [21].
As operators are the objects of interest in solving PDEs, learning such operators from data, which is loosely termed as operator learning, has emerged as a dominant paradigm in recent years for the applications of machine learning to PDEs. A very partial list of architectures for operator learning include operator networks [9], DeepONets [39] and its variants [41, 7], PCA-net [5], neural operators [25] such as graph neural operator [34], Multipole neural operator [35] and the very popular Fourier Neural Operator [33] and its variants [36, 45], VIDON [47], spectral neural operator [14], LOCA [23], NOMAD [52] and transformer based operator learning architectures [8].
Despite the considerable success of the recently proposed operator learning architectures, several pressing issues remain to be addressed. These include, but are by no means restricted to, limited expressivity for some of these algorithms [28] to aliasing errors for others [14] to the very fundamental issue of possible lack of consistency in function spaces for many of them. As argued in a recent paper [2], a structure-preserving operator learning algorithm or representation equivalent neural operator has to respect some form of continuous-discrete equivalence (CDE) in order to learn the underlying operator, rather than just a discrete representation of it. Failure to respect such a CDE can lead to the so-called aliasing errors [2] and affect model performance at multiple discrete resolutions.
Despite many attempts, see [1, 16] and references therein, the absence of a suitable CDE, resulting in aliasing errors, has also plagued the naive use of convolutional neural networks (CNNs) in the context of operator learning, see [63, 33, 2] on how using CNNs for operator learning leads to results that heavily rely on the underlying grid resolution. This very limited use of Convolution (in physical space) based architectures for operator learning stands in complete contrast to the fact that CNNs [30] and their variants are widely used architectures for image classification and generation and in other contexts in machine learning [29, 37, 61]. Moreover, CNNs can be thought of as natural generalizations of the foundational finite difference methods for discretizing PDEs [17, 38]. Given their innate locality, computational and data efficiency, ability to process multi-scale inputs and outputs and the availability of a wide variety of successful CNN architectures in other fields, it could be very advantageous to bring CNN-based algorithms back into the reckoning for operator learning. This is precisely the central point of the current paper where we make the following contributions,
- We propose novel modifications to CNNs in order to enforce structure-preserving continuous-discrete equivalence and enable the genuine, alias-free, learning of operators. The resulting architecture, termed as Convolutional Neural Operator (CNO), is instantiated as a novel operator adaptation of the widely used U-Net architecture.
- In addition to showing that CNO is a representation equivalent neural operator in the sense of [2], we also prove a universality result to rigorously demonstrate that CNOs can approximate the operators, corresponding to a large class of PDEs, to desired accuracy.
- We test CNO on a novel set of benchmarks, that we term as Representative PDE Benchmarks (RPB), that span across a variety of PDEs ranging from linear elliptic and hyperbolic to nonlinear parabolic and hyperbolic PDEs, with possibly multiscale solutions. We find that CNO is either on-par or outperforms the tested baselines on all the benchmarks, both when testing in-distribution as well as in out-of-distribution testing.
Thus, we present a new CNN-based operator learning model, with desirable theoretical properties and excellent empirical performance, with the potential to be widely used for learning PDEs.
2 Convolutional Neural Operators.
Figure 1: Schematic representation of CNO (2.3) as a modified U-Net with a sequence of layers (each identified with the relevant operators on the right, see Section 2) mapping between bandlimited functions. Rectangles represent multi-channel signals. Larger the height, larger is the resolution. Wider the rectangles, more channels are present.
Setting. For simplicity of the exposition, we will focus here on the two-dimensional case by specifying the underlying domain as  $D = \mathbb{T}^2$ , being the 2-d torus. Let  $\mathcal{X} = H^{r}(D,\mathbb{R}^{d\mathcal{X}}) \subset \mathcal{Z}$  and  $\mathcal{Y} = H^{s}(D,\mathbb{R}^{d\mathcal{Y}})$  be the underlying function spaces, where  $H^{r,s}(D,\cdot)$  are Sobolev spaces of order  $r$  and  $s$ . Without loss of generality, we set  $r = s$  hereafter. Our aim would be to approximate continuous operators  $\mathcal{G}^\dagger : \mathcal{X} \to \mathcal{Y}$  from data pairs  $(u_i, \mathcal{G}^\dagger(u_i))_{i=1}^M \in \mathcal{X} \times \mathcal{Y}$ . We further assume that there exists a modulus of continuity for the operator i.e.,
$$
\left\| \mathcal {G} ^ {\dagger} (u) - \mathcal {G} ^ {\dagger} (v) \right\| _ {\mathcal {Y}} \leq \omega \left(\left\| u - v \right\| _ {\mathcal {Z}}\right), \quad \forall u, v \in \mathcal {X}, \tag {2.1}
$$
with  $\omega : \mathbb{R}_+ \to \mathbb{R}_+$  being a monotonically increasing function with  $\lim_{y \to 0} \omega(y) = 0$ . The underlying operator  $\mathcal{G}^\dagger$  can correspond to solution operators for PDEs (see Section 3 for the exact setting) but is more general than that and encompasses examples such as those arising in inverse problems, for instance in imaging [4].
Bandlimited Approximation. As argued in a recent paper [2], Sobolev spaces such as  $H^r$  are, in a sense, too large to allow for any form of continuous-discrete equivalence (CDE), i.e., equivalence between the underlying operator and its discrete representations, which is necessary for robust operator learning. Consequently, one has to consider smaller subspaces of  $H^r$  which allow for such CDEs. In this respect, we choose the space of bandlimited functions [57] defined by,
$$
\mathcal {B} _ {w} (D) = \{f \in L ^ {2} (D): \operatorname {s u p p} \widehat {f} \subseteq [ - w, w ] ^ {2} \}, \tag {2.2}
$$
for some  $w > 0$  and with  $\widehat{f}$  denoting the Fourier transform of  $f$ . It is straightforward to show using (2.1) (see SMA.1) that for any  $\varepsilon > 0$ , there exists a  $w$ , large enough depending on  $r$ , and a continuous operator  $\mathcal{G}^* : \mathcal{B}_w(D) \to \mathcal{B}_w(D)$ , such that  $\| \mathcal{G}^\dagger - \mathcal{G}^* \| < \varepsilon$ , with  $\| \cdot \|$  denoting the corresponding operator norm. In other words, the underlying operator  $\mathcal{G}^\dagger$  can be approximated to arbitrary accuracy by the operator  $\mathcal{G}^*$  that maps between band-limited spaces. Consequently, as shown in SMA.2, one can readily define discrete versions of  $\mathcal{G}^*$  using the underlying sinc basis for bandlimited functions and establish a continuous-discrete equivalence for it.
Definition of CNO. Given the above context, our goal will be to approximate the operator  $\mathcal{G}^*$  in a structure-preserving manner i.e., as the underlying operator maps between spaces of bandlimited functions, we will construct our operator approximation architecture to also map bandlimited functions to bandlimited functions, thus respecting the continuous-discrete equivalence. To this end, we denote the operator  $\mathcal{G}:\mathcal{B}_w(D)\to \mathcal{B}_w(D)$  as a convolutional neural operator (CNO) which we define as a compositional mapping between functions as
$$
\mathcal {G}: u \mapsto P (u) = v _ {0} \mapsto v _ {1} \mapsto \dots v _ {L} \mapsto Q (v _ {L}) = \bar {u}, \tag {2.3}
$$
$$
v _ {l + 1} = \mathcal {P} _ {l} \circ \Sigma_ {l} \circ \mathcal {K} _ {l} (v _ {l}), \quad 1 \leq \ell \leq L - 1. \tag {2.4}
$$
From (2.3), we see that first, the input function  $u \in \mathcal{B}_w(D)$  is lifted to the latent space of bandlimited functions through a lifting layer:
$$
P: \left\{u \in \mathcal {B} _ {w} (D, \mathbb {R} ^ {d _ {\mathcal {X}}}) \right\}\rightarrow \left\{v _ {0} \in \mathcal {B} _ {w} (D, \mathbb {R} ^ {d _ {0}}) \right\}.
$$
Here,  $d_0 > d_{\mathcal{X}}$  is the number of channels in the lifted, latent space. The lifting operation is performed by a convolution operator which will be defined below.
Then, the lifted function is processed through the composition of a series of mappings between functions (layers), with each layer consisting of three elementary mappings, i.e.,  $\mathcal{P}_l$  is either the upsampling or downsampling operator,  $\kappa_{l}$  is the convolution operator and  $\Sigma_l$  is the activation operator. These elementary operators are defined below and are inspired by the modifications of CNNs for image generation in [22]. Finally, the last output function in the iterative procedure  $v_{L}$  is projected to the output space with a projection operator  $Q$ , defined as
$$
Q: \left\{v _ {L} \in \mathcal {B} _ {w} (D, \mathbb {R} ^ {d _ {L}}) \right\}\rightarrow \left\{\bar {u} \in \mathcal {B} _ {w} (D, \mathbb {R} ^ {d _ {y}}) \right\}.
$$
The projection operation is also performed by a convolution operator defined below.
Convolution Operator. For simplicity of exposition, we will present the single-channel version of the convolution operator  $\mathcal{K}_l$  here. See SM A.3 for the multi-channel version for this and other operators considered below. Convolution operations are performed with discrete kernels
$$
K _ {w} = \sum_ {i, j = 1} ^ {k} k _ {i j} \cdot \delta_ {z _ {i j}}
$$
defined on the  $s \times s$  uniform grid on  $D$  with grid size  $\leq 1/2w$ , in-order to satisfy the requirements of the Whittaker-Shannon-Kotelnikov sampling theorem [56], and  $z_{ij}$  being the resulting grid points,  $k \in \mathbb{N}$  being the kernel size and  $\delta_x$  denoting the Dirac measure at point  $x \in D$ . The convolution operator for a single-channel  $\mathcal{K}_w: \mathcal{B}_w(D) \to \mathcal{B}_w(D)$  is defined by
$$
\mathcal {K} _ {w} f (x) = \left(K _ {w} \star f\right) (x) = \int_ {D} K _ {w} (x - y) f (y) d y = \sum_ {i, j = 1} ^ {k} k _ {i j} f (x - z _ {i j}), \quad \forall x \in D,
$$
where the last identity arises from the fact that  $f \in \mathcal{B}_w$ . Thus, our convolution operator is directly parametrized in physical space, in contrast to the Fourier space parametrization of a convolution in the FNO architecture of [33]. Hence, our parametrization is of a local nature.
Upsampling and Downsampling Operators. For some  $\overline{w} > w$ , we can upsample a function  $f \in \mathcal{B}_w$  to the higher band  $\mathcal{B}_{\overline{w}}$  by simply setting,
$$
\mathcal {U} _ {w, \bar {w}}: \mathcal {B} _ {w} (D) \to \mathcal {B} _ {\bar {w}} (D), \quad \mathcal {U} _ {w, \bar {w}} f (x) = f (x), \quad \forall x \in D.
$$
On the other hand, for some  $\underline{w} < w$ , we can downsample a function  $f \in \mathcal{B}_w$  to the lower band  $\mathcal{B}_{\underline{w}}$  by setting  $\mathcal{D}_{w,\underline{w}}: \mathcal{B}_w(D) \to \mathcal{B}_{\underline{w}}(D)$ , defined by
$$
\mathcal {D} _ {w, \underline {{w}}} f (x) = \left(\frac {\underline {{w}}}{w}\right) ^ {2} (h _ {\underline {{w}}} \star f) (x) = \left(\frac {\underline {{w}}}{w}\right) ^ {2} \int_ {D} h _ {\underline {{w}}} (x - y) f (y) d y, \quad \forall x \in D,
$$
where  $\star$  is the convolution operation on functions defined above and  $h_{\underline{w}}$  is the so-called interpolation sinc filter:
$$
h _ {w} \left(x _ {0}, x _ {1}\right) = \operatorname {s i n c} \left(2 w x _ {0}\right) \cdot \operatorname {s i n c} \left(2 w x _ {1}\right), \quad \left(x _ {0}, x _ {1}\right) \in \mathbb {R} ^ {2}. \tag {2.5}
$$
Activation Layer. Naively, one can apply the activation function pointwise to any function. However, it is well-known that such an application will no longer respect the band-limits of the underlying function space and generate aliasing errors [22, 14, 2]. In particular, nonlinear activations can generate features at arbitrarily high frequencies. As our aim is to respect the underlying CDE, we will modulate the application of the activation function so that the resulting outputs fall within desired band limits. To this end, we first upsample the input function  $f \in \mathcal{B}_w$  to a higher bandlimit  $\overline{w} > w$ , then apply the activation and finally downsample the result back to the original bandlimit  $w$  (See Figure 1). Implicitly assuming that  $\overline{w}$  is large enough such that  $\sigma(\mathcal{B}_w) \subset \mathcal{B}_{\overline{w}}$ , we define the activation layer in (2.3) as,
$$
\Sigma_ {w, \bar {w}}: \mathcal {B} _ {w} (D) \rightarrow \mathcal {B} _ {w} (D), \quad \Sigma_ {w, \bar {w}} f (x) = \mathcal {D} _ {\bar {w}, w} (\sigma \circ \mathcal {U} _ {w, \tilde {w}} f) (x), \quad \forall x \in D. \tag {2.6}
$$
Instantiation through an Operator U-Net architecture. The above ingredients are assembled together in the form of an Operator U-Net architecture that has bandlimited functions as inputs and outputs. In addition to the blocks that have been defined above, we also need additional ingredients, namely incorporate skip connections through ResNet blocks of the form,  $\mathcal{R}_{w,\overline{w}}: \mathcal{B}_w(D, \mathbb{R}^d) \to \mathcal{B}_w(D, \mathbb{R}^d)$  such that
$$
\mathcal {R} _ {w, \bar {w}} (v) = v + \mathcal {K} _ {w} \circ \Sigma_ {w, \bar {w}} \circ \mathcal {K} _ {w} (v), \quad \forall v \in \mathcal {B} _ {w} (D, \mathbb {R} ^ {d}). \tag {2.7}
$$
We also need the so-called Invariant blocks of the form,  $\mathcal{I}_{w,\overline{w}}:\mathcal{B}_w(D,\mathbb{R}^d)\to \mathcal{B}_w(D,\mathbb{R}^d)$  such that
$$
\mathcal {I} _ {w, \bar {w}} (v) = \Sigma_ {w, \bar {w}} \circ \mathcal {K} _ {w} (v), \quad \forall v \in \mathcal {B} _ {w} (D, \mathbb {R} ^ {d}). \tag {2.8}
$$
Finally, all these ingredients are assembled together in a modified Operator U-Net architecture which is graphically depicted in Figure 1. As seen from this figure, the input function, say  $u \in \mathcal{B}_w(D,\mathbb{R}^{d_x})$  is first lifted and then processed through a series of layers. Four types of blocks are used i.e.,
downsampling (D) block corresponding to using the downsampling operator  $\mathcal{D}$  as the  $\mathcal{P}$  in (2.4), upsampling (U) block corresponding to using the upsampling operator  $\mathcal{U}$  as the  $\mathcal{P}$  in (2.4), ResNet (R) block corresponding to (2.7) and Invariant (I) block corresponding to (2.8). Each block takes a band-limited function as input and returns another band-limited function (with the same band) as the output. Finally, U-Net style patching operators, which concatenate outputs for different layers as additional channels are also used. As these operations act only in the channel width and leave the spatial resolution unchanged, they conform to the underlying bandlimits. Thus, CNO takes a function input and passes it through a set of encoders, where the input is downsampled in space but expanded in channel width and then processed through a set of decoders, where the channel width is reduced but the space resolution is increased. At the same time, encoder and decoder layers (at the same spatial resolution or band limit) are connected through additional ResNet blocks. Thus, this architectural choice allows for transferring high frequency content via the skip connections, before filtering them out with the sinc filter as we go deeper into the encoder. Hence, the high frequency content is not just recreated with the activation function, but also modified through the intermediate networks. Consequently, we build a genuinely multiscale operator learning architecture.
Continuous-Discrete Equivalence for CNO. We have defined CNO (2.3) as an operator that maps bandlimited functions to bandlimited functions. In practice, like any computational algorithm, CNO has to be implemented in a discrete manner, with discretized versions of each of the above-defined elementary operations being specified in SM A.4. Given how each of the elementary blocks (convolution, up- and downsampling, activation, ResNets etc) are constructed, we prove the following proposition (in SM A.5):
Proposition 2.1. Convolutional Neural Operator  $\mathcal{G}:\mathcal{B}_w(D,\mathbb{R}^{d_{\mathcal{X}}})\to \mathcal{B}_w(D,\mathbb{R}^{d_{\mathcal{Y}}})$  (2.3) is a Representation equivalent neural operator or ReNO, in the sense of [2], Definition 3.4.
Further details about the notion of ReNOs is provided in SM A.4 and we refer the reader to [2], where this concept is presented in great detail and the representation equivalence of CNO is discussed. In particular, following [2], representation equivalence implies that CNO satisfies a form of resolution invariance, allowing it to be evaluated on multiple grid resolutions without aliasing errors.
3 Universal Approximation by CNOs.
We want to prove that a large class of operators, stemming from PDEs, can be approximated to desired accuracy by CNOs. To this end, we consider the following abstract PDE in the domain  $D = \mathbb{T}^2$ ,
$$
\mathcal {L} (u) = 0, \quad \mathcal {B} (u) = 0, \tag {3.1}
$$
with  $\mathcal{L}$  being a differential operator and  $\mathcal{B}$  a boundary operator. We assume that the differential operator  $\mathcal{L}$  only depends on the coordinate  $x$  through a coefficient function  $a\in H^r (D)$ . The corresponding solution operator is denoted by  $\mathcal{G}^{\dagger}:\mathcal{X}^{*}\subset H^{r}(D)\to H^{r}(D):a\mapsto u$ , with  $u$  being the solution of the PDE (3.1). We assume that  $\mathcal{G}^{\dagger}$  is continuous. Moreover, we also assume the following modulus of continuity,
$$
\left\| \mathcal {G} ^ {\dagger} (a) - \mathcal {G} ^ {\dagger} \left(a ^ {\prime}\right) \right\| _ {L ^ {p} \left(\mathbb {T} ^ {2}\right)} \leq \omega \left(\left\| a - a ^ {\prime} \right\| _ {H ^ {\sigma} \left(\mathbb {T} ^ {2}\right)}\right), \tag {3.2}
$$
for some  $p \in \{2, \infty\}$  and  $0 \leq \sigma \leq r - 1$ , and where  $\omega : [0, \infty) \to [0, \infty)$  is a monotonously increasing function with  $\lim_{y \to 0} \omega(y) = 0$ . (3.2) is automatically satisfied if  $\mathcal{X}^*$  is compact and  $\mathcal{G}^\dagger$  is continuous. Under these assumptions, we have the following universality theorem for CNOs (2.3),
Theorem 3.1. Let  $\sigma \in \mathbb{N}_0$  and  $p\in \{2,\infty \}$  as in (3.2),  $r > \max \{\sigma ,2 / p\}$  and  $B > 0$ . For any  $\varepsilon >0$  and any operator  $\mathcal{G}^{\dagger}$ , as defined above, there exists a CNO  $\mathcal{G}$  such that for every  $a\in \mathcal{X}^*$  with  $\| a\|_{H^r (D)}\leq B$  it holds,
$$
\left\| \mathcal {G} ^ {\dagger} (a) - \mathcal {G} (a) \right\| _ {L ^ {p} (D)} <   \varepsilon . \tag {3.3}
$$
In fact, we will prove a more general version of this theorem in SM B, where we also include additional source terms in the PDE (3.1).
4 Experiments.
Training Details and Baselines. We provide a detailed description of the implementation of CNO and the training (and test) protocol for CNO as well as all the baselines in SM C.1. To ensure a level playing field among all the tested models for each benchmark, we follow an ensemble training procedure by specifying a range for the underlying hyperparameters for each model and randomly selecting a subset of the hyperparameter space. For each such hyperparameter configuration, the corresponding models are trained on the benchmark and the configuration with smallest validation error is selected and the resulting test errors are reported, allowing us to identify and compare the best performing version of each model for every benchmark. We compare CNO with the following baselines: two very popular operator learning architectures, namely DeepONet (DON) [39] and FNO [33], a transformer based operator-learning architecture, i.e., Galerkin Transformer (GT) [8], feedforward neural network with residual connections (FFNN) [18] and the very widely-used ResNet [18] and U-Net [50] architectures.
Table 1: Relative median  $L^1$  test errors, for both in- and out-of-distribution testing, for different benchmarks and models.
In/Out	FFNN	GT	UNet	ResNet	DON	FNO	CNO
Poisson	In	5.74%	2.77%	0.71%	0.43%	12.92%	4.98%	0.21%
Equation	Out	5.35%	2.84%	1.27%	1.10%	9.15%	7.05%	0.27%
Wave	In	2.51%	1,44%	1.51%	0.79%	2.26%	1.02%	0.63%
Equation	Out	3.01%	1.79%	2.03%	1.36%	2.83%	1.77%	1.17%
Smooth	In	7.09%	0.98%	0.49%	0.39%	1.14%	0.28%	0.24%
Transport	Out	650.6%	875.4%	1.28%	0.96%	157.2%	3.90%	0.46%
Discontinuous	In	13.0%	1.55%	1.31%	1.01%	5.78%	1.15%	1.01%
Transport	Out	257.3%	22691.1%	1.35%	1.16%	117.1%	2.89%	1.09%
Allen-Cahn	In	18.27%	0.77%	0.82%	1.40%	13.63%	0.28%	0.54%
Equation	Out	46.93%	2.90%	2.18%	3.74%	19.86%	1.10%	2.23%
Navier-Stokes	In	8.05%	4.14%	3.54%	3.69%	11.64%	3.57%	2.76%
Equations	Out	16.12%	11.09%	10.93%	9.68%	15.05%	9.58%	7.04%
Darcy	In	2.14%	0.86%	0.54%	0.42%	1.13%	0.80%	0.38%
Flow	Out	2.23%	1.17%	0.64%	0.60%	1.61%	1.11%	0.50%
Compressible	In	0.78%	2.09%	0.38%	1.70%	1.93%	0.44%	0.35%
Euler	Out	1.34%	2.94%	0.76%	2.06%	2.88%	0.69%	0.59%