SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Francesca Bartolucci<sup>1</sup> Emmanuel de Bezenac<sup>2</sup> Bogdan Raonic<sup>2,3</sup>
Roberto Molinaro $^{2}$  Siddhartha Mishra $^{2,3}$  Rima Alaifari $^{2,3}$
1 Delft University of Technology, Netherlands
2 Seminar for Applied Mathematics, ETH, Zurich, Switzerland
3 ETH AI Center, Zurich, Switzerland
Recently, operator learning, or learning mappings between infinite-dimensional function spaces, has garnered significant attention, notably in relation to learning partial differential equations from data. Conceptually clear when outlined on paper, neural operators necessitate discretization in the transition to computer implementations. This step can compromise their integrity, often causing them to deviate from the underlying operators. This research offers a fresh take on neural operators with a framework Representation equivalent Neural Operators (ReNO) designed to address these issues. At its core is the concept of operator aliasing, which measures inconsistency between neural operators and their discrete representations. We explore this for widely-used operator learning techniques. Our findings detail how aliasing introduces errors when handling different discretizations and grids and loss of crucial continuous structures. More generally, this framework not only sheds light on existing challenges but, given its constructive and broad nature, also potentially offers tools for developing new neural operators.
Operators are mappings between infinite-dimensional function spaces. Prominent examples are solution operators for ordinary and partial differential equations (PDEs) [9] which map function space inputs such as initial and boundary data to the function space valued PDE solution. They are also the natural mathematical framework for inverse problems, both in the context of PDEs [12] and in (medical) imaging [3], where the object of interest is the inverse operator which maps observables to the underlying material property/image that needs to be inferred or reconstructed.
Given the prohibitive cost of traditional physics based algorithms for approximating operators, particularly those associated with PDEs, increasing attention is being paid in recent years to machine-learning based operator approximation. Such techniques for learning operators from data fall under the genre of operator learning. A (by no means comprehensive) list of examples for operator learning architectures include operator networks [6], DeepONets [20], Graph neural operators [18], multipole neural operators [19], PCA-Nets [4], Fourier neural operators (FNO) [17], VIDON [21], spectral neural operators [10], LOCA [13], NOMAD [24], Continuous Generative Neural Networks [1] and transformer based operator learning models [5].
However, there is still a lack of clarity on what constitutes operator learning? Clearly, an operator learning framework, or a neural operator in the nomenclature of [15], should be able to process functions as inputs and outputs. On the other hand, functions are continuous objects and in practice,
one may not have access to functions either at the input or output level. Instead, one can only access functions and perform computations with them on digital computers through their discrete representations such as point values on a grid, cell averages or in general, coefficients of an underlying basis. Hence, in practice, neural operators have to map discrete inputs to discrete outputs. This leads to a dichotomy, i.e., neural operators are designed to process functions as inputs/outputs but only have access to their discrete representations. Consequently, at any finite resolution, possible mismatches between the discretizations and the continuous versions of neural operators can lead to inconsistencies in the underlying function spaces. These inconsistency errors can propagate through the networks and may mar the performance of these algorithms. Moreover, important structural properties of the underlying operator, such as symmetries and conservation laws, hold at the continuous level. Inconsistent discretizations may not preserve these structural properties of the operator, leading to symmetry breaking etc. with attendant adverse consequences for operator approximation.
Addressing this possible inconsistency between neural operators and their discretizations is the central point of this article. To this end, we revisit and adapt existing notions of the relationship between continuous and discrete representations in signal processing, applied harmonic analysis and numerical analysis, with respect to the questions of sampling and interpolation. Roughly speaking, we aim to find a mathematical framework in which the continuous objects (functions) can be completely (uniquely and stably) recovered from their discrete representations (point evaluations, basis coefficients, etc.) at any resolution. Consequently, working with discrete values is tantamount to accessing the underlying continuous object. This equivalence between the continuous and the discrete is leveraged to define a class of neural operators, for which the discrete input-output representations or their continuous function space realizations are equivalent. On the other hand, lack of this equivalence leads to aliasing errors, which can propagate through the neural operator layers to adversely affect operator approximation. More concretely, our contributions are
- We provide a novel, very general unifying mathematical formalism to characterize a class of neural operators such that there is an equivalence between their continuous and discrete representations. These neural operators are termed as Representation equivalent Neural Operators or ReNOs. Our definition results in an automatically consistent function space formulation for ReNOs.
- To define ReNOs, we provide a novel and precise quantification of the notion of aliasing error for operators. Consequently, ReNOs are neural operators with zero aliasing error.
- We analyze existing operator learning architectures to find whether they are ReNOs or not.
- Synthetic numerical experiments are presented to illustrate ReNOs learn operators without aliasing and to also point out the practical consequences of aliasing errors particularly, with respect to evaluations of the neural operators on different grid resolutions.
2 A short background on Frame Theory and Aliasing
We start with a short discussion, revisiting concepts on the relationship between functions and their discrete representations. This is the very essence of frame theory, widely used in signal processing and applied harmonic analysis. We refer to SM D for a detailed introduction to frame theory.
Equivalence between functions and their point samples. For simplicity of exposition, we start with univariate functions  $f \in L^{2}(\mathbb{R})$  and the following question: Can a function be uniquely and stably recovered from its values, sampled from equispaced grid points  $\{f(nT)\}_{n \in \mathbb{Z}}$ ? The classical Whittaker-Shannon-Kotel'nikov (WSK) sampling theorem [25], which lies at the heart of digital-to-analog conversion, answers this question in the affirmative when the underlying function  $f \in \mathcal{B}_{\Omega}$ , i.e., it belongs to the Paley-Wiener space of band-limited functions  $\mathcal{B}_{\Omega} = \{f \in L^{2}(\mathbb{R}) : \mathrm{supp}\hat{f} \subseteq [-\Omega, \Omega]\}$ , for some  $\Omega > 0$ ,  $\hat{f}$  the Fourier transform of  $f$  and sampling rate  $1 / T \geq 2\Omega$ , with  $2\Omega$  termed as the Nyquist rate. The corresponding reconstruction formula is,
$$
f (x) = 2 T \Omega \sum_ {n \in \mathbb {Z}} f (n T) \operatorname {s i n c} (2 \Omega (x - n T)), \quad \operatorname {s i n c} (x) = \sin (\pi x) / (\pi x). \tag {2.1}
$$
We note that the sequence of functions  $\{\phi_n(x) = \mathrm{sinc}(2\Omega x - n)\}_{n\in \mathbb{Z}}$  constitutes an orthonormal basis for  $\mathcal{B}_{\Omega}$  and denote by  $\mathcal{P}_{\mathcal{B}_{\Omega}}:L^{2}(\mathbb{R})\to \mathcal{B}_{\Omega}$  the orthogonal projection operator onto  $\mathcal{B}_{\Omega}$ ,
$$
\mathcal {P} _ {\mathcal {B} _ {\Omega}} f = \sum_ {n \in \mathbb {Z}} \langle f, \phi_ {n} \rangle \phi_ {n} = \sum_ {n \in \mathbb {Z}} f \left(\frac {n}{2 \Omega}\right) \phi_ {n}, \tag {2.2}
$$
where the last equality is a consequence of  $\mathcal{B}_{\Omega}$  being a reproducing kernel Hilbert space with kernel  $\mathcal{K}_{\Omega}(x,y) = \mathrm{sinc}(2\Omega (x - y))$ . Hence,  $\mathcal{P}_{\mathcal{B}_{\Omega}}f$  corresponds to the right-hand side of (2.1) for  $T = 1 / 2\Omega$  and formula (2.1) is exact if and only if  $f\in \mathcal{B}_{\Omega}$ , i.e.,  $f = \mathcal{P}_{\mathcal{B}_{\Omega}}f\Longleftrightarrow f\in \mathcal{B}_{\Omega}$ .
What happens when we sample a function at a sampling rate below the Nyquist rate, i.e. when  $1 / T < 2\Omega$ ? Alternatively, what is the effect of approximating a function  $f \notin \mathcal{B}_{\Omega}$  by  $\mathcal{P}_{\mathcal{B}_{\Omega}}f$ ? Again, sampling theory provides an answer in the form of the aliasing error:
Definition 2.1. Aliasing for bandlimited functions[2, §5] The aliasing error function  $\varepsilon(f)$  and the corresponding aliasing error of  $f \in L^{2}(\mathbb{R})$  for sampling at the rate  $2\Omega$  are given by
$$
\varepsilon (f) = f - \mathcal {P} _ {\mathcal {B} _ {\Omega}} f, \quad a n d \quad \| \varepsilon (f) \| _ {2} = \| f - \mathcal {P} _ {\mathcal {B} _ {\Omega}} f \| _ {2}.
$$
If the aliasing error  $\varepsilon(f)$  is zero, i.e. if  $f \in \mathcal{B}_{\Omega}$ , we say that there is a continuous-discrete equivalence (CDE) between  $f$  and its samples  $\{f(n/2\Omega)\}_{n \in \mathbb{Z}}$ .
Continuous-Discrete Equivalence in Hilbert spaces. Next, we generalize the above described concept of continuous-discrete equivalence to any separable Hilbert space  $\mathcal{H}$ , with inner product  $\langle \cdot, \cdot \rangle$  and norm  $\|\cdot\|$ . A countable sequence of vectors  $\{f_i\}_{i \in I}$  in  $\mathcal{H}$  is a frame for  $\mathcal{H}$  if there exist constants  $A, B > 0$  such that for all  $f \in \mathcal{H}$
$$
A \| f \| ^ {2} \leq \sum_ {i \in I} | \langle f, f _ {i} \rangle | ^ {2} \leq B \| f \| ^ {2}. \tag {2.3}
$$
Clearly, an orthonormal basis for  $\mathcal{H}$  is an example of a frame with  $A = B = 1$ . We will now define maps that will allow us to make the link between functions and their discrete representations, which we will extensively use in the following sections. The bounded operator  $T\colon \ell^2 (I)\to \mathcal{H}$ ,  $T(\{c_i\}_{i\in I}) = \sum_{i\in I}c_if_i$ , is called synthesis operator and its adjoint  $T^{*}\colon \mathcal{H}\to \ell^{2}(I)$ ,  $T^{*}f = \{\langle f,f_i\rangle \}_{i\in I}$ , which discretizes the function by extracting its frame coefficients, is called analysis operator. By composing  $T$  and  $T^{*}$ , we obtain the frame operator  $S\coloneqq TT^{*}$ , which is an invertible, self-adjoint and positive operator. Furthermore, the pseudo-inverse of the synthesis operator is given by  $T^{\dagger}\colon \mathcal{H}\rightarrow \ell^2 (I)$ ,  $T^{\dagger}f = \{\langle f,S^{-1}f_i\rangle \}_{i\in I}$ , which will be used in the following to reconstruct the function from its discrete representation, i.e. its frame coefficients.
With these concepts, one can introduce the most prominent result in frame theory [7], the frame decomposition theorem, which states that every element in  $\mathcal{H}$  can be uniquely and stably reconstructed from its frame coefficients by means of the reconstruction formula
$$
f = T T ^ {\dagger} f = \sum_ {i \in I} \langle f, S ^ {- 1} f _ {i} \rangle f _ {i} = \sum_ {i \in I} \langle f, f _ {i} \rangle S ^ {- 1} f _ {i}, \tag {2.4}
$$
where the series converge unconditionally. Formula (2.4) is clearly a generalization of reconstruction formula (2.1). However, it is worth pointing out that, in general, the coefficients in (2.4) are not necessarily point samples of the underlying function  $f$ , but more general frame coefficients.
In general, it may not be possible to access all the frame coefficients to reconstruct a function in a Hilbert space. Instead, just like in the case of reconstructing functions from point samples, one will need to consider approximations to this idealized situation. This is best encapsulated by the notion of a frame sequence, i.e., a countable sequence  $\{v_{i}\}_{i\in I}\in \mathcal{H}$  which is a frame for its closed linear span, i.e. for  $\overline{\mathrm{span}}\{v_i:i\in I\}$ . With this notion, we are in a position to generalize aliasing errors and the CDE, to arbitrary Hilbert spaces. Let  $\mathcal{H}$  be a separable Hilbert space and let  $\{v_{i}\}_{i\in I}$  be a frame sequence for  $\mathcal{H}$  with  $\mathcal{V} = \overline{\mathrm{span}}\{v_i:i\in I\}$  and frame operator  $S\colon \mathcal{V}\to \mathcal{V}$ . Then, the orthogonal projection of  $\mathcal{H}$  onto  $\mathcal{V}$  is given by  $\mathcal{P}_{\mathcal{V}}f = \sum_{i\in I}\langle f,v_i\rangle S^{-1}v_i$ . Thus, formula (2.4) holds and the function  $f$  can be uniquely and stably recovered from its frame coefficients if and only if  $f\in \mathcal{V}$ . If  $f\notin \mathcal{V}$ , reconstructing  $f$  from the corresponding frame coefficients results in an aliasing error:
Definition 2.2. Aliasing for functions in arbitrary separable Hilbert spaces. The aliasing error function  $\varepsilon(f)$  and the resulting aliasing error  $\| \varepsilon(f) \|$  of  $f \in \mathcal{H}$  for the frame sequence  $\{v_i\}_{i \in I} \subseteq \mathcal{V}$  are given by
$$
\varepsilon (f) = f - \mathcal {P} _ {\mathcal {V}} f, \| \varepsilon (f) \| = \| f - \mathcal {P} _ {\mathcal {V}} f \|.
$$
(a) An alias-free operator's diagram commutes.
(b)  $ReNO$ : an alias-free  $u$  can be constructed by discretizing the operator  $U$  for different discretizations given by  $\Psi, \Phi$ .
(c) ReNO: discrete representations  $u, u'$  for different discretizations are equivalent.
Figure 1: Alias-free framework.  $U$  is the underlying operator,  $u$  its discrete implementation. The synthesis operators and their pseudo-inverses make the link between function and discrete space.
If the aliasing error  $\varepsilon(f)$  is zero, i.e. if  $f \in \mathcal{V}$ , we say that there is a continuous-discrete equivalence (CDE) between  $f$  and its frame coefficients  $\{\langle f, v_i \rangle\}_{i \in I}$ .
3 Alias-Free Framework for Operator Learning
In this section, we will extend the concept of aliasing of functions to operators. We demonstrate that this notion of operator aliasing is fundamental to understanding how the neural operator performs across various discretizations, thereby addressing the shortcomings of possible lack of continuous-discrete equivalence. We will then define Representation equivalent Neural operators (ReNOs), whose discrete representations are equivalent across discretizations.
Assume we posit a (neural) operator  $U$ , mapping between infinite-dimensional function spaces. As mentioned before, this operator is never computed in practice, instead a discrete mapping  $u$  is used. The difficulty in formalizing this is that we should be able to compute this discrete mapping for any discretizations of the input and output function. We formalize this notion below in a practical manner.
Setting. Let  $U$ :  $\operatorname{Dom} U \subseteq \mathcal{H} \to \mathcal{K}$  be an operator between two separable Hilbert spaces, and let  $\Psi = \{\psi_i\}_{i \in I}$  and  $\Phi = \{\phi_k\}_{k \in K}$  be frame sequences for  $\mathcal{H}$  and  $\mathcal{K}$ , respectively, with synthesis operators  $T_{\Psi}$  and  $T_{\Phi}$ . We denote their closed linear spans by  $\mathcal{M}_{\Psi} := \overline{\operatorname{span}}\{\psi_i : i \in I\}$  and  $\mathcal{M}_{\Phi} := \overline{\operatorname{span}}\{\phi_k : k \in K\}$ . We note that by classical frame theory [7], the pseudo-inverses  $T_{\Psi}^{\dagger}$  and  $T_{\Phi}^{\dagger}$ , initially defined on  $\mathcal{M}_{\Psi}$  and  $\mathcal{M}_{\Phi}$ , respectively, can in fact be extended to the entire Hilbert spaces, i.e.  $T_{\Psi}^{\dagger} : \mathcal{H} \to \ell^{2}(I)$  and  $T_{\Phi}^{\dagger} : \mathcal{K} \to \ell^{2}(K)$ .
3.1 Operator Aliasing and Representation Equivalence
Once the discretization is chosen - determined by input and output frame sequences  $(\Psi, \Phi)$  - connecting the continuous operator  $U$  with its discrete counterpart  $u$  is the notion of operator aliasing. Given any mapping  $u \colon \ell^2(I) \to \ell^2(K)$ , we can build the operator  $T_{\Phi} \circ u \circ T_{\Psi}^{\dagger} \colon \mathcal{H} \to \mathcal{K}$ , whose definition clearly depends on the choices of the frame sequences that we make on the continuous level. In other words, any mapping  $u$  can be interpreted as a discrete representation of an underlying continuous operator, which in general, may differ from the operator  $U$ , that is of interest here. Hence, in analogy to Definitions 2.1 and 2.2, we can define the aliasing error of  $U$  relative to the discrete representation  $u$  as,
Definition 3.1. Operator aliasing. The aliasing error operator  $\varepsilon(U, u, \Psi, \Phi)$ :  $\operatorname{Dom} U \subseteq \mathcal{H} \to \mathcal{K}$  is given by
$$
\varepsilon (U, u, \Psi , \Phi) = U - T _ {\Phi} \circ u \circ T _ {\Psi} ^ {\dagger},
$$
and the corresponding scalar error is  $\| \varepsilon (U,u,\Psi ,\Phi)\|$ , with  $\| \cdot \|$  denoting the operator norm.
An aliasing error of zero implies that the operator  $U$  can be perfectly represented by first discretizing the function with  $T_{\Psi}^{\dagger}$ , applying  $u$ , and then reconstructing with  $T_{\Phi}$ , or equivalently, that the diagram in Figure 1a commutes, i.e. the black and the blue directed paths in the diagram lead to the same result. If the aliasing error is zero, we say that  $(U, u, \Psi, \Phi)$  satisfies a continuous-discrete equivalence
(CDE), implying that accessing the discrete representation  $u$  is exactly the same as accessing the underlying continuous operator  $U$ .
Example 3.2. Aliasing with the magnitude squared operator. Consider the operator  $U(f) = |f|^2$  as an operator from  $\mathcal{B}_{\Omega}$  into  $\mathcal{B}_{2\Omega}$ . The choice to discretize inputs and outputs on the same grid  $\left\{\frac{n}{2\Omega}\right\}_{n\in \mathbb{Z}}$  corresponds to choosing  $\Psi = \Phi = \{\mathrm{sinc}(2\Omega x - n)\}_{n\in \mathbb{Z}}$ , and to defining the discrete mapping  $u\colon \ell^2 (\mathbb{Z})\to \ell^2 (\mathbb{Z})$  by  $u(v) = T_{\Psi}^{\dagger}\circ U\circ T_{\Psi}(v) = v\odot \overline{v}$ , where  $\odot$  denotes the entrywise product. Then, for every  $f\in \mathcal{B}_{\Omega}$  such that  $U(f)\in \mathcal{B}_{2\Omega}\setminus \mathcal{B}_{\Omega}$ , we have
$$
\varepsilon (U, u, \Psi , \Phi) (f) = U (f) - T _ {\Phi} \circ T _ {\Phi} ^ {\dagger} (U (f)) = U (f) - \mathcal {P} _ {\mathcal {B} _ {\Omega}} (U (f)) \neq 0,
$$
and we encounter aliasing. This occurs because  $U$  introduces new frequencies that exceed the bandwidth  $\Omega$ . However, we can rectify this by sampling the output functions on a grid with twice the resolution of the input grid. This corresponds to choosing  $\Phi = \{\mathrm{sinc}(4\Omega x - n)\}_{n\in \mathbb{Z}}$  and to defining  $u = T_{\Phi}^{\dagger}\circ U\circ T_{\Psi}$ , which simply maps samples from grid points  $\left\{\frac{n}{2\Omega}\right\}_{n\in \mathbb{Z}}$  into squared samples from the double resolution grid  $\left\{\frac{n}{4\Omega}\right\}_{n\in \mathbb{Z}}$ . This effectively removes aliasing since the equality  $U = T_{\Phi}\circ u\circ T_{\Psi}^{\dagger}$  is satisfied. Furthermore, sampling the input and output functions with arbitrarily higher sampling rate, i.e. representing the functions with respect to the system  $\{\mathrm{sinc}(2\overline{\Omega} x - n)\}_{n\in \mathbb{Z}}$  with  $\overline{\Omega} > 2\Omega$ , yields no aliasing error since  $\{\mathrm{sinc}(2\overline{\Omega} x - n)\}_{n\in \mathbb{Z}}$  constitutes a frame for  $\mathcal{B}_{2\Omega} \supseteq \mathcal{B}_{\Omega}$ .
In practice, the discrete representation  $u$  of the operator  $U$  depends on the choice of the frame sequences. This means that – as mentioned at the beginning of the section – there is one map for every input/output frame sequence  $\Psi, \Phi$ . The consistency between operations  $u, u'$  with respect to different frame sequences can be evaluated using the following error.
Definition 3.3. Representation equivalence error. Suppose that  $u, u'$  are discrete maps with associated frame sequences  $\Psi, \Phi$  and  $\Psi', \Phi'$ , respectively. Then, the representation equivalence error is given by the function  $\tau(u, u') : \ell^2(I) \to \ell^2(K)$ , defined as:
$$
\tau (u, u ^ {\prime}) = u - T _ {\Phi} ^ {\dagger} \circ T _ {\Phi^ {\prime}} \circ u ^ {\prime} \circ T _ {\Psi^ {\prime}} ^ {\dagger} \circ T _ {\Psi}
$$
and the corresponding scalar error is  $\| \tau (u,u^{\prime})\|$
Intuitively, this amounts to computing each mapping on their given discretization, and comparing them by expressing  $u'$  in the frames associated to  $u$ . In the following section we leverage the notion of operator aliasing in the context of operator learning, as well as explore its practical implications with respect to representation equivalence at the discrete level.
3.2 Representation equivalent Neural Operators (ReNO)
Equipped with the above notion of continuous-discrete equivalence, we can now introduce the concept of Representation equivalent Neural Operator (ReNO). To this end, for any pair  $(\Psi, \Phi)$  of frame sequences for  $\mathcal{H}$  and  $\mathcal{K}$ , we consider a mapping at the discrete level  $u(\Psi, \Phi)$ :  $\mathrm{Ran} T_{\Psi}^{\dagger} \to \mathrm{Ran} T_{\Phi}^{\dagger}$ , which handles discrete representations of the functions. Notice how this map is indexed by the frame sequences: this is normal as when the discretization changes, the definition of the function should also change. In order to alleviate the notation, when this is clear from the context, we will refer to  $u(\Psi, \Phi)$  simply as  $u$ . See SM D.1 for an explanation of the condition  $u(\Psi, \Phi)$ :  $\mathrm{Ran} T_{\Psi}^{\dagger} \to \mathrm{Ran} T_{\Phi}^{\dagger}$ .
Definition 3.4. Representation equivalent Neural Operators (ReNO). We say that  $(U, u)$  is a ReNO if for every pair  $(\Psi, \Phi)$  of frame sequences that satisfy  $\mathrm{Dom} U \subseteq \mathcal{M}_{\Psi}$  and  $\mathrm{Ran} U \subseteq \mathcal{M}_{\Phi}$  there is no aliasing, i.e. the aliasing error operator is identical to zero:
$$
\varepsilon (U, u, \Psi , \Phi) = 0. \tag {3.1}
$$
We will write this property in short as  $\varepsilon (U,u) = 0$
In other words, the diagram in Figure 1a commutes for every considered pair  $(\Psi, \Phi)$ . In this case, the discrete representations  $u(\Psi, \Phi)$  are all equivalent, meaning that they uniquely determine the same underlying operator  $U$ , whenever a continuous-discrete equivalence property holds at the level of the function spaces. The domain and range conditions in Definition 3.4 simply imply that the frames can adequately represent input and output functions of  $U$ .
Remark 3.5. If the aliasing error  $\varepsilon(U, u, \Psi, \Phi)$  is zero (as required in Definition 3.4), then the assumption that  $u(\Psi, \Phi)$  maps  $\operatorname{Ran} T_{\Psi}^{\dagger} \subseteq \ell^{2}(I)$  into  $\operatorname{Ran} T_{\Phi}^{\dagger} \subseteq \ell^{2}(K)$  implies that
$$
u (\Psi , \Phi) = T _ {\Phi} ^ {\dagger} \circ U \circ T _ {\Psi}. \tag {3.2}
$$