SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We observe that this definition of  $u(\Psi, \Phi)$  is such that the diagram in Figure 1b commutes. In other words, once we fix the discrete representations  $\Psi, \Phi$  associated to the input and output functions, there exists a unique way to define a discretization  $u(\Psi, \Phi)$  that is consistent with the continuous operator  $U$  and this is given by (3.2). In practice, we may have access to different discrete representations of the input and output functions, which in the theory amounts to a change of reference systems in the function spaces. Note that to avoid any aliasing error, the discrete representation of  $U$  has to depend on the chosen frame sequences, i.e. inevitably,  $u$  must depend on  $\Psi$  and  $\Phi$ , and hence, must be discretization dependent. See A.2 for the proof of Remark 3.5.
In particular, Remark 3.5 directly implies a formula to go from one discrete representation to another, as
$$
u \left(\Psi^ {\prime}, \Phi^ {\prime}\right) = T _ {\Phi^ {\prime}} ^ {\dagger} \circ T _ {\Phi} \circ u \left(\Psi , \Phi\right) \circ T _ {\Psi} ^ {\dagger} \circ T _ {\Psi^ {\prime}}, \tag {3.3}
$$
whenever the pairs of frame sequences  $(\Psi, \Phi)$  and  $(\Psi', \Phi')$  satisfy the conditions in Definition 3.4. In other words, the diagram in 1c commutes.
Formula (3.3) immediately implies Proposition 3.6, which establishes a link between aliasing and representation equivalence. This highlights the contrast to discretization invariance, discussed at length in [15]: while this concept establishes an asymptotic consistency, representation equivalence includes the direct comparison between any two given discretizations and guarantees their equivalence.
Proposition 3.6. Equivalence of ReNO discrete representations. Let  $(U, u)$  be a ReNO. For any two frame sequence pairs  $(\Psi, \Phi)$  and  $(\Psi', \Phi')$  satisfying conditions in Definition 3.4, we have that
$$
\tau (u, u ^ {\prime}) = 0,
$$
where, by a slight abuse of notation,  $u'$  denotes  $u(\Psi', \Phi')$ .
Hence, under the assumption that the discrete map at each discretization is consistent with the underlying continuous operator, we have a unique way to express the operator at each discretization. Moreover, formula (3.3) closely resembles analogous formulas presented in [17, 15, 23] when evaluating single shot super resolution. However, in Section 5, we offer a nuanced perspective, indicating variability across different scenarios.
3.3 Layer-wise Instantiation
As shown in Remark 3.5, we can compute the outputs of the ReNO on the computer by first discretizing the continuous operator  $U$ . Typically, if  $U$  is a neural operator composed of multiple layers, as we will show in this section, it is possible to discretize each layer, while remaining consistent with the underlying operator.
Consider a neural operator  $U$  with  $L$  layers, each a mapping between separable Hilbert spaces:
$$
U = U _ {L} \circ U _ {L - 1} \circ \dots \circ U _ {1}, \quad U _ {\ell}: \mathcal {H} _ {\ell} \rightarrow \mathcal {H} _ {\ell + 1}, \quad \ell = 1, \dots , L, \tag {3.4}
$$
we denote by  $\Psi_{\ell}$  a frame sequence for  $\mathcal{H}_{\ell}$ . The choice of frame sequences for each  $\mathcal{H}_{\ell}$  corresponds to the choice of accessible discrete representations of functions in the underlying function space  $\mathcal{H}_{\ell}$ .
Proposition 3.7. Stability of ReNO under composition. Consider the composition  $U = U_{L} \circ \ldots \circ U_{1}$  as in eq. 3.4, as well as a discrete mapping  $u = u_{L} \circ \ldots \circ u_{1}$ . If each layer  $(U_{\ell}, u_{\ell})$  is a ReNO, the composition  $(U, u)$  also is a ReNO.
As the proof of Proposition 3.7, presented in SM A.1, also shows, if each hidden layer in the operator (3.4) has an aliasing error (2.2), then these errors may propagate through the network and increase with increasing number of layers.
3.4  $\epsilon$ -ReNOs
In practice, we may not need to set the aliasing error to zero. But it is essential to be able to control it and make it as small as desired. The notion of ReNOs, Definition 3.4, can very simply be extended to the case where we allow for a small, controlled amount of aliasing. Indeed we can introduce  $\epsilon$ -ReNOs, which satisfy  $\| \varepsilon (U,u)\| \leq \epsilon$  (for every pair of admissible frame sequences) instead of  $\varepsilon (U,u) = 0$ .
Proposition 3.8. Let  $(U, u)$  be an  $\epsilon$ -ReNO. For any two frame sequence pairs  $(\Psi, \Phi)$  and  $(\Psi', \Phi')$  satisfying conditions in Definition 3.4 and such that  $\mathcal{M}_{\Phi'} \subseteq \mathcal{M}_{\Phi}$ , we have
$$
\| \tau (u, u ^ {\prime}) \| \leq \frac {2 \epsilon \sqrt {B _ {\Psi}}}{\sqrt {A _ {\Phi}}},
$$
Equipped with our definition of Representation equivalent Neural Operators (ReNOs), we analyze some existing operator learning architectures to ascertain whether they are neural operators or not.
Convolutional Neural Networks (CNN). Classical convolutional neural networks are based on the convolutional layer  $k$ , involving a discrete kernel  $f \in \mathbb{R}^{2s + 1}$  and a discrete input  $c$ :
$$
k (c) [ m ] = (f * c) [ m ] = \sum_ {i = - s} ^ {s} c [ m - i ] f [ i ].
$$
We can then analyze this layer using our framework to ask whether this operation can be associated to some underlying continuous operator. Intuitively, if this is the case, the computations conducted on different discretizations effectively representing the input should be consistent; in the contrary case, no associated continuous operator exists and the convolutional operation is not a ReNO.
Consider the case where the discrete input  $c$  corresponds to pointwise evaluation on a grid of some underlying bandlimited function  $f \in \mathcal{B}_{\Omega}$ , for example  $c[n] = f\left(\frac{n}{2\Omega}\right)$ ,  $n \in \mathbb{Z}$  with associated orthonormal basis  $\Psi = \{\mathrm{sinc}(2\Omega x - n)\}_{n \in \mathbb{Z}}$ . Consider now an alternate representation of  $f$ , point samples of a grid twice as fine:  $d[n] = f\left(\frac{n}{4\Omega}\right)$ , with  $\Psi' = \{\mathrm{sinc}(4\Omega x - n)\}_{n \in \mathbb{Z}}$  as basis. Clearly, even though discrete inputs agree, i.e.  $c[n] = d[2n]$ , this is no longer true for the outputs,  $(k*c)[n] \neq (k*d)[2n]$ . This in turn implies that there exist frame sequences  $\Psi, \Psi'$  such that:
$$
T _ {\Psi} \circ k _ {\Psi} (c) \circ T _ {\Psi} ^ {\dagger} \neq T _ {\Psi^ {\prime}} \circ k _ {\Psi^ {\prime}} (d) \circ T _ {\Psi^ {\prime}} ^ {\dagger} \tag {4.1}
$$
thereby defying the representation equivalence property of ReNOs, in the sense of Definition 3.4. This fact is also corroborated with the experimental analysis, Section 5.
Fourier Neural Operators (FNO). FNOs are defined in [17] in terms of layers, which are either lifting or projection layers or Fourier layers. As lifting and projection layers do not change the underlying spatial structure of the input function, but only act on the channel width, these linear layers will satisfy Definition 3.4 of ReNO here. Hence, we focus on the Fourier layer of the form,
$$
v _ {\ell + 1} (x) = \sigma \left(A _ {\ell} v _ {\ell} (x) + B _ {\ell} (x) + \mathcal {K} v _ {\ell} (x)\right), \tag {4.2}
$$
with the Fourier operator given by
$$
\mathcal {K} v = \mathcal {F} ^ {- 1} (R \odot \mathcal {F}) (v).
$$
Here,  $\mathcal{F},\mathcal{F}^{-1}$  are the Fourier and Inverse Fourier transforms.
For simplicity of exposition, we set  $A_{\ell} = B_{\ell} \equiv 0$  and focus on investigating whether the Fourier layer (4.2) satisfies the requirements of a ReNO. Following [17], the discrete form of the Fourier layer is given by  $\sigma(kv)$ , with  $kv = F^{-1}(R \odot F(v))$ , where  $F, F^{-1}$  denote the discrete Fourier transform (DFT) and its inverse.
In SM B.1, we show that the convolution in Fourier space operation for the FNO layer (4.2) satisfies the requirements of a ReNO. However the pointwise activation function  $\sigma(f)$ , applied to a bandlimited input  $f \in \mathcal{B}_{\Omega}$  will not necessarily respect the bandlimits, i.e.,  $\sigma(f) \notin \mathcal{B}_{\Omega}$ . In fact, with popular choices of activation functions such as ReLU,  $\sigma(f) \notin \mathcal{B}_{\omega}$ , for any  $\omega > 0$  (see SM C for numerical illustrations). Thus, the Fourier layer operator (4.2) may not respect the continuous-discrete equivalence and can lead to aliasing errors, a fact already identified in [10]. Hence, FNO may not be a ReNO in the sense of Definition 3.4.
Convolutional Neural Operators (CNO). Introduced in [23], the layers of a convolutional neural operator consist of three elementary operations
$$
v _ {l + 1} = \mathcal {P} _ {l} \circ \Sigma_ {l} \circ \mathcal {K} _ {l} (v _ {l}), \quad 0 \leq l \leq L - 1, \tag {4.3}
$$
where  $\mathcal{K}_l$  is a convolution operator,  $\Sigma_l$  is a non-linear operator whose definition depends on the choice of an activation function  $\sigma: \mathbb{R} \to \mathbb{R}$ , and  $\mathcal{P}_l$  is a projection operator. We show in SM B.2 that CNO layers respect equation (3.2) and consequently CNOs are Representation equivalent Neural Operators (ReLUOs) in the sense of Definition 3.4. This is the result of the fact that the activation layer is defined in a way to respect the band-limits of the underlying function space.
5.1 Assessing Representation Equivalence
In the previous section, we have studied existing neural operator architectures from a theoretical perspective. In this section, for the same architectures, we corroborate these findings from an empirical viewpoint. Aliasing is a quantity that cannot be computed in practice, as we cannot access the underlying operator  $U$  on a computer, we can nonetheless compute representation equivalent errors introduced in Definition 3.3, which is related to aliasing by Propositions 3.7 and 3.8 (as well as being a quantity of interest in itself).
Experimental Setting. We wish to learn an unknown target operator  $Q$  using a neural operator. In this experiment, all neural operators (CNN, FNO and SNO) take as input pointwise evaluations on the grid, and are able to deal with varying input resolutions. A simple way of constructing the target operator  $Q: H \to H$ , where  $H$  is the space of periodic and  $K = 30$ -bandlimited functions, is by sampling input and output pairs in a random fashion. Sampling of a function in  $H$  can be realized as follows: As we know that  $\Psi_K := \{d_K(\cdot - x_k)\}_{k = -K, \dots, K}$  constitutes a frame for  $H$ ,  $d_K$  being the Dirichlet kernel of order  $K$  and  $x_k = \frac{k}{2K + 1}$ , any function  $f \in H$  can be written as  $f(x) = \sum_{k = -K}^{K} f(x_k) d_K(x - x_k)$ . Thus, the discrete representation of  $f$  simply corresponds to its  $2K + 1 = 61$  point-wise evaluations on a grid, i.e.  $\{f(x_k)\}_{k = -K, \dots, K}$ . Note that for simplicity we have used and will use the same frame sequences for both input and output spaces, as these are the same.
Training and Evaluation. Once the data is generated, we train neural operators  $u(\Psi_K, \Psi_K)$  on discretizations associated to the frame sequences, which are simply the point-wise evaluations of the input-target functions in the data. In other words, we regress to the frame coefficients of the target function, with coefficients of the input function as input. Once training is over, we evaluate how the different neural operators behave when dealing with changing input and output frame sequences. The frame sequences here are  $\Psi_M$  for different testing frames, with associated  $2M + 1$  sized grids, with associated operator  $u_M: \mathbb{R}^{2M + 1} \to \mathbb{R}^{2M + 1}$ .
The results of this experiment are presented in Figure 2; we also provide results for additional architectures in Figure 4. The results in Figure 2 clearly show that as predicted by our theory, neither CNN nor FNO are representation equivalent, in the sense of Definition 3.4 and changing the resolution, which amounts to a change of frame, does not keep the operator invariant, causing aliasing errors that materialize themselves as representation equivalence errors (as in Definition 3.3) here. On the other hand, as predicted by the theory, CNO is more likely to be a ReNO – as long as the frames selected satisfy conditions of Definition 3.4 (i.e. “Representation Equivalence” zone). When these conditions no longer hold (“No Equivalence” zone), CNO also generates aliasing errors. Thus, this experiment clearly demonstrates the practical implications of the ReNO framework. In contrast, the
Figure 2: Representation equivalence is computed for three classical architectures, CNN, FNO and CNO trained on a resolution of 61 (yellow dot). The "Resolution Equivalence" zone, located on the right-hand side, denotes the region where discrete representations have an associated frame, while the left-hand side represents the area where this is no longer the case (loss of input/output information). As predicted by the theory, CNN and FNO are not representation equivalent, while CNO is error-free in the equivalence region, but failing to do so outside of it.
discretization invariance [15] condition only ensures that in the infinite resolution limit, the neural operator converges, while not making any predictions about the behavior of the neural operator at finite resolutions.
Minimizing representation equivalence error. Instead of directly tackling aliasing errors from a theoretical perspective as is done in the previous sections, we select an architecture that may have aliasing, i.e. FNO, and we try to minimize the representation equivalence error during training. We observe and detail our findings in SM C.3.
5.2 Assessing Structure Preservation
Representation Equivalence Error (%)	10.38	20.24	42.97	98.55
Translation Equivariance Error (%)	10.54	20.54	45.79	104.44
Table 1: Representation equivalence error vs translation equivariance error. We observe a direct link between aliasing and preserving continuous structures.
Our methodology aims to maintain the structure of the underlying operator, as defined at the continuous level. Fundamental structural characteristics of the underlying operator, like symmetries and conservation principles, are maintained at the discrete level. Discretizations that are not aligned may fail to uphold these intrinsic properties of the operator, resulting in deviations like symmetry breaking which can negatively influence the results.
As an example of this, we consider FNO. Similarly to what is done in [11], the primary structure we aim to uphold is translation equivariance, as defined at the operator level. We look at different trained FNOs – all of which have a different representation equivalence error – and assess whether they are translation equivariant or not. More specifically, these FNOs are trained by minimizing both the regression loss and the representation equivalence error simultaneously, introducing a more or less large multiplier  $\lambda$  on the latter, just like in SM C.3.
As we vary the parameter  $\lambda$ , we observe in Table 1, there is a distinct relationship between the Discrete Aliasing Error (DAE) and the Translation Equivariance Error. Both errors show a similar trend with a nearly perfect positive linear association.
Summary. Although a variety of architectures for operator learning are available since the last couple of years, there is still a lack of clarity about what exactly constitutes operator learning. Everyone would agree that an operator learning architecture should have functions as inputs and outputs. However, in practice, one does not necessarily have access to functions, either at the input or output level. Rather the access is limited to some form of discrete representations of the underlying functions, for instance, point values, cell averages, coefficients with respect to a basis etc. Moreover, one can only perform computations with discrete objects on digital computers. Hence, given the necessity of having to work with discrete representations of functions, it is essential to enforce some relationship between the continuous and discrete representations of the underlying functions or in other words, demand consistency in function space.
Unlike in [15], where the authors advanced a form of asymptotic consistency in terms of discretization parameters, we go further to require a stronger form of structure-preserving equivalence of the continuous and discrete representations of operators. To this end, we leverage the equivalence between continuous and discrete representations of functions in Hilbert spaces in terms of frame theory. The main point about this equivalence is the fact that functions, belonging to suitable function spaces, can be uniquely and stably reconstructed, from their frame coefficients. A failure to enforce this equivalence results in the so-called aliasing errors that quantitatively measure function space inconsistencies.
We extend this notion of aliasing error to operators here and use it to define Representation equivalent Neural Operators (ReNOs), see Definition 3.4. Our framework automatically implies consistency in function spaces and provides a recipe for deriving ReNOs in terms of changing the underlying frames. We also employ our framework to analyze whether or not existing operator learning architectures are ReNOs and also corroborate our results through experiments.
Related Work. Our current paper relies heavily on structure preservation from classical numerical analysis, [22] and references therein, as well as concepts from signal processing and applied harmonic analysis [25] and references therein. Among the emerging literature on operator learning, [15] was one of the first papers to attempt a unifying framework that encompasses many operator learning architectures and codify them through a particular definition of neural operators. We take an analogous route here but with our main point of departure from [15] being that unlike their notion of asymptotic consistency, we require systematic consistency in function space by enforcing the representation equivalence for the underlying operator at each layer. Another relevant work is [10] where the authors flag the issue of possible aliasing errors with specific operator learning architectures. We significantly expand on the approach of [10] by providing a rigorous and very general definition for aliasing errors. Finally, our definition of Representation equivalent Neural Operators, relying on aliasing errors for operators, is analogous to a similar approach for computing with functions, rather than discrete representations, which is the cornerstone of the Chebfun project [8] and references therein.
Limitations and Extensions. Our aim in this paper was to tackle a fundamental question of what defines a neural operator? We have addressed this question here and shown that enforcing some form of equivalence between continuous and discrete representations is needed for the architecture to genuinely learn the underlying operator rather than just a discrete representation of it. What we have not addressed here are quantitative measures of the error associated with a Representation equivalent Neural Operator, introduced in Definition 3.4, in approximating the underlying operator. This is a much broader question than what is addressed here, since sources of error, other than aliasing errors, such as approximation, training and generalization errors also contribute to the total error (see [16, 14, 15]).
One interesting direction for further analysis would involve exploring operators and their discretized counterparts that form  $\epsilon$ -ReNOs and also enforce the ReNO property approximately. Characterizing the implications of this concept within operator learning is a topic for future work.
[1] G. S. Alberti, M. Santacesaria, and S. Scutto. Continuous generative neural networks. arXiv preprint arXiv:2205.14627, 2022.
[2] J. J. Benedetto. Irregular sampling and frames. Wavelets: A Tutorial in Theory and Applications, 2:445-507, 1992.
[3] M. Bertero, P. Bocacci, and C. De Mol. Introduction to inverse problems in imaging. CRC press, 2021.
[4] K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021.
[5] S. Cao. Choose a transformer: Fourier or galerkin. In 35th conference on neural information processing systems, 2021.
[6] T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917, 1995.
[7] O. Christensen. Frames and bases: An introductory course. Springer Science & Business Media, 2008.
[8] T. A. Driscoll, N. Hale, and L. N. Trefethen. Chebfun Guide. Pafnuty Publishers, Oxford, 2014.
[9] L. C. Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
[10] V. Fanaskov and I. Oseledets. Spectral neural operators. arXiv preprint arXiv:2205.10573, 2022.
[11] N. Gruver, M. A. Finzi, M. Goldblum, and A. G. Wilson. The lie derivative for measuring learned equivariance. In *The Eleventh International Conference on Learning Representations*, 2023.
[12] V. Isakov. Inverse Problems for Partial Differential Equations. Springer, 2017.
[13] G. Kissas, J. H. Seidman, L. F. Guilhoto, V. M. Preciado, G. J. Pappas, and P. Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1-63, 2022.
[14] N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for Fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021.
[15] N. B. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces. CoRR, abs/2108.08481, 2021.
[16] S. Lanthaler, S. Mishra, and G. E. Karniadakis. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):tnac001, 2022.
[17] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
[18] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural operator: Graph kernel network for partial differential equations. CoRR, abs/2003.03485, 2020.
[19] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, A. M. Stuart, K. Bhattacharya, and A. Anandkumar. Multipole graph neural operator for parametric partial differential equations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6755-6766. Curran Associates, Inc., 2020.
[20] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-229, 2021.
[21] M. Prasthofer, T. De Ryck, and S. Mishra. Variable input deep operator networks. arXiv preprint arXiv:2205.11404, 2022.
[22] A. Quarteroni and A. Valli. Numerical approximation of Partial differential equations, volume 23. Springer, 1994.
[23] B. Raonic, R. Molinaro, T. De Ryck, T. Rohner, F. Bartolucci, R. Alaifari, S. Mishra, and E. de Bezenac. Convolutional neural operators for robust and accurate learning of PDEs. To appear in 37th conference on neural information processing systems, https://arxiv.org/abs/2302.01178, 2023.
[24] J. H. Seidman, G. Kissas, P. Perdikaris, and G. J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. arXiv preprint arXiv:2206.03551, 2022.
[25] M. Unser. Sampling-50 years after Shannon. Proceedings of the IEEE, 88(4):569-587, 2000.
[26] M. Vetterli, J. Kovacevic, and V. Goyal. Foundations of Signal Processing. Cambridge University Press, 2014.