NOMAD: Nonlinear Manifold Decoders for Operator Learning
Authors: Jacob H Seidman
Georgios Kissas
Paris Perdikaris
George J. Pappas
Venue: NeurIPS 2022 Accept (2022)
Keywords: Operator Learning
Manifold Learning
Functional Data
PDEs
Nonlinear Dimension Reduction

Abstract: Supervised learning in function spaces is an emerging area of machine learning research with applications to the prediction of complex physical systems such as fluid flows, solid mechanics, and climate modeling.  By directly learning maps (operators) between infinite dimensional function spaces, these models are able to learn discretization invariant representations of target functions.  A common approach is to represent such target functions as linear combinations of basis elements learned from data. However, there are simple scenarios where, even though the target functions form a low dimensional submanifold, a very large number of basis elements is needed for an accurate linear representation. Here we present NOMAD, a novel operator learning framework with a nonlinear decoder map capable of learning finite dimensional representations of nonlinear submanifolds in function spaces.  We show this method is able to accurately learn low dimensional representations of solution manifolds to partial differential equations while outperforming linear models of larger size.  Additionally, we compare to state-of-the-art operator learning methods on a complex fluid dynamics benchmark and achieve competitive performance with a significantly smaller model size and training cost.

NOMAD: Nonlinear Manifold Decoders for Operator Learning
Jacob H. Seidman'
Georgios Kissas
Graduate Program in Applied Mathematics
Department of Mechanical Engineering and Applied Mechanics University of Pennsylvania
and Computational Science University of Pennsylvania
seidj0sas upenn. edu
gkissas$seas, upenn edu
Paris Perdikaris
George J. Pappas
Department of Mechanical Engineering and Applied Mechanics University of Pennsylvania pgp0seas. upenn. edu
Department of Electrical and Systems Engineering University of Pennsylvania pappasgóseas upenn. edu
Abstract
Supervised learning in function spaces is an emerging area of machine learning research with applications to the prediction of complex physical systems such as fluid flows, solid mechanics, and climate modeling. By directly learning maps (operators) between infinite dimensional function spaces, these models are able learn discretization invariant representations target functions. common ap- proach is to represent such target functions linear combinations of basis elements learned from data. However, there are simple scenarios where, even though the target functions form low dimensional submanifold, very large number of basis elements is needed for an accurate linear representation. Here we present NOMAD, novel operator learning framework with nonlinear decoder map capable of learning finite dimensional representations of nonlinear submanifolds in function spaces. We show this method is able accurately learn low dimensional represen- tations of solution manifolds to partial differential equations while outperforming linear models of larger size. Additionally, learning methods on a complex fluid dynamics benchmark and achieve competitive performance with significantly smaller model size and training cost.
compare state-ofthheaa operator
Introduction
imput functions
output fincttoos
Machine learning techniques have been ap- plied to great success for modeling func- tions between finite dimensional vector spaces. For example, in computer vision (vectors of pixel values) and natural lan- guage processing (vectors of word embed- dings) these methods have produced state- of-the-art results in image recognition [18] and translation tasks all data has an obvious and faithful repre- sentation as finite dimensional vectors.
L'(X))
L2()))
encoder
decoder
However, not
approximator
Figure 1: The Operator Learning Manifold Hypothesis.
'These authors contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

particular, functional data is mathematically represented as a vector in an infinite dimensional vector space. This kind of data appears naturally in problems coming from physics, where scenarios fluid dynamics, solid mechanics, and kinematics are described by functions of continuous quantities.
Supervised learning in the infinite dimensional setting can be considered for cases where want to map fiunctional inputs to target functional outputs. For example, we might wish to predict the velocity of fluid as function of time given an initial velocity field, or predict the pressure field across the surface of the Earth given temperature measurements. This is similar to a finite dimensional regression problem, except that we are now interested in learning an operator between spaces of functions. We refer to this as a supervised operator learning problem: given a data-set of N pairs of functions {u' ss) s')..... functions such that F(u
....... (u,, }))}. learn an operator F which maps input functions to output s", vi.
One approach to solve the supervised operator learning problem is introduce a parameterized operator architecture and train it to minimize a loss between the model's predicted functions and the true target functions in the training set. One of the first operator network architectures was presented in [8] with accompanying universal approximation guarantees in the uniform norm. These results were adapted to deep networks in (30) and led the DeepONet architecture and its variants [51 [2 21 The Neural Operator architecture, motivated by the composition of linear and nonlinear layers in neural networks, was proposed in [27]] Using the Fourier convolution theorem to compute the integral transform in Neural Operators led to the Fourier Neural Operator (28). Other recent architectures include approaches based on PCA-based representations
2 random feature approaches
(36). wavelet approximations to integral transforms [15]. and attention-based architectures (23
A common feature shared among many of these approaches is that they aim approximate an operator using three maps: an encoder, an approximator, and a decoder, see Figure [and and Section 3 for more details. In all existing approaches embracing this structure, the decoder is constructed as linear map. In doing so, the set of target functions is being approximated with a finite dimensional linear subspace in the ambient target function space. Under this setting, the universal approximation theorems of (8. 24 25 guarantee that there exists a linear subspace of a large enough dimension which approximates the target functions to any prescribed accuracy.
However, as with finite dimensional data, there are scenarios where the target functional data concentrates low dimensional nonlinear submanifold. We refer the phenomenon of data in function spaces concentrating on low dimensional submanifolds as the Operator Learning Manifold Hypothesis, see Figure I For example, it is known that certain classes of parametric partial differential equations admit low dimensional nonlinear manifolds of solution functions [9]. Although linear representations can be guaranteed approximate these spaces, their required dimension can become very large and thus inefficient in capturing the true low dimensional structure of the data.
In this paper, we are motivated by the Operator Learning Manifold Hypothesis to formulate new class of operator learning architectures with nonlinear decoders. Our key contributions can be summarized as follows.
Limitations of Linear Decoders: We describe in detail the shortcomings of operator learning methods with linear decoders and present some fundamental lower bounds along with an illustrative operator learning problem which is subject to these limitations.
Nonlinear Manifold Decoders (NOMAD): This motivates a novel operator learning frame- work with nonlinear decoder that can find low dimensional representations for finite dimensional nonlinear submanifolds in function spaces.
Enhanced Dimensionality Reduction: A collection of numerical experiments involving linear transport and nonlinear wave propagation shows that, by learning nonlinear submani- folds of target functions, we can build models that achieve state-of-the-art accuracy while requiring significantly smaller number of latent dimensions.
Enhanced Computational Efficiency: As a consequence, the resulting architectures con- tain a significantly smaller number of trainable parameters and their training cost is greatly reduced compared to competing linear approaches.
We begin our presentation in Section 2 by providing taxonomy of representative works in the 3 we formally define the supervised operator learning problem and discuss existing approximation strategies, with focus on highlighting open challenges and limitations. In
literature. In Section
2

---

Section 4 we present the main contributions of this work and illustrate their utility through the lens of a pedagogical example. In Section | we provide a comprehensive collection of experiments that demonstrate the performance of using NOMAD against competing state-of-ther
methods for our main findings and discusses lingering limitations and broader impact. Additional details on architectures, hyperparameter selection, and training details are provided in the Supplemental Materials. The code and data accompanying this manuscript are hww.thhwwwo.rrrreddiiiiiiiieeeee
operator learning. SeetioonSssmmante
available at
2 Related Work in Dimensionality Reduction
Low Dimensional Representations in Finite Dimensional Vector Spaces: Finding low dimen- sional representations of high dimensional data has a long history, going back to 1901 with the original formulation of principal components analysis (PCA) [39] PCA is linear method that works best when data concentrates on low dimensional subspaces. When data instead concentrates on low dimensional nonlinear spaces, kernelized PCA [42 and manifold learning techniques such as Isomap and diffusion maps [46 [1 35] can be effective in finding nonlinear low dimensional structure, see [47 for review. The recent popularity of deep learning has introduced new methods for finding low dimensional structure in high dimensional data-sets, most notably using auto-encoders [52
and deep generative models [12 22 Relevant to work, such techniques have found success in
approximating submanifolds in vector spaces corresponding to discretized solutions of parametric partial differential equations (PDEs) 43 [4], where a particular need for nonlinear dimension reduction arises in advection-dominated problems common to fluid mechanics and climate science
26 33
Low Dimensional Representations in Infinite Dimensional Vector Spaces: The principles be- hind PCA generalize in straightforward way to functions residing in low dimensional subspaces of infinite dimensional Hilbert spaces [50]. In the field of reduced order modeling of PDEs this is sometimes referred to proper orthogonal decomposition [ (see (29] for interesting exposition of the discrete version and connections to the Karhunen-LLoève decomposition). Affine representations of solution manifolds parametric PDEs and guarantees when they effective using the notion of linear n-widths 40 have been explored in [9]- As in the case of finite dimensional data, using kernel create feature representation of set of functions, and then performing PCA in the associated Reproducing Kernel Hilbert Space can give nonlinear low dimensional representations 45. The theory behind optimal nonlinear low dimensional representations for of functions is still being developed, but there has been work towards defining what "optimal" should mean in this context and how it relates more familiar geometric quantities [10]
3 Operator Learning
Notation:
Let us first set up some notation and give a formal statement of the supervised operator learning problem. We define C(X; Rd) as the set of continuous functions from set X' to Rd. When X R"," we define the Hilbert space,
L L'(X.R') = f:
I = / [I dr co)
This is an infinite dimensional vector space equipped with the inner product (f.g) = fx f(x)g(x)dz. When X' is compact, have that C(X; Rd) C (2((BB)) We now can present formal statement of the supervised operator learning problem.
Problem Formulation: Suppose we are given training data-set of N pairs of functions (u², st), where w/ E C(X;R))) with compact X' C RRd,, and st E C()) Rd) with compact ) C Rds Assume there is a ground truth operator G C(X'R, + C(); RRd,) such that G(u') = and that the u² are sampled i.i.d. from probability measure on C(X;; Rd)). The goal of the supervised operator learning problem to learn a continuous operator F Rd): + C(); RRd,) to approximate G. To do so, will attempt to minimize the following empirical risk over a class of
3

---

operators Fo. with parameters 0 E θ C Rde,
N
1 (()) = N
[F((')) s8 5'112
An Approximation Framework for Operators: popular approach to learning an operator G: (X) + (3)) acting on a probability measure μ (2)) is to construct an approximation out of three maps [25] (see Figure
GxF:=DoAOE:
The first map, & (x) Rm is known as the encoder. It takes input function and maps a finite dimensional feature representation. For example, could take continuous function to its point-wise evaluations along a collection of m sensors, project function onto basis functions. The next map A R + IR is known as the approximation map. This can be interpreted a finite dimensional approximation of the action of the operator G. Finally, the image of the approximation map is used to create the output functions in ())) by means of the decoding map D R + ()).. We will refer to the dimension, of the domain of the decoder the latent dimension. The composition of these maps can be visualized in the following diagram.
(X)
(2
(3)
D
R
R
Many successful operator learning architectures such as the DeepONet (30). the (pseudo-spectral) Fourier Neural Operator in (24) LOCA (23,, and the PCA-based method in 2 all use linear decoding maps D. A linear 'D can be defined by set of functions E (,2(O)). ,...., n, and acts on a vector R as
Linear Decoders:
D(3) 81m +...+++4 T.
(4)
For example, the functions can be built using trigonometric polynomials as in the --FNO 224]. be parameterized by neural network in DeepONet [3), created the normalized output of kernel integral transform in LOCA 23
Limitations of Linear Decoders: We can measure the approximation accuracy the operator F with two different norms. First is the (p) operator norm,
|F - Gl = E [IF(w) [(w))
(5)
Note that the empirical risk used train model for the supervised operator learning problem (see (T)) is Monte Carlo approximation of the above population loss. The other option to measure the approximation accuracy is the uniform operator norm. For compact set C L2((X') define the error
sup F(u) - ((u))lez well
(6)
When linear decoder is used for F D o data-dependent lower bound to each of these eπrоrs can be derived.
1 lower bound: When the pushforward measure has finite second moment, its covariance operator r L2²())) + ()²( is self-adjoint, positive semi-definite, and trace-class, and thus admits an orthogonal set of eigenfunctions spanning its image, (di, d2....) with associated decreasing eigenvalues A > A2 2 The decay of these eigenvalues indicates the extent to which samples from GHf concentrate along the leading finite-dimensional eigenspaces. was shown in (25) that for any choice of & and A, these eigenvalues give fundamental lower bound to the expected squared I error of the operator learning problem with architectures as in (3) using a linear decoder D.
E [[|D o Ao ((u) ((w)l w~
Σ A

---

This result can be refined to show that the optimal choice of functions T; (see equation (4) for a linear decoder are the leading eigenfunctions of the covariance operator { .... The interpretation of this result is that the best way approximate samples from ## with an n-dimensional subspace to use the subspace spanned by the first n "principal components" of the probability measure 9 Jr. The error incurred by using this subspace is determined by the remaining principal components, namely the sum of their eigenvalues E>>,, Ag. The operator learning literature has noted that for problems with slowly decaying pushforward covariance spectrum (such solutions to advection- dominated PDEs) these lower bounds cause poor performance for models of the form
25 13
Uniform lower bound: In the reduced order modelling of PDEs literature 9 10 26 there exists a related notion for measuring the degree to an n-dimensional subspace can approximate set of functions & C L)))). This is known as the Kolmogorov π-- n-wiith [40], and for compact set S is defined as
(n)
inf ()) #ES vEV. V., subspace dim(V.)=n
sup inf ||s 1: 11112200.
This measure of how well set of functions can be approximated by linear subspace in the uniform norm leads naturally to lower bound for the uniform error (6). To see this, first note that for any E U4 the error from F(u) to (u)) is bounded by the minimum distance from (u)) to the image of F. For linear decoder D IRn > T,2()), define the (at most) n-dimensional V, im(D) (,))). Note that im(F) S Vn, and may write
[IF(u) (()ll 2 inf || (()))))))))
VVV.
Taking the supremum of both sides over U, and then the infimum of both sides over all n-dimensional subspaces Vn gives
sup F (el)e)) 2 ucll
sup inf ||I - G(u)) CLC))( well subspace dim (lm(V,,)= V=
inf
vEV,
The quantity on the right is exactly the Kolmogorov n-width of G()). We have thus proved the following complementary statement to (7) when the error is measured in the uniform norm.
Proposition! 1 Let (x) be compact and consider an operator learning architecture as in 3 where D R - (D' is linear decoder. Then, for any E -I2(X) - RR" and A: R. R", the
uniform norm error of F == D a A E satisfies the lower bound
sup [1 -(())ll>(O)) Z (9(U)) aEll
Therefore, that in both the (p) and uniform norm, the error for operator learning problem with a linear decoder is fundamentally limited by the extent to which the space of output functions "fits" inside finite dimensional linear subspace. the next section we will alleviate this fundamental restriction by allowing decoders that can learn nonlinear embeddings of IR" into L2))..
4 Nonlinear Decoders for Operator Learning
Motivating Example: Consider the problem of learning the antiderivative operator mapping functions to their first-order derivative
g:u = (-
u(y) dy.
(10)
acting on a set of input functions
U = (u(x) 2 cos(2.tt.) 10
(1I)
The set of output functions is given by G(4) = (2 0 < 4 < T). This is one- dimensional curve of functions ([2 1)) parameterized by a single number However, we would not be able to represent this set of functions with one-dimensional linear subspace. In Figure 2b
5

---

0.0
10F
Lineur doedder
Relative Ceeer (lokal
NOMAD
0,5
Diganalue
(--1
0.2
PC3
0.0
-10+
10.4
012
10
10
10
-0.2
0.0
0.2
-22
0.0
PC2
-20
10
20
30
40
50 00
70 80 50 100
Dinousson index
PC1
Latent dimension
(a)
(b)
(c)
Figure Antiderivative Example: (a) log of the leading 100 PCA eigenvalues of G(4); (b) Projection of functions in the image of G(L) on the first three PCA components, colored by the frequency of each projected function; (c) Relative L testing error (log10 scale) function of latent dimension for linear and nonlinear decoders (over 10 independent trials).
we perform PCA on the functions in this set evaluated on a uniform grid of values of I. We see nonzero and relatively constant, suggesting that an operator learning
that the first 20 eigenvalues architecture with a linear or affine decoder would need a latent dimension of at least 20 to effectively approximate functions from G(U).. Figure 2a gives a visualization of this curve of functions projected onto the first three PCA components. We will return to this example in Section 5 and see that an architecture with a nonlinear decoder can in fact approximate the target output functions with superior accuracy compared to the linear case, using a single latent dimension that can capture the underlying nonlinear manifold structure.
Operator Learning Manifold Hypothesis: We now describe an assumption under which a non- linear decoder is expected be effective, and use this formulate the NOMAD architecture. To this end, let be a probability measure on (2)) and g (2) + (2)))) We assume that there exists an n-dimensional manifold M C T,2())) and open subset O C M such that
E
inf PEO
[[S(u) u112a
S
(12)
mewp
In connection with the manifold hypothesis in
An() + (m(())
D
deep learning 5 4), we refer this the Op-
erator Learning Manifold Hypothesis. There are scenarios where is known this assumption holds, such as in learning solutions to parametric PDEs (3
This assumption motivates the construction of nonlinear decoder for the architecture in (3) as follows. For each u, choose v(u)= E O such that
submaniffodd
Nonlitear subumailold
E [I] v(u) [2=] S e. ana
(13)
D
Let :o:00 Ra be coordinate chart for O C M. We can represent v(u) O by its coordinates d(v(w)) E R.. Consider a choice of encoding and approximation maps such that ((E( gives the coordinates for v(u). If the decoder were chosen D = 6-1 then by con- struction, the operator F = D o A will satisfy
Pi
E [[((u) F(u) <. (14)
Figure 3: An example of linear versus nonlinear decoders.
Therefore, interpret learned decoding map as attempting to give finite dimensional coordinate system for the solution manifold. Consider a generalized decoder of the following form
D : R. x -.
(15)

---

This induces a map from D : R - (2())), as D(3) = D(3,)). ). If the solution manifold M is finite dimensional linear subspace in ())) spanned by 1- we would want a decoder use the coefficients along the basis as a coordinate system for M. A generalized decoder could learn this basis as the output of a deep neural network to act as
Dio (3, v) BITI +
(16)
However, if the solution manifold is not linear, then we should learn nonlinear coordinate system given by nonlinear D. A nonlinear version of D can be parameterized by using deep neural network f R x y - IR which jointly takes as arguments (3, (3,y).
D(3,y) y) f(3.
(17)
When used in the context of an operator learning architecture of the form 3 we call nonlinear decoder from [7) NOMAD (NOnlinear MAnifold Decoder). Figure presents visual comparison
between linear and nonlinear decoders.
Under the assumption of the Operator Learning Manifold Hypothesis, we have proposed fully nonlinear decoder (T7) to represent target functions using architectures of the form (3) We next show that using a decoder of the form (17) results in operator learning architectures which can learn nonlinear low dimensional solution manifolds. Additionally, we will see that when these solution manifolds do not "fit" inside low dimensional linear subspaces, architectures with linear decoders will either fail or require a significantly larger number of latent dimensions.
Summary of NOMAD:
Results
In this section we investigate the effect of using linear versus nonlinear decoders as building blocks of operator learning architecture taking the form (3). In all cases, we will use an encoder which takes wist evaluations of the input functions, and an approximator map A given by deep neural network. The linear decoder parametrizes a set of basis functions that are learned as the outputs of an MLP network. In this case, the resulting architecture exactly corresponds to the DeepONet model from [30]. We will compare this against using NOMAD where the nonlinear decoder built using an MLP network that takes inputs the concatenation of 3 E R" and given query point y E y. All models are trained with by performing stochastic gradient descent on the loss function in []. The reported eтгоrs measured in the relative L²())) norm by averaging over all functional pairs in the testing data-set. More details about architectures, hyperparameters settings, and training details are provided in the Supplemental Materials.
First, we revisit the motivating example from Section 4 where the goal is to learn the antiderivative operator (TO) acting on the set of functions (IT). In Figure 2 we see the performance of model with a lincar decoder and NOMAD over a range of latent dimensions n. For each choice of 10 experiments with random initialization seeds were performed, and the mean and standard deviation of testing errors are reported. We see that the NOMAD architecture consistently outperforms the linear one (by one order of magnitude), and can even achieve 10% relative prediction error using only n 1.
Learning the Antiderivative Operator:
Solution Operator of Parametric Advection PDE: Here we consider the problem of learning the solution operator to a PDE describing the transport of a scalar field with conserved energy.
s(x t)
s(z t) 0,
(18)
Øi
Or
over domain (z, [0,2] x 10, 1]. The solution operator maps an initial condition s(z, 0) u(x)
to the solution at all times s(z, t) which satisfies (18). We consider a training data-set of initial conditions taking the form of radial basis functions with very small fixed lengthscale centered at
randomly chosen locations in the interval [o, 1). We create the output functions by evolving these initial conditions forward in time for 1 time unit according to the advection equation (18) (see Supplemental Materials for more details). Figure 4a gives an illustration one such solution plotted over the space-time domain.
Performing PCA on the solution functions generated by these initial conditions shows very slow decay of eigenvalues (see Figure 4b), suggesting that methods with lincar decoders will require
7

---

0.0
10
Relaive (loga)
30
0.5
20
Bignnninn
10
Linesr deooler
10
-1.0
NOMAD
.00
-15
0.75
10-1
0.0
10.511
11.5
1.0
-20
1.5
2.0
0.001
10P
Dimccssio index
10
10
10
10
20
Latent dimension
30
40
30
60 70
80 50 100
(a)
(b)
(c)
Figure 4: Advection Equation: (a) Propagation of an initial condition function (highlighted in black)
through time according to (18): (b) log of the leading 1, 000 PCA eigenvalues of G(U): (c) Relative L testing (log10 scale) function of latent dimension for linear and nonlinear decoders (over 10 independent trials).
a moderately large number of latent dimensions. However, since the data-set was constructed by evolving of functions with single degree of freedom (the center of the initial conditions), would expect the output functions to form a solution manifold very low dimension.
In Figure 4c
compare the performance of linear decoder and NOMAD as function of the latent dimension vi. Linear decoders yield poor performance for small values n, while NOMAD appears to immediately discover good approximation the true solution manifold.
Propagation of Free-surface Waves: As a more challenging benchmark we consider the shallow- water equations; set of hyperbolic equations that describe the flow below pressure surface in fluid [4]] The underlying PDE system takes the form
8U OF aG 0t.
P pvl pu2
pud puf + 290²
+
0, U= =
F =
G
=
pod U2 pug g9
(19)
Ox
Oy
00102
where p(z, y, the fluid height from the free surface, is the gravity acceleration, and e1 (z, f). 22(2, y, denote the horizontal and vertical fluid velocities, respectively. We consider reflective boundary conditions and random initial conditions corresponding to random droplet falling into still fluid bed (see Supplemental Materials). In Figure we show the average testing error of model with a linear and nonlinear decoder as function of the latent dimension. Figure 5b shows snapshots of the predicted surface height function on top of a plot of the errors the ground truth for the best, worst, median, and a random sample from the testing data-set.
We additionally use this example compare the performance NOMAD to other state-of -aee--fffhhhtteeeee art operator learning architectures. For this experiment we include harmonic feature expansion in the DeepONet, improve its performance. We do not include this in the NOMAD architecture for this experiment (see Supplemental Materials for more details). In Table I we present the mean relative error and its standard deviation for different operator learning methods, as well as the prediction that provides the worst error in the testing data-set when compared against the ground truth solution. For each method we also report the number of its trainable parameters, the number of its latent dimension and the training wall-clock time minutes. Since the general form of the FNO (28) does not neatly fit into the architecture given by there is not directly comparable measure of latent dimension for it. also observe that, although the model with NOMAD closely matches the performance LOCA [23], its required latent dimension, total number of trainable parameters, and total training time all significantly smaller.
model with a linear decoder and
was shown in [32] that this can significantly
Discussion
Summary: have presented a novel framework for supervised learning function spaces. The proposed methods aim to address challenging scenarios where the manifold of target functions has

---

NOMAD
0.001
0.0025
10
30
70
0
0.0000
0.000
Relative erorr
-0.5
Decader
-1.0
best
wOTsE
-1.5
5.10
30
70
M
-1.5
Lines Decuder
0.002
0.001
-1.0-
NOMAD
-1.5
1.6
10
30
60
20
300
0.000
0.000
Latent dimensioo
median
random
(b)
Figure Propagation of Free-surface Waves: (a) Relative I testing error (log scale) function of latent dimension for linear and nonlinear decoders (over 10 independent trials); ; (b) Visualization of predicted free surface height p(r, y, 0.31) and point-wise absolute prediction error contours corresponding to the best, worst, and median samples in the test data-set, along with representative test sample chosen random.
Table 1: Comparison of relative L errors (in %) for the predicted output functions for the shallow water equations benchmark against existing state-of-the-aa operator learning methods: LOCA (3) DeepONet (DON) B00, and the Fourier Neural Operator (FNO) 28 The fourth column reports the relative L error for (p, v1, 12) corresponding to the worst case example in the test data-set. Also shown each model's s total number of trainable parameters dg, latent dimension n, and computational cost in terms of training time (minutes).
Method
01
U2
worst case
dg
/1
cost
LOCA
0.040 + 0.015
2.7 + 0.3
2.9 + 0.4
(0.1, 4.2)
0(10%)
480
12.1
DON
0.100 + 0.030
5.5
1.4
(0.6, 11,
0(10%)
480
15.4
FNO
0.140 + 0.060
3.4
3.2
80.4,8.9,8.77
0(10%)
N/A
14.0
NOMAD
0.048 + 0.017
2.0 0.4
0.3
(0.1, 5.1,5.8,499
0(105)
20
5.5
low dimensional structure, but is embedded nonlinearly into its associated function space. Such cases commonly arise across diverse functional observables in the physical and engineering sciences (e.g. turbulent fluid flows, plasma physics, chemical reactions), and pose a significant challenge to the application of most existing operator learning methods that rely linear decoding maps, forcing them to require an excessively large number of latent dimensions to accurately represent target functions. To address this shortcoming proposed fully nonlinear framework that can effectively learn low dimensional representations of nonlinear embeddings in function spaces, and demonstrated that achieves competitive accuracy state-of-the-ar operator learning methods while using a significantly smaller number of latent dimensions, leading to lighter model parametrizations and reduced training cost.
Limitations: Our proposed approach relies on the Operator Learning Manifold Hypothesis (see equation (12).. suggesting that cases where low dimensional manifold structure does not exist will be hard to tackle (c.g. target function manifolds with fractal structure, solutions to evolution equations with strange attractors). Moreover, even when the manifold hypothesis holds, the underlying effective latent embedding dimension is typically not known a-priori, and may only be precisely found via cross-validen valisaatiit..n. Another direct consequence replacing linear decoders with fully nonlinear maps is that the lower bound in (9) needs to be rephrased in terms of nonlinear n-width, which in general can be difficult to quantify. Finally, in this work we restricted ourselves exploring simple nonlinear decoder architectures such as an ML.s with the latent parameters B and query location concatenated as inputs. Further investigation is needed to quantify the improvements that could brought by considering more contemporary deep learning architectures, such as hypernetworks [16] which can

---

define input dependent weights for complicated decoder architectures. One example of this idea in the context of reduced order modeling can be found in Pan et. [37] where the authors propose hypernetwork based method combined with Implicit Neural Representation network [44]
Potential Negative Societal Impacts: Our contributions enables advances in accelerating the
modeling, simulation, and optimization of physical and engineering systems. As with any tool that furthers our understanding and ability to predict the outcomes of complex systems, there may be ill-intentioned use cases, but do not expect any specific negative impact from this work.
Acknowledgments and Disclosure of Funding
G.K. and P.P. would like to acknowledge support from the US Department of Energy under under the Advanced Scientific Computing Research program (grant DE-SC0019116), the US Air Force (grant AFOSR FA9550-20-1--00 100000)),,,,, and US Department of Energy/Advanced Research Projects Agency (grant DE-AR0001201), J.S. and G.P. would like to acknowledge support from the AFOSR under grant 199550019---0266- (Assured Autonomy in Contested Environments) and the NSF Simmons Mathematical and Scientific Foundations of Deep Learning (grant 2031985) We also thank the developers of the software that enabled our research, including JAX 3 Kymatio (. Matplotlib [20). Pytorch (38) and NumPy [17].
References
[1] Mathieu Andreux, Tomás Angles, Georgios Exarchakis, Roberto Leonarduzzi, Gaspar Rochette, Louis Thiry, John Zarka, Stéphane Mallat, Joakim Andén, Fugene Belilovsky, al. Kymatio: Scattering transforms in python. J. Mach. Learn. Res., 21 (60) 2020.
[2] Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric PDEs. The SMAI journal of computational mathematics, 7:121 2021.
[3] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas. Skye Wanderman-Miine and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[4] Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: A manifold disentangicment perspective. IEEE transactions on neural networks and learning systems, 27(10):999-2200 2015.
[5] Lawrence Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1, 2005.
[6] Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven L. Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(455)22445-2245,, 2019.
[7] Anindya Chatterjee. An introduction the proper orthogonal decomposition. Current Science, pages 808-817, 2000.
[8] Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions Neural Networks, 6(4):911-917, 1995.
[9] Albert Cohen and Ronald De Vore. Approximation of high- digh-dimensional parametric PDEs. Acta Numerica, 24:1-159, 2015.
[10] Albert Cohen, Ronald Devore, Guergana Petrova, and Przemyslaw Wojtaszezyk. Optimal stable nonlinear approximation. Foundations of Computational Mathematics, pages 1-42, 2021.
[11] Ronald Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5-30, 2006.
10

---

[12] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1):53-65, 2018.
[13] Maarten De Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew Stuart. The cost- accuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203. 13181, 2022.
[14] Rudy Geelen, Stephen Wright, and Karen Willcox. Operator inference for non-intrusive model reduction with nonlinear manifolds. arXiv preprint arXiv:2205.02304. 2022.
[15] Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- operator learning for differential equations. Advances in Neural Information Processing Systems, 34, 2021.
[16] David Ha, Andrew M. Dai, and Quoe Le. Hypernetworks. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[17] Charles Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585(7825):357-36,, 2020.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rX1v:1606.00815, 2016.
[20] John D Hunter. Matplotlib: A 2D graphics environment. IEEE Annals of the History of Computing, 903)) 95, 2007.
[21] Pengzhan Jin, Shuai Meng, and Lu Lu. MIONet: Learning multiple-input operators via tensor product. arXiv preprint 22XIv:2202.06137, 2022.
[22] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[23] Georgios Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George Pappas, and Paris Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1-63, 2022.
[24] Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, ArtArt, 2021.
[25] Samuel Lanthaler, Siddhartha Mishra, and George Karniadakis. Error estimates for Deep- ONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications. 6(1 ):tma0001, 2022.
[26] Kookjin Lee and Kevin T Carlberg. Model reduction of dynamical systems on nonlinear mani- folds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020.
[27] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003. 034855, 2020.
[28] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadeneshelii Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, et al. Fourier Neural Operator for parametric partial differential equations. In International Conference on Learning Representatiinss, 2020.

---

[29] YC Liang, HP Lee, SP Lim, Lin, KH Lee, and CG1237 Wu. Proper orthogonal decompo- sition and its applications Part I: Theory, Journal of Sound and vibration, 252-3):5 2002.
Part
[30] Lu Lu,, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 22918,2222222 2021.
[31] Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based fair data. arXiv preprint arXiv:2111.055 /2, 2021.
[32] Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data. Computer Methods in Applied Mechanics and Engineering, 393:1 14778, 2022.
[33] Romit Maulik, Bethany Lusch, and Prasanna Balaprakash. Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders. Physics of Fluids, 33()):03710, 2021.
[34] Parviz Moin. Fundamentals of engineering numerical analysis. Cambridge University Press, 2010.
[35] Boaz Nadler, Stéphane Lafon, Ronald R Coifman, and Ioannis G Kevrekidis. Diffusion maps, spectral clustering and reaction coordinates of dynamical systems. Applied and Computational Harmonic Analysis,
13-127, 2006.
[36] Nicholas H Nelsen and Andrew M Stuart. The random feature model for input-output maps between Banach spaces. SIAM Journal on Scientific Computing, 43(5):A3212-33243 2021.
[37] Shaowu Pan, Steven L Brumton, and J Nathan Kutz. Neural implicit flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal data, arXiv preprint arXiv:2204.03216. 2022.
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-888 -037, 2019.
[39] Karl Pearson. Liii on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosoppical magazine and journal of science, 2(11):559-572, 1901.
[40] Allan Pinkus. N-widths in Approximation Theory, volume 7. Springer Science & Business Media, 2012.
[41] Wilhelmus HA Schilders, Henk A Van der Vorst, and Joost Rommes. Model order reduction: theory, research aspects and applications, volume 13. Springer, 2008
[42] Bernhard Schölkopf, Alexander Smola, and Roberr Müller. Nonlinear component analysis as kernel eigenvalue problem. Neural computation, 10(5): 1299-1191, 1998.
[43] Lawrence Sirovich. Turbulence and the dynamics of coherent structures. i. coherent structures. Quarterly of applied mathematics, 45(3):561-571, 1987.
[44] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[45] Jun Song and Bing Li. Nonlinear and additive principal component analysis for functional data. Journal of Multivariate Analysis, 181:104675, 2021.
[46] Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 29015500):319-2223,, 2000.
12

---

[47] Laurens van der Maaten, Eric O. Postma, and Jaap van den Herik. Dimensionality reduction: A comparative review. 2009.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[49] Cornelis Boudewijn Vreugdenhil. Numerical methods Springer Science & Business Media, 1994.
for shallow-water flow, volume 13.
[50] Jane-Ling Wang, Jeng-Min Chiou, and Hans-Georg Müller. Functional data analysis. Annual Review of Statistics and Its Application, 3:257-295, 2016.
[51] Sifan Wang, Hanwen Wang, and Paris Perdikaris. Improved architectures and training algorithms for deep operator networks. arXiv preprint 10.065,,, 2021.
[52] Yasi Wang, Hongxun Yao, and Sicheng Zhao. Auto-encoder based dimensionality reduction. Neurocomputing, 184:232 242, 2016.
13


Reviews:
[{"review_id": "f9UvjM9FvC2", "reviewer": "Reviewer_ixP9", "summary": "The authors contribute a thorough, sound, and clearly explained lower bound to the approximation quality of learned operators with linear decoders. Further, the authors propose NOMAD: an extension of DeepONet to nonlinear decoders which significantly outperforms DeepONet in terms of accuracy at a given number of model parameters on the shallow-water equation, advective flow, and integrating a cosine function.  \n", "questions": "3.1 Is the universal operator approximation theorem valid for NOMAD? DeepONet has shown that a universal operator approximation theorem can be shown for a linear decoder. Replacing the linear decoder with an MLP seems to invalidate that theorem. \n\n3.2 Can NOMAD exploit and learn spatial correlations? Similar to DeepONet, NOMAD uses the space, x, as input. So, there does not seem a step to learn spatial correlations, e.g., via CNNs, GNNs, Fourier transforms, or other.\n\n3.3 Could NOMAD also be used to replace the linear decoder in FNO? Or does NOMAD only work in combination with DeepONet?\n", "limitations": "4.1 The authors clearly state that the efficiency of NOMAD relies on the existence of low-dimensional nonlinear solution submanifolds and clearly state the PDE cases for which this is might not be true. \n\n4.2 It would be helpful for the future reader, if the authors address the mentioned questions in the camera ready version. \n\n4.3 The paper includes a satisfactory paragraph on potential negative societal impact. \n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Strengths: \n\n1.1 The general topic area of learning operators is of very high significance for reduced-order modeling, uncertainty quantification, parameter exploration, and sensitivity analysis in climate science, physics, fluid dynamics and more. Existing architectures, such as DeepONet and FNO, have shown proof of concepts but can be improved in accuracy. \n\n1.2 The authors convey the limitations of current operator learning architectures with linear decoders with great clarity and intuition. The limitations are supported by a sound proof on the lower bound on the approximation and quality references to relevant works in the reduced-ordor modeling literature. The math is sound and all variables are defined with name and dimensions. \n\n1.3 The proposed method to replace a linear with nonlinear decoding is easy to implement. \n\n1.4 The results support the claimed contributions. Indeed, NOMAD is more accurate than DeepONet if the latent space is limited to a few parameters.  \n\n2. Weaknesses:\n\n2.1 The results contain three PDEs that have been selected to highlight the benefit of NOMAD. It would have been interesting to see how NOMAD performs on standard benchmarks that have been used in FNO, DeepONet paper; specifically incompressible  Navier-Stokes at low Reynolds number. It would have also been interesting to include FNO in Table 1. However, the results section is of sufficient quality and extent to support the authors claims. \n\n2.2 NOMAD does not address how to do autoregressive prediction, but it also seems out of scope for the strong theoretical contributions that NOMAD makes.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "f6YjjZfePV9", "reviewer": "Reviewer_Dpah", "summary": "The authors consider a functional regression problem, i.e., given a dataset of pairs of functions, learn an operator which maps input functions to output functions. They represent the operator as a combination of an encoder, an approximator in a latent space, and a decoder that maps a latent representation into a target function. The authors claimed that a typical decoder is linear, i.e., it outputs a linear combination of some basis functions, which depends on the input vector of the predicted target function. However, in the case of a nonlinear manifold, we should use a nonlinear decoder, so the authors proposed to use a nonlinear decoder represented by a neural network. They demonstrated the usefulness of such modification based on several examples, including cases of PDE systems, which maps some functional input condition into an output functional solution.", "questions": "- 79: The authors mentioned “EnhancedComputationalEfficiency”. However, they did not provide any information about the computational efficiency in the text of the paper.\n- 132: here, mu is a probability measure on L^2(X). (5): here, mu is a probability measure on an infinite-dimensional space of output functions. There is some contradiction in notations. What is U in (5)? It is not defined earlier in the text.\n- 260: antidervative -> antidirevative?", "limitations": "- There is no ablation study. The authors did not provide any results on how accuracy depends on various hyperparameters, sampling rates of the input/output functions, etc.\n- There is no comparison with SOTA functional regression approaches.\n- If the input/output functions are complex, we should use some very dense grid to represent them. However, in such case, it is not clear which architectural choices we should make about the structure of the neural networks based encoder/decoder/approximator", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The problem of functional regression is important for applications. So the topic of the paper is important\n- The authors provided some simple theoretical analysis; in particular, they interpreted the approximation error as the Kolmogorov n-width and demonstrated that in the case of a linear decoder, the approximation error is lower bounded by the n-width. I consider this a trivial result; moreover, the authors did not use it anyhow in their algorithm; however, the interpretation can be useful.\n- The authors demonstrated that the nonlinear decoder provides a smaller approximation error\n\nWeakness\n- The novelty of the paper is limited. The main improvement is that the authors used a nonlinear decoder represented by a neural network.\n- The experimental section is very limited. It does not contain any comparison with various competing methods. For example, various neural networks based realizations of the so-called Koopman operator (https://www.ias.informatik.tu-darmstadt.de/uploads/Team/JoeWatson/Damken_Bsc_2020.pdf) can also be used for solving the same type of a problem. Also, what about comparing the considered approach to the approach from https://arxiv.org/abs/1506.07365 , which can be easily adapted to the considered case? A solution of a PDE can be represented as a tensor. What about comparing the proposed approach to SOTA image2image approaches based on deep neural networks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aMA1bZxlrQP", "reviewer": "Reviewer_AwfW", "summary": "This paper proposes a supervised learning method called NOMAD to learn operators (maps between infinite-dimensional function spaces). Like many previous operator-learning methods, they have an encoder (from function space to R^m), an approximator (from R^m to R^n) approximating the operator, and a decoder (from R^n to the new function space). Unlike previous papers with this framework, the decoder can be nonlinear, with the idea that although there are guarantees about linear representations, nonlinear representations can be more efficient/lower-dimensional. They test their method on the antiderivative operator, linear transport & nonlinear wave propagation and compare NOMAD to state-of-the-art methods on the 3rd example. NOMAD has better or close accuracy, but with fewer latent dimensions and increased computational efficiency.", "questions": "FNO is \"resolution-invariant.\" Is a limitation of NOMAD that the grid needs to be consistent across examples?", "limitations": "I thought that the limitations section was well-written and unusually thoughtful.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Originality:* To my understanding, NOMAD's architecture is not very different from some of the previous papers, but has a nonlinear decoder. However, the previous methods are presented in a unified way that is easy to understand, and the authors make a solid argument for these methods being consistently restricted by having a linear decoder. \n\n*Quality:* I think that the claims are sound. The paper could be strengthened with more extensive experiments comparing against previous papers. It's also mentioned in the appendix that they compare against the simplest possible DeepONet so that they can isolate the effect of the decoder. Perhaps this is not a fair representation of the power of DeepONet?\n\n*Clarity:* There is some tricky math in this paper, but overall, I think that the main points were clear. I appreciated the anti-derivative & parametric advection PDE examples for building intuition that there can be some low-dimensional structure that linear decoders cannot capture.\n\n*Significance:* If NOMAD really is a more accurate and more efficient approach to operator learning than methods like DeepONet & FNO, which are getting a lot of attention, then it could be very impactful. As mentioned above, I think the argument could be strengthened with more experiments comparing the methods.\n\n\n\nUPDATE: the response was helpful, and I'm increasing my rating. Thanks!", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D7s2zyVruSe", "reviewer": "Reviewer_uiyi", "summary": "The paper proposes a nonlinear decoder in a computational framework for supervised operator learning, which maps a function space to another function space. The paper has presented a general interpretation of operator learning, which consists of encoder-approximation map-decoder and proposes to use a nonlinear decoder instead of existing linear decoders. ", "questions": "- how does the size of the latent dimension affect the performance? In Fig 4.C, increasing the latent dimension has no effect or even increases the l2 error. \n- would there be any general suggestion or guidance in choosing the size of latent dimension/ neural network choices ? \n- in the basis point of view, how does the dimension of the last to the second layer affect the performance? if the last layer is the linear layer, the columns of the weight matrix in the last layer can be considered as a basis. How does the number of the columns in the last layer affect the performance? ", "limitations": "the authors adequately addressed the limitations in the manuscript. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] well-written and well-motivated\n\n[+] proposes a simple, but essential extension to existing the operator learning paradigm\n\n[-] hyperparameter search to some extent would make the contribution stronger and convincing. In the current presentation, several things are unclear (in particular, deep learning aspects) e.g., how the size of latent dimension affects the performance, how the performance would be if the decoder is more powerful, if a more expressive decoder is employed, even lower dimensional latent space can be employed? would a specific nonlinearity or learning rate result in better performance and so on. Some intuitions or elaborations on rationale behind those choices would be appreciated. \n\n[-] although the authors add references for neural network architectures used in the experiments, those information could be included in the manuscript directly.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2Aar--3ajgG", "reviewer": "Reviewer_MKwA", "summary": "This paper deals with supervised regression when both the input and output data are functions, with application to partial differential equations (PDEs). Classical techniques include a linear modelling of the relationship between the input and output variables, which can then take the form Linear Encoder -> Linear Operator -> Linear Decoder.\n\nIn this work, authors show the theoretical limitations of taking the decoder to be a linear map, and propose a novel neural architecture based on a nonlinear decoder that allows to learn operators with a smaller latent dimension in the \"operator learning manifold hypothesis\" scenario. Experimental results illustrate the benefit of the approach on several benchmarks corresponding to various PDEs.", "questions": "Questions:\n\n- Despite the PDEs being linear, the mapping to learn may no be linear, so why is the linear modelling necessary ? Especially, one could use vector-valued RKHSs to model such relationship, with the additional benefit of being able to regularize the empirical risk using the vv-RKHS norm.\n- Concerning the $f(\\beta, y)$ network, since $y$ is a function does it need to go through the encoder before entering $f$ ? Could you precise the architecture of such a network ?\n\nSuggestions:\n- Line 227 could be clarified: if $\\tilde{D}$ is a generalized decoder, then $D$ does not have to be linear in $y$.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is clear and easy to follow.\n- References are well organized.\n- Topic of operator learning is relevant to the NeurIPS community and is likely to have multi-disciplinary benefits.\n- Experimental section is convincing.\n\nWeaknesses:\n\n- The technical contribution of the paper appears moderate, as it mainly consists in replacing the linear decoder by a nonlinear one obtained with a neural network. The part about the limitations of the linear decoders reads more as a remark than a real theoretical contribution.\n- There are some levels of approximation which are not discussed in the paper. Especially, by adopting the functional space view, the fact that one cannot directly build a neural net $f(\\beta, y)$ when $y$ is a function is swept under the rug.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept