Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors contribute a thorough, sound, and clearly explained lower bound to the approximation quality of learned operators with linear decoders. Further, the authors propose NOMAD: an extension of DeepONet to nonlinear decoders which significantly outperforms DeepONet in terms of accuracy at a given number of model parameters on the shallow-water equation, advective flow, and integrating a cosine function.  

strengths_and_weaknesses: 1. Strengths: 

1.1 The general topic area of learning operators is of very high significance for reduced-order modeling, uncertainty quantification, parameter exploration, and sensitivity analysis in climate science, physics, fluid dynamics and more. Existing architectures, such as DeepONet and FNO, have shown proof of concepts but can be improved in accuracy. 

1.2 The authors convey the limitations of current operator learning architectures with linear decoders with great clarity and intuition. The limitations are supported by a sound proof on the lower bound on the approximation and quality references to relevant works in the reduced-ordor modeling literature. The math is sound and all variables are defined with name and dimensions. 

1.3 The proposed method to replace a linear with nonlinear decoding is easy to implement. 

1.4 The results support the claimed contributions. Indeed, NOMAD is more accurate than DeepONet if the latent space is limited to a few parameters.  

2. Weaknesses:

2.1 The results contain three PDEs that have been selected to highlight the benefit of NOMAD. It would have been interesting to see how NOMAD performs on standard benchmarks that have been used in FNO, DeepONet paper; specifically incompressible  Navier-Stokes at low Reynolds number. It would have also been interesting to include FNO in Table 1. However, the results section is of sufficient quality and extent to support the authors claims. 

2.2 NOMAD does not address how to do autoregressive prediction, but it also seems out of scope for the strong theoretical contributions that NOMAD makes.

questions: 3.1 Is the universal operator approximation theorem valid for NOMAD? DeepONet has shown that a universal operator approximation theorem can be shown for a linear decoder. Replacing the linear decoder with an MLP seems to invalidate that theorem. 

3.2 Can NOMAD exploit and learn spatial correlations? Similar to DeepONet, NOMAD uses the space, x, as input. So, there does not seem a step to learn spatial correlations, e.g., via CNNs, GNNs, Fourier transforms, or other.

3.3 Could NOMAD also be used to replace the linear decoder in FNO? Or does NOMAD only work in combination with DeepONet?

limitations: 4.1 The authors clearly state that the efficiency of NOMAD relies on the existence of low-dimensional nonlinear solution submanifolds and clearly state the PDE cases for which this is might not be true. 

4.2 It would be helpful for the future reader, if the authors address the mentioned questions in the camera ready version. 

4.3 The paper includes a satisfactory paragraph on potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a supervised learning method called NOMAD to learn operators (maps between infinite-dimensional function spaces). Like many previous operator-learning methods, they have an encoder (from function space to R^m), an approximator (from R^m to R^n) approximating the operator, and a decoder (from R^n to the new function space). Unlike previous papers with this framework, the decoder can be nonlinear, with the idea that although there are guarantees about linear representations, nonlinear representations can be more efficient/lower-dimensional. They test their method on the antiderivative operator, linear transport & nonlinear wave propagation and compare NOMAD to state-of-the-art methods on the 3rd example. NOMAD has better or close accuracy, but with fewer latent dimensions and increased computational efficiency.
strengths_and_weaknesses: *Originality:* To my understanding, NOMAD's architecture is not very different from some of the previous papers, but has a nonlinear decoder. However, the previous methods are presented in a unified way that is easy to understand, and the authors make a solid argument for these methods being consistently restricted by having a linear decoder. 

*Quality:* I think that the claims are sound. The paper could be strengthened with more extensive experiments comparing against previous papers. It's also mentioned in the appendix that they compare against the simplest possible DeepONet so that they can isolate the effect of the decoder. Perhaps this is not a fair representation of the power of DeepONet?

*Clarity:* There is some tricky math in this paper, but overall, I think that the main points were clear. I appreciated the anti-derivative & parametric advection PDE examples for building intuition that there can be some low-dimensional structure that linear decoders cannot capture.

*Significance:* If NOMAD really is a more accurate and more efficient approach to operator learning than methods like DeepONet & FNO, which are getting a lot of attention, then it could be very impactful. As mentioned above, I think the argument could be strengthened with more experiments comparing the methods.



UPDATE: the response was helpful, and I'm increasing my rating. Thanks!
questions: FNO is "resolution-invariant." Is a limitation of NOMAD that the grid needs to be consistent across examples?
limitations: I thought that the limitations section was well-written and unusually thoughtful.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors consider a functional regression problem, i.e., given a dataset of pairs of functions, learn an operator which maps input functions to output functions. They represent the operator as a combination of an encoder, an approximator in a latent space, and a decoder that maps a latent representation into a target function. The authors claimed that a typical decoder is linear, i.e., it outputs a linear combination of some basis functions, which depends on the input vector of the predicted target function. However, in the case of a nonlinear manifold, we should use a nonlinear decoder, so the authors proposed to use a nonlinear decoder represented by a neural network. They demonstrated the usefulness of such modification based on several examples, including cases of PDE systems, which maps some functional input condition into an output functional solution.
strengths_and_weaknesses: Strength
- The problem of functional regression is important for applications. So the topic of the paper is important
- The authors provided some simple theoretical analysis; in particular, they interpreted the approximation error as the Kolmogorov n-width and demonstrated that in the case of a linear decoder, the approximation error is lower bounded by the n-width. I consider this a trivial result; moreover, the authors did not use it anyhow in their algorithm; however, the interpretation can be useful.
- The authors demonstrated that the nonlinear decoder provides a smaller approximation error

Weakness
- The novelty of the paper is limited. The main improvement is that the authors used a nonlinear decoder represented by a neural network.
- The experimental section is very limited. It does not contain any comparison with various competing methods. For example, various neural networks based realizations of the so-called Koopman operator (https://www.ias.informatik.tu-darmstadt.de/uploads/Team/JoeWatson/Damken_Bsc_2020.pdf) can also be used for solving the same type of a problem. Also, what about comparing the considered approach to the approach from https://arxiv.org/abs/1506.07365 , which can be easily adapted to the considered case? A solution of a PDE can be represented as a tensor. What about comparing the proposed approach to SOTA image2image approaches based on deep neural networks?
questions: - 79: The authors mentioned “EnhancedComputationalEfficiency”. However, they did not provide any information about the computational efficiency in the text of the paper.
- 132: here, mu is a probability measure on L^2(X). (5): here, mu is a probability measure on an infinite-dimensional space of output functions. There is some contradiction in notations. What is U in (5)? It is not defined earlier in the text.
- 260: antidervative -> antidirevative?
limitations: - There is no ablation study. The authors did not provide any results on how accuracy depends on various hyperparameters, sampling rates of the input/output functions, etc.
- There is no comparison with SOTA functional regression approaches.
- If the input/output functions are complex, we should use some very dense grid to represent them. However, in such case, it is not clear which architectural choices we should make about the structure of the neural networks based encoder/decoder/approximator
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper deals with supervised regression when both the input and output data are functions, with application to partial differential equations (PDEs). Classical techniques include a linear modelling of the relationship between the input and output variables, which can then take the form Linear Encoder -> Linear Operator -> Linear Decoder.

In this work, authors show the theoretical limitations of taking the decoder to be a linear map, and propose a novel neural architecture based on a nonlinear decoder that allows to learn operators with a smaller latent dimension in the "operator learning manifold hypothesis" scenario. Experimental results illustrate the benefit of the approach on several benchmarks corresponding to various PDEs.
strengths_and_weaknesses: Strengths:

- The paper is clear and easy to follow.
- References are well organized.
- Topic of operator learning is relevant to the NeurIPS community and is likely to have multi-disciplinary benefits.
- Experimental section is convincing.

Weaknesses:

- The technical contribution of the paper appears moderate, as it mainly consists in replacing the linear decoder by a nonlinear one obtained with a neural network. The part about the limitations of the linear decoders reads more as a remark than a real theoretical contribution.
- There are some levels of approximation which are not discussed in the paper. Especially, by adopting the functional space view, the fact that one cannot directly build a neural net $f(\beta, y)$ when $y$ is a function is swept under the rug.

questions: Questions:

- Despite the PDEs being linear, the mapping to learn may no be linear, so why is the linear modelling necessary ? Especially, one could use vector-valued RKHSs to model such relationship, with the additional benefit of being able to regularize the empirical risk using the vv-RKHS norm.
- Concerning the $f(\beta, y)$ network, since $y$ is a function does it need to go through the encoder before entering $f$ ? Could you precise the architecture of such a network ?

Suggestions:
- Line 227 could be clarified: if $\tilde{D}$ is a generalized decoder, then $D$ does not have to be linear in $y$.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a nonlinear decoder in a computational framework for supervised operator learning, which maps a function space to another function space. The paper has presented a general interpretation of operator learning, which consists of encoder-approximation map-decoder and proposes to use a nonlinear decoder instead of existing linear decoders. 
strengths_and_weaknesses: [+] well-written and well-motivated

[+] proposes a simple, but essential extension to existing the operator learning paradigm

[-] hyperparameter search to some extent would make the contribution stronger and convincing. In the current presentation, several things are unclear (in particular, deep learning aspects) e.g., how the size of latent dimension affects the performance, how the performance would be if the decoder is more powerful, if a more expressive decoder is employed, even lower dimensional latent space can be employed? would a specific nonlinearity or learning rate result in better performance and so on. Some intuitions or elaborations on rationale behind those choices would be appreciated. 

[-] although the authors add references for neural network architectures used in the experiments, those information could be included in the manuscript directly.  
questions: - how does the size of the latent dimension affect the performance? In Fig 4.C, increasing the latent dimension has no effect or even increases the l2 error. 
- would there be any general suggestion or guidance in choosing the size of latent dimension/ neural network choices ? 
- in the basis point of view, how does the dimension of the last to the second layer affect the performance? if the last layer is the linear layer, the columns of the weight matrix in the last layer can be considered as a basis. How does the number of the columns in the last layer affect the performance? 
limitations: the authors adequately addressed the limitations in the manuscript.
review_title: null

Meta-review: metareview: The focus of the submission is supervised learning with functional inputs and outputs. Particularly, the authors consider the
encoder-approximator-decoder architecture (2)-(3) to tackle this task. After discussing the limitations of linear decoders in this scheme (meant in L^2 and uniform sense; the latter is elaborated in Proposition 1), the authors present the nonlinear NOMAD architecture under the assumption of operator learning manifold hypothesis (12) which captures a low-dimensional output space condition. They demonstrate the efficiency of the approach compared to the DeepONet method relying on linear decoders, Fourier neural operators and LOCA (i) when using stochastic gradient descent on the empirical loss (1), (ii) on the learning problem of the antiderivative operator and learning the solution of two partial differential equations.

Functional data analysis is a fundamental area of machine learning with a large number of applications. The authors present a new method in this context which can be of definite interest to the community as it was assessed by the reviewers.
confidence: Certain
Decision: Accept