Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The work develops Multiwavelet-based neural operator using multiwavelet Transform and orthogonal polynomials. The model applies multiscale transfer and kernel integration. Empirically the model is more accurate compare to Fourier neural operator on many PDEs.
main_review: In my opinion, this work is a natural and novel extension to neural operators. The architecture is very solid and the performance is quite good. I would vote for strong acceptance.

Concerns:
(1) It's understandable that most of the benchmark numbers are copied from the previous papers. For the Burgers' equation, it was reported as 0.0149 on FNO but if the batch norm layers are taken out, the error on FNO will be ~ 0.002. This can also be true for the Korteweg-de Vries equation. This may explain why the gap of MWT and FNO is so larger on Burgers and Korteweg-de Vries, but not so much on Darcy and Navier-stokes. 
To have a fair and solid comparison, I strongly suggest the author use the tuned FNO model and re-run for Burgers and Korteweg-de Vries equation.
(2) MWT has a multi-scale structure, which may imply the overall runtime of MWT can be slower. It will be great if the author could report the runtime of MWT and how it compares to the baseline models.
limitations_and_societal_impact: It will be better if the author can discuss when wavelet does not apply. To my knowledge, while there are many works on wavelets, but in general people still prefer FEM/FDM solvers, etc.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a multiwavelet-based neural operator learning scheme that compresses the associated operator’s kernel using fine-grained wavelets. Using the proposed model, the paper performs experiments on the Korteweg-de Vries (KdV) equation, Burger’s equation, Darcy Flow, and Navier-Stokes equation. Compared with the existing neural operator approaches, this model claims to show significantly higher accuracy and can find the solution of a high-resolution input after learning from lower-resolution data. This paper appears to be a novel proposal, and certainly of interest. However, the method only applies to some toy examples, its application prospect seems narrow.

main_review: Strengths


This paper operates in a general area (neural PDE) of great interest to the machine learning community. The method demonstrates the applicability on higher dimensions of the 2-D Darcy flow equation in Section 3.4, which is a good point. Also, I appreciate the detailed supplementary material provided by the authors.
 
Weaknesses


- The introduction of the theory seems a little indirect and not concise, such as Section 2.1 is a bit redundant.
- Although the paper shows a lot of error curves, the results are rarely shown. To enhance readers' intuition of the predictive power of the presented model, the temporal evolution of the PDE solutions should be visualized.
- The result comparison is not convincing in its current form. It's described in a fairly vague way, and there's a lack of results showing that it works. This part would need significant extensions, or potentially could be left out.
- Table 3 in supplementary material looks like the simulation error is too large to be practical.


Correctness


The technical content of this paper appears to be correct.
 

Clarity


Overall, the exposition of the paper is fairly clear. But some details are confusing:
- MWT model architecture in Figure 2 is confusing. I can't follow the data flow of the network.
- What are the two subfigures in Figure 3?  Please specify the meaning of these two figures.
 


Relation to prior art


This paper compares its MWT model using two different OP bases (Leg, Chb) with the most recent successful neural operators, such as the graph neural operator (GNO), the multipole graph neural operator (MGNO), the LNO which makes a low-rank representation of the operator kernel K(x; y), and the Fourier neural operator (FNO ). The MWT model claims to show the ability to learn the function mapping through lower-resolution data, and be able to generalize to higher resolutions. These methods are compared in the results section, but the demonstration seems to need to be strengthened. Also, it would be helpful to add some citations of recent Neural ODE/PDE solvers, such as papers related to conservation laws, symplectic neural networks, and neural vortex methods.
 
Reproducibility


The method description is clear in general and reasonably easy to follow.


 
Additional Feedback/Questions


How to handle the modal decomposition of a problem with nontrivial boundary conditions?
 



limitations_and_societal_impact: I would love to see some limitations of the proposed method. In addition, the method should discuss more robustness to real noise data.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of approximating the solution operator of a partial differential equation (PDE) that maps the inputs to the solution. The authors begin by reviewing the deep learning techniques that have been recently introduced and the multiple applications of this problem. The main contribution of the paper is to propose a new neural network architecture, based on the multiwavelet transform, to approximate the kernel of the solution operator. Finally, the authors compare their method against several Neural operator techniques (Fourier neural operator, graph neural operator, ...) on a number of numerical PDE examples and achieve lower relative error (2X - 4X).
main_review: ## Originality

The problem of approximating the solution operator is not novel and many techniques have been recently proposed, which the authors reference properly in the introduction. A deep learning approach based on the wavelet transform has also been proposed by [1] and I believe that the authors essentially generalize this approach to employ an orthonormal polynomial basis (Legendre or Chebyshev basis) as basis for the wavelet construction. The paper would gain in quality by explaining the advantages and drawbacks of this approach compared to [1]. The dimension of the subspace generated is larger, which gives higher approximation power, but does it come with higher computational training time for the networks ?

[1] Feliu-Faba, Fan, Ying, J. Comput. Phys., 2020.

## Clarity

The introduction of the paper is well-written and summarizes the potential applications as well as the recent works on the topic. However, I found several places where the paper lacks clarity, beginning with section 2. In the problem setup, the problem is not clearly defined and I do not see what the function a(x) represents for the PDE: is it a forcing term, a coefficient function in the PDE ? 
Similarly, the authors "do not put the restriction of linearity" on the operator T, but represent it as an integral kernel and try to approximate the kernel. This formulation won't hold for the nonlinear PDEs that the authors are considering later.

Section 3.4 discusses standard properties of integral kernels but the definition and property cited are not mathematically rigorous as they omit the definition of the objects or the spatial domain. Furthermore, Property 1 does not hold for the Green's function of the Laplace operator in spatial dimension 3 (the kernel is singular and not continuous along the diagonal as stated by the property).
The experimental setup at the beginning of section 3 also lacks clarity: do the authors learn the operator for 1 input-output pair (a(x),u(x)), evaluated at N points of the domain (i.e. one-shot learning), or do they choose several input-output pairs, which they then sample on the domain ?

Most of the section A of the supplementary material is very standard and the authors could just reference the relevant results later instead of redefining Legendre, Chebyshev polynomials or describing Gram-Schmidt orthogonalization.

## Quality

The authors propose to use a compressed representation of the integral kernel (or Green's function) because learning a direct representation of the kernel is expensive in high dimensions (essentially because the kernel has twice the spatial dimension of the solutions to the PDE).

 However, it is feasible to learn the kernel for the problems that the authors are considering in the paper (spatial dimension 1,2, and simple square geometries) [2,3]. Moreover, even for simple problems (Poisson equation in 1D/2D), the kernel can be challenging to learn due to the singularity along the diagonal. I'm wondering whether the compressed approach discussed in the paper would be able to accurately learn the Green's function in this setting and outperform the methods proposed by [2,3] by comparing the L2 norm of the kernel function. 
In  fact, using a testing set of functions to benchmark the method can be misleading, especially if the testing set is generated from the same distribution as the training set, because the method could simply overfit the training functions and not be accurate for any type of input functions.

Similarly, the choice of the Gaussian random fields for the input functions (e.g. $f \sim \mathcal{N}(0, 7^4(-\Delta+7^2 I)^{-2.5}$)) seems a bit strange and varies across the different examples. Since the method is also assessed on functions generated by the same distribution, it is also a possibility that the method performs well for this particular choice but fails for any other choice. 
Why did the authors not consider the standard squared-exponential covariance kernel ?

Finally, the benchmarks are reported on an average of 3 experiments, but the authors did not report the individual errors, which questions the stability of the technique with respect to multiple runs and the standard deviation of the errors achieved.

[2] Li et al., arxiv:2003.03485, 2020.

[3] Boulle, Earls, Townsend, arxiv:2105.00266, 2021.

## Significance

The gain in the relative error (2X-4X) achieved by the author is moderately significant but is compensated by the different flaws in the numerical experiments pointed out in the previous section. This neural network architecture could be of practical interest, provided it is either computationally faster than existing techniques or more accurate (but validated by more robust numerical experiments). The theoretical justification of the method is a bit vague due to the lack of mathematical rigour in sections 2.1-2.4 and probably does not apply for the range of examples that the authors are considering.
limitations_and_societal_impact: I do not see any negative societal impact in this study. One limitation that the authors could discuss is that the method currently works on simple tensor-products domain, on which simple polynomial bases exist. I do not know if the method naturally extends to more complex geometries or non-uniform spatial discretization.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to use a multiwavelet representation in order to learn operators which compute non-linear PDEs. The input and output functions are represented in multiwavelet bases.  The key idea is that multiwavelets provide sparse representation of large class of Calderon-Zygmund operators with a "non-standard form" which keep both the wavelet and scaling function coefficients, and are thus used in numerical schemes to solve many PDEs such as Burger equations or KdV.  In these cases, the specific non-linear operator corresponding to a PDE is a non-linear map between two sparse representations and are thus expected to be easier to learn then from the original time or spatial functions. The operator learning is performed with neural networks applied at each scale. The algorithm is resolution independent because the resolution parameter is incorporated in the multiwavelet basis. Results are demonstrated over Burger, Korteweg-de Vries and Darcy's flow with better than state of the art results. 
main_review: This paper is presenting an interesting approach to learn non-linear operators arising from PDEs. By using prior information on the nature of the solutions and the underlying operators, it  builds a sparse representations which simplifies the estimation problem. This type of work at the interface between harmonic analysis representations of PDEs solutions and neural networks is interesting and can potentially lead to real applications. At this point, the problems that is studied are very academic (one-dimensional Burger or KdV or Darcy flows) but they are none the less interesting. Whereas the paper spends a lot of time on the multiwavelet representation and its properties to represent Calderon-Zygmund integral operators, which are known in harmonic analysis, they spend little space on the description of the neural networks that perform the learning, the choice of network and an analysis of its properties. It would be very useful to see how the complexity of the network relates to the complexity of the underlying learned operator once represented in a sparse non-standard form. Overall this is a good paper that makes an interesting link between PDE and learning. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The manuscript entitled “Multiwavelet-based Operator Learning for Differential Equations” exploits the fundamental properties of the operator’s kernel which enable numerically efficient representation. The proposed method exhibits a 10X improvement on relative L2 error for some of the PDEs compared with the state-of-the-art neural operators, which is very impressive. The proposed method also shows the ability to find the solution of a high-resolution input after learning from lower-resolution data.
main_review: This paper will have a significant impact in the field of operator learning for differential equations. I am looking forward to its publication with some modifications to address the below issues:

1.	On Line 115 Page 3, should it be V(0, k) instead of V(0, n)?

2.	In Figure 1 on Page 4, the description for this figure is not clear. It is hard to understand (ii) and (iii) by the caption. It doesn’t even mention (iii) explicitly. What is the relation between H(0), G(0), H(1), G(1) the 3d matrix displayed in the figure? On the last line of the caption, it should be i = 0, 1 instead of i = 1, 2.

3.	On Line 178 Page 5, it is not clear what the predefined filter matrices look like. Only looking at Figure 2 and the contexts in the 2.3 Multiwavelet-based Model, it is very hard for the readers to build the same model on their side, hence it is hard to reproduce the experimental results in the paper. The authors should plot another graph for the neural networks so readers can understand how to build the neural network in practice. The author should show at least one detailed neural network for one of its PDE experiments for readers to reproduce the results.

4.	I don’t see the supplementary materials.

In summary, I recommend its publication in NeurIPS after those commends are addressed.

limitations_and_societal_impact: Authors should address the limitation of the filter matrices used in this paper. Should they apply to all PDE problems? Why does this approach result in 10x accuracy improvement for some PDEs while only gives about 1x accuracy improvement (no improvement) for the Navier-Stokes (NS) are 2d time-varying PDEs?
review_title: null

Meta-review: This paper investigates the problem of approximating the solution operator of a partial differential equation (PDE) and proposes a novel neural architecture based on the multi-wavelet transform to approximate the kernel of the solution operator. The key innovation of this paper is a clever application of the multi-wavelet transform that allows for a simpler solution. There is a general consensus among the expert reviewers that this paper is of high quality and offers a new solution to the important problem that would have a high impact on the PDE community. Some reviewers raise a criticism that the paper didn't provide the results on complex real-world scenarios but focus instead on the simple settings. I consider this weakness to be minor for the NeurIPS audience, especially after the authors have satisfactorily described potential directions to improve this work in their responses.

As a result, I recommend this paper for publication at NeurIPS2021.
Decision: Accept (Spotlight)