Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes using the multiwavelets framework to solve coupled partial differential equations in a data-driven manner. Multiwavelets are a generalization of standard scalar wavelets but instead use more than one scaling function and the multiwavelet transform is not a straightforward separable extension of the wavelet transform but contains interaction coefficients due to this multivariable choice. 

At the other hand, in scenarios of coupled partial differential equations in which more than one differential equation needs to be solved with interacting terms, the greens function kernels are not independent and hence not amenable to be solved using well-known neural operators like PINNs, Fourier Neural Operators (FNO), etc in a straightforward manner. 

The authors combine the two scenarios and propose the coupled multiwavelet neural operator (CMWNO). Experiments are demonstrated for 2 problems - Gray Scott Equations and the Mean Field Game Problem. The results show good improvements over baseline neural network approaches by a significant margin. 

strength_and_weaknesses: Strengths 

- The high-level motivation combining multiwavelets and coupled PDEs is very interesting and original (to the best of my knowledge) 
- The results are significant when compared to standard learning-based baseline methods 


Weaknesses and Questions

- My main concern is the assumptions leading to equations 4 and 5. I am not fully on board with why the multiwavelet space automatically guarantees decoupling? I presume this assumption is crucial since the construction of the architecture for CMWNO is hinged on such a separation and solved in an iterative fashion (i.e. fix the coupling dependence and solve for the kernel and vice versa). Despite this, what is the need for multiwavelets at this step? and not simple wavelets or the Fourier transform?  (which can also lead to an interesting baseline) 
- Even though the improvement over shown baselines is good, I am not sure if it is a super convincing comparison since most of these approaches do not seem to be designed for coupled PDE’s. I do feel that the concatenation approach to make these methods work for the coupled system is a fair attempt, but I am not too surprised they don’t work well. 
- The paper focuses on coupling with only two variables. How can the proposed framework be extended for a more sophisticated coupling PDE system with many variables? especially the dice strategy? 
- Broadly, it appears that the positioning of this paper appeals more to a core PDE audience (giving them a new tool with neural networks for complex PDEs like Gray Scott equations, etc) rather than a representation learning conference. However, this paper is not entirely out of place for ICLR, but I feel that the background and motivation for coupled PDEs and Multiwavelets can be made more accessible. For example, a visual schematic, or some visualizations depicting the use cases for multiwavelets and coupled PDEs would go a long way to make everything convincingly self-contained. 
clarity,_quality,_novelty_and_reproducibility: Ok for clarity and reproducibility. Good Quality and Originality. 
summary_of_the_review: Overall, I think this paper is quite decent in its methodology and evaluation, and contains an interesting combination of multiwavelets and coupled PDEs. Despite that, I find some ambiguity in the proposed framework, especially regarding the architecture design and assumptions leading to Eq 4 and 5. I’ll wait for the rebuttal to make a more informed opinion, but as of now, it is a borderline rating from me.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a modification of multiwavelet architecture introduced in [1] for the case of coupled PDEs. Comparisons with the Fourier Neural Operator [2], two versions of the original multiwavelet operator [1], the Pade exponential model [3], and Physics-informed neural networks [4] show that the proposed modification improves relative L2 error.


[1] - Gupta G, Xiao X, Bogdan P. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems. 2021 Dec 6;34:24048-62.

[2] - Li Z, Kovachki N, Azizzadenesheli K, Liu B, Bhattacharya K, Stuart A, Anandkumar A. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895. 2020 Oct 18.

[3] - Gupta G, Xiao X, Balan R, Bogdan P. Non-linear operator approximations for initial value problems. In: International Conference on Learning Representations 2021 Sep 29.

[4] - Raissi M, Perdikaris P, Karniadakis GE. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics. 2019 Feb 1;378:686-707.
strength_and_weaknesses: Strength

1. Main objective (to solve coupled PDEs in the regression setting) is clearly formulated.
2. Most of the main claims are reasonably substantiated (for exceptions see below).
3. Most ideas are clearly explained (for exceptions see below).

Weaknesses

1. Architecture is not described in detail.

    In Section 3 (page 6) authors describe neural networks A, B, C, and T used in multiwavelet neural operators as follows
    ``In our proposed model, for each operator, the neural networks A, B, and C use single-layered convolutional neural networks while T uses a single linear layer.``
    
    This description is incomplete. What are the sizes of convolutions, the strides the number of input and output channels (features)? Are there any nonlinear activations and what are those? Are there multiple multiwavelet kernels (cells) involved and how many of them?

2. The proposed model is a minor modification of [1].

    From the description in the main text and the parameter count in Table 6 (supplementary materials), it looks like the proposed architecture is exactly the same as in [1]. The contribution then is a way how the known architecture can be applied to coupled equations.

3. Awkward notation

    In many places, authors use backslash / to refer the reader to distinct cases. This does not look appropriate when mathematical expressions are involved since / is also used to denote division, quotient group, and other mathematical operations.  For a concrete example see page 5, text right before equation 9.

4. The first claimed contribution is unjustified.

    ``For coupled differential equations, we propose a coupled neural operator learning scheme, named CMWNO. To the best of our knowledge, CMWNO is the first work using pure data-driven method to solve coupled differential equations.``
    
    It is not clear what the authors mean. Any work that solves the Navier-Stokes equation using machine learning deals with coupled PDEs since velocities are coupled. See [2] for a concrete example. Besides that, there are works on multiphase flows [3], weather forecasts [4], e.t.c. Essentially, there are hundreds if not thousands of papers dealing with coupled PDEs in regression and physics-informed setups.

5. The choice of a physics-informed neural network (PINN) as a benchmark is questionable.

    PINN provides a way to solve PDE given initial data. As the present reviewer understands vanilla PINN takes only coordinates for input and produces the solution. That is, PINN is not a neural operator in contrast with all other considered architectures. How PINN is applied precisely in this context? Is it retrained for each new input function? In the opinion of the present reviewer, PINN adds little to the understanding of how the accuracy of the presented model compares to the state-of-the-art.

[1] - Gupta G, Xiao X, Bogdan P. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems. 2021 Dec 6;34:24048-62.

[2] - Stachenfeld K, Fielding DB, Kochkov D, Cranmer M, Pfaff T, Godwin J, Cui C, Ho S, Battaglia P, Sanchez-Gonzalez A. Learned Coarse Models for Efficient Turbulence Simulation. arXiv preprint arXiv:2112.15275. 2021 Dec 31.

[3] - Wen G, Li Z, Azizzadenesheli K, Anandkumar A, Benson SM. U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources. 2022 May 1;163:104180.

[4] - Pathak J, Subramanian S, Harrington P, Raja S, Chattopadhyay A, Mardani M, Kurth T, Hall D, Li Z, Azizzadenesheli K, Hassanzadeh P. Fourcastnet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv preprint arXiv:2202.11214. 2022 Feb 22.
clarity,_quality,_novelty_and_reproducibility: 1. Clarity

    The presentation is clear in most parts. The exception is the description of architectures that the present reviewer described in the previous section.

2. Quality

    Overall, the quality of the text, organization, results and mathematical details, illustrations are reasonably good.

3. Novelty

    The novelty is questionable because, as we mentioned, authors apply architecture from [1] with no significant conceptual changes. However, the way multiwavelet coefficients are mixed is a novel contribution.

4. Reproducibility

    The authors provided a jupyter notebook with the code, but the present reviewer was unable to run it after downloading because some parts of the library developed by the authors were not included in the zip archive. At the current stage, the work is not reproducible, but the present reviewer believes that reproducibility is not going to be a significant issue after the code is supplied. Besides that, the code provided in [1] can be sufficient for reproducibility after appropriate modifications.

[1] - Gupta G, Xiao X, Bogdan P. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems. 2021 Dec 6;34:24048-62.
summary_of_the_review: Overall, the paper is well-written and organized and mostly well-explained. Provided experiments are of good quality and models chosen for benchmarks include state-of-the-art architectures. The approach given by the authors is empirically justified and illustrated with (vague) theoretical considerations.

The main concern of the present reviewer is that the paper does not contain significantly novel techniques. It only proposes a minor correction of architecture from [1].


[1] - Gupta G, Xiao X, Bogdan P. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems. 2021 Dec 6;34:24048-62.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript presents a neural operator approach for solving coupled PDEs, motivated by multiwavelets. The authors demonstrate some improved accuracy on Gray-Scott and nonlocal mean field game problems. The multiwavelet transform is used to represent the coupled information from the coupled PDEs and a dice strategy is employed to mimic the interaction of the information during the training process. 
strength_and_weaknesses: I am not aware of any existing neural operator approaches for coupled PDEs, and the use of the multiwavelet transform is very reasonable in this context. The dice strategy is a nice observation to deal with the imbalance update problem that could occur for each kernel.  The authors compare their approach with many other reasonable neural operators, and some improved accuracy is observed. 

There's an assumption that one can write down decoupled solution operators that solve for each component of the coupled PDEs in (5). I find it hard to assess if this is a mild assumption or not.
clarity,_quality,_novelty_and_reproducibility: The manuscript is well-written and the experiments are reproducible. I believe that the neural operator architecture as well as the dice strategy for training neural operators are two novel contributions.  
summary_of_the_review: The authors are proposing a strategy for solving coupled PDEs that sounds reasonable, and uses a multiwavelet discretization of the spatial domain. There are some improvements in accuracy for the Gray-Scott and nonlocal mean field game problems. Given the assumption in  (5), I find it difficult to assess how well this approach will work as a general approach for coupled PDEs.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a way to solve coupled PDEs using multiwavelet bases and also introduces a "dice strategy".
The authors make their best to explain difficult concepts and ideas, as well as the intuition behind their approach.

Strengths: Good presentation, impressive results on several benchmarks. 
Weaknesses: The overall architecture is not very straightforward and simple, and almost leans towards a specific numerical solver (i.e. one can directly solve the problem in the multiwavelet space, rather then learning something). Reimplementing the ideas and using by others could be really difficult. 
justification_for_why_not_higher_score: The paper is a solid work, but I don't think it brings some significant novelty to the table.
justification_for_why_not_lower_score: All reviewers agree in evaluation, so would I based on my own reading.
Decision: Accept: poster