Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for learning solutions to PDE that are guaranteed to satisfy the constraints. This is done in eq. (5) by learning a basis for the solutions and finding the best linear combination of them. This results in the overall learning objective in eq. (8) that learns the basis that best-minimizes the PDE residual on the training set. The method is evaluated on 1d convection problems (Section 4.1) and 2D Darcy Flow problems (Section 4.2). 
strength_and_weaknesses: Strengths
+ The standard PDE learning methods have no guarantees on how well the predicted solutions match the PDE constraints and having a model class that always satisfies them seems useful.
+ The experimental results on the convection and Darcy Flow problems clearly show the residual losses and relative errors improve on the problems they care about.

Weaknesses
+ The biggest weakness is that the problem sizes are relatively lower that other ones considered in the literature, for example in [PDEBench](https://arxiv.org/pdf/2210.07182.pdf), and the comparisons to the soft-constrained PDE baseline do not appear to be established results. The paper would be stronger if it evaluated on the exact experimental setting from prior work on learning PDE solutions 
+ The inference procedure in eq. (5) seems computationally expensive to solve for every instance
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written.
summary_of_the_review: I recommend to accept the paper as it's a reasonable idea and evaluation that will be influential to help the PDE solving community better-enforce constraints in the predictions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The reviewed work proposes a novel approach for representing solution operators of linear partial differentials as neural networks without the need for training data provided by solution pairs. Given an input function (right-hand side, initial conditions, boundary conditions, etc.) the proposed work uses a Fourier neural operator (FNO) to learn basis functions. Then, a classical linear solver is used to compute a linear combination of these basis functions that best solves the PDE. At training time, this linear system solve is treated as an "implicit layer," and gradients are propagated through it. Thus, the FNO essentially solves a dictionary learning problem where it learns to automatically generate the best basis functions in which to express the solution. 

This approach is compared to a physics-informed deep-O-net, showing minor improvements in accuracy vs. training time. 
strength_and_weaknesses: ### Strengths
Using neural networks for dictionary learning of optimal basis functions is an interesting idea that deserves further investigation. To the best of my knowledge, this idea is novel and given how well-understood the solution of linear systems is, it seems like a good idea to leverage these tools more when using neural networks to solve PDEs.

### Weaknesses
In my opinion, the paper has three main weaknesses. 

The first is the restriction of the proposed approach to linear PDEs. This loses the generality of other neural-network-based approaches, which is arguably one of their greatest strength. 

The second is the requirement to perform a linear solve at test time, which potentially disposes with the cheap online cost of existing physics-informed methods. (It would be helpful if the authors could provide a comparison of the inference cost).

The third weakness is the overall minor improvement over the baseline, especially when accounting for the cost of the implicit layer. If I understand the results in figure D.1 correctly, they show also show a minimal improvement of the learned basis functions over interpolation.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is of reasonable clarity and quality. I am not aware of prior occurrences of the proposed dictionary learning approach combined with implicit layers.
I applaud the authors for including the results of figures A.1, B.1, D.1 in the appendix, but it seems that these are important for assessing the performance of the proposed method. I would therefore suggest including them in the paper.
I was also not able to find the values of $N$ (number of basis functions) used in the experiment. Would the authors mind providing them?  Finally, it seems that comparing the learned basis functions to a simple classical discretization (finite element or similar) would be appropriate to investigate the usefulness of the learned basis functions. 
summary_of_the_review: At present, I believe that the limitations of the reviewed work compared to standard physics-informed FNO/DeepONet, notably the possibly increased inference time and the limitation to linear problems, are excessive given the relatively modest performance gains. Therefore, I recommend the rejection of the paper in its present form. 
However, given how difficult it is to train physics-informed neural operators without training data, an improved version of the manuscript that addresses these issues could merit acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider a setting where the goal is to solve the parameterized partial PDE $\mathcal{F}_\phi(u) = \mathbf{0}$, where $\mathcal{F}_\phi$ is an affine differential operator, the PDE parameter $\phi$ is a function over some domain $\mathcal{X}$, and the solution $u$ is also a function over some domain $\mathcal{X}$. The aim is to train a neural network-based model $u_\theta$ that approximates the mapping from parameters $\phi$ to solutions $u(\phi)$. In this paper, the authors introduce a PDE-constrained layer (implicit layer) within the neural network training process to enforce the PDE constraints over a given set of points in $\mathcal{X}$, and thereby promote constraint satisfaction. Specifically, they let $u_\theta = \sum \omega_i f_\theta^i$, where $f_\theta$ is a neural network and $\omega$ is a weight vector provided/solved for by the implicit layer, and where $f_\theta$ is trained end-to-end with the implicit layer. They show that this approach yields outputs that are close to the true solutions, for the 1D convection and 2D Darcy Flow settings.
strength_and_weaknesses: The overall concept of the paper is interesting: Use a neural network to represent a basis, then explicitly solve for basis weights that attempt to satisfy the PDE constraints over a set of (train or test) points - and during training, train this system end-to-end. Overall, the paper is well-written, the idea is clear, and the experimental validation is illustrative.

The main weakness is that the experimental demonstrations are somewhat incomplete. Notably:
* I was surprised not to see a wall clock time comparison to traditional solvers, as the lack of this comparison makes it difficult to assess whether this particular solution is actually needed / whether it improves upon the non-NN standard for PDEs characterized by affine or linear differential operators. 
* The choice of training points and the choice of testing points, as well as the choice of $N$, presumably have a large effect on performance - however, this is not really discussed or shown.
* The experimental settings seem rather small in scale, and I would have liked to see larger-scale experiments.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and the idea is clever (building closely on prior works in approximating optimization problems with hard constraints, and bringing this idea to the PDE realm). The quality of and need for the work is a little bit harder to assess due to the weaknesses described above.
summary_of_the_review: This paper provides a technically interesting concept, but the need for this concept is unclear without explicit wall clock time comparisons to traditional non-NN baselines (that is, does the new method beat those standard baselines?). A more thorough assessment of training choices (e.g. training/testing data-points and the choice of $N$), as well as (potentially) larger-scale experiments, are also necessary to assess the efficacy of the approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a neural network approach to solve PDEs. The method relies on two key points. The first is outputting coefficients for an overcomplete basis of functions which parameterise the solution. The second is a PDE-constrained layer (e.g. solving a linear system) for incorporating the PDE constraints. The authors explain how to differentiate through the PDE-constrained layer using IFT. Their approach is demonstrated on some simple 1D and 2D problems. Notably, while their method can make use of solution data of the PDE, it doesn’t explicitly require it. Additionally, their approach provides guarantees about satisfying the constraints, rather than just penalising deviations from the constraints.
strength_and_weaknesses: The paper clearly explains the related work and how their approach differs. The paper clearly defines their model setup, and the loss that they train on. The authors explain why using Implicit Differentiation through the PDE-CL layer is crucial, and explain how it is implemented. Tradeoffs of the method are discussed. Special care is taken to explain how the loss function is designed with respect to which residual to penalise to encourage generalisation. Details of the experiments are provided to make it reproducible.

Care is taken to show how their method generalises, and how it improves upon the baseline comparison.

I don’t have any major concerns about the work, but this is partially due to my being not very familiar with the PDE + ML literature. 

Question: Are there ways to bound the relative error of the solution using the PDE residual loss in your method? For example, is it possible for your method to have very low PDE residual loss, but for the relative solution error to not decrease?
clarity,_quality,_novelty_and_reproducibility: The work is clear, high-quality, and novel. Based on reading the appendix, sufficient details are provided to reproduce the experiment.
summary_of_the_review: The work is clear, high-quality, and novel. Based on reading the appendix, sufficient details are provided to reproduce the experiment.

The paper clearly explains the related work and how their approach differs. The paper clearly defines their model setup, and the loss that they train on. The authors explain why using Implicit Differentiation through the PDE-CL layer is crucial, and explain how it is implemented. Tradeoffs of the method are discussed. Special care is taken to explain how the loss function is designed with respect to which residual to penalise to encourage generalisation. Details of the experiments are provided to make it reproducible. Care is taken to show how their method generalises, and how it improves upon the baseline comparison.

I don’t have any major concerns about the work, but this is partially due to my being not very familiar with the PDE + ML literature. This is why I put a lower confidence on my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
In this paper, the authors considered to solve the PDE satisfying the constraints by learning. The authors proposed a PDE-CL layer which linearly combines the basis to ensure the constraints, and exploit an end-to-end way to learn the basis. In the inference stage, the basis is fixed and the PDE-CL layer will output adaptive weights for different PDEs. The method demonstrates promising performances on 1D convection and 2D Darcy Flow problems. 

In sum, most of the reviewers recognize the novelty and signficance of the proposed method. 

The major concerns raised by the reviewers lie in the training/inference complexity and the nonlinear PDE extension. After rebuttal, the authors addressed these concerns. 

justification_for_why_not_higher_score: 
Although the PDE-CL layer is novel in PDE literature, the implicit layer technique has actually been used  in ML community. 
justification_for_why_not_lower_score: 
The method is novel and the empirical results are promising. 

Although there are several concerns about the complexity and nonlinear PDE generality, the authors successfully addressed them.
Decision: Accept: poster