Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a method to incorporate constraints into the output of learnable PDE models. They cover point-wise, differential, and integral constraints. They achieve this by representing the PDE solution in a basis, as is common for the variational method (e.g. pseudo-spectral method, and finite element method). The neural network outputs interpolation coefficients for each time step into the future. To implement the constraints, they have two methods: ‘soft constraints’, whereby a constraint-breaking penalty is applied to the neural network output and ‘hard constraints’, whereby the output of the network is projected on to constraint-satisfying solutions. This latter method is achieved by solving a convex programme.

The authors test on a variety of benchmarks, demonstrating that their method works; although, it is hard to parse the experimental section for whether these results are significant.
main_review: Advantages

- The submission is written well, is clear up until the experimental section.

- The experiments show that the addition of constraints help to learn a model that does not violate said constraints.

- The central method is sound and simple enough to be reimplemented. That said, I think some experimental details are missing and I could not reimplement the experiments. That said, since code will be released, this is not such a big issue.

Queries

- Equation 1: can dynamics F also explicitly depend on t? As written, it does not.

- Equation 7: I’m confused about the dimensionality of \alpha_j. In Equation 6, it is a p-dimensional quantity, because u(t, x) \in R^p. Is it the case that u(t, x) is now scalar-valued, since h: R \to R and \gamma_m: R \to R? This should be made explicit in the text.

- Equation 10: what is g?

- Section 4.1: I think it would be more helpful to the reader if the experimental setups from [37] and [10] were described more completely in the main text. It is difficult to gauge from what is written, what exactly is going on and whether your results are meaningful

- Section 4.2: Could you provide intuition for why the MLP would work better than the GNN? I would hazard a guess that the GNN does not have a large enough receptive field. Since you only use a single layer of message passing from what I see in the appendix, the receptive field of an output neuron is the 1-step neighborhood due the the Delaunay triangulation. By contrast, the MLP receives the full domain as its receptive field.

- There seems to be a lack of baselines in the experimental section after section 4.1. I am not entirely sure why this is. Given that it is mentioned in the related work that there is a rich literature on incorporating constraints, both hard and soft, into learnable PDE models, I would expect to have seen some in the experiments.

- What do the error bars represent in Figures 4 and 7?

Minor notes

- Equation 2: the ordering of x and t in du(x, t)/dt does not align with elsewhere in the submission, where you have used u(t, x).

- Equation 2: Please be more specific what the notation F_\theta(x_i, x_N(i), …) means. For instance what is x_N(i) or u_N(i)? My understanding is that you have replaced the differential operator F with a local parametrised operator F_\theta over neighbourhoods N(i). The notation and underlying assumptions here should be explained more precisely.
summary_of_the_review: I think the method is sound, and up until the experimental section the paper is well laid out and straight-forward to read. My understanding began to breakdown in the experimental section, where information was either referenced in the Appendix or completely in other papers. This made it hard to read and slowly down reviewing a lot. In most of the experiments, comparisons to other methods in the literature are missing and it is therefore hard to gauge the significance of this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces and studies several variants of introducing soft and hard constraints into learned models of PDEs on unstructured meshes.
main_review: In general, the paper is easy to read. Many details remain vague though, e.g., the implementation of the constraints in the experiments, how to train the differentiable hard constraint solver, how exactly are constraints split into gamma_n terms (and why, this seems unnecessary), what is a thin plate spline and why is this a good basis for evaluating constraints.
Section 3.1, 3.2 could have been framed better by noting that this is simply a finite element formulation of constraints, and differential/integral formulations directly correspond to FE differentials/integrals.

As the paper mentions in the introduction, using soft constraints (often on regular grids) in training is an extremely common practice in physics prediction papers. Often, this is not referred to as constraints but as "physics-based"/auxiliary losses (e.g. adding a loss on divergence when modeling incompressible flow), but it amount to the exact same thing.
This paper differentiates itself from these methods by a) using unstructured grids, and b) providing a formalism for solving hard constraints.

However, the paper doesn't make a particular strong case for these choices. First, the results are either on regular grids (4.1), or irregular meshing of a square domain, but with a very uniform edge lengths (4.2-4.3). None of these demonstrate any advantage of irregular meshing, and I expect all examples could have been much more easily solved on regular grids with standard CNN-based methods.
And second, hard constraints are presented as one of the contributions, and the description spans quite a bit of the method section. However as the authors state themselves, their method for solving hard constraints is only feasible for very small systems, and is hence not used for any of the paper's main results (only for a minor example in the appendix).
Similarly, the paper introduces mitigation techniques to make quadratic basis functions (PWQ) work, which turn out to perform strictly worse than a simple linear basis in all settings. 

And finally, I'm a bit unsure what to take away from the results. It's a bit of an obvious finding that putting an auxiliary loss on a certain quantity (e.g. divergence) will produce results with lower values for this quantity. It would have been much more interesting to show examples that actually highlight the contributions of the paper-- i.e. that isn't easily achieved with corresponding loss terms on regular grids. Also, this paper really needs comparisons to other methods, e.g. PDEs on regular grids with physics-based losses, or perhaps comparing constraints on energy to Hamiltonian methods, etc.

Other comments:
- On regular grid in 4.1, wouldn't the used PWL basis be exactly equivalent to FD?
- While there seems to be a bit less noise, I'm not sure that the constrained look 'more similar' to the data in fig 6, 9
- Why does the GNN in 4.2 not work? GNNs generally work very well for such PDEs. An MLP is likely a poor choice for learning local PDE dynamics, particularly when trained on small datasets
summary_of_the_review: While I like aspects of this paper (e.g. I think the FE formulation for constraints on unstructured grids can be useful), in its current form it's not ready for publication. Many of the more interesting contributions (e.g. higher-order basis functions, hard constraints, irregular grids) don't really work that well and aren't used in the main experiments: most results are on square, regular domains, with soft constraints and linear or radial basis functions, which is quite similar to what a lot of physics prediction papers are already doing with auxiliary losses (and for which there aren't any baseline comparisons). 
I'd encourage the authors to work further on the method (e.g. make hard constraints work for nontrivial systems), and choose good experiments to showcase the strengths of constraints on irregular grids. I also think constraints for adversarial methods that the paper touches on could be worth exploring further.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a two-folded method to enforce constraints of different natures (differential, integrals….) on a statistical model learned from physical data.
The constraints are not enforced directly on the model but rather on “interpolant” functions that aims at completing the original model in between the observed grid points.

main_review: The task addressed by the authors is crucial in machine learning since purely data driven approaches blatantly fail at conserving several physical quantities such as momentum or energy. In particular, the enforcement of algebraic constraints (whether differential or integral) on a learned model is a necessary path to learn from/for physical data.

The method can be summarized as follows: Learn the model $F_\theta$ that fits the data constraints (regression, gan etc. ...), hence the solution at “grid points” $\mathbf{u}$. Then fit a simple function of separate space and time $u_f (t,x) = \alpha_j(t) \phi_j(t)$. (linear, quadratic, etc..) to interpolate between the prediction on which can enforce specific constraint, then a fortiori constraining the learned $u$. The experiments seem to support the author’s choices. The experiments are well conducted and present various use cases useful for the whole community.
 
However, the main weakness of the paper is the clarity in the presented interpolation method. Mainly, how it is computed and its link to the learned solution $\mathbf{u}(t)$. Also the estimation of $\phi$ is only very lightly discussed.

Question:

1. How well can the interpolant $u_f$ approximate the original to the problem ? Indeed, I guess this is related to the convergence of a numerical scheme: if the space is well covered (i.e. discrete step size goes down to 0) can we expect to recover a good function $u_f$ using such a prior?
2. Why piecewise cubic approximations do not require extra conditions (do you use Hermite polynomials and continuity of the derivative) compared to piecewise quadratic ? I suggest the authors include a thorough discussion on this topic in the appendices (with the conditions imposed on such local polynomials). 
3. For the gan experiments, does the generator depend on time ? If  not, then the interpolation simply concerns the $\phi$ part ?
4. How to enforce a second order differential constraint when the interpolation is PWL ?

Remark on the presentation:
Despite interesting ideas, there is room for improving the presentation and the clarity of the paper.

a) Since the main focus of the experiments concerns “soft constraints”, I suggest the authors develop the section treating the soft-constraint components (Perhaps including an algorithm to show practical use) and how constraining $u_f$ constrains the learned $\mathbf{u}$ (or more specifically how the losses on $u_f$ impacts the $\theta$ of $F_\theta$), by detailing explicitly the link between the interpolation and the learned solution.


b) The key point of the paper lies in the interpolation scheme, however, in my opinion the paper lacks elements of details for an in depth understanding of the estimation method of the parameters $\alpha_j$ and $\phi$. Since this point is central in the paper, I highly recommend including thorough details on the estimation of the $\alpha$ and $\Phi$ at least in the appendices.

Minor:
 - I suggest for clarity to write the learned solution $u_\theta$ instead of $u$ to avoid the possible confusion between $u$ the solution to (1) and $\mathbf{u}$ the learned numerical solution at the nodes $x_1, .., x_N$). 
- Should $t$ be a variable of $F_\theta$ in eq.2. Does $u_1 = u(x_1)$. ?
- Eq.6. If there are $N$ points ($x_1$, …, $x_n$). Should the sum stop at $N-1$ ?
- P.6 “with a multiplayer perceptron”

summary_of_the_review: The paper proposes a very interesting method to enforce differential / Integral and differential constraints on an interpolation function derived from the learn operator that fits the data. 
The experiments are well conducted but the method’s presentation lacks clarity (see main review) notably on the estimation of the interpolation coefficients and their link to the learned solution function? 

If my concerns are addressed, I will be glad to increase my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to enforce physical constraints in deep learning models. It provides a nice summary of local, differential and integral constraints, and frames them in a Lagrangian setting. For a reason which I could not clearly follow, the paper focuses on GAN early on. This is intuitive to me, as the physical constraints could nicely stand on their own. 

main_review: The paper presents a series of tests which follow a somewhat unintuitive order. First, a regular grid case is presented, which seems to be taken from previous work. Not much detail is given, and the test only confirms that the constraints give very similar results. This test seems like a pure debugging case, as it is based on Cartesian grids instead of the unstructured grids which the paper wants to focus on.

The subsequent sections (4.1 ff) then provide examples with unstructured grids. While the first CH-case is very simplistic, the latter cases contain interesting setups and show some interesting results. E.g., I found it interesting to see how the physical constraints affect the smoothness of the zero divergence fields 

In terms of writing, I found the order and argumentation of the results sub-optimal, as mentioned above. The structured grid case seems mostly out of place. The introduction also sounds strange to me, with current work being "limited" to uniform grids. Previous works have admittedly focused on these, but the methods are still applicable - after all, this submission is applying the methods from uniform grids in a fairly straight forward manner. I would recommend to rephrase this. Also, stylistically, citations shouldn't be used as nouns, and the related work seems to brief to me (e.g., generative models for physical problems are missing).

summary_of_the_review: Overall, I think this submission would benefit from further revisions. It is definintely an interesting direction, but in its current state the paper applies existing methods from regular grids to Lagrangian discretizations. This is a good idea, but not a very fundamental step forward. In combination, the results do not contain many surprises, and the focus on GANs in a way detracts from the main goals of the paper. The presentation of the work could also be further improved. So overall, I'm leaning towards the negative side given the current state of the submission.
review_title: null

Meta-review: The paper introduces a framework for enforcing constraints into deep NNs used for modeling spatio-temporal dynamics characterizing physical systems. The authors consider different types of constraints (pointwise, differential and integral). They start from a formulation approximating PDEs as set of ODEs (method of lines). Their main idea is to approximate the solution of the equations using an interpolant between observations and imposing the constraints on this approximation function. The interpolant is built using basis functions located at observation points. The formalism considers irregular spatial grids and both soft and hard constraints. The main claim is then the introduction of a general formalism for considering different types of constraints on irregular grids. Experiments illustrate the behavior of the proposed method on different types of evolution equations and constraints.

The reviewers agree that the proposed approach is interesting and that some of the ideas are original. However, they also consider that the paper is not convincing enough to demonstrate the interest and novelty of the approach, compared to alternative methods. The experimental section mainly considers (except for one application) regular grids and constraints that could be handled by other methods as well. The authors should present cases where their method provides a clear advantage, distinct from existing solutions. The authors provided a well-argued rebuttal, clarifying several points. However, all reviewers retained their original scores and encourage the authors to further develop the experimental analysis to present a stronger paper. In addition, the presentation could be improved, and some technical aspects better explained (e.g., description of interpolation methods, and some advice on which interpolant to choose for a given problem).
Decision: Reject