Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes learning an underlying PDE through the use of a physics-informed neural network and sparsity regularization on the partial derivatives. A neural network representing the PDE solution is trained simultaneously with a sparse regression on the partial derivatives of this neural network.

The main novelty seems to be combining the model discovery loss of Both et al. 2021 with a sparse prior, relaxing the objective and allowing gradient-based optimization. 

The paper also discusses physics-informed normalizing flows but I feel like it's a bit disconnected / orthogonal to the rest of the paper and not enough information is given; I did not understand the motivation, how it connects to a PDE solution, and how it's applied in the experiments.
main_review: Pros:
  - The task of learning a PDE is interesting.

Cons:
  - But the technical aspect ultimately becomes sparse regression.
  - There are many other differentiable alternatives that are simpler, or newer, that are not explored.

Comments: 
  - This is an arguably straightforward combination of prior works. As such, I'd liked to have seen more analysis on why the chosen approach (SBL) is good. For instance, there are many better sparsity-inducing priors than a Gaussian, such as the horseshoe prior, log-uniform, and other "spike-and-slab" style distributions. See for instance, [1, 2]. A comparison of such approaches would provide more insight. 

  - The baselines warrant more description, as I'm not sure why they fail. For instance, the main alternative method seems to be a greedy search over the mask (after Eq 3: "The mask is updated periodically by some sparse regression technique, and as terms are pruned, the constraint becomes stricter,..."). But more sophisticated search methods should fare better. 

  - Stronger baselines. For instance, before relaxing the mask, what about using REINFORCE gradient to learn binary masks? What about using the Gumbel-softmax or other types of relaxations first? (I imagine a sparsity-inducing approach should still fare better overall, but a comparison to alternative/simpler approaches should be warranted.) There are many other differentiable approaches to sparse regression that the paper does not mention. For a paper titled "Fully differentiable model discovery", I'd expect more exploration among these methods and ultimately a convincing conclusion on why the chosen approach seems the most worthwhile. (Is SBL with the chosen prior particularly suited for PINNs in any way?)

  - PINF: I'm quite lost. (i) Is there's a vector field u(t, x) that is in the PINF, for which the SBL loss is applied on? (ii) I didn't understand the motivation behind a PINF, is it because the Gaussian observation model in PINN may be insufficient? (iii) Perhaps an example would help; what is the underlying PDE for the random walk experiment?

  - Why optimize A and beta all the way, and not jointly with the PINN parameters?

Clarity issues:

  - The paper mentions that this approach can provide "a rigorous criterion for deciding whether a term is active or not". What is the criterion? And why is it "rigorous". Such approach should still requires a threshold for deciding whether a term is active if you ultimately want a deterministic value for the mask.

  - At the top of page 5, "our method does not require backpropagating through the solver" -> What solver is this referring to? Is this referring to optimizing A and beta? (By the way, since there's no closed form expression for A and beta, why not optimize for them at the same time as theta, which should be faster overall?)

  - How does the DeepMoD method differ from what is written in/after Eq 3?

  - When noise is added, is this being added to u only? What if you add noise to x or t? 

  - What exactly is 1% or 100% noise? What is this a percentage of? (i.e. does it correspond to some maximum standard deviation for some gaussian noise distribution?)

Minor:

  - Under Eq 18: what is little "n"? Should that be N?

  - Figure 2: what is OLS? ordinary least squares? Is this referring to DeepMoD?

  - Figure 2b: what is the dashed line?

  - Is it DeePyMod or DeepMoD? Both names seem to be used in the paper.

  - Maybe consider providing in-line labels (i.e. display u_x, u_xx, etc on top of the line), especially for Figure 5e. 

[1] "Handling Sparsity via the Horseshoe" Carvalho et al. (2009)

[2] "Bayesian Compression for Deep Learning" Louizos et al. (2017)
summary_of_the_review: This paper explores only one of many differentiable approaches to learning binary variables. I think a more comprehensive comparison against alternative differentiable approaches, from simple REINFORCE to horseshoe priors, would provide more insight and ultimately a convincing conclusion on why the chosen approach seems the most worthwhile. I do not understand the relevance of the PINF section; perhaps it could be its own standalone paper instead. There are also some clarity issues.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of identifying sets of equations that govern the dynamics of observed natural systems. Building off of work that uses sparse regression to identify a small set of active terms from a large dictionary of possible terms, this paper proposes a soft relaxation of a hard constraint used for selecting out possible terms in the optimization problem. The paper applies this technique to recover the underlying equations from data generated by a few canonical PDEs.
main_review: The paper provides a nice introduction and motivation to the problem of identifying simple governing dynamical equations from observed data.

It seems like there are two main contributions to the method:
Incorporating a soft relaxation of a hard constraint (the discrete mask) used in previous work.
Using sparse Bayesian learning to formulate the sparse regression problem in a fully Bayesian way.
The paper seems to focus on (a), and does not focus on the benefits of (b) in the text.

First, I have a really basic question: in eq (3), why not add a sparsity penalty on the coefficients of $\Theta$ and then get rid of the mask? Would this not also be a soft relaxation of the hard mask constraint? This also seems more in line with prior work such as SINDy. I’m sure there’s some reason why this isn’t feasible, but I couldn’t figure out why.

The text describing the first experimental results (4.2) suggest that two baseline methods were tried, but Figure 2 only compares against one baseline method? What am I missing here?

In addition, more details could be provided about certain choices in the experiments. How sensitive are the results to different hyperparameter choices (choice of the network architecture, number of library terms, regularization hyperparameters, etc).

Some suggestions regarding the figures:
- Add axes labels to all figures. For example, in Figure 2a or 3a, what are the axes? In Figure 2b and 2c or 3b and 3c, it would be helpful if each panel had a short descriptive title.
- It would be helpful to see the recovered equations and reconstruction for at least one of the example applications
- Consider plotting the relative error of each parameter fit, instead of (or in addition to) the actual values (e.g. Fig 3b and 3c).
summary_of_the_review: Overall, I had a bit of a hard time understanding just how significant the improvements were when using this method. Is the differentiable method more robust to hyperparameters? In principle, with careful tuning, it seems like the non-differentiable version should also be to recover the same equations, given that they both use the same model for approximating the dynamics. In addition to significance, the clarity of the paper could also be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: becauseThis work presents a model discovery algorithm to learn the PDEs from data. Their method uses Sparse Bayesian Learning to identify a minimal set of terms (along with their coefficients) from a set of predefined terms that constitute the PDE. Finally, they demonstrate that their algorithm can identify the PDE equations precisely for the following problems -  Korteweg-de Vries, Burgers, Kuramoto-Shivashinsky, and Normalizing flows.
main_review: **Strengths**

The approach presented in this work seems sample efficient (requires more experiments for empirical validation) and can identify equations from noisy datasets as compared to the baselines.

**Weaknesses**

* The authors need to compare their approach against PDE-NET 2.0 [1]. I'd be surprised if PDE-NET 2.0 doesn't perform competitively with the proposed method.

* The authors need to present the exact equations retrieved by all the 3 baselines for each experiment. Also, they need to show which coefficients the colored lines correspond to in the legend (Fig. 2(c), 2(d), 2(e), 3(e)) to improve the readability of the plots.

* I'm not sure if I understand the concerns raised by the authors regarding the regularisation constant in $l_1$ regularization in the following statement in the Introduction - *Applying $l_1$ regularisation can alleviate this problem, but raises the question of how strongly
to apply it*. $\lambda$, the strength of the regularization constant, is a hyperparameter that is varied uniformly. And the model/equation with the least MSE (or any other relevant metric) is picked.

* The authors need to elaborate on the training strategy they used for DeepMod. DeepMod follows a similar learning strategy as [1] (i.e. use L1 regularization to induce sparsity into the Neural Network) to identify concise equations. In [2] the authors have demonstrated that such a learning scheme is susceptible to random restarts. This is also consistent with the observation made by the authors in sec.4.2. For that reason, a grid search has to be performed for different values of $\lambda$ (strength of $l_1$ regularization) and $seeds$. Then the correct model/equation needs to be selected from these candidates using some metric, for instance in [2] the authors used the extrapolation error. Thus model selection is really important and if there doesn't exist a robust model selection strategy that can find the right equation from the pool of candidate equations then finding the right equation just once isn't at all useful. Thus, I'd like for the authors to shed some light on the training curriculum they followed for DeepMod. I'm assuming they didn't run DeepMod with a single set of hyperparameters and thus elaborate on the model selection strategy they used. 

* The authors have shown that their approach is able to identify the correct equations for a very limited number of cases like - Korteweg-de Vries, Burgers, Kuramoto-Shivashinsky, and Normalizing flows. They need to demonstrate that their algorithm can identify a variety of equations. Thus I strongly suggest that they generate PDEs with random terms and coefficients and report the number of cases for which the network succeeds or fails to identify the correct equation as they did in [2].

* (Optional) For the above experiment it'd really insightful to see how training data size affects the quality of the learned equation.

**References**

[1] Z. Long, Y. Lu, X. Ma, and B. Dong. PDE-Net: Learning PDEs from Data. ArXiv e-prints, 2017

[2] Sahoo S, Lampert C, Martius G. 2018 Learning equations for extrapolation and control. In Proc. of the 35th Int. Conf. on Machine Learning, Stockholm, Sweden, 10–15 July 2018 (eds J Dy, A Krause), vol. 80, pp. 4442–4450. Proceedings of Machine Learning Research

summary_of_the_review: The only reason I feel the paper is marginally below the acceptance threshold is that -
* The authors haven't compared their method against one of the most popular methods - PDE-NET 2.0
* Lack of extensive experiments (see above) to demonstrate that their approach can identify a variety of equations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes to combine sparse-Bayesian learning (SBL) with physics informed neural networks (PINNs) to achieve feature/basis selection in learning PDEs.
main_review: This work proposes to denoise noisy data with PINNs, and use SBL on (fixed basis expansions of) the denoised data to provide differentiable basis selection.  The broad idea of combining SBL-like procedures with deep models is natural and has been examined in the context of generative modeling (e.g. Dai and Wipf, 2019), but application to PDE learning appears new.  Numerical experiments seem to demonstrate some improvements, although I don't work on PDE learning so am not completely sure.

My main concern is about the optimization landscape.  For SBL to extract truly sparse coefficients we need its objective to go to infinity; but the PINN loss is a normal likelihood, and will be bounded by the noise level of the data, which is assumed to be finite in this setting (this is different from typical sparse learning scenarios where a low level of noise is assumed).  Therefore, optimizing the sum of the two objectives seems a bad idea.  In particular, one "global optima" would be for the PINN to estimate $\hat u \equiv 0$, incurring a finite reconstruction error, so that the SBL loss tends to negative infinity.  The authors seem to work around this issue by imposing proper hyperpriors, and pretrain the PINN so that the joint optimization starts at an informative initial value, these fixes seem somewhat hackish: proper hyperpriors will restrict the algorithm's ability to obtain truly sparse solutions in practice, or the behavior of the algorithm may depend on the value of the hyperprior parameters $a,b,c,d$ which are difficult to determine a priori.

Also, while I don't work on PINNs so cannot judge the originality, a quick search leads to this work which should be discussed:

Amuthan A. Ramabathiran and Prabhu Ramachandran^, "SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs", Journal of Computational Physics, Volume 445, pages 110600, 2021.

Minor questions/comments:

* In the first experiment, why does SBL without PINN fail in the noiseless setting? Is this due to numerical errors in computing higher-order derivatives?
* $\mathbf{u}_t$(=$\partial_t \hat{\mathbf{u}}$?) isn't defined in the text. The choices of notation might also be optimized.
* Formatting issues: better to use `\eqref` for equations, `Eq.~` as opposed to `Eq. `, etc.

References:

Dai and Wipf (2019), Diagnosing and Enhancing VAE Models, in ICLR.
summary_of_the_review: Pros:
+ The problem of differentiable model discovery on noisy data seems important, and the proposed method appears promising and easy to implement

Cons:
- Optimization landscape appears insensible

**Post-rebuttal update**: while the authors clarified on the experiments and related work, my core concern on the optimization landscape remains.  Therefore I am keeping my score unchanged.
review_title: null

Meta-review: The reviewers are in consensus that this manuscript falls just short of the bar. I recommend that the authors take their recommendations into consideration in revising their manuscript, with a particular focus on comparison to the state of the art.
Decision: Reject