Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper is clearly written, technically sound and innovative.

### Impact:
The paper is bringing an important contribution in the domain of Physics-Informed Machine Learning. It proposes an interesting and technically sounds (and not obvious) way to combine multiple successful components of recent literature in the field.

### Clarity and technical soundness:
The paper is clear, technically sound and manipulates tools in an advanced way.
It would be useful to provide more intuition and high-level interpretation on the most technical aspects of Deep Kernel inference and on the usage of ELBO methods. That would go at the benefit of understanding for the non-specialists of Kernel-based learning who are interested in Physics Informed ML.

### Results:
The results are extensive, are exploiting both simulation-type data and real-world data.
The benchmark used for comparison are relevant.

### Applicability:
It would be interesting to have a more high-level interpretation and analysis of the applicability of the method to 3D simulation data. 
In 4.2, it was not clear to me what N and M stand for. Could you clarify it and provide a more high-level analysis with it too?
review_title: Positive review for Physics Informed Deep Kernel Learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes a deep Gaussian Process (GP) framework for data modeling informed by dynamical systems. 
The analogous to the one of gradient matching, where a GP regression problem is penalised by constraints taking the form of differential equations acting on the GP itself. This study merges this framework to the scalable deep-GP framework of Wilson et al, and proposes an approximated inference setting in which the problem can be optimised through stochastic variational inference. 
The proposed method is tested in several experimental scenarios and compared to standard kernel-based methods, and to the Latent Force Model of Alvarez and colleagues. The results show promising performances in terms of prediction and stability of extrapolation.

The proposed methodology contributes to the domain of gradient-matching and extends the classical approaches to allow for scalability in deep models. In this sense, the comparison with respect to the state of the art appears weak. The comparisons proposed in the study are with respect to either shallow models, or deep-models not allowing the integration of physics-informed contraints. Recent works (e.g. [1,2,3,4]) could provide a fairer benchmark for the proposed approach. In this sense, the feeling is that proposed methodology largely overlaps with these more recent studies, for example for what concerns the use of deep models and variational inference schemes. In particular, the idea of soft-regularisation of deep-GP models has been already explored in [2], and the authors may want to compare the method with respect to this approach. 

It is not clear whether the parameters of the dynamical systems in the term h(Z,\epsilon) of (12) are inferred. If this is the case, the ability of the framework in estimating the systems’ should be assessed and compared to other moment matching approaches. This aspect is overlooked in the paper, although it is quite relevant for interpretability purposes. 
review_title: The framework is sound, but novelty seems questionable

final_score: rating: 5: Marginally below acceptance threshold
review_content: Post-discussion update: The authors have clarified their work considerably, and I believe the work is probably correct. However, the paper still suffers from poor presentation and poorly-motivated or justified modelling choices. The current version of the paper has not been updated, and all below issues thus stand. The paper overall presents a good idea, but I believe the authors made poor modelling choices, which led the kludgy math. 

----

The paper proposes to combine deep kernel learning (DKL) with PDE/ODE prior knowledge for learning spatiotemporal systems. The idea is sound and sensible, although it represents an incremental combination of two existing techniques. The differential assumption clearly improves in spatiotemporal experiments, which also motivates the idea. 

The paper has unfortunately three major mathematical errors. 

First, the very first equation of the method (eq 4) is already incorrect: g(x) is not a function of input “x” but instead it’s a function of solution “f”. The paper itself acknowledges this by giving series of examples which all are functions of “f”. The following GP prior for g(x) is then placed on incorrect inputs and seems misguided or at least insufficient to model the differential. Intuitively this is also easy to see: the differential of the solution is obviously not just a function of space and time (but of solution as well). 

Second, the eq 6 states that a non-linear differential of a GP is some other Gaussian process. This is incorrect: non-linear transforms of a Gaussian process does not retain Gaussianity. Since this is a central definition of the paper, the whole method is likely incorrect.

Third, the paper introduces a bizarre concept of constant zero vectors as random variables, ie. p(0|g). This is clearly wrong, and the probabilistic model is then wrong as well. While this could be easily fixed, the model is an example of GP-matching (which has known pathologies), and this has been extensively studied by many authors (eg. Wenk’19, Wenk’18, Gorbach’17, MacDonalds’15).

Given these obvious errors, the paper needs a major revision.


Technical comments:
o Eq 4 should be g(f,x), since later the \varphi is a function of f and its x-derivatives. Currently none of the examples (eg. burger) follow eq 4. Also eq6 is also wrong with same argument
o Please define (mathematically) the tackled problem and the problem domain. The paper starts by discussing classification, but at sec 3 the context suddenly changes into PDEs without the reader being informed about this.
o The paragraph “to incorporate..” is difficult to follow since its technical but does not open up the math yet. It would help to write this in more conceptual way
o It would greatly help the reader understand the method to include the sup-fig1 in the main paper
o It seems that both f and g are assumed to be separate GPs. This is unclear from text, please clarify
o eq7 is strange, since it defines a dirac between a random variable g and a sample of h. This is nonsensical.
o The 0-sampling is obviously wrong and unnecessary. Double priors are perfectly fine in Bayesian modelling, and eq9 could be defined without the 0-stuff. The double priors are called product of expert priors, see eg. MacDonalds'15 or Wenk’19. They do have their own pathologies, but these papers discuss them extensively.  
 
review_title: review

final_score: rating: 7: Good paper, accept
review_content: ### Summary of my understanding

The authors address the problem of function estimation given noisy observations of the function's values. The estimation is done in a non-parametric manner as posterior inference of Gaussian processes. The authors' proposal is to incorporate physics knowledge into this process. The knowledge here refers to a differential equation on the function to be estimated ($f$), and it may include unknown parameters and an unknown external force term $g$. They derive a joint distribution of 1) observation of $f$, i.e., $y$, 2) "virtual" observations $0$ of $g$, 3) latent variable $Z$ for latent force $g$, 4) randomness between $f$ and $g$, and 5) values of $g$. Afterward, they propose to perform a collapsed inference based on a lower bound of the log marginal likelihood (of observations $y$ and virtual observations $0$).

### Evaluation

The problem setting is clear. The proposed method is technically sound, and the experiments are enough convincing to show its superiority to baselines. Regardless of the common points with the latent force models [Alvarez+ 09] in terms of the problem setting, this paper proposes a somewhat new technique to perform inference under the presence of differential equation-based constraints on a target function. I think this paper is a sound work.

### Note

I think a concern lies in the paper's clear violation of the formatting instruction of ICLR 2021. Especially, the too narrow margins before and after headings seem to compress the main text, which might seem quite unfair. I delegate judgment on this regard to the chairs and for now, I decided the rating ignoring this matter.

review_title: Sound work

Meta-review: The paper presents a framework for incorporating physics knowledge (through, potentially incomplete, differential equations) into the deep kernel learning approach of Wilson et al. The reviewers found the paper addresses an important problem and presents good results.  However, one of the main issues raised by R1 is that, although the proposed method can be applied to broader settings such as that of incomplete differential equations, there are still regimes where the comparison is not only possible but perhaps insightful. An example baseline is the work of Lorenzi and Filippone, “Constraining the Dynamics of Deep Probabilistic Models” (ICML, 2018). Another critical issue, raised by R4, is the insufficient clarity in the presentation. Many of the concerns raised by this reviewer were clarified in the discussion and I thank the authors for their engagement. However, the AC believes some of the points raised by R4 in this regard were left unaddressed in the paper and the manuscript does indeed require at least one more iteration.

The format violation concerns raised during the reviewing process did not affect the decision on this paper, as the PCs confirmed that they did not meet the bar for desk rejection and recommended to assess the paper on its technical merits.
Decision: Reject