Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Paper proposes a methodology to combine ODE with latent force models to efficiently solve ODEs and ground them with observed data. 
main_review: Authors utilize the recent developments in probabilistic numerics to solve  ODE with a latent force in single forward pass. Authors show the usefulness of their methodology on a non-parametric SIRD model applied to COVID19 spread in Germany. 

In general experiments on simulated and real data are done properly and show usefulness of the approach.
The ability to solve ODE model in a single pass is admittedly very useful and something that can make the technique adoptable. However, I would like to see comparisons being done on more sophisticated models and being compared against a MCMC approach in much more detail. Right now authors do a comparison as something of an after thought. Also, all novelty is useful only if first the runtimes for the proposed approach is really short than say NUTS. Secondly, the estimates from the approach are comparable to NUTS or in case of simulated data near to real data. 

Finally, it seems everything authors have proposed here is already done in previous works they have referenced apart from specifically applying it to SIRD models. It might be case of misunderstanding on my side as most of the technical details have been taken away from paper and gives me impression that they have been done before. If not, I would at least like to see in SI a full blown description that can justify or outline the exact methodological contribution by authors.  

Also, I would like authors to claim less novelty about modeling time-varying contact rate. It is something standard in literature. In case of COVID  I would encourage authors to see work done by Walker, Patrick G. T., et al. "The impact of COVID-19 and strategies for mitigation and suppression in low- and middle-income countries." Science, vol. 369, no. 6502, 24 July 2020, pp. 413-22, doi:10.1126/science.abc0035. The supplement has the model description Which was first published as a pre-print in March 2020. The work along this line is used as a standard model for COVID for quite long. This is essentially a problem with our field that we don't look at work done outside the echo chamber. Even the framework byWalker, Patrick G. T., et al. is opensourced here https://github.com/mrc-ide/squire. Even a more sophisticated version than the author's and Walker, Patrick G. T., et al. is used here https://www.medrxiv.org/content/10.1101/2021.01.11.21249564v1. Which is now a Science Translational Medicine article here DOI: 10.1126/scitranslmed.abg4262. All this models have a time varying contact rate in them. 

I have updated my score based on replies from authors and reading comments from other reviewers. My concerns have been mostly meet. So I am updating the score. 
limitations_and_societal_impact: Authors have not even included them. I would first like them to even attempt it before giving any suggestion. They say we do have a model but we don't use it for policy making. This is weird the whole point about this section is authors thinking a bit more constructively about their work rather than passing the buck that it is not our concern. I would encourage authors to at least acknowledge the limitations and assumptions and then say output derived from their work might only be valid under certain specific conditions and any work deriving policy from these estimates should keep them in mind.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper discusses how to combine data with known differential equations of an observed system, to infer the parameters of these differential equations. To this end, the authors combine the state-space framework with ODE solvers and provide an approximate inference procedure based on extended Kalman filters.
main_review: Though the paper doesn’t provide new techniques itself, the combination of recent developments in probabilistic ODE solvers, and combining them with a state-space framework is interesting for inference of dynamical systems. The paper is mostly well written. Integrating knowledge of the underlying dynamic system into state spaced models, and combining it with observations, is important in many applications. The paper lines out nicely the methodology, that achieves this.
The main critique I have, is that the paper is written very high level, and from the main text it was quite difficult to follow what e.g. is happening in Algorithm 1. I think a bit more detail on the Filter equations couldn’t harm here in Sec. 3.4. Also in the experiments some details, e.g. explicitly writing what is $X(t)$, $U(t)$ and $Y$ for the different cases would help readability in my opinion. Then another important point is that there is no baseline in the experiments. I guess it would be simple to at least show the improvement over a normal state space model, which just uses observations. (Much like it is done schematically in Fig. 1, but providing numbers). Otherwise, I think it is a good paper, and with some changes, it could be even better.

limitations_and_societal_impact: + It would be nice to provide some details on the Gauss-Markov priors in the appendix.
+ I miss a discussion, how parameters could be estimated in this framework. For example, if some parameters on the system are unknown (but constant), how would you estimate them. I guess, one would think about some EM procedure, like done in standard state-space models. Also, it would help to say what emission parameters H and R (from Eq. 1) are in the experiments. I couldn’t find any details on them in the text, neither, what are observations. 
+ I noticed, that the values for the diffusion parameter $\sigma_X^2$ were chosen quite high. How were these parameters selected? Is there a heuristic that can be followed?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper "A Probabilistic State Space Model for Joint Inference from Differential Equations and Data" describes a method of how to estimate latent functions in ordinary differential equations using measuremt data on observed parts of the model. This is achieved by combining a (single!) run through an ODE solver with a suitable probabilistic framework (Bayes filter).
main_review: The research question is highly relevant, not only due to COVID-19, the methods are suitable for this problem, and the results are good. The paper is written very clearly, both from the language aspect and from a formal point of view. Suitable experiments are conducted and clearly described. Previous work is cited. This paper is ready to be published at NeurIPS. 

Some minor criticism:
 - The title of the paper could be more what kind of differential equations are being used and what is inferred.
 - It might have been helpfull to mention further possible areas of application.
 - Is it possible to work with a non-autonomous differential equation? (lines 49-50)
 - For the wider machine learning audience: What is a latent force? What are latent force models? 
 - Due to potential ethical concerns: is the "SIR-type Modell" the canonical/only choice to model COVID-19? What are the alternatives?
 - Line 64: how to estimate gamma and eta might be worth a footnote. 
 - How is the ODE grid chosen? equi-distant? adaptive?
 - I would have liked the related work section earlier in the paper.
 - Don't you think that "
    (a) Did you state the full set of assumptions of all theoretical results? [Yes]
    (b) Did you include complete proofs of all theoretical results? [N/A]
   " is a little bit contradictory? I mean one could have said more about function spaces which allow for solutions of the ODEs, existence of solution of ODEs and more aspects of the usual problems when dealing with differential equations. I do not think that this would have helped the paper.

It is a minefield to adequately cite all papers relating differential equations and machine learning. However, the following paper (and references therein) might be suitable to also be cited:
 - Chen et al., Neural ordinary differential equations
 - Dupont et al., Augmented neural odes
 - Jidling et al., Linearly constrained Gaussian processes
 - Lange-Hegermann, Algorithmic linearly constrained Gaussian processes
 - Rubanova et al., Latent odes for irregularly-sampled time series
 - Yang et al., Inference of dynamic systems from noisy and sparse data via manifold-constrained Gaussian processes
limitations_and_societal_impact: See ethical concerns.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a technique for computing approximate ordinary differential equation (ODE) solution posteriors. As shown in previous work [31], the state solutions of mechanistic ODE systems can be approximated by an extended Kalman filter. The authors further extend this methodology by conditioning the filter on observed sequences. Unlike its Monte Carlo alternatives, the resulting inference scheme computes approximate posteriors in a single forward-backward pass. The synthetic data experiments demonstrate that both the state and parameter trajectories are accurately inferred. The vanilla method and a variant that allows non-negative state estimation are tested on a COVID-19 dataset, implying interpretable findings that agree with government measures.
main_review: Overall, the paper is written clearly and very easy to follow. The background is explained in sufficient detail. Figures are highly illustrative and communicate the ideas well (figure-1 might be slightly misleading as the curves seem like point estimates while the inference is probabilistic). Both the simulated and COVID experiments nicely show how high level of accuracy and calibration of uncertainty estimates. 

To my understanding, the model largely builds upon [31]. Since [31] has already defined the prior process and introduced artificial data points $z_{0:M}$, the main theoretical contributions of the manuscript are (i) the additional prior on $u(t)$ and likelihood, and (ii) the inference in log space. In my opinion, these extensions are straightforward and thus NeurIPS might not be a suitable venue. That being said, I believe the COVID experiments pertain to the challenges we globally face today, which boosts the impact of this manuscript. Therefore, if the NeurIPS organizing committee particularly welcomes COVID-related submissions (about which I couldn't find any information) or I miss any major aspect of the paper, I recommend a reject and re-submission to a more suitable venue. Below are more detailed comments:

- The observation model in eq. (1) is linear, which might be too restrictive for more complicated systems. A discussion on how to extend the method into non-linear observation mappings would be nice.
- The performance of the proposed framework should be analyzed thoroughly. Although the comparison against the MCMC baseline is impressive, I wonder how difficult the addressed problems are. For instance, how does the estimation error change with (intermediate) gaps in the observed sequences? Since the inference is already based on discrete filter update equations, it would be interesting to see how simple discrete-time approximations (to the underlying true ODE system) would behave (this is not to show the merits of continuous-time modeling but to show when/if discrete approximations fail).
- How are the prior parameters selected? An ablation study showing how, e.g., length-scale and M impact the overall performance would be nice. Similarly, does $\lambda$ provide a trade-off between data likelihood and accuracy of ODE solutions?

Minor comments:
- Assuming fixed $\gamma$ and $\eta$ parameters may be too simplistic. In the context of the global pandemic, for instance, different COVID variants and the ratio of people who have developed immunity directly affect these parameters. This should at least be addressed verbally.
- In Figures 5-6, what do the thin gray dashed curves represent (look like the "truth" curve in Figure 4)?
- (minor) The state notation $X(t) = (X^{(0)}(t), ..., X^{(\nu)}(t))$ might be updated to reflect that $X(t)$ is concatenated, e.g., $\mathbf{X}(t) = (X^{(0)}(t), ..., X^{(\nu)}(t))$.

**Post-rebuttal update:** Thanks the authors for the rebuttal. After reading the rebuttal, other reviews/responses, and a little bit of extra thinking, I started to think the paper should be accepted. Utilizing Kalman filters to infer unknown latent forces is indeed one idea that could impact further research and developments. That being said, I have three additional comments that should provide the ground for my final rating:
- The approach assumes a dense observed trajectory. The gradient matching-based "ODE likelihood" (7) would start failing with more and more sparse data. I believe the approach should be stress tested.
- The methodology is tested only on very low-dimensional systems with very simple data likelihood models. So, I wonder how generalizable this framework is.
- The inference heavily depends on [31]. This is not a problem but makes me question the originality of the contribution.
limitations_and_societal_impact: The limitations and potential negative societal impact are addressed. Maybe an additional sentence reminding that the findings are highly related to the assumptions made before inference would be nice.
review_title: null

Meta-review: The authors present a method for inferring latent functions in ODE model from observations. Reviewers widely praised the manuscript for clarity and motivation. 

The discussion focussed on a technical aspect: that the method only requires a single pass of an ODE solver. This is achieved through a reworking of the problem of ODE solving as a state-space inference problem. this is an exciting insight that could change how the community thinks about solving these sorts of inverse problems. 

One reviewer raised a serious concern about the lack of a societal impact statement: the authors have provided their statement in the rebuttal with which I am satisfiedL this must make it into the camera-ready edition. 
Decision: Accept (Poster)