Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a method for incorporating physical dynamics into Deep GPs for the purpose of modeling nonlinear dynamical systems. The dynamics are incorporated through a kernel built via the Green's function for an associated dynamical system. 
main_review: Overall, the paper was well motivated and well written.  My main concerns are have to do with related work:

First, I would have appreciated additional summary of how this method compares to previous work. The introduction and related work section discuss a number of methods and techniques that this work builds off of or is similar to (LFMs, DGPs, random Fourier features, etc) as well as different axes along which the related works differ (shallow vs deep, if or how they incorporate physical dynamics, etc). Perhaps include a table that enumerates these related works and which combination(s) of the techniques they studied?

Second, the experimental section could make additional comparisons to other methods for modeling nonlinear dynamical systems. The paper mentions the work of Lorenzi and Filippone as being quite similar, but having a different approach to incorporating dynamical constraints. Shouldn't their method also be included in the experimental evaluation?

Finally, the paper could provide more guidance on how to choose the number of latent forces and the number of random features, in addition to other hyperparameters. How do these choices affect the results?
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a deep GP for nonlinear dynamical systems. Crucially, each of the GPs uses a physics-informed kernel a la the latent force model.
main_review: # Strengths
Learning good models of nonlinear dynamical systems is an important topic that spans many fields such as RL and state-space models to name a few. I think this is a novel combination of deep GPs + random features + latent force models that allows for fast and efficient learning and inference.

# Weaknesses
I think the biggest weakness is the structure of the paper. Firstly, I don't think the idea of using physics-informed random features was well motivated. Specifically, for a simple latent force model the functional form of the dynamics is assumed--providing a very strong inductive bias--and a GP is used to learn the force components. Or for deep CNN, the pro of using multiple convolutional layers is that they extract a hierarchy of visual features. It would have been nice if the authors would have demonstrated some empirical evidence that these physics-informed random features give the model an inductive bias towards certain types of dynamics.

Next, I think there was unnecessary notation that made the paper harder to follow. Specifically, in section 2.2 the authors introduce random feature expansions for latent force models with a generic Green's functions but in section 3.1 they end up using the Green's function for a linear ODE; this led to an increase in, my opinion, unnecessary notation that just made the approach harder to follow. 

Lastly, I am puzzled by the choice of the UCI regression benchmarks task in the experiments section. The goal of this paper was to introduce a method for learning dynamical systems. While I am impressed that the proposed approach works well here, these types of datasets isn't what the approach was motivated for (which ties back to my point about lack of motivation). For instance, there are a number of synthetic dynamical systems that are commonly used to benchmark sequential VAEs (these can be adapted to this instance by having the latent dynamics be fully observed): FitzHugh-Nagumo, Lorenz attractor, Cart-pole, different robots from MuJoCo, etc. I would have been more impressed seeing these types of results in a paper about learning nonlinear dynamical systems.

# Conclusion
While I do think this is a cool idea, I think more time needs to be spent on the writing of the paper and applying the proposed approach on common nonlinear dynamical systems.

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a model aimed to learn time series stemming from Dynamical Systems while incorporating uncertainty. The model is a Deep Gaussian Process where the random features expansion is done using a physics informed approximation derived from the Green Kernels of certain families of ODEs.
main_review: The model presented here is novel and its derivation seems principled enough in my opinion. The results show that DLFM compares favourably to DGPs both in terms of accuracy and uncertainty calibration.

However, I am quite skeptical about the baselines used here: how is a NN with a single hidden layer deep? There are many other architectures (deeper, residual, recurrent, Neural ODE-like models,...) which can be compared in terms of accuracy and sometimes uncertainty calibration to the proposed model. More specifically, I think that the model presented in https://openreview.net/forum?id=Skey4eBYPS is particularly relevant to this work and should be presented as a baseline.

Thus, while I think the ideas proposed here are promising indeed, I don't think that the baselines compared to are strong enough to fully validate them.

Edit after rebuttal: After reading other reviews and the answer of the authors, I am happy to raise my score: This paper brings a novel method to GP-based estimation of non-linear DS which should be of interest to many in the community.
limitations_and_societal_impact: I don't think there are any other than what is already acknowledged in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This manuscript describes Deep Latent Force Models (DLFM), a nonlinear dynamical system with nicely calibrated uncertainty quantification. Similar to deep Gaussian processes (DGPs), the presented framework stacks layers of GP layers with physics-inspired covariance functions following ordinary differential equations (ODEs). The inference is based on the weight space approximation of GPs that maintains variational distributions over the weights and spectral frequencies. The model outperforms (deep) GPs on time series prediction tasks and performs slightly better in regression tasks.
main_review: This submission proposes a very natural extension to LFMs by "making it deeper". Although the idea and motivation are straightforward, the authors have done a great job at describing, formulating, and evaluating the method. The experiments are not very exhaustive but adequate to show clear improvements over both DGPs and LFMs. Also, the manuscript is written very clearly; I had no difficulty following the background and the methodology. All in all, the paper does not lack to address any major concerns of mine and I recommend an accept. 

Below are more detailed comments and a few questions. I would be happy if these are addressed in the rebuttal (and possibly in the paper).

- The authors did not specify a recipe for choosing the form of the differential function. Since it directly impacts the covariance, gaining some intuition would be valuable. Also, is the same differential equation (line 132) used in all experiments?
-  Without any latent layer, the input to the function $f_d(t)$ was time and the ODE helped including the physics into covariance function, both of which make sense for learning dynamical systems. Now in this deep model, how can we interpret the differential equation in the latent space? 
- Why is the VAR-FIXED trick needed? Also, are the same Monte Carlo samples used both for training and testing? If so, does not it imply potential overfitting to these samples?
- I wonder if the model has passed a stress test. How would deeper architectures, wider hidden layer(s), more/fewer random features, and so on affect the results, training speed/ease, etc?
- Why was LFM-RFF trained with 20 random features instead of 100 as in the deeper variant?
- Looking at Figure3b-3c and Figure5, the model seems to achieve similar MSEs as the competing methods but certainly better at uncertainty quantification. Would the authors agree with that statement? If so, stating this explicitly and a discussion on the reason(s) would help the reader better comprehend the methodology.
- In the top-left panel of Figure4, the uncertainty drops at test time (particularly when ABP<-2). Can the authors clarify why might this be happening?
- A note on the computational complexity would be nice.
- This is more of a comment than a question: Instead of this (potentially overly) complicated model, what if we had a simpler two-layer architecture that first transforms the time input non-linearly and then the final output comes from an LFM? Why should the proposed formulation be preferred over this simpler architecture (or something similar)? 

**Post-rebuttal comment:** Thanks the authors for the rebuttal. I didn't have any major concerns, so I keep my rating as it is. Nice piece of work!
limitations_and_societal_impact: Both are addressed in the text.
review_title: null

Meta-review: There was initially some spread in the reviewer scores and perception of this paper, but the reviewer consensus did lean towards acceptance in the end. The additions proposed in the rebuttal are quite extensive but written out in sufficient detail that the reviewers also believe the paper can be accepted. Please, carefully pay attention to the reviewer comments and make sure to include the promised changes in the camera-ready version. In terms of additional experiments, including the results you presented during the rebuttal/discussion phase is sufficient.

Decision: Accept (Poster)