Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper addresses the problem of data-driven identification of latent representations for partially-observed dynamical systems. The proposed method uses an augmented state-space model where the dynamical operator is parameterized using neural networks. Given some training data, the proposed approach jointly estimates the parameters of the model as well as the corresponding unobserved components of the latent space for the training sequences. Experimental results on forecasting using real and synthetic datasets are presented.

Overall I think that the paper makes an interesting contribution. I find very interesting the idea of performing inference on the set of unobserved components in the latent space. The empirical results seem sufficient to me, but I am not familiar with relevant baselines (see below). Please find below some comments and questions.

I am personally not familiar with the literature on this problem, so my assessment might be affected by this. I did not find the paper easy to read and the presentation assumes a lot of previous knowledge. I think that the background and related work section could be more friendly written (considering the ICLR audience).

The training scheme (described in Section 3) uses one-step-ahead forecasting. The temporal consistency of the unobserved component of the latent space is only loosely enforced with the regularization term in (6). One could train using forecasting with more steps (and only doing inference for the initial y_t of the subsequence), as this is closer to what is used at test time. Do you think this would be helpful for having better accuracy when forecasting more steps?

It would be good to provide more details on how to build the forecasting operator (implementing 4-th order Runge-Kutta scheme) and what is exactly the bilinear architecture of Fabelt et al.

Regarding the experimental validation, I like that the paper starts with a simple motivating example and moves to more complex cases. Experimental results are convincing to me, as the model is able to recover the performance of other models that do have access to the full state. I am not familiar with the literature so I'm unable to judge whether all relevant baselines are included. 

Regarding the Latent-ODE baseline, would results change running with different (larger) dimension for the latent space?

The paper should cite the work: Ayed, et al. "Learning Dynamical Systems from Partial Observations." arXiv preprint arXiv:1902.11136 (2019). Would this be a relevant baseline to compare to?

Is the training data regularly-sampled? Would the model be robust the irregularly-sampled training data?

The authors evaluate all methods with one and four step forecasting in the last two experiments. I think that it would be informative to show a wider range of number of steps, to show how performance degrades with longer predictions (more than 4).

Finally, regarding the Modelling Sea Level Anomaly task, all baselines are ran by the authors. It would be informative to also include results of prior art using this dataset, if possible.

Other minor comments:

The citation format is wrong. Most citations should be using the \citep command

In the second paragraph of Section 1, it says: "Unfortunately, When the"

In the caption of figure 2 it says: "according to thr"

A few lines before the "Modelling Sea Level Anomaly" subsection there's an exclamation sign before the text"1e-4 for a one-step..."

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Update: I raised the score from 1 to 3 to acknowledge the authors' consideration for the 2000-2010 literature on learning dynamical systems from partial observations. Unfortunately, the writing is still confusing, some of the claims in the introduction and rebuttal are inexact ([5] does not embed the observations and does work with partially observed environments), and the method lacks originality compared to existing work. Other work that relies on ODE integration and in finding high-dimensional state variables has recently been published and tested on more ambitious datasets, e.g. Ayed et al, "Learning Dynamical Systems from Partial Observations", arXiv 2019.

***

TL;DR: relatively well written (if sometimes confusing) paper that reinvents the inference of latent variables in nonlinear dynamical systems that has been published in the 2000s, and that misses an important chunk of literature (and experiments on dynamical systems such as Lorenz-63) from that time.

This paper proposes an approach for learning dynamical systems from partial observations x, by using an augmented state variable z that follows dynamics that can be described by an ordinary differential equation (ODE) with dynamics f. The authors motivate their work by the problem of dynamical system identification when only partial observations are available. The authors claim that was to date primarily addressed using time-delay embedding, following Takens' theorem. The authors introduce s-dimensional unknown state variables z, dynamical function f (for the ODE on z), flow phi on z, limit cycles on z, observation function H: z -> x that goes from z to n-dimensional observations x, low k-dimensional manifold r (with a map M: x -> r), and state augmentation variable y. The reconstructed state variable u is the concatenation of r and y. One key ingredient of the method is to infer the optimal value of state augmentation variable y during learning (see equations 5 and 6) and inference for forecasting (7); this is not well explained in the abstract and introduction.

I would note that the problem of state space modeling (SSM) and dynamical system identification has been well studied, and the notation and reformulation in this paper is somewhat confusing for those who are used to the notation in SSMs (specifically, expressing the observation approximation as M^{-1}(G(phi(u_{t-1}))). Learning a state-space model involves both learning parameters and inferring the latent states representation (or, in graphical models, the distribution of these latent states) given the parametric models. One approach has been to formulate the state-space model learning by maximum likelihood learning of the model parameters so that the generative model fits observed data x, and this would involve factoring out the distribution of latent states z; the algorithm would rely on Expectation Maximisation, and could involve variational approximations or sampling. While the state space models were hampered by their linearity, several papers in 2000s showed how it is possible to learn nonlinear dynamical models, e.g. [4], [5], [6] and [7] to cite earlier ones. Equations (5) and (6) are similar to the standard equations for a dynamical system expressed in continuous time, with the only difference that the optimisation is with respect to y only, rather than w.r.t. z or u (why not \tilde z or \hat z?).

The paper mentions various initialisation strategies for y (last paragraph of section 3). Why not predict from the past of the observations, like is done in many other similar work?

The literature review mixes older and newer references. For example, on page 1, I would note that the Takens' theorem has been applied in conjuction with Support Vector Regression as early as 1999 [1][2], and with neural networks in 1993 [3].

Most importantly, the ideas of this paper have already been published in [4] (with architecture constraints on the neural network state-space model), in [5] (with any nonlinear neural network state-space model), in [6] (using Restricted Boltzmann Machines) and in [7] (using Gaussian Process latent variable models).
The model is illustrated with experiments on a 2D linear attractor, on the Lorenz-63 attractor. Given the results published in [1] and [2] using SVR on 1D observations of that attractor, and in [5] using a recurrent neural network, I am unconvinced by these results. It seems in particular that the number of training points (around 4000) limits the performance of RNN / LSTM models. The application to Sea Level Anomaly is interesting.

Minor comments:
"Unfortunately, When" (page 1)
There is a missing -1 after M in equation (5) and (10)
In equation (7), should not the sum go from t=0 to T, as x_t is unknown for t>T?
What is prediction and what is extrapolation on Figure 1?
The caption of Fig 1 contains (left)
The figures seem squeezed with the captions / titles un-aesthetically wide. 
Labels on Figure 5 in the appendix seem mixed, and red should be the ground truth

[1] Mattera & Haykin (1999) "Support vector machines for dynamic reconstruction of a chaotic system"
[2] Muller, Smola, Ratsch, Scholkopf, Kohlmorgen & Vapnik (1999) "Using support vector machines for time-series prediction"
[3] Wan (1994) "Time series prediction by using a connectionist network with internal delay lines"
[4] Ghahramani, and Roweis (1999) "Learning nonlinear dynamical systems using an EM algorithm"
[5] Mirowski & LeCun (2009) "Dynamic Factor Graphs for Time Series Modeling"
[6] Taylor, Hinton & Roweis (2006) "Modeling human motion using binary latent variables"
[7] Wang, Fleet & Hertzmann (2006) "Gaussian process dynamical models"

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new deep learning approach based on Takens’s theorem to identify the dynamics of partially observed chaotic systems. In particular, the method augments the state using the solution of an ODE. Experiments on Lorenze-64 dynamics and sea level anomaly demonstrate the advantage of the proposed method over state-of-the-art baselines.

+ The unification of Taken’s embedding theorem and deep learning provides a novel perspective into dynamical systems
+ Impressive experiment results compared with baselines including RNN and latent ODE

- The proposed method requires knowledge of the underlying dynamic model to solve the ODE, which is not fair for other methods
- The model is trained using data from the same initial conditions, which is essentially overfitting. The authors should provide experiments for dataset from different initial conditions.
- The writing is not very clear. For example, how to solve the optimization problem in Eqn (7),  as the augmented states u_{t-1} are unknown? How to find the bijective mapping M for general dynamical systems?

Minor: question mark in section 4 page 6.   Figures 2 plots are difficult to read, pls provide more details in columns and rows.
review_title: Official Blind Review #2

Meta-review: This paper presents an ODE-based latent variable model, argues that extra unobserved dimensions are necessary in general, and that deterministic encodings are also insufficient in general.  Instead, they optimize the latent representation during training.  They include small-scale experiments showing that their framework beats alternatives.

In my mind, the argument about fixed mappings being inadequate is a fair one, but it misses the fact that the variational inference framework already has several ways to address this shortcoming:
1) The recognition network outputs a distribution over latent values, which in itself does not address this issue, but provides regularization benefits.
2) The recognition network is just a strategy for speeding up inference.  There's no reason you can't just do variational inference or MCMC for inference instead (which is similar to your approach), or do semi-amortized variational inference.

Basically, this paper could have been somewhat convincing as a general exploration of approximate inference strategies in the latent ODE model.  Instead, it provides a lot of philosophical arguments and a small amount of empirical evidence that a particular encoder is insufficient when doing MAP inference.  It also seems like a problem that hyperparameters were copied from Chen et al 2018, but are used in a MAP setting instead of a VAE setting.  Finally, it's not clear how hyperparameters such as the size of the latent dimensions were chosen.
Decision: Reject