Learning Latent Dynamics for Partially-Observed Chaotic Systems
Authors: Said ouala
Duong Nguyen
Lucas Drumetz
Bertrand Chapron
Ananda Pascual
Fabrice Collard
Lucile Gaultier
Ronan Fablet
Venue:  (2020)
Keywords: Dynamical systems
Neural networks
Embedding
Partially observed systems
Forecasting
chaos

Abstract: This paper addresses the data-driven identification of latent representations of partially-observed dynamical systems, i.e. dynamical systems whose some components are never observed, with an emphasis on forecasting applications and long-term asymptotic patterns. Whereas state-of-the-art data-driven approaches rely on delay embeddings and linear decompositions of the underlying operators, we introduce a framework based on the data-driven identification of an augmented state-space model using a neural-network-based representation. For a given training dataset, it amounts to jointly reconstructing the latent states and learning an ODE (Ordinary Differential Equation) representation in this space. Through numerical experiments, we demonstrate the relevance of the proposed framework w.r.t. state-of-the-art approaches in terms of short-term forecasting errors and long-term behaviour. We further discuss how the proposed framework relates to Koopman operator theory and Takens' embedding theorem.

Under review as a conference paper at ICLR 2020
LEARNING LATENT DYNAMICS FOR PARTIALLY-OBSERVED CHAOTIC SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper addresses the data-driven identification of latent Ordinary Differential Equation (ODE) representations of partially-observee dynamical systems, i.e. dy- namical systems whose some components are never observed, with an emphasis on forecasting applications and long-term asymptotic patterns. Whereas state- of-the-art data-driven approaches rely on an explicit mapping of the observation series onto a latent space, the proposed approach only relies on the definition of an augmented space, digee--dimensional than the manifold spanned by the ob- served variables, where the dynamics of the observations can be fully described by an ODE. From a numerical point of view, the proposed approach exploits a neural-network ODE representation and the associated variational minimization scheme. Numerical experiments support the relevance the proposed framework w.r.t ofate-of-the-eee art approaches, including neural ODE schemes, both in terms of short-term forecasting errors and long- behaviour. We further discuss how the proposed framework relates to Koopman operator theory and Takens' embedding theorem.
INTRODUCTION
Learning the underlying dynamical representation of some observed variables x, E IR" (where (fo....,T) is the temporal sampling time and n the dimension of the observations) is key challenge in various scientific fields, including control theory, geoscience, fluid dynamics, economics; for
applications ranging from system identification to forecasting and assimilation issues (Lai & Wei.
1982 Abarbanel & Lall 1996 Jeong& Hussain 1995) Koopmans) 1949).
For fully-observed systems, i.e. when the observed variables x, relate to some underlying determin- istic states s, according to a diffcomorphic mapping possibly corrupted by noise, recent advances (Brunton et al. 2016b; Fablet al. 2018; Nguyen al. 2019) have shown that one can identify the governing equations the dynamics of s, from representative dataset of observations (x,, hi- Unfortunately, when the observed variables x; only relate to some but not all the components of underlying states St, these approaches can not apply since no ODE or, more generally, no one-to- one mapping defined in the observation space can represent the time evolution of the observations. In this context, Takens's theorem states the conditions under which delay embedding, formed by lagged versions of the observed variables, guarantees the existence of governing equations in the embedded space (Takens 1981).
Takens's 's theorem has motivated a rich literature of machine learning schemes to identify dynamical representations of
systems using a delay embedding. This comprises both non- parametric schemes based on nearest-neighhors or analogs (Abarbanel 1996a) as well as parametric schemes which include polynomial representations (Paduart et al. (0000) neural network models [aa 1993) and Support Vector Regression (SVR) models (Mattera Haykin 1999. Müller al.,
1999 For all these approaches, the identification of the appropriate delay embedding is critical
issue Abarbanei [996b(c).
From neural network and machine learning perspective, the inference of latent space, within a State Space Model (SSM) framework, for dynamical systems has motivated a broad literature especially for time series forecasting (Ghahramani & Roweis 1999] Wang al. 20066 Mirowski & LeCun 2009; He et al., 2013), Krishnan et al. (0166). Most of those techniques were introduced in the context of reduced order modeling (ROM) to infer low-dimensional manifolds, where the

---

Under review as a conference paper at ICLR 2020
dynamics of the observations be represented. When considering partially-observe systems, these approaches state this issue as the inference of a (non-linean) projection of an input sequence be modeled. This projection is usually computed in a probabilistic framework using Bayesian filtering techniques. However, recovering the attractor's dynamics using iterative predictions is still issue for such models since the explicit modeling of latent space as delay embedding of the observations may limit the expressiveness of the latent states, especially when considering chaotic dynamics.
in latent space where the observations
In this work, we show that we do not need to rely explicitly neither on delay embedding nor on the learning of an inference model map the observation series to a latent space. We only assume that we may define an augmented space, higher-dimensional than the manifold spanned by the observed variables, where the dynamies of the observations can be fully described by an ODE. Using neural-network representations for the parametrization of the dynamical model, it amounts to jointly learning the governing ODE and reconstructing the augmented latent states for given observation dataset using variational framework. We report experiments on linear and chaotic dynamics, which illustrate the relevance of the proposed framework compared to state-of-the-arr approaches. We then further discuss the key features of this framework with respect sfate-of-the- art dynamical systems identification tools such as Koopman operator theory (Koopman 1931).
BACKGROUND AND RELATED WORK
This section introduces the learning of dynamical representations for partially- observed systems and links this problem to recent advances in machine learning.
Let us consider an unobserved state variable s governed by an autonomous system of m differ- ential equations & = -s..). Let us also assume that this system generates flow Φ (8t.) fi ((w E RT with trajectories that are asymptotic to a limit-cyele of dimension d con- tained in RT. We further assume that we are provided with a measurement function H. that maps state variable s to our observations x; H((s) E R".
When considering the data-driven identification a dynamical mapping that governs some observa- tion data, we first need to evaluate whether the dynamics in the observation space can be described using a smooth" ODE. Another way to tackle this question is to find the conditions under which the deterministic properties of the unobserved limit-cycle L are preserved in the observation space in R" such that one can reliably perform forecasts in the observation space. The general condition under which mapping HH preserves the topological properties of the initial limit-cycle involves differential structure. Assuming that L is a smooth compact differential manifold, the topological properties of L are preserved through a mapping H. in 1" if H. is one-to-one and is an immersion of L in R". Under these conditions our observation mapping is called an embedding (Sauer et al., 1991..
The simplest example of an embedding involves an identity observation operator H. With this em- bedding, we have direct access to the state variable s which is governed by a deterministic ODE, This particular case has been widely studied in the literature. Parametric representations have been for decades the most popular models thanks to their simplicity and interpretability (Paduart et al., 2010). Brunton et al. 201 Recently, these approaches have been enriched by neural network and deep learning schemes (Wiewel et al. 2018). (Raissi et al. 2018). In particular, the link between residual networks (Chen et al. 2018) Ouala al. 2019) and numerical integration schemes have opened new research avenues for learning extremely accurate dynamical models even from cregularly-samplee training data. These schemes show greater interpretability and forecasting performance for the data- driven representation of systems governed by an ODE, compared with other state-of-the-art neural networks schemes, including Recurrent Neural Networks (RNN) such as LSTM (Long-Short-TTem Memory).
However, for a wide range of real-world systems, we are never provided with an observation oper- ator that forms an embedding of the unobserved dynamical system. In such situations, we do not have any guarantee on the existence of a smooth ODE that governs the temporal evolution of our observations. From this point of view, the question of finding an appropriate dynamical representa-
'The word smooth here stands for continuously differentiable or C².
2

---

Under review as a conference paper at ICLR 2020
tion of some observed data may not be this straightforward. The fact that our data may come from some unobserved governing equation may restrict the use of the above-mentioned state-of-the-art algorithms. The main difficulty lies in the ability to map observation series to latent space that provides at least a one-to-one mapping between two successive states. From a geometrical point of view, the time delay theorem (Takens, [98TT) provides way to build latent space that preserves the topological properties of the true (unobserved) dynamics limit-cycle. A generalization of this theorem (Sauer et al., 1991) shows that one can reconstruct topologically similar limit-cycles us- ing any appropriate smooth composition map of the observations. The derivation of a dynamical system from such representations however encounters large disparities since no explicit relation- ships between the defined phase space and an ODE formulation have been clearly made. Classical state-of-the-art techniques such as polynomial representations (Brunton et al. 20T6b) and K-Nearest Neighbors (KNN) (L-uensat et al. 2017) algorithms were proposed but they often fail to achieve both accurate short-term forecasting performance and long-term topologically similar reconstructed limit-cycles (see experiments for an illustration).
We may also point out that the limitation of ODE-based representation in deep learning architec- ture has also been pointed out recently in (Dupont et al.) 2019 Zhang et al., 2019) for classification issues. As ODE-derived trajectories do not intersect, it may limit the ability of neural ODE rep- resentations to reach relevant classification performance in a given feature space. To address this issue, (Dupont et al. 2019) and (Zhang et al. 2019) propose to consider an augmented state, simply by augmenting the observed state by a number of zeros to create high-dimensional space in which an ODE representation can be identified. Such strategy cannot apply to time series modeling as successive augmented states cannot be forced to zero for some dimensions.
Advances in the inference of latent spaces in state space models was introduced essentially, from a dynamical systems perspective, to retrieve cow-dimensional manifolds, where the dynamics of the system evolve. When applied to partially-observed systems, the latent variables are typically inferred from sequence of observations through parametric modeling of the posterior distribution as in (He et al. 2015: Krishnan et al.) 2016 Chen et al., 2018) or through marginalization with model constraints as in (Ghahramani & Roweis [999] Wang et al., 2006) However, such models often fail in accounting for long term patterns (as shown in the experiments). This is due to the fact that the latent space is constrained to be a non- non-linear projection of a sequence of observations, which limits the expressiveness of the dynamical model. Interestingly, (Mirowski & LeCun, 2009) does not involve the learning of an inference model as the reconstruction of the latent states is solved as gradient-based minimization of the dynamical prior w.r.t. an observation series. However, the dynamical prior relies on an explicit delay embedding as the dynamics of the latent state depend both on the previous latent state and on delay embedding of the observations.
In this work, we address the identification of a latent embedding, associated with an ODE represen- tation, for partially-observed systems. The core idea of this work is infer an augmented latent space, governed by an ODE, which fully explain the observed time series and their dynamics. In contrast to previous work (He et al., 2015] Krishnan et al. 2016 Chen et al. 2018 Ghahramani & Roweis 1999 Wang et al.J (006). we do not exploit either a delay embedding or an explicit modeling of the inference model (i.e., the reconstruction of the latent states given the observed time series). As such, our scheme only involves the selection of the class of ODEs of interest. The ex- pected benefits are as follows: (i) our model ensures the existance of latent embedding associated with ODE, which may not be guaranteed when considering parametric inference model and/or a delay embedding, (ii) our model reduces the complexity of the overall scheme to the complexity of the ODE representation, (iii) our model guarantees the consistency of the reconstructed latent states w.r.t. the learnt ODE. Our experiments suggest that this may be of key importance to jointly address short-term forecasting performance and long-term chaotic patterns.
3. LEARNING LATENT REPRESENNATIONS OF PARTIALLY-OBSERVIE DYNAMICS
Augmented latent dynamics: Let us consider a continuous m-ddimensional dynamical system S: governed by an autonomous ODE &; = fs:) with o the corresponding flow (st.) = Sr [Sw)... We assume the state 8; be never fully-observed. Formally, we can define an obser-
3

---

Under review as a conference paper at ICLR 2020
vation function H. RT - IR" such that the observation x, follows x, = H((s). An example of interest comprises the situation when one observes only the n first variables of st.
Our goal is to derive an ODE representation of x; R". However, the key question arising here is the extent to which the dynamics expressed in the observations space, reflect the true underlying dynamics in R and consequently, the conditions H under which the predictable deterministic dynamical behavior of the hidden states is still predictable in the observations space. To illustrate this issue, we may consider linear dynamical system in the complex domain governed by the following linear ODE:
S os;
Ste So
with s C a state variable and a E C complex imaginary number. The solution of this problem is
S; = K
(2)
with K constant depending on So. Let us assume now that we are only provided with the real part as direct measurements of the unobserved state i.e. H(.) = Real(.) xx Real(s:).
Proposition 1 : The flow of an ODE cannol represent the time evolution of x
The proof of the proposition is given in the appendix. The intuition exemplified for the 2- dimensional system given by ( is as follows. If we assume that we are only provided the real part as direct measurements X2 of the true states St, no smooth autonomous ODE model in the scalar observation space can describe the trajectories of the observations as the mapping between two observations is not one-to-one For example, assuming that St, and St, correspond to two states that have the same real part but distinct imaginary parts, the associated observed states are equal Xig = Xty However, the time evolution of the states St, and St, differ if they have different imag- inary parts. The observed states X and Xt++&are then no longer equal after any time increment As a consequence, a given observation may have more than one future state and this behavior cannot be represented by smooth ODE in the observation space. Hence, the application of ODE mapping such as (Chen et al. 2018) and (Fablet et al., 2018) for such observations will lead to poor forecasting performance. From naive Neural networks point of view, fitting such model will most likely force the forecasting into an equilibrium point since we may iteratively match the same inputs different output predictions. This simple example emphasizes the need for considering augmented latent states.
In this context, Takens's theorem guarantees the existence of augmented space, in which a one- to-one mapping exists between successive time steps (Takens| 1981). More precisely, for given observation operator H of deterministic underlying dynamical system that governs s, one can de- fine a delay embedding from an observation sequence, i.e. {xt, -54--66- : delay and a, two natural numbers. An ODE representation in a latent space can than be formulated according to (Chen et al. 2018) by sampling the latent states from a parametrized posterior distri- bution given the delay embedding time series. Here, we aim to identify an augmented latent space without resorting to an explicit mapping from the observed series to the latent space, such that the latent dynamics are governed by a smooth ODE and can be mapped to the observations. Formally, let us introduce the following notations. Let u, Rd define a digdmeenioonall augmented latent state formed by an observed componen
xxo+b5) ,.oo with 5 the time
x2 and an unobserved latent component referred to as 2₁-
[x] 2
u
Assuming that the augmented latent state u, is governed by an ODE leads to the the following state space model:
ú = (u,)
(4)
x; = G(u,)
where the dynamical operator fo belongs to a family of smooth operators (in order to guarantee uniqueness (Coddington & Levinson 1955)) parametrized by 0. G is projection matrix that satis- fies x, = -Gu,). It means that operator G only returns the observed component of the augmented
2 For simplicity, the observed component is considered in the equations as the direct observations. However, in general, when considering Reduced Order Models (ROMs), the observed component can be seen as low dimensional projection of the observations. Please refer to the appendix for a similar derivation of the equations in the case of reduced order modeling.
4

---

Under review as a conference paper at ICLR 2020
latent state. Compared with previous work, the key difference lies in the definition of the augmented latent space, where only the observed component is explicitly stated as a function of observation x,. Similarly to neural ODE frameworks (Chen et al. 2018 Fablet et al. 2018). we consider neural-network representation with Lipschitz nonlinearities and finite weights for ODE operator fo- As detailed in the next sections, address the identification of operator fo from dataset of ob- served state series .x0....,X77 as well as the exploitation of the identified latent dynamics for the forecasting of the time evolution of the observed states, for instance unobserved future states [xr+1..... XT++N J. Both issues involve the joint inference of the unobserved component of the augmented latent states. This formulation generalizes reduced-ordes modeling (ROM) (Cham- pion et al. 2019), where observation x, fully derives from a lower-dimensional variable. We let the reader refer the appendix for associated generalization of the proposed framework.
Learning scheme: Given an observation time series (xo...., Xr) we aim to identify the state- space model defined by ( which amounts to learning the parameters of the dynamical operator fo. However, the component 2, of the augmented state u, is never observed, this identification requires the joint optimization the model parameters 0 as well as of the hidden component 2, Formally, this problem is stated the following minimization of the forecasting error observed variables:
i arg min min
X G (Bo., (u:-1)))l
(5)
u,
(u.
Subject to
G(u,) Xt
with o the one-step-ahead diffcomorphic flow associated with operator fe such that:
( = u + f( fo(
In (5), the loss to be minimized involves the one-step-ahead forecasting error for the observed vari- able Xx. The constraints state that the augmented latent state sequence {uo, uyy defined by (3) should be solution of the ODE 4 Here, we numerically minimize the equivalent formulation:
T
min min (Ze)
Σ
|xx - G (20,r (u,- + Al|u (e.c(u
where = [x [xff"00 xf and À weighting parameter: The term ||,., -.. may be regarded as a regularization term such that the inference of the unobserved component 2, of the augmented state u, is not solved independently for each time step.
Using a neural-network parametrization for the ODE operator fe, the corresponding forecasting flow Φo.c is also stated as a neural network based (typically -odder Runge-Kutte Kutta scheme). This architecture, very much similar to ResNet (He ct al. 2015). allows very accurate identification of ODE models (Fablet et al. 2018 Ouala et al. 2019 Hence, for a given observation sequence (xo..... }r). we minimize 6 jointly w.r.t. 0 and unobserved variables (Za,...,Zr].
numerical integration scheme formulation
Application to forecasting: We also apply the proposed framework to the forecasting of the ob- served states X,- Given trained latent dynamical model (4), forecasting future states for x; relies on the forecasting of the entire augmented latent state u, The latter amounts determining ini- tial condition of the unobserved component z, and performing a numerical integration of the trained ODE (
Let us denote by x, t E ft., T) new series of observed states. We aim to forecast future states x3, (T'+1,... .,. T 8T'). Following (6). infer the unobserved component 27 of the augmented state un at time T from the following minimization:
27 arg min min
Σ | - G (0o.. (u 11 Am[na +f..(uu"
z (x))er
5

---

Under review as a conference paper at ICLR 2020
Here, we only minimize w.r.t latent variables {zī) given the trained forecasting operator Pø,t- This minimization relates to a variational assimilation issue with dynamical and observation operators (Lynch & Huang 2010). Similarly the learning step, benefit from the neural-network parameterization of operator Po.t and from the associated automatic differentiation tool to compute the solution of the above minimization using gradient descent.
states and known
We may consider different initialization strategies for this minimization problem. Besides a simple random initialization, we may benefit from the information gained on the manifold spanned by the unobserved components during the training stage. The basic idea comes to assume that the training dataset is likely comprise state trajectories which are similar the new ones. As the training step embeds the inference of the whole latent state sequence, we may pick as initialization for minimization 7 the inferred augmented latent state in the training dataset which leads to the observed state trajectory that is the most similar (in the sense of the I.2 norm) to the new observed sequence x The interest of this initialization scheme two-fold: (i) speeding-up the convergence of minimization (7) as we expect to be closer to the minimum; (ii) considering initial condition which in the basin of attraction of the reconstructed limit-cycle. The latter may be critical as cannot guarantee that the learnt model does not involve other limit-cycles than the ones truly revealed by the training dataset, which may lead to a convergence to a local and poorly relevant minimum.
4 NUMERICAL EXPERIMENTS
In this section, we report numerical experiments to illustrate the key features of proposed framework. We consider three case-studies: a linear ODE case-study; a chaotic system, namely Lorenz-63 dy- namics, and real upper ocean data.
Application to a linear ODE: In order to illustrate the key principles of the proposed framework, we consider the following linear ODE in the complex domain:
$ as;
(8)
Sto = 80
with a = - 0.5j, J2 = -1 and So 0.5. As a E C with Real(a) < 0 and Sp * 0, the solution of this ODE is an ellipse in the complex plane (Fig.
As observations, we consider the real part of the underlying state, i.e. the observation function H C - is given by x, = Real(s;). This is a typical example, where the mapping between two successive observations is not one-to-one mapping since all the states that have the same real part lead to the same observation. As explained in section 3 one cannot identify an autonomous ODE model that will reproduce the dynamical behavior of the observations in the observations space.
We apply the proposed framework to this toy example. We consider a 2-dimensional augmented state u, [x, 2 As netrallnetwork parametrization for operator fe. consider neural network with a single linear fully-connected layer. We use an observation series of 10000 time steps as training data. As illustrated in [ given the same initial condition over the observable state, the inferred latent state dynamics, though different from the true ones, depicts similar spiral pattern. This result is in agreement with the geometrical reconstruction techniques (Takens) 1981 of the latent dynamics up to diffcomorphic mapping. Overall, our model learns a dynamical behavior similar to the true model represented by an elliptic transient and an equilibrium point limit-set. Furthermore, the projection of the augmented latent space and the true solution of Eq. (8) in the real axis illustrate the relevance of the proposed framework in forecasting the observations dynamics (mean square error < 1E 6).
Lorenz-63 dynamics: Lorenz-63 dynamical system a 3-dimensional model that involves, under some specific parametrizations (Lorenz) 1963), chaotic dynamics with a strange attractor. We sim- ulate chaotic Lorenz-63 state sequences with the same model parameters as proposed in (Lorenz) 1963) using the LOSDA ODE solver (Hindmarsh) 1983) with an integration step of 0.01. We sume that only the first Lorenz-63 variable is observed X = St,1- We apply the proposed framework to this experimental setting using training sequence of 4000 time-steps.
For benchmarking purposes, we perform a quantitative comparison with state-of taaee--f-theaatt art approaches using delay embedding representations (Takens 1981). The parameters of the delay embedding

---

Under review as a conference paper at ICLR 2020
Prediction
Preficice Extrapolation
Extrppolaion
solation
solution
Givi G B.N
Tim
Givio Boll))
Tiaa
Figure 1: Illustration for 2-dimensional linear ODE: Forecasted augmented latent space with respect to the true states given (left) the same initial condition as the training sequence and (right) different initial condition. Given the initial condition we illustrate both the prediction (forecast up to the end of the training time) of the trained model and the extrapolation (forecast beyond the training time) performances with respect the true trajectory ,Th projection of the solutions in the real plane illustrates the forecasting of the observations.
representation, namely the lag and the dimension dg of the augmented space were computed using state-of-the-art techniques. Specifically, the lag parameter was computed using both the mutual information and correlation techniques (Abarbanei) 1996b), respectively denoted as TMr and TCorr- Regarding the dimension of the embedding representation, we used the Takens embedding condition dg 2d - 1 with d the dimension of the hidden limit-cycle. The delay embedding dimension was also computed using the False Nearest Neighbors (FNN) method (Abarbanel 1996c). We also tested arbitrary parameters for the delay embedding dimension. Given the delay embedding representation, tested two sate-offtthearr data-driven representations of the dynamics. The Analog Forecasting technique (AF) which is based on the nearest neighbours algorithm (Lguensat et al., 2017) and the Sparse Regression (SR) method on a second order polynomial representation of the delay embedding states. Regarding deep learning models, we compare our method to a stacked Bidirectional LSTM (RNN) and the Latent-ODE model as proposed in (Chen al., 2018). Finally, the proposed framework, referred to as Neural embedding for Dynamical Systems (NbedDyn) was tested for different dimensions of the augmented state space, namely from to (please refer to the apendix for details on the neural networks based architectures
Fig. 2 iillustrates the learning process for the latent space from the initialization to the last training epoch. We also report the analysis of short-term forecasting performance as well the long-term asymptotic behavior characterized by the largest Lyapunov exponent of the benchmarked models in Tab [ The proposed model leads significant improvements in terms of short term forecasting performance with respect to the other approaches. Surprisingly, The Latent-ODE and RNN models lead to the poorest performance both in terms of forecasting error and asymptotic behavior. This is mainly due, in the Latent-ODE case, to the fact that the latent space is seen as a non linear projec- tion of the observed variables through the optimization of the ELBO loss (Krishnan et al. 2016)_ By contrast, our latent embedding formulation optimizes the latent states to forecast the observed variables which explicitly constrain the latent space be an embedding of the true underlying dy- namics. The RNN model in the other hand converges to a periodic solution (please refer to the appendix for forecasting figures) with still a poor short term forcasting performances. Overall, this results on deep learning models suggest that one should use such tools with care to guarantee satis- fying the specifications of the underlying system. Ths The SR model seems to lead to better short term forecast (using ad hoc parameters (r = 6, dg = 3), however, it does not capture well the chaotic
'The results of the neural networks based models were averaged over 5 runs except for NhedDDn dg 4 and dg where we only launched 3 runs. We will include other runs for the final version of the manuscript
7

---

Under review as a conference paper at ICLR 2020
Model
h
4h
À1
TMI =16
(p(FNN)
5.6 3
1.3E
0.85
TMJ =16
dp(T'akens) =
9.9.
244
NaN
AF
Toor =27
dy(FNN) =3
8.9E
2.3E
12.35
Teo =27
de(Takens)
8.53
1.95
NaN
==
dy =
8.0A 4
9.0E 4
0.87
T= 10
de 3
2.1E
4.9E 3
0.60
Tu: =66
dg(FNNN =
7.8 - 2
2.5
0.12
TM1 =16
dp(T'akens) =
4.5% 2
1.7E
NaN
SR
Toor =27
de(FNN)
1.E
4.6E
NaN
Too: =27
dg(Takens)=0
2.1E
8.4E
NaN
7,6
7AE
NaN
== 10
de 3
2.5E
5.7E
0.2535
Latent-ODE
6.9F 2 2.9 -
1.5F 1.5-1- 3F 2
NaN
RNN
- 4.6E
- 1.1E
6.79 0.0
3.2F 1.3F
1.7E
3+
0.81 0.09
NbedDyn
4
1.3E 1000-411 5.2E
5
7.3E
44 2.2
4
0.82 0.06
dg 5
3.&E 41 7.4E
4
2.0
314
4
0.80 1: 0.02
dg 6
3.7F - 4 28F
-
2.0F
3+
1.7E
3
0.92 + 0.02
dg 6 (Best)
9.1E
4.7E
0.92
Table 1: Forecasting performance on the test set of data-driven models for Lorenz-63 dynamics where only the first variahle is observed: first columns mean RMSE for different forecasting time steps, third column largest Lyapunov exponent of a predicted series of length of 10000 time- steps (The true largest Lyapunov exponent of the Lorenz 63 model is 0.91 (Sprott 2003
Epuach
40
Lanee
Figure 2: Evolution of the learnt latent space: starting from a random initialization of the aug- mented states 2; the latent space is optimized according the minimization of Eq. (6) form limit-cycle similar to the true Lorenz 63 attractor. We depict 3-dimensional projections of the learnt latent space for the proposed model with different embedding dimensions from de 3 to dE 6.
patterns, which are associated a positive largest Lyapunov exponent. This may suggest the com- bination of the SR model and a delay embedding may require additional investigation as a good geometrical reconstruction of the phase space as stated in Takens' theorem does not guarantee the existence of parametric ODE model based on the corresponding delay embedding variables. Bet- ter performance reported using an analog forecasting approach. The performance however greatly varies depending on the considered definition of the delay embedding. Using ad hoc parameters (T ( 6, dg = 3), one may retrieve the expected long-term chaotic behavior (Λ₃ = 0.87) with a relatively low short-term forecasting error (8.0e for one-step-ahead forecast). When consid- ering the proposed model, we report for all parametrizations, augmented space dimensions from to 6, performance at least in the same range the best analog forecasting setting. Besides, when
8

---

Under review as a conference paper at ICLR 2020
increasing the dimension of the augmented space, significantly decrease short-term forecasting
errors (< I.e - 4 for a one-step-ahead forecast when considering the best fit for dg = 6, i.c. one order of magnitude compared to the best benchmark model) while keeping an appropriate chaotic long-term pattern (A = 0.92).
Modeling Sea Level Anomaly (SLA): The data driven identification of dynamical representations of real data is an extremely difficult task especially when the underlying processes involve non stable behaviors such as chaotic attractors. This is mainly due to the fact that we do not have any exact knowledge of the closed form of the equations governing the temporal evolution of our variables. Furthermore, the measured quantity may depend on other unobserved variables which makes the exploitation of data-driven techniques highly challenging.
In this context, we report an application to SLA (Sea Level Anomaly) dynamics. which relate to upper ocean dynamics and are monitored by satellite altimeters (Calmant et al. 2008). Sea surface dynamics are chaotic and clearly involve latent processes, typically subsurface and atmospheric processes, The dataset used in experiments is SLA time series obtained using the WMOP product (Juza et al. 2016). The spatial resolution of our data is 0.05% and the temporal resolution h = 1 day. We use the data from January 2009 to December 2014 as training data and we tested our approach on the last month of the year 2014. The considered region is located on south Mallorca (2.5% to 4.25% E, 37 25 N to 39.5 N). Finally, and in order to identify ROM, our observations are mapped to a low dimensional space using a projection matrix defined offline using a PCA as follow r, M(xx) Re with & 15 which amounts capture 92% of the total variance. The proposed framework is applied simply by replacing X by the PCA components r,. Please refere to the appendix for a generalization of the proposed approache in the case of ROM.
We report forecasting performance for our model and include com- parison with analog methods (AF), Sparse regression (SR). LSTM (RNN) and neural ODE setting (Latent-ODE) in Tab. 2 (The re- sults of the neural networks based models were averaged over 5 runs). Regarding the proposed NbedDyn model we consider an augmented latent space with dæ 60. Our model clearly outperforms the three benchmarked schemes with very significant gain for the forecasting performance at one day (relative gain greater than 90 %) and two days (relative gain greater than 90 %). For a 4-day-ahead forecasting, our model still outperforms the other ones though the gain is lower (relative gain of 40%). Finally, and in order to illustrate the influence of adding extra dimensions to define an augmented latent space on real data, we also tested the proposed NbedDyn model directly on the PCA space (dg = & = 15) this model is referred to as NbedDynZERO and the influence of the latent components is clear from the results in Tab. 2 witch allows a relative gain up to 90 % with respect to the same model directly on the PCA space. We let the reader refer to the Supplementary Material for more detailed analysis of these experiments, including visual comparisons of the forecasts.
Model
to
4h
RMSE
0.036
0.049
0.067
AF
96.9716
SR
Com
RMSE
99:428
0014
97,638
90.918
0.837
Laient ODE
RMSE
D.000 1 0.05
0.39%
97.30%
0,031 00631
0.36%
D.000 0.000
0.55%
RNN
Com
RMSE
96 9955 4: 0.4005
0.026 0.003
US 20%
0,008
= 1.73%
0.007
74.97% 4 5.75%
IU063 0.016
NeedDynZERO
RMSE
0.016 0.0
0,023 4:
0.0G8 0.0
99.645
NoolDDnn
RMSE
0,002 0.0003
0.000 0.001
[U0O0 &.00LL 90.01'5 + 0.04%
Com
96 09.9 0.001755 %
99.91% = 0.01
Table 2: SLA Forecasting performance on the test set of data-driven models: RMSE and correlation coefficients for different forecasting time steps.
DISCUSSION
In this work, we address the data-driven identification of latent dynamics for systems which are only partially observed, i.e. when some components of the system of interest are never observed. The reported forecasting performance for Lorenz-63 dynamics is in line with the forecasting performance of otate-of-the-art learning- beaseng-bbssed approaches for norse-free and obbly-observed setting. This is of key interest for real-world applications, where observing systems most often monitor only some components of the underlying dynamics. As typical example, the SLA forecasting experiment

---

Under review as a conference paper at ICLR 2020
clearly motivates the proposed framework in the context of ocean dynamics for which neither in situ nor satellite observing systems can provide direct observations for all state variables (e.g., subsurface velocities, fine-scale sea surface currents).
We may also further discuss how the proposed framework relates to ofate-of-the-ar dynamical sys- tem theory approaches. Most of these approaches rely on delay embedding, as Takens's theorem states the existence of a delay embedding in which the topological properties of the hidden dynami- cal system are equivalent to those of the true systems up to a diffeomorphic mapping. Hence, state- of-the-art approaches typically combine the selection of a delay embedding representation within classic regression models to represent the one-step-ahead mapping in the considered embedding. Here, we consider latent dynamics governed by an unknown ODE (4) but we do not explicitly state the latent space. This however implicit our forecasting framework. By construction, the con- sidered forecasting model relies on the integration of the learnt ODE 4 from an initial condition given as the solution of minimization (7).
Let us consider the following embedding & such that:
T
vb ((Xe)to:T) arg min min (urher
Σ
||x
G ( (u=- + Allu, (o,((u,1)11"
Given this embedding, the resulting one-step-ahead forecasting for the observed variable may writ- ten as:
XT+1 = G (F (+b ((x))==t;r))
(10)
Hence, s defines delay embedding representation implicitly stated through minimization 7 In this embedding. the dynamics of the observed system x is governed by the composition of observa- tion operator G and forecasting operator &, r. Regarding the literature Koopman operator theory, most approaches rely on the explicit identification of eigenfunctions and eigenvalues of the Koop- man operator (Koopman 1931; Brunton et al., (2016a] Tu et al., (2014). Our framework relates the identification of the infinitesimal generator To of the one-parameter subgroup defined by Koopman operator through the ODE representation 4 By construction, the Koopman operator associated with the identified operator fü is also diagonalizable, such that the identification of infinitesimal generator fg provides an implicit decomposition of the Koopman operator of the underlying and unknown dynamical system onto the eigenbasis of the learnt latent dynamics governed by ODE (4)
Future work will further explore methodological aspects, especially the application high- dimensional and stochastic systems. In the considered framework, when considering ROM of the SLA experiment, the projection from the observations space the low dimensional space was car- ried using simple PCA projection. Although for the geociences community, using PCA to reduce the dimensionality motivated by the Galerkin derivation of reduced order models from complex high dimensional governing partial differential equations (Holmes et al. 2012). using auto-encoders have shown promising results in discovering optimal coordinates when trained jointly with a dynamical system. The combination of the proposed framework with the variational setting considered in the Latent-ODE model (Chen et al. (0188) also appears as an interesting direction for future work. The extension to stochastic systems through the identification of Stochastic ODE is also of key in- terest, for instance for future applications of the proposed framework to geophysical random flows, especially to the simulation and forecasting of ocean-atmosphere dynamics in which stochastic com- ponents naturally arise (Chapron et al. 2018).
REFERENCES
Henry D. Abarbanel. Modeling Chaos, pp. 95-114. Springer New York, New York, NY, 1996a. doi: 10.
ISBN 978-1-46122-076-44
URL sattps://dc. /rg/
10. 1007/978 11778-1--611--- 07600763444 4_6
Henry D. I. Abarbanel. Choosing Time Delays, pp. 25-37, Springer New York, New York, NY, 1996b. ISBN 978-1-4611-0763-44, doi: 10.1007/978-1144612077766
URL URLhttps:///ol
Henry D. I. Abarbanel. Choosing the Dimension of Reconstructed Phase Space, pp. 39-67. Springer New York, New York, NY. 1996c, ISBN 978- 1-4612-0763-4. doi: 10.1007/978. 10.1007/97--1-4411222
/ / ott.://ee////loooooo//////////11 4.
10

---

Under review as a conference paper at ICLR 2020
Henry D. I. Abarbanel and Upmanu Lall. Nonlinear dynamics of the great salt lake: system iden- tification and prediction. Climate Dynamics, 12(4))287-22) Mar 1996. ISSN 1432-0894. doi: 10.1007/BF00229500. URL/attpa://aii.oom 100//1111..00.00000 1
10.
Peter Benner, Volker Mehrmann, and Danny C Sorensen. Dimension reduction of large-scale sys- lems, volume 45. Springer, 2005.
Steven I. Brunton, Bingni W Brunton, Joshua L. Proctor, and Nathan Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PloS one, 11(2):0150171, 2016a.
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the Na- tional Academy of Sciences, 113(15):3332-3955 April 2016b. ISSN 0027-8424, 1091-6490. doi: 1..77//paas1151334111 URL http: /twp:://wwww.. hwwp pnas. org/ orgg/oooup/6b/11101111 pnas 1517384113
Stéphane Calmant, Frédérique Scyler, and Jean François Cretaux. Monitoring continental sur- face waters by satellite altimetry. Surveys in Geophysics, 29(4):247-269, Oct 2008. ISSN
1573-0956. doi: 10.1077/107712-008-905555 s 1107122008-9051-11
URL https:://cc.org/////// 1007/
Kathleen Champion, Bethany Lusch, Nathan Kutz, and Steven I. Brunton, Data-driven discovery of coordinates and governing equations. arXiv preprint 1oXiv=190.002/07, 2011
Bertrand Chapron, Pierre Dérian, Etienne Mémin, and Valentin Resseguier. Large-scale flows under
location uncertainty: a consistent stochastic framework. Quarterly Journal of the Royal Meteor-- logical Society, 144(710):551226) 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ-
ential equations. In Advances in Neural Information Processing Systems, pp. 657 6571-6583, 2018.
Earl A Coddington and Norman Levinson. Theory of ordinary differential equations. Tata McGraw-
Hill Education, 1955.
Emilien Dupont, Amaud Doucet, and Yee Whye Teh. Augmented neural odes. arXiv preprint 1rX1v:190..01661, 2019.
R. Fablet, S. Ouala, and C. Herzet. Bilinear residual neural network for the identification and forecasting of geophysical dynamics. In 2018 26th European Signal Processing Conference (EU- SIPCO), pp. 1477-1481, Sep. 2018. doi: 10.239 10/EE9ICCOOOOO000 201.8553999.
Zoubin Ghahramani and Sam T Roweis. Learning nonlinear dynamical systems using em algo- rithm. In Advances in neural information processing systems, pp. 431-437, 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image
Recognition. arXiv:1512.03388 [cs], december 2015. URL http::/erxxv.org//aae 1511..0388
arXiv: 1512.03385.
A. C. Hindmarsh. ODEPACK, a systematized collection of ODE solvers. IMACS Transactions on Scientific Computation, 1:55-64, 1983.
Philip Holmes, John L. Lumley, Gahl Berkooz, and Clarence W. Rowley. Galerkin projection, pp. 106-129. Cambridge Monographs on Mechanics. Cambridge University Press, 2 edition, 2012. doi: 10.1017/CB09778511199977
Jinhee Jeong and Fazle Hussain. On the identification a vortex, Journal of Fluid Mechanics, 285: 69-94, 1995. doi: 10.1017/500221 12095000462.
M. Juza, B. Mourre, L. Renault, S. Gómara, K. Sebastián, S. Lora, J. P. Beltran, B. Frontera, B. Ga- rau, C. Troupin, M. Toreer, E. Heslop, B. Casas, R. Escudier, G. Vizoso, and J. Tintoré. Socib operational ocean forecasting system and multi plauti- validation in the western mediterranean sea. Journal of Operational Oceanography, 9ssp 116, 2016. doi: 10..086//75587 21117700.00101110
2015.1117764. 10
2015.

---

Under review as a conference paper at ICLR 2020
B. O. Koopman. Hamiltonian systems and transformations in hilbert space. Proceedings of the National Academy of Sciences of the United States of America, 17(5):315-35 318, 1931. ISSN
00278424. BRLLE:/:///w/ jstor. wwtpstaalleeeeaaeteeeeeee //611
Tjalling C. Koopmans. Identification problems in economic model construction. Econometrica, 17 (2):125-144) 144125144, 1949. ISSN 00109082 14680262. URL http: www. jator. wwt....oro//tall.....ooo 1905689
Rahul G. Krishnan, Uri Shalit, and David Sontag. Structured Inference Networks for Nonlinear State Space Models. /0ti:/669..990 fcs, stat], September 2016. URL http:://axiv..
org/abs/ /aabs/1609.. 09869
arXiv: 1609.09869.
Tze Leung Lai and Ching Zong Wei. Least squares estimates stochastic regression models with applications to identification and control of dynamic systems. The Annals of Statistics, 10(1):
154-166, 1982. ISSN 00905364. URL http: www. http:://wwww..oo....... jtg.or.rg/goo.......oo..o... wttpo//.rg//ttaae///44555//
Redouane Lguensat, Pierre Tandeo, Pierre Ailliot, Manuel Pulido. and Ronan Fablet. The Analog Data Assimilation. Monthly Weather Review, aug 2017. ISSN 0027-0644, 1520-0493. doi: URL
10. 10.1175/MWR-D--1---05
117175777777 1 ametsso..ooo//oo...oooooo
MWR 1---D-16666 16- 0441144
Edward N. Lorenz. Deterministic Nonperiodic Flow. Journal of the Atmospheric Sciences, 20(2):
130-144 141, March 1963. ISSN 0022-4928. doi: 10. (...77711520000011116))000021111
2. URL httt:::tt:::oraaaaaaae
amet http:////aaaaaaaaaoo/oo/////////
02083C0130: DNF 332 00. C0;2
Peter Lynch and Xiang- Yu Huang. Initialization, pp. 241-260. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010. ISBN 000---3-----77-0---1-0 doi: 10.1007/978-3-540-7444 //dei. 10 0007/978 3- 540- 00000-7773-22
URL https:
Davide Mattera and Simon Haykin. Advances in kernel methods. chapter Support Vector Ma- chines for Dynamic Reconstruction of Chaotic System, pp. 211-241. MIT Press, Cambridge, MA, USA, 1999. ISBN 0-262-19416-3 URL [ttLhhtt://l.... acm. ortgciiaattteoooooooooooooo seggciitaioos cfm?id= 299094 299106
Piotr Mirowski and Yann LeCun. Dynamic factor graphs for time series modeling. In Joint Eu- ropean Conference on Machine Learning and Knowledge Discovery in Databases, pp. 128-143. Springer, 2009.
Klaus-Robert Müller, A Smola, Gunnar Rätsch, Schölkopf, Jens Kohlmorgen, and Vladimir Vapnik. Using support vector machines for time series prediction. Advances kernel meth- ods support vector learning, pp. 243-254, 1999.
Duong Nguyen, Said Ouala, Lucas Drumetz, and Ronan Fablet. Em-like learning chaotic dynamics from noisy and partial observations. SciRate, Mar. 2019. URL
com/
arxiv/1903. 10335
S. Ouala, A. Pascual, and R. Fablet. Residual integration neural network. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3622- 3626, May 2019. doi: 10. N0.1100/CCASSPP2099.8889 86.844777
Johan Paduart, Lieve Lauwers, Jan Swevers, Kris Smolders. Johan Schoukens, and Rik Pin- telon. Identification of nonlinear systems using Polynomial Nonlinear State Space mod- els. Automatica, 46(4):647-656, April 2010. ISSN 0005-1098.
doi: 10.1016/j.automall 2010.01.001. URL http:/ http:://wwe /www. www.::sseinneeer....cc..mmm..eee http://////ccce///e/e/iee//eee/e SSC05109810000211X
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Multistep neural networks for data- driven discovery of nonlinear dynamical systems. arXiv preprint arXiv:1801.012366 2018.
Clarence W Rowley, Tim Colonius, and Richard M Murray. Model reduction for compressible flows using pod and galerkin projection. Physica D: Nonlinear Phenomena,
2004.
12

---

Under review as a conference paper at ICLR 2020
Tim Sauer, James A. Yorke, and Martin Casdagli. Embedology. Journal of Statistical Physies, 65 (3):579 616, Nov 1991. ISSN 1572-9613. doi: 10.1007/BF01005374.. URL https::/doi. 10. 111000000000/00055555555300
Houshang Sohrab. Basic real analysis, volume 231. Springer, 2003.
Julien Clinton Sprott. Chaos and Time-Series Analysis. Oxford University Press, Inc., New York, NY, USA, 2003. ISBN 0198508409.
Floris Takens. Detecting strange attractors in turbulence. In David Rand and Lai-Sang Young (eds.), Dynamical Systems and Turbulence. Warwick 1980, pp. 366-381, Berlin, Heidelberg, 1981. Springer Berlin Heidelberg. ISBN 978-3-540-38845-3
Jonathan Tu, Clarence W Rowley, Dirk M Luchtenburg. Steven I. Brunton, and J Nathan Kutz. On dynamic mode decomposition: Theory and applications. Journal of Computational Dynamics, (2):391-421, 2014.
Eric A Wan. Time series prediction by using connectionist network with internal delay lines. In SANTA FE INSTITUTE STUDIES IN THE SCIENCES OF COMPLEXIYY-RROEEDING VOLUME,, volume 15, pp. 195-195. Kadison-Wesleye publishing co, 1993.
Jack Wang, Aaron Hertzmann, and David J Fleet. Gaussian process dynamical models. In Advances in neural information processing systems, pp. 1441-1448, 2006.
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent-space physies: Towards learning the temporal evolution of fluid flow, 2018.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural ordi- nary differential equations. arXiv preprint 1rXiv:1907.12990 2019.
13

---

Under review as a conference paper at ICLR 2020
APPENDIX
A PROOF OF PROPOSITION
This proposition be easily extended to any observation function that doesn't form embedding of the initial unobserved ODE. However, for the sake of simplicity, we will consider the example given in Eq. (
Lets suppose a smooth ODE in the observation space that governs the time evolution of x from Eq. TT.
[2 f(x() X1o
(11)
This ODE generates a flow = (x(x)).
Since our observation operator not one-to-one, we can assume the existence of some f, f1. l2 where Real @ ( )) = Real(()(s...) with Real(s,,) Real(st,) (& is the flow generated by the unobserved ODE illustrated in Eq. ①.) Projecting this equality to the observation space leads to (x(,,)) = (xx,) with x,, Xts-
Since the above ODE is smooth (or continuously differentiable), we can show that f is locally Lipschitz on any interval containing to (Sohrab) (2003) which garentees by Picard's Existence The- orem the existance of unique solution (Coddington & Levinson] 1955). Formally, for the times 1,2,, (x,),, = (x(x,, if and only Xt, X/2' This contradicts the assumption that X1, A Xt2 and thus, there is no existence of such that (e,,((;(s..)) - Real((g(s..)) with (eal(s;,) ) (8a)(
B AUGMENTED REDUCED ORDER MODELING
With view to providing general formulation which also accounts for reduced-order modeling (ROM) (Champion al.) 2019). we assume that the observations x; are fully characterized by lower-dimensional state M(x,) E R* with M global coordinates chart, mapping each obser- vation coordinates M(x,) E R* Typical examples for operator M includes the identity as in the equations of section 3. In ROM, M may include PCA projections (Benner et al. (2055) for Galerkin derivation of ROM for complex high dimensional partial differential equations (Rowley
al. 2004). Formally, the augmented state space is formulated as
u.² = [M(x)) x2,1
(12)
with the corresponding SSM
ú; fo(u:) x; (GM)))
(13)
Based on the above formulation, we can state the optimization problem stated Eq. 5 as:
θ arg min min
|xt MM (#0. (u,-)))) 12
(2,)
(14)
Subject to
GG
with the corresponding non constrained formulation
T
min min (
X
M- (1o,, (u--1))) 12 2 + Allu, - 1))
(15)
14

---

Under review as a conference paper at ICLR 2020
Finally, when considering forecasting applications of new observations, restate the equation in the context ROM as follow
T+8T
27 arg min min
Σ x MM (u. 11 A/[nK Po.c(uf-))ll (16)
27 {x) heet
1=++1
C LIST OF SYMBOLS
S Unobserved state governing the underlying dynamics in R
X Observed component of R", usually related to some components
M the case ROM (x high dimensional), M a projection operator from RR R with >>>>> k
n
Augmented latent space of dimension de which consists on a concatenation of an observed components (M(x)) and latent components
z Latent component concatenated to MMx) to form the augmented state
f()
Unknown dynamical system governing the unobserved state s
Φc(.)
Flow map of the Unknown dynamical system f(.)
fo(.) Dynamical system governing the augmented state 11
(r,)(.) Flow map of the dynamical system fa(-)
D DIMENSIONALITY ANALYSIS OF THE NBEDDYN MODEL.
One of the Key parameters the proposed model is the dimension of the latent space. Despite the fact that it is extremely challenging to get prior idea of the dimension of the model in the case of real data experiments, one can analyze the spawned manifold of the learnt latent states get an idea of the true dimension of the underlying model (true here stands for sufficient dimension of the latent space). The idea here is compute the modulus of the eigenvalues of the Jacobian matrix for each input of the training data. An eigenvalue does not influence the temporal evolution of the latent state has a modulus that tend zero. The number of non-zero eigenvalues can then be seen sufficient dimension of the latent space.
Regarding the identification of an ODE model governing the first state variable the Lorenz 63 model, Fig. iillustrates the eigenvalues of the Jacobian matrix and their modulus for dimension of the latent space de = 6. Interestingly, only 3 eigenvalues have non-zero modulus and are effectively influencing the underlying dynamics. This result shows that as sufficient dimension to identify an ODE model governing the first state of the Lorenz 63 system which is the same dimension the true Lorenz 63 model.
a 3 dimensional latent- satent-space
15

---

Under review as a conference paper at ICLR 2020
(a) Eigenvalues real part.
(b) Eigenvalues imaginary part.
Moan romaliad 12 nort
75
0.0
(c) Eigenvalues modulus.
Figure 3: Analysis of the eigenvalues of the NbedDyn model Jacobian matrix.: Lorenz-63 case- study with dg = 6
(a) Eigenvalues real part.
Eigenvalues imaginary part.
Moon nornakdd L2 nom
R5
0.2
06
E Eiemoouee
(c) Eigenvalues modulus.
Figure 4: Analysis the eigenvalues of the NbedDyn model Jacobian matrix.: Sea Level Anomaly case-study with dg 60
16

---

Under review as a conference paper at ICLR 2020
The analysis of the eigenvalues of the Sea Level Anomaly model in the other hand are not straight- forward as in the case of the Lorenz model since we do not have any idea on the analytical form of the underlying dynamical model. Fig. 4 iillustrates that using 60 dimensional latent space for the Nye model, only 50 eigenvalues have non-zero modulus and thus, are effectively influencing the underlying dynamics. The conclusion in this case is that the observed SLA data evolve in 50 dimensional latent space parametrised by the dynamical model fo.
E ADDITIONAL FIGURES OF THE LORENZ 63 EXPERIMENT
We illustrate the forecasting performance of the tested models for the Lorenz-63 experiment through an example of forecasted trajectories in Fig. [ Our model with dg = leads to a trajectory similar to the true up to 7 Lyapunov times, when the best alternative approach diverge from the true trajectory beyond 4 Lyapunov times.
15
M
A
M
w
W
100
Lorenz
M JMW
MW
MM
VVVVVVVV
m
W
0-
Figure 5: Generated time series of the proposed models. Given the same initial condition, generated time series of 1000 time steps.
An other interesting experiment is to find the initial condition for new observation data. This issue addressed as presented in section 3 as follow. Given a new noisy and partial observation sequence (Fig. (6). first look for potential initial condition in the inferred training latent state sequence. This intial condition is then optimized using the cost function described by equation (7) to minimize the forecasting error of the new observation sequence.
17

---

Under review as a conference paper at ICLR 2020
16
Nyedlyn forecast
5
-5
-15
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Lorenz time
Figure 6: Forecasted Lorenz 63 state sequence given noisy and partial observations: Given noisy and partial observations, our model optimizes equation (7) to infer an initial condition that minimize the forecasting of the observations.
F ADDITIONAL FIGURES OF THE SEA LEVEL ANOMALY EXPERIMENT
Forecasted states of the Sea Level Anomaly are illustrated in Fig. and The visual analysis of the forecasted SLA states emphasize the relevance of the proposed NbedDyn model. While state of the art approaches generally overestimate the time evolution of some structures such as eddies, our model is the only one to give near perfect forecasting up to 4 days.
PCW
Rescane DCA your (aten--O0)
Foncriae DCA dass (DNN)
PCA cist
Figure 7: Generated time series of the proposed models for the forecasting of the SLA dynamics. Given the same initial condition, generated time series of 150 days.
18

---

Under review as a conference paper at ICLR 2020
Day
forcadt
Foncast
Thu state
0.50
Thue 530s
852
Thue Fal2
Trae db3s
0.2
0.25
0.2
0.25
-025
-323
-0.25
2.5
12:0
125
2.5
50
100
125
5/3
1.5
10.0
125
7.5
12.
E07 moce
20F
mode
EDF
moce
507 1932
(a) Analog forecasting.
Day forecast
Day 2 forecast
Day forecast
Dty foroeast
borestt
Forcett
lereart Tres
Fercest
Volue
Value
38
0.-
-32
-12
-62
-00
(b) Sparse regression.
forecast
Foreast
Pereadt
Fercss:
Transtale
Tha sttla
6.4
Trae stak
Trae stal
02
0.2-
38
-12
-02
-0.2
10
10
10
14
ED0 mice
E00 moce
mode
50 T150
(c) Latent-ODE.
Day forecast
Day 2 foreasst
Day: fareastt
Day forecast
Foreust Traas
Forczst
6,4-
Pertast
Fercust
Trie stal
92
3.0
31
6.0
-3.2
-0.0
10
14
10
14
30F
(d) RNN.
forcces!
Fercast
Farcast
Trasstate
244
True stale
644
Trae stalk
644
Trae state
2.2
32
C2:
0.0-
E0Frrce
Ed8 noce
20F made
805 vod:
(e) NhedDynZERO.
forocast
foreaat
Day
forocast
6.6
Trassttt
True 2382
0.4
Tree stak
0,50
Trae state
Value
2.2
022
0.25
0.0
0.23
-3
-32
10
10
18
14
EDF mice
E04 moct
made
20F Ted0
(I) NbedDyn.
Figure Forecasted EOF components of the proposed models.
19

---

Under review as a conference paper at ICLR 2020
432
(a) Ground truth.
500
402
(b) Analogs forecasting.
duy buneas
day boweas
(c) Sparse regression.
2.55
431
600
442
600
(d) Latent-ODE.
600
RNN.
LaW Aracert
2dW DOc
2az Secutt
(f) NbedDynZERO.
(g) NbedDyn forecast.
Figure 9: Forecasted SLA states of the proposed models.
G NEURAL NETWORKS HYPERPARAMETERS
The neural-netwoott parametrization for the dynamical model operator fo is a bilinear architecture as presented in (Fablet al. 2018). The idea of this model is to embed second order polynomial repre-
20

---

Under review as a conference paper at ICLR 2020
sentations of the input states by combining conneece layers and an element- product. This bilinear layer are then concatenated to classical fully connected layers to perform regression. This architecture incorporates physical knowledge the dynamics since physical dynamical systems often involve bilinear non-linearities. Among classic physical models, one may cite for instance advection-diifusion dynamies shallow water equation.
This model is than integrated numerically to compute the flow-map Φg using fourth order Runge Kutta scheme. In practice, the integration scheme is implemented ResNet with four layers sharing the dynamical model.
G.1 LORENZ 63 EXPERIMENTS HYPERPARA METERS
Parameter
Value
Parameter
Value
Latent dimension
4
Number of LSTM layers
Hidden size
15
Hidden size
10
RNN hidden size
100
Sequence length
30
Learning rate
0.01
Learning rate
0.001
Optimizer
Adam
Optimizer Training data
Adam
Training data
4000
4000
Table 4: Latent-ODE parameters in the Lorenz 63 Experiment, please refer (Chen et al. 2018) for more details.
Table 3: RNN parameters in the Lorenz 63 Experiment.
Parameter
Value
Augmented Latent dimension
6
Number of bilinear layers
6
Number of linear layers
6
Integration scheme
Runge Kutta 4
Learning rate
0.001
Optimizer Training data
Adam
4000
Table NbedDyn parameters the Lorenz 63 Experiment, please refer to (Fablet al. 2018 for
more details.
G.2 SLA EXPERIMENTS HYPERPARAMETERS
Parameter
Value
Parameter
Value
Latent dimension
60
Number of LSTM layers
5
Hidden size
70
Hidden size Sequence length
20
RNN hidden size
200
40
Learning rate
0.01
Learning rate
0.001
Optimizer
Adam
Optimizer Training data
Adam
Training data
2000
2000
Table 7: Latent- parameters in the SLA Experiment, please refer to (Chen et al. 2018) for more details.
Table 6: RNN parameters in the SLA Exper-
iment.
21

---

Under review as a conference paper at ICLR 2020
Parameter
Value
Augmented Latent dimension
60
Number of bilinear layers
60
Number of linear layers
60
Integration scheme
Runge Kutta 4
Learning rate
0.001
Optimizer
Adam
Training data
2000
Table 8: NbedDyn parameters in the SLA Experiment, please refer to (Fablet et al. 2018) for more details.
H SCOOP AND LIMITATIONS
Constraining limit cycles The proposed augmented ODE formulation does suppose any prior knowledge on the underlying dynamics responsible for the temporal evolution of the observations. This can lead in some cases (especially when working on chaotic dynamics) output dynamical representation that has several attracting regions in addition to the one leading to the observations limit cycle. This can lead inappropriate results when trying to find initial condition that fore- casts given observation sequence. The Idea of using the manifold spanned by the augmented training data allows bypass this issue but we believe that adding additional constraints (energy preserving constraints, known symmetries in the models ...etc.) can significantly improve the qual- ity of the learnt dynamical models.
22

---

Under review as a conference paper at ICLR 2020
1
CODE SAMPLE
import libs from venerate_date import generate_data import numpy as np
import torch
from torch. autograd import Variable seed 0
np random. seed (seed)
torch, manual_seed (seed)
# Generate data
class GD:
model = 'Lorenz-63"
class parameters:
sigma 10.0
rho 28.0
beta 8.0/3
dt dt.innegration = 0,01 integration time
nb-loop-data 50.01 size of the catalog
test_ samples = 1000
run the dala generation
catalog. xt, yo generate_data (GD)
X_test
catalog Inalogs[catalon analogs shape[0]-GD. rest.samples:
X-train = catalog analogs [: catalog analogs shape[0]-Gr test-samples ,:1]
Grad np. gradient train [::0]). reshape X train (01
/GD. dt integrationn
Variable (torch from_numpy (a_uayXX_train))) ) float ()) Variable (torch from from_numpy Grad_t), float ())
data_size = X_train shape
#neural net params
params = {
params transition layers ']=
params [ bb linear layers ]=6
params| hom_hidden_liieaat
params[ ''dim-input" '1=1
params 'dim_latent"
params dddim'ddmmmuuuuuu "]=params dimm params lim_latent')
"arams('dtiittegration
GD. dt.integration
Dynamical model
class FC net on Module):
def ..init__ params):
super(FC.net,
y y-aug up. random. uniform ( coaooe=dattazziiie paaamms latent II)
self y-aug = torch nn. Parameter (torch. from fwom. numpy (yg) float self linearCell
torch nn Linear Linear (params| 'dim_ jummoutput']], params dim fim_hidden hidden__inearr' "])
self BlinearCell]
torch. mn. ModuleList()
[torch, mn. Linear (aaamms [ dim dim-output'l, 1)) for i in range (params bi linear layers DDI self. BlinearCell2 torch mn. ModuleList
[torch. nn Linear (params dim dim_output D) for in vange(paranl bi bi.linear_layerr DDI) augmented params [ dim.hidden.linear self trans torch. nn. ModuleList
params linerr '
23

---

Under review as a conference paper at ICLR 2020
[torch.. mn Linear (augmented.size, params| 'iimooutput") I)])
self transs extend
[torch. mn. Linear (params dim output'] 'I. params dim output J
for in r1nge(1, params transition layers'DD) self. outputLayer
torch. on Linear (params dim dim_ootppu")] params dim-output "]
def forward (self, inp. dt): if inp - aug-inp = torch cat ((inp, self, y-aug) dim=1)
dim fimm-input' ]
else:
aug inp inp BP_outp = Variable (torch, zeros ((aug-inp. size (10),
I ('t..linear 1)))
L.outp = self. lelllnnerelll (aug inp) I)):
for in range fag.
self wtw.eeaaCelllli1(cag.nnnool [linearCelll aug 00]
aug-vect torch torch.cat carr((_ou(pp, BP_ou(p), dim=l) 1)):
for in range (paramss"transitioon
aug-vect (self. transLayers [i] (aug-vect)
grad = self outputLayer (aug vect) model = return grad, aug inp
compute flow RK4
class INT INT.net an. Module):
def --init_.. (self, params):
super(INT (INT _net. self).
self add module( ( Dyn net ', FC net paramss) self. Dyn net model
def forward (elf, inp. dt):
kl. aug-inp
self (ynp dt)
inp_k2 = aug inp 0.5 params| dt ott_inegrraton ']*kl
k2, tmp = self Dyn.net ,t)
inp_k3 = aug_inp 0.5* ' dt_integration ']*k2
k3, tmp = self. Dyn_net dt)
inp_k4 = aug-inp params| params['dt_intess "t_itdddearttiiiii 'J*
k4, tmp = self. Dyn_net
pred aug-inp +d+=((1++++2++++43+4444
return pred, kl, inp. aug inp
#Instanciate the model
modelRINN = INT net params)
criterion torch
MSELoss (reduction= elementwise.mean" (eedeettiosddeeeeeveeeeeeeeee
optimizer = torch. optim Adam addammooeel. model. parameters ())
#Pretraining fit the gradient
params[ ntrain |=03000000,000001
for in range (params ntrain ][[]):
pred. grad. inp. aug inp modelRINN (x, params dt integration '))
lossi criterion [:
2)
loss2 criterion (pred -1 1.:], aug ang_inp [I;,:])
loss 0.9*lossl 1.9 1os1
ptint optimizer zero-grad
loss backward retain graphh-Tearprree optimizer. step
# training
for t in range (params nntrainn I[1]))
pred. grad, inp aug aug-inp modelRINN (x, params '' inttegrgaii 'J)
lossl criterion (pred - -I::] aug oug-inp [ ;:])
loss2 criterion (pred[: -1.1:] aug [1:,1:])
24

---

Under review as a conference paper at ICLR 2020
loss 110.0=loss| 1.0+loss? (t,
optimizer. zero g ()
loss backward retain graph=True) aptimizer.step ()
25


Reviews:
[{"review_id": "rJl7V-W6KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I raised the score from 1 to 3 to acknowledge the authors' consideration for the 2000-2010 literature on learning dynamical systems from partial observations. Unfortunately, the writing is still confusing, some of the claims in the introduction and rebuttal are inexact ([5] does not embed the observations and does work with partially observed environments), and the method lacks originality compared to existing work. Other work that relies on ODE integration and in finding high-dimensional state variables has recently been published and tested on more ambitious datasets, e.g. Ayed et al, \"Learning Dynamical Systems from Partial Observations\", arXiv 2019.\n\n***\n\nTL;DR: relatively well written (if sometimes confusing) paper that reinvents the inference of latent variables in nonlinear dynamical systems that has been published in the 2000s, and that misses an important chunk of literature (and experiments on dynamical systems such as Lorenz-63) from that time.\n\nThis paper proposes an approach for learning dynamical systems from partial observations x, by using an augmented state variable z that follows dynamics that can be described by an ordinary differential equation (ODE) with dynamics f. The authors motivate their work by the problem of dynamical system identification when only partial observations are available. The authors claim that was to date primarily addressed using time-delay embedding, following Takens' theorem. The authors introduce s-dimensional unknown state variables z, dynamical function f (for the ODE on z), flow phi on z, limit cycles on z, observation function H: z -> x that goes from z to n-dimensional observations x, low k-dimensional manifold r (with a map M: x -> r), and state augmentation variable y. The reconstructed state variable u is the concatenation of r and y. One key ingredient of the method is to infer the optimal value of state augmentation variable y during learning (see equations 5 and 6) and inference for forecasting (7); this is not well explained in the abstract and introduction.\n\nI would note that the problem of state space modeling (SSM) and dynamical system identification has been well studied, and the notation and reformulation in this paper is somewhat confusing for those who are used to the notation in SSMs (specifically, expressing the observation approximation as M^{-1}(G(phi(u_{t-1}))). Learning a state-space model involves both learning parameters and inferring the latent states representation (or, in graphical models, the distribution of these latent states) given the parametric models. One approach has been to formulate the state-space model learning by maximum likelihood learning of the model parameters so that the generative model fits observed data x, and this would involve factoring out the distribution of latent states z; the algorithm would rely on Expectation Maximisation, and could involve variational approximations or sampling. While the state space models were hampered by their linearity, several papers in 2000s showed how it is possible to learn nonlinear dynamical models, e.g. [4], [5], [6] and [7] to cite earlier ones. Equations (5) and (6) are similar to the standard equations for a dynamical system expressed in continuous time, with the only difference that the optimisation is with respect to y only, rather than w.r.t. z or u (why not \\tilde z or \\hat z?).\n\nThe paper mentions various initialisation strategies for y (last paragraph of section 3). Why not predict from the past of the observations, like is done in many other similar work?\n\nThe literature review mixes older and newer references. For example, on page 1, I would note that the Takens' theorem has been applied in conjuction with Support Vector Regression as early as 1999 [1][2], and with neural networks in 1993 [3].\n\nMost importantly, the ideas of this paper have already been published in [4] (with architecture constraints on the neural network state-space model), in [5] (with any nonlinear neural network state-space model), in [6] (using Restricted Boltzmann Machines) and in [7] (using Gaussian Process latent variable models).\nThe model is illustrated with experiments on a 2D linear attractor, on the Lorenz-63 attractor. Given the results published in [1] and [2] using SVR on 1D observations of that attractor, and in [5] using a recurrent neural network, I am unconvinced by these results. It seems in particular that the number of training points (around 4000) limits the performance of RNN / LSTM models. The application to Sea Level Anomaly is interesting.\n\nMinor comments:\n\"Unfortunately, When\" (page 1)\nThere is a missing -1 after M in equation (5) and (10)\nIn equation (7), should not the sum go from t=0 to T, as x_t is unknown for t>T?\nWhat is prediction and what is extrapolation on Figure 1?\nThe caption of Fig 1 contains (left)\nThe figures seem squeezed with the captions / titles un-aesthetically wide. \nLabels on Figure 5 in the appendix seem mixed, and red should be the ground truth\n\n[1] Mattera & Haykin (1999) \"Support vector machines for dynamic reconstruction of a chaotic system\"\n[2] Muller, Smola, Ratsch, Scholkopf, Kohlmorgen & Vapnik (1999) \"Using support vector machines for time-series prediction\"\n[3] Wan (1994) \"Time series prediction by using a connectionist network with internal delay lines\"\n[4] Ghahramani, and Roweis (1999) \"Learning nonlinear dynamical systems using an EM algorithm\"\n[5] Mirowski & LeCun (2009) \"Dynamic Factor Graphs for Time Series Modeling\"\n[6] Taylor, Hinton & Roweis (2006) \"Modeling human motion using binary latent variables\"\n[7] Wang, Fleet & Hertzmann (2006) \"Gaussian process dynamical models\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syxv3Mo2YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new deep learning approach based on Takens’s theorem to identify the dynamics of partially observed chaotic systems. In particular, the method augments the state using the solution of an ODE. Experiments on Lorenze-64 dynamics and sea level anomaly demonstrate the advantage of the proposed method over state-of-the-art baselines.\n\n+ The unification of Taken’s embedding theorem and deep learning provides a novel perspective into dynamical systems\n+ Impressive experiment results compared with baselines including RNN and latent ODE\n\n- The proposed method requires knowledge of the underlying dynamic model to solve the ODE, which is not fair for other methods\n- The model is trained using data from the same initial conditions, which is essentially overfitting. The authors should provide experiments for dataset from different initial conditions.\n- The writing is not very clear. For example, how to solve the optimization problem in Eqn (7),  as the augmented states u_{t-1} are unknown? How to find the bijective mapping M for general dynamical systems?\n\nMinor: question mark in section 4 page 6.   Figures 2 plots are difficult to read, pls provide more details in columns and rows.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lvkgDO9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of data-driven identification of latent representations for partially-observed dynamical systems. The proposed method uses an augmented state-space model where the dynamical operator is parameterized using neural networks. Given some training data, the proposed approach jointly estimates the parameters of the model as well as the corresponding unobserved components of the latent space for the training sequences. Experimental results on forecasting using real and synthetic datasets are presented.\n\nOverall I think that the paper makes an interesting contribution. I find very interesting the idea of performing inference on the set of unobserved components in the latent space. The empirical results seem sufficient to me, but I am not familiar with relevant baselines (see below). Please find below some comments and questions.\n\nI am personally not familiar with the literature on this problem, so my assessment might be affected by this. I did not find the paper easy to read and the presentation assumes a lot of previous knowledge. I think that the background and related work section could be more friendly written (considering the ICLR audience).\n\nThe training scheme (described in Section 3) uses one-step-ahead forecasting. The temporal consistency of the unobserved component of the latent space is only loosely enforced with the regularization term in (6). One could train using forecasting with more steps (and only doing inference for the initial y_t of the subsequence), as this is closer to what is used at test time. Do you think this would be helpful for having better accuracy when forecasting more steps?\n\nIt would be good to provide more details on how to build the forecasting operator (implementing 4-th order Runge-Kutta scheme) and what is exactly the bilinear architecture of Fabelt et al.\n\nRegarding the experimental validation, I like that the paper starts with a simple motivating example and moves to more complex cases. Experimental results are convincing to me, as the model is able to recover the performance of other models that do have access to the full state. I am not familiar with the literature so I'm unable to judge whether all relevant baselines are included. \n\nRegarding the Latent-ODE baseline, would results change running with different (larger) dimension for the latent space?\n\nThe paper should cite the work: Ayed, et al. \"Learning Dynamical Systems from Partial Observations.\" arXiv preprint arXiv:1902.11136 (2019). Would this be a relevant baseline to compare to?\n\nIs the training data regularly-sampled? Would the model be robust the irregularly-sampled training data?\n\nThe authors evaluate all methods with one and four step forecasting in the last two experiments. I think that it would be informative to show a wider range of number of steps, to show how performance degrades with longer predictions (more than 4).\n\nFinally, regarding the Modelling Sea Level Anomaly task, all baselines are ran by the authors. It would be informative to also include results of prior art using this dataset, if possible.\n\nOther minor comments:\n\nThe citation format is wrong. Most citations should be using the \\citep command\n\nIn the second paragraph of Section 1, it says: \"Unfortunately, When the\"\n\nIn the caption of figure 2 it says: \"according to thr\"\n\nA few lines before the \"Modelling Sea Level Anomaly\" subsection there's an exclamation sign before the text\"1e-4 for a one-step...\"\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an ODE-based latent variable model, argues that extra unobserved dimensions are necessary in general, and that deterministic encodings are also insufficient in general.  Instead, they optimize the latent representation during training.  They include small-scale experiments showing that their framework beats alternatives.

In my mind, the argument about fixed mappings being inadequate is a fair one, but it misses the fact that the variational inference framework already has several ways to address this shortcoming:
1) The recognition network outputs a distribution over latent values, which in itself does not address this issue, but provides regularization benefits.
2) The recognition network is just a strategy for speeding up inference.  There's no reason you can't just do variational inference or MCMC for inference instead (which is similar to your approach), or do semi-amortized variational inference.

Basically, this paper could have been somewhat convincing as a general exploration of approximate inference strategies in the latent ODE model.  Instead, it provides a lot of philosophical arguments and a small amount of empirical evidence that a particular encoder is insufficient when doing MAP inference.  It also seems like a problem that hyperparameters were copied from Chen et al 2018, but are used in a MAP setting instead of a VAE setting.  Finally, it's not clear how hyperparameters such as the size of the latent dimensions were chosen.
Decision: Reject