Learning Latent Dynamics for Partially-Observed Chaotic Systems

1 INTRODUCTION . Learning the underlying dynamical representation of some observed variables xt ∈ Rn ( where t ∈ { t0 , ... , T } is the temporal sampling time and n the dimension of the observations ) is a key challenge in various scientific fields , including control theory , geoscience , fluid dynamics , economics ; for applications ranging from system identification to forecasting and assimilation issues ( Lai & Wei , 1982 ; Abarbanel & Lall , 1996 ; Jeong & Hussain , 1995 ; Koopmans , 1949 ) . For fully-observed systems , i.e . when the observed variables xt relate to some underlying deterministic states st according to a diffeomorphic mapping possibly corrupted by noise , recent advances ( Brunton et al. , 2016b ; Fablet et al. , 2018 ; Nguyen et al. , 2019 ) have shown that one can identify the governing equations of the dynamics of st from a representative dataset of observations { xti } i . Unfortunately , when the observed variables xt only relate to some but not all the components of underlying states st , these approaches can not apply since no ODE or , more generally , no one-toone mapping defined in the observation space can represent the time evolution of the observations . In this context , Takens ’ s theorem states the conditions under which a delay embedding , formed by lagged versions of the observed variables , guarantees the existence of governing equations in the embedded space ( Takens , 1981 ) . Takens ’ s theorem has motivated a rich literature of machine learning schemes to identify dynamical representations of partially-observed systems using a delay embedding . This comprises both nonparametric schemes based on nearest-neighbors or analogs ( Abarbanel , 1996a ) as well as parametric schemes which include polynomial representations ( Paduart et al. , 2010 ) , neural network models ( Wan , 1993 ) and Support Vector Regression ( SVR ) models ( Mattera & Haykin , 1999 ; Müller et al. , 1999 ) . For all these approaches , the identification of the appropriate delay embedding is a critical issue ( Abarbanel , 1996b ; c ) . From a neural network and machine learning perspective , the inference of a latent space , within a State Space Model ( SSM ) framework , for dynamical systems has motivated a broad literature especially for time series forecasting ( Ghahramani & Roweis , 1999 ; Wang et al. , 2006 ; Mirowski & LeCun , 2009 ; He et al. , 2015 ; Krishnan et al. , 2016 ) . Most of those techniques were introduced in the context of reduced order modeling ( ROM ) to infer low-dimensional manifolds , where the dynamics of the observations can be represented . When considering partially-observed systems , these approaches state this issue as the inference of a ( non-linear ) projection of an input sequence in a latent space where the observations can be modeled . This projection is usually computed in a probabilistic framework using Bayesian filtering techniques . However , recovering the attractor ’ s dynamics using iterative predictions is still an issue for such models since the explicit modeling of latent space as a delay embedding of the observations may limit the expressiveness of the latent states , especially when considering chaotic dynamics . In this work , we show that we do not need to rely explicitly neither on a delay embedding nor on the learning of an inference model to map the observation series to a latent space . We only assume that we may define an augmented space , higher-dimensional than the manifold spanned by the observed variables , where the dynamics of the observations can be fully described by an ODE . Using neural-network representations for the parametrization of the dynamical model , it amounts to jointly learning the governing ODE and reconstructing the augmented latent states for a given observation dataset using a variational framework . We report experiments on linear and chaotic dynamics , which illustrate the relevance of the proposed framework compared to state-of-the-art approaches . We then further discuss the key features of this framework with respect to state-of-theart dynamical systems identification tools such as Koopman operator theory ( Koopman , 1931 ) . 2 BACKGROUND AND RELATED WORK . This section introduces the learning of dynamical representations for partially-observed systems and links this problem to recent advances in machine learning . Let us consider an unobserved state variable s governed by an autonomous system of m differential equations ṡt = f ( st ) . Let us also assume that this system generates a flow Φti ( st0 ) =∫ ti t0 f ( zw ) dw ∈ Rm with trajectories that are asymptotic to a limit-cycle L of dimension d contained in Rm . We further assume that we are provided with a measurement function H that maps our state variable s to our observations xt = H ( st ) ∈ Rn . When considering the data-driven identification of a dynamical mapping that governs some observation data , we first need to evaluate whether the dynamics in the observation space can be described using a smooth1 ODE . Another way to tackle this question is to find the conditions under which the deterministic properties of the unobserved limit-cycle L are preserved in the observation space in Rn such that one can reliably perform forecasts in the observation space . The general condition under which a mapping H preserves the topological properties of the initial limit-cycle involves a differential structure . Assuming that L is a smooth compact differential manifold , the topological properties of L are preserved through a mapping H in Rn if H is one-to-one and is an immersion of L in Rn . Under these conditions our observation mapping is called an embedding ( Sauer et al. , 1991 ) . The simplest example of an embedding involves an identity observation operator H. With this embedding , we have direct access to the state variable s which is governed by a deterministic ODE . This particular case has been widely studied in the literature . Parametric representations have been for decades the most popular models thanks to their simplicity and interpretability ( Paduart et al. , 2010 ) , ( Brunton et al. , 2016b ) . Recently , these approaches have been enriched by neural network and deep learning schemes ( Wiewel et al. , 2018 ) , ( Raissi et al. , 2018 ) . In particular , the link between residual networks ( Chen et al. , 2018 ; Ouala et al. , 2019 ) and numerical integration schemes have opened new research avenues for learning extremely accurate dynamical models even from irregularly-sampled training data . These schemes show greater interpretability and forecasting performance for the datadriven representation of systems governed by an ODE , compared with other state-of-the-art neural networks schemes , including Recurrent Neural Networks ( RNN ) such as LSTM ( Long-Short-Term Memory ) . However , for a wide range of real-world systems , we are never provided with an observation operator that forms an embedding of the unobserved dynamical system . In such situations , we do not have any guarantee on the existence of a smooth ODE that governs the temporal evolution of our observations . From this point of view , the question of finding an appropriate dynamical representa- 1The word smooth here stands for continuously differentiable or C1 . tion of some observed data may not be this straightforward . The fact that our data may come from some unobserved governing equation may restrict the use of the above-mentioned state-of-the-art algorithms . The main difficulty lies in the ability to map observation series to a latent space that provides at least a one-to-one mapping between two successive states . From a geometrical point of view , the time delay theorem ( Takens , 1981 ) provides a way to build a latent space that preserves the topological properties of the true ( unobserved ) dynamics limit-cycle . A generalization of this theorem ( Sauer et al. , 1991 ) shows that one can reconstruct topologically similar limit-cycles using any appropriate smooth composition map of the observations . The derivation of a dynamical system from such representations however encounters large disparities since no explicit relationships between the defined phase space and an ODE formulation have been clearly made . Classical state-of-the-art techniques such as polynomial representations ( Brunton et al. , 2016b ) and K-Nearest Neighbors ( KNN ) ( Lguensat et al. , 2017 ) algorithms were proposed but they often fail to achieve both accurate short-term forecasting performance and long-term topologically similar reconstructed limit-cycles ( see experiments for an illustration ) . We may also point out that the limitation of ODE-based representation in deep learning architecture has also been pointed out recently in ( Dupont et al. , 2019 ; Zhang et al. , 2019 ) for classification issues . As ODE-derived trajectories do not intersect , it may limit the ability of neural ODE representations to reach relevant classification performance in a given feature space . To address this issue , ( Dupont et al. , 2019 ) and ( Zhang et al. , 2019 ) propose to consider an augmented state , simply by augmenting the observed state by a number of zeros to create a high-dimensional space in which an ODE representation can be identified . Such a strategy can not apply to time series modeling as successive augmented states can not be forced to zero for some dimensions . Advances in the inference of latent spaces in state space models was introduced essentially , from a dynamical systems perspective , to retrieve low-dimensional manifolds , where the dynamics of the system evolve . When applied to partially-observed systems , the latent variables are typically inferred from a sequence of observations through a parametric modeling of the posterior distribution as in ( He et al. , 2015 ; Krishnan et al. , 2016 ; Chen et al. , 2018 ) or through marginalization with model constraints as in ( Ghahramani & Roweis , 1999 ; Wang et al. , 2006 ) However , such models often fail in accounting for long term patterns ( as shown in the experiments ) . This is due to the fact that the latent space is constrained to be a non-linear projection of a sequence of observations , which limits the expressiveness of the dynamical model . Interestingly , ( Mirowski & LeCun , 2009 ) does not involve the learning of an inference model as the reconstruction of the latent states is solved as gradient-based minimization of the dynamical prior w.r.t . an observation series . However , the dynamical prior relies on an explicit delay embedding as the dynamics of the latent state depend both on the previous latent state and on a delay embedding of the observations . In this work , we address the identification of a latent embedding , associated with an ODE representation , for partially-observed systems . The core idea of this work is to infer an augmented latent space , governed by an ODE , which fully explain the observed time series and their dynamics . In contrast to previous work ( He et al. , 2015 ; Krishnan et al. , 2016 ; Chen et al. , 2018 ; Ghahramani & Roweis , 1999 ; Wang et al. , 2006 ) , we do not exploit either a delay embedding or an explicit modeling of the inference model ( i.e. , the reconstruction of the latent states given the observed time series ) . As such , our scheme only involves the selection of the class of ODEs of interest . The expected benefits are as follows : ( i ) our model ensures the existance of a latent embedding associated with an ODE , which may not be guaranteed when considering a parametric inference model and/or a delay embedding , ( ii ) our model reduces the complexity of the overall scheme to the complexity of the ODE representation , ( iii ) our model guarantees the consistency of the reconstructed latent states w.r.t . the learnt ODE . Our experiments suggest that this may be of key importance to jointly address short-term forecasting performance and long-term chaotic patterns .

Summary: The paper addresses the problem of data-driven identification of latent representations for partially-observed dynamical systems. The proposed method uses an augmented state-space model where the dynamical operator is parameterized using neural networks. Given some training data, the proposed approach jointly estimates the parameters of the model as well as the corresponding unobserved components of the latent space for the training sequences. Experimental results on forecasting using real and synthetic datasets are presented.