Learning Stable Deep Dynamics Models for Partially Observed or Delayed Dynamical Systems

1 Introduction . In this paper , we address the task of learning stable , partially observed , continuous-time dynamical systems from data . More specifically , given access to a data set { ( t0 , yl0 ) , . . . , ( tN , ylN ) } Ll=1 of noisy , partial observations collected along L trajectories of an unknown , stable dynamical system , ż ( t ) = g ( z ( t ) ) , z ( t ) ∈ Rm x ( t ) = h ( z ( t ) ) , x ( t ) ∈ Rn , m ≥ n , yi = x ( ti ) + i , i i.i.d.∼ N ( 0 , σ2 ) , ( 1 ) we would like to learn a model for the dynamics of x ( t ) . Moreover , it should be ensured that the model remains stable ( we will be concerned with exponential convergence to 0 ) on unseen trajectories . Learning such systems in a data-driven way is a key challenge in many disciplines , including robotics [ Wensing et al. , 2017 ] , continuous-time optimal control [ Esposito , 2009 ] or system biology [ Brunton et al. , 2016 ] . One powerful continuous-time approach to non-linear system identification are deep Neural ODEs ( NODE ) , as presented by Chen et al . [ 2018 ] . Since neural networks are very expressive , they can be deployed in a variety of applications [ Rackauckas et al. , 2020 ] . However , because of that expressiveness , little is known about their system theoretical properties after training . Thus , there has been growing interest in regularizing such dynamics models to ensure favorable properties . In the context of ensuring stability of the learned dynamics , Kolter and Manek [ 2019 ] propose to jointly learn a dynamics model and a neural network Lyapunov function , that guarantees global stability via a projection method . Neural network Lyapunov functions have previously been employed by Richards et al . [ 2018 ] to estimate the safe region of a fixed feedback policy and by Chang et al . [ 2019 ] ∗Correspondence to andreas.schlaginhaufen @ epfl.ch , wenkph @ ethz.ch . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to learn a stabilizing feedback policy for given dynamics . Moreover , Boffi et al . [ 2020 ] prove that neural Lyapunov functions can also be learned efficiently from data collected along trajectories . Thus far , all of these approaches are working directly with a standard ODE dynamics model . If the system ’ s states are fully observed and the system is Markovian , this can be a valid choice . However , in many practical settings , partial observations and non-Markovian effects like hysteresis or delays are the norm . To address the limited expressivity of neural ODEs in the classification setting , Dupont et al . [ 2019 ] introduce Augmented Neural ODEs ( ANODE ) . Here , a standard neural ODE is augmented with unobserved states , to extend the family of functions the model is able to capture . While Dupont et al . [ 2019 ] demonstrate that initializing the unobserved states at 0 is sufficient for the classification case , this is certainly not true when deploying ANODE as a dynamical system . In fact , our experiments in Section 4 demonstrate that learning this initial condition is a key problem in practice . Inspired by state-augmentation methods in the time-discrete case , we thus propose to capture partial observability and non-Markovian effects via Neural Delay Differential Equations ( NDDE ) . NDDEs were very recently proposed in the context of classification by Zhu et al . [ 2021 ] and in the context of closure models for partial differential equations by Gupta and Lermusiaux [ 2020 ] . While NDDEs offer an elegant solution to avoid the Markovianity of neural ODEs , again little can be said about stability outside of the training set . In fact , our experiments in Section 4 show that in a sparse observation and high noise setting , a NDDE model that is stable on training trajectories may become unstable along new unseen trajectories . We therefore extend the ideas of neural network Lyapunov functions , originally developed for stability analysis of non-linear ODEs , to time-delay systems and introduce a Lyapunov-like regularization term to stabilize the NDDE . In contrast to ODEs , NDDEs have an infinite-dimensional state space , which requires careful discretization schemes we introduce in this work . We then showcase the applicability of the proposed framework for the stabilization of the NDDE model and for the task of learning a stabilizing feedback policy in delayed feedback control of known open loop dynamics . In summary , we demonstrate the applicability of NDDEs to the case of modeling a partially observed dynamical system . We then leverage classical approaches for stability analysis in the context of delayed systems to develop a novel , Lyapunov-like regularization term to stabilize NDDEs . Furthermore , we provide theoretical guarantees and code for our implementation.2 2 Model and background . The main model of this paper , NDDEs , mathematically belongs to the class of time-delay systems that come with some additional difficulties compared to ODEs , both on the theoretical as well as the numerical side . Thus , we first recall some preliminaries and notation on time-delay systems . Then we continue with the model architecture and stability of time-delay systems . 2.1 Time-delay systems . Suppose r > 0 and consider the infinite-dimensional state space Cr : = C ( [ −r , 0 ] , Rn ) of continuous mappings from the interval [ −r , 0 ] to Rn . Throughout this paper , we endow Rn with the Euclidean norm ‖·‖2 and Cr with the supremum norm ‖φ‖r = sups∈ [ −r,0 ] ‖φ ( s ) ‖2 for φ ∈ Cr . Further on , along a trajectory x ∈ C ( [ −r , tf ] , Rn ) we make use of the notation xt ( · ) : = x ( t+ · ) ∈ Cr to denote the infinite-dimensional state at time t ∈ [ 0 , tf ] . A subset B ⊆ Cr is referred to as invariant if γ+ ( B ) = B , where γ+ ( B ) : = { xt ( ψ ) ∈ Cr : ψ ∈ B , t ≥ 0 } denotes its positive orbit . For a locally Lipschitz function f : Cr → Rn , an autonomous time-delay system is defined by the family of initial value problems : { ẋ ( t ) = f ( xt ) x ( s ) = ψ ( s ) , s ∈ [ −r , 0 ] , ψ ∈ Cr . ( 2 ) In contrast to ODEs , the dynamics are given by a Functional Differential Equation ( FDE ) and the initial condition by a function ψ ∈ Cr . As we are interested in autonomous dynamics , we will always set the initial time to zero . Furthermore , we denote by x ( ψ ) ( t ) ∈ Rn the solution and by xt ( ψ ) ∈ Cr the history state at time t , starting from the initial history ψ . As discussed in Section 2.2 , we will mainly focus on the important special case of retarded delay differential equations with commensurate delays ẋ ( t ) = f ( x ( t ) , x ( t− τ ) , ... , x ( t−Kτ ) ) = f ( xτ−K ( t ) ) . ( 3 ) 2Code is available at : https : //github.com/andrschl/stable-ndde Since some discretization is always necessary for computational tracktability , this implicitly includes a numerical approximation of general FDEs if the number of delays is chosen sufficiently large . It holds that r = Kτ , where for convenience we introduced the short-hand notation xτ−K ( t ) : = ( x ( t ) , x ( t− τ ) , . . . , x ( t−Kτ ) ) . Note that while the instantaneous change ( i.e. , the vector field ) in ( 3 ) depends only on a discrete set of observations xτ−K ( t ) , the initial history ψ ∈ Cr has to be given on the entire interval [ −r , 0 ] in order to have well-defined dynamics for t ≥ τ . As a consequence , in practice we need an interpolation of the initial history , and for numerical integration a specific DDE solver based on the method of steps [ Alfredo Bellen , 2013 ] is required . Apart from this , existence and uniqueness of solutions to ( 2 ) and ( 3 ) follow in a similar fashion as for ODEs [ Hale and Lunel , 1993 , Diekmann , 1995 ] . 2.2 Neural Delay Differential Equations . Model architecture The motivation of the model architecture is the following : We look for a general method to learn continuous non-Markovian time series which occur , for example , in partially observed dynamical systems . As already mentioned before , the temporal evolution of an ODE is uniquely determined by its current state , which makes NODEs inherently Markovian . Instead of augmenting NODEs with additional states , our approach is inspired by neural network based system identification of discrete-time dynamical systems : the latter copes with non-Markovian effects by augmenting the state space with past observations ( i.e. , literally memory states ) in order to lift the problem back into a Markovian setting ( see e.g . [ Chen et al. , 1990 ] ) . A continuous-time analog leads us to a FDE ẋ ( t ) = f ( xt ) where the current change is depending on the history xt ( s ) : = x ( t + s ) , s ∈ [ −r , 0 ] up to some maximal delay r > 0 . Since a neural network can not represent a general non-linear functional f , we discretize the infinite-dimensional memory state xt as in Equation ( 3 ) . This leads us to the NDDE model ẋ ( t ) = fNNθ ( x ( t ) , x ( t− τ ) , . . . , x ( t−Kτ ) ) = fNNθ ( xτ−K ( t ) ) , ( 4 ) which is illustrated in Figure 1 . Here , fNNθ is a feedforward neural network and K the number of delays . Predictions Given an initial history ψ ∈ Cr we integrate equation ( 4 ) to get the prediction at time t x̂ ( t ) = DDESolve ( ψ , fNNθ , t0 , t ) . ( 5 ) However , in practice observations are subject to noise and can not be sampled at an infinite rate . Hence , we need to approximate ψ by a smoothed interpolation . For this purpose we employ Gaussian Process ( GP ) regression . Given a set { ( t0 , y0 ) , . . . , ( tNhist , yNhist } ofNhist observations along the initial history , we fit for each scalar initial history component a zero-mean GP . As a kernel , we choose the Radial Basis Function ( RBF ) kernel kγ , σk ( t , t ′ ) = σ2k exp ( −|t− t ′| 2l2 ) ( 6 ) with length-scale l and kernel variance σ2k . This choice worked well in our experiments . Nevertheless , it is not crucial and other sufficiently smooth kernels such as Matérn 3/2 or Matérn 5/2 may be appropriate as well [ Rasmussen and Williams , 2005 ] . For the smoothed interpolation of the initial history we are then using the posterior mean function , ψ ( t ) i = ktT ( KTT + σ 2I ) −1 Yi , 1 ≤ i ≤ n , ( 7 ) where Yi = ( ( y0 ) i , . . . , ( yNhist ) i ) , T = ( t0 , . . . , tNhist ) , ktT = ( k ( t , t0 ) , . . . , k ( t , tNhist ) , and KTT = ( k ( tj , tk ) ) Nhist j , k=1 . The kernel hyperparameters l , σ 2 k as well as the observation noise variance σ 2 are estimated from data by marginal likelihood maximization . Training For training we proceed similar to NODEs and minimize the least squares loss J = N∑ i=0 ‖yi − x̂ ( ti ) ‖22 ( 8 ) along trajectories . While it is possible to utilize an interpolated continuous adjoint sensitivity method for calculating the loss gradients , differentiation through DDE solvers turned out to be significantly more efficient in our experiments . As discussed by Calver and Enright [ 2016 ] , one reason is that jump discontinuities need to be accounted for that are later propagated in higher order derivatives along the solution of the adjoint state , and the DDE solver needs to be restarted accordingly . We therefore refrain from going into further details about adjoint methods and simply make use of the differentiable DDE solvers provided by Rackauckas and Nie [ 2017 ] . Approximation capabilities As opposed to neural ODEs , we are no longer learning an ordinary differential equation , but a retarded-type delay differential equation with constant delays . An interesting question is under which conditions a NDDE can model the time series corresponding to the partial observations h ( z ( t ) ) of the ODE system ( 1 ) . As discussed in Appendix B , a sufficient condition for this is that the delay coordinate map , E : Rm → R ( K+1 ) n , z ( t ) 7→ xτ−K ( t ) = ( h ( z ( t ) ) , h ( z ( t− τ ) ) , . . . , h ( z ( t−Kτ ) ) ) , ( 9 ) is one-to-one . For dynamical systems confined on periodic or chaotic attractors , the delay embedding theorem by Takens [ 1981 ] indeed shows that this holds true for large enough K ( for more details see Appendix B and the references therein ) . Although we do not assume that the system ( 1 ) is confined to such an attractor , our experimental results in Section 4 demonstrate approximation power and generalization capabilities of NDDEs when applied to dissipative systems .

Summary: This paper aims to learn stable deep dynamics models for partially observed or delayed dynamical systems. The authors build their method upon Neural Delay Differential Equations (NDDEs) [1], where the main idea is to augment the state with the past history, thus avoiding the estimation of unobserved augmented states. To stabilize the learned dynamics, the authors then propose to combine NDDEs with a Lyapunov-like regularization term.  The authors show that NDDEs are powerful tools to model partially observed dynamical systems where the dynamics are non-Markovian. Furthermore, they demonstrate that the proposed method outperforms Augmented Neural ODEs (ANODE) in partially observed systems and is able to learn a stabilizing feedback policy for control systems with input delays.  [1] Qunxi Zhu, Yao Guo, Wei Lin, "Neural Delay Differential Equations"