Learning Stable Deep Dynamics Models for Partially Observed or Delayed Dynamical Systems

1 Introduction . In this paper , we address the task of learning stable , partially observed , continuous-time dynamical systems from data . More specifically , given access to a data set { ( t0 , yl0 ) , . . . , ( tN , ylN ) } Ll=1 of noisy , partial observations collected along L trajectories of an unknown , stable dynamical system , ż ( t ) = g ( z ( t ) ) , z ( t ) ∈ Rm x ( t ) = h ( z ( t ) ) , x ( t ) ∈ Rn , m ≥ n , yi = x ( ti ) + i , i i.i.d.∼ N ( 0 , σ2 ) , ( 1 ) we would like to learn a model for the dynamics of x ( t ) . Moreover , it should be ensured that the model remains stable ( we will be concerned with exponential convergence to 0 ) on unseen trajectories . Learning such systems in a data-driven way is a key challenge in many disciplines , including robotics [ Wensing et al. , 2017 ] , continuous-time optimal control [ Esposito , 2009 ] or system biology [ Brunton et al. , 2016 ] . One powerful continuous-time approach to non-linear system identification are deep Neural ODEs ( NODE ) , as presented by Chen et al . [ 2018 ] . Since neural networks are very expressive , they can be deployed in a variety of applications [ Rackauckas et al. , 2020 ] . However , because of that expressiveness , little is known about their system theoretical properties after training . Thus , there has been growing interest in regularizing such dynamics models to ensure favorable properties . In the context of ensuring stability of the learned dynamics , Kolter and Manek [ 2019 ] propose to jointly learn a dynamics model and a neural network Lyapunov function , that guarantees global stability via a projection method . Neural network Lyapunov functions have previously been employed by Richards et al . [ 2018 ] to estimate the safe region of a fixed feedback policy and by Chang et al . [ 2019 ] ∗Correspondence to andreas.schlaginhaufen @ epfl.ch , wenkph @ ethz.ch . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to learn a stabilizing feedback policy for given dynamics . Moreover , Boffi et al . [ 2020 ] prove that neural Lyapunov functions can also be learned efficiently from data collected along trajectories . Thus far , all of these approaches are working directly with a standard ODE dynamics model . If the system ’ s states are fully observed and the system is Markovian , this can be a valid choice . However , in many practical settings , partial observations and non-Markovian effects like hysteresis or delays are the norm . To address the limited expressivity of neural ODEs in the classification setting , Dupont et al . [ 2019 ] introduce Augmented Neural ODEs ( ANODE ) . Here , a standard neural ODE is augmented with unobserved states , to extend the family of functions the model is able to capture . While Dupont et al . [ 2019 ] demonstrate that initializing the unobserved states at 0 is sufficient for the classification case , this is certainly not true when deploying ANODE as a dynamical system . In fact , our experiments in Section 4 demonstrate that learning this initial condition is a key problem in practice . Inspired by state-augmentation methods in the time-discrete case , we thus propose to capture partial observability and non-Markovian effects via Neural Delay Differential Equations ( NDDE ) . NDDEs were very recently proposed in the context of classification by Zhu et al . [ 2021 ] and in the context of closure models for partial differential equations by Gupta and Lermusiaux [ 2020 ] . While NDDEs offer an elegant solution to avoid the Markovianity of neural ODEs , again little can be said about stability outside of the training set . In fact , our experiments in Section 4 show that in a sparse observation and high noise setting , a NDDE model that is stable on training trajectories may become unstable along new unseen trajectories . We therefore extend the ideas of neural network Lyapunov functions , originally developed for stability analysis of non-linear ODEs , to time-delay systems and introduce a Lyapunov-like regularization term to stabilize the NDDE . In contrast to ODEs , NDDEs have an infinite-dimensional state space , which requires careful discretization schemes we introduce in this work . We then showcase the applicability of the proposed framework for the stabilization of the NDDE model and for the task of learning a stabilizing feedback policy in delayed feedback control of known open loop dynamics . In summary , we demonstrate the applicability of NDDEs to the case of modeling a partially observed dynamical system . We then leverage classical approaches for stability analysis in the context of delayed systems to develop a novel , Lyapunov-like regularization term to stabilize NDDEs . Furthermore , we provide theoretical guarantees and code for our implementation.2 2 Model and background . The main model of this paper , NDDEs , mathematically belongs to the class of time-delay systems that come with some additional difficulties compared to ODEs , both on the theoretical as well as the numerical side . Thus , we first recall some preliminaries and notation on time-delay systems . Then we continue with the model architecture and stability of time-delay systems . 2.1 Time-delay systems . Suppose r > 0 and consider the infinite-dimensional state space Cr : = C ( [ −r , 0 ] , Rn ) of continuous mappings from the interval [ −r , 0 ] to Rn . Throughout this paper , we endow Rn with the Euclidean norm ‖·‖2 and Cr with the supremum norm ‖φ‖r = sups∈ [ −r,0 ] ‖φ ( s ) ‖2 for φ ∈ Cr . Further on , along a trajectory x ∈ C ( [ −r , tf ] , Rn ) we make use of the notation xt ( · ) : = x ( t+ · ) ∈ Cr to denote the infinite-dimensional state at time t ∈ [ 0 , tf ] . A subset B ⊆ Cr is referred to as invariant if γ+ ( B ) = B , where γ+ ( B ) : = { xt ( ψ ) ∈ Cr : ψ ∈ B , t ≥ 0 } denotes its positive orbit . For a locally Lipschitz function f : Cr → Rn , an autonomous time-delay system is defined by the family of initial value problems : { ẋ ( t ) = f ( xt ) x ( s ) = ψ ( s ) , s ∈ [ −r , 0 ] , ψ ∈ Cr . ( 2 ) In contrast to ODEs , the dynamics are given by a Functional Differential Equation ( FDE ) and the initial condition by a function ψ ∈ Cr . As we are interested in autonomous dynamics , we will always set the initial time to zero . Furthermore , we denote by x ( ψ ) ( t ) ∈ Rn the solution and by xt ( ψ ) ∈ Cr the history state at time t , starting from the initial history ψ . As discussed in Section 2.2 , we will mainly focus on the important special case of retarded delay differential equations with commensurate delays ẋ ( t ) = f ( x ( t ) , x ( t− τ ) , ... , x ( t−Kτ ) ) = f ( xτ−K ( t ) ) . ( 3 ) 2Code is available at : https : //github.com/andrschl/stable-ndde Since some discretization is always necessary for computational tracktability , this implicitly includes a numerical approximation of general FDEs if the number of delays is chosen sufficiently large . It holds that r = Kτ , where for convenience we introduced the short-hand notation xτ−K ( t ) : = ( x ( t ) , x ( t− τ ) , . . . , x ( t−Kτ ) ) . Note that while the instantaneous change ( i.e. , the vector field ) in ( 3 ) depends only on a discrete set of observations xτ−K ( t ) , the initial history ψ ∈ Cr has to be given on the entire interval [ −r , 0 ] in order to have well-defined dynamics for t ≥ τ . As a consequence , in practice we need an interpolation of the initial history , and for numerical integration a specific DDE solver based on the method of steps [ Alfredo Bellen , 2013 ] is required . Apart from this , existence and uniqueness of solutions to ( 2 ) and ( 3 ) follow in a similar fashion as for ODEs [ Hale and Lunel , 1993 , Diekmann , 1995 ] . 2.2 Neural Delay Differential Equations . Model architecture The motivation of the model architecture is the following : We look for a general method to learn continuous non-Markovian time series which occur , for example , in partially observed dynamical systems . As already mentioned before , the temporal evolution of an ODE is uniquely determined by its current state , which makes NODEs inherently Markovian . Instead of augmenting NODEs with additional states , our approach is inspired by neural network based system identification of discrete-time dynamical systems : the latter copes with non-Markovian effects by augmenting the state space with past observations ( i.e. , literally memory states ) in order to lift the problem back into a Markovian setting ( see e.g . [ Chen et al. , 1990 ] ) . A continuous-time analog leads us to a FDE ẋ ( t ) = f ( xt ) where the current change is depending on the history xt ( s ) : = x ( t + s ) , s ∈ [ −r , 0 ] up to some maximal delay r > 0 . Since a neural network can not represent a general non-linear functional f , we discretize the infinite-dimensional memory state xt as in Equation ( 3 ) . This leads us to the NDDE model ẋ ( t ) = fNNθ ( x ( t ) , x ( t− τ ) , . . . , x ( t−Kτ ) ) = fNNθ ( xτ−K ( t ) ) , ( 4 ) which is illustrated in Figure 1 . Here , fNNθ is a feedforward neural network and K the number of delays . Predictions Given an initial history ψ ∈ Cr we integrate equation ( 4 ) to get the prediction at time t x̂ ( t ) = DDESolve ( ψ , fNNθ , t0 , t ) . ( 5 ) However , in practice observations are subject to noise and can not be sampled at an infinite rate . Hence , we need to approximate ψ by a smoothed interpolation . For this purpose we employ Gaussian Process ( GP ) regression . Given a set { ( t0 , y0 ) , . . . , ( tNhist , yNhist } ofNhist observations along the initial history , we fit for each scalar initial history component a zero-mean GP . As a kernel , we choose the Radial Basis Function ( RBF ) kernel kγ , σk ( t , t ′ ) = σ2k exp ( −|t− t ′| 2l2 ) ( 6 ) with length-scale l and kernel variance σ2k . This choice worked well in our experiments . Nevertheless , it is not crucial and other sufficiently smooth kernels such as Matérn 3/2 or Matérn 5/2 may be appropriate as well [ Rasmussen and Williams , 2005 ] . For the smoothed interpolation of the initial history we are then using the posterior mean function , ψ ( t ) i = ktT ( KTT + σ 2I ) −1 Yi , 1 ≤ i ≤ n , ( 7 ) where Yi = ( ( y0 ) i , . . . , ( yNhist ) i ) , T = ( t0 , . . . , tNhist ) , ktT = ( k ( t , t0 ) , . . . , k ( t , tNhist ) , and KTT = ( k ( tj , tk ) ) Nhist j , k=1 . The kernel hyperparameters l , σ 2 k as well as the observation noise variance σ 2 are estimated from data by marginal likelihood maximization . Training For training we proceed similar to NODEs and minimize the least squares loss J = N∑ i=0 ‖yi − x̂ ( ti ) ‖22 ( 8 ) along trajectories . While it is possible to utilize an interpolated continuous adjoint sensitivity method for calculating the loss gradients , differentiation through DDE solvers turned out to be significantly more efficient in our experiments . As discussed by Calver and Enright [ 2016 ] , one reason is that jump discontinuities need to be accounted for that are later propagated in higher order derivatives along the solution of the adjoint state , and the DDE solver needs to be restarted accordingly . We therefore refrain from going into further details about adjoint methods and simply make use of the differentiable DDE solvers provided by Rackauckas and Nie [ 2017 ] . Approximation capabilities As opposed to neural ODEs , we are no longer learning an ordinary differential equation , but a retarded-type delay differential equation with constant delays . An interesting question is under which conditions a NDDE can model the time series corresponding to the partial observations h ( z ( t ) ) of the ODE system ( 1 ) . As discussed in Appendix B , a sufficient condition for this is that the delay coordinate map , E : Rm → R ( K+1 ) n , z ( t ) 7→ xτ−K ( t ) = ( h ( z ( t ) ) , h ( z ( t− τ ) ) , . . . , h ( z ( t−Kτ ) ) ) , ( 9 ) is one-to-one . For dynamical systems confined on periodic or chaotic attractors , the delay embedding theorem by Takens [ 1981 ] indeed shows that this holds true for large enough K ( for more details see Appendix B and the references therein ) . Although we do not assume that the system ( 1 ) is confined to such an attractor , our experimental results in Section 4 demonstrate approximation power and generalization capabilities of NDDEs when applied to dissipative systems .

Summary: This paper extends prior work on stable deep neural network dynamics models to learn stable models with partially observed states, and to learn (simple) control policies with time-delayed feedback. To achieve this, the authors minimize a Lyapunov-Razumikhin loss term when training _neural delay differential equations_. They demonstrate the capabilities of their system on a few tasks.