Learning Stable Deep Dynamics Models for Partially Observed or Delayed Dynamical Systems
Authors: Andreas Schlaginhaufen
Philippe Wenk
Andreas Krause
Florian Dörfler
Venue: NeurIPS 2021 Poster (2021)
Keywords: Machine Learning
Neural Delay Differential Equations
Dynamical Systems
Partially Observed Systems
Time-Delay Systems
Stable System Identification
Safe Learning

Abstract: Learning how complex dynamical systems evolve over time is a key challenge in system identification. For safety critical systems, it is often crucial that the learned model is guaranteed to converge to some equilibrium point. To this end, neural ODEs regularized with neural Lyapunov functions are a promising approach when states are fully observed. For practical applications however, {\em partial observations} are the norm. As we will demonstrate, initialization of unobserved augmented states can become a key problem for neural ODEs. To alleviate this issue, we propose to augment the system's state with its history. Inspired by state augmentation in discrete-time systems, we thus obtain {\em neural delay differential equations}. Based on classical time delay stability analysis, we then show how to ensure stability of the learned models, and theoretically analyze our approach. Our experiments demonstrate its applicability to stable system identification of partially observed systems and learning a stabilizing feedback policy in delayed feedback control.

Learning Stable Deep Dynamies Models for Partially Observed or Delayed Dynamical Systems
Andreas Schlaginhaufen'
Philippe Wenk ETH Zürich wenkph@ethz.chh ch
Andreas Krause ETH Zürich krauseaßethz.col ch
Florian Dörfler ETH Zürich dorflerGethz.co
ETH Zürich
andreas aedreas.schlainnaa schlaginhaufen
@epfl.ch
Abstract
Learning how complex dynamical systems evolve over time is a key challenge in system identification. For safety critical systems, it often erucial that the learned model is guaranteed to converge to some equilibrium point. To this end, neural ODES regularized with neural Lyapunov functions are a promising approach when states are fully observed. For practical applications however, partial observations are the norm. As we will demonstrate, initialization of unobserved augmented states can become key problem for neural ODEs. To alleviate this issue, propose to augment the system's 's state with its history. Inspired by state augmentation discrete- time systems, on classical time delay stability analysis, we then show how to ensure stability of the learned models, and theoretically analyze our approach. Our experiments demonstrate its applicability to stable system identification of partially observed systems and learning stabilizing feedback policy in delayed feedback control.
thus obtain neural delay differential equations. Based
Introduction
In this paper, we address the task learning stable, partially observed, continuous-time dynamical systems from data. More specifically, given access to data set {(to v6),... (tN, f of noisy, partial observations collected along L trajectories of an unknown, stable dynamical system,
(t))
(=(())
2t) E R
xxt) h(z(t))
.x(t) E R" m m22n. n,
lid. N(0,0²).
xx(t)) +
would like to learn model for the dynamics of srt). Moreover, it should be ensured that the model remains stable (we will be concerned with exponential convergence to 0) on unseen trajectories.
Learning such systems in data-driven way is a key challenge in many disciplines, including robotics [Wensing et al. 2017]. continuous-time optimal control [Esposito] 2009) or system biology [Brunton] et 2016)) One powerful continuous time approach to non non-linear system identification deep Neural ODEs (NODE), as presented by Chen et al. (2018). Since neural networks are very expressive, they can be deployed in a variety of applications [Rackauckas et al. 2020 However, because of that expressiveness, little is known about their system theoretical properties after training. Thus, there has been growing interest in regularizing such dynamics models ensure favorable properties. In the context of ensuring stability of the learned dynamics, Kolter and Manek [2019] propose to jointly learn dynamics model and neural network Lyapunov function, that guarantees global stability via a projection method. Neural network Lyapunov functions have previously been employed by Richards al. (0018 estimate the safe region of fixed feedback policy and by Chang al. 2019
"Corresoondence andreas. scchlaginhaufeeeeppi ch, wenkph@ethz.co ch.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

to learn a stabilizing feedback policy for given dynamics. Moreover, Boffi et al. [2020] prove that neural Lyapunov functions can also be learned efficiently from data collected along trajectories.
Thus far, all of these approaches are working directly with standard ODE dynamics model. If the system's ' states are fully observed and the system is Markovian, this can be valid choice. However, in many practical settings, partial observations and non- Mon-Maakov effects like hysteresis or delays are the norm. To address the limited expressivity of neural ODEs in the classification setting, Dupont et al.) [2019] introduce Augmented Neural ODEs (ANODE). Here, a standard neural ODE is augmented with unobserved states, to extend the family of functions the model is able to capture. While Dupont iet al.] [2019) demonstrate that initializing the unobserved states at 0 is sufficient for the classification case, this certainly not true when deploying ANODE as a dynamical system. In fact, our experiments in Section 4 demonstrate that learning this initial condition is key problem in practice.
Inspired by state-augmentation methods in the time-discrete case, we thus propose to capture partial observability and Man-Maaoov effects via Neural Delay Differential Equations (NDDE). NDDEs were very recently proposed in the context of classification by [Zhu et al. [2021] and in the context of closure models for partial differential equations by Gupta and Lermusiaux [2020). While NDDEs offer an elegant solution avoid the Markovianity of neural ODEs, again little can be said about stability outside of the training set. In fact, our experiments in Section E show that in a sparse observation and high noise setting, a NDDE model that stable on training trajectories may become unstable along new unseen trajectories. We therefore extend the ideas of neural network Lyapunov functions, originally developed for stability analysis of non- non-linear ODEs, to time-delay systems and introduce a Lyapunov-like regularization term to stabilize the NDDE In contrast to ODEs, NDDEs have an infiite-dimensional state space, which requires careful discretization schemes we introduce in this work. We then showcase the applicability of the proposed framework for the stabilization of the NDDE model and for the task of learning a stabilizing feedback policy in delayed feedback control of known open loop dynamics.
In summary, we demonstrate the applicability of NDDEs the case of modeling a partially observed dynamical system. We then leverage classical approaches for stability analysis in the context of delayed systems to develop novel, Lyapunov-like regularization term to stabilize NDDEs. Furthermore, we provide theoretical guarantees and code for our implementation
2 Model and background
The main model of this paper, NDDEs, mathematically belongs to the class of time-delay systems that come with some additional difficulties compared to ODEs, both on the theoretical as well as the numerical side. Thus, we first recall some preliminaries and notation on time-delay systems. Then we continue with the model architecture and stability of time-delay systems.
2.1 Time-delay systems
Suppose > 0 and consider the mappings from the interval [-r, 0] to R". Throughout this paper, we endow I with the Fuclidean norm 1l-1|2 and C, with the supremum norm || supse[-r.o] [[x())] for á C,.. Further on, along a trajectory
state space C, = C((-r, 0). RR") of continuous
C([-r,ty], R") we make use of the notation (-) == x(t +)) E C, to denote the diiiite-dimensiona state at time & E [0, ty]. A subset B S C, is referred to as invariant if y (B) = B, where 7 (B) == (x+(w) E e,, : e E B, 2 0} denotes its positive orbit. For locally Lipschitz. function C TR. an autonomous time-delay system is defined by the family of initial value problems:
(t)) = f(xz) (x()) = (sss, 8 sEl-r,0) 1b C..
In contrast to ODEs, the dynamics are given by a Functional Differential Equation (FDE) and the ini- tial condition by function v Cr. As are interested in autonomous dynamics. we will always set the initial time to zero. Furthermore, we denote by xx( x() E IR" the solution and by xy (x(()) E C, the history state at time i, starting from the initial history th. As discussed in Section 2.2 we will mainly focus on the important special case of retarded delay differential equations with commensurate delays
= f(t KT) = f(x" (t)).
(3)
"Code is available at: httpp:///gstchbb..cc/maaaettsst

---

Since some discretization is always necessary for computational tracktability, this implicitly includes a numerical approximation of general FDEs if the number of delays is chosen sufficiently large. It holds that
TT, where for convenience we introduced the short-hand notation x x=x(() = (xtt),=(tt-))) -) -)...... (r Kr))- Note that while the instantaneous change (i.e., the vector field) in (3) depends only on discrete set of observations x x=x(t) (t),, the initial history 1 C, has to be given on the entire interval [-r; in order to have well-defined dynamics for l т. As consequence, in practice we need an interpolation of the initial history. and for numerical integration a specific DDE solver based on the method of steps [Alfredo Bellen 2013] is required. Apart from this, existence and uniqueness of solutions to (2 and (3) follow in a similar fashion as for ODEs (Hale and Lunel
1993
Diekmann 1995].
2.2 Neural Delay Differential Equations
Model architecture The motivation of the model architecture is the following: We look for general method to learn continuous non-Markovian time series which occur, for example, in partially observed dynamical systems. As already mentioned before, the temporal evolution of an ODE uniquely determined by its current state, which makes NODEs inherently Markovian. Instead of augmenting NODEs with additional states, our approach is inspired by neural network based system identification of discrete- time dynamical systems: the latter copes with non-Markovian effects by augmenting the state space with past observations (i.e., literally memory states) in order to lift the problem back into a Markovian setting (see e.g. [Chen et al. 1990]). A continuous time f(x) where the current change depending the history (s) == xtt + s), $ E [-r, 0j up to some maximal delay r > O. Since a neural network cannot represent general non-linear functional f. we discretize the infiite-dimmnssione memory state xı as in Equation (3) This leads us to the NDDE model
analog leads to FDE at(t)
a(t)
f N (a((), att 7), -),...,,f(t -T)) = J (x=x(t)),
(4)
which is illustrated in Figure delays.
1 Here, f is feedforward neural network and K the number of
GP initial history &b
NDDE prediction
NN
att)
a
sr(1)
Initial observations
Observations
ax(t KT)
-
Figure 1: Graphical illustration of the NDDE model.
Predictions
Given an initial history de E C, we integrate equation (4) to get the prediction at time /
((t = DDESolve((), NNN, to. t).
(5)
However, in practice observations are subject to noise and cannot be sampled at an infinite rate.
Hence, we need approximate &b by smoothed interpolation. For this purpose we employ Gaussian Process (GP) regression. Given a set {(v, Bo). .................. Nam VN of Nuis observations along the initial history, we fit for each scalar initial history component zero-mean GP. As a kernel, we choose the Radial Basis Function (RBF) kernel
oy (t,t') )²) of exp (_1' 1I--1' l')
(6)
2/2
with length-scale I and kernel variance o.. This choice worked well in our experiments. Nevertheless, it is not crucial and other sufficiently smooth kernels such as Matérn 3/2 or Matérn 5/2 may be appropriate as well (Rasmussen and Williams (2055. For the smoothed interpolation of the initial history we are then using the posterior mean function,
((1,, = ker (KTT + +011) Y,, 15isn,
where Y, ((yo),.... ).) T (to...., (to....,t,..) (Nm)), keT (o(t,to).... l(t,, NN.), and Krr (ki(tz, tx) (x)) The kernel hyperparameters of well the observation noise variance o² estimated from data by marginal likelihood maximization.
3

---

Training For training we proceed similar to NODEs and minimize the least squares loss
My +(4))l2
(8)
along trajectories. While is possible utilize an interpolated continuous adjoint sensitivity method for calculating the loss gradients, differentiation through DDE solvers turned out to be significantly more efficient in our experiments. As discussed by Calver and Enright [2016]. one reason that jump discontinuities need to be accounted for that are later propagated in higher order derivatives along the solution of the adjoint state, and the DDE solver needs to be restarted accordingly. We therefore refrain from going into further details about adjoint methods and simply make use of the differentiable DDE solvers provided by Rackauckas and Nie 2017].
Approximation capabilities As opposed neural ODEs, are longer learning an ordinary differential equation, but a retarded-type delay differential equation with constant delays. An interesting question is under which conditions NDDE can model the time series corresponding to the partial observations h(²(t)) of the ODE system (I) As discussed in Appendix B condition for this that the delay coordinate map,
a sufficient
E IR - IR (K+1)n
(1) x ()=))(((=(())))))))))))))))>>))) (((() (((())) M((() KT))),
one-to-one. For dynamical systems confined on periodic or chaotic attractors, the delay embedding theorem by Takens [181] ] indeed shows that this holds true for large enough K (for more details see Appendix B and the references therein). Although we do not assume that the system ① is confined to such an attractor, our experimental results in Section 4 demonstrate approximation power and generalization capabilities of NDDEs when applied to dissipative systems.
2.3 Stability time-delay systems
We discuss stability analysis for the general class of time delay-systems We assume that the origin is an equilibrium, f(0) = 0, and slightly adjust the definition of exponential stability with respect this equilibrium point as provided by Fridman [2014] our needs:
Definition 1 For fixed set of initial histories S C C, and constunts 7, M > O, we call system (7, exxponniill decaying on S over the time horizon [0, ty) if
[ I Me ((--t) (106 for 05 s s E.
(10)
For some invariant B C Cr with 0 B the time delay system 2 is called (7, -exponnnnet stable on B if is (7, H) -exponennil decaying on B over the time horizon [0, co).
Here, y measures the rate of decay and M is an upper bound on the transient overshoot. In cases where we do not care about the specific values of 7 and M we simply call the system (2) exponentially stable. Note that if time-delay system is (7, M) -xxponnnnnt decaying on a set of initial histories S over [0, oo), then it is also (7, M) -expooetnttttllllyy decaying over [0, oo) on (S).
Since (S) is invariant by definition, (M)) decay on over [0, oo) is equivalent to (y, eexponentia stability on y (S).
Razumikhin's method A key method to prove exponential stability of non-linear ODEs are Lyapunov functions [Lyapunov] 1992]. However, directly applying ODE Lyapunov functions to time-delay systems leads to very restrictive results (e.g., 1-dimensional system would not be allowed oscillate [Fridman] (014)). Nevertheless, along the same lines, two approaches geared towards stability analysis of non-linear time-delay systems exist. Whereas the method of Trapunov-Krasovse method to an infinite-dimension 1956) is to make use of positive-definit Lyapunov functions V 12" + R4, with finite domains familiar from the ODE case and to relax the decay condition. Namely, negative derivative of V<r(t)) at time is required only when we are about to leave the sublevel set VSa = fx E R" V(x) < m of 77 sup.e[-r.0f (z tt + s)). The following theorem establishes sufficient conditions for exponential stability.
functionals [Krasovskii] 1963) a natural extension of Lyapunov's 's direct state space, the idea of Razumikhin-type theorems Razumikhinn

---

Theorem ( (Effimov and Aleksandrov, 2020) Assume there exists differentiable function
V: F. R., positive reals c1, Cz, a, and a constant q > 1 such that along all trajectories starting in o ES C c, the following conditions hold for all t. E (0,ty);
(i) a 11=(0) V(x(t)) =v<(() S 02 (=((
(ii) ((z(t)) 5 -aV (x((tt whenever V(a(t + +)) S (V((())) As E-E
Then system (2) is (7, M) -exponentii decaying min( (a. loRa)/2 and M 02/c1. Moreover, if for an invariant set S C. with 0 B condi- tions (i), (it) hold for all z, E B then the time delay system 2 is exponentially stable on B.
S over [0, T) with decay rate ?
function V establishing stability of some invariant set B by satisfying conditions (1), (id) in Theorem ! is referred to as a Lyapunov-Ren Razumikhnn Function (LRF). However, due to the infinite dimension of C, it hard verify the decay condition (ti) on the entire state space C,. We thus focus on proving (n, -exxponenntia decay along trajectories starting within some fixed set of initial conditions S C Cr, which, as discussed before, is for an infinite time horizon equivalent to (7, -exponennni stability on B -SS)
Note that Theorem D establishes sufficient, but not necessary conditions for exponential stability. The problem is that often it is not strong enough to check the Razumikhin condition
(V(((() ((x(1 s))
(11)
only on the interval 8 E [-r,0), but we should take into account more of the past observations of ((=(()). It therefore be helpful to reinterpret problem ( as in the state space Cry with some rv > r and to apply Theorem [ to that problem. However, in this new larger state space, only initial histories of the form
(s) ( (r (v-) (v()) (s (r- (r- , E [r rv,0], 0],
,8E rr, r rvl
with s/ C,
(12)
need to be considered for stability in C, Hale and Lunel 1993]. Furthermore, Proposition I which we prove in Appendix A shows that also exponential stability follows, albeit at the price of larger bound on the transient overshoot.
Proposition 1 If f is -Lipschhi,, S(0)- = O, and 1b, & defined as in 12 then
j S || ctx(ov-r)
(13)
Centred around this idea, necessary and sufficient Razumikhin-type conditions for discrete-time delay systems are given by Gielen et al. [2013). In the following, we therefore treat rvv as a hyperparameter that has be chosen for the respective problem at hand.
3 Learning stable dynamics
We now propose approach, based neural LRFs, to enforce stability of a parametric DDE. The key idea is to jointly Iearn a neural network Lyapunov- Lyapunov-Razumikaa Razumikhin function and the dynamics model. Similarly to [Richards et al. 20188 Chang et al., 2019] we propose to enforce stability via the loss function. This is in contrast to Kolter and Manek [2019] who use projection- approach to ensure stability in the forward pass. The main reason for this design choice is that projective approach based on LRFs leads to discontinuities
the forward pass, which are problematic for DDE solvers [Alfredo Bellen 2013). Moreover, incorporating the Lyapunov neural network into the forward pass renders the model slow during inference time and a loss function based approach offers the opportunity actively stabilize an initially unstable system, as we demonstrate in Section 4
Lyapunov neural network construction Except for the decay condition along solutions (condition (ii) in Thcorem D an LRF has the same form an ODE Lyapunov function. We thus employ the same Lyapunov neural network as proposed by Kolter and Manek (2019]. The construction based on an Inpur-Covve Convex Neural Network (ICNN) [Amos et al. [016]] The ICNN + 0%
N (z) is convex by construction and any convex function can be approximated by such neural
networks Chen et al. 2019). In order to satisfy the upper bound in condition (i) of Theorem I the
5

---

activation functions o of 96 strict convexity, and to make sure that the global minimum lies at x = 0, a final layer
N' are required to additionally have slope no greater than one. To ensure
(x)= og NN (x) g NN (0)) + c -||==|2
(14)
is chosen. Here, c 0 is small constant. As for the activation function a, having a global minimum at requires (0) = 0. Furthermore, since we want to ensure Lipschitz continuity of the loss derivatives, we use twice continuously differentiable smoothed ReLU version
0
x S 0
x
(
d²
(2d)
0 S < d
(15)
d
x
x > d.
2
This slightly differs from the original a proposed by Kolter and Manek 2019 since they only needed a once continuously differentiable one. This construction ensures that 9
N (x) = O (
as x + and also V. (x:) = ((1x=112) as ||=/|| - xo. We can therefore always find constants C1, C2 such that the conditions (i) in Theorem are satisfied. In the next step, we explain how to employ this neural network architecture to learn neural LRFs and at the same time stabilize a parametric delay differential equation of the form (3)
Crapunov-Razumikinn loss As stated before, V.N. satisfies condition (i) in Theorem by con- struction. The relaxed decay condition (ii) however has to be enforced during training. Since it is practically infeasible to check the Razumikhin condition (II) on the continuous interval [-ry, Oj. we need some discretization that still allows for stability guarantees. As we will analyze in this section, this is satisfied by the the following loss with discretized Razumikhin condition
ELRF (0, B, x
Kv (t))
ReLU (VaNn (x (t)) + aV (V)N(((())) e VV (x(t))
max VNN (x(t - -iti)) IsisKy
(16)
otherwise. Furthermore, for notational simplicity we choose TV S T and such that r = l. for some integer I E N. According to Theorem a zero loss L (0.0, 0, x0x""xxx00 (t)) along a trajectory of length ky) implies exponential decay along this trajectory. Moreover, if for a fixed set of initial histories Suain S C, the loss (16) is zero along all trajectories starting in Swain and over a time horizon [0, oc), then the delay differential equation is stable on 7 (Suain). However, since we cannot check this for ty oc, choose ty large enough to ensure convergence to sufficiently small region around the origin. Theorem 2 then also establishes exponential decay for trajectories starting not necessarily in but close enough Suain- For its proof refer Appendix A
Here, (()) denotes the unit step function with Oss) = if 2 0 and O(s)
Theorem If the dynamics are L-pliihhh and the LRF loss (6) is zero along trajectories starting in Sirain C. Cr over a time horizon [0, t then the time-delay system is (7, M -exponennty decaying on Smain over (y) Moreover, if for another set of initial histories S C Syain and some e > 0, the training set Smain is 8-covering of S (in the (lorm))rm) with å = se ce (I, then the time delay system is (y, MM) expoonentially decaying on S (0) over the time horizon [0, tt) and with M = 2M + 1. Here, (0) = {vb E C. || S $} denotes the e-ball around the origin.
While for a zero loss exponential decay is guaranteed, the discretization of the Razumikhin condition might be introducing additional conservatism by requiring decay in V.NN too often. However, Proposition 2 tells us that the discretized Razumikhin condition holds, TV is small enough, and the current state Ties outside of an e-ball around the origin, then the continuous condition holds for some q > Furthermore, ÿ converges quadratically to g as TV > 0. Remembering that the decay rate in Theorem is = min(a, logg)/2, )/2, it becomes apparent that by discretization we are requiring a slightly larger rate of decay, which can however be controlled by the choice of TV.

---

Proposition 2 Let Kv and TV be such that rv := KVTV 2 r and let z(.) be a solution of i(t)= = f(x,) passing through 21, Cr. Assume || <C and || 2 E. Furthermore, let f be T.p-lipsc and differentiable. Then, if the discretized Razumikhin condition,
VNN (ztto - kTv)) S
,Kk E (.1,,,,Ky
(17)
is satisfied and TV is small enough, then the continuous Razumilhinn condition, Va (zl(o + +)) S V (z((o)), holds for any & E [-rv, 0] and some ((TV) with ((Tv) - O(T)) as TV -
(18)
The condition that the current state lies outside some e-ball around the origin may be replaced by an assumption on the solutions' decay in I1- However, in practice we are usually satisfied with conver- gence to some small neighborhood of the origin, since, as already noted, we cannot choose an infinite time horizon and also as due to observation noise and data scarcity our model will always be subject to some modelling errors. We elaborate more on this issue and prove Proposition in Appendix A
2
Stabilizing NDDEs In order to stabilize an NDDE on fixed set of initial conditions Suain S C. we minimize the stabilizing loss (T6) on set of data points fx(")..... x <(urr) } with x(+) Ra(Kr+1) collected along trajectories starting in Suain. The resulting gradients are added up with those from the NDDE loss (8). While this enables us to stabilize the NDDE on unseen trajectories, we still need an efficient method to generate realistic initial histories. Especially in the setting of partially observed systems we do not know much more about initial histories than that they are contained within a bounded, Lipschitz subset of C,. However, since for our NDDE model the initial history is given by a GP-mean function, it suffices to stabilize the NDDE for initial histories within the subset Naia
ss E Htll((t)
Σ C.ki,o (t, tt)}
(19)
of the reproducing kernel Hilbert space Hg corresponding to kt, (t, ()). Furthermore, boundedness of initial histories translates into bound on the norm of the expansion coefficients [I(c1, ..., CNiss) |1, < A
and Lipschitz continuity can be accounted for by upper bounding the inverse length-scale 1/ S B
and the kernel variance of S C' [Rasmussen and Williams) 2005). To satisfy these constraints, sample at each training iteration initaal histories 3 E Swain as follows: The expansion coefficients (cI,-.., (Na))) ((n are sampled uniformly in 1.2-ball, and 1/1, or on bounded intervals [0, B), C). respectively.
Note, that while another possibility would be to integrate the loss 16 term into the NDDE loss, the discontinuities in (16) turn out to be problematic for DDE solvers.
as continuous regularization
Delayed feedback control The stabilizing loss 16) is essentially applicable to any parametric DDE of the form (3) Equations of this form also occur in delayed feedback control. Assume we want to learn stabilizing state feedback u(t) TT(x)()) for known open loop control system i(t)= = (z(t,, u(t - -)) with input delay. In practice, such delays in the feedback loop are often introduced as consequence of communication latencies and typically cause instability (Krstic] 2009). The resulting closed loop system is a parametric DDE (t)) f (x((), (x((( T))). which can, for small enough delays, be stabilized in data-driven way with our LRF loss (16) If the input delay exceeds some critical value, the system can no longer be stabilized by DDE methods and infinite-dimensional feedback taking into account the inputs history would be required Krstic 2009
Experimental results for delayed feedback stabilization are provided in Section
Choice of hyperparameters Our NDDE model (4 well the stabilizing loss (66) involve hy- perparameters such as number and magnitude of delays, whose choice we discuss in the following. For our NDDE model, the number of delays К clearly controls the representational capabilities. In gen- eral it is sufficient to choose K large enough such that the delay coordinate map (9) is one-to-one. For periodic or chaotic attractors, Takens' Embedding Theorem 3 (see Appendix B) provides a sufficient lower bound on K to ensure this. Moreover, in our experiments, larger values of K ease training and - perhaps surprisingly do not hurt generalization performance. Of course, an overly large number of delays leads to long training time per iteration, thus slowing down training again. Except for the first experiment where we directly compare NDDEs to ANODES, we fix a relatively large number of delays K = 10 throughout the paper. While Takens' Theorem 3 is besides periodicity condition, com- pletely agnostic to the choice of the delay parameter r, various heuristics such as Average Mutual Infor- mation or False Nearest Neighbours exist in practice (for overview see Wallot and Mønster 2018).
7

---

With regard to the stabilizing loss (66. Proposition 2 proves that the number of delays Ky controls the conservatism we introduce through discretization of the Razumikhin condition. Furthermore, as discussed in (12), the maximal considered delay controls the conservatism inherent to Razu- mikhin's Theorem itself. Lastly, the parameters and g are directly related to the rate of decay ? in Theorem via y min(a, log g/rv). We thus choose R log g/rv. Moreover, a too small choice of Ky, TV or too large choice of a, can be detected via a non-zero LRF loss (16
Experiments
Learning partially observed dynamics We first compare the applicability of Vanilla NDDEs and ANODEs for the task of learning partially observed harmonic oscillator,
11 (tt) = dt =2(1))
(-) !) x(t), M(((t)) = (1 (t).
(20)
We train the models over two training trajectories starting from =0,1 - (1, 0) and =0,1 (0, 2) and with zero observation noise. For ANODEs we compare model trained with given true augmented initial conditions (IC) against another model where we initialize the augmented states with zero and learn them via the adjoint method. Moreover, for the NDDE we compare a single delay model with K = 1 to a multiple delay model with K 10. The resulting vector field plots for the ANODE models illustrated in Figures 2a 2c demonstrate that, whereas for true initial conditions the dynamics match the ground truth well, learning the augmented initial conditions turns out to be key problem. In contrast, for our NDDE model the initialization is conveniently provided by the GP interpolation. This is also reflected in the learning curves in Figure 2d where we see that both NDDE models yield a significantly lower train loss for fewer iterations compared to the ANODE models. Moreover, the NDDE with K 10 achieves a better training score. For the rest of the experiments we therefore fix K 10. For more information about the setup and additional experiments we refer to Appendix
10a
ANGUR trne IU NDDE
10
NDDE
10-1_
10
100
2001
300
sterationn
(a) Ground truth
(b) ANODE true IC
(c) ANODE Iearned IC
(d) Train loss
Figure 2: Comparison of ANODES with true and learned initial conditions (IC) and Vanilla NDDEs.
In (a)-(c), the phase portrait for ground truth and ANODE models are provided. Note, that since we are only interested in the first state the direction of rotation is irrelevant for the ANODE models. As it impossible draw phase portrait of the NDDE model, the train losses for all four models are compared in Figure (d). While for given true initial conditions the ANODE model achieves a reasonable training fit, learning the augmented initial condition leads high training error. Moreover, both NDDE models show superior training performance compared to the ANODE models, both in terms of training error and number of iterations.
Learning stable NDDEs Kolter and Manek 2019 show that in the ODE case, a neural network dynamics model trained on stable data may become unstable for long-term prediction. For NDDEs, we observed this to be a problem in the setting of sparse observations, a high noise level, and generalization over initial histories. In particular, compound pendulum, where only the angles of deflection y01 and 22, but not the angular velocities observed. This is complex non-linear dynamical system which, for low friction, exhibits chaotic behavior [Shinbrot et al. 1992]. The governing equations are derived in Appendix C
consider partially observed damped double
For observation noise of variance a = 0.05 and training and test data along 4 trajectories, we compare the generalization performance of Vanilla NDDE and a NDDE stabilized with LRF regularization. We repeat the training for 20 independent weight initializations and noise realizations. The resulting predictions illustrated in Figures
that while the median prediction is stable, the upper 0.95 guantile explodes for the Vanilla NDDE. In contrast, the stabilized NDDE remains stable

---

on all test trajectories. Moreover, whereas the test loss in Figure 3c explodes for the unstable NDDE, the train losses are approximately the same. Thus, the LRF loss guides us stable optimum without sacrificing training performance.
Vanilla NDDE
test prediction
2
ground truth
5.
-2
time
time
(a) Prediction along unseen trajectory Vanilla NDDE
stabilized NDDE
test prediction
2
ground trush
~wmmmmm
wwwmmm
0
-2
20
30
20
30
time
time
(b) Prediction along unseen trajectory for stabilized NDDE
train loss
test loss
Vanilla NDDE
10
Vanilla NDDE
10-1
stabilised NDDE
stabilized NDDE
105
10
10
108
10
0
100
200
300
400
500
100
200
300
400
500
iterations
iterations
(c) Train and test eroo both models
Figure 3: In (a)-(b) the test predictions are shown for one of the test trajectories and in (c) train and test loss for all trajectories are illustrated. The lines indicate the median and the shaded area the 0.05 and 0.95 quantiles from 20 independent weight initializations and noise realizations.
Stabilization with delayed feedback control As first application for learning stabilizing feedback policy of known open loop system, we consider a friction-less inverted pendulum with an input delay 0.03. The open loop dynamics are given by
41 (2(())
sin( (1)) ml2 ult -
xz(()
(21)
Here, the states are (I1, x2) = (pp,g) where (tt) is the angle of deflection with respect to the fully
upright position, 9 indicates the acceleration of gravity, and I and the length and mass of the pendulum. Furthermore, u(t) is the torque which is applied at the pivot point. The goal is to learn stabilizing feedback policy
w(t) +(((t) (t) (t).
(22)
Similar to Chang et al. 2019 we initialize the parameters kj Ai with the values from the Linear
Quadratic Regulator (LQR) feedback policy calculated for the linearization of (T). For the training,
continuously generate new initial histories as follows: We sample ODE initial conditions

---

circle of radius x/ 7/2, assuming zero control for t < 0. Thus. the dynamics are described by an autonomous ODE along initial histories. As depicted in Figure the initially unstable state feedback can be stabilized by means of our Razumikhin loss. Furthermore, the speed of decay can be controlled by the choice the hyperparameters a and q. Moreover, LRR is zero along new test trajectories indicating that we indeed learned valid LRF candidate for this set of initial histories.
840
2-
720
600
480
360
LQR
240
-2
-2
120
0
2
2
0
2
0
time
time
SP
(a) Delayed state-feedback
(b) Learned (left) and randomly initialized (right) V
Figure 4: In (a), the learned delayed state feedhack policy is compared to the LQR control. The left plot in (b) shows the learned LRF candidate V and the right plot random initialization. Whereas LQR is unstable for delayed feedback, the policies learned by minimization of the Lyapunov- Razumikhin loss are stable and the rate of decay can be controlled by the choice of the decay parameters and q.
As second more complex- - experiment, we consider stabilizing a cartpole with a delayed input force acting on the cart. In contrast to the two-dimensional inverted pendulum, this is four- dimensional non-linear system. Its states are sz() (e((), (()))(()))())))) where yp again denotes angle of deflection and & the position the cart. For the exact equations, we refer to (Stimac) 1999]. For the control force acting on the cart we assume a delay of 0.05 and aim at finding a stabilizing feedback policy To (x((t)). Similarly to the inverted pendulum experiment, Figure 5 shows that minimizing the LRF loss (16) enables to find a stabilizing feedback policy from initially unstable LQR feedback. Furthermore, the rate of decay can be controlled by the choice of the hyperparameter and
0.50
0.25
M
0.00
0.25
LQR
0.01
0.50
1.0
2
time
time
timee
time
Figure For the delayed cartpole, we compare the learned state feedback policy to the LQR controller. While LQR becomes unstable for delayed feedback, the feedback policies learned with the syapunov-Razumikhin loss are stable, and the decay rate can controlled by the choice of a and q.
5 Conclusion
In this paper, we demonstrated that NDDEs a powerful tool learn non-Markovian dynamics occuring when observing partially observed dynamical system. Via state augmentation with the past history we avoid the estimation of unobserved augmented states, which we showed to major problem of ANODES when applied to partially observed systems. Based on classical time-delay stability theory, we then proposed a new regularization term based on a neural network Lyapunov- Razumikhin function to stabilize NDDEs. We further showed how this approach can be used to learn a stabilizing feedback policy for control systems with input delays. Besides experiments showeasing the applicability of our approach, we also provide code and theoretical analysis.
10

---

Acknowledgments
This research was supported by the Max Planck ETH Center for Learning Systems. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme grant agreement No 815943 as well as from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545.
References
M. C. Agarana and T. Akinlabi. Mathematical modelling and analysis of human arm as triple pendulum system using euler- lagragian model. IOP Conference Series: Materials Science and Engineering. 413:012010, sep 2018. doi: (0.10088/755--899944111111000000
M. Z. Alfredo Bellen. Numerical Methods for Delay Differential Equations. OXFORD UNIV
PR. Apr. 2013. ISBN 0199671370. URL httt::tt:://eee ebook httpo///////////ooou/////99191 alfredo_bell bellen marino. zennaro www.mummmiiia...m numericallmmthhoss equations.hit htmml
B. Amos, I. Xu, and J. 7. Kolter. Input convex neural networks. CoRR, abs/ abss/169.077152 2016. URL http://arxiv.. wttgg/bb/11609 07152
R. Bellman. The stability of solutions of linear differential equations. Duke Mathematical Journal, 10(4), dec 1943, doi: 00255/00022770444400000333
N. M. Boffi, S. Tu, N. Matni, J.JJ. E. Slotine, and V. Sindhwani. Learning stability certificates from data, 2020.
S. L. Brumton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-391) 2016.
Calver and W. Enright. Numerical methods for computing sensitivities for ODEs and DDEs. Numerical Algorithms, 74(4):1101-111) sep 2016. doi: 10.1007/s11055-016-000
Y.C. Chang, N. Roohi, and S. Gao. Neural 1yapumov control. In Advances in Neural Information Processing Systems, volume 32, pages 3245-3254 Curran As-
sociates, Inc., 2019. URL htttps://proeeedings segrgs..eurips.. www..ipp...cppppe///eett........ cc/ppaprr/019/fiiee 334771ab223000009cfff75333 1200l pdf
R. T. Q. Chen, Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differ- ential equations. In Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL http::ttt/ttcceediiiiiiis neur ips. /eerrippscc/ppppe//1/////f..
pdf
S. Chen, S. A. Billings, and P. M. Grant. Non-linear system identification using neural networks. International Journal of Control, 1166):1111-214, 1990. doi: 10. 10.1880002200779089934422.
URL https://doi. org 2rg/10 N...1008/00027779989992222
Y. Chen, Y. Shi, and B. Zhang. Optimal control via neural networks: A convex approach. In International Conference on Learning Representations, 2019. URL https::/openreviiee net/ Horrm??dd-HIMT22AAS)
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4):303-314 dec 1989. doi: 10. 10.1007//902551277.
O. Diekmann. Delay Equations: Functional-, Complex, and Nonlinear Analysis. Springer New York, New York, NY, 1995. ISBN 9781461242062
E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural odes. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. http:htttss//proeeeeei

---

D. Efimov and A. Aleksandrov. On estimation of rates of convergence in cerppunov--azuuikkin approach. Automatica. 11::108928, jun 2020. doi: 20000/f..utomattica20000000
W. R. Esposito. Dynamic programming- continuous- time optimal controlDynamic Programming:
Continuous-time Optimal Control, pages 844-846. Springer US, Boston, MA, 2009. ISBN
978-0-388-77455--0 doi: 10. 11110077778803337777599-00-000
org/10.
778-0-387-747599 146
E. Fridman. Introduction to time-delay systems analysis and control. Birkhäuser, Cham, 2014. ISBN 1903399.
R. H. Gielen, M. Lazar, and S. Rakovic. Necessary and sufficient razumikhin-type conditions for stability of delay difference equations. IEEE Transactions Automatic Control, 58(10):
2637-2642, oct 2013. doi: 10.1109/ac.20113225555
S. Gowda, Y. Ma, A. Cheli, M. Gwozdz, V. B. Shah, A. Edelman, and C. Rackauckas. High- performance symbolic-numeries via multiple dispatch. arXiv preprint 2rXiv:2105.03944 2021.
A. Gupta and
Lermusiaux. Neural closure models for dynamical systems, 2020.
J. K. Hale and S. M. V. Lunel. Introduction to Functional Differential Equations. Springer New York, 1993. doi: 10.
J. Z. Kolter and G. Manek. Learning stable deep dynamics models. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/ Paper. pdf
N. N. Krasovskii. Stability of motion (translated from the (1959) russian ed. by j. brenner). Stanford University Press, 1963. doi: 10. 00010077/000084395000226666
M. Krstic. Delay Compensation for Nonlinear, Adaptive, and PDE Systems. Birkhäuser Boston, 2009. doi: 00.1007/97-008877--48
A. M. Lyapunov. The general problem of the stability of motion. International Journal of Control, 55(3):531-534, mar 1992. doi: 10. 1080/00000 179208934253.
Y. Ma, S. Gowda, R. Anantharaman, C. Laughman, Shah, and Rackauckas. Modelingtoolkit: composable graph transformation system for equation-based modeling, 2021.
G. M. Phillips. Interpolation and Approximation by Polynomials. Springer New York, 2003. doi: 10. 10.1007//97417.
C. Rackauckas and Q. Nie. Differentiallquaaionss]]] performant and feature-rich ecosystem for solving differential equations in julia. Journal of Open Research Software, 5(1), 2017.
C. Rackauckas, Y. Ma, J. Martensen, C. Warner, K. Zubov, R. Supekar, D. Skinner, A. Ramadhan, and A. Edelman. Universal differential equations for scientific machine learning. arXiv preprins M0Xi::2001.04385.. 2020.
Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions, 2017.
C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
B. Razumikhin. On the stability of systems with a delay (russian). Prikladnaya Matematika i Mekhanika, vol. 20, pp. 500-512, 1956.
M. Richards, F. Berkenkamp, and Krause. The Iyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In Proceedings of The 2nd Conference on Robot Learning, volume 87 Proceedings of Machine Learning Research, pages PMIR, 29-31 Oct 2018
J. C. Robinson. A topological delay embedding theorem for infinite-dimensional dynamical systems. Nonlinearity, 18(5): jul 2005. doi: 10. 11088/0951-7715/118855 URL https
//oi www.oog.//.... 0ww.00///00111115111888111
12

---

T. Sauer, J. A. Yorke, and M. Casdagli. Embedology. Journal of Statistical Physics, 65(3-4):579-616 nov 1991. doi: 10.10077//01053745..
Shinbrot, Grebogi. Wisdom, and J. A. Yorke. Chaos a double pendulum. American Journal of Physics, 60(6):491-49, jun 1992. doi: 10.1119/1.1660.
H. Smith. An Introduction Delay Differential Equations with Applications the Life Sciences. Springer- Verlag GmbH, Sept. 2010. ISBN 9781441976468.
URL
htttp::/www.bboo..dd////o/ //to.//./////prrrouo//////////// www.ddffereeniaaeeeuutitoiioinei petppaationsswtthheeeeee with wwwpthhappppiiiiiiiioio to tt_etett____eeeeeet life siee_eciencesseeeeeeeeseesssssee html
smi th an int.iooooouutioo.ooo..
A. K. Stimac. Standup and stabilization of the inverted pendulum. Master's thesis, Massachusetts Institute of Technology, 1999.
Takens. Detecting strange attractors turbulence. In Dynamical Systems and Turbulence, Warwick 1980, pages 366-381, Berlin, Heidelberg, 1981. Springer Berlin Heidelberg. ISBN 978-3-540- 38945-3.
C. Tsitouras. Runge-kutte kutta pairs of order 5 (4) satisfying only the first column simplifying assumption. Computers & Mathematics with Applications, 62(2):770-775, 2011.
S. Wallot and D. Mønster. Calculation of average mutual information (ami) and false-nearest neighbors (fnn) for the estimation of embedding parameters of multidimensional time series in matlab. Frontiers in Psychology, 9:1679, 2018. ISSN 1664-1078 doi: 10.3389/fpsyg. 201.ffsssyy....8...... 2018.01677 URL wtthh:tt/:/s/tt /tttp://www.... frontiersin wwrg/article//00 2389/fpsyg
P. M. Wensing, S. Kim, and J.-J. E. Slotine. Linear matrix inequalities for physically consistent inertial parameter identification: A statistical perspective on the mass distribution. IEEE Roboties and Automation Letters, 3(1):60-67, 2017.
Q Zhu, Y. Guo, and W. Lin. Neural delay differential equations, 2021.
13


Reviews:
[{"review_id": "b3QNEKeRM8A", "reviewer": "Reviewer_2gyi", "summary": "This paper extends prior work on stable deep neural network dynamics models to learn stable models with partially observed states, and to learn (simple) control policies with time-delayed feedback. To achieve this, the authors minimize a Lyapunov-Razumikhin loss term when training _neural delay differential equations_. They demonstrate the capabilities of their system on a few tasks.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is the novel application of Lyapunov-Razumikhin stability to Neural DDEs. Applicable related work is cited.\n\n## Quality\n\nThe derivation of the Lyapunov-Razumikhin loss term proceeds methodically from the background section. The special care taken with the Lyapunov NN construction and the LRF loss discretization is especially appreciated.\n\nThe authors are forthright in their evaluation of their own work: they clearly highlight potential pitfalls (such as how the stability certification is for each trajectory and not global, etc.) \n\n### Experiments\n\nTheir choice of a two-link pendulum with partially-observed states is a reasonable evaluation task to highlight how the LRF loss prevents blow-up. This is only a single evaluation, though, and it is not obvious to me that ANODE (at least, when used with true initial conditions) will fail at solving the two-link pendulum.\n\nHaving more experiments would make this much better. In particular, if they can find a condition in which ANODE/vanilla NDDEs fail to learn but stable NDDEs succeed, this would further support their claim on LRF acting to regularize training (line 294). One way to do this would be to try this on three- or more-link pendulums as well.\n\nThe delayed feedback control experiment is well-selected.\n\nThe experiment tasks are all relatively simple and small. The paper would be much better served by presenting either a more complex “showcase” task, or a comparison to a common benchmark.\n\n## Clarity\n\nThe submission is well-organized and mostly clear. Some key experimental details are in the appendix, but given that the focus of this paper is on the methodology, this is reasonable. (For the delayed feedback control experiment, I suggest mentioning the selection of $\\tau=0.03$ in the experimental section itself.)\n\nThere is a little ambiguity in the creation of the initial history: in particular, lines 248-254, I’m not sure which data is used to train/fit the GP-mean, and if this is replaced/appended to at each iteration.\n\nAs this is a stability-by-loss-function approach, there is also some concern on how well the stability generalizes. (Answering the question “how likely are other trajectories produced by the model to be stable?”) Presenting some experiment quantifying this would alleviate this concern.\n\n## Significance\nThis work joins a body of previous work on stable neural-network dynamics models using loss terms. Its ability to deal with partially-observable states and delayed control is somewhat interesting. The experimental results don’t showcase what the method could do in practice.\n\nTo make the results compelling depends on what this method can do in practice. Perhaps this could be showcased by performance on a more complicated common control-theory benchmark task (like a delayed or partially-observed cartpole?) \n\n(Minor formatting issue: please consider allowing Latex to position figures instead of embedding them in text flow, It improves readability.) \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper's technical limitations have been addressed in the review. This is basic research, and has no direct negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "WIlirpMwI3n", "reviewer": "Reviewer_Guwy", "summary": "The paper presents a method to learn the dynamics of stable systems. The authors first exploit Neural Delay Differential Equations to model the partial observability and time delay of the system. They then employ Lyapunov-Razumikhin Function to ensure the stability of the learned dynamics. The authors evaluate their methods by presenting the sampled stable trajectories and loss plots of the learned model against other selected models.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength] The paper has the following strength.\n- The paper studies the modeling of the interesting type of time-delayed system and extends to partially observed stable systems. \n- The proposed method is learning stable dynamics based on sampled trajectories in a data driven way. \n- The method can be combined with policy learning to jointly learn a stabilizing policy and the dynamics of the closed loop system.\n\n[Weakness] However, It has the following weakness about the paper which has stopped me from recommending the paper for publication in NeurIPS.\n- The paper discusses little and conducts no comparison against the line of work from recurrent neural networks (RNNs) based effort on modeling system dynamics. Even though RNNs may lack the capability of modeling continuous dynamics, I think a more thorough discussion and empirical experiments here would increase the completeness of the work.\n- The model introduces a few parameters to tune: K, samples of historical observations, r, the length of history to consider, and alpha, q for the exponential stability of the learned system. The parameters are highly tailored with the system dynamics to learn. That is, you would not be able to learn a good model for a (internally) higher dimensional partially observed systems with a small K (this is mentioned by authors as there would not be a one-to-one mapping from internal states to the coordinates). I am expecting a discussion on choosing the parameters and a demonstration/analysis of performance of the method against these parameters.\n- The method bases on optimizing a loss function where a loss term is introduced to encourage stability. This only indirectly ensures stability. I am concerned that unseen observations could lead to non-stabilizing trajectories. If any theoretical guarantee could be established on this, it would increase the theoretical contribution of the work.\n- The experiments has only compared the method against ANODE where ANODE's failure is shown through non-matching phase plots, but the success of the proposed method is only shown through lower loss. This alone is insufficient in showing a better performance of the proposed method as the worse performing ANODE with learned IC achieves lower loss than ANODE with true IC. More experiments could help make the point clearer.\n- The method ports the established method of Neural Delay Differential Equations to the dynamics learning problem. The theoretical analysis of the method bases on established results clearly given in other papers -- The only theorem in this paper is given by (Efimov and Aleksandrov, 2020). The experiments have not shown any special benefits of the method against existing methods. Therefore, I am concerned that the novelty of the work is not up to the standard of NeurIPS.\n\n_________________________________\nPost rebuttal:\nI have read the responses from the authors and appreciate the author's responses to my questions. However, I would like to stick with my original reviews and scoring.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "4iDWxUzc-A5", "reviewer": "Reviewer_Xxju", "summary": "This paper aims to learn stable deep dynamics models for partially observed or delayed dynamical systems. The authors build their method upon Neural Delay Differential Equations (NDDEs) [1], where the main idea is to augment the state with the past history, thus avoiding the estimation of unobserved augmented states. To stabilize the learned dynamics, the authors then propose to combine NDDEs with a Lyapunov-like regularization term.\n\nThe authors show that NDDEs are powerful tools to model partially observed dynamical systems where the dynamics are non-Markovian. Furthermore, they demonstrate that the proposed method outperforms Augmented Neural ODEs (ANODE) in partially observed systems and is able to learn a stabilizing feedback policy for control systems with input delays.\n\n[1] Qunxi Zhu, Yao Guo, Wei Lin, \"Neural Delay Differential Equations\"", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\nThis paper targets an important research question of learning stable deep dynamics models for partially observable scenarios. Deep neural networks are powerful tools to fit the observation data, yet additional treatments have to be in place to ensure that the learned model satisfies certain properties like stability, energy preservation, etc.\n\nThis paper is on the line of adding structures to the learning procedure based on our understanding of the dynamical systems. The authors take a step forward by extending to partially observed or delayed systems, which I believe has the potential to apply to safety-critical systems where it is crucial to have guarantees on convergence.\n\nThe authors provide theoretical analysis and convergence guarantees on the proposed method. \n\n\n[Weakness]\n\nMy primary concern of this paper is its novelty. This work builds on top of Neural Delay Differential Equations (NDDE) [1] by jointly learning a neural network Lyapunov-Razumikhin function. However, similar techniques for stabilizing dynamical systems with neural Lyapunov functions have already been investigated in [2, 3] and many others. Thus, to me, the proposed method seems to be a direct combination of existing methods, making this paper a bit insufficient in technical novelty.\n\nMy second concern is the lack of evaluation tasks. For evaluation, the authors only consider a simple hand-designed system, a partially observed damped double pendulum, and a friction-less inverted pendulum. However, for papers that line the similar direction like [3], they considered much more challenging cases, including wheeled vehicles and humanoid balancing. In [4], they investigated the n-link pendulum and high-dimensional systems like video texture generation. The current set of evaluating environments may be a bit too simple, limiting the significance and potential impact of this paper.\n\n\n[1] Qunxi Zhu, Yao Guo, Wei Lin, \"Neural Delay Differential Equations\"\n\n[2] Spencer M. Richards, Felix Berkenkamp, Andreas Krause, \"The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems\"\n\n[3] Ya-Chien Chang, Nima Roohi, Sicun Gao, \"Neural Lyapunov Control\"\n\n[4] Gaurav Manek, J. Zico Kolter, \"Learning Stable Deep Dynamics Models\"\n\n\n================================\n\nPost-rebuttal comments\n\nI have read the reviews from other reviewers and the rebuttal from the authors. I appreciate the authors' efforts in providing the additional experimental results and the detailed discussions about the novelty of the proposed method and the comparison with previous methods. I encourage the authors to include them in the revised version of the paper.\n\nThe rebuttal has sufficiently addressed my concerns, and I raised my score from 5 (Marginally below the acceptance threshold) to 6 (Marginally above the acceptance threshold).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see my main review for the limitations, and I do not see any obvious potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to NeurIPS.  Although there is still some disagreement between reviewers after the rebuttal period, overall the proposed paper seems to be a solid contribution to the emerging field of modeling continuous-time dynamical systems with delays and/or partial observability.  The work is perhaps a bit incremental in that it is largely a combination of past approaches, including past work in Neural DDEs and past work on ensuring the stability of dynamical systems specified by Lyapunov functions.  But it also combines these past methods in new and rather interesting ways, and I believe is worthy of highlighting at NeurIPS.  Thus, while the paper is somewhat borderline, I am recommending it be accepted, with the strong comment that the authors address the reviewer concerns that _can_ be addressed (i.e., not just the general concerns about significance) as they did in the rebuttal.

Decision: Accept (Poster)