SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Appendix C, we further demonstrate empirically that SNDEs are insensitive to the specific choice of  $\gamma$  over a large range (beyond a minimum value, consistent with Theorem 2). We also provide more intuition about choosing  $\gamma$  and the computational implications of this choice. In practice, we find that SNDEs are easy to use across a wide variety of settings with minimal tuning and incur only moderate training overhead compared to vanilla NDEs. In some cases, SNDEs are even computationally cheaper than vanilla NDEs at inference time despite the cost of computing the pseudoinverse. Finally, Appendix D provides results of additional experiments not included here due to space constraints.
5.1 Two-Body Problem [second order, autonomous, non-chaotic, conservation law]
The motion of two bodies attracting each other with a force inversely proportional to their squared distance (e.g., gravitational interaction in the non-relativistic limit) can be written as
$$
\ddot {x} = - \frac {x}{\left(x ^ {2} + y ^ {2}\right) ^ {3 / 2}}, \quad \ddot {y} = - \frac {y}{\left(x ^ {2} + y ^ {2}\right) ^ {3 / 2}}, \tag {13}
$$
where one body is fixed at the origin and  $x, y$  are the (normalized) Cartesian coordinates of the other body in the plane of its orbit [43]. We stabilize the dynamics with respect to the conserved angular momentum  $L$ , yielding
$$
\mathcal {M} = \{(x, y) \in \mathbb {R} ^ {2}; x \dot {y} + y \dot {x} - L _ {0} = 0 \}, \tag {14}
$$
Figure 2: Top row: Results for the two-body problem experiment, showing a single test trajectory in (a) and averages over 100 test trajectories in (b-c). Middle row: Results for the rigid body rotation experiment, showing a single test trajectory in (d) and averages over 100 test trajectories in (e-f). Bottom row: Results for the DC-to-DC converter experiment, showing the voltage  $v_{1}$  across the first capacitor during a single test trajectory in (g), and averages over 100 test trajectories in (h-i). The vanilla NODE (blue) is unstable in all settings, quickly drifting from the constraint manifold and subsequently diverging exponentially, while the vanilla ANODE (green) is unstable for the rigid body and DC-to-DC converter experiments. In contrast, the SNODE (red) and SANODE (purple) are constrained to the manifold with accurate predictions over a long horizon in all settings. Confidence intervals are not shown as they diverge along with the unstabilized trajectories.
where  $L_{0}$  is the initial value of  $L$ . We train on 40 trajectories with initial conditions  $(x, y, \dot{x}, \dot{y}) = (1 - e, 0, 0, \sqrt{1 - e / 1 + e})$ , where the eccentricity  $e$  is sampled uniformly via  $e \sim U(0.5, 0.7)$ . Each trajectory consists of a single period of the orbit sampled with a timestep of  $\Delta t = 0.1$ .
The top row of Figure 2 shows that SNODEs, ANODEs and SANODEs all achieve stable long-term prediction over multiple orbits, while unstabilized NODEs diverge exponentially from the correct orbit.
5.2 Motion of a Rigid Body [first order, autonomous, non-chaotic, holonomic constraint]
The angular momentum vector  $y = (y_{1},y_{2},y_{3})^{T}$  of a rigid body with arbitrary shape and mass distribution satisfies Euler's equations of motion,
$$
\binom {\dot {y} _ {1}} {\dot {y} _ {2}} = \left( \begin{array}{c c c} 0 & - y _ {3} & y _ {2} \\ y _ {3} & 0 & - y _ {1} \\ - y _ {2} & y _ {1} & 0 \end{array} \right) \binom {y _ {1} / I _ {1}} {\binom {y _ {2} / I _ {2}} {y _ {3} / I _ {3}}}, \tag {15}
$$
where the coordinate axes are the principal axes of the body,  $I_1, I_2, I_3$  are the principal moments of inertia, and the origin of the coordinate system is fixed at the body's centre of mass [43]. The motion of  $y$  conserves the Casimir function  $C(y) = \frac{1}{2}\left(y_1^2 + y_2^2 + y_3^2\right)$ , which is equivalent to conservation of angular momentum in the orthogonal body frame and constitutes a holonomic constraint on the allowed states of the system. We therefore have the manifold
$$
\mathcal {M} = \left\{\left(y _ {1}, y _ {2}, y _ {3}\right) \in \mathbb {R} ^ {3}; y _ {1} ^ {2} + y _ {3} ^ {2} + y _ {3} ^ {2} - C _ {0} = 0 \right\}. \tag {16}
$$
We train on 40 trajectories with initial conditions  $(y_{1},y_{2},y_{3}) = (\cos (\phi),0,\sin (\phi))$ , where  $\phi$  is drawn from a uniform distribution  $\phi \sim U(0.5,1.5)$ . Each trajectory consists of a 15 second sample with a timestep of  $\Delta t = 0.1$  seconds.
The middle row of Figure 2 demonstrates that, unlike vanilla NODEs and ANODEs, SNODEs and SANODEs are constrained to the sphere and stabilize the predicted dynamics over a long time horizon in this first order system.
5.3 DC-to-DC Converter [first order, non-autonomous, non-chaotic, conservation law]
We now consider an idealized DC-to-DC converter [52, 33], illustrated in Figure 3, with dynamics
$$
C _ {1} \dot {v} _ {1} = (1 - u) i _ {3}, \quad C _ {2} \dot {v} _ {2} = u i _ {3}, \quad L _ {3} \dot {i} _ {3} = - (1 - u) v _ {1} - u v _ {2}, \tag {17}
$$
where  $v_{1}, v_{2}$  are the state voltages across capacitors  $C_{1}, C_{2}$ , respectively,  $i_{3}$  is the state current across an inductor  $L_{3}$ , and  $u \in \{0, 1\}$  is a control input (a switch) that can be used to transfer energy between the two capacitors via the inductor. The total energy in the circuit,  $E = \frac{1}{2} (C_{1}v_{1}^{2} + C_{2}v_{2}^{2} + L_{3}i_{3}^{2})$ , is conserved, yielding the manifold
$$
\mathcal {M} = \left\{\left(v _ {1}, v _ {2}, i _ {3}\right) \in \mathbb {R} ^ {3}; C _ {1} v _ {1} ^ {2} + C _ {2} v _ {2} ^ {2} + L _ {3} i _ {3} ^ {2} - E _ {0} = 0 \right\}. \tag {18}
$$
Figure 3: Idealized schematic of a DC-to-DC converter.
We train on 40 trajectories integrated over 10 seconds with a timestep of  $\Delta t = 0.1$  seconds, where  $C_1 = 0.1$ ,  $C_2 = 0.2$ ,  $L_3 = 0.5$ , and a switching period of 3 seconds, i.e., the switch is toggled every 1.5 seconds. The initial conditions for  $(v_{1}, v_{2}, i_{3})$  are each drawn independently from a uniform distribution  $U(0,1)$ .
The bottom row of Figure 2 shows the voltage across  $C_1$  over multiple switching events (g), with the NODE and ANODE quickly accumulating errors every time the switch is applied, while the SNODE and SANODE remain accurate for longer. Panels (h,i) show the familiar exponentially accumulating errors for vanilla NODE and ANODE, versus stable relative errors for SNODE and SANODE.
5.4 Controlled Robot Arm [first order, non-autonomous, non-chaotic, time-dependent control]
Next, we apply SNDEs to solve a data-driven inverse kinematics problem [64], that is, learning the dynamics of a robot arm that satisfy a prescribed path  $p(t)$ . We consider an articulated robot arm consisting of three connected segments of fixed length 1, illustrated in Figure 4(a). Assuming one end of the first segment is fixed at the origin and the robot arm is restricted to move in a plane, the endpoint  $e(\theta)$  of the last segment is given by
$$
e (\theta) = \binom {\cos \left(\theta_ {1}\right) + \cos \left(\theta_ {2}\right) + \cos \left(\theta_ {3}\right)} {\sin \left(\theta_ {1}\right) + \sin \left(\theta_ {2}\right) + \sin \left(\theta_ {3}\right)}, \tag {19}
$$
where  $\theta_{j}$  is the angle of the  $j$ -th segment with respect to the horizontal and  $\theta = (\theta_{1},\theta_{2},\theta_{3})$ . The problem consists of finding the motion of the three segments  $\theta (t)$  such that the endpoint  $e(\theta)$  follows
Figure 4: Controlled robot arm. (a) Schematic of the robot arm. (b) Snapshot of a single test trajectory. After 100 seconds the NODE (blue) has drifted significantly from the prescribed control while the SNODE (red) accurately captures the ground truth dynamics (black). (c) Relative error in the endpoint  $e(\theta)$  averaged over 100 test trajectories. The NODE (blue) accumulates errors and leaves the prescribed path, while the SNODE (red) remains accurate. Shadings in (c) are  $95\%$  confidence intervals.
a prescribed path  $p(t)$  in the plane, i.e.,  $e(\theta) = p(t)$ . Minimizing  $||\dot{\theta}(t)||$ , it can be shown [42] that the optimal path satisfies
$$
\dot {\theta} = e ^ {\prime} (\theta) ^ {T} \left(e ^ {\prime} (\theta) e ^ {\prime} (\theta) ^ {T}\right) ^ {- 1} \dot {p} (t), \tag {20}
$$
where  $e'$  is the Jacobian of  $e$ . These will be our ground truth equations of motion.
We stabilize the SNODE with respect to the (time-dependent) manifold
$$
\mathcal {M} = \left\{(\theta , t) \in \mathbb {S} \times \mathbb {R}; e (\theta) - p (t) = 0 \right\}. \tag {21}
$$
In particular, we prescribe the path
$$
p (t) = e _ {0} - \binom {\sin (2 \pi t) / 2 \pi} {0}, \tag {22}
$$
where  $e_0$  is the initial position of the endpoint, such that  $e(\theta)$  traces a line back and forth on the  $x$ -axis. We train on 40 trajectories of duration 5 seconds, with timestep  $\Delta t = 0.1$  and initial conditions  $(\theta_{1},\theta_{2},\theta_{3}) = (\theta_{0}, - \theta_{0},\theta_{0})$ , where  $\theta_0$  is drawn from a uniform distribution  $\theta_0\sim U(\pi /4,3\pi /8)$ . Additionally, we provide the network with  $\dot{p}$ , the time derivative of the prescribed control.
Figure 4 shows that the unconstrained NODE drifts substantially from the prescribed path during a long integration, while the SNODE implements the control to a high degree of accuracy and without drift.
5.5 Double Pendulum [second order, autonomous, chaotic, conservation law]
Finally, we apply stabilization to the chaotic dynamics of the frictionless double pendulum system. The total energy  $E$  of the system is conserved [7], yielding the manifold,
$$
\mathcal {M} = \left\{\left(\theta_ {1}, \theta_ {2}, \omega_ {1}, \omega_ {2}\right) \in \mathbb {S} ^ {2} \times \mathbb {R} ^ {2}; E \left(\theta_ {1}, \theta_ {2}, \omega_ {1}, \omega_ {2}\right) - E _ {0} = 0 \right\}, \tag {23}
$$
where  $\theta_{i}$  is the angle of the  $i$ -th arm with the vertical and  $\omega_{i} = \dot{\theta}_{i}$ . We refer the reader to Arnold [7] (or the excellent Wikipedia entry) for the lengthy equations of motion and an expression for the total energy. For simplicity we take  $m_{1} = m_{2} = 1\mathrm{kg}$ ,  $l_{1} = l_{2} = 1\mathrm{m}$ , and  $g = 9.81\mathrm{ms}^{-2}$ . We train on 40 trajectories, each consisting of 10 seconds equally sampled with  $\Delta t = 0.05$ , and with initial conditions  $(\theta_{1},\theta_{2},\omega_{1},\omega_{2}) = (\phi ,\phi ,0,0)$ , where  $\phi$  is drawn randomly from a uniform distribution  $\phi \sim U(\pi /4,3\pi /4)$ . We emphasize that this is a highly limited amount of data when it comes to describing the chaotic motion of the double pendulum system, intended to highlight the effect of stabilization in the low-data regime.
Figure 5(a-b) shows that, while initially the SNODE only marginally outperforms the vanilla NODE in terms of the relative error of the state, the longer term relative error in energy is substantially
Figure 5: Results for the double pendulum. (a) Relative error in the state over 300 short test trials, shown with  $95\%$  confidence intervals (shaded). Compared to the SNODE, the NODE diverges rapidly as it begins to accumulate errors in the energy. (b) Relative error in the energy averaged over 5 long test trials. (c) Comparison of the double pendulum's invariant measure estimated by the (hybrid) UDE, with and without stabilization, versus ground truth, with  $95\%$  confidence intervals.
larger for NODE than for SNODE. A certain relative error in state is indeed unavoidable for chaotic systems.
In addition to predicting individual trajectories of the double pendulum, we also consider an additional important task: learning the invariant measure of this chaotic system. This can be motivated by analogy with climate predictions, where one also focuses on long-term prediction of the invariant measure of the system, as opposed to predicting individual trajectories in the sense of weather forecasting, which must break down after a short time due to the highly chaotic underlying dynamics. Motivated by hybrid models of the Earth's climate [36, 71], we choose a slightly different training strategy than before, namely a hybrid setup in line with the UDE approach mentioned above. In particular, the dynamics of the first arm  $\ddot{\theta}_{1}$  are assumed known, while the dynamics of the second arm  $\ddot{\theta}_{2}$  are learned from data. We train on a single trajectory of duration 60 seconds with  $\Delta t = 0.05$ . For each trained model, we then integrate ten trajectories of duration one hour - far longer than the observed data. An invariant measure is estimated from each long trajectory (see Appendix E) and compared with the ground truth using the Hellinger distance.
Figure 5(c) shows that, as  $\gamma$  is increased, our ability to accurately learn the double pendulum's invariant measure increases dramatically due to stabilization, demonstrating that the "climate" of this system is captured much more accurately by the SNDE than by the NDE baseline.
We have introduced stabilized neural differential equations (SNDEs), a method for learning dynamical systems from observational data subject to arbitrary explicit constraints. Our approach is based on a stabilization term that is cheap to compute and provably renders the invariant manifold asymptotically stable while still admitting all solutions of the vanilla NDE on the invariant manifold. Key benefits of our method are its simplicity and generality, making it compatible with all common NDE methods without requiring further architectural changes. Crucially, SNDEs allow entirely new types of constraints, such as those arising from known conservation laws and controls, to be incorporated into neural differential equation models. We demonstrate their effectiveness across a range of settings, including first and second order systems, autonomous and non-autonomous systems, with constraints stemming from holonomic constraints, conserved first integrals of motion, as well as time-dependent restrictions on the system state. SNDEs are robust with respect to the only tuneable parameter and incur only moderate computational overhead compared to vanilla NDEs.
The current key limitations and simultaneously interesting directions for future work include adapting existing methods for NODEs on Riemannian manifolds to our setting, generalizations to partial differential equations, allowing observations and constraints to be provided in different coordinates, and scaling the method to high-dimensional settings such as learning dynamics from pixel observations, for example in fluid dynamics or climate modeling. Finally, we emphasize that high-dimensional, nonlinear dynamics may not be identifiable from just a small number of solution trajectories. Hence, care must be taken when using learned dynamics in high-stakes scenarios (e.g., human robot interactions), especially when going beyond the training distribution.
This work was funded by the Volkswagen Foundation. The authors would like to thank Christof Schötz, Philipp Hess and Michael Lindner for many insightful discussions while preparing this work, as well as the anonymous reviewers whose thoughtful feedback significantly improved the paper. The authors also thank the creators and maintainers of the Julia programming language [13] and the open-source packages DifferentialEquations.jl [66], Flux.jl [46], Zygote.jl [45], ComplexityMeasures.jl [41] and Makie.jl [27], all of which were essential in preparing this work. The authors gratefully acknowledge the European Regional Development Fund (ERDF), the German Federal Ministry of Education and Research and the Land Brandenburg for supporting this project by providing resources on the high performance computer system at the Potsdam Institute for Climate Impact Research.
[1] Pierre Ablin and Gabriel Peyré. Fast and accurate optimization on the orthogonal manifold without retraction. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 5636-5657. PMLR, 2022.
[2] Pierre-Antoine Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008.
[3] Hananeh Aliee, Fabian J. Theis, and Niki Kilbertus. Beyond predictions in neural ODEs: Identification and interventions. arXiv preprint arXiv:2106.12430, 2021.
[4] Hananeh Aliee, Till Richter, Mikhail Solonin, Ignacio Ibarra, Fabian J. Theis, and Niki Kilbertus. Sparsity in continuous-depth neural networks. In Advances in Neural Information Processing Systems, pages 901-914, 2022.
[5] Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceedings of The 33rd International Conference on Machine Learning, pages 1120-1128. PMLR, 2016.
[6] Ludwig Arnold, Christopher K.R.T. Jones, Konstantin Mischeikow, Genevieve Raugel, and Ludwig Arnold. Random Dynamical Systems. Springer, 1995.
[7] Vladimir Igorevich Arnold. Mathematical Methods of Classical Mechanics. Springer New York, 2nd edition, 1989.
[8] Uri M. Ascher and Linda R. Petzold. Computer Methods for Ordinary Differential Equations and Differential-Algebraic Equations. Society for Industrial and Applied Mathematics, 1998.
[9] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? In Advances in Neural Information Processing Systems, pages 4261-4271, 2018.
[10] J. Baumgarte. Stabilization of constraints and integrals of motion in dynamical systems. Computer Methods in Applied Mechanics and Engineering, 1(1):1-16, 1972.
[11] Soren Becker, Michal Klein, Alexander Neitz, Giambattista Parascandolo, and Niki Kilbertus. Predicting ordinary differential equations with transformers. In Proceedings of the 40th International Conference on Machine Learning, pages 1978-2002. PMLR, 2023.
[12] Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Maximillian Nickel, Aditya Grover, Ricky T. Q. Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. In Proceedings of the 39th International Conference on Machine Learning, pages 1749-1763. PMLR, 2022.
[13] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM Review, 59(1):65-98, 2017.
[14] Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In Proceedings of the 37th International Conference on Machine Learning, pages 1045-1055. PMLR, 2020.
[70] Yulia Rubanova, Ricky T.Q. Chen, and David K. Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. In Advances in Neural Information Processing Systems, pages 5320-5330, 2019.
[71] Chaopeng Shen, Alison P. Appling, Pierre Gentine, Toshiyuki Bandai, Hoshin Gupta, Alexandre Tartakovsky, Marco Baity-Jesi, Fabrizio Fenicia, Daniel Kifer, Li Li, Xiaofeng Liu, Wei Ren, Yi Zheng, Ciaran J. Harman, Martyn Clark, Matthew Farthing, Dapeng Feng, Praveen Kumar, Doaa Aboelyazed, Farshid Rahmani, Yalan Song, Hylke E. Beck, Tadd Bindas, Dipankar Dwivedi, Kuai Fang, Marvin Höge, Chris Rackauckas, Binayak Mohanty, Tirthankar Roy, Chonggang Xu, and Kathryn Lawson. Differentiable modelling to unify machine learning and physical models for geosciences. Nature Reviews Earth & Environment, 4(8):552-567, 2023.
[72] Takeshi Teshima, Koichi Tojo, Masahiro Ikeda, Isao Ishikawa, and Kenta Oono. Universal approximation property of neural ordinary differential equations. arXiv preprint arXiv:2012.02414, 2020.
[73] Ch. Tsitouras. Runge-Kutta pairs of order 5 (4) satisfying only the first column simplifying assumption. Computers & Mathematics with Applications, 62(2):770-775, 2011.
[74] James H. Verner. Numerically optimal Runge-Kutta pairs with interpolants. Numerical Algorithms, 53(2-3):383-396, 2010.
[75] Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural ODEs and invertible residual networks. In Proceedings of the 37th International Conference on Machine Learning, pages 11086-11095. PMLR, 2020.
[76] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-Net: Learning Hamiltonian dynamics with control. In 8th International Conference on Learning Representations, 2020.
[77] Aiqing Zhu, Pengzhan Jin, and Yifa Tang. Deep Hamiltonian networks based on symplectic integrators. arXiv preprint arXiv:2004.13830, 2020.
A Differential Algebraic Equations
A differential algebraic equation (DAE) in its most general, implicit form is
$$
F (t, x, \dot {x}) = 0, \tag {24}
$$
where  $x \in \mathbb{R}^n$  and  $F \colon \mathbb{R} \times \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}^n$ . When  $\partial F / \partial \dot{x}$  is nonsingular, Equation (24) is an implicit ODE and by the implicit function theorem may be written as an explicit ODE in the form  $\dot{x} = f(x,t)$  [34]. In the more interesting case of singular  $\partial F / \partial \dot{x}$ , an important special case of Equation (24) is given by semi-explicit DAEs in Hessenberg form, for example,
$$
\dot {y} = f (t, y, z) \tag {25a}
$$
$$
0 = g (t, y, z), \tag {25b}
$$
where  $x = (y, z)$ . We call  $y$  the differential variables, since their derivatives appear in the equations, and  $z$  the algebraic variables, since their derivatives do not. The semi-explicit form of Equation (25) highlights the connection between certain classes of DAEs and ODEs subject to constraints.
It is generally possible to differentiate the constraints in Equation (25b) a number of times and substitute the result into Equation (25a) to obtain a mathematically equivalent ODE. The number of differentiations required to do so is the differential index of the DAE, and corresponds loosely to the "distance" of the DAE from an equivalent ODE. The differential index – and related measures of index not directly based on differentiation – is used extensively to classify DAEs, especially in the context of numerical methods for their solution [15]. Each differentiation of the constraints reduces the index of the system by one (ODEs have index 0).
Of particular interest in the context of this paper are constrained ODEs of the form
$$
\dot {u} = f (t, u) \tag {26a}
$$
$$
0 = g (t, u), \tag {26b}
$$
where  $u \in \mathbb{R}^n$ ,  $f \colon \mathbb{R} \times \mathbb{R}^n \to \mathbb{R}^n$ , and  $g \colon \mathbb{R} \times \mathbb{R}^n \to \mathbb{R}^m$ . Equation (26) can be written as a semi-explicit Hessenberg index-2 DAE, with  $u$  as the differential variables and  $m$  Lagrange multipliers as the algebraic variables, for example,
$$
\dot {u} = f (t, u) - D (u) \lambda \tag {27a}
$$
$$
0 = g (t, u), \tag {27b}
$$
where  $\lambda \in \mathbb{R}^m$  and  $D(u)$  is any bounded matrix function such that  $GD$ , where  $G = g_{u}$  is the Jacobian of  $g$ , is boundedly invertible for all  $t$  [8]. We can therefore approach the task of solving the constrained ODE Equation (26) from the perspective of solving the Hessenberg index-2 DAE Equation (27).
DAEs are not ODEs, however, and a number of additional complications arise when we seek numerical solutions [65]. Generally, the higher the index, the harder it is to solve a given DAE. For this reason, it is common to first perform an index reduction (i.e. differentiate the constraints) before applying numerical methods. However, the numerical solution of the resulting index-reduced system may exhibit drift off from the invariant manifold defined by the original constraints. For this reason, Baumgarte [10] proposed a stabilization procedure for index-reduced DAEs that renders the invariant manifold asymptotically stable. Baumgarte's stabilization is, in turn, a special case of the stabilization procedure later proposed by Chin [21] and adopted by us in this paper. We emphasize, however, that our stabilization procedure addresses a different application and problem than these related methods; while Baumgarte and Chin sought to stabilize drift off from the invariant manifold due to discretization error in the numerical solution of an index-reduced DAE, we seek to constrain some learned dynamics imperfectly approximated by a neural network.