SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, one may ask why a neural network could not be incorporated directly into Equation (27) and the resulting index-2 DAE solved directly. While possible in principle, DAEs require implicit numerical methods, with the result that the computational complexity of computing gradients of solutions – whether via automatic differentiation or adjoint sensitivity analysis – scales with the cube of the system size [51]. In contrast, backpropagating through explicit solvers has linear complexity.
Figure 6: Comparison of the alternative, faster SNDE formulation in Equation (29) with the standard SNDE formulation in Equation (12), applied to the controlled robot arm experiment. (a) After 100 seconds, both the fast SNODE (green) and the standard SNODE (orange, not visible behind green) continue to implement the prescribed path exactly, while the NODE (blue) has drifted significantly. (b) Relative errors in the position of the endpoint, averaged over 10 test trajectories. The fast SNODE (green) admits slightly larger errors than the standard SNODE (orange), although the difference is negligible in this setting.
B Faster Alternatives to the Pseudoinverse
The computational cost of the pseudoinverse is  $\mathcal{O}(m^2 n)$ , where  $m$  is the number of constraints and  $n$  is the dimension of the problem. While cheap to compute for the systems studied in this paper, the pseudoinverse may become prohibitively expensive for applications with a large number of constraints. As an alternative choice for the stabilization matrix  $F(u)$  in Equation (5), we recommend using the transpose of the Jacobian  $G(u) = g_u$ , i.e.,
$$
F (u) = G ^ {T} (u), \tag {28}
$$
which is instead  $\mathcal{O}(1)$ . Since  $G^T(u)G(u)$  is symmetric positive definite, the theoretical guarantees of Theorem 1 and Theorem 2 still apply, such that
$$
\boxed {\dot {u} = f _ {\theta} (u) - \gamma G ^ {T} (u) g (u)} \quad [ \text {a l t e r n a t i v e S N D E} ] \tag {29}
$$
is a valid SNDE.
We apply an SNDE in the form of Equation (29) to the controlled robot arm experiment, in which the Jacobian has dimensions  $2 \times 3$ . In this instance, a single evaluation of  $G^{T}(u)$  is approximately 10 times faster than the pseudoinverse  $G^{+}(u)$ , which translates to a  $10\%$  speedup when evaluating the entire right-hand side of the SNDE (the cost of evaluating of the neural network still dominates in this setting).
Figure 6 shows that the alternative SNDE formulation implements the control effectively and illustrates the tradeoff when choosing the stabilization matrix  $F(u)$ . The pseudoinverse  $G^{+}$  can be expensive to compute but, as an orthogonal projection, yields the most "direct" stabilization to the constraint manifold and therefore the most accurate implementation of the constraint. The transpose  $G^{T}$ , on the other hand, is always cheap to compute and yields a valid stabilization, at the cost of only slightly larger errors in the constraint.
C The Stabilization Parameter  $\gamma$
C.1 Runtime Implications
We assess the computational cost of SNODEs compared to vanilla NODEs. SNODEs require the computation of (and backpropagation through) the pseudoinverse of the Jacobian of the constraint
Table 1: Training time of NODEs vs SNODEs. All experiments are trained for 1,000 epochs on an Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz. Statistics are calculated over 5 random seeds.
Model	γ	Training Time (seconds)
Two-Body Problem	Rigid Body	DC-to-DC Converter
Mean	Std. Dev.	Mean	Std. Dev.	Mean	Std. Dev.
NODE	-	10,580	271	9,730	71	14,000	316
SNODE	0.1	12,060	206	12,000	283	18,060	524
1	12,180	194	12,500	126	18,020	549
2	12,160	102	12,340	301	18,280	240
4	12,980	147	14,160	280	18,020	354
8	14,000	268	15,320	376	18,200	482
16	14,260	162	15,680	519	-	-
32	15,300	167	17,140	680	-	-
function  $g$ . Additionally, as  $\gamma$  is increased, SNODEs may require more solver steps at a given error tolerance, with the SNODE eventually becoming stiff for sufficiently large  $\gamma$ . Naively, one may thus expect a noticeable increase in runtime. However, as described in Section 2, the computational cost of training NODEs also depends on the "complexity" of the learned dynamics, which in turn determines how many function evaluations are required by the solver. This leads to nontrivial interactions between the added computation of enforcing constraints and the thereby potentially regularized "simpler" dynamics, which may require fewer function evaluations by the solver.
In Table 1, we report comparisons of training times between SNODEs and NODEs for different values of  $\gamma$  for three settings. SNODEs take roughly 1.2 to 1.8 times longer to train, with smaller values of  $\gamma$  incurring less overhead. Overall, this is a manageable increase for most relevant scenarios.
We also show inference times in Table 2. Here, the trend reverses and larger values of  $\gamma$  lead to lower inference times. This is because the solver requires fewer steps (has higher acceptance rates of proposed step sizes) for stronger stabilization. Hence, while predictive performance is largely unaffected, one can use the specific choice of  $\gamma$  as a tuning knob that trades off training time versus inference time.
C.2 Constraint Implications
To complement these results, Figure 7 shows that the relative error remains almost unchanged for a large range of  $\gamma$  values, that is, beyond a certain minimum value, SNODEs are not sensitive to the specific choice of  $\gamma$ . Even a value of  $\gamma = 1$  works well in the settings we have considered, indicating that we can typically get away with runtime increases of a factor of 1.2. However, larger values of  $\gamma$  only lead to slightly increased training times, while generally enforcing the constraints to a higher degree of accuracy.
C.3 A Practical Guide to Choosing  $\gamma$
As described in the previous sections, choosing the optimal value of  $\gamma$  requires a tradeoff between accuracy and training time; larger values of  $\gamma$  will enforce the constraints more accurately, at the cost of (mildly) additional training time. Our experience with the systems in this paper is that stabilization begins to work around  $\gamma \sim 1$  and remains effective as  $\gamma$  is increased, until the system becomes stiff around  $\gamma \sim 100$ . Within those limits, we suggest experimenting with a range of values, for example, powers of two.
Finally, we note that the stabilization term can be switched "on and off" at any time during training. For example, if computing the pseudoinverse is expensive, it may be beneficial to start training without stabilization, before switching the stabilization on once  $f_{\theta}$  is close to  $f$ .
Figure 7: Effect of  $\gamma$  on relative errors. Top row: Two-body problem (a-b). Middle row: Rigid body (c-d). Bottom row: DC-to-DC converter (e-f). Beyond a certain value, SNDEs are not highly sensitive to the choice of  $\gamma$ , although larger values may enforce the constraints more accurately.
Table 2: Inference time and (adaptive) solver statistics of SNODEs vs NODEs for the two-body problem experiment. Inference time statistics are calculated using 100 test initial conditions, each of which is integrated for 20 seconds (short enough so that the NODE solution does not diverge). Solver step statistics are reported for a single test trial, intended to illustrate the observed trends in inference time. SNODEs are cheaper at inference time due to significantly fewer rejected solver steps.
Model	Inference Time (seconds)	Solver Steps
Type	γ	Median	Mean	Accepted	Rejected	RHS Evaluations
NODE	-	2.44	2.51 ± 0.04	2,379	3,343	34,335
SNODE	0.1	2.14	2.17 ± 0.03	2,040	2,874	29,487
1	2.19	2.19 ± 0.03	2,054	2,704	28,551
2	2.22	2.27 ± 0.04	2,061	2,541	27,615
4	2.07	2.15 ± 0.05	2,180	2,382	27,375
8	2.05	2.07 ± 0.04	2,355	1,877	25,395
16	1.98	2.03 ± 0.05	2,677	1,437	24,687
32	1.96	1.99 ± 0.04	3,219	1,029	25,491
Table 3: Stable time of NODEs for the same 100 test trials as shown in Figure 2. NODE models (not shown) did not not diverge during any trial.
Experiment	Trial Length (seconds)	NODE Stable Time (seconds)
Min.	Max.	Median	Mean	Std. Dev.
Two-Body Problem	200.0	52.0	182.8	121.7	117.3	30.3
Rigid Body	1600.0	341.7	1600.0	1600.0	1349.9	468.4
DC-to-DC Converter	160.0	19.3	160.0	113.9	110.49	45.6
D.1 Hamiltonian Neural Networks
We train a Hamiltonian neural network (HNN) on the two-body problem, which is a Hamiltonian system (Figure 8). The HNN initially conserves angular momentum, suggesting that it has learned a reasonable approximation of the true Hamiltonina. However, it becomes unstable after approximately 50 seconds, highlighting the benefits of stabilization when data is limited and conservation laws are known.
D.2 Tangent Projection Operator
The invariant manifold of the rigid body experiment is a sphere. To constrain trajectories to the sphere, we can follow an approach similar to Rozen et al. [69] and Ben-Hamu et al. [12] and define a tangent projection operator (TPO),
$$
P (u) = I - \frac {u u ^ {T}}{\| u \| ^ {2}}, \tag {30}
$$
where  $I$  is the identity matrix.  $P(u)$  projects velocities at  $u$  to the tangent space of the sphere  $T_{u}\mathbb{S}^{2}$ . Applying the tangent projection operator of Equation (30) to the standard NDE model  $f_{\theta}$  for rigid body dynamics, we obtain
$$
\dot {u} = P (u) f _ {\theta} (u) \in T _ {u} \mathbb {S} ^ {2}. \tag {31}
$$
Equation (31) is a manifold ODE. In general, specialized numerical methods must be used when integrating manifold ODEs to guarantee the numerical solution remains on the manifold [43]. However, for the sake of this comparison, we find regular ODE solvers to be sufficient in practice.
Figure 8(d) shows that the TPO satisfies the constraint exactly, with relative errors of  $\sim 10^{-10}$ , compared to  $\sim 10^{-4}$  for the SNODE and SANODE. However, this does not translate to improved prediction of the system state (Figure 8(c)), suggesting that, in this setting, there is little practical benefit to the additional accuracy in the constraint.
D.3 Stable Time
The unstabilized, vanilla NODEs of Figure 2 are characterized by an initial drift from the invariant manifold that gives way to a subsequent rapid divergence. In practice, however, certain test trials may remain stable for significantly longer than others. In this section, we characterize the stable time  $T_{\mathrm{stab}}$  of an individual test trial as the time elapsed until the relative error  $E(t)$  in the predicted system state  $\hat{u}(t)$  exceeds a given threshold value  $E_{\mathrm{stab}}$ , i.e.
$$
T _ {\text {s t a b}} = \max  \left\{t \mid E (t) <   E _ {\text {s t a b}} \right\}. \tag {32}
$$
Taking  $E_{\mathrm{stab}} = 10^3$ , Table 3 shows  $T_{\mathrm{stab}}$  for the two-body problem, rigid body, and DC-to-DC converter experiments. Across several thousand test trials in this paper, not one stabilized NDE model diverged.
E Invariant Measure
Given that the double pendulum is a chaotic system, predictions of individual trajectories will break down after short times. We therefore also quantify the performance of NODEs and SNDEs in terms
Figure 8: Top row: Comparison with a Hamiltonian neural network (HNN) for the two-body problem experiment. Bottom row: Comparison with a tangent projection operator (TPO) for the rigid body experiment. Relative errors are calculated using the same 100 test initial conditions as in Figure 2. The HNN (a-b, pink) initially conserves angular momentum but is unstable for some initial conditions. The TPO (c-d, navy blue) conserves the Casimir function exactly. However, this does not translate to better prediction of the system state.
of their ability to capture the double pendulum's invariant measure. We refer to Arnold et al. [6] and Chekroun et al. [17] for detailed definitions of invariant measures; in short, a measure  $\mu$  is said to be invariant under some flow  $\Phi$  if  $\mu (\Phi^{-1}(t)(\mathcal{A})) = \mu (\mathcal{A})$  for all measurable sets  $\mathcal{A}$ . Invariant measures are commonly used to characterize the long-term dynamical characteristics of chaotic dynamical systems (see, for example, Arnold et al. [6] or Chekroun et al. [17] for a discussion of the invariant measure of the paradigmatic Lorenz-63 system). Since the double pendulum is an ergodic system, averages over long times approximate ensemble averages. We can therefore obtain a sample of the invariant measure numerically by integrating the system for a very long time. Concretely, we estimate the invariant measure from a single long trajectory using an algorithm due to Diego et al. [28], implemented in ComplexityMeasures.jl [41], based on a numerical estimate of the transfer operator. We then use the Hellinger distance [22] to compare the resulting probability distribution with the ground truth value for the double pendulum.
Table 4: Additional hyperparameters.
Experiment
Two-Body Problem	Rigid Body	DC-to-DC Converter	Robot Arm	Double Pendulum
γ	8	32	8	16	16
Hidden Layers	2	2	2	2	2
Hidden Width	128	64	64	128	128
Max LR	10-3	10-4	5 × 10-3	10-3	10-2
Min LR	10-5	10-5	10-5	10-5	10-4
Augmented Dimension	2	2	1	-	-
F Architecture and Training
Trajectories are generated using the 9(8) explicit Runge-Kutta algorithm due to Verner [74], implemented in DifferentialEquations.jl [66] as Vern9. To ensure that invariants are satisfied exactly in the training data, we use absolute and relative solver tolerances of  $10^{-24}$  in conjunction with Julia's BigFloat number type. Trajectories for training and validation are independent, that is, a given trajectory is used exclusively either for training or validation. Each trajectory is split using a multiple-shooting approach into non-overlapping chunks of 3 timesteps each.
Networks are implemented using Flux.jl [46] and consist of fully-connected dense layers with ReLU activation functions. All experiments are trained for 1,000 epochs using the AdamW optimizer [56] with weight decay of  $10^{-6}$  and an exponentially decaying learning rate schedule. During training, trajectories are integrated using the 5(4) explicit Runge-Kutta algorithm due to Tsitouras [73], implemented in DifferentialEquations.jl [66] as Tset5, with absolute and relative tolerances of  $10^{-6}$
The stabilization hyperparameter  $\gamma$  as well as network sizes, learning rates, and the number of additional dimensions for ANODEs are optimized for each experiment and are summarized in Table 4.