SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joon-Hyuk Ko†
Department of Physics & Astronomy
Seoul, 08826, South Korea
jhko725@snu.ac.kr
Nojun Park
MA, 02142, United States
bnj11526@mit.edu
Hankyul Koh†
physics113@snu.ac.kr
Wonho Jhe
whjhe@snu.ac.kr
Neural Ordinary Differential Equations (NeuralODEs) present an attractive way to extract dynamical laws from time series data, as they bridge neural networks with the differential equation-based modeling paradigm of the physical sciences. However, these models often display long training times and suboptimal results, especially for longer duration data. While a common strategy in the literature imposes strong constraints to the NeuralODE architecture to inherently promote stable model dynamics, such methods are ill-suited for dynamics discovery as the unknown governing equation is not guaranteed to satisfy the assumed constraints. In this paper, we develop a new training method for NeuralODEs, based on synchronization and homotopy optimization, that does not require changes to the model architecture. We show that synchronizing the model dynamics and the training data tames the originally irregular loss landscape, which homotopy optimization can then leverage to enhance training. Through benchmark experiments, we demonstrate our method achieves competitive or better training loss while often requiring less than half the number of training epochs compared to other model-agnostic techniques. Furthermore, models trained with our method display better extrapolation capabilities, highlighting the effectiveness of our method.
Predicting the evolution of a time varying system and discovering mathematical models that govern it is paramount to both deeper scientific understanding and potential engineering applications. The centuries-old paradigm to tackle this problem was to either ingeniously deduce empirical rules from experimental data, or mathematically derive physical laws from first principles. However, the complexities of systems of interest have grown so much that these traditional approaches are now often insufficient. This has led to a growing interest in using machine learning methods to infer dynamical laws from data.
One line of research stems from the dynamical systems literature, where the problem of interest was to determine the unknown coefficients of a given differential equation from data. While the task seems rather simple, it was found that naive optimization cannot properly recover the desired coefficients
due to irregularities in the optimization landscape, which worsens with increasing nonlinearity and data length [64, 50, 56, 1].
A parallel development occurred in the study of recurrent neural networks (RNNs), which are neural network analogues of discrete maps. It was quickly found that training these models can be quite unstable and results in exploding or vanishing gradients. While earlier works in the field resulted in techniques such as teacher forcing or gradient clipping that are agnostic to the RNN architecture [14, 45], more recent approaches have been directed towards circumventing the problem entirely by incorporating specific mathematical forms into the model architecture to constrain the eigenvalues of the model Jacobian and stabilize RNN dynamics [4, 29, 9, 34, 42]. While these latter methods can be effective, for the purpose of discovering dynamical equations underlying the data, these approaches are inadequate as one cannot ascertain the hidden equations conform to the predetermined rigid mathematical structures - especially so if the dynamics to be predicted is incompatible with the constrained model architecture [57, 41].
Figure 1: Optimization trajectories for our homotopy method (left) and vanilla gradient descent (right). While convention training meanders on the pathological loss landscape, our method provides a series of relaxed landscapes that effectively guide the optimizer to the loss minimum.
In this work, we focus on Neural Ordinary Differential Equations (NeuralODEs) [11]. These models are powerful tools in modeling natural phenomena, bridging the expressibility and flexibility of neural networks with the de facto mathematical language of the physical sciences. This has led to researchers amalgamating NeuralODEs with partial information about the governing equation to produce "grey-box" dynamics models [55, 71], and endowing NeuralODEs with mathematical structures that the system must satisfy [22, 18]. Despite their conceptual elegance, training NeuralODEs tend to result in long training times and sub-optimal results, a problem that is further exacerbated as the length of the training data grows [20, 17]. Similar to the case of RNNs, methods have been proposed to tackle the problem involve placing either strong [12, 25], or semi-strong constraints [17, 30] to the functional form the NeuralODE can take - something the underlying governing equation does not guarantee satisfying.
Contributions We introduce a method to accurately train NeuralODEs on long time series data by leveraging tools from the chaos and mathematical optimization literature: synchronization and homotopy optimization [1, 15]. Through loss landscape analyses, we show that longer training data lead to more irregular loss landscapes, which in turn lead to deteriorating performances of the trained models. We show that synchronizing NeuralODE dynamics with the data can smooth the loss landscape, on which homotopy optimization can be applied to enhance training. Through performance benchmarks, we show that not only does our model far outperform conventional gradient descent training, it also surpasses multiple shooting in terms of convergence speed and extrapolation accuracy.
2.1 Neural ordinary differential equations
A NeuralODE [11] is a model of the form,
$$
\frac {d \mathbf {u}}{d t} = \mathbf {U} (t, \mathbf {u}; \boldsymbol {\theta}), \quad \mathbf {u} \left(t _ {0}\right) = \mathbf {u} _ {0}. \tag {1}
$$
where  $\mathbf{u}_0\in \mathbb{R}^n$  is the initial condition or input given to the model, and  $\mathbf{U}(...;\pmb {\theta}):\mathbb{R}\times \mathbb{R}^n\to \mathbb{R}^n$  is a neural network with parameters  $\pmb {\theta}\in \mathbb{R}^{m}$  that governs the dynamics of the model state  $\mathbf{u}(t)\in \mathbb{R}^n$ . While this model can be used as a input-output mapping<sup>1</sup>, we are interested in the problem of training
In this case, NeuralODEs become the continuous analog of residual networks [11].
NeuralODEs on time series data to learn the underlying governing equation and forecast future dynamics.
Given an atomically increasing sequence of time points and the corresponding measurements  $\{t^{(i)},\hat{\mathbf{u}}^{(i)}\}_{i = 0}^{N}$ , NeuralODE training starts with using an ordinary differential equation (ODE) solver to numerically integrate Equation (1) to obtain the model state  $\mathbf{u}$  at given time points:
$$
\mathbf {u} ^ {(i)} (\boldsymbol {\theta}) = \mathbf {u} ^ {(0)} (\boldsymbol {\theta}) + \int_ {t ^ {(0)}} ^ {t ^ {(i)}} \mathbf {U} (t, \mathbf {u}; \boldsymbol {\theta}) d t = \text {O D E S o l v e} \left(\mathbf {U} (t, \mathbf {u}; \boldsymbol {\theta}), t ^ {(i)}, \mathbf {u} _ {0}\right). \tag {2}
$$
Afterwards, gradient descent is performed on the loss function  $\mathcal{L}(\pmb{\theta}) = \frac{1}{N + 1}\sum_{i}\left|\mathbf{u}^{(i)}(\pmb{\theta}) - \hat{\mathbf{u}}^{(i)}\right|^{2}$  to arrive at the optimal parameters  $\pmb{\theta}^*$ .
2.2 Homotopy optimization method
In topology, homotopy refers to the continuous deformation of one function into another. Motivated by this concept, to find the minimizer  $\pmb{\theta}^{*}\in \mathbb{R}^{m}$  of a complicated function  $\mathcal{F}(\pmb {\theta})$ , the homotopy optimization method [67, 68, 3, 15] introduces an alternative objective function
$$
\mathcal {L} (\boldsymbol {\theta}, \lambda) = \left\{ \begin{array}{l l} \mathcal {G} (\boldsymbol {\theta}), & i f \quad \lambda = 1 \\ \mathcal {F} (\boldsymbol {\theta}), & i f \quad \lambda = 0 \end{array} \right. \tag {3}
$$
where  $\mathcal{G}(\pmb{\theta})$  is an auxiliary function whose minimum is easily found, and  $\mathcal{L}(\pmb{\theta},\lambda):\mathbb{R}^m\times \mathbb{R}\to \mathbb{R}$  smoothly interpolates between  $\mathcal{G}$  and  $\mathcal{F}$  as the homotopy parameter  $\lambda$  varies from 1 to 0. While a commonly used construction is the convex homotopy  $\mathcal{L}(\pmb {\theta},\lambda) = (1 - \lambda)\mathcal{F}(\pmb {\theta}) + \lambda \mathcal{G}(\pmb {\theta})$ , more problem-specific forms can be employed.
Homotopy optimization shines in cases where the target function  $\mathcal{F}$  is non-convex and has a complicated landscape riddled with local minima. Similarly to simulated annealing [33], one starts out with a relaxed version of the more complicated problem at hand and finds a series of approximate solutions while slowly morphing the relaxed problem back into its original non-trivial form. This allows the optimization process to not get stuck in spurious sharp valleys and accurately converge to the minimum of interest.
2.3 Synchronization of dynamical systems
Here, we briefly summarize the mathematical phenomenon of synchronization, as described in the literature [51, 28, 1, 49]. In later sections, we will show that synchronization can be used to mitigate the difficulties arising in NeuralODE training. Consider two states  $\hat{\mathbf{u}}, \mathbf{u} \in \mathbb{R}^n$ , each evolving via
$$
\frac {d \hat {\mathbf {u}}}{d t} = \hat {\mathbf {U}} (t, \hat {\mathbf {u}}; \hat {\boldsymbol {\theta}}), \quad \frac {d \mathbf {u}}{d t} = \mathbf {U} (t, \mathbf {u}; \boldsymbol {\theta}) - \mathbf {K} (\mathbf {u} - \hat {\mathbf {u}}). \tag {4}
$$
Here,  $\hat{\mathbf{U}}, \mathbf{U} : \mathbb{R} \times \mathbb{R}^n \to \mathbb{R}^n$  are the dynamics of the two systems parameterized by  $\hat{\theta}, \theta$ , respectively, and  $\mathbf{K} = \text{diag}(k_1, \dots, k_n)$ , is the diagonal coupling matrix between the two systems. The two systems are said to synchronize if the error between  $\hat{\mathbf{u}}(t)$  and  $\mathbf{u}(t)$  vanishes with increasing time.
Theorem 1 (Synchronization). Assuming  $\mathbf{U} = \hat{\mathbf{U}}$ ,  $\pmb{\theta} \approx \hat{\pmb{\theta}}$ , and  $\mathbf{u}(t)$  in the vicinity of  $\hat{\mathbf{u}}(t)$ , the elements of  $\mathbf{K}$  can be chosen so that the two states synchronize: that is, for error dynamics  $\pmb{\xi}(t) = \mathbf{u}(t) - \hat{\mathbf{u}}(t)$ ,  $||\pmb{\xi}(t)||_2 = ||\mathbf{u}(t) - \hat{\mathbf{u}}(t)||_2 \to 0$  as  $t \to \infty$ .
Proof. By subtracting the two state equations, the error, up to first order terms, can be shown to follow:
$$
\frac {d \boldsymbol {\xi} (t)}{d t} = \frac {d \mathbf {u} (t)}{d t} - \frac {d \hat {\mathbf {u}} (t)}{d t} = \left[ \left. \frac {\partial \mathbf {U} (\mathbf {t} , \mathbf {u} ; \boldsymbol {\theta})}{\partial \mathbf {u}} \right| _ {\mathbf {u} = \hat {\mathbf {u}}} - \mathbf {K} \right] \boldsymbol {\xi} (t) = \mathbf {A} \boldsymbol {\xi} (t). \tag {5}
$$
Increasing the elements of  $\mathbf{K}$  causes the matrix  $\mathbf{A}$  to become negative definite—which in turn, causes the largest Lyapunov exponent of the system,
$$
\Lambda := \lim  _ {t \rightarrow \infty} \frac {1}{t} \log \frac {| | \boldsymbol {\xi} (t) | |}{| | \boldsymbol {\xi} (0) | |}. \tag {6}
$$
It is also possible to use other types of losses, such as the L1 loss used in [18, 31].
3In the literature, these methods are also referred to as "continuation methods"
to also become negative as well. Then  $||\pmb {\xi}(t)||_2 = ||\mathbf{u}(t) - \hat{\mathbf{u}} (t)||_2\rightarrow 0$  as  $t\to \infty$  , which means the dynamics of  $\mathbf{u}$  and  $\hat{\mathbf{u}}$  synchronize with increasing time regardless of the parameter mismatch.
Remark. Our assumption of  $\mathbf{U} = \hat{\mathbf{U}}$ , that the functional form of the data dynamics is the same as the equation to be fitted can also be thought to hold for NeuralODE training. This is because if the model is sufficiently expressive, universal approximation theorem will allow the model to match the data equation for some choice of model parameter.
2.3.1 Constructing homotopy with synchronization
To combine homotopy optimization with synchronization, we slightly modify the coupling term of Equation (4) by multiplying it with the homotopy parameter  $\lambda$ :
$$
\frac {d \mathbf {u}}{d t} = \mathbf {U} (t, \mathbf {u}; \boldsymbol {\theta}) - \lambda \mathbf {K} (\mathbf {u} - \hat {\mathbf {u}}). \tag {7}
$$
With this modification, homotopy optimization is introduced to the problem as follows. When  $\lambda = 1$  and the coupling matrix  $\mathbf{K}$  has sufficiently large elements, synchronization occurs and the NeualODE prediction  $\mathbf{u}(t)$  starts to converge to the data trajectory  $\hat{\mathbf{u}}(t)$  for large  $t$ . As we will confirm in Section 4, this causes the resulting loss function  $\mathcal{L}(\boldsymbol{\theta}, 1)$  to become well-behaved, serving the role of the auxiliary function  $\mathcal{G}$  in Equation (3). When  $\lambda = 0$ , the coupled Equation (4) reduces to the original model form of Equation (1). Accordingly, the corresponding loss function  $\mathcal{L}(\boldsymbol{\theta}) = \mathcal{L}(\boldsymbol{\theta}, 0)$  is the complicated loss function we need to ultimately minimize. Therefore, starting with  $\lambda = 1$  and successively decreasing its value to 0, all the while optimizing for the coupled loss function  $\mathcal{L}(\boldsymbol{\theta}, \lambda)$  allows one to leverage the well-behaved loss function landscape from synchronization while being able to properly uncover the system parameters [66, 58].
3 Related works
Synchronization in ODE parameter estimation After the discovery of synchronization in chaotic systems by Pecora & Carroll [47, 48], methods to leverage this phenomenon to identify unknown ODE parameters were first proposed by Parlitz [44], and later refined by Abarbanel et al. [2, 1]. Parameter identification in these earlier works would proceed by either designing an auxiliary dynamics for the ODE parameters [44, 40], or including a regularization term on the coupling strength in the loss function [53, 1]. The idea of applying homotopy optimization to this problem was pioneered by Vyasarayani et al. [65, 66] and further expanded upon by later works [58, 63, 52].
Homotopy methods in machine learning As homotopy methods can outperform gradient descent for complex optimization problems, it has been used in diverse fields, including Gaussian homotopy for non-convex optimization [27] and training feedforward networks [13, 24, 10]. Curriculum learning [6] is another example of homotopy optimization, with the each task in the curriculum corresponding to a particular value of the homotopy parameter as it is decreased to 0. Not to be confused with the method discussed in this paper is the homotopy perturbation technique [26] used in the dynamical systems literature, which is a method to obtain analytical solutions to nonlinear differential equations.
Improving the training of RNNs and NeuralODEs As previously mentioned, many recent approaches to improve RNN and NeuralODE training involve constraining the model expressivity. The limitations of this strategy is investigated in [57], where it is demonstrated that such "stabilized" models cannot learn oscillatory or highly complex data that does not match the built-in restrictions. A recent development with a similar spirit as ours is Mikhaeil et al. [41], where the authors developed an architecture-agnostic training method for RNNs by revisiting the classical RNN training strategy of teacher forcing [60] and refining it via the chaos theory concept of Lyapunov exponents. Other aspects of NeuralODE training, such as better gradient calculation, have been studied to improve NeuralODE performance [30, 72, 39, 31]. Such methods are fully compatible with our proposed work, and we showcase an example in Appendix G, by changing the gradient calculation scheme to the "symplectic-adjoint-method" from [39].
4 Loss landscapes in learning ODEs
4.1 Complicated loss landscapes in ODE parameter estimation
NeuralODEs suffer from the same training difficulties of ODE fitting and RNNs. Here we illustrate that the underlying cause for this difficulty is also identical by analyzing the loss landscape of NeuralODEs for increasing lengths of training data. The data were generated from the periodic Lotka-Volterra system<sup>4</sup> and the landscapes were visualized by calculating the two dominant eigenvectors of the loss Hessian, then perturbing the model parameter along those directions [69]. We also provide analogous analyses for ODE parameter estimation on the same system in Appendix B to further emphasize the similarities between the problem domains.
Figure 2: Irregular loss landscape in NeuralODE training. (Upper left) Loss landscape and (Lower left) eigenvalue spectrum of the loss Hessian for increasing lengths of train data. (Upper right) Loss landscape and (Lower right) eigenvalue spectrum of the loss Hessian with increasing coupling strength. For clarity, loss values were clipped above at 4000. All Hessian related information was calculated using the PyHessian package [69].
From Figure 2 (upper left), we find that the loss landscape is well-behaved for short data, but becomes pathological as the data lengths. In fact, the landscape for  $t \in [0,9.1]$  features a stiff cliff, which is known to cause exploding gradients in RNN training [14, 45]. Indeed, this picture is also confirmed by an alternative principal component analysis-based visualization of the loss landscape and the optimization path [35] $^{5}$  (Figure 1, right). We also observe that the overall magnitude of the loss increases with data length. This is a direct consequence of model dynamics  $\mathbf{u}(t)$  being independent of the data dynamics  $\hat{\mathbf{u}}(t)$  and therefore tending to diverge away with time.
To further confirm our findings and compensate for the fact that our visualizations are low dimensional projections of the actual loss landscape, we computed the eigenvalue spectrum of the loss function hessian for increasing lengths of training data (Figure 2, lower left). The computed spectrums show that hessian eigenvalues spread away from zero with increasing length of training data, signalling that the loss morphs from a relatively flat to a more corrugated landscape. We also observe large positive eigenvalue outlier peaks, whose values also rapidly increase with the length of training data. As such large outliers have been reported to slow down training [19], this also provides additional insights as to why NeuralODE training stagnates on long time series data.
4.2 Loss landscape of synchronized dynamics
With coupling, the loss function  $\mathcal{L}$  between the model predictions and the data now becomes a function of both the model parameters  $\theta$  and the coupling strength  $k$ . Increasing the coupling strength  $k$  effectively acts as shortening the data because increased coupling causes the two trajectories to start synchronizing earlier in time, after which the error between the two will diminish and contribute negligibly to the total loss. Therefore, synchronizing the model and the data generating equations leads to a more favorable loss landscape that gradient descent based optimizers will not struggle on.
4For more details, see Section 6
For details on the visualization method, see Section B of the supplementary of the cited reference.
Once again, we confirm this point by visualizing the loss landscape and the Hessian eigenvalue spectrum for a long time series dataset while increasing the strength of the coupling. From Figure 2 (upper right) we find that with increasing coupling strength, the steep cliff and the flat basin of the original loss surface are tamed into a well-behaved slope. This is reflected in the eigenvalue spectrum as well (Figure 2, lower right), with the all of the spectrum gathering near zero as the coupling is turned up. In fact, when the coupling strength is increased too much, we find that all the eigenvalues bunch together at zero, meaning that the loss function becomes completely flat. Qualitatively, this corresponds to the case where the coupling completely dominates the synchronized dynamics and forces the model predictions onto the data regardless of the model parameters.
5 Homotopy optimization for NeuralODE training
Having established that applying synchronization between the model and data can tame the irregular loss landscape during training, we now turn towards adapting the synchronization-homotopy approach of Section 2.3.1 to a NeuralODE setting. In doing so, there are couple details that needs addressing to arrive at a specific implementations. We discuss such points below.
5.1 Constructing the coupling term
The coupling term for synchronization  $-\lambda \mathbf{K}(\mathbf{u} - \hat{\mathbf{u}})$  in Equation (7) depends on the data dynamics  $\hat{\mathbf{u}}$ , whose values are only available at discrete timestamps  $\{t^{(i)}\}_{i=0}^{N}$ . This is at a disaccord with the ODE solvers used during training, which require evaluating the model at intermediate time points as well. Therefore, to construct a coupling term defined for arbitrary time, we used a cubic smoothing spline [21] of the data points to supply the  $\hat{\mathbf{u}}$  values at unobserved times. Smoothing was chosen over interpolation to increase the robustness of our approach to measurement noise.
Note that cubic spline is not the only solution - even defining the coupling term as a sequence of impulses is possible, only applying it where it is defined, and keeping it zero at other times. This is an interesting avenue of future research, as this scheme can be shown [53] to be mathematically equivalent to teacher forcing [60].
5.2 A little more on the coupling matrix
In general, the coupling matrix  $\mathbf{K}$  can be any  $n\times n$  matrix, provided it renders the Jacobian of the error dynamics (matrix  $\mathbf{A}$  in Equation (6)) to be negative definite along the data trajectory  $\hat{\mathbf{u}}(t)$ . However, naively following this approach leads to  $n^2$  hyperparameters just for the coupling matrix, which can quickly get out of hand for high dimensional systems. To circumvent this issue, in this work we have used a coupling matrix of the form  $\mathbf{K} = k\mathbf{I}_n$ , where  $\mathbf{I}_n$  is the  $n\times n$  identity matrix, to arrive at single scalar hyperparameter characterizing coupling strength.
5.3 Scheduling the homotopy parameter
The homotopy parameter  $\lambda$  can be decreased in various different ways, the simplest approach being constant steps. In our study, we instead used power law decrements of the form  $\{\Delta \lambda^{(j)}\}_{j = 0}^{n_{step} - 1} = \kappa^j /\sum_{j = 0}^{n_{step} - 1}\kappa^j$  where  $\Delta \lambda^{(j)}$  denotes the  $j$ -th decrement and  $n_{step}$  is number of homotopy steps. This was inspired from our observations of the synchronized loss landscape (Figure 2, right), where the overall shape change as well as the width of the eigenvalue spectrum seemed to be a superlinear function of the coupling strength.
5.4 Hyperparameter overview
Our final implementation of the homotopy training algorithm has five hyperparameters, as described below.
Coupling strength  $(k)$  This determines how trivial the auxiliary function for the homotopy optimization will be. Too small, and even the auxiliary function will have a jagged landscape; too large, and the initial auxiliary function will become flat (Figures 2 and 11, left panel,  $k = 1.0, 1.5$ ) resulting in very slow parameter updates. We find good choices of  $k$  tend to be comparable to the scale of the measurement values.