SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Homotopy-based NeuralODE training
Input: Data  $\{t^{(i)},\hat{\mathbf{u}}^{(i)}\}_{i = 0}^{N}$  , untrained model  $\mathbf{U}(\dots ;\pmb {\theta})$  Hyperparameters: Coupling strength(k), homotopy parameter decrement ratio(  $\kappa$  ), number of homotopy steps(s), epochs per homotopy step(nepoch), learning rate(η)  $\hat{\mathbf{u}} (t) = CubicSmoothingSpline(\{t^{(i)},\hat{\mathbf{u}}^{(i)}\}_{i = 0}^{N})$ $\triangleright$  Construct data interpolant  $\lambda \leftarrow 1$  Initialize homotopy parameter for  $s$  in 0...  $n_{step} - 1$  do for epoch in 0...  $n_{epoch} - 1$  do  $\{\mathbf{u}^{(i)}\}_{i = 0}^{N} = ODESolve(\dot{\mathbf{u}} = \mathbf{U}(t,\mathbf{u};\pmb {\theta}) - \lambda k\mathbf{I}_n(\mathbf{u} - \hat{\mathbf{u}} (t)))$ $\triangleright$  Solve coupled dynamics  $\{\tilde{\mathbf{u}}^{(i)}\}_{i = 0}^{N} = ODESolve(\dot{\tilde{\mathbf{u}}} = \mathbf{U}(t,\tilde{\mathbf{u}};\pmb {\theta}))$ $\triangleright$  Also solve uncoupled dynamics  $MSE(\pmb {\theta}) = \frac{1}{N + 1}\sum_i|\tilde{\mathbf{u}}^{(i)}(\pmb {\theta}) - \hat{\mathbf{u}}^{(i)}|^2$  if  $MSE\leq MSE_{best}$  then  $MSE_{best}\gets MSE$ $\triangleright$  Model performance is gauged by uncoupled dynamics error  $\pmb{\theta}^{*}\gets \pmb{\theta}$  Checkpoint best model parameters end if  $\mathcal{L}(\pmb {\theta}) = \frac{1}{N + 1}\sum_i\left|\mathbf{u}^{(i)}(\pmb {\theta}) - \hat{\mathbf{u}}^{(i)}\right|^2$  Loss function is calculated using coupled dynamics  $\pmb {\theta}\gets$  OptimizerStep(  $\eta ,\pmb {\theta},\nabla_{\pmb{\theta}}\mathcal{L}$  end for  $\lambda \leftarrow \lambda -\kappa^s /\sum_{s = 0}^{n_{steps} - 1}\kappa^s$  Decrease homotopy parameter with power law decrements end for Output: Trained model parameters  $\pmb{\theta}^{*}$ $\lambda = 0$
Homotopy parameter decrement ratio  $(\kappa)$  This determines how the homotopy parameter  $\lambda$  is decremented for each step. Values close to 1 cause  $\lambda$  to decrease in nearly equal decrements, whereas smaller values cause a large decrease of  $\lambda$  in the earlier parts of the training, followed by subtler decrements later on. We empirically find that  $\kappa$  values near 0.6 tend to work well.
Number of homotopy steps  $(s)$  This determines how many relaxed problem the optimization process will pass through to get to the final solution. Similar to scheduling the temperature in simulated annealing, fewer steps results in the model becoming stuck in a sharp local minima, and too many steps makes the optimization process unnecessarily long. We find using values in the range of 6-8 or slightly larger values for more complex systems yields satisfactory results.
Epochs per homotopy step ( $n_{\text{epoch}}$ ) This determines how long the model will train on a given homotopy parameter value  $\lambda$ . Too small, and the model lacks the time to properly converge on the loss function; too large, and the model overfits on the simpler loss landscape of  $\lambda \neq 0$ , resulting in a reduced final performance when  $\lambda = 0$ . We find for simpler monotonic or periodic systems, values of 100-150 work well; for more irregular systems, 200-300 are suitable.
Learning rate  $(\eta)$  This is as same as in conventional NeuralODE training. We found values in the range of 0.002-0.1 to be adequate for our experiments.
To benchmark our homotopy training algorithm against existing methods, we trained multiple NeuralODE models across time-series datasets of differing complexities. The performance metrics we used in our experiments were mean squared error on the training dataset and on model rollouts of  $100\%$  the training data length, which from now on we refer to as interpolation and extrapolation MSE, respectively. For each training method, the hyperparameters were determined through a grid search performed prior to the main experiments. All experiments were repeated three times with different random seeds, and we report the corresponding means and standard errors whenever applicable. For conciseness, we outline the datasets, models and the baseline methods used in the subsequent sections, and refer interested readers to Appendices E and F for extensive details on data generation, model configuration and hyperparameter selection. We provide all of the code for this paper in https://github.com/Jhko725/NeuralODEHomotopy.
6.1 Datasets and model architectures
Lotka-Volterra system The Lotka-Volterra system is a simplified model of prey and predator populations [43]. The system dynamics are periodic, meaning even models trained on short trajectories should have strong extrapolation properties, provided the training data contains a full period of oscillation. We considered two types of models for this system: a "black-box" model of Equation (1), and a "gray-box" model used in [54] given by  $\frac{dx}{dt} = \alpha x + U_1(x,y;\pmb{\theta}_1)$ ,  $\frac{dy}{dt} = -\gamma y + U_2(x,y;\pmb{\theta}_2)$ .
Double pendulum For this system, we used the real-world measurement data from Schmidt & Lipson [59]. While the double pendulum system can exhibit chaotic behavior, we used the portion of the data corresponding to the aperiodic regime. We trained two types of models on this data: a black-box model and a model with second-order structure. Our motivation for including the latter model stems from the ubiquity of second-order systems in physics and from [23], which reports these models can match the performance of Hamiltonian neural networks.
Lorenz system The Lorenz system displays highly chaotic behavior, and serves as a stress test for time series prediction tasks. For this system, we only employed a black-box model.
As the focus of this paper is to effectively train NeuralODEs without employing any specialized model design or constraining model expressivity, this was also the criteria by which we chose the baseline training method for our experiments. Two training methods were chosen as our baseline: vanilla gradient descent, which refers to the standard method of training NeuralODEs, and multiple-shooting method. As the latter also stems from the ODE parameter estimation literature, and its inner working relevant to the discussion of our results, we briefly describe the multiple-shooting algorithm before discussing our results.
Multiple shooting The multiple-shooting algorithm [62, 8, 5, 64] circumvents the difficulty of fitting models on long sequences by dividing the training data into shorter segments and solving multiple initial value problems. To keep the predictions for different segments in sync, the loss function is augmented with a continuity enforcing term. Unlike synchronization and homotopy, which we employ in NeuralODE training for the first time, works applying multiple shooting to NeuralODEs are scant but do exist [54, 38, 61]. As the three previous references differ in their implementations, we chose the scheme of [54] for its simplicity and implemented a PyTorch adaptation to use for our experiments. Further details about our multiple-shooting implementation can be found in Appendix D.
7.1 Training NeuralODEs on time series data of increasing lengths
In the previous sections, we found that longer training data leads to a more irregular loss landscape. To confirm that this does translate to deteriorated optimization, we trained NeuralODEs on Lotka-Volterra system trajectories of different lengths. Inspecting the results (Figure 3, first panel) we find while all training methods show low interpolation MSE for shortest data, this is due to overfitting, as evidenced by the diverging extrapolation  $\mathrm{MSE}^7$ . As the length of the data grows and encompasses a full oscillation period, we find that the model performances from homotopy and multiple-shooting stabilize for both interpolation and extrapolation. This is clearly not the case for the vanilla gradient descent, whose model performances degrade exponentially with as data length increases.
Model capacity is not the problem The reason NeuralODEs struggle on long sequences can easily be attributed to insufficient model capacity. We demonstrate this is not the case by repeating the previous experiment for the mid-length data while decreasing the model size used (Figure 3, second panel). In the case of vanilla gradient descent, the model size is directly correlated to its performance,
<sup>6</sup>While not explicitly mentioned in the paper, DiffEqFlux.j1 library has an example tutorial using this technique to train NeuralODEs.
Indeed, plots of the model prediction trajectory in Appendix G.3 also confirm this fact.
Figure 3: Results for the Lotka-Volterra system. (First) Interpolation and extrapolation errors for increasing data length. (Second) Errors for the mid-length data with decreasing model capacity. (Third) Errors for fixed time interval but with increasing data sparsity. (Fourth) Errors for mid-length data with increasing noise.
lending the claim some support. However, we find that both homotopy and multiple shooting methods can effectively train much smaller models to match the accuracy of a larger model trained with vanilla gradient descent. This signifies that the cause of the problem is not the capacity itself, but rather the inability of training algorithms to fully capitalize on the model capacity.
Robustness against data degradation We performed an additional set of experiments to evaluate the performances of different training methods against increasing data sparsity and noise. This is particularly of interest for our homotopy training as we employ smoothing cubic splines (Section 5.1) whose reliability can be adversely affected by the data quality. From the results (Figure 3, third, fourth panels) we find this is not the case and that our algorithm performs well even for degraded data. One unexpected result is the drastic underperformance of the multiple shooting method with increased sampling period, which is a side effect from the fact we held the time interval fixed for this experiment. Due to the space constraint, we continue this discussion in Appendix G.3.
7.2 Performance benchmarks of the training algorithms
Having confirmed that both the homotopy and multiple-shooting methods successfully mitigate the NeuralODE training problem, we now present a comprehensive benchmark in Figure 4.
Figure 4: Performance benchmarks. (Left) Interpolation and (Center) extrapolation MSEs for each training method. (Right) Epochs where the minimum interpolation MSE was achieved. For all plots, the bar and the error bars denote the mean and standard errors across three runs.
From the results, it is clear both our homotopy method and multiple shooting can both boost the interpolation performance of NeuralODEs, with our method being slightly more performant. Inspecting the extrapolation performance, however, we do find a clear performance gap between our method and multiple shooting, with the latter even performing worse than vanilla gradient descent for the double pendulum dataset (Figure 4, center, Figure 5, left). This underperformance of multiple shooting for extrapolation tasks can be attributed to the nature of the algorithm: during training time, multiple shooting integrates NeuralODEs on shorter, subdivided time intervals, whereas during inference time the NeuralODE needs to be integrated for the entire data time points and then some more for extrapolation. This disparity between training and inference render models trained by
multiple shooting to be less likely to compensate for error accumulation in long term predictions, resulting in poor extrapolation performance.
Another stark difference between our homotopy training and the other methods is in the number of epochs required to arrive at minimum interpolation MSE (Figure 4, right panel). While the exact amount of speedup is variable and seems to depend on the dataset difficulty, we find a  $90\% - 40\%$  reduction compared to vanilla training, highlighting the effectiveness of our approach. One unexpected observation from our experiments was that the epoch at which minimum interpolation MSE epoch was achieved did not always correspond to the homotopy parameter  $\lambda$  being zero (Appendix G.2). While seemingly counterintuitive, this is in line with the literature, where nonzero homotopy parameter has resulted in better generalization performances [36, 24].
What makes homotopy training effective: inspecting training dynamics To obtain a complementary insight to the NeuralODE training dynamics and relate the effectiveness of our method with the properties of the loss landscape, we calculated the trace of the loss hessian during training (Figure 5, right panel). Small, positive trace values indicate more favorable training dynamics [69], and we find that our method results in the smallest trace, followed by multiple shooting then vanilla gradient descent. The fact that the hessian trace maintains a low value throughout the entire homotopy training, even as the loss surface is morphed back to its unsynchronized, irregular state, also confirms that our homotopy schedule works well and managed to gradually guide the optimizer into a well performing loss minimum.
Figure 5: Second-order NeuralODE training results for the double pendulum dataset. (Left) Predicted trajectories. Lines and bands correspond to mean and standard errors for three training runs and the dashed line indicate the extrapolation start time. (Right) Estimation of the hessian trace during training. The homotopy curve stops early due to the difference in the number of maximum training curves used.
8 Limitations and discussion
In this paper, we adapted the concepts of synchronization and homotopy optimization for the first time in the NeuralODE literature to stabilize NeuralODE training on long time-series data. Through loss landscape analysis, we established that the cause of the training problem is identical between differential equations, RNNs and NeuralODEs and that synchronization can effectively tame the irregular landscape. We demonstrated that our homotopy method can successfully train NeuralODE models with strong interpolation and extrapolation accuracies.
Note that our work serves as an initial confirmation that the methods developed in the ODE parameter estimation literature can be adapted to be competitive in the NeuralODE setting. As such, there are multiple opportunities for future research, including theoretical studies on the convergence properties of our algorithm and further testing out ideas in the synchronization literature, such as alternative parametrization for the coupling matrix [28], or incorporating the homotopy parameter into the loss function as a regularization term [1, 58]. Extending our algorithm to high-dimensional systems is another important direction, which we believe will be possible through the use of latent ODE models [70]. Nevertheless, our work hold merit in that it lays the groundwork upon which ideas from ODE parameter estimation, control theory, and RNN/NeuralODE training can be assimilated, which we believe will facilitate further research connecting these domains.
J. Ko and H. Koh contributed equally to the paper.
This work was supported by grants from the National Research Foundation of Korea (No. 2016R1A3B1908660) to W. Jhe.
[1] H. D. I. Abarbanel, D. R. Creveling, R. Farsian, and M. Kostuk. Dynamical State and Parameter Estimation. SIAM Journal on Applied Dynamical Systems, 8(4):1341-1381, 2009.
[2] H. D. I. Abarbanel, D. R. Creveling, and J. M. Jeanne. Estimation of parameters in nonlinear systems using balanced synchronization. Physical Review E, 77(1):016208, 2008.
[3] E. L. Allgower and K. Georg. Numerical Continuation Methods: An Introduction. Springer Berlin Heidelberg, Berlin, Heidelberg, 1990.
[4] M. Arjovsky, A. Shah, and Y. Bengio. Unitary Evolution Recurrent Neural Networks. In Proceedings of The 33rd International Conference on Machine Learning, pages 1120-1128. PMLR, 2016.
[5] E. Baake, M. Baake, H. G. Bock, and K. M. Briggs. Fitting ordinary differential equations to chaotic data. Physical Review A, 45(8):5524-5529, 1992.
[6] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 41-48, New York, NY, USA, 2009. Association for Computing Machinery.
[7] L. Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
[8] H. G. Bock. Recent Advances in Parameter Identification Techniques for O.D.E. In P. Deuflhard and E. Hairer, editors, Numerical Treatment of Inverse Problems in Differential and Integral Equations: Proceedings of an International Workshop, Heidelberg, Fed. Rep. of Germany, August 30 — September 3, 1982, Progress in Scientific Computing, pages 95–121. Birkhäuser, Boston, MA, 1983.
[9] B. Chang, M. Chen, E. Haber, and E. H. Chi. AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks. In International Conference on Learning Representations, volume 7, 2019.
[10] Q. Chen and W. Hao. A homotopy training algorithm for fully connected neural networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 475(2231):20190662, Nov. 2019.
[11] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[12] K. M. Choromanski, J. Q. Davis, V. Likhosherstov, X. Song, J.-J. Slotine, J. Varley, H. Lee, A. Weller, and V. Sindhwani. Ode to an ODE. In Advances in Neural Information Processing Systems, volume 33, pages 3338-3350. Curran Associates, Inc., 2020.
[13] J. Chow, L. Udpa, and S. Udpa. Homotopy continuation methods for neural networks. In 1991., IEEE International Symposium on Circuits and Systems, volume 5, pages 2483-2486, 1991.
[14] K. Doya. Bifurcations of recurrent neural networks in gradient descent learning. IEEE Transactions on neural networks, 1(75):218-229, 1993.
[15] D. M. Dunlavy and D. P. O'Leary. Homotopy optimization methods for global optimization. Technical Report SAND2005-7495, Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA (United States), 2005.
[16] W. Falcon and The PyTorch Lightning team. PyTorch Lightning, Mar. 2019.
[71] Y. Yin, V. Le Guen, J. Dona, E. de Bézenac, I. Ayed, N. Thome, and P. Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124012, 2021.
[72] J. Zhuang, N. Dvornek, X. Li, S. Tatikonda, X. Papademetris, and J. Duncan. Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE. In Proceedings of the 37th International Conference on Machine Learning, pages 11639–11649. PMLR, 2020.
In this section, we present some of the figures that were excluded from the main text for brevity.
Figure 6 below highlights the efficiency of our training approach with respect to the number of training epochs by plotting the trajectory predictions of the second-order models during training.
Figure 6: Second-order model predictions for the double pendulum data during training. Blue and orange lines indicate data and model predictions respectively. (Top) Results from our homotopy approach. (Middle) Results from multiple shooting. (Bottom) Results from vanilla gradient descent.
Figures 7 to 9 show model prediction trajectories corresponding to our benchmark results of Figure 4. One intriguing point is that even though both the gray-box model (Lotka-Volterra system, Figure 7) and the second-order model (double pendulum, Figure 8) contain more prior information about their respective systems, this does not necessarily translate to better predictive capabilities if an effective training method is not used. This suggests that introducing physics into the learning problem can obfuscate optimization, something that has been reported for physics-informed neural networks [32]. It also highlights the effectiveness of our homotopy training algorithm, as our method can properly train such difficult-to-optimize models.
Figure 7: Trajectory predictions for the Lotka-Volterra system from models trained with different methods. Dashed vertical line indicates the start of extrapolation.
Figure 8: Trajectory predictions for the double pendulum from models trained with different methods. Dashed vertical line indicates the start of extrapolation.
Figure 9: Trajectory predictions for the Lorenz system from models trained with different methods. Dashed vertical line indicates the start of extrapolation.
B Loss landscape and synchronization in ODE parameter estimation
Here, we illustrate the similarity between the ODE parameter estimation problem and the NeuralODE training problem by repeating analyses analogous to those of Section 4 in the setting of estimating the unknown coefficients of a given ordinary differential equation.
In the left and middle panels of Figure 10, we show the trajectories of the Lotka-Volterra and the Lorenz systems where a single parameter was perturbed. We can easily observe that the original trajectory and the perturbed trajectories evolve independently in time. Furthermore, while trajectories from the simpler Lotka-Volterra system retain the same periodic behavior and differ only in their amplitudes and phases, trajectories from the Lorenz system display much different characteristics with increasing time. This translates over to the shape of the loss landscape (Figure 10, right) with the loss for the Lotka-Volterra system only becoming steeper with increasing data length, whereas the loss for the Lorenz system displays more and more local minima.
Once synchronization is introduced to the perturbed trajectory, its dynamics tends to converge with the reference trajectory as time increases, with the rate of convergence increasing for larger values of the coupling strength  $k$  (Figure 11, left and middle panels). In terms of the loss landscape (right panel), larger coupling does result in a smoother landscape, with excessive amounts of it resulting in a flat landscape that is also detrimental to effective learning. These results are a direct parallel to our observations in Figure 2 and provide additional justifications as to why techniques developed in the field of ODE parameter estimation also work so well on NeuralODE training.
Figure 10: Dynamics of single parameter perturbed systems. The systems and the parameters used are described in Section 6. (Left) Solutions for the periodic Lotka-Volterra equations with perturbed  $\alpha$  parameter. (Middle) Solutions for the chaotic Lorenz system with perturbed  $\beta$  parameter. (Right) MSE loss function landscape for the Lotka-Volterra equations (right, upper) and Lorenz system (right, lower) for different lengths of time series data for the loss calculation.
Figure 11: Dynamics of coupled systems with varying coupling strengths. The systems and the parameters used are identical to those of Figure 10. (Left) Results for the periodic Lotka-Volterra equations. (Middle) Results for the chaotic Lorenz system. (Right) Loss function landscape of the coupled systems with different coupling strengths.
C Further description of the homotopy optimization procedure
To implement homotopy optimization, one usually selects the number of discrete steps for optimization,  $s$ , as well as a series of positive decrement values for the homotopy parameter  $\{\Delta \lambda^{(k)}\}_{0}^{s - 1}$  that sum to 1. Afterwards, optimization starts with an initial  $\lambda$  value of  $\lambda^{(0)} = 1$ , which gives  $\mathcal{H}^{(0)}(\pmb {\theta}) = \mathcal{G}(\pmb {\theta})$ . At each step, the objective function at the current iteration is minimized with respect to  $\pmb{\theta}$ , using the output from the previous step  $\pmb{\theta}^{*(k - 1)}$  is as the initial guess:
$$
\mathcal {H} ^ {(k)} (\boldsymbol {\theta}) = \mathcal {H} (\boldsymbol {\theta}, \lambda = \lambda^ {(k)}) \rightarrow \boldsymbol {\theta} ^ {* (k)} = \underset {\boldsymbol {\theta}} {\operatorname {a r g m i n}} \mathcal {H} ^ {(k)} (\boldsymbol {\theta}) \tag {8}
$$