Title: Homotopy-based training of NeuralODEs for accurate dynamics discovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural ordinary differential equations

synchronization

homotopy optimization

loss landscape

dynamical systems
Abstract: Neural Ordinary Differential Equations (NeuralODEs) present an attractive way to extract dynamical laws from time series data, as they bridge neural networks with the differential equation-based modeling paradigm of the physical sciences. However, these models often display long training times and suboptimal results, especially for longer duration data. While a common strategy in the literature imposes strong constraints to the NeuralODE architecture to inherently promote stable model dynamics, such methods are ill-suited for dynamics discovery as the unknown governing equation is not guaranteed to satisfy the assumed constraints. In this paper, we develop a new training method for NeuralODEs, based on synchronization and homotopy optimization, that does not require changes to the model architecture. We show that synchronizing the model dynamics and the training data tames the originally irregular loss landscape, which homotopy optimization can then leverage to enhance training. Through benchmark experiments, we demonstrate our method achieves competitive or better training loss while often requiring less than half the number of training epochs compared to other model-agnostic techniques. Furthermore, models trained with our method display better extrapolation capabilities, highlighting the effectiveness of our method.

Homotopy-based training of NeuralODEs for accurate dynamics discovery

Abstract

Neural Ordinary Differential Equations (NeuralODEs) present an attractive way to extract dynamical laws from time series data, as they bridge neural networks with the differential equation-based modeling paradigm of the physical sciences. However, these models often display long training times and suboptimal results, especially for longer duration data. While a common strategy in the literature imposes strong constraints to the NeuralODE architecture to inherently promote stable model dynamics, such methods are ill-suited for dynamics discovery as the unknown governing equation is not guaranteed to satisfy the assumed constraints. In this paper, we develop a new training method for NeuralODEs, based on synchronization and homotopy optimization, that does not require changes to the model architecture. We show that synchronizing the model dynamics and the training data tames the originally irregular loss landscape, which homotopy optimization can then leverage to enhance training. Through benchmark experiments, we demonstrate our method achieves competitive or better training loss while often requiring less than half the number of training epochs compared to other model-agnostic techniques. Furthermore, models trained with our method display better extrapolation capabilities, highlighting the effectiveness of our method.

1 Introduction

Predicting the evolution of a time varying system and discovering mathematical models that govern it is paramount to both deeper scientific understanding and potential engineering applications. The centuries-old paradigm to tackle this problem was to either ingeniously deduce empirical rules from experimental data, or mathematically derive physical laws from first principles. However, the complexities of systems of interest have grown so much that these traditional approaches are now often insufficient. This has led to a growing interest in using machine learning methods to infer dynamical laws from data.

One line of research stems from the dynamical systems literature, where the problem of interest was to determine the unknown coefficients of a given differential equation from data. While the task seems rather simple, it was found that naive optimization cannot properly recover the desired coefficients

due to irregularities in the optimization landscape, which worsens with increasing nonlinearity and data length [64, 50, 56, 1].

A parallel development occurred in the study of recurrent neural networks (RNNs), which are neural network analogues of discrete maps. It was quickly found that training these models can be quite unstable and results in exploding or vanishing gradients. While earlier works in the field resulted in techniques such as teacher forcing or gradient clipping that are agnostic to the RNN architecture [14, 45], more recent approaches have been directed towards circumventing the problem entirely by incorporating specific mathematical forms into the model architecture to constrain the eigenvalues of the model Jacobian and stabilize RNN dynamics [4, 29, 9, 34, 42]. While these latter methods can be effective, for the purpose of discovering dynamical equations underlying the data, these approaches are inadequate as one cannot ascertain the hidden equations conform to the predetermined rigid mathematical structures - especially so if the dynamics to be predicted is incompatible with the constrained model architecture [57, 41].

In this work, we focus on Neural Ordinary Differential Equations (NeuralODEs) [11]. These models are powerful tools in modeling natural phenomena, bridging the expressibility and flexibility of neural networks with the de facto mathematical language of the physical sciences. This has led to researchers amalgamating NeuralODEs with partial information about the governing equation to produce 'grey-box" dynamics models [55, 71], and endowing NeuralODEs with mathematical structures that the system must satisfy [22, 18]. Despite their conceptual elegance, training NeuralODEs tend to result in long training times and sub-optimal results, a problem that is further exacerbated as the length of the training data grows [20, 17]. Similar to the case of RNNs, methods have been proposed to tackle the problem involve placing either strong [12, 25], or semi-strong constraints [17, 30] to the functional form the NeuralODE can take something the underlying governing equation does not guarantee satisfying.

Contributions We introduce a method to accurately train NeuralODEs on long time series data by leveraging tools from the chaos and mathematical optimization literature: synchronization and homotopy optimization [1, 15]. Through loss landscape analyses, we show that longer training data lead to more irregular loss landscapes, which in turn lead to deteriorating performances of the trained models. We show that synchronizing NeuralODE dynamics with the data can smooth the loss landscape, on which homotopy optimization can be applied to enhance training. Through performance benchmarks, we show that not only does our model far outperform conventional gradient descent training, it also surpasses multiple shooting in terms of convergence speed and extrapolation accuracy.

2 Preliminaries

2.1 Neural ordinary differential equations

A NeuralODE [11] is a model of the form,

where u 0 ∈ R n is the initial condition or input given to the model, and U ( ... ; θ ) : R × R n → R n is a neural network with parameters θ ∈ R m that governs the dynamics of the model state u ( t ) ∈ R n . While this model can be used as a input-output mapping 1 , we are interested in the problem of training

NeuralODEs on time series data to learn the underlying governing equation and forecast future dynamics.

Given an monotonically increasing sequence of time points and the corresponding measurements { t ( i ) , ˆ u ( i ) } N i =0 , NeuralODE training starts with using an ordinary differential equation (ODE) solver to numerically integrate Equation (1) to obtain the model state u at given time points:

Afterwards, gradient descent is performed on the loss function L ( θ ) = 1 N +1 ∑ i ∣ ∣ u ( i ) ( θ ) -ˆ u ( i ) ∣ ∣ 2 2 to arrive at the optimal parameters θ ∗ .

2.2 Homotopy optimization method

In topology, homotopy refers to the continuous deformation of one function into another. Motivated by this concept, to find the minimizer θ ∗ ∈ R m of a complicated function F ( θ ) , the homotopy optimization method 3 [67, 68, 3, 15] introduces an alternative objective function

where G ( θ ) is an auxillary function whose minimum is easily found, and L ( θ , λ ) : R m × R → R smoothly interpolates between G and F as the homotopy parameter λ varies from 1 to 0. While a commonly used construction is the convex homotopy L ( θ , λ ) = (1 -λ ) F ( θ ) + λ G ( θ ) , more problem-specific forms can be employed.

Homotopy optimization shines in cases where the target function F is non-convex and has a complicated landscape riddled with local minima. Similarly to simulated annealing [33], one starts out with a relaxed version of the more complicated problem at hand and finds a series of approximate solutions while slowly morphing the relaxed problem back into its original non-trivial form. This allows the optimization process to not get stuck in spurious sharp valleys and accurately converge to the minimum of interest.

2.3 Synchronization of dynamical systems

Here, we briefly summarize the mathematical phenomenon of synchronization, as described in the literature [51, 28, 1, 49]. In later sections, we will show that synchronization can be used mitigate the difficulties arising in NeuralODE training. Consider two states ˆ u , u ∈ R n , each evolving via

Here, ˆ U , U : R × R n → R n are the dynamics of the two systems parameterized by ˆ θ , θ , respectively, and K = diag ( k 1 , ..., k n ) , is the diagonal coupling matrix between the two systems. The two systems are said to synchronize if the error between ˆ u ( t ) and u ( t ) vanishes with increasing time.

Theorem 1 (Synchronization) . Assuming U = ˆ U , θ ≈ ˆ θ , and u ( t ) in the vicinity of ˆ u ( t ) , the elements of K can be chosen so that the two states synchronize: that is, for error dynamics ξ ( t ) = u ( t ) -ˆ u ( t ) , || ξ ( t ) || 2 = || u ( t ) -ˆ u ( t ) || 2 → 0 as t →∞ .

Proof. By subtracting the two state equations, the error, up to first order terms, can be shown to follow:

Increasing the elements of K causes the matrix A to become negative definite -which in turn, causes the largest Lyapunov exponent of the system,

to also become negative as well. Then || ξ ( t ) || 2 = || u ( t ) -ˆ u ( t ) || 2 → 0 as t →∞ , which means the dynamics of u and ˆ u synchronize with increasing time regardless of the parameter mismatch.

Remark. Our assumption of U = ˆ U , that the functional form of the data dynamics is the same as the equation to be fitted can also be thought to hold for NeuralODE training. This is because if the model is sufficiently expressive, universal approximation theorem will allow the model to match the data equation for some choice of model parameter.

2.3.1 Constructing homotopy with synchronization

To combine homotopy optimization with synchronization, we slightly modify the coupling term of Equation (4) by multiplying it with the homotopy parameter λ :

With this modification, homotopy optimization is introduced to the problem as follows. When λ = 1 and the coupling matrix K has sufficiently large elements, synchronization occurs and the NeualODE prediction u ( t ) starts to converge to the data trajectory ˆ u ( t ) for large t . As we will confirm in Section 4, this causes the resulting loss function L ( θ , 1) to become well-behaved, serving the role of the auxillary function G in Equation (3). When λ = 0 , the coupled Equation (4) reduces to the original model form of Equation (1). Accordingly, the corresponding loss function L ( θ ) = L ( θ , 0) is the complicated loss function we need to ultimately minimize. Therefore, starting with λ = 1 and successively decreasing its value to 0, all the while optimizing for the coupled loss function L ( θ , λ ) allows one to leverage the well-behaved loss function landscape from synchronization while being able to properly uncover the system parameters [66, 58].

3 Related works

Synchronization in ODE parameter estimation After the discovery of synchronization in chaotic systems by Pecora & Carroll [47, 48], methods to leverage this phenomenon to identify unknown ODE parameters were first proposed by Parlitz [44], and later refined by Abarbanel et al. [2, 1]. Parameter identification in these earlier works would proceed by either designing an auxillary dynamics for the ODE parameters [44, 40], or including a regularization term on the coupling strength in the loss function [53, 1]. The idea of applying homotopy optimization to this problem was pioneered by Vyasarayani et al. [65, 66] and further expanded upon by later works [58, 63, 52].

Homotopy methods in machine learning As homotopy methods can outperform gradient descent for complex optimization problems, it has been used in diverse fields, including Gaussian homotopy for non-convex optimization [27] and training feedforward networks [13, 24, 10]. Curriculum learning [6] is another example of homotopy optimization, with the each task in the curriculum corresponding to a particular value of the homotopy parameter as it is decreased to 0. Not to be confused with the method discussed in this paper is the homotopy perturbation technique [26] used in the dynamical systems literature, which is a method to obtain analytical solutions to nonlinear differential equations.

Improving the training of RNNs and NeuralODEs As previously mentioned, many recent approaches to improve RNN and NeuralODE training involve constraining the model expressivity. The limitations of this strategy is investigated in [57], where it is demonstrated that such "stabilized" models cannot learn oscillatory or highly complex data that does not match the built-in restrictions. A recent development with a similar spirit as ours is Mikhaeil et al. [41], where the authors developed an architecture-agnostic training method for RNNs by revisiting the classical RNN training strategy of teacher forcing [60] and refining it via the chaos theory concept of Lyapunov exponents. Other aspects of NeuralODE training, such as better gradient calculation, have been studied to improve NeuralODE performance [30, 72, 39, 31]. Such methods are fully compatible with our proposed work, and we showcase an example in Appendix G, by changing the gradient calculation scheme to the 'symplectic-adjoint-method" from [39].

4 Loss landscapes in learning ODEs

4.1 Complicated loss landscapes in ODE parameter estimation

NeuralODEs suffer from the same training difficulties of ODE fitting and RNNs. Here we illustrate that the underlying cause for this difficulty is also identical by analyzing the loss landscape of NeuralODEs for increasing lengths of training data. The data were generated from the periodic LotkaVolterra system 4 and the landscapes were visualized by calculating the two dominant eigenvectors of the loss Hessian, then perturbing the model parameter along those directions [69]. We also provide analogous analyses for ODE parameter estimation on the same system in Appendix B to further emphasize the similarities between the problem domains.

1

2

20

40

t

[0, 3.1]

1

2

50

100

150

t

[0, 5.1]

1

2

1000

2000

t

[0, 7.1]

1

2

1000

2000

3000

4000

t

[0, 9.1]

1

2

1000

2000

3000

4000

k

=0.0

1

2

500

1000

k

=0.2

1

2

50

100

k

=0.4

1

2

5.0

7.5

10.0

k

=0.8

10 2

10 1

10 0

0

10 0

10 1

10 2

10 3

10 4

Data length

10 2

10 1

10 0

0

10 0

10 1

10 2

10 3

10 4

Coupling strength

7

6

5

4

3

2

1

0

log(density)

7

6

5

4

3

2

1

0

1

log(density)

7

6

5

4

3

2

1

0

1

log(density)

7

6

5

4

3

2

1

0

log(density)

Hessian eigenvalue

Hessian eigenvalue

From Figure 2 (upper left), we find that the loss landscape is well-behaved for short data, but becomes pathological as the data lengthens. In fact, the landscape for t ∈ [0 , 9 . 1] features a stiff cliff, which is known to cause exploding gradients in RNN training [14, 45]. Indeed, this picture is also confirmed by an alternative principal component analysis-based visualization of the loss landscape and the optimization path [35] 5 (Figure 1, right). We also observe that the overall magnitude of the loss increases with data length. This is a direct consequence of model dynamics u ( t ) being independent of the data dynamics ˆ u ( t ) and therefore tending to diverge away with time.

To further confirm our findings and compensate for the fact that our visualizations are low dimensional projections of the actual loss landscape, we computed the eigenvalue spectrum of the loss function hessian for increasing lengths of training data (Figure 2, lower left). The computed spectrums show that hessian eigenvalues spread away from zero with increasing length of training data, signalling that the loss morphs from a relatively flat to a more corrugated landscape. We also observe large positive eigenvalue outlier peaks, whose values also rapidly increase with the length of training data. As such large outliers have been report to slow down training [19], this also provides additional insights as to why NeuralODE training stagnates on long time series data.

4.2 Loss landscape of synchronized dynamics

With coupling, the loss function L between the model predictions and the data now becomes a function of both the model parameters θ and the coupling strength k . Increasing the coupling strength k effectively acts as shortening the data because increased coupling causes the two trajectories to start synchronizing earlier in time, after which the error between the two will diminish and contribute negligibly to the total loss. Therefore, synchronizing the model and the data generating equations leads to a more favorable loss landscape that gradient descent based optimizers will not struggle on.

Once again, we confirm this point by visualizing the loss landscape and the Hessian eigenvalue spectrum for a long time series dataset while increasing the strength of the coupling. From Figure 2 (upper right) we find that with increasing coupling strength, the steep cliff and the flat basin of the original loss surface are tamed into a well-behaved slope. This is reflected in the eigenvalue spectrum as well (Figure 2, lower right), with the all of the spectrum gathering near zero as the coupling is turned up. In fact, when the coupling strength is increased too much, we find that all the eigenvalues bunch together at zero, meaning that the loss function becomes completely flat. Qualitatively, this corresponds to the case where the coupling completely dominates the synchronized dynamics and forces the model predictions onto the data regardless of the model parameters.

5 Homotopy optimization for NeuralODE training

Having established that applying synchronization between the model and data can tame the irregular loss landscape during training, we now turn towards adapting the synchronization-homotopy approach of Section 2.3.1 to a NeuralODE setting. In doing so, there are couple details that needs addressing to arrive at a specific implementations. We discuss such points below.

5.1 Constructing the coupling term

The coupling term for synchronization -λ K ( u -ˆ u ) in Equation (7) depends on the data dynamics ˆ u , whose values are only available at discrete timestamps { t ( i ) } N i =0 . This is at a disaccord with the ODE solvers used during training, which require evaluating the model at intermediate time points as well. Therefore, to construct a coupling term defined for arbitrary time, we used a cubic smoothing spline [21] of the data points to supply the ˆ u values at unobserved times. Smoothing was chosen over interpolation to increase the robustness of our approach to measurement noise.

Note that cubic spline is not the only solution - even defining the coupling term as a sequence of impulses is possible, only applying it where it is defined, and keeping it zero at other times. This is an interesting avenue of future research, as this scheme can be shown [53] to be mathematically equivalent to teacher forcing [60].

5.2 A little more on the coupling matrix

In general, the coupling matrix K can be any n × n matrix, provided it renders the Jacobian of the error dynamics (matrix A in Equation (6)) to be negative definite along the data trajectory ˆ u ( t ) . However, naively following this approach leads to n 2 hyperparameters just for the coupling matrix, which can quickly get out of hand for high dimensional systems. To circumvent this issue, in this work we have used a coupling matrix of the form K = k I n , where I n is the n × n identity matrix, to arrive at single scalar hyperparameter characterizing coupling strength.

5.3 Scheduling the homotopy parameter

The homotopy parameter λ can be decreased in various different ways, the simplest approach being constant steps. In our study, we instead used power law decrements of the form { ∆ λ ( j ) } n step -1 j =0 = κ j / ∑ n step -1 j =0 κ j where ∆ λ ( j ) denotes the j -th decrement and n step is number of homotopy steps. This was inspired from our observations of the synchronized loss landscape (Figure 2, right), where the overall shape change as well as the width of the eigenvalue spectrum seemed be a superlinear function of the coupling strength.

5.4 Hyperparameter overview

Our final implementation of the homotopy training algorithm has five hyperparameters, as described below.

Coupling strength ( k ) This determines how trivial the auxillary function for the homotopy optimization will be. Too small, and even the auxillary function will have a jagged landscape; too large, and the initial auxillary function will become flat (Figures 2 and 11, left panel, k = 1 . 0 , 1 . 5 ) resulting in very slow parameter updates. We find good choices of k tend to be comparable to the scale of the measurement values.

Homotopy parameter decrement ratio ( κ ) This determines how the homotopy parameter λ is decremented for each step. Values close to 1 cause λ to decrease in nearly equal decrements, whereas smaller values cause a large decrease of λ in the earlier parts of the training, followed by subtler decrements later on. We empirically find that κ values near 0.6 tend to work well.

Number of homotopy steps ( s ) This determines how many relaxed problem the optimization process will pass through to get to the final solution. Similar to scheduling the temperature in simulated annealing, fewer steps results in the model becoming stuck in a sharp local minima, and too many steps makes the optimization process unnecessarily long. We find using values in the range of 6-8 or slightly larger values for more complex systems yields satisfactory results.

̸

Epochs per homotopy step ( n epoch ) This determines how long the model will train on a given homotopy parameter value λ . Too small, and the model lacks the time to properly converge on the loss function; too large, and the model overfits on the simpler loss landscape of λ = 0 , resulting in a reduced final performance when λ = 0 . We find for simpler monotonic or periodic systems, values of 100-150 work well; for more irregular systems, 200-300 are suitable.

Learning rate ( η ) This is as same as in conventional NeuralODE training. We found values in the range of 0.002-0.1 to be adequate for our experiments.

6 Experiments

To benchmark our homotopy training algorithm against existing methods, we trained multiple NeuralODE models across time-series datasets of differing complexities. The performance metrics we used in our experiments were mean squared error on the training dataset and on model rollouts of 100% the training data length, which from now on we refer to as interpolation and extrapolation MSE, respectively. For each training method, the hyperparameters were determined through a grid search performed prior to the main experiments. All experiments were repeated three times with different random seeds, and we report the corresponding means and standard errors whenever applicable. For conciseness, we outline the datasets, models and the baseline methods used in the subsequent sections, and refer interested readers to Appendices E and F for extensive details on data generation, model configuration and hyperparameter selection. We provide all of the code for this paper in https://github.com/Jhko725/NeuralODEHomotopy .

6.1 Datasets and model architectures

Lotka-Volterra system The Lotka-Volterra system is a simplified model of prey and predator populations [43]. The system dynamics are periodic, meaning even models trained on short trajectories should have strong extrapolation properties, provided the training data contains a full period of oscillation. We considered two types of models for this system: a "black-box" model of Equation (1), and a "gray-box" model used in [54] given by dx dt = αx + U 1 ( x, y ; θ 1 ) , dy dt = -γy + U 2 ( x, y ; θ 2 ) .

Double pendulum For this system, we used the real-world measurement data from Schmidt & Lipson [59]. While the double pendulum system can exhibit chaotic behavior, we used the portion of the data corresponding to the aperiodic regime. We trained two types of models on this data: a black-box model and a model with second-order structure. Our motivation for including the latter model stems from the ubiquity of second-order systems in physics and from [23], which reports these models can match the performance of Hamiltonian neural networks.

Lorenz system The Lorenz system displays highly chaotic behavior, and serves as a stress test for time series prediction tasks. For this system, we only employed a black-box model.

6.2 Baselines

As the focus of this paper is to effectively train NeuralODEs without employing any specialized model design or constraining model expressivity, this was also the criteria by which we chose the baseline training method for our experiments. Two training methods were chosen as our baseline: vanilla gradient descent, which refers to the standard method of training NeuralODEs, and multiple-shooting method. As the latter also stems from the ODE parameter estimation literature, and its inner working relevant to the discussion of our results, we briefly describe the multiple-shooting algorithm before discussing our results.

Multiple shooting The muliple-shooting algorithm [62, 8, 5, 64] circumvents the difficulty of fitting models on long sequences by dividing the training data into shorter segments and solving multiple initial value problems. To keep the predictions for different segments in sync, the loss function is augmented with a continuity enforcing term. Unlike synchronization and homotopy, which we employ in NeuralODE training for the first time, works applying multiple shooting to NeuralODEs are scant but do exist [54, 38, 61] 6 . As the three previous references differ in their implementations, we chose the scheme of [54] for its simplicity and implemented a PyTorch adaptation to use for our experiments. Further details about our multiple-shooting implementation can be found in Appendix D.

7 Results

7.1 Training NeuralODEs on time series data of increasing lengths

In the previous sections, we found that longer training data leads to a more irregular loss landscape. To confirm that this does translate to deteriorated optimization, we trained NeuralODEs on LotkaVolterra system trajectories of different lengths. Inspecting the results (Figure 3, first panel) we find while all training methods show low interpolation MSE for shortest data, this is due to overfitting, as evidenced by the diverging extrapolation MSE 7 . As the length of the data grows and encompasses a full oscillation period, we find that the model performances from homotopy and multiple-shooting stabilize for both interpolation and extrapolation. This is clearly not the case for the vanilla gradient descent, whose model performances degrade exponentially with as data length increases.

Model capacity is not the problem The reason NeuralODEs struggle on long sequences can easily be attributed to insufficient model capacity. We demonstrate this is not the case by repeating the previous experiment for the mid-length data while decreasing the model size used (Figure 3, second panel). In the case of vanilla gradient descent, the model size is directly correlated to its performance,

31

61

91

10

3

10

2

10

1

10 0

Interpolation MSE

2 3

2 4

2 5

10

2

10

1

0.1

0.3

0.5

10

2

10

1

10 0

5

10

20

50

10

2

10

1

10 0

31

61

91

Train data length

10

1

10 1

10 3

Extrapolation MSE

Homotopy

Multishoot

Vanilla

2 3

2 4

2 5

Nodes per layer

10

2

10

1

10 0

0.1

0.3

0.5

Sampling period

t

10

2

10

1

10 0

10 1

5

10

20

50

Noise strength (% mean)

10

2

10

1

10 0

lending the claim some support. However, we find that both homotopy and multiple shooting methods can effectively train much smaller models to match the accuracy of a larger model trained with vanilla gradient descent. This signifies that the cause of the problem is not the capacity itself, but rather the inability of training algorithms to fully capitalize on the model capacity.

Robustness against data degradation We performed an additional set of experiments to evaluate the performances of different training methods against increasing data sparsity and noise. This is particularly of interest for our homotopy training as we employ smoothing cubic splines (Section 5.1) whose reliability can be adversely affected by the data quality. From the results (Figure 3, third, fourth panels) we find this is not the case and that that our algorithm performs well even for degraded data. One unexpected result is the drastic underperformance of the multiple shooting method with increased sampling period, which is a side effect from the fact we held the time interval fixed for this experiment. Due to the space constraint, we continue this discussion in Appendix G.3.

7.2 Performance benchmarks of the training algorithms

Having confirmed that both the homotopy and multiple-shooting methods successfully mitigate the NeuralODE training problem, we now present a comprehensive benchmark in Figure 4.

101

Homotopy

102

4000

Multishoot

3500

MSE (Interpolation)

100

Vanilla

MSE (Extrapolation)

101

3000

Epochs

2500

10-1

100

2000

1500

10-2

10-1

1000

500

Black Box

Black Box Second Order Black Box

10-2

Black Box

Black Box Second Order Black Box

0

Black Box

Black Box Second Order Black Box

Lotka-Volterra

Double Pendulum

Lotka-Volterra

Double Pendulum

Lotka-Volterra

Double Pendulum

From the results, it is clear both our homotopy method and multiple shooting can both boost the interpolation performance of NeuralODEs, with our method being slightly more performant. Inspecting the extrapolation performance, however, we do find a clear performance gap between our method and multiple shooting, with the latter even performing worse than vanilla gradient descent for the double pendulum dataset (Figure 4, center, Figure 5, left). This underperformance of multiple shooting for extrapolation tasks can be attributed to the nature of the algorithm: during training time, multiple shooting integrates NeuralODEs on shorter, subdivided time intervals, whereas during inference time the NeuralODE needs to be integrated for the entire data time points and then some more for extrapolation. This disparity between training and inference render models trained by

multiple shooting to be less likely to compensate for error accumulation in long term predictions, resulting in poor extrapolation performance.

Another stark difference between our homotopy training and the other methods is in the number of epochs required to arrive at minimum interpolation MSE (Figure 4, right panel). While the exact amount of speedup is variable and seems to depend on the dataset difficulty, we find a 90%40% reduction compared to vanilla training, highlighting the effectiveness of our approach. One unexpected observation from our experiments was that the epoch at which minimum interpolation MSE epoch was achieved did not always correspond to the homotopy parameter λ being zero (Appendix G.2). While seemingly counterintuitive, this is in line with the literature, where nonzero homotopy parameter has resulted in better generalization performances [36, 24].

What makes homotopy training effective: inspecting training dynamics To obtain a complementary insight to the NeuralODE training dynamics and relate the effectiveness of our method with the properties of the loss landscape, we calculated the trace of the loss hessian during training (Figure 5, right panel). Small, positive trace values indicate more favorable training dynamics [69], and we find that our method results in the smallest trace, followed by multiple shooting then vanilla gradient descent. The fact that the hessian trace maintains a low value throughout the entire homotopy training, even as the loss surface is morphed back to its unsynchronized, irregular state, also confirms that our homotopy schedule works well and managed to gradually guide the optimizer into a well performing loss minimum.

State

1

0

State

0

103

trace

一

1

-2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

Hessiant

102

10

Zm

20 ↓

State

0

State

0

101

Homotopy

Multi.-shoot

-10

-20

Vanilla

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0

1000

2000

3000

4000

Time t

Time t

Train epochs

8 Limitations and discussion

In this paper, we adapted the concepts of synchronization and homotopy optimization for the first time in the NeuralODE literature to stabilize NeuralODE training on long time-series data. Through loss landscape analysis, we established that the cause of the training problem is identical between differential equations, RNNs and NeuralODEs and that synchronization can effectively tame the irregular landscape. We demonstrated that our homotopy method can successfully train NeuralODE models with strong interpolation and extrapolation accuracies.

Note that our work serves as an initial confirmation that the methods developed in the ODE parameter estimation literature can be adapted to be competitive in the NeuralODE setting. As such, there are multiple opportunities for future research, including theoretical studies on the convergence properties of our algorithm and further testing out ideas in the synchronization literature, such as alternative parametrization for the coupling matrix [28], or incorporating the homotopy parameter into the loss function as a regularization term [1, 58]. Extending our algorithm to high-dimensional systems is another important direction, which we believe will be possible through the use of latent ODE models [70]. Nevertheless, our work hold merit in that it lays the groundwork upon which ideas from ODE parameter estimation, control theory, and RNN/NeuralODE training can be assimilated, which we believe will facilliate further research connecting these domains.

Author Contributions

Ko and H. Koh contributed equally to the paper.

Acknowledgments

This work was supported by grants from the National Research Foundation of Korea (No. 2016R1A3B1908660) to W. Jhe.

References

H. D. I. Abarbanel, D. R. Creveling, R. Farsian, and M. Kostuk. Dynamical State and Parameter Estimation. SIAM Journal on Applied Dynamical Systems , 8(4):1341-1381, 2009.

H. D. I. Abarbanel, D. R. Creveling, and J. M. Jeanne. Estimation of parameters in nonlinear systems using balanced synchronization. Physical Review E , 77(1):016208, 2008.

E. L. Allgower and K. Georg. Numerical Continuation Methods: An Introduction . Springer Berlin Heidelberg, Berlin, Heidelberg, 1990.

M. Arjovsky, A. Shah, and Y. Bengio. Unitary Evolution Recurrent Neural Networks. In Proceedings of The 33rd International Conference on Machine Learning , pages 1120-1128. PMLR, 2016.

E. Baake, M. Baake, H. G. Bock, and K. M. Briggs. Fitting ordinary differential equations to chaotic data. Physical Review A , 45(8):5524-5529, 1992.

Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning , ICML '09, pages 41-48, New York, NY, USA, 2009. Association for Computing Machinery.

L. Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.

H. G. Bock. Recent Advances in Parameter Identification Techniques for O.D.E. In P. Deuflhard and E. Hairer, editors, Numerical Treatment of Inverse Problems in Differential and Integral Equations: Proceedings of an International Workshop, Heidelberg, Fed. Rep. of Germany, August 30 - September 3, 1982 , Progress in Scientific Computing, pages 95-121. Birkhäuser, Boston, MA, 1983.

B. Chang, M. Chen, E. Haber, and E. H. Chi. AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks. In International Conference on Learning Representations , volume 7, 2019.

Q. Chen and W. Hao. A homotopy training algorithm for fully connected neural networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 475(2231):20190662, Nov. 2019.

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

K. M. Choromanski, J. Q. Davis, V. Likhosherstov, X. Song, J.-J. Slotine, J. Varley, H. Lee, A. Weller, and V. Sindhwani. Ode to an ODE. In Advances in Neural Information Processing Systems , volume 33, pages 3338-3350. Curran Associates, Inc., 2020.

J. Chow, L. Udpa, and S. Udpa. Homotopy continuation methods for neural networks. In 1991., IEEE International Sympoisum on Circuits and Systems , volume 5, pages 2483-2486, 1991.

K. Doya. Bifurcations of recurrent neural networks in gradient descent learning. IEEE Transactions on neural networks , 1(75):218-229, 1993.

D. M. Dunlavy and D. P. O'Leary. Homotopy optimization methods for global optimization. Technical Report SAND2005-7495, Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA (United States), 2005.

W. Falcon and The PyTorch Lightning team. PyTorch Lightning, Mar. 2019.

C. Finlay, J.-H. Jacobsen, L. Nurbekyan, and A. Oberman. How to Train Your Neural ODE: The World of Jacobian and Kinetic Regularization. In Proceedings of the 37th International Conference on Machine Learning , volume 37, pages 3154-3164. PMLR, 2020.

M. Finzi, K. A. Wang, and A. G. Wilson. Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 13880-13889. Curran Associates, Inc., 2020.

B. Ghorbani, S. Krishnan, and Y. Xiao. An Investigation into Neural Net Optimization via Hessian Eigenvalue Density. In Proceedings of the 36th International Conference on Machine Learning , pages 2232-2241. PMLR, 2019.

A. Ghosh, H. Behl, E. Dupont, P. Torr, and V. Namboodiri. STEER : Simple Temporal Regularization For Neural ODE. In Advances in Neural Information Processing Systems , volume 33, pages 14831-14843. Curran Associates, Inc., 2020.

P. J. Green, B. W. Silverman, P. J. Green, and B. W. Silverman. Nonparametric Regression and Generalized Linear Models: A Roughness Penalty Approach . Number 58 in Monographs on Statistics and Applied Probability. Chapman & Hall /CRC, Boca Raton, Fla., 2000.

S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian Neural Networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

N. Gruver, M. A. Finzi, S. D. Stanton, and A. G. Wilson. Deconstructing the Inductive Biases of Hamiltonian Neural Networks. In International Conference on Learning Representations , volume 9, 2021.

C. Gulcehre, M. Moczulski, F. Visin, and Y. Bengio. Mollifying Networks. In International Conference on Learning Representations , volume 5, 2017.

R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu. Liquid Time-constant Networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7657-7666, 2021.

J.-H. He. Homotopy perturbation technique. Computer Methods in Applied Mechanics and Engineering , 178(3-4):257-262, 1999.

H. Iwakiri, Y. Wang, S. Ito, and A. Takeda. Single Loop Gaussian Homotopy Method for Non-convex Optimization. Advances in Neural Information Processing Systems , 35:7065-7076, 2022.

G. A. Johnson, D. J. Mar, T. L. Carroll, and L. M. Pecora. Synchronization and imposed bifurcations in the presence of large parameter mismatch. Physical Review Letters , 80(18):39563959, 1998.

S. Kanai, Y. Fujiwara, and S. Iwamura. Preventing Gradient Explosions in Gated Recurrent Units. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

P. Kidger, R. T. Q. Chen, and T. J. Lyons. "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms. In Proceedings of the 38th International Conference on Machine Learning , pages 5443-5452. PMLR, 2021.

S. Kim, W. Ji, S. Deng, Y. Ma, and C. Rackauckas. Stiff neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlinear Science , 31(9):093122, 2021.

A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. In Advances in Neural Information Processing Systems , volume 34, pages 26548-26560. Curran Associates, Inc., 2021.

P. J. M. Laarhoven and E. H. L. Aarts. Simulated Annealing: Theory and Applications . Springer Netherlands, Dordrecht, 1987.

M. Lezcano-Casado and D. Martínez-Rubio. Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group. In Proceedings of the 36th International Conference on Machine Learning , pages 3794-3803. PMLR, 2019.

H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the Loss Landscape of Neural Nets. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

X. Lin, Z. Yang, X. Zhang, and Q. Zhang. Continuation path learning for homotopy optimization. In Proceedings of the 40th International Conference on Machine Learning , pages 21288-21311. PMLR, 2023.

I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations , volume 7, 2019.

S. Massaroli, M. Poli, S. Sonoda, T. Suzuki, J. Park, A. Yamashita, and H. Asama. Differentiable Multiple Shooting Layers. In Advances in Neural Information Processing Systems , volume 34, pages 16532-16544. Curran Associates, Inc., 2021.

T. Matsubara, Y. Miyatake, and T. Yaguchi. Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory. In Advances in Neural Information Processing Systems , volume 34, pages 20772-20784. Curran Associates, Inc., 2021.

A. Maybhate and R. E. Amritkar. Use of synchronization and adaptive control in parameter estimation from a time series. Physical Review E , 59(1):284-293, 1999.

J. Mikhaeil, Z. Monfared, and D. Durstewitz. On the difficulty of learning chaotic dynamics with RNNs. Advances in Neural Information Processing Systems , 35:11297-11312, 2022.

J. Miller and M. Hardt. Stable Recurrent Models. In International Conference on Learning Representations , volume 7, 2019.

J. D. Murray. Mathematical Biology. 1: An Introduction . Number 17 in Interdisciplinary Applied Mathematics. Springer-Verlag GmbH, Berlin Heidelberg, softcover reprint of the hardcover 3rd edition 2002, corrected second printing edition, 2004.

U. Parlitz. Estimating Model Parameters from Time Series by Autosynchronization. Physical Review Letters , 76(8):1232-1235, 1996.

R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning , pages 1310-1318. PMLR, 2013.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 , 2019.

L. M. Pecora and T. L. Carroll. Synchronization in chaotic systems. Physical Review Letters , 64(8):821-824, 1990.

L. M. Pecora and T. L. Carroll. Driving systems with chaotic signals. Physical Review A , 44(4):2374-2383, 1991.

L. M. Pecora and T. L. Carroll. Synchronization of chaotic systems. Chaos: An Interdisciplinary Journal of Nonlinear Science , 25(9):097611, 2015.

M. Peifer and J. Timmer. Parameter estimation in ordinary differential equations for biochemical processes using the method of multiple shooting. IET Systems Biology , 1(2):78-88, 2007.

J. H. Peng, E. J. Ding, M. Ding, and W. Yang. Synchronizing hyperchaos with a scalar transmitted signal. Physical Review Letters , 76(6):904-907, 1996.

V. G. Polisetty, S. K. Varanasi, and P. Jampana. Stochastic state-feedback control using homotopy optimization and particle filtering. International Journal of Dynamics and Control , 10(3):942-955, 2022.

J. C. Quinn, P. H. Bryant, D. R. Creveling, S. R. Klein, and H. D. I. Abarbanel. Parameter and state estimation of experimental chaotic systems using synchronization. Physical Review E , 80(1):016201, 2009.

C. Rackauckas, M. Innes, Y. Ma, J. Bettencourt, L. White, and V. Dixit. DiffEqFlux.jl - A Julia Library for Neural Differential Equations, 2019.

C. Rackauckas, Y. Ma, J. Martensen, C. Warner, K. Zubov, R. Supekar, D. Skinner, A. Ramadhan, and A. Edelman. Universal Differential Equations for Scientific Machine Learning, 2021.

J. O. Ramsay, G. Hooker, D. Campbell, and J. Cao. Parameter estimation for differential equations: A generalized smoothing approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 69(5):741-796, 2007.

A. H. Ribeiro, K. Tiels, L. A. Aguirre, and T. Schön. Beyond exploding and vanishing gradients: Analysing RNN training using attractors and smoothness. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages 2370-2380. PMLR, 2020.

K. Schäfer, K. Flaßkamp, J. Fliege, and C. Büskens. A Combined Homotopy-Optimization Approach to Parameter Identification for Dynamical Systems. PAMM , 19(1), 2019.

M. Schmidt and H. Lipson. Distilling Free-Form Natural Laws from Experimental Data. Science , 324(5923):81-85, 2009.

N. B. Toomarian and J. Barhen. Learning a trajectory using adjoint functions and teacher forcing. Neural Networks , 5(3):473-484, 1992.

E. M. Turan and J. Jäschke. Multiple shooting for training neural differential equations on time series. IEEE Control Systems Letters , 6:1897-1902, 2022.

B. van Domselaar and P. Hemker. Nonlinear parameter estimation in initial value problems. Technical Report MC-NW-75-18, Amsterdam Univ. Math. Cent. Num. Sci., 1975.

S. K. Varanasi, P. Jampana, and C. P. Vyasarayani. Minimum attention stochastic control with homotopy optimization. International Journal of Dynamics and Control , 9(1):266-274, 2021.

H. U. Voss, J. Timmer, and J. Kurths. Nonlinear Dynamical System Identification from Uncertain and Indirect Measurements. International Journal of Bifurcation and Chaos , 14(06):1905-1933, 2004.

C. P. Vyasarayani, T. Uchida, A. Carvalho, and J. McPhee. Parameter identification in dynamic systems using the homotopy optimization approach. Multibody System Dynamics , 26(4):411424, 2011.

C. P. Vyasarayani, T. Uchida, and J. McPhee. Single-shooting homotopy method for parameter identification in dynamical systems. Physical Review E , 85(3):036201, 2012.

L. T. Watson. Globally convergent homotopy methods: A tutorial. Applied Mathematics and Computation , 31:369-396, May 1989.

L. T. Watson and R. T. Haftka. Modern homotopy methods in optimization. Computer Methods in Applied Mechanics and Engineering , 74(3):289-305, 1989.

Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney. PyHessian: Neural Networks Through the Lens of the Hessian. In 2020 IEEE International Conference on Big Data (Big Data) , pages 581-590. IEEE, 2020.

C. Yildiz, M. Heinonen, and H. Lahdesmaki. ODE2VAE: Deep generative second order ODEs with Bayesian neural networks. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Y. Yin, V. Le Guen, J. Dona, E. de Bézenac, I. Ayed, N. Thome, and P. Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124012, 2021.

J. Zhuang, N. Dvornek, X. Li, S. Tatikonda, X. Papademetris, and J. Duncan. Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE. In Proceedings of the 37th International Conference on Machine Learning , pages 11639-11649. PMLR, 2020.

A Additional figures

In this section, we present some of the figures that were excluded from the main text for brevity.

Figure 6 below highlights the efficiency of our training approach with respect to the number of training epochs by plotting the trajectory predictions of the second-order models during training.

State

1,

2

State

1,

2

Epoch = 99

Epoch = 199

Epoch = 900(best)

State

1,

2

State

1,

2

Epoch = 99

Epoch = 199

Epoch = 3833(best)

State

1,

2

State

1,

2

Multi.shoot. Epoch = 59

Epoch = 99

Epoch = 199

Epoch = 3286(best)

Figures 7 to 9 show model prediction trajectories corresponding to our benchmark results of Figure 4. One intriguing point is that even though both the gray-box model (Lotka-Volterra system, Figure 7) and the second-order model (double pendulum, Figure 8) contain more prior information about their respective systems, this does not necessarily translate to better predictive capabilities if an effective training method is not used. This suggests that introducing physics into the learning problem can obfuscate optimization, something that has been reported for physics-informed neural networks [32]. It also highlights the effectiveness of our homotopy training algorithm, as our method can properly train such difficult-to-optimize models.

0

5

10

State

x

Black-box

0

5

10

State

x

Gray-box

Data

Homotopy

Multi.-shoot

Vanilla

Extrap. start time

0

2

4

6

8

10

12

Time

t

0

2

4

6

State

y

0

2

4

6

8

10

12

Time

t

0

2

4

6

State

y

0.0

0.5

1.0

1.5

2.0

0

2

State

1

Black-box

0.0

0.5

1.0

1.5

2.0

0

2

Second-order

Data

Homotopy

Multishoot

Vanilla

0.0

0.5

1.0

1.5

2.0

2.5

0.0

State

2

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.0

0.5

1.0

1.5

2.0

10

0

10

State

1

0.0

0.5

1.0

1.5

2.0

10

0

10

0.0

0.5

1.0

1.5

2.0

Time

t

20

0

20

State

2

0.0

0.5

1.0

1.5

2.0

Time

t

20

0

20

0

2

4

6

Time

t

10

5

0

5

10

15

20

State

x

Homotopy

Multi.-shoot

Vanilla

Extrap. start time

0

2

4

6

Time

t

10

0

10

20

State

y

0

2

4

6

Time

t

0

10

20

30

40

State

z

B Loss landscape and synchronization in ODE parameter estimation

Here, we illustrate the similarity between the ODE parameter estimation problem and the NeuralODE training problem by repeating analyses analogous to those of Section 4 in the setting of estimating the unknown coefficients of a given ordinary differential equation.

In the left and middle panels of Figure 10, we show the trajectories of the Lotka-Volterra and the Lorenz systems where a single parameter was perturbed. We can easily observe that the original trajectory and the perturbed trajectories evolve independently in time. Furthermore, while trajectories from the simpler Lotka-Volterra system retain the same periodic behavior and differ only in their amplitudes and phases, trajectories from the Lorenz system display much different characteristics with increasing time. This translates over to the shape of the loss landscape (Figure 10, right) with the loss for the Lotka-Volterra system only becoming steeper with increasing data length, whereas the loss for the Lorenz system displays more and more local minima.

Once synchronization is introduced to the perturbed trajectory, its dynamics tends to converge with the reference trajectory as time increases, with the rate of convergence increasing for larger values of the coupling strength k (Figure 11, left and middle panels). In terms of the loss landscape (right panel), larger coupling does result in a smoother landscape, with excessive amounts of it resulting in a flat landscape that is also detrimental to effective learning. These results are a direct parallel to our observations in Figure 2 and provide additional justifications as to why techniques developed in the field of ODE parameter estimation also work so well on NeuralODE training.

0

5

10

x

0%

+50%

50%

0

2

4

6

8

10

0

5

y

0

20

x

0

25

y

0

2

4

6

8

10

0

25

z

0

5

10

(

)

40

20

0

20

40

Parameter perturbation (%)

0

50

100

150

(

)

0

5

10

x

k=0.0

k=0.1

k=1.0

Ref.

0

2

4

6

8

10

0.0

2.5

5.0

y

0

20

x

k=0.0

k=0.3

k=1.5

Ref.

0

20

y

0

2

4

6

8

10

0

25

z

0

5

10

(

.

K

)

k = 0.0

k = 0.1

k = 1.0

40

20

0

20

40

Parameter perturbation (%)

0

50

100

(

.

K

)

k = 0.0

k = 0.3

k = 1.5

C Further description of the homotopy optimization procedure

To implement homotopy optimization, one usually selects the number of discrete steps for optimization, s , as well as a series of positive decrement values for the homotopy parameter { ∆ λ ( k ) } s -1 0 that sum to 1. Afterwards, optimization starts with an initial λ value of λ (0) = 1 , which gives H (0) ( θ ) = G ( θ ) . At each step, the objective function at the current iteration is minimized with respect to θ , using the output from the previous step θ ∗ ( k -1) is as the initial guess:

Afterwards, λ decremented to its next value, λ ( k +1) = λ ( k ) -∆ λ ( k ) , and this iteration continues until the final step s where λ ( s ) = 0 , H ( s ) ( θ ) = F ( θ ) , and the final minimizer θ ∗ ( s ) = θ ∗ is the sought-after solution to the original problem F ( θ ) .

D Multiple shooting algorithm

Our implementation of the multiple shooting algorithm for training NeuralODEs closely mirrors the example code provided in the DiffEqFlux.jl package [54] of the Julia programming language ecosystem.

Given time series data { t i , ˆ u i } N i =0 , multiple shooting method partitions the data time points into m overlapping segments:

During training, the NeuralODE is solved for each time segment, resulting in m segmented trajectories:

Taking into account the overlapping time point between segments, these trajectories can then all be concatenated to produce the full trajectory prediction:

From this, the data loss is defined identically to conventional NeuralODE training: L data ( θ ) = 1 N +1 ∑ i | u i ( θ ) -ˆ u i | 2 .

However, due to the segmented manner in which the above trajectory is generated, optimizing only over this quantity will result in a model that could generate good piecewise predictions, but is unable to generate a proper, continuous global trajectory. Therefore, to ensure the model can generate smooth trajectories, continuity constraints u ( i -1) n = u ( i ) 0 , ( i ∈ 1 . . . m -1) must be enforced. How this is achieved differs across the literature, and our implementation - following that of [54] - introduces a regularization term in the loss:

Finally, the train loss for the multiple shooting method is then defined as

and is minimized using gradient descent, where β is a hyperparameter that tunes the relative importance of the two terms during training.

E Experiment details

Our experiments were implemented in PyTorch [46], using the pytorch-lightning [16] framework. The wandb library [7] was used to keep track of all experiments as well as to perform hyperparameter sweeps. In all experiments, the AdamW optimizer [37] was used to minimize the respective loss functions for the vanilla and homotopy training. Each experiment was repeated using random seed values of 10, 20, and 30 to compute the prediction means and standard errors.

E.1 Lotka-Volterra system

The Lotka-Volterra system is a simplified model of predator-prey dynamics given by,

where the parameters α, β, γ, δ characterize interactions between the populations.

Data preparation. Following the experimental design of Rackauckas et al. [55], we numerically integrate Equation (9) in the time interval t ∈ [0 , 6 . 1] , ∆ t = 0 . 1 with the parameter values α, β, γ, δ = 1 . 3 , 0 . 9 , 0 . 8 , 1 . 8 and initial conditions x (0) , y (0) = 0 . 44249296 , 4 . 6280594 . Continuing with the recipe, Gaussian random noise with zero mean and standard deviations with magnitude of 5% of the mean of each trajectory was added to both states. For both data generation and NeuralODE prediction, integration was performed using the adaptive step size dopri5 solver from the torchdiffeq package [11] with an absolute tolerance of 1e-9 and a relative tolerance of 1e-7.

For the experiments of Figure 3, training data were generated in a similar manner, but with specific parameters varied to match the corresponding experiments. Data varying train data length (Figure 3, left panel) were generated using time spans of t ∈ [0 , 3 . 1] , [0 , 6 . 1] , [0 . 9 . 1] respectively with shared parameter values of ∆ t = 0 . 1 and noise amplitude of 5% of the mean. Data with differing sampling periods (Figure 3, third panel) were generated with a fixed time span of t ∈ [0 , 6 . 1] , noise amplitude of 5% of the mean and sampling periods of ∆ t = 0 . 1 , 0 . 3 , 0 . 5 . Data with different noise amplitudes (Figure 3, fourth panel) were generated using a time span of t ∈ [0 , 6 . 1] , sampling period of ∆ t = 0 . 1 and noise amplitudes of 5%, 10%, 20%, 50% of the mean.

Model architecture. We used two types of models for this dataset, a black-box NeuralODE of Equation (1), and a gray-box NeuralODE used in Rackauckas et al. [54] that incorporates partial information about the underlying equation, given by:

In our default setting, the black-box NeuralODE had 3 layers with [2, 32, 2] nodes respectively. The gray-box NeuralODE had 4 layers with [2, 20, 20, 2] nodes, with each node in the output layer corresponding to the output of U 1 , U 2 of Equation (10). Following the results from [31], a nonsaturating gelu activation was used for all layers except for the final layer, where identity activation was used. For the model capacity experiment (Figure 3, second panel), the number of nodes in the hidden layer was changed accordingly.

E.2 Double pendulum

The double pendulum system is a canonical example in classical mechanics, and has four degrees of freedom θ 1 , θ 2 , ω 1 , ω 2 corresponding to the angles and angular velocities of the two pendulums with respect to the vertical. The governing equation for the system can be derived using Lagrangian formulation of classical mechanics and is given by:

where ∆ θ = θ 2 -θ 1 , m 1 , m 2 are the masses of each rod, and g is the gravitational acceleration.

Data preparation. While simulated trajectories for the double pendulum can be generated using the equations above, we instead used the experimental data from Schmidt & Lipson [59]. This consists of two trajectories from the double pendulum, captured using multiple cameras. The noise in the data is subdued due to the LOESS smoothing performed by the original authors. For our experiments, we used the first 100 points of the first trajectory for training and the next 100 to evaluate the extrapolation capabilities of the trained model.

Model architecture. Two types of models were used for this dataset: a black-box NeuralODE (Equation (1)) and a NeuralODE with secondorder structure[23], which for this system, takes the form:

The black-box NeuralODE had 4 layers with [4, 50, 50, 4] nodes, with the input and output node numbers corresponding to the degrees of freedom of the system. The second-order model also had 4 layers with [4, 50, 50, 2] nodes. Note that there are now 2 output nodes instead of 4 since incorporating second-order structure requires the neural network to only model the derivatives of ω 1 and ω 2 . For both models, gelu activations were used for all layers except the last, which used identity activation. Identical to the previous dataset, the NeuralODEs were integrated using the dopri5 solver from the torchdiffeq package [11] with an absolute tolerance of 1e-9 and a relative tolerance of 1e-7.

E.3 Lorenz system

The Lorenz system is given by the equations

Data preparation. To generate the data, we followed experimental settings of Vyasarayani et al. [66] and used parameter values σ, ρ, β = 10 , 28 , 8 / 3 and the initial condition x 0 , y 0 , z 0 = 1 . 2 , 2 . 1 , 1 . 7 . These conditions, upon integration, gives rise to the well-known 'butterfly attractor". The training data was generated in the interval t ∈ [0 , 3 . 1] , and still adhering to the paper, a Gaussian noise of mean 0 and standard deviation 0.25 was added to the simulated trajectories to emulate experimental noise. For data generation and NeuralODE prediction, the adaptive step size dopri5 solver from the torchdiffeq package [11] was used with an absolute tolerance of 1e-9 and a relative tolerance of 1e-7. ODE solver and the tolerance values were kept identical to the previous Lotka-Volterra experiment.

Model architecture. We used a single NeuralODE for this dataset - a black-box model having 4 layers with [3, 50, 50, 3] nodes. The number of nodes for the input and output layers correspond to the three degrees of freedom of the state vector u = [ x, y, z ] T of the system. The activations for the model was kept identical to the previous experiments - gelu activation for all layers, except for the last layer which used identity activation.

F Hyperparameter selection

For all combinations of models and datasets used, hyperparameters for the optimization were chosen by running sweeps prior to the experiment with a fixed random seed of 10, then selecting hyperparameter values that resulted in the lowest mean squared error value during training.

Note that for the experiment of Figure 3, the same hyperparameters used in Figure 4 was used for all experiments, due to the sheer cost of sweeping for the hyperparameters for every change of independent variables.

Vanilla gradient descent. Vanilla gradient descent has learning rate as its only hyperparameter. Due to finite computation budget, we set the maximum training epochs to 4000 for all sweeps and experiments. We list the values used in the hyperparameter sweep as well as the final selected values for each dataset in Table 1. We found that larger learning rates lead to numerical underflow in the adaptive ODE solver while training on the more difficult Lorenz and double datasets; hence, the sweep values for the learning rate parameters were taken to be lower than those for the Lotka-Volterra dataset.

Multiple shooting. The multiple shooting method has three hyperparameters: learning rate, number of segments, and continuity penalty. For the majority of experiments, we set number of segments to 5, and performed hyperparameter sweep on the remaining two parameters.

Homotopy optimization. Our homotopy-based NeuralODE training has five hyperparameters: learning rate ( η ), coupling strength ( k ), number of homotopy steps ( s ), epochs per homotopy steps ( n epoch ), and homotopy parameter decrement ratio ( κ ).

While sweeping over all five hyperparameters would yield the best possible results, searching such a high dimensional space can pose a computation burden. In practice, we found that sweeping over only the first two hyperparameters and fixing the rest at predecided values still returned models that functioned much better than their vanilla counterparts. We list the predecided hyperparameter values for each dataset in Table 3. Note that we increased the epochs per homotopy step from 100 the Lotka-Volterra model to 300 for the Lorenz system and double pendulum datasets to account for the increased difficulty of the problem.

Table 4 shows the sweeped hyperparameters as well as the final chosen values for each dataset. The total number of training epochs is given by multiplying the number of homotopy steps with epochs per homotopy steps. This resulted in 600 epochs for the Lotka-Volterra dataset, and 1800 epochs for both the Lorenz system and the double pendulum datasets.

G Further results

G.1 Table of the benchmark results

Here, we present the benchmark results of Figure 4 in a table format as an alternative data representation.

G.2 Example of the training curves

Here, we include some of the training curves for our experiments. Figure 13 displays the averaged training curves for the black-box model trained on the Lotka-Volterra dataset. From the right panel, it can clearly be seen that our homotopy method optimizes the MSE much rapidly than the other methods, arriving at the noise floor in less than 500 epochs. The abrupt jumps in the MSE curve for homotopy is due to the discontinous change in the train loss that occurs every the the homotopy parameter is adjusted. To make this connection clearer, we show the train loss as well as the homotopy parameter for our method in the left panel of the same figure.

10

2

10

1

10 0

10 1

Train loss

L

(

;

)

0

200

400

600

Epoch

0.0

0.5

1.0

0

500

1000

1500

2000

2500

3000

3500

4000

Epoch

10

2

10

1

10 0

10 1

10 2

10 3

Mean squared error

Noise floor

Homotopy

Multi.-shooting

Vanilla

We present analogous results for the second-order model trained on the double pendulum dataset in Figure 14. Once again, we find that our homotopy method converges much quicker than its competitors.

10

3

10

2

10

1

10 0

10 1

Train loss

L

(

;

)

0

300

600

900

1200 1500 1800

Epoch

0.0

0.5

1.0

0

500

1000

1500

2000

2500

3000

3500

4000

Epoch

10

3

10

2

10

1

10 0

10 1

Mean squared error

Noise floor

Homotopy

Multi.-shooting

Vanilla

G.3 Further discussion on the Lotka-Volterra system results

In this section, we present additional results regarding our experiment of Figure 3 and continue our discussion.

Increasing data length. Figure 15 depicts the predicted trajectories for differently trained models. For the shortest data, it can be seen that all models did overfit on the training data and failed to capture the periodic nature of the system. This justifies our attributing the large extrapolation error of the models in Figure 3 to overfitting. However, we emphasize that this overfitting is not due to failings in the training algorithms, but rather due to the insufficient information in the training data, as the models cannot be expected to learn periodicity without being given at least a single period worth of data. As the data is increased, we find that both homotopy and multiple shooting properly capture the dynamics of the system, whereas the vanilla method was unable to properly learn the system for the longest data.

0

5

10

State

x

Data

Homotopy

Multishoot

Vanilla

0

5

10

0

5

10

0

5

10

15

Time

t

0

2

4

6

State

y

0

5

10

15

Time

t

0

2

4

6

0

5

10

15

Time

t

0

2

4

6

Reducing model capacity. In Figure 16, we show the prediction trajectories corresponding to the model capacity experiment. It is clear from the results that both the homotopy and multiple shooting methods can produce models that accurately portray the dynamics, regardless of the reduction in the model size. In contrast, predictions from vanilla training deteriorate as the model size decreases, with the smallest model inaccurately estimating both the amplitude and the phase of the oscillations.

0.0

2.5

5.0

7.5

State

x

Data

Homotopy

Multishoot

Vanilla

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

0

5

10

Time

t

0

2

4

6

State

y

0

5

10

Time

t

0

2

4

6

0

5

10

Time

t

0

2

4

6

Increasing sampling period. The model trajectories for decreasing sampling period in the training data is shown below in Figure 17. Intriguingly, we find that the multiple shooting method results suffer greatly with the increased data sparsity. This is a side effect of our experiment setting, where we set the time interval constant while increasing the sampling period. To elaborate, our choice of fixed time interval causes the number of training data points decrease as the period increases. However, as multiple shooting trains by subdividing the training data, it struggles on small datasets because this leads to each segment containing even less data points that do not convey much information about the data dynamics.

Another interesting observation that can be made is that vanilla training gives better results when sparser data is used, which is also confirmed by the error values in the third panel of Figure 3. This suggests an alternate training improvement strategy for training NeuralODEs on long time series data - by training on subsampled data, then if necessary, gradually anneal the data intervals back to its original form as training proceeds. Of course, the effectiveness of such a scheme will need to be tested for more complex systems, which is outside the scope of this paper.

0.0

2.5

5.0

7.5

State

x

Data

Homotopy

Multishoot

Vanilla

0

5

0.0

2.5

5.0

7.5

0

5

10

Time

t

0

2

4

6

State

y

0

5

10

Time

t

0

2

4

6

0

5

10

Time

t

0.0

2.5

5.0

As our homotopy training method utilized a cubic smoothing spline to supply the coupling term, we also inspected the quality of this interpolant as the data period was increased. From Figure 18, we find that the cubic spline relatively resistant to the increased data sparsity, which may be one of the reasons why our homotopy method is very robust against this mode of data degradation.

0

5

10

State

x

Data

Ground truth

Cubic spline

0

5

10

0

5

10

0

2

4

6

Time

t

0

2

4

6

State

y

0

2

4

6

Time

t

0

2

4

6

0

2

4

6

Time

t

0

2

4

6

Increasing data noise. Finally, Figure 19 shows the model trajectories as the noise in the training data is increased. Here, we do find that while all methods return deteriorating predictions as the noise is increased, our homotopy method is found to be most robust to noise, followed by multiple shooting, and finally vanilla gradient descent. A part of our algorithm's robustness to noise can be explained by the use of the cubic smoothing splines. As our use of synchronization couples the NeuralODE dynamics to the cubic spline trajectory during training, one could argue that the denoising effect of cubic splines is thus directly transferred to the model.

0

5

10

State

x

Data

Homotopy

Multishoot

Vanilla

0

5

10

0

5

10

0

5

10

0

5

10

Time

t

0

2

4

6

State

y

0

5

10

Time

t

0

2

4

6

0

5

10

Time

t

0

2

4

6

0

5

10

Time

t

0

2

4

6

However, this does not seem to be the full picture. Figure 20 shows the cubic smoothing splines used during modeling training for each of the noise levels. As we held the amount of smoothing constant throughout our experiments, we see that for larger noise amplitudes, the spline fails to reject all noise, and displays irregular high frequency behaviors. On the other hand, the corresponding trained trajectories for the homotopy method (Figure 19, third and fourth panels) do not mirror such irregular oscillations, indicating that the homotopy optimization procedure itself also has an intrinsic robustness to noise.

0

5

10

State

x

Data

Ground truth

Cubic spline

0

5

10

0

5

10

0

5

10

0

2

4

6

Time

t

0

2

4

6

State

y

0

2

4

6

Time

t

0

2

4

6

0

2

4

6

Time

t

0

2

4

6

0

2

4

6

Time

t

0

2

4

6

G.4 Comparison to a non-deep learning algorithm: SINDy

In this section, we briefly compare our algorithm to a more traditional symbolic regression-based method. We choose the well-known SINDy algorithm, which is readily available in the pysindy package. This algorithm takes as input, time series measurements of the states of the dynamical system in question, as well as a dictionary of candidate terms that could constitute the governing equation. Afterwards, the time derivative of the states is estimated numerically, and a sparse regression is performed to determine which terms exist in the governing equation, as well as what their coefficient values are.

Compared to neural network-based approaches, the unique feature of SINDy is that its results are given in the form of analytical expressions. Therefore if the governing equation for the data is simple (e.g. consists of low-order polynomial terms), or if the user has sufficient prior information about the system and can construct a very small dictionary that contains all the terms that appear in the true equation, SINDy can accurately recover the ground truth equation.

On the other hand, if the system is high dimensional so that the space of possible candidate terms start to grow drastically, or if the governing equation has a arbitrary, complex form that cannot be easily guessed, SINDy either does not fit the data properly, or even if it does, recovers inaccurate equations that are difficult to assign meaning to. These correspond to situations where NeuralODEs, and hence our method for effectively training them, shines.

To illustrate this point, we choose the double pendulum dataset and compare the predicted trajectories from SINDy and our homotopy algorithm. As one can see from Equation (13), the governing equation of this system has an extremely complicated form that cannot be guessed easily. To reflect this difficulty, we chose the basis set of the dictionary of the candidate terms to be the state variable themselves, their sines and cosines, and the higher harmonics of the sines and cosines.

0.0

0.5

1.0

t

1.0

0.5

0.0

0.5

1.0

State

1

0.0

0.5

1.0

t

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

State

2

0.0

0.5

1.0

t

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

State

1

0.0

0.5

1.0

t

15

10

5

0

5

10

15

20

State

2

Data

PySINDy

Homotopy

From Figure 21, we find that SINDy is able to fit the data to a modest accuracy. While this may seem weak compared to the result from our homotopy method, the accuracy of the SINDy prediction is likely to improve if the candidate dictionary is updated to better reflect the actual form of Equation (13). However, this amounts to supplying the algorithm with excessive amounts of prior information about the system, which is rarely available in practice. This is also constrasted with NeuralODE training with our homotopy method - which does not require any additional information about the system to produce the results above.

G.5 Changing the gradient calculation scheme

As we commented in Section 3, methods that focus on other aspects of NeuralODE training, such as alternate gradient calculation schemes, are fully compatible with our homotopy training method. To demonstrate this point, we used the symplectic-adjoint method [39] provided in the torch-symplectic-adjoint library as a substitute for the direct backpropagation used for gradient calculation in our experiments.

Comparing the above results to our benchmark results in Figure 4 and Table 5, we see that the overall results remain the same - our homotopy method is able to train the models effectively, in much smaller number of epochs.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for training NeuralODEs, based on synchronization and homotopy optimization. They show that the addition of the synchronization module can smooth the loss landscape, on which homotopy optimization can be applied to enhance training. The new training method achieves competitive results in convergence speed and interpolation and extrapolation accuracy when compared with other baseline methods, especially for long training data. In addition,they demonstrate the robustness of the method experimentally.\n\nSTRENGTHS:\n1. The method of homotopy optimization has been introduced into the training of various neural networks, but this paper combines the idea of homotopy optimization with synchronization, and proposes a method that is very suitable for training NeuralODE on dynamical system-related tasks.\n2. The new training method can effectively improve the efficiency of network training and has the potential to alleviate a series of problems caused by irregular loss landscape.\n3. Compared with other methods, this method has no restrictions on the model structure or parameters at all, so it is more general.\n4. The experiments are valid.\n\nWEAKNESSES:\n1. The explanations in this paper are all experimental and lack rigorous theoretical support. For example, whether it is possible to analyze if the hessian trace of loss can be strictly bounded by the traditional method after adding coupling items for homotopy optimization.\n2. The authors only test the proposed method in low-dimensional toy models.\n3. It is mentioned in the paper that there are many stabilized models that improve training by limiting the expressivity of the model, but these methods are not compared experimentally with this new method to verify the advantages of this new method.\n\nQUESTIONS:\n1. Is it possible to add some theoretical discussions on the smoothness of the loss landscape？\n2. Could you provide experimental results on more highly complex real-world datasets?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTraining neural ODE models on long sequences of data from a dynamical system is difficult. The authors argue empirically that this is due to a poorly conditioned loss landscape leading to difficulties in optimization. To rectify this difficulty the authors use tools from the literature on synchronization (which gives conditions for when two dynamical systems will converge based on an error correction coupling term. They provide a homotopy between the synchronized dynamical system and the original one and argue that the optimization dynamics will be nice on the sychronized system and progressively less nice along the homotopy to the original system. However by gradually training the parameters and moving along the homotopy the method “ratchets up the difficulty” in a way that makes the ultimate optimization problem easier. \n\nThey demonstrate the success of the system on a collection of commonly used dynamical systems taken from the neural ODE literature when compared to vanilla SGD and a multiple-shooting algorithm that handles the longer time horizons by breaking them up. The authors show that their method archives better accuracy, robustness, and convergence than the baselines on these problems.\n\nSTRENGTHS:\nThe work combines the idea of synchonization with the standard Neural ODE training to improve the performance on long-horizon data. Since “long-horizon” data is not that long when training Neural ODEs, this is a welcome development. The idea of synchronization is intuitive and theoretically justified as well as empirically demonstrated to be useful through the loss landscape Hessian. The empirical results are not broad but they are deep; the authors study many variations of the problem on a small number of dynamical systems (3) against a small number of baselines (2). However, the method performs well under noise, sparsity, and long data on these and with better test performance and more quickly.\n\nWEAKNESSES:\nThe theoretical explanation leaves a little bit lacking. I would love to know more about the properties of the dynamical system as it is more or less synchronized. If there were traces of such in an image I think it would be intuitive. It took me a while to work out that as \\lambda -> \\infty any setting of the parameters is optimal and I think it might be good to explicitly say / visualize that. Any more detailed theoretical statements about this method would also be welcomed.\n\nQUESTIONS:\nHow is K chosen? I couldn’t find it in the main paper and the idea is intuitive even if K=I but I’d like to know?\n\nWould this method work for control systems as well? Seems like the counterfactual nature of the problem could be interesting.\n\nLIMITATIONS:\nThis seems pretty general and seems to widely apply to the breadth of neural ODE architectures and dynamical systems.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new method for training Neural Ordinary Differential Equations (NeuralODEs), a modeling approach that merges neural networks with the paradigm of differential equations from physical sciences. While NeuralODEs offer significant potential for extracting dynamic laws from time series data, they often suffer from long training times and subpar results. In this paper, the authors propose a training method built on synchronization and homotopy optimization. Their results demonstrate that the new method achieves competitive or even better training loss, often requiring fewer epochs compared to traditional model-agnostic techniques, and also exhibits better extrapolation capabilities.\n\nSTRENGTHS:\n1. The paper addresses a notable challenge in the field of NeuralODEs: the long training times and subpar results. Their proposed solution is both innovative and logical, introducing a novel concept in the NeuralODE literature.\n2. The authors demonstrate the advantages of their method convincingly, showing that it improves the training loss and requires fewer epochs for training.\n\nWEAKNESSES:\n1. The method's extension to high-dimensional systems or partially observed states is yet to be explored, limiting its current applicability.\n2. The paper could have benefited from more real-world application examples, demonstrating the utility of the method beyond the context of benchmark experiments.\n\nQUESTIONS:\n1. How does the efficacy of homotopy and multiple-shooting methods change with the size of the NeuralODE model? Is it possible to find a balance between model size and training method for optimal performance?\n2. What are the effects of data sparsity and noise on the performance of different training methods for NeuralODEs, particularly the homotopy training method?\n3. How can this methodology be extended to high-dimensional systems or systems with partially observed states?\n4. What are the practical implications and potential applications of this training method in various industries or scientific disciplines?\n\nLIMITATIONS:\nThe paper mainly uses synthetic datasets for testing the performance of training algorithms. The findings might not generalize to real-world datasets, which can exhibit unique complexities and noise structures.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a homotopy-based training method for NeuralODE models, in particular for the case with cases with long sequence of training data.  Comprehensive experimental results demonstrated the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The proposed method is based on homotopy method, which is a classic global optimization method, with sound theoretical foundation.\n2. Comprehensive experimental results. Good results when compared with the baseline methods\n3. The paper explains key concepts and results well.\n\nWEAKNESSES:\n1.  The argument on the loss function landscape lacks theoretical insights. The connection is established by empirical results. \n2.  Although the overall idea of using homotopy is good, some details are not properly addressed. See the questions and limitation part for more details. \n3. A few key references related to homotopy methods are not cited. The references by JH He really put homotopy methods in the mainstream of scientific computing: https://scholar.google.com/citations?user=tzM7c2cAAAAJ&hl=en\n\nQUESTIONS:\n1. I question the statement around Line 196, on the usage of cubic splines and its potential consequences. Cubic spline is a relatively high-order method that may be inaccurate when the underlying dynamic system is **stiff** (or there are widely spread time constants). Will the cubic spline interpolation be able to handle responses from _stiff_ systems? \n\n2. I question the convergence of the homotopy method when constant $\\Delta \\lambda$ is used. The introduction of homotopy to the original ODE effectively increases the order of the ODE by 1, and transform it into a DAE.  The authors should address whether there is the potential of non-convergence, and when it happens, how to address it.\n\nLIMITATIONS:\nNo negative societal impact perceived.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new training method for Neural Ordinary Differential Equations (NeuralODEs) that aims to improve their performance in extracting dynamical laws from time series data. The proposed method is based on synchronization and homotopy optimization, which does not require changes to the model architecture. The authors demonstrate that their method achieves competitive or better training loss while often requiring less than half the number of training epochs compared to other techniques. Furthermore, models trained with their method display better extrapolation capabilities.\n\nSTRENGTHS:\n[+] The paper is well-organized and clearly written, which makes it easy to follow and understand the proposed method. The introduction to the relevant background is sufficient, and the presentation of the experimental part is well-structured.\n\n[+] The proposed method is well motivated, and some visualizations play a significant role in understanding the problems with traditional methods and the performance of the newly proposed method.\n\n[+] The newly proposed method is very easy to integrate with models. In fact, the new method does not change the model but only changes the training process. It is simple to use and has excellent results.\n\n[-] Considering that there are five main hyperparameters, if there are targeted sensitivity experiments in the experiments to demonstrate how the selection of hyperparameters affects the experimental results to some extent, the experimental results might be more convincing.\n\n[-] The few systems involved in the experimental part are low-dimensional ODE systems. Can the proposed algorithm be applied to higher-dimensional problems? What new challenges might be encountered?\n\n[?] What is the relationship between the K in the algorithm framework and the hyperparameters: Coupling strength (k)?\n\n[?] It would be better if there is an analysis of computational complexity or a comparison of computation time with the baseline algorithm.\n\nWEAKNESSES:\nsee above\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}