SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we briefly compare our algorithm to a more traditional symbolic regression-based method. We choose the well-known SINDy algorithm, which is readily available in the pysindy package. This algorithm takes as input, time series measurements of the states of the dynamical system in question, as well as a dictionary of candidate terms that could constitute the governing equation. Afterwards, the time derivative of the states is estimated numerically, and a sparse regression is performed to determine which terms exist in the governing equation, as well as what their coefficient values are.
Compared to neural network-based approaches, the unique feature of SINDy is that its results are given in the form of analytical expressions. Therefore if the governing equation for the data is simple (e.g. consists of low-order polynomial terms), or if the user has sufficient prior information about the system and can construct a very small dictionary that contains all the terms that appear in the true equation, SINDy can accurately recover the ground truth equation.
On the other hand, if the system is high dimensional so that the space of possible candidate terms start to grow drastically, or if the governing equation has a arbitrary, complex form that cannot be easily guessed, SINDy either does not fit the data properly, or even if it does, recovers inaccurate equations that are difficult to assign meaning to. These correspond to situations where NeuralODEs, and hence our method for effectively training them, shines.
To illustrate this point, we choose the double pendulum dataset and compare the predicted trajectories from SINDy and our homotopy algorithm. As one can see from Equation (13), the governing equation of this system has an extremely complicated form that cannot be guessed easily. To reflect this difficulty, we chose the basis set of the dictionary of the candidate terms to be the state variable themselves, their sines and cosines, and the higher harmonics of the sines and cosines.
Figure 21: Comparison between our homotopy method and SINDy on the double pendulum dataset.
From Figure 21, we find that SINDy is able to fit the data to a modest accuracy. While this may seem weak compared to the result from our homotopy method, the accuracy of the SINDy prediction is likely to improve if the candidate dictionary is updated to better reflect the actual form of Equation (13). However, this amounts to supplying the algorithm with excessive amounts of prior information about the system, which is rarely available in practice. This is also constrained with NeuralODE training with our homotopy method - which does not require any additional information about the system to produce the results above.
G.5 Changing the gradient calculation scheme
As we commented in Section 3, methods that focus on other aspects of NeuralODE training, such as alternate gradient calculation schemes, are fully compatible with our homotopy training method. To demonstrate this point, we used the symplectic-adjoint method [39] provided in the torch-symplectic-adjoint library as a substitute for the direct backpropagation used for gradient calculation in our experiments.
Table 6: Benchmark results with the gradient calculation scheme changed to the symplectic-adjoint method.
Dataset	Lotka-Volterra	Double Pendulum	Lorenz System
Model Type	Black Box	Gray Box	Black Box	Second Order	Black Box
Best train epochs
Baseline	(3999, 3999, 3853)	(3997, 3999, 3992)	(3909, 3384, 3998)	(3983, 3998, 3972)	(3994, 3999, 3990)
Multi-. Shoot.	(3982, 3968, 3670)	(3901, 3958, 3996)	(3805, 3913, 3820)	(3988, 3970, 3942)	(3977, 3418, 3954)
Homotopy	(299, 208, 228)	(598, 265, 291)	(1201, 1799, 1774)	(900, 1799, 900)	(1799, 918, 1799)
Mean Squared Error (×10-2)
Baseline (interp.)	2.47±0.13	187±133	1.69±0.20	1.02±0.12	184±60.1
Multi-. Shoot.(interp.)	1.20±0.02	0.95±0.02	0.76±0.01	0.61±0.02	29.1±15.8
Homotopy (interp.)	0.77±0.04	1.68±0.76	1.08±0.12	0.25±0.03	33.3±5.43
Comparing the above results to our benchmark results in Figure 4 and Table 5, we see that the overall results remain the same - our homotopy method is able to train the models effectively, in much smaller number of epochs.