SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: To show our framework applicability to different NODE variants, here we illustrate HBNODE and MoHBNODE performance on PP data. The left panel illustrates PCA embeddings of the latent ODE state inferred by HBNODE and MoHBNODE, where each color matches one trajectory. The figures on right compare HBNODE and MoHBNODE reconstructions on test trajectory sequences. Note that the models take the dashed area as input, and darker area represents the trajectory length seen during training. predator-prey data.
where  $\mathbb{U}[\cdot, \cdot]$  denotes a uniform distribution. Validation and test data consist of  $N_{\mathrm{val}} = N_{\mathrm{test}} = 50$  trajectories with sequence length  $T_{\mathrm{val}} = 50$  and  $T_{\mathrm{test}} = 150$ , respectively. We add noise to the data following the implementation of [Rubanova et al., 2019].
The obtained test MSE demonstrates that our framework improves all aforementioned methods, namely, NODE, SONODE, and the state-of-the-art HBNODE (see Fig. 2, Fig. 11, and Fig. 12). In particular, our framework improves generalization to new dynamics and far-horizon forecasting as reported in Table 2 columns 2 and 3. In addition, modulating the motion via the dynamics modulator variable leads to interpretable latent ODE state trajectories  $\mathbf{z}(t)$ , see Fig. 2. More specifically, the obtained latent trajectories  $\mathbf{z}(t)$  have qualitatively a comparable radius topology to the observed amplitudes in the data space. By contrast, the latent space of NODE does not have such structure. Similar results are obtained for MoHBNODE (App. D, Fig. 12).
Training and inference times To showcase that our proposed framework is easy to train we have plotted the validation MSE versus wall clock time during training for sinusoidal data, please see App. Fig. 10. As it is apparent from the figure, our framework is easier to train than all baseline methods. We further compute the inference time cost for the sin data experiment, where the test data consists of 50 trajectories of length 150. We record the time it takes NODE and MoNODE to predict future states while conditioned on the initial 10 time points. Repeating the experiment ten times, the inference time cost for NODE is  $0.312 \pm 0.050$  while for MoNODE is  $0.291 \pm 0.040$ .
5.1.2 Predator-prey (PP) data
Next, we test our framework on the predator-prey benchmark [Rubanova et al., 2019, Norcliffe et al., 2021], governed by a pair of first-order nonlinear differential equations (also known as Lotka-Volterra
system, Eq. 28). The training data consists of  $N = 600$  trajectories of length  $T = 100$ . For every trajectory the four parameters of the differential equation are sampled, therefore each trajectory specifies a different interaction between the two populations. Validation and test data consist of  $N_{\mathrm{val}} = N_{\mathrm{test}} = 100$  trajectories with sequence length  $T_{\mathrm{val}} = 100$  and  $T_{\mathrm{test}} = 300$ . Similarly to sinusoidal data, we add noise to the data following [Rubanova et al., 2019].
The results show that our modulator framework improves the test accuracy of the existing methods for both, generalization to new dynamics as well as for forecasting, see Table 2. Moreover, examining the latent ODE state embeddings reveals that our framework results in more interpretable latent space embeddings also for PP data, see Fig. 3 for HBNODE and App. D Fig. 13 for NODE. In particular, the latent space of MoHBNODE and MoNODE captured the same amplitude relationship across trajectories as in observation space. For visualisation of the SONODE, MoSONODE, NODE, and MoNODE trajectories see App. D, Fig. 14, Fig. 13.
5.1.3 Bouncing ball (BB) with friction
To investigate the performance of our framework on video sequences we test it on a bouncing ball dataset, a benchmark often used in temporal generative modeling [Sutskever et al., 2008, Gan et al., 2015, Yildiz et al., 2019]. For data generation, we modify the original implementation of Sutskever et al. [2008] by adding friction to every data trajectory, where friction is sampled from  $\mathbb{U}[0,0.1]$ . The friction slows down the ball by a constant factor and is to be inferred by the dynamics modulator. We use  $N = 1000$  training sequences of length  $T = 20$ , and  $N_{\mathrm{val}} = N_{\mathrm{test}} = 100$  validation and test trajectories with length  $T_{\mathrm{val}} = 20$  and  $T_{\mathrm{test}} = 40$ . Our framework improves predictive capability for video sequences as shown in Table 4. As visible in Fig .15, the standard NODE model fails to predict the position of the object at further time points, while MoNODE corrects this error. For the second-order variants, LSONODE and MoLSONODE, we again observe our framework improving MSE from 0.0181 to 0.014.
5.1.4 Informativeness metric
Next, we quantify how much information latent representations carry about the unknown factors of variation (FoVs) [Eastwood and Williams, 2018], which are the parameters of the sinusoidal, PP, and BB dynamics. As described in [Schott et al., 2021], we compute  $R^2$  scores by regressing from latent variables to FoVs. The regression inputs for MoNODE are the dynamics modulators  $\mathbf{d}$ , while for NODE the latent trajectories  $\mathbf{z}_{1:T}$ . Note that  $R^2 = 1$  corresponds to perfect regression and  $R^2 = 0$  indicates random guessing. Our framework ob
Table 3:  $R^2$  scores to predict the unknown FoVs from inferred latents. Higher is better.
NODE	MoNODE
Sine	0.90	0.99
PP	-1.35	0.39
BB	-0.29	0.58
tains better  $R^2$  scores on all benchmarks (Table 3), implying better generalization capability of our framework. Therefore, as stated in [Eastwood and Williams, 2018], our MoNODE is better capable of disentangling underlying factors of variations compared to NODE.
5.2 Static modulator variable
To investigate the benefit of the static modulator variable, we test our framework on Rotating MNIST dataset, where the dynamics are fixed and shared across all trajectories, however, the content varies. The goal of the modulator prediction network is to learn the static features of each trajectory and pass it as an input to the decoder network. The data is generated following the implementation by [Casale et al., 2018], where the total number of rotation angles is  $T = 16$ . We include all ten digits and the initial rotation angle is sampled from all possible angles  $\theta^n \sim \{1, \dots, 16\}$ . The training data consists of  $N = 1000$  trajectories with length  $T = 16$ , which corresponds to one cycle of rotation. Validation and test data consist of  $N_{\mathrm{val}} = N_{\mathrm{test}} = 100$  trajectories with sequence length  $T_{\mathrm{val}} = T_{\mathrm{test}} = 45$ . At test time, the model receives the first  $T_z$  time frames as input and predicts the full horizon  $(T = 45)$ . We repeat each experiment 3 times and report the mean and standard deviation of the MSEs computed on the forecasting horizon (from  $T = 15$  to  $T = 45$ ). For further training details see App.B, while for a complete overview of hyperparameter see App. C Table 7.
The obtained test MSE confirms that the static modulator variable improves forecasting quality  $(T = 45)$ , see Table 4 and App. D fig. 16 for qualitative comparison. In addition, the latent ODE
Figure 4: PCA embeddings of the latent ODE state for Rotating MNIST dataset as inferred by NODE (top) and MoNODE (bottom). We generate 16 trajectories from a single digit where the initial angle of every trajectory is incremented by  $24^{\circ}$  degrees, starting from  $0^{\circ}$  until  $360^{\circ}$  degrees. Circle: the start of the trajectory (the initial angle), line: ODE trajectory. The color gradient corresponds to the initial angle of the trajectory in observation space.
Table 4: A Comparison of NODE and MoNODE test MSEs and standard deviations on BOUNCING BALL, ROT.MNIST, and MOCAP datasets.
BOUNCING BALL	ROT.MNIST	MOCAP	MOCAP-SHIFT
NODE	0.0199(0.001)	0.039 (0.003)	72.2(12.4)	61.6(6.2)
MoNODE	0.0164(0.001)	0.030 (0.001)	57.7(9.8)	58.0(10.7)
states of MoNODE form a circular rotation pattern resembling the observed dynamics in data space while for NODE no correspondence is observed, see App. D Fig. 16. Moreover, as shown in Fig. 4, the latent space of MoNODE captured the relative distances between the initial rotation angles, while NODE did not. The TSNE embeddings of the static modulator indicate a clustering per digit shape, see App. D Fig. 17. For additional results and discussion, see App. D.
5.3 Are modulators interchangeable?
To confirm the role of each modulator variable we have performed two additional ablations with the MoNODE framework on: (a) sinusoidal data with static modulator instead of dynamics, and (b) rotating MNIST with dynamics modulator instead of static. We report the test MSE across three different initialization runs with standard deviation. For sinusoidal data with MoNODE + static modulator, the test MSE performance drops to  $2.68 \pm 0.38$  from  $0.29 \pm 0.11$  (MoNODE + dynamic modulator). For rotating MNIST, MoNODE + dynamics modulator performance drops to  $0.096 \pm 0.008$  from  $0.030 \pm 0.001$  (MoNODE + static modulator). In addition, we examined the latent embeddings of the dynamics modulator for rotating MNIST. Where previously for the content modulator we observed clusters corresponding to a digit's class, for dynamics modulator such a topology in the latent space is not present (see App. D Fig. 17). Taken together with Table 3, the results confirm that the modulator variables are correlated with the true underlying factors of variation and play their corresponding roles.
5.4 Real world application: modulator variables
Next, we evaluate our framework on a subset of CMU Mocap dataset, which consists of 56 walking sequences from 6 different subjects. We pre-process the data as described in [Wang et al., 2007], resulting in 50-dimensional data sequences. We consider two data splits in which  $(i)$  the training and test subjects are the same, and  $(ii)$  one subject is reserved for testing. We refer to the datasets as MOCAP and MOCAP-SHIFT (see App. B for details). In test time, the model receives the first 75 observations as input and predicts the full horizon (150 time points). We repeat each experiment five times and report the mean and standard deviation of the MSEs computed on the full horizon. As shown in Table 4 and Fig. 5, our framework improves upon NODE. See Table 12 for ablations with different latent dimensionalities and with only one modulator variable (static or dynamics) present.
Figure 5: MoNODE and NODE reconstructions on a Mocap test sequence. Note that the models take the first  $T_{\mathrm{in}} = 75$  time points as input and predict for  $T = 150$  time steps.
The present work introduces a novel modulator framework for NODE models that allows to separate time-evolving ODE states from modulator variables. In particular, we introduce two types of modulating variables: (i) dynamics modulator that can modulate the dynamics function and (ii) static modulator that can modulate the observation mapping function. Our empirical results confirm that our framework improves generalization to new dynamics and far-horizon forecasting. Moreover, our modulator variables better capture the true unknown factors of variation as measured by  $R^2$  score, and, as a result, the latent ODE states have an equivalent correspondence to the true observations.
Limitations and future work The dynamical systems explored in the current work are limited to deterministic periodic systems that have different underlying factors of variation. The presented work introduces a framework that builds upon a base NODE model, hence, the performance of Mo*NODE is largely affected by the base model's performance. The current formulation cannot account for epistemic uncertainty and does not generalize to out-of-distribution modulators, because we maintain point estimates for the modulators. A straightforward extension would be to apply our framework to Gaussian process-based ODEs [Hegde et al., 2022] stochastic dynamical systems via an auxiliary variable that models the noise, similarly to Franceschi et al. [2020]. Likewise, rather than maintaining point estimates, the time-invariant modulator parameters could be inferred via marginal likelihood as in van der Wilk et al. [2018], Schwöbel et al. [2022], leading to a theoretically grounded Bayesian framework. Lastly, the separation of the dynamic factors and modulating factors could be explicitly enforced via an additional self-supervised contrasting loss term [Grill et al., 2020] or by more recent advances in representation learning [Bengio et al., 2013]. Lastly, the concept of dynamics modulator could also be extended to object-centric dynamical modeling [Kabra et al., 2021], which would allow accounting for per-object specific dynamics modulations while using a single dynamic model.
Acknowledgements The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217. Căgatay Yildiz funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC-Number 2064/1 - Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 950086).
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 2017.
F. P. Casale, A. Dalca, L. Saglietti, J. Listgarten, and N. Fusi. Gaussian process prior variational autoencoders. Advances in neural information processing systems, 31, 2018.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07532, 2019.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
Y. Rubanova, R. T. Chen, and D. K. Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
L. Schott, J. Von Kugelgen, F. Trauble, P. Gehler, C. Russell, M. Bethge, B. Scholkopf, F. Locatello, and W. Brendel. Visual representation learning does not generalize strongly within the same domain. arXiv preprint arXiv:2107.08221, 2021.
P. Schwöbel, M. Jørgensen, S. W. Ober, and M. Van Der Wilk. Last layer marginal likelihood for invariance learning. In International Conference on Artificial Intelligence and Statistics, pages 3542-3555. PMLR, 2022.
A. Solin, E. Tamir, and P. Verma. Scalable inference in sdes by direct matching of the fokkerplanck-kolmogorov equation. Advances in Neural Information Processing Systems, 34:417-429, 2021.
I. Sutskever, G. E. Hinton, and G. W. Taylor. The recurrent temporal restricted boltzmann machine. Advances in neural information processing systems, 21, 2008.
M. Tenenbaum and H. Pollard. Ordinary differential equations: an elementary textbook for students of mathematics, engineering, and the sciences. Courier Corporation, 1985.
M. van der Wilk, M. Bauer, S. John, and J. Hensman. Learning invariances using the marginal likelihood. Advances in Neural Information Processing Systems, 31, 2018.
J. M. Wang, D. J. Fleet, and A. Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence, 30(2):283-298, 2007.
H. Xia, V. Suliafu, H. Ji, T. Nguyen, A. Bertozzi, S. Osher, and B. Wang. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems, 34:18646-18659, 2021.
C. Yildiz, M. Heinonen, and H. Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Y. D. Zhong, B. Dey, and A. Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint arXiv:1909.12077, 2019.
A Model Encoder Adjustments
SONODE In the original implementation by [Norcliffe et al., 2020] the initial velocity is computed by a 3-layer MLP with ELU activation functions and only the initial time point,  $T_{z} = 1$ , is passed as an input. For the initial position the first observation time point is passed, e.g.  $\mathbf{x}_0$ . The resulting model fails to fit on sinusoidal data, see fig. 6 for performance on validation trajectories.
Figure 6: Original SONODE [Norcliffe et al., 2020] performance on sine validation sequences with  $T_{z} = 1$ . Green dots is ground truth, blue line, model's prediction.
As it can be seen in figure 6 the model only fits the first data point correctly, therefore, we extend the number of input time points passed to the model to  $T_{z} = 5$  to compute the initial velocity. This results in notable performance improvements, see Fig. 7.
Figure 7: SONODE [Norcliffe et al., 2020] performance on sine validation sequences with  $T_{z} = 5$ . Green dots is ground truth, blue line, model's prediction.
Consequently, we increase the number of input frames used for SONODE, as well as investigate replacing the MLP architecture with a RNN. We report the resulting test MSEs in App. D Table 10. For the parameters used in the experiments please see Table 7.
HBNODE In the original implementation by [Xia et al., 2021] the encoder is a 3-layer MLP with TANH activation functions that autoregressively takes every ground truth data point as input, e.g.  $\mathbf{x}_i$  for  $i = [0,\dots ,T]$  and predicts the latent representation  $\mathbf{z}_i$  which is also fed as input to the proceeding time points  $i > 1$ . The differential function is modeled by linear layer projection. As the present work is focused on model forecasting capabilities we adjust the original HBNODE to be compatible with problem set-ups where the ground truth data is not available. Meaning that once the model has reached the time point where there is no more ground truth data available ( $T > 50$ ), the model only uses it's own latent state predictions  $\mathbf{z}_i$  to compute the next latent state  $\mathbf{z}_{i + 1}$ . As the authors of HBNODE claim that the model is designed to better fit long-term dependencies we initially reduce the number of subsequent increments,  $N_{\mathrm{incr}}$  from 10 to 3 (see App. B for training details). In Fig. 8 we show the qualitative performance of HBNODE on validation trajectories. Even though the model fits perfectly the ground truth data within  $T < 50$ , it fails to extrapolate to future time frames. We investigate the cause of this issue by, first, changing the differential function from a linear projection to a 3-layer MLP with TANH activations, and, second, by replacing the original autoregressive encoder with a RNN encoder that learns the initial position  $\mathbf{z}_0$ .
Figure 8: HBNODE [Xia et al., 2021] performance on sine validation sequences. Green dots is ground truth, blue line, model's prediction.
Table 5: Ablation results for HBNODE with RNN encoder,  $T_{z} = 10$ , with different  $N_{\mathrm{incr}}$  on sine data.
\( N_{incr} \)	Validation MSE
1	3	5	10
HBNODE	0.164	0.135	0.144	0.066
Increasing the modelling capacity of the differential function did not resolve the observed issue with forecasting. This implies that the observed over-fitting of the model might be caused due to its autoregressive encoder that takes every ground truth time point as input. By replacing the autoregressive encoder with an RNN encoder we obtain significant forecasting improvements, see Fig. 9. Therefore, in all our reported experiments we replace the autoregressive encoder with an RNN and performed additional tests on the number of data increments  $N_{\mathrm{incr}} = [1,3,5,10]$  as well as the number of input frames  $Tiv$  to compute the initial state  $\mathbf{z}_0$ . We find in our experimentation that  $N_{\mathrm{incr}} = 10$  brings the best model performance (see Table 5) and subsequently test different  $T_z$  frames, 10 and 40, respectively. We report the obtained test MSE for each set-up across 3 model runs with different seeds in App. D Table 10 and for the final model parameter settings used in the experiments please see App. C.
Figure 9: HBNODE [Xia et al., 2021] with RNN encoder,  $N_{\mathrm{incr}} = 10$  and  $T_z = 10$  performance on sine validation sequences. Green dots is ground truth, blue line, model's prediction.
All models for sine, PP and rotating MNIST have been trained on a single GPU (GTX 1080 Ti) with 10 CPUs and 30G memory.
Sine data In the same vein as Rubanova et al. [2019], Norcliffe et al. [2021], we first define a set of time points  $T = \{0, 0.1, 0.2, \dots, t_i, \dots, 4.9\}$ . Then each sequence  $\mathbf{y}^n$  is generated as follows:
$$
a ^ {n} \sim \mathbb {U} [ 1, 3 ] \tag {15}
$$
$$
\omega^ {n} \sim \mathbb {U} [ 0. 5, 1. 0 ] \tag {16}
$$
$$
\phi \sim \mathbb {U} [ 0, 1. 0 ] \tag {17}
$$
$$
\epsilon^ {n} \sim \mathbb {U} [ 0., \eta ] \tag {18}
$$
$$
x _ {i} ^ {n} = a ^ {n} \sin \left(\omega^ {n} t _ {i} + \phi^ {n}\right) \tag {19}
$$
$$
y _ {i} ^ {n} = x _ {i} ^ {n} + \epsilon^ {n} \tag {20}
$$
where  $\mathbb{U}$  denotes the uniform distribution and  $\eta$  is set to  $\eta = 0.2$ . We use  $\Delta t = 0.1$  to generate  $N = 300$  training trajectories of length  $N_{\mathrm{test}} = 50$ ,  $N_{\mathrm{val}} = 50$  validation trajectories of length  $T_{\mathrm{val}} = 50$ , and  $N_{\mathrm{test}} = 50$  test trajectories of length  $T_{\mathrm{test}} = 150$ . Test trajectories are longer to test model forecasting accuracy.
During training we set batch size to 10. We incrementally increase the sequence length of training trajectories starting from  $T = 5$  until  $T = 50$ . We perform this in 10 increments for NODE, MoNODE, HBNODE and MoHBNODE, and in 4 increments for SONODE and MoSONODE. All models are trained for 600 epochs. The model with lowest validation MSE is used for evaluation on test trajectories.
Predator-Prey data To generate Predator-Prey trajectories we define the following generative process:
$$
\alpha^ {n} \sim \mathbb {U} [ 1., 3. 5 ] \tag {22}
$$
$$
\gamma^ {n} \sim \mathbb {U} [ 1.,. 3. 5 ] \tag {23}
$$
$$
\delta^ {n} \sim \mathbb {U} [ . 2,.. 3 ] \tag {24}
$$
$$
\beta^ {n} \sim \mathbb {U} [. 5, . 6 ] \tag {25}
$$
$$
\epsilon^ {n} \sim \mathbb {U} [ 0., \eta ] \tag {26}
$$
$$
\mathbf {x} _ {0} ^ {n} \sim \mathbb {U} [ 1., 5. ] \tag {27}
$$
$$
\mathbf {x} _ {i} ^ {n} = \mathbf {x} _ {0} ^ {n} + \int_ {0} ^ {t _ {i}} \left[ \begin{array}{l} \alpha^ {n} x _ {1} ^ {n} (\tau) - \beta^ {n} x _ {1} ^ {n} (\tau) x _ {2} ^ {n} (\tau) \\ \delta^ {n} x _ {1} ^ {n} (\tau) x _ {2} ^ {n} (\tau) - \gamma^ {n} x _ {2} ^ {n} (\tau) \end{array} \right] d \tau \tag {28}
$$
$$
\mathbf {y} _ {i} ^ {n} = \mathbf {x} _ {i} ^ {n} + \epsilon^ {n} \tag {29}
$$
where  $\mathbb{U}$  denotes the uniform distribution and  $\eta$  is set to  $\eta = 0.3$ . We use  $\Delta t = 0.1$  to generate  $N = 600$  training trajectories of length  $T = 100$ ,  $N_{\mathrm{val}} = 100$  validation trajectories of length  $T_{\mathrm{val}} = 100$ , and  $N_{\mathrm{test}} = 100$  test trajectories of length  $T_{\mathrm{test}} = 300$ . Test trajectories are longer to test model forecasting accuracy.
During training we set the batch size to 20. We incrementally increase the sequence length of training trajectories starting from  $T = 10$  until  $T = 100$ . We perform this in 10 increments for NODE, MoNODE, HBNODE and MoHBNODE, and in 2 increments for SONODE and MoSONODE. All models are trained for 1500 epochs. The model with lowest validation MSE is used for evaluation on test trajectories.