SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rotating MNIST For the data generation we base our code upon the image rotation implementation provided by [Solin et al., 2021]. We set the total number of rotation angles to  $T = 16$  and sample the initial rotation angle from all the possible angles  $\theta^n \sim \mathbf{U}[1, T]$ . We generate  $N = 1000$  training trajectories with length  $T = 16$  and  $N = 100$  validation and test trajectories with length  $T = 45$ .
During training we set the batch size to 25 and learning rate to 0.002. We train all models for 400 epochs and for evaluation use the trained model with the lowest validation mse.
Bouncing ball with friction For data generation, we use the script provided by Sutskever et al. [2008]. As a tiny update, each observed sequence has a friction constant drawn from a uniform distribution  $\mathbb{U}[0,0.1]$ . We set the initial velocities to be unit vectors with random directions.
Table 6: Details of the datasets. Below,  $N$  denotes the number of sequences (for training, validation, and test),  ${N}_{t}$  is the sequence length,  ${\Delta t}$  denotes the time between two observations,and  $\sigma$  is the standard deviation of the added noise.
DATASET	N	Nval	Ntest	T	Tval	Testt	Δt	η
SINUSOIDAL DATA	300	50	50	50	50	150	0.1	0.2
LOTKA-VOLTERRA	600	100	100	100	100	300	0.1	0.3
ROTATING MNINST	2000	100	100	16	45	45	0.1	0.0
BOUNCING BALL	1000	100	100	20	20	40	0.1	0
Motion capture The walking sequences we consider are 07_01.amc, 08_03.amc, 35_02.amc, 35_12.amc, 35_34.amc, 39_08.amc, 07_02.amc, 16_15.amc, 35_03.amc, 35_13.amc, 38_01.amc, 39_09.amc, 07_03.amc, 16_16.amc, 35_04.amc, 35_14.amc, 38_02.amc, 39_10.amc, 07_06.amc, 16_21.amc, 35_05.amc, 35_15.amc, 39_01.amc, 39_12.amc, 07_07.amc, 16_22.amc, 35_06.amc, 35_28.amc, 39_02.amc, 39_13.amc, 07_08.amc, 16_31.amc, 35_07.amc, 35_29.amc, 39_03.amc, 39_14.amc, 07_09.amc, 16_32.amc, 35_08.amc, 35_30.amc, 39_04.amc, 07_10.amc, 16_47.amc, 35_09.amc, 35_31.amc, 39_05.amc, 07_11.amc, 16_58.amc, 35_10.amc, 35_32.amc, 39_06.amc, 08_02.amc, 35_01.amc, 35_11.amc, 35_33.amc, 39_07.amc. The number of training, validation and test samples in MOCAP and MOCAP-SHIFT splits are 46-5-5 and 43-5-8, respectively. Since sequences are already dense, we skip every other data point. For ease of implementation, we take the last 300 time points, leading to sequences of length  $T = 150$ . We experiment with different latent dimensionalities and report the findings in Table 12.
C Architecture and Hyperparameter Details
Table 7: Experiment parameters.  $T_{z}$  is the number of input frames to condition the initial state on, and  $T_{e}$  is the number of input frames used to compute the modulator/content variables. Further, we note the dimensionality of the dynamics state  $q_{z}$  (output of the ODESolve, where the differential equation is modeled by an MLP), as well as the latent dimensionality of the dynamics modulator  $q_{d}$  and the static  $q_{s}$ . lr stands for learning rate.
DATASET	MODEL	\(T_{z}\)	\(T_{e}\)	\(q_{z}\)	\(q_{d}\)	\(q_{s}\)	lr	N# parameters
SIN	NODE	10	-	8	-	-	0.002	24666
MonODE	3	10	4	4	-	0.002	24598
SONODE	10	-	2	-	-	0.002	21802
MoSONODE	10	10	2	4	-	0.002	23346
HBNODE	10	-	8	-	-	0.002	24404
MoHBNODE	10	10	4	4	-	0.002	24938
PREY-PREDATOR	NODE	40	-	16	-	-	0.002	28022
MonODE	8	40	8	8	-	0.002	26976
SONODE	40	-	4	-	-	0.002	29204
MoSONODE	40	40	4	8	-	0.002	31382
HBNODE	40	-	16	-	-	0.002	26586
MoHBNODE	10	40	8	8	-	0.002	26744
ROTATING MNIST	NODE	5	-	32	-	-	0.001	513561
MonODE	5	15	16	16	-	0.001	558681
MOCAP	NODE	75	-	24	-	-	0.002	45112
MonODE	75	75	8	8	8	0.002	51360
MOCAP-SHIFT	NODE	75	-	24	-	-	0.002	45112
MonODE	75	75	8	8	8	0.002	51360
BOUNCING BALL	NODE	5	-	212	-	-	0.001	162343
MonODE	5	5	8	4	-	0.001	150479
Table 8: Model architecture overview
Dataset	Model	Position Encoder	Velocity Encoder	Modulator P. Network	Differential Function	ODE Solver	Decoder
SIN	NODE	RNN	-	-	MLP	rk4	MLP
PREY-PREDATOR	MONODE	RNN	-	RNN	MLP	rk4	MLP
MOCAP	SONODE	-	MLP	-	MLP	rk4	-
MOCAP-SHIFT	MoSONODE	-	MLP	RNN	MLP	rk4	-
ROTATING MNIST	NODE	CNN	-	-	MLP	rk4	CNN
MonODE	CNN	-	RNN	MLP	rk4	CNN
BOUNCING BALL	NODE	CNN	CNN	-	MLP	rk4	CNN
MonODE	CNN	CNN	CNN	MLP	rk4	CNN
Figure 10: Model training time comparison on sinusoidal data. x-axis validation MSE, y-axis wall clock time in seconds. Thick line mean value across 3 different runs (different initialization seeds), shaded area standard deviation.
Table 9: Test MSE across different solvers for sinusoidal data for NODE Chen et al. [2018] and MoNODE (ours).
model	seq. length	euler	rk4	dopri5
NODE	T=50	0.13(0.03)	0.13 (0.03)	0.19(0.09)
MoNODE	0.07(0.02)	0.04 (0.01)	0.05(0.01)
NODE	T=150	2.48(1.26)	1.84 (0.70)	2.11(0.56)
MoNODE	0.42(0.21)	0.29 (0.11)	0.31(0.04)
Table 10: Ablation results for different model architectures and hyperparameters for Sinusoidal and Predator-Prey data. For final model parameters used see section app. C
Data	Model	Velocity Encoder	\(T_{z}\)	\(T_{e}\)	\(N_{incr}\)	Test MSE (std)
T=50	T=150
SINUSOIDAL DATA	MonODE	-	3	10	10	0.04 (0.01)	0.29 (0.11)
-	10	10	10	0.05 (0.01)	0.37 (0.12)
SONODE	MLP	5	-	10	2.18 (0.06)	3.05 (0.02)
MLP	10	-	4	1.81 (0.01)	3.05 (0.07)
RNN	10	-	10	2.19 (0.15)	3.05 (0.07)
MoSONODE	MLP	5	10	10	0.04 (0.00)	0.29 (0.04)
MLP	10	10	4	0.04 (0.00)	0.32 (0.06)
RNN	10	10	10	0.05 (0.01)	0.35 (0.10)
T=100	T=300
PREDATOR-PREY	MonODE	-	8	40	10	0.74 (0.05)	4.33 (0.19)
-	40	40	10	0.57 (0.04)	23.05 (1.49)
SONODE	MLP	10	-	10	15.801 (0.748)	40.921 (0.816)
MLP	40	-	2	5.130 (0.221)	39.260 (3.367)
RNN	40	-	10	5.101(0.339)	41.890 (8.444)
MoSONODE	MLP	10	40	10	1.459 (0.284)	6.695 (0.828)
MLP	40	40	2	1.093 (0.059)	6.342 (0.982)
RNN	40	40	10	1.294 (0.322)	8.639 (1.468)
HBNODE	RNN	40	-	10	0.879 (0.096)	3346.625 (2119.239)
RNN	10	-	10	13.803 (0.138)	2833.051 (3254.703)
MoHBNODE	RNN	40	40	10	0.870 (0.088)	225.012 (274.485)
RNN	10	40	10	0.943 (0.092)	10.205 (1.429)
Figure 11: Qualitative performance improvement by MoSONODE on SONODE on sinusoidal data. Both models have the same architecture: MLP Velocity encoder,  $T_{z} = 5$ , and  $N_{\mathrm{incr}} = 4$ . Lined area represents number of input frames used to compute initial position and velocity, while grey area represent number of frames seen during training.
Figure 12: The left panel illustrates PCA embeddings of the dynamics state inferred by HBNODE and MoHBNODE, where each color matches one trajectory. The figures on right compare HBNODE and MoHBNODE reconstructions on test trajectory sequences. Note that the models take the dashed area as input, and darker area represents the trajectory length seen during training.
Figure 13: The left panel illustrates PCA embeddings of the dynamics state inferred by NODE and MoNODE, where each color matches one trajectory. The figures on right compare NODE and MoNODE reconstructions on test trajectory sequences. Note that the models take the dashed area as input, and darker area represents the trajectory length seen during training. Prey-Predator data.
Figure 14: Qualitative performance improvement by MoSONODE on SONODE on Predator-Prey data. Both models have the same architecture: MLP Velocity encoder,  $T_{z} = 40$ , and  $N_{\mathrm{incr}} = 2$ . Lined area represents number of input frames used to compute initial position and velocity, while grey area represent number of frames seen during training.
Figure 15: NODE and MoNODE predictions on five bouncing ball sequences. For each couple of plots, the first and second rows correspond to NODE and MoNODE predictions, respectively. The dark circles are the ground truth ball positions and the colored background is the model prediction. As can be seen, MoNODE tracks the ball much better compared to NODE baseline.
(a) Rotating MNIST reconstruction for NODE
(b) Rotating MNIST reconstruction for MoNODE
(c) PCA embeddings NODE
(d) PCA embeddings MoNODE
Figure 16: Top two figures are (a) NODE and (b) MoNODE reconstruction of 3 different test trajectories, where top line is the ground truth for every digit. Bottom row illustrates the corresponding PCA embeddings of each trajectory plotted above, where (c) corresponds to NODE and (d) corresponds to MoNODE.
Figure 17: (left) TSNE embeddings of the static modulator as inferred by MoNODE on test data for roating MNIST. Each color represents one of the possible 10 digits. For digits that have a distinct shape, such as "0" and "1", we can discern a clear clustering and separation from the other digits, while for the remaining digits the per digit clusters exhibit more overlap. This is to be expected as we do not constraint the model to learn spatially separated clusters, as well as some digits are more similar either due to their appearance when rotated, such as "6" and rotated "9", or due to their writing style. (right) For further illustration, we repeat the same experiment with dynamics modulators instead of static. Unlike static modulator latents, for dynamics modulator no topology in the latent space is not present. In all, the results confirm that the modulator variables are correlated with the true underlying factors of variation
(d) PCA embeddings MoNODE
Figure 18: Failure cases on Rotating MNIST for (a) NODE and (b) MoNODE reconstruction of 2 different test trajectories, where top line is the ground truth for every digit. Bottom row illustrates the corresponding PCA embeddings of each trajectory plotted above, where (c) corresponds to NODE and (d) corresponds to MoNODE. Despite MoNODE failing to reconstruct the correct shape of the digit the latent PCA embedding plots reveal that the model has still captured the circular rotation motion.
Table 11: Ablation studies on Rotating MNIST data to compare different parameter affects on NODE and MoNODE. As it can be noted the test MSE for reconstruction within training sequence length  $T = 15$  is lower for NODE than for MoNODE, while for forecasting MoNODE outperforms NODE. This discrepancy can be explained by the fact that the current results indicate that MoNODE is harder to optimize as it's test msec can be grouped in two modes: success cases were the reconstructions are great and the error remains low for forecasting 16 or failure cases where the model fails to reconstruct already within the training sequence length  $T = 15$  as in fig 18. This explains the higher test MSE for MoNODE at  $T = 15$  and it remaining almost unchanged for  $T = 45$ , while for NODE the test MSE grows as the sequence length increases.
model	qz	qs	#N parameters	Test MSE (std)
T=15	T=45
NODE	16	-	461161	0.020	0.098
MoNODE	6	10	513637	0.031	0.034
8	8	516089	0.031	0.032
NODE	24	-	487361	0.014	0.062
MoNODE	8	16	532481	0.030	0.031
12	12	537385	0.035	0.038
NODE	32	-	513561	0.013	0.042
MoNODE	8	24	548873	0.035	0.037
12	20	553777	0.033	0.033
16	16	558681	0.031	0.032
Table 12: Ablation studies on mocap data to compare NODE and different variants of our model (with/without dynamics modulator and content variable). We also experiment with different latent dimensionalities.
| qz |	qd | qs |	MOCAP | MOCAP-SHIFT
NODE	6	-	61.6 ± 9.1
12	-	71.2 ± 20.0
24	-	72.2 ± 12.4
MoNODE	3	3	76.4 ± 12.7
6	6	79.7 ± 14.4
12	12	66.7 ± 12.3
3	-	72.6 ± 7.9
6	-	64.6 ± 9.3
12	-	60.0 ± 17.6
2	2	79.2 ± 8.7
4	4	78.8 ± 16.7
8	8	57.7 ± 9.8