SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ilze Amanda Auzina
i.a. auzina@uva.nl
Cagatay Yildiz
Tübingen AI Center
Matthias Bethge
Neural ordinary differential equations (NODEs) have been proven useful for learning non-linear dynamics of arbitrary trajectories. However, current NODE methods capture variations across trajectories only via the initial state value or by autoregressive encoder updates. In this work, we introduce Modulated Neural ODEs (MoNODEs), a novel framework that sets apart dynamics states from underlying static factors of variation and improves the existing NODE methods. In particular, we introduce time-invariant modulator variables that are learned from the data. We incorporate our proposed framework into four existing NODE variants. We test MoNODE on oscillating systems, videos and human walking trajectories, where each trajectory has trajectory-specific modulation. Our framework consistently improves the existing model ability to generalize to new dynamic parameterizations and to perform far-horizon forecasting. In addition, we verify that the proposed modulator variables are informative of the true unknown factors of variation as measured by  $R^2$  scores.
Differential equations are the de facto standard for learning dynamics of biological [Hirsch et al., 2012] and physical [Tenenbaum and Pollard, 1985] systems. When the observed phenomenon is deterministic, the dynamics are typically expressed in terms of ordinary differential equations (ODEs). Traditionally, ODEs have been built from a mechanistic perspective, in which states of the observed system and the governing differential equation with its parameters are specified by domain experts. However, when the parametric form is unknown and only observations are available, neural network surrogates can be used to model the unknown differential function, called neural ODEs (NODEs) Chen et al. [2018]. Since its introduction by Chen et al. [2018], there has been an abundance of research that uses NODE type models for learning differential equation systems and extend it by introducing a recurrent neural networks (RNNs) encoder [Rubanova et al., 2019, Kanaa et al., 2021], a gated recurrent unit [De Brouwer et al., 2019, Park et al., 2021], or a second order system [Yildiz et al., 2019, Norcliffe et al., 2020, Xia et al., 2021].
Despite these recent developments, all of the above methods have a common limitation: any static differences in the observations can only be captured in a time-evolving ODE state. This modelling approach is not suitable when the observations contain underlying factors of variation that are fixed in time, yet can (i) affect the dynamics or (ii) affect the appearance of the observations. As a concrete example of this, consider human walking trajectories. The overall motion is shared across all subjects, e.g. walking. However, every subject might exhibit person-specific factors of variation, which in turn could either affect the motion exhibited, e.g. length of legs, or could be a characteristic of a given subject, e.g. color of a shirt. A modelling approach that is able to (i) distinguish the dynamic
Figure 1: Schematic illustration of our method. Standard continuous-time latent dynamical systems such as Chen et al. [2018] assume a single ODE state  $\mathbf{z}(t)$  transformed by a surrogate neural differential function. Our method augments the latent space with dynamics and static modulators to account for exogenous variables governing the dynamics and observation mapping (decoding).
(e.g. position, velocity) from the static variables (e.g. height, clothing color), and (ii) use the static variables to modulate the motion or appearance, is advantageous, because it leads to an improved generalization across people. As an empirical confirmation, we show experimentally that the existing NODE models fail to separate the dynamic factors from the static factors. This inevitably leads to overfitting, thus, negatively effecting the model generalization to new dynamics, as well as far-horizon forecasting.
As a remedy, this work introduces a modulated neural ODE (MoNODE) framework, which separates the dynamic variables from the time-invariant variables. We call these time-invariant variables modulator variables and we distinguish between two types: (i) static modulators that modulate the appearance; and (ii) dynamics modulators that modulate the time-evolution of the latent dynamical system (for a schematic overview see Fig. 1). In particular, MoNODE adds a modulator prediction network on top of a NODE, which allows to compute the modulator variables from data. We empirically confirm that our modular framework boosts existing NODE models by achieving improved future predictions and improved generalization to new dynamics. In addition, we verify that our modulator variables are more informative of the true unknown factors of variation by obtaining higher  $R^2$  scores than NODE. As a result, the latent ODE state of MoNODE has an equivalent sequence-to-sequence correspondence as the true observations. Our contributions are as follows:
- We extend neural ODE models by introducing modulator variables that allow the model to preserve core dynamics while being adaptive to modulating factors.
- Our modulator framework can easily be integrated into existing NODE models [Chen et al., 2018, Yildiz et al., 2019, Norcliffe et al., 2020, Xia et al., 2021].
- As we show in our experiments on sinusoidal waves, predator-prey dynamics, bouncing ball videos, rotating images, and motion capture data, our modulator framework consistently leads to improved long-term predictions as measured by lower test mean squared error (an average of  $55.25\%$  improvement across all experiments).
- Lastly, we verify that our modulator variables are more informative of the true unknown factors of variation than NODE, as we show in terms of  $R^2$  scores.
While introduced in the context of neural ODEs, we believe that the presented framework can benefit also stochastic and/or discrete dynamical systems, as the concept of dynamic or mapping modulation is innate for most dynamical systems. We conclude this work by discussing the implications of our method on learning object-centric representations and its potential Bayesian extensions. Our official implementation can be found at https://github.com/IlzeAmandaA/MoNODE.
We first introduce the basic concepts for Neural ODEs, following the notation by Chen et al. [2018].
Ordinary differential equations Multivariate ordinary differential equations are defined as
$$
\dot {\mathbf {z}} (t) := \frac {\mathrm {d} \mathbf {z} (t)}{\mathrm {d} t} = \mathbf {f} (t, \mathbf {z} (t)), \tag {1}
$$
where  $t \in \mathbb{R}_+$  denotes time, the vector  $\mathbf{z}(t) \in \mathbb{R}^{q_z}$  captures the state of the system at time  $t$ , and  $\dot{\mathbf{z}}(t) \in \mathbb{R}^{q_z}$  is the time derivative of the state  $\mathbf{z}(t)$ . In this work, we focus on autonomous ODE systems, implying a vector-valued (time) differential function  $\mathbf{f}: \mathbb{R}^{q_z} \mapsto \mathbb{R}^{q_z}$  that does not explicitly depend on time. The ODE state solution  $\mathbf{z}(t_2)$  is computed by integrating the differential function starting from an initial value  $\mathbf{z}(t_1)$ :
$$
\mathbf {z} \left(t _ {2}\right) = \mathbf {z} \left(t _ {1}\right) + \int_ {t _ {1}} ^ {t _ {2}} \mathbf {f} (\mathbf {z} (\tau)) \mathrm {d} \tau . \tag {2}
$$
For non-linear differential functions  $f$ , the integral does not have a closed form solution and hence is approximated by numerical solvers [Tenenbaum and Pollard, 1985]. Due to the deterministic nature of the differential function, the ODE state solution  $\mathbf{z}(t_2)$  is completely determined by the corresponding initial value  $\mathbf{z}(t_1)$  if the function  $\mathbf{f}$  is known.
2.1 Latent neural ODEs
Chen et al. [2018] proposed neural ODEs for modeling sequential data  $\mathbf{x}_{1:T} \in \mathbb{R}^{T \times D}$ , where  $\mathbf{x}_i \equiv \mathbf{x}(t_i)$  is the  $D$ -dimensional observation at time  $t_i$ , and  $T$  is the sequence length. We assume known observation times  $\{t_1, \ldots, t_T\}$ . Being a latent variable model, NODE infers a latent trajectory  $\mathbf{z}_{1:T} \in \mathbb{R}^{T \times q_z}$  for an input trajectory  $\mathbf{x}_{1:T}$ . The generative model relies on random initial values, their continuous-time transformations, and finally an observation mapping from latent to data space:
$$
\mathbf {z} _ {1} \sim p \left(\mathbf {z} _ {1}\right) \tag {3}
$$
$$
\mathbf {z} _ {i} = \mathbf {z} _ {1} + \int_ {t _ {1}} ^ {t _ {i}} \mathbf {f} _ {\theta} (\mathbf {z} (\tau)) \mathrm {d} \tau \tag {4}
$$
$$
\mathbf {x} _ {i} \sim p _ {\xi} \left(\mathbf {x} _ {i} \mid \mathbf {z} _ {i}\right) \tag {5}
$$
Here, the time differential  $\mathbf{f}_{\theta}$  is a neural network with parameters  $\theta$  (hence the name "neural ODEs"). Similar to variational auto-encoders [Kingma and Welling, 2013, Rezende et al., 2014], the "decoding" of the observations is performed by another non-linear neural network with a suitable architecture and parameters  $\xi$ .
3 MoNODE: Modulated Neural ODEs
We begin with a brief description of the dynamical systems of our interest. Without loss of generality, we consider a dataset of  $N$  trajectories with a fixed trajectory length  $T$ , where the  $n$ 'th trajectory is denoted by  $\mathbf{x}_{1:T}^{n}$ . First, we make the common assumption that the data trajectories are generated by a single dynamical system (e.g., a swinging pendulum) while the parameters that modulate the dynamics (e.g., pendulum length) vary across the trajectories. Second, we focus on the more general setup in which the observations and the dynamics might lie in different spaces, for example, video recordings of a pendulum. A future video prediction would require a mapping from the dynamics space to the observation space. As the recordings might exhibit different lighting conditions or backgrounds, the mapping typically involves static features that modulate the mappings, e.g., a parameter specifying the background.
3.1 Our generative model
The generative model of NODE [Chen et al., 2018] involves a latent initial value  $\mathbf{z}_1$  for each observed trajectory as well as a differential function and a decoder with global parameters  $\theta$  and  $\xi$ . Hence, by construction, NODE can attribute discrepancies across trajectories only to the initial value as the remaining functions are modeled globally. Subsequent works [Rubanova et al., 2019, Dupont et al., 2019, De Brouwer et al., 2019, Xia et al., 2021, Iakovlev et al., 2023] combine the latent space of NODE with additional variables, however, likewise, they do not account for static, trajectory-specific factors that modulate either the dynamics or the observation mapping. The main claim of this work is that explicitly modeling the above-mentioned modulating variables results in better extrapolation and
generalization abilities. To show this, we introduce the following generative model:
$$
\mathbf {d} \sim p (\mathbf {d}) \quad / / d y n a m i c s \text {m o d u l a t o r} \tag {6}
$$
$$
\mathbf {s} \sim p (\mathbf {s}) \quad / / \text {s t a t i c m o d u l a t o r} \tag {7}
$$
$$
\mathbf {z} _ {1} \sim p \left(\mathbf {z} _ {1}\right) \quad / / \text {l a t e n t O D E s t a t e} \tag {8}
$$
$$
\mathbf {z} _ {i} = \mathbf {z} _ {1} + \int_ {t _ {1}} ^ {t _ {i}} \mathbf {f} _ {\theta} (\mathbf {z} (\tau); \mathbf {d}) \mathrm {d} \tau \tag {9}
$$
$$
\mathbf {x} _ {i} \sim p _ {\xi} \left(\mathbf {x} _ {i} \mid \mathbf {z} _ {i}; \mathbf {s}\right). \tag {10}
$$
We broadly refer to  $\mathbf{d}$  and  $\mathbf{s}$  as dynamics and static modulators, and thus name our framework Modulated Neural ODE (MoNODE). We assume that each observed sequence  $\mathbf{x}_{1:T}^{n}$  has its own modulators  $\mathbf{d}^n$  and  $\mathbf{s}^n$ . As opposed to the ODE state  $\mathbf{z}(t)$ , the modulators are time-invariant.
We note that for simplicity, we describe our framework in the context of the initial neural ODE model [Chen et al., 2018]. However, our framework can be readily adapted to other neural ODE models such as second-order and heavy ball, NODEs, as we demonstrate experimentally. For a schematic overview, please see Fig. 1. Next, we discuss how to learn the modulator variables along with the global dynamics, encoder, and decoder.
3.2 Learning latent modulator variables
A straightforward approach to obtain time-invariant modulator variables is to define them globally and independently of each other and input sequences. While optimizing for such global variables works well in practice [Blei et al., 2017], it does not specify how to compute variables for an unobserved trajectory. As the focus of the present work is on improved generalization to unseen dynamic parametrizations, we estimate the modulator variables via amortized inference based on encoder networks  $\mathbf{g}_{\psi}$  and  $\mathbf{h}_v$ , which we detail in the following.
(i) Static modulator To learn the unknown static modulator  $\mathbf{s}^n\in \mathbb{R}^{q_s}$  that captures the time-invariant characteristics of the individual observations  $\mathbf{x}_i^n$  , we compute the average over the observation embeddings provided by a modulator prediction network  $\mathbf{g}_{\psi}(\cdot)$  (e.g., a convolutional neural network) with parameters  $\psi$  ..
$$
\mathbf {s} ^ {n} = \frac {1}{T} \sum_ {i = 1} ^ {T} \mathbf {s} _ {i} ^ {n}, \quad \mathbf {s} _ {i} ^ {n} = \mathbf {g} _ {\psi} \left(\mathbf {x} _ {i} ^ {n}\right). \tag {11}
$$
By construction,  $\mathbf{s}^n$  is time-invariant (or more rigorously, invariant to time-dependent effects) as we average over time. In turn, the decoder takes as input the concatenation of the latent ODE state  $\mathbf{z}_i^n$  and the static modulator  $\mathbf{s}^n$ , and maps the joint latent representation to the observation space (similarly to Franceschi et al. [2020]):
$$
\mathbf {x} _ {i} ^ {n} \sim p _ {\boldsymbol {\xi}} \left(\mathbf {x} _ {i} ^ {n} \mid \left[ \mathbf {z} _ {i} ^ {n}, \mathbf {s} ^ {n} \right]\right) \forall i \in [ 1, \dots , T ]. \tag {12}
$$
Note that the estimated static modulator  $\mathbf{s}^n$  is fed as input for all time points within a trajectory.
(ii) Dynamics modulator Unlike the static modulator, the dynamics modulator can only be deduced from multiple time points  $\mathbf{x}_{i + T_e}^n$ . For example, the dynamics of a pendulum depend on its length. To compute the length of the pendulum one must compute the acceleration for which multiple position and velocity measurements are needed. Thereby, the dynamics modulators,  $\mathrm{d}_i$ , are computed from subsequences of length  $T_{e}$  from a given trajectory. To achieve time-invariance we likewise average over time:
$$
\mathbf {d} ^ {n} = \frac {1}{T - T _ {e}} \sum_ {i = 1} ^ {T - T _ {e}} \mathbf {d} _ {i} ^ {n}, \quad \mathbf {d} _ {i} ^ {n} = \mathbf {h} _ {\boldsymbol {v}} \left(\mathbf {x} _ {i: i + T _ {e}} ^ {n}\right), \tag {13}
$$
where  $\mathbf{h}_{\pmb{\nu}}$  is a modulator prediction network (e.g., a recurrent neural network) with parameters  $\pmb{v}$ . The differential function takes as input the concatenation of the latent ODE state  $\mathbf{z}^n (\tau)\in \mathbb{R}^{q_z}$  and the estimated dynamics modulator  $\mathbf{d}^n\in \mathbb{R}^{q_d}$ . Consequently, we redefine the input space of the differential function  $\mathbf{f}_{\pmb{\theta}}:\mathbb{R}^{q_z + q_d}\mapsto \mathbb{R}^{q_z}$ , implying the following time differential:
$$
\frac {\mathrm {d} \mathbf {z} ^ {n} (t)}{\mathrm {d} t} = \mathbf {f} _ {\boldsymbol {\theta}} \left(\mathbf {z} ^ {n} (t), \mathbf {d} ^ {n}\right) \tag {14}
$$
Table 1: A taxonomy of the state-of-the-art ODE systems. We empirically demonstrate that our modulating variables idea is straightforward to apply to NODE, SONODE, and HBNODE, and consistently leads to improved prediction accuracy. Where with * we refer to: MoNODE, MoSONODE, MoLSONODE, MoHBNODE.
Model	Reference	Latent dynamics	Time invariant modulator	Temporal data
Neural ODE (NODE)	Chen et al. [2018]	✓	X	✓
Augmented NODE (ANODE)	Dupont et al. [2019]	X	X	X
Neural Controlled DE (NCDE)	Kidger et al. [2020]	✓	X	X
Second Order NODE (SONODE)	Norcliffe et al. [2020]	X	X	✓
Latent SONODE (LSONODE)	Yildiz et al. [2019]	✓	X	✓
NODE Processes (NODEP)	Norcliffe et al. [2021]	✓	X	✓
Heavy Ball NODE (HBNODE)	Xia et al. [2021]	X	X	✓
Modulated *NODE	this work	✓	✓	✓
The resulting ODE system resembles in a way the augmented neural ODE (ANODE) [Dupont et al., 2019]. However, their appended variable dimensions are constant and serve a practical purpose of breaking down the diffeomorphism constraints of NODE, while ours models time-invariant variables. We treat  $T_{e}$  as a hyperparameter and choose it by cross-validation.
**Optimization objective** The maximization objective of MoNODE is analogous to the evidence-lower bound (ELBO) as in [Chen et al., 2018] for NODE, where we place a prior distribution on the unknown latent initial value  $\mathbf{z}_1$  and approximate it by amortized inference. Similar to previous works [Chen et al., 2018, Yildiz et al., 2019], MoNODE encoder for  $\mathbf{z}_1$  takes a sequence of length  $T_z < T$  as input, where  $T_z$  is a hyper-parameter. We empirically observe that our framework is not sensitive to  $T_z$ . The optimization of the modulator prediction networks  $\mathbf{g}_{\psi}$  and  $\mathbf{h}_v$  is implicit in that they are trained jointly with other modules while maximizing the ELBO objective.
Neural ODEs Since the neural ODE breakthrough [Chen et al., 2018], there has been a growing interest in continuous-time dynamic modeling. Such attempts include combining recurrent neural nets with neural ODE dynamics [Rubanova et al., 2019, De Brouwer et al., 2019], where latent trajectories are updated upon observations, as well as upon Hamiltonian [Zhong et al., 2019], Lagrangian [Lutter et al., 2019], second-order [Yildiz et al., 2019], or graph neural network based dynamics [Poli et al., 2019]. While our method MoNODE has been introduced in the context of latent neural ODEs, it can be directly utilized within these frameworks as well.
Augmented dynamics Dupont et al. [2019] augment data-space neural ODEs with additional latent variables and test their method on classification problems. Norcliffe et al. [2021] extend neural ODEs to stochastic processes by means of stochastic latent variables, leading to NODE Processes (NODEP). By construction, NODEP embeddings are invariant to the shuffling of the observations. To the best of our knowledge, we are the first to explicitly enforce time-invariant modulator variables.
Learning time-invariant variables The idea of averaging for invariant function estimation was used in [Kondor, 2008, van der Wilk et al., 2018, Franceschi et al., 2020]. Only the latter proposes using such variables in the context of discrete-time stochastic video prediction. Although relevant, their model involves two sets of dynamic latent variables, coupled with an LSTM and is limited to mapping modulation.
To investigate the effect of our proposed dynamics and static modulators, we structure the experiments as follows: First, we investigate the effect of the dynamics modulator on classical dynamical systems, namely, sinusoidal wave, predator-prey trajectories and bouncing ball (section 5.1), where the
Table 2: Test MSE and its standard deviation with and without our framework for NODE [Chen et al., 2018], SONODE [Norcliffe et al., 2020], and HBNODE Xia et al. [2021]. We test model performance within the training regime and for forecasting accuracy. Each model is run three times with different initial seed, we report the mean and standard deviation across runs. Lower is better.
Model	Sinusoidal data	Predator-prey data
T=50	T=150	T=100	T=300
NODE	0.13 (0.03)	1.84 (0.70)	0.85 (0.08)	23.81 (2.29)
MoNODE (ours)	0.04 (0.01)	0.29 (0.11)	0.74 (0.05)	4.33 (0.19)
SONODE	2.19 (0.15)	3.05 (0.07)	15.80 (0.75)	40.92 (0.82)
MoSONODE (ours)	0.05 (0.01)	0.35 (0.10)	1.46 (0.28)	6.70 (0.83)
HBNODE	0.16 (0.02)	3.36 (0.33)	0.88 (0.10)	3346.62 (2119.24)
MoHBNODE (ours)	0.05 (0.01)	0.65 (0.30)	0.94 (0.09)	10.21 (1.43)
parameterisation of the dynamics differs across each trajectory. Second, to confirm the utility of the static modulator we implement an experiment of rotating MNIST digits (section 5.2), where the static content is the digit itself. Lastly, we experiment on real data with having both modulator variables present for predicting human walking trajectories (section 5.4). In all experiments, we test whether our framework improves the performance of the base model on generalization to new trajectories and long-horizon forecasting abilities.
Implementation details We implement all models in PyTorch [Paszke et al., 2017]. The encoder, decoder, differential function, and modulator prediction networks are all jointly optimized with the Adam optimizer [Kingma and Ba, 2014]. For solving the ODE system we use torchdiffeq [Chen, 2018] package. We use the 4th-order Runge-Kutta numerical solver to compute ODE state solutions (see App. D for ablation results for different solvers). For the complete details on data generation and training setup, we refer to App.B. Further, we report the architectures, number of parameters, and details on hyperparameter for each method in App. C Table 7.
Compared methods We test our framework on the following models: (i) Latent neural ODE model [Chen et al., 2018] (NODE), (ii) Second-order NODE model [Norcliffe et al., 2020] (SONODE), (iii) Latent second-order NODE model [Yildiz et al., 2019] (LSONODE), (iv) current state-of-the-art, second-order heavy ball NODE [Xia et al., 2021] (HBNODE).
In order for SONODE and HBNODE to have a comparable performance with NODE we adjust the original implementation by the authors by changing the encoder architecture, while keeping the core of the models, the differential function, unchanged. For further details and discussion, see App. A. We do not compare against ANODE [Dupont et al., 2019] as the methodology is presented in the context of density estimation and classification. Furthermore, we performed preliminary tests with NODEP [Norcliffe et al., 2021]; however, the model predictions fall back to the prior in datasets with dynamics modulators. Hence, we did not include any results with this model in the paper as the base model did not have sufficiently good performance. Finally, we chose not to compare against Kidger et al. [2020] as their Riemann-Stieltjes integral relies on smooth interpolations between data points while we particularly focus on the extrapolation performance for which the ground truth data is not available. For an overview of the methods discussed see Table 1.
5.1 Dynamics modulator variable
To investigate the benefit of the dynamics modulator variable, we test our framework on three dynamical systems: sine wave, prey-predator (PP) trajectories, and bouncing ball (BB) videos. In contrast to earlier works [Norcliffe et al., 2020, Rubanova et al., 2019], every trajectory has a different parameterization of the differential equation. Intuitively, the goal of the modulator prediction network is to learn this parameterisation and pass it as an input to the dynamics function, which is modelled by a neural network.
5.1.1 Sinusoidal data
The training data consists of  $N = 300$  oscillating trajectories with length  $T = 50$ . The amplitude of each trajectory is sampled as  $a^n \sim \mathbb{U}[1,3]$  and the frequency is sampled as  $\omega^n \sim \mathbb{U}[0.5,1.0]$ ,
Figure 2: The left panel illustrates PCA embeddings of the latent ODE state inferred by NODE and MoNODE, where each color matches one trajectory. The figures on the right compare NODE and MoNODE reconstructions on test trajectory sequences. Note that the models take the dashed area as input, and the darker area represents the trajectory length seen during training.