Modulated Neural ODEs
Authors: Ilze Amanda Auzina
Cagatay Yildiz
Sara Magliacane
Matthias Bethge
Efstratios Gavves
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural ODEs
Modulator Variables
Dynamical Systems
Disentanglment
TL;DR: The present work introduces a novel modulator framework for NODE models that allows to separate time-evolving ODE states from modulator variables.

Abstract: Neural ordinary differential equations (NODEs) have been proven useful for learning non-linear dynamics of arbitrary trajectories. However, current NODE methods capture variations across trajectories only via the initial state value or by auto-regressive encoder updates. In this work, we introduce Modulated Neural ODEs (MoNODEs), a novel framework that sets apart dynamics states from underlying static factors of variation and improves the existing NODE methods.  In particular, we introduce *time-invariant modulator variables* that are learned from the data. We incorporate our proposed framework into four existing NODE variants. We test MoNODE on oscillating systems, videos and human walking trajectories, where each trajectory has trajectory-specific modulation. Our framework consistently improves the existing model ability to generalize to new dynamic parameterizations and to perform far-horizon forecasting. In addition, we verify that the proposed modulator variables are informative of the true unknown factors of variation as measured by $R^2$ scores.

Modulated Neural ODEs
llze Amanda Auzina University of Amsterdam i. a. auzina@uva.co nl
Çagatay Yildız University of Tübingen Tübingen Al Center
Sara Magliacane University of Amsterdam
MIT-IBM Watson Al Lab
Matthias Bethge University of Tübingen Tübingen AI Center
Efstratios Gavves University of Amsterdam
Abstract
Neural ordinary differential equations (NODEs) have been proven useful for learn- ing non-linear dynamics of arbitrary trajectories. However, current NODE methods capture variations across trajectories only via the initial state value or by auto- regressive encoder updates. In this work, we introduce Modulated Neural ODEs (MoNODEs), novel framework that sets apart dynamics states from underlying static factors variation and improves the existing NODE methods. In particular, we introduce time-invarian modulator variables that are learned from the data. We incorporate our proposed framework into four existing NODE variants. We test MoNODE on oscillating systems, videos and human walking trajectories, where each trajectory has rajectory-specifi modulation. Our framework consistently improves the existing model ability to generalize to dynamic parameterizations and to perform far- hor-ooizon forecasting. In addition, we verify that the proposed modulator variables are informative of the true unknown factors of variation measured by R² scores.
1 Introduction
Differential equations are the de facto standard for learning dynamics of biological [Hirsch et al. 2012] and physical [Tenenbaum and Pollard| 1985) systems. When the observed phenomenon deterministic, the dynamics are typically expressed in terms of ordinary differential equations (ODEs). Traditionally, ODEs have been built from a mechanistic perspective, in which states of the observed system and the governing differential equation with its parameters are specified by domain experts. However, when the parametric form unknown and only observations are available, neural network surrogates can be used to model the unknown differential function, called neural ODEs (NODEs) Chen al. 2018 Since its introduction by Chen al.) [2018), there has been an abundance of research that uses NODE type models for learning differential equation systems and extend it by introducing a recurrent neural networks (RNNs) encoder [Rubanova et al. 2019 Kanaa et al.) 2021]
gated recurrent unit [De Brouwer et al. 2019 Park al.) 2021]. or second order system Yildiz
ct al. 2019 Norcliffe et al. 2020 Xia et al. 2021].
Despite these recent developments, all of the above methods have common limitation: any static differences in the observations can only be captured in a time-evolving ODE state. This modelling approach is not suitable when the observations contain underlying factors of variation that are fixed in time, yet can (i) affect the dynamics or (ii) affect the appearance of the observations. As a concrete example of this, consider human walking trajectories. The overall motion is shared across all subjects, e.g. walking. However, every subject might exhibit person-specific factors of variation, which in turn could either affect the motion exhibited, e.g. length of legs, or could be characteristic of given subject, e.g. color of shirt. A modelling approach that is able (i) distinguish the dynamic
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Dynamies modulator
Modulatur Prodiction Network
ODE state
Encoding Network
Forward Simulation (ODE)
Decoding
Netwurk
Modulator Prodiction
Network
Figure 1: Schematic illustration of our method. Standard continuous-time latent dynamical systems such as Chen et al. [2018) assume single ODE state x(t) transformed by a surrogate neural differential function. Our method augments the latent space with dynamics and static modulators to account for exogenous variables governing the dynamics and observation mapping (decoding).
(e.g. position, velocity) from the static variables (e.g. height, clothing color), and (ii) use the static variables to modulate motion or appearance, advantageous, because it leads to an improved generalization across people. As an empirical confirmation, we show experimentally that the existing NODE models fail to separate the dynamic factors from the static factors. This inevitably leads to overfitting, thus, negatively effecting the model generalization to new dynamics, as well as far-horizon forecasting.
As remedy, this work introduces modulated neural ODE (MoNODE) framework, which separates the dynamic variables from the time-invariant variables. We call these time-invariant variables modulator variables and we distinguish between two types: (i) static modulators that modulate the appearance; and (ii) dynamics modulators that modulate the time-evolution of the latent dynamical system (for schematic overview see Fig. []. In particular, MoNODE adds modulator prediction network on top of NODE, which allows to compute modulator variables from data. We empiri- cally confirm that our modular framework boosts existing NODE models by achieving improved future predictions and improved generalization to new dynamics. addition, we verify that our modulator variables more informative of the true unknown factors of variation by obtaining higher H² scores than NODE. As a result, the latent ODE state of MoNODE has an equivalent sequence-to-sequence correspondence as the true observations. Our contributions are as follows:
We extend neural ODE models by introducing modulator variables that allow the model to preserve core dynamics while being adaptive to modulating factors.
Our modulator framework can easily be integrated into existing NODE models Chen et al. 2018 Yildiz et al. 2019 Norcliffe al. 2020 Xia et al. [021]]
As we show in our experiments on sinusoidal waves, predator-pre prey dynamics, bouncing ball videos, rotating images, and motion capture data, our modulator framework consistently leads to improved long-term predictions as measured by lower test mean squared error (an average of 55.25% improvement across all experiments).
Lastly, verify that our modulator variables more informative of the true unknown factors of variation than NODE, as we show terms of R²2 scores.
While introduced in the context of neural ODEs, believe that the presented framework can benefit also stochastic and/or discrete dynamical systems, as the concept of dynamic or mapping modulation innate for most dynamical systems. We conclude this work by discussing the implications of our method on learning object-centric representations and its potential Bayesian extensions. Our official implementation can be found at htthtttp://giiieii hhtpb//tttttttt///// com IlzeAmandak/KoooDDS
2 Background
We first introduce the basic concepts for Neural ODEs, following the notation by Chen et al. 2018
Ordinary differential equations Multivariate ordinary differential equations are defined as
2(1) dz(t)
(1)))
dt

---

where t E R denotes time, the vector z(t) E R captures the state of the system at time t, and 2(t E It the time derivative of the state z(t). this work, we focus on autonomous ODE systems, implying vector-valued (lime) differential function f IR - IR%- that does not explicitly depend on time. The ODE state solution x(12) is computed by integrating the differential function starting from an initial value (tt)):
{ f(z(T))dr.
2((2) z(tz)
For non-linear differential functions f. the integral does not have closed form solution and hence is approximated by numerical solvers [Tenenbaum and Pollard 1985]. Due to the deterministic nature of the differential function, the ODE state solution 2(t2) completely determined by the corresponding initial value z(4) if the function f is known.
2.1 Latent neural ODEs
Chen et al. 2018 proposed neural ODEs for modeling sequential data X1:7 E RTXD where X; x(t4) the D-dimensional observation at time t,, and is the sequence length. We assume known observation times (ti....,rr) |r.. Being a latent variable model, NODE infers a latent trajectory Z1:r _xxxx for input trajectory x1:r. The generative model relies on random initial values, their continuous-time transformations, and finally an observation mapping from latent to data space:
Z1 p(z1)
2, = 21 + 1" fZ(2(T))dr
(4)
x; ~ (x |z;)
(5)
Here, the time differential fo is neural network with parameters f (hence the name "neural ODEs"). Similar to variational auto-encoders [Kingma and Welling 2013 Rezende et al. [2014), the "decoding" of the observations is performed by another non-linear neural network with suitable architecture and parameters E.
3 MONODE: Modulated Neural ODEs
We begin with a brief description the dynamical systems of our interest. Without loss of generality, consider dataset of N trajectories with a fixed trajectory length T, where the 'thh trajectory denoted by x".7. First, we make the common assumption that the data trajectories are generated by single dynamical system (e.g., swinging pendulum) while the parameters that modulate the dynamics (e.g., pendulum length) vary across the trajectories. Second, we focus on the more general setup in which the observations and the dynamics might lie in different spaces, for example, video recordings of pendulum. A future video prediction would require a mapping from the dynamics space to the observation space. As the recordings might exhibit different lighting conditions or backgrounds, the mapping typically involves static features that modulate the mappings, e.g., a parameter specifying the background.
3.1 Our generative model
The generative model of NODE [Chen al. 2018 involves latent initial value z₁ for each observed trajectory as well as differential function and decoder with global parameters @ and &. Hence, by construction, NODE can attribute discrepancies across trajectories only to the initial value as the remaining functions are modeled globally. Subsequent works [Rubanova et al., 2019 Dupont et al. 2019 De Brouwer et al. 2019 Xia et al., 22021] lakovlev et al. 2023] combine the latent space of NODE with additional variables, however, likewise, they do not account for static, trajectory-specife factors that modulate either the dynamics or the observation mapping. The main claim of this work that explicitly modeling the above- modulating variables results in better extrapolation and

---

generalization abilities. To show this, we introduce the following generative model:
d ~ p(d)
// dynamics modulator
(6)
p(s)
// static modulator
(7)
21 ~ p(z1)
// latent ODE state
(8)
/ (z(((T); dr
2. z
(9)
PE (x(x;
Z; s).
(10)
We broadly refer to d and as dynamics and static modulators, and thus name our framework Modulated Neural ODE (MoNODE). We assume that each observed sequence x1.r has its own modulators d" and s" As opposed to the ODE state z(t), the modulators are time-invrrian.
We note that for simplicity, we describe our framework in the context of the initial neural ODE model [Chen et al., 2018). However, our framework can be readily adapted to other neural ODE models such as second-order and heavy ball, NODEs, as we demonstrate experimentally. For schematic overview, please see Fig. [ Next, we discuss how to learn the modulator variables along with the global dynamics, encoder, and decoder.
3.2 Learning latent modulator variables
straightforward approach to obtain time-invariant modulator variables to define them globally and independently of each other and input sequences. While optimizing for such global variables works well in practice [Blei et al.) 2017]. it does not specify how to compute variables for an unobserved trajectory. As the focus of the present work is on improved generalization to unseen dynamic parametrizations, we estimate the modulator variables via amortized inference based on encoder networks g& and h which detail in the following.
To learn the unknown static modulator s" R that captures the time- invariant characteristics of the individual observations x we compute the average over the obser- vation embeddings provided by a modulator prediction network g (.) (e.g., a convolutional neural network) with parameters ab:
(i) Static modulator
S = = (x.).
(11)
T
By construction, s" time-invariant (or more rigorously, invariant to time- demeddepndent effects) as we average over time. In turn, the decoder takes as input the concatenation of the latent ODE state z," and the static modulator s", and maps the joint latent representation to the observation space (similarly to Franceschi et al. (2020):
PE
xx
2. s'))
Vi E [I.
(12)
Note that the estimated static modulator s" is fed as input for all time points within a trajectory.
(ii) Dynamics modulator Unlike the static modulator, the dynamics modulator can only be deduced from multiple time points x For example, the dynamics of pendulum depend on its length. To compute the length of the pendulum and velocity measurements are needed. Thereby, the dynamics modulators, d,, are computed from subsequences of length T, from given trajectory. To achieve time- invariiiie we likewise average over time:
must compute the acceleration for which multiple position
T d
d T
Σ
- hy (xH+tt))
(13)
where h, is a modulator prediction network (e.g., a recurrent neural network) with parameters v. The differential function takes as input the concatenation of the latent ODE state z" (T) E R and the estimated dynamics modulator d 94 Consequently, we redefine the input space of the differential function fg 19 + 19a, implying the following time differential:
dz (t)
to (z d1)
(14)
dt

---

Table I: A taxonomy of the state-of-the-art ODE systems. We empirically demonstrate that modulating variables idea is straightforward to apply to NODE, SONODE, and HBNODE, and con- sistently leads to improved prediction accuracy. Where with we refer to: MoNODE, MoSONODE, MOLSONODE, MOHBNODE.
Time invariant modulator
Model
Reference
Latent dynamics
Temporal data
Neural ODE (NODE)
Chen
2018
X
Augmented NODE (ANODE)
Dupont et a. 2019
X
X
Neural Controlled DE (NCDE)
Kidger et al. 2020
/
X
X
Second Order NODE (SONODE)
Norchiff ct a.. 2020
X
X
Latent SONODE (LSONODE)
Yildız et al. 2019
X
NODE Processes (NODEP)
Norcliffe et al. 2021
X
Heavy Ball NODE (HBNODE)
Xia et 2021
X
Modulated "NODE
this work
The resulting ODE system resembles way the augmented neural ODE (ANODE) [Dupont et 2019]. However, their appended variable dimensions are constant and serve practical purpose of breaking down the diffeomorphism constraints of NODE, while ours models inverinvariant variables. We treat T. as a hyperparameter and choose it by cross-validation.
Optimization objective The maximization objective of MONODE analogous to the evidence- lower bound (ELBO) as in [Chen et al. 2018 for NODE, where place prior distribution on the unknown latent initial value z and approximate it by amortized inference. Similar to previous works [Chen et al. 2018 Yildiz al. 2019]. MONODE encoder for Z1 takes sequence of length T. < T We empirically observe that our framework not sensitive
as input, where T3 is to T.. The optimization of the modulator prediction networks g and h, implicit in that they trained jointly with other modules while maximizing the EIBOO objective.
4 Related work
Neural ODEs Since the neural ODE breakthrough [Chen et al.) 2018]. there has been a growing interest in continuous-tine dynamic modeling. Such attempts include combining recurrent neural nets with neural ODE dynamics [Rubanova et al. 2019, De Brouwer et al. 2019]. where latent trajectories updated upon observations, well upon Hamiltonian [Zhong et al.] 2019), Lagrangian [Lutter jet al. 2019], second-order (Yildiz et al.) 2 or graph neural network based dynamics [Poli et al. 2019 While our method MoNODE has been introduced in the context of latent neural ODEs, it can be directly utilized within these frameworks as well.
Augmented dynamies Dupont et al. 2019 augment data-space neural ODEs with additional latent variables and test their method on classification problems. Norcliffe et al. 2021 extend neural ODEs stochastic processes by means of stochastic latent variables, leading to NODE Processes (NODEP). By construction, NODEP embeddings are invariant to the shuffling of the observations. To the best of our knowledge, we are the first to explicitly enforee
modulator variables.
Learning time-invariant variables The idea of averaging for invariant function estimation was used [Kondor 2008 van der Wilk et al. 2018 Franceschi et al. 2020]. Only the latter proposes using such variables in the context of discrete-time stochastic video prediction. Although relevant, their model involves two sets of dynamic latent variables, coupled with an LSTM and is limited to mapping modulation.
5 Experiments
To investigate the effect of our proposed dynamics and stafic modulators, we structure the experiments as follows: First, we investigate the effect of the dynamics modulator on classical dynamical systems, namely, sinusoidal wave, predator-prey trajectories and bouncing ball (section 5.1). where the
5

---

Table 2: Test MSE and its standard deviation with and without our framework for NODE [Chen al. 2018). SONODE [Norcliffe et al., 2020]. and HBNODE Xia al. [202]]. We test model performance within the training regime and for forecasting accuracy. Each model is run three times with different initial seed, we report the mean and standard deviation across runs. Lower is better.
Sinusoidal data
Predator- prey data
Model
T 50
T 150
T 100
T 300
NODE
0.13 (0.03)
1.84 (0.70)
0.85 (0.08)
23.81 (2.29)
MONODE (ours)
0.04 (0.01)
0.29 (0.11)
0.74 (0.05)
4.33 (0.19)
SONODE
2.19 ( (0.15)
3.05 ( (0.77)
15.80 (0.75)
40.92 (0.82)
MOSONODE (ours)
0.05 ( 0.01)
0.35 (0.10)
1.46 (0.28)
6.70 (0.83)
HBNODE
0.16 ( 0.02)
3.36 ( 0.33)
0.88 (0.10)
3346.62 (2119.24)
MOHBNODE (ours)
0.05 (0.01)
0.65 ( (.300
0.94 (0.09)
10.21 (1.43)
parameterisation of the dynamics differs across each trajectory. Second, to confirm the utility of the static modulator we implement an experiment of rotating MNIST digits (section [.2]. where the static content is the digit itself. Lastly, we experiment on real data with having both modulator variables present for predicting human walking trajectories (section 5.4). all experiments, we test whether our framework improves the performance of the base model on generalization to new trajectories and long-horizon forecasting abilities.
We implement all models in PyTorch [Paszke et al. 2017]. The encoder, decoder, differential function, and modulator prediction networks are all jointly optimized with the Adam optimizer [Kingma and Ba 2014). For solving the ODE system we use torchdiffeq [Chen] 2018) package. We use the 4th-order Runge-Kutte numerical solver to compute ODE state solutions (see App. D for ablation results for different solvers). For the complete details data generation and training setup, we refer to A Further, we report the architectures, number of parameters, and details on hyperparameter for each method in App. C Tablle7
Implementation details
Compared methods We test our framework on the following models: (i) Latent neural ODE model (Chen et al. 2018] (NODE), (ii) Second-order NODE model [Noreliffe al. 2020) (SONODE), (iii) Latent second-order NODE model (Yildiz et al. 2019 (LSONODE), (iv) current state-of-the-art, second-order heavy ball NODE (Xia et al. 2021 (HBNODE).
In order for SONODE and HBNODE to have a comparable performance with NODE we adjust the original implementation by the authors by changing the encoder architecture, while keeping the core of the models, the differential function, unchanged. For further details and discussion, see App.[A] We do not compare against ANODE (Dupont et al. [2099] the methodology is presented in the context of density estimation and classification. Furthermore, we performed preliminary tests with NODEP [Noclliffe et al. 202T; ; however, the model predictions fall back to the prior in datasets with dynamics modulators. Hence, we did not include any results with this model in the paper as the base model did not have sufficiently good performance. Finally, we chose not to compare against Kidger et al. [2020) as their Riemann-Stielijes integral relies on smooth interpolations between data points while we particularly focus on the extrapolation performance for which the ground truth data is not available. For an overview of the methods discussed see Table [
5.1 Dynamics modulator variable
To investigate the benefit of the dynamics modulator variable, we test our framework on three dynamical systems: sine wave, prey-predator (PP) trajectories, and bouncing ball (BB) videos. In contrast to earlier works [Norcliffe al. 20200] Rubanova et al. 2019), every trajectory has a different parameterization of the differential equation. Intuitively, the goal of the modulator prediction network is to learn this parameterisation and pass as an input to the dynamics function,
which is modelled by neural network.
5.1.1 Sinusoidal data
The training data consists of N 300 oscillating trajectories with length = 50. The amplitude of each trajectory is sampled as a" (][13] and the frequency is sampled as w U[0.5, 1.0,,

---

Thepslway
NODE
Trijecooy id.
Gound Truth
Erajecoory
Ground Trush
Trijectory
Eralectory
Trapctorr
MaNODE
Truecto it
Trijectory
Trajectory id:
NODE
MoNODE
Figure 2: The left panel illustrates PCA embeddings of the latent ODE state inferred by NODE and MoNODE, where each color matches one trajectory. The figures the right compare NODE and MoNODE reconstructions on test trajectory sequences. Note that the models take the dashed area as input, and the darker area represents the trajectory length seen during training.
Trigecory at
TIINODE
Trejectory
Prajjoctory
Grcund Touth
"Dijeittery idi
Trajectory W
Tlee wof
Trandey 18.9
MoHBNNDD
Trenetor LE
Trajectory
Trajectory
HBNODE
MoHBNODE
Figure 3: To show our framework applicability to different NODE variants, here we illustrate HBNODE and MOHBNODE performance on PP data. The left panel illustrates PCA embeddings of the latent ODE state inferred by HBNODE and MOHBNODE, where each color matches one trajectory. The figures right compare HBNODE and MOHBNODE reconstructions test trajectory sequences. Note that the models take the dashed area as input, and darker area represents the trajectory length seen during training. predator-per prey data.
where U[- -I denotes a uniform distribution. Validation and data consist of Nval Nust 50 trajectories with sequence length Teal 50 and Tist 150, respectively. We add noise to the data following the implementation of [Rubanova et al. 2019].
The obtained test MSE demonstrates that our framework improves all aforementioned methods, namely, NODE, SONODE, and the state-of-the-arl HBNODE (see Fig. 2 Fig. IT and Fig. F2). In particular, our framework improves generalization to new dynamics and far-horizon forecasting as reported in Table Tabieecolumns and 3. In addition, modulating the motion via the dynamics modulator variable leads to interpretable latent ODE state trajectories z(t), see Fig. 2 More specifically, the obtained latent trajectories z(t) have qualitatively a comparable radius topology the observed amplitudes in the data space. By contrast, the latent space of NODE does not have such structure. Similar results are obtained for MOHBNODE (App. D Fig.
Training and inference times plotted the validation MSE versus wall clock time during training for sinusoidal data, please see App. Fig. 10 As it is apparent from the figure, our framework is easier to train than all baseline methods. We further compute the inference time cost for the sin data experiment, where the test data consists of 50 trajectories of length 150. We record the time takes NODE and MoNODE to predict future states while conditioned on the initial 10 time points. Repeating the experiment ten times, the inference time cost for NODE is 0.312 + 0.050 while for MoNODE is 0.291 + 0.040.
To showcase that our proposed framework is easy to train we have
5.1.2 Predator-prey (PP) data
Next, we test our framework on the predator-prey benchmark (Rubanova et al. 2019 Norcliffe et al., 2021 governed by pair of first-order nonlinear differential equations (also known Volterolterra
7

---

The training data consists of N 600 trajectories of length T = 100. For every trajectory the four parameters of the differential equation are sampled. therefore each trajectory specifies different interaction between the two populations. Validation and test data consist of Nua = Ne = 100 trajectories with sequence length Taat 100 and Tiest = 300. Similarly to sinusoidal data, we add noise to the data following (Rubanova et al. [019].
system, Eq. 28).
The results show that our modulator framework improves the test accuracy of the existing methods for both, generalization to new dynamics as well as for forecasting, see T Moreover, examining the latent ODE state embeddings reveals that our framework results more interpretable latent space embeddings also for PP data, see Fig. for HBNODE and App. Di [3] for NODE. In particular, the latent space of MoHBNODE and MONODE captured the same amplitude relationship across trajectories as in observation space. For visualisation of the SONODE, MoSONODE, NODE, and MONODE trajectories see App. D Fig. 14 Fig.
5.1.3 Bouncing ball (BB) with friction
To investigate the performance of our framework on video sequences we test it on a bouncing ball dataset, a benchmark often used in temporal generative modeling [Sutskever et al. 2008 Gan et al. 2015 Yildiz et al.] 2019). For data generation, we modify the original implementation of Sutskever al. 2008 by adding friction to every data trajectory. where friction is sampled from U[0, The friction slows down the ball by a constant factor and is to be inferred by the dynamics modulator. We use N 1000 training sequences of length T 20, and Noa1 = Nus = 100 validation and test trajectories with length T. = 20 and Tiss= 40. Our framework improves predictive capability for video sequences as shown in Table 4 As visible in Fig [15] the standard NODE model fails to predict the position of the object at further time points, while MONODE corrects this error. For the second-order variants, LSONODE and MOLSONODE, we again observe our framework improving MSE from 0.0181 to 0.014.
5.1.4 Informativeness metric
Next, we quantify how much information latent representa- tions carry about the unknown factors of variation (FoVs)
Table 3: R² scores predict the unknown FoVs from inferred latents. Higher is better.
[Easwwood and Williams 2018), which are the parameters of the sinusoidal, PP, and BB dynamics. As described in [Schott] ct al. 2021 we compute R² scores by regressing from latent variables to Fo The regression inputs for MONODE are the dynamics modulators d, while for NODE the latent trajecto- ries Z1. Note that R 1 corresponds to perfect regression and R = indicates random guessing. Our framework ob- tains better R² scores all benchmarks (Table 3). iimllyng better generalization capability of our framework. Therefore, as stated in [Eastwood and Williams) 2018). our MONODE is better capable of disentangling underlying factors of variations compared to NODE.
NODE
MoNODE
Sine PP -1.35
0.90
0.99
0.39
BB
-0.29
0.58
5.2 Static modulator variable
To investigate the benefit of the static modulator variable, we test our framework on Rotating MNIST dataset, where the dynamics are fixed and shared across all trajectories, however, the content varies. The goal of the modulator prediction network is to learn the static features of each trajectory and pass it as an input to the decoder network. The data is generated following the implementation by (Casale cct al. 20188. where the total number of rotation angles is T 16. We include all ten digits and the initial rotation angle is sampled from all possible angles lt ~ {1..... 16). The training data consists of N = 1000 trajectories with length T' = 16, which corresponds to one cycle of rotation. Validation and test data consist of N.ai = NisA = 100 trajectories with sequence length Ta = Tiu 45. At test time, the model receives the first T. time frames as input and predicts the full horizon (T=45) We repeat each experiment 3 times and report the mean and standard deviation of the MSEs computed on the forecasting horizon (from T = 15 to T = 45). For further training details see ApplB while for complete overview of hyperparameter see App- C 7
The obtained test MSE confirms that the static modulator variable improves forecasting quality (T' = 45), see Table and App. D fig. 16 for qualitative comparison. In addition, the latent ODE
8

---

26'
QU"
PCA-2 (0.14)
119
PCA 2 (0.38 (0.34)
119"
144
167"
192"
192"
216
239
263
263"
335
335"
360"
360
PCA-1 (0.19)
PCA-1 (0.36)
Figure 4: PCA embeddings of the latent ODE state for Rotating MNIST dataset inferred by NODE (top) and MoNODE (bottom). We generate 16 trajectories from single digit where the initial angle of every trajectory incremented by 24 degrees, starting from 0 until 360 degrees. Circle: the start of the trajectory (the initial angle), line: ODE trajectory. The color gradient corresponds to the initial angle of the trajectory in observation space.
Table 4: Comparison of NODE and MoNODE test MSEs and standard deviations on BOUNCING BALL, MNIST, and MOCAP datasets.
BOUNCING BALL ROT.MNIST
MOCAP
MOCAP-SHIFT
NODE
0.0199(0.001)
0.039 (0.003)
72.2(12.4)
61.6(6.2)
MONODE 0.0164(0.001)
||
0.030 (0.001) 57.7(9.8)
||
58.0(10.7)
states of MoNODE form circular rotation pattern resembling the observed dynamics in data space while for NODE no correspondence is observed, see App. Dop.DFiis Fig. [6 Moreover, as shown in Fig. +
the latent space MoNODE captured the relative distances between the initial rotation angles. while NODE did not. The TSNE embeddings of the static modulator indicate clustering per digit
shape, see App. Fig. 17
For additional results and discussion, see App. D
5.3
Are modulators interchangeable?
To confirm the role of each modulator variable we have performed two additional ablations with the MoNODE framework on: (a) sinusoidal data with static modulator instead of dynamics, and (b) rotating MNIST with dynamics modulator instead of static. We report the test MSE across three different initialization runs with standard deviation. For sinusoidal data with MoNODE + static modulator, the test MSE performance drops to 2.68 = 0.38 from 0.29 0.11 (MoNODE + dynamic modulator). For rotating MNIST, MoNODE + dynamics modulator performance drops 0.096 = 0.008 from 0.030 + 0.001 (MoNODE static modulator). In addition. we examined the latent embeddings of the dynamics modulator for rotating MNIST. Where previously for the content modulator we observed clusters corresponding to a digit's s class, for dynamics modulator such topology in the latent space is not present (see App. D Fig. 17). Taken together with Table 3 the results confirm that the modulator variables are correlated with the true underlying factors of variation and play their corresponding roles.
5.4 Real world application: modulator variables
Next, we evaluate our framework on a subset of CMU Mocap dataset, which consists of 56 walking sequences from 6 different subjects. We pre-process the data as described in [Wang et al. 2007]. resulting in 50-dimensional data sequences. We consider two data splits in which (i) the training and test subjects are the same, and (ii) one subject reserved for testing. We refer to the datasets as MOCAP and MOCAP-SHIFT (see Bpp.BBoo details), In test time, the model receives the first 75 observations as input and predicts the full horizon (150 time points). We repeat each experiment five times and report the mean and standard deviation of the MSEs computed on the full horizon, As shown in Table and Fig. 5) our framework improves upon NODE. See Table for ablations with different latent dimensionalities and with only one modulator variable (static dynamics) present.

---

Time t=1
Time t=30
Time t=
Time t=90
Time t=120
Time 1=150
NODR
MoNODE
Figure 5: MoNODE and NODE reconstructions Mocap test sequence. Note that the models take the first Ti = 75 time points as input and predict for T 150 time steps.
Discussion
The present work introduces novel modulator framework for NODE models that allows to separate time-evolving ODE states from modulator variables. In particular, we introduce two types of modulating variables: (i) dynamics modulator that can modulate the dynamics function and (ii) static modulator that can modulate the observation mapping function. Our empirical results confirm that our framework improves generalization to new dynamics and far-horizon forecasting, Moreover, our modulator variables better capture the true unknown factors of variation as measured by R score, and, as a result, the latent ODE states have an equivalent correspondence to the true observations.
Limitations and future work The dynamical systems explored in the current work are limited to deterministic periodic systems that have different underlying factors of variation. The presented work introduces a framework that builds upon a base NODE model, hence, the performance of Mo*NODE largely affected by the base model's performance. The current formulation cannot account for epistemic uncertainty and does not generalize to out-o--disrribution modulators, because we maintain point estimates for the modulators. A straightforward extension would be apply our framework Gaussian process-basse variable that models the noise, similarly to FFanceschh al. (222). Likewise, rather than maintaining point estimates, the time-invarian invariant modulator parameters could be inferred via marginal likelihood as van der Wilk al. (2018), [Schwöell et al. [2022]. leading to theoretically grounded Bayesian framework. Lastly, the separation of the dynamic factors and modulating factors could be explicitly enforced via additional self-supervised contrasting loss term [Grill et al.) 2020) or by more recent advances in representation learning [Bengio et al.) 2013]. Lastly, the concept of dynamics modulator could also be extended to object-centric dynamical modeling [Kabra et al., 2021], which would allow accounting for per-object specific dynamics modulations while using single dynamic model.
ODEs (Hegde et al., [022] stochastic dynamical systems via an auxiliary
Ackmowledgements The data used in this project was obtained from mocap.... cs cmu. edu. The database was created with funding from NSF EIA-0196217 Çağatay Yildiz funded by the Deutsche Forschungsgemeinschaie (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXCCNumber 2064/1 Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 950086).
References
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1798 2013.
D. M. Blei, A. Kucukelbir, and D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 2017.
P. Casale, A. Dalca, Saglietti, Listgarten, and N. Fusi. Gaussian process prior variational autocncoders. Advances in neural information processing systems, 31, 2018.
10

---

R. Т. Chen, Y. Rubanova. J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations.
Advances in neural information processing systems, 31, 2018.
R. Q. Chen. torchdiffeq, 2018. URL /tt.hhttwee/gggttoo...c./////oea
E. De Brouwer, J. Simm, A. Arany, and Y. Moreau. Gru-ode-bayes: Continuous modeling of sporadically- time series. Advances in neural information processing systems, 32, 2019.
E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural ODEs. In Advances Neural Information Processing Systems, 2019.
C. Eastwood and C. K. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference Learning Representations, 2018.
J.-Y. Franceschi, Delasalles, M. Chen, S. Lamprier, and Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pages 3233-3246. PMLR, 2020.
Z. Gan, C. Li, R. Henao, D. E. Carlson, and L. Carin. Deep temporal sigmoid belief networks for sequence modeling. Advances in Neural Information Processing Systems, 28, 2015.
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires,
Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to sele-super learning. Advances neural information processing systems, 33:21271-21288 2020.
Hegde, Ç. Yıldız, H. Lähdesmäki, S. Kaski, and M. Heinonen. Variational multiple shooting for bayesian odes with gaussian processes. In Uncertainty in Artificial Intelligence, pages 790-799. PMLR, 2022.
M. W. Hirsch, Smale, and Devaney. Differential equations, dynamical systems, and an introduction to chaos. Academic press, 2012.
V. lakovlev, C. Yildiz, M. Heinonen, and H. Lähdesmäki. Latent neural ODEs with sparse bayesian multiple shooting. The Eleventh International Conference on Leurning Representations, 2023. URL htttp:://openreviet
R. Kabra, D. Zoran, G. Erdogan, L. Matthey, A. Creswell, M. Botvinick, A. Lerchner, and C. Burgess. Simone: View-invariant, cemporally-asstrace
object representations via unsupervised video decomposition. Advances in Neural Information Processing Systems, 34:20146-2015 2021.
D. Kanaa, V. Voleti, S. E. Kahou, and C. Pal. Simple video generation using neural odes. arXiv preprint arxiv: 032992 2021.
P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-7707, 2020.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6:412,.998. 2014.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.611 2013.
I. R. Kondor. Group theoretical methods in machine learning. Columbia University, 2008.
M. Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv preprint 1rXivv:907.0044000 2019.
A. Norcliffe, C. Bodnar, B. Day, N. Simidjievski, and P. Liò. On second order behaviour in augmented neural odes. Advances in neural information processing systems, 33:5911-5921, 2020.
A. Norcliffe, C. Bodnar, B. Day, J. Moss, and P. Liò. Neural ode processes. ICLR, 2021.
S. Park, K. Kim, J. Lee, J. Choo, J. Lee, S. Kim, and E. Choi. Vid-ode: Contimuous-time video generation with neural ordinary differential equation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2412-2422, 2021.

---

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Vetito, Z. Lin, A. Desmaison, L. Antiga, and Lerer. Automatic differentiation pytorch. 2017.
M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park. Graph neural ordinary differential equations. arXiv preprint arXiv: 2019.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
Y. Rubanova, R. T. Chen, and K. Duvenaud. Latent ordinary differential equations for irregularly- sampled time series. Advances in neural information processing systems, 32, 2019.
L. Schott, J. Von Kügelgen, F. Träuble, P. Gehler, C. Russell, M. Bethge, B. Schölkopf. F. Locatello, and W. Brendel. Visual representation learning does not generalize strongly within the same domain. arXiv preprint arXiv:2 2021.
P. Schwöbel, M. Jørgensen, S. W. Ober, and M. Van Der Wilk. Last layer marginal likelihood for invariance learning. In International Conference on Artificial Intelligence and Statistics, pages 3542-3555, PMLR, 2022.
A. Solin, E. Tamir, and P. Verma. Scalable inference in sdes by direct matching of the fokker- Jlonco-kolmogoro equation. Advances in Neural Information Processing Systems, 34:417-429, 2021.
I. Sutskever, G. Hinton, and G. W. Taylor. The recurrent temporal restricted boltzmann machine. Advances in neural information processing systems, 21, 2008
M. Tenenbaum and H. Pollard. Ordinary differential equations: elementary textbook for students of mathematics, engineering, and the sciences. Courier Corporation, 1985.
M. van der Wilk, M. Bauer, S. John, and Hensman. Learning invariances using the marginal likelihood. Advances in Neural Information Processing Systems, 31, 2018.
J. M. Wang, J. Fleet, and A. Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence, 30(2):283-298, 2007.
H. Xia, V. Suliafu, H. Ji, T. Nguyen, A. Bertozzi. S. Osher, and B. Wang. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems, 34:18646-18651 2021.
C. Yildiz, M. Heinonen, and H. Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. Advances Neural Information Processing Systems, 32, 2019.
Y. D. Zhong, B. Dey, and A. Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint arXiv: 1909 192999999277.. 2019.
12

---

A Model Encoder Adjustments
SONODE In the original implementation by [Norelifte et al. 2020] the initial velocity is computed by 3-layer MLP with ELU activation functions and only the initial time point, T. = 1, is passed as an input. For the initial position the first observation time point is passed, e.g. Xp. The resulting model fails to fit on sinusoidal data, see fig. for performance on validation trajectories.
60
Figure 6: Original SONODE [Norlliffe et al. 2020 performance on sine validation sequences with T2 = 1. Green dots ground truth, blue line, model's 's prediction.
As it can be seen in figure 6 the model only fits the first data point correctly, therefore, we extend the number of input time points passed to the model to T- 5 to compute the initial velocity. This results in notable performance improvements, see Fig.
20
40
60
80
100
Figure 7: SONODE [Noceliffe et al. 2020 performance on sine validation sequences with T, 5. Green dots is ground truth, blue line, model's 's prediction.
Consequently, we increase the number of input frames used for SONODE, as well as investigate replacing the MIP architecture with a RNN. We report the resulting test MSEs App. D Table T0 For the parameters used in the experiments please see Table 7
In the original implementation by [iia et al. 2021 the encoder is a 3-layer MLP with TANH activation functions that autoregressively takes every ground truth data point input, e.g. x; for = [0..... T' and predicts the latent representation z, which is also fed as input to the proceeding time points 1. The differential function is modeled by linear layer projection. As the present work is focused on model forecasting capabilities we adjust the original HBNODE to be compatible with problem set-ups where the ground truth data is not available. Mcaning that once the model has reached the time point where there is no more ground truth data available (T" > 50), the model only uses it's own latent state predictions z; to compute the next latent state Z,+1 As the authors of HBNODE claim that the model is designed to better fit long-term dependencies we initially reduce the number of subsequent increments, Niner from 10 to (see App. B training details). In Fig. we show the qualitative performance of HBNODE on validation trajectories. Even though the model fits perfectly the ground truth data within < 50, fails to extrapolate to future time frames. We investigate the cause of this issue by, first, changing the differential function from a linear projection to a 3-layer ML. with TANH activations, and, second, by replacing the original autoregressive encoder with RNN encoder that Iearns the initial position Zo.
HBNODE
13

---

20
40
60
80
100
Figure 8: HBNODE Xia et al. 2021 performance on sine validation sequences. Green dots is ground truth, blue line, model's 's prediction.
Table 5: Ablation results for HBNODE with RNN encoder, T, 10, with different Niner sine data.
Validation MSE
Niner HBNODE 0.164
3
5
0.135
0.144
0.066
Increasing the modelling capacity of the differential function did not resolve the observed issue with forecasting. This implies that the observed over-fitting of the model might be caused due its autoregressive encoder that takes every ground truth time point input. By replacing the autoregressive encoder with an RNN encoder we obtain significant forecasting improvements, see replace the autoregressive encoder with RNN and performed additional tests on the number of data increments Niner = [1,3, 5, 10] well the number of input frames Tic to compute the initial state Zo. We find our experimentation that Niner 10 brings the best model performance (see Table (5) and subsequently test different T' model runs with different seeds in App. D Table 10 and for the final model parameter settings used in the experiments please see App.
Fig. Therefore, in all reported experiments
frames, 10 and 40, respectively. We report the obtained test MSE for each set-up
Figure 9: HBNODE [iaa et al. 2021 with RNN encoder, Niner 10 and T3 10 performance on sine validation sequences. Green dots is ground truth, blue line, model's prediction.
14

---

B Experiment details
All models for sine, PP and rotating MNIST have been trained on single GPU (GTX 1080 Ti) with 10 CPUs and 30G memory.
Sine data the same vein as Rubanova et al. time points T {0, 011,0 0.2,
2019 Norcliffe et al. 2021 we first define a set of
................................ 4.9). Then each sequence y" is generated as follows:
a" ~ U|1,3] 3]
(15)
w U|0.5, 1.0]
(16)
Φ U|o, 1.0]
(17)
E" UiO., rj]
(18)
x = a sin (w 4 6²)
(19)
yf of
(20)
(21)
where denotes the uniform distribution and 7) is set to 17 = 0.2. We use At 0.1 to generate
N 300 training trajectories of length Nes 50, Naa 50 validation trajectories of length
Tval 50, and Nes 50 test trajectories of length Tust 150. Test trajectories are longer test
model forecasting accuracy.
During training set batch size 10. We incrementally increase the sequence length of training trajectories starting from T = until T = 50. We perform this in 10 increments for NODE, MoNODE, HBNODE and MOHBNODE, and in increments for SONODE and MOSONODE. All models are trained for 600 epochs. The model with lowest validation MSE is used for evaluation on test trajectories.
Predator-Prey data process:
To generate Predator-Prey trajectories we define the following generative
a" 3.5. 7" 3355]
(22)
(23)
8" 3[
(24)
8 6
(25)
" 1)
(26)
x U[1.,5.] 5.]
(27)
x x + / t
dr
(28)
y x t
(29)
(30)
where T denotes the uniform distribution and 1) is set to 17 0.3. We use At = 0.1 to generate
N 600 training trajectories of length T 100, Naa 100 validation trajectories of length
Teal 100, and Nies 100 test trajectories of length Tist = 300. Test trajectories are longer to test model forecasting accuracy.
During training we set the batch size to 20. We incrementally increase the sequence length of training trajectories starting from T 10 until T 100. We perform this in 10 increments for NODE. MoNODE, HBNODE and MOHENODE, and in 2 increments for SONODE and MoSONODE. All models are trained for 1500 epochs. The model with lowest validation MSE used for evaluation on test trajectories.
Rotating MNIST For the data generation we base our code upon the image rotation implementation
provided by [Solin et al. 2021 We set the total number of rotation angles to T 16 and sample the initial rotation angle from all the possible angles d U[1, T). We generate = -1000 training trajectories with length T 16 and N 100 validation and test trajectories with length T 45.
15

---

During training we set the batch size to 25 and learning rate to 0.002. We train all models for 400 epochs and for evaluation the trained model with the lowest validation mse.
Bouncing ball with friction For data generation, we use the script provided by Sutskever et al. (2008). As a tiny update, each observed sequence has friction constant drawn from a uniform distribution UOO, 0.1]. We set the initial velocities to be unit vectors with random directions.
Table 6: Details of the datasets. Below, N denotes the number of sequences (for training, validation, and test), N₂ is the sequence length, At denotes the time between two observations, and ar is the standard deviation of the added noise.
DATASET
N
Naal
N
T
T
Tist
At
SINUSOIDAL DATA
300
50
50
50
50
150
0.1
0.2
LOTKA-VOLTERRA
600
100
100
100
100
300
0.1
0.3
ROTATING MNINST
2000
100
100
45
45
0.1
0.0
BOUNCING BALL
1000
100
100
20
20
40
0.1
0
Motion capture
The walking sequences we consider are 07 07_01.amg 08_03 amc, 35_02.ame
35_12.amc, 35 35_34. amc, 39_08. amc, 07_02. amc, 16_15. amc, 35_03. amc, 35_13.am amc,
38_01.ame 39_09. amc, 07_03. amc, 16_16. amc, 35_04. amc, 35_14. amc, a0_02.amc,
39_10.ame 07 07_06. amc, 16 16_21 amc, 35_05.amc; 35_15. amc, 39 39_01.
39_12. 39_12.amc, amc, 35_06. ame, 35_28. amc, 39_02. amc, 39_13. amc, 07_08. amc, 16_31.amc, 15_07.amc, 35_29 amc, 39_03. amc, 39_ 39_14. 29_14.amc, amc, 07 am_,,,am,, amc,
07_07 amc,
amc,
35_08.amm amc, 39_04 amc, 07_10.amc, 16_47 amc, 35_09. 35_31 amc,
39_05.amc, 07_11.amc, 16_58. amc, 35_10.amc, 35_32. amc, 39_06. 08_02. amc, 35_01. amc, 35_11 15.11.am 35 35_33.__ amc, 39_07. amc. The number of training, validation and test samples MOCAP and MOCAP-SHIFT splits are 46-5-5 and 43-5-8, respectively. Since sequences are already dense, skip every other data point. For ease of implementation, points, leading to sequences of length T 150. We experiment with different latent dimensionalities and report the findings in Table 12
take the last 300 time
16

---

C Architecture and Hyperparameter Details
Table 7: Experiment parameters. T3 the number of input frames condition the initial state on.
and T. is the number of input frames used to compute the modulator/content variables. Further, we note the dimensionality of the dynamics state yz (output of the ODESolve, where the differential equation is modeled by MI.P), well as the latent dimensionality of the dynamics modulator qa and the static qa- Ir stands for learning rate.
DATASET
MODEL
Tg
T.
4s
qd
qs
Ir
N# parameters
NODE
10
8
0.002
24666
MONODE
3 10
4
4
0,002
24598
SIN
SONODE
10
2
0.002
21802
MoSONODE
10
10
2
4
0.002
23346
HBNODE
10
8
0.002
24404
MOHBNODE
10
10
4
4
0.002
24938
NODE
40
16
0.002
28022
MONODE
8
40
8
8
0.002
26976
PREY-PREDATOR
SONODE
40
4
0.002
29204
MoSONODE
40
40
4
8
0.002
31382
HBNODE
40
16
0.002
26586
MoHBNODE
10
40
8
8
0.002
26744
NODE MoNODE
5
32
0.001
513561
ROTATING MNIST
5
15
16
16
0.001
558681
NODE
75
24
0.002
45112
MOCAP
MoNODE
75
75
8
8
8
0.002
51360
NODE
75
24
0.002
45112
MOCAP-SHIFT
MoNODE
75
75
8
8
8
0.002
51360
NODE
5
212
0,001
162343
BOUNCING BALL
MoNODE
5 5
8
4
0.001
150479
Table 8: Model architecture overview
Position
Velocity Encoder
Modulator P. Network
Differential Function
ODE Solver
Dataset
Model
Decoder
Encoder
SIN
NODE MONODE
RNN
MLP
rk4
MLP
PREY-PREDATOR
RNN
RNN
MIP
rk4
MLP
MOCAP
SONODE
MLP
MIP
rk4
MOCAP-SHIFT
MoSONODE
MLP
RNN
MLP
rk4
NODE
CNN
MLP
rk4
CNN
ROTATING MNIST
MoNODE
CNN
RNN
MLP
rk4
CNN
NODE
CNN
CNN
MLP
rk4
CNN
BOUNCING BALL
MoNODE
CNN
CNN
CNN
MLP
rk4
CNN
17

---

D
Additional Results
Figure 10: Model training time comparison on sinusoidal data. x-axis validation MSE, y-axis wall clock time in seconds. Thick line mean value across 3 different runs (different initalization seeds), shaded area standard deviation.
Table 9: Test MSE across different solvers for sinusoidal data for NODE Chen et al. MONODE (ours).
2018 and
model
seq. length
euler
rk4
dopri5
NODE
T 50
0.13( 0.03)
0.13 (0.03)
0.19( 0.09)
MONODE
0.07( 0.02)
0.04 (0.01)
0.05( 0.01)
NODE MONODE
T 150
2.48( 1.26)
1.84 (0.70)
2.11( 0.56)
0.42( 0.21)
0.29 (0.11)
0.31( 0.04)
Table 10: Ablation results for different model architectures and hyperparameters for Sinusoidal and Predator-Prey data. For final model parameters used see section app. g
Velocity
Test MSE (std)
Data
Model
Encoder
T.
T-
Niner
T = 50
T' = 150
10
0.04 (0.01)
0.29 (0.11) 0.37 (0.12)
MoNODE
10
10
10
0.05 (0.01)
MLP
5
10
2.18 (0.06)
3.05 (0.02)
SINUSOIDAL
SONODE
MLP
10
4
1,81 ( (0.01)
3.05 ( 0.07)
DATA
RNN
10
10
2.19 (0.15)
3.05 (0.07)
MLP
5
10
10
0.04 (0.00)
0.29 ( (.04)
MOSONODE
MLP
10
10
4
0.04 (0.00)
0.32 ( (0.66)
RNN
10
10
0.05 (0.01)
0.35 (0.10)
T = 100
T 300
8
40
10
0.74 (0.05)
4.33 (0.1)
MONODE
40
40
10
0.57 (0.04)
23.05 ( (1.49)
MLP
10
10
15,801 (0.748)
40.921 (0.816)
SONODE
MLP
40
2
5.130 (0,221)
39.260 (3,367)
RNN
40
10
5.101
41.890 (8.444)
PREDATOR- PREY
MLP
10
40
10
1.459 (0.284)
6.695 (0.828)
MOSONODE
MLP
40
40
2
1.093 (0.059)
6.342 (0.982)
RNN
40
40
10
1.294 (0.322)
8.639 (1.468)
RNN
40
10
0.879 (0.096)
3346.625 (2119.23)) 2339)
HBNODE
RNN
10
10
13.803 138)
2833.051 (3254.703)
MoHBNODE
RNN
40
40
10
0.870 (0.088)
225.012 (274.485)
RNN
10
40
10
0.943 (0.092)
10.205 (1.429)
18

---

Trajectory Id: 14,
Ground Truth
Trajectory id: 14
Ground Truth
Trajectory id: 10
Trajectory id:
Trajectory id:
Trajectory id:
SONODE
MoSONODE
Figure 11: Qualitative performance improvement by MoSONODE SONODE sinusoidal
data. Both models have the same architecture: MLP Velocity encoder, T- = 5, and Niner = 4. Lined area represents number of input frames used to compute initial position and velocity, while grey represent number of frames seen during training.
Toppcwar 33
HENODE
Drjjctory
Prjeectory
MM
Trijuclory id: 10
tory of 10
Plki Rock
MOHBNODE
Trijectory
Tragectory
Toqedioyy at
Trqpeitory
HBNODE
MoHBNODE
Figure 12: The left panel illustrates PCA embeddings of the dynamics state inferred by HBNODE and MOHBNODE, where each color matches one trajectory. The figures on right compare HBNODE and MOHBNODE reconstructions test trajectory sequences. Note that the models take the dashed area as input, and darker area represents the trajectory length seen during training.
Trajectrs
Trujedery 10.9 WVb
Trepecery
NODG
Trojjetoory
Trajeetocy id.
Lrapectory ad.
roin aml
Tejesary 9
Tegectery
MoNODE
Trajectocy
NODE
McNODE
Figure 13: The left panel illustrates PCA embeddings the dynamics state inferred by NODE and MoNODE, where each color matches one trajectory. The figures right compare NODE and MoNODE reconstructions on test trajectory sequences. Note that the models take the dashed area as input, and darker area represents the trajectory length during training. Prey-Predator data.
19

---

Praaectoryy id:
Ground Truth
Trajectory id: 9
Ground Truth
Trajectory id:
Trajectory
Projectory id:
Trajectory
SONODE
MoSONODE
Figure 14: Qualitative performance improvement by MoSONODE on SONODE Predator-Prey
data. Both models have the same architecture: MLP Velocity encoder, T, = 40, and Nincr 2. Lined area represents number of input frames used to compute initial position and velocity, while grey area represent number of frames seen during training.
Figure 15: NODE and MONODE predictions on five bouncing ball sequences. For each couple of plots, the first and second rows correspond NODE and MONODE predictions, respectively. The dark circles are the ground truth ball positions and the colored background is the model prediction. As can be seen, MONODE tracks the ball much better compared NODE baseline.
20

---

Figure 17: (left) TSNE embeddings of the static modulator as inferred by MoNODE on test data for roating MNIST. Each color represent one of the possible 10 digits. For digits that have distinct shape, such as "O" and "1", we can discern clear clustering and separation from the other digits, while for the remaining digits the per digit clusters exhibit more overlap. This is to be expected as we not constraint the model learn spatially separated clusters, as well as some digits are more similar either due to their appearance when rotated, such "6" and rotated "9", or due to their writing style. (right) For further illustration, we repeat the same experiment with dynamics Rotating MNIST reconstruction NODE
(b) Rotating MNIST reconstruction MONODE
Number Narthe Number
Number Number Number
POA 800
PSk-1 (023)
POL1 (035)
(c) PCA embeddings NODE
(d) PCA embeddings MONODE
Figure 16: Top two figures are (a) NODE and (b) MONODE reconstruction of different test trajectories, where top line is the ground truth for every digit. Bottom row illustrates the correspond- ing PCA embeddings of each trajectory plotted above, where (c) corresponds to NODE and (d)
corresponds to MONODE.
Static modulator TSNE emboddings
Dynamics modulator TSNE embeddings
2
3
6
-63
modulators instead of static. Unlike static modulator latents, for dynamics modulator topology in the latent space is not present. In all, the results confirm that the modulator variables are correlated with the true underlying factors of variation
21

---

aa) Rotating MNIST reconstruction NODE
(b) Rotating MNIST reconstruction for MONODE
Number
(030)
(030
(c) PCA embeddings NODE
(d) PCA embeddings MONODE
Figure 18: Failure cases on Rotating MNIST for (a) NODE and (b) MONODE reconstruction of 2 different test trajectories, where top line is the ground truth for every digit. Bottom row illustrates the corresponding PCA embeddings of each trajectory plotted above, where (c) corresponds to NODE and (d) corresponds to MONODE. Despite MONODE failing to reconstruct the correct shape of the digit the latent PCA embedding plots reveal that the model has still captured the circular rotation motion.
Table 11: Ablation studies on Rotating MNIST data to compare different parameter affects on NODE and MONODE, As can be noted the MSE for reconstruction within training sequence length T=15 is lower for NODE than for MONODE, while for forecasting MONODE outperforms NODE. This discrepancy can be explained by the fact that the current results indicate that MONODE harder to optimize as it's test mse can be grouped in two modes: success cases were the reconstructions are great and the error remains low for forecasting o6 failure cases where the model fails reconstruct already within the training sequence length T=15 as in fig [8] This explains the higher test MSE = 15 and remaining almost unchanged for T 45, while for NODE the test MSE grows as the sequence length increases.
MONODE
#N parameters
Test MSE (std)
model
4.
T 15
T 45
NODE
16
461161
0.020
0.098
6
10
513637
0.031
0.034
MONODE
8
516089
0.031
0.032
NODE
24
487361
0.014
0.062
8
16
532481
0.030
0.031
MONODE
12
12
537385
0.035
0.038
NODE
32
513561
0.013
0.042
548873
0.035
0.037
MONODE
12
20
553777
0.033
0.033
558681
0.031
0.032
22

---

Table 12: Ablation studies on mocap data to compare NODE and different variants of our model (with/without dynamics modulator and content variable). We also experiment with different latent dimensionalities.
||
9=
Qa
9s
MOCAP
MOCAP-SHIFT
6.1
81.6 = 39.5
NODE
12
71.2 + 20.0
63.1 + 8.0
24
72.2 12.4
61.6 6.2
3
76.4 + 12.7
72.6 6.2
6
79.7 : 14.4
83.2 = 23.0
66.7 + 12.3
78.9 = 12.7
3
72.6 7.9
67.0 18.4
MONODE
6
6
64.6 64.6+990 9.3
63.9 + 16.2
12
12
60.0 + 17.6
62.5 + 11.0
2
2
79.2 1 8.7
56.3 7.8
78.8 + 16.7
57.0 7.8
8
8
57.7 9.8
58.0 = 10.7
23


Reviews:
[{"review_id": "1c84AQF4gj", "reviewer": "Reviewer_dvMp", "summary": "The paper propose to improve long term forecasting abilities of NODEs by incorporating the dynamics with learnt static parameters. The system is composed of a static modulator and a dynamic modulator, the final solution is sampled from a distribution conditioned on the static modulator and the dynamic generated using the dynamic modulator. Empirical results suggests improvements over NODE on time simple series datasets.", "questions": "How are the networks trained? There are so many different networks. Also, how is this method significantly different from existing literature?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "Presentation is clear, and the paper is well written.\n\nThe empirical improvements on time series data is significant on the long run.", "weaknesses": "Similar work has been proposed, see https://arxiv.org/abs/2302.13262 for an example. Although the methods are slightly different, the fundamental idea of incorporating with static parameters are the same. \n\nThere are no clear theoretical guarantees that the method will work better.\n\nThe network structure is very complicated, the reviewer is worried about the training difficulties. It would be helpful to see some ablation and show it is easy to train.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work. Please find below our response to the concerns voiced. \n> How are the networks trained? There are so many different networks. \n\nWe have a simple feedforward architecture,  which we can train easily end-to-end, no more complex than NODE [1] or ODE2VAE [2]. Specifically, MoNODE optimizes ELBO like NODE, that is the optimization of the additional modulator encoder networks is implicit, with no extra loss terms:\n\n$\\textup{log} \\ p(X)  \\geq \\textup{ELBO} = \\sum_n E_{q_{\\nu}} \\left[  \\textup{log} \\ p\\big(x_{1:T}^{n} \\mid z_1^{n} \\big)  \\right] - \\textup{KL}\\left[ q_{\\nu}( z_1^{n}\\mid x^n_{1:T_z}) || p(z_1) \\right].$\n\nWe would like to clarify that there are only 3 networks in the proposed method: two of which are the standard building blocks of a VAE (encoding, decoding network) as in [1] and the third, our addition, the modulator prediction network. We explicitly name the networks to clarify the intuition, following the narrative style of ODE2VAE [2].\n\n[1] Chen, R. T., Rubanova, Y., Bettencourt, J., \\& Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31.\n\n[2] Yildiz, C., Heinonen, M., \\& Lahdesmaki, H. (2019). ODE2VAE: Deep generative second order ODEs with Bayesian neural networks. Advances in Neural Information Processing Systems, 32.\n\n > It would be helpful to see some ablation and show it is easy to train.\n\nThe proposed model is very easy to train, and not sensitive to hyperparameters. Our model performs consistently well under different hyperparameterization:\n\n(i) the solver used (Table 9., Supp. mat. D); \n\n(ii) the number of input frames $T_z$ used (Table 10., Supp. mat. D); \n\n(iii) the latent dimensionality of the latent ode state $q_z$ and modulator variable $q_s$ (Table 11., Supp. mat. D). \n\nIn addition, to showcase that our proposed framework is easy to train we have plotted the validation MSE versus wall clock time during training for sinusoidal data, please see Figure 1 (additional PDF, global response). As it as apparent from the figure, our framework is easier to train than all baseline methods. We realize that these ablations were not easily accessible, so we will move a highlight of them to the main paper.\n\n> no clear theoretical guarantees\n\nSimilarly to earlier works in this area of research like ODE2VAE [2], LatSegODE [3] and Neural Event ODE [4], we introduce an additional structure on top of the flexible neural ODE model class without a theoretical investigation. Nonetheless, we show empirically that in practice our framework brings a significant accuracy improvement with an average of 55.25\\% across all experiments, and a high of 88.38\\% improvement on PP dataset. We leave a theoretical study as a fantastic next step for future work, to explore the limits of our ideas, especially with respect to the complexity of the static signal.\n\n[3] Shi, R., & Morris, Q. (2021, July). Segmenting hybrid trajectories using latent odes. In International Conference on Machine Learning (pp. 9569-9579). PMLR.\n\n[4] Chen, R. T., Amos, B., & Nickel, M. (2020). Learning neural event functions for ordinary differential equations. arXiv preprint arXiv:2011.03902.\n\n> Also, how is this method significantly different from existing literature?\n\nWe compare with a great array of state-of-the-art methods, a visual summary is in Table 1 in the main paper. Our method is significantly different from the existing methods as \n\n(i) we introduce modulator variables that allow the model to preserve core dynamics while being adaptive to modulating factors. This, in turn, allows us to model underlying factors of variation that are fixed in time.\n\n(ii) our framework addresses a common limitation of existing NODE type models: poor long-term forecasting performance.\n\n(iii) our modulator variables are informative of the true unknown factors of variation as measured by $R^2$ scores. \n\n(iv) it is a *general* framework that can be combined with a large variety of existing NODE type models, bringing on average across all experiments 55.25\\% accuracy improvements.\n\n> similar work has been proposed, see https://arxiv.org/abs/2302.13262\n\nWe do appreciate your knowledge of the related work, even of its newest additions. As you might have noticed, this arxiv submission is very related to ours and we are very aware of it. On the other hand, the paper isn't published yet, and we have very good reasons not to cite it.\n\n"}, {"review_id": "DNWto08kSd", "reviewer": "Reviewer_8J7y", "summary": "Existing neural ordinary differential equations (NODEs) present the problem of variation across trajectories only through initial state values. As a solution to the problem, Modulated Neural ODE (MoNODE), which improves the existing NODE method by distinguishing dynamic and static states, is presented. In MoNODE, 1) static modulators and 2) dynamics modulators are used to divide dynamic variables from time-invariant variables. By attaching these two modulators to existing NODE studies, they show good performance in various datasets.", "questions": "In experiments 5.1 and 5.2, static variable and dynamics variable data were separately tested to evaluate each static modulator and dynamics modulator. Was the dynamics modulator also applied to static variables in these experiments? If the dynamics modulator was used even in the static variable experiment, you should remove the dynamics modulator.\n\nIf you did not use dynamics modulator in static variable experiment, you should experiment using dynamics modulator. In other words, through these experiments, it is necessary to check whether each modulator is playing a role.\n", "limitations": "There is no limitations", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. They presented the problem of handling static variables well in NODE, and suggested dynamics modulator and static modulator as a solution. This is very intuitive.\n\n2. It is very attractive that it can be applied to any NODE model and does not have to be divided into static and dynamic variables in advance.\n\n3. They showed the performance using various datasets to evaluate dynamic and static modulators.", "weaknesses": "1. They experimented by dividing 5.1 dynamics modulator variable and 5.2 static modulator variable data to evaluate their modulators, but they could not show whether each modulator actually played its role.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review, please find below our response to the concerns voiced.\n> Was the dynamics modulator also applied to static variables in these experiments? If the dynamics modulator was used even in the static variable experiment, you should remove the dynamics modulator. If you did not use dynamics modulator in static variable experiment, you should experiment using dynamics modulator. In other words, through these experiments, it is necessary to check whether each modulator is playing a role.\n\nTo confirm the role of each modulator variable we have performed 2 additional ablations with the MoNODE framework on: (a) Sinusoidal data with static modulator instead of dynamics; and (b) rot. MNIST with dynamics modulator instead of static.  We report the test MSE across 3 different initialization runs with standard deviation. For Sinusoidal data for MoNODE + static mod. the test MSE performance drops to 2.68 (0.38) from MoNODE + dyn. mod. 0.29 (0.11). For Rot. MNIST MoNODE + dyn. mod. the performance drops to 0.096 (0.008) from MoNODE + static mod. 0.030 (0.001). In addition, we examined the latent embeddings of the dynamics modulator for Rot. MNIST. Where previously for the content modulator we observed clusters corresponding to a digit's class (Fig. 16, Supp. mat. D), for dynamics modulator such a topology in the latent space is not present, see Fig. 2 (additional PDF, general response). In all, the results confirm that the modulator variables are correlated with the true underlying factors of variation as indicated by the obtained $R^2$ scores (Table 3., main paper) and latent embedding plots (Fig. 16, Supp. mat. D), and, as such, play their corresponding roles.  "}, {"review_id": "ASDIrjSDg3", "reviewer": "Reviewer_VxYT", "summary": "The authors introduce Modulated Neural ODEs by incorporating a static modulator that averages observation embeddings across the whole time and a dynamics modulator that averages observation embeddings by time subsequences. The proposed model consistently improves the existing model's ability to generalize to new dynamic parameterizations.", "questions": "See weaknesses.", "limitations": "The authors have already discussed the limitations of this work, e.g., the current exploration is only limited to deterministic systems and the performance of the work is largely affected by the base model’s performance.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The presentation is pretty straightforward and the idea is simple yet effective, as shown by experiments.", "weaknesses": "- As the output of the static modulator and the dynamics modulator is used by concatenation, I would like to see the experimental results without the static modulator or the dynamics modulator to determine if they are both essential.\n- I would like the authors to add the inference time cost comparison with and without those modulators.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work, please find below the response to the concerns voiced. \n\n> As the output of the static modulator and the dynamics modulator is used by concatenation ...\n\nWe never concatenate static and dynamic modulators. They are additional inputs to the decoder and differential function, respectively. We do concatenate the dynamic state $z$ and each modulator variable separately.\n\n> ... I would like to see the experimental results without the static modulator or the dynamics modulator to determine if they are both essential.\n\nWe do not always use both modulator variables, but rather the one(s) that match the problem: \n\n(i) for low-dimensional problems with changing dynamic's parametarization we use **only** dynamics modulator (Section 5.1)\n\n(ii) for high-dimensional problems with changing object features, but fixed dynamics, we use **only** static modulator (Section 5.2) \n\n(iii) for problems that have both, changing dynamic's parametrization and changing object features, we use **both** modulator variables (Section 5.3)\n\nThat said, we investigated the utility of using both modulator variables versus only one for Mocap (Section 5.3). Using both modulator variables resulted in the lowest test MSE as compared to using only one, either static or dynamics modulator, as reported in Table 12 (Supplementary material D, Section 'Additional results'). This performance gap can be attributed to the data, which contains both static (length of limbs) and dynamics (length of legs) modulating signals, therefore, both variables are needed for optimal performance. \n\nIf for problems (i) and (ii) the corresponding modulator is switched to the opposite ((i) Sinusoidal data with static modulator instead of dynamics; and (ii) rot. MNIST with dynamics modulator instead of static) we see a drop in performance with respect to test MSE. Specifically, for Sinusoidal data for MoNODE + static mod. the performance drops to 2.68 (0.38) from MoNODE + dyn. mod. 0.29 (0.11). For Rot. MNIST MoNODE + dyn. mod. the performance drops to 0.096 (0.008) from MoNODE + static mod. 0.030 (0.001). In all, this confirms that the modulator variable selection is essential, depends on the problem at hand and our framework can accommodate multitude of problem set-ups. \n\n> I would like the authors to add the inference time cost comparison with and without those modulators.\n\nWe compute the inference time cost for the Sinusoidal data experiment, where the test data consists of 50 trajectories of length 150. We record the time it takes NODE and MoNODE to predict future states while conditioned on the initial 10 time points. We perform the evaluation 10 times and report the mean (in seconds) and std. The inference time cost for NODE is 0.312 (0.050), while for MoNODE is 0.291 (0.040). We are faster because the number of parameters for MoNODE is lower than for NODE, 24598 versus 24666, respectively. As such, we would like to highlight that the inference time costs of our framework is on par with the base models where the exact time is dependent on the total number of parameters, which is comparable between all base and Mo*NODE models (Table 7., Supp. mat. C). We will add these results to the final version. \n    "}, {"review_id": "INMigJhgKU", "reviewer": "Reviewer_m2DA", "summary": "The paper proposes an approach for generalizing NeuralODE to different system parameters, in the setting of a single dynamical system with different parameters. A dynamic and static modulators are applied to specific parts of the encode-process-decode architecture: the dynamic modulator acts on the NeuralODE part, while the static modulator only acts on the decoder. They are learned in an end-to-end manner.\nEvaluation is performed on synthetic datasets, as well as a real dataset (walking humans), by using different NODE variants. It showcases the generalization capabilities of the 2 modulators.\nThe paper’s idea is interesting, and I found the experiments are convincing.", "questions": "- Is there a way the modulations spaces could be used for interpolation between different systems?\n- Could this work be extended to process PDE equations?", "limitations": "The work only studies dynamical systems up to 3 dimensions, would it be able to scale to higher dimensions?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework is easy-to-integrate to existing NODE methods to increase their performance.\n- The latent space is more interpretable than with vanilla NODE, and provides high correlation wrt the dynamics parameters\n- The proposed method drastically improves the used NODE methods for long-range prediction", "weaknesses": "- ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review, please find below our response for the questions voiced.\n\n> Is there a way the modulations spaces could be used for interpolation between different systems?\n\nThank you for the interesting question. It touches upon a possible Bayesian extension of the presented work. In particular, in the current set-up the modulator variables are point estimates, however, this could be adjusted to learning possibly multi-modal modulator variable distributions. Ideally, each mode of the inferred posterior could correspond to a particular system (e.g., a handful of particles with no/weak/strong interactions). That being said, the completely black-box nature of our framework could make it difficult to interpret/identify modulators and therefore the systems.\n\n> Could this work be extended to process PDE equations?\n\nBased on the recent work on neural PDEs [1], it is possible to parameterize the systems of ODEs resulting from spatial discretization of a PDE in the same way as our framework. An interesting research question would be, for instance, learning modulators as a function of spatial coordinates. \n\n[1] Learning Space-Time Continuous Neural PDEs from\nPartially Observed States. Iakovlev, Heinonen, Lähdesmäki, 2023."}, {"review_id": "ryDYx1VsWB", "reviewer": "Reviewer_Y7gY", "summary": "This paper proposes a framework for Neural ODEs called Modulated Neural ODEs, which combines the conventional latent Neural ODEs with two Modulator Prediction Networks that are used to extract the time-invariant static and dynamic modulator variables, respectively. Their empirical results demonstrate that this framework can improve generalizations to new dynamic parameterization and long-time forecasting, and can better capture the true unknown factors of variation as well.", "questions": "1.\tWhen comparing with NODE, could you provide a comparison of the number of parameters and set up a more reasonable experiment to prove that the performance improvement is not caused by the increase in the number of parameters?\n\n2.\tWhether the MoNODE framework can outperform the conventional NODE on the task of learning more complex dynamical systems, such as trajectories generated by a family of chaotic systems controlled by different parameters?\n\n3.\tFor the CMU Mocap datasets, due to the addition of the modulator variables, the MoNODE is supposed to significantly improve the performance of NODE in the Mocap-Shift task, but in fact, the improvement effect is not obvious, which is far less than that of Mocap. Can you further explain this phenomenon?\n\n4.\tCould you directly provide some visual description and intuitive interpretation of the modulator variables extracted by the modulator prediction networks?\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe proposed MoNODE framework has better capability of generalizing to new dynamic parameterizations and long-term forecasting.\n\n2.\tThey experimentally verify that the MoNODE framework is able to better capture the true unknown factors of variation with modulator variables.\n", "weaknesses": "1.\tExperimental examples are not convincing enough. The dynamical systems chosen in this paper are relatively simple and naive. For more complex systems such as trajectories generated by a family of chaotic systems controlled by different parameters, it is doubtful whether the MoNODE framework can still extract effective modulator variables.\n\n2.\tThough NODE as the baseline group maintains the same dimension as MoNODE, additional parameters are still introduced in MoNODE, such as parameters in the Modulator Prediction Networks. It is not clear whether the slight improvement in performance is due to the additional number of parameters.\n\n3.\tThis paper does not directly provide a visual presentation or intuitive explanation of the static and dynamics modulator variables extracted by the modulator prediction networks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review. Please find below our response to the concerns voiced. \n\n> Experimental examples are not convincing enough.\n\nWe are slightly surprised and sorry to hear that our findings fail to convince the reviewer. We exhaustively test our ideas on four NODE variants (standard, SONODE, HBNODE, LSONODE) on five datasets that were all previously used to evaluate similar approaches. Our Mo*NODE framework improved by 88.38\\% test MSE for PP experiment and on average by 55.25\\% across all experiments, as reported in Table 2. and Table 4. That being said, we would be happy to know what other experiments would help us convince the reviewer.\n\n> When comparing with NODE, could you provide a comparison of the number of parameters and set up a more reasonable experiment to prove that the performance improvement is not caused by the increase in the number of parameters?\n\nThank you for voicing this concern, as indeed one candidate explanation for the observed performance improvement could be increased parameter count. To account for this fact we approximately match the number of parameters in the base network and the Modulated counterpart. For a complete overview of the number of parameters of each model for each experiment please see Table 7 in the supplementary material, Section C, Architecture and Hyperparameter Details. In particular, for sin, PP, and BB, NODE model actually has a higher parameter count than its MoNODE counterpart (We also control overfitting by choosing the best model via cross-validation). Furthermore, to also account for the fact that the performance boost is not caused just by a larger latent dimensionality, we match the latent dimensionality between the base model and the modulated counterpart. As such, we are confident to say that the observed performance boost is not caused simply by an increase in number of parameters, but rather by the framework itself. \n\n> Whether the MoNODE framework can outperform the conventional NODE on the task of learning more complex dynamical systems, such as trajectories generated by a family of chaotic systems controlled by different parameters?\n\nWe perform an additional ablation on a Lorenz Attractor dataset, with varying parameters as suggested. MoNODE outperforms NODE on both short- and long-horizon forecasting. Specifically, the test MSEs for NODE and MONODE are 36.19 vs 33.90 for short term and 61.28 vs 46.56 for long term. We would like to remind that NODEs are known to suffer from solver stability issues on chaotic systems [1].\n\n[1] Allauzen, A., Dardis, T. P. M., & Plath, H. (2022). Experimental study of Neural ODE training with adaptive solver for dynamical systems modeling. arXiv preprint arXiv:2211.06972.\n\n> For the CMU Mocap datasets, due to the addition of the modulator variables, the MoNODE is supposed to significantly improve the performance of NODE in the Mocap-Shift task, but in fact, the improvement effect is not obvious, which is far less than that of Mocap. Can you further explain this phenomenon?\n\nThis outcome can be attributed to the chosen data split. In particular, we chose to leave a particular subject out for testing since the other subjects had either too many or too few sequences (notice that leaving out a subject with too many/few sequences would lead to a huge decrease in the training data size or biased testing). Thanks for raising your concern, we will include a detailed explanation for this situation in our paper.\n\n> Could you directly provide some visual description and intuitive interpretation of the modulator variables extracted by the modulator prediction networks?\n\nDynamics modulators: intuitively they capture information about the true underlying differential equation parameters (experiments sin, PP, BB) or in the case when the dynamic's function in closed form is unknown (Mocap), then the parameters that affect this unknown function (length of legs). \n\nTo confirm that this intuitive explanation indeed holds, we computed the $R^2$ scores by performing regression from latent variables to the true parameters, as described in Schott et al. 2021. The regression input for MoNODE are the dynamics modulators, while for NODE it is the latent trajectories. For all experiments in Section 5.1, sinusoidal, predator-prey and bouncing ball with friction, our proposed framework obtains a higher $R^2$ score (a score of 1 corresponds to perfect regression) implying that the learned dynamics modulators have captured information of the underlying parameters of the system.\n\nStatic modulators: intuitively they capture static characteristic features of the data, in the case when the data is image sequences, for example. In this case, the static modulator would capture the color, shape and other static features of the moving object.\n\nTo assess whether this assumption is true we evaluate the learned vector space of the static modulator for the rotating MNIST experiment. In particular, in Figure 16., Supplementary material D, Additional results, we see that, even though we have not explicitly enforced any topology in the latent vector space, we can observe a clustering of the data points corresponding to a given digit style (0 versus 1 for example). This qualitative observation indicates that the static modulator has captured information of the underlying characteristic of the observations. Albeit the separation of the clusters is not perfect, the explanation of this is two-fold: visually examining the data points it can be seen that some digits from different classes look very similar due to the handwriting, and we have not enforced in the latent space that each digit class should be embedded as far as possible from each other, hence the observed overlap is reasonable. \n\nWe will provide a sufficient visual description in the final version of the paper.\n\n"}, {"review_id": "ThPdislolr", "reviewer": "Reviewer_Znd9", "summary": "This paper proposes a modulated neural ODE (MoNODE) that can separately model the dynamics and time-invariant modulating variables (e.g., parameters of dynamics or contents of agents).  Such modulating variables are modeled by averaging the embedded trajectories of given observations (i.e., the outputs of modulator prediction networks for a observed trajectories) with respect to the time, similar to the tranformation-invariant (TI) pooling approach. The proposed approach can be integrated into various NODE models simply. The authors validate the proposed MoNODE for various dynamical systems, including some synthetic examples (sinusoidal, Lotka–Volterra Hamiltonian, bouncing ball), images dynamics (rotated MNIST), and CMU motion capture dataset. The experimental results demonstrate the proposed MoNODE is better than NODE in terms of the interpretablility well as long-term predictability of dynamics.\n\n***\n\nThis paper proposes a concept of separating and learning dynamics and time-invariant modulator variables, and I believe it could be beneficial for various time-series prediction problems in scientific and engineering fields. However, I have doubts about whether the proposed model maintains the time-invariance property explicitly. Additionally, there are some concerns regarding comparisons with similar papers and validation in more complicated chaotic systems. Therefore, I currently consider this paper to be at a borderline reject.", "questions": "- I think it would be great to experimentally demonstrate that a trained modulating prediction network indeed guarantees time-invariance in the long-term time horizon. For example, it would be helpful to plot a graph to visualize the change in $d$ or $s$ over time (ideally to be a constant line).\n \n- Is the performance of the trained MoNODE still good when $T_z = T_e \\ll T_{train}$? It seems that an ablation study is needed to check this.\n\n- I am curious whether MoNODE would be applicable to chaotic dynamics. For instance, one could consider the parameters ($\\sigma$, $\\rho$, $\\beta$) of the Lorenz system as dynamic modulating variables. Can MoNODE effectively learn and infer them?\n\n- Max pooling [2] is commonly used as the pooling function for invariant mappings (e.g., $s^n = \\max s_{i}^{n}$ for the static modulator case). Is there a specific reason for using averaging? It would be nice if the authors could conduct a comparison between max and average pooling invariant maps.\n\n***\n\n[2] Laptev, D., Savinov, N., Buhmann, J. M., & Pollefeys, M. Ti-pooling: transformation-invariant pooling for feature learning in convolutional neural networks. CVPR 2016.", "limitations": "The authors mention some limitations (limited to deterministic systems, generalization to out-of-distribution modulators, ...) of the proposed method in the Discussion section. Another limitation worth mentioning is the weakness section of my review: not entirely guaranteed time-invariance, limited application when $T_z < T_e$, and the missing relevant reference.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach is simple and principled, making it easy to integrate into various NODE-type models.\n\n- The paper is well-written and easy to follow.\n\n- The experimental results demonstrate that the learned latent dynamics of MoNODEs are more convincing and interpretable compared to those of NODEs. This may be attributed to the explicit decomposition of the time-invariant contents of agents from the dynamical behavior itself.", "weaknesses": "- The time-invariance of the moduator prediction network is only gauranteed for the finite and fixed time horizon of $T_e$. As a result, it is not entirely clear whether the proposed model can effectively capture the time-invariant aspects of observations over an arbitrary time span.\n\n- Similarly, the proposed model requires the observation of length $T_e$ to predict the time-invariant modulating variables. Therefore, it is not greatly applicable for the case when $T_z < T_e$.\n\n- The paper lacks a comparison with a relevant previous work [1], which factorizes the latent space into content and motion variables and models the dynamics using the (Hamiltonian) latent NODE.\n\n\n***\n\n[1] Khan, A., & Storkey, A. J. Hamiltonian latent operators for content and motion disentanglement in image sequences. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review, please find below our response to the concerns voiced. \n> I think it would be great to experimentally demonstrate that a trained modulating prediction network indeed guarantees time-invariance in the long-term time horizon. For example, it would be helpful to plot a graph to visualize the change in or over time (ideally to be a constant line). \n\nWe would like to clarify that the modulator prediction network does not guarantee time-invariance with respect to the predicted dynamic state $z$, but rather the modulator variables $d$ and $s$. Our proposed framework accumulates prediction error as commonly known with auto-regressive frameworks, yet minor compared to the existing base models (NODE, SONODE, HBNODE, LSONODE) as shown in Figures 2-3 and Figures 10-15 (Supp. mat. D) and indicated by the accuracy improvement with respect to test MSE: Sin by 84.47 \\%, PP by 88.38 \\%, BB by 20.12 \\%, Rot.MNIST by 23\\%, and MOCAP by 20.08\\%.\n\n> Is the performance of the trained MoNODE still good when $T_z = T_e << T_{train}$ It seems that an ablation study is needed to check this.\n\nIn general, the choice of $T_e$ (number of timepoints used for learning modulator variables) is independent of $T_{train}$ (total training sequence length) but rather depends on the underlying dynamical system's properties. Specifically, a subsequence of length $T_e$ has to have sufficient information of the change in dynamics to correctly estimate the system parameters. Hence, if the underlying dynamics are changing very slowly, a higher number of timepoints (larger $T_e$) are needed to estimate the underlying parameters.\nSecondly, the performance of MoNODE is good when $T_z = T_e$. As reported in Table 7 (Supp. mat., C) for most datasets these two parameters are matched and additional ablations are reported in Table 10 (Supp. mat. D). \nLastly, we have paid detailed attention to match $T_z$ of NODE with $T_e$ of Mo-*NODE to make sure that the performance boost is not simply caused by a longer input sequence. \n\n> similarly, the proposed model requires the observation of length $T_e$ to predict the time-invariant modulating variables. Therefore, it is not greatly applicable for the case when $T_z < T_e$.\n\nOur framework is applicable to settings when $T_z < T_e$. Please see Table 10. (Supp. mat. D), where our framework outperforms base models also when $T_z < T_e$. \n\n> I am curious whether MoNODE would be applicable to chaotic dynamics. For instance, one could consider the parameters of the Lorenz system as dynamic modulating variables. Can MoNODE effectively learn and infer them?\n\nWe perform an additional ablation on a Lorenz Attractor dataset, with varying parameters as suggested. MoNODE outperforms NODE on both short- and long-horizon forecasting. Specifically, the test MSEs for NODE and MONODE are 36.19 vs 33.90 for short term and 61.28 vs 46.56 for long term. We would like to remind that NODEs are known to suffer from solver stability issues on chaotic systems [4].\n\n[4] Allauzen, A., Dardis, T. P. M., & Plath, H. (2022). Experimental study of Neural ODE training with adaptive solver for dynamical systems modeling. arXiv preprint arXiv:2211.06972.\n\n> Max pooling [2] is commonly used as the pooling function for invariant mappings (e.g., for the static modulator case). Is there a specific reason for using averaging? It would be nice if the authors could conduct a comparison between max and average pooling invariant maps.\n\nWe performed an additional ablation with Max pooling [2] operator instead of averaging for the static modulator on the rotating MNIST experiment and for the dynamics modulator on the sinusoidal data experiment. We report the obtained test MSE across 3 runs with different initialization seeds and corresponding standard deviation. For MoNODE + max pooling on rotating MNIST we obtain test MSE of 0.035 (0.004), which is  better than the baseline model NODE 0.039 (0.003) but is worse than MoNODE + averaging 0.030 (0.001). For MoNODE + max pooling on sinusoidal data we obtain test MSE of 0.28 (0.03), which is on par with MoNODE + averaging 0.29 (0.11) and better than the base model 1.84 (0.70). We chose the averaging operator in our work as modulator variables should capture features that remain constant in time, however, other functions are also a viable option, for example, one could also consider summation [3].\n\n[3] Franceschi, J. Y., Delasalles, E., Chen, M., Lamprier, S., & Gallinari, P. (2020, November). Stochastic latent residual video prediction. In International Conference on Machine Learning (pp. 3233-3246). PMLR.\n\n>  The paper lacks a comparison with a relevant previous work [1], which factorizes the latent space into content and motion variables and models the dynamics using the (Hamiltonian) latent NODE. ([1] Khan, A., \\& Storkey, A. J. Hamiltonian latent operators for content and motion disentanglement in image sequences. NeurIPS 2022.)\n\nThe code base for [1] unfortunately is incomplete. However, in Section 3.5 of [1] they also perform an experiment on Rotating MNIST (single digit), where ODE2VAE outperforms their method. In our work, we also compare our framework to the non-Bayesian version of ODE2VAE under the name of LSONODE (Section 5.1). Therefore, we perform an additional ablation of ODE2VAE performance on our implementation of Rotating MNIST experiment (all digits) with MoODE2VAE. We report the mean across 3 runs with std. The test MSE for ODE2VAE is 0.068 (0.002), while MoODE2VAE obtains 0.035 (0.002) outperforming ODE2VAE by a large margin. As such, we are confident to say that our method improves the current state-of-the-art and we will contact authors of [1] to include a complete comparison in the camera ready version. "}]

Author rebuttal:
We sincerely appreciate your time and effort in providing valuable feedback. Since our findings and core ideas haven't resonated with you as much as they do with us, we'd like to take this opportunity to first try to better convince you of the overall approach and significance of our work. For specific comments, we have addressed them in the respective "rebuttal" sections.

**Why modulators:** 
Standard NODEs and subsequent works aim to learn a single dynamics function that has a single latent ODE state.  In practice, however, different observations might have different fixed underlying parameterization of the dynamics function or different observed characteristics. Whereas standard methods can only model this variation through a **changing** latent ODE state resulting in a complicated dynamics function that fails to extrapolate (see Fig 2-4, Table 3), Modulated NODEs explicitly accounts for this variation through the time-invariant modulator variables. 

**Benchmarking:** 
We assessed our method on five established NODE benchmarks, enhancing their intricacy by sampling all parameters, incorporating friction, utilizing all digits, adding more walking sequences, and so on. It was striking to observe how easily prior methods, including the state-of-the-art HBNODE, collapsed due to these minor dataset modifications. 
We believe that this highlights a general limitation of earlier methods and affirms the importance of our approach.


**Findings:**
Our method on average improves long-term forecasting test MSE by 55.23\% across all baselines and datasets. Moreover, the modulator variables' latent representation offers insights into the true unknown factors of variation: (i) they align closely with the ground truth factors as measured by $R^2$ (Table 3), (ii) they exhibit a similar topology to the observation space (Figure 16, Supp. mat. D). As a consequence, also the latent ODE state is more consistent with the true underlying dynamics: (i) the amplitudes of latent trajectories heavily correlate with the observed sequences (Figures 2-3), (ii) the rotating digit sequences overlap as expected (Fig 4).
In conclusion, our method demonstrably excels at identifying meaningful representations and offers an elegant approach that opens new avenues in dynamic representation learning, given its wide applicability.

**Applicability:**
Our intent was to propose a "simple-yet-effective" solution, enabling easy integration into various NODE models. The empirical results on four NODE variants attest to the immediate advantages our modulator variables offer. We would be pleased if you agree that the simplicity and versatility of our method warrants positive recognition.

Meta-review: Six experts reviewed this paper with mixed scores. The area chairs agree that this work makes a very important contribution by introducing new Modulated Neural ODEs. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper
Decision: Accept (poster)