Title: Modulated Neural ODEs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural ODEs

Modulator Variables

Dynamical Systems

Disentanglment
Abstract: Neural ordinary differential equations (NODEs) have been proven useful for learning non-linear dynamics of arbitrary trajectories. However, current NODE methods capture variations across trajectories only via the initial state value or by auto-regressive encoder updates. In this work, we introduce Modulated Neural ODEs (MoNODEs), a novel framework that sets apart dynamics states from underlying static factors of variation and improves the existing NODE methods.  In particular, we introduce *time-invariant modulator variables* that are learned from the data. We incorporate our proposed framework into four existing NODE variants. We test MoNODE on oscillating systems, videos and human walking trajectories, where each trajectory has trajectory-specific modulation. Our framework consistently improves the existing model ability to generalize to new dynamic parameterizations and to perform far-horizon forecasting. In addition, we verify that the proposed modulator variables are informative of the true unknown factors of variation as measured by $R^2$ scores.

Modulated Neural ODEs

Abstract

Neural ordinary differential equations (NODEs) have been proven useful for learning non-linear dynamics of arbitrary trajectories. However, current NODE methods capture variations across trajectories only via the initial state value or by autoregressive encoder updates. In this work, we introduce Modulated Neural ODEs (MoNODEs), a novel framework that sets apart dynamics states from underlying static factors of variation and improves the existing NODE methods. In particular, we introduce time-invariant modulator variables that are learned from the data. We incorporate our proposed framework into four existing NODE variants. We test MoNODE on oscillating systems, videos and human walking trajectories, where each trajectory has trajectory-specific modulation. Our framework consistently improves the existing model ability to generalize to new dynamic parameterizations and to perform far-horizon forecasting. In addition, we verify that the proposed modulator variables are informative of the true unknown factors of variation as measured by R 2 scores.

1 Introduction

Differential equations are the de facto standard for learning dynamics of biological [Hirsch et al., 2012] and physical [Tenenbaum and Pollard, 1985] systems. When the observed phenomenon is deterministic, the dynamics are typically expressed in terms of ordinary differential equations (ODEs). Traditionally, ODEs have been built from a mechanistic perspective, in which states of the observed system and the governing differential equation with its parameters are specified by domain experts. However, when the parametric form is unknown and only observations are available, neural network surrogates can be used to model the unknown differential function, called neural ODEs (NODEs) Chen et al. [2018]. Since its introduction by Chen et al. [2018], there has been an abundance of research that uses NODE type models for learning differential equation systems and extend it by introducing a recurrent neural networks (RNNs) encoder [Rubanova et al., 2019, Kanaa et al., 2021], a gated recurrent unit [De Brouwer et al., 2019, Park et al., 2021], or a second order system [Yildiz et al., 2019, Norcliffe et al., 2020, Xia et al., 2021].

Despite these recent developments, all of the above methods have a common limitation: any static differences in the observations can only be captured in a time-evolving ODE state. This modelling approach is not suitable when the observations contain underlying factors of variation that are fixed in time, yet can (i) affect the dynamics or (ii) affect the appearance of the observations. As a concrete example of this, consider human walking trajectories. The overall motion is shared across all subjects, e.g. walking. However, every subject might exhibit person-specific factors of variation, which in turn could either affect the motion exhibited, e.g. length of legs, or could be a characteristic of a given subject, e.g. color of a shirt. A modelling approach that is able to (i) distinguish the dynamic

Dynamics modulator

ModulatorPrediction

P

Network

ODE state

Encoding

Z1

Simulation

Forward

Decoding

Network

Network

(ODE)

Staticmodulator

ModulatorPrediction

S

Network

(e.g. position, velocity) from the static variables (e.g. height, clothing color), and (ii) use the static variables to modulate the motion or appearance, is advantageous, because it leads to an improved generalization across people. As an empirical confirmation, we show experimentally that the existing NODE models fail to separate the dynamic factors from the static factors. This inevitably leads to overfitting, thus, negatively effecting the model generalization to new dynamics, as well as far-horizon forecasting.

As a remedy, this work introduces a modulated neural ODE (MONODE) framework, which separates the dynamic variables from the time-invariant variables. We call these time-invariant variables modulator variables and we distinguish between two types: (i) static modulators that modulate the appearance; and (ii) dynamics modulators that modulate the time-evolution of the latent dynamical system (for a schematic overview see Fig. 1). In particular, MONODE adds a modulator prediction network on top of a NODE, which allows to compute the modulator variables from data. We empirically confirm that our modular framework boosts existing NODE models by achieving improved future predictions and improved generalization to new dynamics. In addition, we verify that our modulator variables are more informative of the true unknown factors of variation by obtaining higher R 2 scores than NODE. As a result, the latent ODE state of MONODE has an equivalent sequence-to-sequence correspondence as the true observations. Our contributions are as follows:

We extend neural ODE models by introducing modulator variables that allow the model to preserve core dynamics while being adaptive to modulating factors.

Our modulator framework can easily be integrated into existing NODE models [Chen et al., 2018, Yildiz et al., 2019, Norcliffe et al., 2020, Xia et al., 2021].

As we show in our experiments on sinusoidal waves, predator-prey dynamics, bouncing ball videos, rotating images, and motion capture data, our modulator framework consistently leads to improved long-term predictions as measured by lower test mean squared error (an average of 55.25% improvement across all experiments).

Lastly, we verify that our modulator variables are more informative of the true unknown factors of variation than NODE, as we show in terms of R 2 scores.

While introduced in the context of neural ODEs, we believe that the presented framework can benefit also stochastic and/or discrete dynamical systems, as the concept of dynamic or mapping modulation is innate for most dynamical systems. We conclude this work by discussing the implications of our method on learning object-centric representations and its potential Bayesian extensions. Our official implementation can be found at https://github.com/IlzeAmandaA/MoNODE .

2 Background

We first introduce the basic concepts for Neural ODEs, following the notation by Chen et al. [2018].

Ordinary differential equations Multivariate ordinary differential equations are defined as

where t ∈ R + denotes time , the vector z ( t ) ∈ R q z captures the state of the system at time t , and ˙ z ( t ) ∈ R q z is the time derivative of the state z ( t ) . In this work, we focus on autonomous ODE systems, implying a vector-valued (time) differential function f : R q z ↦→ R q z that does not explicitly depend on time. The ODE state solution z ( t 2 ) is computed by integrating the differential function starting from an initial value z ( t 1 ) :

For non-linear differential functions f , the integral does not have a closed form solution and hence is approximated by numerical solvers [Tenenbaum and Pollard, 1985]. Due to the deterministic nature of the differential function, the ODE state solution z ( t 2 ) is completely determined by the corresponding initial value z ( t 1 ) if the function f is known.

2.1 Latent neural ODEs

Chen et al. [2018] proposed neural ODEs for modeling sequential data x 1: T ∈ R T × D , where x i ≡ x ( t i ) is the D -dimensional observation at time t i , and T is the sequence length. We assume known observation times { t 1 , . . . , t T } . Being a latent variable model, NODE infers a latent trajectory z 1: T ∈ R T × q z for an input trajectory x 1: T . The generative model relies on random initial values, their continuous-time transformations, and finally an observation mapping from latent to data space:

Here, the time differential f θ is a neural network with parameters θ (hence the name 'neural ODEs'). Similar to variational auto-encoders [Kingma and Welling, 2013, Rezende et al., 2014], the 'decoding' of the observations is performed by another non-linear neural network with a suitable architecture and parameters ξ .

3 MONODE: Modulated Neural ODEs

We begin with a brief description of the dynamical systems of our interest. Without loss of generality, we consider a dataset of N trajectories with a fixed trajectory length T , where the n 'th trajectory is denoted by x n 1: T . First, we make the common assumption that the data trajectories are generated by a single dynamical system ( e.g. , a swinging pendulum) while the parameters that modulate the dynamics ( e.g. , pendulum length) vary across the trajectories. Second, we focus on the more general setup in which the observations and the dynamics might lie in different spaces, for example, video recordings of a pendulum. A future video prediction would require a mapping from the dynamics space to the observation space. As the recordings might exhibit different lighting conditions or backgrounds, the mapping typically involves static features that modulate the mappings , e.g. , a parameter specifying the background.

3.1 Our generative model

The generative model of NODE [Chen et al., 2018] involves a latent initial value z 1 for each observed trajectory as well as a differential function and a decoder with global parameters θ and ξ . Hence, by construction, NODE can attribute discrepancies across trajectories only to the initial value as the remaining functions are modeled globally. Subsequent works [Rubanova et al., 2019, Dupont et al., 2019, De Brouwer et al., 2019, Xia et al., 2021, Iakovlev et al., 2023] combine the latent space of NODE with additional variables, however, likewise, they do not account for static, trajectory-specific factors that modulate either the dynamics or the observation mapping. The main claim of this work is that explicitly modeling the above-mentioned modulating variables results in better extrapolation and

generalization abilities. To show this, we introduce the following generative model:

We broadly refer to d and s as dynamics and static modulators , and thus name our framework Modulated Neural ODE (MONODE). We assume that each observed sequence x n 1: T has its own modulators d n and s n . As opposed to the ODE state z ( t ) , the modulators are time-invariant .

We note that for simplicity, we describe our framework in the context of the initial neural ODE model [Chen et al., 2018]. However, our framework can be readily adapted to other neural ODE models such as second-order and heavy ball, NODEs, as we demonstrate experimentally. For a schematic overview, please see Fig. 1. Next, we discuss how to learn the modulator variables along with the global dynamics, encoder, and decoder.

3.2 Learning latent modulator variables

A straightforward approach to obtain time-invariant modulator variables is to define them globally and independently of each other and input sequences. While optimizing for such global variables works well in practice [Blei et al., 2017], it does not specify how to compute variables for an unobserved trajectory. As the focus of the present work is on improved generalization to unseen dynamic parametrizations, we estimate the modulator variables via amortized inference based on encoder networks g ψ and h υ , which we detail in the following.

(i) Static modulator To learn the unknown static modulator s n ∈ R q s that captures the timeinvariant characteristics of the individual observations x n i , we compute the average over the observation embeddings provided by a modulator prediction network g ψ ( · ) ( e.g., a convolutional neural network) with parameters ψ :

By construction, s n is time-invariant (or more rigorously, invariant to time-dependent effects) as we average over time. In turn, the decoder takes as input the concatenation of the latent ODE state z n i and the static modulator s n , and maps the joint latent representation to the observation space (similarly to Franceschi et al. [2020]):

Note that the estimated static modulator s n is fed as input for all time points within a trajectory.

(ii) Dynamics modulator Unlike the static modulator, the dynamics modulator can only be deduced from multiple time points x n i + T e . For example, the dynamics of a pendulum depend on its length. To compute the length of the pendulum one must compute the acceleration for which multiple position and velocity measurements are needed. Thereby, the dynamics modulators, d i , are computed from subsequences of length T e from a given trajectory. To achieve time-invariance we likewise average over time:

where h υ is a modulator prediction network ( e.g. , a recurrent neural network) with parameters υ . The differential function takes as input the concatenation of the latent ODE state z n ( τ ) ∈ R q z and the estimated dynamics modulator d n ∈ R q d . Consequently, we redefine the input space of the differential function f θ : R q z + q d ↦→ R q z , implying the following time differential:

The resulting ODE system resembles in a way the augmented neural ODE (ANODE) [Dupont et al., 2019]. However, their appended variable dimensions are constant and serve a practical purpose of breaking down the diffeomorphism constraints of NODE, while ours models time-invariant variables. We treat T e as a hyperparameter and choose it by cross-validation.

Optimization objective The maximization objective of MONODE is analogous to the evidencelower bound (ELBO) as in [Chen et al., 2018] for NODE, where we place a prior distribution on the unknown latent initial value z 1 and approximate it by amortized inference. Similar to previous works [Chen et al., 2018, Yildiz et al., 2019], MONODE encoder for z 1 takes a sequence of length T z < T as input, where T z is a hyper-parameter. We empirically observe that our framework is not sensitive to T z . The optimization of the modulator prediction networks g ψ and h υ is implicit in that they are trained jointly with other modules while maximizing the ELBO objective.

4 Related work

Neural ODEs Since the neural ODE breakthrough [Chen et al., 2018], there has been a growing interest in continuous-time dynamic modeling. Such attempts include combining recurrent neural nets with neural ODE dynamics [Rubanova et al., 2019, De Brouwer et al., 2019], where latent trajectories are updated upon observations, as well as upon Hamiltonian [Zhong et al., 2019], Lagrangian [Lutter et al., 2019], second-order [Yildiz et al., 2019], or graph neural network based dynamics [Poli et al., 2019]. While our method MONODE has been introduced in the context of latent neural ODEs, it can be directly utilized within these frameworks as well.

Augmented dynamics Dupont et al. [2019] augment data-space neural ODEs with additional latent variables and test their method on classification problems. Norcliffe et al. [2021] extend neural ODEs to stochastic processes by means of stochastic latent variables, leading to NODE Processes (NODEP). By construction, NODEP embeddings are invariant to the shuffling of the observations. To the best of our knowledge, we are the first to explicitly enforce time-invariant modulator variables .

Learning time-invariant variables The idea of averaging for invariant function estimation was used in [Kondor, 2008, van der Wilk et al., 2018, Franceschi et al., 2020]. Only the latter proposes using such variables in the context of discrete-time stochastic video prediction. Although relevant, their model involves two sets of dynamic latent variables, coupled with an LSTM and is limited to mapping modulation.

5 Experiments

To investigate the effect of our proposed dynamics and static modulators , we structure the experiments as follows: First, we investigate the effect of the dynamics modulator on classical dynamical systems, namely, sinusoidal wave, predator-prey trajectories and bouncing ball (section 5.1), where the

parameterisation of the dynamics differs across each trajectory. Second, to confirm the utility of the static modulator we implement an experiment of rotating MNIST digits (section 5.2), where the static content is the digit itself. Lastly, we experiment on real data with having both modulator variables present for predicting human walking trajectories (section 5.4). In all experiments, we test whether our framework improves the performance of the base model on generalization to new trajectories and long-horizon forecasting abilities.

Implementation details We implement all models in PyTorch [Paszke et al., 2017]. The encoder, decoder, differential function, and modulator prediction networks are all jointly optimized with the Adam optimizer [Kingma and Ba, 2014]. For solving the ODE system we use torchdiffeq [Chen, 2018] package. We use the 4 th-order Runge-Kutta numerical solver to compute ODE state solutions (see App. D for ablation results for different solvers). For the complete details on data generation and training setup, we refer to App.B. Further, we report the architectures, number of parameters, and details on hyperparameter for each method in App. C Table 7.

Compared methods We test our framework on the following models: (i) Latent neural ODE model [Chen et al., 2018] (NODE), (ii) Second-order NODE model [Norcliffe et al., 2020] (SONODE), (iii) Latent second-order NODE model [Yildiz et al., 2019] (LSONODE), (iv) current state-of-the-art, second-order heavy ball NODE [Xia et al., 2021] (HBNODE).

In order for SONODE and HBNODE to have a comparable performance with NODE we adjust the original implementation by the authors by changing the encoder architecture, while keeping the core of the models, the differential function, unchanged. For further details and discussion, see App. A. We do not compare against ANODE [Dupont et al., 2019] as the methodology is presented in the context of density estimation and classification. Furthermore, we performed preliminary tests with NODEP [Norcliffe et al., 2021]; however, the model predictions fall back to the prior in datasets with dynamics modulators. Hence, we did not include any results with this model in the paper as the base model did not have sufficiently good performance. Finally, we chose not to compare against Kidger et al. [2020] as their Riemann-Stieltjes integral relies on smooth interpolations between data points while we particularly focus on the extrapolation performance for which the ground truth data is not available. For an overview of the methods discussed see Table 1.

5.1 Dynamics modulator variable

To investigate the benefit of the dynamics modulator variable, we test our framework on three dynamical systems: sine wave, prey-predator (PP) trajectories, and bouncing ball (BB) videos. In contrast to earlier works [Norcliffe et al., 2020, Rubanova et al., 2019], every trajectory has a different parameterization of the differential equation. Intuitively, the goal of the modulator prediction network is to learn this parameterisation and pass it as an input to the dynamics function, which is modelled by a neural network.

5.1.1 Sinusoidal data

The training data consists of N = 300 oscillating trajectories with length T = 50 . The amplitude of each trajectory is sampled as a n ∼ U [1 , 3] and the frequency is sampled as ω n ∼ U [0 . 5 , 1 . 0] ,

Trajectory id 14

Trajectory id 10

Trajectory id 7

NODE

PCA-2 (0.07)

Trajectoryid:14

GroundTruth

Trajectoryid:14

Ground Truth

T

AAAAWMM

AAAAWMN

3

Trajectory id:10

Trajectoryid:10

-100

-80

PCA-1 (0.73)

-60

-40

-29

Trajectory id 14

Trajectory id 10

MoNODE

Trajectory id 7

(0) )

[Trajectory id:7

[Trajectory id:7

0

20

40

60

80

100

120

140

0

20

40

60

80

100

120

140

NODE

MoNODE

PCA-1 (0.46)

Trajectory id 9

HBNODE

PCA-2 (0.23)

80

Trajectory id 7

Trajectory id 8

30

40

Trajectory id:9

Ground Truth

30

40

Trajectory id:9

GroundTruth

20

20

60

10

10

100

0

0

-150

20

Trajectoryid:8

20

Trajectoryid:8

-100

10

10

Trajectory id 9

MoHBNODE

Trajectory id 7

Trajectory id 8

U

PCA-2 (0.26)

20

Trajectory id:7

20

Trajectory id:7

10

10

0

50

100

150

200

250

300

0

50

100

150

200

250

300

10

PCA-1 (0.34)

HBNODE

MoHBNODE

where U [ · , · ] denotes a uniform distribution. Validation and test data consist of N val = N test = 50 trajectories with sequence length T val = 50 and T test = 150 , respectively. We add noise to the data following the implementation of [Rubanova et al., 2019].

The obtained test MSE demonstrates that our framework improves all aforementioned methods, namely, NODE, SONODE, and the state-of-the-art HBNODE (see Fig. 2, Fig. 11, and Fig. 12). In particular, our framework improves generalization to new dynamics and far-horizon forecasting as reported in Table 2 columns 2 and 3. In addition, modulating the motion via the dynamics modulator variable leads to interpretable latent ODE state trajectories z ( t ) , see Fig. 2. More specifically, the obtained latent trajectories z ( t ) have qualitatively a comparable radius topology to the observed amplitudes in the data space. By contrast, the latent space of NODE does not have such structure. Similar results are obtained for MOHBNODE (App. D, Fig. 12).

Training and inference times To showcase that our proposed framework is easy to train we have plotted the validation MSE versus wall clock time during training for sinusoidal data, please see App. Fig. 10. As it is apparent from the figure, our framework is easier to train than all baseline methods. We further compute the inference time cost for the sin data experiment, where the test data consists of 50 trajectories of length 150. We record the time it takes NODE and MONODE to predict future states while conditioned on the initial 10 time points. Repeating the experiment ten times, the inference time cost for NODE is 0 . 312 ± 0 . 050 while for MONODE is 0 . 291 ± 0 . 040 .

5.1.2 Predator-prey (PP) data

Next, we test our framework on the predator-prey benchmark [Rubanova et al., 2019, Norcliffe et al., 2021], governed by a pair of first-order nonlinear differential equations (also known as Lotka-V olterra

system, Eq. 28). The training data consists of N = 600 trajectories of length T = 100 . For every trajectory the four parameters of the differential equation are sampled, therefore each trajectory specifies a different interaction between the two populations. Validation and test data consist of N val = N test = 100 trajectories with sequence length T val = 100 and T test = 300 . Similarly to sinusoidal data, we add noise to the data following [Rubanova et al., 2019].

The results show that our modulator framework improves the test accuracy of the existing methods for both, generalization to new dynamics as well as for forecasting, see Table 2. Moreover, examining the latent ODE state embeddings reveals that our framework results in more interpretable latent space embeddings also for PP data, see Fig. 3 for HBNODE and App. D Fig. 13 for NODE. In particular, the latent space of MOHBNODE and MONODE captured the same amplitude relationship across trajectories as in observation space. For visualisation of the SONODE, MOSONODE, NODE, and MONODE trajectories see App. D, Fig. 14, Fig. 13.

5.1.3 Bouncing ball (BB) with friction

To investigate the performance of our framework on video sequences we test it on a bouncing ball dataset, a benchmark often used in temporal generative modeling [Sutskever et al., 2008, Gan et al., 2015, Yildiz et al., 2019]. For data generation, we modify the original implementation of Sutskever et al. [2008] by adding friction to every data trajectory, where friction is sampled from U [0 , 0 . 1] . The friction slows down the ball by a constant factor and is to be inferred by the dynamics modulator. We use N = 1000 training sequences of length T = 20 , and N val = N test = 100 validation and test trajectories with length T val = 20 and T test = 40 . Our framework improves predictive capability for video sequences as shown in Table 4. As visible in Fig .15, the standard NODE model fails to predict the position of the object at further time points, while MONODE corrects this error. For the second-order variants, LSONODE and MOLSONODE, we again observe our framework improving MSE from 0 . 0181 to 0 . 014 .

5.1.4 Informativeness metric

Next, we quantify how much information latent representations carry about the unknown factors of variation (FoVs) [Eastwood and Williams, 2018], which are the parameters of the sinusoidal, PP, and BB dynamics. As described in [Schott et al., 2021], we compute R 2 scores by regressing from latent variables to FoVs. The regression inputs for MONODE are the dynamics modulators d , while for NODE the latent trajectories z 1: T . Note that R 2 = 1 corresponds to perfect regression and R 2 = 0 indicates random guessing. Our framework ob-

Table 3: R 2 scores to predict the unknown FoVs from inferred latents. Higher is better.

tains better R 2 scores on all benchmarks (Table 3), implying better generalization capability of our framework. Therefore, as stated in [Eastwood and Williams, 2018], our MONODE is better capable of disentangling underlying factors of variations compared to NODE.

5.2 Static modulator variable

To investigate the benefit of the static modulator variable, we test our framework on Rotating MNIST dataset, where the dynamics are fixed and shared across all trajectories, however, the content varies. The goal of the modulator prediction network is to learn the static features of each trajectory and pass it as an input to the decoder network. The data is generated following the implementation by [Casale et al., 2018], where the total number of rotation angles is T = 16 . We include all ten digits and the initial rotation angle is sampled from all possible angles θ n ∼ { 1 , . . . , 16 } . The training data consists of N = 1000 trajectories with length T = 16 , which corresponds to one cycle of rotation. Validation and test data consist of N val = N test = 100 trajectories with sequence length T val = T test = 45 . At test time, the model receives the first T z time frames as input and predicts the full horizon ( T = 45 ). We repeat each experiment 3 times and report the mean and standard deviation of the MSEs computed on the forecasting horizon (from T = 15 to T = 45 ). For further training details see App.B, while for a complete overview of hyperparameter see App. C Table 7.

The obtained test MSE confirms that the static modulator variable improves forecasting quality ( T = 45 ), see Table 4 and App. D fig. 16 for qualitative comparison. In addition, the latent ODE

0°

0°

24°

24°

48°

6

48°

72°

72°

96°

女

96°

4

119°

4

119°

1.

144°

3

2

144°

167°

167°

PCA-2

0

192°

PCA-2

2

0

192°

216°

216°

239°

P

2

239°

-2

263°

263°

288°

4

288°

312°

312°

335°

6

335°

360°

360°

6

4

PCA-1 (0.19)

2

0

2

6

4

PCA-1 (0.36)

2

0

2

4

6

states of MONODE form a circular rotation pattern resembling the observed dynamics in data space while for NODE no correspondence is observed, see App. D Fig. 16. Moreover, as shown in Fig. 4, the latent space of MONODE captured the relative distances between the initial rotation angles, while NODE did not. The TSNE embeddings of the static modulator indicate a clustering per digit shape, see App. D Fig. 17. For additional results and discussion, see App. D.

5.3 Are modulators interchangeable?

To confirm the role of each modulator variable we have performed two additional ablations with the MONODE framework on: (a) sinusoidal data with static modulator instead of dynamics, and (b) rotating MNIST with dynamics modulator instead of static. We report the test MSE across three different initialization runs with standard deviation. For sinusoidal data with MONODE + static modulator, the test MSE performance drops to 2 . 68 ± 0 . 38 from 0 . 29 ± 0 . 11 (MONODE + dynamic modulator). For rotating MNIST, MONODE + dynamics modulator performance drops to 0 . 096 ± 0 . 008 from 0 . 030 ± 0 . 001 (MONODE + static modulator). In addition, we examined the latent embeddings of the dynamics modulator for rotating MNIST. Where previously for the content modulator we observed clusters corresponding to a digit's class, for dynamics modulator such a topology in the latent space is not present (see App. D Fig. 17). Taken together with Table 3, the results confirm that the modulator variables are correlated with the true underlying factors of variation and play their corresponding roles.

5.4 Real world application: modulator variables

Next, we evaluate our framework on a subset of CMU Mocap dataset, which consists of 56 walking sequences from 6 different subjects. We pre-process the data as described in [Wang et al., 2007], resulting in 50 -dimensional data sequences. We consider two data splits in which (i) the training and test subjects are the same, and (ii) one subject is reserved for testing. We refer to the datasets as MOCAP and MOCAP-SHIFT (see App. B for details). In test time, the model receives the first 75 observations as input and predicts the full horizon ( 150 time points). We repeat each experiment five times and report the mean and standard deviation of the MSEs computed on the full horizon, As shown in Table 4 and Fig. 5, our framework improves upon NODE. See Table 12 for ablations with different latent dimensionalities and with only one modulator variable (static or dynamics) present.

Time t=1

Timet=30

Time t=60

Time t=90

Time t=120

Time t=150

NODE

MoNODE

6 Discussion

The present work introduces a novel modulator framework for NODE models that allows to separate time-evolving ODE states from modulator variables. In particular, we introduce two types of modulating variables: (i) dynamics modulator that can modulate the dynamics function and (ii) static modulator that can modulate the observation mapping function. Our empirical results confirm that our framework improves generalization to new dynamics and far-horizon forecasting. Moreover, our modulator variables better capture the true unknown factors of variation as measured by R 2 score, and, as a result, the latent ODE states have an equivalent correspondence to the true observations.

Limitations and future work The dynamical systems explored in the current work are limited to deterministic periodic systems that have different underlying factors of variation. The presented work introduces a framework that builds upon a base NODE model, hence, the performance of Mo*NODE is largely affected by the base model's performance. The current formulation cannot account for epistemic uncertainty and does not generalize to out-of-distribution modulators, because we maintain point estimates for the modulators. A straightforward extension would be to apply our framework to Gaussian process-based ODEs [Hegde et al., 2022] stochastic dynamical systems via an auxiliary variable that models the noise, similarly to Franceschi et al. [2020]. Likewise, rather than maintaining point estimates, the time-invariant modulator parameters could be inferred via marginal likelihood as in van der Wilk et al. [2018], Schwöbel et al. [2022], leading to a theoretically grounded Bayesian framework. Lastly, the separation of the dynamic factors and modulating factors could be explicitly enforced via an additional self-supervised contrasting loss term [Grill et al., 2020] or by more recent advances in representation learning [Bengio et al., 2013]. Lastly, the concept of dynamics modulator could also be extended to object-centric dynamical modeling [Kabra et al., 2021], which would allow accounting for per-object specific dynamics modulations while using a single dynamic model.

Acknowledgements The data used in this project was obtained from mocap.cs.cmu.edu . The database was created with funding from NSF EIA-0196217. Ça˘ gatay Yıldız funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC-Number 2064/1 - Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 950086).

References

Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association , 2017.

P. Casale, A. Dalca, L. Saglietti, J. Listgarten, and N. Fusi. Gaussian process prior variational autoencoders. Advances in neural information processing systems , 31, 2018.

T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

T. Q. Chen. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffeq .

De Brouwer, J. Simm, A. Arany, and Y. Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems , 32, 2019.

Dupont, A. Doucet, and Y. W. Teh. Augmented neural ODEs. In Advances in Neural Information Processing Systems , 2019.

Eastwood and C. K. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations , 2018.

J.-Y. Franceschi, E. Delasalles, M. Chen, S. Lamprier, and P. Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning , pages 3233-3246. PMLR, 2020.

Gan, C. Li, R. Henao, D. E. Carlson, and L. Carin. Deep temporal sigmoid belief networks for sequence modeling. Advances in Neural Information Processing Systems , 28, 2015.

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Hegde, Ç. Yıldız, H. Lähdesmäki, S. Kaski, and M. Heinonen. Variational multiple shooting for bayesian odes with gaussian processes. In Uncertainty in Artificial Intelligence , pages 790-799. PMLR, 2022.

W. Hirsch, S. Smale, and R. L. Devaney. Differential equations, dynamical systems, and an introduction to chaos . Academic press, 2012.

Iakovlev, C. Yildiz, M. Heinonen, and H. Lähdesmäki. Latent neural ODEs with sparse bayesian multiple shooting. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=moIlFZfj_1b .

Kabra, D. Zoran, G. Erdogan, L. Matthey, A. Creswell, M. Botvinick, A. Lerchner, and C. Burgess. Simone: View-invariant, temporally-abstracted object representations via unsupervised video decomposition. Advances in Neural Information Processing Systems , 34:20146-20159, 2021.

Kanaa, V. Voleti, S. E. Kahou, and C. Pal. Simple video generation using neural odes. arXiv preprint arXiv:2109.03292 , 2021.

Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems , 33:6696-6707, 2020.

P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

R. Kondor. Group theoretical methods in machine learning . Columbia University, 2008.

Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv preprint arXiv:1907.04490 , 2019.

Norcliffe, C. Bodnar, B. Day, N. Simidjievski, and P. Liò. On second order behaviour in augmented neural odes. Advances in neural information processing systems , 33:5911-5921, 2020.

Norcliffe, C. Bodnar, B. Day, J. Moss, and P. Liò. Neural ode processes. ICLR , 2021.

Park, K. Kim, J. Lee, J. Choo, J. Lee, S. Kim, and E. Choi. Vid-ode: Continuous-time video generation with neural ordinary differential equation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 2412-2422, 2021.

Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.

Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07532 , 2019.

J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

Rubanova, R. T. Chen, and D. K. Duvenaud. Latent ordinary differential equations for irregularlysampled time series. Advances in neural information processing systems , 32, 2019.

Schott, J. V on Kügelgen, F. Träuble, P. Gehler, C. Russell, M. Bethge, B. Schölkopf, F. Locatello, and W. Brendel. Visual representation learning does not generalize strongly within the same domain. arXiv preprint arXiv:2107.08221 , 2021.

Schwöbel, M. Jørgensen, S. W. Ober, and M. Van Der Wilk. Last layer marginal likelihood for invariance learning. In International Conference on Artificial Intelligence and Statistics , pages 3542-3555. PMLR, 2022.

Solin, E. Tamir, and P. Verma. Scalable inference in sdes by direct matching of the fokkerplanck-kolmogorov equation. Advances in Neural Information Processing Systems , 34:417-429, 2021.

Sutskever, G. E. Hinton, and G. W. Taylor. The recurrent temporal restricted boltzmann machine. Advances in neural information processing systems , 21, 2008.

Tenenbaum and H. Pollard. Ordinary differential equations: an elementary textbook for students of mathematics, engineering, and the sciences . Courier Corporation, 1985.

van der Wilk, M. Bauer, S. John, and J. Hensman. Learning invariances using the marginal likelihood. Advances in Neural Information Processing Systems , 31, 2018.

M. Wang, D. J. Fleet, and A. Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence , 30(2):283-298, 2007.

Xia, V. Suliafu, H. Ji, T. Nguyen, A. Bertozzi, S. Osher, and B. Wang. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems , 34:18646-18659, 2021.

Yildiz, M. Heinonen, and H. Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. Advances in Neural Information Processing Systems , 32, 2019.

D. Zhong, B. Dey, and A. Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint arXiv:1909.12077 , 2019.

A Model Encoder Adjustments

SONODE In the original implementation by [Norcliffe et al., 2020] the initial velocity is computed by a 3-layer MLP with ELU activation functions and only the initial time point, T z = 1 , is passed as an input. For the initial position the first observation time point is passed, e.g. x 0 . The resulting model fails to fit on sinusoidal data, see fig. 6 for performance on validation trajectories.

0

20

40

60

80

100

0

20

40

60

80

100

As it can be seen in figure 6 the model only fits the first data point correctly, therefore, we extend the number of input time points passed to the model to T z = 5 to compute the initial velocity. This results in notable performance improvements, see Fig. 7.

0

20

40

60

80

100

0

20

40

60

80

100

Consequently, we increase the number of input frames used for SONODE, as well as investigate replacing the MLP architecture with a RNN. We report the resulting test MSEs in App. D Table 10. For the parameters used in the experiments please see Table 7.

HBNODE In the original implementation by [Xia et al., 2021] the encoder is a 3-layer MLP with TANH activation functions that autoregressively takes every ground truth data point as input, e.g. x i for i = [0 , . . . , T ] and predicts the latent representation z i which is also fed as input to the proceeding time points i > 1 . The differential function is modeled by linear layer projection. As the present work is focused on model forecasting capabilities we adjust the original HBNODE to be compatible with problem set-ups where the ground truth data is not available. Meaning that once the model has reached the time point where there is no more ground truth data available ( T > 50 ), the model only uses it's own latent state predictions z i to compute the next latent state z i +1 . As the authors of HBNODE claim that the model is designed to better fit long-term dependencies we initially reduce the number of subsequent increments, N incr from 10 to 3 (see App. B for training details). In Fig. 8 we show the qualitative performance of HBNODE on validation trajectories. Even though the model fits perfectly the ground truth data within T < 50 , it fails to extrapolate to future time frames. We investigate the cause of this issue by, first, changing the differential function from a linear projection to a 3-layer MLP with TANH activations, and, second, by replacing the original autoregressive encoder with a RNN encoder that learns the initial position z 0 .

0

20

40

60

80

100

0

20

40

60

80

100

Table 5: Ablation results for HBNODE with RNN encoder, T z = 10 , with different N incr on sine data.

Increasing the modelling capacity of the differential function did not resolve the observed issue with forecasting. This implies that the observed over-fitting of the model might be caused due to its autoregressive encoder that takes every ground truth time point as input. By replacing the autoregressive encoder with an RNN encoder we obtain significant forecasting improvements, see Fig. 9. Therefore, in all our reported experiments we replace the autoregressive encoder with an RNN and performed additional tests on the number of data increments N incr = [1 , 3 , 5 , 10] as well as the number of input frames Tiv to compute the initial state z 0 . We find in our experimentation that N incr = 10 brings the best model performance (see Table 5) and subsequently test different T z frames, 10 and 40, respectively. We report the obtained test MSE for each set-up across 3 model runs with different seeds in App. D Table 10 and for the final model parameter settings used in the experiments please see App. C.

0

20

40

60

80

100

0

20

40

60

80

100

B Experiment details

All models for sine, PP and rotating MNIST have been trained on a single GPU (GTX 1080 Ti) with 10 CPUs and 30G memory.

Sine data In the same vein as Rubanova et al. [2019], Norcliffe et al. [2021], we first define a set of time points T = { 0 , 0 . 1 , 0 . 2 , . . . , t i , . . . , 4 . 9 } . Then each sequence y n is generated as follows:

(21)

where U denotes the uniform distribution and η is set to η = 0 . 2 . We use ∆ t = 0 . 1 to generate N = 300 training trajectories of length N test = 50 , N val = 50 validation trajectories of length T val = 50 , and N test = 50 test trajectories of length T test = 150 . Test trajectories are longer to test model forecasting accuracy.

During training we set batch size to 10. We incrementally increase the sequence length of training trajectories starting from T = 5 until T = 50 . We perform this in 10 increments for NODE, MONODE, HBNODE and MOHBNODE, and in 4 increments for SONODE and MOSONODE. All models are trained for 600 epochs. The model with lowest validation MSE is used for evaluation on test trajectories.

Predator-Prey data To generate Predator-Prey trajectories we define the following generative process:

(30)

where U denotes the uniform distribution and η is set to η = 0 . 3 . We use ∆ t = 0 . 1 to generate N = 600 training trajectories of length T = 100 , N val = 100 validation trajectories of length T val = 100 , and N test = 100 test trajectories of length T test = 300 . Test trajectories are longer to test model forecasting accuracy.

During training we set the batch size to 20. We incrementally increase the sequence length of training trajectories starting from T = 10 until T = 100 . We perform this in 10 increments for NODE, MONODE, HBNODE and MOHBNODE, and in 2 increments for SONODE and MOSONODE. All models are trained for 1500 epochs. The model with lowest validation MSE is used for evaluation on test trajectories.

Rotating MNIST For the data generation we base our code upon the image rotation implementation provided by [Solin et al., 2021]. We set the total number of rotation angles to T = 16 and sample the initial rotation angle from all the possible angles θ n ∼ U [1 , T ] . We generate N = 1000 training trajectories with length T = 16 and N = 100 validation and test trajectories with length T = 45 .

During training we set the batch size to 25 and learning rate to 0.002. We train all models for 400 epochs and for evaluation use the trained model with the lowest validation mse.

Bouncing ball with friction For data generation, we use the script provided by Sutskever et al. [2008]. As a tiny update, each observed sequence has a friction constant drawn from a uniform distribution U [0 , 0 . 1] . We set the initial velocities to be unit vectors with random directions.

Motion capture The walking sequences we consider are 07_01.amc, 08_03.amc, 35_02.amc, 35_12.amc, 35_34.amc, 39_08.amc, 07_02.amc, 16_15.amc, 35_03.amc, 35_13.amc, 38_01.amc, 39_09.amc, 07_03.amc, 16_16.amc, 35_04.amc, 35_14.amc, 38_02.amc, 39_10.amc, 07_06.amc, 16_21.amc, 35_05.amc, 35_15.amc, 39_01.amc, 39_12.amc, 07_07.amc, 16_22.amc, 35_06.amc, 35_28.amc, 39_02.amc, 39_13.amc, 07_08.amc, 16_31.amc, 35_07.amc, 35_29.amc, 39_03.amc, 39_14.amc, 07_09.amc, 16_32.amc, 35_08.amc, 35_30.amc, 39_04.amc, 07_10.amc, 16_47.amc, 35_09.amc, 35_31.amc, 39_05.amc, 07_11.amc, 16_58.amc, 35_10.amc, 35_32.amc, 39_06.amc, 08_02.amc, 35_01.amc, 35_11.amc, 35_33.amc, 39_07.amc . The number of training, validation and test samples in MOCAP and MOCAP-SHIFT splits are 46-5-5 and 43-5-8, respectively. Since sequences are already dense, we skip every other data point. For ease of implementation, we take the last 300 time points, leading to sequences of length T = 150 . We experiment with different latent dimensionalities and report the findings in Table 12.

C Architecture and Hyperparameter Details

D Additional Results

NODE

MoNODE

HBNODE

MoHBNODE

2:5

MoSONODE

SONODE

JSW uogePIRA

1.0

3SW uogeps

0.6

1.5F

G4

1.0

9.2

Wall Clock Time (seoonds)

3000

Wall Clock Time (seconds)

4000

Wall Clock: Time (seconds)

t250

17'50

300C

Trajectory id: 14

Ground Truth

Trajectory id: 14

GroundTruth

Trajectory id: 10

Trajectory id: 10

[Trajectory id: 7

[Trajectory id: 7

20

40

60

80

100

120

140

20

40

60

80

100

120

140

SONODE

MoSONODE

Trajectory id 14

Trajectory id 10

HBNODE

PCA-2 (0.21)

Trajectory id 7

Trajectoryid:14

GroundTruth

Trajectoryid:14

GroundTruth

0

-5

-10

15

Trajectoryid:10

Trajectoryid:10

20

-20

10

PCA-1 (0.28)

MoHBNODE

PCA-2 (0.31)

Trajectory id: 7

[Trajectory id:7

Trajectory id 14

Trajectory id 10

Trajectory id 7

0

20

40

HBNODE

60

80

100

120

140

0

20

40

MoHBNODE

60

80

100

120

140

PCA-1 (0.32)

Trajectory id 9

Trajectory id 8

Trajectory id 7

NODE

PCA-2 (0.18)

40

30

Trajectory id:9

Ground Truth

40

30

Trajectoryid:9

Ground Truth

20

20

10

10

0

20

Trajectory id: 8

20

Trajectory id:8

PCA-1 (0.29)

$0

Trajectory id 9

10

10

Trajectory id 8

MoNODE

Trajectory id 7

PCA-2 (0.22)

Trajectory id:7

Trajectory id:7

20

20

10

10

0

50

100

NODE

150

200

250

300

0

50

100

MoNODE

150

200

250

300

PCA-1 (0.24)

40

TWYTY

Trajectory id: 9

GroundTruth

40

Trajectory id: 9

GroundTruth

30

30

20

20

10

10

0

0

20

Trajectory id: 8

20

Trajectory id: 8

AAM

10

10

Trajectory id: 7

Trajectory id: 7

20

20

10

10

50

100

150

200

250

300

0

50

100

150

200

250

300

SONODE

MoSONODE

中

D

7

3

2 (0.21)

(a) Rotating MNIST reconstruction for NODE

(b) Rotating MNIST reconstruction for MONODE

(c) PCA embeddings NODE

(d) PCA embeddings MONODE

888888888

8∞888∞88∞888888∞8

Number1

Number 9

Number 1

Number 9

Number8

Number 8

PCA-2 (0.32)

2

PCA-1 (0.23)

。

2

PCA-1(0.35)

PCA-2

Staticmodulator TSNE embeddings

Dynamics modulator TSNE embeddings

6

0

0.2

0

1

1

2

2

0.1

3

3

2

+

4

4

0.0

0

5

5

6

6

-2

7

0.1

7

8

8

4

9

0.2

9

6

0.3

6

4

2

0

2

4

6

8

0.3

0.2

0.1

0.0

0.1

0.2

PCA-2 (0.25)

(a) Rotating MNIST reconstruction for NODE

(b) Rotating MNIST reconstruction for MONODE

(c) PCA embeddings NODE

(d) PCA embeddings MONODE

Number8

Number 8

Number 7

Number 7

PCA-2 (0.28)

PCA-1(0.30)

PCA-1(0.30)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a modulated neural ODE (MoNODE) that can separately model the dynamics and time-invariant modulating variables (e.g., parameters of dynamics or contents of agents).  Such modulating variables are modeled by averaging the embedded trajectories of given observations (i.e., the outputs of modulator prediction networks for a observed trajectories) with respect to the time, similar to the tranformation-invariant (TI) pooling approach. The proposed approach can be integrated into various NODE models simply. The authors validate the proposed MoNODE for various dynamical systems, including some synthetic examples (sinusoidal, Lotka–Volterra Hamiltonian, bouncing ball), images dynamics (rotated MNIST), and CMU motion capture dataset. The experimental results demonstrate the proposed MoNODE is better than NODE in terms of the interpretablility well as long-term predictability of dynamics.\n\n***\n\nThis paper proposes a concept of separating and learning dynamics and time-invariant modulator variables, and I believe it could be beneficial for various time-series prediction problems in scientific and engineering fields. However, I have doubts about whether the proposed model maintains the time-invariance property explicitly. Additionally, there are some concerns regarding comparisons with similar papers and validation in more complicated chaotic systems. Therefore, I currently consider this paper to be at a borderline reject.\n\nSTRENGTHS:\n- The proposed approach is simple and principled, making it easy to integrate into various NODE-type models.\n\n- The paper is well-written and easy to follow.\n\n- The experimental results demonstrate that the learned latent dynamics of MoNODEs are more convincing and interpretable compared to those of NODEs. This may be attributed to the explicit decomposition of the time-invariant contents of agents from the dynamical behavior itself.\n\nWEAKNESSES:\n- The time-invariance of the moduator prediction network is only gauranteed for the finite and fixed time horizon of $T_e$. As a result, it is not entirely clear whether the proposed model can effectively capture the time-invariant aspects of observations over an arbitrary time span.\n\n- Similarly, the proposed model requires the observation of length $T_e$ to predict the time-invariant modulating variables. Therefore, it is not greatly applicable for the case when $T_z < T_e$.\n\n- The paper lacks a comparison with a relevant previous work [1], which factorizes the latent space into content and motion variables and models the dynamics using the (Hamiltonian) latent NODE.\n\n\n***\n\n[1] Khan, A., & Storkey, A. J. Hamiltonian latent operators for content and motion disentanglement in image sequences. NeurIPS 2022.\n\nQUESTIONS:\n- I think it would be great to experimentally demonstrate that a trained modulating prediction network indeed guarantees time-invariance in the long-term time horizon. For example, it would be helpful to plot a graph to visualize the change in $d$ or $s$ over time (ideally to be a constant line).\n \n- Is the performance of the trained MoNODE still good when $T_z = T_e \\ll T_{train}$? It seems that an ablation study is needed to check this.\n\n- I am curious whether MoNODE would be applicable to chaotic dynamics. For instance, one could consider the parameters ($\\sigma$, $\\rho$, $\\beta$) of the Lorenz system as dynamic modulating variables. Can MoNODE effectively learn and infer them?\n\n- Max pooling [2] is commonly used as the pooling function for invariant mappings (e.g., $s^n = \\max s_{i}^{n}$ for the static modulator case). Is there a specific reason for using averaging? It would be nice if the authors could conduct a comparison between max and average pooling invariant maps.\n\n***\n\n[2] Laptev, D., Savinov, N., Buhmann, J. M., & Pollefeys, M. Ti-pooling: transformation-invariant pooling for feature learning in convolutional neural networks. CVPR 2016.\n\nLIMITATIONS:\nThe authors mention some limitations (limited to deterministic systems, generalization to out-of-distribution modulators, ...) of the proposed method in the Discussion section. Another limitation worth mentioning is the weakness section of my review: not entirely guaranteed time-invariance, limited application when $T_z < T_e$, and the missing relevant reference.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for Neural ODEs called Modulated Neural ODEs, which combines the conventional latent Neural ODEs with two Modulator Prediction Networks that are used to extract the time-invariant static and dynamic modulator variables, respectively. Their empirical results demonstrate that this framework can improve generalizations to new dynamic parameterization and long-time forecasting, and can better capture the true unknown factors of variation as well.\n\nSTRENGTHS:\n1.\tThe proposed MoNODE framework has better capability of generalizing to new dynamic parameterizations and long-term forecasting.\n\n2.\tThey experimentally verify that the MoNODE framework is able to better capture the true unknown factors of variation with modulator variables.\n\nWEAKNESSES:\n1.\tExperimental examples are not convincing enough. The dynamical systems chosen in this paper are relatively simple and naive. For more complex systems such as trajectories generated by a family of chaotic systems controlled by different parameters, it is doubtful whether the MoNODE framework can still extract effective modulator variables.\n\n2.\tThough NODE as the baseline group maintains the same dimension as MoNODE, additional parameters are still introduced in MoNODE, such as parameters in the Modulator Prediction Networks. It is not clear whether the slight improvement in performance is due to the additional number of parameters.\n\n3.\tThis paper does not directly provide a visual presentation or intuitive explanation of the static and dynamics modulator variables extracted by the modulator prediction networks.\n\nQUESTIONS:\n1.\tWhen comparing with NODE, could you provide a comparison of the number of parameters and set up a more reasonable experiment to prove that the performance improvement is not caused by the increase in the number of parameters?\n\n2.\tWhether the MoNODE framework can outperform the conventional NODE on the task of learning more complex dynamical systems, such as trajectories generated by a family of chaotic systems controlled by different parameters?\n\n3.\tFor the CMU Mocap datasets, due to the addition of the modulator variables, the MoNODE is supposed to significantly improve the performance of NODE in the Mocap-Shift task, but in fact, the improvement effect is not obvious, which is far less than that of Mocap. Can you further explain this phenomenon?\n\n4.\tCould you directly provide some visual description and intuitive interpretation of the modulator variables extracted by the modulator prediction networks?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
The paper proposes an approach for generalizing NeuralODE to different system parameters, in the setting of a single dynamical system with different parameters. A dynamic and static modulators are applied to specific parts of the encode-process-decode architecture: the dynamic modulator acts on the NeuralODE part, while the static modulator only acts on the decoder. They are learned in an end-to-end manner.
Evaluation is performed on synthetic datasets, as well as a real dataset (walking humans), by using different NODE variants. It showcases the generalization capabilities of the 2 modulators.
The paper’s idea is interesting, and I found the experiments are convincing.

STRENGTHS:
- The proposed framework is easy-to-integrate to existing NODE methods to increase their performance.
- The latent space is more interpretable than with vanilla NODE, and provides high correlation wrt the dynamics parameters
- The proposed method drastically improves the used NODE methods for long-range prediction

WEAKNESSES:
-

QUESTIONS:
- Is there a way the modulations spaces could be used for interpolation between different systems?
- Could this work be extended to process PDE equations?

LIMITATIONS:
The work only studies dynamical systems up to 3 dimensions, would it be able to scale to higher dimensions?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

comments: SUMMARY:
The authors introduce Modulated Neural ODEs by incorporating a static modulator that averages observation embeddings across the whole time and a dynamics modulator that averages observation embeddings by time subsequences. The proposed model consistently improves the existing model's ability to generalize to new dynamic parameterizations.

STRENGTHS:
The presentation is pretty straightforward and the idea is simple yet effective, as shown by experiments.

WEAKNESSES:
- As the output of the static modulator and the dynamics modulator is used by concatenation, I would like to see the experimental results without the static modulator or the dynamics modulator to determine if they are both essential.
- I would like the authors to add the inference time cost comparison with and without those modulators.

QUESTIONS:
See weaknesses.

LIMITATIONS:
The authors have already discussed the limitations of this work, e.g., the current exploration is only limited to deterministic systems and the performance of the work is largely affected by the base model’s performance.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting neural ordinary differential equations (NODEs) present the problem of variation across trajectories only through initial state values. As a solution to the problem, Modulated Neural ODE (MoNODE), which improves the existing NODE method by distinguishing dynamic and static states, is presented. In MoNODE, 1) static modulators and 2) dynamics modulators are used to divide dynamic variables from time-invariant variables. By attaching these two modulators to existing NODE studies, they show good performance in various datasets.\n\nSTRENGTHS:\n1. They presented the problem of handling static variables well in NODE, and suggested dynamics modulator and static modulator as a solution. This is very intuitive.\n\n2. It is very attractive that it can be applied to any NODE model and does not have to be divided into static and dynamic variables in advance.\n\n3. They showed the performance using various datasets to evaluate dynamic and static modulators.\n\nWEAKNESSES:\n1. They experimented by dividing 5.1 dynamics modulator variable and 5.2 static modulator variable data to evaluate their modulators, but they could not show whether each modulator actually played its role.\n\nQUESTIONS:\nIn experiments 5.1 and 5.2, static variable and dynamics variable data were separately tested to evaluate each static modulator and dynamics modulator. Was the dynamics modulator also applied to static variables in these experiments? If the dynamics modulator was used even in the static variable experiment, you should remove the dynamics modulator.\n\nIf you did not use dynamics modulator in static variable experiment, you should experiment using dynamics modulator. In other words, through these experiments, it is necessary to check whether each modulator is playing a role.\n\nLIMITATIONS:\nThere is no limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

comments: SUMMARY:
The paper propose to improve long term forecasting abilities of NODEs by incorporating the dynamics with learnt static parameters. The system is composed of a static modulator and a dynamic modulator, the final solution is sampled from a distribution conditioned on the static modulator and the dynamic generated using the dynamic modulator. Empirical results suggests improvements over NODE on time simple series datasets.

STRENGTHS:
Presentation is clear, and the paper is well written.

The empirical improvements on time series data is significant on the long run.

WEAKNESSES:
Similar work has been proposed, see https://arxiv.org/abs/2302.13262 for an example. Although the methods are slightly different, the fundamental idea of incorporating with static parameters are the same. 

There are no clear theoretical guarantees that the method will work better.

The network structure is very complicated, the reviewer is worried about the training difficulties. It would be helpful to see some ablation and show it is easy to train.

QUESTIONS:
How are the networks trained? There are so many different networks. Also, how is this method significantly different from existing literature?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-14