Latent Space Energy-based Neural ODEs
Authors: Sheng Cheng
Deqian Kong
Jianwen Xie
Kookjin Lee
Ying Nian Wu
Yezhou Yang
Venue: Accepted by TMLR ()

Abstract: This paper introduces novel deep dynamical models designed to represent continuous-time sequences. Our approach employs a neural emission model to generate each data point in the time series through a non-linear transformation of a latent state vector. The evolution of these latent states is implicitly defined by a neural ordinary differential equation (ODE), with the initial state drawn from an informative prior distribution parameterized by an Energy-based model (EBM). This framework is extended to disentangle dynamic states from underlying static factors of variation, represented as time-invariant variables in the latent space. We train the model using maximum likelihood estimation with Markov chain Monte Carlo (MCMC) in an end-to-end manner.  Experimental results on oscillating systems, videos and real-world state sequences (MuJoCo) demonstrate that our model with the learnable energy-based prior outperforms existing counterparts, and can generalize to new dynamic parameterization, enabling long-horizon predictions.

Published in Transactions on Machine Learning Research (02/2025)
Latent Space Energy-based Neural ODEs
Sheng Cheng i
schenng53 Gasu edu
School of Computing and Augmented Intelligence, Arizono State University
Deqian Kong t
Segia edu
Department of Statistics and Data Science, University of California, Los Angeles
Jianwen Xie
jianeen Gucla edu
Akool Reseorch
Kookjin Lee
Gookjin.lee@@uuu. edu
School of Computing and Augmented Intelligence, Arizona State University
Ying Nian Wu $
ywru@stal. ucla edu
Department of Statistics and Data Science, University of California, Los Angeles
Yezhou Yang &
yz.yong@asu. edu
School of Computing and Augmented Intelligence, Arizona State University
Reviewed OpenReview: https: https://openrevieu
openreview. net/ forum? id= @dhhCttlflli
Abstract
This paper introduces novel deep dynamical models designed to represent contimuous-time sequences. Our approach employs neural cmission model to generate each data point in the time series through a non-linear transformation of latent state vector. The evolution of these latent states is implicitly defined by neural ordinary differential equation (ODE), with the initial state drawn from an informative prior distribution paramcterized by an Energy-based model (EBM). This framework is extended to disentangle dynamic stales from underlying static factors of variation, represented as time-invariant. variables in the latent space. We train the model using maximum likelihood estimation with Markov chain Monte Carlo (MCMC) in an end-to-end manner. Experimental results on oscillating systems, videos and real-world state sequences (MuJoCo) demonstrate that our model with the learnable encrgy-based prior outperforms existing counterparts, and can generalize to new dynamic parameterization, enabling long-horizon predictions.
1 Introduction
Top-down dynamic generators, such as Pathak et al. (2017); Tulyakov et al. 2018 ; Xie et al. 2019 2020). are the predominant models for representing and generating high-dimensional, sequences, such as text and video. However, they are less suited for irregularly sequences, commonly found in medical datasots (Goldberger et al., 2000). physical science (Karniadakis et al. 2021). Neural ordinary differential equations (ODEs) (Chen al. 2018) have emerged as a great approach for the continuous representation of time sequences, demonstrating impressive results in synthesizing new continuous-time sequences. Neural ODEs model describe hidden state dynamics using an ODE defined by neural network, which takes the current state and time-step as input and outputs the derivative. Given an initial state, a neural ODE defines a continuous-time trajectory of hidden states through numerical ODE solver. As Chen et al. (2018) has shown, the solver's gradient with respect to the neural network parameters
aegularly-sample. discrete-time
continuous time
Equal contribution, Equal advising

---

Published in Transactions on Machine Learning Research (02/2025)
can be efficiently computed, enabling neural ODEs to serve as building blocks in advanced deep learning frameworks.
Neural ODEs have been adapted for continuous-time sequences through approaches like ODE-RNNs and Latent ODEs (Rubanova et al. 2019). ODE-RNNs generalize RNNs with contimuous-imme hidden dynamics, while Latent ODEs (Chen et al. [2018] Rubanova et al.) 2019) extend latent variable sequential models to continuous dynamies. They assumee the iitiil state follows an isotropic Gaussian distribution as the prior and recruit an additional inference network for variational inference (Kingma & Welling 2014) during training.
However, latent ODEs face three fundamental challenges: (1) The design complexity of the inference net work, which must effectively capture continuous-time hidden dynamics. (2) The optimization challenges stemming from the evidence lower bound, which introduces a non-zero Kullback Leibler (KL) divergence between the true posterior and the approximated Gaussian posterior. (3) The limitations of simple Gaussian priors in capturing the complexity of initial latent state spaces.
To address these challenges comprehensively, we introduce ODE-LEBM, novel latent space neural-ODE based dynamic generative model for continuous-timg sequences. Our solution tackles each challenge through specific design choices: (1) To climinate the inference network complexity, we employ an empirical Bayes approach with Markov chain Monte Carlo (MCMC)-based inference, directly sampling from the posterior distribution without requiring a separate inference network. This approach maintains statistical rigor while simplifying the model architecture. (2) To resolve the optimization challenges from the evidence lower bound, we utilize maximum likelihood estimation (MLE) combined with MCMC sampling. This eliminates the KL divergence gap present in variational approaches, as we directly optimize the likelihood using samples from the true posterior distribution rather than approximated onc. (3) To address the limitations of simple Gaussian priors, incorporate an energy-based model (EBM) prior (Pang et al.) 2020) for the initial ODE state. This significantly enhances the model's expressiveness in capturing complex initial state distributions.
Our training process unifics these solutions through MCMC sampling of latent initial states from both EBM prior and posterior distributions. We update the EBM priors based on the statistical difference between these samples, while the ODE-based generator is updated using posterior samples and observed data.
Our extensive experiments on irregular time scries, rotating MNIST, bounding balls, MuJoCo demonstrate that the proposed ODE-LEBM, coupled with an MCMC-based learning algorithm, effectively eliminates the need for complex inference network design, enabling robust capabilities in interpolation and extrapolation beyond the training distribution, and discovering interpretable representations through its expressive EBM priors.
Our work makes the following contributions:
We introduce the latent space energy-based neural ODE (ODF-LERM), novel ODF-based dynamic generative model with an energy-based prior, designed for continuous-ting time sequence generation.
2. We train the ODE-LEBM model integrating MLE combined with MCMC-based inference. This training method is principled, statistically rigorous and eliminates the need for inference networks.
3. We present two variants of ODE-LEBM that can discover static and dynamic latent variables in complex systems, improving the model's generalizability and interprotability.
4. We conduct extensive experiments to assess the effectiveness and performance of the proposed ODE- LEBM model and the associated learning algo These experiments evaluate the model's ability to generate realistic continuous-timg sequences, learn disentangled and interpretable representations, and outperform existing approaches.
2 Related Work
Neural ODEs Neural ODEs (Chen et al. 2018) aim to model continuous-time soquential data by using neural networks to parameterize the continuous dynamics of latent states. Rubanova al. (2019) and Brouwer et al. (2019) propose continuous-time RNNs by incorporating neural ODE dynamics. Rubanova et. al. (219
2

---

Published in Transactions on Machine Learning Research (02/2025)
ODE Solver(fyr &og Lo (T)
Zto Pa
Zt
ZtT
Latent space
Data
space
Xxy
Xto
Xt
XtT
Lo
L1
LT
Figure 1: An illustration of ODE-LEBM. The initial latent state 2ie follows a learnable EBM prior distribution (a)) (Eq. (]) Subsequent latent states (Rt are generated using a neural ODE (Eq. (2). All
latent stales are then mapped to the data space through an emission model (Eq. (3)
and Yildiz et al. (2019) extend the state transition of dynamic latent variable models contimuous time dynamics specified by neural ODEs and train these models within the VAE framework {Kingma & Welling) 2014h. Our work also falls under the category of based latent variable models. However, unlike Chen et al. (2018); Rubanova et al. (2019); Yildiz et al. (2019); Cheng et al. (2023). our model Icarns an dynamic generator instead of using an uninformative Gaussian prior. Additionally, we train our model via MCMC inference, eliminating the need for an additional assisting network for variational inference.
informative prior for the neural-ODE-basee
Latent space encrgy-based model As an intriguing branch of deep energy-based models (Xic et al., 2016
Nijkamp el al. 2019 Latent space EBMs (LEBMs) (Pang et al., 2020 have demonstrated that an EBM can serve as an informative prior model the latent space of top-down generator, and can be jointly trained with the generator using maximum likelihood. LEBMs have been successfully applied to a variety of tasks, including text generation (Pang & Wu
2021 Yu et al.) 2022), image generation (Zhu et al.] 2023 Cui PPng et al.
et al. 2023 trajectory generation
2021 ), salient object detection (Zhang et al. 2021 2023
unsupervised foreground extraction
Yu et al.
2021 molecule design (Kong et al. 2023
2024a)
and offline
reinforcement learning (Kong et al.
2024b).
From a modeling perspective, Zhuu et al. (0023)
integrate an
LEBM and a neural radiance field (NeRF)
Mildenhall et al., 2022), while Zhang et al.
2021 combines the
LEBM with a vision transformer. Cui et al.
(2023) incorporate the LEBM into a hicrarchical generator, and
Yu et
(2023 pair the LEBM with diflusion model (Ho et al. 2020
Our work leverages an LEBM
alongside ODE-based generator for modeling contimous-tine sequences.
Dynamical system. The dynamic texture model (Doretto et al. 2003) employs lincar transition and emission models. In contrast, non-linear dynamic generator model has been used to approximate chaotic systems (Pathak et al. 2017), where innovation vectors are given as inputs, and the model is deterministic. The dynamic generator (Tulyakov et al. 2018 Xie et al. 2019 2020) uses independent innovation vectors at each discrete time step, making the model stochastic. Our model does not incorporate randomness in the dynamical system and is focused on generating continuous-time sequences by leveraging neural ODEs (Chen)
et al. 2018
3 Latent Space Energy-based Neural ODE
Here, we present our method. Let x (Ir,, i = 1,.... T') denote a sequence of data points, where 14,1=11 1, are their timesteps and 7 is the sequence length. The time I; is assumed to be known for the observation Tty-
We present generative model, ODE-LEBM, to represent continuous-time sequences. The model consists of (1) latent energy-based prior model for the initial state of the latent trajectory, (2) a neural ODE describe the dynamics of the latent trajectory, and (3) an emission model that produces the time series from
3

---

Published in Transactions on Machine Learning Research (02/2025)
the latent trajectory. The model is illustrated in Figure 1
To be specific, we have
Eto (oo()
(1)
2e, = 21 + / f=(2(t), dt,
(2)
Ft. Pal (2 1..)
(3)
The energy-based prior model for the initial state in Eq. can be defined as the exponential tilting of Gaussian distribution:
Pa (a( = Z exp (po))))))))))
(4)
where fa is multilayer perceptron (MLP) with parameters and ~ ~N(00, x0) is known Gaussian distribution. Z == (=, Jdz, is the intractable normalizing constant. The emission model in Eq. is represented by po(x)),,) = (a; fa(z)), where fo is a neural network with parameters B.
Eq. 2 fr noural network paramctorized by y, which specifics a neural ODE (Chen et al. 2018 given by:
dz(t) df
= f))
55)
Given an initial state z(to) = zto the trajectory of the latent states z(t) is implicitly defined by the ODE, and can be evaluated at any desired time using a numerical ODE solver:
24, t2 btr ODESolve(f;; 2t, tu. t .t..t.....)).
(6)
Following Chen et al. 2018 we can treat the ODE solver as differentiable building block for constructing more complex neural network system.
For notation simplicity, let f = (a, 7) be the parameters of the whole model, and ø = (B,7) be those the ODF-based generation model. Although each It, generated by its latent state 2t,, all (2,, 1, 1,... T} are generated by the initial state 2to We can actually write a summarized form for our model
x = (a(ara) te, e~N(0,0) zto ~ (o(oo)))
(7)
where Fs composes Jn (i.e., ODFSolve) and S3-
The marginal distribution of our ODE-LEBM defined as
px(x)= = ff po(x, d200 (Pa )des = /
II (o (2. 1pp
T
(el (d2,
(8)
Suppose we observe a training set of continuous-time sequences (xm: m = 1,... M). The model can be learned by maximizing the likelihood of the training set. The gradient of the log- likelihodood is
V (ox) Epo(=1, x)) V log (x 21,) Epo xx) Va + V (x
(9)
which involves the computation of
E Ep xx) Va log (a(6)) = Epa(=eo (x) ( [afa(2t)) Epu(sso)) / (afa(())))
(10)
and
EpoCen, (x) Vo (x =F P
[x Fo(zt)))
(11)
Estimating the expectations Eq. 10) and Eq. 11 relies on Markov chain Monte Carlo sampling evaluate the prior distribution (=t))) in Eq. 14 and the posterior distribution pa (x).
4

---

Published in Transactions on Machine Learning Research (02/2025)
We could use Langevin dynamics (Neal et al. 2011 to sample from target distribution of the form (=) & exp(-U(z)) by performing r-step iterative procedure with step size é, starting with an initial value z(0) sampled from a standard normal distribution,
x = = [z) + ,20el", ~ ~N(, ~ I), for T T= T= 1, 1-,,,, r.
(12)
For the prior distribution (z) the gradient of the potential function is given by V, -Vafaa) + z/o². For the posterior distribution Pa (2|x) x p(x, 2) (x))))((x)))) the gradient of the potential function is given by V. po x) V. [||x - (=(=))l - [=(() + 11=11//==///20 ||==]]
Algorithm provides complete description of the training algorithm for our ODE-LEBM.
Algorithm Learning algorithm for our ODE-LEBM
Input: (1) Training sequences fxm hm=1: (2) Number of learning iterations J; (3) Numbers of Langevin steps for prior and posterior } (4) Langevin step sizes for prior and posterior (6=,8"): (5) Learning rates for energy-based prior and ODE-based generator (Kn,Ca); (6) batch size n'. Output: Parameters
(3,7) ) for the neural- neural-ODE--al based generator and a for the energy- based prior model 1: Initialize a, 8, and y. do
2: for j j
3: Sample observed sequences (x.. {mm]==1
For each Xm, sample the prior 2 ~ Pa,, (z) using I" Langevin steps in Eq. (12 with step size 8-
5: For each Xm, sample the posterior zm Po, po, (2|x..) using n Langevin steps in Eq. 12) with a step size 5+.
Update energy-based prior using Adam with the gradient Va computed in Eq. [10) and learning rate &a-
7: Update the ODE-based dynamic generator using Adam with the gradient Vá computed in Eq. [11 and a
learning rate Ed.
8: end for
Algorithm 2 provides complete description of the test phase algorithm for our ODE-LEBM. In testing, given initial, partial or irregular sampled observations x, we generate the remaining sequences y as conditional generation o(y[x)= = p(2]x)p(x(2 x)dz P (ptx,,)/((y22,,x) [p(y|2,)))
Algorithm 2 Test-phase algorithm for our ODE-LEBM
Input: (1) Testing sequences x; (2) Numbers of Langevin steps for posterior sampling I; (3) Langevin step sizes for posterior sampling 6; (4) Testing prediction timesteps T; (5) Learned paraneters Φ = (3,7) for the neural-ODE-bn generator and for the energy-based prior model.
Output: Prediction sequences y.
1: For each x, sample the posterior distribution of the inital state (o (x) using I' Langevin steps in Eq. with a step size 8.
12
2: Given Stg, predict the latent states (z,))_ sequence using Eq. 2 and generate the sequences using Eq.
3
Algorithms [ and provide a general framework for learning ODE-LEBM, where latent variables serve as initial latent states (2t,) using MCMC-based inference. While our generator model incorporates a neural ODE in the latent space (Eq. (2) and an emission model that maps latent states to the data space (Eq. (3) we aim to enhance model interpretability through more explicit design of latent variables in above generation process. Specifically, we introduce additional latent variables to disentangle trajectory-specific features in both the neural ODE (Section 4.3.2) and emission procedure (Section 4.3.1) The additional latent variable in the neural ODE, which we refer to as the dynamic variable (za), captures the hyper-parameters of dynamic evolution that cannot be represented by initial states 24,- Similarly, the additional latent variable in the emission model, termed the static variable (z,); characterizes global environmental features that are not directly involved in dynamic evolution. Both model specifications can be learned using Algorithm leveraging the flexibility of MCMC-based learning and inference algorithms.
5

---

Published in Transactions on Machine Learning Research (02/2025)
4 Experiments
This section investigates the performance and adaptability of our proposed ODE-LEBM in various scenarios. By incorporating an EBM prior with an explicit probability density, ODE-LEBM provides more expressive generative model compared to latent, ODE. In the experiments, we evaluate the model's ability to capture complex dynamics from irregularly sampled sequences, discover different time-invariant variables, detect out-of-disttrbuttion samples, and demonstrate its applicability in real-world scenarios.
Bascline methods All our bascline methods use neural ODEs as the dynamical model. Their primary difference is in how they encode the latent space variables. Latent ODE (Chen et al., 2018) encodes the initial latent space using an RNN. To address the issue of irregular sampling, Rubanova et al [2019) proposes
the ODE-RNN an encoder. Additionally, to identify time-invariant variables, Modulated Neural ODEs
(MoNODEs) AAuzina et al. 2024 employs an additional inference network for encoding.
Implementation details The network architectures, training details and physical dynamics of the dataset. are shown in the Appendix
4.1 Overview
Our empirical study adopts the convention from Neural ODEs and test our model in the following scenarios.
Irregularly-sampled time series We generate a synthetic dataset of 1000 one-dimensional trajectories, with each containing T = 100 irregularyy sampled time points within the interval [o, 5j. The trajectories are generated using a sinusoidal function with a fixed amplitude of 1, frequency randomly sampled from [0.5, 1], and starting points sampled from N(p= = 1, 0.1). The dataset is split into 80% for training and 20% for testing. The data generation setting follows Rubbanvaa al. (2019). During training and testing, the latent variable is inferred using 7T randomly sampled timesteps from the total T timesteps to predict all T samples, where y represents the observation ratio (0.1, 0.2, 0.3, or 0.5).
Rotating MNIST The data is generated following the implementation by Casale et al. (2018); Auzina ct al. (2024), with the total number of rotation angles set to 16. We include all ten digits from MNIST dataset, with the inl rotation angle sampled from all possible angles 0 ~ {0", 24", , 312", 336"). The training data consists of N = 1000 trajectories, each with a length of T 15, corresponding to one cycle of rotation. The validation and test data consist of Nvai = Nust = 100 trajectories, each with a sequence length of Tva = 15 and Trest 45. During training and validation, latent variables are inferred from observations over T timesteps to predict within the same interval. For testing, we use the first T timesteps predict sequence of length Test. This process involves both interpolation (first T' timesteps) and extrapolation (remaining timesteps).
Bouncing balls with friction We use a modified version of the bouncing ball vidco sequences, benchmark commonly employed in temporal generative modeling (Gan et al.; 2015 Yildiz et al.] 2019 Sutskever et al. 2008). The dataset is generated using the original implementation by SStskever et al. 2008). with an additional friction term sampled from U00, 0.1] added to each trajectory. The dataset consists of 1000 training sequences of length T 20, and 100 validation and 100 test trajectories with lengths of Teat 20 and Tiest 40, respectively. This setup allows for comprehensive evaluation of the model's ability to learn and predict the dynamics of the bouncing ball under varying friction. The data generation setting follows the implementation by Auzina et al. (2024). The training, validation, and testing protocol follows the same procedure described for Rotating MNIST.
MuJoCo physics simulation MuJoCo physics simulation Todorov et al. 2012 widely used for training reinforcement learning models, is employed in our experiments with three physical environments: Hopper, Swimmer, and HalfCheetah using the DeepMind control suite (Tunyasuvunakool et al. (020). We sampled 10,000 trajectories for each environment with 100 timesteps each, with initial states randomly selected. The data split into training and testing sets at 80/20 ratio. During training, the sequence length is set. to
6

---

Published in Transactions on Machine Learning Research (02/2025)
T = 50, while for testing, the sequence length is extended to Tiest 100. The training and testing protocol follows the same procedure described for Rotating MNIST.
4.2 Prrgularly-Sampler Time Series
First, we aim to verify that our proposed ODE-LEBM, along with its associated posterior inference and learning algorithm, performs effectively. Among various choices in the sinusoidal function dataset, those with irregular timesteps pose the greatest challenge. RRubanova et al. (2019) claims that RNN-based inference networks struggle to model time series with non-uniform intervals, proposing ODE-RNNs as alternative to handle arbitrary time gaps between observations. Instead, we propose using an MCMC-based posterior inference method to avoid the intricate design of inference nelworks for learning latent ODEs. By incorporating the EBM prior, we are interested in the model's capability to handle irregularly sampled time series.
In training, we randomly subsample small percentage of time points simulate sparse observations. For evaluation, we measure the mean squared error (MSE) on the full time series, testing the learned model's interpolation capability. Experiments in Table T show the mean squared error for models trained different. percentages of observed points ranging from 0.1 to 0.5. The visualizations
shown in Figure 2 Our ODE-LEBM outperforms Latent ODE with both the RNN encoder and ODE-RNN variant, as well as NCDSSM (Ansari et al. 2023). ) which a state space model designed for irregular sampling, validating that DDE-LEBM with MCMC- posterior inference achieves better performance while climinating the complexity of designing an inference nelwork. Similarly lo Latent ODEs, ODE-LEBMs reconstruct the time series pretty well even with sparse observations and improve performance with increasing observations.
Table 1: Test Mean Squared Error (MSE) on the irregularly-sampleed sinusoidal dataset.
Observation ratio ?
0.1
0.2
01.3
0.5
GRU-D
2.3562
0.0599
0.0483
0.0411
RNN-VAE
0.4200
0.4189
0.4180
0.4160
NCDSSM
0.1710
0.1105
0.1072
0.1029
Latent ODE (RNN enc)
0.1524
0.0374
0.0311
0.0327
Latent ODE (ODE-RNN enc)
0,0713
0.0311
0.0303
0.0246
ODE-LEBM
0.0700
0.0304
0.0266
0.0233
10% Observed points
30% Observed points
50% Observed points
Prior Samples
2
0
2
Time
Time
Interpolation results on irregularly-sampled time series. The latent initial state sampled based on partial observations, 2t, po(22, (x), and then used predict the entire sequence. For comparison, also show the results of directly sampling from the learned latent EBM prior, Zio Pa (Ztg) in the last column.
Figure 2:
4.3 Disentangle Trajectory-specific Latent Variables
Discovering variables that steer the evolution of underlying dynamics in complex dynamical systems can improve generalization new dynamics and enhance interoperability. Latent variable modeling facilitates the learning of underlying dynamics, particularly when observations and dynamics reside in different spaces, such as in video recording of moving car on the road.
The down latent variable model (i.e,, our ODE- LOE-LEBM) can be viewed as a specifically designed model for trajectory-speeffic variable discovery. The EM-type MLE learning algorithm updates the model paramcters using the gradient Epe(s/o]Ve log po (xx, 2)] in Eq. 0 The learning process consists of two steps: (1) Infer the latent. variable p=(= from each trajectory using MCMC. The Langevin sampling of the posterior
7

---

Published in Transactions on Machine Learning Research (02/2025)
distribution can be viewed as gradient-basee learning of the trajectory-specffe parameters, i.c., 2. Update the model parameters using Vo log po(x; 2), where all trajectories share the same model parameters 0.
(2)
To further enhance model generalization, we disentangle the latent variable into dynamic latent variables Ed, which play a role in ODE evolution, and static variables 2s, which capture environment statistics unrelated to dynamics. For simplicity, we concatenate these variables with the initial latent state 2t, and let them share the joint energy- based prior distribution. Unlike MoNODEs (Auzina et al., 2024], which infer deterministic dynamic and static variables using the average of the observation embeddings, approach infers both latent variables based on their posterior distribution using Langevin dynamics.
4.3.1 Trajectory-specine static variables
The first variant of our proposed ODE-LEBM includes timo-invariant static variable 2, to capture variations not directly related to dynamics. 2s plays role in the emission model, where It; ~ pa(as, [e; 2.). We assume (z), where 2 | Fel, and [ denotes concatenation. The model, learned by MLE using
Algorithm 1 is summarized as follows:
[Zror 2d] ~ Da Pat =n]),
(13)
21 21, f fz(2(t,,
(14)
F4, ~ (p 2s).
(15)
We invostigate model's capability on the rotating-MNIST dataset in sccnarios where sequence dynamics remain the same but observations differ due to varying static variables in the environment. We explore the following aspects: (1) Interpolation and extrapolation with the learned model; (2) Interpretability of both 2to and žej (3) Learned energy functions for out-of-distributio (OOD) detection.
O
O
Figure 3: Interpolation and extrapolation results the Dotating-MNIS" test set. The first 15 steps represent. interpolation, while the last 30 steps represent extrapolation. The first row shows the model predictions for each digit, and the second row presents the ground-truth observations.
Table 2: Mean Squared Error (MSE) on Rotating-MNISS
T=15
T=45
NMI
Latent ODE
0.015 0.001)
0.039(0.001)
ApproxSE
0.086 0.3))
0.099(0.0033
Neural ODE Processor
0.032 004)
(0.000)
Modulated ODE ODE-LEBM
(0.00))
0.030(0.001)
0.131
0.020(0.000)
0.024(0.001) 0.576
Interpolation and Extrapolation To ensure fair comparison with the baseline model (Auzina al. 2024), we set the length of the sequences to 15 during training. For evaluation, measure the MSE on test sequences with lengths of 15 and 45 to demonstrate our model's ability in both interpolation and extrapolation tasks. The interpolation performance is assessed on sequences of the same length as the training data, while the extrapolation performance is evaluated on three times longer sequences. Figure B visualizes the model predictions for sequences with 45 time steps, showcasing the model's ability to generate accurate prodictions beyond the training sequence length. The experimental results, presented in Table indicate that our model outperforms MoNODEs, which encode static variables with an additional convolutional neural network, in
8

---

Published in Transactions on Machine Learning Research (02/2025)
143"
366
164
235
240
1587
332"
111
<0 520
810 30
(a) PCA plot of Zio-
(b) PCA plot of 2x.
(c) PCA plot of Zto-
(d) PCA plot of 2,-
Figure 4: PCA embeddings of both 24 and 2, for the Rotating MNIST dataset, as inferred by posterior sampling. We generate 16 trajectories from single digit, incrementing the initial angle of each trajectory by 24", starting from 0 until 360". In (a) and (c), circles denote the start of the trajectory (the initial angle), and lines represent the ODE trajectory. The color gradient corresponds to the initial angle of the trajectory in the observation space. (a) and (b) are from the same sequences with time reset, while (c) and (d) are from the same sequences without time reset.
both interpolation and extrapolation tasks. Additionally, our model outperforms recent approaches like LatentApprox&DE (Solin et al., 2021) and Neural ODE Processor (Nrcllife et al. 2021). These findings demonstrate the effectiveness of the learned energy functions in the latent space, enabling our model to capture the underlying dynamics and generalize well to longer sequences.
Interpretability of initial states 2t, and static variables 2, To analyze how the model represents rotatod digits in its latent space, we conducted experiments by laking a single digit and generating 16 sequences with the length of 16 timesteps each. These sequences show the digit rotating from 09
3122, 336", For each sequence, we inferred the latent variables (zo, and zæ) using posterior sampling (i.e., | 2.] p([=ta 2a] [x)). To visualize the high-dimensional latent space, we projected these variables into 2D using principal component analysis (PCA).
We investigated two scenarios using the same rotated images but with different time labeling strategies. In the first scenario (Figures Aa| and 4b), we reset the timesteps for each sequence to start from to 15, simulating that all sequences begin at 0. Our results show that it, remains constant across all sequences, while 2, captures the rotational information ranging from 0 to 336". This suggests that by resetting the starting time step, the ODEs focus on the underlying dynamics, which are consistent they originate from the same digit. In the second scenario (Figures 4c and Ad), we preserved the original timesteps, where each sequence begins at different time points (c.g., first sequence: t = O,.... 15; second sequence: z= 1,...., 16; and so on until t= 15,...,30 30). In this case, Sto incorporates the rotational information since it needs to account for different starting positions time, while 2, captures other invariant features of the digit. This experiment. demonstrates that the allocation of information between 2t, and 2, is influenced by time-step labeling, even when the underlying rotated images remain unchanged. Furthermore, projecting 2t, into 2D reveals closely aligned trajectories of the 16 sequences, indicating that our model learns consistent dynamics within the dataset, while observation differences are attributed to static variables.
To investigate whether 2, can capture global information, such as the label of the digits, we randomly sample sequences for each digit, with the initial rotation angle randomly set.. We project the 2, of each sequence into 2D using t-SNE (Van der Maaten & Hinton 2008), as shown in Figure The projection reveals sequences corresponding to the same digit cluster logether, indicating that 2, successfully encodes the digit label global information. Furthermore, we compute the Normalized Mutual Information (NMI) of the clustering results to assess the quality of the learned representations quantitatively. MoNODE achieves an NMI of 0. 03131, while our method obtains a significantly higher NMI of 0.576, demonstrating that our approach produces more interpretable and informative latent variables regarding digit labels.
nut-of-distributtin detection Energy-based model has been widely used to detoct the out-of-distributioo samples by treating the energy function as generative classifier (Elftein et al.) 2021)). To verify this ability, we generate OOD samples by randomly shuffling the sequence order of real samples. We then visualize the
9

---

Published in Transactions on Machine Learning Research (02/2025)
Digit
Digit
2.5.0-155114
Digit 3
Digit 4
Digit
Rigit
Digit 9
Energy
Figure 5: t-SNE plot of static variable with randomly sampled sequences.
Figure 6: Out-of-distribution detec-
tion by energy function.
energy distribution of both real and OOD samples in Figure S predominantly fall into the high-energy region, while real samples are primarily located in the low-energy region.
The visualization shows that OOD samples
4.3.2 Trajectory-specifi dynamic variables
We investigate whether latent variables that steer the latent dynamics evolution emerge with the simplest inductive bias, representing time-invariant hyperparamcters in ODEs. We use dataset of bouncing balls with friction, where each trajectory is sampled from a sequence with different friction coefficinnss drawn from UO0 0.1]. During training, we observe videos of bouncing balls with length T 20. At test time, we generate sequences with T 20 for interpolation and 40 for extrapolation. During training and testing, only the first 10 timesteps are used to infer 2to and zd.
The model, learned by MLE as in Algorithm can be summarized as follows:
[zto za] Pa (lāto za]),
(16)
zt: 2o + I de
(=(= l)dl,
(17)
Ft. ~ (
(18)
Table 3: Mean Squared Error (MSE) bouncing balls with friction (BB).
T-20
T-40
R
Latent ODE
0.0178(0.001)
0.0199(0.001)
-0.29
Modulated ODE ODE-LEBM
0.0110(0.000)
0.0164(0.001)
0.58
(0.09)
0.0119(0.001)
0.62
Table shows the MSE on the test set, demonstrating that our model with dynamic latent variables outperforms other ODE-basod counterparts. This highlights the offectiveness incorporating trajectory-specific dynamic variables to make accurate predictions.
4.4 Real-world application: MuJoCo physics simulation
We demonstrate that our ODE-LEBM can be adapted to complex systems like MuJoCo simulation Todorov et al. 2012). We experiment with three physical environments: Hopper, Swimmer, and Cheetah. To evaluate our method's ability to interpolate and extrapolate these complex state sequences, we use a subsequence of length 50 during training. At test time, the model observes the first. 50 time steps, infers 2tg and predicts
10

---

Published in Transactions on Machine Learning Research (02/2025)
the entire 100 time steps. The first 50 steps resemble the interpolation setting, while the last 50 steps require extrapolation beyond the training distribution. Figure 7
visualizes the prediction results compared to the ground truth. Table | shows that ODE-LEBM significantly outperforms latent ODE, demonstrating the capability of our proposed model in real-world applications.
Table 4: Mean Squared Error (MSE) on MuJoCo physics simulation.
Hopper
Swimmer
HalfChootah
T-50
T-100
T-50
T-100
T-50
T- 100
Latent ODE
0.0097
0.0145
5.8c-05
6.3c-05
0.0112
0.0087
ODE-LEBM
0.0076
0.0138
3.0e-06
4.9e-06
0.0081
0.0079
A
N
Figure 7: Visualization of prediction results on the MuJoCo Hopper environment with time interval 20 steps. The upper row shows the ground truth, while the lower row presents the model predictions. The first 50 time steps (first three columns) represent interpolation, where the model is conditioned on observed data, and the remaining 50 time steps (last three columns) correspond to extrapolation.
4.5 Ablation Studies
Table 5: Ablation study on the number of steps in Langevin sampling.
Steps
20
50
100
1000
T=
0.020
0.014
0.012
0.011
T=45
45
0.024
0.022
0.020
0.019
Number of Langevin sampling steps during test We conduct the experiment the rotating MNIST, starting from 20 steps, which set for training. We show that increasing the number of Langevin steps for the posterior during the test can further improve the performance as shown in Table 5
The necessity of Latent EBM prior and MCMC-based inference To demonstrate the significance of the EBM prior model and MCMC-based inference in posterior inference, we conducted rescarch using the Spiral2D experiment from Chen et al. (2018). We generate 1000 samples each for training and validation, and 200 samples for testing. The sequence length for training and validation is 100, while the test sequence length is 500. We train four modcls: Latent ODE, MCMC-based posterior inference without EBM prior, Latent ODE with EBM prior, and MCMC-based posterior inference with EBM prior. The test MSE losses for these models are shown in Table D The substantial improvement achieved by using the EBM prior and MCMC-based sampling underscores their importance.
Time Complexity We analyze the computational complexity of ODE-LEBM. The analysis conducted under the same experimental sctup as described in Table 0 We report the training and testing speeds of the latent ODE and ODE-LEBM, considering Langevin sampling steps set 20 (the default setting for all other experiments), 50, and 100. The results shown in Table 7
11

---

Published in Transactions on Machine Learning Research (02/2025)
Table 6: The necessity of Latent EBM prior and MCMC-based inference.
MCMC-based inference
EBM prior
Test MSE
22.76
/
17.92
x
8.33
1.36
Table 7: Ablation study on time complexity.
Methods
Latent ODE
ODE-LEBM (20 stops)
ODE-LEBM (50 steps)
ODE-LEBM (100 steps)
Training phase (s/iter)
0.27
4.5
10.7
21.3
Testing phase (s/iter)
0.27
4.2
10.4
21.0
5
Conclusion
This paper presents a novel time series model, which is latent space energy-based neural ODE (ODE-LEBMs) designed for continuous-time sequences. Our model specifies the dynamics of latent stales using neural ordinary differential equations (Neural ODEs), represents the initial state space through latent energy-based prior model, and maps latent states to data space using top-down neural network. Through extensive experiments, we demonstrate that our model can effectively interpolate and extrapolate beyond the training distribution, unveil meaningful representations, and adapt to various scenarios, including learning from irregularly sampled sequences and identifying different time-invariant variables. Compared to its counterpart model, Latent ODE, our approach more statistically rigours, accurale, and design friendly.
Limitation This paper proposes using posterior inference learn ODE-LEBM. Posterior sampling using Langevin dynamics as form of test-time computation provides flexible sampling process at the cost of multi-step model forward and backward computation. In the future, shall study the potential behavior of sampling using Langevin dynamics a stochastic differential equation or persistent Markov Chain to speed up the sampling process. In the future, we shall leverage the expressive ODE-LEBM to tackle more challenging large-scale real-world applications.
Acknowledgments
SC and YY were supported by NSF Cyber-Physical System (CPS) program grant #2038666 and the Engineering Research and Development Center Information Technology Laboratory (ERDC-ITL) under Contract No. W912HZ24C0002. KL acknowledges support from NSF under grant IIS #2338909. YW was partially supported by NSF DMS-2015577, NSF DMS-2415226, and a gift fund from Amazon. JX acknowledgess support from XSEDE grant CIS210052. The authors acknowledge resources and support. from the Research Computing facilities Arizona State University. The views and opinions of the authors expressed herein do not necessarily state reflect those of the funding agencies and employers.
12

---

Published in Transactions on Machine Learning Research (02/2025)
References
Abdul Fatir Ansari, Alvin Heng, Andre Lim, and Harold Soh. Neural cantinuous-discceed state space models for irregularly-sampler time scries. In International Conference on Machine Learning (ICML), pp. 926-951. PMLR, 2023.
Tlze Amanda Auzina, Çagatay Yıldiz, Sara Magliacane, Matthias Bethge, and Efstratios Gavves. Modulated neural odes. In Advonces in Neural Information Processing Systems (NeurIPS), volume 36, 2024.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Sru-ode-bayes: Contimuous modeling of sporadically-obsered time series. In Advances in Neural Information Processing Systems (NeurIPS), pp- 7377 7388, 2019.
Francesco Paolo Casale, Adrian Dalca, Luca Saglietti, Jennifer Listgarten, and Nicolo Fusi. Gaussian process prior variational antoencoders. In Advances in Newral Information Processing Systems (NeurIPS), volume 31, 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6572 0223,,63, 2018.
Sheng Cheng, Yezhou Yang, Yang Jiao, and Yi Ren. Supe-supe learning to discover physical objects and prodict their interactions from raw videos. In NeurIPS 2023 AI for Science Workshop, 2023.
Jiali Cui, Ying Nian Wu, and Tian Han. Learning hierarchical features with joint latent space energy-basod prior. International Conference on Computer Vision (ICCV), pp. 2218 2227, 2023.
Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto. Dynamic textures. Int. J. Comput. Vis., 51(2)911-,09, 2003.
Sven Elfiein, Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. On out-o--distribution detection with energy-based models. arXiv preprinl orXim:2107.08788 2021.
Zhe Gan, Chunyuan Li, Ricardo Henao, David E Carlson, and Lawrence Carin. Deep temporal sigmoid belief networks for sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015.
A. L.. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mictus, G. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PoysioToolkii, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation, (23 e220, 2000.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, (ICLR), 2014.
Deqian Kong, Bo Pang, Tian Han, and Ying Nian Wu. Molecule design by latent space energy-based modeling and gradual distribution shifting. In Uncertainty in Artificial Intelligence (UAI), volume 216, pp. 1109-1120, 2023.
Degian Kong, Yuhao Huang, Jianwen Xie, Edouardo Honig, Ming Xu, Shuanghong Xue, Pei Lin, Sanping Zhou, Sheng Zhong, Nanning Zheng, and Ying Nian Wu. Molecule design by latent prompt transformer. In Advances in Neural Information Processing Systems (NewrIPS), 2024a.
Deqian Kong, Dehong Xu, Minglu Zhao, Bo Pang, Jianwen Xie, Andrew Lizarraga, Yuhao Huang, Sirui Xic, and Ying Nian Wu. Latent plan transformer for trajectory abstraction: Planning as latent space inference. Tn Advances in Neural In Information Processing Systems (NeurIPS), 2024b.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes neural radiance fields for view synthesis. Commun. ACM, (5(1):99-106, 2022.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Erik Nijkamp, Mitch Hill, Song-Chun Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run MCMC toward energy-based model. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5233 5243, 2019.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural {ode} processes. In International Conference on Learning Representations (TCLR), 2021.
Bo Pang and Ying Nian Wu. Latent space energy-based model of symbol-vector coupling for text generation and classification. In International Conference on Machine Learning (ICML), volume 139, pp. $359-8370, 2021.
Bo Pang, Tian Han, Erik Nijkamp, Bong-Chun Zhu, and Ying Nian Wu. Learning latent space energy-basod prior model. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Bo Pang, Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy-based model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11814-11824, 2021.
Jaideep Pathak, Zhixin Lu, Brian R Hunt, Michelle Girvan, and Edward Ott. Using machine learning to replicate chaotic attractors and calculate lyapunov exponents from data. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27() 2017.
Yulia Rubanova, Tian Qi Chen, and David Duvenaud. Latent ordinary differential equations for irregularly- sampled time series. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5321-5331, 2019.
Arno Solin, Ella Tamir, and Prakhar Verma. Scalable inference in sdes by direct matching of the fokker Holooooro equation. In Advances in Neural Information Processing Systems (NewrIPS), volume 34, pp. 417-429, 2021.
Iyy Sutskever, Geoffrey E Hinton, and Graham W Taylor. The recurrent temporal restricted boltzmann machine. In Advances in neural information processing systems (NewrIPS), volume 21, 2008.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: N.1109//RO0.2012..3350
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kantz. Mocogan: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp- 1526 1535, 2018
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dicontrol: Software and tasks for continuous control. Softwore Impacis, 6: 6:100022, 2020. ISSN 2665-9638.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using L-sne. Journal of machine learning research, 9(11), 2008.
Jianwen Xie, Yang La, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning (ICML), pp. 2635-2644, 2016.
Jianwen Xie, Ruiqi Gao, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Learning dynamic generator model by alternating back- propagaaatn through time. In The Thirty- Third AAAI Conference on Artificial Intelligence, (AAAT), pp. 5498 5507. AAAT Press, 2019.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Jianwen Xie, Ruiqi Gao, Zilong Zheng, Song- Chun Zhu, and Ying Nian Wu. Motion- generator model: Unsupervised disentanglement of appearance, trackable and intrackable motions in dynamic patterns. In The Thirty-Foure AAAI Conference on Artificial Intelligence (AAAI)), pp. 12442 12451. AAAI Press, 2020.
Çagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. ODE2VAE: deep generative second order odes with bayesian neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 13412-13421, 2019.
Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song- Chun Zhu. Unsupervised foreground extraction via deep region competition. In Advances in Neural Information Processing Systems (NewrIPS), pp. 14264-14279, 2021.
Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruiqi Gao, Yixin Zhu, Song- Chun Zhu, and Ying Nian Wu. Latent diffusion energy-based model for interpretable text modeling. In International Conference on Machine Learning (ICML), 2022.
Peiyu Yu, Yaxuan Zhu, Sirui Xie, Xiaojian Ma, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Learning energy-based prior model with diffusion-amortizeed MCMC. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Jing Zhang, Jianwen Xie, Nick Barnes, and Ping Li. Learning generalive vision transformer with energy-based latent space for saliency prediction. In Advances in Neural Information Processing Systems (NeurIPS), pp. 15448-15463, 2021.
Jing Zhang, Jianwen Xie, Nick Barnes, and Ping Li. An energy-based prior for generative saliency. IEEE Trans. Pattern Anal. Mach. Intell., 45(11):13100 111111118000-1111,,, 2023.
Yaxuan Zhu, Jianwen Xie, and Ping Li. Likelihood-basse generative radiance field with latent space energy- based model for 3d-aware disentangled image representation. In International Conference on Artificial Intelligence and Statistics (AISTATS), volume 206, pp. 4164-4180 PMLR, 2023.
A Experiment details
Model Architecture For the emission model and neural ODE model, the network architecture designs are the same as the baseline model in Auzina et al. (2024): Rubanova et al. (211)). For EBM prior, all experiments use layers MLP with activation GELU. The latent space dimension for each experiment is shown in Table g
Table 8: The latent space dimension
Experiment name
Zt
Za
Rd
Trregular sampling
10
Bouncing balls
8
4
Rotating MNIST
16
16
MuJoCo
15
Training details of prior model and MCMC-based inference The mumber of steps in Langevin sampling is 20 for training. We list the remaining training details in Table g
Computing resources for training All experiments can be done within an 11GB GPU like GTX 1080Ti. However, to speed up the training process, we use single V100 for training.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Table The training details prior model
Experiment name
hidden dim
step size
learning rate
Irregular sampling
25
0.01
le-4
Bouncing balls
20
0.5
le-5
Rotating MNIST
20
0.5
le-5
MuJoCo
25
0.001
1c-5
Physical Dynamics in Dataset Bouncing balls with friction: We utilize the script provided by Sutskever al. (2008), with a minor modification. Each observed sequence assigned friction constant y, drawn from uniform distribution U|0, 0.1). The velocity evolves as u(t) to yt until the ball collides with wall, at which point the direction reverses.
Rotating MNIST Dataset: We use the image rotation implementation provided by Solin et al. [2021] The total number of rotation angles is set to T = 16, with rolations performed around the center of the image. At each timestep, the image rotates by x/8.
MuJoCo Physics: The dynamics MuJoCo follow a dynamical system as implemented in Todorov al. 2012
B
More visualization
In this section, we will provide more visualization results.
B.1 Bouncing balls with friction
We present the reconstruction results over 20 timesteps in Figure S demonstrating that our model can almost
reconstruct the movement of the ball. In contrast, according to Auzina al. (2024). the latent ODE is unable to achieve this, and MoNODE's reconstruction is also inadequate.
Figure 8: Visualization of prediction results on the Bouncing balls with friction. The first row the ground truth and the second row is the prediction.
B.2 Hopper
We present more pair of Hopper environment results in Figure
16

---

Published in Transactions on Machine Learning Research (02/2025)
N
Figure 9: Visualization of prediction results the MuJoCo Hopper environment. The first row is the ground truth and the second row is the prediction.
C
Derivations
The derivations of Eqs.
to
11
follows.
V log (x()
(x(x) Vepe(x))
Po (x) Vo f pooox, =0o ddzo
22t (x) Vo / po(x, 21o ddze
po (x, 20d)
(19)
po po (x) po(x, 2no) Vopa (x, zro Jdzo
[ pulzro (x) Vo log (x, Reo ddzro
Exp (x) V log pa (x, z1o)]
EpolEin (x) V log (2ta)) + V log Po (x)21)))
With pa(z))) = exp (po and Z exp (Zt. )oo (Zro Jdzea:
Epo(ze, (x) Vv) log Epo(zeo x) Va Po (2t,)
log 2)]
1
/ exp (fa(zt))) ddza
Epo V la (2)0)
V.
Z
1
/
Epo xc) Va
V. exp (fa (Zta) )ol )dzzo
(20)
/
1
Epo(ses xx) Va log
exp
(z VaJa (a ddz
z
/
Epo Va log
Pa Va
Epo [ap Bpa(ze)) (afa(2
vafa(210)
With p(x =
E Epo(en, bx) Vo log (x = Exo(zegg (x)
V
20 xx - F2(
(21)
Expo (Reg (x)
o (x - Fo(21))) Fz(
17

---

Published in Transactions on Machine Learning Research (02/2025)
D More Quantitative result on Rotating MNIST
The importance of static variables 2, We run additional experiment of proposed method without infer the static variables zz. The results are presented in the third row of the Table T0 Our results indicate that our method, without explicitly inferring the statie latent variable, achieves better performance than the vanilla latent ODE. However, it does not match the performance of the Modulated ODE, which leverages additional encoders encode the static latent variable. Notably, when our model infers the static latent variable directly, it perforrs the Modulated ODE.
Table 10: The importance of static variables 2, on Rotating-MNIST.
Method
Static latent variable 2, Using Encoder
T 45
Latent ODE
0.039
Modulated ODE
0.030
ODE-LEBM
0.036
ODE-LEBM
0.024
The necessity of Latent EBM prior and MCMC-based inference Similar to the Spiral2D experiment, we conducted an additional ablation study to assess the contributions of the Latent EBM prior and MCMC- based inference, as shown in Table
Table 11: The necessity of Latent EBM prior and MCMC-based inference on Rotating-MNIST.
Method
MCMC-based inference
EBM prior
Lateut ODE
X
0,039
ODE-LEBM
0.025
ODE-LEBM
0.031
ODE-LEBM
0.024
18


Reviews:
[{"review_id": "vfB9P7mse5", "reviewer": "Reviewer_gDHo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The idea is simple and straightforward. \n- The authors evaluate their method on vast experiments.\n- Inference network designing is not necessary through this formulation.\n\nWeakness\n- While simple, it was quite difficult to go through Section 3. I would encourage authors to add a simple intuitive example in the beginning of Section 3\n- Introduction section is not direct enough. I had a hard time connecting the introduction to the contribution. A good revision can fix this.\n- Utilizing EBM could introduce additional time complexity. Can the authors do a runtime experiment comparing the models?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents a time series model that utilizes latent space for neural ODE prediction, and moves back to the data space from latent space using a top-down NN. The authors evaluate their work on multiple benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weakness.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "csta1IJbJA", "reviewer": "Reviewer_r7Sf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is quite well-written overall.\n- The proposed model is conceptually quite simple, I think it makes intuitive sense to have a learnable prior in the form of an EBM.\n- The proposed model seems to perform well overall compared to alternative neural ODE models. The ablation results in Table 6 seem quite strong.\n\n\n\n\n\nWeaknesses:\n- Section 4 is not ideally structured, I found it somewhat difficult to follow some parts.\n- The aspect of computational cost (both during training and at test-time) is not really discussed at all.\n\n\n\nQuestions/suggestions:\n- How dos the computational cost scale with the number of Langevin steps, during training and at test-time? For example in Table 5 and Table 6, what is the computational cost of those model variations?\n- Seems a bit strange to introduce model variations in Section 4.3.1 (eq. (13) - (15)) and Section 4.3.2 (eq. (16) - (18)), could it not make more sense to describe these in Section 3 instead?\n- What would the performance of the original model in eq. (1) - (3) be in Table 2 and Table 3? Are the model variations introduced in Section 4.3.1 and Section 4.3.2, respectively, required for ODE-LEBM to perform well there?\n- What model variation is used in Section 4.4? (the model in Section 3, Section 4.3.1 or Section 4.3.2?)\n- In Figure 6, is the Hopper falling through the floor in the predicted sequence? Is this not a bit strange? Also, the prediction seems to be quite far away from the ground truth towards the end of the sequence, is this expected? Are there any other types of models which could perform better in this task?\n- Could be nice to provide an algorithm also for how the model is used at test-time? This is a bit unclear to me.\n- The ablation study in Table 6 is interesting and the results seem quite strong. However, why do you introduce a new dataset/task here which hasn't been mentioned before? And, could this model comparison perhaps be done also on some of the other datasets/tasks?\n- Given that the paper is shorter than 10 pages, \"Limitations\" and \"Additional Related Work\" could be moved from the appendix to the main text?\n\n\n\n\n\n\nMinor things:\n- The text formatting next to Figure 4 is a bit strange, with just \"region\" super close to the caption.\n- Section 1, \"We train the ODE-LEBM model MLE combined with\" --> \"We train the ODE-LEBM model using MLE combined with\"?\n- Section 2, \", and molecule design\" --> \", molecule design\"?\n- Section 2, \"a LEBM and a neural radiance field\" --> \"an LEBM and a neural radiance field\"?\n- Figure 4 is placed after Figure 5, and is also referred to after Figure 5 in the text? (what's currently called Fig 4 should be called Fig 5?)\n- Section 4.3.1, \"Energy-based model has been\" --> \"Energy-based models have been\"?\n- Section 4.3.1, \"generative classifier Elflein et al. (2021)\" --> \"generative classifier (Elflein et al., 2021)\"? Same citation formatting issue also in the first sentence of Section 4.4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose ODE-LEBM, a generative model for continuous-time sequences. It consists of an energy-based model prior for the initial latent state, a neural ODE for the latent trajectory dynamics, and an emission model producing the time series from the latent trajectory. All model components are jointly trained using MLE with MCMC-based sampling.\n\nThey conduct experiments on synthetic 1D data, the \"bouncing ball video sequences\" benchmark, rotating MNIST, and three MuJoCo environments (Hopper, Swimmer, HalfCheetah).\n\nThey compare their ODE-LEBM with other neural ODE model variants, achieving quite promising performance overall.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think this could be a quite solid paper, it could definitely be relevant for the TMLR audience.\n\nHowever, I think the current version requires some clarifications, see questions above.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hoP98Ktlkf", "reviewer": "Reviewer_rpdn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The use of an energy-based prior distribution for the initial states in an ODE-based latent dynamical model, combined with MCMC-based sampling to estimate the expectations with respect to the posterior distribution instead of relying on an encoder, is a novel approach. Extensive experiments conducted by the authors have demonstrated the effectiveness of the proposed method compared to other ODE-based latent dynamical models.\n\n- The proposed method is grounded in solid theoretical analysis with reasonable assumptions.\n\n- The proposed method is evaluated on several different datasets, varying from simple one-dimensional synthetic time series data to physical simulation data. And different aspects of the model are tested. Adequate experimental evidence has proven the superiority of the proposed method.\n\n- The paper is well-structured, clearly presented and easy to follow.\n\nWeaknesses:\n\nAlthough the paper is concise in its presentation, it lacks certain details in both the theoretical derivations and the experimental implementations. See requested changes for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel latent dynamical model in which the latent dynamics are represented by an ODE with an energy-based prior for the initial states. The model is trained using MCMC-based inference to estimate both the prior and posterior distributions of the latent variables. Furthermore, the authors propose two model variants designed to capture global static and dynamic latent variables. Comprehensive experiments are conducted to evaluate the method's performance across various aspects, including its ability to handle irregularly sampled temporal data, interpolate and extrapolate sequential data, disentangle global static and dynamic variables, and apply effectively to real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have summarized the requested changes into two parts: theoretical presentation and experimental implementation.\n\nFor the theoretical presentation:\n\n- To enhance the clarity of the proposed method, it would be helpful if the authors included the derivation of the learning objectives in equations (10) and (11) in the appendix.\n\n- It would also enhance comprehension if the authors could include the algorithm for inference during the test phase of the proposed model.\n\nFor the experimental implementation:\n\n- In Section 4.1, the authors briefly describe the different datasets used for evaluation. However, the underlying true dynamics used to generate three of the datasets—the bouncing balls with friction, rotating MNIST, and MuJoCo physics—are not clearly explained. \nBesides, the latent dimensions of $z_t$, $z_s$ and $z_d$ used for experiments are not specified.\n\n- The presentation in Section 4.3.1 on the interpretability of initial states $z_{t_0}$ and static variables $z_s$ is difficult to follow. In particular, it is unclear what it meant by “resetting the timesteps” for a given sequence of observations and how this affects the model’s processing. Providing an example to illustrate this operation would greatly enhance the readers’ understanding.\n\n- In both Figure 6 and Figure 8, it appears that the extrapolation generated by the proposed model tends to produce static motion patterns, with limited variation in object movement. Could the authors provide an explanation for this phenomenon?", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a latent ODE as an state-space like emission model, where a latent ODE is coupled with a decoder. The initial state distribution posterior is inferred through MCMC, which presents a simple training objective. The method requires inference at training-time to establish the initial state distribution for test sequences. The paper also proposes a more flexible initial distribution as a transformed Gaussian. 

The reviews generally agree that the idea is sensible and well-presented, and comes with extensive empirical evaluation. One reviewer found results incremental, and the added cost of the method as negative. All reviewers agree on the paper presenting a simple and useful method with good results. The authors improved the submission based on the comments by incorporating studies on model interpretability, cost and by clarifying the presentation wrt claims.

This is a well-written paper that proposes a useful, well-presented, and sufficiently novel method in the domain, that solves some of the problems in earlier methods, while achieving good performance.
Decision: Accept (poster)