SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equations
Authors: Grigory Bartosh
Dmitry Vetrov
Christian A. Naesseth
Venue: ICML 2025 poster (2025)
Keywords: diffusion
generative models
SDE
time series
variational inference
Primary area: deep_learning->sequential_models_time_series

Abstract: The Latent Stochastic Differential Equation (SDE) is a powerful tool for time series and sequence modeling. However, training Latent SDEs typically relies on adjoint sensitivity methods, which depend on simulation and backpropagation through approximate SDE solutions, which limit scalability. In this work, we propose SDE Matching, a new simulation-free method for training Latent SDEs. Inspired by modern Score- and Flow Matching algorithms for learning generative dynamics, we extend these ideas to the domain of stochastic dynamics for time series modeling, eliminating the need for costly numerical simulations. Our results demonstrate that SDE Matching achieves performance comparable to adjoint sensitivity methods while drastically reducing computational complexity.

SDE Matching:
Scalable and Simulation-Free Training of Latent Stochastic Differential Equations
Grigory Bartosh
Dmitry Vetrov?
Christian A. Naesseth
Abstract
Table 1. Asymptotic complexity comparison. L and R are ber of sequential and parallel evaluations of drift/diffusion terms, respectively, and D is the number of parameters/states.
The Latent Stochastic Differential Equation (SDE) is powerful tool for time series and se- quence modeling. However, training Latent SDEs typically relies on adjoint sensitivity methods, which depend on simulation and backpropaga- tion through approximate SDE solutions, which limit scalability. In this work, we propose SDE Matching, new simulation-free method for train- ing Latent SDEs. Inspired by modern Score- and Flow Matching algorithms for learning generative dynamics, we extend these ideas to the domain of stochastic dynamics for time series and sequence modeling, climinating the need for costly numer- ical simulations. Our results demonstrate that SDE Matching achieves performance comparable to adjoint sensitivity methods while drastically reducing computational complexity.
Method
Memory
Time
Forward Pathwise
(Yang & Kushner, 1991)
0(1)
O(LD)
(Gobet & Munos, 2005)
Backprop through Solver
O(L)
O(L)
(Giles & Glasserman, 2006)
Stochastic Adjoint (Li al., 2020)
0(1)
O(Llog L))
Amortized Reparameterization (Course & Nair, 2023)
O(R)
O(R)
SDE Matching (this paper)
((1)
O(1)
Differential Equations (SDEs) (Movellan et al., 2002; Ha et al., 2018: Tzen & Raginsky, 2019; Peluchetti & Favaro, 2020; Hodgkinson al., 2020). Latent SDEs have been shown to be more robust to data shifts (Oh al., 2024), and applications are numerous, from neuroscience (ElGazzar & van Gerven, 2024) to video prediction (Daems et al., 2024).
1. Introduction
Differential equations are a natural choice for modeling continuous-time dynamical systems and have recently re- ceived significant interest in machine learning. Since Chen al. (2018) introduced the adjoint sensitivity method for learning Ordinary Differential Equations (ODEs) in memory-efficient manner, ODE-based approaches became popular in deep learning for density estimation (Grathwohl et al., 2019) and to model irregularly observed time series (Yildiz et al., 2019; Rubanova et al., 2019).
To enable more memory- learning, Li et al. (2020) extended the adjoint sensitivity method from ODEs to SDEs. Despite these advancements, training of ODE and SDE models is iimulation-bbseld it relies costly numerical integration of differential equations and backpropagation through the solutions. With training algorithms that are difficult to parallelize on modern hardware, Latent SDEs have till now resisted truly scaling.
However, ODEs describe deterministic systems and encode all uncertainty into their initial conditions. This limits the applicability of ODE-based approaches when modeling stochastic and chaotic processes. To address these limi- tations, several works study Neural (or Latent) Stochastic
In parallel, Score Matching (Ho et al., 2020: Song et al., 2021b) and Flow Matching methods (Lipman al., 2023; Albergo al., 2023; Liu et al., 2023) have demonstrated that continuous-tima dynamics for generative modeling can be learned efficiently in simulation-free manner- without requiring numerical integration. These techniques have proven computationally efficient and scalable for high- dimensional problems. Inspired by these developments, we develop a simulation- approach for learning SDEs.
University of Amsterdam *Constructor University, Bre- Correspondence Grigory Bartosh waaabrrr@@uvaalll Dmitry Vetrov <dvetrov constrrccooruuiveesstyy Christian A. Naesseth <c..nacssseth @uva.nl>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SDE Matching: Scalable and Aimulation-FFee Training of Latent SDEs
We introduce SDE Matching-as simulation-free framework for learning Latent SDE models. The key idea we adopt from matching-based approaches is direct access to samples from the latent posterior approximation at any time step. This eliminates the need to integrate SDEs during training. The SDE Matching objective is estimated using the Monte Carlo method, achieving (1) memory and time complexity. We summarize the complexity comparisons in Table 1.
Zo E RR from the initial prior distribution po(zo). Next, infer the latent continuous dynamics Z (=(()) by
integrating the following SDE:
dz = (Ze, t)dt + go (Zt, t)dw,
(I)
where hg TRD x 10, + RD is the drift term, (0 () RD x [0,1] + RDxD is the diffusion term, and w is a standard Wiener process. The SDE in Equation (1), together with the prior distribution pe pe(z)), defines a sequence of probability distributions po (2())) at each time step E [0, 1]. Once the trajectory of the latent variables Z is sampled, we independently sample observations 2t, from the conditional distributions pa (T,, =x,) ) for each time step 4..
We summarize contributions follows:
1. We establish a clear connection between diffusion mod- els and Latent SDEs, which motivates the development of a more efficient, simulation-free training procedure.
The goal of the Latent SDE is to find a set of parameters 0 that best fits a dataset of observed time series or sequences. Unfortunately, the posterior distribution of the latent vari- ables, pe ()),,, is generally intractable. However, varia- tional inference can be used to train the Latent SDE. Similar to the prior process, we introduce an approximate posterior process, which is conditioned on the observations X. The posterior process consists of two components: the initial posterior distribution qu (2o (X and a conditional SDE:
2. We propose an efficient parameterization of the La- tent SDE posterior process and use it to develop SDE Matching, simulation-fre free training procedure for La- tent and Neural SDEs.
3.
We demonstrate that SDE Matching achieves compa- rable performance to the adjoint sensitivity method on both synthetic and real data, while significantly reduc- ing the computational cost of training and improving convergence.
dz, = fx 1, X)dt + go(ze; f)dw,
(2)
4.
We show that SDE Matching enables the application of Latent SDEs to high- nigh-dimensional problems, where training with the adjoint sensitivity method is compu- tationally infeasible.
where fi((z, 1, X) defines the drift term of the SDE condi- tionally on the observations X. It is important to note that, despite having different drift term, the posterior shares the same diffusion term ge (ze, t) as in Equation (1).
In Section 2. we provide background on the Latent SDE model. In Section 3, we discuss the connection between Latent SDEs and matching-based methods, which serve as motivation for SDE Matching. Then, in Section 4, we in- troduce the parameterization of the Latent SDE posterior process and formulate the SDE Matching training algorithm. In Section 5, we demonstrate that our method achieves com- parable performance on both synthetic and real-world data while requiring significantly less computational and mem- ory cost for training. In Section we discuss and contrast SDE Matching related work. Finally, conclude with discussion of the limitations of our approach and potential directions for future work.
The training objective of the Latent SDE is a variational bound on the log-marginal likelihood of the observations:
log Pe (X) < L = Eprior + Lditf + Lree
(3)
Cpiir DKL (go (
(=o (Ipo(2o)
(4)
1 || (2:,4, L, (,,,,,)))1,1
Laitt a (2(x))
(5)
2
N
Crec
E qe (Z)))
log pa (2 12))
(6)
where ro t, X) satisfies
go(2:: ( t, X) ho((z, t) fe
(7)
2. Background
Obtaining an estimate of the objective & for stochastic opti- mization requires joint numerical integration of the poste- rior process SDE (Equation tion (5). While there exist methods to improve the memory and computational efficiency of numerical integration, most are simulation-bass and require backpropagation through numerical solutions of SDEs. Besides being computation- ally expensive, these methods are difficult to parallelize and suffer from numerical instabilities. Altogether, these challenges make training the Latent SDE a difficult task.
Consider a series of observations X Tty Tta...... Tens where each Ft, E X where the space X depends on the application. For simplicity of notation, we assume that all time steps l; belong to the interval [0, 1j. The extension to intervals of arbitrary lengths is straightforward.
(2)) and the integral in Equa-
The Latent SDE model assumes that this series is gener- ated constructively by the following stochastic process, re- ferred to as the prior process. First, sample latent variable
2

---

SDE Matching: Scalable and Simulation-Free Training of Latent SDEs
3. Diffusion Models as Latent SDEs
(2) invert the time direction for the entire model, mapping the time interval [0, 1] into [1.
In contrast to Latent SDEs, recent advancements in diffu- sion and flow-based modeling demonstrate that continuous dynamics can be learned efficiently in a simulation-free man- ner, without requiring numerical integration. At first glance, these generative models seem quite different from Latent SDEs. Instead of generating data autoregressively, they produce single data point through an iterative refinement process, gradually reconstructing corrupted data. However, similar to Latent SDEs, continuous diffusion models can be thought of as learning an SDE in latent space. This connection is useful for understanding and developing SDE Matching.
First, the reverse SDE for the forward process can be ob- tained from the Fokker-Planck equation, similar to the derivation of the marginal reverse SDE in Equation (9).
Second, after inverting the time direction, we obtain: dz [f(t) g2) (t) log [z)]dt + g(t)dw. (11)
Except for the change in time direction, the generative pro- cess remains unchanged:
dæ = [f(1)z + O (1))0(22, () + g())dw.
(12)
Conventional diffusion models are defined through two pro- cesses: the forward (or noising) process and the reverse (or generative) process. The forward process takes a data point E R.. and perturbs it over time by injecting noise. This dynamic can be expressed as an SDE with a linear drift term and intteeeedeee diffusion term:
From this perspective, diffusion models can be seen as a special case of Latent SDEs with a specific form of the posterior process and only a single observation step 1. Moreover, substituting this parameterization of the prior and reverse processes into the Latent SDE objective in Equation (3), we find that it corresponds to a reweighted denoising score matching objective:
time
dzz- f(()=zdl + g())dw,
1h' 2g (t) [1tt + (())s(((z).))
where f [0. 1] + g: [0,1] r Rt, and g(zg|a)) m 6(zo- x). Due to the linearity of Equation (8), the con- ditional marginal distribution is available in closed form ((<//x) = (N= a, T), where Og of are determined by gt)). The functions f(t) and g(t) are typically chosen to ensure that g(=1 (z) z ((z 0, I). The genera- tive process then reverses this transformation, starting from the prior p(2))= = ((22; 0, I) and following the (marginal) reverse SDE:
E z
/|/ dt
f(itz - (t) log (
+C (13)
g2(t)) |so((),)) t) V., log 10
E ((1)g(
+
2
(14)
This result is particularly meaningful since it is well known that denoising score matching (Equation (10)) corresponds to a reweighted variational bound on the likelihood of diffu- sion models (Song et al., 2021a).
dæ,- = [f((t) (t) log ((())dt + g()
Here, w denotes a standard Wiener process, where time flows backward. Diffusion models approximate this reverse process by learning the score function v., log ((2)) using denoising score matching loss:
This connection between diffusion models and Latent SDEs, along with the fact that diffusion models can be trained in a simulation-fre manner, motivates us to extend and develop a simulation- free framework for training Latent SDEs.
E [1 t)- V. log d(gg(()/)))12
(10)
(()) (a)
where u!) represents uniform distribution over the in- terval [0, 1J, and sy : gd x [0, + Rd is a learnable approximation of the score function.
4. SDE Matching
Diffusion models can be trained in simulation-free manner because they do not require full simulation of the noising process. Instead, they allow direct sampling of latent vari- ables 2 from the marginal distribution (<=](x). ). In contrast, Latent SDEs are often trained using a posterior process parameterized by a general-form SDE. This renders the marginal posterior distribution (p X) intractable, which prevents simulation-free training.
A key property of the denoising score matching objective is that it can be learned in a simulation-free manner. Due to the simplicity of the forward process, instead of integrat- ing the SDE in Equation (8), we can directly sample from (<2)(z)) and estimate the expectation in Equation (10) using the Monte Carlo method. This property distinguishes dif- fusion models from Latent SDEs. However, as we will see diffusion models are in fact a special case of Latent SDEs.
To address this limitation, we propose SDE Matching a framework for simulation-fis training of Latent SDE mod- els. The key idea behind SDE Matching is to parameter- ize the posterior process' conditional SDE via a learnable
To demonstrate this connection we: (1) derive the reverse SDE for the conditional forward process (Equation (8)), and
3

---

SDE Matching: Scalable and Simulation-Fre Training of Latent SDEs
function Fe 1, X) that directly defines the marginal dis- tributions (=<2((X). This design inherently allows direct sampling of latent variables without numerical integration. Importantly, SDE Matching simplifies only the training pro- cedure while leaving the generative dynamics of the Latent SDE prior process fully flexible.
sampling. The specific parameterization of F.p and the de- pendence of 2, on observations X are user design choices.
Conditional ODE, We assume that F&p is differentiable with respect to and t, and invertible with respect to E. Fixing ob- servations X and noise e, and varying t from 0 to 1, results in a smooth trajectory from so to 21. Differentiating these trajectories over time yields a velocity field that generates the conditional distribution qo (2(2x))),
4.1. Generative Model
the conditional ODE:
In SDE Matching, the prior process is identical to the stan- dard Latent SDE model (see Section 2):
dz = ( X)dt, fo(ent,x)= 0 (e, X)
where
(17)
(18)
dz,- = ha(2,, t)dt + t)d..
(15)
0t
s= sx..'(e..))))
In general, the functions hy and go may be parameterized us- ing neural networks of arbitrary form. However, the choice of parameterization involves some trade-offs, which we will discuss later.
Thus, if we sample za ~ (2 X) and solve Equation (17) up to time t, we obtain a sample 2, ~ 9, (x).
The time derivative of Fg can be computed efficiently using forward-mode differentiation in modern frameworks such as PyTorch (Paszke et al., 2017) or JAX (Bradbury et al., 2018).
Observations Fe, are likewise assumed to be sampled con- ditionally independently from the likelihood distributions It. ~ pe 2 (t.)) for each time step i..
Conditional SDE. Finally, we derive the conditional SDE that defines the posterior process.
4.2. Posterior Process
Given access both the conditional ODE and the score function Zz, log (p(2) X), we follow Song et al. (2021b)
Instead of defining the posterior process through condi- tional SDE (Equation (2)) and then deriving its analytical solution, we propose an alternative approach. We first de- fine the conditional marginal distribution of latent variables (2x for each t, and then derive the corresponding con- ditional SDE with the desired marginal distributions.
and define conditional SDE with marginal distributions (2g X) as follows:
dz, = fo, t, (x)dt + g0 t)dw, where
(19)
(2,, 1, x)
f4(2,, ())
(20)
We construct the posterior process by following three steps:
1
+
2 go (zt. t)g0 (2e, V,, log 9p(2 (X)
1. Define the marginal distribution implicitly through a parameterized function of noise;
+ 2
2., [90 t) (ze, t)]
2. Construct a conditional ODE that satisfies the target marginals;
Here, the diffusion term go is the matrix-valued function from the prior process. It is crucial that the posterior process has the same diffusion term the prior process to ensure that the variational bound is finite. Notably, ge affects only the distribution of trajectories 2(t)xejo.1) while the marginal distributions q (2x (X) do not depend on go.
3. Transition from the ODE to SDE while preserving the marginal distributions.
A detailed description of each step with proofs is provided in Appendix A.
We would like to emphasis that the correspondence be- tween the posterior SDE in Equation (19) and its marginals 4-(22 (2, X) is exact, not approximate.
Marginal Distribution. We define the condi- tional marginal distribution 4p(2,] X) implicitly. First, we introduce a function that, given time t and observations X, transforms noise E into latent variable 21:
Posterior
Evaluating the drift term in Equation (19) presents several challenges. First, it requires access to the conditional score function V., log (* X). which can be computationally expensive. However, for F.p functions that enable efficient log-determinant computation of the Jacobian matrix, the score function can be computed efficiently (e.g., functions linear in E or RealNVP architectures (Dinh et al., 2017: Kingma & Dhariwal, 2018)). The calculation of this score function further discussed in Appendix A.4.
2 = (,)
(16)
where S ~ g(E) = NNe;; 0, 1). This implicitly defines the conditional distribution of latent variables q (xx Al- though, in general, we do not have an explicit form for this distribution, the above definition inherently enables efficient
4

---

SDE Matching: Scalable and Simulation-Fre Training of Latent SDEs
Algorithm 1 Training of SDE Matching
val [0, 1J. u(i) represents a uniform distribution over the set 1, ,.... N, and Te.p(2z, 1, X) satisfies:
Require: X. pe (20), heg go. (o(2)12z)) F.p for learning iterations do
go ()r t, X)= ho(ze, t) (0 2, X). (25)
# Calculation of prior loss Sprior
en 1),
20 = F& (So. 0,
The objective in Equation (21) consists of three terms: the prior loss Ppiirr. the diffusion loss Lait, and the recon- struction loss Lree. The optimization of Cpiior with respect to parameters 0 can be omitted if we are not interested in unconditional sampling or in learning the initial prior pa (zo) example, if we are solely interested in forecast- ing. Otherwise, this term can still be optimized efficiently.
Sprior = DKL (p.(9p((e)
# Calculation of diffusion loss Laitt
~ u(t)
E, 1)
2 X))
(=,, 1, X) 90 (z,, 1) (ho(zz,))) () - (2,, X))
Laitt = || t, X)I22
The other two terms, Laitt and Cree, which previously re- quired the numerical integration of the conditional SDE, can now be estimated more efficiently. Specifically, for each term, we can first sample timestep i, then sample 2e ~ go (x) and finally evaluate the corresponding func- tion. Importantly, SDE Matching allows the estimation of Laif and Free with an arbitrary number of samples evalu- ated in parallel. It also enables inference of the posterior pro- cess (Equation (19)) as special case. However, we found that using single sample was sufficient for our experiments. We summarize the training procedure in Algorithm 1.
# Calculation of reconstruction loss free
i ~ u(i)
Eu I(
= = Po(su, 4x.X) X)
Sree= = - log po ( )2(,)
# Optimization
L Lree + Caift + prior
Gradient step A and yo w.r.t. L
end for
The second challenge involves computing the last term in Equation (20), which can be expensive in -isheiinnniooaal settings. However, it can be computed efficiently if for ex- ample the diffusion term go is of the form ao( (z, (t)) for scalar-valued og and matrix-valued L'g. Another option is if go is a diagonal matrix, where each element (2,. x) depends only on the k-th coordinate of 2,- Furthermore, if go does not depend on the state 2t, this term vanishes. Notably, while this structure limits flexibility, it is identical to constraints in mnemory-efficienn implementations of stan- dard Latent SDE training (Li et al., 2020), which means that SDE Matching does not introduce additional restrictions.
Note that while we use the functions fep (Equation (17)) and fo,p (Fquation (19)) to describe the conditional SDE in the posterior process, these functions are ultimately defined by the reparameterization function F., (Equation (16)) and go. which are the functions we actually parameterize.
The full Bayesian treatment is also possible straightfor- ward extension of the bound in Equation (21) for given prior with a separate variational approximation for 0.
We would like to emphasize that the SDE Matching objec- tive is exactly the same variational bound as used in the adjoint sensitivity method. The key difference lies in the es- timation strategy for the objective. We discuss the tightness of the variational bound in Equation (21) in Appendix B and empirically validate convergence to the true posterior for a linear SDE in Appendix C.I.
4.3. Optimization
I Like standard Latent SDE training, SDE Matching opti- mizes the parameters 0 and yo end-to-end by optimizing the same variational objective. However, the training procedure of SDE Matching can be made simulation- -fee by leverag- ing the above posterior process and rewriting the objective from Equation (3) as follows:
4.4. Sampling
Unconditional sampling at inference- or test-time for SDE Matching follows the same procedure as sampling from a Latent SDE. First, we sample an initial latent variable 20 ~ po(2)) and then integrate the unconditional SDE in Equation (15) using any off the--shhhhff SDE solver. Then, at each desired time step f, we project the latent states 2; to the
- -log (X) S L Lprior + Laitt + Lree
(21)
Lprior = DKI. (gp {)))))(20)
(22)
Lai = E
I (2e, t,
(23) data space by sampling I, from the distribution pi (x))
u(t)g (2a (X)
2
However, if we have access to partial observations X Dt, Br....., Ttn and are interested in forecasting for > tN. we can instead first sample the latent state StN from (2(N (X) and then integrate the prior process dynamics
L N
E u(i)a((a,
log Pa (a 1,)
(24)
where u(t) represents a uniform distribution over the inter-
5

---

SDE Matching: Scalable and Simulation-Free Training of Latent SDEs
using this sample initial condition. This follows because the Latent SDE is Markovian with observations that ditionally independent given the latent state. Importantly, because SDE Matching provides direct access to an approxi- mation to the posterior (smoothing) marginals it can be used for simulation-Irs sampling (inference) of the latent state, whereas conventional parameterization of the posterior pro- cess would require simulating the conditional SDE up to time step in first.
Similarly, interpolation can be performed by inferring only the posterior process dynamics. Alternatively, for more gen- eral conditioning events and steering one could leverage Sequential Monte Carlo methods (Naesseth et al., 2019; Chopin et al., 2020; Wu et al., 2023) for conditional sam- pling, but a detailed investigation of these approaches is beyond the scope of this work.
(a) Training data distribution
Due to the strong connection between SDE Matching and diffusion models-and given that diffusion models have demonstrated exceptional performance in deterministic sam- pling-it is natural to ask whether it is possible to sample deterministically from SDE Matching. Indeed, the prior process and the initial distribution po())) together define sequence of marginal probability distributions pe(2)). While it is possible to learn an ODE corresponding to Pa (2+), it is important to clarify that solving this ODE does not neces- sarily produce meaningful time-series samples.
(b) Adjoint sensitivity method
(c) SDE Matching
Figure I. Training data distribution learned dynamics from 3D stochastic Lorenz attractor. Results for Latent SDE trained with adjoint sensitivity method (bottom left) and SDE Matching (bottom right). Both methods successfully learn the underlying dynamics.
Although the ODE preserves marginal distributions, al- ters the joint distribution of latent variable trajectories. The same applies to the observed variables: while the marginal distributions remain correct, the joint distribution becomes distorted. Therefore, using ODE for deterministic sam- pling of time series data does not generally preserve the correct structure of the generated trajectories.
10a
Adjoint sensitivity
SDE Matching
NELBO
10a
5. Experiments
SDE Matching is simulation-free free framework for training Latent SDE models. Therefore, the goal of this section is to demonstrate that SDE Matching indeed enables computa- tionally efficient training of Latent SDEs. Our objective is not to outperform all other Latent for time series modeling. Instead, SDE Matching is com- patible with existing extensions and can be combined with them for further improvements.
x10
10a
approaches
1000
2000
3000
4000
5000
Iteration
Figure 2. NELBO (Equation (21)) objective with respect to itera- tion step on 3D stochastic Lorenz attractor data. Results for Latent SDE trained with adjoint sensitivity method and SDE Matching. SDE Matching demonstrates approximatly 10 times faster conver- gence terms of iterations.
To this end, we provide various experiments on both syn- thetic and real data. In all experiments, we use the same hyperparameters for both the conventional parameterization of Latent SDE and the parameterization described Sec- tion Additional details on parameterization and training provided in Appendix D. When using the SDE Matching training procedure, jointly optimize the parameters of the generative model 0 in the prior pz()), drift he ha t),
diffusion go (2,, t), and observation model po (=))2)) as well as the parameters of the posterior reparameterization func- tion Fip. All parameters are optimized jointly by minimizing
6

---

SDE Matching: Scalable and Simulation- -rree Training of Latent SDEs
the variational bound in Equation (21). We do not apply annealing or additional regularization techniques during
Table 2. Test MSE on motion capture datasct. We report an average performance based on 10 models trained with deferent random seeds and 95% confidence interval based on t-statistic. The first section of the table contains simulation-based approaches and the second part contains simulation approaches. 'results from (Yildiz et al., 2019), "from (Li ct al., 2020) and 'from (Course & Nair, 2023).
training.
The experiments demonstrate that SDE Matching achieves comparable or better accuracy than simulation-based La- tent SDE training, while significandly reducing computa- tional complexity. Not only does SDE Matching reduce the computational cost of each training iteration, the pa- rameterization of the posterior process also leads to faster convergence in terms of the number of iterations for further gains compared to alternatives. The findings suggest that SDE Matching enhances the scalability of Latent SDEs to longer and nigher-dimensione time series.
Model
Test MSE 1
DTSBN-S (Gan et al., 2015)
34.86 + 0.021 22.96'
npODE (Heinonen et al., 2018)
NeuralODE (Chen et al., 2018)
22.49 + 0.881
ODE (Yildiz et al., 2019)
10.06 = 1.4*
ODE (Yildiz al., 2019)
8.09 = 1.95
The code is available at https://gitte com/ GrigoryBartosh/ wetryyBarrtosssseeeeeeeee ael_mmatcinng.
Latent ODE (Rubanova et al., 2019)
5.98 + 0.28
Latent SDE (Li et al., 2020)
4.03 + 0.2*
ARCTA (Course & Nair, 2023)
7.62 0.93#
5.1. Synthetic Datasets
SDE Matching (ours)
4.50 + 0.32
For the synthetic data experiment, we consider the 3D stochastic Lorenz attractor process from Li et al. (2020) with identical observation generation procedure.
indicating significantly greater stability for long time series modeling.
As shown in Figure 1, both models-on trained using the adjoint sensitivity method and the other with SDE Match- oug-successfully learned the underlying dynamics. How- ever, SDE Matching required only a single evaluation of the drift term in the posterior process for each iteration, whereas the adjoint sensitivity method required 100 simulation steps for this experiment. In terms of absolute runtime, single training iteration with SDE Matching was approximately five times faster.
In Appendix C.2 we provide additional experiments demon- strating robustness of the SDE Matching training procedure.
5.2. Motion Capture Dataset
To validate SDE Matching on real-world data, we follow Li et al. (2020); Course & Nair (2023) and evaluate it on the motion capture dataset from Gan et al. (2015). This dataset consists of motion recordings from subject 35 walking, rep- resented as 50-dimensional time series with 300 observa- tions each. The dataset is split into 16 training sequences, 3 validation sequences, and 4 test sequences, following the preprocessing method from Wang et al. (2007).
Morcover, as demonstrated in Figure 2, SDE Matching leads to faster convergence of the model. We attribute this to the parameterization of the posterior process through the repa- rameterization function Fip (Equation (16)). We hypothesize that since this function directly models the marginal distribu- tions of the posterior process, the model can more efficiently learn compared to integrating conditional SDEs.
We use the same hyperparameters as Li et al. (2020). includ- ing setting the latent state dimensionality to 6 and keeping the number of training iterations unchanged. In Table 2, we report the average performance on the training set over 10 models trained from different random seeds. SDE Match- ing achieves similar performance the adjoint sensitivity method while being significantly more computationally effi- cient.
The combined, per-iteration and convergence, runtime speed-up over 50x compared to the adjoint sensitivity approach for this experiment.
To evaluate robustness with respect to time horison, we compare the gradient norms of the objective function with respect to model parameters for both SDE Matching and the adjoint sensitivity method. When integrating over time hori- zons T E 1,2,5, 10, the adjoint sensitivity method yields the following logno-based based gradient norms: 6.26 + 0.14, 6.95 + 0.20, 7.91 + 0.23 and 8.82 + 0.28. These results demonstrate that the gradient norms grow exponentially with the time horizon for adjoint sensitivity methods
SDE Matching also outperforms the other simulation- approach, ARCTA Course Nair (2023), which can be seen as a special case of SDE Matching. Notably, ARCTA requires drawing around 100 latent samples and evaluations of SDEs per batch, whereas SDE Matching requires only one. We attribute this result to the fact that ARCTA em- ploys a less flexible prior and posterior parameterization, where the volatility function ga is stateinndependent and distribution of latent states q (2) depends heavily on observations close to time t. This dependence makes it
In contrast, SDE Matching maintains a stable gradient norm of 4.92 + 0.24 (in logio scale) across all time horizons,
7

---

SDE Matching: Scalable and Simulation- Training of Latent SDEs
ently encode all uncertainty into the initial conditions the dynamics is completely deterministic. This approach is in- adequate for modeling fundamentally stochastic processes, especially when observations are sparse or uninformative.
(a) Adjoint sensitivity method
Traditionally, Monte Carlo methods have been used to train SDEs (Movellan et al., 2002; Beskos et al., 2006; van der Meulen & Schauer, 2017). Another line of work (Archam- beau et al., 2007a:b) has focused on variational inference techniques for inferring the Latent SDEs's training objective. Discretization is used by Ryder et al. (2018) to enable more expressive variational approximations, and Ha et al. (2018) draw the connection to stochastic optimal control for more flexibility.
(b) SDE Matching
Figure Qualitative comparison of generated video sequences depicting moving pendulum. Samples generated by the model trained with adjoint sensitivity method and SDE Matching with same number of training iterations.
More recently, Li et al. (2020) extended the adjoint sensitiv- ity method from ODEs SDEs. Unlike Neural ODEs, La- tent SDEs are better suited for modeling inherently stochas- tic and chaotic processes and are more robust to data shifts.
challenging to learn long-term dependencies and limits the ability to propagate learning signals effectively.
An extended discussion on the importance of state- dependent volatility functions go. along with experiments on the state-dependent stochastic Lotka-Voltens system, is provided in Appendix C.3.
Many studies have proposed modifications to objective func- tions, introduced regularized dynamics, and improved com- putational efficiency and numerical stability for both Neural ODEs (Kelly et al., 2020; Finlay et al., 2020; Kidger et al., 2021a) and Latent SDEs (Kidger et al., 2021b). However, most of these methods still rely backpropagation through the numerical solutions of differential equations limiting their scalability. In Verma et al. (2024) the authors instead rely on site-based Gaussian approximation and suggest a fixed-point training algorithm.
5.3. Video Dataset
Finally, we evaluate the performance of SDE Matching high-dimennional setting by modeling sequences of 32 x 32 ine. video data data-depiccing moving pendulum.
Using an identical number of training iterations (20k), SDE Matching successfully learns the underlying dynamics in approximately 20 minutes on a single GPU. In contrast, the adjoint sensitivity method requires around 30 hours for the same number of iterations and still fails accurately capture the system's 's dynamics.
Closest to our work Course & Nair (2023) that develops a method for training Latent SDEs in simulation-free man- ner. This approach is restricted to posterior processes with Gaussian marginals and does not support state-dependent diffusion terms in the prior process. Additionally, this method assumes that the latent state in the posterior pro- cess depends only on few temporally closest observations. As a result, for effective optimization requires drawing approximately 100 latent samples, rather than per batch during training. This method can be seen special case of SDE Matching with the corresponding restrictions on the parameterization of the prior process and posterior process.
Visualizations of the generated sequences are provided in Figure 3.
6. Related Work
Differential equations a widely recognized technique for modeling continuous dynamics. Recently they have seen surge of interest in machine learning after the introduction of Neural ODEs by Chen et al. (2018) and Rubanova et al. (2019). Neural ODEs demonstrated strong performance in time-series modeling compared to recurrent neural networks, particularly for irregularly sampled time series. However, Neural ODEs have several limitations.
Recently, Zhang et al. (2024) proposed a simulation-free technique for time-series modeling. However, this ap- proach learns dynamics directly in the original data space rather than in latent space. Furthermore, model non- Markovian processes the authors condition the generative dynamics on a fixed set of past observations, making sim- ulations more computationally expensive and limiting the model's ability to capture long-range dependencies. This can be viewed as a special case of SDE Matching, where the latent space is constructed by concatenation of the most recent observations.
First, they rely on adjoint sensitivity methods for training, which requires numerical integration of gradients and back- propagation through the solutions. Aside from being com- putationally expensive and difficult to parallelize on modern hardware, ODEs can sometimes suffer from numerical insta- bilities (Lea al., 2000). Second, ODE-based models inher-
Another line of research that explores learning stochastic
8

---

SDE Matching: Scalable and Aimulation-Free Training of Latent SDEs
dynamics in a simulation-free manner is diffusion models (Ho al., 2020; Song al., 2021b). As we demonstrated in Section 3, conventional diffusion models can be seen special case of Latent SDEs with only a single observation. The key property that enables efficient training in conven- tional diffusion models is their simple and fixed noising process, i.e., the posterior process.
8. Conclusion
We introduced SDE Matching, a simulation-free framework for training Latent SDEs. By leveraging insights from score- based generative models, we formulated a method that elim- inates the need for costly numerical simulations while main- taining the expressiveness of Latent SDEs. Our approach directly parameterizes the marginal posterior distributions, enabling efficient training and conditional inference.
Recendly Singhal et al. (2023); Bartosh et al. (2023); Nielsen et al. (2024); Sahoo et al. (2024); Bartosh et al. (2024); Du et al. (2024) have shown that the noising process in diffusion models can be more complex, and even learnable, while still preserving the simulation- freeeerffeeeee property. These approaches can also be viewed as special cases of SDE Matching by the re-interpretation of diffusion models as Latent SDEs.
Through both synthetic and real-world experiments, we demonstrated that SDE Matching achieves performance better than existing adjoint sensitivity- based methods, while significantly reducing computational complexity. The ability to estimate latent trajectories with- out solving high-dimensional SDEs makes our method par- ticularly suitable for large-scale time-series and sequence modeling.
comparable
7. Limitations and Future Work
The simulation-free properties of SDE Matching come with certain trade-offs. First, SDE Matching parameterizes the posterior process through the function Fe,e,,, X) (Equa- tion (16)). This function must not only be smooth and invertible with respect to E, it must also provide access to the conditional score function Va, log yo (z) (Equa- tion (20)). These requirements restrict the flexibility of F'p and, consequently, the posterior distribution approximation. Nevertheless, to the best of our knowledge, SDE Match- ing offers the most flexible parameterization that enables simulation-fire access to latent samples 2n ~ (2 X) of the conditional SDE posterior process.
Despite its advantages, SDE Matching introduces trade-off's in terms of posterior parameterization flexibility. Neverthe- less, the best of our knowledge, SDE Matching proposes the most flexible parameterization of the posterior process that allows simulation-free sampling of latent variables. Ex- ploring more expressive function classes for posterior distri- butions and further extending SDE Matching are promising directions for future work.
Furthermore, our approach provides foundation for scaling Latent SDEs to previously infeasible sequential domains such as audio and video. We believe that the scalability and efficiency of SDE Matching will enable broader applica- tions in scientific modeling, finance, and healthcare, where structured uncertainty modeling is critical. Future research may also explore extensions to simulation-fis training of Latent ODEs and connections reinforcement learning and stochastic optimal control.
Another limitation is the computational cost of evaluating the posterior process' SDE drift term for high-dimensional (z. (Equation (15)) of general form, as discussed in Section 4.2. However, it is worth reiterating that other memory-efficient training of Latent SDEs faces this same limitation.
SDE Matching does not constrain the flexibility of the prior process, which defines the generative model. Nevertheless, developing methods that allow even more flexible parame- terization of F., and diffusion terms go remains a promising direction for future research. Additionally, understanding the impact of this flexibility in modeling latent dynamics versus dynamics in the original data space is an interesting open question.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
We believe that thanks to its simulation-free properties, SDE Matching has the potential to scale Latent SDE-based mod- eling to nigh-dimensional applications like audio and video generation and many other applications where they were previously infeasible. It could potentially also contribute to the development of simulation- free methods for training Latent ODE models and learning policies in reinforcement learning. We leave these directions for future work.
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: A unifying framework for flows and diffusions, 2023. URL https://arxiv. Abs/2333.08879
Archambeau, C., Cornford, D., Opper, M., and Shawe- Taylor, J. Gaussian process approximations of stochastic differential equations. In Gaussian Processes in Practice, pp. 1-16. PMLR, 2007a.
9

---

SDE Matching: Scalable and Simulation-Fre Training of Latent SDEs
Archambeau, C., Opper, M., Shen, Y., Cornford, D., and Shawe-Taylor, J. Variational inference for diffusion pro- cesses. Advances in neural information processing sys- tems, 20, 2007b.
Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP. In International Confer- ence Learning Representations, 2017. URL https: / //openreview. www.:/eeetfoouteeeoe http:/oommmmmmddiaeeeeae
Bartosh, G., Vetrov, D., and Naesseth, C. A. Neural diffu- sion models. arXiv preprint arXiv:2310.08337,, 2023.
Du, Y., Plainer, M., Brekelmans, R., Duan, C., Noé, F. Gomes, C. P. Aspuru-Guzik, A., and Neklyudov, K. Doob's lagrangian: A sample-efficient variational approach to transition path sampling. arXiv preprint arXiv:2410.07974 2024.
Bartosh, G., Vetrov, D., and Naesseth, C. A. Neural flow dif- fusion models: Learnable forward process for improved diffusion modelling, arXiv preprint wrXiv22404.12944 2024.
ElGazzar, A. and van Gerven, M. Generative modeling of neural dynamics via latent stochastic differential equa- tions. arXiv preprint arXiv:2412.12113 2024.
Beskos, A., Papaspiliopoulos, O., Roberts, G. O., and Fearn- head, P. Exact and computationally efficient likelihood- based estimation for discretely observed diffusion pro- cesses (with discussion). Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3):333-382, 2006.
Finlay, C., Jacobsen, J.-H., Nurbekyan, L.,, and Oberman, A. How to train your neural ode: the world of jacobian and kinetic regularization. In International conference on machine learning, pp. 3154-3164. PMLR, 2020.
Biloš, M., Sommer, J., Rangapuram, S. S., Januschowski, T., and Günnemann, S. Neural flows: Efficient alterna- tive to neural odes. In Advances in neural information processing systems, volume 34, pp. 22337, 2021.
Gan, Z., Li, C., Henao, R., Carlson, D. E., and Carin, L. Deep temporal sigmoid belief networks for sequence modeling. Advances in Neural Information Processing Systems, 28, 2015.
Bogachev, V. I. Measure theory. Springer, 2007.
Giles, M. and Glasserman, P. Smoking adjoints: Fast Monte Carlo greeks. Risk, 19(1):88-92, 2006.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclauni,, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman S., and Zhang, Q. JAX: composable transformations Python+NumPy programs. 2018. URL http://gittien ctm/ goon///ooll////ooooooooo jax.
Gobet, E. and Munos, R. Sensitivity analysis using Itô- Malliavin calculus and martingales, and application to stochastic optimal control. SIAM Journal on control and optimization, 1765)11676- 2005.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Grathwohl, W., Chen, R. T. Q.. Bettencourt, J., and Du- venaud, D. Scalable reversible generative models with free-form continuous dynamics. International Confer- ence on Learning Representations, 2019. URL https: // //opnreeiie www.iiiweettte...... tiree/forumee forum? d=TJxgknCKKR.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learn- ing phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint 146.v:1406.1078 2014.
Ha, J.-S., Park, Y-J., Chae, H.-J., Park, S.-S., and Choi, H.- L. Adaptive path-integral autoencoders: Representation learning and planning for dynamical systems. Advances in Neural Information Processing Systems, 31, 2018.
Chopin, N., Papaspillopooloss, O., et al. An introduction to sequential Monte Carlo, volume 4. Springer, 2020.
Heinonen, M., Yildiz, C., Mannerström, H., Intosalmi, J., and Lähdesmäki, H. Learning unknown ode models with gaussian processes. In International conference on ma- chine learning, pp. 1959-1968. PMLR, 2018.
Course, K. and Nair, P. Amortized reparametrization: ef- ficient and scalable variational inference for latent sdes. Advances in Neural Information Processing Systems, 36, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6855 2020.
Daems, R., Opper, M., Crevecoeur, G., and Birdal, T. Vari- ational inference for SDEs driven by fractional noise. In The Twelfth International Conference on Learning Representations, 2024. URL https: openrevie.. net/ forum? d=rtxx894 8B94JMS.
Hodgkinson, L., van der Heide, C., Roosta, F., and Mahoney, M. W. Stochastic normalizing flows. arXiv preprint arXiv:200..09547, 2020.
10

---

SDE Matching: Scalable and Simulation-Fre Training of Latent SDEs
Kelly, J., Bettencourt, J., Johnson, M. J., and Duvenaud, D. K. Learning differential equations that easy to solve. Advances in Neural Information Processing Sys- tems, 33:4370-43388 2020.
Nielsen, B. M. G., Christensen, A., Dittadi, A., and Winther, O. Diffenc: Variational diffusion with learned encoder. In The Twelfth International Conference Learning Representations, 2024. URL ptttps://openzevier net / forum? d= bQWTG.
Kidger, P., Chen, T., and Lyons, T. J. hey, that's not an ode": Faster ode adjoints via seminorms. In ICML, pp. 5443-5452, 2021a.
Oh, Y., Lim, D.-Y., and Kim, S. Stable neural stochastic differential equations in analyzing irregular time series data. arXiv preprint arXiv:2402.14989, 2024.
Kidger, P., Foster, J., Li, X. C.,, and Lyons, T. Efficient and accurate gradients for neural sdes. Advances in Neu- ral Information Processing Systems, 34:18747-18761, 2021b.
Oksendal, B. Stochastic differential equations. Springer, 2003.
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617-2680, 2021.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint 1411v::412.6990 2014.
Kingma, D. and Dhariwal, P. Glow: Generative flow with invertible Ixl convolutions. Advances in neural information processing systems, 31, 2018.
Park, B., Lee, H., and Lee, J. Amortized control of contin- uous state space feynman-kan model for irregular time series. arXiv preprini arXiv:2410.05660) 2024.
Lea, D. J., Allen, M. R., and Haine, T. W. Sensitivity analysis of the climate of a chaotic system. Tellus A: Dynamic Meteorology and Oceanography, 52(5):523- 532, 2000.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E.,
DeVito, Z., Lin, Z., Desmaison, A., Antiga, L.,, and Lerer, A. Automatic differentiation in pytorch. 2017.
Peluchetti, S. and Favaro, S. Infinitely deep neural networks diffusion processes. International Conference Ar- rificial Intelligence and Statistics, pp. 2020.
Lee, J. M. Introduction to Smooth Manifolds. Springer, 2012.
PMLR,
Li, X.. Wong, T.-K. L., Chen, R. T., and Duvenaud, D. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882. PMLR, 2020.
Peters, J., Bauer, S., and Pfister, N. Causal models for dy- namical systems. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 671-690. 2022.
Lipman, Y., Chen, R. T. Q. Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference Learning Represen- tations, 2023. URL https: / ppw.revee...ee.......eee forum? wtw.::ddd@PwwwPoPP.....
Rubanova, Y., Chen, R. T., and Duvenaud, D. K. Latent ordinary differential equations for iregularly-sampped time series. Advances in neural information processing systems, 32, 2019.
Rudin, W. Differential equations, dynamical systems, and linear algebra. Tata McGraw- Education, 2006.
Liu, X., Gong, C., and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference Learning Representations, 2023, URL httphhtttttssseeeeeee //wp///eppeeeeee//////epe fort/?foum? i JTTIIITTTnn
Ryder, T., Golightly, A., McGough, A. S., and Prangle, D. Black-box variational inference for stochastic differen- tial equations. In International Conference on Machine Learning, pp. 4423 PMLR, 2018.
Mider, M., Schauer, M., and Van der Meulen, F. Continuous- discrete smoothing of diffusions. Electronic Journal of Statistics, 15(2):4295-4445 2021.
Sahoo, S. S., Gokaslan, A., Sa, C. D., and Kuleshov, Diffusion models with learned adaptive noise processes, 2024. URL https: /ttp//ppeeeeeeeeeeeeeeeeeeeeeeee net/ forum? i id=8g2ttBnrpi.
Movellan, R., Mineiro, P., and Williams, R. J. A monte carlo em approach for partially observable diffusion pro- cesses: theory and applications to neural networks. Neu- ral computation, 14(7):1507-1544, 2002.
Sarkka, S. and Solin, A. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019,
Naesseth, C. A., Lindsten, F., Schön, T. B., et al. Elements of sequential Monte Carlo. Foundations and Trends@ in Machine Learning, 12(3):307-392, 2019.
Singhal, R., Goldstein, M., and Ranganath, R. Where to diffuse, how to diffuse, and how to get back: Automated learning for multivariate diffusions. In The Eleventh

---

SDE Matching: Scalable and Simulation-Free Training of Latent SDEs
International Conference on Learning Representations, 2023. URL https::/p/............. http::::///ppeeeeeee..oom forum? id=osei3I20ia,
Zhang, X., Pu, Y., Kawamura, Y., Loza, A., Bengio, Y., Shung, D. I.., and Tong, A. Trajectory flow matching with applications to clinical time series modeling, In Advances in Neural Information Processing Systems, volume 37, 2024.
forumm
Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. Ad- vances in Neural Information Processing Systems, 34: 1415-1428, 2021a.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- cling through stochastic differential equations. In In- ternational Conference on Learning Representations, 2021b. URL htt.t:ppp///peeeeeeee..... http://wwwwweeeerrreeeeee 1d=PxTIG12RRHS,
Tzen, B. and Raginsky, M. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv: (0t.:::99000088880 2019.
Vadillo, F. Comparing stochastic lotka-volterra predator- prey models. Applied Mathematics and Computation, 160:181-189, 2019.
van der Meulen, F. and Schauer, M. Bayesian estima- tion of discretely observed multi-dimensional diffusion processes using guided proposals. Electronic Journal of Statistics, 11(1):2358 2396, 2017. doi: 10.1214/ 17-EJS1290. URL https: http:://dooi.ee otg///..org......//oo////o1 1214/ 17-EJS1290.
Verma, P., Adam, V., and Solin, A. Variational Gaussian process diffusion processes. In International Conference Artificial Intelligence and Statistics, pp. 1909-1917. PMLR, 2024.
Wang, J. M., Fleet, J., and Hertzmann, A. Gaussian pro- cess dynamical models for human motion. IEEE transac- tions on pattern analysis and machine intelligence, 30(2): 283-298, 2007.
Wu, L., Trippe, B., Naesseth, C. A., Blei, D., and Cumning- ham, P. Practical and asymptotically exact conditional sampling in diffusion models. In Advances in Neural Information Processing Systems, volume 36, pp. 31372- 31403, 2023.
Yang, J. and Kushner, H. A Monte Carlo method for sensitivity analysis and parametric optimization of non- linear stochastic systems. SIAM journal on control and optimization, 29: 1991.
Yildiz, C., Heinonen, M., and Lahdesmaki, H. Ode2vae: Deep generative second order odes with Bayesian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
12

---

SDE Matching: Scalable and Simulation-Free -Fee Training of Latent SDEs
A. Detailed Description of the Posterior Process
A.1. Marginal Distribution
The posterior marginal distribution is defined by a transformation F from noise for each t and X:
2 = t,, E~ g(E) = O,
(26)
Assuming that Pe invertible and differentiable in E for each and X. Then, we can use the standard change of variables result for distributions (Rudin, 2006; Bogachev, 2007) to derive the density &z
OF (z,, xX
(x(2, q(E) c= 1
(27)
dz
where 1-11 the inverse of Pg and denotes the absolute value of the determinant. This a well-known result of normalizing flows (Papamakarios al., 2021) applied to our setting.
A.2. Conditional ODE
We leverage results based on flows, solutions to ODEs, inspired by Biloš et al. (2021); Lee (2012); Bartosh et al. (2024) to turn a flow Fep into its corresponding generating ODE f.p. Proposition A.1. Let F.p RD x X R.D be smooth function such that F, X) is invertible Vt, X. Then, () is flow with the infinitesimal generator
dFF (F,(e,t,
(28)
dl
which is smooth vector field J such that
OF (E,
()),
(29)
Øt
Proof. The result is consequence of the Fundamental Theorem on Flows (Lee, 2012, Theorem 9.12) for the flow defined by X))
Using Proposition A.I with = F (2,,t, X), identifying that (F (2,, X),,, = 2g, lets us construct the conditional ODE whose solution matches those of the flow 2, = Fe t)):
dz. = [p(2t, L, X)dl,
where
(30)
OF (E,t,,
Folze,t,
(31)
Ot
e= (xx,,)))
Initializing using 2g ~ [0(20| X) and solving the above conditional ODE in Equation (30) ensures 2, Fo X. for
8 ~ q(E), where denotes equal in distribution.
A.3. Conditional SDE
To derive the conditional SDE we leverage a result by Song et al. (2021b) that relates the marginal distributions of an SDE with its corresponding (probability flow) ODE, restated in Proposition A.2 for convenience. Proposition A.2. The marginal distributions Pe(zs) of the SDE
dz, h((2,, t)dt + g(2e, f)dw,
(32)
are, under suitable conditions on the drift and diffusion terms and g, identical to the distributions induced by the ODE
dz: [b(2e))) g(2, t)g T (2x, V logg((z)) 1 Vz. (g(2,, t)g (z (zc. dt.
(33)
2
13

---

SDE Matching: Scalable and Aimulation-Free Training of Latent SDEs
Proof. See Song et al. (2021b, Appendix D.I).
Note that this equivalence lets us easily find the corresponding (probability flow) ODE. However, we can equally well use it in the reverse order with a known ODE to construct the corresponding SDE by selecting a diffusion term g and using Equation (33) compute the drift term h.
For the prior process diffusion term (=(,) with velocity field Equation (30) apply Proposition A.2 to find the conditional SDE with marginals (2+ X)
dæ fa L, X)dt + go(ze, t)dw, where
(34)
1 (z t, X) = fo, ()) (z t)gg (Ze, V. log (z (X) V [g0 t)ga (z,, 0).
(35)
This result follows by simply identifying terms in Proposition A.2 with our specific case
h(z, t) fo,p(2t, g(z,,)) t) = ())
A.4. Parametcrization
As discussed in Section 4.2, evaluating the posterior SDE (Equation (19)) requires access to the conditional score function
V.. log [2 X). Since we parameterize the posterior marginals (2 X) implicitly through an invertible transformation
Ppp (Equation (16)) of random variable ~ ((E) into 2t, the score function can in general be computed using Equation (27):
Vx log (o = V.. [ogg(()
OF, (2e, X
+ log
(36)
c= (x)
Oz.
The first term Equation (36) represents the log-density of the noise distribution, which straightforward to compute. The second term is the log-determinant of the Jacobian matrix of the inverse transformation. this Jacobian is available, the score function can be efficiently computed using automatic differentiation tools.
In this work, we parameterize the function Fsp as follows:
t,, X) Mu (X, ox t)s, and (z, (2,,,X) = 1hp (X,
(37)
where oi is matrix valued function. Substituting this parameterization into Equation (36), the score function simplifies to:
V log (2 = [o X, t)o (X, (x)))= (Xp(X,t) t) - 21) = -0,-" (X, ( F (x.x.))
(38)
The SDE Matching framework allows for other, potentially more flexible, parameterizations the function Fg. However, we leave the exploration of such alternatives for future research.
B. Theoretical Considerations
In this section, we provide an extended discussion on the connection between the prior and posterior processes. Our primary goal is to address the question: under what conditions does the variational bound in Equation (21) become tight and match the true likelihood of the model?
The variational bound becomes tight when the approximate posterior distribution exactly matches the true posterior. In the case of the Latent SDE, this means that the posterior process must match the true posterior of the prior process.
The prior process (Equation (15)) is given by:
dz, ho ho(2,, t)dt + go t)dw.
(39)
14

---

SDE Matching: Scalable and Simulation-Free Training of Latent SDEs
Given series of observations X, the true posterior process be derived via Doob's h-trranforrm (Mider et al., 2021; Park al., 2024), yielding:
dz = ho(2,, (0(2,, t)9 (zt, tVV,, logpo (X1l2t) dt go(2,, D)dw
(40)
where X, {its : Is E X, I) denotes the set of future observations from time onward.
We can also consider the corresponding deterministic process that follows the posterior marginals pe pa(ee) X)
dz: hol(zet,
(41)
dt
1 Vz log (2x x)
1 V..
= =ho(zz 1) + go 1)90 (Ex, 1) Vx, log (Xe|zs) -
[gu0 (z)0 0] (42)
2
Now, consider the approximate posterior process (Equation (19)) of the form:
dæp = (z, t, X)dt + go (2,, t)dw.
(43)
The approximate process matches the true posterior process if the drift terms are equal: (a, X) ho(z,, L, X) or equivalently, if the corresponding deterministic processes (Equations (17) and (42)) match: (t, x) ho t, X). In terms of the reparameterization function (e, X) (Equation (16)) that defines the approximate posterior process, tthe variational bound becomes tight, and the approximate posterior process matches the true posterior, if F.p learns the solution trajectories of the ODE defined by ha z,, 1, X) (Equation (42)).
Unfortunately, as is typical in variational inference, the true posterior intractable. Nevertheless, its analytical form provides valuable insight into the behavior that the approximate posterior should approximate. Therefore, with sufficiently flexible reparameterization function (e, I, X), the SDE Matching approach should, in principle, be capable of learning the true posterior process and the variational bound becomes equal to the negative log-marginal likelihood.
C. Additional Results
C.1. Convergence to Likelihood
To empirically validate the tightness of the ELBO SDE Matching, as discussed in Appendix B, we set up a linear stochastic system of the form:
dæ, F(t)2,dt + L(f)dw
(44)
H()= Tt.
(45)
where r2 ~ For this system, the true log-marginal likelihood can be computed using the Kalman filter (Sarkka Solin, 2019).
In this experiment, we use simple coefficients: F(t) -t, L(t) t, (1) 1, and R(t) 0.01. The model is parameterized identically
experiment the 3D stochastic Lorenz attractor (Section 5.1). As shown in Appendix C.3, the SDE Matching ELBO closely converges to the true log- marginal likelihood. Furthermore, similar to results in Section 5.1, SDE Matching achieves significantly faster convergence than the adjoint sensitivity method and requires approximately 20x less time per training iteration.
C.2. Robustness of Gradients
We present additional experiments to assess the robustness of SDE Matching.
Using the setup from Appendix C.1, we compare the gradient norms and variances over training time. As shown in Figure 4, SDE Matching consistently exhibits both lower gradient norms and reduced variance compared to the adjoint sensitivity method.
15

---

SDE Matching: Scalable and Simulation- Free Training of Latent SDEs
AIjARE pomsltuuty S0e Moleting
38
177
121
Nt.00
121
:01
10
30"
13"
2000
4000
5000
8023
10000
Itarsion
Rorution
(b) Norm of gradient
(c) Variance of gradients
Convergence of NELBO
Figure 4. Linear system training dynamics for Latent SDE trained with adjoint sensitivity method and SDE Matching.
Ground Truth
Scate Incepardent vclutillty pulf!
Stats dapendent voletilty gylt.
13-
2.
2-
Figure 5. Lotka-VVlterra system, learned trajectories for SDE Matching with state-independent and state-dependent volatility.
C.3. Importance of State-Dependent Diffusion Term
We also present an example that highlights the importance of state-dependent diffusion term (volatility), a feature not supported by the previous simulation-free approach ARCTA (Course & Nair, 2023).
To demonstrate this, we design experiment based on the stochastic Lotka-Volteraa system with dynamics:
dr (ar + Bry)dl + ardw
(46)
dy (ōry + ry)dt aydw.
(47)
We follow the setup from Course & Nair (2023) (Appendix G.1) with volatility coefficient 0.15 and use the same parameterization in Section 5.1.
We then apply SDE Matching to train models with both state-independent and state-dependent volatility functions. As illustrated in Figure 5, the model with state-independee volatility fails to accurately capture the correct trajectory structure, whereas the state-dependent model succeeds.
State-dependant volatility is often a relevant property many dynamical systems e.g. finance (Oksendal, 2003), biology (Vadillo. 2019), neuroscience (ElGazzar & van Gerven, 2024), and causal inference (Peters al., 2022), where Latent SDEs be applied for modelling. The Lotke-Volterra system which we this experiment is an example of such biological system. In Neural SDEs, state-dependant volatility also leads dynamics that is more robust to distribution shift (Oh et al., 2024). Finally, the state-dependant volatility is an important contributing factor in our MOCAP experiment (Section 5.2), where the state-dependent SDE Matching and Li et al. (2020) both perform much better than the state-independenl ARCTA (Course & Nair, 2023). Therefore, we believe that, together with more general marginals of the posterior process through normalising flows, enabling simulation free training with state-dependant volatility is also an important contribution SDE Matching.
16

---

SDE Matching: Scalable and Simulation- Frree Training of Latent SDEs
D. Implementation Details
For the experiments presented in Section 5, we follow the setup from Li al. (2020) for both the 3D stochastic Lorenz attractor and the motion capture dataset. We use the same parameterizations and hyperparameters for the prior process, including the observation models pe (24 (x) and the coordinate-wise diagonal diffusion term go (2t,
We make slight modification to the parameterization of the posterior process. First, we do not use encoder, i.e., separate network that defines the initial conditions of the posterior process. Second, Li et al. (2020) propose using time-reversal GRU layer (Cho et al., 2014) to aggregate information from observations into what they refer to as context. This context is then used to predict the drift term of the posterior SDE via an MLP. To maintain consistency with Cho et al. (2014), we use the same hyperparameters for the GRU and ML.. However, instead of predicting the drift term, we use the MLP to predict the functions μ4 and log dp (Equation (37)) based on the context computed from all observations and the time step I. We use diaganal parameterization dep-
For the moving pendulum video experiment, follow the setup from Course & Nair (2023) for both data generation and the parameterization of the prior process. For the posterior process, we use the same parameterization as in previous experiments, but replace the MLP with a convolutional encoder from Course & Nair (2023) to process image observations.
For optimization, we use the same training hyperparameters, including the Adam optimizer (Kingma, 2014) and the same number of training iterations. However, we do not apply any additional regularization, reweighting, or annealing techniques.
We believe that there exists much more efficient ways parameterize the posterior process for SDE Matching. However, in this work, primary goal to provide fair comparison with the standard Latent SDE training approach rather than focusing on specific design choices such as prior and posterior model architectures.
17


Reviews:
[{"review_id": "T66gmWa3Ss", "reviewer": "Reviewer_TN6r", "summary": "This paper builds on the observation that the reverse process in score-based diffusion models can be seen as a neural SDE (as fundamentally it is a process with a parameterized drift). Then the authors try to exploit this connection to develop a simulation-free training scheme for latent SDEs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable feedback, which helps us to improve the paper. Below, we address the questions and comments raised in the review.\n\n**Other Comments:**\n\nWe apologise for any confusion regarding terminology. Allow us to provide a clearer definition of the term “simulation-free”.\n\nIn the diffusion literature, simulation-free implies that explicit simulation of dynamics (numerical integration) is not required during training. For example, the adjoint sensitivity method [1] requires numerical simulation of the posterior SDE (Eq. 2) to sample latent states $z_t \\sim q_\\phi(z_t|X)$ to evaluate the loss function (Eq. 5). In contrast, thanks to our alternative parameterization of the posterior process, SDE Matching allows, by design, direct sampling of $z_t$ via inference of the reparameterization function $F_\\phi(\\epsilon, t, X)$ (Eq. 16), eliminating the need for numerical simulation of the posterior SDE. This makes the training process simulation-free.\n\n**Questions:**\n\n1. Consistency Models (CM) as originally introduced in [2] learns a function $f_\\theta(x_t, t)$ that approximates the solutions of a fixed marginal ODE starting at $x_t$ and ending at some $x_0 \\sim p_{data}(x_0)$.\n    \n    In SDE Matching, we introduce a reparameterization function $F_\\phi(\\epsilon, t, X)$ (Eq. 16). Based on $F_\\phi$, we derive a conditional ODE (Eq. 17) and then a conditional SDE (Eq. 19) that define the posterior process.\n    \n    While both approaches involve a pair of an ODE and its solver, the nature of these ODEs differs: in CM, the goal is to approximate solutions of an unconditional ODE, whereas in SDE Matching we derive an exact ODE as an intermediate step used to define a conditional SDE. Thus, while there is a connection, they are fundamentally different approaches.\n    \n2. SDE Matching does not incur additional learning cost by parameterizing the posterior process via the reparameterization function $F_\\phi(\\epsilon, t, X)$ (Eq. 16) — this is a central point of our approach.\nUnlike diffusion models, the posterior process in a Latent SDE is not parameter-free. It is a complicated process conventionally defined by the drift function $f_\\phi(z_t, t, X)$ in the SDE (Eq. 2), and sampling from the posterior requires simulating this SDE.\n    \n    By reparameterizing the posterior process, we neither introduce new parameters nor increase the complexity of the dynamics. Instead, we reparameterise the same dynamics in a different way, enabling direct sampling of latent variables $z_t$ from the posterior marginals without numerically integrating the SDE.\n    \n    The key advantage of SDE Matching is that a single training iteration takes asymptotically less time (see rebuttal to JkpK). As discussed in Section 5.1 (lines 320–322), for the 3D Lorenz attractor dataset, one iteration of SDE Matching takes approximately $5$ times less time. Figure 2, while not the main result, additionally shows that SDE Matching not only has faster iterations but also converges more quickly.\n    \n3. To demonstrate the scalability of SDE Matching, we provide two additional experiments.\n    \n    First, to evaluate scalability with respect to time, we use the experimental setup described in Section 4.1 and compare the gradient norms of the objective function with respect to the model parameters for both SDE Matching and the adjoint sensitivity method. When integrating over time horizons $T \\in {1, 2, 5, 10}$, the adjoint sensitivity method yields the following $\\log_{10}$-based gradient norms: $6.26 \\pm 0.14$, $6.95 \\pm 0.20$, $7.91 \\pm 0.23$, and $8.82 \\pm 0.28$. This demonstrates that the gradient norms grow exponentially as the time horizon increases.\n    \n    In contrast, SDE Matching maintains a stable $\\log_{10}$-based gradient norm of $4.92 \\pm 0.24$ across all time horizons, indicating better stability for long time series modelling.\n    \n    Second, to demonstrate scalability with respect to dimensionality, we designed an experiment where the model learns a sequence of images (a video) depicting a moving pendulum. SDE Matching successfully captures the underlying dynamics and generates realistic samples.\n    \n    You can find examples of generated sequence at the anonymised link: https://imgur.com/a/aMfSuHB.\n    \n4. Please refer to rebuttal to Reviwer C8BY (section “Other Comments”), where we discuss the tightness of the variational bound in SDE Matching.\n5. We hope that the clarifications highlighting the efficiency of our approach, along with additional discussion and experimental results demonstrating the scalability of SDE Matching, will strengthen your support for the acceptance of our paper.\n\nWe will include clarifications and additional results, along with detailed explanations of the experimental setups, in the camera-ready version if accepted.\n\n[1] Li et al. “Scalable gradients for stochastic differential equations”\n\n[2] Song et al. “Consistency models”", "claims_and_evidence": "The main claim of this paper is that the authors propose a \"simulation-free\" (definition of which wasn't made clear, see my question below) method for training a latent SDE. The derivation is provided and some experimental results support the claims - although not comprehensive.", "methods_and_evaluation_criteria": "The paper evaluates the training method on some benchmark problems. Although, it can be definitely said that the evaluation is not comprehensive.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experiments include two main parts, synthetic datasets and the motion capture dataset. While these experiments produce reasonable results, similar papers however include more examples, typically, e.g. moving MNIST or bouncing balls, and similar. I will return to this point in my questions.", "relation_to_broader_scientific_literature": "This is a continuation of a long line of papers in latent SDEs. Typical methods require long simulations, whereas the current method tries to avoid it; in that sense, it has a distinct purpose.", "essential_references_not_discussed": "none", "other_strengths_and_weaknesses": "See my comments/suggestions - questions.", "other_comments_or_suggestions": "1) please define simulation-based and simulation-free methods more precisely. Do not assume that every reader knows exactly what you mean by these.", "questions_for_authors": "1) The paper essentially tries to formulate the posterior distribution in a latent SDE as a pushforward or some simple distribution. This feels like it is similar to consistency models - can authors elaborate on this comparison?\n\n2) I am not totally convinced that the extra cost of learning the pushforward mapping makes the method really efficient overall w.r.t. simulation-based methods. Figure 2 shows fast convergence, but not in computation time, but in iterations. Could you plot the same graph for computational cost, including any pretraining?\n\n3) Is this method scalable after all? Earlier papers (see my earlier comment) have various image experiments.\n\n4) Is there any scope for theory, assuming that the pushforward measure is accurate enough? Of course, this depends on what's out there for latent SDEs already, but worth discussing.\n\n5) My final comment about this paper is that it falls short of sufficient demonstration of the costs and behaviour of this method. To be clear, I am not interested in to see that this method beats all other baselines - as authors noted, this method can  probably also be used in other baseline methods. However, authors should clearly demonstrate via a series of detailed experiments the behaviour and scalability of this method, even if the results are partially negative. In the current paper, the results are thin and not comprehensive, making it unclear whether this method is sufficiently explored.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HqZRgtEw4z", "reviewer": "Reviewer_C8BY", "summary": "This paper improves the method of Course & Nair for variational inference in latent SDEs in 2 ways:  A better recognition network and more flexible marginals through normalizing flows.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive feedback regarding the speed, simplicity, and sensibility of our method, as well as the discussion of its connections to diffusion models. Below, we address the questions and comments raised in the review.\n\n**References:**\n\n- We will cite the *Nature* paper by Course & Nair in the camera-ready version if accepted.\n\n**Weaknesses:**\n\n1. To demonstrate the scalability of SDE Matching, we provide additional experiments (please refer to rebuttal to TN6r, Question 3).\n2. We would like to point out that, compared to the work of Course & Nair, in addition to allowing more flexible marginals of the posterior process, SDE Matching also supports a state-dependent diffusion term $g_\\theta(z_t,t)$. This is important in cases where the underlying dynamics exhibit state-dependent volatility. \n    \n    To demonstrate this difference, we designed an experiment similar to the setup in Section 4.1, using a stochastic Lotka–Volterra system with a dynamic of the following form:\n    \n    $dx=(\\alpha x-\\beta xy)dt+\\sigma xdw$, $dy=(\\delta xy-\\gamma y)dt+\\sigma ydw$\n    \n    We then applied SDE Matching to train models with both state-independent and state-dependent volatility functions. Visualisations of the learned trajectories are available at the anonymised link: https://imgur.com/a/jh8sM0Q. It is evident that the model with state-independent volatility fails to capture the correct form of the trajectories.\n    \n3. Please refer to the rebuttal to JkpK for the discussion on training and clarifications on inference (including interpolation).\n\n**Other Comments:**\n\nWe would like to highlight that, in contrast to methods such as Flow Matching, both SDE Matching and the adjoint sensitivity method do not match marginals. Instead, they match the distributions of trajectories of the posterior and prior processes. SDE Matching essentially proposes an alternative parameterization of the posterior process that, by design, enables training of the Latent SDE model by minimising the same objective, but in a simulation-free manner.\n\nNevertheless, the question of the limitations on the tightness of the ELBO in SDE Matching is interesting. Due to the limited space, we may not include rigorous derivations, but we outline the theoretical aspects of this question below.\n\nIn the case of the Latent SDE, the variational bound becomes tight when the posterior process matches the true posterior of the prior process.\n\nThe prior process (Eq. 15) is given by:\n\n$dz_t=h_\\theta(z_t,t) dt+g_\\theta(z_t,t)dw$\n\nGiven a series of observations $X$, the true posterior process can be derived via Doob’s $h$-transform, yielding:\n$dz_t=\\Big[\\overbrace{h_\\theta(z_t,t)+g_\\theta(z_t,t)g^\\top_\\theta(z_t,t)\\nabla_{z_t}\\log p_\\theta\\left(X_t|z_t\\right) }^{h_\\\\theta(z_t,t,X)} \\Big] dt+g_\\theta(z_t,t)dw,$\n\nwhere $X_t=\\\\{x_s:x_s\\in X,t\\leq s\\\\}$ denotes the set of future observations from time $t$ onward.\n\nWe can also consider the corresponding deterministic process that follows the true posterior marginals $p_\\theta(z_t|X)$:\n\n$$\\frac{dz_t}{dt}=\\bar{h}^\\theta(z_t,t,X)=h_\\theta(z_t, t)+g_\\theta(z_t,t)g^\\top_\\theta(z_t,t)\\left[ \\nabla_{z_t}\\log p_\\theta\\left(X_t|z_t\\right)-\\frac{1}{2}\\nabla_{z_t}\\log p_\\theta\\left(z_t|X\\right)\\right]-\\frac{1}{2}\\nabla_{z_t}\\cdot\\left[g_\\theta(z_t,t)g^\\top_\\theta(z_t,t)\\right]$$\n\nNow, consider the approximate posterior process (Eq. 19):\n\n$d z_t=f_{\\theta,\\phi}(z_t,t,X)dt+g_\\theta(z_t,t)dw$\n\nThe approximate process matches the true posterior process if the drift terms are equal: $f_{\\theta,\\phi}(z_t,t,X)\\equiv h_\\theta(z_t,t,X)$, or equivalently, if the corresponding deterministic processes (Eq. 17) match: $\\bar{f}^{\\theta,\\phi}(z_t,t,X)\\equiv \\bar{h}^\\theta(z_t,t,X)$. In terms of the reparameterization function $F_\\phi(\\epsilon,t,X)$ (Eq. 16) that defines the approximate posterior process, we may say that the variational bound becomes tight, and the approximate posterior process matches the true posterior, if $F_\\phi$ learns the solution trajectories of the ODE defined by $\\bar{h}_\\theta$.\n\nTherefore, with a sufficiently flexible reparameterization function $F_\\phi$, the SDE Matching approach should, in principle, be capable of making the variational bound tight.\n\n**Questions:**\n\nSDE Matching shows similar performance to Li et al. as there is an overlap of the confidence intervals. We attribute the slightly better performance of Li et al. to a suboptimal parameterization of the posterior process. In this work, we intentionally kept the parameterization as close as possible to the conventional setup from Li et al. to ensure a fair comparison. In principle, with a sufficiently flexible parameterization, SDE Matching should demonstrate comparable performance.\n\nWe will include additional discussions and experimental results with detailed setup explanations in the camera-ready version. We trust these clarifications, highlighting the novelty and scalability of our work, will enhance your support for acceptance!", "claims_and_evidence": "The claims are basically that it's a fast and flexible approach.  Also that it's much faster than the Sota in terms of predictive accuracy, Li et al.  However I don't think the speed claim was actually verified empirically (though I believe them).", "methods_and_evaluation_criteria": "Just argumentation and a few small-scale experiments.", "theoretical_claims": "The motivations and connections in section 3 are spot on, make sense, and are well-explained.  There aren't really any theorems but I don't think there need to be.", "experimental_designs_or_analyses": "There are 2 medium-scale experiments.  Not much math.  But it's such a sensible method I don't mind.", "relation_to_broader_scientific_literature": "This is well-discussed in the paper, pointing out that this is bringing ideas from diffusion models to latent SDE inference (using normalizing flows to match marginals)", "essential_references_not_discussed": "Maybe the Nature version of the Course & Nair work.", "other_strengths_and_weaknesses": "Weaknesses:\n - Still using the mocap experiment (introduced in 2019) as the largest experiment.  I guess it's a pain to re-run Li et al but that was 5 years ago, and seems pretty small now.\n - The similarity to Course & Nair.  however, the simplicity and better perf of the new version make this OK.\n - didn't include a time comparison on the mocap experiments.  As it stands, it's not clear what the pareto curve for speed vs accuracy is.", "other_comments_or_suggestions": "\"Similarly, interpolation can be performed by inferring only the posterior process dynamics\" not clear what this means.\n\nWould love to see more discussion of to what extent matching marginals limits the tightness of the ELBO.", "questions_for_authors": "Why do you think your method didn't match Li et al's performance on mocap?  Should it be able to, in principle?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I69f3IDNpY", "reviewer": "Reviewer_JFqV", "summary": "The author(s) proposed SDE-matching, a simulation-free method to fit latent SDE models. The key idea is to use a differentiable normalizing flow method to learn the Markovianization of the posterior SDE and match the probability flow ODE defined by the normalizing flow to get back the SDE.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that the reviewer finds our claims to be well supported theoretically, the experiments well designed, and the discussions insightful. We are especially grateful that the reviewer took the time to reproduce our experiments. We sincerely thank the reviewer for such attention to our paper and for the valuable feedback. In response to the comments and questions raised, we provide the following clarifications and commitments.\n\n**Other Comments**\n\n1. Indeed, in SDE Matching we do not explicitly solve any SDE during training. Instead, we parameterize the posterior process in such a way that allows us to estimate the objective function in $\\mathcal{O}(1)$ space and time. The purpose of Table 1 is primarily to reflect the computational budget in terms of number of drift/diffusion term evaluations per training iteration.\n    \n    Nevertheless, we would like to note that in SDE Matching, when sampling latent variables $z_t$ to estimate the diffusion loss (Eq. 23), we sample them from the posterior marginals $q_\\phi(z_t|X)$, which, by design, are solutions of the posterior SDE (Eq. 19). Thus, during optimization, we are effectively computing gradients with respect to the solutions of the posterior SDE.\n    \n2. In the Latent SDE model, training essentially requires two things: the ability to sample from the posterior marginals $q_\\phi(z_t|X)$ and access to the posterior SDE. With the conventional parameterization, we directly define the SDE via its drift, which necessitates numerical simulation in order to sample from the unknown marginals.\n    \n    The high-level idea of SDE Matching is to define the posterior process in a different order: we first parameterize a sampler of the marginal distribution and, based on it, derive the posterior SDE. Therefore, we may say that sampling from $q_\\phi(z_t|X)$ implicitly solves the posterior SDE.\n    \n    It is also important to emphasize that the correspondence between the posterior SDE and its marginals in SDE Matching is exact, not approximate.\n    \n\n**Questions:**\n\n1. There is a connection with PINNs in the tools used to define the model. Like PINNs, we use automatic differentiation to compute time derivatives (Eq. 18) and score functions (Eq. 26). We also employ a partial differential equation (the Fokker–Planck equation) to establish connections between elements of the dynamical system.\n    \n    However, unlike typical PINNs that solve PDEs, we use similar mechanics to compute coefficients of an SDE based on its solutions. So, while we believe there are conceptual connections, these are significantly different methods solving different problems.\n    \n2. The difficulty of solving the posterior SDE is reflected in the flexibility of the reparameterization function $F_\\phi(\\epsilon,t,X)$ (Eq. 16). As discussed in Section 7, the SDE Matching parameterization limits the flexibility of the posterior process.\n    \n    If the true posterior process is simple and has tractable marginals, solving it should not be computationally expensive. However, if the posterior is complex, an insufficiently flexible parameterization may not accurately capture the underlying dynamics.\n    \n    How complex the posterior latent process needs to be is an open question. Especially in the presence of a flexible observation model.\n    \n3. In this work, we chose a simple parameterization for the posterior process in order to stay close to the setup of Li et al. However, the reviewer’s observations motivate us to explore more expressive parameterizations, and we plan to include those results in a future revision.\n    \n    In general, we believe it may be more challenging to marginalize posterior processes of more complex forms, and that the results could be more sensitive to architectural choices in the parameterization network.\n    \n    We believe that with denser observations, the posterior process may be better approximated by Gaussian marginals. In contrast, when observations are sparse, intermediate distributions tend to resemble the unconditional marginals, which may have arbitrary shapes.\n    \n4. Small values of  $g_\\theta$ may introduce numerical instability, however this issue is not specific to SDE Matching, as the term involving $g_\\theta^{-1}$ appears in the ELBO of Latent SDE models in general.\n5. Yes, Eq. 27 implies that the posterior marginals $q_\\phi(z_t|X)$ are conditionally Gaussian.\n6. Yes, providing time steps to the reparameterization function $F_\\phi$ makes perfect sense when working with observations at arbitrary time points.\n7. In practice, the terms in the objective (Eq. 21) may indeed have different scales (e.g., if the observation model has very low variance). However, the total variational bound remains valid, and all terms should still be minimized to match the true dynamics.\n\nWe trust that the clarifications and additional discussions will strengthen your support for acceptance! If accepted, we will update the camera-ready version to reflect this discussion.", "claims_and_evidence": "The claim that the methods can scalably and simulation freely solve latent SDE seems well supported theoretically but not very well tested in experiments.", "methods_and_evaluation_criteria": "Yes, the method is designed to learn time series with latent SDE, and the paper tested whether the learned latent SDE will predict the time series in both simulated and real data experiments.", "theoretical_claims": "There are not many theories. Majority of the results are known.", "experimental_designs_or_analyses": "The design is on par with the original latent SDE method and is well-designed. I would appreciate if there are a bit more low dimensional SDE being shown like Fig.1 but it is not necessary.", "relation_to_broader_scientific_literature": "- Clarifying what feature made the diffusion model, as a special case of latent SDE simulation free is very insightful. \n- Time series modeling with latent SDE is useful in broad scientific fields.", "essential_references_not_discussed": "Not off the top of my head.", "other_strengths_and_weaknesses": "- The paper has enough details to reproduce without a code-base. \n- Generally well written. \n- Clarified why diffusion model can be simulation free as a latent SDE.", "other_comments_or_suggestions": "- I think Table 1 is misleading. The proposed method does not provide a gradient over parameters w.r.t. SDE solutions while all other methods do. The proposed method is, in fact, *avoiding* such calculation, but Table 1 could be interpreted as the proposed method could provide a gradient in $O(1)$ time and space which is not true. \n\n- I encourage the author(s) to provide more intuition on why the method could avoid the burden of numerically solving SDE: e.g., is numerically solving SDE provide information that is not necessary, or we implicitly solved the SDE with some NN that are parallelizable so the burden is transformed into training which can be scaled, or any other intuition.", "questions_for_authors": "1) This may not be a real question. But when I tried to reproduce the paper with realNVP, it felt a lot like implementing a physics-informed neural net (PINN), where you have a network approximating solutions of PDE, and in this case, I believe the normalizing flow actually solves the Fokker-Plank equation so maybe it can be seen as a PINN. I wonder if there is a connection. \n\n2) I think in general a difficulty cannot magically go away. I wonder if the author(s) can comment if the difficulty of solving SDE was absorbed into training the normalizing flow. The difficulty here is we need to know the solution of posterior SDE and the author(s) experiment suggests that knowing the marginal seems to be enough, but we still need to approximate it.\n\n3) this is connected to my second question. The authors used only a linear model for posterior SDE. When I tried realNVP it was much harder to train (strongly depends on hyperparameter tuning and takes large epochs, maybe it is just my implementation). To what extent does the success depend on the sampling rate in the data being relatively high so that distributions in between can be approximated with some Gaussians, and if one wants to model low sampling rate data, will the difficulty of learning the (unseen) marginals become more problematic? If time permits, can the author try a different sampling rate in the Lorenz experiment?  \n\n4) The objective (23) has $g^{-1}$ in it while $\\bar{f}_{\\theta,\\varphi}$ is itself not scaled by $g$, will this cause numerical issue when $g$ is small?\n\n5) Will eq. 27 imply Gaussian marginals? \n\n6) Will it be more useful to let eq.16 also take the time $(t_1, t_2, ... t_N)$ so some smoothness can be used especially when there are multiple time series taken with different times in data?\n\n7) What if losses in eq. 21 have vastly different scales e.g., the data has very little noise?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IkFujfe3TF", "reviewer": "Reviewer_JkpK", "summary": "This paper introduces SDE Matching, a novel simulation-free method for training Latent Stochastic Differential Equations (SDEs).  Traditional training of Latent SDEs relies on adjoint sensitivity methods, which are computationally expensive due to numerical integration and backpropagation through SDE solutions. SDE Matching addresses this limitation by leveraging connections with Score and Flow Matching techniques used in generative modeling, by directly parameterizing the marginal posterior distributions of the latent SDE, thereby obviating the need for SDE simulation during training.\n\n## Update after rebuttal\nMy score remains the same.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted to see that the reviewer finds our approach Innovative, efficient and note the reduction in computational cost of training. Below, we address the questions raised in the review.\n\n**Questions:**\n\nIn all our experiments, including the 3D stochastic Lorenz attractor, the motion capture dataset, and additional experiments (see rebuttal to TN6r, Question 3), with the same parameterization and training hyperparameters (such as the number of training steps), SDE Matching requires approximately $5$ times less computation time than the adjoint sensitivity method per iteration. Asymptotically, a single training iteration of SDE Matching takes $\\mathcal{O}(1)$ time, while the adjoint sensitivity method takes $\\mathcal{O}(L)$, where $L$ is the number of steps in the simulation of the posterior SDE (Eq. 2). However, in practice, the exact training time depends not only on the length of simulation, but also on the evaluation cost of other components, such as the calculation of the prior and reconstruction losses (Eqs. 22 and 24), which affects the actual ratio.\n\nAdditionally, as demonstrated in Figure 2, SDE Matching may exhibits significantly faster convergence compared to the adjoint sensitivity method, further accelerating the overall training procedure. When combining the per-iteration improvement and the convergence speed improvement SDE Matching is ~500 times faster than the baseline in the experimental setup from Section 4.1.\n\nAt the same time, in SDE Matching, the generative model defined by the prior process (Section 4.1) is exactly the same as in training the Latent SDE using the adjoint sensitivity method. Therefore, the inference time for unconditional sampling is identical for both approaches. However, as discussed in Section 4.4, in the case of forecasting, if we have access to partial observations $x_{t_1}, \\dots, x_{t_N}$, SDE Matching enables sampling of the latent state $z_{t_N}$ at the time of the last observation $t_N$ in $\\mathcal{O}(1)$ time. In contrast, the conventional parameterization of the posterior process requires simulating the conditional SDE up to  $t_N$. This property of SDE Matching can significantly reduce inference time when observations span a long period. However, in our experiments, we only have access to $3$ samples, all close to $t = 0$. Consequently, most of the inference time is spent simulating the prior process from $t_N$, making the forecasting inference time nearly equal for SDE Matching and the adjoint sensitivity method.\n\nWe would like to note that, in principle, SDE Matching allows simulation-free sampling of latent states $z_t$ at arbitrary time steps, potentially enabling fully simulation-free interpolation and forecasting in $\\mathcal{O}(1)$ time. However, such inference of the posterior process would require a corresponding training procedure in which the model observes only a few early observations. Otherwise, the posterior process may produce biased samples.", "claims_and_evidence": "The central claims are that SDE Matching enables scalable and computationally efficient training of Latent SDEs, achieving performance on par with adjoint sensitivity methods.  The paper provides experimental evidence on both synthetic (3D stochastic Lorenz attractor) and real-world (motion capture) datasets.  Results demonstrate that SDE Matching achieves comparable or slightly better performance and faster convergence compared to adjoint sensitivity methods.", "methods_and_evaluation_criteria": "The paper introduces SDE Matching, which involves parameterizing the posterior marginal distribution and deriving a conditional ODE and SDE. The evaluation metrics used include the Negative Evidence Lower Bound (NELBO) for training convergence and Test Mean Squared Error (MSE) on a motion capture dataset for performance evaluation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "The proposed methodology builds upon and extends the literature on Latent SDEs, Neural ODEs, and simulation-free training methods such as Score and Flow Matching in generative models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Innovative and efficient simulation-free training framework for Latent SDEs.\n- Significant reduction in computational cost and faster convergence demonstrated empirically.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Can you provide a more detailed comparison of the training and inference wall-time for SDE Matching versus adjoint sensitivity methods, especially for the motion capture dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper builds on the observation that the reverse process in score-based diffusion models can be seen as a neural SDE that can be used for a simulation-free training scheme for latent SDEs. This submission was reviewed by 4 reviewers in the field, and the paper ended up with an average overall recommendation of 4.0 (min: 2, max: 5). The score did not change during the discussion.

The reviewers appreciated the theoretical aspects of the work, while several of the reviewers had concerns related to the experiments. Many of the more minor concerns were resolved during the discussion phase. After the internal discussion, the decision reads that even if the paper would benefit from improvement, it is still over the bar for acceptance. 

For the camera-ready, please go through the reviewer comments and try to improve the paper for clarity to ensure proper impact of your work.
Decision: Accept (poster)