Variational Neural Stochastic Differential Equations with Change Points
Authors: Yousef El-Laham
Zhongchang Sun
Haibei Zhu
Tucker Balch
Svitlana Vyetrenko
Venue: Accepted by TMLR ()

Abstract: In this work, we explore modeling change points in time-series data using neural stochastic differential equations (neural SDEs). We propose a novel model formulation and training procedure based on the variational autoencoder (VAE) framework for modeling time-series as a neural SDE. Unlike existing algorithms training neural SDEs as VAEs, our proposed algorithm only necessitates a Gaussian prior of the initial state of the latent stochastic process, rather than a Wiener process prior on the entire latent stochastic process. We develop two methodologies for modeling and estimating change points in time-series data with distribution shifts. Our iterative algorithm alternates between updating neural SDE parameters and updating the change points based on either a maximum likelihood-based approach or a change point detection algorithm using the sequential likelihood ratio test. We also discuss theoretical implications of the proposed change point detection scheme. Finally, we present an empirical evaluation that demonstrates the expressive power of our proposed model, showing that it can effectively model both classical parametric SDEs and some real datasets with distribution shifts.

Published in Transactions on Machine Learning Research (03/2025)
Variational Neural Stochastic Differential Equations with Change Points
Yousef El-Laham J.P. Morgan AT Research
jouseffellchomm@ppmhheee
Zhongchang Sun
zhongcha@buuffllo.cc edu
University at Buffalo
Haibei Zhu
haibei.zhu@fpmmchasee
J.P. Morgan Al Research
Tucker Balch
tucker. buukerrbblcchhjppmccaseeccmmm
J.P. Morgan AI Research
Svitlana Vyetrenko
svitlana. o
J.P. Morgan Al Research
Reviewed on OpenReview:
Abstract
In this work, we explore modeling change points in time-series data using neural stochastic differential equations (neural SDEs). We propose novel model formulation and training procedure based on the variational autoencoder (VAE) framework for modeling time-series as a neural SDE. Unlike existing algorithms training neural SDEs as VAEs, our proposed algorithm only necessitates a Gaussian prior of the initial state of the latent stochastic process, rather than a Wiener process prior on the entire latent stochastic process. We develop two methodologies for modcling and estimating change points time-scries data with distribution shifts. Our iterative algorithm alternates between updating neural SDE parameters and updating the change points based on either maximum likelihood-baase approach or change point detection algorithm using the sequential likelihood ratio test. We provide a theoretical analysis of this proposed change point detection scheme. Finally, we present an empirical evaluation that demonstrates the expressive power of our proposed model, showing that it can effectively model both classical parametric SDEs and some real datasets with distribution shifts.
1 Introduction
Stochastic differential equations (SDEs) are a class of probabilistic models frequently used model continuous- time stochastic processes (Lelièvre & Stoltz, 2016; Soboleva & Pleasants, 2003; Huillet, 2007). They have broad range of applications in fields such quantitative finance, physics, biology, and engineering (Sauer, 2011; Browning et al., 2020). SDEs comprise two main components: a drift function, which models the deterministic evolution of the stochastic process over time, and a diffusion function, which captures the stochastic component of the process. In traditional SDE modeling, domain experts design simple parametric models for the drift and diffusion functions to encapsulate the key propertics of the system of interest. Model parameters are then learned using statistical estimation approaches, such as the method of moments estimation maximum likelihood estimation (Casella & Berger, 2024; Kay, 1993). While this SDE learning

---

Published in Transactions on Machine Learning Research (03/2025)
process is feasible for variety of applications, such as population ecology or mathematical finance, it can be challenging to apply in more complex systems. Recently, the concept of neural SDEs was introduced by integrating neural networks with SDES (Li et al., 2020; Tzen & Raginsky, 2019; Hodgkinson et al., 2020). This offers a more adaptable approach to modcling real-world time-series, climinating the need to define the structure of the drift and diffusion functions a prior.
Following the introduction of neural ordinary differential equations (neural ODEs), a wealth of research has emerged neural SDEs to model the dynamics of stochastic process (X, In (Kidger et al., 2021), connection was established between neural SDEs and Wasserstein generative adversarial networks (W-GANs), demonstrating that certain types of neural SDEs can be interpreted and trained within an infinite-dimensionaal GAN framework. An alternative approach to training neural SDEs involves the use of the variational autoencoder (VAE) framework, which has been adopted in various studies (Hasan et al., 2021; Li et al., 2020). The VAE framework was introduced (Hasan et al., 2021) to learn latent SDEs from noisy observations, assuming a prior distribution for the latent variable at each time step. In (Li et al., 2020), the training of SDEs as VAEs was also explored, assuming a prior over a latent stochastic process characterized by an SDE with a diffusson term for tractability of the evidence lower bound (ELBO). However, both approaches assume a prior over the entire latent stochastic process {Zt (E which may be too strong an assumption, as the training data may not always conform to this prior. Therefore, in this paper, we propose a new framework for training SDEs as VAEs that does not require such a strong prior in the latent. space.
While much of the existing research on neural SDEs has primarily focused on time-series modeled by a single SDE, the underlying dynamics of real-world time-series data often surpass the complexity that single model can capture. Scenarios where the dynamics of time-series abruptly change over time, such as the distributional shifts in stock prices during the COVID period, present significant challenges for existing approaches. In training neural SDEs, it's often assumed that the drift and diffusion terms exhibit Lipschitz continuity, a requirement nccessary to ensure the convergence of SDE solvers (Kidger et al., 2021). However, this assumption can be restrictive, as a single SDE with Lipschitz continuous drift and diffusiin terms may struggle to accurately model time-series with sharp distributional shifts. This limitation motivates our invostigation into the problem of change point detection for neural SDEs. With the detected change point, the time-series can be further modeled using multiple SDEs conditioned on the occurrence of change point. Similar work in this line of research includes the previously proposed neural jump SDE (Jia & Benson, 2019), which augments the neural ODE model with a temporal point process to model sharp changes in the ODE dynamics, without considering the stochastic nature (i.c., diffusion) of the timo-serics. In (Sun et al., 2024), a neural SDE model with change points is proposed based on the W-GAN framework; however, since the W-GAN framework based on an implicit generative model, it is difficult to derive thcoretical results regarding the convergence of the training algorithm.
In this paper, we introduce framework for training SDEs VAEs and develop algorithm for change point detection in neural SDEs based on this VAE framework. Specifically, we propose an iterative algorithm for change point detection under unknown SDE dynamics, which alternately updates the change point estimate and the neural SDE model parameters. The algorithm is summarized in two steps: (1) Update model parameters: Given the current change point estimate, we train different SDE models based on our proposed VAE framework; and (2) Update the change points: Given the current model parameters, we run a likelihood ratio test sequentially to refine the change point estimates. Our specific contributions are follows:
We propose a novel framework to train SDEs as VAEs. Unlike existing approaches, which require a prior over the latent stochastic process [Z. our formulation only necessitates specifying a prior over the initial state zg:
2. Leveraging our proposed VAE framework, we develop two approaches for learning change points in time-series model as latent neural SDEs: a method based on the idea of maximum likelihood estimation and a change detection algorithm based on the sequential likelihood ratio test. We utilize the Euler-Maruyama approximation to SDE solutions and apply suitable stochastic filtering methodologies to obtain an unbiased estimator of both the marginal likelihood of the change point and the test statistic in the sequential likelihood ratio test;
2

---

Published in Transactions on Machine Learning Research (03/2025)
3. We develop iterative algorithm to jointly learn the SDE model parameters and the unknown change points. Under certain conditions, we demonstrate that our iterative algorithm achieves performance guarantees regarding the estimation accuracy;
Lastly, we demonstrate the generative power of the neural SDE model on our proposed distributional shift generation benchmark datasets, showing that our model outperforms state-of-tho-art deep generative models across a variety of metrics.
2 Problem Formulation
Let = h denote a du--dimensional Brownian motion with admissíble filtration IF (Fe) on the interval [0, 27. This work is concerned with modeling the distribution of an & -aluue contimuous- time stochastic process X = {Xx) defined on the filtered probability space (2, F.,,F.P),, P), which assumed to the solution of an SDE of the following form:
dX. = [(,, X.)dl - glt, X.),W,, LE (0,
(1)
where Xo ~ μr is the initial state following the initial distribution po, f: [0, x gd, + Rd is called the drift function, and g: [0, 7) Rd, ddx is called the diffusion function. The drift and diffusion functions are typically assumed to satisfy some regularity conditions. Mainly, the drift and diffusion functions are bounded above by linear function:
(t, =) (lo x) 2n [=])), EERL, te [0,T)
(2)
and also satisfy Lipschitz smoothness condition:
(tf - f(t, x2)1 + |lg(t, x1)) - gt,, æ2)) & 72 - x211. Φ1 D2 Rd,, t [0,
(3)
for some constants 71, 72 > O. Under these assumptions, the stochastic process X is said to be a strong solution of the SDE in (1) if it satisfies (1) for each sample path of the Wiener process (W.)eetorr) and for all in the defined time interval almost surely (Oksendal, 2013). These conditions are also sufficient for proving strong order convergence of the Euler-Maruyama method (Kloeden et al., 1992).
Our goal in this work is to learn the drift and diffusion of the SDE defined in (1) given irregularly sampled time-series Bobs = (Zt, .t....., Zxx), where te (0,7) for all k. An ideal methodology would be robust to potential distribution shifts and could potentially model change points in the timo-scries (see Figure 1).
3 Related Work
Due to the large capacity of neural networks for function approximation, neural SDEs have been proposed to allow for data-driven learning of SDEs. In neural SDEs, the drift and diffusion modeled via neural networks, rather than "simple" pre-defined parametric functions. Neural SDEs can be trained using the VAE framework (Li et al., 2020; Hasan et al., 2021) where it is assumed that there is an underlying latent stochastic process (Ze with some prior distribution. In the following, we review an existing approach for training neural SDEs using the VAE framework.
3.1 Neural SDEs under the Variational Autoencoder Framework
Training SDEs as VAEs has been studied in (Li et al., 2020), where the prior is defined over the dy-dimensional latent stochastic process Z = (Zi hejo, which is characterized by an SDE:
dZ. (a t) + 9a Z,, t), t (0,
(4)
where 2u ~ Ho denotes the initial state of with corresponding initial distribution μa and a denotes a of hyperparameters. The posterior of ( (Zi)eejo,r) is approximated the solution of another SDE, which is of
3

---

Published in Transactions on Machine Learning Research (03/2025)
Xo Ho
Model 0
X tsv
Value
0
Model 1
X V1 <ISV2
-2-
Model 2
-4
V
V1 t S
V/2
> V2
X t>v2
-6
20
40
60
80
100
Time
Figure 1: An example of a non-stationary lime-series of length T = 100. Within the highlighted segments, the time-series is stationary and can be easily modeled with generative models, such as neural SDEs. The sharp distributional shifts occurring at the change points complicate the modeling of such a time-series with single att-of-the-box generative model.
the form:
~ 900 (obb)
(5)
dZ. = l (z, t)dt + g6 (2,, t),,,
1 E (0,
(6)
where & denotes the parameters of the variational approximation. Given the latent variable Ze, we assume that the observation X, has a distribution characterized by:
X, = ho ( Z.) E
where 9 denotes set of parameters, E, are i.id. noise terms usually assumed to be Gaussian distributed and independent of Z, Here, the function he can be thought of as decoder that decodes each sampled Z, to the mcan of the original stochastic process X sampled at the same time point X,
In (Li et. al., 2020), is assumed that the diffusion lerms for the prior SDE and posterior SDE are the same, i.., (zz t) = 90 (2,, t). Let ( (obe)) denote the marginal posterior of z; for all t E [o, 2J. Then, the pover-bound to the marginal likelihood called the ELBO, denoted by E(p, 0: xobw), can be established follows:
log (Tobs) 2 E(Φ,, 0; Xobe)
(8)
K
log po pa (2 - -f"
Eggs
2 Il Ze, 1)
(9)
where uz( be optimized by maximizing the ELBO. Exact evaluation E(Φ, 0; zobs)) intractable, but Monte Carlo approximation be obtained by sampling from the variational approximation:
(z,,t)). Therefore, the parameters of the neural SDE model can
J
1 E( 0: Robs) ~ J
ΣΣ (x ( 1 Σ
1 2
[ lddt
where 206) = (2(,) approximation and J is the total number of sampled trajectories. In (Li al., 2020), stochastic gradients
(1) denotes sampled trajectory the stochastic process Z from the variational
4

---

Published in Transactions on Machine Learning Research (03/2025)
obtained by using the adjoint sensitivity method; although other solvers, such as the Euler-Maruyama solver, can trivially be applied. At inference time, samples from the generative model are obtained by sampling from the latent trajectory from the Wiener process prior, which are then processed by the decoder. Hereafter, we refer to the approach proposed by (Li et al., 2020) as the LatentSDE model.
3.2 Identifying Change Points with Latent SDEs
A change point detection scheme based on the aforementioned variational framework was proposed (Ryzhikov et al., 2022). The authors propose to utilize a sequential likelihood ratio test (SLRT) to detect changes in given time-scries using trained SDE model based on the VAE framework. To elaborate, in their work, a single LatentSDE model is trained to model given time-series dataset. Once the model is trained, it is used as a means for online change detection using the following derived test statistic approximated with a Monte Carlo average based on samples drawn from the variational posterior:
p(zel() plarett 11)
CCPD(#:)
Σ log
(10)
| (Z::) Idzu:
Σ log
(11)
p 1) (20:1 Cobe ddzon-1/
1ml
1 EM N (2elf(z( (mm)) C.
M
log
(12)
M Em=IN N (=ra)(((=²))))) ((((
1=1
M
(m) where p(ær(t) denotes the likelihood of observing at time t, *0 ~ 90 (o0zzzl/zza Robe) denotes a sampled trajectory from the variational posterior of the trained LatentSDE model, f Rd: + Rdx denotes the decoder function, C' denotes a prior covariance matrix, and L denotes a lag. An important distinction of this work from ours is that their proposed method locused on the online detection task and does not explicitly include change points in the modeling of the latent SDE. This implies that their model cannot be used for generation of timo-scries with distributional shifts, but only a mcans to detecting shifts in the data. (or future data). Moreover, theoretical insights shown in their work focused only on the analytical form of the test statistic, rather than the theoretical properties of their algorithm (e.g., convergence guarantees and rates). We want to re-pasias that the goal of our work is to design & neural SDE model to accurately capture the dynamics of timo-serics data exhibiting distributional shifts, which requires capturing the change points in an offline manner. This is in contrast to the goal of the work in (Ryzhikov et al., 2022), which purely focuses on the detection task.
3.3 Neural SDEs Trained as GANs
As alternative to the VAE formulation modeling neural SDEs, an approach based on W-GAN formulation was also proposed in (Kidger et al., 2020), which we refer to SDECAN hereafter. An important distinction between this approach and its VAE counterpart is the the input noise distribution in SDEGAN only defines the initial state of the latent SDE. An approach for modeling change points in neural SDEs has already been proposed based on the W-GAN framework (Sun et al., 2024), which extends the original work on training neural SDEs as infnite-dimmeniinnal GANs by (Kidger et al., 2021). In this work, change points were directly modeled in the latent SDE dynamics (via the W-GAN generator network). The training of the model alternated between two phases: (1) updating the W-GAN parameters with fixed change points; and (2) updating the change points using CUSUM-type algorithm (Page, 1954) with test statistic based on the difference in discriminator scores between two consecutive windows of time-scrios datasct. The proposed lest statistic turns out to be connected to the Wasserstein-1 distance, and so the algorithm can be viewed as performing an approximate Wasserstein two-sample test (see (Ramdas et al., 2017)) for making change point updates. While the approach proposed in (Sun et al., 2024) demonstrated good empirical performance for generation of time-series with distributional shift, the theoretical validity of the method remains open question. Furthermore, recent works have shown that W-GANs provide inaccurate measures
5

---

Published in Transactions on Machine Learning Research (03/2025)
to the Wasserstein distance (Mallatto et al., 2019; Stanczuk et al., 2021) and therefore, the justification of the approach based on Wasserstein two-sample testing becomes questionable.
3.4 Limitations of Existing Approaches
Most works on VAE-based neural SDEs are structured in a manner similar to the aforementioned approach, where the prior is assumed over the entire latent process (e-g,, one can assume that a priori (2 is Wiener process). This modeling assumption, however, may be too restrictive in practice since the training data might not always conform to this prior latent SDE, which may degrade the generative performance of the model. To elaborate, the ELBO objective function in (9) is comprised of two-terms: a reconstruction term (expected log-likelihood);; and penalty term (KI.D between prior and posterior stochastic processes). The penalty term will ensure that solution of the posterior SDE will not be too far away from the Wiener process prior. The adverse effect of this regularization is that the decoder function needs to be powerful enough to transform trajectories from a Wiener process to trajectories from the underlying data distribution. Another important point is that, in the training of neural SDEs, it's common to assume that the drift function f, and the diffusion function 9, have Lipschitz continuity which ensures the existence of a unique and strong solution the SDE (1). Assuming smooth drift and diffusion, however, may limit the model's capability accurately model time-series with sudden distributional shift (e-g., sharp changes in the mean or volatility). A simple way to account for distributional shift is to incorporate change points into the LatentSDE model by utilizing different decoders before and after the change point. Unfortunately, this type of modeling does not explicitly capture how the dynamics of the underlying SDE change before and after the change point a single Wiener process prior would be utilized at inference time to generate trajectories and the decoder function is responsible for capturing the changes.
4 Proposed Methodology
In this work, we design a novel algorithm for training neural SDEs that does not require a strong prior in the latent space to train the SDEs as VAFs. Furthermore, within the VAE framework, we propose an algorithm incorporate change points to identify distribution shifts in the times series. Given the change points, we model the time-series as multiple SDEs based on the change points. Specifically, procedure that alternately updates the change point estimate and the SDE model parameters. To simplify the presentation, in the following, we consider the case where there is one change point. Our algorithm can be generalized to the case with multiple change points. A high-level overview of our modeling approach is summarized in Figure 2. In Fig 2, a time-series sample is first passed into an encoder (e.g., LSTM or neural CDE), which outputs the variational posterior parameters of the initial state of the latent SDE (Z.) An SDE solver (with SDE dynamics based on 0o) employed to sample the stochastic process {Ze1220 at times t < v (before change point). The terminal latent state of this sample Z, is then passed to second SDE solver (with SDE dynamics based on 01), which samples the latent SDE until time T. To obtain the corresponding samples in the original time-scries space, a probabilistic decoder (c.g., fully connected network) is used to decode each sampled latent SDE code Z, into its corresponding value in the original data space X,-
propose an optimization
4.1 System Model
To incorporate change points in our model, we assume that a change occurs at an unknown time v E (0, T). That is, the latent process (Ziheto.r) in our model is characterized by two different SDEs before and after the change point:
2o p(zo),
(13)
dZ. fo, (Z,, t) + go, (Z,, t)dW.
t tE (0,u), t L (u,
(14)
dZ. = fo, (Z,, t) + go, (Z,, f)dW,
(15)
where for and go, are the latent drift and diffusion neural networks (parameterized by 0u) before the change point u, and So, and go, are the latent drift and diffusion neural networks (parameterized by 01) after the
6

---

Published in Transactions on Machine Learning Research (03/2025)
Time-Series Dataset
Generated Dataset
D = (x()) 1==
S
1> V
Decoder
Decoder
Encoder
SDE Solver with 0o
SDE Solver with 9
96 (zolx)
{Zetsv
(Zot>
Latent SDE Model with Change Point
Figure A simplified flow diagram of the latent SDE model considered this work that accounts for potential change points in the time-series dataset.
change point. The observation process is modeled as:
X; ho, (z)) + et:
(16)
where he, Rd + Rds is assumed to be a fully connected neural network with standard activations and E, ~ NO, of a. We highlight that the decoder homogencous across time and is thus not impacted by the change point.
4.2 Algorithm Summary
Let. 0 = (0o. 0,, On} denote the "decoder' parameters. Consider a variational approximation (0(=o (toh))))))))))))))))))))))))(( of the posterior the initial state, where & denote the parameters of the variational approximation. We train the neural SDE model with change points using an itcrative algorithm, where cach itcration of the algorithm has two steps. In the first step, given the current change point estimate u(f-1), we update the model parameters (0) ( and the variational parameters (f (b(1) by maximizing the ELBO. Tn the second step, given the current value of the model parameters ((+), we update the change point u(t) y(i-1)
by maximizing the marginal likelihood of the observed data. We present pseudocode for the training algorithm in Algorithm and discuss each of the two steps in more details in the following.
4.3 Model Parameter Updates
We update the model parameters 0 (given v) using variational inference, by introducing variational approximation over the posterior of the initial state of the latent stochastic process zo given the observed data Tobs- Let v denote our current guess of the change point at iteration our algorithm. In our work, the parameters are updated by maximizing the following lower bound on the log-evidence the case of fixed change point (();
log po (a (10) (0. (Xotss)
where (Z.bbe) ) is defined as
(Zoor) A -DKL (20/o/)) ([p(2o)) + Eng [log pe (Zobs/z)))
7

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm Variational Neural SDEs with Change Points (CP-SDEVAE)
Initialize model paramcters ((0) {0) 1 (00) ), variational paramcters Φ (0) and change point estimate v(0)
for i= 1 to E do
Number of training epochs
Update model parameters:
Fixing u(t-1), update 0 c ((-1) and Φ5(+) - (bli-1) by minimizing the loss function in (21).
Update change point:
Fixing 0 = 0(1, update the change point pl6) +- ((t1) by maximizing the marginal likelihood given
0. This can be done exactly using the greedy maximum likelihood- based update or approximately with the fast detoction-based update.
Return: ((E), ()) v(B)
A key distinction between this ELBO and the one utilized in (Li et al., 2020) is that the variational postcrior is defined only over the initial state. This pushes the influence of the latent SDE dynamics into the expected log-likelln likkiiiioood term, rather than the KLD penalty. This choice gives us mainly two advantages:
1. If p(z)) is Gaussian and the choice of the variational approximation (0(=ol (oba) is Gaussian, the KLD penalty can be analytically computed. In (Li et al., 2020), tractability of the KLD penalty is achieved by making the more restrictive choice that the prior and posterior diffusion are the same.
After training, the learned latent neural SDE dynamics are utilized to generate samples. Our sampling procodure direct analog the GAN-based approach presented (Kidger et al., 2020), which has been shown to work practically well on variety of datasets, where the initial state of the latent SDE is generated from random noise and then propagated through the GAN generator (VAF, decoder in our case).
The challenge of utilizing our variational formulation now the tractability the expected log-likelihode Ey (log pa (Dons| [2)]], which we discuss the following.
4.3.1 Expected Log- Lige-iikldhood
Let Zobs denote the latent stochastic process (Z, sampled at same time steps Tobs. By the law of total probability, can write:
(Fobsl 20) = f pe (oroll Rooa (p (Zobal 20) ddzobs
K
= / pa (Zobss
pe (Zie z(re) II
dzobs.
Thus, the expected log-likelihood term can be written as nested expectation:
(Zob())))) A Eng [log po (2)b
K
= Egg log
E
Po (Izy (212) 20
(17)
k=
where the inner expectation is taken with respect to Po( m=(zob</20) For almost all choices of latent drift and diffusion of the neural SDE, this expression is intractable, but can be approximated using a nested Monte Carlo estimator:
Coo((Fors) = = log
7
Σ
II
pe (rty 1212 (..m)))
M
8

---

Published in Transactions on Machine Learning Research (03/2025)
(1m) pz(zbba|=0 (m)) is sampled via an SDE solver and 2 g0l (Fohs) for j 1...., J and where Robs ,,...,,. M. The mean-squared error (MSE) of this estimator converges to 0 at a rate of O + # (Rainforth et al., 2018), implying that the estimator is consistent (i.c., converges in probability to the true expected log-likelihood). By standard results in stochastic oplimization, this should guarantee that the optimization of the ELBO will converge (in expectation) to a local optimum of the model parameters, since one component of the ELBO can be approximated via consistent estimator (expected log- oog-likelihood and the other component can be computed analytically (KLD). In practice, stochastic gradients of & can be obtained 'backpropagating through the SDE solver', for which there are a variety of different approaches including the Euler-Maruyama method, the pathwise method (Yang & Kushner, 1991), and the adjoint sensitivity method (Li et al., 2020). The Euler-Maruyana method solves the SDE using finite differences, generating a sample path using a sequence of Gaussian state transitions. The reparameterization trick typically employed in VAE training can casily be applicd since each transition is Gaussian, enabling low variance stochastic gradients. The pathwise method is considered to be the continuous-time analog to the reparameterization trick, while the adjoint sensitivty method works by solving a SDE whose solution is the desired gradient. In this work, we consider the Euler-Maruyama solver, but note that both the pathwise method and the adjoint sensitivity method can easily be utilized as well. For more information on the advantages and disadvantages of each solver, we refer (Ti et al., 2020), which provides a nice summary.
4.3.2 Expected Predictive Log-Likelihood
In recent works, the problem of noise underestimation in LatentSDE models has been investigated (Heck et al., 2024), a phenomenon that is analogous to a recurring observation made in underestination of posterior variance variational inference. Since our work also employs variational inference, a weakness of the training loss for the model paramcters of the SDEVAE model is it will also lead to an underestimation of noise variance, since it is also balancing an expected log-likelihood term (reconstruction error) and a KLD term.
To understand specifically why this is problematic for our model, let consider simple example. Consider that the decoder of our model is linear, that is let ha, (z) = α + B,, where ar E Rd,xd, and B, Rd When utilizing a single trajectory a single trajectory Zobs for the nested Monte Carlo estimator (J = 1, = 1), the expected log- likelihood when using a linear decoder is approximated as:
K
(Xob)))
=
log pe (aty Izon):
(18)
1=
Σ k=1
log No N(rre 2t
Br. oh Ia.);
(19)
K (1t2 (a),21 + +B,))2
Kdg log (2T
(20)
2
207
As can be seen from (20), the expected log-likelihood term focuses on optimizing the model parameters, such that the marginals distributions of each It, are well-calibrated (the underlying dynamics of the latent SDE are not influencing the expected log-likelihood). We note that hyperparameter tuning of does not solve the noise estimation issue, as changing of also only ellects the marginal distribution of æt, and thus places emphasis on calibrating the marginal distribution of æt, given zt, for all k.
To improve the generative quality of our model, we propose to improve noise estimation by regularizing the ELBO with the following additional term:
K Σ log
E pe (ar ( 20 /20 [=]]]
Ci (Foba) 2 Egg
where the inner expectation is taken with respect to p(Z (20). We refer to Co (Fobs) as the expected predictive bikeiioli Just like the standard expected log- licellol C (xobe) can be approximated with nested MC estimator. For our estimator, we use first-order Taylor approximation to obtain Gaussian
9

---

Published in Transactions on Machine Learning Research (03/2025)
approximation for the distribution po( [ approximation typically used in extended Kalman filtering, which is designed for non-linear state-space models with additive Gaussian (Kalman, 1960; Smith et al., 1962). A key difference between Le.s and L8.0 is that maximizing Le.0 encourages well-calibrated conditional distributions po(zx|z)) rather than well-calibrated marginal distributions po(z)). We have found that empirically, this improves the generative performance of our model in terms of capturing noise properties in the time-series. To summarize, when updating the model parameters, for a fixed change point y and observed time-series Tobs we minimize the following loss function:
E(0, Φ; Tobs: v) DLL
Cared MaD (zo (obs) llp(=))) Aa (Tobs) 00 (Eobs):
(21)
where Aka O, Ann > O, and Apred > are regularization constants. In practice, we often train generative models collection of timo-scrics fae /n=1> in which we minimize the average loss across all time-series (amortized inference) to make model parameter updates:
(
arg min
Σ 10, Φ; x (a) obs ((-1))
(22)
BER N
where θ and 4 denote the feasibile sets of the model parameters and the variational approximation parameters, respectively.
4.4 Change Point Updates
We present two approaches for updating the change points: a greedy approach based on exact maximum likelihood estimate; and an online approach based on the sequential likelihood ratio test. For simplicity, we assume that the change point belongs to the set of sampled time points T (f2....,fx). We refer the reader to the Appendix for exlension to the case where the change point can occur at any time index in (0, 7)- Before delving into each approach, we provide an overview of particle filtering methods and how they can be used for obtaining an estimator of the change point likelihood pe pa(Cobalv = f), which is critical quantity for the change point update.
4.4.1 Particle Filtering for Change Point Likelihood Estimation
Particle filtering stochastic filtering methodology for approximating the posterior distribution of a latent process given sampled observations from another stochastic process. Consider the system model in Section 4.1 under the assumption that the change point fixed to T. The system model can approximately be expressed in terms of system of probability distributions:
po (Zie (ite-);;
th (before change)
State Equation:
Zth po Zta L1 1-V=
pa, (2ca ((v-1); tp > 7 (after change)
Observation Equation:
Tr ~ pa, (1ta (Zte)
The goal of a particle filtering method is to obtain a sample-based (discrete random measure) approximation to the filtering distribution Pa(z,, Dty V = r) or the smoothing distribution po po(zto. | :== 7) by using importance sampling. For example, this system model, the smoothing distribution po (o (2tr.)) can be expressed in terms of the joint distribution pe (z Ftsa IV 7) and the normalizing constant pe ( /v T):
Po (z (zb12) po (Ztou Drick IM = T) po(211el /V T)
x p(zo) II pen (xi, (z(.)
II 5:t, Sv
Pa. (Z1, (z)-.))
II Bit.
pe, (Zi, (
Pe(2 (Ro,U=T)> I Po(zi, 111, womr)
10

---

Published in Transactions on Machine Learning Research (03/2025)
The fundamental idea behind the particle filtering approach is sequential importance sampling, which utilizes proposal distribution at time la that is factorized in manner similar to the Markov process defining the state equation:
g( (xnee-)) (z0) II (( 12t, 1, (t,),
time instant tx, the (unnormalized) importance weight of a trajectory sampled from =0) ~ g(zas, (o denoted by w!/h) is weighted according to the smoothing distribution po( F((x) can be recursively computed as follows:
(3) (i) ( x () pen (1tr 1 Doo (Z
(200=1212(4 -V= j=l....,d
g( 1 22 Dta))
(5) (1)
The pairs of sampled trajectories and their weights in particle filtering provides a means for obtaining estimators of quantities related to the smoothing distribution. An variation of particle filtering is bootstrap particle filtering (BPF), which samples trajectories according to the assumed state model, i.e,, (t,,,, po(z,. 2t, 1: = 7) and includes an additional resampling step avoid the path degeneracy problem. In this case, the importance weights are proportional to the likelihood function:
( oie x po, (ate =())
due to the fact that if the particle streams are resampled at each time instant, then w.)) tt-1 x J for all j. Finally, we discuss the utility of particle filtering in the context of this work, which that it can be used to evaluate the marginal likelihood of a particular change point (which is used in our maximum likelihood update of the change point) and it can be used to compute likelihood ratios (which is used to in detector based update of the change point).
Marginal likelihood of change point: An important quantity in this work the marginal likelihood of the change point being equal to a particular value (over time horizon T tx), which can be approximated a product of the average importance weight:
Po( r) 2,
(23)
Under weak assumptions, this estimator unbiased and converges almost surely to the true marginal likelihood (Crisan & Doucet, 2002).
Approximation of the likelihood ratio for change point detection: The likelihood ratio is a fundamental quantity statistics, lypically used to construct a lest statistic for hypothesis test. For instance, for change point detection, being able to compute the log- likelihood ratio A(xaa), which we define
Poo Iv p 1 T)
A( A log
(24)
( 7)
where the numerator in (24) corresponds to the likelihood the change point occurs at time and the denominator corresponds to the likelihood the change point does not occur at time т, but at later time. Under both models, the latent trajectories generated up until time t4 are the same they are both generated by latent SDE with parameter 0g. The difference in these likeliloods comes from the fact that in the case of the mumerator, Zte.s sampled by propagating the previous latent state 2t, with post-change SDE (with parameters 0 rather than the pre-change SDE (with paramcters 0p). It turns out this quantity can be approximated with BPF by taking the ratio of their average importance weights. To be precise, we can form the following approximation of the marginal likelihoods for each hypothesis at time step T l% for each
11

---

Published in Transactions on Machine Learning Research (03/2025)
model:
11
D (2tia V== tx)
J
th
(1.1)
II
J
1/
(.0)
(25)
(' Mi tx)
W
(5,0)
(26)
j=1
where (2.00) and ((d1) denote the importance weights of the jth particle stream when propagated by the pre-change SDE and post-change SDE at the instant lez respectively. Given these two approximations, we can estimate the log-likelihood ratio A ( using an estimator (xxea) at time instant 4 as
F ( V = 1x)
A(D log
(27)
p (xtea) 1/ > tx)
((:1))
k-
w(1.0)
log
j=1
to
=1
(28)
(1:0)
Σ ン
v5(,00
J
log
j
tt
(1.1)
log
1 ト
(1:0) to
(29)
j=1
j=1
4.4.2 Greedy Update: Maximum Likelihood
Now that we have discussed particle filtering methods, we can now elaborate how change points can be updated in our algorithm. Change point updates are made by finding the optimal value of the change points given the most recently updated model parameter. We define the optimal change point update v() as the one that maximizes the marginal likelihood of the data:
( arg max p( = 7).
(30)
EET
By the chain rule of probability, we can write:
(uobbll=tt = II p(au V t)
(31)
where Dictz denotes the observed data such before time tg. While for general models p(Zobs/V t) is an intractable integral, can be recursively estimated using Bayesian filtering techniques. In this work, we use particle filtering (Djuric et al., 2003), which provides straightforward way to obtain consistent estimator D(aotaly f) for pl t) (please see (23)). We call the maximun likelihood update for the greedy update because it requires O(T runs of the BPF to estimate the marginal likelihood for all candidate values v E T (see Algorithm 2). This may not be practical for long sequences and so we propose an alternative approach for a faster update based on the sequential likelihood ratio test.
Note that the approximation of log-tikelihood ratio (27), the number of particles generated for both pro-/post- SDE are assumed to the same (l.e., J trajectories); however, one can generalize the estimator consider different. numbers of generated trajectories for the pre//post- change (i.e., Jo for the pre-change SDE and J₁ for the post change SDF).
12

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm 3 Detection- based CP Update
Initialize particle filtering particles. Initialize log Zo = 0.
Algorithm 2 Maximum Likelihood CP Update
for k = to K do P Number of sampled times Propagate particle using assuming no change point and assuming change point:
Initialize particle filtering particles. Initialize log 20.50 = 0. for == 1 to K do > Number of sampled times
Run particle filter with model parame- ters fixed to 0 and obtain marginal like- lihood estimator:
Run PF to approximate log marginal likeli- hood under Ho, luz
Run PF from time 4 to time tx and ap- proximate of the logarithm of the marginal likelihood log p(zorell = tx):
log po > (x) log Zt.
log Z + log 212l
log P( = tx) log Zumts
Run PF to approximate log marginal likeli- hood under H1,x ta::
= log 20m0 + log 2
log log 1x) log Zipp -log Zx-1 + log Zi:
Note: Our estimator is composed of two compo- nents: log Zort, and log Z, The compo- nent log 0:tz can be obtained from particles recycled from the previous PF run.
Approximate log-likelihood ratio:
Change point greedy approximation:
log (x log Z: log 1/2
i- = arg max log Z EET
If log A(x);;))) > 7:
Return: 1 = tk
Return: D.
Else:
Set: log 2x = log Z1zte
Return: D = arg max log A A(x))a))
4.4.3
Fast Update: Sequential Likelihood Ratio Detector
A fast and online method for updating the change points at each training iteration is to use a sequential change point detection scheme (Polunchenko & Tartakovsky, 2012). Notably, the CUSUM algorithm has been applied for detecting change points in neural SDEs trained as W-GANs, where an approximated Wasserstein distance based on the learned W-GAN critic is used to detect the change point in single forward pass of O(ITI) segments of the time-series (obtained via a sliding window). Practically speaking, it is only useful for neural SDEs trained under the W-GAN framework, since a proxy for computing the Wasscrstein distance is required. Furthermore, the learned change point does not have any theoretical guarantees. Unlike W-GANs, which are implicit generative models, VAËs are explicit generative models and provide easy access to the probability measures of the latent and observed processes. This allows us to utilize the sequential likelihood ratio test for detecting the change point, test for which theorctical implications have been well-studied.
Specifically, our change point updates are inspired by the classical sequential testing framework, where we consider the candidate change points r to belong to the set of sampled time points T. At each time is E T we decide between two hypotheses:
Ho Fta. p( tx),
Hi : Pey. p( 1V tx).
where Tty. = (a.,...., ).,)) The null hypothesis Ho that the change occurs after time tg (and thus, the detection algorithm continues to run) and the alternative hypothesis H, is that the change occurs precisely at ls (and thus, we stop the detection algorithm and adopt lg as the change point). We adopt the
13

---

Published in Transactions on Machine Learning Research (03/2025)
change point update as the value of & that rejects the mull hypothesis, i.c. when
log A (.. & log p( /V = t&) log p 2 /v > tx) 2%
(32)
where ) denotes the likelihood ratio of the test at time and is threshold determined by the pre-specified false alarm probability of the test In practice, the log-likelihood ratio is typically monitored as the test statistic. Importantly, evaluation of the likelihood ratio involves the integration over Ziae (in both the numerator and denominator) and thus, is generally an intractable quantity. Similar to the greedy approach for updating the change points, we use a BPF to sequentially obtain an estimator of A(x based on (27)-(29):
1 (Z |V (x) ( = 1' (a (x)
(33)
where J denotes the number trajectories sampled in the BPF. The advantage of the sequential testing approach that maximum of [T] BPF steps are needed detect the change, which can all be done using a single run of the BPF, reducing the change point update complexity O((T)) BPF steps.
4.5 Theoretical Insights
In this section, we provide some thcoretical insights of our proposed work. Mainly, we show that under certain assumptions, the training algorithm converges to stationary point w.r.t. the ELBO. We also show that our detection scheme, under certain assumptions, also achieves optimal error probability, further justifying it as method for estimating the change point in our algorithm. We note that all theoretical results presented in this work consider the ELBO defined in (17) without the predictive expected log- aigliielihhod regularizer term. A theoretical analysis of this regularizer is left for future work.
4.5.1 Convergence of Training Algorithm to Stationary Point
To prove that our algorithm converges to stationary point, we need to make few assumptions about the efficiency of the updates at each iteration of the algorithm. Mainly, we assume that both model updates and change point updates lead to an improvement based on their respective criterion. Mainly, model parameter updates improve the ELBO and change point updates improve the marginal likelihood. We also make the assumption that the inference gap as a result of the variational approximation does not widen after change points are updated. A detailed description of the assumptions can be found in Appendix A.1.1.
In the following theorem, show that our training algorithm converges to stationary point of the ELBO mainly that after each update in the algorithm the ELBO either stays the same or increases in value. We provide a visualization of the result in Fig 3.
Theorem 1. As E + oo, our algorithm (under marimam likelihood updates for the change points) reaches stationary point w.r.t. a lower bound on the marginal likelihood, i.e.,
E (609))6)00 pas (Fobs) 2 1) 11 (Zobs))
for all i E N, where denotes the natural numbers.
Proof Sketch. To prove this result, we needed to show that change point updates (which we assume yield an improvement in marginal likelihood) imply an improvement w.r.t. the ELBO as well. The difference between the logarithm of the marginal likelihood can be shown be sum of two components: the improvement in the ELBO and the change in accuracy in the variational approximation (based on the KLD between the variational approximation and the true posterior distribution) after change point updates made. Under the assumption that change point updates do not vastly impact the accuracy of the variational approximation, we directly arrive at the desired result. The detailed proof can be found in Appendix A.2.
4.5.2 Optimality of the Detector
In the following theorem, we provide a thcoretical insight into the performance of our online change point update. Specifically, demonstrate that at each time 4, our update asymptotically achieves the optimal
14

---

Published in Transactions on Machine Learning Research (03/2025)
log, (8..M,) BLBO
log (L.
9l...)
ELD.
log ML (0.,)((-1)
GAP.(u00)
log LogMl, (f.,vv)))
log logM).(9,,x0)
EL.D
EL..
Updatte
Change
Update
Point
log ML (fte))) (011
log (ou 12 v()
GAY (00, 4000 pi
lug ML 200
ELBO 12 اh
EL.50 (10 (0)
Figure 3: Stationary point convergence based on training algorithm. Under the assumption that change point updates do not widen the inference gap, the result is evident and demonstrated in this diagram.
error probability as the number of sampled trajectories tends to infinity. This result is siguificant as it provides a thaoretical guarantee for the performance of our proposed method.
Theorem 2. As J co, have that P(A(I))))) 2 H/H) - P(A(ZA..)) 2 >/H6) and P (A( < <1121) ((A( <+)H)).
Proof Sketch. We begin by showing that as J + oc, the likelihoods / (It = t) and p (a Iv > t) converge almost surely to p( | () and p( j > (), respectively. This achieved by applying standard convergence results of bootstrap particle filters (BPFs). The contimuous mapping theorem then implics that the likelihood ratio ((Fr,.)) converges to A(Eraa) almost surely. We then demonstrate that our lest achieves the optimal error probability. The detailed proof can be found in the Appendix A.3.
Remark on rate of convergence: With a little abuse of notation, let p((t)a))) and E( the mean and covariance matrix of a respectively. By standard results in Monte Carlo, we have that v J (u(x,,)) converges distribution to (0, E(ar,a)). Since we sample independent trajectories of Btya in the BPF, from the convergence of the delta method (Fisher, 1925), we have that the estimate A A (mt,.) converges to the ground truth A ( at the rate of VJ*
4.6 Practical Considerations
In this section, we highlight several important aspects consider in order to ensure success training of the CP-SDEVAE algorithm.
Model architecture: Our model architecture comprises several key components designed to effectively capture and process time-series data. The encoder utilizes an LSTM network, which is well-suited for sequential data processing. For scenarios involving irregularly sampled time-series, an alternative approach such as neural CDE could be considered. The decoder is implemented as fully connected network, providing flexibility in output generation. The core of the model lies in the latent SDE components. Both the drift and diffusion networks of the latent SDE are implemented as fully connected networks with LipSwish aclivation functions. This design choice introduces an important tradeoff: while more complex drift and diffusion networks can potentially capture more intricate dynamics, they tend to reduce the meaningfulness of detocted change points. This phenomenon was observed our ablation study conducted on both real and synthetic data, detailed in Section of the Appendix. For the SDE solver, we employ the Euler- Marey-aa Although experimented with alternative approaches based on the adjoint sensitivity method, we found
15

---

Published in Transactions on Machine Learning Research (03/2025)
significant performance differences, leading to favor the simpler Euler method for its efficiency and case of implementation.
Optimizer and stochastic weight averaging: Our optimization strategy is carefully crafted to ensure robust model training. We utilize the Adam optimizer with a learning rate of 1 x 10 and a weight decay of x 10- The training process continues for a maximum of E = 10000 epochs or until convergence is reached, as determined by the ELBO loss. To enhance training stability, we incorporate stochastic weight averaging, a technique that has shown promise in previous work on training neural SDEs, such as the SDEGAN approach.
Initialization of change points: The initialization of change points plays a crucial role in model performance. We explored two methods: random initialization and initialization based on mean shift using the ruptures library in Python (Truong et al., 2020). Our findings strongly favor the latter approach, as the model exhibits sensitivity to poorly initialized change points. The ruptures library provides a more informed starting point, leading to improved overall performance.
To further enhance the robustness of our change point detection, we implement a warm-start period of = 50 epochs before making any change point updates in the training process. This warm-start period is essential because the accuracy of change point detection is intrinsically linked the overall model performance. Mismatches in model paramcters can lead to degradation in both the maximum likelihood estimation and detection-based approaches for estimating change points. By allowing the model to stabilize initially, we mitigate these potential issues and improve the reliability of our change point estimates.
Detection threshold: For a given threshold 7, the fast detection-based update corresponds to certain level of tolerance for false alarms. In online settings, it's crucial to set this threshold before deploying the detection algorithm. Much of the literature on sequential testing frameworks focuses on calibrating this threshold for various statistical models to meet specific tolerances for false alarm probabilities. However, the focus of this work on using the detector to estimate the change point in an offline manner. In this context, the threshold can be seen as hyperparameter of the CP-SDEVAE model, which be tuned to enhance the quality of generative performance. It's important to note that the detection threshold introduces a trade-off. A larger value of y means that a change point will only be detected in the event of a more extreme distributional shift. Conversely, a smaller value of y increases the likelihood of detecting change point in response to minor and possibly insignificant changes.
Extension to multiple change points: Our proposed mathematical formulation provides a method to incorporate a single change point in modeling neural SDEs. To extend to change points, : MP, a variety of approaches can be used, For the greedy approach based on maximum likelihood, if there are multiple change points, one can update each change point va by maximizing the marginal likelihood, holding all other change points and the model parameters fixed to their most recently updated values:
arg max S1522111
P(Xabalua 1, V-d):
where Đーa denotes the current estimate of all other change points and we define in = 0 and io44 = T. For the detection-based update, one continues rumning the detector until the desired number of change points are detected. If the number of detected change points is less than the number of change points assumed in the model, an adaptive threshold can be utilized. With regards to choosing the number of change points, standard model selection approaches, such cross-validation marginal likelihood maximization, can be considered. With regards to marginal likelihood maximization, we recall that particle filtering methods can be utilized to obtain an unbiased estimator of the marginal likelihood (i.e., using the estimator in (23)). This particular estimator can be used to compare generalization performance between difference classes of models (i.e., different number of change points). This strategy has been repeatedly employed Bayesian inference settings (see (Llorente et al., 2023) for a review on the use of the marginal likelihood for model selection).
Greedy vs. fast update: In our work, we compare two primary approaches for estimating change points: the greedy approach and the fast (detection-basse) approach. The greedy approach proves superior in terms
16

---

Published in Transactions on Machine Learning Research (03/2025)
of accuracy, it preciscly determines the change points that maximize the marginal likelihood. For single change point, this method tests K hypotheses, each requiring O(T) propagation steps in the particle filter. When dealing with multiple change points (1 (1< D < K), the computational complexity increases significantly, with the number of hypotheses to be tested growing to
(5) As previously discussed, the number of hypotheses needed to be tested at each epoch can be reduced using coordinate-ascent style updates; however, this does not solve the long time horizon issuc.
In contrast, the fast approach, while potentially less accurate, offers significant computational advantages. It requires only a single ( propagation step through the particle filter, regardless of the mumber of change points. This makes it particularly suitable for scenarios involving long time-scrics with multiple change points, where the greedy approach may become computationally infeasible.
The choice between these approaches ultimately depends on the specific characteristics of the data being analyzed, including the length of the time-scries and the number of time-scries samples. For shorter timo-serics when computational resources allow, the greedy approach provides the most accurate results. However, for longer time-series or when dealing with large datasets, the fast approach offers a practical alternative that balances accuracy with computational efficiency.
5 Experiments
Here, we present numerical experiments to verify the validity of the proposed CP-SDEVAE model. To that end, we conduct two different types of experiments. First, we conduct experiments on syuthetic data generated from an Ohhsteen-UUlennbeck (OU) process. We use the OU process experiment to compare different variants of the CP-SDEVAE method (e.g., with/withoud /withoutttuuttt change points, MLE-based change point updates vs. detoction-basor change point updates). We also use this dataset of different hyperparameters and the impact of the proposed predictive negative log- log-likelihood regularizer. The description and results of the ablation studics can be found in the Appendix C. For all methods, we use a detection threshold of y = O for the log-likeli log-likelihood ratio as a means to detect the change point.
means to conduct basic ablations to understand the effect
5.1 Toy Data
We consider synthetic univariate time-series dalaset generated from an OU process. In the first example, we compare the generative performance of our proposed SDEVAE model with the LatentSDE model described in Scction 3.1. In this example, we assume the same switching OU process that is utilized for our ablation studies in Appendix C. second example, we compare different variants of our proposed approach for an OU process with a single change point. In the third example, we test the robustness of the proposed method by introducing multiple change points.
5.1.1 OU Process with Single Change Point (Slow Change)
We consider time-series generated from a switching OU process that is the solution to the following SDE
dxx, = Po(Ho - Xt) + ondW, t (0,z)
dX,, 01(m1 - -x)dtt + oodW,
t T]
where we consider the parameter settings 0, 0.05, pn = 4, do = 0.15, 01 0.03, f1 = 2,, 01 =1 15, a change point of 25, and a time-horizon of T = 50. We assume that for initial state Xo is Gaussian distributed with mean and variance 0.01. Using Euler solver with step-size A, = for all 4, we simulate N = 500 trajectories to construct a time-series dataset. Our goal is to compare the performance of the LatentSDE model and SDEVAE under different settings of the observation noise." 2 Similar to the LatentSDE implementation, we utilize a GRU-based encoder and 2 layer MLP with 100 hidden units for the latent drift, latent diffusion, and decoder networks.
TThe implementation of the LatentSDE model utilized is based on an implementation found in the torchade library https://github. con/ goggggo-oeogerrceee torehhee for modeling Lorenz attractor. We adopted the implementation into our rodebase and utilized an analogous architecture for fair comparison.
17

---

Published in Transactions on Machine Learning Research (03/2025)
LatentSiDE with Obs. Noise 0.01
LatentSDE with Obs Noise 0.10
SOEVAE with Obs Nake 0.01
SDEVAE with Obs Noise 0:10
real synthabi
[eal kyrithani
rCa synchebi
synthat
Figure 4: Comparison of generative performance of LatentSDE and SDEVAE (ours) models for different observation noise settings.
We summarize the results in Figure 4. As evidenced by the figure, in the case of low observation noise of 0.01, the LatentSDE model underestimates both the noise of the time-series, with smooth trajectories that approximate that capture the mean of the time-series dataset well. When increasing the observation noise to 0.10, although the marginal distribution better captured, the trajectories are still too smooth accurately capture the distribution of the observed datasct. In contrast, SDEVAE is able to capture the trajectories well under both noise assumptions, with the observation noise of 0.10 providing more realistic samples.
5.1.2 OU Process with Single Change Point (Sharp Change)
We consider the same SDE as the previous experiments, except with parameter settings 0o = 0.2, no = 4, Oo = 1, 0 = 0.5, μI = -4, o₁ = 1, a change point v = 25, and a horizon of T = 50. Under this parameterization of the switching OU process, the distribution shift more sharp (since the reversion parameters fg and 81 are larger). We assume that for initial state X, is Gaussian distributed with mean 3 and variance 1. Using an Fuler solver with step-size A, = 1 for all 1, we simulate N 100 trajectories to construct time-series dataset. For each baseline model, we standardize the dataset using the global mean and variance taken across all time-scries. For the basclines in this experiment, we consider four different variants of our method: (1) CP-SDEVAE assuming no change points (which we also refer to as SDEVAE); (2) CP-SDEVAE assuming a single change point with maximum likelihood-basee change point updates; (3) CP-SDEVAE assuming a single change point with detection-based change point updates; and (4) CP-SDEVAE assuming two change points with detection-based ML updates. For each of the methods, we assume the following hyperparameter settings: for the encoder architecture with 2-layer fully-connected neural network with standard ReLU activation functions; the latent dimension of the SDE is assumed to be 32; for all latent drift/diffusion functions, we use 2-layer fully-connected neural network with LipSwish activations; for the decoder network, we use a 1-layer fully-connected network with ReLU activations; we use the Adam optimizer with a woight decay of x 10-4 and = 5 trajectorics for cach MC estimator of the ELBO. As previously mentioned, we utilize this example as a means to conduct an ablation study to test the effectiveness of different components of our model. For more information about the parameter settings of the ablation study and the key finds, please see Appendix C.
Results: A summary figure showing the results of the generated time-serics from each model (along with the detected change point) is shown in Figure As can be demonstrated from Figure 5, the change point variants of the proposed approach outperform the variant without any change points assumed. This is evident by looking the ELBO metric shown in the title of each subfigure, where the change point based approaches achieve superior value, which indicates that the generated datasct with the change point variants demonstrate a higher degree of realism. Moreover, an interesting point to add is that overparamelerization in terms of the number of change points does not impact the model's ability to capture the dataset; however, leads to over representation terms of the number of change points. We can see that when a second change point is assumed for the CP-SDEVAE, the second detected change point does not actually reflect a shift in distribution. Lastly, we point out that both the ML.F-based change point update and the dedeetion-based change point.
18

---

Published in Transactions on Machine Learning Research (03/2025)
CP-SDEVAE No Change Points ELBO -7.53
CR-SDEVAE MLEE 1 Change Point ELBO = -4,62
Value
Value
Time Index
Time Index
CP-SDEVAE SLRT 1 Change Point
CP-SDEVAE SURT 2 Change Points
ELBO -4.62
ELBO -4.65
Value
Value
Time Index
Time Index
Figure 5: Time-series plots for each of the baseline methodologies.
Log Evidence
Log Evidence Ratio (SLRT)
Mean
20
95%
-100
10-
200
N -300
400
-10
-500
Mean Lag Evidence
-20
15
20
25
30
35
5
10
Figure 6: Comparison of log- likelioel ratio for detection-based change point updates and log evidence for ML.E-based change point updates.
update lead the same overall detected change point in the algorithm. As can be seen Figure 6, the log marginal likelihood achieves its maximum value the true change point value. For the detection-based update, we track the log-evidence ratio remains relatively stable until approach the change point value. It is clear that for this example of distribution shift, both approaches are easily able to identify the change point.
5.1.3 OU Process with Multiple Change Points
Consider a Lime-series generated from a switching OU process that the solution to the following SDE:
dX, Ao(μo X,)dt + oodW,,
E (0, v1]
dX,, (M1 X,)dt dW,,,
t (v, U2]; t E (12, T].
dX, Oz(22 Xe)dt ondW,
19

---

Published in Transactions on Machine Learning Research (03/2025)
CP-SDEVAE No Change Points ELBO -1833
CP-SDEVAE SLRT Change Point
ELBO = -13.71
Value
Time Index
Time Index
Log Evidence Ratio (SLRT)
CP-SDEVAE SLRT Change Points
ELBO -13.51
85% CI
Value
-20
0
10
20
30
40
50
60
70
Time Index
Figure 7: Time-series plots for each of the baseline methodologies for the multi-change example. The log- lieeliil ratio statistic also shown as a function of time.
where we consider the parameter settings On 0.2, 1o 4, oo = 1, 0, 0.5, 1 -4, 01 1, 02 = 0.5, #2 2, 02 0.5, change point values of 11 25 and 1/2 75, and time- Hime-horizon of T 100. We utilize the same hyperparameter settings as the previous example. Using an Euler solver with step-size A 1 for all t, we simulate N = 100 trajectories to construct a time-series dataset. Here, we also test three different. variants of our method: (1) CP-SDEVAE assuming no change points; (2) CP-SDEVAE assuming one change point with with iiocctio-bbased change point updates; (3) CP-SDEVAE assuming two change points with detection-basod change point updates.
Results: A summary figure showing the results of the generated time-series from each model (along with the detected change point) and the the log- likeliild ratio over time is shown in Figure 7. As we can see from Figure7, the CP-SDEVAE variants which assume a change point are able to better capture the shift the distribution, where assuming a single change point shows better performance than assuming no change points at all and assuming two change points shows the best performance overall. We can also see that the log-likelihood ratio flips signs exactly at the location of the change points, demonstrating the methodologies' ability to reffect both small and large shifts in the distribution.
5.2 Real Data Experiments
We conducted experiments utilizing five bascline models on four datascts as benchmarks to evaluate the proposed models in the time-series data generation task. Then, evaluated the generated data based three metrics, including marginal distribution, indistinguishability, and predictiveness. The evaluation results are shown in Table 1.
5.2.1 Baseline Models and Datasets
We included five representative timo-series models as comparison basclines: TimeVAE (Desai et al., 2021), TimeGAN (Yoon et al., 2019), QuantGAN (Wiese et. al., 2020), T.S4 (Zhou et al., 2023), and SDEGAN
20

---

Published in Transactions on Machine Learning Research (03/2025)
(Li et al., 2020; Kidger et al., 2021). Each model brings a distinct approach to the generation of synthetic time-series data.
Specifically, TimeVAE leverages a variational autoencoder structure with convolutional layers to capture the temporal dynamics and dependencies inherent in time-serics. TimeGAN, QuantGAN, and SDEGAN are built on the generative adversarial network framework to maintain temporal correlations within the data. The LS4 model addresses the challenge of capturing long-term dependencies within time-series by introducing a state space ordinary differential equations (ODE) framework for latent variables. We note that our model is denoted by CP-SDEVAEL, where L denotes the mumber of change points assumed.
We ran experiments on four datasets, including the S&P500 prices, S&P500 intraday prices, cryptocurrency prices, and air quality measurements. Supplementary Material provides a detailed description of these datasets. Time-series samples in all datasets have a fixed length of 120, while different datasets contain different numbers of samples. To evaluate the performance of the modcls, cach datasct was split into two subsets, a training and a lesting set. This split was conducted following the "80-20" rule that 80% of the samples were randomly selected to form the training set, and the remaining 20% was the testing set.
5.2.2 Evaluation Metrics
Our quantitative evaluation framework in this work encompasses three distinct metrics, each targeting a specific aspect of data quality and utility. The metrics include marginal similarity, data indistinguishabiiiy via classification, and prodictive quality:
Marginal Distribution Similarity: First, we assess the similarity between the marginal dis- tributions of the generated and real timo-series data. This evaluation is conducted based on a histogram-based difference. It calculates the density histogram of the real data, which serves as a reference for synthetic data. Here, we fixed the number of histogram bins for density calculation. Then, the comparison calculates the absolute difference in densities across correspouding distributions. Specifically, the final marginal score is obtained by averaging the discrepancies across all bins and data dimensions. The minimum value of the marginal score would be zero, indicating perfect match in marginal distributions between real and synthetic data, while the upper bound of the marginal score cannot. be directly determined without any constraint. Thus, the smaller the marginal score, the better the capability of the proposed model to replicate the distributional properties of the real data. We denote this score as "Marginal (4)" in our experiments.
Synthetic Data Indistinguishaaiiitt:: The second metric evaluates the indistinguishability of synthetic data from real data through a classification approach (Yoon et al., 2019). A downstream classifier, built upon the structured state space model, or the $4 model mentioned in (Zhou et al., 2023), is trained to difforentiate between synthetic and real data. Specifically, the S4 model maps the input time-series data to a higher dimensional space via a linear encoder to capture the temporal dynamics within the data and produces classification scores via a linear decoder. Real and generated synthetic data with the same sample size are concatenated to form unified dataset, which is then split into training and testing sets. Binary labels are assigned to indicate the source of each time-scrics. The classifier is trained on the labeled dataset using binary cross-entropy (BCE) loss to distinguish real and synthetic samples. The outcome of this evaluation represents the model's accuracy in classifying synthotic versus real data in the testing set. We took the absolute value of the accuracy after subtracting 0.5, which is the value that indicates that the model cannot distinguish real and synthetic samples. Thus, lower classification scores indicate a higher degree of indistinguishabiliyy suggesting that the synthetic data closely mimics the real data. This metric directly addresses the proposed model's capability of generating data that is qualitatively indistinguishable from real data. We denote this score as "Classification (L)" in our experiments.
Synthetic Data Predictive Quality: The third aspect of evaluation is the predictive quality of synthetic data. This metric provides insights into how well synthetic data can be a proxy for real data in prediction tasks. We utilized the S4 model with the same model structure as the predictor but enabled it to predict corresponding future values given time-series data. Unlike training
21

---

Published in Transactions on Machine Learning Research (03/2025)
combined dataset to evaluate the classification quality, the predictor was trained only the syuthetic data and tested on the real data. The prediction accuracy is calculated using the mean squared error (MSE) between the prediction results and the actual values. Thus, the lower the score, the better the proposed model's capability to generate high-fidelity synthetic data. We denote this score as "Prediction (4) in our experiments.
5.2.3 Results and Discussion
The experimental results, presented in Table 1, demonstrate that CP-SDEVAE outperforms most baseline modcls, even without assuming any change points (L = 0) in real datasots. Among the competitors, the LS4 model emerges as the closest rival, with CP-SDEVAE achieving comparable performance across nearly all datasets. Notably, CP-SDEVAE significantly outshines its GAN-based counterpart, SDE-GAN, even scenarios without change points. This superior performance is attributed to the greater stability and efficacy of VAE-based generative models compared to GAN-based models, particularly with smaller datasets. For the S&P 500 datasels and the oryptocurrency datasets, the baselines approaches such TimeVAE, TimeGAN, QuantGAN and SDEGAN tend to perform relatively worse than LS4 and CP-SDEVAE. This is due to the fact that those datasets contain stronger distributional shifts than the Air Quality dataset, which has the most comparable results across all methods. As a note, the Air Quality dataset demonstrates repeated fluctuations; making a poor candidate for SDE modeling in the first place.
Further discussion is warranted on the necessity of incorporating change points in our model. The architecture used in these experiments features underlying latent SDE represented by an MLP with 3 hidden layers and 64 hidden units, which is adept at capturing nonlinear shifts in drift and diffuusion. In the datasets tested, shifts occur more gradually rather than abruptly, reducing the importance of change points for some datasets. However, the utility of change points becomes evident with the S&P 500 data, where employing = 2 change points significantly improved the classification score. To explore this hypothesis further, we conducted ablation studics on the S&P 500 sectors dataset, varying the number of layers and hidden units, as shown in Table in Section C of the Appendix. These studies reveal that the complexity of the underlying neural SDE model increases, the need for change points to enhance performance diminishes.
Finally, we compare the performance of CP-SDEVAE and LS4, as LS4 remained competitive across all datascts. Unlike SDEVAE and CP-SDEVAE, which are based on latent SDEs, LS4 employs latent ODE with a structured state-space model. By leveraging a convolutional representation, LS4 bypasses explicit hidden state evaluations, significantly improving computational efficiency. Notably, LS4 has been shown in (Zhou et al., 2023) to perform well on stochastic data with sharp transitions. However, CP-SDEVAE offers two key advantages over LS4. First, if the application requires segmenting time-series into different regimes, CP-SDEVAE provides this capability, whereas LS4 assumes single underlying model. This segmentation allows for learning distinct temporal dynamics and generating diverse lime-series patlerns during inference. Second, for interpretability, CP-SDEVAE explicitly models the latent diffusion term, whereas LS4 does not separately approximate it. This feature can be beneficial for analyzing the role of stochasticity in timo-scrics evolution.
6 Conclusions
In this work, we introduced novel formulation of neural SDEs within the VAE framework, enabling seamless integration of change points using principles from maximum likelihood estimation and classical change point detection theory. We presented theoretical results demonstrating the convergence of change points and VAE model parameters to a stationary point, as well as the optimality of the Bayesian detector used in our method, which minimizes the probability of error in the test. We evaluated our algorithm on various real-world datascts, finding that our generative model achioves competitive performance compared to other deep generative models for Lime-series data and effectively captures distributional shifts.
Future Work
There are various research directions that can be considered as future work. Importantly, one limitation of the CP-SDEVAE method is that it requires that the number of change points be specified priori. While selecting
22

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Time Series Experiments Summary Table
Baseline Models
Proposed Models
Data (dimension)
Metric
TimeVAR
TimeGAN
Quant
LS44
SDEGAN
CP- SDEVARg
CP- SDEVAR;
CP- SDEVARg
Marginal
0.089 1
0.002
0.080 1
0.015
0.122 1
0.065
0.013 1
0.001
0.076 1
5000
0.030 1
0,000
0.027 1
0.006
0.026 1
0.006
S&P 500 (606, 120)
Classifica-
tion 4
0.300
0.107
0.261
0.130
0.500
0.000
0.045
0.105 =
(.000 1.070
1.000
= 02800 = 1074
F9000
2900
0.160
10000
09000
=
0.061
0.035 = +
Prediction +
0.617 +
0.273
0.365
0.475
0.582
0.422
0.004
00003
0.009
1000
0,016
S&P 500 intraday (500, 120)
Marginal
Classifica-
0.475
0.086
0.002
0.036
0.485
0.006
0.500 1
0.100
0.032
0,0700
0.016 1
0,000
0,033
T
0.0000 1
0000
04330
0/020
1
00700
20000
1 2.010
1800
2000
1,000
T000 00000
00000
T 21100
1000
+ 21100
2000 1,000
00 2801
T 10000
1
tion 4
0.032
0.012
0.000
0.886 T
+ 6611
T 16110
T 09800 0
Prediction +
1.634
0.438
1.448
1.022
3.257 +
2:330
0.144
0.010
1.422
03030
1200
2220 1200 0
T 1107 0.100
1,104
Marginal J
0.120
0.012
0.102
0.012
0.022
0.122
0.033
0.178 1
0.033
0.107 1
0.015
0,010
1010
Crypto currency (12, 120)
Classifica-
0.300
0.300
0.200 1
0.200 1
0.300 1
0.200 1
10100
0.200
0.23
1.100 =
0300
tion 4
0.245
0.245
0.245
0.245
0.245
0.240
Prediction
0.073 +
0.022
1.180 +
0.962
1.344 +
1.222
0.105 +
0.053
0.441 +
0.446
0.559 +
0.404
0.507 T
0/414
00
24697 T 1 8
0.063
0.040
0.133
0.034
0.109 +
0.057
0.044
0.056 +
Marginal J
0.003
0.004
0.049
0.006
0.015
0.024
0.005
0.012
Classifica-
0.220 +
0.260 +
0.500 +
0.220 +
0.380 +
0.220 +
0.260 +
0.220 +
quality (60, 120)
tion 4
0.098
0.150
0.000
0.098
0.098
0.160
0.150
0.160
0.723
1.189 1
4.696
0.657 +
3.845
1,272
0.894
0.938 1
Prediction
0,308
0.722
2.175
0.335
4.274
0.582
0.232
0.230
among multiple candidate models straightforward (using model selection techniques like cross-validation or marginal likelihood maximization), it can be computationally expensive. An interesting extension of this work would consider that the number of change points need not be specified, and furthermore, would not require that all observed lime-series share the same change point (i.e., heterogeneous change points). Another interesting direction for future work is to theoretically understand why the predictive log-likelihood regularizer improves the performance of the generative model (i.e., the model's noise estimation). Lastly, the current version of the CP-SDEVAE model uses a numerical algorithm that alternately optimizes the model parameters and the change points, due the fact that the joint optimization of the model parameters (which are real valued) and the change points (which are discrete valued). While this work showed that theoretically the numerical algorithm converges to stationary point of the ELBO, introduces the additional computational burden of having to use particle filtering to resolve the change point updates, which can be expensive in practice. A practically useful extension of this work would allow for differentiable change point modeling, possibly using temporal point processes (see Koley et al. (2023)). This would enable joint optimization of the model parameters and the change points, which comes with standard convergence guarantees of stochastic optimization algorithms.
Broader Impact Statement
The presented work proposes algorithm for modeling change points in time scries data, which can potentially be applied in a variely of disciplines including ecology, finance, and robotics, among others. The proposed work can have a positive impact on society used responsibly, as offers the capability to model distribution shifts time scries data. We highlight that the proposod work can also be incorporated into risk forecasting systems and potentially provide more robustness forecasting uncertainty when distribution shifts may occur. Importantly, the scenario considered in this work addressed change points modeled agnostic of exogenous
23

---

Published in Transactions on Machine Learning Research (03/2025)
drivers (i.e., covariate information). In applications where distribution shifts are known to be driven by certain covariates, responsible usage of this work would entail incorporating this covariate information into the generative model in order to detect and model change points.
References
Alexander P Browning, David Warne, Kevin Burrage, Ruth Baker, and Matthew Simpson. Identifiability analysis for stochastic differential equation models in systems biology. Journal of the Royal Sociely Interface, 17(113):20200652, 2020.
George Casella and Roger Berger. Statistical inference. CRC Press, 2024.
Dan Crisan and Arnaud Doucet. A survey of convergence results on particle filtering methods for practitioners. IEEE Transactions on signal processing, 50(3):736 746, 2002.
Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Timevae: variational auto-encoder for multivariate time series generation. arXiv preprint 11:::2111.0000000 2021.
Petar M Djuric, Jayesh H Kotecha, Jianqui Zhang, Yufci Huang, Tadesse Ghirmai, Mónica F Bugallo, and Joaquin Miguez. Particle filtering. IEEE signal processing magazine, 20(5):19 38, 2003.
Ronald Aylmer Fisher. Theory of statistical estimation. In Mathematical proceedings of the Cambridge philosophical sociely, volume 22, pp. 700-725. Cambridge University Press, 1925.
Ali Hasan, Joao M Percira, Sina Farsiu, and Vahid Tarokh. Identifying latent stochastic differential equations. IEEE Transactions on Signal Processing, 70:89 104, 2021.
Linus Heck, Maximilian Gelbrecht, Michael T Schaub, and Niklas Boers. Improving the noise estimation of latent neural stochastic differential equations. arXie preprint arxiv:2412. 17499, 2024.
Liam Hodgkinson, Chris van der Heide, Fred Roosta, and Michael W. Mahoney. Stochastic normalizing flows, 2020.
Thierry Huillet. On Wright-Fisher diffusion and its relatives. Journal of Stotistical Mechanics: Theory and Experiment, 2007(11):11006, nov 2007.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
Rudolph Emil Kalman. A new approach to lincar filtering and prodiction problems. 1960.
Steven Kay. Statistical signal processing: estimation theory. Prentice Hall, 1:Chapter 3, 1993.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Newral Information Processing Systems, 33:6696 1566696-6707, 2020.
Patrick Kidger, James Foster, Xuechen Ti, and Terry J Lyons. Neural sdes infinite-dimensional gans. Tn International conference on machine learning, pp. 5453 5463. PMLR, 2021.
Peter E Kloeden, Eckhard Platen, Peter E Kloeden, and Eckhard Platen. Stochastic differential equations. Springer, 1992.
Paramita Koley, Harshavardhan Alimi, Shrey Singla, Sourangshu Bhattacharya, Niloy Ganguly, and Abir De. Differentiable change-poie point detection with temporal point processes. In International Conference on Artificial Intelligence and Statistics, pp. 6940-6955. PMLR, 2023.
T. Lelièvre and G. Stoltz. Partial differential equations and stochastic methods molecular dynamics. Acta Numerica, 25:681-888 880, 2016. doi: 10.1017 0010077/0096249991600000
24

---

Published in Transactions on Machine Learning Research (03/2025)
Xuechen Li, Ting- Kam Leonard Wong, Ricky TQ Chen, and David K Duvenaud. Scalable gradients and variational inference for stochastic differential equations. In Symposium on Aduances in Ayprocimate Bayesian Inference, pp. 1 1-28. PMLR, 2020.
Fernando Llorente, Luca Martino, David Delgado, and Javier Lopez-Santiago. Marginal likelihood computation for model selection and hypothesis testing: an extensive review. SIAM review, 65(1):3-55 -8,, 2023.
Anton Mallasto, Guido Montúfar, and Augusto Gerolin. How well do wgans estimate the wasserstein metric? arXiv preprint orXiv:1910.09895 2019.
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science Business Media, 2013.
Ewan S Page. Continuous inspection schemes. Biometrika, 41(1/2):100-115, 1954.
Aleksey S Polunchenko and Alexander G Tartakovsky. State-of-the--as in sequential change-point detection. Methodology and computing in applied probability, 14:649-684 684, 2012.
Tom Rainforth, Robert Cornish, Hongscok Yang, Andrew Warrington, and Frank Wood. On nesting monte carlo estimators, 2018.
Aaditya Ramdas, Nicolás García Trillos, and Marco Cuturi. On Wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47, 2017.
Artem Ryzhikov, Mikhail Hushchyn, and Denis Derkach. Latent neural stochastic differential equations for change point detection. arXiv preprint arXiv:2208. 10317, 2022.
Timothy Sauer. Numerical solution of stochastic differential equations in finance, In Handbook of computational finance, pp. 529-550 Springer, 2011.
Gerald L Smith, Stanley F Schmidt, and Leonard A McGee. Application of statistical filter theory to the optimal estimation of position and velocily on board circumlunar vehicle, volune 135. National Aeronautics and Space Administration, 1962.
T. K. Soboleva and A. B. Pleasants. Population growth as nonlinear stochastic process. Mathematical and Computer Modelling, 38(11):1437-1442 2003.
Jan Stanczuk, Christian Etmann, Lisa Maria Kreusser, and Carola-Bibian Schönlicb. Wasserstein gans work because they fail (lo approximate the wasserstein distance). arXiv preprint arXiv:2103.0167. 016077 2021.
Zhongchang Sun, Yousef Fl-Lahhm, and Svitlana Vyetrenko. Neural stochastic differential equations with change points: A generative adversarial approach. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1965-6969. IEEE, 2024.
Charles Truong, Laurent Oudre, and Nicolas Vayatis. Selective review of offline change point detection methods. Signal Processing, 167:107299, 2020.
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. orXiv preprint arXiv:1905.0990 2019.
Magnus Wiese, Robert Knobloch, Ralf Korn, and Peter Kretschmer. Quant gans: Deep generation of financial time series. Quantitative Finance, 20(9):1419-1 -14104900,0 2020.
Jichuan Yang and Harold Kushner. A monte carlo method for sensitivity analysis and parametric optimization of nonlincar stochastic systems. SIAM journal on control and optimázation, 29(5): 1991.
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. A duancee in Newral Information Processing Systems, 32, 2019.
25

---

Published in Transactions on Machine Learning Research (03/2025)
Shuyi Zhang, Bin Guo, Anlan Dong, Jing Hc, Ziping Xu, and Song Xi Chen. Cautionary tales on air air-quality improvement in beijing. Proceedings of the Royal Society A: Malhemalical, Physical and Engineering Sciences, 473(2205), 2017.
Linqi Zhou, Michael Poli, Winnie Xu, Stefano Massaroli, and Stefano Ermon. Deep latent state space models for time-series generation. In International Conference on Machine Learning, pp. 42625 42643. PMLR,
2023.
A Appendix
A.1
Theoretical Proofs
In this part of the appendix, provide proofs for the propositions presented in the paper. For convenience, restate our assumptions for each proposition and provide short justification of the assumption.
A.1.1 Assumptions for Convergence to a Stationary Point
To prove that the training algorithm for CP-SDEVAE converges to a stationary point, we made a few assumptions (which we find reasonable). We state these assumptions in the following and provide a short justification:
Assumption 1. [Model update phase always leads to an ELBO improvement/ Let 6' + and ø' + Φ denote the model and variational porameter updates oblained from numerically mazimizzng ( We assume. that for fized change point v, our model parameter updates lead to an improvement in the ELBO:
A 0, A Eor (Xoots)) - ( 2 6m: 5m 20
Justification: The model update phase aims to update the model parameters and the variational approximation parameters & by maximizing the ELBO for a fixed change point. Standard stochastic optimization approaches guarantee that these updates improve the objective function (in expectation). Therefore, it is reasonable assumption.
Assumption 2. [Change point update phase always leads to a marginal likelihood improvement) Let v/ + v denote the change point update obtained from numerically mazimizing the marginal likelihood: pa Poooaall = t). We assume that for fized model parameters 0, our change point updates lead to an improvement in the marginal likelihood:
AC(V ,v) A log pep (aoobll = v') - -log Po = deps
dep 2
Justification: With the maximum likelihood approach, the change point updates are made to maximize lo = 1) w.r.t. the change point Since assume that the change point occurs at a sample time t e T this is a discrete optimization problem that can numerically be solved with Bayesian filtering approaches. In this work, we utilized the bootstrap particle filter (BPF), which provides an estimator of the marginal likelihood that converge almost surely to the true marginal likelihood. Therefore, the greedy change point update guarantees the assumption in the limit infinite particles used the BPF form the estimator of the marginal likelihood (justified by the strong law of large numbers).
Assumption 3. [Change point updates do not widen the inference gap] Let v +v denote the change point update oblained from numerically mazimizing the marginal likelihood: po po(#otzlu= = t)- We assume for fixed variational parameter &, the KLD between the variational opprorimation and the posterior distribution of the initial state zu smaller than the improvement the marignal likelihood:
DKL(go (z0 Lobe) Ilpo, (zolxora)) PKI((a) (zo (oka)) llp, Roba)) S 8KL, Sep 2 δKL.
where dep denoles the marimum improvement in the marginal likelihood during the change point update phase.
26

---

A.2 Proof for Convergence to a Stationary Point Published in Transactions on Machine Learning Research (03/2025)
Justification: Unlike the aforementioned assumptions, this assumption is non-standard and requires proper mathematical justification based on this specific problem setting. Let ADKil(V' (v',,) v) denote the change in the KLD between the variational approximation and the true posterior. We can manipulate this expression as follows:
(zol (bb))) - DKL Dobs) ADKL(V, DRL(Oo Fobeoo/ooollloooooo (iiioo,r/zolaaann
q (20) (obba)
}]
lep (z (Robs)
Eus
log
Eys
log
Pa,v' (zo (2
po.v (2 (oohs)
4p ( (obbs)
Pe.v ( Zu) Dobe)
Bue
log
+1gg log
(Manipulate logarithm)
Pst zo @obs)
pe, o' (z)) (obs))
Po,u (((0)) pe. (Bouss)
Egg
log
(Bayes' theorem)
Po.e (mobs Fo (0)
Pe.rr (Bobs))
Pa.er (Dobs)
pa., (Xobs =0)
Eye
log
Eg
log
(Manipulate logarithm)
po.v (Zobe)
pe. vi (Dobal (0)
pe. (Doba)
Pa @obbs 20)
loog
Egs
log
Pa. (Fota))
Pa. ( (Eohas 2o)
6ep
8n
Therefore, to justify our assumption that ADKL (1', 1) S ôKL, where dep ôKL, need to justify the incquality ònr < 0 which is equivalent to showing that:
log Po (Xotsl (2)) 2 log (T 22), Ep E Rdd
The key difference between this inequality and Assumption 2 is the additional conditioning on the initial state 20. We can write each of these conditional likelihood as integral mcasures with respect to Zobs:
Pe. (Zobelbbl = (1I po (ote zrc )o. ( dz A-
dzix
We can see that as K grows, the dependence of the latent state 2o gets smaller. This something we should expect from SDE models, since the dependence of the initial state zo on the observed trajectory Lobs should be minimal for long sequences. Since the change point updates are made to maximize marginal likelihood, average we expect that log Pe, ( =0) be larger than ( =0) and therefore it is reasonable to assume 6μ 0. This implics that dKL I dep and the assumption is verified. We emphasize that the validity of this assumption is conditional on the fact that the time-series being considered are of sufficient length.
In this subsection, we prove Theorem 1.
Proof. We would like to show that for any i E N:
A 0 Φ', o, v/ & Eor os (Kobma) Eo E. (Pobs) 2
We can show this in two steps by showing that:
Eg ,' (Fobe) Eo w (Nobe) 2 Eg.s.(uEbbe
Change Point Update
Model Update
By Assumption 1, if the updates to the model parameters are efficient (for fixed change point v), then we the first inequality is trivial, i.e.
AE 0 Φ. ()) > 5m
Eo (2 Ed.
27

---

Published in Transactions on Machine Learning Research (03/2025)
To show the second inequality, capitalize Assumption 3 which claims the variational inference gap does not widen after change point updates. The change in the logarithm of the marginal likelihood (pre/post- change point updates) is given by:
AC(V', = (Eor , (Xoba) Eor .s v ( +
AE(U),))
(DKL(g (zol (oba) ) lIpor w (20 (ubb)) - DKL (1 (20 Dabs Ilpor ( Pobs))
ADKL(V'N))
Tt. follows from Assumption 2 that AC(V'v)) 2 ADKI(V (V/,) v) and therefore,
AE ,1) 1) 20
Thus, conclude that AE 0, v,) 2.
A.3 Proof of Asymptotic Optimality of Test Statistic
Proof. In the following, we show simple proof for the case of using a Monte Carlo estimator of the marginal likelihood of the change point. We note; however that this proof can be trivially extended to any estimator that converges almost surely (e-g., particle filtering-based estimators of the marginal likelihood). For brevity in the notation, also remove the dependency in the discussed distributions on O since assume that it is fixed and the locally optimal value.
(m) Consider set of i.i.d. samples 2 p(zu:ll== T) for -,..., M. By the strong law of large numbers, we have that
M
lim
_V==) ,u= M p(ax (2 (mm)))
Σ
(m)
p( 7),
(34)
almost surely. Consider the function (xx +. The set of discontinuity points D, of fxx satisfies P(x D₂) = 0. By the continuous mapping theorem, we have that
lim M
(35)
M EM
p(x, v =) p( D=
almost surely. Therefore, by Slutsky's theorem, readily deduce that
lim M p(z 24 M-300 EM
M
(m)
w =)
p(Z = > 7) p( v > T)
(36)
pl
(B
(m)
17
almost surely. Note that for the binary hypothesis testing problem (in the case of change point detection), the optimal likelihood ratio test is defined as follows
{ 1,
p(a+|v=r) 2 7
7*(),)
]
uSt
(37)
0,
p( v>T) < 7.
28

---

Published in Transactions on Machine Learning Research (03/2025)
For the type-I error probability, have that
T)
P
lim
M
m
27)
(m) T) (m))
m=1 pl
=P
lim
M m=1
p( p(
T)
7, 4-000
lim
M
m=1
P( 2
(m)
V =V
plac T)
M-700
2m p1 21
M
(m)
1V2 7)
m=
M
p
V 7 T) Zt
(m)
p( V 7)
1 M
m== P
(m) r)
m= ((
(m) vV= r)
p Ma =
IP
lim
M
7, lim
M M-000
M 2m= (
M
(m)
v/ 7)
M +00
m=1
M
P(
// k>+) T) (m)
p
M
m=1 (( (m) .V=7 T)
M
m== P
M
v V=
p(ayvy T)
P
lim
2
lim
M-00
M > ((
|ン 7)
M+
M 2m=1 (( 1> 7)
M
p(znvu>])
M7 2m= p( (m)) V
M
p( V = T)
P
lim
M
M Σ m= p(
V .> T)
p(a,
= p( = T)
p > T) 2 /
(38)
where the last inequality is from (36). We can also derive the same result for the type-Il error probability.
B Datasets and Preprocessing
This work used four datasets to evaluate the proposed and baseline models. We selected these datasets to demonstrate the effectiveness of change point detection in synthetic time-series data generation so that the datasets contain clear value shifts. We preprocessed all datasets by normalizing time-series per sequence with a zero mcan and one variance. Let X be dataset with N data points 2i, where = 1......, .N. The normalization of X is performed as follows:
First, calculate the mean μ and standard deviation a of X:
(39)
2m
N Σ (x. - 14)2
(40)
Then, normalize each data point z; to obtain the normalized value 2, using:
Tt P
24
(41)
0
B.1 S&P 500 Dataset
We collected stock tickers from the S&P 500 and utilized per-sequence normalized price as the first dataset, which contains 504 tickers total, as shown in Figure 8. We downloaded these original price timo-scrics via the Yahoo Finance Python package 3. The time range of this dataset starts from January 2020 and ends in June 2020. The length of each sample is 120. This datasct covers the significant stock price drop at the beginning of the COVID-19 pandemic in March 2020.
B.2
S&P 500 Intraday Dataset
Like the S&P 500 dataset mentioned above, we collected the intraday prices on Feb 5th, 2024, to form the second dataset. The length of this dataset was also set to 120. The intraday prices were originally parsed as
Shttpe://github. com/rranarousss//yfinnnee.
29

---

Published in Transactions on Machine Learning Research (03/2025)
Norma Vallues
20
40
60
80
100
120
Timestep
Figure 8: Normalized stock prices in S&P 500.
Normalized
20
60
80
100
120
Timestep
Figure 9: Normalized intraday stock prices in S&P 500.
per-min prices across the training day and then down-sampled to 120 steps for each stock. However, four stocks were removed because they had less than 120 data points because of a limited number of executed trades on the selocted day. Thus, the total number of timo-scrios samples is 500.
B.3 Crypto Dataset
The third dataset is collected from cryptocurrency prices We downloaded twelve cryptocurrency price time-series via the Yahoo Finance AP1 and normalized them using the method mentioned above. The time range of this datasct starts in carly March 2021 and ends in June 2022 with total length of 120 time steps. This dataset covers when cryptocurrency prices significantly increased and varied in the first. half of 2021.
B.4 Air Quality Dataset
We also used the 'Beijing Multi-Site Air-Quality Dataset', which is available Kaggle (Zhang al., 2017). This dataset is extensive collection of air quality mcasurements from 12 monitoring stations across Beijing.
30

---

Published in Transactions on Machine Learning Research (03/2025)
BTC
ITC
ETH
BNG
DOCE
BCH
Normal Vallues
XRP
ADA
XLM
LOS
20
40
60
80
100
120
Timestep
Figure 10: Normalized cryptocurrency prices.
Normalized
20
60
80
100
120
Timestep
Figure 11: Normalized air quality measures.
It records various pollutants like PM2.5 and PM10 hourly from 2013 to 2018. Here, we aggregated the original hourly-based data into weekly-based data by taking the mean across corresponding values. We constructed the dataset by selecting five pollutants, including PM2.5, PM10, SO2, NO2, and CO. This dataset contains 60 time-series samples with a fixed length 120.
C Ablation Studies
In this section, present ablation studics conducted on both the synthetic and real datascts considered in this work.
C.1 Synthetic OU Dataset
We consider a synthetic OU dataset with a single change point and train our proposed neural SDE model without change points. The goal of this ablation study test the various hyperparameters of the base CP-SDEVAE model. Table shows the different hyperparameters tested, along with their assumed default value when the hyperparameter is assumed to be held fixed. In the following, conduct ablation study
31

---

Published in Transactions on Machine Learning Research (03/2025)
Parameter
Default
Search Space
Parameter Description
latent fiie hidden dim encoder
32
[4, 16, 32]
latent space dimension
128
[128]
# of encoder hidden units # of encoder layers
num layers_encodee
[1, 2, 3]
hidden dim sde num layers_so sde var _de_dee
64
[32, 64, 128]
# of drift/ srift/diee /diffusion hidden units
[1, 2, 3]
# of drift/ /diffusion layers
1.0
[0.01, 0.1, 1.0]
decoder variance latent diffusion type
is diffusion homoscedastic
True
[False, True]
latent_dde Lilii_dffffssoonvee_ -va
1.0
[0.01, 0.1, 1.0]
latent diffusion value flag for training diffusion # of decoder hidden units weight of NLL in loss weight of KLD loss weight of predictive NLL # of SDE trajectories step size for Euler solver
train diffusion
False
[False, True]
decoder type nll weight kld weightt pred nll weight
'mlp'
['linear, 'mlp"]
1.0
[0.001, 1.0]
1.0
[0.001, 1.0]
0.05
[0.0, 0.01, 0.05, 0.1]
num tite_trajectories
5
[1, 5, 10]
culer_step__t
1.0
[0.05, 0.1, 1.0]
Table 2: Hyperparameters: Default values, ablation study configurations, and concise descriptions
by varying the value of two hyperparameters at a time according to their corresponding search space as shown in Table 2. To test performance, we plot the generated trajectories from each trained model after E 250 epochs, along with the corresponding ELBO.
C.1.1 Latent SDE Size vs. Number of SDE Layers
In this part of the ablation study, we want to understand varying both the number of layers in the SDE drifts and diffusion and the number of hidden neurons had a big impact on performance. We observe a weak trend that when the number of layers in the neural SDE drift and diffusion networks is small (1-3 layer), increasing the number of neurons achieves higher ELBO. The best parameter configuration here was to utilize 3 hidden layers with 64 neurons per layer, achieving ELBO value of -5.40. The worst performing model was the most complex one (3 hidden layers with 128 neurons per layer), which achieved an ELBO of -7.24.
C.1.2 Latent Space Size vs. Decoder Type
In this part of the ablation study, our goal is to compare different types of decoder: either linear decoder (denoted by the configuration 'hidden diide decoder=None') or an single-layer MLP with 128 neurons. Along with the type of decoder, we also vary the size of the latent space. In general, observe the trend that the linear decoder (across all latent dimension sizes), achieves equal or better performance as compared to the MLP decoder. This can be attributed to the fact that the decoder in this example is actually compressing the latent variable (whose size is greater than the time-series dimension). We notice that as the latent dimension gets larger, the performance of the the two decoders bocomes more similar.
C.1.3 Latent Space Size vs. Number of Encoder Layers
In this part of the ablation study, we vary both the number of encoder layers and the dimension the latent random variable. Here, we observe the trend that utilizing a larger number of layers improves performance in terms of ELBO, across all latent dimension sizes. We note that the best performing model was achieved for the most complex model in hyperparameter search space, where the size of the latent variable was 32 and the number of encoder layers was 3.
C.1.4 Latent Space Size vs. Number of SDE Trajectories
In this part of the ablation, we study the impact of sampling additional SDE trajectorics in our nested Monte Carlo estimator of the ELBO over a fixed number of iterations. We study this across different. sizes of
32

---

Published in Transactions on Machine Learning Research (03/2025)
hitcen_dim_ jayers_dde
hidden_din_sde layera_dde
F180 -6.7
F180 -2.25
FL8O -5.65
valus
value
Valtun
Time Index
Time Index
Time Index
hiddde_dim__s byers.pdeeeeee.
hidden dim_ddd
hidden_dim_se 125,
EUBO a 5597
ELBO a 6666
EL8O 5.11
Vilue
Valuu
hisiden_dne hiside__dim_om dimmmoemmeeeeeeeeeooooooom = 32, num, byers.pde 3
aiccen_dim_rde = 54, num, jayers_dee ELBO -5.26
hidden_dim_see 128, num Jayers_sde =
EUBO 5,33
ELBO 5.66
Volue
Voluee
Value
Index
Index
Index
Figure 12: hidden
dim sde vs. m layers_ sde.
33

---

Published in Transactions on Machine Learning Research (03/2025)
latent_dii dim 4, hidden dim decoder None ELBO -6.47
latent_ dim = 4, hidden dim decoder 128 ELBO -6.36
Value
Value
16
58
30
Time Index
Time Index
latent dim = 8, hidden dim decoder None
latent_ dim = 8, hidden dim decoder 128 ELBO -5.92
ELBO
Value
Value
Time Index
Time Index
latent_ding 16, hidden ceadenndim_aae decoder = None
latent_ dim 16. hidden_ddin decoder = 128
ELBO 5.47
ELBO -600
Value
Value
Time Index
Time Index
latent dlim 32, hidden dim decoder = None
latent dim
32, hidden dim decoder = 128 ELBO -5.21
ELBO -6.14
Value
Value
Time Index
Time Index
Figure 13: latent dim vs. decoder _tpp..
34

---

Published in Transactions on Machine Learning Research (03/2025)
latent dim
layers encoder EL9O
latent dim num, layers ancoder 2 ELBO
latent, dim
layerz encoder ELBO
Valus
Time Index
Time Index
Time Index
latend dimm mum_leyers_eee enccoee
laleni dim n.num_llyee arccddr
laterd dim encoder
ELSO - -5.22
EL8O .5.85
EL8O -5.91
Value
Time Index
Time Index
Index
latert dam 16, rum layers encoder 1
latent cim 16, num, layers, encoder -2
latert dim - 16, rum layers uncoder -3
ELSO -6.0
ELBO - -6.2
ELBO -5.48
Volue
Time Index
Time Index
Time Index
latert, dim - 32, rum, layers, encoder 1 EUBO -6.64
latent, cim - 32, num, layers, ancoder -2
Iatert, dim - 32, num layers encoder -3
EUBO -6:35
ELBO 55.30
Value
Time Index
Time Index
Time Index
Figure 14: latent dim vs. num layers encoder.
35

---

Published in Transactions on Machine Learning Research (03/2025)
the latent random variable. Across all sizes of the latent variable, we observe the trend that increasing the number of SDE, trajectories in the Monte Carlo estimator improves the model performance in terms of ELBO. We add that, based on the results in the figure, the generated samples also appear more realistic and capture the distribution of the underlying data better.
Intent_dm dim 4, num sde, trajectories 11 ELBO 3339
Intent_oim oim 6. num sde trajectories m ELBO 6.69
Intent_dim- m numuude trajectories 10
ELBO - 4.62
Volue
velue
Valuue
Time Index
Time Index
Index
latent,dim dim 8, num_sde praeccootes -1
latent_olm B, numudde trajecto 5
latent_dim- num sde, trajectories 10
ELBO 11.77
ELBO
ELBO
Volue
Value
Time Insek
Time Index
Time Indox
latent_cim- num_rde see trajectories
latent 16, num_dde traeectories =
Istent_ dimnt_dim-aa www.-eetaaaectorieee. 10
EL80 14554
ELBO 5.99
ELBO
Value
Time Index
Time Index
Time Index
latent clim 32, trajectories
lalent dim trajectories
latert trajectories 10
ELSO - -10.22
ELBO -5.99
ELBO - .339
Value
Value
Valuer
Index
Index
Indox
Figure 15: latent dim vs. num sde trajectories.
C.1.5 NLL Weight vs. Predictive NLL Weight
In this part of the ablation study, test to see whether or not the predictive negative log-likelihood regularizer is adding any value to the model performance. We vary the value of the regularization parameter of the predictive NLL regularizer under negligible NLL component (NLL weight = 0.001) and under vanilla NLL component (NLL weight- = 1). We can see that while removing the predictive NLL results in larger overall ELBO, the generated trajectories under the configuration of (pred NLL weight = 0) are noiseless, implying that the latent diffusion is meaningless (akin to a latent neural ODE). We can see that by incorporating the predictive NLL component, although the ELBO slightly decreases, the trajectories become noisier and thus are more realistic and better suited to modeling stochastie processes.
36

---

Published in Transactions on Machine Learning Research (03/2025)
LLMegi - 0EC PHS get- 0.0
HLMSSR 3071, pEdT wagt +033
HENAgR - OZE. 2103 wettt------------------- 500
+1,neght- ED0, potettage -1
420
veggt
<Leeggi- Neght-
(8 weget
1800 4.0
016 2 2.0
C.80 20---.05
Figure 16: is diffusion homoscedastic vs. diffusion value vs. train diffusion.
C.1.6 Homoscedastic Heteroscedastic Diffusion
As part of ablation study, we investigated three distinct approaches for modeling the latent diffusion in the neural SDE framework: 1.) learnable heteroscedasticc diffusion; 2.) learnable homoscedastic diffusion; 3.) fixed homoscedastic diffusion. Our analysis revealed that all three configurations yielded comparable generated timc-series, both in terms of trajectory dynamics and corresponding ELBO values. We posit that this similarity in performance can be altributed to the higher dimensionality of the SDE's latent space relative to the time-series data. This dimensional disparity allows even less complex latent dynamics to adoquately capture the obscrved timc-scrics dynamics. It noteworthy that the heteroscodastic latent diffusion assumption marginally outperformed the other approaches in terms of ELBO. We hypothesize that this superior performance stems from the increased flexibility inherent in the heteroscedastic model.
train diffusion True, latent diffusion type homosceda:
train diffusion True, latent diffusion type heterosceda ELBO = -4.56
ELBO = 5.88
Value
Value
Time Index
Time Index
Lrain_eiffasion Falia, difusioon
train, ciffusion ciffusion val
train diflusion diffusion val
ELBO -675
ELBO -559
tolu
Time Index
Time Indox
Time Index
Figure 17: hidden dim sde vs. num layers sde.
37

---

Published in Transactions on Machine Learning Research (03/2025)
C.1.7 Decoder Variance vs. Latent Diffusion Value
Figure 18 shows the tradeoff in model performance when varying the variance of the decoder and the variance of the latent diffusion (in the case of fixed, homoscedastic variance). We can see that decoder variance plays a larger role, as the ELBO is much worse when the decoder variance is too small. The best performance is obtained when the variance of decoder is 0.1 and the latent diffusion is 0.1.
decoder- 0.01, latent diffuston 0.01
0.01, latent diflusion val P180 -3 -1544
deceder 0.02, diffusion val
El 80 -123.00
Ft -126.59 59
valuo
Time Index
Index
Index
de_eeedder-- 0.1, latend, diffusionn_oe val 0,01
ver decodedeerrr 0.1, latert diffusion yii 0.1
vir ver_decoder 0. latent ciffusinn yal 1.0
E1800 3,67
EL8O - 19.49
ELBO - 18.64
Vauu
Time Index
Time Index
Time Index
var deccdddder- 1.0, latent diffusiion_vil val 0.01
var decocer 1.0, lalert diflssion val - 0.1
var clecoder 3.0 Ialant, ciffusion yal - 1.0
EL8O
ELBO
EL5O
Value
Value
Time Index
Time Index
Index
Figure 18: var decoder vs. diffusion value.
C.2 S&P 500 Dataset
In this part of the ablation study, we test the need for change points in the real data experiments, with results for different number of SDE layers and different SDE hidden neuron sizes shown in Table 3. We provide an analysis of the results of this ablation in the main text.
38

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Ablation Study on the S&P500 Sectors Dataset
Latent Size
Latent Size 32
Latent Size 64
Hidden Layers
Metric
SDE
SDE
SDE
SDE
SDE
SDE
SDE
SDE (1 CP)
SDE
(0 CP)
(1 CP)
CPs)
(0 CP)
(1 CP)
(2 CPs)
CP)
CPs)
0.036::
0.0364
0.031+
0.037
0.031
0.032
0.041
0.033:t
0.034:
Marginal 4
0.004
0.009
0.009
0.005
0.002
0.004
0.006
0.002
0.003
0.2731
0.255.t
0.20011
0.26411
0.2181
0.1911
0.364
0.27311
0.227.1
Classification
0.147
0.164
0.046
0.142
0.078
0.105
0.104
0.111
0.104
0.243
0.096+
0.127
0.142
0.080+
0.101
0.121=
0.160
0.100
Prediction 4
0.209
0.006
0.030
0.020
0.017
0.018
0.029
0.138
0.046
0.0294
0.0584
0.028
0,0364
0.0301
0.032+
0.035=
0.0384
0.035:
Marginal 4
0.003
0.037
0.004
0.006
0.008
0.003
0.004
0.006
0.003
0.21811
0.336.1
0.145.1
0.245.1
0.2271
0.1091
0.218
0.30911
0.200.1
Classification
0.053
0.084
0.093
0.130
0.119
0.062
0.133
0.060
0.111
0.093#
0.381
0.076:
0.103
0.092+
0.085=
0.117=
0.123
0.078
Prediction +
0.020
0.029
0.292
0.013
0.054
0.018
0.030
0.100
0.028
0.0334
0.0314
0.032-1
0,0344
0,0334
0.031
0.031=
0.0344
0.0334
Marginal $
0.005
0.002
0.005
0,006
0.006
0.005
0.004
0.006
0.003
Classification
0.109.1
0.089
0.173.1
0.145
0.236.1
0.034
0.1091
0.084
12001
0,080
4990
02277
0.076
0.1001
0.067
0.2001
0.124
0,2001
0.084
0.072.1
0.100.L
0.0591
0.0741
TT400
0005.1
0.063.1
Prediction |
0.014
0.079
0.008
0.018
0.018
0.028
0.006
0.008
0.014
39


Reviews:
[{"review_id": "idM1e2b60D", "reviewer": "Reviewer_Pvqi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper is generally well-written and well-structured. \n- Focusing on change points in time-series data is a promising and impactful research direction given a range of applications with abrupt distributional shifts.\n- The overall VAE-based framework and alternative training methodology (between neural SDE parameter updates and change point detection) seem to be novel. \n- Theoretical guarantees are provided, and the resulting method achieves robust and competitive performance in the presented experiments. \n\n**Weaknesses:**\n- It remains unclear how strong the method really is because of the following considerations (which are not sufficiently evaluated in real-world scenarios):\n\t- the number of change points needs to be known in advance.\n\t- the computational complexity seems to increase significantly with the number of change points.\n\t- increasing the network complexity can diminish the importance of change point detection.\n\t- other methods such as LS4 achieve comparable performance.\n- The method could be better motivated and explained (see the comments below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The present paper introduces a novel framework for modeling time-series data with distributional shifts using neural stochastic differential equations (SDEs). Leveraging a VAE framework, the proposed method relies on a variational prior for the initial state of a latent neural SDE instead of priors on the full process as used in previous work. Moreover, the method integrates change points into the models by representing the dynamics with different neural SDEs between different change points. The model alternates between training neural SDE parameters and refining change points either using greedy maximum likelihood-based updates or sequential change point detection schemes. The paper provides theoretical insights regarding convergence guarantees for the training algorithm and optimality for the detection scheme under specific conditions. Finally, the method is empirically validated on synthetic data, such as Ornstein-Uhlenbeck processes with distributional shifts, and real-world datasets, including financial and air quality datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Motivation:** \n- It remains unclear why a prior over the entire latent stochastic process might be too strong an assumption. The reasoning \"as the training data may not always conform to this prior\" should be further explained. Also, one could argue that regularization can be helpful in penalizing large deviations from the prior drift (see (8)). It also remains questionable whether summarizing all the observations into a single initial condition for the SDE is a good modeling paradigm. \n\n**Minor issues:**\n- Neural SDE also fall under the \"traditional SDE modeling\" as formulated in the introduction, i.e., they have \"parametric models for the drift and diffusion functions\" and \"model parameters are then learned using [e.g.] maximum likelihood estimation.\"\n- $p_\\alpha$ is defined as initial distribution; however, it seems to be used differently in (8). Generally, this section is hard to follow.\n- The variational distribution and its parameters $\\phi$ appear in the Algorithm Summary in Section 4.2; however, they are not properly defined before.  \n- The Lipschitz continuity of the drift and diffusion terms seems to be required in the spatial dimension. In the temporal dimension, Hölder continuity (with $\\alpha=0.5$) seems to be sufficient for, e.g., Euler-Maruyama.  \n\n**Questions:** \n- One version (\"sequential likelihood ratio test\") of the second step of the proposed algorithm (\"Update the change points\") seems similar to the work by Ryzhikov et al. (2022). In that sense, the distinction between *offline* and *online* algorithms in Section 3.2 seems somewhat artificial. \n- Is \"backpropagating through the SDE solver\" not implicitely using \"the reparametrization trick\". It would be good to clarify the discussion before Section 4.3.2.\n- It would enhance accessibility to add further explanations to Section 4.3.2, both on the motivation/derivation and the estimator. Is it correct that this regularization is not considered in the theoretical results? \n- The \"assumption that the inference gap as a result of the variational approximation does not widen after change points are updated\" seems to be quite strong. Can the authors comment on its validity?\n- Why are the MLE-based change point updates not evaluated for the experiment in Section 5.1.2?\n- Can the authors evaluate and discuss the performance vs. cost tradeoff (e.g., in terms of runtime/flops) of the considered methods in Table 1? In particular, it would be interesting to observe the empirical scaling of the costs of the proposed method with different numbers of change points (including > 2). Specifically, the observation that the \"complexity of the underlying neural SDE model increases, the necessity for change points to enhance performance diminishes\" raises the question of whether the additional cost for estimating the change points is justified. \n- Why does the experiment in Section 5.2 not consider the method by Sun et al. (2024)?\n- Can the authors comment on the shortcoming that the number of change points needs to be known in advance?\n\n**Typos:**\n- \"a prior\" -> a priori\n- In several places, `\\citet` instead of `\\citep` should be used.\n- \"identity distribution shifts\" -> identify", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f1khuJ8yKA", "reviewer": "Reviewer_8zqg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengthes**\n\n1. The proposed strategy is simple, and the algorithm is straightforward.\n\n2. The authors evaluate the proposed method across various settings, ranging from controlled experiments to practical scenarios.\n\n3. The manuscript is easy to follow, even for non-experts in the field.\n\n**Weaknesses**\n\n1. Although the motivation of the paper is clearly stated, it is unclear which empirical results validate the effects of the weaker assumption on the prior. A more in-depth analysis, such as experiments on toy datasets comparing these two priors, would make the paper's motivation more convincing.\n\n2. The number of change points is treated as a hyperparameter, meaning it must be decided before model training. While the authors examined cases with different numbers of change points on toy datasets, further analysis on how the choice of this parameter affects performance (e.g., setting it to more than 2 in Sections 5.1) would enhance the validity of the proposed approach.\n\n3. The motivation for introducing the predictive log-likelihood in Section 4.3.2 is unclear. The explanation provided was not sufficient for me to understand its purpose. Further elaboration would help readers follow this section more effectively.\n\n4. Although Theorem 1 provides convergence analysis for the proposed update scheme, it is not clear how close the stationary point is to the (global) optimal solution. If discussing this gap is difficult, it should at least be mentioned as a limitation of the method.\n\n5. Theorem 2 addresses only the extreme case $J \\to \\infty$, which is not helpful in understanding how large $J$ should be in practice or how $J$ affects the approximation error.\n\n6. The manuscript contains some errors and undefined quantities in the formulations, which can be easily corrected.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new framework for training SDEs as VAEs, requiring only a prior on the initial state $\\boldsymbol{z}_0$, instead of the entire trajectory. It develops methods for modeling change points in time-series modeled as latent neural SDEs, including an iterative algorithm for joint learning of SDE parameters and change points. The approach is demonstrated to achieve comparable performances to state-of-the-art models on distributional shift generation benchmark datasets, offering improved modeling of complex time-series with abrupt dynamic changes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weeknesses listed above.\n\nAdditionally, I recommend that the authors clarify the differences between the proposed method and LS4 in terms of their strategies or underlying ideas. This clarification is particularly important given that LS4, which does not explicitly model change points, achieves competitive or better performance than the proposed method.\n\n**Minor comments**\n\n- $d_z$ is not defined.\n\n- What is $\\boldsymbol{\\sigma}_{\\boldsymbol{\\phi}}$ on page 2? I could not find its definition.\n\n- In the first paragraph of Section 4.3.2, $\\alpha$ and $\\beta$ should be defined.\n\n- The first equation in Section 4.3.2 (i.e., the definition of $\\mathcal{L}^{\\mathrm{pred}}$) contains $p\\_{\\boldsymbol{\\theta}}(\\boldsymbol{z}\\_{t\\_{k}}|\\boldsymbol{x}\\_{t\\_{k-1}})$, but should this instead be $p\\_{\\boldsymbol{\\theta}}(\\boldsymbol{x}\\_{t\\_{k}}|\\boldsymbol{z}\\_{t\\_{k-1}})$?\n\n- Does Eq. (20) contain typos? Should it be $p(\\boldsymbol{x}\\_{\\mathrm{obs}}|\\nu=\\tau)=\\prod_{k=1}^K p(\\boldsymbol{x}\\_{t_k}|\\boldsymbol{x}\\_{t<t_k},\\nu=\\tau)$?\n\n- Eq. (21) should be consistent with Eq. (16); however, there are several discrepancies. For instance, $\\tau$ is not clearly connected with $t\\_{k+1}$ in Eq. (16). Additionally, the meaning of $t$ is unclear in Eq. (22).\n\n- The definition of $\\hat{p}$ in Eq. (22) is required.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tvtiWBR0DN", "reviewer": "Reviewer_EA6f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths and Weaknesses\n\n## Strengths:\n- **Solid Background**: The paper provides a thorough background section, giving readers a solid foundation in related work and existing methodologies.\n- **Strong Theoretical Findings**: The theoretical analysis, including convergence proofs and optimality guarantees, adds credibility to the proposed approach.\n- **Well-Organized Experiment Design**: The experiments are well-structured, covering both synthetic and real-world datasets, and are supported by ablation studies to assess the impact of various model components.\n- **Vast Practical Considerations**: The authors include practical details such as initialization strategies, optimizer configurations, and warm-start training procedures, making the model implementation-friendly.\n- **Innovative Methodology**: The combination of change point detection with variational neural SDEs in a VAE framework is a novel approach that adds significant value to time-series modeling and change point analysis.\n- **Practical Relevance**: The model’s applications in finance, healthcare, and environmental monitoring underscore its practical importance.\n- **Clear Explanation of Training Steps**: The practical training steps provided help bridge the gap between theory and application.\n- **Comprehensive Ablation Studies**: The paper’s ablation studies help highlight the significance of different model components.\n\n## Weaknesses:\n- **Some Theoretical Statements Are Unclear**: Certain theoretical explanations, such as those related to the proofs, could be more detailed for clarity.\n- **Insufficient Number of Experiments on Real Data**: While experiments on synthetic data are strong, the number of real-world datasets used is limited, which raises questions about the model's generalizability.\n- **No Limitations Discussion**: The paper does not sufficiently address the potential limitations of the algorithm, such as scenarios where it may not perform well.\n- **Algorithm Scalability Issues**: The proposed method is not well-scalable when dealing with an increasing number of change points, potentially limiting its use for longer time-series with frequent changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission presents a novel approach to the change point detection problem using neural stochastic differential equations (SDEs) within a variational autoencoder (VAE) framework. The authors propose a new offline neural SDE change point detection algorithm and introduce a modified variational inference method with an updated evidence lower bound (ELBO). This ELBO incorporates a new prior that is based on a more common and relaxed assumption, making the variational inference simpler and more tractable compared to previous approaches. This innovation facilitates more efficient training and improves the model's ability to detect change points in an offline setting.\n\nThe iterative algorithm alternates between training the neural SDE model and updating change point estimates through either a maximum likelihood-based method or a sequential likelihood ratio test. The paper includes theoretical guarantees, demonstrating convergence to a stationary point for the training algorithm and proving optimal error probability for the change point detection scheme under specific conditions.\n\nAdditionally, use of particle filtering is employed for estimating marginal likelihoods and log-likelihood ratios, enhancing the accuracy of change point detection within the VAE framework. Also, a predictive log-likelihood term is incorporated into the ELBO to improve the generative performance of the model, enabling it to capture complex time-series dynamics more effectively.\n\nThe model's effectiveness is validated through comprehensive experiments on synthetic and real-world datasets, such as S&P500 prices and air quality measurements. The results are supported by ablation studies and various evaluation metrics, assessing the model’s performance in terms of marginal distribution similarity, data indistinguishability, and predictive quality.\n\nThe authors also provide practical considerations for model training, including initialization strategies, optimizer configurations, and warm-start techniques to enhance stability and reliability. \n\n\nAll these thorough theoretical conclusions, well-described background, and thorough practical considerations demonstrate high expertise of the authors and relative consistency of the provided results. The submission presents a substantial contribution to time-series modeling by combining innovative algorithmic design, simplified variational inference, and robust practical applicability. However, some manuscript aspects need to be addressed before the publication (see below).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Major Areas for Improvement:\n\n* **Handling Multiple Change Points**: While the paper mentions extending the algorithm to multiple change points, more detail on how this generalization is practically managed would strengthen the discussion on scalability. Now it seems each additional change point in a time series requires an extra SDE model with its own trainable parameters.\n\n* **Types of Anomalies Detected**: Clarifying which types of anomalies (e.g., mean shifts, trends, volatility, seasonality) the algorithm can detect would help readers better understand its capabilities and practical applications.\n\n* **Evaluation on More Datasets**: The use of four real datasets, while valuable, might be insufficient to assert the model’s generalizability. Expanding to more datasets or explaining the rationale behind the choice would strengthen this claim. Also, it is strongly recommended to use common change-point detection benchmarks (datasets corpuses + metrics), or explain the current choice of metrics.\n\n* **Source Code for Reproducibility**: Publishing the source code is recommended for enhancing reproducibility and enabling peer validation.\n\n* **Proof Sketch Enhancement**: The proof sketch on page 13 relies on a strong assumption regarding the impact of change point updates on variational approximation accuracy. Including a note that \"The detailed proof can be found in the Appendix\" would add more credibility to the \"Under the assumption that change point updates do not vastly impact the accuracy of the variational approximation\" statement.\n\n* **Explanation of Table Metrics**: Metric names in Table 1 should be explained (Prediction=Synthetic Data Predictive Quality etc.), or have a caption to clarify their meanings. Adding abbreviations or justifying the chosen metrics over other standard benchmarks would improve clarity.\n\n* **Other clarifications required**: \n1. In 4.3.2, how the expected log-likelihood differs from (13)? How correct that substitution is? Does this means you decompose the standard log-likelihood L by two terms: \\alpha*L+(1-\\alpha)*L_pred in order to force the predictive capabilities of the autoecnoder\n1. In (22), how to assess p(xt1:k|ν > t) ?\n\n# Minor Areas for Improvement:\n\n* **Discussion**: In the discussion section, the limitations could be expanded to discuss challenges related to scalability, robustness to noise, and highly non-linear data structures. Addressing potential drawbacks gives a balanced view of the model’s applicability. Also, some potential improvements with further research plan can be discussed in the section. Particularly, potential encoder's LSTM substitution with smth more recent and efficient can be discussed here.\n\n* **Not all formulas are enumerated**: ... while some of them should. For instance, in the \"Expected Predictive Log-Likelihood\" subsection\n\n* **Algorithm Description and Pseudocode**: In Algorithm figures, the current pseudocode format is quite suspicious. For instance, the steps in Algorithm 1, particularly the \"Update change point\" phase, need more detailed non-textual pseudocode. Also, in Algorithm 2&3 the \"PF\" stage should be revealed\n\n\n* **X-Scale Consistency in Figures**: In Figure 6, aligning the X-scale across subplots (especially the Log-Evidence Ratio plot) would aid in direct comparison.\n\n\n# Additional comment\n\nBasically, the change-point updates iteration looks like an M-step of EM-algorithm. On the other hand, the ELBO estimate and maximization can be considered as E- and M- steps respectively. With this regard, can be embed the change-point updates iteration into the VAE training pipeline to make it end-to-end? You can add this thought to the Discussion section too as a part of further research plan", "broader_impact_concerns": "The proposed work on variational neural SDEs for change point detection holds significant promise for advancing the fields of time-series analysis and predictive modeling, with strong potential applications in diverse areas such as finance, healthcare, and environmental monitoring. This contribution is commendable for its focus on detecting distributional shifts, which can enhance decision-making processes, improve system resilience, and foster proactive responses in dynamic environments.\n\nOne notable positive aspect is the potential for this technology to empower organizations to identify critical changes in data patterns more effectively, leading to better risk management and adaptive strategies. In finance, this could mean earlier detection of market shifts, while in healthcare, it could enable timely identification of significant changes in patient data trends, contributing to improved outcomes and patient care.\n\nAdditionally, the proposed algorithm could democratize access to sophisticated change point detection methods, enabling a wider range of industries to leverage advanced tools that were previously available only to those with substantial technical resources. This is a meaningful step toward making robust analytical capabilities more broadly accessible, which aligns with the overall push for equity in technological advancements.\n\nThe authors’ inclusion of practical considerations, such as initialization methods and warm-start strategies, reflects a strong awareness of real-world implementation challenges. This focus on practical application enhances the usability of the model and paves the way for seamless integration into existing analytical workflows, benefiting both research and industry practitioners.\n\nWhile there are potential concerns, such as data privacy and fairness, the positive impact of this work can be amplified by proactive measures. By embedding best practices in transparency and responsible data use, this model could set a benchmark for ethical AI deployment. The authors should consider adding a **Broader Impact Statement** to highlight how this technology could be responsibly utilized, ensuring it supports fair, transparent, and equitable applications across various domains.\n\nSecurity and resilience to adversarial scenarios are also areas where this model could shine. Developing robust training practices and emphasizing safeguards would further underscore the reliability and trustworthiness of the technology.\n\nOverall, the manuscript presents a forward-thinking approach that could greatly benefit fields reliant on time-series data. By addressing broader societal and ethical considerations, the authors can reinforce the model's positive contributions and help guide its responsible adoption. A **Broader Impact Statement** would be a valuable addition, showcasing the authors' commitment to maximizing the positive societal impact of their work while acknowledging and mitigating potential risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is based on a principled framework and is accessible. The reviewers all favor publication. 

The authors have largely taken the comments of the reviewers into account, with some exceptions that I find important to include in the final version (see comments below). I also have the following comments that I would like the authors to address before publication:

- Please clarify what distribution p(zt) in Eq (11) is. 
- What is the approximation that is done from Eq (10) to (11)?  Please add an explanation to the paper.
- From the notation, I guess the samples in Eq (12) are the samples from the variational posterior? Please clarify in the paper. (If is the variational posterior, using q_phi(z_t|xobs)) as before Eq (8) would be more suitable notation than the p(zt) in (11)
- Sec 4.3.2, first paragraph. Note the typo "is the it will" in the sentence ", a weakness of the training loss for the model parameters of the SDEVAE model is the it will also lead to an underestimation of noise variance"

- Results: You point out that LS4 is the "closest rival". Please add a discussion about the pros and cons of LS4 and your method, and when to use which. Please note that Reviewer 8zqg asked a similar question, but I couldn't spot a discussion in the paper. Please add a sufficiently detailed one, or let me know where you have included it if done already.

- I cannot see the "broader impact statement" requested by Reviewer EA6f. Can you please add a brief paragraph?
Decision: Accept (poster)