Continuous Latent Process Flows
Authors: Ruizhi Deng
Marcus A Brubaker
Greg Mori
Andreas Lehrmann
Venue: NeurIPS 2021 Poster (2021)
Keywords: continuous dynamics
time series
stochastic differential equation
normalizing flow

Abstract: Partial observations of continuous time-series dynamics at arbitrary time stamps exist in many disciplines. Fitting this type of data using statistical models with continuous dynamics is not only promising at an intuitive level but also has practical benefits, including the ability to generate continuous trajectories and to perform inference on previously unseen time stamps. Despite exciting progress in this area, the existing models still face challenges in terms of their representational power and the quality of their variational approximations. We tackle these challenges with continuous latent process flows (CLPF), a principled architecture decoding continuous latent processes into continuous observable processes using a time-dependent normalizing flow driven by a stochastic differential equation. To optimize our model using maximum likelihood, we propose a novel piecewise construction of a variational posterior process and derive the corresponding variational lower bound using trajectory re-weighting. Our ablation studies demonstrate the effectiveness of our contributions in various inference tasks on irregular time grids. Comparisons to state-of-the-art baselines show our model's favourable performance on both synthetic and real-world time-series data.

Continuous Latent Process Flows
Ruizhi Deng!.2 Marcus A. Brubaker ,3,4
Greg Mori
Andreas M. Lehrmann' 4 VVector Institute
Borealli AI
Simon Fraser University
"Yok University
Abstract
Partial observations of continuous time-series dynamics at arbitrary time stamps exist in many disciplines. Fitting this type of data using statistical models with con- tinuous dynamics not only promising at an intuitive level but also has practical benefits, including the ability to generate continuous trajectories and to perform inference on previously unseen time stamps. Despite exciting progress in this area, the existing models still face challenges in terms of their representation power and the quality of their variational approximations, We tackle these challenges with continuous latent process flows (CLPF), principled architecture decoding contin- uous latent processes into continuous observable processes using time-dependent normalizing flow driven by stochastic differential equation. To optimize our model using maximum likelihood, we propose a novel piecewise construction a variational posterior process and derive the corresponding variational lower bound using importance weighting of trajectories. An ablation study demonstrates the effectiveness of our contributions and comparisons to state-of- offthhe-rr art basclines show our model's ' favourable performance on both synthetic and real-world data.
Introduction
Sparse and irregular observations of continuous dynamics are common many areas of science, including finance [15] 36), healthcare [16), and physics [30]. Time-series models driven by stochastic differential equations (SDEs) provide an elegant framework for this challenging scenario and have recently gained popularity in the machine learning community [1] 24). The SDEs typically implemented by neural networks with trainable parameters and the latent processes defined by the SDEs then decoded into observable space with complex structure. Due to the lack of closed- form transition densities for most SDEs, dedicated variational approximations have been developed to maximize the observational log-likelihoods 22 [8 24
Despite the progress existing works, challenges still remain for SDE-based models to applied to irregular time-series data. One major challenge is the model's representation power. The continuous- time flow process (CTFP; (TI) uses series of invertible mappings continuously indexed by time to transform simple Wiener process to more complex stochastic process. The use of a simple latent process and invertible transformations permits CTFP models to evaluate the exact likelihood of observations any time grid efficiently, but they also limit the set of stochastic processes that CTFP can express to some specific form which can be obtained using Ito's Lemma and excludes many commonly seen stochastic processes. Another constraint of representation power in practice the Lipschitz property of transformations in the models. The latent SDE model proposed by Hasan et al. 18 and CTFP both transform latent stochastic process with constant variance an observable one using injective mappings. Due to the Lipschitz property existing in many invertible neural network architectures, some processes that be written non-Lipschitz transformation of simple process, like geometric Brownian motion, cannot be expressed by these models unless specific choices of non-Lipschitz decoders are used. Apart from the model's representation power, variational inference is another challenge in training SDE-based models. The latent SDE model in the work of Li et al. (24)
"Tis work was done during an internship at Borealis AI. Correspondance to wsdmdeng @ gmail.com.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

abservable
process
latert SDE
Ema-dapendent decoding
time
procoss
time
1/2
Figure 1: Overview. Our architecture uses stochastic differential equation (SDE; left) to drive time-dependent normalizing flow (NE; right). At time 11, 12 (grey bars), the values of the SDE trajectories (colored trajectories on the left) serve conditioning information the right) into a complex observable process (colored trajectories on the right). The color gradient on the right shows the individual trajectories of this transformation, which is driven by an augmented neural ODE, Since all mon--coninuuus,
the decoding a simple base process (grey trajectories on
stochastic processes and mappings continuous process, enabling modelling continuous dynamics and inference on irregular time grids.
model observed data as partial realizations of
uses principled method of variational approximation based on importance weighting of trajectories between a variational posterior and prior process. The variational posterior process constructed using single SDE conditioned on all observations and is therefore restricted to be Markov process. This approach may lack the flexibility to approximate the true posterior process well enough in complex inference tasks, e.g., in an online setting or with variable-length observation sequences.
In this work we propose Continuous Latent Process Flows (CLPFs model that is governed by latent dynamics defined by an expressive generic stochastic differential equation. Inspired by [I], then use dynamic normalizing flows decode each latent trajectory into continuous observable process. Driven by different trajectories of the latent stochastic process continuously evolving with time, the dynamic normalizing flow can map a simple base process diverse class of observable processes. We illustrate this process in Fig. This decoding is critical for the model to generate continuous trajectories and be trained fit observations on irregular time grids using a variational approximation. Good variational approximations and proper handling of complex inference tasks like online inference depend on a flexible variational posterior process. Therefore, we also propose principled method of defining and sampling from non-Markor variational posterior process that is based on piecewise evaluation SDEs and can adapt to new observations. The proposed model excels at fitting observations on irregular time grids, generalizing to observations on more dense time grids, and generating trajectories continuous time.
Contributions. In summary, we make the following contributions: (1) We propose now-based decoding of a generic SDE as principled framework for continuous dynamics modeling of irregular time-series data. (2) We improve the variational approximation of the observational likelihood through a flexible non-Markov posterior process based on a piecewise evaluation the underlying SDE; (3) We validate the effectiveness of our contributions in a series of ablation studies and comparisons to state-of-the- time-series models, both on synthetic and real-world datasets.
Preliminaries
2.1 Stochastic Differential Equations
SDEs can be viewed as a stochastic analogue of ordinary differential equations (ODEs) the sense that dz, di p(Z,, 1) + random noise a aZZ,,,) 1). Let Z, a variable which continuously evolves with time. An m-dimensional SDE describing the stochastic dynamics of Z, usually takes the form
dZ. / Z,, + a Ze, dW,,
where maps to an m-dimensional vector, or is an x k matrix, and W; is k-dimensional Wiener process. The solution of an SDE is continuous- time stochastic process Z, that satisfies the integral equation Z, = Zo + So μ Z., 8) ds + J o ZZ,, dW, with initial condition Zo, where
available http:///ttaa.oo//eaaaaaaaaaaatto paratennt--poosssffooos

---

the stochastic integral should be interpreted as a traditional Itō integral [27 Chapter 3. For each W. the stochastic process Z, maps w different trajectory (w).
sample trajectory
Latent Dynamics and Variational Bound. SDEs have been used as models of latent dynamics in variety of contexts 18 As closed-form fiiite-dimensioall solutions SDEs are rare, variational approximations are often used in practice. Li et al. [24] propose a principled way of re-weighting latent SDE trajectories for variational approximations using Girsanov's 's theorem [27 Chapter 8.6]. Specifically, consider a prior process and a variational posterior process in the interval [0. 7] defined by two stochastic differential equations dZ, = fa Zz. L) + (Z, () dW, and t) dt + (Z,,t) t) dW respectively. Furthermore, let p( Ze) denote the probability of observing a conditioned on the trajectory of the latent process Z, in the interval [o, T). If there
d2, = μ2l t))
exists mapping u. RT [0. T] + R$ such that o azz (z, () μz2 () f4 (z, () and u
satisfies Novikov's condition 27 Chapter 8.6], we obtain the variational lower bound
log p(x) = log Blp((1 a Z,)l = log E(p((1) 2,) MT] Z Eloogp(<| Ž.) + log MT]:
(2)
where Mr = exp( 10= 2/2 u (() dt - --" u (Z, t)7 dW.). See 24 for formal proof.
2.2 Normalizing Flows
employ bijective mapping f R + Rd to transform a random variable Y with simple base distribution py to a random variable X with a complex target distribution px. We can sample from a normalizing flow by first sampling ~ Py and then transforming it to æ = f(y). As a result of invertibility, normalizing flows can also be used for density estimation. Using the change-of variaalls formula, we have log Px (z)) 0g Oar
Normalizing flows 3 &
13 21 22 23 28 29 31
log PY (g(x)) + log det
where g is the inverse of f-
More recently, normalizing flows have been augmented with a continuous For instance, the continuous-time flow process (CTFP; (II) models irregular observations of continuous-time stochastic process. Specifically, CTFP transforms a simple d- dimensional Wiener process W, to another continuous stochastic process X, using the transformation X, = f(W,.t), t). where f(w,t) t) is an invertible mapping for each t. Despite its benefits of exact dinenssonni distributions, the expressive power of
Continuous Indexing.
index 6 10
logliikelhood computation of arbitrary CTFP to model stochastic processes is limited in the following two aspects: (1) An application of Itô's lemma
27 Chapter 4.2] shows that CTFP can only represent stochastic processes of the form
af di (W.,1) () + Tr.H., ()) di + (W,, dw,
dS(W,,
where f is the Hessian matrix of f with respect to w and Vw.f is the derivative. A variety of stochastic processes, from simple processes like the SaateinnUhhenbeckk (OU) process to more complex non-Markov processes, fall outside of this limited class and cannot be learned using CTFP (see Appendix A for formal proofs); (2) Many normalizing flow architectures are compositions of It is therefore challenging to model certain stochastic processes that are non-Lipschitz transformations of simple processes using CTFP without prior knowledge about the functional form of the observable processes and custom-tailored normalizing flows with non-Lipschitz transformations (see Appendix B for an example).
Lipschitz-comtinnous transformations 7 8 17..
latent variant of CTFP is further augmented with a static latent variable to introduce non-Mart -anMarkkvvinn behavior. It models continuous stochastic processes as X₁ f(W, t: Z), where Z is a latent variable with standard Gaussian distribution and f 2) is CTFP model that decodes each sample of Z into stochastic process with continuous trajectories. Latent CTFP can be used to estimate nimen-dimensional distributions using a variational approximation. However, is unclear how much a latent variable Z with finite dimensions can improve CTFPs representation power.
3 Model
Equipped with these tools, we can now describe our problem setting and model architecture. Let { t)) denote sequence of d-dimensional observations sampled at arbitrary points in time,
3

---

where Tt, and ts denote the value and time stamp of the observation, respectively. The observations are assumed to be partial realizations of a continuous- stochastic process X,, Our training objective is the maximization of the observational log-likelihood induced by X, on given time grid,
C log PX., X., ( Lt,))
(4)
for an inhomogeneous collection of sequences with varying lengths and time stamps. At test-time, in addition to the maximization of log-liielihoods, we are also interested in sampling sparse, dense, or irregular trajectories with finit-dimenniionll distributions that conform with these log-likelihoods. We model this challenging scenario with Continuous Latent Process Flows (CLPF), In Section 3.1 we present our model in more detail. Training and inference methods will be discussed in Section 3.2
3.1 Continuous Latent Process Flows
A Continuous Latent Process Flow consists of two major components: an SDE describing the continu- ous latent dynamics of an observable stochastic process and a continuously indexed normalizing flow serving as time-dependent decoder. The architecture of the normalizing flow itself can be specified in multiple ways, e.g., as an augmented neural ODE [14] or as a series of affine transformations [10]
The following paragraphs discuss the relationship between these components in more detail.
Continuous Latent Dynamics. Analogous to our overview in Section 2.1 we model the evolution of an m-dimensional time-continuous latent state Z, in the time interval [o. 7 using flexible stochastic differential equation driven by an m-dimmensiona Wiener Process W..
(5) where y denotes the (shared) learnable parameters of the drift function μa and variance function a. In experiments, we implement μ and using deep neural networks (see Appendix E for details). Importantly, the latent state Z, cxists for each E [0, I] and can be sampled on any given time grid, which can be irregular and different for each sequence.
dz, He, (Z. t) dt + o. (Z, dW.
Time- DepenDeepnndnt Decoding. Latent variable models decode a latent state into an observable variable with complex distribution. As an observed sequence {( (1)) ( realization of a continuous-tine stochastic process, continuous trajectories of the latent process Z, should be decoded into continuous trajectories of the observable process X,, and not discrete
is assumed to be partial
distributions. Following recent advances in dynamic normalizing flows [0 [1], we model X, as
X Fo; Z,,t). t),
(6)
where O₂ is a d-dimensional stochastic process with closed-form transition density and Fo( ze, t) is a normalizing flow parameterized by 0 for any z,, t. The transformation Fp decodes each sample path of Z, into a complex distribution over continuous trajectories X, Fg a continuous mapping and the sampled trajectories of the base process O are continuous with respect to time 1. Unlike [II. who use Wiener process as base process, we use the stationary marginal distribution and bounded variance. As result, the variance of the observation process does not increase due to the increase of variance in the base process and is primarily determined by the latent process Z, and flow transformation Fe.
(OU) process, which has
Flow Architecture. The continuously indexed normalizing flow Fo(.; 21, 1) can implemented in multiple ways. Deng et al. [I]] use ANODE [14, defined as the solution to the initial value problem
h2 (((T) (a(T)
(o(h(().a(T))))) (o(a(T),T)
O: hh (n(TT) = (z) (a(T))
dr
where T E [To. TII. h(T) E Rd, (()) E 1=+ fo Rd x Rm+1 x [TO, TI] + Rdd go 1=++1 x [to, [1] + R, and Fe is detined as the solution of h(()) at = TI. Note the difference between and T: while t E jo, T] describes the continuous process dynamics, 7 E [To, 7I] describes the continuous time- denendependent decoding each time step t. Alternatively, Cornish al. [10] propose variant of continuously indexed normalizing flows based on a series of N affine transformations ft.
h ((+1) = f.(h)) (0) =..t) t) k(h (1) exp( sol--u'e 0 t)) o. (zz, t)).
(8)
Mooe precisely, the conditional distribution po., 1o,, (or, (o,)) must exist in closed form for any t, <

---

Z
(a) Generation
(b) Inference
Figure 2: Graphical Model Generation and Inference. Red circles represent latent variables Zi Unfilled blue circles represent samples from the OU process discrete time points O,,, Blue diamonds in Fig. 22 indicate each X1, is the result of deterministic mapping of Zi, and O... Filled blue circles in Fig. represenn observed variables X1,-
where h2 o,, h (M) 24. (+)) and 2 are unconstrained transformations, and le is an invertible mapping like residual flow [8. The temporal relationships among Z₁, O₁, and X, from a graphical model point of view are shown in Fig. 2a
3.2 Training and Inference
With the model fully specified, we can now focus our attention on training and inference. Computing the observational log-likelihood (Eq. induced by time-dependent decoding of an SDE (Eg. 6 on an arbitrary time grid is challenging, because only few SDEs have flosed-frrm transition densities. Consequently, variational approximations are needed for flexible SDEs such 5 We propose principled way of approximating the observational fog-likelihood with a variational lower bound based on novel piecewise construction of the posterior distribution of the latent process. In summary, first express the observational log-likelihood as a marginalization over piecewise factors conditioned on latent trajectory, then approximate this intractable computation with a piecewise variational posterior process, and finally derive a variational lower bound for it.
Observational Lige-iiel The observational log- likelhhod can written as an expectation over latent trajectories of the conditional likelihood, which can be evaluated in closed form.
L log Px,, Xtn (x), .......,))). Tt,))
log E Ew
px,, X.,, 1z (2t1....... (I.,,(((([[
log E W.
I Ix,, 74 (It, [t,-, Zz,(w), Zz.,
where (w) denotes a sample trajectory of Z, driven by a ~ W,. For simplicity, we assume w.l.o.g. and in this section only Zo, Xo be given. As result of invertibility, the conditional likelihood terms Px.,IX.... Z.,,.... can be computed using the change-of-variide formula,
logpx., [X._ 2,,1 (0t, [2:; Ze,(w), Ze, (w))
= llog (on, log det (Fo Z, (w), 4:) where o1;
(10)
do.
F (2 Zt, tw),
Piecewise Construction of Variational Posterior. Directly computing the observational log- likelihood intractable and we use a variational approximation during both training and inference. Good variational approximations rely on variational posteriors that are close enough to the true posterior of the latent trajectory conditioned on observations. Previous methods [2]] use a single SDE to propose the variational posterior conditioned on all observations. Instead, we develop more flexible method and naturally adapts to different time grids. Our posterior process is not constrained by the Markov property of SDE solutions. Moreover, the proposed method serves as the basis for principled approach to online inference tasks using variational posterior processes.
can update the posterior process parameters when new observation seen

---

Our construction makes use of a further decomposition of the observational log-likeihood based on the following facts: (Ws++ 13o is also a Wiener process Vs > 0 and the solution to Eq 5 is
Fi=1 be series of probability spaces on which independent m-dimensiooal Wiener processes W (f) are defined. We can sample complete trajectory of the Wiener process W, in the interval [o, I by sampling independent trajectories
Markov process. Specifically, let {(2()) Fl ( P
)())
(6) of length 4-1 from ( and adding them, i.e., We [id.<0 Wi,-t-1 + w 1' where max t < t) + 1. As result, we can solve the latent stochastic differential equations in piecewise manner. Samples of Zı: can be obtained by solving the stochastic differential equation
(*")
d2, 1 (Ze, + 4-1 di + a, (Z,: 1 11-1) dw,"
(11)
in the interval [0, 1-1.. with Z as the initial condition. The log- -igelike of the observations can thus be rewritten as
II pl (2t.. R1, Re, (((())
C = log E (
w!!))
= log E
w
plar,
Deos Rt, Zko wll))
(12)
w( pl [r, 20p 24, (1) R By (1 w((- [...............
worth noting that the value of Tt, is determined by 2t, and Oi, With the distribution of O depending on the value of o and thus on _ and Tt,- after marginalizing over o,, each conditional likelihood term in Eq. 12 conditioned on Tt;- 2t,, and -, For simplicity we also
omit conditioning of each term on Wiener process trajectories up to 1 ie.. wn))
In
preparation of variational approximation can now introduce one posterior SDE
dž, = Ms, (Ze, t 1 dt + o., (Z + t--1) dw
(13)
for each expectation w
p( Di, 1 21 zt: (.0) E w0th Buth
in Eq.
12
Variational Lower Bound with Piecewise Importance Weighting. form the basis for a variational approximation of the expectations in Eq.(12) Specifically, sampling I from the posterior SDE, the expectation can be rewritten as
The posterior SDEs in Eq 13
E W
(( Iot... E,,, Zt. )
M (4) (w)()) B
I...
(14)
where M = exp(- fo 1 Hu 2,, 8) ds for p u 2., 8)² dw serves as importance
weight for the sampled trajectory between the prior latent SDE (Eq. (T) and posterior latent SDE 4-11 (==))(zz()) 8) = Pe. (2.8 4-1 (2, 8 + 14-1). We can define such a posterior latent SDE for each interval. As result, a sample b,, can be obtained by solving the posterior SDEs defined on the intervals up to t; given the initial value E,, and sample (+),
(Eq. 13), ), with u satisfying o,
paths of Wiener processes up to 4, i.e., (1) (2) (+)) After applying the importance weight to each interval and Jensen's inequality, we can derive the following evidence lower bound (ELBO) of the ligelioo::hood:
w) (plze, wiis ãe, Eio (()) M ...E wl [p(x,, (o Ee En ( (1) ...]]
C = log E
n
= log E
wIl) w x
p(ae, [x Ei,, R (1) ) I
(0) (M
(w)(o)
E w(l) ((t)),,...,.))))W)))) x (
log (xx D 21, بيي (()) + Σ log (w(())
(15)
The bound above can be further extended into a tighter bound in IWAE [5] form by drawing multiple independent samples from each W(()) The variational parameter di the output of an encoder RNN that takes as inputs the sequence of observations up to t,, {xt,.... ...., Ft. J. and the sequence

---

of previously sampled latent states, {zi,..... = The parameter óz is thus updated based on both the latest observation and past history. As result, the variational posterior distributions of the latent states Zi are no longer constrained to be Markov and the parameterization of the variational posterior can flexibly adapt to different time grids. The dependency structure between latent and observed variables during inference is depicted in Fig. 2b
4
Related Work
The introduction of neural ODEs 7 unified modeling approaches based on differential equations and modern machine learning. Explorations of time-series modeling leveraging this class of approach have been conducted, which we review here.
The latent ODE model equations. As a result, the entire latent trajectory is solely determined by its initial value. Even though latent ODE models have continuous latent trajectories, the latent state is decoded into observations at each time step independently. Neural controlled differential equations (CDEs) 2 and rough differential equations (RDEs) [25] propagate hidden state across time continuously using controlled differential equations driven by functions of time interpolated from observations on irregular time grids. Neural ODE Processes (NDPs) [26] construct a distribution over neural ODEs order to effectively manage uncertainty over the underlying process that generates the data.
32 propagates a latent state across time using ordinary differential
Among existing time-series works with continuous dynamics, the latent SDE model 24 is most similar to ours. The SDE model includes an adjoint sensitivity method for training SDEs; the derivation of the variational lower bound in our proposed model is based the same principle of trajectory importance weighting between two stochastic differential equations. The posterior process there is defined as a global stochastic differential equation. Our model further exploits the given observation time grid of each sequence to induce a piecewise posterior process with richer structure.
The continuous-time flow process I1 (CTFP) models irregular time-series data as incomplete realization of continuous-tin stochastic processes obtained by applying normalizing flows to the Wiener process. We have discussed the limits of CTFP in Sec. [.22 Because CTFP is generative model that is guaranteed to generate continuous trajectories, we use it as the decoder of latent process for better inductive bias in modeling continuous dynamics. Apart from CTFP, there also works outside the deep learning literature that apply invertible transformations to stochastic processes 33 34) Warped Gaussian Processes 3 transform Gaussian processes to non-Gaussia
processes in observation space using monotonic functions. Copula Processes [34] extend the concept of copulas from multivariate random variables stochastic processes. They transform a stochastic process via a series of marginal cumulative distribution functions to obtain another process while
preserving the underlying dependency structure.
Alternative training frameworks have also been explored for neural SDEs [18 19]. Stochastic
differential equations can be learned latent dynamics with variational approximation (18). Connections between generative adversarial network (GAN) objectives and neural SDEs have been drawn 19.. Brownian motion inputs be mapped time-series outputs. Alongside CDE-based discriminator GAN-based training can be conducted to obtain continuous time generative models.
Experiments
We compare our proposed architecture against several baseline models with continuous dynamics that can be used to fit irregular time-series data, including CTFP, latent CTFP, latent SDE, and latent ODE. We also run experiments with Variational RNN (VRNN) [9, a model that can be used to fit sequential data. However, VRNN does not define continuous dynamical model and cannot generate trajectories with finite-dimensional distributions that are consistent with its log- likelioid estimation. All implementation and training details can be found in the supplementary material.
5.1 Synthetic Data
We evaluate our model on synthetic data sampled from known stochastic processes to verify its ability to capture variety of continuous dynamics. We use the following processes:
7

---

Table 1: Quantitative Evaluation (Synthetic Data). We show test negative log-likelihoods (NLLs) four synthetic stochastic processes across different models. Below each process, we indicate the intensity of the Poisson process from which the time stamps for the test sequences were sampled for testing. [GBM: geometric
Brownian motion (ground truth NLLs:
2, À 20] [0.388, -0. 788); LSDE: linear SDE; CAR:
continuous auto-regressiv process; SLC: stochastic Lorenz curve]
Model
GBM
LSDE
CAR
SLC
À 2
A = 20
À == 2
À = 20
A=2 2
À = 20
À = 20
À = 40
VRNN
0.425
-0650
-0.634
-1.665
1.832
2.675
2.237
1.753
Latent ODE 52
1.966
1.7%
0,900
0.847
4,872
4.765
9.117
9.115
2.940
0.678
00.71
-1.778
383.593
0.489
Latent CTFP
1
1.422
-0.158
-0.468
-1.784
249.839
43.007
1.419
-0.077
SDE
1.243
1.778
0.082
3.594
3.603
7.740
8.256
CLPF (ours)
0.444
-0.698
-0881
-1.939
1.322
-0.077
-2,620
-3,963
Geometric Brownian Motion d.X, ,xX,, dt + ox, dW, Even though geometric Brownian motion can theoretically be captured by the CTFP model, it would require the normalizing flow to be non-Lipschitz; there is no such constraint for the proposed CLPF model.
Linear SDE dX,, = (a(l)x; + b(t)) di + (1) dW, The drift term of the SDE is a linear transformation of X, and the variance term is deterministic function of time An application of Itô's lemma shows that the solution is stochastic process that cannot be captured by CTFP.
Continuous AR(4) Process. This process tests model's ability to capture non-Markov processes (see Appendix C for implementation details):
0
X, = [d, 0,O. dY, AY, dt + dw,,'
0
0
1
0
where A =
(16)
0
0
a1
02
d3
04_
Stochastic Lorenz Curve. We evaluate our model's performance on multi multi-diimnnsioall (3D) data:
dX,- = o(Y, - -X,) dt + a dW,,
dY,- (X,(p- Z,) Y)) dt By dW,,
(17)
dZ: = (X,Y, -B,,) di + a. dW,
In all cases, we sample the observation time stamps from a homogeneous Poisson process with intensity X. To demonstrate our model's 's ability to generalize to different time grids, we evaluate it using different intensities A. An approximate numerical solution to the SDEs obtained using the Euler-Maruyama scheme for the Itô integral.
Our results on synthetic data are displayed in Table We train and evaluate all models on observations of sample trajectories in the interval [0, 30] with observation time stamps sampled from Poisson process with intensity \ 2, except for the stochastic Lorenz curve which sampled in the interval [0, 2). The results demonstrate the proposed CLPF model's ' favourable performance across the board. In particular, our model outperforms both CTFP variants on all four synthetic datasets. We attribute this competitive edge to the expressive power of a generic SDE over a static latent variable. The CTFP models also perform relatively poorly on the Continuous AR(4) process, which is mon-Markov. As the Continuous AR(4) process has underlying 4-dim. stochastic process Y, and is generated by projecting Y, to a I-dim. observation space X,, models driven by a high-dimensional latent process like latent SDE and CLPF show better performance in this case.
To evaluate our model's performance in capturing continuous dynamics, we increase the (average) density of observations and generate observation time stamps from a Poisson process with intensity = 20 (A- 40 for SLC). The results show that models that can generate continuous trajectories, including CTFP, latent CTFP and CLPF, generalize better to dense observations than the other models. In the first row of Fig. 3 we visualize sample sequences from CLPF and VRNN models trained on LSDE data and compare them with samples from an LSDE ground truth process. We run both CLPF and VRNN on a time grid between 0 and 30, with gap of 0.01 between steps. Samples from the CLPF models share important visual properties with samples from the ground truth process. On the other hand, VRNN fails to generate sequences visually similar to the ground truth, despite its competitive density estimation results sparse time grids.
8

---

Table 2: Likelihood Estimation (Real-World Data), We show test negative log-likelihoods (NLLs; lower is better). For CTFP, the reported values are exact; for the other models, report IWAE bounds using K 125 samples. CLPF-ANODE stands for a CLPF model implemented with augmented neural ODEs. CLPF-iRes stands for a CLPF model implemented with indexed residual flows.
Table 3: Ablation Study (Synthetic Data), We show test negative log-likelihoods across different variants of the proposed model. We report IWAE bounds using K = 125 samples with observation time stamps sam- pled from Poisson point process with
(CLPF. Global: single global posterior SDE in latent SDE style (24]; CI.PF-Independent: independent decoder
instead of CTFP-decnder; CI.PF-Winne: Wiener hase
Model
Mujoco [2]
BAQD 35
PTBDB [4
process instead of Or-pceees]
VRNN 99
-15.876
-1.204
.2.035
Model
GBM
LSDE
CAR
SLC
Latent ODE
32
23.551
2.540
-0.533
CLPP-Global
0.447
-0.821
1.552
-3.304
Latent SDF
3,071
1,512
CLPPF- I.ne-endeenn
0.800
-0.326
4.970
7.924
CTFP I
-7.598
-0.170
-1.281
CLPP Wiener
0.390
-0.790
1.041
-1.885
CTFP
-12.693
Latent SDE
1.243
0.082
3.594
7.740
CL-F-ANODE (ours)
-14.694
-0.619
-1.575
CLPF
0.444
-0.831
1.522
-2.620
CLPF-iRes (ours)
-10.873
-0.486
Table 4: Sequential Prediction (Real-World Data). We report the average L2 distance between prediction results and ground truth observations in a sequential prediction setting. The prediction is based on the average of 125 samples. Results are reported in the format mean, [25th percentile, 75th percentile]
Model
Mujoco 3
BAQD 35
PTBDB
VRNN
0
1,599, [0.196. 1,221]
0.519, [0. 168 0.681]
0.037, [0.005, 0.032]
Latent ODE
13,959, [9.857, 15.673]
1.416, [0.936. 1.731]
0.224. [0.114, 0.322]
Latent SDE
PA
7.627. [2.384, 8.381]
0.848, [0.454, 1,042]
0.092, [0.032, 0.111]
CTFP
1,969, [0.173. 1.826]
0.694, [0.202. 0.966)
(0.006. 0.046]
Latent CTFP
|||
1,983, [0.167. 1.744]
0.680, [0.18% 0.943]
0.065, [0.007, 0.059]
CLPF-ANODE (ours)
1,629, [0.149, 1.575]
0.542. [0. 150, 0.726]
0.048, [0.005, 0.041]
C1 IP-iiisss (ours)
18846, [0.177. 1.685]
0.582, [0, 0.805]
0,055, (0.006, 0.049]
5.2 Real-world Data
Many real-world systems, despite having continuous dynamics, are recorded via observations at a fixed sampling rate. Diagnostic Database (PTBDB) [6] are examples of such datasets. We create training/testing data at irregular times by drawing time stamps from a Poisson process and mapping them to the nearest observed sample points (see Appendix D for details).
[32], Beijing Air-Quality Dataset (BAQD) (35) and PTB
We demonstrate our model's 's ability to fit this type of observation as a surrogate for its ability to capture real-world continuous dynamics. Additionally, we also evaluate CLPF on a sequential prediction task to illustrate its benefits in an online inference setting. The results for these experiments are reported in Table 2 and Table 4 In the likelihood estimation task our model achieves competitive performance with respect to other methods that generate continuous dynamics. We do observe low negative log- ligleeiissoods (NLLs) by VRNN on these data, though note that VRNN does not generate true continuous dynamics. In the sequential prediction task CLPF outperforms the other baselines with continuous dynamics; the performance gap between CLPF and VRNN significantly reduced compared to likelihood estimation.
In the second row of Fig. 3] also visualize one dimension of sample sequences generated from CLPF and VRNN models trained on a dense time grid of BAQD data (gap between steps: 0.01) and compare with ground truth samples from the dataset. We can see that VRNN fails to generate trajectories visually similar to samples from the ground truth data on a dense time grid, despite its favourable NLL on a sparse and irregular time grid. The sampled trajectories tend to converge and show little variance after running VRNN for a large number of iterations (1500 for = 1.5). We also observe that VRNN fails to generate trajectories with consistent visual properties when we sample observations on different time grids (see supplementary material for more qualitative samples).
5.3 Ablation Study
We compare the proposed CLPF models against three variants that are obtained by making changes to key components of CLPF. The results with these models are shown in Table CLPF-Global is a variant created by replacing our piecewise construction with global posterior process similar to latent SDE (24) CLPF-Independent is variant where the continuously indexed normalizing flows
9

---

-10
(a) CLPF
(b) VRNN
(c) Ground Truth
Figure 3: Qualitative Samples on Dense Time Grid CLPF and VRNN Models. We generate the sample sequences from CLPF and VRNN trained with LSDE (first row) and BAQD (second row) data and compare the samples generated from the model to samples from the ground truth process (LSDE) training set (BAQD).
and CTFP- CTFP-decoding are replaced with static distributions and independent decoders. CLPF-Wiener a variant that uses a Wiener process as the base process for the CTFP decoder.
CLPF-Global differs from CLPF in manner similar to the difference between latent SDE and JLPF-Independent. In both comparisons we can see modest improvements of performance when replace the posterior process defined by single SDE with a piecewise constructed posterior. Both the comparisons between CLPF and CLPF-Independent and between CLPF-Global and latent SDE show that CTFP-style decoding into continuous trajectories is indeed better inductive bias when modeling data with continuous dynamics. Although the comparison with CLPF-Wien does not show clear trend, CLPF still delivers overall better performance the synthetic datasets. Therefore we use the OU process as our base process in all CLPF experiments.
Discussion and Conclusion
Limits. While being expressive than the CTFP model, the proposed CLPF model is not capable of exact likelihood evaluation and relies on variational approximations for estimation and optimization. differential equations that need be numerically solved, potentially incurring higher computational cost. Fortunately, this computational cost can be controlled via additional parameters: (1) The approximation error of numerical ODE/SDE solutions can be controlled through the tolerance parameters of numerical solvers and the inversion error of residual flows can be controlled through the number of fixed-point iterations. 3 (2) The tightness of the IWAE bound for likelihood estimation can be controlled through the number of latent samples.
The CLPF model also comprise up
Broader Impact. As expressive generative model for continuous time-series, believe CLPF can be used to model continuous dynamics from partial observations in a wide range of areas, including physics, healthcare, and finance. Furthermore, it can potentially be leveraged in a variety of downstream tasks, including data generation, forecasting, and interpolation. However, care must be taken that CLPF models are not used for adverse inference on restricted data. Protection of critical unobserved information is an active area of research that can be explored as an orthogonal component in future versions of model.
Conclusion. We have presented Continuous Latent Process Flows (CLPF), a generative model of continuous dynamics that enables inference on arbitrary real time grids, a complex operation for which we have also introduced powerful piecewise variational approximation. Our architecture is built around the representation power of a flexible stochastic differential equation driving continuously indexed normalizing flow. An ablation study well as comparison to state-of of-th--rrtttttt- baselines for continuous dynamics have demonstrated the effectiveness of our contributions on both synthetic and real-world datasets. A set of qualitative results support our findings. In the future, we plan to explore the theoretical properties of our model in more detail, including an analysis of its universal approximation properties and its ability to capture non-stationary dynamics.
10

---

References
[1] Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Tlya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016.
[2] Cedric Archambeau, Dan Cornford, Manfred Opper, and John Shawe-Taylor. Gaussian Process Ap- proximations of Stochastic Differential Equations. Proceedings of Machine Learning Research (PMLR),
2007.
[3] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and -een-Henrik Jacobsen. Invertible Residual Networks. International Conference on Machine Learning (ICML), 2019.
[4] R Bousseljot, Kreiseler, and A Schnabel. Nutzung der EKG-Signaldaen SinG-iiiaalaannbankk CARDIODAT der PTB über das Internet. Biomedizinische Technik/Biomedical Engineering, 1995.
[5] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance Weighted Autoencoders. International Conference on Learning Representations (ICLR), 2016.
[6] Anthony Caterini, Rob Cornish, Dino Sejdinovic, and Arnaud Doucet. Variational Inference with Continuously-Indee
Normalizing Flows, arXiv:2007.00 200.00546.4 2020.
[7] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[8] Ricky T. Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual Flows for Invertible Generative Modeling. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[9] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A Recurrent Latent Variable Model for Sequential Data. Advances in Neural Information Processing Systems (NeurIPS), 2015.
[10] Rob Comish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. International Conference Machine Learning (ICML), 2020.
[11] Ruizhi Deng, Bo Chang, Marcus A Brubaker, Greg Mori, and Andreas Lehrmann. Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows. Advances in Neural Information Processing Systems (NeurlPS), 2020.
[12] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation. International Conference on Learning Representations (/CLR), 2015.
[13] Laurent Dinh, Jascha Sohl-Dicksttein, and Samy Bengio. Density Estimation using Real NVP. International Conference on Learning Representations (ICLR), 2017.
[14] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented Neural ODEs. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[15] Ramazan Gençay, Michel Dacorogna, Ulrich A Muller, Olivier Pictet, and Richard Olsen. An Introduction to High-frequency Finance. Elsevier, 2001.
[16] Ary L. Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. PhysioBank, Poosio and PhysioNet: Components of a New Research Resource for Complex Physiologie Signals. Circulation, 2000.
[17] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable Reversible Generative Models with Free-form Continuous Dynamics. International Conference on Learning Representations (ICLR), 2019.
[18] Ali Hasan, João M Pereira, Sina Farsiu, and Vahid Tarokh. Identifying Latent Stochastic Differential Equations with Variational Auto-Encoders. arXiv:2007.0600 2020.
[19] Patrick Kidger, James Foster, Xuechen Li. Harald Oberhauser, and Terry Lyons. Neural SDEs as Infinite- dimensional GANs. International Conference Machine Learning (ICML). 2021.
[20] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural Controlled Differential Equations for Irregular Time Series. Advances in Neural Information Processing Systems (NewrIPS), 2021.
[21] Durk P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 convolutions. Advances in Neural Information Processing Systems (NeuriPS), 2018.
[22] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved Variational Inference with Inverse Autoregressive Flow. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[23] Ivan Kobyzev, Simon Prince, and Marcus A Brubaker. Normalizing Flows: Introduction and Ideas. 1908. 09250, 2019.

---

[24] Xuechen Li, Ting-Kam Leonard Wong. Ricky T. Q. Chen, and David Duvenaud. Scalable Gradients for Stochastic Differential Equations. International Conference Artificial Intelligence and Statistics (AISTATS), 2020.
[25] James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Tery Lyons, Neural Rough Differential Equations for Long Time Series. International Conference Machine Learning (ICML), 2021.
[26] Alexander Norcliffe, Cristian Bodnar, Ben Day. Jacob Moss, and Pietro Lio. Neural ODE Processes. International Conference on Learning Representations (ICLR), 2021.
[27] Bernt Oksendal. Stochastie Differential Equations: Introduction with Applications. Springer Science Business Media, 2013.
[28] George Papamakarios, Theo Pavlakou, and lain Murray. Masked Autoregressive Flow for Density Estimation. Advances in Neural Information Processing Systems (NeurlPS), 2017.
[29] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research (JMLR), 2021.
[30] Kira Rehfeld, Norbert Marwan, Jobst Heitzig, and Jürgen Kurths. Comparison of Correlation Analysis Techniques for Irregularly Sampled Time Series. Nonlinear Processes in Geopkysics, 2011.
[31] Danilo Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. International Conference on Machine Learning (ICML), 2015.
[32] Yulia Rubanova, Ricky Q. Chen, and David Duvenaud. Latent Ordinary Differential Equations for tregularly-Samplerr Time Series. Advances Neural Information Processing Systems (NeurlPS), 2019.
[33] Edward Snelson, Carl Edward Rasmussen, and Zoubin Ghahramani. Warped Gaussian Processes. Advances in Neural Information Processing Systems (NeurIPS), 2004.
[34] Andrew Wilson and Zoubin Ghahramani, Copula Processes. Advances Neural Information Processing Systems (NeurIPS), 2010.
[35] Shuyi Zhang. Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Chen. Cautionary Tales on Air Quality Improvement in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2017.
[36] Gilles Zumbach and Ulrich Müller. Operators on Inhomogeneous Time Series. International Journal of Theoretical and Applied Finance, 2001.
12


Reviews:
[{"review_id": "wpKFfY5j5Pc", "reviewer": "Reviewer_GfLk", "summary": "The paper introduces a new form of generative model for continuous time-series consisting of a latent neural SDE transformed by a continuously indexed normalizing flow. This is motivated by the fact that continuously indexed normalizing flows with latent Brownian motions can only model a restricted class of SDE dynamics while neural SDE can only model Markov processes.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses with the important problem of modeling discretely sampled continuous dynamics. The work is a development on recent work on continuous normalizing flows. Specifically, it can be seen as an improvement of the continuous-time flow processes [1]. \n\nThe main difference between [1] and the current work is the use of a latent neural SDE instead of a the Wiener process as latent process. This greatly increases the representational flexibility of the approach since many Ito processes cannot be obtained by transforming a Weiner process with a bijector. All in all, the approach is natural improvement of [1] which fixes a clearly identifiable problem. The resulting method is however less elegant as it combines two separate parameterized components, each of which would work as stand-alone generative model. The novelty is limited since the approach is a natural combination of two popular methods without key new insights and model-specific derivations.  The paper also introduces a somehow novel variational inference method to infer the latent form the discretely sampled observations. \n\nStrengths:\n- The paper addresses a clearly defined problem with important applications. It introduces a new method that solves some clearly stated limitation of state-of-the-art techniques. \n\nWeaknesses:\n- The novelty of the approach is rather limited, being the direct combination of two existing methods. \n-The experiments are on the weak side for NeurIPS. I would have liked to see more experiments both on simulated timeseries and on real data. Given the setting, it would have also be informative to compare the result with Gaussian process and neural process baselines. The results are difficult to interpret since the paper does not provide error bars. The results on real-world data are rather underwhelming, especially for the CLPF-ANODE version, which is the one described in the methods section. There is a big difference between the continuous approaches and the much simpler VRNN in terms of likelihood. The proposed method does not seem to perform much better than the much simpler CTFP.\n\n[1] Deng, Ruizhi, et al. \"Modeling continuous stochastic processes with dynamic normalizing flows.\" 34th Conference on Neural Information Processing Systems\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of their approach are only briefly discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "vZqtXXBRwit", "reviewer": "Reviewer_73E9", "summary": "This paper introduces CLPF---novel parametrization of a latent SDE---that has higher expressivity than previous approaches. CLPF is obtained as a time-dependent invertible transformation of a base process.\nThis transformation is implemented as a normalizing flow whose parameters are a function of another SDF.\nIn addition, the authors introduce a novel non-Markov piecewise variational posterior process that elegantly circumvents the fact that SDE solutions are Markov. The method is evaluated on four well-designed syntethic datasets, whose purpose is to test different properties of CLPF. The efficacy to model more complicated data is tested on a real-world dataset, while an ablation study evaluates the relative importance of different model components.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n- This work combines several existing approaches and algorithmic improvements. Namely, CLPF extends Continuous-Time Flow Process (CTFP). CTFP is defined as an invertible transformation of a base process. This work changes the base process from Wiener to OU, which prevents the variance of the process from growing over time. It also introduces a time-varying flow, whose parameters are a function of another (latent) SDE. Finally, the authors introduce a novel piecewise variational posterior process which improved learning of the generative model and inference. \n- It is clear how this work differs from the predecessors, which are adequately cited (I might have missed some relevant works as I'm not an expert in neural SDEs).\n\nQuality:\n\n- I am not familiar with SDE-based ELBOs, so I cannot comment on the correctness of the derivations.\n\nThe good:\n- This paper seems to be well done. \n- It is well written. The method is clearly explained. \n- The experiments are well designed---I especially like the synthetic data experiments, where each experiment has a clear purpose (as it tests a different aspect of the model).\n- The piecewise posterior approximation is novel and it is cleverly designed to circumvent a known limitation.\n\nThe bad:\n- I think that in Eq 15, when going from line 1 to line 2, the authors missed the expectations with respect to previous latent variables. This is a common thing to not include them in sequential VAEs. However, this is incorrect if not stated explicitly. \n- The authors do not discuss any limitations of the method. I wonder what they are, if any?\n- In experiments, the VRNN (a discrete-time baseline) often outperforms continuous methods in terms of the marginal likelihood. I find that weird. While the authors mention that VRNN is unable to produce samples consistent with the likelihood, I do not understand why that is, and it is not explained.\n- Fig. 3 contains qualitative evaluation but compares the proposed method only with a weak discrete-time baseline. Please include a better continuous-time alternative.\n\nQuestions:\n- NODEs are notoriously slow and expensive to compute. CLPFs include three different neural SDEs (base for the likelihood, posterior and a latent one). How does that affect computational cost with respect to a vanilla NODE? How about with respect to VRNN (used as a baseline)?\n- In a discrete-time system, a state could be engineered such that a Markov system becomes a non-Markov due to inclusion of additional information in the state. While I understand that your piecewise posterior idea solves the issue, I wonder if a similar solution to a discrete-time process could also be employed?\n- Have you thought about using a deterministic neural ODE as an encoder instead of a discrete-time RNN? I think this could gracefully handle non-uniform temporal sampling. This could be formulated as integrating some hidden state between observation time-stamps, and discontinuous state updates (conditioned on an observation) at time-stamps with available observations.\n\nClarity:\n\n- The paper is generally well-written and contains a sufficient amount of information about the method. It does not contain any details about the neural nets used or optimisation. The authors mention that the details are deferred to the appendix, but I haven't checked.\n- It would be nice to have a few comments on the interpretation of certain quantities, e.g., what does M_T in Eq. 2 correspond to? Can I interpret M^{(i)} in Eq. 14 as an importance weight? Was does the u_2 - u_1 term in L82 mean?\n- How do I interpret Eq. 3? As a non-expert in ODEs, I do not understand why that form is limiting.\n- Eq 11 and 13 could be easier to read with a change of variables e.g. \\hat{t_i} = t_{i_1} + \\tilde{t} or similar.\n\nSignificance:\n- The problems studied in this paper are currently very popular within the community.\n- This work is quite original and significantly improves on prior art by improving benchmark results as well as meaningfully expanding the methodological toolkit.\n- In my opinion this is a very significant piece of work.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors do not discuss any limitations of this work (e.g. computational or memory cost, where it would fail compared to alternatives). A societal-impact discussion is absent but also not necessary for this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "USTbk1hTOkd", "reviewer": "Reviewer_XbGb", "summary": "This paper proposes a continuous latent time series model called the Continuous Latent Process Flow (CLPF). Irregular time series observations are modeled as invertible transformations (i.e. normalizing flows) of an underlying stochastic process (the base process), where the flow transformation is itself parameterized by a time-varying stochastic process (the latent process), modeled by a (learned) stochastic differential equation (SDE). The authors also develop a piecewise variational approximation to the latent process, which they train by maximizing the importance-weighted ELBO. In contrast to prior work like Continuous Time Flow Processes (CTFP) and latent SDES, the authors show that CLPFs better model synthetic and real-world time series, including processes which require non-Lipschitz transforms of base processes, and non-Markovian processes. CLPFs also better capture qualitative trends in real world data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This was a well written paper that by-and-large demonstrated its quality beyond existing methods, both empirically and mathematically. While the clarity of the figures and exposition of the math could use a bit of improvement, the experiments are thorough, and are well-designed to illustrate the differences between CLPFs and prior work, making the work of significance to the field. I especially appreciated the inclusion of ablation studies to show the importance of each component. The use of an SDE to parameterize a flow-based transform is also original, to my knowledge, as is the piece-wise variational approximation. In light of the originality, quality and significance of the proposed method, I recommend acceptance of the paper.  Greater clarity would have improved my score to an 8. I just have some clarificatory questions, and some suggestions to improve how the model and its math are communicated.\n\nMy main clarificatory question is why it's important to have _both_ a base process (which is transformed to the observed process) and a latent process (which parameterizes the transformation) in order to achieve modeling flexibility. I understand how this choice improves over CTFPs, which do not handle non-Lipschitz transformations explicitly. But what about something like the latent SDE with suitable augmentations: \n1. Model the base process using an SDE (instead of the OU or Wiener process)\n2. Transform that using a CTFP-style continuously-indexed flow to the observable process\n3. Use a piecewise SDE variational approximation so that the posterior can be non-Markovian\n\nWould the worry about non-Lipschitz transforms apply to this? (And is this actually one of the ablated baselines tested? I couldn't tell). Because this would avoid the need to have two unobserved stochastic processes, which seems like it would reduce computational cost. Basically, I think it'd be great to better explain why you need both the $O_t$ process and the $Z_t$ process, instead of just transforming the $Z_t$ process to the $X_t$ process directly. (The other alternative is transforming the $O_t$ process to the $X_t$ process directly, which I understand would be a variant of CTFP.)\n\nApart from that, I wanted to share a few points that were confusing to me about the figures and the math. For Figure 1, it was not immediately obvious to me, without reading the paper closely, that the latent process wasn't the same as the base process. It's also generally not clear to me what is illustrated on the right-hand-side of the picture. Are the grey lines showing (stochastic) transformations) of specific values of the base process to the observed process? Is the blue line showing the evolution of the observed process, or something else? I think it'd be helpful to show samples of both the base process and the observed process (like in the CTFP) paper, and then on top of that show how the latent process parameterizes transformations from one sample of the base process to one sample of the observed process.\n\nFor Figure 2, the graphical model was confusing to me, because it omits / marginalizes out the base process $O_t$. Intuitively, I was thinking of a CLPFs as a HMM-style graphical model with *two* hidden states at each $t$, $O_t$ and $Z_t$, and so I was surprised to see the non-Markovian dependence of $X_{t_i}$ on $Z_{t_{i-1}}$. It was only after I thought a while about the math shown on Eq. 9 that I realized that marginalizing out $O_t$ leads to the dependence of $X_{t_i}$ on $Z_{t_{i-1}}$, but this wasn't clearly shown / derived. I think it would be a good idea to revise both the Equations and Figure 2 to better explain how this dependence arises.\n\nFinally, I think it might be a good idea to call the reweighting term $M^{(i)}$ in Eq. 14 an importance weight, since that is a readily recognized term for reweighting samples from a proposal distribution (in this case, the approximate posterior).\n\n== Post-Rebuttal ==\n\nThank you to the authors for their detailed responses, clarifications, and additional experiments. I continue to believe this is significant paper that makes meaningful methodological contributions to the field, and I think this has been further demonstrated by the additional experiments. I also appreciate that the author's are aware of the method's limitations and social impact, and look forward to seeing their inclusion in the final paper. I will maintain my score of 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think this paper could do with more discussion of limitations, especially relative to existing methods. Besides the issues with numerical approximation that were (very briefly) noted, one potential limitation that comes to mind is that CLPFs have a higher training, likelihood evaluation, and inference cost, due to the absence of a tractable likelihood. This is in contrast to CTFPs, which can take advantage of a deterministic transformation between the base and observed process, and do not require variational inference for training. Presumably this means that CLPFs might require more time / compute / energy train to reach the same performance on some tasks, and to perform likelihood evaluations, which might motivate using other models (even if they are less flexible).\n\nThe authors did not discuss societal impacts. I would encourage them to do so -- perhaps there are positive/negative impacts of being able to model real-world continuous time data more faithfully, e.g. for forecasting and scientific modeling -- but I think it is their prerogative, given that this is more of a \"general methodology\" paper rather than an application-focused paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "PtPm_6wzwO_", "reviewer": "Reviewer_1ACw", "summary": "The paper addresses probabilistic modeling of sparse or irregularly sampled time series. The proposed method captures the systems dynamics by a rather simple latent stochastic process, which is then warped into a more complex one using a time-indexed ANODE-like normalizing flow. The free parameters of the dynamical system are identified by maximizing the ELBO that lower bounds the likelihood of the observed sequence. The solution aims to overcome the limited expressive power of normalizing flows stemming for the necessity to use invertible transformations that also need to satisfy Lipschitz continuity restrictions.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance of Novelty\n--\n\nA key design choice is to assume the latent dynamics to be governed by an Ornstein Uhlenbeck process, which is a rather trivial stochastic process with analytical solution. The beauty of the solution comes when this simple process is warped by a normalizing flow into dynamical models with arbitrary complexity. This is a compelling solution, and however simple, a novel use of stochastic differential equations in modern machine learning. The suggested variational inference algorithm, especially piecewise reweighting part is also interesting and novel for the context in question. However, there do exist alternative solutions to all the problems pointed out in the paper, which appear to work in the reported experiments better than the proposed method. The lack of empirical evidence on the benefit of the approach impairs its significance.\n\nPresentation\n--\nSection 2 gives a really nice, concice, and clear overview into the essential material to be used later. It is remarkable how a variety of complex concepts are introduced in such clear terms. However, I find the structure of Section 3 a bit convoluted. It takes a while to grasp how the pieces come together in the second and third paragraphs of Section 3.1. Further, the novelty claims are also a bit hidden. I recommend that the next revision of the paper includes a dedicated paragraph that explain in technical depth what exactly is different in the proposed solution from the state of the art. Lastly, presenting the whole method also as pseudocode in the main paper as Algorithm 1 would substantially improve the reader experience.\n\nExperimental Evaluation\n--\n\nBoth the baselines and the experiment setups are well-chosen and well-justified. However, the same is not the case for the evaluation metrics. The paper reports only the test log-likelihood in all experiments. While obviously being a useful score, it is not sufficient to draw conclusions about the prediction success by looking only at the log likelihood. The authors should definitely provide the plain RMSE scores of the same experiments in the rebuttal phase. Furthermore, the proposed method does not seem to improve the evaluated performance metric in the real-world data sets with respect to existing alternative solutions. It could well be that the proposed method has some properties more favorable than its competitors beyond prediction accuracy, but then there should be a corresponding experiment or evaluation metric that quantifies this favorableness.\n\nQuestions:\n--\n\nFig 2 shows that in the data generating process, X_t depends on both Z_t and Z_{t-1}. Where does the dependency on Z_{t-1} come from? Which equation in the paper clarifies this dependency? Eq 6 is a non-Markoving mapping.\n\nOverall\n--\n\nInteresting problem selection. The solution has some not groundbreaking but fairly novel aspects. Scientific writing is overall OK, but the novelty presentation (Sec 3) could be improved. Experimental evaluation is extremely weak. The solution falls behind multiple baselines. Evaluation wrt RMSE missing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not find the answer of the authors to the checklist question 1 (b) convincing. Having closely read Section 3, I am not able to point out a clear discussion anywhere in the paper about the limitations of the \"original solution\" proposed by this work. The paper answers to some extent previously unsolved problem this paper has solved, but does not really talk about what its leaves unsolved or at the expense of what the solution has been achieved (e.g. increased computational cost, additional assumptions on the latent space, loss of interpretability etc.).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "This is a generic methods development paper. Hence I do not see any identifiable ethical concerns specific to the problem it addresses or the solutions it proposes. However, I would still expect the paper to provide more thoughts on this somewhere in the manuscript. The answers provided in the checklist on this issue are too generic and superficial."}}]

Author rebuttal:

Meta-review: This paper explores a continuous latent time series model, the Continuous Latent Process Flow (CLPF). 

CLPFs can be seen as a generalisation of CTFPs where, instead of transforming a base Wiener process with a time-conditional flow, it first introduces a latent process SDE which conditions a CLPF-like model (that uses an OU base process instead of a Wiener process). In that sense, CLPFs combine elements from VRNNs, latent SDEs and CTFPs.

In terms of motivation/discussions, it is clear why CLPFs should be strictly better than CTFPs, but it is not clear why they should be better than latent SDEs (at least not "a priori") so adding more discussions on this could improve the draft.

The experiments show good performance on real-world data relative to CTFPs, latent SDEs and VRNNs. In spite of some hesitance amongst reviewers, I find the experimental results compelling. But I also agree that the paper could benefit from more detailed experiments comparing to VRNNs, for instance to clarify if the difference with respect to VRNN is due to implementation differences (e.g. passing absolute time or not, or small architectural differences).
Decision: Accept (Poster)