Deep Latent State Space Models for Time-Series Generation
Authors: Linqi Zhou
Michael Poli
Winnie Xu
Stefano Massaroli
Stefano Ermon
Venue: Submitted to ICLR 2023 (2023)

Abstract: Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series. In addition to high computational overhead due to explicitly computing hidden states recurrence, existing ODE-based models fall short in learning sequence data with sharp transitions - common in many real-world systems - due to numerical challenges during optimization. In this work, we propose LS4, a generative model for sequences with latent variables evolving according to a state space ODE to increase modeling capacity. Inspired by recent deep state space models (S4), we achieve speedups by leveraging a convolutional representation of LS4 which bypasses the explicit evaluation of hidden states. We show that LS4 significantly outperforms previous continuous-time generative models in terms of marginal distribution, classification, and prediction scores on real-world datasets in the Monash Forecasting Repository, and is capable of modeling highly stochastic data with sharp temporal transitions. LS4 sets state-of-the-art for continuous-time latent generative models, with significant improvement of mean squared error and tighter variational lower bounds on irregularly-sampled datasets, while also being x100 faster than other baselines on long sequences.

Under review as a conference paper at ICLR 2023
DEEP LATENT STATE SPACE MODELS FOR TIME-SERIES GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Methods based on ordinary differential equations (ODEs) are widely used build generative models of time-series. In addition to high computational overhead due to explicitly computing hidden states recurrence, existing short in learning sequence data with sharp transitions common in many real- world systems due to numerical challenges during optimization. this work, we propose LS4, generative model for sequences with latent variables evolv- ing according to a state space ODE to increase modeling capacity. Inspired by recent deep state space models (S4), we achieve speedups by leveraging con- volutional representation of LS4 which bypasses the explicit evaluation of hid- den states. We show that LS4 significantly outperforms previous continuous time generative models in terms of marginal distribution, classífication, and prediction scores on real-world datasets in the Monash Forecasting Repository, and is ca- pable of modeling highly stochastic data with sharp temporal transitions. LS4 latent generative models, with signifi-
models fall
sets state state-of-tue off-the-ar art for continuous-ting cant improvement of mean squared error and tighter variational lower bounds on uregularly-samplee datasets, while also being 100 faster than other baselines on long sequences.
INTRODUCTION
Time series are a ubiquitous data modality, and find extensive application in weather (Hersbach al. 2020) engincering disciplines, biology (Peng et al.) (1955,, and finance (Poli al. 2019). The main existing approaches for deep generative learning of temporal data can be broadly categorized
into autoregressive (Oord al. 2016), latent autoencoder models (Chen et al.) 2018 Yildiz et al.,
2019 Rubanova et al.) 2019), normalizing flows (de Bézenac et al. 20200), generative adversarial
Yoon al., 2019 Yu et al. 2022] Brooks al.] 2022), and diffusion (Rasul et al. [202]] Among these, continuous-time methods (often based underlying ODEs) are the preferred approach for irregularly-samples sequences because they can make predictions at arbitrary time steps and can handle sequences of varying lengths. Unfortunately, existing ODE- based methods (Rubanova al. 2019 Yildiz et al.] 2019) often fall short in learning models for real-world data (c.g., with stiff dynamics) due to their limited expressivity and numerical instabilities during backward gradient computation (Hochreiter 1998 Niesen & Hall 2004 Zhuang et al. 2022).
A natural way to increase the flexibility of ODE-based models is to increase the dimensionality of their (deterministic) hidden states. However, that leads to quadratic scaling in the hidden dimension- ality due to the need of explicitly computing hidden states by unrolling the underlying recurrence over time, thus preventing scaling to long sequences. An alternative approach to increasing model- ing capacity is to incorporate stochastic latent variables into the model, highly successful strategy in generative modeling (Kingma & Welling 2013) Chung et al. 2015: Song et al.) 2020 Ho et al.) 2020). However, this leads to computational costs, and existing models like latent neural ODE mod- els (Rubanova et al. 2019) inject stochasticity only at the initial condition of the system. In contrast, we introduce LS4, latent generative model where the sequence of latent variables is represented as the solution of linear state space equations (Chen) 1984). Unrolling the recurrence equation shows an autoregressive dependence in the sequence of latent variables, the joint which is highly ex- pressive in representing time series distributions. The high dimensional structure of the latent space, being equivalent to that of the input sequence, allows LS4 to learn expressive latent representations and fit the distribution of sequences produced by a family of dynamical systems, common setting

---

Under review as a conference paper at ICLR 2023
resulting from non son-stationarity. We further show how LS4 can learn the dynamics of stiff (Shampine & TTommson 2007) dynamical systems where previous methods fail to do so. Inspired by recent works on deep state space models, or stacks of linear state spaces and non-linearities (Gu et al. 2020 2021), we leverage a convolutional kernel representation to solve the recurrence, bypassing any explicit computations of hidden states via the recurrence equation, which ensures log-linear scaling in both the hidden state space dimensionality as well as sequence length.
We validate our method across variety of time series datasets, benchmarking LS4 against an ex- tensive set of baselines. We propose a set of 3 metrics to measure the quality of generated time series samples and show that LS4 performs significantly better than baselines on datasets with stiff transitions and obtains on average 30% lower MSE scores and ELBO, On sequences with = 20K lengths, our model trains 100 faster than the baseline methods.
RELATED WORK
Rapid progress on deep generative modeling of natural language and images has consolidated dif- fusion (Ho et al., 2020) Song et al.] 22200; Song & Ermon 2019 [Sohl Sooh--Dicsttein et al.) (015)) and autoregressive techniques (Brown et al. (020) as the state-of ofate-ofthhe thrt Although various approaches have been proposed for generative modeling of time series and dynamical systems, consensus on the advantages and disadvantages of each method has yet to emerge.
Deep generative modeling of sequences. All the major paradigms for deep generative modeling have seen application to time series and sequences. Most relevant to our work are latent continuous- time autoencoder models proposed by Chen et al. (2088;; Yildiz et al. (2019); Rubbanva et al. (2019), where a neural differential equation encoder is used to parametrize as distribution of initial conditions for the decoder. Massaroli et al. () proposes variant that parallelizes computation in time by casting solving the ODE as rool finding problem. Beyond latent models, other continuous- time approaches are given in Kidger et al. (2020). which develops a GAN formulation using SDEs.
State space models. State space models. (SSMs) are at the foundation of dynamical system the- ory (Chen 1984) and signal processing (Oppenheim) 1999). and have also been adapted to deep generative modeling. CChung et al. (2015): Bayer & Osendorfer (2014) propose VAE variants of time RNNs, generalized later by (Franceschi al., 2020), among others. These models all unroll the recurrence and are thus challenging scale to longer sequences.
Our work inspired by recent advances deep architectures built as stacks linear SSMs, notably $4 (Gu et al. 202T). Similar to S4, our generative model leverages the convolutional representation of SSMs during training, thus bypassing the need to matcrialize the hidden state of each recurrence.
3 PRELIMINARIES
We briefly introduce relevant details of continuous-timg time SSMs and their different representations. Then introduce preliminaries of generative models for sequences.
3.1 STATE SPACE MODELS (SSM)
A single-input single-output (SISO) linear state space model is defined by the following differential equation
d h, Ab, + Br,
dt
yt Ch, + Dr
with scalar input 2; R. state h, E N and scalar output yt E IR. The system fully characterized
by the matrices A NxN B E Nx1 C. II D Rixl Let x, y E C(la, b). (..RR be absolutely continuous real signals on time interval Ja, BJ. Given an initial condition hy E R the SSM realizes mapping z y.
SSMs are a common tool for processing continuous input signals. We consider single input single output (SISO) SSMs, noting that input sequences with more than single channel can be processed
2

---

Under review as a conference paper at ICLR 2023
by applying multiple SISO SSMs in parallel, similarly regular convolutional layers. We such SSMs as building blocks to map each input dimension to each output dimension in our generative model.
Discrete recurrent representation. practice, continuous input signals are often sampled at time interval A and the sampled sequence is represented by (Fit Pty... ....... (tt,) where ts+1 +.A.. The discretized SSM follows the recurrence
h
Ah + Ba
Vts Cha, + Dat.
where Á cA B A- B)B with the assumption that signals constant during the sampling interval. Among many approaches to efficiently computing eA Gu et al. (2021) use bilinear transform estimate eA (I AAA) (1 +
This recurrence equation can be used to iteratively solve for the next hidden state h4g 11. allowing the states be calculated like an RNN Neural ODE (Chen et al. 2018 Massaroli et 2020).
Convolutional representation. Recurrent representations of SSM are not practical in training be- cause explicit calculation of hidden states for every time step requires (N in time and O L) in space for a sequence of length - This materialization of hidden states significantly restricts RNN-based methods in scaling long sequences. To efficiently train an SSM, the recurrence equa- tion can be fully unrolled, assuming zero initial hidden states, as
h = Brioo
h ABa + Brtt
h Bz + AB + B
St, = CB
yt. = C. CAB + CBB
yt. = CA + CABr, + CBa
and more generally as,
V1 = BB + CAA-1 Br + ..... CBBN
For an input sequence x (Itg Tty:........... Ft.)) one can observe that the output sequence y (ytur ....... Ytg can be computed using convolution with skip connection
y CK t De, where K (B, ... A' B.-BBAA' B
This is the well-known connection between SSM and convolution (Oppenheim & Schafer 1975 Chen 1984 Chilkuri & Eliasmith, 2021 Romero et al., 202T Gu et al. 2020; 2021] 2022) and it be computed very efficiently with Fast Fourier Transform (FFT), which scales better than explicit matrix multiplication at each step.
3.2
VARIATIONAL AUTOENCODER (VAE)
VAEs (Kingma Welling, 2013 Burda et al. 2015) are a highly successful paradigm learning latent representations of high dimensional data and is remarkably capable at modeling complex distributions. A VAE introduces joint probability distribution between a latent variable z and an observed random variable of the form
(x(x,z) = Po po z)p(z))
where 0 represents learnable parameters.
The prior p(z) the latent is usually chosen standard Gaussian distribution, and the ditional distribution (x(x z) is defined through a flexible non-linear mapping (such as neural network) taking z input. Such highly flexible non-linear mappings often lead an intractable posterior Pa (z x). Therefore, an inference model with parameters & parametrizing (o x) introduced as an approximation which allows learning through a variational lower bound of the marginal likelihood:
log (x) 2 - DKL [x)lp(z)) + Egg (xix) [log (x z)]
(4)
where DKL(-Il-) is the Kullback-Leebler divergence between two distributions.
Further explanations in Appendix A.1

---

Under review as a conference paper at ICLR 2023
VAE for sequences. Sequence data can be modeled in many different ways since the latent space can be chosen to encode information at different levels of granularity, i.e. can be a single variable encoding entire trajectories sequence of variables of the same length the trajectories. We focus on the latter.
Given observed sequence variables xgr up time T discretized into sequence (Xto:-... X of length where fz-1 = T, a sequence VAE model with parameters 8, A, ø learns a generative and inference distribution
L-1
1:
II i=
(x(:
po
Xxct, 25t))px zet,)
L-1
96(2c12-1 X5te-1)
II ==
9 (Z,, (zg.))
where Betz = (Zio...... Lt is the corresponding latent variable sequence. The approximate posterior do explicitly factorized product of marginals due efficiency reasons shall discuss in the next section. Given this form of factorization, the variational lowerbound has been considered for discrete sequence data (Chung et al., (2015) with the objective
[
Eyslesr Lixsee 1)
DKu(ge(Ze Ixsx))lxx(z zet.)) + log pa (xt, (xc2)
(5)
4 METHOD
In this section, we introduce Latent S4 (LS4), a latent variable generative model parameterized using SSMs. We show how SSMs can parametrize the generative distribution pe (x )px (ZgT), the prior distribution (ZgT) and the inference distribution (z<TT xşr) effectively. For the purpose of exposition, we can assume 2t, 3y are scalars at any time step Their generalization to arbitrary dimensions is discussed in Section
We first define structured state space model with two input streams and use this building block for our generative model. It is an SSM of the form
d h; Ah, + Brg + E2,
dt
(6)
1/t = Ch, Dx. + Fz.
continuous real signals on time interval [0, We denote [ 2, A, B, E, hp, = Hg(x, bo. f), where 3 denotes trainable parameters A, B, E, as the deterministic function mapping from signals x, 2 h,, the state time t, given initial state hy at time O. The above SSM can be compactly represented by
where 2, C[0,
31 = CCHg(r, z, ho, + Da, + F2,
When the continuous-time input signals are discretized into discrete-time sequences (Ito.... _.,..)) and tional view (assuming D = F for simplicity)
), the corresponding hidden state at time tx has a convolu-
V/, CK2, Tze + CKK,, Ate where K., ABB, K, A'E
which can be evaluated efficiently using FFT. Additionally, A is HiPPO-initialized (Gu et al. 2021 for all such SSM blocks.
4.1 LATENT SPACE STRUCTURED STATE SPACE
The goal of the prior model is to realize sequence of random variables (Et, ....... 2t, which the prior distribution px px(zgrz) models autoregressively. Suppose (Ztor Rti......... 2t,.) is a sequence of latent variables up time in. we define the prior distribution of 2t,, autoregressively as
Px (2 =2t.)) =N(pz (Zet,, A). oz (z)t.))))
(9)
4

---

Under review as a conference paper at ICLR 2023
where the mean M2,n and standard deviation d., deterministic functions of previously generated 2<1, parameterized by x. To parameterize the above distribution, we first define an intermediate building block, stack of which will produce the wanted distribution.
LS4 prior block. The forward pass through our SSM is a two-step procedure: first, we consider the fn-) where simply leverage Equation 6 to define the hidden states to follow H (0, 2, 0, t). Second, on (1 fn], since no additional & is available in this interval, we ignore additional input signals in the ODE and only include the last given latent, i.e. it, as an auxiliary signal for the outputs, which can be compactly denoted, with final GELU non-linearity,
latent dynamics of
as
Vz.n GELU H3, (0, O, H3, (O, Plta.fa Nem-11 O, 0,1n-1,, tw)) + Fe.
(10)
We call this LS4 prior layer, which we use to build our LS4 prior block that is built upon ResNet structure with a skip connection, denoted as
LS4prior (2 .)) LayerNorm(Gy,.ee + bys) + Sta-1
(11)
where & denotes the union of parameters B., : Fu. Gy. by. We define the final parameters Mz,n and Jz.n for the conditional distribution in the autoregressive model the result of stack of LS4 prior blocks. During generation, as an initial condition, 2t, ~ (Mz,0, oi.u) where Ma 0, 2s,0 are learnable parameters, and subsequent latent variables are generated autoregressively. We specify architecture in Appendix C and use À to denote the union of all trainable parameters.
4.2 GENERATIVE MODEL
Given the latent variables, we now specify a decoder that represents the distribution po (arge, (2g(,) Suppose zstz is latent path generated via the latent state space model, the output path Tstz also follows the state space formulation. Assuming we have generated (Iior.... ,t, Lt,
and
(Zt. the conditional distribution of zt, is parametrized as
Po |rct, *st..) = (Nu ( zst., 0),
(12)
where is pre-defined observation standard deviation and Ma,n is deterministic function of agt. and Ter..
LS4 generative block. Different from the prior block, both observation and latent sequences are input into our model, and we define intermediate outputs 9x,n and 92.1 as
h4 = Hs, (0, 2:n-1: H&, (2]to 11. Attain 1). O, fn,,,tn),tte
9.. GELU 9. Ne., + D Dt. + Fg. mnn)
(13)
Jz.n GELU 9.. he, + Dr. + Fg. 2tm)
which are used to build a LS4 generative block defined as
In.n = LayerNormi
(eanNoormGG,g.aa.. Gg. + bgg)) ) + Ita
9=.n LayerNorm( G Gg.92.n by.) 2t.
(14)
LS4 (I[to..n-11)) ((tto.
(0x,n1 92,m)
where denotes all parameters inside the block. Note that the generative block gives streams of outputs, which can be used as inputs for the next stack. We then define the final mean value Pann the result of stack of LS4 generative blocks. The initial condition for generation is given as Tto ~ N (Pz,0 0), aw)) where 0a,0 exactly follows our formulation while taking only 2t, as input. The subsequent vt.," 's are generated autoregressively. We specify our architecture Appendix and use f to denote the union of all trainable parameters.
4.3 INFERENCE MODEL
The latent variable model up time t, has intractable posterior po (zge, Igg..)) Therefore, we approximate this distribution with (= (rge.)) using variational inference.
We parameterize the inference distribution at time in to depend only on the observed path rsst,
( xgg.)) N (iza, (Iis... (pstn..)) Φ), 02 (I x))
(15)
5

---

Under review as a conference paper at ICLR 2023
LS4 inference block. The inference block is defined
Dz.n GELU(C 0. Ha, (2)o...)) 0, 1n- 0,0,t.11) + Dg. Fin)
(16)
[ 4) LayerNorm( G. Dn.n + bg.) + It...
Notice that input æ is fully present in Ito tm] unlike in the generative model. fazt and os, are each deep stack of our inference blocks similarly defined as before. Here also justify our choice of factorizing posterior as a product of marginals as Equation 5 By having each 2t explicitly depending ggt. only,
leverage the fast convolution operation to obtain all in parallel, thus achieving fast inference time, unlike the autoregressive nature of the prior and generative model.
4.4 LS4: PROPERTIES AND PRACTICE
We highlight some properties of LS4. In particular, we compare in the following proposition the expressive power of our generative model against structured state space models.
Proposition 4.1. (LS4 subsumes S4.) Given any autoregressive model r(x) with conditionals
r(In [<<a) parameterized via deep S4 models, there exists choice of 0, X, é such that po,x r(x) and (o.x(2)2) = (x),,,), iee. the variational lower bound (ELBO) is tight.
proof sketch provided in Appendix B This result shows that LS4 subsumes autoregressive generative models based on vanilla S4 (Gu et al., 2021), given that the architecture between SSM layers is the same. Crucially, with the assumption that we are able to globally optimize the ELBO training objective, LS4 will fit the data at least as well as vanilla S4.
Scaling to arbitrary feature dimensions.
far we have assumed the input and latent signals are real numbers. The approach can be scaled to arbitrary dimensions of inputs and latents by constructing L.S4 layers for each dimension which are input into mixing linear layer. We call such parallelized SSMs heads and provide pscudo-code in Appendix C
Proposition 4.2. (Ffficiency.) For a SSM with heads, an observation sequence of length and hidden dimension N can be calculated in (H log(Le L + N)) time and ([H(L+N)) space.
Note that our model is much more efficient in both time and space than RNN/ODE baseed methods (which requires (N in time and O(NL) space as discussed in Section B. To demonstrate the computation efficiency, we additionally provide below pseudo- code for a single LS4 prior layer I0 The other blocks can be similarly constructed.
We provide proof in Appendix B
def LS4 As4_prror_layerr x @ materialize_kereer (z, A, h) # or ( log (L+N)) time fft_conv (K, 2) ((Toog)) time and (1.) space
A, B, C, F, h_0): $ 2: (B, L, 1)
y gelu (CH F**)
return
Note that practice, A HiPPO initialized (Gu al. 2020) and the materialized kernel includes C' so that the convolution is computed directly in the projected space, bypassing materializing the ligh-dimensional hidden states.
EXPERIMENTS
In this section, we verify the modeling capability LS4 empirically. There are three main questions we seek to answer: (1) How effective is LS4 in modeling stiff sequence data? (2) How expressive is LS4 in scaling to real time-series with a variety of temporal dynamies? (3) How efficient training and inference is LS4 in terms of wall-clock time?
5.1 LEARNING TO GENERATE DATA FROM STIFF SYSTEMS
Modeling data generated by dynamics with widely separated time scales has been proven to be particularly challenging for vanilla ODE- approaches which make of standard explicit solvers for inference and gradient calculation. Kim et al. (202T) showed that as the learned dynamics stiffen up track data paths, the ODE numeries start cafastrophically fail; the inference cost

---

Under review as a conference paper at ICLR 2023
raises drastically and the gradient estimation process becomes ill ill-conditioned. These issues can be mitigated by employing implicit ODE solvers or ad-hoc rescalings of the learned vector field (see (Kim et al. 2021) for further details).
In turn, state space models do not suffer from stiffness of dynamics as the numerical methods are sidestepped in favor of an exact evaluation of the convolution operator. We hereby show that LS4 is able to model data generated by a prototype stiff system.
FLAME problem We consider simple model of flame growth (FLAME) (Wanner & Hairer 1996), which has been extensively studied as representative of highly stiff systems:
-
21
where p E (3,4,.... 10). For each p. 1000 trajectories are generated for t E [0, 1000) with unit increment.
Generation. In Figure Lap we show the mean trajectories and the distribution at each time step and that our samples closely match the ground-truth data. The Latent ODE (Rubanova et al. 2019) instead fails to do and produces non-stiff samples drastically different from the target.
Density
Density
Value
0.9
6.11
8:5
543
0.4
805
316
825
0.5%
steps
1.4% steps
0.4
Real
Real
Density
Density
Ours
Latent ODE
Latent ODE
Step
33
3.7%
steps
10
10.0% steps
(a) Generation of the stiff system.
(b) Marginal histograms steps equally spaced between 0.5% and 10% steps log scale.
Marginal Distribution. We plot the marginal distribution of the real data and the generated data from both our model and Latent ODE. Since the stiff transitions are mostly distributed before 10% of total steps, we visualize the marginal histograms at 4 time steps equally spaced between the 0.5% and 10% steps in log scale (see Figure Tb). We observe that the empirical histogram matches the ground truth distribution significantly better than what is produced by Latent ODEs, also qualitatively visible from the samples in (a).
5.2 GENERATION WITH REAL TIME-SERIES DATASETS
We investigate the generative capability of LS4 on real time-series data. We show that model is capable fitting a wide variety of time-serics data with significantly different temporal dynamics. We leave implementation details Appendix D.I
Data. We use Monash Time Series Repository (Godahewa et al. 2021 a comprehensive benchmark containing 30 time-series datasets collected in the real world, and choose FRED-MD, NN5 Daily, Temperature Rain, and Solar Weekly as our target datasets. Each dataset exhibits unique temporal dynamics, which makes generative learning challenging task. A sample from each dataset can be visualized in Figure 2
FRED-MD
NN5 Daily
Temperature Rain
Solar Weekly
Figure 2: Monash data. The selected datasets exhibit variety of temporal dynamics ranging from relatively smooth to stiff transitions.
Metrics. We propose 3 different metrics for measuring generation performance, namely Marginal, Classification, and Prediction scores. Marginal scores calculate the absolute difference between

---

Under review as a conference paper at ICLR 2023
empirical probability density functions of two distributions the lower the better (Ni et al. 2020 Following Kidger et al. (202T), we define Classification scores as using a sequence model to classify whether sample is real generated and use its cross-entropy loss a proxy for generation quality - the higher the less distinguishable the samples, thus better the generation. Similarly, Prediction seq2seq model to predict ls steps into the future the lower the more predictable, thus better the generation. We use a I-layer S4 (Gu et al. (2021) for both Classification and Prediction scores (see Appendix [D]] for more details). We will discuss the necessity of all 3 metrics in the following discussion section.
scores use rran-on-synthetic-teste.
We compare our model with several time-series generative models, namely the such as RNN- VAE (Rubanova et al. 2019), GP- PP- (Fortuin et al. 2020). ODE2VAE (Yildiz et al.
methods
2019), Latent ODE (Rubanova et al. 2019). GAN-based methods such as TimeGAN (Yoon et al,
2019 and SDE GAN (Kidger et al. 2021 and SaShiMi (Goel et al. 2022). We show quantitative
results in T
Data
Metric
RAN-VAE
GP-VAE
ODE'VAE
Lasent ODE
TimeGAN
SDEGAN
SuShiMi
1.44 (Ours)
FRED-MI
Marginal L
0.132
0.152
0.122
0.0416
0.0813
0.0841
0.0482
0.0221
Class, f
0,0362
0,0158
0,0282
0.327
0.0294
0.501
0.00119
0,544
Prodiction
1.47
2,05
0,567
0.0132
0.0575
0.677
0.232
0.0373
NN5 Daily
Marginal
0.137
0.117
0,211
0.107
0.0396
0.0852
0.0199
0.00671
Class.
0,800319
0.00246
0.00102
DO00381
0,00160
0.0852
Prodiction
0.967
1,169
1.19
1.04
1.34
1.01
0.849
0.241
Temp Rain
Murginal
0.0174
1.831
0.990
0.0834
Class. f
0.00000212
0,0000123
0,0000319
0,0300419
0.00271
0,0169
0.0000167
0.976
Prediction
159
2.305
1.133
145
1.96
2.46
2.12
0.521
Solar Woekly
Marginal L. Class.
0.0003
0,308
0.153
0.0853
0.0496
0.147
0.173
0.0459
0,0524
0,000731
0.0998
0,0521
0.6189
0.591
0.00102
0.683
Prodiction
1.25
1.47
0.761
0.973
0.237
0.976
0.578
0.141
Table 1: Generation results on FRED-MD, NN5 Daily, Temperature Rain, and Solar Weekly,
Our model significantly outperforms the baselines on all datasets. We note that baseline models have hard time modeling NN5 Daily and Temperature Rain where the transition dynamics are stiff. For Temperature Rain where most data points lie around r-axis with sharp spikes throughout, latent ODE generates mostly closely to the z-axis, thus achieving lower marginal scores, but its generation is easily distinguishable from data, thus making it a less favorable generative model. We demonstrate that Marginal scores alone are an insufficient metric for generation quality. SaShiMi, an autoregressive model based on S4, does not perform as well on time series generation in the tasks considered. We further discuss the reason in Appendix D.I
5.3 INTERPOLATION & EXTRAPOLATION
We also show that model is expressive enough to fit to ireeullarly data and performs favorably in terms of interpolation and extrapolation. Interpolation refers to the task of predicting missing data given subset of sequence while extrapolation refers to the task that data is separated into 2 segments each with half the length of the full sequence, and one predicts the latter segment given the former.
Data. Following Rubanova et al. (011); Schirmer et al. (2022). we use USHCN and Physionet as our datasets of choice. The United States Historical Climatoloyy Network (USHCN) (Menne al., 2015) is a climate dataset containing daily measurements form 1,218 weather stations across the US for precipitation, snowfall, snow depth, minimum and maximum temperature. Physionet (Silva) et al. 2012) is a dataset containing health measurements of 41 signals from 8,000 ICU patients in their first hours. We follow preprocessing steps of SShhrmere et al. (2022) for training and testing.
Metrics. We use mean squared error (MSE) to evaluate both interpolation and extrapolation.
We compare our model with RNN (Rumelhart et al.) 1985 RNN- VAE (Chung et al. 2014 Rubanova et al., 2019) ODE-RNN (Rubanova et al. 2019), GRU-D (Rubanova et al.) 2019), Latent
ODE (Chen et al. 2018 Rubanova et 2019), and CRU Schirmer et al. (2022). Results are shown in Table 2
Numbers taken from the original paper. We keep the significant digits unchanged
8

---

Under review as a conference paper at ICLR 2023
Task
Data
RNN
RNN-VAR
ODE-RNN
GRU-D
Latent ODE
CRU
1.44 (Ours)
Physionet
2.918
5.930
2.234
3.325
8.341
1.82
0.6287
Interp.
USHCN
4.322
7.561
2.473
3.395
6.859
0.16
0.0594
Physionet
3,406
3.064
3,014
3.120
4.212
6.29
4.942
Extrap.
USHCN
9.474
9.083
9.045
8.964
8.959
12.73
2.975
Table 2:
Interpolation and extrapolation MSE (x10-), scores. Lower scores are better.
We observe that our model outperforms previous continuous-time methods. Our model performs less well extrapolation for Physionet compared ODE-RNN and latent ODE. We postulate that this is due to the high variability granted by our latent space. Since new latent variables are generated as we extrapolate, our model generates paths that are more flexible (hence less predictable) than those of Latent ODE, which instead uses single latent variable to encode an entire path. We additional observe that our model achieves ELBO of -69.00 and 250.22 on Physionet interpolation and extrapolation tasks respectively. These bounds are much tighter lower bounds than other VAE- based methods, i.c. RNN-VAE, which reports 412.8 and 2220.2, and latent ODE, which reports -41033 and - 168.5. We leave additional ELBO comparisons in Appendix D
5.4 RUNTIME
We additionally verify the computational efficiency of our model for both training and inference. We do so by training and inferring on synthetic data with controlled lengths specified below.
of synthetic datasets with lengths {80, 320, 1280, 5120, 20480} investigate scaling of training/inference time with respect to sequence length. Training is done with 100 itera- tions through the synthetic data, and inference is performed given one batch of synthetic data (see Appendix D.3
Data. We create
Metrics. We use wall-clock runtime measured in milliseconds.
Training Time
Inference Time
10
Runtime ms
107
Runtime ms
30a
Latent ODE
ODE-RNN
10a
1.P
ODE2VAE
SDE GAN
10a
Ours
10a
101
320
1280
20480
320
20480
Sequence Length
Sequence Length
Figure 3: Runtime comparison. The y-axis shows run-time (ms) of each setting in log scale. Our runtime stays consistently lower across all sequence lengths investigated.
Figure 3 shows model runtime in log scale. ODE*VAE fails to finish training on the last sequence length within a reasonable time frame, so we omit its plot of the last data point. Our model performs consistently and significantly lower than baselines, which are observed to scale linearly with input lengths, and is x100 faster than baselines in both training and inference on 20480 length.
6 CONCLUSION
We introduce LS4, powerful generative model with latent space evolution following a state space ODE. Our model built with deep stack of LS$4 prior/generaiive/nfeeencc blocks, which are trained via standard sequence VAE objectives. We also show that under specific choices of model parameters, LS4 subsumes autoregressive S4 models. Experimentally, we demonstrate the modeling power of LS4 on datasets with wide variety of temporal dynamics and show significant improve- quality. In addition, our model shows x100 speed-up
ment in generation/intert http://ollotoio/eetttpllaaaaeaae in training and inference time on long sequences. LS4 demonstrates improved expressivity and com- putational efficiency, and we believe that it has a further role to play in modeling general time-series sequences.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint arXiv:1411.7611 2014.
Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming- Yu Liu, Alexei A Efros, and Tero Karras. Generating long videos of dynamic scenes. arXiv preprint Xiiv:2206.034299 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint 1rXiv:1509.000 2015.
Chi-Tsong Chen. Linear system theory and design. Saunders college publishing, 1984.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Narsimha Reddy Chilkuri and Chris Eliasmith. Parallelizing legendre memory unit training. In International Conference Machine Learning, pp. 1898-1907. PMLR, 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Junyoung Chung. Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. recurrent latent variable model for sequential data. Advances neural information pro- cessing systems, 28, 2015.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems, 33:2995-3007, 2020.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically time series. Advances in neural information processing sys- tems, 32, 2019.
Otto Fabius and Joost Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.658/ 2014.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. Gp-vae: Deep probabilis- tic time series imputation. International conference on artificial intelligence and statistics, pp. 1651-1661. PMLR, 2020.
Jean- Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pp. 3233-3246. PMLR, 2020.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state- space models. arXiv preprint arXiv:2202.09729) 2022.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00390 2021.
10

---

Under review as a conference paper at ICLR 2023
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. arXiv preprint arXhv:2206.1182) 2022.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabatt Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):199--2000 2049, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Snowledge-Base Systems, 6(02): 107-116, 1998.
Patrick Kidger, James Morrill, James Foster, and Tenry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 6707, 2020.
Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes infinite-dimensional gans. In International Conference on Machine Learning, pp. 5453-5463. PMLR, 2021.
Suyong Kim, Weiqi Ji, Sili Deng, Yingbo Ma, and Christopher Rackauckas. Stiff neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlinear Science, (199))9931 2021.
Diederik P Kingma and Max Welling. 1312.61312.61/ 14, 2013.
Auto-encoding
variational
bayes.
arXiv preprint
Xuechen Li, Ting-Kon Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882. PMLR, 2020.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. Advances in Neural Information Processing Systems, 15:3952-3963. 2020.
Stefano Massaroli, Michael Poli, Sho Sonoda, Taiji Suzuki, Jinkyoo Park. Atsushi Yamashita, and Hajime Asama. Differentiable multiple shooting layers. Advances in Neural Information Pro- cessing Systems, 34: 6532-16544, 2021.
MJ Menne, CN Williams Jr, and RS Vose. Long-tem term daily climate records from stations across the contiguous united states, 2015.
Hao Ni, Lukasz Szpruch, Magnus Wiese, Shujian Liao, and Baoren Xiao. Conditional sig- wasserstein gans for time series generation. arXiv preprint arXiv:2006.05421 2020.
Jitse Niesen and Trinity Hall. On the global error of discretization methods for ordinary differential equations. PhD thesis, Citeseer, 2004.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: (609.034999 2016.
Alan V Oppenheim. Discrete-tine signal processing. Pearson Education India, 1999.
Alan V Oppenheim and Ronald W Schafer. Digital signal processing(booo), Research supported by the Massachusetts Institute of Technology, Bell Telephone Laboratories, and Guggenheim Foun- dation. Englewood Cliffs, N. J., Prentice-Hall, Inc., 1975. 598 p, 1975.
C-K Peng, Shlomo Havlin, H Eugene Stanley, and Ary L Goldberger. Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series. Chaos: an interdis- ciplinary journal of nonlinear science, 5(1 (1))822-8)) 87, 1995.
Michael Poli, Jinkyoo Park, and Ilija Ilievski. Wattnet: learning to trade fx via hierarchical spatio- temporal representation of highly multivariate time series. arXiv preprint arXiv: 1909.1000,, 2019.
11

---

Under review as a conference paper at ICLR 2023
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising dif- fusion models for multivariate probabilistic time series forecasting. International Conference Machine Learning, pp. 8857-8868 PMLR, 2021.
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Ck- conv: Continuous kernel convolution for sequential data. arXiv preprini arXiv:2102.06611 2021.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irreuularly-sapper time series. Advances in neural information processing systems, 32, 2019.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by emor propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. International Conference on Machine Learning, pp- 19388-19405. PMLR, 2022.
Lawrence F Shampine and Skip Thompson. Stiff systems. Scholarpedia, 2(3):2855, 2007.
Ikaro Silva, George Moody, Daniel Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Com- puting in Cardiology, pp. 245 248. IEEE, 2012.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1.Xiv:2011.13456 2020.
Gerhard Wanner and Ernst Hairer. Solving ordinary differential equations II, volume 375. Springer Berlin Heidelberg New York, 1996.
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial net- works. Advances in neural information processing systems, 32, 2019.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint arXiv:2202.10571. 2022.
Juntang Zhuang, Nicha Dvoreck. Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In International Conference on Machine Learning, pp. 11639-11649, PMLR, 2020.
12


Reviews:
[{"review_id": "sFoy_ojRH0a", "reviewer": "Reviewer_PiDm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an extension of S4 layers [Gu et al. 2021] with latent variables, which are used to define a generative model. \nTo this end, the authors define an auto-regressive prior distribution over latent variables leveraging the S4 parametrisation. Furthermore, the approximate posterior is defined as a product of marginals, and also leverages S4 parametrisation. \n", "summary_of_the_review": "I think this paper has great potential and it is overall good work. However, in the current version, I find several things unclear, and would therefore vote for rejection. Many questions and clarification could be addressed in the rebuttal though.", "strength_and_weaknesses": "While the paper is generally well written, clarification is needed for several details:\n\nFor instance, the model and posterior before Eq. (5) makes assumptions that need explanation: \n- The emission model is both auto-regressive in all observations x and all latent variables z. Can you motivate why you consider the dependence structure on x additionally to z? And why use auto-regressive instead Markovian models which are more common in SSM literature?\n- The inference model assumes that the (approximate) posterior factorises as a product of marginals, which is very restrictive. Why is this choice made as a starting point?\n- Eq. (5) appears to be in conflict with the equation for the approximate posterior above. Whereas above, you assume that the approximate posterior factorises as a product, here you use a factorisation into conditionals (which is not restrictive). Which factorisation did you actually consider? And is Eq. (5) the objective you use in the end? It's not obvious as that is in the background section.\n- In Eq (5), the posterior is conditioned on observation only until some intermediate x_ti, but it should be the full trajectory of observations, otherwise you have an additional gap in the ELBO. This has been discussed e.g. in [1]. (Unless you would do a filtering-based approximation for all the conditionals p(y_t | y_<t), but that seems also not the case.)\n\nAnother thing that confused me is that in the beginning x is used as inputs and y as target observations. And later, x are the observations of the generative model. I think it would be good to either use a different symbol for one, or clarify this better.\n\nI don’t understand Eq. (8). x_tk and z_tk are scalars, and h_tk is the feature vector at time-step t. How does the convolution operation apply here? Maybe you want to write the convolution for the sequence, similar to Eq. (3)?\n\nWhat is H_\\beta? In the inline text above its mentioned that it is the function mapping x, z to h. What is that function? Only dh/dt has been specified, but not h itself, or how it is approximated. Bilinear transformation?\nLater, there is H_\\beta_1 and other subscripts. This is undefined. What is the index?\n\nIn 4.1, before starting the paragraph \"LS4 prior block\", I am missing a connecting sentence. It took me a while to understand that the block parametrises the conditional distribution. Because the next equation has \"y\" as output and there is no other equation for z or mu_z and sigma_z, this paragraph read as unconnected to that equation. This can be streamlined better for the reader.  \n\nI am also a bit worried and uncertain in my understanding about how a latent trajectory is generated. For the posterior, as far as I understand, you can compute the approximate posterior marginals in parallel, because you factorise it as a product of marginals? If so, this should be noted. For the prior, can you only compute the hiddens h and y in parallel, and then have to sample auto-regressively? Is this the reason why there are no forecasting experiments? If this is a limitation, it would be great to state this explicitly. \n\nWhy is the prior on z discrete-time, whereas h is continuous-time? I think this is never mentioned. \n\nEq (11), what is G_yz?  suppose its just an additional matrix. Why this additional transformation here?\n\n“In turn, state–space models do not suffer from stiffness of dynamics as the numerical methods are sidestepped in favor of an exact evaluation of the convolution operator.”\nI would like to learn a bit more about this. As far as I understand, the convolution operator is applied to the *discretized* system. How does this sidestep numerical problems? \n\nHow do you deal with initialisation? From what I understand from the previous work, the Hippo initialisation is one of the key things for the empirical performance. I think this was not discussed, despite its importance. And there is also no theory regarding how to adjust this for the additional latent variables z. \n\nRelated work.\nI think the paper is missing a quite large literature of related work on deep state-space models that are not of the S4 type. Among the vast literature on deep variational state space models, several methods even use conditional linear or linearised state space models, for instance [2,3,4,5].\n\n\nExperimental evaluation.\nThe performed experiments show that the model is able to generate realistic samples even for quite challenging data distributions. This seems quite promising. However, I find the choice of experiments and baselines not ideal. In particular, \n- why not compare to the obvious baseline S4? Both in terms of speed and performance.\n- since the model uses latent variables to express uncertainty, I would have expected to see some experiments and evaluation metrics that would benefit from this, e.g. probabilistic forecasting evaluated with the CRPS score.\nThe current set of experiments leave me somewhat undecided. On the one hand, the model seems to be able to generate realistic data very well. I am not sure if this implies that the latent transition model is also learned properly, or whether there is potentially a short-cut, e.g. using only the initial latent state and ignoring the dynamics. This speculation would be easily debunked through a forecasting experiment. In the experiments in 5.3, the worse performance on Physionet is explained as if the randomness in the latent space was a limitation, but that should be actually the selling point of this model, as certain data just is not perfectly predictable. \n\n\n[1] Bayer et al. 2021, Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models\n[2] Rangapuram et al. 2018, Deep State Space Models for Time Series Forecasting\n[3] Fraccaro et al., 2017, A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning\n[4] Kurle et al. 2020, Deep Rao-Blackwellised Particle Filters for Time Series Forecasting\n[5] Klushyn et al. 2021, Latent Matters: Learning Deep State-Space Models", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jSsN0vXvrlH", "reviewer": "Reviewer_5ELF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed LS4 a sequential latent state space model with the latent state evolving according to a discretized approximation of ODE. The structure of the latent state dynamics induces an efficient convolutional implementation. The model is trained in a VAE framework. Experiment results on challenge datasets including data from stiff systems show strong performance of the model", "summary_of_the_review": "I appreciate the extension of S4 models to its latent variant which leverages both the representation power of sequential latent variable models and the efficiency of S4 models. But I'm especially concerned about the lack of comparison against models with more flexible latent variable models like VRNN and the recently proposed latent SDE.", "strength_and_weaknesses": "Strength:\n\n1. The motivation of introducing a latent variable for S4 model is strong. The model leverages the strong representation power of latent variables and efficient convolutional implementation of S4 Model.\n2. The work compares against a broad set of baseline models. The experiment results show strong performance and efficiency of the proposed model.\n\n\nWeakness:\n1. Some important baselines are missing from the comparison including VRNN[1] and latent SDE[2].\n2. For efficient convolutional implementation, the dynamics of the latent variable are limited to linear dynamics. It’s not clear if such transition dynamics are going to limit the representation power of the latent state. That’s also why I think models with more flexible transition dynamics like VRNN and latent SDE should be compared.\n3. The work reports ELBO in experiments but it is not a tight bound of log-likelihood. IWAE as well as its particle filtering version for sequential latent variable model FIVO are tighter than ELBO and can better reflect how well the generative model fits the data.\n4. The technical novelty of the proposed model is incremental. The model can be viewed as a special case of the existing sequential latent variable model or latent extension of the S4 model. This is a minor concern as the efficiency improvement based on convolution could compensate for the lack of novelty.\n\n\n[1] Chung, Junyoung, et al. \"A recurrent latent variable model for sequential data.\" Advances in neural information processing systems 28 (2015).\n\n[2] Li, Xuechen, et al. \"Scalable gradients for stochastic differential equations.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6292B0RiFY", "reviewer": "Reviewer_rvUd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes LS4 which is a generative model for sequences inspired by the deep state space model (SSM) S4. LS4 performs latent space evolution following a state space ODE and is trained via sequence VAE objectives. State-of-the-art results on selected datasets for continuous-time latent generative models are reported.", "summary_of_the_review": "The paper is well written and easy to follow. Nevertheless, important information needs to be added in the description of the experimental process to be able to correctly assess the presented results.\n\n### Minor Questions\n\n - Why wasn’t the FID used for comparing the generated with the real distribution in section 5.2? It is a well-established metric for generative models and e.g. the bottleneck layer of an Autoencoder could\n   have been used to compute the Frechet Distance.", "strength_and_weaknesses": "\n - [+] The paper is well written and easy to follow. All necessary theoretical and conceptual background information is concisely introduced.\n - [+] The reviewer also appreciates that comprehensive source code is added in the supplementary materials.\n - [-] In several passages of the text “significant” performance gains are claimed although no statistical tests are performed. Furthermore, there is no indication that several replicates per method were\n       performed and there are no error bars in the respective result tables.\n - [-] The authors do not explain how the hyperparameters for the respective methods were selected. In section D only one specific hyperparameter setting of LS4 is explained: how was this selected and what\n       is the exact hyperparameter search space of LS4 and the baseline methods?\n - [-] It is not explained how 4 of 30 datasets were selected in section 5.2. Why weren’t datasets from baseline methods like SaShiMi or SDEGAN selected?\n - [-] Since SaShiMi is specifically designed for modeling raw audio waveforms, wouldn’t S4 or a variant thereof have been a more suitable baseline? This is also briefly discussed by the authors in the last\n       paragraph of D.1.\n - [-] The information in Figure 1 is not clearly illustrated, and the main caption is missing. The plot in (a) is quite overloaded and the y-axis label should be more specific (e.g. “value of x”). It is not\n       clear how well LS4 approximates the ground truth. Is it possible to plot the mean trajectories with standard deviations for a fixed p? Some more information on the FLAME problem would be helpful. How\n       do different trajectories arise given a fixed p, does the variation come from varying initial conditions?\n\n### Questions\n\n - What are the runtimes of SaShiMi and other VAE-based methods in Table 1? Can the authors explain why the runtimes of the baseline methods experience exponential growth, shouldn’t this be on scale\n   O(N^2 L)? Why do the runtimes of Latent ODE drop by increasing the sequence length from 80 to 320? It would also be interesting to compare memory consumption in an experiment since shorter runtime can\n   come at the cost of more overhead and the practical implementation does not necessarily have to follow the theoretical result in Proposition 4.2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject