Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs
Authors: Ilan Naiman
N. Benjamin Erichson
Pu Ren
Michael W. Mahoney
Omri Azencot
Venue: ICLR 2024 poster (2024)
Keywords: Time Series Generation
Koopman Theory; Variational Autoencoder; Generative Modeling
Primary area: generative models
TL;DR: We introduce Koopman VAE (KoVAE) for time series generation, which is based on a novel design for the model prior, and which can be optimized for either regular and irregular training data.

Abstract: Generating realistic time series data is important for many engineering and scientific applications. 
Existing work tackles this problem using generative adversarial networks (GANs).
However, GANs are unstable during training, and they can suffer from mode collapse. 
While variational autoencoders (VAEs) are known to be more robust to the these issues, they are (surprisingly) less considered for time series generation. 
In this work, we introduce Koopman VAE (KoVAE), a new generative framework that is based on a novel design for the model prior, and that can be optimized for either regular and irregular training data. 
Inspired by Koopman theory, we represent the latent conditional prior dynamics using a linear map. 
Our approach enhances generative modeling with two desired features: (i) incorporating domain knowledge can be achieved by leveraging spectral tools that prescribe constraints on the eigenvalues of the linear map; and (ii) studying the qualitative behavior and stability of the system can be performed using tools from dynamical systems theory. 
Our results show that KoVAE outperforms state-of-the-art GAN and VAE methods across several challenging synthetic and real-world time series generation benchmarks. 
Whether trained on regular or irregular data, KoVAE generates time series that improve both discriminative and predictive metrics. 
We also present visual evidence suggesting that KoVAE learns probability density functions that better approximate the empirical ground truth distribution.

Published as a conference paper at ICLR 2024
GENERATIVE MODELING OF REGULAR AND IRREGU- LAR TIME SERIES DATA VIA KOOPMAN VAES
Ilan Naiman
N. Benjamin Erichson LBNL and ICSI eri erichsen@lbll gov
Pu Ren
Ben-Gurion University, UC Berkeley na naimani
LBNL
i @post. wgw..agu............... ac
pren@1bl .ollbl.oo gov
Michael W. Mahoney
Omri Azencot
ICSL, LBNL, and UC Berkeley mmahoney@stat. www.aatw..eeeeeeeeeee.ooooeeeee. edu
Ben-Gurion University
azencotêcs bgu.
ABSTRACT
Generating realistic time series data is important for many engineering and scien- tific applications. Existing work tackles this problem using generative adversarial networks (GANs). However, GANs are unstable during training, and they can suffer from mode collapse. While variational autoencoders (VAEs) are known be more robust to the these issues, they are (surprisingly) less considered for time series generation. In this work, we introduce Koopman VAE (KoVAE), new gen- erative framework that is based on a novel design for the model prior, and that can be optimized for either regular and irregular training data. Inspired by Koopman theory, we represent the latent conditional prior dynamics using a linear map. Our approach enhances generative modeling with two desired features: (i) incorporat- ing domain knowledge can be achieved by leveraging spectral tools that prescribe constraints on the eigenvalues of the linear map; and (ii) studying the qualitative behavior and stability of the system can be performed using tools from dynam- ical systems theory. Our results show that KoVAE outperforms -tate-of-the-art GAN and VAE methods across several challenging synthetic and real-world time series generation benchmarks. Whether trained on regular or irregular data, Ko- VAE generates time series that improve both discriminative and predictive metrics. We also present visual evidence suggesting that KoVAE learns probability density functions that better approximate the empirical ground truth distribution.
INTRODUCTION
Generative modeling is an important problem in modern machine learning (Kingma & Welling, 2014: Goodfellow et al., 2014: Sohl-Dickstein et al., 2015), with recent surge in interest due to results in natural language processing (Brown et al., 2020) and computer vision (Rombach al., 2022; Ramesh et al., 2022). While image and text data have benefited from the recent development of generative models, time series (TS) data has received relatively little attention. This is in spite of the importance of generating TS data in various scientific and engineering domains, including seismology, climate studies, and energy analysis. Since these fields can face challenges in collect- ing sufficient data, e.g., due to high computational costs or limited sensor availability, high-qualit generative models could be invaluable. However, generating TS data presents its own unique set of challenges. First, synthetic TS must preserve the related statistical distribution to fit into downstream forecasting, uncertainty quantification, and classification tasks. Second, advanced objectives such supporting irregular sampling and integrating domain knowledge require models that respect the underlying dynamics (Che et al., 2018; Kidger et al., 2020; Jeon et al., 2022; Coletta et al., 2023).
Several existing state-of-the-art (SOTA) generative TS models are based generative adversarial networks (GAN). For example, TimeGAN (Yoon et al., 2019) learns an embedding space where adversarial and supervised losses are optimized to mimic the data dynamics. Unfortunately, GANs are unstable during training and are prone to mode collapse. where the learned distribution is not sufficiently expressive (Goodfellow, 2016; Lucic al., 2018: Saxena & Cao, 2021). In addition,

---

Published as a conference paper at ICLR 2024
train sets with missing values or more generally, non-equispaced preeullarl-sammpleer train sets, may be not straightforward to support in GAN architectures. For instance, Jeon et al. (2022) combine multiple different technologies to handle tregularly-sampll data, resulting in complex system with many nyper-parameters. These challenges suggest that other generative paradigms should be also considered for generating time series information.
Surprisingly, variational autoencoders (VAEs) are not considered as strong baselines in generative time series benchmarks (Ang al., 2023), although they suffer less from unstable training and mode collapse. In VAE, an approximate posterior is learned via a neural network to match a certain prior distribution that spans the data statistics. Recent methods including TimeVAE (Desai et al., 2021) and CR- VAE (Li et al., 2023) employ variational viewpoint, however, their prior modeling is based on non-sequential standard normal priors. Thus, the learned posterior may struggle to properly rep- resent the latent dynamics (Girin et al., 2021). This issue is further exacerbated when designing advanced sequential objectives. For instance, it is unclear how to incorporate domain knowledge about dynamical systems (c.g., stable dynamics, slow fast converging or diverging dynamics) with unstructured Gaussian prior distributions, One promising direction to modeling latent dynamics is via linear approaches, that recently have been shown improve performance and analysis (Zeng et al., 2023; Orvieto et al., 2023). More generally, this line of work aligns with theoretical and nu- merical tools from Koopman literature (Koopman, 1931; Rowley et al., 2009; Takeishi al., 2017). Koopman theory offers a dual representation for autonomous dynamical systems via linear, albeit mnfnite-dimensionall operators. Harnessing this viewpoint facilitates autoencoder design, yielding finite-dimensional approximate Koopman operators that model the latent dynamics.
In this work, we propose Koopman VAE (KoVAE), novel model that leverages linear latent Koop- man dynamics within VAE setup. KoVAE employs a prior stating that sequential latent data is governed by a linear dynamical system. Namely, the conditional prior distribution of the next latent variable, given the current variable, can be represented using a linear map. Under this assumption, we train an approximate posterior distribution that learns nonlinear coordinate transformation of the inputs to linear latent representation. Our approach offers two benefits: (i) it models the under- lying dynamics and it respects the sequential nature of input data; and (ii) it seamlessly allows one to incorporate domain knowledge and study the qualitative behavior of the system by using spectral tools from dynamical systems theory. Moreover, we integrate into Ko VAV a time-continuou module based on neural controlled differential equations (NCDE) (Kidger et al., 2020) support irregularly- sampled time series information during training. The advantages of VoVAE are paramount as they facilitate the capturing of statistical and physical features of sequential information, and, in addition, imposing physics-constrain and analyzing the dynamics is straightforward.
Contributions.
The main contributions of our work can be summarized as follows:
We propose a new variational generative TS framework for regular and irregular data that is based a novel prior distribution assuming implicit linear latent Koopman representation. Our design and modeling yield a flexible, casy-to-code, and powerful generative TS model.
We show that our approach facilitates high-level capabilities such as physics-constrained TS gen- eration, by penalizing the eigenvalues of the approximated Koopman operator. We also perform stability analysis by inspecting the spectrum and its spread.
We show improved state-of-the-arr results in regular and irregular settings on several synthetic and real-world datasets, often surpassing strong baselines such as TimeGAN and GT-GAN by large margins. For instance, on the regular and irregular discriminative task we report a total mean relative improvement of 58% and 49 % with respect to the second best approach, respectively.
2 RELATED WORK
Our work is related to generative models for TS as well Koopman-basee methods. Both areas enjoy increased interest in recent years, and thus, we focus our discussion on the most related work.
Generative Models for Time Series. Recurrent neural networks (RNNs) and their step-wise pre- dictions have been used to generate sequences in Teacher forcing (Graves, 2013) and Professor forc- ing (Goyal et al., 2016) approaches. Autoregressive methods such WaveNet (van den Oord al., 2016) represent the predictive distribution of each audio sample by probabilistic conditioning on all previous samples, whereas TimeGCI (Jarrett al., 2021) also incorporates GANs and optimizes
2

---

Published as a conference paper at ICLR 2024
transition policy where the reinforcement signal is provided by global energy framework trained with contrastive estimation. Flow-based models for generating time series information utilize nor- malizing flows for an explicit likelihood distribution (Alaa et al., 2020). Long-range sequences were recently treated via state-space models (Zhou et al., 2023).
GANs and VA GAN-based architectures have been shown to be effective for generating TS data. C-RNN-GAN (Mogren, 2016) directly applied GAN to sequential data, using LSTM networks for the generator and the discriminator. Recurrent Conditional GAN (Esteeban et al., 2017) extends the latter work by dropping the dependence on the previous output while conditioning on additional inputs. TimeGAN (Yoon et al., 2019) is still one of the SOTA methods, and jointly optimizes unsupervised and supervised loss terms to preserve the temporal dynamics of the training data during the generation process. COT-GAN (Xu et al., 2020) exploits optimal transport and temporal causal constraints to devise new adversarial loss. Recently, GT-GAN (Jeon et al., 2022) was proposed as one of the few approaches that support irregular sampling, and it employs autoencoders, GANs, and Neural ODEs (Chen et al., 2018). While GAN techniques are more common in generative TS, there are also a few VAE-based approaches. A variational RNN was proposed in (Chung et al., 2015). Time VAB (Desai et al., 2021) models the whole sequence by global random variable with normal Gaussian distribution for a prior, and they introduce trend and seasonality building blocks into the decoder. CR-VAE (Li et al., 2023) learns Granger causal graph by predicting future data from past observations and using a multi-head decoder acting separately on latent variable coordinates. Additional related VAE works include (Rubanova al., 2019; Li et al., 2020; Zhu et al., 2023).
Koopman-based Approaches. Koopman techniques have gained increasing interest over the past two decades, with applications ranging across analysis (Rowley al., 2009; Schmid, 2010; Takeishi et al., 2017; Lusch et al., 2018; Azencot et al., 2019), optimization (Dogra & Redman, 2020; Redman et al., 2022), forecasting (Erichson al., 2019; Azencot et al., 2020; Wang et al., 2023; Tayal et al., 2023), and disentanglement (Berman et al., 2023), among many others (Budišić et al., 2012; Brunton et al., 2021). Most related our work are Koopman-basse based probabilistic models, which have received less attention in the literature. Deep variational Koopman models were introduced in Morton et al. (2019), allowing to sample from distributions over latent observables for prediction tasks. In Srinivasan & Takeishi (2020), the authors sample via Markov Chain Monte Carlo tools over transfer operators. A mean-field variational inference method with guaranteed stability was suggested in Pan & Duraisamy (2020) for the analysis and prediction of nonlinear dynamics. Finally, Han et al. (2022) designed a stochastic Koopman neural network for control that models the latent observables via Gaussian distribution. To the best of our knowledge, our work is the first to combine VAES and Koopman-based methods for generating regular and irregular TS information.
3 BACKGROUND
Below, we discuss background information essential to our method. In App. A and App. B, we also cover VAE and NCDE, used in our work to support mregularly-samplee sequential train sets.
Koopman theory and practice. The underlying theoretical justification for our work is related to dynamical systems and Koopman theory (Koopman, 1931). Let M C I.m be dimeessiiai domain and 9 M + .M be a dynamical system defined by
T++1 = p(xx).
where E M, and t E N is a discrete variable that represents time. Remarkably, under some mild conditions (Eisner et al., 2015), there exists an fimitee-immesiioall operator known as the Koopman operator K. that acts on observable functions ʃ M + C c F and it fully characterizes the dynamics. The operator K.p is given by
(x)) = f p((x))
where f y denotes composition of transformations. It can be shown that K.p is linear, while yp may be nonlinear. Further, if the eigendecompossiton of the Koopman operator exists, the eigenvalues and eigenvectors bear dynamical semantics. Spectral analysis of Koopman operators currently still being researched (Mezié, 2013; Arbabi & Mezic, 2017; Mezic, 2017; Das & Giannakis, 2019). For instance, eigenfunctions whose eigenvalues lie within the unit circle are related to global stabil- ity (Mauroy & Mezić, 2016), and to orbits of the system (Mauroy & Mezić, 2013; Azencot
al.,
3

---

Published as a conference paper at ICLR 2024
Zurr
21/7
NCDE
FisT
embedding
encoder
sampling
decoder
GRU
lincar
re-paramcterization trick
BN
sigmoid
Ap:
#1:T
Koopman
Figure I: A) The posterior is composed of an embedding layer (NCDE), an encoder (GRU + BN), mean/variance computation and sampling (linear + repr. trick), and decoder (GRU + linear sigmoid). B) The prior consists of GRU, linear, a repr. trick layer, and our novel Koopman module.
2013; 2014). In practice, several methods have been devised to approximate the Koopman operator, of which, the dynamic mode decomposition (DMD) (Schmid, 2010) is perhaps the most well-known technique (Rowley et al., 2009). In this context, our work employs a learnable module akin to DMD in the model prior of our KoVAE detailed below. In general, our approach can be viewed as treating the input data as states Tı.r, whereas the learnable linear dynamics we compute in KoVAE are parametrized by the latent observable functional space F.
Sequential VAEs. We denote by TS input 21.7 = . Tr, where Ia E Rd for all t. Below, we focus on a general approach (Girin al., 2021), where the joint distribution is given by
T
p(ULT, Z..) = p(z.. (tt.r))((((11)1)) = -II p(=)ac)) III p(x.).))
where 2t is the prior latent code associated with It, depends on all past variables z<t, and given 2y, one can recover iz via a neural decoder p(xx(=e). The approximate posterior is modeled by
((ITTIETT)
II 0()))
(2)
namely, is the posterior latent code, it is generated using the learned encoder q(2//<20 Zgz), and the dynamic VAE loss is composed of reconstruction and regularization terms and it reads
CVAE =
[1r) KL[g(z11) (1:T) p(21:))
(3)
KOOPMAN VARIATIONAL AUTOENCODERS (KOVAE)
To describe generative model, we detail its sequential posterior in Sec. 4.1, its novel linear Koopman- bassdd prior Sec. 4.2, and its support for incorporating domain knowledge Sec. 4.3.
4.1 VARIATIONAL SEQUENTIAL POSTERIOR
To realize Eq. 2, our approximate posterior (Fig. 1A) is composed of embedding, encoder, sam- pling, and decoder modules. Given the irregular TS Tt,dy, an NCDE embedding layer extracts regularly-sampled TS æι.T, followed by an encoder module consisting of a gated recurrent unit (GRU) layer (Cho et al., 2014) and batch normalization (BN) layer that learn the dynamics and output the latent representation εT... Then, two linear layers produce the posterior mean f(ă) and variance (2e)) for every t, allowing to employ the re-parameterization trick to generate the posterior series 21:7 via
(4) We feed the sampled code to the decoder that includes another GRU, linear layer, and sigmoid activation function. Note that the embedding NCDE layer is not used the regular setting, i.c., when the input TS F1. is regularly sampled.
N (((()) (((()))((())))
4

---

Published as a conference paper at ICLR 2024
4.2 A NOVEL LINEAR VARIATIONAL SEQUENTIAL PRIOR
In general, one of the fundamental tasks in variational autoencoders' design and modeling is the choice of prior distribution. For instance, static VAEs define to be a normal Gaussian distribution, i.e., p(z) = No, T), among other choices, e.g., (Huszár, 2017). In the sequential setting, a common choice for the prior is a sequence of Gaussians (Chung et al., 2015) given by
(5) where the mean and variance are learned with NN, and 21:T represents the latent sequence. Now, the question arises whether one can model the prior of generative TS models p(z1.T) = II= (<z) in Eq. 1 so that it associates better with the underlying dynamics. In what follows, we propose a novel inductive bias leading to a new VAE modcling paradigm.
p(ze(zee) = (N((2<<;;) o" (2<+))))
Prior modeling. In this work. instead of considering a nonlinear sequence of Gaussian as in Eq. 5, we assume that there exists a learnable nonlinear coordinate transformation mapping inputs to a linear latent space. That is, the inputs Ttysty are transformed to #1:T whose dynamics governed by a linear sequence of Gaussians, see inset for an illustration. Formally, the forward propagation in time of žt is governed by matrix, i.e.,
nonlinear prior linear prior
l
to
2x := EAAA [Az;-1],
(6)
for every 1, where A E k is sampled from a space of linear operators A. In practice, we implement the prior (Fig. IB) and produce 21:7 using a gated recurrent unit (GRU) layer, a sampling component, and a Koopman module (Takeishi et al., 2017). Given āg == O. the GRU yields ε that is fed to two linear layers which produce the mean p(2)) and variance o² (2, for every t, allowing to sample & from ž via Eq. 5. To compute A, we construct two matrices Zo. Z with žo:r-1 and ž1:r in their columns, respectively, where =g ~ (1(00;)) o² (0; 0)). Then, we solve the linear system for the best such that AZo 2, similar to DMD (Schmid, 2010). Finally, is defined to be = A
p(z
Training objective. Effectively, it may be that A induces some error, i.e., 21 * & for some e 7.. [1,...;
Thus, we introduce an additional predictive loss term that promotes linearity in the prior latent variables by matching between ž1.T and 21:7, namely,
(F #1:T) Es,r ~p logp(2z17 [irr
Combining the penalties from Sec. and the loss Eq. 7, we arrive at the following training objective which includes reconstruction term, prediction term, and regularization term,
L = -E.. [log p (w [21:))]] + arpea (21:1, Z1T) 3KL[q ( ((2:)))
(8)
where ar, IR+ + are user weights, similar to (Higgins et al., 2016). In App. C, we provide technical details on C, and in App. D, we show that our objective is penalized evidence lower bound (ELBO).
4.3 PHYSICS- CONSTRAINED GENERATION AND ANALYSIS
The matrix A computed in Sec. 4.2 encodes the latent dynamics in a linear form. Thus, it can be constrained and evaluated using spectral tools from dynamical systems theory (Strogatz, 2018). For instance, the eigenvalues X E C.j = 1,.... ki are associated with growth (Ayy > 1) and decay (|Ay < 1), whereas the eigenvectors O; E C encode the dominant modes. Our constrained generation and interpretability results are based on the relation between and dynamical systems.
Often, prior knowledge of the problem can be utilized in the generation of time series information, see c.g., the example in Sec. 5. Our framework allows us directly incorporate such knowledge by constraining the eigenvalues of the system, as was recently proposed in (Berman et al., 2023). Specifically, we denote by C1, C, C C with S several known constant values, and we define the following penalty we add to the optimization that yields A whose eigenvalues are denoted by Aj.
Loig =
Σ | - c,12
5

---

Published as a conference paper at ICLR 2024
where 1Ayl is the modulus of complex number. The loss term 9 can be used during training or inference, or both. For instance, stable dynamical systems as we consider in Sec. 5.2, have Koopman operators with eigenvalues the unit circle, ie.. 1Ayl = 1. Thus, when constraining the dynamics, we can fix some c1,...,6 Cr = 1, and leave the rest to be unconstrained. In addition to constraining the spectrum, we also analyze the spectrum to study the stability behavior of the system, similarly to (Erichson et al., 2019; Naiman & Azencot, 2023), see also Sec. 5.2.
5
EXPERIMENTS
In this section, we detail the results of our extensive experiments. Details related to datasets and baselines are provided in App. E. Our code is available at GitHub.
5.1 GENERATIVE TIME SERIES RESULTS
In what follows, we demonstrate our model's generation capabilities empirically, and we evaluate method both quantitatively and qualitatively in the tests we describe below.
Quantitative evaluation. Yoon et al. (2019) suggested two tasks for evaluating generative time series models: the discriminative task and the predictive task. In the discriminative test, we measure the similarity between real and artificial samples. First, we generate collection of artificial samples 1f and we label them 'fake'. Second, we train a classifier to discriminate between the
fake and the real dataset { Finally, we report the value 1 - acc|, where acc is the accuracy a held-out set. Thus, lower discriminative score implies better generated
of the discriminator data as it fools the discriminator. The predictive task is based on the "train on synthetic, test on real" protocol. We we train predictor the generated artificial samples. Then, the predictor is evaluated on the real data. A lower mean absolute error (MAE) indicates improved predictions.
We consider the discriminative and predictive tasks in evaluation cases: regular, where the entire dataset; and irregular, where we omit a portion the dataset. Specifically, in the irregular setting, we follow Jeon et al. (2022), and we randomly omit 30%, 50%, and 70% of the observations. We show in Tab. 1 and Tab. the discriminative and predictive scores, respectively. Tab. lists at the bottom the relative improvement (RI) of the best score $1 with respect to the second best score 82, i.e., و2 $1 /22. Tab. 2 details the predictive scores obtained when the real data is used in the task (original). Both tables highlight the best scores. The results show that our approach (KoVAE) attains the best scores on both tasks. In particular, KoVAE demonstrates significant improvements on the discriminative tasks, yielding remarkable average RI of 58%. In Tab. 3, we detail the discriminative and predictive scores for the irregular setting. Similarly to the regular case, KoVE presents the best measures, except for predictive Stocks 50% and discriminative Energy 70%. Still, average RIs significant: 48 %, 55 %, and 43% % for the three different irregular settings.
Qualitative evaluation. Next, we use qualitative metrics to examine the similarity of the generated sequences to the real data. We consider two visualization techniques: (i) we project the real and
Table 1: Regular TS, discriminative task
Table 2: Regular TS, predictive task
Method
Sines
Stocks
Energy
MuloCo
Method
Sines
Stocks Energy
MuJoCo
KoVAE (Ours)
0.005
0.009
0.143
0.076
KoVAE (Ours)
0.093
0.037
0.251
0.038
GT-GAN
0.012
0.077
0.221
0.245
GT-GAN
0.097
0.040
0.312
0.055
VAE
0.016
0.036
0.323
0.224
TAE
0.093
0,037
0.254
0.039
TimeGAN
0.011
0.102
0.236
0.409
TimeGAN
0.093
0.038
0.273
0.082
CR-VAE
0.342
0.320
0.475
0.464
CR-VAE
0.143
0.076
0.277
0.050
RCGAN
0.022
0.196
0.336
0.436
RCGAN
0.097
0.040
0.292
0.081
C-RNN-GAN
0.229
0.399
0.499
0.412
C-RNN-GAN
0.127
0.038
0.483
0.055
T-Forcing
0.495
0.226
0.483
0.499
T-Forcing
0.150
0.038
0.315
0.142
P-Forcing
0.430
0.257
0.412
0,500
P-Forcing
0.116
0.043
0,303
0.102
WaveNet
0.158
0.232
0.397
0.385
WaveNet
0.117
0.042
0.311
0.333
WaveGAN
0.277
0.217
0.363
0.357
WaveGAN
0.134
0.041
0.307
0.324
RI
54.54%
75.00% % 35.29%
66.07% %
Original
0.094
0.036
0.250
0.031

---

Published as a conference paper at ICLR 2024
Table Irregular time series (30%, 50% and 70% of observations are dropped)
30%
50%
70%
Sines Stocks Energy
MuloCo
Sines Stocks Energy
MuloCo
Sines
Stocks Energy
MuJoCo
KOVAE (Ours)
0.035
0.162
0.280
0.123
00.030 0.092
0.298
0.117
0.065
0.101
0.392
0.119
GT-GAN
0,363 0.251
0.333
0.249
0.372 0.265
0.317
0.270
00.278
0.230
0.325
0.275
Discriminative Score
TimeGAN-/A AL
0.494
0,463
0,448
0.471
0.496
0.488
0.479
0,483
0.500
0.488
0.496
0.494
RCGAN-At
0.499
0,436
0.500
0.500
0.406 0.478
0.500
0,500
0.433
0.381
0,500
0.500
C-RNN-GAN- At
0,500
0.500
0,500
0,500
0,500
0.500
0.500
0,500
0,500
0.500
0.500
0.500
T-Forcing -At
0.395
0.305
0.477
0.348
0.408
0.308
0.478
0.486
0.374
0.365
0.468
0.428
P-Forcing- At
0.344 0.341
0,500
0.493
0.428
0,388
0.498
0.491
0.288
0.317
0.500
0.498
TimeGAN-D
0.496 0.411
0.479
0.463
0,500
0.477
0.473
0.500
0..998
0,485
0.500
0.492
RCGAN-D
0,500 0.500
0,500
0.500
0,500
0.500
0.500
0,500
0.500
0,500
0.500
0.500
C-RNN-GAN-D
0.500
0,500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0,500
T-Foreing-D
0.408
0.409
0.347
0.494
0.430
0.407
0.376
0.498
0.436
0.404
0.336
0.493
P-Foreing-D
0.500 0.480
0.491
0.500
0.499
0.500
0.500
0.500
0.500
0.449
0.494
0.499
KoVAE (Ours)
0.074
0.019
0.049
0.043
0.072
0.019
0.049
0.042
0.076
0.012
0.052
0.044
GT-GAN
0.099
0.021
0.066
0.048
0.101
0.018
0.064
0.056
0.088
0.020
0.076
0.051
TimeGAN-At
0.145
0.087
0.375
0.118
0.123
0.058
0.501
0,402
0.734
0.072
0.496
0.442
RCGAN-Ar
0.144 0.181
0.351
0.433
0.142 0.094
0.391
0.277
0.218 0.155
0.498
0.222
Predictive Score
C-RNN-GAN-Ar
0.754 0,091
0.500
0.447
0.741
0.089
0.500
0,448
0.751 0.084
0.500
0.448
T-Forcing-4 At
0.116 0,070
0.251
0.056
0.379
0.075
0.251
0.069
0.113 0.070
0.251
0.053
P-Forcing-AL At
0.102 0.083
0.255
0.089
0.120 0.067
0.263
0.189
0.123 0.050
0.285
0.117
TimeGAN-D
0.192 0,105
0.248
0.098
0.169 0.254
0.339
0.375
0.752 0.228
0.443
0.372
RCGAN-D
0.388 0.523
0.409
0.361
0,519
0.333
0.250
0.314
0.404
0.441
0.349
0.420
C-RNN-GAN-D
0.664 0.345
0,440
0.457
0.754
0.273
0.438
0.479
0.632
0.281
0.436
0.479
T-Forcing-D
0.100 0.027
0.090
0.100
0.104
0.038
0.090
0.113
1.2 0.031
0.091
0.114
P-Forcing-D
0.154 0.079
0.147
0.173
0.190
0.089
0.198
0.207
0.278
0.107
0.193
0.191
Original
0.071 0.011
0.045
0.041
0.071
0.011
0.045
0.041
0.071
0.011
0.045
0.041
synthetic data into a two-dimensional space using t S-S (Van der Maaten & Hinton, 2008); and (ii we perform kernel density estimation by visualizing the probability density functions (PDF). In Fig. 2, we visualize the synthetic and real two-dimensional point clouds in the irregular 50% setting for all datasets via t-SNE (top row), and additionally, we visualize their corresponding probability density functions (bottom row). Further, we also show the PDF of GT-GAN (Jeon et al., 2022) in dashed-black curves at the bottom row. Overall, our approach displays strong correspondences in
Sines 50%
Stocks 50%
Energy 50%
MujoCo 50%
10
6-
t-SNE Embedding
6
8-
10
0-
10: 4.
-6-
-8-
synthetic
-6-
-14
14
-14
0
14
-14
0
14
-14
14
Data Density Estimate
5.50
real
KoVAE
GAN
2-
2.75
0.00
0-
0.2
0.8
1.4
-0.4
0.4 Data Value
1.2
0.2
0.5 Data Value
0.8
0.30
0.45
0.65
Data Value
Data Value
Figure 2: We qualitatively evaluate our approach with two-dimensional S- plots of the synthetic and real data (top row). In addition, we show the probability density functions of the real data, and for Ko VAE and GT-GAN synthetic distributions (bottom row).
7

---

Published as a conference paper at ICLR 2024
KoVAE
Constrained KoVAE
1
1
3m(A)
Sm(A)
0
-1
-1
Re(A))
Re(A)
Figure 3: On the left, we show the spectrum of the approximate Koopman operator without con- straints during the training. On the right, we show the spectrum of the approximate Koopman operator for the model that is trained with stability constraints. We can see that, indeed, the absolute value of two eigenvalues is approximately 1.
both visualizations, where in Fig. 2 (top row), we observe a high overlap between real and synthetic samples, and in Fig. 2 (bottom row), the PDFs show similar trend and behavior. On Stocks 50% we identify lower variability in KoVAE in comparison to the real data that present a larger variance. In App. H, we provide additional results for the regular and irregular 30%, and 70% cases.
5.2 PHYSICS- CONSSRRINCD GENERATION
In this section, we demonstrate how to incorporate prior knowledge of the problem when generating time series information. While we could study the Sines dataset, we opted for a more challenging test case, and thus, we focus on the nonlinear pendulum system. Let 1 and = 9.8 denote the length and gravity, respectively. Then, we consider the following ordinary differential equation (ODE) that describes the evolution of the angular displacement from an equilibrium 8,
d² 0
9 sin 0 = 0, 0(0) 0, ((0) 0.
d +
(10)
To generate N different sequences, we uniformly sample 0² (0) 4/..5, 2.7) for j over the time interval t = [0, 17], where the time step is defined by At = 0.1. This results in a set ( IN with T' = 170 and each time sample is two-dimensional, i.e., E IR2 To simulate real- world noisy sensors, we incorporate additive Gaussian noise to each sample. Namely, we sample p N0, and we define the train set samples via He 2, + 0.08 p.
,,...,N
We evaluate the nonlinear pendulum on three different models: (i) a KOVAE with a = 0; (ii) a KoVAE; and (ii) a KoVAE with eigenvalue constraint described in Sec. 4.3. Specifically, we train all models with a fixed latent size x of 2, to be k = 4. For the constrained version, we additionally add Leig:
11 2 | - c,l
(11)
where cp = 1, Cg = 1 and Ap and >g are the largest eigenvalues during training. We define this con- straint since the nonlinear pendulum is a stable dynamical system that governed by two modes. The rest of the eigenvalues, Ae where i * p, q could be additionally constrained to be equal to zero. However, we observe that when we leave the rest of the eigenvalues be unconstrained, we get similar behavior. Fig. 3 shows the spectra of the linear operators associated with Ko and con- strained Ko VAE. We can analyze and interpret the learned dynamics by investigating the spectrum. Specifically, if [Ayl < 1, it is associated with exponentially decaying modes since limy [А]] = 0. When |Λyl 1, the associated mode has infinite memory, whereas |Λyl > is associated with un- stable behavior of the dynamics since lim;-poe |X5| = oc. In Fig. 3, we observe that training without the constraint results in decaying dynamics where all eigenvalues < (yellow). In contrast, analyz- ing our approximate operator reveals that as expected, IXpl and |>g (purple) are approximately one, indicating stable dynamics and the rest (yellow) approximately zero, as desired. Finally, also compute the spectra associated with the computed operators in the regular setting and discussed the results in App. F.
8

---

Published as a conference paper at ICLR 2024
real
KOVAE, ==
KoVAE
constrained KoVAE
12
B 12
12
2.30
Data Density Estimate
D-SNE Emoeda ng
1.15
-12
-12
0.00
10
10
-10
10
-10
0
10
0.0
0.5
1.0
t-SNE Embedding
t-SNE Embedding
t-SNE Embedding
Data Value
Figure 4: The t SNE plots of KoVAE with a = 0 (A), KoVAE (B), and constrained Ko (C),
and their probability density functions (D) are compared to the true nonlinear pendulum data.
In addition to analyzing the spectrum, we visualize in Fig. 4 the t-SNE plots of the real data in comparison the data generated using the three trained models, KoVAE with a 0 in Fig. 4A, KoVAE in Fig. 4B, and constrained Ko VAE in Fig. 4C. Further, we also plot the probability density functions of the three models in Fig. 4D. The results clearly show that the synthetic samples gener- ated with the constrained KoVAE (red) better match the distribution and PDF of the true process in comparison KoVAE (green) and KoVAE with a (gray).
5.3 ABLATION STUDY
We ablate our approach on the discriminative task in the regular and 30% and 50% irregular settings. We eliminate the linear prior by fixing a = O, and we also train another baseline without the linear prior and without the recurrent (GRU) component. Tab. 4 details our ablation results, and we observe that Ko VAE outperforms all other regular ablation baselines and the majority of irregular baselines. We conclude that by introducing novel linear dynamical prior. VAE improves generative results.
Table Discriminative ablation results with regular and irregular 30% and 50% data on our method (KoVAB), without the linear prior (a = 0), and also without the RNN (a O. no GRU).
regular
30%
50%
Sines Stocks Energy MuloCo Sines Stocks Energy MuJoCo
Sincs Stocks Energy MuloCo
KoVAE
0.005
0.009
0,143
0.076
0.035
0.162
0,280
0.123
0.030
0.092
0.298
0.1
0.006 0.023
0.155
0.087
0.038
0.172
0.295
0.131
0.040
0.181
0.306
0.108
0, no GRU
0.268
0.272
0.436
0.267
0.275
0.225
0.444
0.305
6 CONCLUSION
Generative modeling of time series data is often modeled with GANs which are unstable to train and exhibit mode collapse. In contrast, VAEs are more robust to such issues, however, current generative time series approaches employ non-sequential prior models. Further, introducing con- straints that impose domain knowledge is challenging these frameworks. We propose Koopman VAE (KoVAB) a new variational autoencoder that is based on a novel dynamical linear prior. Our method enjoys the benefits VAEs, it supports regular and irregular time series data, and it fa- cilitates the incorporation of coossis-conssrrints, and analysis through the lens of linear dynamical systems theory. We extensively evaluate our approach on generative benchmarks in comparison to strong baselines, and we show that KoVAE significantly outperforms existing work on several quan- titative and qualitative metrics. We also demonstrate on real-world challenging climate dataset that KoVAE approximates well the associated density distribution and it generates accurate temperature long-term signals. Future work will explore the utility of KoVAE for scientific and engineering problems in more depth.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
OA was partially supported by an ISF grant 668/21, an ISF equipment grant, and by the Israeli Council for Higher Education (CHE) via the Data Science Research Center, Ben-Gurion University of the Negev, Israel. MWM would like to acknowledge NSF and ONR for providing partial support of this work. NBE would like to acknowledge NSF, under Grant No. 2319621, and the U.S. Depart- ment of Energy, under Contract Number DE-AC22-05CHI 1231, for providing partial support of this work. IN was partially supported by the Lynn and William Frankel Center of the Computer Science Department, Ben-Gurion University of the Negev. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and official endorsement should be inferred.
REFERENCES
Ahmed Alaa, Alex James Chan, and Mihaela van der Schaar. Generative seme-series modeling with Fourier flows. In International Conference on Learning Representations, 2020.
Yihao Ang, Qiang Huang, Yifan Bao, Anthony KH Tung, and Zhiyong Huang. TSGBench: time series generation benchmark. arXiv preprint arXiv:2309.0375: 2023.
Hassan Arbabi and Igor Mezic. Ergodic theory, dynamic mode decomposition, and computation of spectral properties of the Koopman operator. SIAM Journal Applied Dynamical Systems, 16 (4):2096-2126, 2017.
Omri Azencot, Mirela Ben-Chen, Frédéric Chazal, and Maks Ovsjanikov. An operator approach to tangent vector field processing. In Computer Graphics Forum, volume 32, pp. 73-82. Wiley Online Library, 2013.
Omri Azencot, Steffen Weißmann, Maks Ovsjanikow, Max Wardetzky, and Mirela Ben-Chen. Func- tional fluids on surfaces. In Computer Graphics Forum, volume 33, pp. 237-246. Wiley Online Library, 2014.
Omri Azencot, Wotao Yin, and Andrea Bertozzi. Consistent dynamic mode decomposition, SIAM Journal on Applied Dynamical Systems, 1863):1565-1588 2019.
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent Koopman autoencoders. In International Conference Machine Learning, pp. 475-485, PMLR, 2020.
Nimrod Berman, Ilan Naiman, and Omri Azencot. Multifactor sequential disentanglement via struc- tured Koopman autoencoders. In The Eleventh International Conference Learning Represen- tations, ICLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Steven L Brunton, Marko Budišié, Eurika Kaiser, and J Nathan Kutz. Modern Koopman theory for dynamical systems. SIAM Rev., 2021.
Marko Budišić, Ryan Mohr, and Igor Mezić. Applied koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science, 22(4), 2012.
Luis Candanedo. Appliances energy prediction. UCI Machine Learning Repository, 2017.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085,
2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
10

---

Published as a conference paper at ICLR 2024
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing, EMNLP, pp. 1724-1734. ACI., 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. Advances in neural information pro- cessing systems, 28, 2015.
Andrea Coletta, Sriram Gopalakrishan, Daniel Borrajo, and Svitlana Vyetrenko. On the constrained time-series generation problem. arXiv preprint arXiv:2307.01717, 2023.
Suddhasattwa Das and Dimitrios Giannakis. Delay-coordinate maps and the spectra of Koopman operators. Journal of Statistical Physics, 175(6):107-1145 2019.
Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Time variational auto- encoder for multivariate time series generation. arXiv preprint arXiv:2111.08005 2021.
Carl Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05900 2016.
Akshunna S Dogra and William Redman. Optimizing neural networks via Koopman operator theory.
Advances in Neural Information Processing Systems, 33:2087-2097, 2020.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In International Conference Learning Representations, 2019.
Tanja Eisner, Bálint Farkas, Markus Haase, and Rainer Nagel. Operator theoretic aspects of ergodic theory, volume 272. Springer, 2015.
N Benjamin Erichson, Michael Muchlebach, and Michael W Mahoney. Physics-ifformed autoen- coders for Lyapunov-stable fluid flow prediction. arXiv preprint a90iv: 1006666 2019.
Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional GANs. arXiv preprint 1rXiv:1706.000 02633, 2017.
Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, and Xavier Alameda- Pineda. Dynamical variational autoencoders: A comprehensive review. Found. Trends Mach. Learn., 15(1-2):1-155, 2021.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks.
arXiv preprint
17X1v:1701.00160. 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Advances in neural information processing systems, 27, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Ben- gio. Professor forcing: new algorithm for training recurrent networks. Advances in Neural Information Processing Systems 29, pp. 4601-4609, 2016.
Alex Graves. Generating sequences with recurrent neural networks. (0ti:1138.0888 2013.
arXiv preprint
Minghao Han, Jacob Ruler-Rolle, and Robert K. Katzschmann. DeSKO: stability-assured robust control with deep stochastic Koopman operator. In The Tenth International Conference Learning Representations, ICLR, 2022.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabatee Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The ERAS global reanalysis. Quarterly Journal of the Royal Meteorological Society, 1499730):199--02 2020.
11

---

Published as a conference paper at ICLR 2024
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2016.
Ferene Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: 2017.
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Time-series generation by contrastive imi- tation. Advances in Neural Information Processing Systems, 14:22968-289885 2021.
Jinsung Jeon, Jeonghak Kim, Haryong Song, Seunghyeon Cho, and Noseong Park. GT-GAN: gen- eral purpose time series synthesis with generative adversarial networks. Advances in Neural Information Processing Systems, 35:36999-37010, 2022.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 2020.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICIR, 2014.
Bernard O Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences, 17(5):315-11, 1931.
Hongming Li, Shujian Yu, and José C. Príncipe. Causal recurrent variational autoencoder for med- ical time series generation. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI, pp. 8562-8570, 2023.
Xuechen Li, Ting-Kan Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882. PMLR, 2020.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? a large-scale study. Advances in neural information processing systems, 31, 2018.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications,, (1):4950 2018.
Alexandre Mauroy and Igor Mezić. A spectral operator-theoreti framework for global stability. In 52nd IEEE Conference on Decision and Control, 5234-5239. IEEE, 2013.
Alexandre Mauroy and Igor Mezić. Global stability analysis using the eigenfunctions of the Koop- man operator. IEEE Trunsactions on Automatic Control, (111))33356333 2016.
Igor Mezić. Analysis of fluid flows via spectral properties of the Koopman operator. Annual Review of Fluid Mechanics, 45:357-378, 2013.
Igor Mezic. Koopman operator spectrum and data analysis. arXiv preprint 1rXhv:1702.075977 2017.
Olof Mogren. C-RNN-GAN: continuous recurrent neural networks with adversarial training. arXiv preprint 161 1.166/..00 109904, 2016.
Jeremy Morton, Freddie D. Witherden, and Mykel J. Kochenderfer. Deep variational Koopman models: inferring Koopman observations for uncertainty- dynamics modeling and control. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence, IJCAI, pp. 3173-3179, 2019.
Ilan Naiman and Omri Azencot. An operator theoretic approach for analyzing sequence neural networks. Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 9268- 9276, 2023.
Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülgehre, Razvan Pas- canu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference Machine Learning, JCML., volume 202, pp. 15670-26695 2023.
12

---

Published as a conference paper at ICLR 2024
Shaowu Pan and Karthik Duraisamy. Physics-informed probabilistic learning of linear embeddings of nonlinear dynamics with guaranteed stability. SIAM Journal on Applied Dynamical Systems, 19(1):880-500 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint (xXiv:2204.06122 1(2):3 2022.
William T Redman, Maria Fonoberova, Ryan Mohr, loannis G Kevrekidis, and Igor Mezić. An operator theoretic view on pruning deep neural networks, In The Tenth International Conference on Learning Representations, ICLR, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Clarence W Rowley, Igor Mezić, Shervin Bagheri, Philipp Schlatter, and Dan S Henningson. Spec- tral analysis of nonlinear flows. Journal of fluid mechanics, 641:115-122 15-177, 2009.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for srapule time series. Advances in neural information processing systems, 32, 2019.
Divya Saxena and Jiannong Cao. Generative adversarial networks (GANs) challenges, solutions, and future directions. ACM Computing Surveys (CSUR), 54(3):1-42, 2021.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656:5-28, 2010.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Anand Srinivasan and Naoya Takeishi. An MCMC method for uncertainty set generation via operator-theoretic metrics. In 59th IEEE Conference on Decision and Control, CDC, pp. 2714- 2719, 2020.
Steven H Strogatz. Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering. CRC press, 2018.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning Koopman invariant subspaces for dynamic mode decomposition. Advances in neural information processing systems, 30, 2017.
Kshitij Tayal, Arvind Renganathan, Rahul Ghosh, Xiaowei Jia, and Vipin Kumar. Koopman in- vertible autoencoder: Leveraging forward and backward dynamics for temporal modeling. arXiv preprint arXiv:2309.10220 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1 0..100//ROS.201 2. 63666009.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. WaveNet: generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop, pp. 125, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Rui Wang, Yihe Dong, Sercan Ö. Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. In The Eleventh International Conference Learning Repre- sentations, ICLR, 2023.
Tianlin Xu, Li Kevin Wenliang, Michael Munn, and Beatrice Acciaio. COT-GAN: generating quential data via causal optimal transport. Advances in neural information processing systems, 33:8798-8809, 2020.
13

---

Published as a conference paper at ICLR 2024
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial net- works. Advances in neural information processing systems, 32, 2019.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, pp. 11121-11128, 2023.
Linqi Zhou, Michael Poli, Winnie Xu, Stefano Massaroli, and Stefano Ermon. Deep latent state space models for time-series generation. In International Conference on Machine Learning, pp. 42625-42643, PMLR, 2023.
Harrison Zhu, Carles Galsells-Rodaa and Yingzhen Li. Markovian Gaussian process variational au- toencoders, In International Conference on Machine Learning, pp. 42938 42981. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2024
A STATIC VARIATIONAL AUTOENCODERS.
We recall a few governing equations from (Kingma & Welling, 2014; Doersch, 2016). In genera- tive modeling, we want to compute the probability density function (PDF) P(z) using maximum likelihood framework. The density P(x) reads
P(x) - p(x| 0)p(2)
(12)
where: latent representation associated with z. The model p(1|2; 0) ideally approaches for some z. In variational inference, we typically have that p(r| 0) = (=; I), i.e., Gaussian distribution where the mean is generated with a neural network f. However, two issues arise with the above PDF: 1) how to define and 2) how to integrate over z.
VAES provide a solution for both issues. The first issue is resolved by assuming that z is drawn from predefined distribution, NO,, 1). The second issue can be potentially solved by approx- imating Eq. 12 with sum, i.e., P(x) & Σ, p(x]=,).
challenge, however, that n needs to be huge in high-dimensions. Instead, observe that for most 2, the term p(z[2) will be nearly zero, and thus we can focus on sampling x that are likely to produce x. Consequently, we need an approximate posterior (==(xr) which allows compute Eg-gp( (ar where is obtained using the re-parametrivation trick.
To guide training so that the prior and approximate posterior match, employ the Kullback-Lieble Divergence KL between g(=) and p(z(z)). The model is trained using the evidence lower bound (ELBO) loss Ez-g [(<|=]] KL[g(=|<)) (xr) p(=)], which is one of the core equations of VAB. Notice that the objective takes the form of reconstruction and regularization penalties, respectively.
B NEURAL CONTROLLED DIFFERENTIAL EQUATIONS
Irregularly sampled TS information d for, e.g., with discrete-time architectures such as recurrent neural networks (RNN). Therefore, the contin- uous analog of RNN is considered for Tiyty based on neural controlled differential equations (NCDE) (Kidger et al., 2020) that are given by
TJ] cannot be modeled directly
J. pl+t f(h(t): ((N()):))dX() 0))
((++1)= h(t)))
(13)
where 11(44) is the hidden code associated with Ft, (t) is a continuous-time trajectory generated from st via an interpolation method with (t+) := It,, and f is a neural network parametrized by Of whose role is to learn the latent infinitesimal factor.
C LOsS FUNCTION EVALUATION
Our model is trained using the objective function C, defined in Eq. 8, and repeated below,
C = -E pt [ + aCprre Æir) - [KL[g(<11111 (x),tx)) ) p(()r)),
where the first addend is the reconstruction term, the second addend is the predictive loss term, and the last addend is the KL regularization term. To evaluate L in practice, assumptions, followed by straightforward computations. First, we assume that p( ( follows Gaussian distribution whose mean is the output of the model and its variance is some constant. Namely, p(Z R1rr) = N (x1r:: @irr(Zer, 0), o²), where (ie((2rr) denotes the output of the decoder whose learnable parameters are given by 0. This is common probabilistic assumption (Goodfellow et al., 20) under which the term Bz (log p(11:r [1:rr) becomes simple mean squared error (MSE) between F1.T and T.. Second, a similar reasoning is applied to Cpred. yielding an MSE evaluation between 11.7 and ă Finally, the regularization term involves the distribution p(21.)) := 5( (21:7 - A )p( (1:r), ). where 5(.) is the Dirac delta distribution. However, the KL divergence is always evaluated in our framework
make a few standard
batches 21.T computed from ΣT.7. Therefore, we approximate 6(21. - A by 1, leading to ((( se p(λ)r). Then, we can compute the KL term using closed-form formulation given the mean and variance of two distributions.
15

---

Published as a conference paper at ICLR 2024
D VARIATIONAL PENALIZED EVIDENCE LOWER BOUND
In what follows, we present the derivation of the variational penalized evidence lower bound for our method. Our objective is minimize [1 under the Koopman con- straint (2ge ReT) = Gagr=pllogp(2277/3
KLoleerleer)llllerrlsssrr) + + Be bogroploogppss
/ ((ggrlag)) log ((2gr(=gr) degr Bogropoogp(2er11 p(zcr|agr)
/ ((sgr(eg)) log
( (er) dzer +
p(2gT, IST)
/
((<TT(ETT) dzgr + Bagr=-
log
p (gT)
(=gT] (gT)
logp(zsr) ((gg)lagr)) log
dzer E Begn=ploogp
(ggT
( IgT) dzgr Bagroplogppasrrk22
logp(=<r) ((2gr|<5)) log
m(zs)p((egr(2s7)
logp(zgr) ((2grlægr)]og [(ggT|ag])
logg(ggrl (egg l|
((gT)
logg(=zs) B loog ((ggrlagr) log (ogeg:))
p(2ST)
-lgy(<gr)) + K /agp::/::/ppppo/oaaa/eeee/e////e (eer + Bagrop
Thus we have the following:
logp(#şr) + |
+ -Bggrgloopppptprekee
Now, rearranging the equation, we yield:
logp(#şr) - KLlg(LLLlllaaa_e le
Thus finally:
logp(æşr) + Bogropp [gT)]
(14)
Eg Bog + KLig(zer) |agp + Boggr=l
16

---

Published as a conference paper at ICLR 2024
E DATASETS AND BASELINE METHODS
We consider four synthetic and real-world datasets with different characteristic properties and statis- tical features, following challenging benchmarks on generative time series (Yoon al., 2019; Jeon et al., 2022). Sines, is multivariate simulated data, where each sample =ī( := sin (2TTtt + f), with 1j ~ [0, 1], ~ U([-T, x|. and channel j E (I,...: 5) for every i in the dataset. This dataset is characterized by its continuity and periodic properties. Stocks, consists of the daily historical Google stocks data from 2004 to 2019 and has six channels including high, low, opening. closing, and adjusted closing prices and the volume. In contrast to Sines, Stocks is assumed to include ran- dom walk patterns and is generally aperiodic. Energy, is a multivariate UCI appliance energy prediction dataset (Candanedo, 2017), with 28 channels, correlated features, and it is characterized by noisy periodicity and continuous-van valued measurements. Finally, MuJoCo (Multi-Join dynam- ies with Contact) (Todorov et al., 2012), is peneral-purpose physics generator, which we use to simulate time series data with 14 channels.
We compare our method with several state-of-the-art (SOTA) generative time series models, such as TimeGAN (Yoon et al., 2019), RCGAN (Esteban et al., 2017), C-RNN-GAN (Mogren, 2016), WaveGAN (Donahue et al., 2019), WaveNet (van den Oord et al., 2016), T-Foreing (Graves, 2013), P-Forcing (Goyal et al., 2016), Time (Desai et al.. 2021), CR- VAE (Li et al., 2023), and the recent irregular method GT-GAN (Jeon al., 2022). All the methods, except for GT-GAN, are not designed to handle missing observations; thus, we follow GT-GAN and compare them with their re-designed versions. Specifically, extending regular approaches to support irregular TS requires the conversion of dynamical module to its time-continuous version. For instance, we converted GRU layers to GRU-At and GRU-D to exploit the time difference between observations and to learn the exponential decay between samples, respectively. We denote the re-designed methods by adding At or D postfix to their name, such as TimeGAN-AI and TimeGAN-D.
F CONSTRAINED GENERATION AND ANALYSIS
In addition to the analysis of the constrained system in Eq. 10, we also compute the approximate Koopman operators for all the regular systems we consider in the main text. Specifically, we show in Fig. 5 four panels, corresponding the spectral eigenvalue plots for Sines (left), Stocks (middle left), Energy (middle right), and MuJoCo (right). Green cigenvalues are within the unit circle, whereas red cigenvalues are located outside the unit circle. Noticeably, the learned dynamics for Sines, Stocks and MuJoCo are stable, i.e., their corresponding eigenvalues are within the unit circle, and thus, their training, inference and overall long-term behavior is expected be numerically stable and not produce values that present large deviations. In contrast, the spectrum associated with the model learned for the Energy dataset reveals unstable dynamics where the largest X; >>> 1. We hypothesize that we can benefit from incorporating spectral constraints to regularize the dynamics and achieve stable dynamics. In the future, we will investigate whether spectral constraints can enhance the generation results and overall behavior of such models.
Sines
Stocks
Energy
MujoCo
Slm(A)
-1
-1
-1
-1
0
1
-1
Re(A)
Re(AA)
Re(A)
Re(A)
Figure 5: The spectral distribution of the approximate Koopman operator of the prior for each dataset in the regular setting.
17

---

Published as a conference paper at ICLR 2024
Monitoring stations Points interest.
Figure 6: Comparison of the aenporally-averaged temperature distribution of the generated signals and ground truth data in California region. left: California region map. middle: ground truth average. right: generated average.
G CONDITIONAL GENERATION FOR WEATHER DATA
Accurate weather data is essential for various applications, such as agriculture, water resource man- agement, and disaster early warning. However, due to the limited availability of weather stations, scientists are faced with the challenge of the inherent sparsity of observational weather data, espe- cially in specific geographical areas of interest. Generative models can produce synthetic weather data to supplement sparse or incomplete observational datasets. In this paper, we explore the poten- tial of Ko model for modeling temperature dynamics based sparse measurement data. Our objective is to generate the temperature data at any location within a specific region of interest. Hence, the proposed KoVAE model here conditioned the geospatial coordinates of the spe- cific region. By leveraging conditional generative models, we hope to provide valuable insights into weather dynamics in specific regions and enhance the -aaiing processes that are reliant on accurate weather information. Our model is trained with geospatial coordinates as conditional vari- ables. To support the conditional training and generation, we made two simple modifications: (i) we add very simple MLP to support spatial embeddings s; and (ii) we augment the decoder and prior to support conditional generation: p( at [21,T, s) and p(x 21:7, s) = p( 21:T (s)p(( (rr.. s).
We focus on two representative regions in the United States: California and Central America areas. California has diverse and non-stationary weather changes due to the complex interactions between seas, lands, and mountains. Central America areas present more stable and relatively easier tem- perature dynamics compared with the California region. Both of these regions include grid of 80 80, which means we have 6400 time series samples in total. Moreover, the dataset in this task comprises temperature at 2-meter height above the surface from ERA5 reanalysis dataset (Hersbach al., 2020). We select 4-month observation data within the specific area for training and evaluat- ing our VAE model. It contains 120 time steps for each time series sample. We split the train and test with a ratio of 80% and 20%. The generated time series samples shown in Fig. 9 and Fig. 10. We can see that our KoVAE model can accurately capture the underlying dynamics of tem- perature data in both California and Central America regions. Furthermore, Fig. and Fig. 7 exhibit temperature distribution between the generations and
the comparisons of the temporally- ground truth in California and Central America regions. The left panels in these figures show the points of interest with respect to the given monitoring stations, which can be used to condition our model. The temperature distribution patterns the generations align well with those of the ground truth in the entire domain. Similarly, we also observe that the generation and ground truth match well when comparing minimum and maximum values over time, as shown in Fig. Nevertheless, the fine-scale spatial details are lacking in our generation due the absence of spatial constraints. In the future, we will enhance our conditional Ko model by incorporating spatial continuity and prior knowledge for spatiotemporal generation.
18

---

Published as a conference paper at ICLR 2024
Mondoding Fations
Figure 7: Comparison of the temporall--aeeraged temperature distribution between the generations and ground truth in Central America region. left: Central America region map. middle: ground truth average. right: generated average.
H ADDITIONAL GENERATION RESULTS
For brevity, in the main text, we provided tables without standard deviation (std). In this section, we provide extended tables that include the std for each metric. Tab. 6 and Tab. 7 present the results for the regular observed data of discriminative and predictive scores, respectively. In Tab. 8, Tab. and Tab. 10 we show the results for the irregular sampled of 30%, 50%, and 70% missing observation, respectively. Each table includes both discriminative and predictive scores. Furthermore, we pro- vide qualitative results to compare our model with GT-GAN, the second-best generation model for both regularly and irregularly sampled data. In Fig. 11 Fig. 12, Fig. 13, we visualize the t-SNE pro- jection of both ground truth data and generated data (first row) and the PDF of ground truth data generated data (second row). We perform the visualization for all regularly and irregularly sampled datasets.
A
E
F
Figure 8: The maximum and minimum plots for both California and Central America regions. (A) The maximum value over time of the ground truth in the California region. (B) The maximum value over time of the generation in the California region. (C) The maximum value over time of the ground truth the Central America region. (D) The maximum value over time of the generation in the Central America region. (E) The minimum value over time of the ground truth in the California region. (F) The minimum value over time of the generation in the California region. (G) The minimum value over time of the ground truth in the Central America region. (H) The minimum value over time of the generation in the Central America region.
19

---

Published as a conference paper at ICLR 2024
310
M
300
300
290
290
280
Ground truth
Ground truth
280
Generated
Generated
270
20
40
60
80
100
120
20
40
60
80
100
120
300
290
Ground truth
290
Generated
280
MM
280
270
270
Ground truth
260
Generated
260
20
40
60
80
100
120
20
40
60
80
100
120
300
Ground truth
WWW
290
Generated
290
280
270
288
Ground truth
260
Generated
20
40
60
80
100
120
20
40
60
80
100
120
294
Ground truth
300
Ground truth
Mmm M
Generated
Generated
292
M
290
280
290
270
260
20
40
60
80
100
120
20
40
60
80
100
120
300
WM
300
Ground truth
290
Generated
290
280
280
Ground truth
270
270
Generated
260
20
40
60
80
100
120
20
40
60
80
100
120
Figure 9: The representative generations in the California area.
COMPUTATIONAL RESOURCES
Here we compare the model complexity (number of parameters), and the wall clock time mea- surement with GT-GAN. We present the details in Tab. 5. Both models were trained the same software and hardware for fair comparison. The software environments we use are: CentOS Linux 7 (Core) and PYTHON 3.9. 16, and the hardware is: NVIDIA RTX 3090. We observe that
20

---

Published as a conference paper at ICLR 2024
300
300
MWW mn Mn m Mad
WWW
Ground truth Generated
280
280
260
Ground truth Generated
260
MM
20
40
80
100
120
20
40
60
80
100
120
300
Ground truth
Ground truth
290
290
Generated
Generated
280
280
270
270
260
260
20
40
100
120
20
40
100
120
300
Pammy
290
1 milh
290
280
270
280
Ground truth
Ground truth Generated
Generated
260
20
40
80
100
120
20
40
60
100
120
300
Ground truth
Generated
280
yom IM M
290
260
My
Ground truth Generated
280
0
20
40
60
80
100
120
20
40
60
80
100
120
300
Ground truth
290
290
Generated
280
280
270
270
Ground truth Generated
260
wM
260
250
20
40
60
80
100
120
20
40
60
80
100
120
Figure 10: The representative generations the Central America area.
more parameters on the Energy and MuJoCo datasets, but the training time significantly faster on the same software and hardware.
21

---

Published as a conference paper at ICLR 2024
Method
Copm.
Sines
Stocks
Energy
MuJoCo
# Parameters
32,929
33,410
230,860
195,902
KoVAE (Ours)
Wall Clock Time
2h 49m
2h 49m
5h 16m
3h 46m
# Parameters
41,913
41,776
57,104
47,346
GT-GAN
Wall Clock Time 10h 12m
12h 20m
10h 39m
13h 12m
Table 5: Comparison of number (#) of parameters and wall clock time.
Stocks
Energy
MujoCo
10
1:5NE Embesding
-10
Syrthet
-18
18
-19
19
-15
15
-17
17
Data Density Estimate
5.50
Origina
7.0
14
KiVAS (Ours)
GT-GAN
2
2.75
3.5
0.4
0.9
0.9
0.30
0.45
0.65
Data Value
Data Value
Data Value
Data Value
Figure 11: We qualitatively evaluate our approach on regularly sampled data with two- wwo-dimensional t-SNE plots of the synthetic and real data (top row). In addition, show the probability density functions the real data, and for KoVAE and GT-GAN synthetic distributions (bottom row).
J HYPERPARAMETERS ROBUSTNESS
We also explore how stable our model is to hyperparameter choice. To this end, we perform an extensive grid search over the following space for:
a, (1.0, 0.9, 0.7, 0.5, 0.3, 0.1, 0.09, 0.07, 0.05, 0.03, 0.01, 0.009, 0.007, 0.005, 0.003, 0.001, 0.0009, 0.0007, 0.0005)
for the stocks dataset. Fig. 14 shows the discriminative score for each combination of a and Most of the values are lower than the second- best model for this task.
K RECONSTRUCTION RESULTS
In addition to studying the generative capabilities of KoVAE, we also investigate its reconstruction and inference features below. We show Fig. 15 the reconstructed signals the regular setting. Each subplot represent separate feature, where for datasets with more than five channels, we plot the first five features. Solid lines represent ground-truth truth data, whereas dashed lines are the recon- structions our model outputs. For simplicity, we omitted axis labels. For all subplots, the r-axis represents time, and the y-axis is the feature values. Moreover, we also plot in Fig. 16 the inferred signals in the irregular 50% setting. In this case, half of the signal is removed during training, and thus, the reconstructions present the inference capabilities of our model. Our results indicate that Ko VAE is able to successfully reconstruct and infer the data.
22

---

Published as a conference paper at ICLR 2024
Sines 30%
Stocks 30%
Energy
MujoCo 30%
8
109
(-SNE Embedding
10-3.
-10
Synthetic
-8
-18
18
-19
19
-15
-17
17
Data Density Estimate
5,50
Original
8
17.0
KoVAE (Ours]
GT GAN
9.5
2.75
4
0.00
0.0
0.2
0.8
1.4
-0.20
0.55
1.30
0.2
0.5 Data Value
0.8
0.300
0.475
0.650
Data Value
Data Value
Data Valuee
Figure 12: We qualitatively evaluate our approach on 30% irregularly sampled data with two- dimensional t-SNE plots of the synthetic and real data (top row). In addition, we show the probability density functions of the real data, and for KoVAE and GT-GAN synthetic distributions (bottom row).
Sines 70%
Stacks 70%
Energy 70%
MujaCo 70%
8
10c
L:SNE Embedding
10-2.
Originel Synthetic
-10
-8
-18
18
-19
19
-17
Data Density Estimate
5,50
7.50
13.0
Original
41
2.75
KoVAE [Ours)
3.75
6.5
GT GAN
0.00
0.00
0.0
0.2
0.7
1.2
0.11
0.8
1.7
0.2
0.5 Data Value
0.8
0.300
0.475
0.650
Data Value
Data Value
Data Valuee
Figure 13: We qualitatively evaluate our approach on 70% irregularly sampled data with two- dimensional t-SNE plots of the synthetic and real data (top row). In addition, we show the probability density functions of the real data, and for KoVAE and GT-GAN synthetic distributions (bottom row).
23

---

Published as a conference paper at ICLR 2024
10
-019
-016
10-3
0.23
0.30
20
0.07
10-1
10-3
10a4 10+4
Figure 14:
Each point this plot represents the discriminative score for stocks regularly sampled data when trained with specific a and values. The marker denotes the best discriminative score for some = 9e-3, 3, 3 = 9e-4 values. In addition, most of the values the figure present better performance than the second-best method.
Table Regular TS, discriminative task
Method
Sines
Stocks
Energy
MuJoCo
KoVAE (Ours)
.005..003
.009:.006
.143+:.011
.076+.001
GT-GAN
.012=.014 .077=.031
.221:.068
.245:.022
TimeGAN
.011=.008 .102=.021
.236:t.012
.409:.028
Time
.016:.010 .036=.033 .323#.029 .2244.026
CR-VAE
.322..177 .320+.095
.475+.054
.644.012
RCGAN
.022+0008 .196+.027
.336+t0017
.436+t001
C-RNN-GAN
.2291.040 .3992.028
.499:.001
.412:.095
T-Foreing
.455.001 .2261.0355
.4831.004
.4994.000
P- P-Forcing
430+.227 .257=.026
412+.006
.500+.000
WaveNet
.158+.011
.232+.028
.397:.010
.385..025
WaveGAN
.27+-.013
.277
363+.002
.357:.017
RI
54.54%
75.00%
35.29%
66.07%
Table 7: Regular TS, predictive task
Method
Sines
Stocks
Energy
MuloCo
KoVAE (Ours)
.093:.000
.037:.000
.251=.000
.038+.002
GT-GAN
.097+..00
.04011.000
.312:.002
.055+t.000
TAE
.093:.000 .037.:.03
.254+1.000 .039::.02
TimeGAN
.093:.019 .038:.001 .273:.004 .082:.006
CR-VAE
.143:.002 .0764.013
.2774.001 .050+.000
RCGAN
.097+-001 .040+.001
.292+.005 .081+.003
C-RNN-GAN
.127+.004 .038+.000
.483+.005
.055+.004
T-Forcing
.150+.022 .038+.001
.315+.005
.142+.014
P-Forcing
.1161.004 .0431.001
.3034.006
.1024.013
WaveNet
.117+.008 .042+.001
.311..005
.333+.004
WaveGAN
.134..013
.041#.001
.3074.007
.324+.006
Original
0.094
0.036
0.250
0.031
24

---

Published as a conference paper at ICLR 2024
Table 8: Irregular time series (30%
dropped)
Method
Sines
Stocks
Energy
MuJoCo
VKAE (Ours)
.035::0233
.1621:006 .280 .123+.018
GT-GAN
.3631.033
.251=.097
.3333.063
.2491.035
Diseriminative Score
TimeGAN-ZAt
49411.012 .463...02
.4481.027
.4711.016
RCGAN-At
499:.000 4336:.064
.500=.000
.500=.000
C-RNN-GAN At
.500:.000 .5001.001
.5001.000
.500=.000
T-Forcing-AAl
.395+0033 .305+=0002
.477+.011
.348+0041
P-Foreing-AN
.44+.127
.341+.035
.500++000 .493+.010
TimeGAN-D
496+.008
:499::010
-479+.000.4634..222 .463+.025
RCGAN-D
500:1000 .500:..00
.5001.000 .0000...000000..0000 5500.000
C-RNN-GAN-D
.5001.000 .5001.000 .5001:000 .500..0000
T-Forcing-D
408+.087 .409
.347:.046 .494+.004
P-Forcing-D D
550
.480+.000 .911:.020 .500=.0000
KoVAE (Ours)
.0741.005 .0191..01
.0491.002
.0331.004
GT-GAN
.099::004 .021= _066+.001
.048+000
TimeGAN-Zt
.145:.025 .08::.001 .375..011
.118..0111118..033 .118..032
Predictive Score
RCGAN-AI
.144:.028 .181:.014 .3511.056 .33
C-RNN-GAN-A/
.7544.000 .091#.007
.500+.000
.447..000
T-Foreing-Ar
.116:.002 .0701.013
.251
.0561.001
P-Forcing-44 At
.122+.002 .083:.018
.255#.001
.089#.011
TimeGAN-D
.192:.082 .1053.053
.248:.024
.0981.006
RCGAN-D
388+.113 .523+.020 .409+.020
.361+.073
C-RNN-GAN-D
.664..001 .345:.002
.440#.000
.457+.001
T-Forcing-D
.100+.002 .027:.002
.0901.001
1001.0001
P-Forcing-D
.154:.004
.0791.008 .1774.001
.173..002
Original** **
.071=.004 .011= .055#.001 .011
Table 9: Irregular time series (50%
dropped)
Method
Sines
Stocks
Energy
MuJoCo
Ko VAE (Ours)
.0301.007
.092:..075
.2981..01 .117:.019
GT-GAN
.3724.128
.265+.003
.3174.010
.270+-0016
TimeGAN-AA
.464..008
.4874.019
.479:.020
.483+-023
Discriminative Score
RCGAN-At
.061t.165 .478:.049
.5001.000
55001.000
C-RNN-GAN-At
.5001.000
.5001.000
.5001.000
55001.000
T-Forcing -At
408 .308+.010
.478+.011
486+-005
P-Forcing At
.28 .388+.026
.498+.005
.911.012
TimeGAN-D
5004.000 .4774.021
.7331.015
500++.000
RCGAN-D
500+.000
5500:0000
.500+.000
.500+.000
C-RNN-GAN-D
550+..00 500++000
.500+.000
500++0000
T-Forcing-D
430+..11 .4074.034
.3761.046
4981.001
P-Forcing-D
.4991.000
55001.000
.500
5000
KoVAE (Ours)
.0721.002 .01911001
.0491.001
.0421.001
GT-GAN
1011:0000 .018+.002
.0644.001
056+.003
TimeGAN-At
1233.040 .0583.003
.5011.008
402 1.02T
Predictive Score
RCGAN-AI
.42+.005 .094+-013
.391+:014
.277+0061
C-RNN-GAN-AL
.711+.026 .089+.001
.500+.000
448++001
T-Forcing At
3794.029 .075:.032
.5114.000
.099+.002
P-Forcing- Af
1204.005 .0674.014
2633:.003
11891.026
TimeGAN-D
16991.074 .254::007
.3391.029
.375=.01T
RCGAN-D
5191.006 .33
.20+..010
.314+.023
C-RNN-
.7541.000 .2734.000
.384.0000
.479+.000
T-Forcing-D
.104+.001 .038+-003
.090+.000
.13++001
P-Foreing-D
.190+-002
.089+-010
198++005
207+-008
Original
.0711.004 001
00451.001
.0411.002
25

---

Published as a conference paper at ICLR 2024
Table 10:
Irregular time series (70% dropped)
Method
Sines
Stocks
Energy
MuJoCo
KOVAE (Ours)
.065:.012
.101=.040
.3922:.004 .119:1.014
GT-GAN
.288.0022 .230#.053
.325:.047 .275+.023
Discriminative Score
TimeGAN-AV
-500+:.000
.488:.009
.4961..008
.49411.009
RCGAN-AI
.433+1142
.381+.086
.500++000
.500+=000
C-RNN-GAN-At
-500=.000 .500=.000
.5001.000 .5001.000
T-Foreing-Ai At
.374..087 .365+.027
.4682.008 428:1.022
P-Forcing-AAt
.888+.047 .317=.019
50000 -500==000
.498+-003
TimeGAN-D
.4982.006 .4854.022
.5001.000 449::.0099
RCGAN-D
.500=0000 .500=.000 :5000:0:00000++0000000000
.500=.000 .500+.000
C-RNN-GAN-D
.500+.000 .500:1.000 .500:1.000 5500+.000
T-Foreing-D
4361..67 .4044.008
.3361.032
49331.005
P-Forcing-D D
1000:0000.449.1155....0 .94.1.00..444.11111111111 ,99--.0000 4499+.000
KoVAE (Ours)
.0761.004 .012.1000
.052.1.001 .0441.001
GT-GAN
.0881.005 .020:1.005 .0661.001
.051+.001
limeGAN-2t
.7342.000 .0722.000 .4961.000
.40
Predictive Score
RCGAN-At
218 .155#.009
.498:1.000
.222+.041
C-RNN-GAN-AL
.751+.014 .084+.002
.500+0000
448++.001
T-Forcing-AAt
.113+.001 .070=.022
.251+.000
.053++0002
P-Foreing-44
.133.004 .0501.002
.2851.006 .1172.034
TimeGAN-D
.752+.001
.228:.000
.4433.000
.372:.0899
RCGAN-D
.044.034 .441
.349:.027 42011.056
C-RNN-GAN-D
.632-.001 .281=.019
.436+.000 .479+.001
T-Forcing-D
.102+.001 .031 00F.:0022.911..0000 .091=.000 .11...0001144.00 .114+.003
P-Forcing-D
.788..045 .107 1933 +91
Original
.071 .0140.00.00.22200000.00 100. .041-.002
sine
stock
energy
mujoco
Figure 15: We plot the original signals (solid lines) vs. reconstructed signals (dashed lines) for Sine,
Stock, Energy, and Mujoco the regular setting. Overall, our model matches the data well.
26

---

Published as a conference paper at ICLR 2024
sine
stock
energy
mujoco
Figure 16: We plot the original signals (solid lines) vs. inferred signals (dashed lines) for Sine, Stock, Energy, and Mujoco in the irregular 50% case. Overall, model recovers the data well, except for stock, where the output signal represents the average.
27


Reviews:
[{"review_id": "Oe3GZn8gvn", "reviewer": "Reviewer_kti2", "summary": "This paper proposes a variational autoencoder, Koopman VAE (KVAE), for time series data based on Koopman theory. The idea is to use a linear map to represent the prior dynamics, alongside a nonlinear coordinate transformation (the encoder) that maps the data to a linear representation. The main features of KVAE is that (i) it can incorporate domain knowledge (in the prior) by placing constraints on the eigenvalues of the linear map; (ii) the behaviour of the system can be analysed using dynamical systems theory tools. The results in the paper are promising, showing that KVAE outperforms SOTA GANs and VAEs across synthetic and real world time series generation benchmarks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Experimental results indicate strong performance compared to GANs and VAEs\n- The use of linear latent dynamics simplifies the learning of the latent dynamics and allows for adding physical constraints, as indicated in section 4.3", "weaknesses": "- A large literature of sequential VAEs for time series data generation is omitted e.g. [1,2,3,4], despite a large number of baselines being used in the experiments section. Considering there is heavy development in this area, it would be useful to compare KVAE to these methods.\n- More discussion in the experiments section is required on the topic of analysing \"the behaviour of the system...using dynamical systems theory tools\" in order to claim this as an additional feature of KVAE\n\n\n[1] Chung et al. (2015). A Recurrent Latent Variable Model for Sequential Data\n\n[2] Rubanova et al. (2019). Latent ODEs for Irregularly-Sampled Time Series\n\n[3] Li et al. (2020). Scalable Gradients for Stochastic Differential Equations\n\n[4] Zhu et al. (2023). Markovian Gaussian Process Variational Autoencoders", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g1t3Qkv8G2", "reviewer": "Reviewer_SWgv", "summary": "A variant of VAE for time-series data is proposed. Technically a notable point of the proposed model lies in its prior model. It first samples a sequence $\\bar{y}$ based on the outputs of a GRU, thus the dynamics of $\\bar{y}$ can be nonlinear. Each $\\bar{y}$ is refined to be $y$ by the linear transformation with the DMD matrix computed on the sequence of $\\bar{y}$. Then there appears a regularization term to minimize the discrepancy between $y$ and $\\bar{y}$, which effectively imposes \"soft\" linearity on the dynamics of $y$, the final output of the prior model.", "questions": "(1) As stated above, the presence of GRU in the decoder makes it a little difficult to assess the real utility of the linear structure in the prior model. Do you have some observations when you did not use a nonlinear sequence-to-sequence model in the decoder?\n\n(2) In practice, how linear the sequence of $y$ is? In my understanding, the linearity of the dynamics of $y_{1:T}$ is not a hard constraint but rather is imposed in a soft manner as regularization. I am curious to what extent the $y$ could become linear with such a soft constraint.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is reasonable, and the experiments are convincing enough to see the superiority of the method especially in terms of generation. \n\nThe literature is nicely reviewed, and the paper is adequately placed in the relevant contexts.\n\nI cannot really assess the novelty and the significance in terms of time-series generation. On the other hand, in terms of Koopman-operator-based neural net architectures, the proposed model seems somewhat novel yet technically straightforward.", "weaknesses": "From a purely technical point of view, the contribution might look rather incremental. So the paper should be assessed rather in the context of time-series generation models, on which I am not really an expert and thus cannot provide an accurate evaluation.\n\nThere is a GRU in the decoder part, which makes it a little difficult to assess the benefit of the Koopman-based prior model. As GRU can provide a nonlinear sequence-to-sequence transformation, it is unclear if the linear structure of $y_{1:T}$ was really beneficial when generating $x_{1:T}$. The results could be more convincing if the decoder did not have the GRU; instead, it should have had a nonlinear **pointwise** (i.e., not sequence-to-sequence) transformation such as a multilayer perceptron applied to each timestep independently. An ablation study with such a change of architecture would be highly informative.\n\n----\n\nBelow are minor points.\n\n- Why do you use two different letters, $y$ and $z$, for the prior part and the posterior part, respectively? Usually in VAE papers, the latent variable is always $z$, and we just say $p(\\cdot)$ for prior and $q(\\cdot)$ for posterior. The current notation in the paper might also be okay, but I just wondered if there could be particular intention to use the two letters.\n- Although the paper focuses on the generation capability of the models, some more experiments on the reconstruction / inference capablity could also be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TDLL2R3Nli", "reviewer": "Reviewer_HuLD", "summary": "In this study, the authors propose a generative model using a variational auto-encoder. The variational auto-encoder employs the neural controlled differential equations to consider irregular time series and uses a GRU to march in time. The authors proposed to use a prior distribution with a linear transition function. It is shown that the proposed model outperforms previous models in the generation task.", "questions": "1. I know trying to optimizer both the prior and posterior in a loss function has become a trend in the deep learning community. However, theoretically speaking, trying to optimize the prior and posterior jointly in the KL loss leads to a ill-posed problem, where a unique solution does not exist. Simply put, it becomes a ping-pong game between the prior and posterior. You can easily show it by computing the parameters of the distributions in a local minima. How do you deal with this ill-posedness? \n\n2. What is $z_t$ and what is $y_t$? Are they different random variables? Based on the paper, it looks like both $y_t$ and $z_t$ denote the latent code, meaning that they are the same variable. I understand that the authors used $y_t$ and $z_t$ to distinguish between the prior and posterior latent code. But the way it is formulated now is not correct. For example, how do you define $KL[q(z)\\|p(y)]$ in eq. (3)? Shouldn't It be $KL[q(z)\\|p(z)]$ or $KL[q(y)\\|p(y)]$?\n\n3. What is the probabilistic model for $p(x_{1:t}|z_{1:t})$? Is it a parametric distribution, e.g., normal distribution? How do you compute the log likelihood function?\n\n4. If the modulus of the eigenvalues of $A$ is not strictly 1, i.e., $|\\lambda| =1$, the system either grows or decays exponentially fast. It should be a hard constraint, not a soft constraint. How do you guarantee this?\n\n5. Based on Eq (6), $y_t$ becomes deterministic once $y_{t-1}$ is observed. Then the probability distribution becomes a delta function, $p(y_t|y_{1:t}) = p(y_t|y_{t-1}) = \\delta (y_t - Ay_{t-1})$. How do you compute the KL divergence of the Dirac delta distribution?\n\n6. How do you find the correct initial condition $y_0$ to represent $x_{1:T}$. As discussed by the authors, $x_{1:T}$ is transformed to $y_{1:T}$. Then, since the model is linear, once $y_0$ is determined, the rest of the sequence is determined as $y_t = A^t y_0$. Hence, it is crucial to find $y_0$ that describes $x_{1:T}$ the best. How do you choose $y_0$ and how do you guarantee that the choice of $y_0$ is the optimal?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The authors proposed to use a linear operator for the prior in the training of the variational auto-encoder. The proposed method is relatively straightforward and it is shown that the method outperforms some of the existing models for generation tasks.", "weaknesses": "The novelty of the study is limited. While the authors claim their method is based on the Koopman operator, their model is, in fact, more similar to the dynamic linear model, or Kalman filter. Also, one of the claims is that the model can take care of irregular time series, the capability is simply using a pre-existing model, neural controlled differential equations. Moreover, the manuscript is not well written. The probabilistic description and the models are not clearly defined. There are a few concerns about their model. See the comments below,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a3kqYqyqKt", "reviewer": "Reviewer_nGir", "summary": "Although the existing GAN-based time-series generation showed good performance, they proposed a Koopman-based VAE model, citing problems such as unstable training or mode collapse. In addition, KVAE is a relatively simple model but it shows better performance in regular and irregular time-series than baseline models.", "questions": "I am curious about the role of the predictive loss term. In this paper, a predictive loss term was added to the object function. Therefore, I am curious about how much the predictive loss term affects the performance that is superior to existing baseline models.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality: There was an existing model that applied the VAE model to time-series generation, but its performance was lacking. This paper showed good performance in time-series generation by applying Koopman to VAE. In addition, while the existing GAN-based model had many hyperparameters due to unstable training, KVAE is very simple and shows good performance.\n\nQuality: In addition to various experiments conducted in previous research, the effectiveness of KAVE is clearly demonstrated through additional experiments such as physics-constrained generation.\n\nClarity: Very well written and easy to read.\n\nSignificance: Time-series generation suffered from many problems due to complex training. However, in this paper, it shows good performance with very easy learning.", "weaknesses": "1. There is a lack of explanation about Koopman in the Background section. In the case of this paper, the main point is Koopman, and readers may also want to know more about Koopman. Therefore, an explanation of Koopman should be in the main paper.\n\n2. There are no results for the predictive loss term in the ablation study in Section 5.5.\n\nMinor issues\n1. For Section 5, Experiments seems to be a more appropriate word than Results.\n2. It seems that 0 should be excluded from the MuJoCo results in Table 10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)