TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation
Authors: Abhyuday Desai
Cynthia Freeman
Zuhui Wang
Ian Beaver
Venue: ICLR 2022 Submitted (2022)
Keywords: VAE
Variational Auto Encoder
Time Series
Data Generation
GAN
Generative Adversarial Network

Abstract: Recent work in synthetic data generation in the time-series domain has focused on the use of Generative Adversarial Networks. We propose a novel architecture for synthetically generating time-series data with the use of Variational Auto-Encoders (VAEs).  The proposed architecture has several distinct properties: interpretability, ability to encode domain knowledge, and reduced training times.  We evaluate data generation quality by similarity and predictability against four multivariate datasets.  We experiment with varying sizes of training data to measure the impact of data availability on generation quality for our VAE method as well as several state-of-the-art data generation methods.  Our results on similarity tests show that the VAE approach is able to accurately represent the temporal attributes of the original data.  On next-step prediction tasks using generated data, the proposed VAE architecture consistently meets or exceeds performance of state-of-the-art data generation methods. While noise reduction may cause the generated data to deviate from original data, we demonstrate the resulting de-noised data can significantly improve performance for next-step prediction using generated data.  Finally, the proposed architecture can incorporate domain-specific time-patterns such as polynomial trends and seasonalities to provide interpretable outputs.  Such interpretability can be highly advantageous in applications requiring transparency of model outputs or where users desire to inject prior knowledge of time-series patterns into the generative model.

Under review as a conference paper at ICLR 2022
TIME A VARIATIONAL AUTO-ENCODER FOR MULTIVARIATE TIME SERIES GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work in synthetic data generation in the time-series domain has focused on the use of Generative Adversarial Networks. We propose novel architec- ture for synthetically generating time-series data with the use of Variational Auto- Encoders (VAEs). The proposed architecture has several distinct properties: inter- pretability, ability to encode domain knowledge, and reduced training times. We evaluate data generation quality by similarity and predictability against four mul- tivariate datasets. We experiment with varying sizes of training data to measure the impact of data availability on generation quality for our VAE method as well several state-of afrt data generation methods. Our results similarity tests show that the VAE approach is able to accurately represent the temporal attributes of the original data. On next-step prediction tasks using generated data, the pro- posed VAE architecture consistently meets or exceeds performance of state-of- the-art data generation methods. While noise reduction may cause the generated data to deviate from original data, we demonstrate the resulting de-noised data can significantly improve performance for next-step prediction using generated data. Finally, the proposed architecture can incorporate domain-specific time-patterns such as polynomial trends and seasonalities to provide interpretable outputs. Such interpretability can be highly advantageous in applications requiring transparency model outputs or where users desire to inject prior knowledge of time-series patterns into the generative model.
INTRODUCTION
Interest in generative models has increased considerably in recent years as the usage of deep learning in industry and academic has exploded. Data generators are useful scenarios that involve unavail- ability of any sufficient amount of real data, restrictions data usage due to privacy reasons, need to simulate situations not yet encountered in reality, simulating exceptional cases, or the need to create datasets for specific testing scenarios such presence of outliers changepoints. Data generators also can help alleviate the limitation of deep-learning models that are data-hungry.
Generative models fall under two broad categories, 1) trained generators that learn from real-world data, and 2) generators that use Monte Carlo sampling methods with user-defined distributions. Generative Adversarial Networks (GAN), Variational Auto-Encoders (VAE), and language models such as GPT (Radford et al. 2019) fall under the first category. Simple examples of the Monte Carlo generation include methods used to generate the classic donut-problem (Beiden al. 2003), or X-OR problem data (Gomes et al., 2006). The major benefit of trained generators is that they can faithfully represent the patterns observed in the real-world data without requiring manual analysis, but their primary drawback is they can require large training data sets and long training times in order to learn to simulate the real data. Monte Carlo methods on the other hand, while being manual to set up, are simpler and more convenient use. The generated data from them is interpretable and these methods also enable users to control the generated data, inject subject matter expertise, and simulate specific situations. The limitation is that generated data be substantially different from real-world data. Therefore, a hybrid method that combines both approaches would be valuable for many downstream tasks such as prediction, forecasting, classification, and testing.
In the domain of time-series data, synthetic data generation is a challenging task due to the temporal patterns in the data. The generative process must capture both the distributions in features well

---

Under review as a conference paper at ICLR 2022
as the temporal relationships. Deep learning methods are especially well-suited for modeling such complex relationships. However, in many real-world cases involving time-series data, the amount of available data can be limited in terms of number of samples, or in length of history. Examples include stock market predictions involving companies soon after initial public offering or predieting a retail organization's s staffing needs in newly opened locations. Such situations require data generation method that functions well despite low volume of data and also allows users to introduce specific constructs of time-series patterns that are known to exist in the specific use-case.
Recent work for synthetic data generation has largely focused on use of GANs (Yoon et al., 2019) Esteban al., 2017), primarily by using recurrent neural networks for both generation and discrimi- nation. However, due to the complexity introduced by temporal relationships, the standard approach of real versus synthetic binary discrimination insufficient to capture temporal dependencies. As result, special mechanisms are required within GAN networks to overcome this challenge. An example of such a special mechanism the combination of supervised training traditionally used in autoregressive models with the unsupervised training of GAN in Yoon et al. (2019).
In order to leverage the strengths of both general data generation approaches we propose a new model based on Variational Auto-Encoders (VAEs) with decoder design that enables user-defined distributions, which we hereafter refer to as Time VAE. We demonstrate that Time VAE can accu- rately model the temporal components of world data. Furthermore, mechanism between the encoder and decoder that is inherent to VAFs acts to denoise the data which may help downstream tasks such as forecasting. Our method allows injection of custom temporal constructs such as level, trend, and seasonality to produce interpretable signals. Our experiments demonstrate ViE meets the performance of top generative models when measuring similarity with original data, is computationally efficient to train, and outperforms available methods on the sext-step prediction task. Finally, we show that our proposed method is superior to existing methods as the size of the available real-worke training data diminishes.
show that the bottleneck
2 RELATED WORKS
In Yoon et al. ) an unsupervised GAN approach is combined with the power of autoregres- sive models, creating TimeGAN. The GAN makes up for the deficiencies that autoregressive mod- els have, namely that they are deterministic. GANs, however, struggle to adhere to the temporal correlations in time series data which autoregressive models excel at. This method is the current state-of thetthheattttt in synthetic timeseries data generation.
Recurrent Conditional GAN (RCGAN) was invented to generate eICU data for privacy (Esteban et al. 2017). A recurrent GAN architecture is used for generating real-valued sequential data. More specifically, RNNs are used for both the generator and discriminator and are conditioned on auxiliary information, consisting of real-valued time-series data with associated labels.
C-RNN-GAN was developed to generate classical music data MMggen 2016). Randomly generated noise is input to deep LSTM modules generate hidden representations which are then sent the fully-connected layers to get the final generated sequential data samples. These samples and also the real music samples are then sent to the discriminator. This adversarial training helps the RNN generate music that varies in the number of tones used and the intensities.
Fabius & Van Amersfoort 2014)) implemented a recurrent VAE which used RNNs for the encoder and decoder. The architecture used the last encoded state of the encoder RNN as representational of the imput sequence feeding into the reparameterizzttio trick used in VAEs. The sample from the encoded representation passes into the decoder. However, Hyland et all Esteban et al. (2017) reported inconsistent results when tested on generated sine wave data, with results being clearly inferior to those obtained by the RCGAN method.
Common limitations of the above generative models include: (1) difficult to train: the trained model is commonly subject to mode collapse, i.e. learning to sample synthetic data without diversity (Sri-) vastava et al. 2017); (2) time consuming: the TimeGAN and RCGAN methods needed more than day to be trained with 5,000 epochs by V100 GPU on larger datasets; (3) Need sufficient data to train GAN- models. It is not easy to obtain sufficient data in many real-world forecasting situations. Our proposed model attempts to address these limitations.
2

---

Under review as a conference paper at ICLR 2022
3 METHODS
Our goal is to meet the following two objectives. First, we want to select an architecture for the en- coder and decoder components the VAE that maximizes our ability to generate realistic samples of timeseries data. Second, we want the architecture that allows us to inject specific temporal struc- tures the data generation process. These temporal structures can be used to create an interpretable generative process and inject domain expertise in cases where we have insufficient real-world data to train the generator model.
3.1 VARIATIONAL AUTO-ENCOOEE AS A GENERATIVE MODEL
We will first provide brief overview of VAßs. The encoder in VAE outputs the distribution of the embedding instead of a point estimate as in traditional Auto-Encoder We are given an input dataset X which consists of N i.i.d. samples of some continuous discrete variable æ. Our main goal is to generate samples that accurately represent distributions in original data. Specifically, we want to model the unknown probability function (x=).
The VAE described by Kingma & Welling 2013) assumes that the observed data are generated by a two step-process. First, value z is generated from some prior distribution pa and second, value is generated from some conditional distribution po (x) While true values of the prior pe (z) and the likelihood pe (a|(² are unknown to us, we assume they are differentiable WRT both 0 and z. Now, as follows: the prior is Po (z), the likelihood is pa (a) z), and the posterior is po (z|z).
define the relationships between the input data and the latent representation
The computation of Po (z) is very expensive and, quite commonly, intractable. To overcome this the VAE introduces an approximation of the posterior distribution as follows:
q (=|x) (z|xx)
With this framework, the encoder serves to model the probabilistic posterior distribution qu (z|x), while the decoder serves to model the conditional likelihood po (oxx2) (z)
Typically, this prior distribution z is chosen to be the Gaussian distribution, more specifically, the standard normal. Then the posterior is regularized during training ensure that the latent space wrapped around this prior. We do this by adding the KL divergence between the variational approximation of the posterior and the chosen prior the loss function.
Since we embed the given inputs into a chosen prior, we can conveniently sample z directly from the prior distribution and then pass this z through the decoder. This essentially converts the VAE's decoder into a generative model.
3.2 VAE Loss FUNCTION
The loss function for the VAE, also named Evidence Lower Bound loss function (ELBO), can be written follows:
Lo.e = - EE (x)x) [z]p) + Dx (x]
The first term on the RHS the negative log-likeliing likelihoo of our data given sampled from go (z(2(ar) (r)). The second term the RHS is the KL-Divergence loss between the encoded latent space distribu- tion and the prior. Note that the process involves sampling z from qo (z] x) which would normally make the operation non-differentiable. However, VAEs the so-called reparameterization trick which makes the VAE end-to-end trainable (Kingma & Welling 2013).
3.3 BASE TIMEVAE ARCHITECTURE
Note that we have not yet described the encoding and decoding functions. One may choose any models for these as long the loss function described above is differentiable. Our method uses combination of traditional deep learning layers such as dense and convolutional layers and custom layers to model time-series specific components such as level, multi-pol muulti-polynomill trend, and seasonal patterns. Figure | provides a block -diorr-a of the base version Time VAE. The base version excludes the custom temporal structures. It does require any time-series specific knowledge.
3

---

Under review as a conference paper at ICLR 2022
Time-VAE (Base)
Encoder (Base)
Decoder (Base)
Dense
Tinnk Dist Dencel
X
Carvib
Corvib
Flatton
Denoe
Dense
Reshappe
ComAD
X
Dense
trediiation Bamok)) Ganooat
Figure 1: Block diagram of components in Base Time VAE
Time-VAE (Interpretable)
Decoder (Interpretable)
Trend Blok
X
X
Encoder (Base)
NG.
Caussirr)
(sampte)
Stawor al Block
(flan))
Figure 2: Block diagram components in Interpretable TimeVAR
The input signal X into the encoder is a 3-dimensional array of size N x T x D, where N is the batch size, T the number of time steps, and number of feature dimensions. given data have variable length sequences, then the sequences are padded with zeroes at the beginning ensure all sequences have same length The encoder passes the inputs through a series of convolutional layers with ReLU activation. Next, the data flattened before passing through fully- fully-connected (dense) linear layer. m is the number of chosen latent dimensions, representing dimensions of the multi-variate Gaussian, then this last layer has 2m number of neurons. We the output parameterize the multivariate Gaussian. The size of latent space m is a key model hyper-parameter.
Next, sample the vector from the multivariate Gaussian using the reparameterizztion trick. The decoder takes the sampled latent vector z which is of length m. It is passed through fully- connected linear layer. Then the data are reshaped into 3-dimensional array before passing through a series of transposed convolutional layers with ReLU activation. Finally, the data passes through Lime-distributed fully connected layer with dimensions such that the final output shape is the same as the original signal X.
3.4 INTERPRETABLE TIMEVAE
We achieve interpretability of the modeled data generation process by injecting temporal structures the data generation process in the decoder. More specifically, we use decomposition of time series into level, trend and seasonal components, common approach in forecasting models; for example the well-known Holt-Winters exponential smoothing technique (Holt) 1957 Winters 1960).
The Interpretable VAE (Figure 2 uses the same encoder structure that of the Base Time The decoder has a more complex architecture because it imparts specific temporal structures to the decoding process. The decoder uses parallel blocks representing different temporal structures that are added together to produce the final output. The structures are of two types, namely, trend, and seasonality.
We next define the trend and seasonality blocks (see Figure 3). For sections below, let N be the batch size, D be the number of features, T be the number of time steps (i.e. number of epochs).
Trend Block: We reuse the trend architecture developed for the N-Beats forecasting model by Ore- shkin al. (019).. Trend is modeled as monotonic function. Let P be the number of degrees of

---

Under review as a conference paper at ICLR 2022
Trend Block
1th Stasonality Block
Dense
Dénna
Reshedk
Frr
Dense
Reshaee
0
Mabiix Mull
Transpest
Vir
Season Areey
V
Poy Spare
Seaeoon nnlices
K
Figure 3: Trend and Seasonality Blocks in Interpretable Time VAE
polynomial specified by the user. We model the trend polynomials p = 0, 1, as two step process. First, we use the latent space vector to estimate the basis expansion coefficients ier for trend. Orr is dimensionality N x D P. Next, we use Oer to reconstruct the trend Vtr in the original signal. The trend reconstruction in the signal in matrix form is follows:
Ver Rr
R = [1 r..,, rP] is the matrix of powers of r where ... 1]/T --1/ is a time vector. R of dimensionality P T. We perform matrix multiplication of 0rr and R and then transpose axes 1 and 2, resulting in final trend matrix Vor which has the dimensionality N T x D.
The matrix Our renders interpretability to the Trend block. Values from Otr specifically define the 9th 1st 2nd pth order trend for each sample N and feature dimension D.
Note that at = O, we get flat trend (i.e. no upward/downward trend), which is equivalent to the level component in traditional time-series modeling nomenclature. Level refers to the average value of the series.
Let S be the number of different seasonality patterns to be modeled. Each seasonality pattern, indexed by j. is defined by two parameters: m as the number of seasons, and the duration of each season. For example, to represent day-of-the- theetweweeeekk seasonality for daily data, m is 7 and d is 1. On the other hand, for hourly level data, day-of-the-wen seasonality is modeled and equal to 24.
Scasonality Block:
with equal
For each seasonality pattern j. ViM performs two steps. First, the latent space vector z used estimate the matrix of basis expansion coefficients 0g, which has the dimensionality an corresponding to specific season for each time- step of X to retrieve the seasonal pattern values Vi en which is of shape N
N x D x m. Next we index the elements in 05
TensorFlow, we use the gather function to perform this indexing using the season indexing array K. The final seasonality estimates VAn are the element- wise summation of all Vi over j 1, 1,2, S.
sn
In the seasonal block j. the matrix 0 provides interpretability of the seasonal pattern for each sample N and feature dimension D. One can index elements in sn. to identify impact of each of the m seasons within seasonal cycle.
The Interpretable Time architecture also allows use of the original Base Decoder (Figure] [T as residual branch the decoder. fact, one may choose to enable or disable any of the Trend, Seasonality, Base Decoder branches shown in Figure 2
Base Decoder as Residual Block:
Interpretable Decoder Output: The final output from the Interpretable Decoder is the element- wise summation of the Trend block output Ver, seasonality block outputs Vi. for j = 1, 2, ..., and the output from the residual Base Decoder (if used).
3.5 VAM OBJECTIVE FUNCTION
We train Time VAE using the ELBO loss function defined earlier with one modification. We use a weight the reconstruction
which works to increase decrease the emphasis placed on the reconstruction loss compared to the KL-Divergence loss between approximated posterior qa (2 x)) and the prior po(z). Weight factor reconstruction error ranged between 0.5 and 3.5 during
5

---

The lower Under review as a conference paper at ICLR 2022
following experiments and can be chosen by visually inspecting quality of generated samples, or through hyper- perer-eeeeterr tuning the generated samples are used in a supervised learning task in a downstream application.
4 EXPERIMENTS
4.1 DATA SETS
We consider 4 multivariate data sety in our experiments. The data sets include:
1. sines:
We generate 10, 000 samples of 5-dimensional sinusoidal sequence of varying fre- quencies, amplitudes, and phases where each feature is correlated with others. For each di- mension E {1, 2.., 5). (t) = asin(2mry + +0) where 7 ~ 200.1, 0.15), w(0, 2x] and a ~ u[1.0, 3.0].
2. stockv: 3, 919 samples (10 years) of 6-dimensional daily values of stock from Yahoo Finance.
3. energy: 711 samples of 28 38-dimensional appliances energy prediction data set con- sisting of continuous-valun valuud measurements from the UCI Machine Learning repository.
4. air: 9, 333 samples of 15-features of hourly-averaged responses from air quality sensors from the UCI Machine Learning repository.
4.2 COMPARISON METHIODOLOGIES
We compare VAE framework RCGAN, RNN trained with leacher-forcing (T-Forcing). and TimeGAN. These were the top performing methods compared to TimeGAN in Yoon et al. (2019
T-Forcing, ceacher-forcingg is deterministic technique for training autoregressive recurrett neural network where the ground truth is used as input instead of model output from a prior time step (Yoon al. 2019). As Yoon al. (2019) do not provide their version of T-Forcing code, follow the supplementary description of their implementation to the best of our ability: 3-layer GRUs with hidden dimensions 4 times the size of the input features, tanh activation function, sigmoid as the output layer activation function, [0,1] min-max scaled outputs, À = 1, and 10.
Instead of a deterministic autoregressive approach, RCGAN mentioned Esteban al. (2017) applies GAN architecture to sequential data but also drops dependence on the previous output while conditioning on additional input. We make use of the RCGAN code inthtps:: //g thub. fommrattshh ab/ RGAN adjusting the settings to account for dimension and sample size of the different data sets.
TimeGAN is generative time-series model combining adversarial GAN methodologies with au- toregressive model techniques to account for temporal dynamics. directly apply the authors' code on htttps://gittbb..cmm//
4.3 COMPARISON METRICS
To assess the quality of the generated data, we analyse 2-dimensional t-SNE plots of the original and synthetic data where the temporal dimension flattened. We also train a classification model to distinguish between the original and synthetic data as supervised task. The discriminative score (accuracy) 0.5) on the held-out set. score close is better indicating the generated data is hard to distinguish from original data. Finally, we consider predictive scores; more specifically, we train a post-hoc sequence model to predict next-step temporal vectors. This 2-layer I. LTM trained on synthetic data and evaluated using the mean absolute error on the original data set.
These comparison metrics were all considered Yoon al. (2019). and we use the authors' avail- able code for these metrics on /tttpp::/githui com the scores, the better.
A link data and experiment
codes will he made available here publication.
'The source code for C-RNN-GAN was obtained from Mogren (2016), but we were unable to reproduce similar performance numbers reported in Yoon et al. (2019) and observed that the method was very unstable the tasks. Therefore, chose exclude this method from experiments.

---

Under review as a conference paper at ICLR 2022
ThreO USME SRL
11
JOONT C
wilpes
10
-1
-1
23
12
MIID-20
wwwgy 1
CREWN
15 mide 5
-1
11
70
20
KEENg p-
MILD
energ
Figure 4: t-SNE plots for the TimeGAN (top left), TimeVAE (top right), RCGAN (bottom left), and T-Forcing (bottom right) models for the 4 data sets under various training size percentages (p). Red for original data, and blue for synthetic data. An empty t-SNE chart appears when not enough training data is present for the model to generate synthetic data.
4.4 PROCEDURE
For each data set mentioned in subsection 4.1 we use 100%. 20%, 10%, 5%, and 2% as training data for each methodology. For example, we consider how well RCGAN performs when only trained on the last 20% of the air data set. The synthetic data generated by these trained models is then used to train the post-hoc sequence models obtain discrimination and prediction scores deseribed in subsection[4.3 The amount of data generated for training the post-hoc sequence models equivalent the percentage of original training data used train the generators.

---

Under review as a conference paper at ICLR 2022
Table 1: Discriminator scores for all data set, model, and training percentages. N/A's exist when enough data was available for the model to generate synthetic data. Best performance is in bold.
model
% train
air
energy
sine
stockv
Time VAR
100
0.381 +/- 0.037
0.498 +/- 0.006
0.021 +/- 0.040
0.009 +/- 0.009
TimeGAN
0.387 +/- 0.026
0.499 +/- 0,000
0.437 +/- 0.023
0.011 +/- 0.013
T-Forcing
0.495 +/- 0.010
0.499 +/- 0.001
0.484 +/- 0.006
0.450+/-00 +000+++00000 0.099
RCGAN
0.495 +/- 0.002
0,500 +/- 0.000
0.382 +/- 0.075
0.494 +/- 0.006
Time
20
0.350 +/- 0.089
0.499 +/- 0.002
0.039 +/- 0.030
0.176 4L176+/-0.20 +/20.
TimeGAN
0.355 +/- 0.045
0.493 +/- 0.007
0.374 0.12
0.042 +/- 0.068
T-Forcing
0.500 +/- 0.000
0,500 +/- 0.001
+/-0000-0003
0.372 +/- 0.241
RCGAN
0,500 +/- 0.000
0.500 +/- 0.000
0.281 1-281+-0.111
0.479 +/- 0.028
Time VAE
10
0.425 +/- 0.067
0.499 +/- 0.001
0.053 +/- 0.045
0.080 +/-
TImeGAN
0.257 +/- 0.093
0.500 +/- 0.001
0.382 +/- 0.055
0.068 +/- 0.106
F-Frrcnng
0.500 +/- 0.000
0.500 +/- 0.001
0.482
0.006
0.474 4/474/-O0.
RCGAN
0.500 +/- 0.000
0,500 +/- 0,000
0/246 0.234
0.474 +/-4777
Time VAE
0.292 +/- 0.207
0.500 +/- 0.001
0.051 +/- 0.068
0.191 0.14.
TImeGAN
0.355 +/- 0.230
0.497 +/- 0.003
0.281
+/-0.18 185
0.040 +/- 0.029
T-Forcing
0.497 +/- 0.006
0.499 +/- 0.003
0.484
+/-
0.449 +/- 0.069
RCGAN
0.500 +/- 0.000
0,500 +/- 0.000
0/499//-0000 0.003
0.500 +/- 0,000
Time VAE
0.154 +/- 0.163
0.492 0.018
0.048 +/- 0.058
0.300 +/- 0.147
TimeGAN
0.167 +/- 0.242
0.468 +/- 0.062
0.192 +/- 0.273
N/A
T-Forcing
0.500 +/- 0.000
0.500 +/- 0.000
0.295 +/- 0.320
0.300 +/- 0.316
ROGAN
0.494 +/- 0.017
0.500 +/- 0.000
0.492 +/- 0.021
N/A
5 RESULTS/DISCUSSION
Figure displays the t-SNE charts of data generated from various generators for each of the datasets and training thresholds. The VAE generated data consistently shows heavy overlap with origi- nal data for all datasets and training thresholds even at the 2% training threshold. We see de- deiiiig in effect particularly for noisy datasets such as stockv and energy. Compared to T-Forcing and RC- GAN, TimeGAN has superior results on the air, energy and stockv datasets. However, performance of TimeGAN is sub-optimal on the sine dataset. In fact, all methods except Time VAE had inferior- quality generated data on the sine dataset. One can also note that performance of other generators deteriorates at smaller training sizes. For example, while TimeGAN produces good quality gener- ated data on the air dataset at the 100% threshold, the quality is inferior at thresholds of 2%, 5%, and 10%. For these reasons, the t-SNE plots indicate that the synthetic data generated by Time is superior compared to generated data from other methods.
Table I contains the discriminator scores for all scenarios. At the 100% threshold, Time VAE pro- duces the best results on the air, sine and stockv datasets. All generators perform poorly on the discrimination test on the energy dataset. Note that the energy dataset has 28 features and is also the largest in number of samples. As was noted from the t-SNE charts, TimeVVA performs signifi- cantly better than other generators on the sine dataset. At lower thresholds of training sizes, results clearly show that Time and TimeGAN methods are superior to T-Forcing and RCGAN. These results do not conclusively identify the better method between Time and TimeGAN, except on the sine dataset. However, TimeGAN and RCGAN could not train successfully on 2% of stockv.
Results from the predictor tests are displayed in Table 2 solute Error on the next-step prediction tasks. Results show that the TimeVAE method consistently meets or exceeds the performance from the other generators. On the sine dataset, Time VAE pro- duces the best results. Quite remarkably, performance of ViE is almost on-par with those of the original datasets themselves on the stocky and sine datasets, even at the 2% threshold. In cases where Time VAE does not have the best MAE score for other datasets, it is very close. For exam- plc, on the stockv dataset, Time VAE and TimeGAN are indistinguishable in performance within the confidence interval. Similarly, the energy dataset, Time and T-Forcing are the best methods.
The predictor scores measure Mean Ab-
8

---

Under review as a conference paper at ICLR 2022
Table 2: Predictor scores for all data set, model, and training percentages. Original means the LSTM for the predictor was trained using original data instead of synthetic data generated from any model and should always return the best performance possible. Best performance that is NOT Original is in bold. N/A's exist when not enough data was available for the model to generate synthetic data.
model
% train
air
energy
sine
stockv
Original
100
0.004 0.000
0.229
0.002
0.213 0.000
0.019 +/- 0.004
Vim
0.013 +/- 0.002
0.268 +/- 0.004
0.213 +/- 0.000
0.019 +/- 0.001
TimeGAN
0.005 +/- 0.000
0.298 +/- 0.002
0.251 +/- 0.027
0.021 +/ 0.001
T-Forcing
0.101 +/- 0.028
0.287 +/- 0.035
0.220 +/- 0.010
0.082 +/- 0.004
RCGAN
0,043 +/- 0,002
0.277 +/- 0,011
0.262 +/- 0,024
0,025 +/- 0.002
Original
20
0.004 0,001
0.187 0.003
0.215 0,000
0.049 +/- 0.001
Time VAE
0,019 +/- 0,003
0.288 0,002
0.215 +/- 0,000
0.052 +/ 0,001
TimeGAN
0.007 +/- 0.002
+/-0000
0.287 +/- 0.051
0.050 +/- 0.001
T-Forcing
0.139 +/- 0.061
0.256 0.006
0.219 +/- 0.007
0.091 +/- 0.024
RCGAN
0.480 +/- 0.315
0.751 0.43.
0.254 +/- 0.001
0.164 +/- O. 122
Original
10
0.002 +/- 0,001
0.162 +/- 0.004
0.214 +/- 0.000
0.072 +/- 0,000
VAE
0.005 +/- 0.003
0.275 0.275 0.001
0.215 +/- 0.000
0.075 +/- 0.001
TimeGAN
0.003 0.001
0.318 +/- 0.006
0.300 +/- 0.059
0.081 +/- 0.008
I-Forcing
0.157 +/- 0.027
0.262 +/- 0.014
0.216 +/- 0.002
0.118 +/- 0.040
RCGAN
0.605 +/- 0.618
0.740 +/- 0.371
0.241 +/- 0.030
0.150 +/- 0.094
Original
0.022 +/- 0.001
0.144 +/- 0.015
0.217 +/- 0.000
0.078 +/- 0.002
Time
0.040 +/- 0.002
0.262 +/- 0.002
0.218 +/- 0,001
0.084 +/- 0.004
TimeGAN
0.046 +/- 0,005
0,329 +/- 0,010
0.262 +/- 0.032
0.080 +/- 0.001
T- T-Forcing
0.185 +/- 0.012
0.263
0.011
0.221 +/- 0,006
0.131 +/- 0.026
RCGAN
0.428 +/- 0.045
0.499 +/- 0.054
0.222 +/- 0.002
0.669 +/- 0.252
Original
0,026 +/- 0,003
+/- 0,002
0.222 +/- 0,002
0.144 0.006
Tim
0.056 +/- 0.005
0.260 +/- 0.003
0.223 +/- 0.001
0.153 +/- 0.008
TimeGAN
0.057 +/- 0.003
+/-0003
0.270 +/- 0.004
N/A
T-Foreing
0.185 0.086
0.266 0.005
0.225 0.005
0.155 +/- 0.003
RCGAN
0.361 0.079
1.451 +/- 0.024
0.320 +/- 0.005
N/A
Table 3: Training times (in seconds) for all models using 100% of the data. Times were all obtained using the same ml. xlaaxe AWS instance with 4 vCPU, 1 V100 GPU, and 16 GB of memory.
model
air
energy
sine
stockv
VAE
442.39
1,661.40
386.20
95.40
TimeGAN
3,042.88
3,174.45
3,618.66
3,035.34
T-Foreing
115.14
134.91
117.59
109.19
RCGAN
9,102.56
19,806.19
4,365.80
1,442.70
CONCLUSIONS
In this paper, we introduce a novel Variational Auto-Encoder architecture for generating multi- variate time-series data. We propose two architectures Base Tim and an Interpretable Time The Base Time model is evaluated on multiple datasets and compared with current of-te--f--heearrr timeseries data generation methods. T-SNE charts comparing similarity of generated data with original data showed that VAE consistently produced high quality generated data and its performance was superior to that of other methods. Scores from discriminator and next-step prediction tests all datasets also indicate that the Time method meets exceeds the current kfae-oof-the-al in time-series data generation. Finally, from Table is clear that Time VAE requires significantly less computing time, and therefore cost, to train than existing GAN-based methods.
The Time VAE method is also novel because it allows injection of domain-speciffi temporal con- structs, which allows the method produce interpretable results. Our future work this method will focus on further proving the value of data generation methods with such temporal constructs on more complex tasks such as multi-step time-series forecasting with limited amount of data.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sergey V Beiden, Marcus A Maloof, and Robert F Wagner. A general model for finite-sample effects in training and testing of competing classifiers. IEEE Transactions on pattern analysis 2003.
and machine intelligence, 25 25(12):1561-150
Cristóbal Esteban, Stephanie 1. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv:1706.00 02633, 2017.
Otto Fabius and Joost R Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6588. 2014.
Carla Gomes, Ashish Sabharwal, and Bart Selman. Near-uniform sampling of combinatorial spaces using xor constraints, In NIPS, pp. 481 1884488, 2006.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. Technical report, Carnegie Institute of Technology, Pittsburgh, PA, 1957.
Diederik Kingma and Max Welling. arXiv: 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Olof Mogren. C-mnn-gnn Continuous recurrent neural networks with adversarial training. arXiv preprint arXiv:1611.09904 2016.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint urXiv:1905 10437, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 3310-3320 2017.
Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management sci- ence, 6(3):324-34,, 1960.
Jinsung Yoon, Daniel Jarrett, and Mihacla van der Schaar. Time- generative adversarial net- works. In Proceedings of the 33rd International Conference Neural Information Processing Systems, pp. 5508-5516 2019.
10


Reviews:
[{"review_id": "uVOESFjnmf", "reviewer": "Reviewer_f3YU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is motivated by the limited data access for the time-series domain, so the authors propose a synthetic data generator that they claim will be valuable for deep learning research. However, I would doubt how much the proposed method can contribute to the machine learning community. In real cases, we are motived to solve problems or develop new methods based on practical challenges. It will be difficult to evaluate such a synthetical data generator can capture the necessary needs for specific questions. \n\n1. There is no discussion about the interpretability in the results session.\n2. In Table 1, there is no obvious better performance for the proposed model besides the dataset sine. ", "summary_of_the_paper": "This paper proposes an architecture for synthetically generating time-series data with the use of VAE. The authors claim that the contributions for this paper are interpretability, capable of encoding domain knowledge, and reduced training time. ", "summary_of_the_review": "The authors proposed a synthetic data generator for the time-series domain. Based on the results, I didn’t see significant performance achieved compared to the other state-of-the-art methods. Meanwhile, it didn’t discuss the interpretability of this method which is claimed as one key contribution of this method. \n\nStrengths\n\nWell written paper\n\nWeaknesses\n\nNo interpretation was discussed since it was claimed as the main contribution \nThe overall contribution to this field is limited\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YjQAZ6UovRg", "reviewer": "Reviewer_CnbM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengths *\nThe main strength of this work is that the proposed model is a sensible and general choice for learning from arbitrary time-series data. The paper is clear and generally well written, and I've found it to be technically sound. While the use of VAEs for time-series data is not novel, the proposal to combine deep architectures with interpretable components is interesting. I would suggest the authors pursue this line of investigation further as interpretability is still an open problem with deep generative models (incorporating domain knowledge can ultimately help with generalization as well). Results on the proposed datasets and proposed metrics clearly show the advantages of TimeVAE over a set of chosen GAN-based models.\n\n* Weaknesses *\nI do not think this paper is ready for publication as there are substantial limitations in its current version. One of the main issues is that this paper lacks general context of the literature in a number of areas: state-space models, deep generative models for sequential prediction tasks, particle fitlers, and so on. Its technical contribution is thus unclear: it is unlikely that there can be a single general-purpose deep generative model for time-series. Different domains require drastically different architectures and tailored learning algorithms. \n\n- The description of the methods and the background on generative models is odd. The categorization of generative models as those that \"learn from data\" and those that \"use Monte Carlo sampling\" is confusing. I imagine the authors refer to simulation-based models (sometimes also referred to as implicit models due to the lack of a closed-form likelihood) for the second category. The explanation of VAEs is a bit nonstandard --  I would suggest framing the method as a generative model, and subsequently proposing the use of variational inference and the reparameterization trick as the way to approximately minimize the intractable marignal log likelihood.\n- A number of imprecise or not very formal claims are made, such as \"the computation of p(z) is very expensive and often intractable\". I imagine the authors mean the posterior p(z|x)? Another example is \"The GAN makes up for the deficiencies that autoregressive models have, namely that they are deterministic\". Under the common definition of autoregressive models, these are probabilistic.\n- To my understanding, the proposed architecture encodes the whole sequence into a global random variable modelled with a Gaussian distribution. The generative model decodes the latent vector to produce the whole output sequence. Having a single global lantent vector for a whole sequence seems like a very limiting design -- most state space model incorporate per time-step latent variable. Have the authors considered such a design?\n\nThere are also some important issues in the experimental methodology.\n- While interpretable components in the decoder are introduced, these are presumably not used at all in the experiments and no evaluation is provided as to the benefit and/or intepretability of these components.\n- If the purpose is to compare and find the best types of generative models for time-series, the experiments would benefit from including probabilistic autoregressive models that are generally well suited to capturing time correlations. Similarly, I'm not sure why the next-step prediction evaluation requires post-hoc learning of an LSTM model. Why not build into the VAE design the ability to predict next step (cf state-space models)?\n- All the experiments are done on a number of synthetic or novel datasets unavailable elsewhere. This makes it difficult to compare the proposed method to other pieces of work in the literature. \n- How does the proposed discriminator score compare to similar types of metrics that compare model samples and data samples (e.g. FID score)? ", "summary_of_the_paper": "This work proposes a generative model architecture for time-series that is trained using the variational auto-encoder framework. The main motivation for this model is (a) to obtain a more sample efficient model compared to comparable models based on generative adversarial networks (GANs), and (b) optionally incorproate domain knowledge in its design.\nThe authors design a fairly general-purpose VAE architecture which can also incorporate intepretable decoder blocks. Results are carried out on a number of new datasets and compared to the aformentioned GAN based time-series models. The authors use two different metrics to evaluate the mismatch between genereated and real samples, and show that their TimeVAE performs better than the alternative models.", "summary_of_the_review": "The paper has some interesting ideas regarding incorporating domain knowledge and interpretability into a deep generative model for time-series, but falls short in a large number of aspects. The literature review is insufficient, the background and methods should be described in more precise terms, and the experiments are not applied to well-known datasets, or extensive enough such that we can robustly see the benefits of the proposed model compared to the state-of-the-art. The experiments also lack any type of evaluation of the interpretability aspect of the proposed model, thus I do not see it as an important contribution. Overall, I suggest a major revision to the paper before recommending its acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8XdBV6d5Ebt", "reviewer": "Reviewer_X2k3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and I think clearly conveys the main points of their models as well as the related work.\n\nPrimarily however, it is not clear to me why the proposed models (and definitely the proposed \"base\" model) are not just essentially a standard VAE (or conditional VAE) applied to time series data. The architecture even includes convolutional layers (which yes have been showed to be sometimes useful for time series task) as opposed to maybe including some form of recurrent layer which might distinguish it more - this seems a bit strange. \n\nA counter might be that \"well TimeGAN is just a GAN applied to time series data\" but there at least they introduce additional loss functions in order to try and capture stepwise marginal distributions as well for example, there doesn't appear to be any of that sort of thing in this paper which makes it hard to see how it can be claimed that TimeVAE better captures the temporal aspects of the data.\n\nThe VAE loss function is just presented as is with little discussion as to exactly how it comes about. I'm sure most readers will be familiar, but I think it's worth discussing it's derivation. This is especially the case as the paper goes on to actually weight the terms, and a further discussion as to why that's appropriate or reasonable would be useful.\n\nThe empirical evaluation I think is done well and is compelling that TimeVAE does produce realistic time series trajectories - they could use an expanded set of benchmarks though given how many methods there are out there in the generative models space - the TimeGAN for example contains a number of extras that are not considered here.\n\nEquations should be formatted correctly in the middle of the line and numbered.", "summary_of_the_paper": "The paper presents a method for a generative model of time series data based on variational auto-encoders. They suggest two models, a simplified base version as well as a conditional \"interpretable\" one that can be used to customise trend and seasonality aspects of the generated data.", "summary_of_the_review": "The proposed methods simply do not seem to be presenting any sort of substantial originality that would make for me to recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0gffk67PcC-", "reviewer": "Reviewer_Gi2q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- Well written paper\n- Interesting approach to include methods from “classical” timeseries \n\n### Weaknesses\n- The introduced decoder building blocks seem to also introduce a set of new hyper-parameters. There are no details provided on how to best select these hyper-parameters nor how to perform model selection regarding these hyper-parameters.\n- The authors motivate the newly introduced building blocks with interpretability. In the empirical evaluation, there is no experiment or ablation that would provide any evidence on how interpretable the newly created building blocks are.\n\n### Questions\n- What about GP-VAE (Fortuin et al: Deep Probabilistic Multivariate Time Series Imputation, 2020)? How related is time-series imputation to next step prediction? I would be interested in the authors thoughts on that.\n- Table 1: I am not sure if I read the results correctly in Table 1. According to the description on the discriminator score in section 4.3. a value close to zero indicates a good performance of the model (the score is 0.5 - accuracy). For the 100% version of the training set, performance numbers are often worse, than for the 2% version of the training set. This is counter-intuitive to me. Could you please elaborate on that?\n- Are the t-SNE plots bin figure 4 generated using reconstructions of samples or randomly generated samples? I could not find the respective information.", "summary_of_the_paper": "The authors present a VAE for multivariate time series generation. The authors add additional decoder building blocks modelling level, trend and seasonality components of time series. Their method, named TimeVAE, is evaluated on four different datasets. Performance measures of generated samples are the similarity to real samples and the predictive quality of generated, “future” time steps. ", "summary_of_the_review": "The authors provide a simple and interesting extension to VAEs for multivariate timeseries. Although I like the idea of the paper, the limited empirical evaluation can be improved in my opinion. Although the authors perform experiments on 4 datasets to compare to previous works, the hyper-parameters of the core building blocks are not evaluated properly. Ablation studies and the selection of values are missing. Therefore, this paper is not ready for publication at ICLR in my opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a VAE-based architecture for generating multivariate time series. The base version of TimeVAE models a distribution over a fixed-length sequences of observations using a latent vector of fixed dimensionality and a convolutional encoder and decoder. The Interpretable TimeVAE model incorporates additional features from traditional time series models such as explicit modelling of trends and seasonality. TimeVAE is compared to several baselines such as TimeGAN on four small times series dataset and seems to perform competitively according to two custom evaluation metrics and a visualization. 

The reviewers thought that the paper was interesting but not ready for publication due to the following:
-The paper's contributions and their significance are not clear
-Interpretable VAE was not used in the experiments and its interpretability has not been verified
-Coverage of related work is insufficient
Decision: Reject