TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation

1 INTRODUCTION . Interest in generative models has increased considerably in recent years as the usage of deep learning in industry and academic has exploded . Data generators are useful in scenarios that involve unavailability of any or sufficient amount of real data , restrictions on data usage due to privacy reasons , need to simulate situations not yet encountered in reality , simulating exceptional cases , or the need to create datasets for specific testing scenarios such as presence of outliers or changepoints . Data generators also can help alleviate the limitation of deep-learning models that are data-hungry . Generative models fall under two broad categories , 1 ) trained generators that learn from real-world data , and 2 ) generators that use Monte Carlo sampling methods with user-defined distributions . Generative Adversarial Networks ( GAN ) , Variational Auto-Encoders ( VAE ) , and language models such as GPT ( Radford et al. , 2019 ) fall under the first category . Simple examples of the Monte Carlo generation include methods used to generate the classic donut-problem ( Beiden et al. , 2003 ) , or X-OR problem data ( Gomes et al. , 2006 ) . The major benefit of trained generators is that they can faithfully represent the patterns observed in the real-world data without requiring manual analysis , but their primary drawback is they can require large training data sets and long training times in order to learn to simulate the real data . Monte Carlo methods on the other hand , while being manual to set up , are simpler and more convenient to use . The generated data from them is interpretable and these methods also enable users to control the generated data , inject subject matter expertise , and simulate specific situations . The limitation is that generated data can be substantially different from real-world data . Therefore , a hybrid method that combines both approaches would be valuable for many downstream tasks such as prediction , forecasting , classification , and testing . In the domain of time-series data , synthetic data generation is a challenging task due to the temporal patterns in the data . The generative process must capture both the distributions in features as well as the temporal relationships . Deep learning methods are especially well-suited for modeling such complex relationships . However , in many real-world cases involving time-series data , the amount of available data can be limited in terms of number of samples , or in length of history . Examples include stock market predictions involving companies soon after initial public offering or predicting a retail organization ’ s staffing needs in newly opened locations . Such situations require a data generation method that functions well despite low volume of data and also allows users to introduce specific constructs of time-series patterns that are known to exist in the specific use-case . Recent work for synthetic data generation has largely focused on use of GANs ( Yoon et al. , 2019 ; Esteban et al. , 2017 ) , primarily by using recurrent neural networks for both generation and discrimination . However , due to the complexity introduced by temporal relationships , the standard approach of real versus synthetic binary discrimination is insufficient to capture temporal dependencies . As a result , special mechanisms are required within GAN networks to overcome this challenge . An example of such a special mechanism is the combination of supervised training traditionally used in autoregressive models with the unsupervised training of GAN in Yoon et al . ( 2019 ) . In order to leverage the strengths of both general data generation approaches we propose a new model based on Variational Auto-Encoders ( VAEs ) with a decoder design that enables user-defined distributions , which we hereafter refer to as TimeVAE . We demonstrate that TimeVAE can accurately model the temporal components of real-world data . Furthermore , we show that the bottleneck mechanism between the encoder and decoder that is inherent to VAEs acts to denoise the data which may help downstream tasks such as forecasting . Our method allows injection of custom temporal constructs such as level , trend , and seasonality to produce interpretable signals . Our experiments demonstrate TimeVAE meets the performance of top generative models when measuring similarity with original data , is computationally efficient to train , and outperforms available methods on the next-step prediction task . Finally , we show that our proposed method is superior to existing methods as the size of the available real-world training data diminishes . 2 RELATED WORKS . In Yoon et al . ( 2019 ) , an unsupervised GAN approach is combined with the power of autoregressive models , creating TimeGAN . The GAN makes up for the deficiencies that autoregressive models have , namely that they are deterministic . GANs , however , struggle to adhere to the temporal correlations in time series data which autoregressive models excel at . This method is the current state-of-the-art in synthetic timeseries data generation . Recurrent Conditional GAN ( RCGAN ) was invented to generate eICU data for privacy ( Esteban et al. , 2017 ) . A recurrent GAN architecture is used for generating real-valued sequential data . More specifically , RNNs are used for both the generator and discriminator and are conditioned on auxiliary information , consisting of real-valued time-series data with associated labels . C-RNN-GAN was developed to generate classical music data ( Mogren , 2016 ) . Randomly generated noise is input to deep LSTM modules to generate hidden representations which are then sent to the fully-connected layers to get the final generated sequential data samples . These samples and also the real music samples are then sent to the discriminator . This adversarial training helps the RNN generate music that varies in the number of tones used and the intensities . Fabius & Van Amersfoort ( 2014 ) implemented a recurrent VAE which used RNNs for the encoder and decoder . The architecture used the last encoded state of the encoder RNN as representational of the input sequence feeding into the reparameterization trick used in VAEs . The sample from the encoded representation passes into the decoder . However , Hyland et all Esteban et al . ( 2017 ) reported inconsistent results when tested on generated sine wave data , with results being clearly inferior to those obtained by the RCGAN method . Common limitations of the above generative models include : ( 1 ) difficult to train : the trained model is commonly subject to mode collapse , i.e . learning to sample synthetic data without diversity ( Srivastava et al. , 2017 ) ; ( 2 ) time consuming : the TimeGAN and RCGAN methods needed more than a day to be trained with 5,000 epochs by a V100 GPU on larger datasets ; ( 3 ) Need sufficient data to train GAN-based models . It is not easy to obtain sufficient data in many real-world forecasting situations . Our proposed model attempts to address these limitations . 3 METHODS . Our goal is to meet the following two objectives . First , we want to select an architecture for the encoder and decoder components in the VAE that maximizes our ability to generate realistic samples of timeseries data . Second , we want the architecture that allows us to inject specific temporal structures to the data generation process . These temporal structures can be used to create an interpretable generative process and inject domain expertise in cases where we have insufficient real-world data to train the generator model . 3.1 VARIATIONAL AUTO-ENCODER AS A GENERATIVE MODEL . We will first provide a brief overview of VAEs . The encoder in a VAE outputs the distribution of the embedding instead of a point estimate as in a traditional Auto-Encoder . We are given an input dataset X which consists of N i.i.d . samples of some continuous or discrete variable x . Our main goal is to generate samples that accurately represent distributions in original data . Specifically , we want to model the unknown probability function p ( x ) . The VAE described by Kingma & Welling ( 2013 ) assumes that the observed data are generated by a two step-process . First , a value z is generated from some prior distribution pθ ( z ) , and second , a value x is generated from some conditional distribution pθ ( x|z ) . While true values of the prior pθ ( z ) and the likelihood pθ ( x|z ) are unknown to us , we assume they are differentiable WRT both θ and z . Now , we can define the relationships between the input data and the latent representation as follows : the prior is pθ ( z ) , the likelihood is pθ ( x|z ) , and the posterior is pθ ( z|x ) . The computation of pθ ( z ) is very expensive and , quite commonly , intractable . To overcome this the VAE introduces an approximation of the posterior distribution as follows : qΦ ( z|x ) ≈ pθ ( z|x ) With this framework , the encoder serves to model the probabilistic posterior distribution qΦ ( z|x ) , while the decoder serves to model the conditional likelihood pθ ( x|z ) . Typically , this prior distribution of z is chosen to be the Gaussian distribution , more specifically , the standard normal . Then the posterior is regularized during training to ensure that the latent space is wrapped around this prior . We do this by adding the KL divergence between the variational approximation of the posterior and the chosen prior to the loss function . Since we embed the given inputs into a chosen prior , we can conveniently sample z directly from the prior distribution and then pass this z through the decoder . This essentially converts the VAE ’ s decoder into a generative model . 3.2 VAE LOSS FUNCTION . The loss function for the VAE , also named Evidence Lower Bound loss function ( ELBO ) , can be written as follows : Lθ , φ = −Eqφ ( z|x ) [ logpθ ( x|z ) ] + DKL ( qφ ( z|x ) ||pθ ( z ) ) The first term on the RHS is the negative log-likelihood of our data given z sampled from qφ ( z|x ) . The second term on the RHS is the KL-Divergence loss between the encoded latent space distribution and the prior . Note that the process involves sampling z from qφ ( z|x ) which would normally make the operation non-differentiable . However , VAEs use the so-called reparameterization trick which makes the VAE end-to-end trainable ( Kingma & Welling , 2013 ) . 3.3 BASE TIMEVAE ARCHITECTURE . Note that we have not yet described the encoding and decoding functions . One may choose any models for these as long as the loss function described above is differentiable . Our method uses a combination of traditional deep learning layers such as dense and convolutional layers and custom layers to model time-series specific components such as level , multi-polynomial trend , and seasonal patterns . Figure 1 provides a block-diagram of the base version of TimeVAE . The base version excludes the custom temporal structures . It does not require any time-series specific knowledge . The input signal X into the encoder is a 3-dimensional array of size N × T ×D , where N is the batch size , T is the number of time steps , and D is number of feature dimensions . If given data have variable length sequences , then the sequences are padded with zeroes at the beginning to ensure all sequences have same length T. The encoder passes the inputs through a series of convolutional layers with ReLU activation . Next , the data are flattened before passing through a fully-connected ( dense ) linear layer . If m is the number of chosen latent dimensions , representing dimensions of the multi-variate Gaussian , then this last layer has 2m number of neurons . We use the output to parameterize the multivariate Gaussian . The size of latent space m is a key model hyper-parameter . Next , we sample the vector z from the multivariate Gaussian using the reparameterization trick . The decoder takes the sampled latent vector z which is of length m. It is passed through a fullyconnected linear layer . Then the data are reshaped into 3-dimensional array before passing through a series of transposed convolutional layers with ReLU activation . Finally , the data passes through a time-distributed fully connected layer with dimensions such that the final output shape is the same as the original signalX .

Summary: The authors present a VAE for multivariate time series generation. The authors add additional decoder building blocks modelling level, trend and seasonality components of time series. Their method, named TimeVAE, is evaluated on four different datasets. Performance measures of generated samples are the similarity to real samples and the predictive quality of generated, “future” time steps.