TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation

1 INTRODUCTION . Interest in generative models has increased considerably in recent years as the usage of deep learning in industry and academic has exploded . Data generators are useful in scenarios that involve unavailability of any or sufficient amount of real data , restrictions on data usage due to privacy reasons , need to simulate situations not yet encountered in reality , simulating exceptional cases , or the need to create datasets for specific testing scenarios such as presence of outliers or changepoints . Data generators also can help alleviate the limitation of deep-learning models that are data-hungry . Generative models fall under two broad categories , 1 ) trained generators that learn from real-world data , and 2 ) generators that use Monte Carlo sampling methods with user-defined distributions . Generative Adversarial Networks ( GAN ) , Variational Auto-Encoders ( VAE ) , and language models such as GPT ( Radford et al. , 2019 ) fall under the first category . Simple examples of the Monte Carlo generation include methods used to generate the classic donut-problem ( Beiden et al. , 2003 ) , or X-OR problem data ( Gomes et al. , 2006 ) . The major benefit of trained generators is that they can faithfully represent the patterns observed in the real-world data without requiring manual analysis , but their primary drawback is they can require large training data sets and long training times in order to learn to simulate the real data . Monte Carlo methods on the other hand , while being manual to set up , are simpler and more convenient to use . The generated data from them is interpretable and these methods also enable users to control the generated data , inject subject matter expertise , and simulate specific situations . The limitation is that generated data can be substantially different from real-world data . Therefore , a hybrid method that combines both approaches would be valuable for many downstream tasks such as prediction , forecasting , classification , and testing . In the domain of time-series data , synthetic data generation is a challenging task due to the temporal patterns in the data . The generative process must capture both the distributions in features as well as the temporal relationships . Deep learning methods are especially well-suited for modeling such complex relationships . However , in many real-world cases involving time-series data , the amount of available data can be limited in terms of number of samples , or in length of history . Examples include stock market predictions involving companies soon after initial public offering or predicting a retail organization ’ s staffing needs in newly opened locations . Such situations require a data generation method that functions well despite low volume of data and also allows users to introduce specific constructs of time-series patterns that are known to exist in the specific use-case . Recent work for synthetic data generation has largely focused on use of GANs ( Yoon et al. , 2019 ; Esteban et al. , 2017 ) , primarily by using recurrent neural networks for both generation and discrimination . However , due to the complexity introduced by temporal relationships , the standard approach of real versus synthetic binary discrimination is insufficient to capture temporal dependencies . As a result , special mechanisms are required within GAN networks to overcome this challenge . An example of such a special mechanism is the combination of supervised training traditionally used in autoregressive models with the unsupervised training of GAN in Yoon et al . ( 2019 ) . In order to leverage the strengths of both general data generation approaches we propose a new model based on Variational Auto-Encoders ( VAEs ) with a decoder design that enables user-defined distributions , which we hereafter refer to as TimeVAE . We demonstrate that TimeVAE can accurately model the temporal components of real-world data . Furthermore , we show that the bottleneck mechanism between the encoder and decoder that is inherent to VAEs acts to denoise the data which may help downstream tasks such as forecasting . Our method allows injection of custom temporal constructs such as level , trend , and seasonality to produce interpretable signals . Our experiments demonstrate TimeVAE meets the performance of top generative models when measuring similarity with original data , is computationally efficient to train , and outperforms available methods on the next-step prediction task . Finally , we show that our proposed method is superior to existing methods as the size of the available real-world training data diminishes . 2 RELATED WORKS . In Yoon et al . ( 2019 ) , an unsupervised GAN approach is combined with the power of autoregressive models , creating TimeGAN . The GAN makes up for the deficiencies that autoregressive models have , namely that they are deterministic . GANs , however , struggle to adhere to the temporal correlations in time series data which autoregressive models excel at . This method is the current state-of-the-art in synthetic timeseries data generation . Recurrent Conditional GAN ( RCGAN ) was invented to generate eICU data for privacy ( Esteban et al. , 2017 ) . A recurrent GAN architecture is used for generating real-valued sequential data . More specifically , RNNs are used for both the generator and discriminator and are conditioned on auxiliary information , consisting of real-valued time-series data with associated labels . C-RNN-GAN was developed to generate classical music data ( Mogren , 2016 ) . Randomly generated noise is input to deep LSTM modules to generate hidden representations which are then sent to the fully-connected layers to get the final generated sequential data samples . These samples and also the real music samples are then sent to the discriminator . This adversarial training helps the RNN generate music that varies in the number of tones used and the intensities . Fabius & Van Amersfoort ( 2014 ) implemented a recurrent VAE which used RNNs for the encoder and decoder . The architecture used the last encoded state of the encoder RNN as representational of the input sequence feeding into the reparameterization trick used in VAEs . The sample from the encoded representation passes into the decoder . However , Hyland et all Esteban et al . ( 2017 ) reported inconsistent results when tested on generated sine wave data , with results being clearly inferior to those obtained by the RCGAN method . Common limitations of the above generative models include : ( 1 ) difficult to train : the trained model is commonly subject to mode collapse , i.e . learning to sample synthetic data without diversity ( Srivastava et al. , 2017 ) ; ( 2 ) time consuming : the TimeGAN and RCGAN methods needed more than a day to be trained with 5,000 epochs by a V100 GPU on larger datasets ; ( 3 ) Need sufficient data to train GAN-based models . It is not easy to obtain sufficient data in many real-world forecasting situations . Our proposed model attempts to address these limitations . 3 METHODS . Our goal is to meet the following two objectives . First , we want to select an architecture for the encoder and decoder components in the VAE that maximizes our ability to generate realistic samples of timeseries data . Second , we want the architecture that allows us to inject specific temporal structures to the data generation process . These temporal structures can be used to create an interpretable generative process and inject domain expertise in cases where we have insufficient real-world data to train the generator model . 3.1 VARIATIONAL AUTO-ENCODER AS A GENERATIVE MODEL . We will first provide a brief overview of VAEs . The encoder in a VAE outputs the distribution of the embedding instead of a point estimate as in a traditional Auto-Encoder . We are given an input dataset X which consists of N i.i.d . samples of some continuous or discrete variable x . Our main goal is to generate samples that accurately represent distributions in original data . Specifically , we want to model the unknown probability function p ( x ) . The VAE described by Kingma & Welling ( 2013 ) assumes that the observed data are generated by a two step-process . First , a value z is generated from some prior distribution pθ ( z ) , and second , a value x is generated from some conditional distribution pθ ( x|z ) . While true values of the prior pθ ( z ) and the likelihood pθ ( x|z ) are unknown to us , we assume they are differentiable WRT both θ and z . Now , we can define the relationships between the input data and the latent representation as follows : the prior is pθ ( z ) , the likelihood is pθ ( x|z ) , and the posterior is pθ ( z|x ) . The computation of pθ ( z ) is very expensive and , quite commonly , intractable . To overcome this the VAE introduces an approximation of the posterior distribution as follows : qΦ ( z|x ) ≈ pθ ( z|x ) With this framework , the encoder serves to model the probabilistic posterior distribution qΦ ( z|x ) , while the decoder serves to model the conditional likelihood pθ ( x|z ) . Typically , this prior distribution of z is chosen to be the Gaussian distribution , more specifically , the standard normal . Then the posterior is regularized during training to ensure that the latent space is wrapped around this prior . We do this by adding the KL divergence between the variational approximation of the posterior and the chosen prior to the loss function . Since we embed the given inputs into a chosen prior , we can conveniently sample z directly from the prior distribution and then pass this z through the decoder . This essentially converts the VAE ’ s decoder into a generative model . 3.2 VAE LOSS FUNCTION . The loss function for the VAE , also named Evidence Lower Bound loss function ( ELBO ) , can be written as follows : Lθ , φ = −Eqφ ( z|x ) [ logpθ ( x|z ) ] + DKL ( qφ ( z|x ) ||pθ ( z ) ) The first term on the RHS is the negative log-likelihood of our data given z sampled from qφ ( z|x ) . The second term on the RHS is the KL-Divergence loss between the encoded latent space distribution and the prior . Note that the process involves sampling z from qφ ( z|x ) which would normally make the operation non-differentiable . However , VAEs use the so-called reparameterization trick which makes the VAE end-to-end trainable ( Kingma & Welling , 2013 ) . 3.3 BASE TIMEVAE ARCHITECTURE . Note that we have not yet described the encoding and decoding functions . One may choose any models for these as long as the loss function described above is differentiable . Our method uses a combination of traditional deep learning layers such as dense and convolutional layers and custom layers to model time-series specific components such as level , multi-polynomial trend , and seasonal patterns . Figure 1 provides a block-diagram of the base version of TimeVAE . The base version excludes the custom temporal structures . It does not require any time-series specific knowledge . The input signal X into the encoder is a 3-dimensional array of size N × T ×D , where N is the batch size , T is the number of time steps , and D is number of feature dimensions . If given data have variable length sequences , then the sequences are padded with zeroes at the beginning to ensure all sequences have same length T. The encoder passes the inputs through a series of convolutional layers with ReLU activation . Next , the data are flattened before passing through a fully-connected ( dense ) linear layer . If m is the number of chosen latent dimensions , representing dimensions of the multi-variate Gaussian , then this last layer has 2m number of neurons . We use the output to parameterize the multivariate Gaussian . The size of latent space m is a key model hyper-parameter . Next , we sample the vector z from the multivariate Gaussian using the reparameterization trick . The decoder takes the sampled latent vector z which is of length m. It is passed through a fullyconnected linear layer . Then the data are reshaped into 3-dimensional array before passing through a series of transposed convolutional layers with ReLU activation . Finally , the data passes through a time-distributed fully connected layer with dimensions such that the final output shape is the same as the original signalX .

Summary: This work proposes a generative model architecture for time-series that is trained using the variational auto-encoder framework. The main motivation for this model is (a) to obtain a more sample efficient model compared to comparable models based on generative adversarial networks (GANs), and (b) optionally incorproate domain knowledge in its design. The authors design a fairly general-purpose VAE architecture which can also incorporate intepretable decoder blocks. Results are carried out on a number of new datasets and compared to the aformentioned GAN based time-series models. The authors use two different metrics to evaluate the mismatch between genereated and real samples, and show that their TimeVAE performs better than the alternative models.