PSA-GAN: Progressive Self Attention GANs for Synthetic Time Series

1 INTRODUCTION . In the past years , methods such as ( Salinas et al. , 2020 ; Franceschi et al. , 2019 ; Kurle et al. , 2020 ; de Bézenac et al. , 2020 ; Oreshkin et al. , 2019 ; Rasul et al. , 2021 ; Cui et al. , 2016 ; Wang et al. , 2016 ) have consistently showcased their effectiveness of deep learning in time series analysis tasks . Although these deep learning based methods are effective when sufficient clean data is available , this assumption is not always met in practice . For example , sensor outages can cause gaps in IoT data , which might render the data unusable for machine learning applications ( Zhang et al. , 2019b ) . An additional problem is that time series panels often have insufficient size for forecasting tasks , leading to research in meta-learning for forecasting ( Oreshkin et al. , 2020b ) . Cold starts are another common problem in time series forecasting where some time series have little and no data ( like a new product in a demand forecasting use case ) . Thus , designing flexible and task-independent models that generate synthetic , but realistic time series for arbitrary tasks poses an important challenge . Generative adversarial networks ( GAN ) are a flexible model family that has had success in other domains . However , for their success to carry over to time series , synthetic time series data must be of realistic length , which current state-of-the-art synthetic time series models struggle to generate because they often rely on recurrent networks to capture temporal dynamics ( Esteban et al. , 2017 ; Yoon et al. , 2019 ) . In this work , we make three contributions : i ) We propose PSA-GAN , a progressively growing , convolutional time series GAN model augmented with self-attention ( Karras et al. , 2018 ; Vaswani et al. , 2017 ) . PSA-GAN scales to long time series because the progressive growing architecture starts modeling the coarse-grained time series features and moves towards modeling fine-grained details during training . The self-attention mechanism captures long-range dependencies in the data ( Zhang et al. , 2019a ) . ii ) We show empirically that PSA-GAN samples are of sufficient quality and length to boost several downstream forecasting tasks : far-forecasting and data imputation during inference , data imputation of missing value stretches during training , forecasting under cold start conditions , and data augmentation . Furthermore , we show that PSA-GAN can be used as a forecasting model and has competitive performance when using the same context information as an established baseline . iii ) Finally , we propose a Frechet Inception distance ( FID ) -like score ( Salimans et al. , 2016 ) , ContextFID , leveraging unsupervised time series embeddings ( Franceschi et al. , 2019 ) . We show that the lowest scoring models correspond to the best-performing models in our downstream tasks and that the Context-FID score correlates with the downstream forecasting performance of the GAN model ( measured by normalized root mean squared error ) . Therefore , the Context-FID could be a useful general-purpose tool to select GAN models for downstream applications . We structure this work as follows : We discuss the related work in Section 2 and introduce the model in Section 3 . In Section 4 , we evaluate our proposed GAN model using the proposed Context-FID score and through several downstream forecasting tasks . We also directly evaluate our model as a forecasting algorithm and perform an ablation study . Section 5 concludes this manuscript . 2 RELATED WORK . GANs ( Goodfellow et al. , 2014 ) are an active area of research ( Karras et al. , 2019 ; Yoon et al. , 2019 ; Engel et al. , 2019 ; Lin et al. , 2017 ; Esteban et al. , 2017 ; Brock et al. , 2019 ) that have recently been applied to the time series domain ( Esteban et al. , 2017 ; Yoon et al. , 2019 ) to synthesize data ( Takahashi et al. , 2019 ; Esteban et al. , 2017 ) , and to forecasting tasks ( Wu et al. , 2020b ) . Many time series GAN architectures use recurrent networks to model temporal dynamics ( Mogren , 2016 ; Esteban et al. , 2017 ; Yoon et al. , 2019 ) . Modeling long-range dependencies and scaling recurrent networks to higher lengths is inherently difficult and limits the application of time series GANs to short sequence lengths ( less than 100 time steps ) ( Yoon et al. , 2019 ; Esteban et al. , 2017 ) . One way to achieve longer realistic synthetic time series is by employing convolutional ( van den Oord et al. , 2016 ; Bai et al. , 2018 ; Franceschi et al. , 2019 ) and self-attention architectures ( Vaswani et al. , 2017 ) . Convolutional architectures are able to learn relevant features from the raw time series data ( van den Oord et al. , 2016 ; Bai et al. , 2018 ; Franceschi et al. , 2019 ) , but are ultimately limited to local receptive fields and can only capture long-range dependencies via many stacks of convolutional layers . Self-attention can bridge this gap and allow for modeling long-range dependencies from convolutional feature maps , which has been a successful approach in the image ( Zhang et al. , 2019a ) and time series forecasting domain ( Li et al. , 2019 ; Wu et al. , 2020a ) . Another technique to achieve long sample sizes is progressive growing , which successively increases the resolution by adding layers to generator and discriminator during training ( Karras et al. , 2018 ) . Additionally , the synthetic time series should preserve the dynamics of the real data . This can be achieved by using the latent state of an autoencoder network ( Yoon et al. , 2019 ) . In PSA-GAN , we explore whether unsupervised techniques could instead provide meaningful embeddings that can act as an additional signal to the GAN to model realistic time series . Our proposal , PSA-GAN , synthesizes progressive growing with convolutions , self-attention , and embeddings into a novel architecture particularly geared towards time series . Another line of work in the time series field is focused on developing suitable loss functions for modeling financial time series with GANs where specific challenges include heavy tailed distributions , volatility clustering , absence of autocorrelations , among others ( Cont , 2001 ; Eckerli & Osterrieder , 2021 ) . To this end , several models like QuantGAN ( Wiese et al. , 2020 ) , ( Conditional ) SigWGAN ( Ni et al. , 2020 ; 2021 ) , and DAT-GAN ( Sun et al. , 2020 ) have been proposed ( see ( Eckerli & Osterrieder , 2021 ) for review in this field ) . This line of work targets its own challenges by developing new loss functions for financial time series , which is orthogonal to our work , i.e . we focus on neural network architectures for time series GANs and show its usefulness in the context of time series forecasting . Another challenge is the evaluation of synthetic data . While the computer vision domain uses standard scores like the Inception Score and the Frechet Inception Distance ( FID ) ( Salimans et al. , 2016 ; Heusel et al. , 2018 ) , such universally accepted scores do not exist in the time series field . Thus , researchers rely on a Train on Synthetic–Test on Real setup and assess the quality of the synthetic time series in a downstream classification and/or prediction task ( Esteban et al. , 2017 ; Yoon et al. , 2019 ) . In this work , we build on this idea and assess the GAN models through downstream forecasting tasks . Additionally , we suggest a Frechet Inception Distance-like score that is based on unsupervised time series embeddings ( Franceschi et al. , 2019 ) . Critically , we want to be able to score the fit of our fixed length synthetic samples into their context of ( often much longer ) true time series , which is taken into account by the contrastive training procedure in ( Franceschi et al. , 2019 ) . As we will later show , the lowest scoring models correspond to the best performing models in downstream tasks . 3 MODEL . Problem formulation We denote the values of a time series dataset by zi , t ∈ R , where i ∈ { 1 , 2 , . . . , N } is the index of the individual time series and t ∈ { 1 , 2 , . . . , T } is the time index . Additionally , we consider an associated matrix of time feature vectors X1 : T = ( x1 , . . . , xT ) in RD×T . Our goal is to model a time series of fixed length τ , Ẑi , t , τ = ( ẑi , t , . . . , ẑi , t+τ−1 ) , from this dataset using a conditional generator function G and a fixed time point t. Thus , we aim to model Ẑi , t , τ = G ( n , φ ( i ) , Xt : t+τ−1 ) , where n ∈ Rτ is a noise vector drawn from a Gaussian distribution of mean zero and variance one ; φ is an embedding function that maps the index of a time series to a vector representation , that is concatenated to each time step of Xt : t+τ−1 . An overview of the model architecture is shown in Figure 1 and details about the time features are presented in Appendix A. Spectral Normalised Residual Self Attention with Convolution The generator and discriminator use a main function m that is a composition of convolution , self attention and spectral normalisation m ◦ f : Rnf×l → Rnf×l x 7→ γ SA ( f ( x ) ) + f ( x ) ( 1 ) where f ( x ) = LR ( SN ( c ( x ) ) ) and m ( y ) = γ SA ( y ) +y , c is a one dimensional convolution operator , LR the LeakyReLU operator ( Xu et al. , 2015 ) , SN the Spectral Normalisation operator ( Miyato et al. , 2018 ) and SA the Self Attention module . The variable nf is the number of in and out-channels of c , l is the length of the sequence . Following the work of ( Zhang et al. , 2019a ) , the parameter γ is learnable . It is initialized to zero to allow the network to learn local features directly from the building block f , and is later enriched with distant features as the absolute value of gamma increases , hereby more heavily factoring the self-attention term SA . The module m is referenced as Residual Self-Attention in Fig . 1 ( Right ) . Downscaling and Upscaling The following sections mention upscaling ( UP ) and downscaling ( DOWN ) operators that double and halve the length of the time series , respectively . In this work , the upscaling operator is a linear interpolation and the downscaling operator is the average pooling . PSA-GAN PSA-GAN is a progressively growing GAN ( Karras et al. , 2018 ) ; thus , trainable modules are added during training . Hereby , we model the generator and discriminator as a composition of functions : G = gL+1 ◦ ... ◦g1 andD = d1 ◦ ... ◦dL+1 where each function gi and di for i ∈ [ 1 , L+1 ] corresponds to a module of the generator and discriminator . GENERATOR As a preprocessing step , we first map the concatenated input [ n , φ ( i ) , Xt : t+τ−1 ] from a sequence of length τ to a sequence of length 8 , denoted by Z̃0 , using average pooling . Then , the first layer of the generator g1 applies the main function m : g1 : Rnf×2 3 → Rnf×2 3 Z̃0 7→ Z̃1 = m ◦ f ( Z̃0 ) ( 2 ) For i ∈ [ 2 , L ] , gi maps an input sequence Z̃i−1 to an output sequence Z̃i by applying an upscaling of the input sequence and the function m ◦ f : gi : Rnf×2 i+1 → Rnf×2 i+2 Z̃i−1 7→ Z̃i = m ◦ f ( UP ( Z̃i−1 ) ) ( 3 ) The output of gi is concatenated back to the time features Xt : t+τ−1 and forwarded to the next block . Lastly , the final layer of the generator gL+1 reshapes the multivariate sequence Z̃L to a univariate time series Ẑi , t , τ of length τ = 2L+3 using a one dimensional convolution and spectral normalisation . DISCRIMINATOR The architecture of the discriminator mirrors the architecture of the generator . It maps the generator ’ s output Ẑi , t , τ and the time features Xt : t+τ−1 to a score d. The first module of the discriminator dL+1 uses a one dimensional convolution c1 and a LeakyReLU activation function : d ′ L+1 : R1+D , τ → Rnf , τ ( Z̃L+1 , Xt : t+τ−1 ) 7→ ỸL = SN ( LR ( c1 ( Z̃L+1 , Xt : t+τ−1 ) ) ( 4 ) For i ∈ [ L+ 1 , 2 ] , the module di applies a downscale operator and the main function m : di : Rnf×2 i+2 → Rnf×2 i+1 Yi 7→ Yi−1 = DOWN ( m ( Yi ) ) ( 5 ) The last module d1 turns its input sequence into a score : d1 : Rnf×2 3 → R Y1 7→ Y0 = SN ( FC ( LR ( SN ( c ( m ( Y1 ) ) ) ) ) ) ( 6 ) where FC is a fully connected layer . PSA-GAN-C We another instantiation of PSA-GAN in which we forward to each generator block gi knowledge about the past . The knowledge here is a subseries Ẑi , t−LC , LC in the range [ t− LC , t− 1 ] , with LC being the context length . The context Ẑi , t−LC , LC is concatenated along the feature dimension , i.e . at each time step , to the output sequence of gi . It is then passed through a two layers perceptron to reshape the feature dimension and then added back to the output of gi .

Summary: This paper proposes a GAN-based approach (PSA) for generating realistic time-series data that can be used to improve several downstream tasks such as imputation of missing data and forecasting. The generator and discrimination model in the proposed GAN framework, PSA-GAN consists of progressively growing blocks consisting of convolutions and self-attention. The paper also proposes a metric for evaluating the quality of synthetic time series data. Experiments show that the proposed approach is able to achieve better performance than previous GAN-based approaches for time series and is also able to improve on downstream forecasting tasks by augmenting forecasting models.