PSA-GAN: Progressive Self Attention GANs for Synthetic Time Series

1 INTRODUCTION . In the past years , methods such as ( Salinas et al. , 2020 ; Franceschi et al. , 2019 ; Kurle et al. , 2020 ; de Bézenac et al. , 2020 ; Oreshkin et al. , 2019 ; Rasul et al. , 2021 ; Cui et al. , 2016 ; Wang et al. , 2016 ) have consistently showcased their effectiveness of deep learning in time series analysis tasks . Although these deep learning based methods are effective when sufficient clean data is available , this assumption is not always met in practice . For example , sensor outages can cause gaps in IoT data , which might render the data unusable for machine learning applications ( Zhang et al. , 2019b ) . An additional problem is that time series panels often have insufficient size for forecasting tasks , leading to research in meta-learning for forecasting ( Oreshkin et al. , 2020b ) . Cold starts are another common problem in time series forecasting where some time series have little and no data ( like a new product in a demand forecasting use case ) . Thus , designing flexible and task-independent models that generate synthetic , but realistic time series for arbitrary tasks poses an important challenge . Generative adversarial networks ( GAN ) are a flexible model family that has had success in other domains . However , for their success to carry over to time series , synthetic time series data must be of realistic length , which current state-of-the-art synthetic time series models struggle to generate because they often rely on recurrent networks to capture temporal dynamics ( Esteban et al. , 2017 ; Yoon et al. , 2019 ) . In this work , we make three contributions : i ) We propose PSA-GAN , a progressively growing , convolutional time series GAN model augmented with self-attention ( Karras et al. , 2018 ; Vaswani et al. , 2017 ) . PSA-GAN scales to long time series because the progressive growing architecture starts modeling the coarse-grained time series features and moves towards modeling fine-grained details during training . The self-attention mechanism captures long-range dependencies in the data ( Zhang et al. , 2019a ) . ii ) We show empirically that PSA-GAN samples are of sufficient quality and length to boost several downstream forecasting tasks : far-forecasting and data imputation during inference , data imputation of missing value stretches during training , forecasting under cold start conditions , and data augmentation . Furthermore , we show that PSA-GAN can be used as a forecasting model and has competitive performance when using the same context information as an established baseline . iii ) Finally , we propose a Frechet Inception distance ( FID ) -like score ( Salimans et al. , 2016 ) , ContextFID , leveraging unsupervised time series embeddings ( Franceschi et al. , 2019 ) . We show that the lowest scoring models correspond to the best-performing models in our downstream tasks and that the Context-FID score correlates with the downstream forecasting performance of the GAN model ( measured by normalized root mean squared error ) . Therefore , the Context-FID could be a useful general-purpose tool to select GAN models for downstream applications . We structure this work as follows : We discuss the related work in Section 2 and introduce the model in Section 3 . In Section 4 , we evaluate our proposed GAN model using the proposed Context-FID score and through several downstream forecasting tasks . We also directly evaluate our model as a forecasting algorithm and perform an ablation study . Section 5 concludes this manuscript . 2 RELATED WORK . GANs ( Goodfellow et al. , 2014 ) are an active area of research ( Karras et al. , 2019 ; Yoon et al. , 2019 ; Engel et al. , 2019 ; Lin et al. , 2017 ; Esteban et al. , 2017 ; Brock et al. , 2019 ) that have recently been applied to the time series domain ( Esteban et al. , 2017 ; Yoon et al. , 2019 ) to synthesize data ( Takahashi et al. , 2019 ; Esteban et al. , 2017 ) , and to forecasting tasks ( Wu et al. , 2020b ) . Many time series GAN architectures use recurrent networks to model temporal dynamics ( Mogren , 2016 ; Esteban et al. , 2017 ; Yoon et al. , 2019 ) . Modeling long-range dependencies and scaling recurrent networks to higher lengths is inherently difficult and limits the application of time series GANs to short sequence lengths ( less than 100 time steps ) ( Yoon et al. , 2019 ; Esteban et al. , 2017 ) . One way to achieve longer realistic synthetic time series is by employing convolutional ( van den Oord et al. , 2016 ; Bai et al. , 2018 ; Franceschi et al. , 2019 ) and self-attention architectures ( Vaswani et al. , 2017 ) . Convolutional architectures are able to learn relevant features from the raw time series data ( van den Oord et al. , 2016 ; Bai et al. , 2018 ; Franceschi et al. , 2019 ) , but are ultimately limited to local receptive fields and can only capture long-range dependencies via many stacks of convolutional layers . Self-attention can bridge this gap and allow for modeling long-range dependencies from convolutional feature maps , which has been a successful approach in the image ( Zhang et al. , 2019a ) and time series forecasting domain ( Li et al. , 2019 ; Wu et al. , 2020a ) . Another technique to achieve long sample sizes is progressive growing , which successively increases the resolution by adding layers to generator and discriminator during training ( Karras et al. , 2018 ) . Additionally , the synthetic time series should preserve the dynamics of the real data . This can be achieved by using the latent state of an autoencoder network ( Yoon et al. , 2019 ) . In PSA-GAN , we explore whether unsupervised techniques could instead provide meaningful embeddings that can act as an additional signal to the GAN to model realistic time series . Our proposal , PSA-GAN , synthesizes progressive growing with convolutions , self-attention , and embeddings into a novel architecture particularly geared towards time series . Another line of work in the time series field is focused on developing suitable loss functions for modeling financial time series with GANs where specific challenges include heavy tailed distributions , volatility clustering , absence of autocorrelations , among others ( Cont , 2001 ; Eckerli & Osterrieder , 2021 ) . To this end , several models like QuantGAN ( Wiese et al. , 2020 ) , ( Conditional ) SigWGAN ( Ni et al. , 2020 ; 2021 ) , and DAT-GAN ( Sun et al. , 2020 ) have been proposed ( see ( Eckerli & Osterrieder , 2021 ) for review in this field ) . This line of work targets its own challenges by developing new loss functions for financial time series , which is orthogonal to our work , i.e . we focus on neural network architectures for time series GANs and show its usefulness in the context of time series forecasting . Another challenge is the evaluation of synthetic data . While the computer vision domain uses standard scores like the Inception Score and the Frechet Inception Distance ( FID ) ( Salimans et al. , 2016 ; Heusel et al. , 2018 ) , such universally accepted scores do not exist in the time series field . Thus , researchers rely on a Train on Synthetic–Test on Real setup and assess the quality of the synthetic time series in a downstream classification and/or prediction task ( Esteban et al. , 2017 ; Yoon et al. , 2019 ) . In this work , we build on this idea and assess the GAN models through downstream forecasting tasks . Additionally , we suggest a Frechet Inception Distance-like score that is based on unsupervised time series embeddings ( Franceschi et al. , 2019 ) . Critically , we want to be able to score the fit of our fixed length synthetic samples into their context of ( often much longer ) true time series , which is taken into account by the contrastive training procedure in ( Franceschi et al. , 2019 ) . As we will later show , the lowest scoring models correspond to the best performing models in downstream tasks . 3 MODEL . Problem formulation We denote the values of a time series dataset by zi , t ∈ R , where i ∈ { 1 , 2 , . . . , N } is the index of the individual time series and t ∈ { 1 , 2 , . . . , T } is the time index . Additionally , we consider an associated matrix of time feature vectors X1 : T = ( x1 , . . . , xT ) in RD×T . Our goal is to model a time series of fixed length τ , Ẑi , t , τ = ( ẑi , t , . . . , ẑi , t+τ−1 ) , from this dataset using a conditional generator function G and a fixed time point t. Thus , we aim to model Ẑi , t , τ = G ( n , φ ( i ) , Xt : t+τ−1 ) , where n ∈ Rτ is a noise vector drawn from a Gaussian distribution of mean zero and variance one ; φ is an embedding function that maps the index of a time series to a vector representation , that is concatenated to each time step of Xt : t+τ−1 . An overview of the model architecture is shown in Figure 1 and details about the time features are presented in Appendix A. Spectral Normalised Residual Self Attention with Convolution The generator and discriminator use a main function m that is a composition of convolution , self attention and spectral normalisation m ◦ f : Rnf×l → Rnf×l x 7→ γ SA ( f ( x ) ) + f ( x ) ( 1 ) where f ( x ) = LR ( SN ( c ( x ) ) ) and m ( y ) = γ SA ( y ) +y , c is a one dimensional convolution operator , LR the LeakyReLU operator ( Xu et al. , 2015 ) , SN the Spectral Normalisation operator ( Miyato et al. , 2018 ) and SA the Self Attention module . The variable nf is the number of in and out-channels of c , l is the length of the sequence . Following the work of ( Zhang et al. , 2019a ) , the parameter γ is learnable . It is initialized to zero to allow the network to learn local features directly from the building block f , and is later enriched with distant features as the absolute value of gamma increases , hereby more heavily factoring the self-attention term SA . The module m is referenced as Residual Self-Attention in Fig . 1 ( Right ) . Downscaling and Upscaling The following sections mention upscaling ( UP ) and downscaling ( DOWN ) operators that double and halve the length of the time series , respectively . In this work , the upscaling operator is a linear interpolation and the downscaling operator is the average pooling . PSA-GAN PSA-GAN is a progressively growing GAN ( Karras et al. , 2018 ) ; thus , trainable modules are added during training . Hereby , we model the generator and discriminator as a composition of functions : G = gL+1 ◦ ... ◦g1 andD = d1 ◦ ... ◦dL+1 where each function gi and di for i ∈ [ 1 , L+1 ] corresponds to a module of the generator and discriminator . GENERATOR As a preprocessing step , we first map the concatenated input [ n , φ ( i ) , Xt : t+τ−1 ] from a sequence of length τ to a sequence of length 8 , denoted by Z̃0 , using average pooling . Then , the first layer of the generator g1 applies the main function m : g1 : Rnf×2 3 → Rnf×2 3 Z̃0 7→ Z̃1 = m ◦ f ( Z̃0 ) ( 2 ) For i ∈ [ 2 , L ] , gi maps an input sequence Z̃i−1 to an output sequence Z̃i by applying an upscaling of the input sequence and the function m ◦ f : gi : Rnf×2 i+1 → Rnf×2 i+2 Z̃i−1 7→ Z̃i = m ◦ f ( UP ( Z̃i−1 ) ) ( 3 ) The output of gi is concatenated back to the time features Xt : t+τ−1 and forwarded to the next block . Lastly , the final layer of the generator gL+1 reshapes the multivariate sequence Z̃L to a univariate time series Ẑi , t , τ of length τ = 2L+3 using a one dimensional convolution and spectral normalisation . DISCRIMINATOR The architecture of the discriminator mirrors the architecture of the generator . It maps the generator ’ s output Ẑi , t , τ and the time features Xt : t+τ−1 to a score d. The first module of the discriminator dL+1 uses a one dimensional convolution c1 and a LeakyReLU activation function : d ′ L+1 : R1+D , τ → Rnf , τ ( Z̃L+1 , Xt : t+τ−1 ) 7→ ỸL = SN ( LR ( c1 ( Z̃L+1 , Xt : t+τ−1 ) ) ( 4 ) For i ∈ [ L+ 1 , 2 ] , the module di applies a downscale operator and the main function m : di : Rnf×2 i+2 → Rnf×2 i+1 Yi 7→ Yi−1 = DOWN ( m ( Yi ) ) ( 5 ) The last module d1 turns its input sequence into a score : d1 : Rnf×2 3 → R Y1 7→ Y0 = SN ( FC ( LR ( SN ( c ( m ( Y1 ) ) ) ) ) ) ( 6 ) where FC is a fully connected layer . PSA-GAN-C We another instantiation of PSA-GAN in which we forward to each generator block gi knowledge about the past . The knowledge here is a subseries Ẑi , t−LC , LC in the range [ t− LC , t− 1 ] , with LC being the context length . The context Ẑi , t−LC , LC is concatenated along the feature dimension , i.e . at each time step , to the output sequence of gi . It is then passed through a two layers perceptron to reshape the feature dimension and then added back to the output of gi .

Summary: The paper proposes a type of GAN to generate synthetic time series. The authors use the data generated by their proposed model to train forecasting networks, which improves the baselines. They also show that the proposed GAN can itself be used as  a forecasting model and that it has competitive performance to baseline models. Finally, the paper proposes an adaptation of the FID score for time series.