PSA-GAN: PROGRESSIVE SELF ATTENTION GANS
Year: 2021

Abstract:
Realistic synthetic time series data of sufficient length enables practical applications in time series modeling tasks, such as forecasting, but remains a challenge. In this paper we present PSA-GAN, a generative adversarial network (GAN) that generates long time series samples of high quality using progressive growing of GANs and self-attention. We show that PSA-GAN can be used to reduce the error in several downstream forecasting tasks over baselines that only use real data. We also introduce a Frechet-Inception Distance-like score for time series, Context-FID, assessing the quality of synthetic time series samples. We find that Context-FID is indicative for downstream performance. Therefore, Context-FID could be a useful tool to develop time series GAN models.

heading: 1 INTRODUCTION
text: In the past years, methods such as (Salinas et al., 2020; Franceschi et al., 2019; Kurle et al., 2020; de Bézenac et al., 2020; Oreshkin et al., 2019; Rasul et al., 2021; Cui et al., 2016; Wang et al., 2016) have consistently showcased their effectiveness of deep learning in time series analysis tasks. Although these deep learning based methods are effective when sufficient clean data is available, this assumption is not always met in practice. For example, sensor outages can cause gaps in IoT data, which might render the data unusable for machine learning applications (Zhang et al., 2019b). An additional problem is that time series panels often have insufficient size for forecasting tasks, leading to research in meta-learning for forecasting (Oreshkin et al., 2020b). Cold starts are another common problem in time series forecasting where some time series have little and no data (like a new product in a demand forecasting use case). Thus, designing flexible and task-independent models that generate synthetic, but realistic time series for arbitrary tasks poses an important challenge. Generative adversarial networks (GAN) are a flexible model family that has had success in other domains. However, for their success to carry over to time series, synthetic time series data must be of realistic length, which current state-of-the-art synthetic time series models struggle to generate because they often rely on recurrent networks to capture temporal dynamics (Esteban et al., 2017; Yoon et al., 2019).
In this work, we make three contributions: i) We propose PSA-GAN, a progressively growing, convolutional time series GAN model augmented with self-attention (Karras et al., 2018; Vaswani et al., 2017). PSA-GAN scales to long time series because the progressive growing architecture starts modeling the coarse-grained time series features and moves towards modeling fine-grained details during training. The self-attention mechanism captures long-range dependencies in the data (Zhang et al., 2019a). ii) We show empirically that PSA-GAN samples are of sufficient quality and length to boost several downstream forecasting tasks: far-forecasting and data imputation during inference, data imputation of missing value stretches during training, forecasting under cold start conditions, and data augmentation. Furthermore, we show that PSA-GAN can be used as a forecasting model and has competitive performance when using the same context information as an established baseline. iii) Finally, we propose a Frechet Inception distance (FID)-like score (Salimans et al., 2016), ContextFID, leveraging unsupervised time series embeddings (Franceschi et al., 2019). We show that the lowest scoring models correspond to the best-performing models in our downstream tasks and that the Context-FID score correlates with the downstream forecasting performance of the GAN model (measured by normalized root mean squared error). Therefore, the Context-FID could be a useful general-purpose tool to select GAN models for downstream applications.
We structure this work as follows: We discuss the related work in Section 2 and introduce the model in Section 3. In Section 4, we evaluate our proposed GAN model using the proposed Context-FID score and through several downstream forecasting tasks. We also directly evaluate our model as a forecasting algorithm and perform an ablation study. Section 5 concludes this manuscript.

heading: 2 RELATED WORK
text: GANs (Goodfellow et al., 2014) are an active area of research (Karras et al., 2019; Yoon et al., 2019; Engel et al., 2019; Lin et al., 2017; Esteban et al., 2017; Brock et al., 2019) that have recently been applied to the time series domain (Esteban et al., 2017; Yoon et al., 2019) to synthesize data (Takahashi et al., 2019; Esteban et al., 2017), and to forecasting tasks (Wu et al., 2020b). Many time series GAN architectures use recurrent networks to model temporal dynamics (Mogren, 2016; Esteban et al., 2017; Yoon et al., 2019). Modeling long-range dependencies and scaling recurrent networks to higher lengths is inherently difficult and limits the application of time series GANs to short sequence lengths (less than 100 time steps) (Yoon et al., 2019; Esteban et al., 2017). One way to achieve longer realistic synthetic time series is by employing convolutional (van den Oord et al., 2016; Bai et al., 2018; Franceschi et al., 2019) and self-attention architectures (Vaswani et al., 2017). Convolutional architectures are able to learn relevant features from the raw time series data (van den Oord et al., 2016; Bai et al., 2018; Franceschi et al., 2019), but are ultimately limited to local receptive fields and can only capture long-range dependencies via many stacks of convolutional layers. Self-attention can bridge this gap and allow for modeling long-range dependencies from convolutional feature maps, which has been a successful approach in the image (Zhang et al., 2019a) and time series forecasting domain (Li et al., 2019; Wu et al., 2020a). Another technique to achieve long sample sizes is progressive growing, which successively increases the resolution by adding layers to generator and discriminator during training (Karras et al., 2018). Additionally, the synthetic time series should preserve the dynamics of the real data. This can be achieved by using the latent state of an autoencoder network (Yoon et al., 2019). In PSA-GAN, we explore whether unsupervised techniques could instead provide meaningful embeddings that can act as an additional signal to the GAN to model realistic time series.
Our proposal, PSA-GAN, synthesizes progressive growing with convolutions, self-attention, and embeddings into a novel architecture particularly geared towards time series.
Another line of work in the time series field is focused on developing suitable loss functions for modeling financial time series with GANs where specific challenges include heavy tailed distributions, volatility clustering, absence of autocorrelations, among others (Cont, 2001; Eckerli & Osterrieder, 2021). To this end, several models like QuantGAN (Wiese et al., 2020), (Conditional) SigWGAN (Ni et al., 2020; 2021), and DAT-GAN (Sun et al., 2020) have been proposed (see (Eckerli & Osterrieder, 2021) for review in this field). This line of work targets its own challenges by developing new loss functions for financial time series, which is orthogonal to our work, i.e. we focus on neural network architectures for time series GANs and show its usefulness in the context of time series forecasting.
Another challenge is the evaluation of synthetic data. While the computer vision domain uses standard scores like the Inception Score and the Frechet Inception Distance (FID) (Salimans et al., 2016; Heusel et al., 2018), such universally accepted scores do not exist in the time series field. Thus, researchers rely on a Train on Synthetic–Test on Real setup and assess the quality of the synthetic time series in a downstream classification and/or prediction task (Esteban et al., 2017; Yoon et al., 2019). In this work, we build on this idea and assess the GAN models through downstream forecasting tasks. Additionally, we suggest a Frechet Inception Distance-like score that is based on unsupervised time series embeddings (Franceschi et al., 2019). Critically, we want to be able to score the fit of our fixed length synthetic samples into their context of (often much longer) true time series, which is taken into account by the contrastive training procedure in (Franceschi et al., 2019). As we will later show, the lowest scoring models correspond to the best performing models in downstream tasks.

heading: 3 MODEL
text: Problem formulation We denote the values of a time series dataset by zi,t ∈ R, where i ∈ {1, 2, . . . , N} is the index of the individual time series and t ∈ {1, 2, . . . , T} is the time index. Additionally, we consider an associated matrix of time feature vectors X1:T = (x1, . . . ,xT ) in
RD×T . Our goal is to model a time series of fixed length τ , Ẑi,t,τ = (ẑi,t, . . . , ẑi,t+τ−1), from this dataset using a conditional generator function G and a fixed time point t. Thus, we aim to model Ẑi,t,τ = G(n, φ(i),Xt:t+τ−1), where n ∈ Rτ is a noise vector drawn from a Gaussian distribution of mean zero and variance one; φ is an embedding function that maps the index of a time series to a vector representation, that is concatenated to each time step of Xt:t+τ−1. An overview of the model architecture is shown in Figure 1 and details about the time features are presented in Appendix A.
Spectral Normalised Residual Self Attention with Convolution The generator and discriminator use a main function m that is a composition of convolution, self attention and spectral normalisation
m ◦ f :Rnf×l → Rnf×l
x 7→ γ SA(f(x)) + f(x) (1)
where f(x) = LR(SN(c(x))) and m(y) = γ SA(y)+y, c is a one dimensional convolution operator, LR the LeakyReLU operator (Xu et al., 2015), SN the Spectral Normalisation operator (Miyato et al., 2018) and SA the Self Attention module. The variable nf is the number of in and out-channels of c, l is the length of the sequence. Following the work of (Zhang et al., 2019a), the parameter γ is learnable. It is initialized to zero to allow the network to learn local features directly from the building block f , and is later enriched with distant features as the absolute value of gamma increases, hereby more heavily factoring the self-attention term SA. The module m is referenced as Residual Self-Attention in Fig. 1 (Right).
Downscaling and Upscaling The following sections mention upscaling (UP) and downscaling (DOWN) operators that double and halve the length of the time series, respectively. In this work, the upscaling operator is a linear interpolation and the downscaling operator is the average pooling.
PSA-GAN PSA-GAN is a progressively growing GAN (Karras et al., 2018); thus, trainable modules are added during training. Hereby, we model the generator and discriminator as a composition of functions: G = gL+1 ◦ ...◦g1 andD = d1 ◦ ...◦dL+1 where each function gi and di for i ∈ [1, L+1] corresponds to a module of the generator and discriminator.
GENERATOR As a preprocessing step, we first map the concatenated input [n, φ(i),Xt:t+τ−1] from a sequence of length τ to a sequence of length 8, denoted by Z̃0, using average pooling. Then, the first layer of the generator g1 applies the main function m:
g1 : Rnf×2 3 → Rnf×2 3
Z̃0 7→ Z̃1 = m ◦ f(Z̃0) (2)
For i ∈ [2, L], gi maps an input sequence Z̃i−1 to an output sequence Z̃i by applying an upscaling of the input sequence and the function m ◦ f :
gi : Rnf×2 i+1 → Rnf×2 i+2
Z̃i−1 7→ Z̃i = m ◦ f(UP(Z̃i−1)) (3)
The output of gi is concatenated back to the time features Xt:t+τ−1 and forwarded to the next block.
Lastly, the final layer of the generator gL+1 reshapes the multivariate sequence Z̃L to a univariate time series Ẑi,t,τ of length τ = 2L+3 using a one dimensional convolution and spectral normalisation.
DISCRIMINATOR The architecture of the discriminator mirrors the architecture of the generator. It maps the generator’s output Ẑi,t,τ and the time features Xt:t+τ−1 to a score d. The first module of the discriminator dL+1 uses a one dimensional convolution c1 and a LeakyReLU activation function:
d ′
L+1 : R1+D,τ → Rnf ,τ
(Z̃L+1,Xt:t+τ−1) 7→ ỸL = SN(LR(c1(Z̃L+1,Xt:t+τ−1)) (4)
For i ∈ [L+ 1, 2], the module di applies a downscale operator and the main function m:
di : Rnf×2 i+2 → Rnf×2 i+1
Yi 7→ Yi−1 = DOWN(m(Yi)) (5)
The last module d1 turns its input sequence into a score:
d1 : Rnf×2 3
→ R Y1 7→ Y0 = SN(FC(LR(SN(c(m(Y1))))))
(6)
where FC is a fully connected layer.
PSA-GAN-C We another instantiation of PSA-GAN in which we forward to each generator block gi knowledge about the past. The knowledge here is a subseries Ẑi,t−LC ,LC in the range [t− LC , t− 1], with LC being the context length. The context Ẑi,t−LC ,LC is concatenated along the feature dimension, i.e. at each time step, to the output sequence of gi. It is then passed through a two layers perceptron to reshape the feature dimension and then added back to the output of gi.

heading: 3.1 LOSS FUNCTIONS
text: PSA-GAN is trained via the LSGAN loss(Mao et al., 2017). We used the LSGAN loss because it has been shown to address mode collapse (Mao et al., 2017). Furthermore, least-squares type losses in embedding space have been shown to be effective in the time series domain (Mogren, 2016; Yoon et al., 2019). Additionally, we use an auxiliary moment loss 7 to match the first and second order moments between the batch of synthetic samples and a batch of real samples:
ML ( Ẑτ ,Zτ ) = ∣∣∣µ(Ẑτ)− µ (Zτ )∣∣∣+ ∣∣∣σ (Ẑτ)− σ (Zτ )∣∣∣ (7) where µ is the mean operator and σ is the standard deviation operator. The real and synthetic batches have their time index and time series index aligned. We found this combination to work well for PSA-GAN empirically. Note that the choice of loss function was not the main focus of this study and we think that our choice can be improved in future research.
Training procedures GANs are notoriously difficult to training, have hard-to-interpret learning curves, and are susceptible to mode collapse. The training procedure to address those issues is presented in the appendix.

heading: 4 EXPERIMENTS
text: The evaluation of synthetic time series data from GAN models is challenging and there is no widely accepted evaluation scheme in the time series community. We evaluate the GAN models by two guiding principles: i) Measuring to what degree the time series recover the statistics of the training dataset. ii) Measuring the performance of the GAN models in challenging, downstream forecasting scenarios.
For i), we introduce the Context-FID (Context-Frechet Inception Distance) score to measure whether the GAN models are able to recover the training set statistics. The FID score is widely used for evaluating synthetic data in computer vision (Heusel et al., 2018) and uses features from an inception network (Szegedy et al., 2015) to compute the difference between the real and synthetic sample statistics in this feature space. In our case, we are interested how well the the synthetic time series windows ”fit” into the local context of the time series. Therefore, we use the time series embeddings from Franceschi et al. (2019) that uses a contrastive learning scheme to learn embeddings of time series that fit into the local context. Note that we train the embedding network for each dataset separately. This allows us to directly quantify the quality of the synthetic time series samples.
For ii), we set out to mimic several challenging time series forecasting tasks that are common for time series forecasting practitioners. These tasks often have in common that practitioners face missing or corrupted data during training or inference, which increases the error of deep-learning based forecasting models. Here, we set out to use the synthetic samples to complement an established baseline model, DeepAR, in these forecasting tasks. These tasks are: Far-forecasting and missing values during inference, missing values during training, cold starts, and data augmentation. We evaluate these tasks by the normalized root mean squared distance (NRMSE) Additionally, we evaluate PSA-GAN model when used as a forecasting model. Note that, where applicable, we re-train our GAN models on the modified datasets to ensure that they have the same data available as the baseline model during the downstream tasks.
We also considered NBEATS (Oreshkin et al., 2020a) and Temporal Fusion Transformer (TFT) (Lim et al., 2019) as alternative forecasting models. However, we found that DeepAR performed best in our experiments and therefore we report these results in the main text. Please refer to the Appendix for the evaluation of NBEATS and TFT (Tables 5-9 and Figures 13-16).
In addition, we perform an ablation study of our model and discuss whether the Context-FID scores are indicative for downstream forecasting tasks and their correlation with forecasting performance of the GAN model. For reproducibility, we will make the model code available on Github along with scripts to replicate the experiments.

heading: 4.1 CONTEXT-FID SCORE ON TIME SERIES EMBEDDINGS
text: To compute the Context-FID, we replace InceptionV3 (Szegedy et al., 2015) with the encoder E of Franceschi et al. (2019), which we train separately for each dataset. Our proposed Context-FID score is computed as follows: we first select a time range [t, t + τ ]. We then sample a batch of synthetic time series Ẑt,τ = [Ẑ1,t,τ , ..., ẐN,t,τ ] and a batch of real time series Zt,τ = [Z1,t,τ , ..., ZN,t,τ ] that we encode with E into Ẑet,τ and Z e t,τ respectively. Finally, we compute the FID score of the embeddings.

heading: 4.2 DATASETS AND BASELINES
text: We use the following public, standard benchmark datasets in the time series domain: M4, hourly time series competition data (414 time series) (Makridakis et al., 2020); Solar, hourly solar energy collection data in Alabama State (137 stations) (Lai et al., 2017); Electricity, hourly electricity consumption data (370 customers) (Dheeru & Karra Taniskidou, 2017); Traffic: hourly occupancy rate of lanes in San Francisco (963 lanes) (Dheeru & Karra Taniskidou, 2017). Unless stated otherwise, we split all data into a training/test set with a fixed date and use all data before that date for training. For testing, we use a rolling window evaluation with a window size of 32 and seven windows. We minmax scale each dataset to be within [0, 1] for all experiments in this paper (we scale the data back before evaluating the forecasting experiments). In lieu of finding public datasets that represent the downstream forecasting tasks, we modify each the datasets above to mimic each tasks for the respective experiment (see later sections for more details).
We compare PSA-GAN with different GAN models from the literature (TIMEGAN (Yoon et al., 2019) and EBGAN (Zhao et al., 2017)). In what follows PSA-GAN-C and PSA-GAN denote our proposed model with and without context, respectively. In the forecasting experiments, we use the GluonTS (Alexandrov et al., 2019) implementation of DeepAR which is a well-performing forecasting model and established baseline (Salinas et al., 2019).

heading: 4.3 EVALUATION RESULTS
text: Direct Evaluation with Context-FID Scores: Table 1 shows the Context-FID scores for PSA-GAN, PSA-GAN-C and baselines. For all sequence lengths, we find that PSA-GAN or PSA-GAN-C consistently produce the lowest Context-FID scores. For a sequence length of 256 time steps, TIMEGAN is the second best performing model for all datasets. We show samples from our GAN model in the dowstream task section (Figures 3) and in the Appendix. Note that even though using a context in the PSA-GAN model results in the best overall performance, we are interested to use the GAN in downstream tasks where the context is not available. Thus, the next section will use PSA-GAN without context, unless otherwise stated.
Far-forecasting: In this experiment, we forecast far into the future by assuming that the data points between the training end time and the rolling evaluation window are not observed. For example, the last evaluation window would have 32 ∗ 6 unobserved values between the training end time and the forecast start date. This setup reflects two possible use cases: Forecasting far into the future (where no context data is available) and imputing missing data during inference because of a data outage just before forecasting. Neural network-based forecasting models such as DeepAR struggle under these conditions because they rely on the recent context and need to impute these values during inference. Furthermore, DeepAR only imputes values in the immediate context and not for the lagged values. Here, we use the GAN models during inference to fill the missing observations with synthetic data. As a baseline, we use DeepAR and impute the missing observations of lagged values with a moving average (window size 10) during inference. Here, we find that using the synthetic data from GAN models drastically improve over the DeepAR baseline and using samples from PSA-GAN results into the lowest NRMSE for three out of four datasets (see left Table in Figure 2). Figure 2 also shows the NRMSE as a function of the forecast window. When more values are unobserved, the error of DeepAR increases sharply while using the PSA-GAN samples during inference results in lower error. Figure 3 shows an example of a imputed time series and the resulting forecast from the electricity dataset when using PSA-GAN.
Missing Value Stretches: Missing values are present in many real world time series applications and are often caused by outages of a sensor or service that generates a time series (Zhang et al., 2019b). Therefore, missing values in real-world scenarios are often not evenly distributed over the time series but instead form missing value ”stretches”. In this experiment, we simulate missing value stretches and remove time series values of length 50 and 110 from the datasets. This results into 5.4-7.7% missing values for a stretch length of 50 and and 9.9-16.9% missing values for a stretch length of 110 (depending on the dataset.) Here, we split the training dataset into two parts along the time axis and only introduce missing values in the second part. We use the first (unchanged) part of the training dataset to train the GAN models and both parts of the training set to train DeepAR. We then use the GAN models to impute missing values during training and inference of DeepAR.
NRMSE
Dataset Electricity M4 Solar Traffic Model
DeepAR 3.31±0.78 0.84±0.44 1.43±0.02 1.0±0.15 EBGAN 2.46±0.3 1.26±0.29 2.02±0.07 0.9±0.22
TIMEGAN 1.57±0.25 1.32±0.13 1.21±0.05 0.86±0.17 PSA-GAN 0.99±0.44 0.62±0.3 1.22±0.05 0.59±0.2
1 2 3 4 5 6 7 Forecast window
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 NR M SE
DeepAR EBGAN TIMEGAN PSA-GAN (ours)
Figure 4 shows that using the GAN models to impute missing values during training DeepAR on data with missing value stretches reduces its forecasting error. While all GAN models reduce the NRMSE in this setup, PSA-GAN is most effective in reducing the error in this experiment. The Appendix shows a detailed Figure with a split by dataset.
Cold Starts: In this experiment, we explore whether the GAN models can support DeepAR in a cold start forecasting setting. Cold starts refer to new time series (like a new product in demand forecasting) for which little or no data is available. Here, we randomly truncate 10%, 20%, and 30% of the time series from our datasets such that only the last 24 (representing 24 hours of data) values before inference are present. We only consider a single forecasting window in this experiment. We then again use the GAN models to impute values for the lags and context that DeepAR conditions for
NRMSE
Dataset Electricity M4 Solar Traffic Model
DeepAR 0.49±0.03 0.26±0.02 1.07±0.01 0.45±0.01 EBGAN 0.64±0.05 0.32±0.09 1.18±0.03 0.50±0.03
TIMEGAN 0.56±0.05 0.28±0.02 1.06±0.02 0.47±0.004 PSA-GAN 0.64±0.22 0.3±0.04 1.07±0.03 0.50±0.11
forecasting the cold start time series. Cold start time series are not used for training the GAN models. Figure 5 shows the forecasting error of the different models for the cold start time series only. In this experiment, PSA-GAN and TIMEGAN improve the NRMSE over DeepAR and are on-par overall (mean NMRSE 0.70 and 0.71 for PSA-GAN and TIMEGAN, respectively). The Appendix shows a detailed Figure with a split by dataset.
Data Augmentation: In this experiment, we average the real data and GAN samples to augment the data during training. 1 During inference, DeepAR is conditioned on real data only to generate forecasts. In our experiments, none of the GAN models for data augmentation consistently improves over DeepAR. Overall, TIMEGAN is the best-performing GAN model but plain DeepAR still performs better. This finding is aligned with recent work in the image domain where data augmentation with GAN samples does not improve in a downstream task (Ravuri & Vinyals, 2019). We hypothesize that the GAN models cannot improve in the data augmentation setup because they are trained to generate realistic samples and not necessarily to produce relevant invariances. Furthermore, the dataset sizes might be sufficient for DeepAR to train a well-performing model and therefore the augmentation might not be able to reduce the error further. More research is required to understand whether synthetic data can improve forecasting models via data augmentation.
Forecasting Experiments: In this experiment, we use the GAN models directly for forecasting (Table 3). We can see that DeepAR consistently performs best. This is expected as DeepAR takes into account context information and lagged values. This kind of information is not available to the GAN models. To test this we further consider PSA-GAN-C, i.e. PSA-GAN with 64 previous time series values as context, and further evaluate DeepAR with restricted lags to the previous 64 values (DeepAR-C). We can see that in this case PSA-GAN-C outperforms DeepAR-C in 3 out of 4 datasets and PSA-GAN performs on par with DeepAR-C. Moreover, both PSA-GAN and PSA-GAN-C are the best performing GAN models. Adding lagged values to the GAN models as
1We also tried using only the GAN generated data for training and experimented with ratios of mixing synthetic and real samples, similar to the work in (Ravuri & Vinyals, 2019). Furthermore, we tried different weights and scheduled sampling but this did not improve the results.
context could further improve their performance and adversarial/attention architectures have been previously used for forecasting (Wu et al., 2020a).
Ablation Study: Figure 6 shows the results of our ablation study where we disable important components of our model: moment loss, self-attention, and fading in of new layers. We measure the performance of the ablation models by Context-FID score. Overall, our proposed PSA-GAN model performs better than the ablations which confirms that these components contribute to the performance of the model.
Low Context-FID Score Models Correspond to Best-performing Forecasting Models: One other observation is that the lowest Context-FID score models correspond to the best models in the data augmentation and far-forecasting experiments. PSA-GAN and TIMEGAN produce the lowest Context-FID samples and both models also improve over the baseline in most downstream tasks. Overall, PSA-GAN has lowest Context-FID and also outperforms the other models in the downstream tasks, except for the cold start task. Additionally, we calculated of the Context-FID scores for the ablation models mentioned in Figure 6 (but with a target length of 32) and the RMSE of these models in the forecasting experiment (as in Table 3). We find that the Pearson correlation coefficient between the Context-FID and forecasting NRMSE is 0.71 and the Spearman’s rank correlation coefficient is 0.67, averaged over all datasets. All datasets (except Traffic) have a correlation coefficient of at least 0.73 in either measure (see Table 4 in the Appendix).This suggests that the Context-FID score could be a useful score to develop GAN models for downstream use in forecasting applications.

heading: 5 CONCLUSION
text: We have presented PSA-GAN, a progressive growing time series GAN augmented with self-attention, that produces long realistic time series and improves downstream forecasting tasks that are challenging for deep learning-based time series models. Furthermore, we introduced the Context-FID score to assess the quality of synthetic time series samples produced by GAN models. We found that the lowest Context-FID scoring models correspond to the best-performing models in downstream tasks. We believe that time series GANs that scale to long sequences combined with a reliable metric to assess their performance might lead to their routine use in time series modeling.

heading: A TIME SERIES FEATURES
text: We use time-varying features to represent the time dimension for the GAN models to enable training on time series windows sampled from the full-length time series . We use HourOfDay, DayOfWeek, DayOfMonth and DayOfYear features. Each feature is encoded using a single index number (for example, between [0, 365[ for DayOfYear) and normalized to be within [-0.5, 0.5]. We also use an age feature to represent the age of the time series to be log(2.0 + t) where t is the time index. Additionally, we embed the index of the individual time series into a 10 dimensional vector with an embedding network to and use it as a non time-varying feature. DeepAR uses the same features in forecasting.

heading: B TRAINING PROCEDURE
text: PSA-GAN uses spectral normalization and progressive fade-in of new layers to stabilize training.
Progressive fade in of new layers As training progresses, new layers are added to double the length of time series. However, simply adding new untrained layers drastically changes the number of parameters and the loss landscape, which destabilizes training. Progressive fade in of new layers is a technique introduce by Karras et al., 2018 to mitigate this effect where each layer gi and di is smoothly introduced. For i ∈ [2, L] the fade in of new layers is written as:
gi :Rnf×2 i+1 → Rnf×2 i+2
Z̃i−1 7→ Z̃i = αm(UP(Z̃i−1)) + (1− α)UP(Z̃i−1) (8)
di :Rnf×2 i+2 → Rnf×2 i+1
Yi 7→ Yi−1 = αDOWN(m(Yi)) + (1− α)DOWN(Yi) (9)
where α is a scalar initialized to be zero and grows linearly to one over 500 epochs.
Spectral Normalisation Both the generator and the discriminator benefit from using spectral normalisation layers. In the discriminator, Spectral Normalisation stabilizes training (Miyato et al., 2018). It constrains the Lipschitz constant of the discriminator to be bounded by one. In the generator the spectral normalisation stabilises the training and avoids escalation of the gradient magnitude (Zhang et al., 2019a).

heading: C TRAINING DETAILS
text: PSA-GAN has been trained over 6500 epochs, where a new block is added every 1000 epochs and is faded over 500 epochs. At each epoch, PSA-GAN trains on 100 batches of size 512. It optimises its parameters using the Adam (Kingma & Ba, 2017) with a learning rate of 0.0005 and betas of (0.9, 0.999) for both the generator and the discriminator.
PSA-GAN (and its variants) has been trained on ml.p2.xlarge Amazon instances for 21 hours.
The number of in-channels and out-channels nf in the Main Block (Figure 1, Right) equals 32. The number of out-channels in the convolution layer c equals 1 (Eq.6 ). The number of time features D equals 15.
For the case of PSA-GAN-C, the MLP block is a two layers perceptron with LeakyReLU. It incorporates knowledge of the past by selecting the LC = 64 time points and concatenates them with its input.

heading: D HYPERPARAMETER TUNING
text: We did limited hyperparameter tuning in this study to find default hyperparemters that perform well across datasets. We performed the following grid-search: the batch size, the number of batches used per epoch, the presence of Self-Attention and the presence of the Progressive fade in of new layers mechanism explained in section B, paragraph B. The range considered for each hyper-parameter is: batch size : [64, 128, 256, 512], the number of batches used per epoch: [50, 100, 200], the presence of Self-Attention: [True,False] and the presence of the Progressive fade in of new layers: [True,False].
The set of hyperparameters is shared across the four datasets we train PSA-GAN on (which means we do not tune hyperparameters per datasets). We select the final hyperparameter set by lowest Context-FID score on the training set, averaged across all datasets.

heading: E FORECASTING EXPERIMENT DETAILS
text: We use the DeepAR (Salinas et al., 2020) implementation in GluonTS (Alexandrov et al., 2019) for the forecasting experiments. We use the default hyperparameters of DeepAR
with the following exceptions: epochs=100, num batches per epoch=50, dropout rate=0.01, scaling=False, prediction length=32, context length=64, use feat static cat=True. We use the Adam optimizer with a learning rate of 1e−3 and weight decay of 1e−8 . Additionally, we clip the gradient to 10.0 and reduce the learning rate by a factor of 2 for every 10 consecutive updates without improvement, up to a minimum learning rate of 5e−5. DeepAR also uses lagged values from previous time steps that are used as input to the LSTM at each time step t. We set the time series window size that DeepAR samples to 256 and truncate the default hourly lags to fit the context window, the prediction window, and the longest lag into the window size 256. We use the following lags: [1, 2, 3, 4, 5, 6, 7, 23, 24, 25, 47, 48, 49, 71, 72, 73, 95, 96, 97, 119, 120, 121, 143, 144, 145].
For NBEATS (Oreshkin et al., 2020a), we use the default parameters and set the trainer settings to epochs=100, num batches per epoch=50. Note that in the original paper, NBEATS is an ensemble of 180 networks. Due to the high computational cost, we only ensemble three networks with the settings context length=64, context length=96, context length=128 and mean absolute percentage error as the loss function.
For TFT Lim et al. (2019), we use the default hyperparameters except: epochs=100, num batches per epoch=50, context length=128.
For all methods, we used the implementations available in GluonTS (Alexandrov et al., 2019).

heading: F ADDITIONAL EXPERIMENTS

heading: G SAMPLE FORECASTS
text: We present more sample forecasts under the framework of the forecasting experiment. In Figs. 9, 10, 11, 12 we present sample forecast for Electricity, M4, Solar Energy, and Traffic datasets, respectively. We can see that the synthetic time series generated by PSA-GAN and PSA-GAN-C consistently convey a better approximation to the ground truth values. One can further observe that this task is particularly challenging for GANs as data generation is located in a time-frame that is not covered while training.

Review summaries:
The authors propose a new GAN-based algorithm for time series synthesis. They use progressive growing of GAN architectures to improve the performance of GAN and self-attention to enhance the expressive capability of neural networks. The experimental results validate the superiority of the proposed PSA-GAN algorithm.

The paper proposes a type of GAN to generate synthetic time series. The authors use the data generated by their proposed model to train forecasting networks, which improves the baselines. They also show that the proposed GAN can itself be used as  a forecasting model and that it has competitive performance to baseline models. Finally, the paper proposes an adaptation of the FID score for time series.

This paper proposes a GAN-based approach (PSA) for generating realistic time-series data that can be used to improve several downstream tasks such as imputation of missing data and forecasting. The generator and discrimination model in the proposed GAN framework, PSA-GAN consists of progressively growing blocks consisting of convolutions and self-attention. The paper also proposes a metric for evaluating the quality of synthetic time series data. Experiments show that the proposed approach is able to achieve better performance than previous GAN-based approaches for time series and is also able to improve on downstream forecasting tasks by augmenting forecasting models.