PSA-GAN: Progressive Self Attention GANs for Synthetic Time Series
Authors: Paul Jeha
Michael Bohlke-Schneider
Pedro Mercado
Shubham Kapoor
Rajbir Singh Nirwan
Valentin Flunkert
Jan Gasthaus
Tim Januschowski
Venue: ICLR 2022 Poster (2022)
Keywords: Synthetic Time Series
GAN
Generative Modeling
Time Series
Forecasting

Abstract: Realistic synthetic time series data of sufficient length enables practical applications in time series modeling tasks, such as forecasting, but remains a challenge. In this paper we present PSA-GAN, a generative adversarial network (GAN) that generates long time series samples of high quality using progressive growing of GANs and self-attention. We show that PSA-GAN can be used to reduce the error in several downstream forecasting tasks over baselines that only use real data. We also introduce a Frechet-Inception Distance-like score for time series, Context-FID, assessing the quality of synthetic time series samples. We find that Context-FID is indicative for downstream performance. Therefore, Context-FID could be a useful tool to develop time series GAN models. 

Published as a conference paper at ICLR 2022
PSA-GAN: PROGRESSIVE SELF ATTENTION GANs FOR S SYNTHETIC TIME SERIES
Paul Jeha*s
Michael "
Pedro Mercado AWS Al Labs
Technical University of Denmark paujeedtu.dk
AWS AI Labs boh kem@ ama zon com
pedroml 0 @amazon,
Shubham Kapoor
Rajbir Singh Nirwan AWS Labs n rwwrramazaaaaaooa...mm com
Valentin Flunkert AWS AI Labs fl flunkert@amazzn com
AWS Al Labs
kapooshueann ama zon com
Jan Gasthaus
Tim Januschowski Zalando SE tim. jaa.aa/ascchossleelaaddeded
AWS AI Labs
gasthaust amazon. com
ABSTRACT
Realistic synthetic time series data of sufficient length enables practical applications in time series modeling tasks, such as forecasting, but remains a challenge. In this paper, present PSA-CAN, generative adversarial network (GAN) that generates long time series samples of high quality using progressive growing of GANs and self-attention. We show that PSA-GAN can be used reduce the error in several downstream forecasting tasks over baselines that only use real data. We also introduce Frechet Inception distance-lit score for time series, Context-FID, assessing the quality of synthetic time series samples. We find that Context-FID is indicative for downstream performance. Therefore, Context-FIl could useful tool to develop time series GAN models.
INTRODUCTION
In the past years, methods such (Salinas al., 2020; Franceschi et al., 2019; Kurle al., 2020; de Bézenac et al., 2020; Oreshkin et al., 2020a; Rasul et al., 2021; Cui et al., 2016; Wang et al., 2017) have consistently showcased the effectiveness of deep learning in time series analysis tasks. Although these deep learning based methods are effective when sufficient and clean data is available, this assumption is not always met in practice. For example, sensor outages can cause gaps IoT data, which might render the data unusable for machine learning applications (Zhang et al., 2019b). An additional problem is that time series panels often have insufficient size for forecasting tasks, leading to research in meta-learning for forecasting (Oreshkin et al., 2020b). Cold starts another common problem in time series forecasting where some time series have little and no data (like product in demand forecasting case). Thus, designing flexible and task-independent models that generate synthetic, but realistic time series for arbitrary tasks are an important challenge. Generative adversarial networks (GAN) are flexible model family that has had success in other domains. However, for their success to carry over to time series, synthetic time series data must be of realistic length, which current ofate-of-the-eee synthetic time series models struggle to generate because they often rely on recurrent networks to capture temporal dynamics (Esteban et al., 2017; Yoon et al., 2019).
In this work, we make three contributions: We propose PSA-GAN progressively growing, convolutional time series GAN model augmented with self-attention (Karras et al., 2017: Vaswani al., 2017). PSA- GAN scales to long time series because the progressive growing architecture starts
"Equal contribution.
Work done while Amazon/AWS AT Labss

---

Published as a conference paper at ICLR 2022
modeling the coarse saase--rained time series features and moves towards modeling fine- tne-ggaaind details during training. The self-attention mechanism captures long-range dependencies in the data (Zhang al., 2019a). ii) We show empirically that PSA-GAN samples are of sufficient quality and length to boost several downstream forecasting tasks: far-forecasting and data imputation during inference, data imputation of missing value stretches during training, forecasting under cold start conditions, and data augmentation. Furthermore, we show that PSA -GAN can be used as a forecasting model and has competitive performance when using the same context information as an established baseline. iii) Finally, we propose a Frechet Inception distance (FID)-like score (Salimans et al., 2016), Context- FID, leveraging unsupervised time series embeddings (Franceschi et al., 2019). We show that the lowest scoring models correspond to the best-performing models in our downstream tasks and that the Context-FID score correlates with the downstream forecasting performance of the GAN model (measured by normalized root mean squared error). Therefore, the Context-FID could be useful general-purposs tool to select GAN models for downstream applications.
We structure this work as follows: We discuss the related work in Section 2 and introduce the model in Section 3. In Section 4, we evaluate our proposed GAN model using the proposed Context-FID score and through several downstream forecasting tasks. We also directly evaluate model as forecasting algorithm and perform an ablation study. Section 5 concludes this manuscript.
2 RELATED WORK
GANs (Goodfellow al., 2014) are an active area of research (Karras al., 2019; Yoon et al., 2019; Engel et al., 2019; Lin et al., 2017; Esteban et al., 2017: Brock et al., 2018) that have recently been applied to the time series domain (Esteban et al., 2017; Yoon et al., 2019) to synthesize data (Takahashi et al., 2019; Esteban al., 2017), and to forecasting tasks (Wu et al., 2020). Many time series GAN architectures use recurrent networks to model temporal dynamics (Mogren, 2016; Esteban et al., 2017: Yoon et al., 2019). Modeling long-range dependencies and scaling recurrent networks to long sequences is inherently difficult and limits the application of time series GANs to short sequence lengths (less than 100 time steps) (Yoon et al., 2019; Esteban et al., 2017). One way to achieve longer realistic synthetic time series is by employing convolutional (van den Oord et al., 2016; Bai et al., 2018; Franceschi al., 2019) and self-attention architectures (Vaswani et al., 2017). Convolutional architectures are able to learn relevant features from the raw time series data (van den Oord et al., 2016; Bai et al., 2018; Franceschi et al., 2019), but are ultimately limited to local receptive fields and can only capture long-range dependencies via many stacks of convolutional layers. Self-attention can bridge this gap and allow for modeling long-range dependencies from convolutional feature maps, which has been a successful approach in the image (Zhang et al., 2019a) and time series forecasting domain (Li et al., 2019; Wu et al., 2020). Another technique to achieve long sample sizes is progressive growing, which successively increases the resolution by adding layers to generator and discriminator during training (Karras et al., 2017). Our proposal,
synthesizes progressive growing with convolution and self-attentic into novel architecture particularly geared towards time series.
Another line of work in the time series field is focused on developing suitable loss functions for modeling financial time series with GANs where specific challenges include heavy tailed distributions, volatility clustering, absence of autocorrelations, among others (Cont, 2001: Eckerli & Osterrieder, 2021). To this end, several models like QuantGAN (Wiese et al., 2020), (Conditional) SigWGAN (Ni al., 2020; 2021), and DAT-GAN (Sun et al., 2020) have been proposed (see (Eckerli & Osterrieder, 2021) for review in this field). This line of work targets its own challenges by developing new loss functions for financial time series, which is orthogonal to our work, i.e. we focus on neural network architectures for time series GANs and show its usefulness in the context of time series forecasting.
Another challenge is the evaluation of synthetic data. While the computer vision domain uses standard scores like the Inception Score and the Frechet Inception distance (FID) (Salimans et al., 2016; Heusel et al., 2017), such universally accepted scores do not exist in the time series field. Thus, researchers rely on Train Synthetic-Test in a downstream classification and/or prediction task (Esteban et al., 2017; Yoon et al., 2019). In this work, we build on this idea and assess the GAN models through downstream forecasting tasks. Additionally, we suggest a Frechet Inception distance-like score that is based on unsupervised time series embeddings (Franceschi et al., 2019). Critically, we want to be able to score the fit of our fixed
Real setup and assess the quality of the synthetic time series
2

---

Published as a conference paper at ICLR 2022
length synthetic samples into their context of (often much longer) true time series, which is taken into account by the contrastive training procedure in Franceschi et al. (2019). As we will later show, the lowest scoring models correspond to the best performing models in downstream tasks.
MODEL.
Problem formulation We denote the values of a time series dataset by 21,0 E R. where i e , is the index of the individual time series and {1,2, T" is the time index. Additionally, we consider an associated matrix of time feature vectors X1:T = (x1....,xp .,r) in RDxT. Our goal is to model a time series of fixed length T, 2r,.r = (Ei, 2. from this dataset using& a conditional generator function G and fixed time point Thus, we aim to model Z = (n, (i), 71), where n E RT is noise vector drawn from a Gaussian distribution of mean zero and variance one: is embedding function that maps the index of time series to vector representation, that concatenated to each time step of XI An overview the model architecture is shown in Figure I and details about the time features are presented in Appendix A.
Generator
Discriminator
Main Block
me Fertverr
Residuel
Main Blook
Upsccalin
Main Block
Fale lmngt 2.
Conveltin
Speciall I Nam
Dowesccoling
Main Block
Main Block
Spiural Nomm In Cerencce
Sulf Amertion
Spoce
Leoky ReLU
Bew lach
2
Spectral
Convolution
Figure I: Left: Architecture of our proposed model, PSA GAN The generator contains blocks where each doubles the size of the output, by linear interpolation. contains (dashed drawing) a multi layer perceptron block, that incorporates knowledge of the past in the generator. This block is used in the PSA-GAN-C model. The discriminator contains n blocks that halve the using average pooling. Right: The main block used in the generator and discriminator.
size of the input
Spectral Normalised Residual Self- Celf-Attention with Convolution The generator and discriminator use main function m that a composition of convolution, self-attention and spectral normalisation
f F - Rayxi x + (S((f()) + f(x)
where f(x) = (R(SN(c(z))) and m(y) 7 SA( + y. e is a one dimensional convolution operator, I.R the LeakyReLU operator (Xu et al., 2015), SN the spectral normalisation operator (Miyato et al., 2018) and SA the self-attention module. The variable n, is the number of in and out-channels of is the length of the sequence. Following the work of (Zhang et al., 2019a), the parameter ? is learnable. It is initialized to zero to allow the network to leam local features directly from the building block of and is later enriched with distant features as the absolute value of gamma increases. hereby more heavily factoring the self-attention term SA. The module is referenced as residual self-attention Figure (right).
Downscaling and Upscaling The following sections mention upscaling (UP) and downscaling (DOWN) operators that double and halve the length of the time series, respectively. In this work, the upscaling operator is linear interpolation and the downscaling operator is the average pooling.
PSA-GAN PSA-GAN is progressively growing GAN (Karras et al., 2017); thus, trainable modules added during training. Hereby,
model the generator and discriminator as a composition of functions: G = 91++1 o... o 91 and D = d, o... d where each function g; and d, for i É [1,L+ ---+1 corresponds to module of the generator and discriminator.
3

---

Published as a conference paper at ICLR 2022
GENERATOR As preprocessing step, first map the concatenated input [n, o(1), Xt:++-1] from sequence of length to sequence of length 8, denoted by Zo, using average pooling. Then, the first layer of the generator 9i applies the main function m.
91 : R
(2)
Zo + 21 = m f(Zo)
For i [2, L.. 9: maps input sequence 7 to output sequence Z, by applying upscaling of the input sequence and the function m o f:
R X2
12""X2"
9i
2 + 2, m f(UP(2;-1))
The output gi is concatenated back the time features X and forwarded to the next block.
Lastly, the final layer of the generator gr.-1 reshapes the multivariate sequence 2r. univariate time series Zi,,r of length 2 using a one dimensional convolution and spectral normalisation.
DISCRIMINATOR
The architecture of the discriminator mirrors the architecture of the generator. It maps the generator's output ,,,,, and the time features X to score d. The first module of the discriminator d++1 uses one dimensional convolution e1 and a LeakyReLUU activation function:
dun1: 1 + g.
(4)
(ăL+1, X + Y EN(LR(cI(Z411 (241
For [L+
+ 1, 2). the module de applies downscale operator and the main function m:
d, x2 + RR x2
(5)
Y DOWN(m((Y))
The last module dj turns its input sequence into score:
d1 R x2 + R
(6)
Y Yo SXVECLR(((((((())))
where FC is fully connected layer.
PSA-GAN-C We introduce another instantiation of PSA-GAN which we forward to each generator block 9 knowledge about the past. The knowledge here is a sub-series 2,,-- - Lc.Le in the range !- Lc, 1]. with Le being the context length. The context 2i, I.c is concatenated along the feature dimension, i.e. at each time step, to the output sequence of gt- then passed through a two layers perceptron to reshape the feature dimension and then added back to the output of g,.
3.1 LOss FUNCTIONS
PSA PAA-GAN is trained via the LSGAN loss (Mao et al., 2017), since it has been shown to address mode collapse (Mao et al., 2017). Furthermore, least-squares type losses embedding space have been shown to be effective in the time series domain (Mogren, 2016; Yoon et al., 2019). Additionally, we use auxiliary moment loss match the first and second order moments between the batch of synthetic samples and batch of real samples:
ML (r,, z. ( - -
(2.) (Z.)
where y is the mean operator and is the standard deviation operator. The real and synthetic batches have their time index and time series index aligned. We found this combination to work well for PSA-GAB empirically. Note that the choice of loss function was not the main focus of this study and think that our choice be improved in future research.
4

---

Published as a conference paper at ICLR 2022
Training procedures GANs are notoriously difficult to train, have hard-to-interpret learning curves, and are susceptible to mode collapse. The training procedure to address those issues together with other training and tuning details is presented in Appendix B-E.
4 EXPERIMENTS
The evaluation of synthetic time series data from GAN models is challenging and there is widely accepted evaluation scheme in the time series community. We evaluate the GAN models by two guiding principles: i) Measuring what degree the time series recover the statistics of the training dataset. ii) Measuring the performances of the GAN models in challenging, downstream forecasting scenarios.
For i), we introduce the Context-FID (Context-Fiechhe Inception distance) score to measure whether the GAN models are able to recover the training set statistics. The FID score widely used for evaluating synthetic data in computer vision (Heusel et al., 2017) and uses features from an inception network (Szegedy
al., 2016) to compute the difference between the real and synthetic sample statistics in this feature space. In our case, we are interested in how well the the synthetic time series windows "fit" into the local context of the time series. Therefore, we use the time series embeddings from Franceschi al. (2019) to learn embeddings of time series that fit into the local context. Note that we train the embedding network for each dataset separately. This allows us to directly quantify the quality of the synthetic time series samples (see Appendix D for details).
For ii), we set out to mimic several challenging time series forecasting tasks that are common for time series forecasting practitioners. These tasks often have in common that practitioners face missing or corrupted data during training or inference. Here, we set out to use the synthetic samples to complement established baseline model, DeepAR, in these forecasting tasks. These tasks are: far-forecasting and missing values during inference, missing values during training, cold starts, and data augmentation. We evaluate these tasks by the normalized root mean squared distance (NRMSE). Additionally, we evaluate P applicable, we re-train our GAN models on the modified datasets to ensure that they have the same data available as the baseline model during the downstream tasks.
model when used as forecasting model. Note that, where
We also considered NEEATS (Oreshkin et al., 2020a) and Temporal Fusion Transformer (IFT) (Lim et al., 2021) as alternative forecasting models. However, we found that DeepAR performed best in our experiments and therefore we report these results in the main text (see Appendix F for experiment details). Please refer to Appendix G for the evaluation of NBEATS and TFT (Tables S2-S6 and Figures S3-S6).
In addition, we perform an ablation study of our model and discuss whether the Context-FID scores are indicative for downstream forecasting tasks.
4.1 DATASETS AND BASELINES
We use the following public, standard benchmark datasets in the time series domain: M4, hourly time series competition data (414 time series) (Makridakis et al., 2020); Solar, hourly solar energy collection data in Alabama State (137 stations) (Lai et al., 2018): Electricity, hourly electricity consumption data (370 customers) (Dheeru & Karra Taniskidou, 2017); Traffic: hourly occupancy rate of lanes in San Francisco (963 lanes) (Dheeru & Karra Taniskidou, 2017). Unless stated otherwise, we split all data into a training/test set with a fixed date and use all data before that date for training. For testing, we use rolling window evaluation with window size of 32 and seven windows. We minmax scale each dataset to be within [0, 1] for all experiments in this paper (we scale the data back before evaluating the forecasting experiments). In lieu of finding public datasets that represent the downstream forecasting tasks, modify each the datasets above to mimic each tasks for the respective experiment (see later sections for more details).
We compare PSA GAN with different GAN models from the literature (TIMEGAN (Yoon al., 2019) and EBGAN (Zhao et al., 2017)). In what follows PSA-CAN-C and PSA-GAN denote our proposed model with and without context, respectively. In the forecasting experiments, we use the GluonTS (Alexandrov et al., 2020) implementation of DeepAR which is a werl-peeforming forecasting model and established baseline (Salinas et al., 2020).
5

---

Published as a conference paper at ICLR 2022
Dataset
Length
EBGAN
TIMEGAN
PSA-GAN
PSA-GAN-C
64
2.07+0.3
0.77+v.12
0.018+t.0099
0.011+0.002
Electricity
128
1.93
1.03:0.13
0.0310.01
0.022:0001
256
3.02:0.25
1.58
0.042:1.001
0.036:1002
64
0.93
0.93
0.113 051
0.158:+0.08s
M4
128
2.53:
0.6:0.07
0.156:00.072
0.22
256
2.08
0.89
0.356:1.125
0.235 100008
64
0.48=0.11
0.13
0.012#t0.00 003
0.004
Solar-Energy
128
1.87
0.00
0.007 0.0.00
0.002 #0,001
256
2.38+0.43
0.09
0.064
0.004
64
2.6520.15
0.5922.06
0.241 10.041
0.058
Traffic
128
2.09
2.13+0.12
0..829
0.0444
256
1.86
1.82+v.12
0.13 +o.017
0.488 +0.027
Table 1: Context FID-scores (lower is better) of PSA-GAN and baselines. We score 5120 randomly selected windows and report the mean and standard deviation.
4.2 DIRECT EVALUATION WITH CONTEXT-FID SCORES
Table 1 shows the Context-FID scores for PSA-GAN, PSA-GAN-C and baselines. For all sequence lengths, we find that PSA-GAI PSA-GAN-O consistently produce the lowest Context-FID scores. For a sequence length of 256 time steps, TIMEGAN is the second best performing model for all datasets. Note that even though using context in the PSA-GAN model results in the best overall performance, we are interested to use the GAN in downstream tasks where the context is not available. Thus, the next section will use PSA-GAN without context, unless otherwise stated.
4.3 EVALUATION ON FORECASTING TASKS
In this section, we present the results of the forecasting tasks. We find that synthetic samples do not improve over baselines in all cases. However, we view these results as a first attempt to use GAN models in these forecasting tasks and believe that future research could improve over our results.
For-forecasting: In this experiment, we forecast far into the future by assuming that the data points between the training end time and the rolling evaluation window are not observed. For example, the last evaluation window would have 32 6 unobserved values between the training end time and the forecast start date. This setup reflects two possible use cases: Forecasting far into the future (where no context data is available) and imputing missing data during inference because of a data outage just before forecasting. Neural network- based forecasting models such as DeepAR struggle under these conditions because they rely on the recent context and need to impute these values during inference. Furthermore, DeepAR only imputes values in the immediate context and not for the lagged values. Here, we use the GAN models during inference to fill the missing observations with synthetic data. As a baseline, we use DeepAR and impute the missing observations of lagged values with a moving average (window size 10) during inference. Here, we find that using the synthetic data from GAN models drastically improve over the DeepAR baseline and using samples from PSA-GAN results into the lowest NRMSE for three out of four datasets (see left Table in Figure 2). Figure 2 also shows the NRMSE as a function of the forecast window. Figure 3 shows an example of a imputed time series and the resulting forecast from the Electricity dataset when using PSA-GAN. -GAN.
Missing values are present in many real world time series applications and are often caused by outages of sensor or service (Zhang et al., 2019b). Therefore, missing values in real-world scenarios are often not evenly distributed over the time series but instead form missing value "stretches". In this experiment, we simulate missing value stretches and remove time series values of length 50 and 110 from the datasets. This results into 5.4-7.7% missing values for a stretch length of 50 and and 9.9-16.95 missing values for a stretch length of 110 (depending on the dataset.) Here, split the training dataset into two parts along the time axis and only introduce missing values in the second part. We use the first (unchanged) part of the training dataset to train the GAN models and both parts of the training set train DeepAR. We then the GAN models
Missing Value Stretches:

---

Published as a conference paper at ICLR 2022
NRMSE
ESGAN
Dataset Model
Electricity
M4
Solar
Traffic
TIME
(ours)
2.5
2.0
DeepAR
3.3140.79 0.84
1.43
1,0
1.5
KRGAN
2.4620.3
1.2622.22
2.0220.07
0.9
TIMEGAN
1.5720.25
1.32
1..0
0.86
PSA-GAN
0.99
0.62
1.2220.05
0.59
Forecast Window
Figure 2: NRMSE of far-forecasting experiments (lower is better). Mean and confidence intervals are obtained by re-running each method ten times. Left: NRMSE average over different forecast windows. DeepAR is only run the real time series data. The other models correspond to DeepAR and one of the GAN models for filling missing observations. Right: NRMSE by forecast window. The first window will have no missing values during inference (we forecast the first 32 steps after the training range) and we increase the missing values during inference with each window (the last window will have 32 6 missing values). For DeepAR, the NRMSE increases noticeably at the fourth forecast window while using PSA-GAN to impute the missing values at inference keeps the NRMSE low.
Imputed sequence
Figure 3: Example of the far- forecasting experiment for the Elec- tricity dataset. True data shown in blue and the imputed sequence and forecasts the model orange. The orange line between the dashed lines corresponds to the PSA- GGN imputed sequence and is used by DeepAR for forecasting. Note that this part is unobserved in this periment. The imputed sequence al- lows to generate reasonable fore- cast, even if many data points before the forecast start date are unobserved.
300
275
Electrolty consuniption
250
225
175
150
125
26
27
28
29
30
31
01
02
03
04
05
06
Sep
2014
to impute missing values during training and inference DeepAR. Figure 4 shows that using the GAN models to impute missing values during training DeepAR data with missing value stretches reduces its forecasting error. While all GAN models reduce the NRMSE in this setup, PSA PSA-GAN is most effective in reducing the in this experiment. See Figure S1 for detailed split by dataset.
Stretch length 50
Stretch length 110
Figure 4: Performance of using different GAN models for imputation in the missing value stretch length experiment. Markers denote the mean NRMSE averaged over datasets and error bars the 68% confidence interval over ten runs. For 50 and 110 missing value stretch length, us- ing PSA-GAN to impute missing values results in the lowest overall error, whereas DeepAR returns the highest.
NRMSE
1.4
1.2
1,0
DeepAR
EBGAN
TIMGGAN
PSA-GAN
DeepAR
EBGAN
TIMEGAN
PSA-GAN
Cold Starts: In this experiment, we explore whether the GAN models can support DeepAR in cold start forecasting setting. Cold starts refer to new time series (like new product in demand forecasting) for which little or no data is available. Here, we randomly truncate 10%, 20%, and 30% of the time series from datasets such that only the last 24 (representing 24 hours of data) values before inference are present. We only consider single forecasting window in this experiment. We then again use the GAN models impute values for the lags and context that DeepAR conditions for
7

---

Published as a conference paper at ICLR 2022
Table 2: NRMSE accuracy compari-
NRMSE
son of data augmentation experiments (lower is better, best method in bold). Mean and 95% confidence intervals are obtained by re-running each method five times. DeepAR is only run on the real time series data. The other models correspond to DeepAR and one of the GAN models for data augmentation.
Dataset Model
Electricity
M4
Solar
Traffic
DeepAR
0.49
0.26=0.02
1.07
0.45#
EBGAN
0.64
1.18 0.32+0000
0.50#b.03
TIMEGAN
0.56110.05
0.28:10.02
1.06:0.02
0.47 0.0004
PSA-GAN
0.64
0.310.04
1.07
0.5010.11
forecasting the cold start time series. Figure 5 shows the forecasting error of the different models for the cold start time series only. In this experiment, PSA-GAN and TIMEGAN improve the NRMSE over DeepAR and are on-pat overall (mean NMRSE 0.70 and 0.71 for PSA-GAN and TIMEGAN, respectively). See Figure $2 for detailed split by dataset.
Cold start percentage
Cold start percentage
Cold start percentage 30
0.9
NRMSE
0.8
0.7
0.6
DeepAR
EBGAN
TIMEGAN
PSA-GAN
DeepAR
EBGAN
TIMEGAN
PSA-GAN
DeepAR
EBGAN
TIMEGAN
PSA-GAN
Figure 5: Performance of using different GAN models for imputation the cold start experiment. Markers denote the mean NRMSE averaged over datasets and error bars the 68% confidence interval over ten runs. Note that this figure only shows the error of the cold start time series. Overall, TIMEGAN and P PA GAN improve the NRMSE over DeepAR in this setup, while PSA GAN the best method when 30% of time series are cold starts.
Data Augmentation: In this experiment, we average the real data and GAN samples to augment the data during training." During inference, DeepAR is conditioned real data only to generate forecasts. In Table 2 we we can see that none of the GAN models for data augmentation consistently improves over DeepAR. Overall, TIMEGAN the best-performing GAN model but plain DeepAR still performs better. This finding is aligned with recent work in the image domain where data augmentation with GAN samples does not improve a downstream task (Ravuri & Vinyals, 2019). We hypothesize that the GAN models cannot improve in the data augmentation setup because they are trained to generate realistic samples and not necessarily to produce relevant invariances. Furthermore, the dataset sizes might be sufficient for DeepAR to train well well-performing model and therefore the augmentation might not be able to reduce the error further. More research is required to understand whether synthetic data can improve forecasting models via data augmentation.
Forecasting Experiments: In this experiment, we use the GAN models directly for forecasting (Table 3, example samples in Appendix H). We can see that DeepAR consistently performs best. This is expected as DeepAR takes into account context information and lagged values. This kind of information is not available the GAN models. To test this we further consider P PSAGGAN-C, i.e. PSA-GAN with 64 previous time series values as context, and further evaluate DeepAR with drawing lags only from the last values (DeepAR-C). We can see that this case PSA-CAN-C outperforms DeepAR-C in 3 out of datasets and (SAGAA performs par with DeepAR-C. Morcover, both eSA-GAN and P PSA-GAN- C are the best performing GAN models. Adding lagged values to the GAN models as context could further improve their performance and adversarial/attentoon architectures have been previously used for forecasting (Wu et al., 2020).
We also tried using only the GAN generated data for training and experimented with ratios of mixing synthetic and real samples, similar to the work in (Ravuri & Vinyals, 2019). Furthermore, we tried different weights and scheduled sampling but this did not improve the results.
8

---

Published as a conference paper at ICLR 2022
NRMSE
Table NRMSE accuracy compari- son of forecasting experiments (lower is better, best method bold). Mean and 95% confidence intervals are ob- tained by re-running each method five times. DeepAR-C and PSA G use the same 64 previous values as con- text. Among GAN models PSA-GAN
Dataset Model
Electricity
M4
Solar
Traffic
DeepAR
0.49-0.03
0.20 0.26:=0.22
1.07=0.01
0.45++0.01
DeepAR-C
1.43+0.40
0.73
0.95+t..1
0.89_0.16
EBGAN
3.6310.07
1.14:0.01
2.31
2.02-0.00
TIMEGAN
3.0410.00
1.12:0.00
1.83:0.00
1.13=0.00
PSA-GAN
1.53
0.61
1.35+0.00
0.66=0.00
and PSA-GAN-C perform best.
PSA-GAN-C
1.1840.00
0.33=+0.00
1.32+0.00
0.57=0.00
4.4 ABLATION STUDY
Figure 6 shows the results of our ablation study where we disable important components of our model: moment loss, self-attention, and fading in of new layers. We mcasure the performance of the ablation models by Context-FID score. Overall, our propost logog ed PSA-GAN model performs better than the ablations which confirms that these components contribute to the performance of the model.
Figure 6: We perform abla- tion study by disabling moment loss, self-attention, and fading in of new layers of PSA-GAN. We show the mean performance over three runs on four datasets and the 68% confidence inter- val. PSA- GAN has the lowest Context-FID score which con- firms that the proposed model requires these components for good performance. SA: Self- attention, MI: Moment loss, fade in: Progressive fade in of new layers
MSAGAN
wio
fade
all, fadk
wio Mt. wio wis fade
whth SA
wie MI, wh. SA
nio MR.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
6.8
Context
4.5 Low CONTEXT-FID SCORE MODELS CORRESPOND TO BEST-PERFORMING FORECASTING MODELS
One other observation is that the lowest Context-FID score models correspond to the best models in the data augmentation and forecsstnn experiments. PSA-GAN and TIMEGAN produce the lowest Context-FID samples and both models also improve the baseline in most downstream tasks. Overall, PSA-GAM has lowest Context-FID and also outperforms the other models in the downstream forecasting tasks, except for the cold start task. Additionally, for the ablation models mentioned in Figure 6 (but with a target length of 32) and the NRMSE of these Table 3). We find that the Pearson correlation coefficient
calculated the Context-FID scores
models the forecasting experiment between the Context-FID and forecasting NRMSE is 0.71 and the Spearman's is rank correlation coefficient is 0.67, averaged over all datasets. All datasets (except Traffic) have correlation coefficient of at least 0.73 in either measure (see Table S1 in the Appendix).
5 CONCLUSION
We have presented PSA-CAN, progressive growing time series GAN augmented with self-attention, that produces long realistic time series and improves downstream forecasting tasks that are challenging for deep barning-basel time series models. Furthermore, we introduced the Context-FID score to assess the quality of synthetic time series samples produced by GAN models. We found that the lowest Context-FID scoring models correspond the best-performing models in downstream tasks. We believe that time series GANs that scale to long sequences combined with a reliable metric to assess their performance might lead their routine in time series modeling.

---

Published as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
Details necessary for reproducing our experiments are given in the Appendix. In particular, details on training are provided Sections and C, together with hyperparameter tuning Section E and further experimental settings in Section F. The code we used in the paper is available under: https: : / h///ttthii thub.
wtwmmmohhhhhhhhhccc deeeeeeiiee//eee / -gan and we will additionally dissemi- nate PSA GAN via GluonTS: https: /ttpt//ttttttb//// com/ aws laba/ aabs/gluon-te
REFERENCES
Alexander Alexandrov. Konstantinos Benidis, Michael Bohlke-Sccheideer Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research, 21(116):1-6, 2020.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv: 2018.
Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. 1xXiv:8809.11990 2018.
R. Cont. Empirical properties of asset returns: stylized facts and statistical issues. Quantitative Finance, 1(2):223-22 236, feb 2001.
Zhicheng Cui, Wenlin Chen, and Yixin Chen. Multi-Scale Convolutional Neural Networks for Time Series Classification. arXiv:/603.06995, 2016.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems, 33, 2020.
Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository, 2017. URL http: //archive. ics. uci.udui eediiedu/mm..
Florian Eckerli and Joerg Osterrieder. Generative adversarial networks in finance: An overview. Machine Learning eJournal, 2021.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. GANSynth: Adversarial neural audio synthesis. In International Conference on Learning Representations, 2019.
Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs. arXiv: 1706.02633. 2017.
Eves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, 2019.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv: 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv e-prints, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
10

---

Published as a conference paper at ICLR 2022
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv: 1412.694, 2014.
Richard Kurle, Syama Sundar Rangapuram, Emmanuel de Bézenac, Stephan Günnemann, and Jan Gasthaus. Deep rao- noo-blackwellised particle filters for time series forecasting. Advances in Neural Information Processing Systems, 2020.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '18, 2018.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, 2019.
Bryan Lim, Sercan Ó. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4), 2021.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-ting Sun. Adversarial ranking for language generation. In Advances in Neural Information Processing Systems, 2017.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54-74, 2020.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 201 8.
Olof Mogren. C-RNN-GAN: Continuous recurrent neural networks with adversarial training. 1oXiv:/611.090000 2016.
Hao Ni, Lukasz Szpruch, Magnus Wiese, Shujian Liao, and Baoren Xiao. Conditional Sig- Wasserstein GANs for Time Series Generation. arXiv:2006.05422 05421, 2020.
Hao Ni, Lukasz Szpruch, Mare Sabate- Vidales, Baoren Xiao, Magnus Wiese, and Shujian Liao. Sig- Wasserstein GANs for Time Series Generation. arXiv:2111.01200 2021.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020a.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications to zero-shot time-series forecasting, arXiv preprint urXiv:2002.02887. 2020b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1911.01703,, 2019. 01703.
URL htth: /ttpp:////xivv...// wrw.vorrg/aae abs 1912.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference Machine Learning, 2021.
Suman Ravuri and Oriol Vinyals. Seeing is Not Necessarily Believing: Limitations of BigGANs for Data Augmentation. March 2019.
11

---

Published as a conference paper at ICLR 2022
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training GANs. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting. 36(3): 1181-1191, 2020.
He Sun, Zhun Deng, Hui Chen, and David C. Parkes. Decision-Aware Conditional GANs for Time Series Data. arXiv:2009. 12682, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Computer Vision and Pattern Recognition, 2016.
Shuntaro Takahashi, Yu Chen, and Kumiko Tanaka-Ishii. Modeling financial time-series with generative adversarial networks. Physica A Statistical Mechanics and its Applications, 527, 2019.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: Generative Model for Raw Audio. (rXXv:1600003499, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from seratch with deep neural networks: A strong baseline. In 2017 International Joint Conference on Neural Networks (IJCNN) 2017.
Magnus Wiese, Robert Knobloch, Ralf Korn, and Peter Kretschmer. Quant gans: deep generation of financial time series. Quantitative Finance, 20(9):1419 1440, 2020.
Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. In Advances in Neural Information Processing Systems, 2020.
Bing Xu, Naiyan Wang. Tianqi Chen, and Mu Li. Empirical Evaluation of Rectified Activations in Convolutional Network. TrXxv:/505.00833, 2015.
Jinsung Yoon, Daniel Jarrett, and Mihacla van der Schaar. Time-series generative adversarial networks. In Advances Neural Information Processing Systems, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference Machine Learning, 2019a.
Yi-Fan Zhang, Peter Thorburn, Wei Xiang, and Peter Fitch. Ssim a deep learning approach for recovering missing time series sensor data. IEEE Internet of Things Journal, 6(4):6618-662) 2019b.
Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial networks. In International Conference on Learning Representations, 2017.
12

---

Published as a conference paper at ICLR 2022
APPENDIX
A TIME SERIES FEATURES
We use varying features represent the time dimension for the GAN models to enable training on time series windows sampled from the full-length time series. We use OourofDay, DayOfWeek, DayOfMonth and DayOfYear features. Each feature is encoded using single index number (for example, between [0, 365 for DayOfYear) and normalized to be within [-0.5, 0.5]. We also use an age feature represent the age of the time series to be log(2.0 + L) where is the time index. Additionally, we embed the index of the individual time series into a 10 dimensional vector with an embedding network and it as a non time-varying feature. DeepAR uses the same features in forecasting.
B TRAINING PROCEDURE
PSA GANN uses spectral normalization and progressive fade-in of new layers to stabilize training.
Progressive fade in of new layers As training progresses, new layers are added to double the length of time series. However, simply adding new untrained layers drastically changes the number of parameters and the loss landscape, which destabilizes training. Progressive fade in of new layers is technique introduce by Karras et al., 2017 to mitigate this effect where each layer g, and di is smoothly introduced. For E [2, L] the fade in of new layers written as:
91 x2 B x2
(8)
7:-1 + 2; am (1
d, x2 - R x2
(9)
- DO))( + ai) DOWN(Y.)
where a is scalar initialized to be zero and grows linearly to one over 500 epochs.
Spectral Normalisation Both the generator and the discriminator benefit from using spectral normalisation layers. In the discriminator, Spectral Normalisation stabilizes training (Miyato et al., 2018). It constrains the Lipschitz constant of the discriminator be bounded by one. In the generator the spectral normalisation stabilises the training and avoids escalation of the gradient magnitude (Zhang et al., 2019a).
C TRAINING DETAILS
All GAN models in this paper have been implemented using PyTorch from Paszke al. (2019).
PSA-GAN has been trained over 6500 epochs, where a new block is added every 1000 epochs and is faded over 500 epochs. At each epoch, PSA-GAN trains 100 batches of size 512. It optimises its parameters using the Adam (Kingma & Ba, 2014) with learning rate of 0.0005 and betas of (0.9, 0.999) for both the generator and the discriminator.
PSA-GAN (and its variants) has been trained on ml..2.xlaage Amazon instances for 21 hours.
The number of in-channels and out-channels ny in the Main Block (Figure 1, Right) equals 32. The number of out-channels in the convolution layer e equals (Eq.6) The number of time features D equals 15.
For the case of PSA-GAN-C, the MLP block is two layers perceptron with LeakyReLU. incorporates knowledge the past by selecting the Le = 64 time points and concatenates them with its input.
13

---

Published as a conference paper at ICLR 2022
D CONTEXT-FID SCORE ON TIME SERIES EMBEDDINGS
To compute the Context-FID,, we replace Inception V3 (Szegedy et al., 2016) with the encoder E of Franceschi et al. (2019), which train separately for each dataset.
Our proposed Context-FID score is computed as follows: we first select a time range [t, + T|. We then sample batch of synthetic time series Zt,7 [?1,4.,,... ..., NN,,rl and batch of real time series Ze [2,, .... [r,, that we encode with E into Zi and ,,, respectively. Finally, compute the FID score of the embeddings.
E HYPERPARAMETER TUNING
We did limited hyperparameter tuning in this study to find default hyperparemters that perform well across datasets. We performed the following grid-search: the batch size, the number of batches used per epoch, the presence of Self-ate -fff-atteniion and the presence of the Progressive fade in of new layers mechanism explained in section paragraph B. The range considered for each hyper-parameter is: batch size: [64, 128, 256, 512). the number batches used per epoch: [50, 100, 200), the presence of Self-at falf-attentio on: (True, False] and the presence of the Progressive fade in of new layers: [True, False].
The set of hyperparameters shared across the four datasets we train PSA-GAN on (which means we do not tune hyperparameters per datasets). We select the final hyperparameter set by lowest Context-FID score on the training set, averaged across all datasets.
F FORECASTING EXPERIMENT DETAILS
We use the DeepAR (Salinas et al., 2020) implementation in GluonIS (Alexandrov et al., 2020) for the forecasting experiments. We use the default hyperparameters of DeepAR with the following exceptions: epochs=100, num batches per epoch=100, dropout rate=0. 01, scaling=False, prediction 1ength=32, context length=64, feat static cat=Tzue. We use the Adam optimizer with learning rate of le-3 and weight decay of le-8. Additionally, we clip the gradient 10.0 and reduce the learning rate by a factor of 2 for every 10 consecutive updates without improvement, up to a minimum learning rate of 5e-5. DeepAR also uses lagged values from previous time steps that are used as input the LSTM each time step t. We set the time series window size that DeepAR samples to 256 and truncate the default hourly lags fit the context window, the prediction window, and the longest lag into the window size 256. We use the following lags: [1, 2, 3, 4, 5, 6, 7, 23, 24, 25, 47, 48, 49, 71, 72, 73, 95, 96, 97, 119, 120, 121, 143, 144, 145].
For NBEATS (Oreshkin et al., 2020a), we use the default parameters and set the trainer settings to epochs=100 00, num batches per epoch=50. Note that the original paper, NBEATS an ensemble of 180 networks. Due to the high computational cost, we only ensemble three networks with the settings context length=644 64, context length-96, context 1ength-128 and mean absolute percentage error the loss function.
For TFT (Lim et al., 2021), we use the default hyperparameters except: epochs-100, num batches per epoch=50, context length=128.
For all methods, we used the implementations available in GluonTS (Alexandrov et al., 2020).
G ADDITIONAL EXPERIMENTS
14

---

Published as a conference paper at ICLR 2022
Pcarson
Spcarman
Traffic
0.597
0.609
M4
0.858
0.455
Solar 0.852
0.773
Electricity 0.533
0.891
Table $1: Pearson and Spearman rank-order correlation coefficient between the Context-FID score and NRMSE for different datasets. These results suggest that low Context-FID scores indicate low error forecasting and therefore Context-FID could be useful score to develop GAN models the time series domain.
dataset electricity Stretch length 50
detaset electricity Sttetch length
3.5
3.5
3.0
3.0
NRMSB
2.5
2.5
2.0
1.5
2.0
1.0
1.5
solar Stretch
dataset
length
1.6
NRASE
1.5
1.4-
1.2
1.1
dataset m4 hourly Stretch length 50
dataset m4, mddhhouryy Stretch length 110
1.
1.2
NRMBE
1.1
1.0-
12:
0.9
1.1
0.8
1.0
0.7
length
dataset traffic Stretch length
1
1.1
NRMSE
10-
0.65
0.60
0.55
06
DoepAR
EBGAN
TMEGAN
PSA-GAN
Deedi
ERGAN
TINEGAN
PSA-GAN
Figure S1: Detailed performance of using different GAN models for imputation in the missing value stretch length experiment. Markers denote the mean NRMSE averaged over ten runs and error bars represent the 95% confidence interval. For 50 and 110 missing value stretch length, using PSA-GAN to impute missing values results in the lowest error for three out of four datasets.
15

---

Published as a conference paper at ICLR 2022
600204: elcardyyy Cold start percentage 10 dataset elecifiily Cold start perconiage 20
dstaset - excrricyy Cold start parceniage
0.59
0.55
0.30
2.50
0.45
0.45
0.45
0.35
0.35
0002.007 m4_hourfy Cold start percentage
10
dataset m4_hourfy Cold start porceniage
20 dataset m4_hourlt Cold stort percentage
0.45
0:45
0.40
0.35
0.30
0.30
0.20
0.20
0.15
0.20
solee-enegy
10dersset
Cold
20datase
Gold
1.7
1.0
3.6
1.
1.7
1.6
1.5
1.4
1.4
1.3
1.3
0.82
0.21
0.84
0.51
0.30
0.82
0.30
0.74
0.80
0.78
0,77
Deep48
BB04N
TMPGAN
PSA.AAN
Deepil
BRGW
TAMSAAN
PSU.GAN
Deveslt
ESGAN
TIMEGAN
PSA-GAN
Figure S2: Detailed performance of using different GAN models for the cold start experiment. Markers denote the mean NRMSE averaged over ten runs and error bars the 95% confidence interval. T TIMEGAN and PSA-GAN improve over DeepAR in three of four datasets.
NRMSE Electricity
Dataset Model
M4
Solar
Traffic
NBEATS
5.29 0.28
1.92 0.04
2.08
0.0
0.94 0.02
EBGAN
3.59 = 0.21
1.76 0.03
2.07
+ 0.0
0.96
TIMEGAN
2.06 = 0.02
1.75 0.01
2.07
0.0
0.8
PSA-GAN
1.30 + 0.04
0.4 + 0.01
2.07 0.0
0.65 0.0
Table S2: NRMSE of far-forecasting experiments with NBEATS as the forecasting model (lower is better). Mean and confidence intervals are obtained by re-running each method ten times.
16

---

Published as a conference paper at ICLR 2022
Stretch length 50
Stretch length 110
NRMSE
2.2
2.0
1.8
1.6
NBEATS
EBGAN
TIMEGAN
PSA PSA-GAN
NBEATS
EBGAN
TIMEGAN
PSA-GAN
Figure S3: Performance of using different GAN models for imputation in the missing value stretch length experiment using NBEATS as the forecasting model. Markers denote the mean NRMSE averaged over datasets and error bars the 68% confidence interval ten runs.
Cold start percentage 10
Cold start percentage 20
Cold start percentage 30
1.1
NAMSE
1.0
0.9
0.8
NBEATS
EBGAN
TIMEGAN
PSA-GAN
NBEATS
EBGAN
TIMEGAN
PSA-GAN
NBEATS
EBGAN
TIMEGAN
PSA-GAN
Figure S4: Performance of using different GAN models for imputation in the cold start experiment using NBEATS as the forecasting model. Markers denote the mean NRMSE averaged over datasets and eтror bars the 68% confidence interval over ten runs. Note that this figure only shows the error of the cold start time series.
NRMSE Electricty
Dataset Model
M4
Solar
Traffic
NBEATS
0.86 = 0.06
0.28 + 0.01
2.07 + 0.0
0.58 +
EBGAN
0.74 + 0.03
0
2.08
0.0
0.58 + 0.01
TIMEGAN
0.58 0.01
0.28 + 0.01
2.08
+ 0.0
0.51 0.0
PSA-GAN
0.92 = 0.06
0.3 + 0.01
1.97 + 0.02
0.54 + 0.01
Table S3: NRMSE accuracy comparison of data augmentation experiments (lower is better) using NBEATS as the forecasting model. Mean and 95% confidence intervals are obtained by re-running each method five times.
NRMSE Electricity
Dataset Model
M4
Solar
Traffic
TFT
4.51 + 0.02
1.78 - 0.04
1.74 + 0.02
0.94 + 0.0
EBGAN
3.54 + 0.22
1.5 1.5
2.5 2.5 0.13
0.76 0.02
TIMEGAN
2.03 + 0.05
1.71 + 0.04
1.26 + 0.03
0.82 + 0.03
PSA-GAN
1.97 0.05
0.34 0.01
2.15 + 0.13
0.70
Table $4: NRMSE of far-forecasting experiments with Temporal Fusion Transformer (TFT) as the forecasting model (lower is better). Mean and confidence intervals are obtained by re-running each method ten times.
17

---

Published as a conference paper at ICLR 2022
Stretch length 50
Stretch length 110
2.50
NRMSE
2.00
1.75
1.50
1.25
TFT
EBGAN
TIMEGAN
PSA-GAN
TFT
EBGAN
TIMEGAN
PSA
Figure S5: Performance of using different GAN models for imputation in the missing value stretch length experiment using Temporal Fusion Transformer (TFT) as the forecasting model. Markers denote the mean NRMSE averaged over datasets and error bars the 68% confidence interval over ten runs.
Cold start percentage 10
Cold start percentage = 20
Cold start percentage 30
1.5
NRMSE
I
1.0
TFT
EBGAN
TIMEGAN
P5A-GAN
TFT
EBGAN
TIMEGAN
PSA-GAN
IFT
EBGAN
TIMEGAN
PSA-GAN
Figure S6: Performance of using different GAN models for imputation in the cold start experiment using Temporal Fusion Transformer (TFT) as the forecasting model. Markers denote the mean NRMSE averaged datasets and error bars the 68% confidence interval over runs. Note that this figure only shows the eтror of the cold start time series.
NRMSE Electricity
Dataset Model
M4
Solar
Traffic
TFT
0.63 + 0.04
0.26 + 0.02
1.29
0.04
0.40 0.0
EBGAN
0.9 + 0.05
0.31 + 0.01
1.21 + 0.05
0.64 0.01
TIMEGAN
0.83 + 0.04
0.85 + 0.06
1.28 - 0.03
0.51 + 0.01
PSA-GAN
0.94 + 0.03
0.35 + 0.02
1.64 = 0.04
0.42 - 0.0
Table S5: NRMSE accuracy comparison of data augmentation experiments (lower is better) using Temporal Fusion Transformer (TFT) as the forecasting model. Mean and 95% confidence intervals are obtained by re-running each method five times.
18

---

Published as a conference paper at ICLR 2022
NRMSE
Dataset Model
Electricity
M4
Solar
Traffic
DeepAR
0.49=0.03
0. 0.262
1.07
0.45
NEEATS
0.86+0.06
0.28+0.01
2.07=0.00
0.58
TFT
0,63+t0.04
0.26+0.02
=.29
0.40#t0.0
DeepAR-C
1.4310.40
0.73
0.95=0.01
2.89
EBGAN
3.6310.07
1.14:0.01
2.31=0.01
2.02=0.00
TIMEGAN
3.04
1.1220.00
1.83=0.00
1.13:0.00
PSA-GAN
1.53+0.00
0.61
1.35=0.00
0.66=2.00
PSA-GAN-C
1.1840.00
0.33
1.32=0.00
0.57
Table S6: Extended NRMSE accuracy comparison of forecasting experiments with NBEATS and Temporal Fusion Transformer (TFT) as additional forecasting models (lower is better).
19

---

Published as a conference paper at ICLR 2022
H SAMPLE FORECASTS
We present more sample forecasts under the framework of the forecasting experiment. Figures S7- S10 we present sample forecast for Electricity, M4, Solar Energy, and Traffic datasets, respectively. We can see that the synthetic time series generated by convey a better approximation the ground truth values. One can further observe that this task is particularly challenging for GANs as data generation is located in time-frame that is not covered while training.
and PSA GAN-G CC consistently
Figure S7: Sample forecasts for dataset Electricity
(a) EBGAN
(b) TIMEGAN
(c) PSA-GAN
(d) PSA-GAN-C
Figure S8: Sample forecasts for dataset M4
(a) EBGAN
(b) TIMEGAN
(c) PSA-GAN
(d) PSA-GAN-C
Figure S9: Sample forecasts for dataset Solar
(a) EBGAN
(b) TIMEGAN
(c) PSA-GAN
(d) PSA-GAN-C
Figure S10: Sample forecasts for dataset Traffic
(a) EBGAN
(b) TIMEGAN
(c) PSA-GAN
(d) PSA-GAN-C
20


Reviews:
[{"review_id": "UOENm6EKHsO", "reviewer": "Reviewer_RLDM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n1. The proposed approach is novel for generating time series data.\n2. The paper introduces a metric for evaluating the quality of synthetic time series data.\n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. \n4. The paper is well written and easy to follow.\n\nConcerns:\n1. Could the authors clarify how the loss function in Equation 7 is used to train the generator and the discriminator?\n2. What are the time features (referred as X in the paper)? \n3. How does the concatenation happen in the generator with the time features since the sequence length after upscaling keeps increasing?\n4. Similarly, the authors should provide more details on how does the approach deals with the fixed-length context in PSA-GAN-C? How does the summation work with varying lengths of sequences?\n5. Table 3 compares the forecasting performance of GAN models with the DeepAR architecture. It shows that DeepAR achieves much better performance. Based on this experiment, wouldn't it be better to use DeepAR for imputing/predicting the time series in the far-forecasting experiment than imputing with the GAN? The authors should compare to this baseline or clarify why this is not possible. \n6. Table 2 shows interesting results. Could the authors explain why the data augmentation leads to degradation of the forecasting performance instead of improving it? \n\n\nAdditional Comments:\n1. The discriminator architecture in Figure 1 and equation 5 don't agree with each other. Could the authors clarify/align them?\n2. Remove the additional 'of' in section 4 para 1.", "summary_of_the_paper": "This paper proposes a GAN-based approach (PSA) for generating realistic time-series data that can be used to improve several downstream tasks such as imputation of missing data and forecasting. The generator and discrimination model in the proposed GAN framework, PSA-GAN consists of progressively growing blocks consisting of convolutions and self-attention. The paper also proposes a metric for evaluating the quality of synthetic time series data. Experiments show that the proposed approach is able to achieve better performance than previous GAN-based approaches for time series and is also able to improve on downstream forecasting tasks by augmenting forecasting models. ", "summary_of_the_review": "This paper proposes a novel approach for generating realistic time-series data. It also introduces a metric for evaluating the quality of synthetic time series data which has been lacking in this domain. The paper also provides comprehensive experiments to show the effectiveness of the proposed framework.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7IaeRFXaygu", "reviewer": "Reviewer_pbaT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n1. The proposed GAN-model seems to outperform the baselines significantly.\n2.  Adapting the FID score to time series seems like a good idea. This could be an interesting measure of performance for future work.\n\n### Weaknesses\nThere are multiple issues in the description of the model which are unclear to me. I think that Chapter 3 (and Figure 1) would benefit from being partially rewritten.\n1. How is $\\phi(i)$ used in the model?\n2. What is meant by the time features $\\mathbf{X}$? Do these features encode the position in time, like the e.g. sine-cosine in the transformer? What does the “number of time features” mean? Is this the dimensionality of the feature vector at each time?\n3. Page 3 says: “As a preprocessing step, we first map the input features from a sequence of length τ to a sequence of length 8, denoted by $\\tilde{Z}_0$, using average pooling.” What is meant by the input features? Is that the sum $\\mathbf{X} + \\mathbf{n}$? \n4. What is the dimensionality of the Gaussian noise $\\mathbf{n}$?\n5. According to Figure 1, the output of the generator has length $n * 16$, where $n$ is the number of blocks. Since each block doubles the size of its input, shouldn’t the output have size $2^n * 8$?\n6. Shouldn’t the output of equation (4) be named $Y_L$ instead of $\\tilde{Z}_L$, since the latter denotes a variable inside the generator?\n7. p. 4: The time range of the $\\hat{Z}_{i, t-L_C, L_C}$ is said to be $[t-L_C, t-1]$, but according the definition of $\\hat{Z}_{i, t, \\tau}$ on p. 2, the time range would be $[t-L_C, t]$.\n\n**Technical Novelty And Significance:**\n\n* Context-FID: (1) Why is the encoder of Franceschi used - comparison to other encoders or why is it particularly good? (2) Why is the network trained separately for each dataset? This is different to the standard FID, where the Inception network is trained on ImageNet.\nMinor:\n* Page 3 says: “The module $m: f(x) \\rightarrow m(f(x))$ is referenced as Residual Self-Attention in Fig. 1 (Right).” This module should not be denoted by $m$ because the variable m already denotes the main function defined in equation (1). So the sentence should be something like: “The module $m_1: f(x) \\mapsto m(x)$ …”.\n\n**Typos**\n* “The discriminator contains n blocks that halves the size of the input using an average pooling.” -> The discriminator contains n blocks that *halve* the size the size of the input *using average* pooling.\n* “The training procedure to address those issues are presented in the appendix.” -> *is* presented in the appendix\n", "summary_of_the_paper": "The paper proposes a type of GAN to generate synthetic time series. The authors use the data generated by their proposed model to train forecasting networks, which improves the baselines. They also show that the proposed GAN can itself be used as  a forecasting model and that it has competitive performance to baseline models. Finally, the paper proposes an adaptation of the FID score for time series.", "summary_of_the_review": "The proposed model outperforms the baselines and the proposed FID score for time series looks interesting. However, there are a few unclarities in the description of the model. I hope that these things can be explained during the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0nnFVbVXK9b", "reviewer": "Reviewer_DRP7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of progressive growing of GANs is convincing for synthesizing time series. The time series data is difficult to fit due to high volatility. Thus it is feasible to fit the distribution gradually from easy case to difficult case. Self-attention is capable of promoting the capability of fitting abnormal circumstances such as data imputation. So, the idea is convincing. \n\nBut there are two drawbacks for this submission.\n1) The novelty is not good enough for ICLR. The two ideas are both borrowed from existing papers.\n2) The SOTA algorithms are not compared except TimeGAN, such as QuantGAN, SigCWGAN, and DAT-GAN. EBGAN is not usually applied  in this scenario. Besides, it is not the SOTA algorithm in the GAN field.\n\n", "summary_of_the_paper": "The authors propose a new GAN-based algorithm for time series synthesis. They use progressive growing of GAN architectures to improve the performance of GAN and self-attention to enhance the expressive capability of neural networks. The experimental results validate the superiority of the proposed PSA-GAN algorithm.", "summary_of_the_review": "The idea is interesting. But the novelty and experiments are not good enough for ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-BNciJ9icdU", "reviewer": "Reviewer_4v5L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper adapts the known concepts for image GAN literature to time-series, without substantial extra inventions. It poses a concern for novelty. \n\n- The architecture is just composed of standard layers from image GAN literature, based on self-attention. Whereas for the experiments, the Author use the RNN-based DeepAR. What happens if an RNN based architecture is used for generation or if a self-attention based architecture is used for forecasting? \n\n- The loss function choice seems heuristic and not very well explained. How does the loss help with periodicity? What happens if the signals have high vs. low frequency components? What happens when the signals have peaks? Different impactful time-series cases should be considered and the section should be expanded accordingly. \n\n- I don't think the FID score would have significant value for time-series as the perceptual quality of time-series to humans is different than images. Beyond proposing the score, its usefulness should be demonstrated. \n\n- The procedure for hyperparameter tuning is not well explained. How does one construct a validation set and what validation reward to use? \n\n- I think experiments section should be better organized and the writing should be improved for a more clear presentation. The results on far-forecasting and data imputation seem quite promising. The improvements in cold start and data augmentation seem negligible. For forecasting, the gap with DeepAR is still significant. Overall, rather than focusing on a lot of results, I suggest emphasizing on the key capabilities the data synthesis brings, with significant improvements. Fig. 6 is just a cherry-picked example and it does not tell much. \n\n- For data augmentation, how can one optimize what ratio of synthetic data to use and is there a way of more intelligent sampling of synthetic data specifically focusing on the low data coverage regimes? The naive application does not seem promising and the authors need to think further to show strong results. \n\n- Many recent papers show significant outperformance over DeepAR. That aspect is orthogonal to the contributions of the paper but I suggest benchmarking with at least one more forecasting model. Specifically for far-forecasting, RNN based models are known to suffer from error propagation without teacher forcing. So direct decoding approaches that generate all the multi-horizon forecasts would be better candidates, and likely will reduce the gains with synthetic data. ", "summary_of_the_paper": "PSA-GAN is proposed, based on progressively generating the output time-series with self-attention. Via extensive experiments, its benefits in synthesizing realistic samples are demonstrated. ", "summary_of_the_review": "Overall, the paper has strong experimental results and solid contributions. But the novelty is not very high, and also several aspects of the paper need improvements. I am willing to increase my score if the authors address the points above. \n\n---------------------------\n\nI am updating my score as the reviewers have addressed some of my concerns with extra experiments. I believe the paper still needs significant edits to reflect the points in the reviewer responses. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adapts the idea of progressive growing of GANs to time series synthesis. The reviewers thought that the idea was well motivated. DRP7 initially expressed concern w.r.t. novelty. They were also concerned with the lack of certain baselines. The authors responded, highlighting its contributions w.r.t. Evaluation (Context-FID score) and extensiveness of the evaluation. The authors also added missing references but pushed back on the additional baselines. DRP7 raised their score.  Reviewer pbaT was also positive about the work though had some questions and suggestions for improving clarity. They had initially given a low score for “correctness” but raised this, indicating they were satisfied their clarity concerns were addressed. Reviewer RLDM (whose code-name happens to match a ML conference) thought the work was novel and appreciated the introduction of a new metric for evaluating the quality of generated time series data. They remarked on the thoroughness of the experiments and the quality of the presentation. They asked some clarifying questions to which the authors provided a response. Reviewer 4v5L also had a concern with novelty, felt the loss function was “heuristic” and didn’t see the utility of the FID-based score. They also presented several clarifying questions. The authors provided a lengthy response to that reviewer’s concerns, having run additional analysis, and the reviewer upgraded their score in response. With all reviewers on the accept side of the fence I am inclined to recommend acceptance. Please note 4v5L’s comment that “the paper still needs significant edits to reflect the points in the reviewer responses”.
Decision: Accept (Poster)