Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new GAN-based algorithm for time series synthesis. They use progressive growing of GAN architectures to improve the performance of GAN and self-attention to enhance the expressive capability of neural networks. The experimental results validate the superiority of the proposed PSA-GAN algorithm.
main_review: The idea of progressive growing of GANs is convincing for synthesizing time series. The time series data is difficult to fit due to high volatility. Thus it is feasible to fit the distribution gradually from easy case to difficult case. Self-attention is capable of promoting the capability of fitting abnormal circumstances such as data imputation. So, the idea is convincing. 

But there are two drawbacks for this submission.
1) The novelty is not good enough for ICLR. The two ideas are both borrowed from existing papers.
2) The SOTA algorithms are not compared except TimeGAN, such as QuantGAN, SigCWGAN, and DAT-GAN. EBGAN is not usually applied  in this scenario. Besides, it is not the SOTA algorithm in the GAN field.


summary_of_the_review: The idea is interesting. But the novelty and experiments are not good enough for ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a type of GAN to generate synthetic time series. The authors use the data generated by their proposed model to train forecasting networks, which improves the baselines. They also show that the proposed GAN can itself be used as  a forecasting model and that it has competitive performance to baseline models. Finally, the paper proposes an adaptation of the FID score for time series.
main_review: ### Strengths
1. The proposed GAN-model seems to outperform the baselines significantly.
2.  Adapting the FID score to time series seems like a good idea. This could be an interesting measure of performance for future work.

### Weaknesses
There are multiple issues in the description of the model which are unclear to me. I think that Chapter 3 (and Figure 1) would benefit from being partially rewritten.
1. How is $\phi(i)$ used in the model?
2. What is meant by the time features $\mathbf{X}$? Do these features encode the position in time, like the e.g. sine-cosine in the transformer? What does the “number of time features” mean? Is this the dimensionality of the feature vector at each time?
3. Page 3 says: “As a preprocessing step, we first map the input features from a sequence of length τ to a sequence of length 8, denoted by $\tilde{Z}_0$, using average pooling.” What is meant by the input features? Is that the sum $\mathbf{X} + \mathbf{n}$? 
4. What is the dimensionality of the Gaussian noise $\mathbf{n}$?
5. According to Figure 1, the output of the generator has length $n * 16$, where $n$ is the number of blocks. Since each block doubles the size of its input, shouldn’t the output have size $2^n * 8$?
6. Shouldn’t the output of equation (4) be named $Y_L$ instead of $\tilde{Z}_L$, since the latter denotes a variable inside the generator?
7. p. 4: The time range of the $\hat{Z}_{i, t-L_C, L_C}$ is said to be $[t-L_C, t-1]$, but according the definition of $\hat{Z}_{i, t, \tau}$ on p. 2, the time range would be $[t-L_C, t]$.

**Technical Novelty And Significance:**

* Context-FID: (1) Why is the encoder of Franceschi used - comparison to other encoders or why is it particularly good? (2) Why is the network trained separately for each dataset? This is different to the standard FID, where the Inception network is trained on ImageNet.
Minor:
* Page 3 says: “The module $m: f(x) \rightarrow m(f(x))$ is referenced as Residual Self-Attention in Fig. 1 (Right).” This module should not be denoted by $m$ because the variable m already denotes the main function defined in equation (1). So the sentence should be something like: “The module $m_1: f(x) \mapsto m(x)$ …”.

**Typos**
* “The discriminator contains n blocks that halves the size of the input using an average pooling.” -> The discriminator contains n blocks that *halve* the size the size of the input *using average* pooling.
* “The training procedure to address those issues are presented in the appendix.” -> *is* presented in the appendix

summary_of_the_review: The proposed model outperforms the baselines and the proposed FID score for time series looks interesting. However, there are a few unclarities in the description of the model. I hope that these things can be explained during the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a GAN-based approach (PSA) for generating realistic time-series data that can be used to improve several downstream tasks such as imputation of missing data and forecasting. The generator and discrimination model in the proposed GAN framework, PSA-GAN consists of progressively growing blocks consisting of convolutions and self-attention. The paper also proposes a metric for evaluating the quality of synthetic time series data. Experiments show that the proposed approach is able to achieve better performance than previous GAN-based approaches for time series and is also able to improve on downstream forecasting tasks by augmenting forecasting models. 
main_review: Positives:
1. The proposed approach is novel for generating time series data.
2. The paper introduces a metric for evaluating the quality of synthetic time series data.
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. 
4. The paper is well written and easy to follow.

Concerns:
1. Could the authors clarify how the loss function in Equation 7 is used to train the generator and the discriminator?
2. What are the time features (referred as X in the paper)? 
3. How does the concatenation happen in the generator with the time features since the sequence length after upscaling keeps increasing?
4. Similarly, the authors should provide more details on how does the approach deals with the fixed-length context in PSA-GAN-C? How does the summation work with varying lengths of sequences?
5. Table 3 compares the forecasting performance of GAN models with the DeepAR architecture. It shows that DeepAR achieves much better performance. Based on this experiment, wouldn't it be better to use DeepAR for imputing/predicting the time series in the far-forecasting experiment than imputing with the GAN? The authors should compare to this baseline or clarify why this is not possible. 
6. Table 2 shows interesting results. Could the authors explain why the data augmentation leads to degradation of the forecasting performance instead of improving it? 


Additional Comments:
1. The discriminator architecture in Figure 1 and equation 5 don't agree with each other. Could the authors clarify/align them?
2. Remove the additional 'of' in section 4 para 1.
summary_of_the_review: This paper proposes a novel approach for generating realistic time-series data. It also introduces a metric for evaluating the quality of synthetic time series data which has been lacking in this domain. The paper also provides comprehensive experiments to show the effectiveness of the proposed framework.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: PSA-GAN is proposed, based on progressively generating the output time-series with self-attention. Via extensive experiments, its benefits in synthesizing realistic samples are demonstrated. 
main_review: - The paper adapts the known concepts for image GAN literature to time-series, without substantial extra inventions. It poses a concern for novelty. 

- The architecture is just composed of standard layers from image GAN literature, based on self-attention. Whereas for the experiments, the Author use the RNN-based DeepAR. What happens if an RNN based architecture is used for generation or if a self-attention based architecture is used for forecasting? 

- The loss function choice seems heuristic and not very well explained. How does the loss help with periodicity? What happens if the signals have high vs. low frequency components? What happens when the signals have peaks? Different impactful time-series cases should be considered and the section should be expanded accordingly. 

- I don't think the FID score would have significant value for time-series as the perceptual quality of time-series to humans is different than images. Beyond proposing the score, its usefulness should be demonstrated. 

- The procedure for hyperparameter tuning is not well explained. How does one construct a validation set and what validation reward to use? 

- I think experiments section should be better organized and the writing should be improved for a more clear presentation. The results on far-forecasting and data imputation seem quite promising. The improvements in cold start and data augmentation seem negligible. For forecasting, the gap with DeepAR is still significant. Overall, rather than focusing on a lot of results, I suggest emphasizing on the key capabilities the data synthesis brings, with significant improvements. Fig. 6 is just a cherry-picked example and it does not tell much. 

- For data augmentation, how can one optimize what ratio of synthetic data to use and is there a way of more intelligent sampling of synthetic data specifically focusing on the low data coverage regimes? The naive application does not seem promising and the authors need to think further to show strong results. 

- Many recent papers show significant outperformance over DeepAR. That aspect is orthogonal to the contributions of the paper but I suggest benchmarking with at least one more forecasting model. Specifically for far-forecasting, RNN based models are known to suffer from error propagation without teacher forcing. So direct decoding approaches that generate all the multi-horizon forecasts would be better candidates, and likely will reduce the gains with synthetic data. 
summary_of_the_review: Overall, the paper has strong experimental results and solid contributions. But the novelty is not very high, and also several aspects of the paper need improvements. I am willing to increase my score if the authors address the points above. 

---------------------------

I am updating my score as the reviewers have addressed some of my concerns with extra experiments. I believe the paper still needs significant edits to reflect the points in the reviewer responses.
review_title: null

Meta-review: This paper adapts the idea of progressive growing of GANs to time series synthesis. The reviewers thought that the idea was well motivated. DRP7 initially expressed concern w.r.t. novelty. They were also concerned with the lack of certain baselines. The authors responded, highlighting its contributions w.r.t. Evaluation (Context-FID score) and extensiveness of the evaluation. The authors also added missing references but pushed back on the additional baselines. DRP7 raised their score.  Reviewer pbaT was also positive about the work though had some questions and suggestions for improving clarity. They had initially given a low score for “correctness” but raised this, indicating they were satisfied their clarity concerns were addressed. Reviewer RLDM (whose code-name happens to match a ML conference) thought the work was novel and appreciated the introduction of a new metric for evaluating the quality of generated time series data. They remarked on the thoroughness of the experiments and the quality of the presentation. They asked some clarifying questions to which the authors provided a response. Reviewer 4v5L also had a concern with novelty, felt the loss function was “heuristic” and didn’t see the utility of the FID-based score. They also presented several clarifying questions. The authors provided a lengthy response to that reviewer’s concerns, having run additional analysis, and the reviewer upgraded their score in response. With all reviewers on the accept side of the fence I am inclined to recommend acceptance. Please note 4v5L’s comment that “the paper still needs significant edits to reflect the points in the reviewer responses”.
Decision: Accept (Poster)