Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Time-Transformer AAE. The proposed method is an adversarial autoencoder approach, and the Time-Transformer is a component of the decoder. Each Time-Transformer block is comprised of (1) two parallel modules: a TCN block and a transformer block, and (2) a bi-directional cross-attention. The experimental results show that the proposed method could significantly outperform SOTA methods.
strength_and_weaknesses: Strengths:
1. The proposed Time-Transformer is a novel architecture for time series generation.
2. Experimental results show that the proposed method could significantly improve the results.
3. Writing is clear and easy to follow.


Weaknesses and questions:
1. How will your model perform if you use GAN rather than the adversarial auto-encoder framework?
2. The encoder is much simpler than the decoder, what's the motivation behind your choice?
3. How does Time-Transformer perform if you use two parallel transformer blocks rather than a TCN and a transformer?
clarity,_quality,_novelty_and_reproducibility: Clarity: the writing is clear and easy to follow.

Quality: the proposed approach looks technically sound, and the evaluation is comprehensive. However, some

Novelty: good.

Reproducibility: good since implementation details are provided.
summary_of_the_review: In general, this paper introduces a novel method for time series generation, and the results show that it can significantly outperform the SOTA methods. However, the motivations for some choices and designs are not quite clear (please see weaknesses). In my opinion, the strengths outweigh the weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce Time-Transformer AAE, a generative model for time series that incorporates a new "time-transformer" architecture into the adversarial autoencoder (AAE) generative framework. The time-transformer consists of parallel temporal convolutional and transformer blocks interleaved with bidirectional cross-attention to share information. Empirical studies on a variety of datasets show that this approach achieves good coverage and captures the predictive information in the original data, while ablations show that the specific architecture of the time-transformer block outperforms reasonable alternatives.
strength_and_weaknesses: Strengths:
- The paper is well written and the main ideas are clearly presented. The figures and supplementary details are relevant and informative.
- The model is studied through a broad set of experiments that show it consistently outperforms recent, relevant competitor methods in terms of some standard quality metrics for generative modes: predictive value of the synthetic data for a real test set, difficulty of discriminating between real and generated sequences, and overlap of the empirical and generative distributions.
- The ablation study (Table 4) provides convincing evidence that the specifically parallel structure of the time-transformer block is key to achieving the results above. This goes some way towards alleviating weakness #1 below.

Weaknesses:
- The novelty of the proposal is modest overall. The specific innovation here is the parallel arrangement of two well-known architectures as part of a decoder block in a well-known generative framework. 
- Throughout the paper, the authors repeatedly claim that simultaneous learning of local and global structure in time series is a major challenge that is specifically overcome by their approach. Despite this, this problem is not defined at any level of technical detail, the property is not established to obtain with any level of rigor in the provided data, and there is no evidence provided the superior performance of the proposed method is due to solving this specific problem.
- There are no training details whatsoever in the main paper. The supplement describes the learning rate schedule but not the algorithmic details of training. The algorithmic details of training need to be included somewhere in the paper or supplement.
clarity,_quality,_novelty_and_reproducibility: The writing and presentation is clear. The "weaknesses" section includes some items that I think detract from the quality and limit the overall novelty of the method. However, the quality of the experiments and results appears to be good. Code is provided, but reproducibility would be improved by including details of the training algorithm in the paper itself.
summary_of_the_review: The authors propose an architecture and framework for generative time series modeling that borrows significantly from previous work, yet is shown to perform well relative to state-of-the-art competitors in a broad range of experiments. Moreover, the quality of these results seems to depend specifically on the novel way in which these familiar components are combined. Other issues in the presentation would be straightforward to address and could improve my overall evaluation of the work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduced a new time series generative model called `Time-Transformer AAE` that consists of an adversarial autoencoder and a newly designed architecture called `Time-Transformer`, where temporal properties are learnt by both a TCN layer and a Transformer block. This model was proposed to better learn local correlations as well as global dependencies. Empirical results demonstrated that this model can achieve state-of-the-art performance on datasets, especially datasets with long term dependencies.

The contributions of this paper comes from the creation of this Transformer based adversarial auto-encoder model, proposing the layer-wise parallel design & interaction to simultaneously learn local & global features, and demonstrating its performance via empirical experiments.

strength_and_weaknesses: Generally this paper does well in presenting its idea and performance, but struggles to provide a coherent story to help us understand how the idea was developed and how the claim of better capturing local correlation & global pattern is supported.

Strengths of this paper:
1. The figures are very informative in demonstrating the model architecture & bidirectional cross attention.
2. Extensive experiments are conducted over multiple aspects to understand the performance difference between the proposed model and multiple baselines.
3. The experiment results look very promising, huge improvement over previous state-of-the-art.
4. Ablation study demonstrated the proposed bidirectional cross attention mechanism is very useful.

Weaknesses and how to improve:
1. The datasets are a bit limited and on a small scale. Can you try a larger dataset like [eletricity](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014) or [traffic](https://archive.ics.uci.edu/ml/datasets/PEMS-SF) and see if there's any difference in the results?
2. While the paper claimed to better capture local correlation and global dependency, there's no clear analysis to support this claim. Case study or explanation with characteristics of the dataset and performance could mitigate this gap.
3. The discriminative score and predictive score are generated with a weak baseline (2 layer LSTM), in real applications people use more sophisticated methods like a Transformer based model to do classification & prediction. Would you please try better classifiers or predictors and see if the result changes? This is because these new methods do better in capturing subtle long term dependency and local patterns.
4. The motivation of using the proposed architectures is not well explained. There're more choices in how to capture local / long term dependency. Would you please add some motivations to help me understand the necessities in using the proposed architecture?

clarity,_quality,_novelty_and_reproducibility: This paper is written in good quality & clarity, main methods and designs are well presented in the Introduction, Method and Experiments section. Figures & tables are very useful for understanding the overall design. Experiments covered most aspects of time series generation, and ablation study clearly demonstrated the impact of the cross-attention layer. On originality, the use of TCN and Transformer are common in time series applications, the originality comes only from the design bidirectional cross attention. Therefore, the originality is limited.

summary_of_the_review: Overall, this paper proposed an interesting idea of combining TCN & Transformer in a Time-Transformer block for adversarial autoencoder time series generation. The presentation of model architecture and performance are clear. On the other hand, the motivation of such a proposed model is a bit unclear, and the application of such a model in new areas seems limited. It also appears unclear to me whether the performance gain comes from combining different layers of previous state-of-the-art model layers or it's a genuine change. Therefore, I recommend this paper to be rejected.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a decoder architecture for time-series generation. The proposed model, namely the Time-Transformer, uses both a temporal convolutional network (TCN) and a Transformer to process information at different timescales. While the TCN focuses on learning local features, the Transformer block aims to capture global features, which are then fused via a cross-attention operation. The decoder network stacks several of these Time-Transformer layers. The proposed decoder is trained via the adversarial autoencoder framework to learn a latent space. The model is evaluated on a variety of time series benchmarks (Sine, Stock, and Energy datasets). It demonstrates better performance compared to the baselines.
strength_and_weaknesses: Strengths:

1- The paper leverages the adversarial autoencoder in the temporal domain.

2- The proposed Time-Transformer block controls access to local and global information via a convolution and a self-attention layer, respectively. This design addresses the main concern of the paper: learning local and global features explicitly. It also seems to perform well in the evaluations. 

---

Weaknesses:

1- Generative modeling of time-series data is a vast literature involving a number of areas, such as deterministic autoregressive approaches [1*, 2*], state-space models [3*, 4*], and contrastive learning [7*]. Furthermore, [5*, 6*] augment TCNs with a hierarchy of latent variables to capture short-and long-term dependencies. The discussion in the related work section and the baselines in the experiments ignore this line of research. I can suggest another set of ablations. To show the contribution of the proposed Time-Transformer architecture, other model classes could be used in the decoder (i.e., after the De-Convolutional block).

2- The evaluations are performed on "toy" datasets. Sequences of 24 or 128 steps are rather short and hence easily explainable by a fixed-dimensional latent variable. I think this is a limitation as the entire sequence is encoded via a single, global, fixed-dimensional latent space. It is hard to tell how the model would perform when the task requires modeling longer sequences like sequential image generation.

3- Finally, the performance of the baselines, TimeGAN and RCGAN, looks different from the reported performance in the TimeGAN paper. In the t-SNE plots on the preliminary "Sine_Sim" and the "Stock" datasets (see Appendix E), the baselines perform poorly. However, in the TimeGAN paper, (see Fig. 3 a and b in https://proceedings.neurips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf), the generated and the real data are not distinguishable. Could it be possible that the baseline models have not converged? Could the authors explain the source of the difference?

---

Additional References

[1*] Van den Oord, Aaron, et al. "Conditional image generation with pixelcnn decoders." Advances in neural information processing systems 29 (2016).

[2*] Van Den Oord, Aäron, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." International conference on machine learning. PMLR, 2016.

[3*] Chung, Junyoung, et al. "A recurrent latent variable model for sequential data." Advances in neural information processing systems 28 (2015).

[4*] Fraccaro, Marco, et al. "Sequential neural models with stochastic layers." Advances in neural information processing systems 29 (2016).

[5*] Lai, Guokun, et al. "Stochastic wavenet: A generative latent variable model for sequential data." arXiv preprint arXiv:1806.06116 (2018).

[6*] Aksan, Emre, and Otmar Hilliges. "STCN: Stochastic temporal convolutional networks." arXiv preprint arXiv:1902.06568 (2019).

[7*] Jarrett, Daniel, Ioana Bica, and Mihaela van der Schaar. "Time-series generation by contrastive imitation." Advances in Neural Information Processing Systems 34 (2021): 28968-28982.

clarity,_quality,_novelty_and_reproducibility: The proposed method is easy-to-follow. The Time-Transformer architecture is inspired by the recent Mobile-Former work (Chen et al., 2021) as mentioned in the paper. It can be considered a temporal extension of the Mobile-Former concept.
summary_of_the_review: It is an interesting line of work, and the paper presents strong performance in the evaluations. However, considering the originality of the architecture, I think the paper requires a revision to provide more insights. The related work discussion also misses relevant time-series models. I believe providing more comprehensive evidence for the contribution of the proposed model would be very helpful for the community. The following questions can be a starting point. Is the Time-Transformer also effective in other frameworks (i.e., seq2seq VAE or autoregressive setting instead of the adversarial autoencoders)? How would other decoder classes perform in the benchmarks?

I am leaning towards a rejection for the current version of the submission as it would require a major revision. However, I am willing to change my score upon clarification and new evidence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a new time series generative model for bridging Temporal Convolutional Networks and Transformer via a layer-wise parallel structure. While the paper contains some interesting idea, reviewers raised major weakness concerns about weak empirical evaluations on toy data, limited novelty, and lack of convincing justification and insufficient comparisons with existing work. While the authors tried to respond to the review questions, some of major review concerns remain and the overall quality of this work is below the acceptance bar.
Decision: Reject