Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposed a transformer-based generative model, named anoformer, to detect anomaly for time series in an unsupervised manner. Specifically, anoformer combined the Transformer model and the GAN model by adopting transformer-blocks as the generator and discriminator module of GAN. Besides, a two-step masking mechanism was designed to improve the model robustness. Through extensive experiments on 4 public datasets, including real-world and synthetic datasets, the superiority of anoformer was verified. 
strengths_and_weaknesses: Strengths:
+ This paper is easy to follow with some well-designed figure to visualize the key modules of the Anoformer framework. In particular, each key module of Anoformer is elaborated on in following sections.
+ The introduction and the related work section provide a clear and brief introduction of the transformer-based generative models and the difference between the Anoformer and SOTA models.
+ The two-step mask is designed to avoid the generator just copying the input as the output. Specifically, the mask pool in the random mask step provides different views of the input; the exclusive mask in the re-mask step makes the model consider all parts of the input; the entropy-base mask in the re-mask step makes the model consider the uncertain part.   
+ The extensive ablation experiment on different datasets verify the contribution of each key module of the Anoformer. 


Weakness:
- Anoformer detects anomaly based on the entire input or the subsequence of the input. The length of the input time series impacts the delay of the anomaly detection service.
- Too many hyper-parameters are introduced in different sections. A summary table of these parameters should be added for a better review experience.
- The referred paper [30] was cited as both action recognition and text classification reference in line 81-82, while [30] only relates to action recognition. 
questions: please clarify the question below
* One sample in this paper is the entire time series or the subsequence of the time series? If it is subsequence, how do you do the sub-sampling. 

* Whether the Anoformer can provide the anomaly detection in a real-time manner ? Please states the computing complexity and execution speed information. 

* The formulation of the critic C is missing in the section 3.4, please further clarify it. 
limitations: We have concerns on the following points:

* The experiment section only introduces the F1-based metrices, while F1-score highly depends on the threshold. Other metrics, e.g., AUC, the complexity, the running time, should be added.

* It is not very clear Anoformer provides the anomaly detection service on the data-point level or the time-series level? It will be great differences on performance and execution complexity.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a novel transformer-based GAN framework called AnoFormer for time series anomaly detection. In addition, the proposed model is enhanced by a two-step masking strategy consisting of 1) random masking and 2) entropy-based re-masking. The experimental results show the proposed AnoFormer achieves state-of-the-art performance on several benchmark datasets on time series anomaly detection.
strengths_and_weaknesses: Strengths

+ The paper is easy to follow and well-organized.
+ The transformer-based GAN framework and the two-step masking strategy work well.
+ The performance gains are significant.

Weaknesses

- Lack of motivation. The paper only talks about how it is designed, while not elaborating clearly on why it is designed like this. E.g. Why do we make the input signal discrete instead of continuous? Why do we build the mask pool like this? Why do we re-mask 50% of parts masked in Step 1? Why the hyper-parameter $\epsilon$ is set to random?
- Lack of novelty and related work. Actually, the transformer-based GAN framework for time series is not novel and has been proposed in [1], which is not cited and discussed in this paper. The related work section should be more specific (e.g. time series related or time series anomaly detection related literature). The baselines, such as BeatGAN[2], TadGAN[3], RAMED[4], and Anomaly Transformer[5], should be discussed in this section.
- Some notations and presentations are confusing. E.g. $\epsilon$ in Line 185, how it is randomly sampled, with uniform random or normal random? And what is $P_{X'}$ in Line 186?
- The performance is good but codes are not provided.

[1] Adversarial Sparse Transformer for Time Series Forecasting. NeurIPS 2020

[2] BeatGAN: Anomalous Rhythm Detection using Adversarially Generated Time Series. IJCAI 2019

[3] TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks. IEEE International Conference on Big Data 2020

[4] Time Series Anomaly Detection with Multiresolution Ensemble Decoding. AAAI 2021

[5] Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy.  ICLR 2022
questions: Why do we make the input signal discrete instead of continuous?

Why do we build the mask pool like this?

Why do we re-mask 50% of parts masked in Step 1?

Why the hyper-parameter $\epsilon$ is set to random? What is the distribution $P_{X'}$ in Line 186?

The performance of the paper is attractive, however, considering the novelty and the unclear elaboration, I vote for borderline reject. 
limitations: N/A
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper tackles the task of detecting and localizing anomalies in time series data. It follows a transformer approach where time series are automatically masked, then the transformer attempt to reconstruct the masked region. The reconstruction error is used as an anomaly score. A few technical tricks are incorporated such as using a GAN loss in training and an adaptive masking strategy. The method is evaluated on a few datasets and achieves better results than a set of deep learning approaches.  
strengths_and_weaknesses: Originality: The main premise of using inpainting as a task for reconstruction-based anomaly detection in not new (e.g. [1] but there are many). Nor is using a GAN discriminator (also found in [1]). The main difference here is using a transformer, the particular masking approach and the particular hyperparameter choices. This is an original contribution but a modest one. 

Quality: The method appears sound as far as it goes. The GAN ablation is somewhat worrisome as it appears to be useful only in a particular configuration and could be due to randomness in the data or a particular hyperparameter choice. However, the evaluation is not very convincing. The NeurIPS-TS datasets were generated by the authors, and are really easy. Note the original paper had classical baselines as doing much better than deep learning methods - but the original datasets were much smaller than the ones generated here. Also the other 3 benchmarks are not standard. It is not clear to me how they were selected. Also simple/classical baselines were not attempted. Several papers (e.g. NeurIPS-TS cited in this paper or [2]) have recently show that deep learning methods do not outperform classical baselines in many cases. Another issue I did not not see highlighted - the transformer + discretization approach might find it hard scale to multivariate time series (as the number of token to be predicted might expand by two orders of magnitude, in some cases). Although one of the benchmarks had 2 dimensions, it is not clear to me how this method would scale to standard benchmarks used by the deep learning time series AD community (SWaT, MSL, WADI etc.). 

Clarity: The paper is well written, was a pleasure to read.

Significance: Due to the concerns mentioned above and the limited originality, I think this work is at risk of having low significance.   

[1] Yan, Xudong, et al. "Learning semantic context from normal samples for unsupervised anomaly detection." AAAI'21
[2] Kim, Siwon, et al. "Towards a rigorous evaluation of time-series anomaly detection." AAAI'22.
questions: * How were the benchmark datasets chosen?
* How do the classical benchmarks for NeurIPS-TS/[2] etc. perform?
* How well does the method scale to multivariate TS (e.g. 100 dimensions)?


limitations: Some were discussed in the appendix. A few more were highlighted in the review. A more extensive discussion would have been helpful.
review_title: null

Meta-review: metareview: This paper proposes a transformer-based GAN method and a two-step masking mechanism for time series anomaly detection. The proposed method is demonstrated on a variety of datasets. 

After rebuttals, both Reviewer 73J5 and Reviewer W6VQ remained negative. The main concern is the novelty and significance of the proposed method. 
confidence: Certain
Decision: Reject