Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary:  The paper propose a cycle gan variants combined with RNN for time series anomaly detection. The setting is assuming training model on a given normal data, then applying the trained model to detect anomalies.  Different detecting criteria are studied in the experiments. 

1. It is not clear the advantage of using GANs in anomaly detection of the proposed algorithm, and it is questionable if using GANs is really useful. The authors only provide hand-waving explanation.  For example, the papers says  " Therefore, once the Critic is trained, it should assign more or less stable scores to the normal sequences and a significantly different score to an anomalous sequence." in p5.  I'm doubt if it is true.  The GAN objective only says the score on x~p(x) and g(z) are similar. Also, the score for normal samples can also have a distribution since we only do mean matching in W-1 distance.  I think a stronger analysis with certain assumptions is necessary for making this statement and justifying the proposed algorithm.  Some possible route can be found in 

Chang et al., Kernel Change-Point Detection with Auxiliary Deep Generative Models, ICLR 2019.  

Although their setting is slightly different, where they focus on change point detection.  They provide a testing power lower bound explanation of using the critic of GANs, but they require some early stopping. Otherwise the guarantee won't hold.  I'm wondering if the analysis can be extended to here. Also, if the proposed algorithm requires the early stopping or not? If not, why?  

2. The experiments are not conclusive.  The simple LSTM predictions beats the proposed algorithms in some cases.   Given there are only two datasets, it's hard to say if the proposed algorithm is really better. Also, the author proposed so many combinations, it is also not clear which one should be favored based on the table. 

3. Many related works are missing.  In addition to Chang el al (2019) mentioned above, there are many related works of using GANs in time series detection problem, e.g. BeatGAN: Anomalous Rhythm Detection using Adversarially Generated Time Series, and many others. 


To summarize, the motivation and the advantage of using GANs is not well justified in addition to experiments.  Also, the authors only study two datasets, and the results are not very conclusive. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a GAN model with cycle-consistent loss function for anomaly detection on timeseries. The loss function is the combination of reconstruction error (L2 norm) and both discriminators (generator and decoder) loss. Once trained, the anomaly score is computed as the mean-var-normalized product of the reconstruction score and the discriminator score. The framework is adapted to timeseries by using LSTM networks for the generator and the encoder and 1D CNN for the discriminators (critics). For the calculation of the reconstruction error, the authors study three approaches: point-wise, dynamic time warping and area difference. The method is applied to two datasets (NASA Spacecraft telemetry, Yahoo traffic) and is compared to simple baselines (LSTM, ARIMA, DenseAE). The results shows an improvement over the baselines. Variations of the models are also studied (with and without critic, different similarity measures for the reconstruction errors). The results show that the discrimination score (critic) does overall provide an improvement over the reconstruction error alone. And for the reconstruction score, the area differences is best, followed by DTW and point-wise.

PROS:

* A somewhat novel approach to anomaly detection in timeseries is proposed that combines GAN discriminator score with a simple area-based similarity measure for the  reconstruction error.
* A solid adaptation of CycleGAN to timeseries.
* The simple area-based similarity measure is interesting and seems to be a good match for timeseries reconstruction as slight shifts are not penalized.
* The paper is well written, well organized and easy to follow.
* The technical content is sound and the math correct.


CONS:

* not really novel (CycleGAN for timeseries) nor DTW as reconstruction error. 
* no comparison to State-of-the-art GAN models for anomaly detection, such as AnoGAN and ADGAN.
* Too few datasets are used (2) for the experiments, making it hard to draw conclusions as to which variation is better.
* Baseline approaches are just mentioned by name. A minimal description would be desirable.
* The ground-truth anomalies and the anomaly threshold are not marked in the plots, making it hard to evaluate them.

Overall the paper proposes a method that improves over baseline but is not compared to other GAN-based SOT models. The novelty is not very high as the main architecture is from CycleGAN and the proposed similarity measures for the reconstruction error are not really novel.


review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper trains a GAN on univariate time series data and uses reconstruction errors in combination with the critic's output to predict anomalous subsequences. The method is applied on two real-world data sets and compared to three simple baselines.

I have several reservations about this manuscript:
- The methodology isn't very original: The GAN architecture is essentially a slightly modified CycleGAN trained with Wasserstein loss.
- Many design choices appear to be ad-hoc: there has been no principled selection of the GAN's hyperparameters, nor has their effect on the experimental results been studied. For computing the reconstruction error the authors use the integral over the difference between two time series without taking the absolute value, which is a very unusual choice. (Standard choices would be e.g. L1 or L2 norm.) While this turns out to have worked "surprisingly well" on the studied datasets, it is not difficult to construct scenarios where this choice will fail. 
- Another design choice that should be discussed in more detail is the smoothing of the time series data using a moving average filter; in the context of anomaly detection this can have a significant impact, and it may not work equally well for different data sets, so a principled approach for determining the level of filtering is paramount. Same goes for the de-trending, and the actual parameters both of the moving averages and the de-trending functions should be reported.
- It is not clear to me how exactly the anomaly scores (supposedly for different sequence lengths l?) are used to predict the subsequence(s) containing an anomaly. It seems there is no incentive to keep the predicted subsequences as short as possible, i.e. if the anomaly score indicates there might be an anomaly present, then the safe approach is to just flag the entire sequence as anomalous (it doesn't hurt the sensitivity they way it's computed)?
- The description of the experiments requires more detail. Are both datasets labelled? After dividing the datasets into rolling chunks of length 100, how many samples do the training and test sets contain? How many of the test set samples contain anomalies?
- The baselines need to be described in more detail. What method was used, e.g. to select and fit the ARIMA models? What type of reconstruction loss was used? How were the anomalous subsequences predicted?
- Is there any way to compare the proposed method with Li et al's (who also used a GAN for anomaly detection in time series data)?

Detailed comments:
- abstract: "particular hard" -> "particularly hard"
- p.3, paragraph starting with "To support...": shouldn't this be "E(x) with x ~ P_X"?
- p.4 and p.6: broken citations "?"

------------------

I acknowledge that I have read the authors' response, but it doesn't change my assessment that several major revisions are needed:
- further motivating the design choices and comparing them to the state-of-the-art;
- formalizing (ideally providing a model for) the sort of anomalies that the proposed method aims to detect;
- discussing limitations of the proposed approach.
review_title: Official Blind Review #1

Meta-review: The paper proposes a cycle-consistent GAN architecture with measuring the reconstruction error of time series for anomaly detection.

The paper aims to address an important problem, but the current version is not ready for publication. We suggest the authors consider the following aspects for improving the paper:
1. The novelty of the proposed model: motivate the design choices and compare them with state-of-art methods
2. Evaluation: formalize the target anomalies and identify datasets/examples where the proposed model can significantly outperform existing solutions. 


Decision: Reject