Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposed an unsupervised anomaly detection method for the scenarios where the training data not only includes normal data but also a lot of anomaly data. The basic idea of this paper is to iteratively refine the normal data subset, selected from the whole training data set. Specifically, the paper first train an auto-encoder (AE) and determine which are the normal data samples according to the reconstruction errors. Then, using the normal data to retrain the AE again, and re-select the normal samples. Repeat the above two steps until convergence. 

Overall, the novelty of this paper is in doubt. Detecting anomaly by reconstruction error of AE has been explored thoroughly, and this paper only extends it to iteratively select the normal samples. The extension seems to be very straightforward. 

Also, the refining process is also problematic. It will highly depend on the initial selection, and the error will be propagated to subsequent detections. How to determine which data samples are anomalous is a key to the success of the model, but the proposed method based on the variance assumption is too intuitive and not convincing.

In addition, the experimental results on the very simple MNIST task is very poor, putting the effectiveness of the proposed model in doubt.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In this paper the authors propose a framework for anomaly detection. The method is based on autoencoders and reconstruction error, but instead of training the autoencoder using all the data-points, the method iteratively uses some form of clustering to determine the points which presumably belong to the normal set, and uses them for training the autoencoder. This helps make the method robust when the portion of anomalous data-points is high.

The paper is well-written and clear and the proposed architecture is novel to my knowledge. Nevertheless, I have the following concerns about the paper. Given clarifications in the author response, I would be willing to increase the score. 

- In terms of novelty, separating the dataset into normal + outliers/noise is not novel (Zhou 2017 cited in the paper). The novel part here is perhaps using variance and clustering for making the separation. However, using variance is not well motivated and it is referred to Figure 4, in which the argument is not clear. Similarly, why the reconstruction error is included in the latent representation (eq 5) is not clear. 

- Given the similarity of the idea to Zhou 2017, the comparison seems important (if the code is available), and also proper discussion of such similar works is required, which currently is not presented.

- How is the performance affected by very low ratio of anomalies? This can be shown by including %2,%3 of anomalies in Table 2. 

- The sensitivity of the results to the choice of hyper-parameters: p0, p, and r is not clear, and how these parameters are chosen is not discussed. It would be interesting to see how the performance is affected by different choices of the hyper-parameters.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In the article, the authors solve the problem of anomaly detection using a fully unsupervised approach. They try to deal with the main challenge of anomaly detection: a lack of certainty on what defines normal data and anomaly one. For this purpose, the authors iteratively use: 1) autoencoders to learn the representation of the data; 2) applying in the latent space clustering to get a new training set and retrain autoencoders. The experimental results show that the author’s method performed better results than such a baseline model as one-class SVM and one-class NN. 
 
The proposed algorithm looks robust and well-motivated, but the text of the article and the experiments can be improved. As the proposed approach is a heuristic, the experiments should be done more persuasively, including more metrics used and more alternative algorithms considered. 

The key comments are the following:
1. The formatting of the article needs to be improved e.g.:
2. there is no comma between rows in the equation (1) ;
<<to be accepted into the “training” set, .>> - there is an extra comma;
3. round brackets in the equation (6) should be bigger;
4. Table 3 is bigger than the page sizes.
5. The quality of the pictures should be improved:
- Increase the captions font size in Figure 2;
- The captions and the legends in Figure 3 are practically not visible;
6.  Is the DAGMM method SOTA in anomaly detection with deep autoencoder? There are many other methods with similar ideas. We expect that we should provide a comparison with other methods: 
https://arxiv.org/pdf/1809.02728.pdf - IGMM-GAN
https://papers.nips.cc/paper/7915-generative-probabilistic-novelty-detection-with-adversarial-autoencoders.pdf - GPND AE
6. Also, DAGMM works badly according to the experiments in the article with max AUROC in Table 1 only 50.3 (so it seems that it is no better than the coin-flipping)
7. Why was the only selected digit for analysis 4? Usual for comparison anomaly detection on MNIST dataset apply the following procedure: for each figure in dataset consider corresponded class as anomaly data, and the rest of the digits are used as normal data, e.g.:
https://arxiv.org/pdf/1802.06222.pdf
https://arxiv.org/pdf/1906.11632.pdf 
8.  Class imbalances can affect the value of the AUROC metric. Possibly, the other metrics like AUPRC, F1-scores will better reflect the work of the algorithms for comparison. Also, AUROC is not representative when it comes to the selection of the threshold for anomaly detection. Precision and Recall can help to get more insights.
9. In Table 3, the result of applying the proposed algorithm presented with standard deviation, but other methods are represented by one metric value. Why? The explanation is required.
review_title: Official Blind Review #3

Meta-review: The authors propose an approach for anomaly detection in the setting where the training data includes both normal and anomalous data.  Their approach is a fairly straightforward extension of existing ideas, in which they iterate between clustering the data into normal vs. anomalous and learning an autoencoder representation of normal data that is then used to score normality of new data.  The results are promising, but the experiments are fairly limited.  The authors argue that their experimental settings follow those of prior work, but I think that for such an incremental contribution, more empirical work should be done, regardless of the limitations of particular prior work.
Decision: Reject