Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Review: The paper proposes a technique for anomaly detection. It presents a novel method that unifies the current classification-based approaches to overcome generalization issues and outperforms the state of the art. This work also generalizes to non-image data by extending the transformation functions to include random affine transformations. A lot of important applications of anomaly detection are based on tabular data so this is significant. The “normal” data is divided into M subspaces where there are M different transformations, the idea is to then learn a feature space using triplet loss that learns supervised clusters with low intra-class variation and high inter-class variation. A score is computed (using the probabilities based on the learnt feature space) on the test samples to obtain their degree of anomalousness. The intuition behind this self-supervised approach is that learning to discriminate between many types of geometric transformations applied to normal images can help to learn cues useful for detecting novelties. 

Pros:
- There is an exhaustive evaluation and comparison across different types of data with the existing methods along with the SOTA. 
- It is interesting to see how random transformations indeed helped to achieve adversarial robustness.
- The method is generalized to work on any type of data with arbitrary number of random tasks. It can even be used in a linear setting if needed for small datasets.

Cons:
- While I liked that an analysis was done to see the robustness of the method on the contaminated data, I would be interested to see a more rigorous comparison in this fully unsupervised setting. 


Comments/Question:
Does the selection of the transformation types affect the method performance at all? 

In the Results section on Page 7, there are a couple of “??” instead of table numbers.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper proposes a novel approach to classification-based anomaly detection for general data. Classification-based anomaly detection uses auxiliary tasks (transformations) to train a model to extract useful features from the data. This approach is well-known in image data, where auxiliary tasks such as classification of rotated or flipped images have been demonstrated to work effectively. The paper generalizes to the task by using the affine transformation y = Wx+b. A novel distance-based classification is also devised to learn the model in such as way that it generalizes to unseen data. This is achieved by modeling the each auxiliary task subspace by a sphere and by using the distance to the center for the calculation of the loss function. The anomaly score then becomes the product of the probabilities that the transformed samples are in their respective subspaces. The paper provides comparison to SOT methods for both Cifar10 and 4 non-image datasets. The proposed method substantially outperforms SOT on all datasets. A section is devoted to explore the benefits of this approach on adversarial attacks using PGD. It is shown that random transformations (implemented with the affine transformation and a random matrix) do increase the robustness of the models by 50%. Another section is devoted to studying the effect of contamination (anomaly data in the training set). The approach is shown to degrade more gracefully than DAGMM on KDDCUP99. Finally, a section studies the effect of the number of tasks on the performance, showing that after a certain number of task (which is probably problem-dependent), the accuracy stabilizes.


PROS:

* A general and novel approach to anomaly detection with SOT results.

* The method allows for any type of classifier to be used. The authors note that deep models perform well on the large datasets (KDDCUP) while shallower models are sufficient for smaller datasets.

* The paper is relatively well written and easy to follow, the math is clearly laid out.


CONS:

* The lack of a pseudo-code algorithm makes it hard to understand and reproduce the method

* Figure 1 (left) has inverted colors (DAGMM should be blue - higher error).
* Figure 1 (right) - it is unclear what the scale of the x-axis is since there is only 1 label. Also the tick marks seem spaced logarithmically, which, if i understand correctly, is wrong.

* The paragraph "Number of operations" should be renamed "Number of tasks" to be consistent. Also the sentence "From 16 ..." should be clarified, as it seems to contrast accuracy and results, which are the same entity. The concept of 'stability of results' is not explained clearly. It would suffice to say: 'From 16 tasks and larger, the accuracy remains stable'.

* In section 6, the paragraph "Generating many tasks" should be named "Number of tasks", to be consistent with the corresponding paragraph in section 5.2. Also the first sentence should be: "As illustrated in Figure 1 (right), increasing the number of tasks does result in improved performance but the trend is not linear and beyond a certain threshold, no improvements are made. And again the concept of 'stability' is somewhat misleading here. The sentence '...it mainly improves the stability of the results' is wrong. The stability is not improved, it is just that the performance trend is stable.

* The study on the number of tasks should be carried on several datasets. Only one dataset is too few to make any claims on the accuracy trends as the number of task is increased.

* The authors should coin an acronym to name their methods.

Overall this paper provides a novel approach to classification-based semi-supervised anomaly detection of general data. The results are very encouraging, beating SOT methods by a good margin on standard benchmarks.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: UPDATE:
I acknowledge that I‘ve read the author responses as well as the other reviews.

I appreciate the clarifications, additional experiments, and overall improvements made to the paper. I updated my score to 6 Weak Accept. 


####################

This paper proposes a deep method for anomaly detection (AD) that unifies recent deep one-class classification [6] and transformation-based classification [3, 4] approaches. The proposed method transforms the data to $M$ subspaces via $M$ random affine transformations and identifies with each such transformation a cluster centered around some centroid (set as the mean of the respectively transformed samples). The training objective of the method is defined by the triplet loss [5] which learns to separate the subspaces via maximizing the inter-class as well as minimizing the intra-class variation. The anomaly score for a sample is finally given by the sum of log-probabilities, where each transformation-/cluster-probability is derived from the distance to the cluster center. Using random affine transformations, the proposed method is applicable to general data types in contrast to previous works that only consider geometric transformations (rotation, translation, etc.) on image data [3, 4]. The paper conclusively presents experiments on CIFAR-10 and four tabular datasets (Arrhythmia, Thyroid, KDD, KDD-Rev) that indicate a superior detection performance of the proposed method over baselines and deep competitors.

I think this paper is not yet ready for acceptance due to the following main reason: 
(i) The experimental evaluation needs clarification and should be extended to judge the significance of the empirical results.

(i) I think the comparison with state-of-the-art deep competitors [6, 4] should consider at least another image dataset besides CIFAR-10, e.g. Fashion-MNIST or the recently published MVTec [1] for AD. On CIFAR-10, do you also consider geometric transformations however using your triplet loss or are the reported results from random affine transformations? I think reporting both would be insightful to see the difference between image-specific and random affine transformations.
On the tabular datasets, how do deep networks perform in contrast to the final linear classifier reported on most datasets? Especially when only using a final linear classifier, the proposed method is very similar to ensemble learning on random subspace projections. Figure 1 (right) shows an error curve that is also typical for ensemble learning (decrease in mean error and reduction in overall variance). I think this should be discussed and ensemble baselines [2] should be considered for a fair comparison. Table 2 also seems incomplete with the variances missing for some methods?
Further clarifications are needed. How many transformations $M$ do you consider on the specific datasets? How is hyperparameter $s$ chosen?
Finally, I think the claim that the approach is robust against training data contamination is too early from only comparing against the DAGMM method on KDDCUP (Is Figure 1 (left) wrong labeled? As presented DAGMM shows a lower classification error).

Overall, I think the paper proposes an interesting unification and generalization of existing state-of-the-art approaches [6, 4], but I think the experimental evaluation needs to be more extensive and clarified to judge the potential significance of the results. The presentation of the paper also needs some polishing as there are many typos and grammatical errors in the current manuscript (see comments below).


####################
*Additional Feedback*

*Positive Highlights*
1. Well motivated anomaly detection approach that unifies existing state-of-the-art deep one-class classification [6] and transformation-based classification [3, 4] approaches that indicates improved detection performance and is applicable to general types of data.
2. The work is well placed in the literature. All relevant and recent related work is included in my view.

*Ideas for Improvement*
3. Extend and clarify the experimental evaluation as discussed in (i) to infer statistical significance of the results.
4. I think many details from the experimental section could be moved to the Appendix leaving space for the additional experiments.
5. Maybe add some additional tabular datasets as presented in [2, 7].
6. Maybe clarify “Classification-based AD” vs. “Self-Supervised AD” a bit more since unfamiliar readers might be confused with supervised classification.
7. Improve the presentation of the paper (fix typos and grammatical errors, improve legibility of plots)
8. Some practical guidance on how to choose hyperparameter $s$ would be good. This may just be a default parameter recommendation and showing that the method is robust to changes in s with a small sensitivity analysis.

*Minor comments*
9. The set difference is denoted with a backslash not a forward slash, e.g. $R^L \setminus X$.
10. citet vs citep typos in the text (e.g. Section 1.1, first paragraph “ ... Sakurada & Yairi (2014); ...”)
11. Section 1.1: “ADGMM introduced by Zong et al. (2018) ...” » “DAGMM introduced by Zong et al. (2018) ...”.
12. Eq. (1): $T(x, \tilde{m})$ in the first denominator as well.
13. Section 2, 4th paragraph: $T(x, \tilde{m}) \in R^L \setminus X_{\tilde{m}}$.
14. $m$, $\tilde{m}$, and $m'$ are used somewhat inconsistently in the text.
15. Section 3: “Note, that it is defined everywhere.”?
16. Section 4: "If $T$ is chosen deterministicaly ..." >> "If $T$ is chosen deterministically ..."
17. Section 5, first sentence: “... to validate the effectiveness our distance-based approach ...” » “... to validate the effectiveness of our distance-based approach ...”.
18. Section 5.1: “We use the same same architecture and parameter choices of Golan & El-Yaniv (2018) ...” » “We use the same architecture and parameter choices as Golan & El-Yaniv (2018) ...”
19. Section 5.2: “Following the evaluation protocol of Zong et al. Zong et al. (2018) ...” » “Following the evaluation protocol of Zong et al. (2018) ...”.
20. Section 5.2: “Thyroid is a small dataset, with a low anomally to normal ratio ...” » “Thyroid is a small dataset, with a low anomaly to normal ratio ...”.
21. Section 5.2, KDDCUP99 paragraph: “Tab. ??” reference error.
22. Section 5.2, KDD-Rev paragraph: “Tab. ??” reference error.


####################
*References*

[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.
[2] J. Chen, S. Sathe, C. Aggarwal, and D. Turaga. Outlier detection with autoencoder ensembles. In SDM, pages 90–98, 2017.
[3] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
[4] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[5] X. He, Y. Zhou, Z. Zhou, S. Bai, and X. Bai. Triplet-center loss for multi-view 3d object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1945–1954, 2018.
[6] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[7] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.
review_title: Official Blind Review #3

Meta-review: The paper presents a method that unifies classification-based approaches for outlier detection and (one-class) anomaly detection. The paper also extends the applicability to non-image data.

In the end, all the reviewers agreed that the paper makes a valuable contribution and I'm happy to recommend acceptance.
Decision: Accept (Poster)