Classification-Based Anomaly Detection for General Data

Abstract: Anomaly detection, finding patterns that substantially deviate from those seen previously, is one of the fundamental problems of artificial intelligence. Recently, classification-based methods were shown to achieve superior results on this task. In this work, we present a unifying view and propose an open-set method, GOAD, to relax current generalization assumptions. Furthermore, we extend the applicability of transformation-based methods to non-image data using random affine transformations. Our method is shown to obtain state-of-the-art accuracy and is applicable to broad data types. The strong performance of our method is extensively validated on multiple datasets from different domains.  

Reviews and author responses:

UPDATE:
I acknowledge that I‘ve read the author responses as well as the other reviews.

I appreciate the clarifications, additional experiments, and overall improvements made to the paper. I updated my score to 6 Weak Accept. 


####################

This paper proposes a deep method for anomaly detection (AD) that unifies recent deep one-class classification [6] and transformation-based classification [3, 4] approaches. The proposed method transforms the data to $M$ subspaces via $M$ random affine transformations and identifies with each such transformation a cluster centered around some centroid (set as the mean of the respectively transformed samples). The training objective of the method is defined by the triplet loss [5] which learns to separate the subspaces via maximizing the inter-class as well as minimizing the intra-class variation. The anomaly score for a sample is finally given by the sum of log-probabilities, where each transformation-/cluster-probability is derived from the distance to the cluster center. Using random affine transformations, the proposed method is applicable to general data types in contrast to previous works that only consider geometric transformations (rotation, translation, etc.) on image data [3, 4]. The paper conclusively presents experiments on CIFAR-10 and four tabular datasets (Arrhythmia, Thyroid, KDD, KDD-Rev) that indicate a superior detection performance of the proposed method over baselines and deep competitors.

I think this paper is not yet ready for acceptance due to the following main reason: 
(i) The experimental evaluation needs clarification and should be extended to judge the significance of the empirical results.

(i) I think the comparison with state-of-the-art deep competitors [6, 4] should consider at least another image dataset besides CIFAR-10, e.g. Fashion-MNIST or the recently published MVTec [1] for AD. On CIFAR-10, do you also consider geometric transformations however using your triplet loss or are the reported results from random affine transformations? I think reporting both would be insightful to see the difference between image-specific and random affine transformations.
On the tabular datasets, how do deep networks perform in contrast to the final linear classifier reported on most datasets? Especially when only using a final linear classifier, the proposed method is very similar to ensemble learning on random subspace projections. Figure 1 (right) shows an error curve that is also typical for ensemble learning (decrease in mean error and reduction in overall variance). I think this should be discussed and ensemble baselines [2] should be considered for a fair comparison. Table 2 also seems incomplete with the variances missing for some methods?
Further clarifications are needed. How many transformations $M$ do you consider on the specific datasets? How is hyperparameter $s$ chosen?
Finally, I think the claim that the approach is robust against training data contamination is too early from only comparing against the DAGMM method on KDDCUP (Is Figure 1 (left) wrong labeled? As presented DAGMM shows a lower classification error).

Overall, I think the paper proposes an interesting unification and generalization of existing state-of-the-art approaches [6, 4], but I think the experimental evaluation needs to be more extensive and clarified to judge the potential significance of the results. The presentation of the paper also needs some polishing as there are many typos and grammatical errors in the current manuscript (see comments below).


####################
*Additional Feedback*

*Positive Highlights*
1. Well motivated anomaly detection approach that unifies existing state-of-the-art deep one-class classification [6] and transformation-based classification [3, 4] approaches that indicates improved detection performance and is applicable to general types of data.
2. The work is well placed in the literature. All relevant and recent related work is included in my view.

*Ideas for Improvement*
3. Extend and clarify the experimental evaluation as discussed in (i) to infer statistical significance of the results.
4. I think many details from the experimental section could be moved to the Appendix leaving space for the additional experiments.
5. Maybe add some additional tabular datasets as presented in [2, 7].
6. Maybe clarify “Classification-based AD” vs. “Self-Supervised AD” a bit more since unfamiliar readers might be confused with supervised classification.
7. Improve the presentation of the paper (fix typos and grammatical errors, improve legibility of plots)
8. Some practical guidance on how to choose hyperparameter $s$ would be good. This may just be a default parameter recommendation and showing that the method is robust to changes in s with a small sensitivity analysis.

*Minor comments*
9. The set difference is denoted with a backslash not a forward slash, e.g. $R^L \setminus X$.
10. citet vs citep typos in the text (e.g. Section 1.1, first paragraph “ ... Sakurada & Yairi (2014); ...”)
11. Section 1.1: “ADGMM introduced by Zong et al. (2018) ...” » “DAGMM introduced by Zong et al. (2018) ...”.
12. Eq. (1): $T(x, \tilde{m})$ in the first denominator as well.
13. Section 2, 4th paragraph: $T(x, \tilde{m}) \in R^L \setminus X_{\tilde{m}}$.
14. $m$, $\tilde{m}$, and $m'$ are used somewhat inconsistently in the text.
15. Section 3: “Note, that it is defined everywhere.”?
16. Section 4: "If $T$ is chosen deterministicaly ..." >> "If $T$ is chosen deterministically ..."
17. Section 5, first sentence: “... to validate the effectiveness our distance-based approach ...” » “... to validate the effectiveness of our distance-based approach ...”.
18. Section 5.1: “We use the same same architecture and parameter choices of Golan & El-Yaniv (2018) ...” » “We use the same architecture and parameter choices as Golan & El-Yaniv (2018) ...”
19. Section 5.2: “Following the evaluation protocol of Zong et al. Zong et al. (2018) ...” » “Following the evaluation protocol of Zong et al. (2018) ...”.
20. Section 5.2: “Thyroid is a small dataset, with a low anomally to normal ratio ...” » “Thyroid is a small dataset, with a low anomaly to normal ratio ...”.
21. Section 5.2, KDDCUP99 paragraph: “Tab. ??” reference error.
22. Section 5.2, KDD-Rev paragraph: “Tab. ??” reference error.


####################
*References*

[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.
[2] J. Chen, S. Sathe, C. Aggarwal, and D. Turaga. Outlier detection with autoencoder ensembles. In SDM, pages 90–98, 2017.
[3] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
[4] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[5] X. He, Y. Zhou, Z. Zhou, S. Bai, and X. Bai. Triplet-center loss for multi-view 3d object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1945–1954, 2018.
[6] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[7] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.

We thank the reviewer for the dedicated and positive review and are pleased the reviewer recognized the novelty of the approach, its state-of-the-art performance and computational scalability.

As requested by the reviewer, pseudo code for the algorithm was added to the paper.

The labels were indeed mislabeled in Fig.1, our approach achieved the better performance. We updated the figure to elucidate all issues brought to our attention by the reviewer.

We run the “number of tasks” experiment on the other datasets, they are shown in the appendix. For all datasets, increasing the number of tasks increases performance up to a certain point. From this point increasing the number of tasks mainly decreases variance between runs. For the smaller datasets, accuracy improves up to a higher number of transformations. We presented the results with the maximal number of transformations in the appendix. We elucidated the text related to this experiment.

We coined the acronym GOAD and use it for our method in the text. Thank you for this helpful suggestion.

We thank the reviewer for the dedicated and mostly positive review. We are pleased the reviewer recognized that our approach is interesting and well motivated and that it convincingly outperforms the state-of-the-art competitors on standard benchmarks.

We sincerely apologize that the editorial quality of the paper was not of the high standard that the reviewer naturally expected. We have significantly revised and improved it, including all the stylistic issues the reviewer highlighted. We believe the quality is now of a high standard.

“consider at least another image dataset besides CIFAR-10, e.g. Fashion-MNIST”: The results for FashionMNIST were added to the paper in the appendix. Overall our method achieves the best performance of all methods.

“On CIFAR-10, do you also consider geometric transformations”: We are using exactly the same geometric transformations as Golan and El-Yaniv [4]. As noted in the paper, geometric transformations are a special case of the affine transformation class. For CNNs to be maximally effective the transformation needs to be locality preserving. Using random affine matrices for images classified by CNNs did not perform competitively as it removed pixel locality information exploited by CNNs. This is different for tabular data where there is no order between different features, making random matrices a good choice of transformation. We updated this insight in the manuscript.

“how do deep networks perform in contrast to the final linear classifier reported on most datasets?”: Results of deep classifiers are significantly better than linear methods for KDD, KDDRev which are data rich (linear accuracy forKDDCup99, KDDRev was around 80%). Results for deep classifiers are roughly similar to linear classifiers for data poor tasks (Thyroid, Arrhythmia).

“ensemble baselines [2] should be considered”: we compared our method an ensemble baseline similar to [2] (implemented by the PyOD package) in exactly the same setting as our experiments. Results can be seen in the appendix. Our method outperforms the ensemble method on the tested datasets. The results are most remarkable on the larger datasets, on which deep classifiers have a distinct advantage.

“Table 2 also seems incomplete with the variances missing”:  Results are copied from Zong et al. did not contain variance, the variance values for these methods are missing in the table. All methods that we ran report variance results. We revised the paper to clarify this.

“How many transformations do you consider on the specific datasets?”: A graph with the accuracies for all datasets is shown in the appendix. Above a certain threshold the number is not critical, the reported experiments used 32 transformations were used for all datasets but Arrhythmia which used 64 (due to its high variance owing to its small size). We present in the appendix results for a larger number of transformations on the smaller datasets (1024 on Arrhythmia and Thyroid) with performance increases on these smaller datasets.

“How is hyperparameter s chosen?”: the value is not very sensitive. We found that a value of s=1.0 performed well in all datasets and is the recommended starting point. Although originally we ran Cifar10 using s=0.1, we present the same experiment with s=1.0 in the appendix with very similar numbers.

We followed the further ideas for improvement proposed by the reviewer. The style and editorial quality of the paper were much improved. The requested experiments were added. We will add further tabular experiments in the final version of the paper. The clarifications requested by the reviewer were added. We also clarified the “Classification” and “Self-supervised” terms.

We are thankful for the reviewer’s detailed comments, which we believe have improved the paper.

We thank the reviewer for the dedicated and positive review. We are pleased that the reviewer recognized the novelty and strong performance of our method across many data types, the novel adversarial robustness that it brings and its scalability across different computational regimes.

We presented the contamination results on KDDRev as this was the comparison made in the DA-GMM paper. To address the reviewer’s request for further experiments on contaminated data, we computed the results on the other datasets (Thyroid did not have enough anomalies to perform this experiment). The graphs are presented in the appendix. The trend is similar to that observed in the KDDRev experiment.

The type of transformation affects the results but not very significantly. We present results in the appendix for the affine transformation restricted to (i) permutation and (ii) rotation matrices. The results in line with the full affine transformation (typically a little lower). We have previously experimented with using randomized neural networks as the auxiliary transformations however in our preliminary experiments, the results were not as good as for the linear (affine, rotation, permutation) transformation classes.

We fixed the missing table reference in the revised version of the submission.


This paper proposes a novel approach to classification-based anomaly detection for general data. Classification-based anomaly detection uses auxiliary tasks (transformations) to train a model to extract useful features from the data. This approach is well-known in image data, where auxiliary tasks such as classification of rotated or flipped images have been demonstrated to work effectively. The paper generalizes to the task by using the affine transformation y = Wx+b. A novel distance-based classification is also devised to learn the model in such as way that it generalizes to unseen data. This is achieved by modeling the each auxiliary task subspace by a sphere and by using the distance to the center for the calculation of the loss function. The anomaly score then becomes the product of the probabilities that the transformed samples are in their respective subspaces. The paper provides comparison to SOT methods for both Cifar10 and 4 non-image datasets. The proposed method substantially outperforms SOT on all datasets. A section is devoted to explore the benefits of this approach on adversarial attacks using PGD. It is shown that random transformations (implemented with the affine transformation and a random matrix) do increase the robustness of the models by 50%. Another section is devoted to studying the effect of contamination (anomaly data in the training set). The approach is shown to degrade more gracefully than DAGMM on KDDCUP99. Finally, a section studies the effect of the number of tasks on the performance, showing that after a certain number of task (which is probably problem-dependent), the accuracy stabilizes.


PROS:

* A general and novel approach to anomaly detection with SOT results.

* The method allows for any type of classifier to be used. The authors note that deep models perform well on the large datasets (KDDCUP) while shallower models are sufficient for smaller datasets.

* The paper is relatively well written and easy to follow, the math is clearly laid out.


CONS:

* The lack of a pseudo-code algorithm makes it hard to understand and reproduce the method

* Figure 1 (left) has inverted colors (DAGMM should be blue - higher error).
* Figure 1 (right) - it is unclear what the scale of the x-axis is since there is only 1 label. Also the tick marks seem spaced logarithmically, which, if i understand correctly, is wrong.

* The paragraph "Number of operations" should be renamed "Number of tasks" to be consistent. Also the sentence "From 16 ..." should be clarified, as it seems to contrast accuracy and results, which are the same entity. The concept of 'stability of results' is not explained clearly. It would suffice to say: 'From 16 tasks and larger, the accuracy remains stable'.

* In section 6, the paragraph "Generating many tasks" should be named "Number of tasks", to be consistent with the corresponding paragraph in section 5.2. Also the first sentence should be: "As illustrated in Figure 1 (right), increasing the number of tasks does result in improved performance but the trend is not linear and beyond a certain threshold, no improvements are made. And again the concept of 'stability' is somewhat misleading here. The sentence '...it mainly improves the stability of the results' is wrong. The stability is not improved, it is just that the performance trend is stable.

* The study on the number of tasks should be carried on several datasets. Only one dataset is too few to make any claims on the accuracy trends as the number of task is increased.

* The authors should coin an acronym to name their methods.

Overall this paper provides a novel approach to classification-based semi-supervised anomaly detection of general data. The results are very encouraging, beating SOT methods by a good margin on standard benchmarks.


Review: The paper proposes a technique for anomaly detection. It presents a novel method that unifies the current classification-based approaches to overcome generalization issues and outperforms the state of the art. This work also generalizes to non-image data by extending the transformation functions to include random affine transformations. A lot of important applications of anomaly detection are based on tabular data so this is significant. The “normal” data is divided into M subspaces where there are M different transformations, the idea is to then learn a feature space using triplet loss that learns supervised clusters with low intra-class variation and high inter-class variation. A score is computed (using the probabilities based on the learnt feature space) on the test samples to obtain their degree of anomalousness. The intuition behind this self-supervised approach is that learning to discriminate between many types of geometric transformations applied to normal images can help to learn cues useful for detecting novelties. 

Pros:
- There is an exhaustive evaluation and comparison across different types of data with the existing methods along with the SOTA. 
- It is interesting to see how random transformations indeed helped to achieve adversarial robustness.
- The method is generalized to work on any type of data with arbitrary number of random tasks. It can even be used in a linear setting if needed for small datasets.

Cons:
- While I liked that an analysis was done to see the robustness of the method on the contaminated data, I would be interested to see a more rigorous comparison in this fully unsupervised setting. 


Comments/Question:
Does the selection of the transformation types affect the method performance at all? 

In the Results section on Page 7, there are a couple of “??” instead of table numbers.


Meta-review: The paper presents a method that unifies classification-based approaches for outlier detection and (one-class) anomaly detection. The paper also extends the applicability to non-image data.

In the end, all the reviewers agreed that the paper makes a valuable contribution and I'm happy to recommend acceptance.

Decision: accept-poster