Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work investigates a classic unsupervised outlier detection problem, in which we do not have any label information and need to learn a detection model from those unlabeled data to identify any inconsistent data points as outliers. The key approach here is to apply existing self-supervised contrastive feature learning methods to extract feature representations and then apply a cluster-based method to calculate outlier scores. It also presents two ways to leverage labeled outlier data if available, including an improved mahalanobis distance method and the application of supervised contrastive learning methods proposed recently. The methods, including unsupervised and semi-supervised methods that use a few labeled outlier data, are evaluated using four datasets. As I read through the paper, I find the following major issues.

1. the question this work intends to answer, "Can we design an effective outlier detector with access to only unlabeled data from training distribution?", is a classic and well-studied problem in the anomaly/outlier detection community. There have been many studies over this problem. We cannot simply ignore those previous work  and states it as a new problem. see resources like Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3), 1-58. or Aggarwal, C. C. (2015). Outlier analysis. Springer, Cham. for numerous previous work on using shallow methods to address this problem.

2. There are a number of studies on self-supervised outlier detection approaches as well as what is called few-shot outlier detection approaches, but I cannot find any discussion of those work and the empirical comparison to these methods. The authors may refer to some recent survey papers, such as "Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2020). Deep Learning for Anomaly Detection: A Review. arXiv preprint arXiv:2007.02500.", to find some of these studies. Some closely related methods are: self-supervised methods such as GT and E3Outlier that learns feature representations using a pre-text task in a self-supervised way; unsupervised outlier detection methods such as RDA, REPEN, ALOCC, OCGAN, etc.; methods that use a few labeled outlier data such as Deep SAD, DevNet, REPEN, etc. Please see table 1 in that survey paper for more details. The authors are suggested to discuss and differentiate their method from these existing studies, and to compare empirical comparisons to these closely related methods.

3. The claiming of "a parameter-free detector" is misleading and incorrect. Similar to other methods, the presented methods still have a huge number of hyper-parameters. They may be able to work without parameter tuning on each dataset, but this is not parameter-free. Also, many existing methods can also work in this way.

4. The way that the presented outlier detection methods utilizes the labeled outlier data may less effective than previous work, because the cluster-based anomaly scoring here is separated from the representation learning. More advanced approaches (see some of the methods mentioned above) can unify the anomaly scoring and the representation learning together and the labeled outlier data is used to optimized the entire anomaly detection pipeline, rather than the representation learning stage only. 

5. The experiment settings are not properly designed to justify the paper's arguments. First, closely related deep unsupervised and semi-supervised methods are missing in the comparison in tables 1 and 3. Second, I think it is important to include some popular baselines here. For example, how is the performance of using traditional outlier detectors such as iforest, lof, and knn distance on feature representations extracted with a pre-trained resnet-50? How many benefits does the computationally extensive contrastive representation learning gain compared to those simple solutions?  Third, why is the cluster-based outlier scoring method used? can we use other traditional outlier detectors such as iforest, lof, and knn distance?
review_title: Limited novelty; major issues in experiment settings; important references are missing 

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Summary] 
The paper addresses the OOD problem without learning from class labels. It proposes to learn the feature representation with unsupervised learning, then applies Mahalanobis distance to measure how far a test data is away from the in-distribution data. The proposed framework also considers the cases of when class labels or OOD data are available. The experiments on image datasets (mainly cifar-10/100) demonstrate its advantage over other unsupervised OOD methods.

[Reasons for score]
Bullet 1 in the cons is a significant concern. However, the method and analysis provide interesting insights. Overall, the pros and cons are equally substantial. 

[Strengths]
1. The proposed framework is sounded and can well handle a variety of cases when class labels or OOD data are available.
2. The way it utilizes a small amount of OOD data is novel (eq.3). 
3. The experiments and analysis are sufficient to support its major claim while providing additional insights (ex: Figures 2 and 4, Table 4).

[Weaknesses]
1. The eq.3 assumes OOD data are from a cluster that has the same mean and variance. This assumption is not valid in general cases. The experiments have a setup that favors this assumption; therefore, bias exists. The paper should explicitly point out this as a substantial limitation and provide some experiments (ex: use SVHN+CIFAR10 for the labeled OOD data) to elaborate it.
2. The evaluations are mainly on toy datasets. Although appendix table 6 uses ImageNet, it will be interesting if more realistic OOD datasets are included.
3. The use of the word “unlabeled” sometimes is confusing. There are two types of labels: class label and in/out-of-distribution label. The paper should clearly say its method is doing OOD without class labels. Please consider polishing the use of these terms in sections 1 and 2.

======================
POST REBUTTAL

The updated results make sense. The limitation/assumption mentioned in weaknesses 1 must be sufficiently disclosed in section 3.2. 

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors tackle out-of-distribution detection without requiring additional detailed labels by leveraging the recent advancements in self-supervised methods combined with Mahalanobis distance metrics in feature space. They find that their approach matches or slightly outperforms supervised baselines, and widely outperforms other unsupervised approaches. They also consider a few-shot version of the task and extend their method to take advantage of supervision when available.

Pros:
Performance on outliers is a significant challenge for parametric models, so if outliers can be detected accurately this may help address potential failure modes in real-world systems. There is clear motivation for the problem.
The authors show clear gains over existing baselines.

Cons:
Their method is a pretty-straightforward combination of two established methods, contrastive self-supervised representation learning and the Mahalanobis distance as a metric of distance from a point to a distribution. 
Their method seems to rely heavily on the efficacy of contrastive self-supervised learning. It would be good to see an ablations showing the performance of the Mahalanobis distance-based approach with the same embedding features as some of the poorly-performing unsupervised baseline methods to understand how much of the gains are due to the more powerful representation method. For example, could the authors add Mahalanobis distance on top of the learned representations from the autoencoder or VAE models? I know that the authors might have to get creative here, but as it stands it is not clear to me how much of these performance gains in the unsupervised setting are due to the author’s contribution, as opposed to just due to SimCLR.


Nits:

 Naming their method SSD seems unnecessarily confusing due to the popularity of the Single-shot multibox detector (SSD) model. I would recommend rethinking the acronym for clarity.

I would prefer to see the related work discussed before the authors proposed approach, so that their method can be better placed into context with the existing literature by the reader.

“Scaling with eigenvalues remove the bias” -> Scaling with eigenvalues removes the bias

review_title: Borderline - could use additional analysis and ablations to separate improvements based on their method from improvements due to the SimCLR self-supervised representation

final_score: rating: 7: Good paper, accept
review_content: Overview: This paper proposes an outlier detection scheme based on contrastive self-supervised training for representation learning and cluster-conditioned detection using Mahalanobis distance. It shows 0.8% improvement in AUROC over the previous state-of-the-art by Winkens when trained using labeled in-distribution data. When training data is unlabeled, it performs significantly better than existing OOD detection algorithms based on AE, VAE, PixelCNN++, Deep-SVDD, and Rotation-loss and it performs only slightly worse than the labeled case. This is not surprising since I don't believe label information is crucial for OOD detection anyway, but it's good to confirm it. The authors show additional gain is possible when some OOD images are available during training, i.e., one or five images per class. This paper is very well written and I think the novelty and contributions are significant enough to merit its acceptance.

Comments: It's interesting to see the availability of only 1~5 samples per class for OOD data can result in some noticeable gains. I am not sure if covariance matrices have some scaling factors in (3), but how about using different weights in (3) for in-distribution and OOD data to take care of differences between sample sizes of in-distribution and OOD data? Would it make sense to have a higher weight for OOD data since it has fewer samples than in-distribution data?

There's a big performance gap between existing unlabeled schemes and the proposed scheme. Which one contributes most to the difference? NT-Xent? Mahalanobis distance? Cluster-conditioned detection? Some ablation studies that remove one or more of these would be interesting.

Authors say "higher eigenvalues dominates Euclidean distance but are least helpful for outlier detection", but wouldn't too small eigenvalues be less helpful for outlier detection because they are more sensitive to noise? Can authors provide an explanation on why higher eigenvalues are least helpful for outlier detection? It's strange that very small eigenvalues give very high AUROC values in Fig. 2.

In (3), why there's no minimization over m as in (2)? Is single cluster assumed for both in-distribution and OOD data in (3)?

Reference (SimCLR) missing for NT-Xent in equation (1). 

Please consider including and comparing with the following references.

[X1] Denouden, et al., Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance, https://arxiv.org/abs/1812.02765

[X2] Vernekar, et al., Out-of-distribution Detection in Classifiers via Generation, https://arxiv.org/abs/1910.04241

[X3] Goyal, et al., DROCC: Deep Robust One-Class Classification, https://arxiv.org/abs/2002.12718

review_title: Good paper

Meta-review: There is some positive consensus on this paper, which improved somewhat after
the very detailed rebuttal comments by the authors. The use of limited amounts of OOD data is interesting and novel. There were some experimental design problems, but these were well-addressed in rebuttal.

A reviewer points out that
anomaly/outlier detection does not explicitly assume that there is only one
class within the normal class (or in-distribution data). The one-class
assumption is mainly made in some popular anomaly detection methods, such as
one-class classification-based approaches for anomaly detection. The authors
should take this into careful consideration when preparing a final version of
this work.

Decision: Accept (Poster)