SSD: A Unified Framework for Self-Supervised Outlier Detection
Authors: Vikash Sehwag
Mung Chiang
Prateek Mittal
Venue: ICLR 2021 Poster (2021)
Keywords: Outlier detection
Out-of-distribution detection in deep learning
Anomaly detection with deep neural networks
Self-supervised learning

Abstract: We ask the following question: what training information is required to design an effective outlier/out-of-distribution (OOD) detector, i.e., detecting samples that lie far away from training distribution? Since unlabeled data is easily accessible for many applications, the most compelling approach is to develop detectors based on only unlabeled in-distribution data. However, we observe that most existing detectors based on unlabeled data perform poorly, often equivalent to a random prediction. In contrast, existing state-of-the-art OOD detectors achieve impressive performance but require access to fine-grained data labels for supervised training. We propose SSD, an outlier detector based on only unlabeled in-distribution data. We use self-supervised representation learning followed by a Mahalanobis distance based detection in the feature space. We demonstrate that SSD outperforms most existing detectors based on unlabeled data by a large margin. Additionally, SSD even achieves performance on par, and sometimes even better, with supervised training based detectors.  Finally, we expand our detection framework with two key extensions. First, we formulate few-shot OOD detection, in which the detector has access to only one to five samples from each class of the targeted OOD dataset. Second, we extend our framework to incorporate training data labels, if available. We find that our novel detection framework based on SSD displays enhanced performance with these extensions, and achieves state-of-the-art performance. Our code is publicly available at https://github.com/inspire-group/SSD.

Published as a conference paper at ICLR 2021
SSD: A SUPERVISED OUTLIER DETECTION
UNIFIED FRAMEWORK
FOR SELF-
Vikash Sehwag
Mung Chiang Purdue University ch angepurdue.
Prateek Mittal
Princeton University vvi kash@pri inw.rrrninntttteeeee
Princeton University pmi ttal @pr1 @princetor. incetonn.... edu
ABSTRACT
We ask the following question: what training information is required to design an effective cifiirr/uutoo-disstbbutttt (OOD) detector, i.e., detecting samples that lie far away from the training distribution? Since unlabeled data is easily accessible for many applications, the most compelling approach on only unlabeled in-distribution data. However, we observe that most existing detectors based on unlabeled data perform poorly, often equivalent a random prediction. In contrast, existing state-of oate-of-theeart OOD detectors achieve impressive performance but require access to fine- aie-ggrined data labels for supervised training- We propose SSD, outlier detector based only unlabeled in in-distribution data. We use self-supervised representation learning followed by a Mahalanobis distance based detection in the feature space. We demonstrate that SSD outperforms most existing detectors based on unlabeled data by a large margin. Additionally, SSD achieves performance training based detectors. Finally, we expand our detection framework with two key extensions. First, formulate few-shot 0OD detection, in which the detector has access to only one to five samples from each class of the targeted OOD dataset. Second, we extend our framework incorporate training data labels, if available. We find that our novel detection framework based on SSD displays enhanced performance with these extensions, and achieves state-of-te ohtte-ar--- performance"
develop detectors based
par, and sometimes even better, with supervised
INTRODUCTION
Deep neural networks are at the cornerstone of multiple safety-critical applications. ranging from autonomous driving (Ramanagopal et al., 2018) biometric authentication (Masi al. 2018 Günther et al.] (017). When trained on particular data distribution, referred to as in-distribution data, deep neural networks are known to fail against test inputs that lie far away from the training distribution, commonly referred to as outliers out-of-distributioo (OOD) samples (Grubbs) 1969 Hendrycks & Gimpel 2017). This vulnerability motivates the use of an outlier detector before feeding the input samples to the downstream neural network modules. However, key question is to understand what training information is crucial for effective outlier detection? Will the detector require fine- -raiiga annotation of training data labels process?
access
of outliers in the training
Since neither data labels nor outliers ubiquitous, the most compelling option is to design outlier detectors based on only unlabeled in-distribution data. However, we observe that most of the existing outlier detectors based on unlabeled data fail scale up to complex data modalities, such as images. For example, autoencoder (AE) (Hawkins et al. 2002) based outlier detectors have achieved success in applications such as intrusion detection (Mirsky et al.) 2018), and fraud detection (Schreyer et al.) 2017 However, this approach achieves close to chance performance on image datasets. Similarly, density modeling based methods, such as PixeICNN+ (Salimans et al. 2017) and Glow (Kingma) & Dhariwal 2018) known to assign even a higher likelihood to outliers in comparison to in- distribution data (Nalisnick al. 2019)).
In contrast, existing state-of-the-arr OOD detectors achieve high success image datasets but assume the availability of fine-grained labels for in-distribution samples (Hendrycks Gimpel| 2017
Our code is publicly available at

---

Published as a conference paper at ICLR 2021
Bendale & Boult 2016 Liang al. 2018: Dhamija al., 2018 Winkens et al. 2020). This is
a strong assumption since labels, in particular ar fine- one-grained labels, can be very costly to collect in some applications (Google AI Pricing 2020), which further motivates the use of unlabeled data. The inability of supervised detectors to use unlabeled data and poor performance of existing unsupervised approaches naturally give rise to the following question.
(OOD) data detector with access to only unlabeled data from training distribution?
Can we design an effective
framework for outlier detection with unlabeled date involves two key steps: 1) Learning a good feature representation with unsupervised training methods 2) Modeling features of in in-distribution data without requiring class labels. For example, autoencoders attempt to learn the representation with bottleneck layer, under the expectation that successful reconstruction requires learning good set of representations. Though useful for tasks such as dimensionality reduction, we find that these representations are not good enough sufficiently distinguish in-distribution data and outliers. We argue that if unsupervised training can develop a rich understanding of key semantics in in-distribution data then absence of such semanties in outliers can cause them lie far away in the feature space, thus making it easy to detect them. Recently, self-supervised representation learning methods have made large progress, commonly measured by accuracy achieved on a downstream classification task (Chen) et al. 2020 He et al. 20200 Oord et al. 2018: Misra & Maaten 2020 Tian et al.) (020). We leverage these representations
our proposed cluster-conditioned framework based on the Mahalanobis distance (Mahalanobis [936). Our key result is that self-supervised representations are highly effective for the task of outlier detection in our self-supervised outlier detection (SSD) framework where they not only perform far better than most of the previous unsupervised representation learning methods but also perform on par, and sometimes even better, than supervised representations.
What if access to fraction of OOD data or training data labels is available? How do move past a detector based on unlabeled data and design a framework which can take advantage of such information? Though access to outliers during training is strong assumption, it may be feasible to obtain a few prior instances of such outliers (Görnitz et al.) 2013). We characterize this setting as few-shot OOD detection, where we assume access to very few, often one five, samples from the targeted set of outliers. While earlier approaches (Liang et al. 2018: Lee et al. 2018b) mostly use such data to calibrate the detector, we find that access to just few outliers can bring an additional boost in the performance of our detector. Crucial to this success is the reliable estimation of first and second order statisties of OOD data the high dimensional feature space with just few samples.
Finally, if class labels are available in the training phase, how can we incorporate them in the SSD) framework for outlier detection? Recent works have proposed the addition of the supervised cross- entropy and self-supervised learning loss with tunable parameter, which may require tuning for optimal parameter setting for each dataset (Hendrycks et al. 2019b; Winkens et al. 2020). We demonstrate that incorporating labels directly in the contrastive loss achieves 1) tuning parameter- free detector, and 2) srute-of-the--ae performance.
1.1 KEY CONTRIBUTIONS
SSD for unlabeled data. We propose SSD, an unsupervised framework for outlier detection based on unlabeled in-distribution data. We demonstrate that SSD outperforms most existing unsupervised outlier detectors by large margin while also performing par, and sometimes even better than supervised training based detection methods. We validate our observation across four different datasets: CIFAR-10, 100, STL-10, and ImageNet.
Extensions of SSD. We provide two extensions SSD to further improve its performance. First, we formulate few-shot OOD detection and propose detection methods which can achieve a significantly large gain in performance with access to only a few targeted OOD samples. Next, we extend SSD, without using any tuning parameter, to also incorporate in-distribution data labels and achieve state-of-the-ar performance.
We refer OOD detection without using class labels of in-distrihution data as unsupervised OOD detection.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
OOD detection with unsupervised defectors. Interest in unsupervised outlier detection goes back to Grubbs (1969). We categorize these approaches in three groups 1) Reconstruction -error error based detection using Auto-encoders (Hawkins et al., 2002] / Mirsky et al. 2018: Schreyer et al. 2017 7) or
Variational auto-encoders (Abati et al. [019 An & Cho 2015) 2) Classification based, such as Deep-
SVDD (Ruff ct al. 2018 EI- Yaniv & Wiener 2010 Geifman El- Yaniv 2017) and 3) Probabilistic
detectors, such as density models like Glow and PixcICNN+ (Ren et al. 2019 Nalisnick et al. 2019
Salimans et al. 2017 Kingma& & Dhariwal 2018). We compare with detectors from each category and find that SSD outperforms them by wide margin.
OOD detection with supervised learning. Supervised detectors have been most successful with complex input modalities, such as images and language (Chalapathy et al. 2018a DeVrss & Taylor
2018; Dhamija et al., 2018: Jiang et al. 2018) Yoshihashi et al. 2018: Lee et al. (2188)). Most of
these approaches model features of in-distribution data at output (Liang et al. (008 [onndrreeyyycs & Gimpel, 2017 Dhamija et al. 2018) or in the feature space (Lee et al. 2018b] Winkens et al. 2020) for detection. We show that SSD can achieve performance on par with these supervised detectors, without using data labels. A subset of these detectors also leverages generic OOD data to boost performance (Hendrycks al. Mohseii
Mohseni et al., 2020).
Access to OOD data at training time. Some recent detectors also require OOD samples for hyperparameter tuning (Liang et al. 2018: Lee et al.
20T Zisselman & Tamar 2020). We extend SSD to this setting but assume access to only few OOD samples, referred to as few-shot OOD detection, which our framework can efficiently utilize to bring further gains in performance.
In conjunction with supervised training. Vyas et al. (2018) uses ensemble of leave-one-out classifier, Winkens et al. (2020) uses contrastive self-supervised training, and Hendrycks et al.) (2019b) uses rotation based supervipe loss, in conjunction with supervised cross- cotsopy loss to achieve sate-of-the-ar performance in OOD detection. Here we extend SSD, to incorporate data labels, when available, and achieve better performance than existing state-of-the-art
Anomaly detection. In parallel to OOD detection, this research direction focuses on the detection of semantically related anomalies in applications such intrusion detection, spam detection, disease detection, image classification, and video surveillance. We refer the interested reader to Pang al. (2020) for detailed review. While large number of works focuses on developing methods particularly for single-class modeling in anomaly detection (Perera et al., 2019, Schheel et al. 2017 Ruff et 2018 Chalapathy et al.] 2018b Golan El- Yaniv] [2018] Wang al. [2019), ) some recent work achieve success in both OOD detection and anomaly detection Tack et al. (2020); |ergman
Hoshen (200)). We provide a detailed comparison of our approach with previous work in both categories.
SSD: SELF-SUPERVISED OUTLLIER/OUTOOF--ISTTRIIEE DETECTION
In this section, we first provide the necessary background on iiiicrrouut-o-diissbutttooo (OOD) detection and then present the underlying formulation of our self-supervised detector (SSD) that relies on only unlabeled in-distribution data. Finally, we describe two extensions of SSD to (optionally) incorporate targeted OOD samples and in-distribution data labels (if available).
Notation. We represent the input space by X' and corresponding label space as y. We assume in-distribution data is sampled from Pinn Pixy- In the absence of data labels, is sampled from marginal distribution Pir We sample out-of-distribution data from pood We denote the feature extractor by f X - Z, where d-dimensional feature space (Z). The feature extractor is often parameterized by a deep neural network. In supervised learning, we obtain classification confidence for each class by go X' - IR.. In most cases, g is parameterized by shallow neural network, generally linear classifier.
Rdd, a function which maps sample from the input space to the
wttpr//ttttoo-dssibuuuuii (OOD) detection. Given a collection of sam- ples from Pin x Pood the objective is to correctly identify the source distribution, i.c., Pir or Pood for each sample. We use the term supervised OOD detectors for detectors which use in-distribution data labels, i.e., train the neural network (g f) pén X'xy using supervised training techniques.
Problem Formulation:
3

---

Published as a conference paper at ICLR 2021
Unsupervised OOD detectors aim to solve the aforementioned OOD detection tasks, with access to only Px In this work, we focus on developing effective unsupervised OOD detectors.
Background:
Contrastive self-supervised representation learning. Given unlabeled training data, aims to train a feature extractor, by discriminating between individual instances from data, to learn good set of representations. Using image transformations, first creates views of each image, commonly referred to as positives. Next, it optimizes to pull each instance close to its positive instances while pushing away from other images, commonly referred as negatives. Assuming that (z)) are positive pairs for the ith image from a batch of N images and h(.)) is projection header, is the temperature, contrastive training minimizes the following loss, referred to Normalized cemperature-scale cross-entropy (NT-Xent), over each batch.
2N
E us/T
(f(x))
Lbutch
2.V
log
Ex 1(k + ua/7
U [h(f(z)))ll
(1)
3.1 UNSUPERVISED OUTLIER DETECTION WITH SSD
Leveraging contrastive self-supervised training. In the absence of data labels, SSD consists of two steps: 1) Training feature extractor using unsupervised representation learning, 2) Devel- oping an effective OOD detector based on hidden features which isn't 't conditioned on data labels. We leverage contrastive self-supervisee training for rep- resentation learning in our outlier detection framework, particularly due to its otate-of-the-art performance (Chen et al., 2020; Tian et al., 2020). We will discuss the ef-
65
60
feet of different representation learning methods later in
AUROC
Section 4.2
50
Cluster-conditooned detection. In absence of data la- bels, we develop a cluster-conditions detection method in the feature space. We first partition the features for in-distribution training data in clusters. We represent features for each cluster as Z... We use k-means clustering method, due to its effectiveness and low computation cost. Next, we model features in each cluster independently, and calculate the following outlier score (sg) min D(z, Z...)
100 15 20
Figenvaluess
Figure I: AUROC along individual prin- ciple eigenvector with CIFAR-IO as in- distribution and CIFAR-100 as OOD. Higher eigenvalues dominates euclidean distance, but are least helpful for outlier detection. Mahalnobis distance avoid this bias with appropriate scaling and performs much better.
m
for each test input x, where -)...) is distance metric in the feature space. We discuss the choice of the number of clusters in Section 4.2
Choice of distance metric: Mahalanobis distance. We Mahalanobis distance calculate the outlier score follows:
min( Z.r (mm) Em= (Zx - (m))
7/
where pm and En are the sample mean and sample covariance of features ( (2) of the in-distribution training data. We justify this choice with quantitative results in Figure [ With eigendecomposition of
sample covariance (Emm Am 82 min (QT pmm) A (Q -mm)) which
is equivalent to euclidean distance scaled with eigenvalues in the eigenspace. We discriminate between in-distribution (CIFAR-10) and OOD (CIFAR- data along each principal eigenvector (using AUROC, higher the better). With euclidean distance, i.c.. in absence of scaling, components with higher cigenvalues have more weight but provide least discrimination. Scaling with eigenvalues removes the bias, making Mahalnobis distance effective for outlier detection in the feature space.
3.2 FEW-SHOT OOD DETECTION (SSD;)
In this extension of the SSD framework, we consider the scenario where a few samples from the OOD dataset used at inference time also available at the time of training. We focus one-shot and five-shot detection, which refers to access to only one and fives samples, from each class of the targeted OOD dataset, respectively. Our hypothesis is that in-distribution samples and OOD samples
4

---

Published as a conference paper at ICLR 2021
will be closer to other imputs from their respective distribution in the feature space, while lying further away from each other. We incorporate this hypothesis by using following formulation outlier score.
(Zx -Mm)n 20n))"2,m (Zx (in) (Ze Mood)
"ond
(Mood)
where Pín Ein and Mood: Lood are the sample mean and sample covariance in the feature space for in-distribution and OOD data, respectively.
Challenge. The key challenge few samples. Sample covariance is not an accurate estimator of covariance when the number of samples is less than the dimension of feature space (Stein 1975), which is often in the order of thousands for deep neural networks.
reliably estimate the statistics for OOD data, with access to only
Shrunk covariance estimators and data augmentation. We overcome this challenge by using following two techniques: 1) we shrunk covariance estimators (eedoit & Wolf 2004), and 2) amplify number of OOD samples using data augmentation. We use shrunk covariance estimators due to their ability to estimate covariance better than sample covariance, especially when the number of samples is even less than the feature dimension. To further improve the estimation we amplify the number of samples using data augmentation at the input stage. We common image transformations, such as geometric and photometric changes to create multiple different images from a single source image from the OOD dataset. Thus given set of OOD samples {ui, U2, Uk we first create set of & x n samples using data augmentation, 2/ = fuu..... "i, Uk:...... }. Using this set, we calculate the outlier score for test sample in the following manner.
(3x Min) 2i (3x
pin) (zx mv) Sy (Zx 1 mu)
where ju and Sy are the sample mean and estimated covariance using shrunk covariance estimators for the set U, respectively.
3.3 How TO BEST USE DATA LABELS (SSD+)
If fine-grained labels for in-distribution data are available, immediate question is how to incorporate them in training to improve the success in detecting OOD samples.
Conventional approach: Additive self-supervised and supervised training loss. A common theme in recent works (Hendrycks et al. 2019b Winkens et al. 2020) to add self-supervised (Leat) and supervised (Leup) training loss functions. i.c., Ltraining Leup aLasi where the hyperparameter a is chosen for best performance on OOD detection. A common loss function for supervised training is cross-entropy.
Our approach: Incorporating labels in contrastive self-supervised training. As we show in Section 4.2 even without labels, contrasting between instances using self-supervised learning is highly successful for outlier detection. We argue for a similar instance-based contrastive training, where labels can also be incorporated to further improve the learned representations. To this end, we use the recently proposed supervised contrastive training loss function (Khosla et al. 2020)). which uses labels for more effective selection of positive and negative instances for each image. In particular, we minimize the following loss function.
2N
1 + 1)T = vi) SeeTTunn un/T
1
2N
Dbatch
log
(5)
2N
Σ1 == 1 + i)eu? un/T
2N
where Ny, refers number of images with label y; in the batch. In comparison to contrastive NT-Xent loss (Equation []. now we use images with identical labels in each batch as positives. We will show the superior performance of this approach compared to earlier approaches, and note that it is also a parameter-fi -fre approach which doesn't 't require additional OOD data to tune parameters. the proposed cluster-conditioned framework with Mahalnobis distance, results in better performance than using data labels. We further summarize our framework in A Algorithm
We further
find
5

---

Published as a conference paper at ICLR 2021
Algorithm I: Self-supervised outlier detection framework (SSD)
Input :Kin, Neest, feature extractor (f), projection head (h), Required True-positive rate (7),
Optional: Xond: Vin # Xin E X Nood E IPoed
Output: outlier or not? Vx E Keest
Function getFeatures( )):; return (f(=)//ll
Function SSDScore( 2, M4, Σ): return {(2 1= k) 2};
Function SSDkScore(2 2, pins Lin: Hoods Lood):
return {(z [4n) )mm)'" "n (z pin)
(ood)
2ood( "oodd - Mood), V 2 E 3 }:
end
Parition Xin in training set (Atrain) and calibration set (Reat); if Vnn is not available then
Caarch
2N Σ.N -log
of ug/T Ets 3(12a)e og/7""
h(f(ee)) 18(f(=2))
; TTrain feature extractor
else
(1 "Tup// Tua/t
Coetch
2N EiN -log
Ex T(Kgas)) MA/T
end
Train feature extractor (f) by minimizing Lbatch over Xtrain;
Ztrain = get Features FetterrressArnai Zeal getFeatures(Aall
Ztest = getFeatures (Meett) ) if Xood is available: Zood = get
if X is not available then
Spal = SSDSccre (Zett, M Etrain); # Sample mean and convariance of Ztrain
Stest SSDScore Ztests Mtrains Etraia) # outlier score;
else
# Using convariance estimation techniques from Section 3.2 for 2ood
8aa = SSDk B Mtrain; Earainn (Mods Soodd):
Stest = SookScore Ziests Ptroin; Etrains Pood: Good);
end
I; E Meett is an outlier if iest > (Seal threshold at TPR = T);
EXPERIMENTAL RESULTS
4.1 COMMON SETUP ACROSS ALL EXPERIMENTS
We use recently proposed NT-Xent loss function from SimCLR (Chen et al. 2020 method for self-supervised training. We use the ResNet-50 network in all key experiments but also provide ablation with ResNet-18, ResNet-34, and ResNet-100 101 network architecture. We train each network, for both supervised and self-supervised training, with stochastic gradient descent for 500 epochs, 0.5 starting learning rate with cosine decay, and weight decay and batch size set to le-4 and 512, respectively. We the temperature parameter 0.5 in the NT-Xent loss. We evaluate our detector with three performance metrics, namely FPR (at TPR=95%), AUROC, and AUPR. For the supervised training baseline, we use identical training budget as SSD while also using the Mahalanobis distance based detection in the feature space. Due to space constraints, we present results with AUROC, which refers area under the receiver operating characteristic curve, in the main paper and provide detailed results with other performance metrics in Appendix B.5 Our setup incorporates six image datasets along with additional synthetic datasets based on random noise. We report the average results over three independent runs in most experiments.
Number of clusters. We find the choice the number of clusters dependent on which layer we extract the features from in the Residual neural networks. While for the first three blocks, we find an increase in AUROC with the number of clusters, the trend is reversed for the last block (Appendix (B.2)) Since the last block features achieve the highest detection performance, model the in-distribution features as a single cluster in subsequent experiments.
4.2 PERFORMANCE OF SSD
Comparison with unsupervised learning based detectors. We present this comparison in Table We find that SSD improves average AUROC by up to 55, compared to standard outlier detectors

---

Published as a conference paper at ICLR 2021
Table 1: Comparison of SSD with different outlier detectors using only unlabeled training data.
In-distribution
CIFAR-10 (SVHN)
CIFAR-I0 (CIFAR 100)
CIFAR-100 (SVHN)
CIFAR-I0D 100 (CIFAR
Averuge
(Out of distribution)
Autoencoder Hawkkis et al.
2002
51.3
3.0
51.4
27.0
VAE (Kingmu & Welling 2014
52.8
2.6
47.1
26.2
PixeICNN+ Salmmas cl al
15.8
52.4
Deep-SVDD
Rult ct
14.5
52.1
16.3
51.4
33,5
Rotation-loss
(Giaars
2018
97.9
81.2
94.4
50.1
80.9
CSI Tack et
2020
99.8
89.2
SSD
99.6
90.6
94,9
69.6
88.7
SSD (A 5)
99.7
93.1
99.1
78.2
92.5
based on Density modeling (PixelCNN+ (Salimans et al.) 2017)). input reconstruction (Auto- encoder (Hawkins et al. 2002), Variational Auto-encoder [Kingma & Welling 2014)), and One-class classification (Deep-SV VVDD Ruff et al. (2018). A common limitation of each these three detectors is to find images from the SVHN dataset as more in-distribution when trained on CIFAR-10 or CIFAR- dataset. In contrast, SSD able to successfully detect a large fraction of outliers from SVHN dataset. We also experiment with Rotation-loss Gidaris et al. (2018), non-contrastive self-supervised training objective. We find that SSD with contrastive NT-Xent loss achieves 9.6% higher average AUROC compared to using Rotation-loss.
Ablation studies. We ablate along individual parameters self-supervised training with CIFAR-10 as in-distribution data (Figure 2). While architecture doesn't 't have a very large effect AUROC for most OOD dataset, we find that the number of training epochs and batch size plays a key role in detecting outliers from the CIFAR-100 100 dataset, which is hardest to detect among the four OOD datasets. We also find an increase in the size of training dataset helpful in the detection of all four OOD datasets.
251
0004
WiH
Techure
CRAR 122 V+H
Teaturs
000 delinet
No webrrllaayynr
Model size.
(b) Number of epochs.
(c) Batch size.
(d) Size of training
Figure 2: Ablating across different training parameters in SSD under following setup: In-distribution
dataset = CIFAR-10, OOD dataset = CIFAR-100, Training epochs = 500, Batch size 512.
Comparison with supervised representations. We earlier asked the question whether data labels are even necessary to learn representations crucial for OOD detection? To answer it, we compare SSD with supervised network, trained with an identical budget as SSD while also using Mahalanobis distance in the feature space, across sixteen different pairs of in in-distribution and out-of-distribution datasets (Table 2 We observe that self-supervised representations even achieve better performance than supervised representations for 56% of the tasks in Table 2
Success in anomaly detection. We now measure the performance of SSD in anomaly detection where we consider one of the CIFAR- 10 classes as in-distribution and the rest of the classes as a source of anomalies. Similar the earlier setup, train ResNet-50 network using self-supervised training with NTt-Xent loss function. While the contrastive loss attempt to separate individual instances in the feature space, find that adding an l2 regularization in the feature space helps in improving performance. In particular, we add this regularization (with scaling coefficient of 0.01) to bring individual instance features close the mean of all feature vectors the batch. Additionally, we reduce the temperature from 0.5 to 0.1 to reduce the separability of individual instances due to the contrastive loss. Overall, we find that our approach outperforms all previous works and achieves competitive performance with the concurrent work of Tack et al. (2020) (Table 3).
4.3 FEW-SHOT OOD DETECTION (SSD,)
Setup. We focus one-shot and five-shot OOD detection, i.e., set l to or 5 in Equation and use Ledoit-Woll (Ledoit Wolf 2004) estimator for covariance estimation. To avoid a bias on selected samples, report average results over 25 random trials.
7

---

Published as a conference paper at ICLR 2021
Table Comparing performance of self-supervised (SSD) and supervised representations. We also provide results for few-shot OOD detection (SSD)) for comparison with our baseline SSD detector.
distribution
In-
OOD
Supervised
SSDs
k-5
distriburion
In-
O0I
Superivsed
SSD
CIFAR-100
90.6
90.6
91.7
93.0
CIFAR-100
94.8
$4.0
90.1
90.0
SVHN
99.6
99.6
99.9
99.7
STL-10
SVHN
98.7
95.7
98.77
99.4
CIFAR-10
75.5
85.7
Blobs
98.8
99.9
99.7
100.0
Blobs
96.4
88.6
96.5
99.9
LSUN
96.5
93.8
97.5
97.8
LSUN
88.5
66.8
94,1
91.5
Places365
95.2
92.7
96.7
97.3
Placo3655
88.3
64.9
95.4
95.6
CIFAR-10
78.3
SVHN
99.1
CIFAR-100
Textune
SVHN
94.9
82.9
94.5
988,8
96.8
99.5
99.1
94.2
ImageNet
Texture
Blobs
99.5
95.4
85.1
98.4
100.0
94.7
100.0
97.3
Blahs
98.1
57.3
98.1
100.0
Gaussian Noise
100.0
100.0
100.0
100.0
LSUU
79.5
69.4
92.3
93.4
ImageNet-OO
45.2
75.5
89.4
93.3
Places365
79.6
62.6
90.7
92.7
Table 3: Comparison of SSD with other detectors for anomaly detection task on CIFAR- 10 dataset.
Aipllane
Automobile
Bird
Cat
Deer
Dog
Frog
Homse
Shap
Truck
Average
Randomly Initialized network
44.1
62.4
44.1
59.8
73.8
53.7
57.5
VAE Kingma & Welling 2014
70.0
38.6
67.9
53.5
74.8
52.3
68.7
49.3
69.6
38.6
58.3
OCSV'T
44.0
64.9
48.7
64.9
50.8
58.5
AnoGAN StBlegl at al
67.1
54.7
52.9
$1.5
65.1
50.3
58.5
62.5
75.8
66.5
61.8
PixiCNN den Lini
16
53.1
99.5
47.6
51.7
73.9
54.2
59.2
78.9
34.0
66.2
61.8
DSVID
65.9
50.3
59.1
73.1
OCGAN (Verre e di
2019
75.7
53.1
64.0
62.0
72.3
62.0
72.3
57.5
$2.0
55.4
65.6
RCAE
72.0
63.1
71.7
60.5
72.8
64.0
64.9
63.6
74.7
74.5
68.2
DROCU (Cooal al MBA0
81.7
76.7
66.7
67.1
73.6
74.4
74.4
71.4
80.0
76.2
74,2
Deep SAD Kalr 2019
77.9
E3Outlerr
Wang 2019
GT Giolan Yrrm
2011
74.7
95.7
78.1
72.4
87.8
87.8
83.4
95.5
91.3
91.3
86.0
levAL
[haang al. 2019
78.5
89.8
86.1
77.4
90.5
84.5
89.2
92.0
85.5
86.5
GOAD Beteman Toshen
A120
77.2
96.7
83.3
77.7
87.8
87.8
90.0
96.
93.8
92.0
88.2
CST
1ack al 2020
89.9
99.9
93.1
85.4
93.9
93.2
98.1
98.7
97.9
95.5
94.3
98.5
84.2
91.7
94.4
90.0
Results. Compared to the baseline SSD detector, one-shot and five-shot settings improve the average AUROC, across all OOD datasets, by 1.6 and 2.1, respectively (Table In particular, we observe large gains with CIFAR- 100 as in-distribution and CIFAR-10 as OOD where five-shot detection improves the AUROC from 69.6 78.3. We find the use of shrunk covariance estimator most critical in the success of our approach. Use of shrunk covariance estimation itself improves the AUROC from 69.6 to 77.1. Then data augmentation further improves it 78.3 for the five-shot detection. With an increasing number of transformed copies of each sample, also observe improvement in AUROC, though it later plateaus close to ten copies (Appendix(B.2) B.3).
What if additional OOD images are available Note that some earlier works, such as Liang et al. 20188). assume that 1000 OOD inputs are available for tuning the detector. We find that with access to this large amount of OOD samples, SSD; can improve the state-of-the-art by larger margin. For example, with CIFAR-100 as in-distribution and CIFAR-10 10 as out-of-distribution, it achieves 89.4 AUROC, which is 14.2% higher than the current state-of-the-ate (Winkens al. 2020
4.4 SUCCESS WHEN USING DATA LABELS (SSD+)
Now integrate labels of training data in framework and compare it with the existing of the-art detectors. We report our results in Table 4 Our approach improves the average AUROC by 0.8 over the previous state-of-the-ar detector. Our approach also achieves equal better performance than previous state-of-the-an across individual pairs of in and out-distribution dataset. For example, using labels in our framework improves the AUROC of Mahalanobis detector from 55.5 72.1 for CIFAR-100 as in-distribution and CIFAR-10 as the OOD dataset. Using the simple softmax probabilities, training a two-layer MLP on learned representations further improves the AUROC to 78.3. Combining SSD+ with a five-shot OOD detection method further brings gain of 1 1.4 in the average AUROC.
5 DISCUSSION AND CONCLUSION
On tuning hy hyperparameters in SSD. In our framework, we either explicitly avoid the use of additional maramn-ssrameters (such as when combining self-supervised and supervised loss functions
8

---

Published as a conference paper at ICLR 2021
Table 4: Comparison of SSD+, i.c., incorporating labels in the SSD detector, with state-of-the-ar detectors based on supervised training.
Io-distribution
CIFAR-I0 (CIFAR-10))
CIFAR-I0 (SVHN)
CIFAR-100 (CIFAR-I0)
CIFAR-100 (SVHN)
)itt-of-iistrhhttoon)
Averago
Softmax-probs (Heudrycks & Gimpel 2017
89.8
95.9
78.0
78.9
85.6
(DINLLian 21120188
20188
89.6
96.4
77.9
60.9
81.2
Mahalnobis el 2018b]
99.4
55.3
Residual Flowa Ziscelmnn & Tamar 2020
89.4
99.1
77.1
97.5
90.7
Gram Matrix (Sistr Oire 2019
79.0
99.5
67.9
96.0
85.6
Outier exposurc yold
201%
75.7
88.6
Rotation-lsss Supervised Hendryyks di 2019h
90.9
98.9
Contrastive + Supervised Winkens et al. 2020
92.9
99.5
78.3
95.6
91.6
CS1 (Tack 2020
SSD
93.4
99.9
78.3
98.2
92.4
SS02,+ (R S)
94.1
99.6
84.1
97.4
93.8
Uses wider ResNet-50 network, Requires additional out-of-distributin data tuning.
in SSD+) or refrain from tuning the existing set of parameters for each OOD dataset. For example, we use standard set of parameters for sulf-supervised training and model the learned features with a single-cluster.
Why contrastive self-supervised learning is effective in the SSD framework? We focus on the NT-Xent loss function, which is parameterized by a temperature variable (r). Its objective is to pull positive instances, i.e., different transformations of an image, together while pushing away from other instances. Earlier works have shown that such contrastive training forces the network to learn a good set of feature representations. However, a smaller value of temperature quickly saturates the loss, discouraging it to further improve the feature representations. We find that the performance of SSD also degrades with lower temperature, suggesting the necessity of learning a good set of feature representation for effective outlier detection (Table
Table 5: Test Accuracy and AUROC with different temperature values in NT: Xent (Equation I loss. Using CIFAR- 10 as in-distribution and CIFAR- as OOD dataset with ResNet18 network.
Temperature
0.001
0.01
0.1
0.5
Test AAccrrcy
70.8
76.7
86.9
90.5
AUROC
66.7
71.6 85.5
89.5
How discriminative ability of feature representations evolves over the course of training. We analyze this effect in Figure where we compare both SSD) and supervised training based detector over the course of training. While discriminative ability of self- supervied training in SSD is lower at the start, it quickly catches up with supervised representations after half of the training epochs.
Performance of SSD improves with the amount of available unlabeled data. A compelling advantage of un- supervised learning is to learn from unlabeled data, which
90.0
Supervised
can be casily collected. As presented in Figure 2 we find that performance of SSD) increases with the size of training dataset. We conduct another experiment with the STL-10
AUROC
85.0
82.5
dataset, where in addition to the 5k training images, we also use additional 10k images from the unlabeled set. This improves the AUROC from 94.7 to 99.4 for CIFAR- 100 as the OOD dataset, further demonstrating the success of SSD in leveraging unlabeled data (Appendix B.4). In conclusion, our framework provides an effective & flexible approach for outlier detection using unlabeled data.
80.0
100
200 300
400
500
Epochs in course of training
Figure 3: AUROC over the course of training with CIFAR-10 as in- distribution and CIFAR-100 OOD set.
Acknowledgments We would like to thank Chong Xiang, in part by the National Science Foundation under grants CNS-1553437 and CNS-170410S, by Qualcomm Innovation Fellowship, by the Army Research Office Young Investigator Prize, by Army Research Laboratory (ARL) Army Artificial Intelligence Institute (A212), by Office of Naval Research (ONR) Young Investigator Award, by Facebook Systems for ML award, by Schmidt DataX Fund, and by Princeton E-ffiliates Partnership.
Liwei Song, and Arjun Nitin Bhagoji for their helpful feedback on the paper. This work was supported
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 481-490, 2019.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2:1-18, 2015.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Liron Bergman and Yedid Hoshen. Classification based anomaly detection for general data. In International Conference Learning Representations, 2020. URL wttpt::tt//ttt..c... http:ttss//pppereviiiiee
nert/ 2im 1d==1 1K 1_
Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:/802.00 1802.06360, 2018a.
Raghavendra Chalapathy, Aditya Krishma Menon, and Sanjay Chawla. Anomaly detection using one-class neural networks. arXiv preprint 3822v:8802.063000 2018b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215-223, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255. leee, 2009.
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv: 04865, 2018.
Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. In Advances in Neural Information Processing Systems, pp. 9175-9186, 2018.
Ran El- Yaniv and Yair Wiener. On the foundations noise-free selective classification. Journal of Machine Learning Research, ([MMa)):105516441 2010.
Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems, pp. 4878 4887, 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL
net/ htt.:p/aaweereeeeeeoooooooooaaaa wtt.:/ommmmmmma22m11
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In Advances in Neural Informaiion Processing Systems, pp. 9758 9769, 2018.
API Google AI Pricing. Data Labelling Pricing Google Al Platform, 2020. URL https: 7/cloud. goog le. com/ wttpeeecmmmmmaaaaaaaoooooo form http:///pppatttoo//tttttaaaaaaaa wttp//aaaaaaaaaaaaaaaaaaaaeaeeee
Nico Görnitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection.
Journal of Artificial Intelligence Research, 46:235-262,, 2013.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateck Jain. Droce:
Deep robust one-class classification. In International Conference on Machine Learning, 2020.
10

---

Published as a conference paper at ICLR 2021
Frank E Grubbs. Procedures for detecting outlying observations in samples. Technometrics, 11(1): 1-21, 1969.
Manuel Günther, Steve Cruz, Ethan M Rudd, and Terrance E Boult. Toward open-set face recognition. In Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. IEEE, 2017.
Simon Hawkins, Hongxing He, Graham Williams, and Rohan Baxter. Outlier detection using replicator neural networks. In International Conference on Data Warehousing and Knowledge Discovery, pp. 170-180. Springer, 2002.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019a. URL https
/oopenreieww net/ forum? (eeerummmiddxxyyxxhxxxxxe
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances Neural Information Processing Systems, pp. 15663-15674, 2019b.
Chaoqing Huang, Jinkun Cao, Fei Ye, Maosen Li, Ya Zhang, and Cewu Lu. Inverse-transform autoencoder for anomaly detection. arXiv preprint 19Xi::191..16777 2019.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In Advances in Neural Information Processing Systems, pp. 5546-5551 2018.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, 2020.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and
Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014 Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
7/arxiv. otg/ 1aabs/131 1312.1112 6114
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in neural information processing systems, pp.
2018.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Olivier Ledoit and Michael Wolf. Honey, i shrunk the sample covariance matrix. The Journal of Portfolio Management, 30(4):110- 119, 2004.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrate classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributiio samples and adversarial attacks. In Advances in Neural Information Processing Systems, pp. 7167-7177, 2018b.
Shiyu Liang, Yixuan Li. and R. Srikant. Enhancing the reliability of out-of-distribbtion image detection in neural networks. In International Conference on Learning Representations, 2018. URL htR..htttps://ppnneeee http:/ewwww...eeeeeeeeeeeeeeeeee nwt//forumti V
Prasanta Chandra Mahalanobis, On the generalized distance in statistics. National Institute Science of India, 1936.
11

---

Published as a conference paper at ICLR 2021
lacopo Masi, Yue Wu, Tal Hassner, and Prem Natarajan. Deep face recognition: A survey. In 2018 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), pp. 471-478. IEEE, 2018.
Yisroel Mirsky, Tomer Doitshman, Yuval Elovici, and Asaf Shabtai. Kitsune: an ensemble of autoencoders for online network intrusion detection. arXiv preprint arXiv: 180.00988000089 2018.
Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6707-6717 17, 2020.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. In AAAI, pp. 5216-5223, 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In International Conference on Learning Representations, 2019. URL
net/ fore:/forum?iddii xwNhCcYm.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 2011.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 2018.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton van den Hengel. Deep learning for anomaly detection: review. arXiv preprint XXiv:2007.02500 2020.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Oegan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2898-2906, 2019.
Manikandasriram Srinivasan Ramanagopal, Cyrus Anderson, Ram Vasudevan, and Matthew Johnson- Roberson. Failing to learn: autonomously identifying perception failures for self-driving cars. IEEE Robotics and Automation Letters, (80))3870---67,00
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-o--distributtin detection. In Advances in Neural Information Processing Systems, pp. 14707-14718, 2019.
Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Lucas Deecke, Shoaib A. Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 4393-4402, 2018.
Lukas Ruff, Robert A Vandermeulen. Nico Gömniz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. International Conference on Learning Representations, 2019.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In International Conference Learning Representations, 2017.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distriibttion examples with in- distribution examples and gram matrices. arXiv preprint 19X.v:: 2019.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurie and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, In International conference on information processing in medical imaging, pp. 146-157. Springer, 2017.
Bernhard Schölkopf, John C Platt, John Shawe-Taylee Alex J Smola, and Robert C Williamson. Estimating the support of nigh-dimensional distribution. Neural computation, 13(7):1443-147) 2001.
12

---

Published as a conference paper at ICLR 2021
Marco Schreyer. Timur Sattarov, Damian Borth, Andreas Dengel, and Bernd Reimer. Detection of anomalies in large scale accounting data using deep autoencoder networks. arXiv preprint arXiv:1709.05254, 2017.
C. Stein. Estimation of a covariance matrix. 39th Annual Meeting IMS, Atlanta, GA, 1975, 1975. URL http::/::/cc...... .ni..ar.. ac jp/na winaai///ia 002001111111 852
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in Neural Information Processing Systems, 33, 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint vrXiv:2005.00244 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems, pp. 4790 4798, 2016.
Apoory Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke. distr--disttitution detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 550-564, 2018.
Siqi Wang, Yijie Zeng, Xinwang Liu, En Zhu, Jianping Yin, Chuanfu Xu, and Marius Kloft, Effective end-to-end unsupervised outlier detection via inlier priority of discriminative network. In Advances in Neural Information Processing Systems, pp. 5962-5975 2019.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribttion detection. arXiv preprini arXiv:2007.05500 2020.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. Clasiffcation-rcconstuuiiii learning for open-set recognition. arXiv preprint arXiv:1812.04246, 2018.
Ev Zisselman and Aviv Tamar. Deep residual flow for novelty detection. arXiv preprint arXiv:2001.05419, 2020.
A ADDITIONAL DETAILS ON EXPERIMENTAL SETUP
A.1 TRAINING AND EVALUATION SETUP FOR DEEP NEURAL NETWORKS.
We use ResNet-50 architecture for our major experiments and ResNet-18 for ablation studies. We also provide results with ResNet-34 and ResNet-101 architecture. We use two-layer fully connected network as the projection header (h(.)). To contrast with a large number of negatives, NT-Xent loss requires much larger batch size compared to the supervised cross-entropy loss function. We train it using a batch size of 512. When evaluating self-supervised models, even when we incorporate labels in SSD, we achieve the best performance when modeling in-distribution features with only a single cluster. However, for supervised training, which refers to the supervised baseline in the paper, we find that increasing the number of clusters helps. For it, we report the best of the results obtained from cluster indexes or using true labels of the data. For each dataset, we use the test set partition, if it exists, as the OOD dataset. For consistent comparison, we re-implement & Gimpel 2017). ODIN (Liang et al. 2018), and Mahalanobis detector (Lee et al.) 2018b) and evaluate their performance on the identical network, trained with supervised training for 500 epochs. We set the perturbation budget to 0.0014 and temperature to 1000 for ODIN, since these are the most successful set of parameters reported in the original paper (Liang et al., 20188). We primarily focus on one-shot and five-shot OOD detection, i.e., set & to one or five. It implies access to one and five images, respectively, from each class of the targeted OOD dataset. We create ten randomly transformed samples from each available OOD image in the SSD; detector. With very small k. such as one, we find that increasing number of transformations may degrade performance in some cases. In this case we simply resort to using only transformation per sample. To avoid any
deeimax--robbaillee
(Hendrycks)
13

---

Published as a conference paper at ICLR 2021
hyperparameter selection, set the image augmentation pipeline be the as the one used in training. Finally, we use the Ledoit- Wolf method (Ledoit the OOD samples.
Wolf 2004) to estimate the covariance of
A.2 PERFORMANCE METRICS FOR OUTLIER DETECTORS
We use the following three performance metrics to evaluate the performance of outlier detectors.
FPR at TPR=95%. It refers to the false positive rate (= FP (P/(FP+TN), when true positive rate (= / /TP FN)) is equal to 95%. Effectively, its goal is measure what fraction of outliers go undetected when it is desirable to have true positive rate of 95%.
AUROC. It refers to the area under the receiver operating characteristic curve. We measure it by calculating the area under the curve when we plot TPR against FPR.
AUPR. refers to the area under the precision-recall curve, where precision = TP (TP+FP) and recall = TP (TP+FN). Similar to AUROC, AUPR is also threshold independent metric.
A.3 DATASETS USED THIS WORK
We the following datasets in this work. Whenever there is mismatch between the resolution of images in in-distribution and out-of-distribution (OOD) data, we appropriately scale the OOD images with bilinear scaling. When there is an overlap between the classes of the in-distribution and OOD dataset, we remove the common classes from the OOD dataset.
CIFAR-10 (Krizhevsky et al. 2009). It consists of 50,000 training images and 10,000 test images from 10 different classes. Each image size is 32 pixels.
CIFAR-100 Krizhevsky al.) 2009). CIFAR-I00 also has 50,000 training images and 10,000 test images. However, it has 100 classes which are further organized in 20 sub-classes. Note that its classes aren't identical to the CIFAR 10 dataset, with slight exception with class truck in CIFAR- 10 and pickup truck in CIFAR- 100. However, their classes share multiple similar semantics, making it hard catch outliers from the other dataset.
SVHN (Nezeer et al., 2011). SVHN is a real-world street-view housing number dataset. It has 73,257 digits available for training, and 26,032 digits for testing. Similar to the CIFAR-10/100 dataset, the size of its images is also 32x32 pixels.
STL-10 (Coates et al., 2011) STL-10 has identical classes as the CIFAR-10 dataset but focuses on the unsupervised learning. It has 5,000 training images, 8,000 test images, and a set of 100,000 unlabeled images. Unlike the previous three datasets, the size of its images 96x96 pixels.
DTD Cimpoi et al., 2014). Describable Textures Dataset (DTD) is collection of textural images in the wild. It includes total of 5.640 images, split equally between 47 categories where the size images range between 300x300 and 640 640 pixels.
ImageNet DDng et al., 2009). ImageNet is large scale dataset of 1,000 categories with 1.2 Million training images and 50,000 validation images. has high diversity in both inter- and intra-class images and is known to have strong generalization properties to other datasets.
Blobs. Similar to Hendrycks et al. (001aa), we algorithmically generate these amorphous shapes with definite edges.
Gaussian Noise. We generate images with Gaussian noise using mean of 0.5 and standard deviation of 0.25. We clip the pixel value to the valid pixel range of [0, 1].
Uniform Noise. It refers to images where each pixel value is uniformly sampled from the [0, 1] range.
We refer the commonly used ILSC 2012 release of ImageNet dataset.
14

---

Published as a conference paper at ICLR 2021
B ADDITIONAL EXPERIMENTAL RESULTS
B.I LIMITATIONS OF OUTLIER DETECTORS BASED ON SUPERVISED TRAINING
Existing supervised training based detector assumes that fine-grained data labels are available for the training data. What happens to the performance of current detectors if we relax this assumption by assuming that only coarse labels are present. We simulate this setup by combining consecutive classes from the CIFAR- 10 dataset into two groups, referred to as CIFAR-2, or five groups referred to as CIFAR-5. We use CIFAR 100 as the out-of -it-o---iitiibttiin dataset. We find that the performance of existing detectors degrades significantly when only coarse labels are present (Figure D. In contrast, SSD operates on unlabeled data thus doesn't suffer from similar performance degradation.
AUROC
200
80
XUROC
60
C1AR-38 CF40-5
OTAR-2
Unioxed
First
Res/Ntt-1 Block
Sacond
Fourth
CIAAR 100
000 dataeet
BiVIs
Figure 4: Existing supervised detector requires fine-graine grained labels. In contrast, SSD can achieve similar performance with only unlabeled data.
Figure 6: Using extra unla- beled training data can help further improve the perfor- mance of SSD.
Figure 5: Relationship AU- ROC with clusters depends on which block we use as the fea- ture extractor.
B.2 ON CHOICE OF NUMBER OF CLUSTERS
We find that the choice of optimal number of clusters is dependent on which layer we use as the feature extractor in Residual Neural network. We demonstrate this trend in Figure with CIFAR-10 as in-distribution dataset and CIFAR-100 as last layer of each block in the residual network and measure the SSD performance with them. While for the first three blocks, we find an increase in AUROC with number of clusters, the trend is reversed for the last block (Figure S). Since last block features achieve highest detection performance, we model in-distribution features using a single cluster.
dataset. We extract features from the
B.3 ABLATION STUDY FOR FEW-SHOT OOD DETECTION
For few shot OOD detection, we ablate along the number of transformations used for each sample. We choose CIFAR- 100 in-distribution and CIFAR-10 as OOD dataset with SSD set & to five, and choose ResNet- 18 network architecture. When increasing number of transformations from 1, 5, 10, 20, 50 the AUROC of detector is 74.3, 75.7, 76.1, 76.3, 76.7. To achieve balance between the performance and computational cost, we use ten transformations for each sample in our final experiments.
B.4 PERFORMANCE SSD IMPROVES WITH AMOUNT OF UNLABELED DATA
With easy access to unlabeled data, it is compelling to develop detectors that can benefit from the increasing amount of such data. We earlier demonstrated this ability of SSD for the CIFAR-10 dataset in Figure 2 Now we present similar results with the STL-10 dataset. We first train self-supervised network, and an equivalent supervised network with 5,000 training images from the STL-10 dataset. We refer to these networks by SSD-5k and Sup-5k, respectively. Next, we include additional 10,000 images from the available 100k unlabeled images in the dataset. As we show in Figure SSD is able to achieve large gains in performance with access to the additional unlabeled training data.
B.5 RESULTS WITH DIFFERENT PERFORMANCE METRICS
We provide our detailed experimental results for each component in the SSD framework with three different performance metrics in Table 7
15

---

Published as a conference paper at ICLR 2021
Table 6: Experimental results of SSD detector with multiple metrics for ImageNet dataset.
95183
In-
distribation
OOD
Supervised
SSD
Superivsed
S3D:
Supervised
SSD,
I-1
k-5
I=1
1-5
km1
kmd
ImagoNitt
SVHN
1.3
0.6
0.0
0.0
99.4
99.1
10000
100.0
98.4
96.5
100.0
100.0
Testure
$7.2
23.2
20.1
11.4
85.4
95.4
95.4
97.4
41.7
75.8
7X.6
54.2
Blobs
0.0
0.0
0.0
98.4
99.5
100.0
100.0
81.1
91.5
100.0
100,0
Caneeian
0.0
0.0
100.0
100.0
100.0
100.
16

---

Published as a conference paper at ICLR 2021
k=5
92.9
99.8
98.9
81.7
99.0
92.2
100.0
98.0
100.
87.7
99.7
SSD,
k=
92.5
100.0
98.1
100.0
76.2
99.1
91.6
100.0
98.7
100.0
87.1
99.5
SSD
92.3
99.9
97.3
100.0
65.3
70.7
98.2
93.3
98.9
83.7
86.7
AUPR 1
k=5
100.0
99.0
100.0
77.2
91.9
100.0
92.3
99.7
99,9
Table 7: Experimental results of SSD detector with multiple metrics over CIIAR 10, CIFAR 100. and STI. 10 dataser.
SSD,
k=
90.5
100.0
98.4
99.4
69.3
95.0
97.8
92.6
99.5
92.2
SSD Supervised
89.5
99.8
96.7
99.9
97.5
97.9
82.5
99. 8
95.6
98.
72.9
97.8
95.1
99.5
82.6
92.1
k=5
94.0
99.6
99.2
100.0
84.0
97.4
94.5
1000
97.8
99.9
99.8
SSDA
99.9
98.6
100.0
97.3
93.8
99.8
90.4
99.7
SSD+
93.4
99.9
98.5
71.0
98.2
81.2
98.3
92.4
96.9
85.8
92.9
AUROC
k=5
93.0
99.4
100.0
94.2
100.0
90.0
84.5
SSD,
k=
91.7
99.7
98.7
96.5
Superivsed
90.6
97.8
99.9
55.3
57.3
95.7
75.5
88.6
SSD
97.6
98.8
69.6
82.9
94.8
85.8
96.4
1=5
0.0
0.3
SSD2
k=
36.3
0.5
6.4
0.0
72.1
11.3
24.3
0.0
8.6
0.4
46.2
0.1
FPR (TPR = 95%) J
SSD+
38.5
0.2
7.7
0.0
89.5
7.9
68.1
3.6
40.0
18.6
51.8
67.8
k=5
2.7
4.5
25.1
2.4
39.5
SSDz
k=
44.7
0.2
5.1
4.4
85.3
2.7
16.4
1.1
32.9
5.8
50.0
14.6
SSD Supervised
47.4
12.4
0.0
964
28.4
23
95.6
73.2
26.2
69.4
88.9
50.7
2.0
4.3
89.4
20,9
65.8
29.9
53.0
16.3
CIFAR-100 100
OOD
SVHN
Texture
Blobs
SVHN
Texture
Blobs
SVHN
Texture
Blobs
distribution
CITAR-10
STL-10
CIFAR-IN
17


Reviews:
[{"review_id": "xDBLcIlb6rF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] \nThe paper addresses the OOD problem without learning from class labels. It proposes to learn the feature representation with unsupervised learning, then applies Mahalanobis distance to measure how far a test data is away from the in-distribution data. The proposed framework also considers the cases of when class labels or OOD data are available. The experiments on image datasets (mainly cifar-10/100) demonstrate its advantage over other unsupervised OOD methods.\n\n[Reasons for score]\nBullet 1 in the cons is a significant concern. However, the method and analysis provide interesting insights. Overall, the pros and cons are equally substantial. \n\n[Strengths]\n1. The proposed framework is sounded and can well handle a variety of cases when class labels or OOD data are available.\n2. The way it utilizes a small amount of OOD data is novel (eq.3). \n3. The experiments and analysis are sufficient to support its major claim while providing additional insights (ex: Figures 2 and 4, Table 4).\n\n[Weaknesses]\n1. The eq.3 assumes OOD data are from a cluster that has the same mean and variance. This assumption is not valid in general cases. The experiments have a setup that favors this assumption; therefore, bias exists. The paper should explicitly point out this as a substantial limitation and provide some experiments (ex: use SVHN+CIFAR10 for the labeled OOD data) to elaborate it.\n2. The evaluations are mainly on toy datasets. Although appendix table 6 uses ImageNet, it will be interesting if more realistic OOD datasets are included.\n3. The use of the word “unlabeled” sometimes is confusing. There are two types of labels: class label and in/out-of-distribution label. The paper should clearly say its method is doing OOD without class labels. Please consider polishing the use of these terms in sections 1 and 2.\n\n======================\nPOST REBUTTAL\n\nThe updated results make sense. The limitation/assumption mentioned in weaknesses 1 must be sufficiently disclosed in section 3.2. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XWoP_v4oYoA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates a classic unsupervised outlier detection problem, in which we do not have any label information and need to learn a detection model from those unlabeled data to identify any inconsistent data points as outliers. The key approach here is to apply existing self-supervised contrastive feature learning methods to extract feature representations and then apply a cluster-based method to calculate outlier scores. It also presents two ways to leverage labeled outlier data if available, including an improved mahalanobis distance method and the application of supervised contrastive learning methods proposed recently. The methods, including unsupervised and semi-supervised methods that use a few labeled outlier data, are evaluated using four datasets. As I read through the paper, I find the following major issues.\n\n1. the question this work intends to answer, \"Can we design an effective outlier detector with access to only unlabeled data from training distribution?\", is a classic and well-studied problem in the anomaly/outlier detection community. There have been many studies over this problem. We cannot simply ignore those previous work  and states it as a new problem. see resources like Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3), 1-58. or Aggarwal, C. C. (2015). Outlier analysis. Springer, Cham. for numerous previous work on using shallow methods to address this problem.\n\n2. There are a number of studies on self-supervised outlier detection approaches as well as what is called few-shot outlier detection approaches, but I cannot find any discussion of those work and the empirical comparison to these methods. The authors may refer to some recent survey papers, such as \"Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2020). Deep Learning for Anomaly Detection: A Review. arXiv preprint arXiv:2007.02500.\", to find some of these studies. Some closely related methods are: self-supervised methods such as GT and E3Outlier that learns feature representations using a pre-text task in a self-supervised way; unsupervised outlier detection methods such as RDA, REPEN, ALOCC, OCGAN, etc.; methods that use a few labeled outlier data such as Deep SAD, DevNet, REPEN, etc. Please see table 1 in that survey paper for more details. The authors are suggested to discuss and differentiate their method from these existing studies, and to compare empirical comparisons to these closely related methods.\n\n3. The claiming of \"a parameter-free detector\" is misleading and incorrect. Similar to other methods, the presented methods still have a huge number of hyper-parameters. They may be able to work without parameter tuning on each dataset, but this is not parameter-free. Also, many existing methods can also work in this way.\n\n4. The way that the presented outlier detection methods utilizes the labeled outlier data may less effective than previous work, because the cluster-based anomaly scoring here is separated from the representation learning. More advanced approaches (see some of the methods mentioned above) can unify the anomaly scoring and the representation learning together and the labeled outlier data is used to optimized the entire anomaly detection pipeline, rather than the representation learning stage only. \n\n5. The experiment settings are not properly designed to justify the paper's arguments. First, closely related deep unsupervised and semi-supervised methods are missing in the comparison in tables 1 and 3. Second, I think it is important to include some popular baselines here. For example, how is the performance of using traditional outlier detectors such as iforest, lof, and knn distance on feature representations extracted with a pre-trained resnet-50? How many benefits does the computationally extensive contrastive representation learning gain compared to those simple solutions?  Third, why is the cluster-based outlier scoring method used? can we use other traditional outlier detectors such as iforest, lof, and knn distance?", "extra_scores": {}, "extra_text": {}}, {"review_id": "IpikhaQbjzJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: This paper proposes an outlier detection scheme based on contrastive self-supervised training for representation learning and cluster-conditioned detection using Mahalanobis distance. It shows 0.8% improvement in AUROC over the previous state-of-the-art by Winkens when trained using labeled in-distribution data. When training data is unlabeled, it performs significantly better than existing OOD detection algorithms based on AE, VAE, PixelCNN++, Deep-SVDD, and Rotation-loss and it performs only slightly worse than the labeled case. This is not surprising since I don't believe label information is crucial for OOD detection anyway, but it's good to confirm it. The authors show additional gain is possible when some OOD images are available during training, i.e., one or five images per class. This paper is very well written and I think the novelty and contributions are significant enough to merit its acceptance.\n\nComments: It's interesting to see the availability of only 1~5 samples per class for OOD data can result in some noticeable gains. I am not sure if covariance matrices have some scaling factors in (3), but how about using different weights in (3) for in-distribution and OOD data to take care of differences between sample sizes of in-distribution and OOD data? Would it make sense to have a higher weight for OOD data since it has fewer samples than in-distribution data?\n\nThere's a big performance gap between existing unlabeled schemes and the proposed scheme. Which one contributes most to the difference? NT-Xent? Mahalanobis distance? Cluster-conditioned detection? Some ablation studies that remove one or more of these would be interesting.\n\nAuthors say \"higher eigenvalues dominates Euclidean distance but are least helpful for outlier detection\", but wouldn't too small eigenvalues be less helpful for outlier detection because they are more sensitive to noise? Can authors provide an explanation on why higher eigenvalues are least helpful for outlier detection? It's strange that very small eigenvalues give very high AUROC values in Fig. 2.\n\nIn (3), why there's no minimization over m as in (2)? Is single cluster assumed for both in-distribution and OOD data in (3)?\n\nReference (SimCLR) missing for NT-Xent in equation (1). \n\nPlease consider including and comparing with the following references.\n\n[X1] Denouden, et al., Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance, https://arxiv.org/abs/1812.02765\n\n[X2] Vernekar, et al., Out-of-distribution Detection in Classifiers via Generation, https://arxiv.org/abs/1910.04241\n\n[X3] Goyal, et al., DROCC: Deep Robust One-Class Classification, https://arxiv.org/abs/2002.12718\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EN0oRTt5JO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle out-of-distribution detection without requiring additional detailed labels by leveraging the recent advancements in self-supervised methods combined with Mahalanobis distance metrics in feature space. They find that their approach matches or slightly outperforms supervised baselines, and widely outperforms other unsupervised approaches. They also consider a few-shot version of the task and extend their method to take advantage of supervision when available.\n\nPros:\nPerformance on outliers is a significant challenge for parametric models, so if outliers can be detected accurately this may help address potential failure modes in real-world systems. There is clear motivation for the problem.\nThe authors show clear gains over existing baselines.\n\nCons:\nTheir method is a pretty-straightforward combination of two established methods, contrastive self-supervised representation learning and the Mahalanobis distance as a metric of distance from a point to a distribution. \nTheir method seems to rely heavily on the efficacy of contrastive self-supervised learning. It would be good to see an ablations showing the performance of the Mahalanobis distance-based approach with the same embedding features as some of the poorly-performing unsupervised baseline methods to understand how much of the gains are due to the more powerful representation method. For example, could the authors add Mahalanobis distance on top of the learned representations from the autoencoder or VAE models? I know that the authors might have to get creative here, but as it stands it is not clear to me how much of these performance gains in the unsupervised setting are due to the author’s contribution, as opposed to just due to SimCLR.\n\n\nNits:\n\n Naming their method SSD seems unnecessarily confusing due to the popularity of the Single-shot multibox detector (SSD) model. I would recommend rethinking the acronym for clarity.\n\nI would prefer to see the related work discussed before the authors proposed approach, so that their method can be better placed into context with the existing literature by the reader.\n\n“Scaling with eigenvalues remove the bias” -> Scaling with eigenvalues removes the bias\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is some positive consensus on this paper, which improved somewhat after
the very detailed rebuttal comments by the authors. The use of limited amounts of OOD data is interesting and novel. There were some experimental design problems, but these were well-addressed in rebuttal.

A reviewer points out that
anomaly/outlier detection does not explicitly assume that there is only one
class within the normal class (or in-distribution data). The one-class
assumption is mainly made in some popular anomaly detection methods, such as
one-class classification-based approaches for anomaly detection. The authors
should take this into careful consideration when preparing a final version of
this work.

Decision: Accept (Poster)