CADet: Fully Self-Supervised Anomaly Detection With Contrastive Learning
Authors: Charles Guille-Escuret
Pau Rodriguez
David Vazquez
Ioannis Mitliagkas
Joao Monteiro
Venue: Submitted to ICLR 2023 (2023)
Keywords: Contrastive learning
OOD detection
adversarial detection
MMD
ImageNet-O
Anomaly detection

Abstract: Handling out-of-distribution (OOD) samples has become a major stake in the real-world deployment of machine learning systems. This work explores the application of self-supervised contrastive learning to the simultaneous detection of two types of OOD samples: unseen classes and adversarial perturbations. Since in practice the distribution of such samples is not known in advance, we do not assume access to OOD examples. We show that similarity functions trained with contrastive learning can be leveraged with the maximum mean discrepancy (MMD) two-sample test to verify whether two independent sets of samples are drawn from the same distribution. Inspired by this approach, we introduce CADet (Contrastive Anomaly Detection), a method based on image augmentations to perform anomaly detection on single samples. CADet compares favorably to adversarial detection methods to detect adversarially perturbed samples on ImageNet. Simultaneously, it achieves comparable performance to unseen label detection methods on two challenging benchmarks: ImageNet-O and iNaturalist. Additionally, CADet is fully self-supervised and requires neither labels for in-distribution samples nor access to out-of-distribution examples.

Under review as a conference paper at ICLR 2023
CADET: FULLY SELF-SUPERVISHE ANOMALY DETEC- TION WITH CONTRASTIVE LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Handling out-of of-iiiibibiutiionnnnn (OOD) samples has become a major stake in the real- world deployment of machine learning systems. This work explores the application of self-supprvised contrastive learning to the simultaneous detection of two types of OOD samples: unseen classes and adversarial perturbations. Since in practice the distribution of such samples is not known in advance, we do not assume access to OOD examples. We show that similarity functions trained with contrastive learning can be leveraged with the maximum mean discrepancy (MMD) two-sample test to verify whether two independent sets of samples are drawn from the same distribution. Inspired by this approach, we introduce CADet (Contrastive Anomaly Detection), a method based on image augmentations perform anomaly detection single samples. CADet compares favorably to adversarial detection methods to detect adversarially perturbed samples on ImageNet. Simultaneously, it achieves comparable performance to unseen label detection methods on two challenging benchmarks: ImageNet-O and iNaturalist. CADet is fully self-supervised and requires neither labels for in-distribution samples access OOD examples.
INTRODUCTION
While modern machine learning systems have achieved countless successful real-world applications, handling -ut-of-distributiion (OOD) inputs remains tough challenge of significant importance. The problem is especially acute for high-dimensional problems like image classification. Models are typically trained close-world setting but inevitably faced with novel input classes when deployed in the real world. The impact can range from displeasing customer experience to dire consequences in the case of safety-criticn applications such as autonomous driving (Kitt et al. 2010) or medical analysis (Schlegl et al., 2017a). Although achieving high accuracy against all meaningtul distributional shifts is the most desirable solution, it is particularly challenging. An efficient method to mitigate the consequences of unexpected inputs is to perform anomaly detection, which allows the system to anticipate its inability to process unusual inputs and react adequately.
Anomaly detection methods generally rely on of three types of statistics: features, logits, and softmax probabilities, with some systems leveraging a mix of these (Wang et al., (022). An anomaly score f(x) is computed, and then detection with threshold performed based on whether f(x) > т. The goal of a detection system is to find an anomaly score that efficiently discriminates between in-distribution and out-of-distribution samples. However, the common problem of these systems is that different distributional shifts will unpredictably affect these statistics. Accordingly, detection systems either achieve good performance on specific types of distributions or require tuning on OOD samples. In both cases, their practical use is severely limited. Motivated by these issues, recent work has tackled the challenge of designing detection systems for unseen classes without prior knowledge
of the unseen label set access to OOD samples (Winkens al. 2020: Tack et al. 2020 Wang cct al. 2022).
We first investigate the use of maximum mean discrepancy two-sample test (MMD) (Gretton et al. 2012) in conjunction with self-supervised contrastive learning to assess whether two sets of samples have been drawn from the same distribution. Motivated by the strong testing power of this method, then introduce a statistic inspired by MMD and leveraging contrastive transformations. Based on this statistic, propose CADet (Contrastive Anomaly Detection), which is able to detect OOD samples

---

Under review as a conference paper at ICLR 2023
from single inputs and performs well both label- and adversarial detection benchmarks, without requiring access to any OOD samples to train or tune the method.
Only few works have addressed these tasks simultancously. These works either focus on particular in-distribution data such as medical imaging for specific diseases (Uwimanal & Senanayake (02T)) or evaluate their performances 2009). SVHN (Netzer et al.) 2011), and LSUN (Yu et al.] al.] (013), resulting in simple benchmarks that do not translate general real world applications (Lee et al. 2018).
datasets with very distant classes such CIFARTO (Krizhevsky)
Contributions: Our main contributions are as follows:
We use similarity functions learned by supervipd contrastive learning with MMD to show that the test sets of CIFARIO and CIFARIO.1 (Recht et al. 2019) have different distributions.
We propose novel improvement to MMD and show it can also be used to confidently detect distributional shifts when given a small number of samples.
We introduce CADet, fully self-supervised method for anomaly detection, and show it outperforms current methods in adversarial detection tasks while performing well on class-based OOD detection.
The outline is as follows: in Section 2 we discuss relevant previous work. Section 3 describes the supervid contrastive method based on SimCLRv2 (Chen et al.) 2020b) used in this work. Section [ explores the application of learned similarity functions in conjunction with MMD to verify whether two independent sets of samples drawn from the same distribution. Section 5 presents CA Det and evaluates its empirical performance. Finally, we discuss results and limitations in Section 6
2 RELATED WORK
We propose a self-supervised contrastive method for anomaly detection (both unknown classes and adversarial attacks) inspired by MMD. Thus, our work intersects with the MMD, label-based OOD detection, adversarial detection, and self-supervised contrastive learning literature.
MMD two- wvo-sample test has been extensively studied (Gretton et 2012; Wenliang et al. 2019
Gretton et al. 2009 [Sutherland et al., 2016)] Chwialkowski et al. [2015 Jitkrittum et al. 6)., though
is, to the best of our knowledge, the first time a similarity function trained via contrastive learning is used in conjunction with MMD. Liu et al. (2020a) uses MMD with deep kernel trained
fraction of the samples to argue that CIFARIO 10 and CIFARIO. have different test distributions. We build upon that work by confirming their finding with higher confidence levels, using fewer samples.
Label-based OOD detection methods discriminate samples that differ from those in the training distribution. We focus unsupervised OOD detection in this work, i.e., do not assume access to data labeled as OOD. Unsupervised OOD detection methods include density-
(Zhai et al. 2016 Nalisnick et al. 20188] 2019 Choi al. 2018 Du & Mordatch| 2019: Ren et al. 2019 Serra et al.
2019;
Grathwohl et al. 2019; Liu et al., 2020b Dinh et al., 2016)) reconstruction based (Schlegl
al. 20176 Zong al. 2018 Deecke al.] 2018 Pichorskyi et al. 20188) Perera al. 2019 Choi
Zong
ct al. 2018), one-class classifiers (Schölkopl et al. 1999 Rulf et al. (018)) self-supervised (Golan &
| Bergman & Hoshen 2020 Tack al., 2020), and supervised approaches (Liang et 2017 Hendrycks & (impel.2016)) though some works do not fall into any of these categories (Wang et al. 2022).
EI- Yaniy Hendrycks al.
A Adversarial detection discriminates adversarial samples from the original data. Adversarial samples are generated by minimally perturbing actual samples to produce change in the model's output, such as missaassification. Most works rely on the knowledge of some attacks for training (Abusnaina
cct al. 2021 Metzen et al. 2017 Feinman et al. 2017 Lust & Condurache] 2020 Zuo & Zeng 2021
Papernot & McDaniel 2018 Ma et al. 2018), with the exception of Hu al. (2019)
Self-supervised contrastive learning methods (Wu et al. 2018 He et al. 2020 Chen et al. 2020a b)
commonly used pre-train model from unlabeled data to solve a downstream task such as image classification. Contrastive learning relies on instance discrimination trained with a contrastive
loss (Hadsell et al. 2006) such infoNCE (Gutmann & Hyvärinen 2010)
2

---

Under review as a conference paper at ICLR 2023
Contrastive learning for OOD detection aims find good representations for detecting OOD samples in a supervised (Liu & Abbeel| 2 Khalid et al. 2022) or unsupervised (Winkens et al. 2020. Mohseni et al., 2020, Sehwag al. (221) setting. Perhaps the closest work in the literature is CSI (Tack et al. 2020), which found SimCLR features to have good discriminative power for unknown classes detection and leveraged similarities between transformed samples in their score. However, this method is not well-suited for adversarial detection. CSI ignores the similarities between different transformations of a same sample, an essential component to perform adversarial detection (see Section 6.2). In addition, CSI scales their score with the norm of input representations. While efficient on samples with unknown classes, is unreliable on adversarial perturbations, which typically increase representation norms.
3 CONTRASTIVE MODEL
We build our model on top of SimCLRv2 (Chen et al., 2020b) for its simplicity and efficiency. is composed of an encoder backbone network fo as well as 3-layer contrastive head hgs Given an in- distribution sample X, similarity funetion sim, and distribution of training transformations Tirain- the goal is to simultaneously maximize ff and minimize Ez ft..g-.i:o..taTTTe.... [si o fo(to (x)), hg o ft (1)))), i.., we want to learn rep- resentations in which random transformations of same example are close while random transforma- tions of different examples are distant.
[sim(hgr o fo (f)),,,),, har o fo (x)))]
To achieve this, given an input batch {z:)i=1.....N we compute the set {ar 6) ) 1==,1==1,.... by applying two transformations independently sampled from Tirain each F₁. We then compute the embeddings 2,
(5) = hag o fo (z) and apply the following contrastive loss:
L(z) =
Σ $=
log
(1)
Eje(hn.ny
cim(
3/T EjE(L...N cim((s 2/1
where 7 is the temperature hyperparameter and sim(2z v) = ||>||||11|
(xlv)
is the cosine similarity.
Hyperparameters: We follow as closely as possible the setting from SimCLRv2 with a few mod- ifications to adapt to hardware limitations. particular, we use the LARS optimizer (You et al.. 2017) with learning rate 1.2, momentum 0.9, and weight decay 10-4, Iteration-wise, we scale up the learning rate for the first 40 epochs linearly, then use an iteration-wic cosine decaying schedule until epoch 800, with temperature size of 1024. We compute the contrastive loss on all batch samples by aggregating the embeddings computed by each GPU. We use synchronized BatchNorm and fp32 precision and do not use memory buffer. We use the same set of transformations, i.e., Gaussian blur and horizontal flip with probability 0.5, color jittering with probability 0.8, random crop with scale uniformly sampled in [0.08, 1]. and grayscale with probability 0.2.
0.1. We train 8 V100 GPUs with accumulated batch
For computational simplicity and comparison with previous work, we use a ResNet50 encoder architecture with final features of size 2048. Following SimCLRv2, we use a three-layer fully connected contrastive head with hidden layers of width 2048 using ReLU activation and batchNorm and set the last layer projection to dimension 128. For evaluation, we use the features produced by the encoder without the contrastive head. We do not, at any point, use supervised fine-tuning.
4 MMD TWO-SAMPLE TEST
The Maximum Mean Discrepancy (MMD) is statistic used in the MMD two-sample test to assess whether two sets of samples S3z and Se are drawn from the same distribution. It estimates the expected difference between the intra-set distances and the across-sets distances.
Definition 4.1 (Gretton et al. (2012). ) Let : X' x X + R be the kernel of a reproducing Hilbert space Ha. with feature maps k( x)) Ha. Let X, X' ~ IP and Y, Y' ~ Q. Under mild integrability
3

---

Under review as a conference paper at ICLR 2023
conditions,
(M 0:H+) = sup
[F [[((
(2)
SEH.lls
=V X k(Y, (XY
(3)
Given two sets of n samples Sp = (Xilign and S = (Y respectively drawn from P and Q we can compute the following unbiased estimator Liu al. (2020a):
MMD (q; k) = n(n 1)
Σ #
X)) A(Y, V2) A(2, k(V,,
(4)
Under the null hypothesis bo IP Q. this estimator follows a normal distribution of mean 0 Grettonn et al.] 2012). Its variance can be directly estimated (Gretton al., 2009), but it is simpler to perform permutation test as suggested in (Sttherland et al. (2016)). which directly yields a p-value for bo- The idea to use random splits X, of the input sample sets obtain 7perm different (though not independent) samplings of MMD3 k), which approximate the distribution of (Sp Sp; k) under the null hypothesis.
Liu et a (2020a) train deep kernel to maximize the power of the MMD two-sample test training split of the sets of samples to test. We propose instead to use our learned similarity function without any fine-tuning, Note that we return the p-value Apera+T (1 + D Tperm
1 (p: 2 est)) instead of Direrm (p& 2 est). Indeed, under the null hypothesis P = Q, est and pt are drawn from Opprro
the same distribution, so for j E {0,1, ., Rperm }. the probability for est to be smaller than exactly j elements of {p:} is Piperm +I' Therefore, the probability that j elements or less of {p:}: are larger
than est is E/= AI Rparm +I While this change has a small impact for large values of Operm, it is essential to guarantee that we indeed return a correct p-value. Notably, the algorithm of Liu et al. (2020a) has probability hypothesis.
1+1
O to return an output of 0.00 even under the null
Additionnally, we propose an improvement of MMD called MMD-CC (MMD with Clean Calibration). Instead of computing ps based on random splits of Sr U Sq. we require as input two disjoint sets of samples drawn from IP and compute p: based on random splits of (( U Sf (see Algorithm| 1). This change requires to use twice many samples from F. but reduces the variance induced by the random splits of Sp Se. which is significant when the number of samples is small.
4.1 DISTRIBUTION SHIFT BETWEEN CIFAR- 10 AND CIFAR-10.1 TEST SETS
After years of evaluation of popular super- vised architectures on the test set of CIFAR- 10 (Krizhevsky/ 2009). modern models may overfit it through their hyperparameter tuning and structural choices. CIFAR-10.1 (Recht et al.) 2019) was collected to verify the performances of these models on Iruly independent sam- ple from the training distribution. The authors note a consistent drop in accuracy across mod- els and suggest it could be due a distribu- tional shift, though they could not demonstrate it. Recent work (Liu et al. 2020a) leveraged the two-sample test to provide strong evidence of distributional shifts between the test sets of
Algorithm MMD-CC two-sample test
Input: s(1) S SQ. Rperm, sim
(2)
est MM D2 (()
(S()) Sqjsim)
for i = 1,2, 7perm do
Randomly split s()) US$ into two disjoint
sets X, Y of equal size
p: +
end for Output: p: 1t (1
E Pperm I (p. est))
CIFAR- 10 and CIFAR-10.1. We run MMD-
CC and MMD two-sample tests for 100 different samplings of (!!)) s Sg. using every time
0.05. We also report results using cosine similarity applied to the features of supervised models as a comparative baseline. We report the results in Table I for range of sample sizes. We compare the results to three
Mperr 500, and rejecting bo when the obtained paluue is below the threshold
4

---

Under review as a conference paper at ICLR 2023
1.0
08
08
0.6
06
Imagelet-O
samples
Naturalist
500 samples
PGD
1000 samples
OW
2000 samples
FCSM
00
random
0.
0.0
02
04
0.6
0.8
100
0.0
02
04
0.6
0.8
1.0
false positive
false positive rate
(a)
(b)
Figure 1:
ROC curves for MMD-CC two-sample test on CIFARIO vs CIFARTO.1 using different
sample sizes (left) and for CADet anomaly detection different out-distributions (right).
Table I:
Average rejection rates of ho on CIFAR- 10 vs CIFAR-10.1 for a 0.05 across different sample sizes n, using ResNet50 backbone.
n=2000
n=1000
n=500
n=200
n=100
n=50
ME- ME-Chhwee Chwialkowski et al. (0155)
0.588
12ST-L~Cheng & Cloninge (2019)
0.529
MMD-D~ Liu et al. (2020a)
0.744
MMD SimCLRy2 (ours)
1.00
1.00
0.997
0.702
0.325
0.154
MMD-CC + SimCLRv2 (ours)
1.00
1.00
0.997
0.686
0.304
0.150
MMD Supervised (ours)
1.00
1.00
0.884
0.305
0.135
0.103
MMD-CC + Supervised (ours)
1.00
1.00
0.870
0.298
0.131
0.096
competitive methods reported in Liu et al. (2020a): Mean embedding (ME) (Chwialkowski et al.
2015 Jitkrittum al. 2016), MMD-D (Liu et | 2022a), and C2ST-L (Cheng & Cloninger] (2019). Finally, y, we show in Figure Ta the ROC curves of the proposed model for different sample sizes.
Other methods in the literature do not external data for pre-training, as do with ImageNet, which makes a fair comparison difficult. However, it is noteworthy that our learned similarity can very confidently distinguish samples from the two datasets, even in settings with fewer samples available. Furthermore, while we achieve excellent results even with supervised network, our model trained with contrastive learning outperforms the supervised alternative very significantly. We note however that with such high number of samples available, MMD-CC performs slightly worse than MMD. Finally, we believe the confidence obtained with our method decisively concludes that CIFARI0 10 and 1IFAR10.1 have different distributions, which is likely the primary explanation for the significant drop in performances across models on CIFAR 10.A as conjectured by Recht et al. (2019). The difference in distribution between CIFAR10 and CIFAR10. 10.1 is neither based on label set adversarial perturbations, making it an interesting task.
4.2 DETECTION OF DISTRIBUTIONAL SHIFTS FROM SMALL NUMBER OF SAMPLES
Given small set of samples with potential unknown classes or adversarial attacks, we can similarly use the waops test with our similarity function to verify whether these samples are in-distribution (Gao et al. 2021). In particular, we test for samples drawn from ImageNet-O, iNaturalist, and PGD perturbations, with sample sizes ranging from 3 to 20. For these experiments, we sample {! and 5(22 5000 times across all of ImageNet's validation set and compare their MMD and MMD-CC estimators to the one obtained from Sy and Sq- We report in Table the AUROC of the resulting detection and compare it to the ones obtained with supervised ResNet50 the baseline.
Such a setting where we use several samples assumed to be drawn from a same distribution to perform detection is uncommon, and not aware of prior baselines in the literature. Despite using very
5

---

Under review as a conference paper at ICLR 2023
Table AUROC for detection using two-sample on to 20 samples drawn from ImageNet and from ImageNet-O, iNaturalist or PGD perturbations, with a ResNet50 backbone.
ImageNet-O
NNtuuralist
PGD
nL samples
10
20
10
MMD SimCLRv2
64.3
72.4
86.9
97.6
88.3
97.6
99.5
99.5
35.2
53.8
86.6
98.8
MMD-CC SimCLRv2
65.3
73.2
88.0
97.7
95.4
99.2
99.5
99.5
70.5
84.0
96.6
99.5
MMD Supervised
62.7
69.7
83.2
96.4
91.8
98.7
99.5
99.5
20.0
22.5
33.0
57.5
MMD-CC Supervised
62.6
71.0
85.5
97.2
98.0
99.5
99.5
99.5
57.4
61.3
70.5
85.8
few samples (3 S. v < 20), our method can detect OOD samples with high confidence. We observe particularly outstanding performances iNaturalist, which is easily explained by the fact that the subset we are using (cf. Section[] [1) only contains plant species, logically inducing an abnormally high similarity within its samples. Furthermore, observe that MMD-CC performs significanty better than MMD, especially on detecting samples perturbed by PGD.
Although our method attains excellent detection rates for sufficient numbers of samples, the require- ment to have set of samples all drawn from the same distribution to perform the test makes it unpractical for real-wort world applications. In the following section, we present CADet, detection method inspired by MMD but applicable to anomaly detection with single inputs.
CADET: CONTRASTIVE ANOMALY DETECTION
While the numbers in Section 4 demonstrate the reliability of two-sample test coupled with contrastive learning for identifying distributional shifts, it requires several samples from the same distribution, which is generally unrealistic for practical detection purposes. This section presents CADet, method to leverage contrastive learning for anomaly detection on single samples from OOD distributions.
sud--super contrastive learning trains similarity function s to maximize the similarity between augmentations of the same sample, and minimize the similarity between augmentations of different samples. Given an input sample teaa we propose leverage this property to perform anomaly detec- tion on test taking inspiration from MMD two-sample test. More precisely, given a transformation distribution Teat, we compute Yhrs random transformations zfeat of Etests as well as Ntrr random
transformations (k) each sample (k) of held-out validation dataset val. intra-similarity and out-similarity:
We then compute the
Sattest xdestt 1513
Σ
Σ Es(atert ,0y
(K))
200 200 Ex)
(1) is
m in (x test
m out (x test
(5)
Thers (Mere + 1)
" (x
vall
We finally define the following statistic to perform detection:
scorec = in + rym
(6)
since we do assume knowledge of OOD samples, is difficult priori to tune 7 although crucial to balance information between intra-sample similarity and cross-sample similarity. As workaround, we calibrate y by equalizing the variance between m. and yym on a second set of validation samples x-
Calibration:
Var (ar(mi"(2);2 E x(2)
Var (md x), (22)
Rather than evaluating the false positive (FPR) for range of possible thresholds the hypothesis testing approach to compute the p-value:
Peatue (2 lest) = 1 E ((2 val
s.t. score (x) < scorec
(8)
(2) (xx(2) val ||+1

---

Under review as a conference paper at ICLR 2023
Algorithm 2 and Algorithm - detail the calibration and the testing steps, respectively. Setting threshold p [0, for the Pratue will result in a FPR of mean p, with a variance dependant of [
Section 5. further describes our experimental setting.
Algorithm 2 CADet calibration step
Input: x val: x Teval, learned similarity function various hyperparameters used below;
1: for x(() E xx))) do
2:
i= 1........ Tarrs do
3:
Sample from Tevat
Algorithm CADet testing step
4:
20")
t(x(1)
Input: transformed samples: x. (1) scores:
5: end for
Scores, test sample: z coefficient:
6: end for
7: k: <-00 0
trasnformation set: Teval;
8: for (2)
E do vgl
for 1,2,..., Dtrs do
9:
for i- 1,2,.... Ters do
2: Sample t from Teval
10:
Sample from Teval
xtest + ((xtest)
11:
x.
(2) + (xx(2))
4: end for
12:
end for
5: m f
Σ s(ztest 843
2 se
13:
m
Mive
Σ { ...
(2)
(2))
Σ Nevo
141
out
14:
m
+ out
Σ
Σ s(x) 20) (2))
x
xx
7: score te
(1lro +
mn
m
n # (Xx!))
15:
k f k +
8: rank +0
16: end for
for score"al scores do
17: m
aeoo (Deva +1)
10: if score < score then
out
out
11: rank rank +
12: end if
19: Vmn. Voat = Var(m"n) V Var(m
13: end for
20:
Vout
Vin
14: Peal
# [ccores
rank41
21: k:
Output: p-value: Peal
22: for (2)
E x222 do
23:
scores + m + x m
24:
k
25: end for
Output: coefficient: 7, scores: scores,
transformed samples: x; (1)
5.1 EXPERIMENTS
For all evaluations, we use the same transformations as SimCLRv2 except color jittering, Gaussian blur and grayscaling. We fix the random crop scale to 0.75. We use [(x (2) H = 2000 in-distribution samples, (X(!)) = 300 separate samples to compute cross-similarities, and 50 transformations per sample. We pre-train a ResNet50 with ImageNet as in-distribution.
Unknown classes detection: we use two challenging benchmarks for the detection of unknown classes. iNaturalist using the subset in Huang & Li (202l) made of plants with classes that do not intersect ImageNet. Waang et al. (2022) noted that this dataset is particularly challenging due to proximity of its classes. We also evaluate on ImageNet-O (Hendrycks al. 2021); explicitly designed to be challenging for OOD detection with ImageNet as in-iiitriuution. We compare to
recent works and report the AUROC scores in Table 3
Adversarial detection: for adversarial detection, we generate adversarial attacks on the validation partition of ImageNet against pre-trained ResNet50 using three popular attacks: PGD (Madry et al.
7

---

Under review as a conference paper at ICLR 2023
Table AUROC for OOD detection on ImageNet-O and iNaturalist with ResNet50 backbone.
Training
iNaturalist
ImageNet-O
Average
MSP Hendrycks & Gimpel
20166
88.58
56.13
72.36
Energy Liu et al. 2020c
80.50
53.95
67.23
ODIN Liang et al. (2017
86.48
52.87
69.68
MaxLogit Hendrycks et al.
(2019a
86.42
54.39
70.41
KL Matching Hendrycks et 22019a
Supervised
90.48
67.00
78.74
Sun al. [2021
87.27
68.02
77.65
Mahalanobis
Lee el 2018
89.48
80.15
84.82
Residual Wang et al.
2022
84,63
81.15
82.89
ViM Wang (2022
89.26
81.02
85.14
Supervised Self-supervised
95.28
70.73
83.01
CADet (ours)
83.42
82.29
82.86
(contrastive)
Table AUROC for adversarial detection on ImageNet against PGD, CW and FGSM attacks, with
ResNet50 backbone.
Tuned on Adv
Training
PGD
CW
FGSM
Average
Supervised
62.30
60.29
68.10
63.56
ODIN
Liang et al. 2017
Yes
Contrastive
59.91
60.23
64.99
61.71
Supervised
84.31
84.29
77.95
82.18
Hu Hu et al. 2019
Yes
Contrastive
94.80
95.19
78.18
89.39
Supervised
66.40
59.58
71.02
65.67
Hu Hu et al.
2019
self-calibration
No
Contrastive
75.69
75.74
69.20
73.54
Supervised 75.25
71.02
83.45
76.57
CADet (ours)
No
Contrastive
94.88
95.93
97.56
96.12
2017). CW (Carlini & Wagner 2017). 7) and FGSM (Goodfellow et al. 2014). We follow the tuning suggested by Abusnaina et al. (2021), i.e. PGD: norm Loc, 0.02, step size 0.002, 50 iterations; CW: norm Lg, ò = 0.10, learning rate of 0.03, and 50 iterations; FGSM: norm Los 5 = 0.05. We compare our results with ODIN (Liang et al. 2017). which achieves good performances in Lee et al. (2018) despite not being designed for adversarial detection, and to Hu et al. (2019). Most other existing adversarial detection methods assume access to adversarial samples during training (see Section 2). While both of these works use adversarial samples to tune hyperparameters, additionally propose a modification to Hu et al. (2019) to perform auto-calibration based on the mean and variance of the criterions clean data, similarly to CADet's calibration step. We report the AUROC scores in Table 4 and illustrate them with ROC curves against each anomaly type in
Figure lb
DISCUSSION
6.1
RESULTS
CADet performs particularly well on adversarial detection, surpassing alternatives wide margin. We argue that self-supervised contrastive learning is suitable mechanism for detecting classification attacks due its inherent label-agnostic nature. Interestingly, Hu et al. (2019) also benefits from contrastive pre-training, achieving much higher performances than with supervised backbone. However, is very reliant on calibrating on adversarial samples, since we observe a significant drop in performances with auto-calibration. Simultancously. CADet performs well on detecting unknown classes, although not beating the best existing methods on iNaturalist.
Notably, applying CADet to supervised network achieves state-of-the-art performances on iNatural- ist with ResNet50 architecture, suggesting CADet can reasonable standalone detection method some benchmarks, independently of contrastive learning. In addition, the poor performances of the supervised network on ImageNet-O and adversarial attacks show that contrastive learning is essential address the trade-off between different type of anomalies.
8

---

Under review as a conference paper at ICLR 2023
Overall, our results show CADet achieves an excellent trade-off when considering both adversarial and label-based OOD samples.
6.2 THE PREDICTIVE POWER OF IN-SIMILARITIES AND OUT-SIMILARITIES
Table reports the mean and variance of m and m _ and the rescaled mean ymm across all distributions. Interestingly. we see that out-similarities m better discriminate label-based OOD samples, while in-similarities m better discriminate adversarial perturbations. Combining in- similarities and out-similarities is thus an essential component to simultaneoush detect adversarial perturbations and unknown classes.
6.3 LIMITATIONS
Computational cost: To perform detection with CADet, need to compute the fea- tures for a certain number of transforma- tions of the test sample, incurring significant overhead. Figure 2 shows that reducing the number of transformations to minimize com- putational cost may not significantly affect performances. While the calibration step can be expensive, we note that it only needs
Table Mean and variance of m and mour
IN-1K
1Nat
IN-0
FGD
CW
FGSM
0,972
0.967
0.969
0.954
0.951
0,948
0.296
0.071
0.066
0,061
0.068
0.07
0.069
Var
8.3e-415
7.8-005
10c04
2.e-04
20c-04
2.lo-04
be run once for given in-distribution. The coefficient and scores are all one-dimensional values
that can be easily stored, and purposely use small number of validation samples | (x(2)) 300 to make their embedding easy to memorize.
self superviedd trastive learning is computationally expensive, only evaluated method on ResNet50 architecture. In Wang et al. (2022), the authors achieve significantly superior performances when using larger, recent architectures. The performances achieved with ResNet50 are in- sufficient for real-world usage, and the question of how our method would scale larger archi- tectures remains open.
Architecture scale:
97.5
96.0
92.5
ImageNet-O
AUROC
90.0
INaturalist
PGD
87.5
FGSM
82.5
6.4 FUTURE DIRECTIONS
80.0
While spurious correlations with background features problem in supervised learning. aggravated in self-supervised contrastive learning, where background features are highly relevant to the training task. We conjecture the poor performances of CADet on iNaturalist OOD detection are explained by the background similarities with ImageNet images, obfuscating the differences relevant features. A natural way to alleviate this issue is to incorporate background transformations to the training pipeline, as was successfully applied in Ma et al. (2018). This process would come the cost of being unable to detect shifts in background distributions, but such case is generally less relevant to deployed systems. We leave to future work the exploration of how background transformations could affect the capabilities of CADet.
number of transformations
Figure 2: AUROC score of CADet against the number of transformations.
6.5 CONCLUSION
We have presented CADet, a method for both OOD and adversarial detection based on self-supervised contrastive learning. CADet achieves excellent trade-off in detection power across different anomaly types. Additionally, we discussed how MMD could be leveraged with contrastive learning assess distributional discrepancies between sets of samples.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ahmed Abusnaina, Yuhang Wu, Sumpreet Arora, Yizhen Wang, Fei Wang, Hao Yang, and David Mohaisen. Adversarial example detection using latent neighborhood graph. International Conference on Computer Vision, 2021.
Liron Bergman and Yedid Hoshen. Classification-basee anomaly detection for general data, arXiv preprint TrXiv:2005.02355 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Symposium on Security and Privacy, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks. arXiv preprint wrX.iv:909.1129000 2019.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust
anomaly detection. arXiv preprint arXiv: 181000139,,
Kacper Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample testing with analytic representations of probability measures. arXiv preprint 150.::150000.7755, 2015.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference on machine learning and knowledge discovery in databases, 2018.
Laurent Dinh, Jascha bocksDick,,,,,, and Samy Bengio. Density estimation using real nvp. arXiv preprint wwXiv:1605.08800 2016.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, 2019.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.004110 2017.
Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han 0003, Tongliang Liu, Gang Niu 0001, and Masashi Sugiyama. Maximum mean discrepancy test aware of adversarial attacks. In Marina Meila and Tong Zhang 0001 (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 3564435755 PMLR, 2021. URL http://proceedings. .mw.iinnns.m....mmmmmmm.mm.mm... mww. mww.ppesss.vv1/ [gao21b. ht.ll
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: http:::412..6..... 2014.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:19/2.03263. 2019.
Arthur Gretton, Kenji Fukumizu, Zaid Harchaoui, and Bharath K. Sriperumbudur. A fast, consistent kernel two-sample test. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, 2009.
10

---

Under review as a conference paper at ICLR 2023
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In Computer Vision and Patten Recognition, 2006.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Conference Computer Vision and Pattern Recognition, 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-diisrriution examples in neural networks. arXiv preprint arXiv: 0600.000021111 0200022336 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint Xrxiv: /9/1. 132, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, 2019b.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.071127 2021.
Shengyuan Hu, Tao Yu, Chuan Guo, Wei-Lun Chao, and Kilian Q Weinberger. A new defense against adversarial images: Turning a weakness into a strength. In Advances in Neural Information Processing Systems, 2019.
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distributt detection for large semantic space. arXiv preprint arXiv:2105.01877 2021.
Wittawat Jitkrittum, Zoltan Szabo, Kacper Chwialkowski, and Arthur Gretton. Interpretable distribu- 2016.
tion features with maximum testing power. arXiv preprint
Umar Khalid, Ashkan Esmaeili, Nazmul Karim, and Nazanin Rahnavard. Rodd: A self- sulf-sueervieed approach for robust out-of-distribution detection. arXiv preprint urXiv:2204.025:: 2022.
Bernd Kitt, Andreas Geiger, and Henning Lategahn. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In 2010 IEEE Intelligent Vehicles Symposium, pp. 486-492, 2010. doi: 10.1
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribttion image detection in neural networks. arXiv preprint arXiv:/706.02690, 2017.
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J Sutherland. Learning deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning, 2020a.
Hao Liu and Pieter Abbeel. Hybrid discriminative- training via contrastive learning. arXiv preprint arXiv:2007.09070, 2020.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based outtt--diitributiin detection. In Advances in Neural Information Processing Systems, 2020b.
11

---

Under review as a conference paper at ICLR 2023
Weitang Liu, Xiaoyun Wang. John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. arXiv preprint arXiv:2010.03759, 2020c.
Julia Lust and Alexandru Paul Condurache. Gran: an efficient gradient-norm based detector for adversarial and misclassified examples. arXiv preprint arXiv:2004.09117 09179, 2020.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck,
Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint
2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 13XXv:1706.0660 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint 1rXiv:1702.04267, 2017.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distributin detection. In AAAI Conference on Artificial Intelligence, 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? arXiv preprint 1rX0v118/0.00916,, 2018.
Eric T Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint arXiv: 02994, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIS Workshop on Deep Learning and Unsupervised Feature Learning 201 1, 2011.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.0470 2018.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Conference Computer Vision and Pattern Recognition, 2019.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In Advances neural information processing systems, 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10871 2019.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, 2019.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International Conference on Machine Learning, 2018.
Thomas Schlegl, Philipp Seebück, Sebastian M. Waldstein, Ursula Schmidt-Erfune and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. arXiv preprint arXiv: arXiv: 1703.0591, 2017a
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative ad adversarial networks to guide marker discovery. In International conference on information processing in medical imaging. 2017b.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe- Shawe-Taylor, and John Platt. Support vector method for novelty detection. In Advances in neural information processing systems, 1999.
12

---

Under review as a conference paper at ICLR 2023
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self self-super outlier detection. In International Conference on Learning Representations, 2021. URL https
Eorummid=vse /
[XpmR8J
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-based generative models. arXiv preprint arXiv: 11480, 2019.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. arXiv preprint arXiv:2111. 2021.
Danica J. Sutherland, Hsiao-Yu Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. arXiv preprint / 04488, 2016.
Jihoon Tack, Sangwoo Mo. Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Anisie Uwimanal and Ransalu Senanayake. Out of distribution detection and adversarial attacks on deep neural networks for robust medical image analysis. arXiv preprint arXiv:2107.0488) 2021.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: But-of-distribution with virtual- logit matching. In Conference on Computer Vision and Pattenn Recognition, 2022.
Li Wenliang, Dougal Sutherland, Heiko Strathmann, and Arthur Gretton. Learning deep kernels for exponential family densities. In International Conference on Machine Learning. 2019.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Conference Computer Vision and Pattern Recognition, 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:/708 7708003888 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506 03365, 2015.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International Conference Machine Learning, 2016
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018
Fei Zuo and Qiang Zeng. Exploiting the sensitivity of adversarial examples to erase-and-restore. In Asia Conference Computer and Communications Security, 2021.
13


Reviews:
[{"review_id": "xN9UkdL7Iu0", "reviewer": "Reviewer_TDRY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present research on the intersection of outlier detection using MMD on two sets of samples and self-supervied pretraining.\nFirst of all they use self-supervision to pretrain a similarity function which is then to be used for outlier detection using MMD and derived statistics. \n\nThey present two improvements over Liu et al. 'Learning deep kernels for non-parametric two-sample tests.' when computing statistics. One is a better calibration of the p-value calculation. The other is a different way to compute the MMD statistic, in which they double the size of the validation set, and likely are able to achieve a lower variance of the p-scores if the test distribution is different from the validation distribution.\n\nThey show two results in these parts. One is stronger evidence for a distribution shift between CIFAR10 and CIFAR10.1 than shown in previous work. The second are results for outlier detection against sets of outliers taken form other datasets and obtained by adversarial optimization.\n\nThe third contribution they state as contrastive anomaly detection. This is meant to identify a single test sample whether it could be an outlier. \n\nIn this they take the test sample, and create a set of augmented variants from it, using augmentations similar to those used in self supervised pretraining. They do the same with a set of validation samples. Then they define a statistic somewhat reminiscent of the one used in MMD to obtain a score, which then is used to compute a p-value. The p-value is based on comparing scores computed between two augmented validation sets from the in distribution, and a score computed between one augmented validation set and one set obtained by augmenting the test sample.\n\nThey show results in Table 3 and 4 which are mixed for outlier detection and good for adversarial detection.\n", "summary_of_the_review": "The paper has two smaller statistical improvements over MMD. It advocates the use of self-supervision for outlier detection when combined with MMD. That is an insight.\n\nThe strongest issue with respect to acceptance in the eyes of this reviewer is the last contribution on single sample outlier detection which seems to depart from MMD without any motivation for doing so.\n\n*edit* after reading the rebuttal, the reviewer increased his score from 5 to 6 , \n\nhowever, if accepted to ICLR please clarify that it is inspired by MMD but differs in that it gives up / departs from the geometric interpretation used in MMD (when switching the sign) + provide the motivation stated in the rebuttal. ", "strength_and_weaknesses": "Strengths: \n1 well written paper\n2 experiments with out-of sample and adversarial data\n3 an improvement in detection in setups where one has a set of test samples due to the two statistical changes\n4 incremental novelty in combining self-supervision and MMD\n\nweaknesses:\n\n1 There might be problems in the formulation of the third contribution:\n\n1.1 First of all, a naming criticism: The similarity is fixed. This the authors also acknowledge themself. This is okay but it is unclear why it is named contrastive, as no change or training of similarity is performed based on the augmentations of the test sample. Yes, they initialize the similarity using contrastive learning, but the test sample is not involved in this first step. It is averaging over copies obtained by data augmentation, and thus not more contrastive than the original MMD with self-supervised initialization. \n\nContrastively-initialized outlier detection is more precise here.\n\n1.2 Secondly, and most importantly, equation (6) departs from the idea of an MMD score substantially. The MMD score as in eq (4) measures a difference between self-similarities and cross-similarities. Equation (6) performs an addition where MMD as in eq (4) would perform a subtraction.\n\nComparing to eq (4) also a self-similarity term between validation data is missing. For thresholding it can be omitted, but it would impact the variance of the estimators. However the main issue is the mentionned addition in place of substraction of the cross-similarity.\n\nequation (6) becomes by that a statistic without any motivation behind it. Why it is chosen as addition when (4) suggest a substraction ?\n\n1.3 Furthermore by matching equation (7) with algorithm 2 (on page 7) one can see that the calibration to compute gamma uses only data from the same in-distribution. But then asymptotically gamma should become 1, as m^{out} and m^{in} are computed over data drawn from the same distribution an thus their variance should be the same.\n\nIf gamma is not 1, as in Table 5, then something in the implementation seems to be unexpected (although gamma seems to be a constant across the datasets in Table 5).\n\n1.2 and 1.3 look as if they wanted to do something slightly different. gamma becomes non-trivial if it involves a mix out validation and augmented variants of the test sample.\n\n\n2 it would be good to try adversarial detection with other types of adversarial attacks, which are generated by different principles\n\n3 self-supervised outlier detection is not novel itself (https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf , https://arxiv.org/abs/2103.12051 ), but see strengths, point 4", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wzn_-YpPAmg", "reviewer": "Reviewer_FoD6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel out-of-distribution detection algorithm inspired by MMD-based two-sample testing. The paper first shows that contrastive learning, when used with MMD two-sample test, can be highly effective in recognizing distribution shifts in image statistics. Then, the paper extends this finding to build an out-of-distribution detection algorithm that can detect images that contain an object that is not in the training set or is adversarially modified to fool an image classifier.", "summary_of_the_review": "It is a well-written paper with algorithmically novel contributions. It would be a more appealing paper if the performance in Table 3 was stronger and the fair comparisons were made in Table 4.", "strength_and_weaknesses": "# Strength\n\n* As far as I know, the application of MMD-based two-sample testing on OOD detection is a novel approach. There have been some efforts to apply the two-sample testing idea to OOD detection, but they were not as successful as this paper.\n* The paper is well-written. The expositions are clear and easy to follow.\n* The paper does a good job of summarizing the existing literature and providing the technical information required to reproduce the experiments.\n* The paper provides a helpful discussion on the limitations and future directions of the work in Section 6.3.\n\n# Weaknesses\n\nI think the followings are more like questions rather than weaknesses.\n\n* (\"Supervised\" adversarial attack) Although it is stated in Section 5.1, it would be nicer to emphasize that the adversarial samples considered in this paper are generated with respect to a supervised classifier. The reason is that an adversarial sample can also be generated with respect to an OOD detector, and OOD detectors are known to be very vulnerable to such attacks, see for example, https://arxiv.org/abs/2106.04260 . Nonetheless, limiting the scope of the paper to the \"supervised\" adversarial attack would not be the reason for rejection, I think.\n* (Adversarial detection performance of contrastive representation) I am worried that the comparison between supervised representation and contrastive representation in Table 4 is not fair, because these adversarial samples are generated with respect to the supervised network being tested. From the information provided in the paper, it is not clear whether the attack would transfer to a network trained with contrastive learning. Even though it translates well, the attack is usually the strongest against a network the attack is targeting. Therefore, a supervised network can be said to be handicapped. I believe it is a little bit too early to conclude that contrastive representation is more effective in adversarial detection. A few more supporting arguments should help resolve this issue.\n* (The number of inlier samples in testing) For me, it is surprising that so few inlier samples are needed to perform successful two-sample testing with respect to outlier classes, and I wonder if there is a good explanation regarding this. Naively, if there are 1,000 classes in the in-distribution dataset, shouldn't we need at least 1,000 samples that represent each class? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pk-_vLd5CY", "reviewer": "Reviewer_txxW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes two sample test method that measures the maximum mean discrepancy (MMD) of two sample sets for out-of-distribution sample set detection. The main idea is to use self-supervised features for MMD. On top of the self-supervised features, the paper proposes a method called MMD-CC (MMD with clean calibration) that reduces the variance of MMD in the cost of computations which is effective when sample set sizes are small.\nThen, the authors propose a method to apply MMD and MMD-CC to a single outlier instance detection (anomaly detection) by simulating a two sample test on a single test sample by populating the test sample with multiple transformations (augmentations).\nTwo terms are used to measure anomaly score; intra similarity score which measures the similarity between transformed test samples, and out similarity score which measures the similarity between transformed training and test samples. The calibration step is proposed to find a balancing hyper-parameter between the two similarity score terms. Instead of using the similarity score directly as an anomaly score, the paper proposes to perform hypothesis testing and use the p-value as an anomaly score. The proposed methods are evaluated on 4 datasets for out-of-distribution detection tasks (a task that assumes a set of test samples are given), and evaluated on two image datasets and 3 adversarial scenarios (on the ImageNet validation set) for anomaly detection tasks (single instance detection).\n", "summary_of_the_review": "The paper proposes a two-sample-test-based out-of-distribution detection and anomaly detection.\nThe experiments are extensive with meaningful results. Although there are a few missing ablation studies and details, the paper is well-written overall.\n", "strength_and_weaknesses": "## Strength\n\n### Various experiments support the effectiveness of the method\n\n### Diverse applications\n\nOut-of-distribution detection from two sample sets, single instance anomaly detection, and adversarial attach detections are shown in the experiment section.\n\n### Diverse datasets are used for the experiments.\n\nFour datasets are used for out-of-distribution detection tasks (tasks that assume a set of test samples are given).  Two image datasets are used for anomaly detection.\nThree adversarial scenarios are shown (on the ImageNet validation set) for anomaly detection tasks (single outlier instance detection).\n\n### Ablation studies\n\nSome necessary ablation studies are presented in the paper including varying sample set sizes (Figure1 and Table1), and the number of transformations used for anomaly detection (figure 2).\n\n### The limitations of the paper are addressed.\n\n\n\n## Weakness\n\n### Computational cost\n\nThe paper also addresses their computational cost requirement. Since two sample test is a statistical test on two sample sets, their computation requires quadratic order in terms of sample sizes. Moreover, the paper proposes to repeat this test multiple times to reduce the MMD variance.\n\n### Pretraining on a large-scale dataset required\n\nThe method relies on well trained feature representation. The experiments are performed on a pre-trained ResNet 50 or a self-trained ResNet 50.\n\n### Missing ablation studies\n\n1 Computing calibration parameter gamma in Eq (7) costs n_trs * n_trs * |X_val1| * ||X_val2| computations. How much is this calibration process important? What is the performance gain from this process? What if gamma=1 is used without any calibration process?\n\n2 The paper uses hypothesis testing instead of directly using the score in equation (8). How much performance gain was obtained from this?\n\n### (Minor point) The method may be sensitive to different transformations.\n\nIn the first paragraph of section 5.1, transformations except ‘color jittering, gaussian blur, and gray scaling’ are used. What is the rationale behind this? Is the method sensitive to different types of transformations?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "juAQZ44s0t", "reviewer": "Reviewer_qf3u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the detection of out-of-distribution (OOD) instances (novel classes and adversarial attacks) in vision using similarities learned via self-supervised contrastive learning (SimCLRv2) with a maximum mean discrepancy (MMD) two-sample test. This approach is first used to show that there exists a significant distributional shift between CIFAR10 and CIFAR10.1 (which has been proposed as a corrected update of CIFAR10). Then, an extension of the approach for single samples is proposed via using augmentations of a sample to create a set, called CADet (Contrastive Anomaly Detection). An experimental evaluation on OOD detection of (i) novel classes (ImageNet vs. iNaturalist, ImageNet-O) and (ii) adversarial examples (generated with PGD, CW, and FGSM attacks) shows that CADet performs similar on detecting unknown classes and favorably on detecting adversarial attacks over previous methods.", "summary_of_the_review": "The paper includes interesting findings regarding differences between using self-supervised vs. supervised embeddings for OOD, in particular wrt adversarial examples, but is limited in providing further analysis and insights into why one might expect the shown improvements in detecting adversarial examples.", "strength_and_weaknesses": "+ I find it interesting to see that self-supervised embeddings improve adversarial attack detection over using supervised embeddings, also for other previous methods. I'm not aware of previous work making this observation.\n+ The paper is well integrated into the existing literature, making a comparison to recently established OOD methods.\n\n- Overall, I find the experimental results to be not too convincing. For detecting novel classes, the proposed method does not show a consistent improvement over previous methods (the supervised embeddings also being better over the self-supervised one in 2/3 cases). The adversarial examples detection is more convincing, which I think should be extended/elaborated on.\n- While CADet empirically seems to work particularly well for adversarial attacks, I am missing some explanation or intuition on why this might be the case. I think the paper might be stronger focusing and digging deeper on this case, since the differences over previous methods here are intriguing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject