CADet: Fully Self-Supervised Out-Of-Distribution Detection With Contrastive Learning
Authors: Charles Guille-Escuret
Pau Rodriguez
David Vazquez
Ioannis Mitliagkas
Joao Monteiro
Venue: NeurIPS 2023 poster (2023)
Keywords: Contrastive learning
OOD detection
adversarial detection
MMD
ImageNet-O
Anomaly detection
CIFAR-10.1
TL;DR: We leverage self-supervised contrastive learning first with MMD two-sample test to discriminate between two distribution, then in a novel method called CADet for detection of both novel classes and adversarial attacks on single samples.

Abstract: Handling out-of-distribution (OOD) samples has become a major stake in the real-world deployment of machine learning systems. This work explores the use of self-supervised contrastive learning to the simultaneous detection of two types of OOD samples: unseen classes and adversarial perturbations. First, we pair self-supervised contrastive learning with the maximum mean discrepancy (MMD) two-sample test. This approach enables us to robustly test whether two independent sets of samples originate from the same distribution, and we demonstrate its effectiveness by discriminating between CIFAR-10 and CIFAR-10.1 with higher confidence than previous work. Motivated by this success, we introduce CADet (Contrastive Anomaly Detection), a novel method for OOD detection of single samples. CADet draws inspiration from MMD, but leverages the similarity between contrastive transformations of a same sample. CADet outperforms existing adversarial detection methods in identifying adversarially perturbed samples on ImageNet and achieves comparable performance to unseen label detection methods on two challenging benchmarks: ImageNet-O and iNaturalist. Significantly, CADet is fully self-supervised and requires neither labels for in-distribution samples nor access to OOD examples.

CADet: Fully Self-Supervised Anomaly Detection With Contrastive Learning
Charles Guille-Escuret ServiceNow Research, Mila, Université de Montréal gui guillech@mile l quebee
Pau Rodriguez
ServiceNow Research
pau. puu.roddigguzzbeee..cooo..cm
David Vazquez
ServiceNow Research
david. vazquezr@sericceoor.com
Toannis Mitliagkas
Mila, Université de Montréal,
Canada CIFAR AI chair
ioannis la quebec
Joao Monteiro
ServiceNow Research
joao www.teiroodserviienoo.coo
Abstract
Handling out-of-distribution (OOD) samples has become major stake in the real-world deployment of machine learning systems. This work explores the use of self-supervised contrastive learning the simultaneous detection of two types of OOD samples: unseen classes and adversarial perturbations. First, pair self-supervised contrastive learning with the maximum mean discrepancy (MMD) two-sample test. This approach enables us to robustly test whether two indepen- dent sets samples originate from the same distribution, and we demonstrate its effectiveness by discriminating between CIFAR- 10 and CIFAR- 10.1 with higher confidence than previous work. Motivated by this success, we introduce CADet (Contrastive Anomaly Detection), novel method for OOD detection of single samples. CADet draws inspiration from MMD, but leverages the similarity be- tween contrastive transformations of same sample. CADet outperforms existing adversarial detection methods identifying adversarially perturbed samples on ImageNet and achieves comparable performance unseen label detection methods on two challenging benchmarks: ImageNet-O and iNaturalist, Significantly, CADet is fully self-supervised and requires neither labels for in-distribution samples nor access to OOD examples
Introduction
While modern machine learning systems have achieved countless successful real-world applications, handling out-of-distributin (OOD) inputs remains a tough challenge of significant importance. The
Our code to compute CADet scores is publicly available as an OpenOOD fork at at/ttps:: htttt:::///gthb..... com/ char werrresGG//iie SassG//Opennn -ttp://pppOOOOOOOO00 CADet
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

problem is especially acute for high- nigh-dimensional problems like image classification. Models are typically trained in a world setting but inevitably faced with novel input classes when deployed in the real world. The impact can range from displeasing customer experience to dire consequences in the case of safety-critical applications such as autonomous driving [31] or medical analysis [55]. Although achieving high accuracy against all meaningful distributional shifts is the most desirable solution, it is particularly challenging. An efficient method to mitigate the consequences of unexpected inputs is to perform anomaly detection, which allows the system to anticipate its inability to process unusual inputs and react adequately.
Anomaly detection methods generally rely on one of three types of statistics: features, logits, and softmax probabilities, with some systems leveraging mix of these [66]. An anomaly score f f() is computed, and then detection with threshold I is performed based on whether f(x) > т. The goal of a detection system is find an anomaly score that efficiently discriminates between in-distribution and out-of-distribution samples. However, the common problem of these systems is that different distributional shifts will unpredictably affect these statistics. Accordingly, detection systems either specific types of distributions or require tuning on OOD samples. In
achieve good performance both cases, their practical use is severely limited. Motivated by these issues, recent work has tackled the challenge of designing detection systems for unseen classes without prior knowledge of the unseen label set or access to OOD samples 68 63 66].
We first investigate the use of maximum mean discrepancy two-sample test (MMD) 19 in con- junction with self-supervised contrastive learning to assess whether two sets of samples have been drawn from the same distribution. Motivated by the strong testing power of this method, we then introduce a statistic inspired by MMD and leveraging contrastive transformations. Based on this statistic, we propose CADet (Contrastive Anomaly Detection), which is able to detect OOD samples from single inputs and performs well on both label-based and adversarial detection benchmarks, without requiring access to any OOD samples to train tune the method.
Only few works have addressed these tasks simultaneously. These works either focus on particular in nn-distrbuttio data such as medical imaging for specific diseases (65) or evaluate their performances
on datasets with very distant classes such as CIFARIO (32), SVHN [47, and LSUN (73). resulting in
simple benchmarks that do not translate to general real world applications 33 5T
Contributions Our main contributions
follows:
We use similarity functions learned by sulf-supervised contrastive learning with MMD to show that the test sets of CIFARIO and CIFARIO.1 (52) have different distributions.
We propose novel improvement to MMD and show it can also be used to confidently detect distributional shifts when given a small number of samples.
We introduce CADet, fully self-supervised method for OOD detection inspired by MMD, and show it outperforms current methods in adversarial detection tasks while performing well on label-based OOD detection.
The outline is as follows: in Section 2 we discuss relevant previous work. Section 3 describes the self-supervised contrastive method based on SimCIRv2 (5) used in this work. Section the application of learned similarity functions in conjunction with MMD to verify whether two independent sets of samples are drawn from the same distribution. Section 5 presents CADet and evaluates its empirical performance. Finally, we discuss results and limitations in Scction 6
4 exxplores
2 Related work
We propose self-supervised contrastive method for anomaly detection (both unknown classes and adversarial attacks) inspired by MMD. Thus, our work intersects with the MMD, label-based OOD detection, adversarial detection, and self-supervised contrastive learning literature.
MMD two- sample test has been extensively studied [9 of our knowledge, the first time a similarity function trained via contrastive learning is used in conjunction with MMD. Liu et al. (35) uses MMD with deep kernel trained on a fraction of the samples to argue that CIFAR 10 and CIFAR 10.1 have different test distributions. We build upon that work by confirming their finding with higher confidence levels, using fewer samples. Dong al. explored applications of MMD to OOD detection.
18 29), though is, to the best

---

Label-based OOD defection methods discriminate samples that differ from those in the training distribution. We focus on unsupervised OOD detection in this work, i.e., we do not assume access to data labeled as OOD. Unsupervised OOD detection methods include density-based [74 45 46
12 53 59 self-supervised [5
17
37
10
reconstruction-basee
[56 75
[ 50 49 77, one-class classifiers 5
25
2 63 and supervised approaches [34 23 though some works do not fall
into any of these categories 666, 60). We refer to Yang et al.
7 for an overview of the many recent
works in this field.
Adversarial detection discriminates adversarial samples from the original data. Adversarial samples are generated by minimally perturbing actual samples to produce a change in the model's 's output. such as misclassification. Most works rely on the knowledge of some attacks for training 39 76 48 40), with the exception of [27].
13
Suue-supe contrastive learning methods 69 22
are commonly used to pre-train a model from unlabeled data to solve a downstream task such as image classification. Contrastive learning relies on instance discrimination trained with a contrastive loss [|] such as infoNCE [20].
Contrastive learning for OOD detection aims to find good representations for detecting OOD
samples in supervised (36 30 unsupervised [68 44 58) setting. Perhaps the closest work in the literature CSI 663, which found SimCLR features to have good discriminative power for unknown classes detection and leveraged similarities between transformed samples in their score. However, this method is not well-suited for adversarial detection. CSI ignores the similarities between different transformations of same sample, essential component to perform adversarial detection (see Section 6.2). In addition, CSI scales their score with the norm of input representations. While efficient samples with unknown classes, is unreliable on adversarial perturbations, which typically increase representation norms. Finally, we failed to scale CSI to ImageNet.
3 Contrastive model
We build our model on top of SimCLRv2 [5] for its simplicity and efficiency. It is composed of an encoder backbone network fo well as 3-layer contrastive head hgr. Given an in-distribution sample X, a similarity function sim, and a distribution of training transformations Terains the goal is to simultaneously maximize
Eo [sim(her fo(to(z)), ha fo(ti(z))))
and minimize
to Tevain [sim(hg o Jo(lo(z)),, hg (o)((
ie., we want to learn representations in which random transformations of same example are close while random transformations of different examples are distant.
To achieve this, given an input batch {a:].=......, we compute the set {af (1) )= by applying two transformations independently sampled from Terein to each T;. We then compute the embeddings 2(") her fo (o) and apply the following contrastive loss:
I(z)=
Σ log Σ
Vi Ny (ui + Uz
Eje
where
Mi.j sim(s, 1/T and U1.j 1 g )/T
is the temperature hyperparameter and sim(r, u) 1 |T
(m/y) is the cosine similarity.
Hyperparameters: We follow as closely as possible the setting from SimCLRv2 with few modifi- cations to adapt to hardware limitations. In particular, we use the LARS optimizer [72) with learning rate 1.2, momentum 0.9, and weight decay 10-4, Iteration- the first 40 epochs linearly, then use an iteration- ceration-wise cosine decaying schedule until epoch 800, with temperature - 0.1. We train on 8 V100 GPUs with an accumulated batch size of 1024. We compute the contrastive loss on all batch samples by aggregating the embeddings computed by each GPU. We use synchronized NarchNorm and fp32 precision and do not use a memory buffer. We use the same set of transformations, i.c., Gaussian blur and horizontal flip with probability 0.5, color
we scale up the learning rate for
3

---

jittering with probability 0.8, random crop with scale uniformly sampled in [0.08, 1], and grayscale with probability 0.2.
For computational simplicity and comparison with previous work, we use a ResNer50 encoder architecture with final features of size 2048. Following SimCLRv2, we use a three-layer fully connected contrastive head with hidden layers of width 2048 using ReLU activation and batchNorm and set the last layer projection to dimension 128. For evaluation, we use the features produced by the encoder without the contrastive head. We do not, at any point, use supervised fine-tuning.
4 MMD two-sample test
The Maximum Mean Discrepancy (MMD) is statistic used in the MMD two-sample test to assess whether two sets of samples Sp and Sg are drawn from the same distribution. It estimates the expected difference between the intra-set distances and the across-sets distances.
Definition 4.1 (Gretton al. [19]] Let k X' X + R. be the kernel of reproducing Hilbert space Ha. with feature maps k k(,,,r) He. Let X, X and Y, Y" Q. Under mild integrability conditions,
(MD(P,Q:HA) = sup
[B[[(X] - B [f()]]
(2)
SEH.1Sll
XXX + 2k(X, Y)
(3)
Given two sets of n samples Sp = (ii and Se = (Yi)ign respectively drawn from and Q. we can compute the following unbiased estimator (35
MMD k) Se: n(n 1)
(k(X,, x,) A(Y. V,) k(,,, Y,) A(Y,
(4)
Under the null hypothesis ho P Q this estimator follows normal distribution of mean O 19). Its variance can be directly estimated [18]. but it is simpler to perform a permutation test suggested in Sutherland et al. 622.. which directly yields p-value for ha- The idea is to random splits X, of the input sample sets to obtain Mperm different (though not independent)
samplings of M M k),, which approximate the distribution of M M Sp; k) under the null hypothesis.
Liu et al. 355 trains deep kernel to maximize the test power of the MMD two-sample on training split of the sets of samples to test. We propose instead to use our learned similarity function without any fine-tuning. Note that we return the p-value Apern+I (1 Σ Npprr
1 (p, 2 est)) instead (p: 2 est). Indeed, under the null hypothesis IP = Q, est and p. drawn from the same distribution, so for j E (0, 1,.., Rperm }. the probability for est to be smaller than exactly j elements of {p.} is Tiperm +1' Therefore, the probability that j elements or less of {p:), ) are larger
of Elperm
Aperr
than est is E/=0 Agerm Mpperm
While this change has a small impact for large values of pperm it is essential to guarantee that we indeed return a correct p-value. Notably, the algorithm of O to return output of 0.00 even under the null hypothesis.
j+1
Liu et al. 35 has probability Ryerm
Additionnally, we propose novel version of MMD called MMD-CC (MMD with Clean Calibration). Instead of computing p: based on random splits of Se So, we require as input two disjoint sets of samples drawn from P and compute p; based on random splits of ((1) U ((2)
S(²2)) (see Algorithm 1p.
This change requires to use twice as many samples from P., but reduces the variance induced by the
random splits of Sp USg which is significant when the number of samples is small. Note that s(") S(2) and Sg must always have the same size. Under the null hypothesis, = Q. s(") s(2) and So are identically distributed, so the pæ conserve the same distribution as for MMD (this is not the case when P o Q, hence the difference testing power). Thus, the validity of MMD-CC follows from the validity of MMD.

---

1.0
10
0.-
08-
0.6
06
samples
samples
ImagelietOO
samples
samples
1000 samples
CW
2000 samples
FGSM
00
random
0.
random
0.0
02
0.4
0.6
0.8
10
0.0
0.2
04
0.6
0.8
10
false positive rate
taise positive rate
(a) ROC curves of MMD two-sample test on CIFAR- (b) ROC curves of CADet on different distribution
10.1 against CIFAR- 10 for different sample sizes.
shifts of ImageNet.
Algorithm MMD-CC two-sample test
Input: 5 ( Sq pperms sim
est <- MM 2 (1)
So: sim)
for i = 1,2,..., perm do
Randomly split Sp" US$
(2) into two disjoint sets X, Y of equal size
<
end for
Output: p: I+Pperm (1+ Ei
Pperm 1 (p, 2 est))
4.1 Distribution shift between CIFAR-10 and CIFAR-10.1 test sets
After years evaluation popular supervised architectures on the test set of CIFAR-10 [32 modern models may overfit it through their hyperparameter tuning and structural choices. CIFAR-IO. 10.1 [52]
was collected to verify the performances of these models on truly independent sample from the training distribution. The authors note consistent drop in accuracy across models and suggest could be due distributional shift, though they could not demonstrate it. Recent work (35] leveraged the two-sample test to provide strong evidence of distributional shifts between the test sets CIFAR- and CIFAR-1O.1. We run MMD-CC and MMD two-sample tests for 100 different
samplings of s()) .s(²), Sg. using every time Rpperm = 500, and rejecting hu when the obtained p-value is below the threshold a 0.05. We also report results using cosine similarity applied to the features of supervised models as comparative baseline. We report the results in Table for range of sample sizes. We compare the results to three competitive methods reported in Liu al.
35:: Mean embedding (ME) g 29 MMD-D 35 and C2ST-L (6). Finally, we show in Figure la ROC curves of the proposed model for different sample sizes.
Other methods in the literature do not use external data for pre-training, as we do with ImageNet, noteworthy that our learned similarity can
which makes a fair comparison difficult. However, very confidently distinguish samples from the two datasets, even in settings with fewer samples available. Furthermore, while achieve excellent results even with supervised network, our model trained with contrastive learning outperforms the supervised alternative very significantly. We note however that with such high number of samples available, MMD-CC performs slightly worse than MMD. Finally, we believe the confidence obtained with our method decisively concludes that CIFARIO and CIFARI0.1 have different distributions, which likely the primary explanation for significant drop in performances across models CIFAR
110.1 10.1, as conjectured by Recht et al. [22 The difference in distribution between CIFAR10 10 and CIFAR 10.1 is neither based on label nor adversarial perturbations, making it interesting task.

---

Table 1: Average rejection rates of hu CIFAR-I0 vs CIFAR-10.1 for 0.05 across different sample sizes n, using ResNet50 backbone.
n=2000
n=1000
n=500
n=200
n=100
n=50
ME 8
0,588
C2ST-L
0.529
MMD-D
35
0.744
MMD SimCLRv2 (ours)
1.00
1.00
0.997
0.702
0.325
0.154
MMD-CC + SimCLRv2 (ours)
1.00
1.00
0.997
0.686
0.304
0.150
MMD Supervised (ours)
1.00
1.00
0.884
0.305
0.135
0.103
MMD-CC Supervised (ours)
1.00
1.00
0.870
0.298
0.131
0.096
Table AUROC for detection using two-sample test on to 20 samples drawn from ImageNet and from ImageNet-O,, iNaturalist or PGD perturbations, with a ResNet50 backbone.
ImageNet-O
iNaturalist
PGD
samples
10
20
3
20
MMD SimCIRR2
64.3
72.4
86.9
97.6
88.3
97.6
99.5
99.5
35.2
53.8
86.6
98.8
MMD-CC SimCLRv2
65.3
73.2
88.0
97.7
95.4
99.2
99.5
99.5
70.5
84.0
96.6
99.5
MMD Supervised
62.7
69.7
83.2
96.4
91.8
98.7
99.5
99.5
20.0
22.5
33.0
57.5
MMD-CC+ + Supervised
62.6
71.0
85,5
97.2
98.0
99.5
99.5
99.5
57.4
61.3
70.5
85.8
4.2 Detection of distributional shifts from small number samples
Given a small set of samples with potential unknown classes or adversarial attacks, we can similarly use the two-sample test with our similarity function to verify whether these samples are in-distribution [4.. In particular, we test for samples drawn from ImageNet-O, O, iNaturalist, and PGD perturbations, with sample sizes ranging from 3 to 20. For these experiments, sample s(1) and Sf 5000 times across all ImageNet's validation set and compare their MMD and MMD-CC estimators to the one obtained from Sp and Sg. We report in Table 2 the AUROC of the resulting detection and compare it to the ones obtained with a supervised ResNet50 as the bascline.
Such a setting where we use several samples assumed to be drawn from a same distribution to perform detection is uncommon, and we are not aware of prior baselines in the literature. Despite using very few samples (3 < n < 20), our method can detect OOD samples with high confidence. We observe particularly outstanding performances on iNaturalist, which is easily explained by the fact that the subset we using (cf. Section[] [] only contains plant species, logically inducing abnormally high similarity within its samples. Furthermore, we observe that MMD-CC performs significantly better than MMD, especially on detecting samples perturbed by PGD.
Although our method attains excellent detection rates for sufficient numbers of samples, the require- ment to have set of samples all drawn from the same distribution to perform the test makes it unpractical for real-world applications. In the following section, we present CADet, detection method inspired by MMD but applicable to anomaly detection with single inputs.
5 CADet: Contrastive Anomaly Detection
While the numbers in Section 4 demonstrate the reliability of two-sample test coupled with contrastive learning for identifying distributional shifts, it requires several samples from the same distribution, which is generally unrealistic for practical detection purposes. This section presents CADet, method to leverage contrastive learning for OOD detection on single samples.
superviedd contrastive learning trains similarity function s to maximize the similarity between augmentations of the same sample, and minimize the similarity between augmentations of different samples. Given an input sample teat, we propose leverage this property to perform anomaly detec- tion on feat taking inspiration from MMD two-sample test. More precisely, given a transformation distribution Teat, we compute 7hr random transformations zfest of Erests as well as Mire random
(1) We then compute the transformations z; on each sample (+) of held-out validation dataset X val.
(k)

---

intra-similarity and out-similarit:
Σ Es 20(())
saafeot saje (4.
Σ
xleex !!) 1d
m² (xtest)
and m out (slest)
(5)
Terrs (Rers + 1)
nhro x (x vall
We finally define the following statistic to perform detection:
scorec == m. + +ym. out
(6)
Intuition: note that while CADet is inspired by MMD, it has some significant differences. min is computed across transformations of a same sample, while Mout is computed across transformations and across samples. Therefore, even under the null hypothesis, Min and Mout cannot be expected to have the same distribution. Besides, both score can be expected to be smaller on OOD samples. Indeed, OOD samples can be expected to have representations further from clean data in average, thus decreasing Mout- Furthermore, since the self-supervised model was trained to maximize the similarity between transformations of same sample, it reasonable to expect that the model will perform this task better in-distribution than out-of -ittriiiittibttionn, and thus that min will be higher in-distribution than on OOD data. This intuition justifies adding mi to mout. instead of the subtraction operated by MMD. Finally, note that we do not consider the intra-similarity between validation samples: the validation is fixed, so is necessarily a constant.
since we do not assume knowledge of OOD samples, it is difficult priori to tune although crucial to balance information between intra-sample similarity and cross-sample similarity. As workaround, we calibrate y by equalizing the variance between m in and ym on second set of validation samples x(2).
Calibration:
Var (m)mm())) E X(2)
[ar(mout (x), E x!2))
It is important note that the calibration of y is essential, because min and Mout have very different means and variances. Since Mout measures the out-similarity between transformations of different samples. it can be expected to be low in average. with high variance. On the contrary, min measures the similarity between transformations of a same sample, and can thus be expected to have higher mean and lower variance, which is confirmed in Table 5 Therefore, simply setting 7 would likely results in one of the score dominating the other.
Rather than evaluating the false positive rate (FPR) for a range of possible thresholds т, we use the hypothesis testing approach to compute the p-value:
Hx E x s.t. scorec(z) (2 ++
(8)
val x
+1
The full pseudo-codes of the calibration and testing steps are given ins Appendix A Setting
threshold p E [0, 1] for P. will result FPR of mean p, with a variance dependant of vall
x(2)
Section 5.1 further describes our experimental setting.
5.1 Experiments
For all evaluations,
the same transformations SimCLRv2 except color jittering, Gaussian blur and grayscaling. We fix the random crop scale 0.75. We use ((x(2) val } 2000 in-distribution
samples, (x(2)) 300 separate samples to compute cross-similarities, and 50 transformations per sample. We pre-train a ResNee55 with ImageNet as in-distribution.
Unknown classes detection: we use two challenging benchmarks for the detection of unknown classes. iNaturalist using the subset in 28 made of plants with classes that do not intersect ImageNet. Wang et al. (66] noted that this dataset is particularly challenging due to proximity of its classes. We
also evaluate on ImageNet-O 26); explicitly designed to be challenging for OOD detection with
ImageNet in-distribution. We compare to recent works and report the AUROC scores in Table 3
7

---

Table 3: AUROC for OOD detection on ImageNet-O and iNaturalist with ResNet50 backbone.
Training
iNaturalist
ImageNet-O
Average
MSP
23
88.58
56.13
72.36
Energy 38
80.50
53.95
67.23
ODIN 34
86.48
52.87
69,68
MaxLogit 24
86.42
54.39
70,41
KL Matching [24]
Supervised
90,48
67.00
78.74
ReAct [6]]
87.27
68.02
77.65
Mahalanobis 33
89.48
80.15
84.82
Residual 666
84.63
81.15
82.89
ViM 66
89.26
81.02
85.14
Supervised Self-supervised
95.28
70.73
83.01
CADet (ours)
83.42
82.29
82.86
(contrastive)
Table AUROC adversarial detection on ImageNet against PGD, CW and FGSM attacks, with
ResNet50 backbone.
Tuned on Adv
Training
PGD
CW
FGSM
Average
Supervised
62.30
60.29
68.10
63.56
ODIN
34
Yes
Contrastive
59.91
60.23
64.99
61.71
Supervised
91.39
87.77
95.12
91.43
FS
Contrastive
94.71
88.00
95.98
92.90
Supervised
93.13
91.08
93.55
92.59
LID
Yes
Contrastive
91.94
89.50
93.56
91.67
Supervised
84.31
84.29
77.95
82.18
Hu 27
Yes
Contrastive
94.80
95.19
78.18
89.39
Supervised 66.40
59.58
71.02
65.67
Hul 27
+ self-calibration
No
Contrastive
75.69
75.74
69.20
73.54
Supervised 92.71
91.01
96.92
93.55
Mahalanobis 33
No
Contrastive 84.14
83.78
79.90
82.61
Supervised 75.25
71.02
83.45
76.57
CADet (ours)
No
Contrastive
94.88
95.93
97.56
96.12
For Mahalanobis [3], since tuning on ood samples is not permitted, we use as score the mahalanobis score the last layer as done ViM [66] Furthermore, isolate the effect of the training scheme from the detection method, we present in Appendix the performance of previous methods using the same supervseee backbone CADet, conjunction with a trained linear layer.
Adversarial detection: for adversarial detection, we generate adversarial attacks on the validation partition of ImageNet against pre-trained ResNet50 using three popular attacks: PGD [2]. CW (3) and FGSM [66]. We follow the tuning suggested by Abusnaina et al. 1. i.e. PGD: norm To 5 0.02, step size 0.002, 50 iterations; CW: norm Le. 8 0.10, learning rate of 0.03, and 50 iterations: FGSM: norm Loc = 0.05. We compare our results with ODIN (34), which achieves good performances in Lee et 33 despite being designed for adversarial detection. Feature Squeezing (FS) 770), Local Intrinsinc Dimensionality (LID) 41). Hu et al. [27]. and Mahalanobis 33 (using the mahalanobis score of the last layer as for unknwon classes detection). Most existing adversarial detection methods assume access to adversarial samples during training (see Section 2). additionally propose a modification Hu et 27 to perform auto-calibration based on mean and variance of the criterions clean data, similarly to CADet's calibration step. We report the AUROC scores in Table 4 and illustrate them with ROC curves against each anomaly type in
Figure lb

---

Table 5: Mean and variance of m m and m ou.
IN-IK
1Nat
IN-0
PGD
CW
FGSM
0.972
0.967
0.969
0.954
0.954
0.948
Mean
0.321
0.296
0.275
0.306
0.302
0.311
7ym
0.071
0.066
0.061
0.068
0.067
0.069
7.8e-05
2.1e.44
2.0e-04
2.Te-04
Var
1.7e-03
7.0c-04
2.3
7.0c00
1.7e-03
1.1e-03
Discussion
6.1 Results
CADet performs particularly well on adversarial detection, surpassing alternatives by a wide margin. We argue that self-supervised contrastive learning is suitable mechanism for detecting classification attacks due to its inherent label-agnostic nature. Interestingly. Hu et al. (27] also benefits from contrastive pre-training, achieving much higher performances than with supervised backbone. However, it is very reliant on calibrating adversarial samples, since we observe a significant drop performances with auto-calibration.
We explain the impressive performances of CADet on adversarial detection by the fact that adversarial perturbations will negatively affect the capability of the self-supervised model to match different transformations of the image. In comparison, the model has high capability on clean data, having been trained specifically with that objective.
While CADet does not outperform existing methods on label-based OOD detection, performs comparably, impressive feat of generality considering adversarially perturbed samples and novel labels have significantly different properties, and that CADet does not require any tuning on OOD samples.
Notably, applying CADet to supervised network achieves state-of-the-arr performances on iNatural- ist with ResNet50 architecture, suggesting CADet can be reasonable standalone detection method on some benchmarks, independently of contrastive learning. addition, the poor performances of the supervised network on ImageNet-O) and adversarial attacks show that contrastive learning is essential address the trade-off between different type of anomalies.
Overall, our results show CADet achieves an excellent trade-off when considering both adversarial and label-based OOD samples.
6.2 The predictive power of in-similarities and ouf-similarities
Table reports the mean and variance of m and m out and the rescaled mean yymout across all distributions. Interestingly, we see that out- similarities out better discriminate label-based OOD samples, while in ai-iiillassiess m better discriminate adversarial perturbations. Combin- ing in- in-similarities and out-similarities is thus essential component to simultaneously detect adversarial perturbations and unknown classes,
97.5
95.0
ImageNet-O
AUROC
90.0
Naturalist
PGD
87.5
CW
85.0
82.5
6.3 Limitations
80.0
Computational cost: To perform detection with CADet, we need compute the features for certain number of transformations of the test sample, incurring significant overhead. Figure 2 shows that reducing the number of transforma- tions to minimize computational cost may not significantly affect performances. While the cal- ibration step can seem expensive, only required less than minutes on a single A100 GPU, which is far from prohibitive. Morcover, it only needs to be run once for given in in-distribution. The
20
40
60
80
100
120
140
number of transferraatins
Figure 2: AUROC score of CADet against the number of transformations.

---

coefficient and scores are all one-dimensional values that can be easily stored, and we purposely use small number validation samples H-x 300 to make their embedding easy memorize.
Architecture scale: as self-supervised contrastive learning computationally expensive, we only evaluated our method ResNet50 architecture. In Wang et al. [66], the authors achieve significantly superior performances when using larger, recent architectures. The performances achieved with ResNet50 are insufficient for real-world usage. and the question of how method would scale to larger architectures remains open.
Adaptive attacks: we do not evaluate on adaptive attacks, and as such our detection method should be considered vulnerable to them. However, as shown in Tramer al. [64], modern detection methods are typically weak to adaptive attacks. Since robustness to any adaptive attack is impossible to prove, we contend this a limitation of detection methods in general, unspecific to CADet.
6.4 Future directions
While spurious correlations with background features are problem in supervised learning. it is aggravated in self-supervised contrastive learning, where background features are highly relevant to the training task. We conjecture the poor performances of CADet on iNaturalist OOD detection are explained by the background similarities with ImageNet images, obfuscating the differences in relevant features. A natural way to alleviate this issue is incorporate background transformations to the training pipeline, as was successfully applied in Ma al. [40]. This process would come at the cost of being unable to detect shifts in background distributions, but such a case is generally less relevant to deployed systems. We leave to future work the exploration of how background transformations could affect the capabilities of CADet.
6.5 Conclusion
We have presented CADet, method for both OOD and adversarial detection based on self-supervised contrastive learning. CADet achieves excellent trade-off in detection power across different anomaly types, without requiring tuning on OOD samples. Additionally, we discussed how MMD could be leveraged with contrastive learning to assess distributional discrepancies between two sets of samples, and proved with high confidence that CIFARIO 10 and CIFAR10.1 have different distributions.
Acknowledgments and Disclosure of Funding
We thank Issam Laradji for his precious help in fixing cluster issues and NeurIPS 2023 reviewers and AC for their constructive criticism and for taking the time to engage in discussions. We believe their feedback has significantly improved the quality of our submission.
10

---

References
[1] A. Abusnaina, Y. Wu, S. Arora, Y. Wang. F. Wang. H. Yang, and D. Mohaisen. Adversarial example detection using latent neighborhood graph. In International Conference on Computer Vision, 2021.
[2] L. Bergman and Y. Hoshen. Classification-bas anomaly detection for general data. arXiv preprint arXiv:2005.00 2005002990, 2020.
[3] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In Symposium on Security and Privacy, 2017.
[4] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[5] T. Chen, S. Korbblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi supi-supe learners. arXiv preprint arXiv:2006. 10029, 2020.
[6] X. Cheng and Cloninger. Classification logit two-sample testing by neural networks. arXiv preprint arXiv: 12299 2019.
[7] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint 1rXiv:1810.01
2018.
[8] K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. arXiv preprint 10%iv:1506.0477 2015.
[9] L. Deecke, R. Vandermeulen, L. Ruff, S. Mandt, and M. Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference machine learning and knowledge discovery in databases, 2018.
[10] L. Dinh, J. Sohl-Dickstinn, and S. Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.
[11] X. Dong, J. Guo, A. Li, W.-T. Ting, C. Liu, and H. T. Kung. Neural mean discrepancy for efficient -fitt--diitiiuuuiin detection, 2022.
[12] Y. Du and Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, 2019.
[13] R. Feinman, R. R. Curtin, S. Shintre, and A. B. Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv: 170: 004 10, 2017.
[14] R. Gao, F. Liu. J. Zhang, B. H. 0003, T. Liu, G. N. 0001, and M. Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In M. Meila and T. Z. 0001, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages PMLR, 2021. URL http::ttt:://eeettieieie
walr.press//ee gttp//goobbbbbb..g. aatmll..mmmll
[15] I. Golan and R. Elniv... Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, 2018.
[16] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.657.2 2014.
[17] W. Grathwohl, K.-C. Wang, J.-H. Jacobsen, D. Duvenaud, M. Norouzi, and K. Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03266. 2019.
[18] A. Gretton, K. Fukumizu, Z. Harchaoui, and B. K. Sriperumbudur. A fast, consistent kernel two-sample test. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, 2009.
[19] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-77,, 2012.
[20] M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference Artificial Intelligence and Statistics, 2010.
[21] R. Hadsell, S. Chopra, and LeCun. Dimensionality reduction by learning an invariant mapping, In Computer Vision and Pattern Recognition, 2006.
11

---

[22] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Conference on Computer Vision and Pattern Recognition, 2020.
[23] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1eX.v:660.002.366 2016.
[24] D. Hendrycks, S. Basart, M. Mazeika, A. Zou, Kwon, M. Mostajabi, Steinhardt, and D. Song. Scaling -ittof--istributionn detection for real-world settings. arXiv preprint arXiv:191 1911:11111.1 2019.
[25] D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, 2019.
[26] D. Hendryeks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. arXiv preprint arXiv: I 1907.07174, 2021.
[27] S. Hu, T. Yu, C. Guo, W.-L. Chao, and K. Q. Weinberger. A new defense against adversarial images: Turning a weakness into a strength. In Advances in Neural Information Processing Systems, 2019.
[28] R. Huang and Y. Li. Mos: Towards scaling out-of-distribbtion detection for large semantic space. urXiv preprint arXiv:2105.01879, 2021.
[29] W. Jitkrittum, Z. Szabo, K. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. arXiv preprini arXiv: 1605.66766, 2016.
[30] U. Khalid, Esmaeili, N. Karim, and N. Rahnavard. Rodd: A sule-supe approach for robust out-of
detection. arXiv preprint arXiv:2204.02555 2022.
[31] B. Kitt, A. Geiger, and H. Lategahn. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In 2010 IEEE Intelligent Vehicles Symposium, pages 486-492, 2010. doi: 10. 00.1109/VVS20010555505 123.
[32] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto. 2009.
[33] K. Lee, K. Lee, H. Lee, J. Shin. A simple unified framework for detecting ouitt--distriuution samples and adversarial attacks. In Advances in Neural Information Processing Systems, 2018.
[34] S. Liang. Y. Li, and R. Srikant. Enhancing the reliability of out-of -uttof-distribution image detection neural networks. arXiv preprint arXiv: (006: 02690, 2017.
[35] F. Liu, W. Xu, J. Lu, G. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning, 2020.
[36] H. Liu and P. Abbeel. Hybrid discriminnaivv--eeeraai training via contrastive learning. arXiv 2020.
preprint
[37] W. Liu, X. Wang, J. Owens, and Y. Li. Energy-based dittr--distribution detection. In Advances in Neural Information Processing Systems, 2020.
[38] W. Liu, X. Wang, J. D. Owens, and Y. Li. Energy-bassed out-of-distribute detection. arXiv preprint arXiv:2010.037559 2020.
[39] J. Lust and A. P. Condurache. Gran: an efficient gradient-norm based detector for adversarial and misclassified examples. arXiv preprint arXiv:2004.09/79 2020.
[40] X. Ma, B. Li, Y. Wang, S. M. Erfani, S. Wijewickrema, G. Schoenebeck, D. Song, M. F. Houle, and J. Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint 1rXiv:1801.0266
2018
[41] X. Ma, B. Li, Y. Wang. S. M. Erfani, S. N. R. Wijewickrema, M. E. Houle, G. Schoenebeck, D. Song, and J. Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. CoRR, 0tbs/1801.02660
2018. URL hnttp:///axx www.v.org//bbae orgg/bs//8a0 180...10....00 02613
[42] A. Madry, A. Makelov, I.. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint
2017.
[43] J. H. Metzen, T. Genewein, V. Fischer, and B. Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv: 1702.04267,,00
[44] S. Mohseni, M. Pitale, J. Yadawa, and 7. Wang- Self-supervised learning for generalizable out-of-distribution detection. In AAAI Conference on Artificial Intelligence, 2020.
12

---

[45] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan. Do deep generative models know what they don't 't know? arXiv preprint arXiv:1810.09/36, 2018.
[46] E. T. Nalismick, A. Matsukawa, Y. W. Teh, and B. Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using a test for typicality. arXiv preprint wrXiv:/906002999 2019.
[47] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in nat- ural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[48] N. Papernot and P. McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:/803.04760 2018.
[49] P. Perera, R. Nallapati, and B. Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Conference on Computer Vision and Pattern Recognition. 2019.
[50] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In Advances in neural information processing systems, 2018.
[51] J. Raghuram, V. Chandrasekaran, S. Jha, and S. Banerjee. A general framework for detecting anomalous inputs to dnn classifiers. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8764-8775. PMLR, 18-24 Jul 2021. JRRL.https://procediiin
mlr. prrssrssseeee. ww11//rgguurarann
html
[52] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv: 108/ 1, 2019.
[53] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Process- ing Systems, 2019.
[54] L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, 2018.
[55] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. arXiv preprint arXiv: 1703.05921, 2017.
[56] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Interna- tional conference on information processing in medical imaging, 2017.
[57] B. Schölkopf. R. C. Williamson, A. Smola, J. Shawe-Taylo Taylor, and J. Platt. Support vector method for novelty detection. Advances in neural information processing systems, 1999.
[58] V. Sehwag, M. Chiang, and P. Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021. URL https //openreview net/forum?idd=vgg
[59] Serrå, D. Álvarez, V. Gómez, O. Slizovskaia, Núñezz and Luque. Input complexity and out-of-distribbttio detection with likelihood-based generative models. arXiv preprint arXiv:1909. / 1480, 2019.
[60] K. Sohn, C.-L. Li. J. Yoon, M. Jin, and T. Pfister. Learning and evaluating representations for deep one-class classification, 2021.
[61] Y. Sun, C. Guo, and Y. Li. React: Out-of-distribution detection with rectified activations. arXiv 2021.
preprint arXiv:2111.
[62] D. Sutherland, H.-Y. Tung, H. Strathmann, De, A. Ramdas, A. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. arXiv preprint arXiv: 1611.4441 2016.
[63] Tack, S. Mo, J. Jeong, and J. Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
[64] Tramer, N. Carlini, W. Brendel, and A. Madry. On adaptive attacks to adversarial example defenses, 2020.
13

---

[65] A. Uwimanal and R. Senanayake. Out of distribution detection and adversarial attacks on deep neural networks for robust medical image analysis. arXiv preprint arXiv:2107.04882 2021.
[66] H. Wang, Z. Li, L. Feng, and W. Zhang. Vim: Out-of-distributioo with virtual-logit matching. In Conference on Computer Vision and Puttern Recognition, 2022.
[67] L. Wenliang, D. Sutherland, H. Strathmann, and A. Gretton. Learning deep kernels for expo- nential family densities. In International Conference on Machine Learning, 2019.
[68] J. Winkens, R. Bunel, A. G. Roy, R. Stanforth, V. Natarajan, J. R. Ledsam, P. Mac MacWilliams, Kohli, A. Karthikesalingam, S. Kohl, et al. Contrastive training for improved out-of- distribution detection, arXiv preprint arXiv:2007.05566 2020.
[69] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In Conference on Computer Vision and Pattern Recognition, 2018.
[70] W. Xu, D. Evans, and Y. Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. CoRR, abs/ 1tts//704001155, 2017.
[71] Yang, K. Zhou, Y. Li, and Z. Liu. Generalized out-of-distribution detection: A survey. arXiv
preprini arXiv:21 10. 1334, 2021.
[72] Y. You, I. Gitman, and B. Ginsburg. Large batch training of convolutional networks, arXiv preprint PrXiv:17055 03888, 2017.
[73] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsum: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 106.::1506.03365, 2015.
[74] S. Zhai, Y. Cheng, W. Lu, and Z. Zhang. Deep structured energy based models for anomaly detection. International Conference on Machine Learning. 2016.
[75] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, Cho, and Chen. Deep autoencod- ing gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018.
[76] F. Zuo and Q Zeng. Exploiting the sensitivity of 12 adversarial examples to erase-and-restore. In Asia Conference on Computer and Communications Security, 2021.
14

---

A
Algorithms pseudo-code
We give the full pseudo codes for CADet's calibration step and inference step in Algorithm and
Algorithm 2 CADet calibration step
Input: x x Teval, learned similarity function various hyperparameters used below;
1: for (1)) E (1) vil do
2: for 1,2,..., Pitr do
3:
Sample from Teval
4: ar(1)) (xx(1)
5: end for
6: end for
7: k: 1
8: for (2)2
E (2) do
9: for 1, 2,...... Pars do
Sample from Teval
(2) - (2)
11:
end for
13: m
Σ
5
(2)
I
(2)
m
E
8(1 102)
(1)
(2))
x(1)
15: k+ +
end for
17: m
m
Nere (Reva++1
18: f my
ni,- (XX.))
19: Vin, Vont = Var(m"n) out
20: 7 Vin
Voul
21: k
22: for (2) E (2) do
val
23: scorek < m in
+ x m
24: #+ ++
25: end for
Output: coefficient: scores: score;, transformed samples: x;
(1)
15

---

Algorithm 3 CADet testing step
Input: transformed samples: at scores: scores, sample: wteet, coefficient: 7, trasnformation set: Teval:
1: for i 1, 2,,2.... ...., Mere do
2: Sample from Tenal
3: arest + ((xtest)
4: end for
Dere
5: min
2 s(atest (est) 17j
6: m mut
Σ
L s(ztest at )
x0))x
(1)
7: scorete
m=
Ber (Mers+)))
Or #1. 1
8: rank:
9: for score"al E scores do
if' score score' then
11: rank & rank +
end if'
13: end for
rank+1
Peal
# (scores) ) +1
Output: p-value: Poal
B Ablation on backbone
We present in table 6 the AUROC of previous methods iNaturalist and ImageNet-O using self-supervised contrastive ResNet50 backbone, as an ablation to table Since most methods require class logits, we train linear layer on top of the self-supervised features using all training labels. Thus, these methods still possess an unfair advantage over CADet by exploiting label information.
Table 6: AUROC for OOD detection on ImageNet-O and iNaturalist with ResNet50 backbone using self-supervised contrastive learning.
Training
iNaturalist
ImageNet-O
Average
MSP
23
76.93
54.09
65.51
Energy
38
69.55
57.91
63.73
ODIN
(34]
81.83
52.10
66.97
MaxLogit
2
75.62
52.35
63.99
KL Matching
24
Self-supervised (contrastive)
79.27
67.71
73.49
ReAct 66]
78.95
68.08
73.52
Mahalanobis 33
72.60
76.49
74.55
Residual 66
78.60
80.99
79.80
ViM 666
72.17
85.11
78.64
C Tuning of baselines
For previous works, when specified otherwise we use the same tuning strategies described in the original papers.
For Mahalanobis
33 employ the of the last layer only as done in ViM 66 For LID 41,
we try for in 10, 20, 100] (and find & = 20 to perform best on all benchmarks) and a in the range [0, 10). For ODIN, for OOI) detection,
the default values described in Liang al. 34 (since we assume no access to ood samples), and tune them for adversarial detection. For FS [70], we use the best joint detection of multiple squeezers, which was tuned on ImageNet, ic. Bit Depth 5-bit, Median Smoothing 2x2, and Non-Local Mean 11-3-4.
16


Reviews:
[{"review_id": "VT2nP9xGHq", "reviewer": "Reviewer_kyFG", "summary": "The paper presents a novel method for Out-of-Distribution (OOD) detection called CADet (Contrastive Anomaly Detection). The authors leverage self-supervised contrastive learning and the maximum mean discrepancy two-sample test(MMD) to assess whether two sets of samples have been drawn from the same distribution. The method is designed to detect OOD samples from single inputs and performs well on both label-based and adversarial detection benchmarks, without requiring access to any OOD samples nor previous classes to train or tune the method.", "questions": "1. I understand ResNet50 is a popular architectures that has been used for OOD and adversarial benchmarks. But there has been significant developments in those areas too, did you have a chance to see how would the method perform on other architectures beyond ResNet50?", "limitations": "Authors have done a good job presenting the limitations of their paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors propose a novel method for OOD detection that does not require access to neither in distribution samples nor OOD samples to train or tune the method.\n2. They use similarity functions learned by self-supervised contrastive learning with MMD to assess the distribution even with few shot images.\n3. Paper shows the method outperforms current methods in adversarial detection tasks while performing well on label-based OOD detection.\n", "weaknesses": "I do not see major weaknesses for this paper as of now. I will be updating this section if need be later.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback. \n\n**Q1:** It is a very pertinent question and recent works such as \"ViM: Out-Of-Distribution with Virtual-logit Matching\" by Wang et al. shows more recent architectures like ViT have distinct and promising features when it comes to OOD detection. Unfortunately, self-supervised contrastive pre-training is expensive and our computation resources did not allow us to experiment with larger architectures, as admitted in L292-L301. \n\nWe seem to have noticed the reviewer have reduced their score from the initial review and would like to inquire if they have any novel question or concerns we might address ? If such decision was motivated by other reviews, we invite the reviewer to also consider our rebuttals to these reviews. "}, {"review_id": "Kp0cVNXe3Q", "reviewer": "Reviewer_X9Xp", "summary": "The paper proposes Maximum Mean Discrepancy with Clean Calibration (MMD-CC), as an improvement of MMD when the number of samples is small. Moreover, it introduces CADet, a novel anomaly detector for both OOD and adversarial detection inspired by MMD. ", "questions": "### Suggestions:\nGiven the amount of OOD and adversarial detectors now available, I would expect a strong validation of a method to prove its effectiveness. I think that the most important concern that should be addressed is how CADet, given its resource limitations, can be a significant improvement over existing unsupervised OOD/adversarial detectors such as [1].\n\n### Typos:\n- Line 140, \"Additionnally\".\n\n[1] Raghuram, J., Chandrasekaran, V., Jha, S., & Banerjee, S. (2021). A General Framework For Detecting Anomalous Inputs to DNN Classifiers. Proceedings of the 38th International Conference on Machine Learning, 8764–8775. \n", "limitations": "The authors addressed the main limitations of their method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In line with recent works, unsupervised anomaly detection in deep learning is a relevant research objective. The paper is well-written and proposes a novel approach for anomaly detection, improving existing works on Maximum Mean Discrepancy. Moreover, I appreciated the idea of improving MMD to overcome its limitation of requiring sets of test samples.", "weaknesses": "- Experimental setting. I did not understand why for benchmarking adversarial detection you kept ODIN [1] and discarded the Mahalanobis detector. To the best of my knowledge, ODIN is mainly designed for OOD detection, furthermore in [2] the Mahalanobis detector achieves better performance than both LID [3] and ODIN for adversarial detection. Moreover, no adaptive attack was tested [4].\n- Strength of the results. In Table 3, CADet has a high variance of the results depending on the training strategy  (iNaturalist vs ImageNet-O) and is not the overall best performing. Moreover, only the AUROC was reported, while other commonly employed metrics are missing (FPR, AUPR). Regarding MMD-CC, it should be better clarified its significance given that it is significantly better only on Table 2, with PGD and small n_samples.\n- Missing related works. Recently, other unsupervised anomaly detection (for both OOD and adversarial) algorithms have been proposed, such as [5]. A comparison with such method should be performed.\n- Reproducibility. No code was released. Some hyperparameters are missing (e.g., number of neighbours for LID [3], perturbation size for Mahalanobis [2] and ODIN [1]). Given how challenging it is to evaluate and compare OOD detectors, this is an important weakness.\n- Architecture scale. Given the existence of other unsupervised methods for anomaly detection ([5]) and the lack of convincing performance improvements over related methods, the computational limitations of CADet are a significant weakness.\n\n[1] Liang, S., Li, Y., & Srikant, R. (2018). Enhancing the reliability of out-of-distribution image detection in neural networks. ICLR 2018.\n\n[2] Lee, K., Lee, K., Lee, H., & Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS 2018.\n\n[3] Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S. N. R., Schoenebeck, G., Song, D., Houle, M. E., & Bailey, J. (2018). Characterizing adversarial subspaces using local intrinsic dimensionality. ICLR 2018.\n\n[4] Tramer, Florian, et al. \"On adaptive attacks to adversarial example defenses.\" Advances in neural information processing systems 33 (2020): 1633-1645.\n\n[5] Raghuram, J., Chandrasekaran, V., Jha, S., & Banerjee, S. (2021). A General Framework For Detecting Anomalous Inputs to DNN Classifiers. Proceedings of the 38th International Conference on Machine Learning, 8764–8775. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback.\n\n1- In [2], the Mahalanobis detector uses a score based on a logistic regression of layer-wise Mahalanobis distances. The regression weights are trained on validation outliers. Training directly a classifier on outliers has long been discarded by the community and makes their comparison unfair and explain their superior performance. This is the reason why we kept ODIN, which is indeed designed as an OOD detector but has been shown in prior works to be somewhat reasonable in detecting adversarial samples, and discarded the detector of [2] which operates in a much easier and less relevant setting where training on outliers is allowed (ODIN does some hyperparameter tuning on outliers but it is a lighter violation as they merely do a grid search on two hyperparameters and show their tuning to transfer well). With that said, for OOD detection, we followed the steps of prior works such as ViM and adapted the Mahalanobis score by only considering it for the last feature layer. Doing this removes the need to train on outliers. We are not aware of any prior works having followed the same process for adversarial detection, but it seems like a natural and logical thing to do. We will immediately start this evaluation, and hopefully will be able to provide results before the discussion phase ends. \n\nRegarding adaptive attacks, it is difficult enough to fit our submission within the page limit. Evaluating against adaptive attacks is a comprehensive subject and would require a dedicated paper. Moreover, while robustness to adaptive attacks is desirable, it is well known that ood detection method are weak against them, and being able to detect attacks in a black box setting is already useful. We believe our method, like its peers, would be weak against adaptive attacks.\n\n2. We admit CADet is not best performing for novel class detection. However, its performance on adversarial detection alone is impressive and in our opinion could warrant publication. The fact that it is also concurrently close to SotA on novel class detection, without requiring any change of tuning, is a significant plus. Would the reviewer find our results more convincing if we only reported adversarial detection ? \n\nIt is true that we only report AUROC. The reason for this is that in our experience, AUROC FPR95 and AUPR typically follow the same pattern, and yields the same rankings across methods. We are not aware of any case where the performance (relative to other methods) vary significantly across metrics. As thus, in our opinion this set of metrics has become redundant and makes results less readable. \n\nMMD-CC is an alternative we propose to classic MMD, which can be advantageous at smaller sample sizes. In Table 2, MMD-CC outperforms MMD on all benchmarks and for all sample sizes (though they are all small), with some ties on iNaturalist. \n\n3. We will cite and discuss [5]. Note that [5] introduces a meta-algorithm which it experiments with existing methods on small datasets, but scaling e.g. aK-LPE to ImageNet requires significant computation ressources. Since most existing methods already follow the abstraction of their meta algorithm, experimenting against these methods, in a sense, is an experimentation against a specific realization of their meta algorithm. \n\n4. For LID, we used k=20 as was used by authors on CIFAR and MNIST, and will specify it. We agree that the absence of code is an important issue. We intend to release our code publicly, the only obstacle being that it is pending legal review. While this legal review has been ongoing for an absurdly long time, we have been promised we should get clearance soon, much before camera ready deadlines. We intend to release our code as a fork of OpenOOD, which should make it especially easy to use and reproduce by the community.\n\n5. As stated above, [5] proposes a meta-algorithm rather than a specific method. They also introduce two specific realisations of their meta-algorithm using scores from previous works. aK-LPE does not scale well either to large datasets, and while we are not absolutely certain, we suspect that the other method of using p-value normalization at each layer *and layer pairs* likely does not scale well to modern architecture either. While CADet has a significant overhead, it is entirely parallelizable (all the overhead is induced by having to do multiple forward passes) and the overhead factor is independent of the architecture. Given its jointly good performances on adversarial detection and novel class detection without ever requiring any labels, we believe it is still an interesting contender that is relevant to the community. \n\nSuggestion: While [1] report clock time including training in Table 9, they fail to report inference time, which is most relevant and is likely to be high. Our reported overhead is purely for inference time. \n"}, {"review_id": "RIKYvIYe3H", "reviewer": "Reviewer_tQNn", "summary": "The authors propose to handle the OOD detection problem, exploring self-supervised contrastive learning to detect samples from previously unseen classes and adversarially perturbed samples. They use self-supervised contrastive learning with the maximum mean discrepancy (MMD) to test if two sets of samples originate from the same distribution. They propose CADet which takes advantage of the similarity of the contrastive representation/transformation of the same sample. The proposed approach is thoroughly tested in two scenarios: the scenario with previously unseen classes and the scenario with adversarially perturbed samples.", "questions": "+ What is the intuition behind the fact that self-supervised contrastive learning works well with MMD? ", "limitations": "+ The paper proposes an effective approach (self-supervised CL + MMD), but there are other mainstream alternatives to MMD such as the KL divergence and Mahalanobis distance. Why do you pick MMD in particular, instead of KL-divergence and Mahalanobis distance (both of which can be adapted to perform on both instance-level and distribution-level)? Is there any comparison on why KL-divergence based and Mahalanobis distance based approaches aren't as effective?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The problem of OOD detection is very important in a lot of aspects of machine learning, and it's a worthwhile problem to study and research.\n+ The proposed approach, CADet, achieves good performance on the benchmarks.\n+ The combination of MMD and self-supervised contrastive learning seems to be effective.", "weaknesses": "- The work is motivated by the claim that OOD samples will be encountered in the wild when the machine learning models are deployed in the wild, but this work is not tested in the wild, in a smart computing or cyber-physical system setting. This is a minor complaint as it is tested on (solely) datasets, but it is advised that the authors revise the paper to not include such claims.\n\n- The work fails to be evaluated on \"fake\" OOD samples, or samples that might originate from a different distribution than the training samples but belong in the same classes. For example, the Webcam domain in the Office-31 dataset and the Amazon domain in the same dataset, have the same classes, but distribution-wise, the samples are different. Will CADet consider Webcam's samples of class A to be OOD from Amazon's samples of class A? \n\n- The Related Works section is too high-level: it's evident that the authors are aware of the recent advance in the field of OOD detection, but a more detailed explanation (one sentence or two) for (most of) the mentioned works would be desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback.\n\n**Weaknesses:**\n\n1- The reviewer is correct that we only evaluate on homogenous benchmark. Evaluating in a realistic setting is extremely difficult, as even data collection can pose legal challenges. Nonetheless, the general motivation for trying to detect OOD samples is that they will be encountered in the wild, even though our evaluations are only an approximation of this phenomenon. While this setting is not perfect, it is the standard approach in the hundreds of OOD detection papers published every year. \n\n2- Even when samples have the same classes, significant covariate shift can lead to misclassification with overconfidence. Besides, the separation into class is arbitrary, and there is no objective reason to consider that covariate shifts are not a change of class (does a picture of a cat and a drawing of a cat belong to the same class ? there is no objective answer, and certain real world systems trained to recognise animals may need to react differently when presented with a drawing). As such, the general formulation of the OOD detection problem is to detect samples that are out-of-distribution, regardless of their semantic classes. While a few works have tried to focus on detecting unknown classes and NOT covariate shifts, in particular \"Full-Spectrum Out-of-Distribution Detection\" by Yang et al, again, the benchmarks we use are standards in the field. \n\n3- We had to decrease the size of the related work section to fit page constraints. Another issue is that the literature of OOD detection has been growing very quick, and it is difficult to give a good representation of the MANY recent works on the topic while also giving insights into specific works. We will add a reference to \"Generalized Out-of-Distribution Detection: A Survey\" by Yang et al. which is an excellent survey of recent works, so that the curious reader can gather more detailed insights into concurrent works.\n\n**Questions:**\n\n1- That is an excellent question. While traditionally self-supervised contrastive learning is used only to learn useful representations, these representations only intervene in the training loss through the similarity function $s=cosine(z1,z2)$. In particular, self-supervised contrastive learning trains the similarity function $s$ to be close to $1$ on pairs of samples that are semantically close (which is approximated by random transformations of a same image) and close to $0$ on pairs of samples that are semantically far (which is approximated by random transformations of different images). Moreover, since the model does not access training label, it does not discard information that is not directly useful for classification, and does not overfit training classes. In MMD, a kernel is used to estimate the proximity between all pairs of samples within a set. For large dimensional, unstructured data like images, traditional kernels often fall short. However, self supervised contrastive learning was precisely train to identify such proximity. Thus, it is a very natural choice.\n\n**Limitations:**\n\nMahalanobis as it is used in the OOD detection literatures, works by learning a class-conditional gaussian model of representations. In the case of e.g. CIFAR10 vs CIFAR10.1, on what data would this gaussian model be fit ? Since the neural network was trained on ImageNet, and does not make class predictions, it could only find a large OOD score for both CIFAR10 and CIFAR10.1, making it an inadequate approach. Training on a subset of CIFAR10/CIFAR10.1 test set would also prove difficult given their small sizes. As for learning a gaussian model directly on the raw images, it is well known that it is very unadapted to such high dimensional unstructured data. For KL divergence, a similar approach is generally used: assuming and learning a gaussian distribution to estimate KL divergence between OOD samples and in-distribution. However, this method needs to be performed on the representations of a neural network or other model (as the raw data is too high dimensional and unstructured). The same issues then arise: training on a subset of test datasets is not sample efficient and not realistic in this case, and training on an external dataset (such as ImageNet) will yield distribution estimation that are extremely far from both sample sets. \n\nThis is where self-supervised contrastive learning provide a convincing solution: the similarity function learned on imagenet is capable of comparing two out-of-distribution samples z1 and z2 to one another directly, whereas KL or Mahalanobis could only compare z1 and z2 to its training distribution, separately. "}, {"review_id": "mIIqxEY3uY", "reviewer": "Reviewer_1ZdX", "summary": "This paper studies the OOD detection of new classes and adversarial attacks with learned similarities via self-supervised contrastive learning in conjunction with MMD two-sample test. To enable MMD applicable for OOD detection on single samples, they improve the idea by using augmentations to create a set. Experimental results on new classes and adversarial examples detections show that their method can perform favorably against previous arts.", "questions": "See weakness", "limitations": "Limitations have been discussed in the manuscript.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-organized.\n\n2. The paper does a good job summarizing preliminary ideas and providing detailed settings reproducing the overall algorithm.\n\n3. The paper provides helpful discussions regarding the efficiency of the method.", "weaknesses": "1. According to Table 3. it seems that the proposed method does not show superior performance against previous arts when using supervised training. Discussions regarding this matter could be included to better justify the method.\n\n2. I'm not an expert on OOD detection. I'm wondering why there are no error bars in the reported results, given that different hyper-parameters could easily improve (or decrease) the performance, and the differences between the compared arts are not that significant reported in Table 3. \n\n3. Some hyper-parameters are empirically selected without proper justifications, especially those regarding augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback.\n\n1- As mentioned L195-L198, CADet is designed to leverage the similarity function of self-supervised contrastive learning (SSCL). Indeed, the CADet score is computed based on the cosine similarity between the representations of random transforms of a same image (for m_in) or of distinct images (for m_out). Those are the exact quantities that SSCL models are trained with: these models are trained to maximise the similarity between random transforms of a same image, and minimise the similarity between random transforms of two different images. This alignement between the training objectifs of SSCL and the quantities used by CADet is what makes CADet particularly suited for SSCL. Note that on iNaturalist, the supervised backbone still performs better because iNaturalist classes are very close to certain ImageNet classes, and having to differentiate plant species within ImageNet allow supervised classifiers to learn very fine-grained features, which are useful to detect such close OOD examples. On all other benchmarks, SSCL backbones are better suited for CADet. \n\nSince CADet was designed specifically with SSCL in mind, it is predictable that its performance with a supervised backbone, which we report as a mean of comparison, would be inferior. Surprisingly, it is not so far from SotA, ranking only behind Mahalanobis and ViM. Note that while CADet is not SotA on novel classes detection benchmarks, it beats previous works in detecting adversarial attacks (Table 4), and still performs close to best methods on novel classes detection (Table 3) without requiring any change of tuning, and without requiring any training label either. We believe such results are sufficient to warrant publication.\n\n2- Since it is now generally accepted in the field that detection methods should not assume access to outliers for validation (or they would overfit outlier distributions, without necessarily transferring to other distribution shift types), we refrain from tuning hyperparameters on outliers. Effectively, we use the standard setting of SimCLRv2 for training, and for instance our hyperparameter gamma is autocalibrated on the training set in a deterministic way. As such, it is not clear to us which different hyperparameters values should be used to compute error bars. \n\nIdeally, error bars would be computed wrt k independent trainings (with separate seeds) for the underlying network. However, as self-supervised contrastive learning is very expensive, and we insist on evaluating at a scale that is relevant to the real world, doing such independent runs is outside of our computational means. \n\n3- The transformations used are the default ones reported in \"Big Self-Supervised Models are Strong Semi-Supervised Learners\" (SimCLRv2) by Chen et al. Unfortunately, since we refrain from seeing outliers, we do not have any objective ways to tune hyperparameters, and resort to default settings. In the case of SimCLRv2, these hyperparameters were likely optimized to maximise post-finetuning validation accuracy, and thus were not exposed to outliers. In the case of gamma, we use an heuristic (balancing the variance of m_in and m_out) which is likely highly suboptimal, but without access to outliers, we do not see viable alternatives."}, {"review_id": "rJFy78j6q3", "reviewer": "Reviewer_hQ4Y", "summary": "This paper consists of two mostly unrelated parts.  In the first part, the authors aim to detect whether two sets of samples are drawn from the same distribution.  To do this, they apply the unbiased estimate of the MMD, in the manner suggested by Sutherland et al., to produce a conformal p-value.  Then they propose a small variation on Sutherland's approach, which they call MMD-CC, that requires twice as many samples but supposedly reduces some variance (although no further explanation or analysis is given).  They apply these techniques to detecting distribution shift between CIFAR-10 and CIFAR-10.1 using a feature-generating network trained on an external dataset (ImageNet), similar to \"outlier exposure\" methods.\n\nIn the second part of the paper, they aim to detect whether a single sample is drawn from same distribution as a set of inliers or not.  For this, they propose the CADet method, which has no discernable relationship to MMD (3) or the unbiased MMD estimate in (4).  Rather, CADet is closer to the CSI approach from Tack et al. and the ICLR'21 approach of Sohn et al., in that it relies on contrastive learning and distributional shifts.  Because key parts of CADet are never explained by the authors, it's difficult to summarize further.  The authors test CADet on experiments where ImageNet is the inliers and other datsets are the outliers, but they never compare to the state-of-the-art CSI approach even though they write \"the closest work in the literature is CSI\" on line 86.", "questions": "1. Regarding the perturbations in Table 2, they are never clearly defined.  Are these perturbations of the Supervised ImageNet-trained classifier used in the bottom two rows?\n\n2. There seems to be a mistake in the contrastive loss equation (1), since the \"j\" in the numerator is undefined.  Should the numerator be \"u_{i,i}\"?\"\n\n3. There seems to be a mistake in the CADet definition (5).  The denominator of m^out contains the number of terms in the numerator sum, but the denominator of m^in does not.  Should the latter be \"ntr(ntr-1)\"?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The combination of Sutherland's MMD with SimCLRv2 gives good empirical performance in detecting distributional shift between CIFAR-10 and CIFAR-10.1.", "weaknesses": "1. Claims about the existing literature are overblown and key citations are missing.  \na. On line 46, the authors claim that \"only a few works detect OOD samples from single inputs ... without requiring access to OOD samples to train or tune the method.\"  But there are dozens, if not hundreds, of papers proposing such methods.  \nb. Although the authors acknowledge that the CSI paper by Tack et al. is very close in spirit to CADet, they do not cite an even closer method in the ICLR'21 paper by Sohn et al.  \nc. Although the authors frequently compute conformal p-values, they don't cite any relevant literature.\nd. The authors do not cite previous work that uses MMD for OOD, such as Dong et al.'s \"Neural Mean Discrepancy for Efficient Out-of-Distribution Detection\".\n\n2. Throughout the paper, the p-values are invalid.  In the best case, the conformal p-values computed by the authors hold only only marginally, i.e., in expectation over the validation data.  They do NOT hold for a fixed validation set.  For a detailed explanation, see Bates et al. \"Testing for Outliers with Conformal p-values\" or Magesh et al. \"Multiple testing framework for out-of-distribution detection\" or Angelopoulos et al. \"Conformal Prediction: A Gentle Introduction\".\n\n3. In some cases, the p-values are invalid for more fundamental reasons.  In MMD-CC Algorithm 1, for example, the conformal p-values are computed as if n_perm was the size of an i.i.d. validation dataset.  But n_perm permutations are not independent and so the p-values computed in this way are invalid.  For example, there is nothing preventing the user from specifing a value of n_perm that is higher than the total number of possible permutations.  In fact, the MMD-CC method is entirely heuristic and comes with no analysis, which is a serious problem. \n\n4. The MMD-CC method is underwhelming at a practical level as well.  First, it requires twice as many samples as Sutherland's MMD.  Despite that limitation, it performs worse in Table 1.\n\n5. For the CIFAR-10-vs-CIFAR-10.1 experiment, the proposed feature generators were trained using ImageNet (see line 160).  This is problematic for several reasons.  First, using an external datset for OOD detection is a form of outlier exposure that violates the main claim of the paper stated on line 45.  Second, the other methods under test do NOT use an external datset, making the comparison unfair.  Third, the use of an external dataset goes against the instructions on line 94, which explicitly state that SimCLRv2 is to be trained on \"in-distribution samples.\" Fourth, alhough the authors wrote that \"a fair comparison is difficult,\" that's not true; they could have easily trained their feature generator on the CIFAR-10 inlier dataset and it would have avoided all these problems.\n\n6. The authors claim a weakness of CSI is that it can't detect adversarial perturbations, but in doing so the authors are conflating two entirely different versions of the OOD problem.  The first version is where one is given a trained supervised classification network and asked to detect samples different from those used to train it.  In this problem, we start with a network and so adversarial perturbations are well defined.  The second version (solved by CSI) is where one is given an inlier distribution and are asked to detect samples different from it.  Here, there is no network, and thus no adversarial perturbations, and so it is unfair/illogical to claim that CSI is \"not well suited for adversarial detection.\" CSI solves an entirely different problem.\n\n7. The section \"detection of distribution shifts from a small number of samples\" hides the fact that a huge number of samples are required to train the feature generating network.  In fact, by training on an external dataset, the authors are implementing a form of \"outlier exposure\" that goes against the main requirement stated on line 45.\n\n8. Critical information about the CADet method (the main contribution according to the title/abstract) is missing, making it impossible to fully evaluate the paper.  In particular, the similarity function \"s\" in (5) is never specified.  On line 195 it says \"self-supervised contrastive learning trains a similarity function,\" but that is incorrect: SSCL trains a feature generating network that maps an input \"x\" to a set of features \"z,\" as specified on line 99.  SSCL does NOT train a similarity function that maps two inputs \"x1\" and \"x2\" to a scalar.\n\n9. The choice to linearly combine the m^in and m^out scores in (6) is heuristic, and probably highly suboptimal.  The framework of multiple hypothesis testing gives principled ways to combine scores.  For more on multiple hypothesis testing, see Candes' STATS 300C lecture notes.\n\n10. The CSI method by Tack et al. is \"the closest work in the literature\" according to line 86. Yet the authors curiously omit it from all the numerical experiments, choosing much easier competitors. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1a. We consider that works that tuned their methods on calibration OOD datasets and evaluated on different test benchmarks are still tuning on OOD samples since these test and calibration benchmarks typically involve similar types of distribution shifts, applied from similar inlier distributions. We searched extensively for works that truly do not have any outlier contamination and found very few of them, such as the excellent ViM.\n\nb. While the method in Sohn et al. is interesting, their method relies on learning the distribution of representations, and do not leverage transformations at test time. We will cite their work but we believe it is not more closely related than CSI.\n\nc. For MMD, the p-value we compute are not conformal p-value. They're classic p-values: under the null hypothesis, the scores are identically distributed and thus their ranks are uniformly distributed. Interestingly, in L224-L227, the p-value we compute is indeed a conformal p-value, and we admit its dependence on $X_{val}$ L227. However we only use it as a computation trick to compute the AUROC more efficiently. We verified independently that laborious AUROC computation differ at most in 0.02% in AUROC scores from the ones we report.\n\nd. We were not aware of that specific work and will cite it.\n\n2. See point 1.c. above. The only part where we compute conformal p-values is in fact just a (valid) computation trick for AUROC measurement. In MMD experiments, the ranks are not computed wrt a separate validation set, and thus there is no such set it could hold marginally on. This is independent from the framework of conformal p-values, and instead relies on the much simpler (and older) framework of p-values.\n\n3. What does the reviewer mean by \"n_perm permutations are not independent\". The permutations are chosen at random and are of course iid. This is the same principle as the original MMD method. If n_perm is larger than the number of possible permutations, certain permutations will occur multiple times. There is no problem with that. Each score is computed with a random permutations of variables that, under the null hypothesis, are iid. This is sufficient to guarantee that the rank of the scores are uniformly distributed under the null hypothesis. This is aligned with the classic MMD method.\n\nCompared to \"Learning deep kernels for non-parametric two-sample tests\" from Sutherland, we even corrected a small mistake in p-value computations (see L131-L139), which made their p-values technically invalid.\n\n4. As we argue in L144, the benefits of MMD-CC are expected to be significant with small sample sizes. This is confirmed in Table 2 where MMD-CC convincingly outperforms MMD. MMD-CC is an alternative we propose, and we never claim it is always preferable to MMD. It performs better in certain settings (few samples) which is valuable even if it doesnt outperforms prior works in every scenario. Regarding the number of samples required, we only need twice as many samples from one of the two distributions. In practice, we often have access to a much larger number of inliers, so this constraint is irrelevant. When we don't, MMD is likely preferable.\n\n5. The claim in L45 refers to CADet (not MMD), which is trained on ImageNet and **always** evaluated with ImageNet as inlier distribution. Thus the reviewer's observation does not contradict L45. We agree the previous methods do not use an external dataset, and thus that these results should be considered with caution, and it is specified at L60-L61. The problem is that existing methods do not use inlier training data either. So even if we trained on CIFAR as the reviewer suggested, the comparison would still not be fair. We believe to be the first work distinguishing CIFAR10 and 10.1 with such confidence, with or without external data. \n\n6. As stated L242-L243 the attacks are computed against a fixed pre-trained network. Therefore, they are black-box attacks and all methods are evaluated against the same images. It is well known that adversarial attacks transfer relatively well across convolutional networks. Since it is a fixed distribution shift, we don't think the reviewer's comment applies.\n\n7. Again, L45 explicitly refers to CADet, not MMD. We believe for hypothesis testing on CIFAR10.1 vs CIFAR10, outlier exposure or contamination does not make sense. The goal is not to evaluate detection metrics on CIFAR10.1 and claim that such metrics will generalise to other out-distributions, but simply to find that these two specific distributions are indeed different. Anything is fair as long as the p-values are valid. \n\n8. As specified on L100, the similarity function we refer to is the cosine similarity on top of learned representations. In the training of SSCL, the representations only appear through the intermediary of cosine(z1, z2). Thus it is not incorrect to claim that SSCL learns a similarity function.\n\n9. It is indeed heuristic. Multiple hypothesis testing is also a valid choice to combine $m_{in}$ and $m_{out}$, but it is often highly suboptimal too. For instance, when introducing random scores with identical distribution in and out of distribution, the performance drop sharply. We could have investigated multiple options to combine scores but then how would we choose between them without using OOD samples ? Thus we opted for a simple heuristic.\n\n10. Similarly to \"OpenOOD: Benchmarking Generalized Out-of-Distribution Detection\", we failed to scale CSI to ImageNet sizes.\n\nQ1. The perturbations are defined later in part 5.1 for the CADet experiment which is an issue. We will clarify in section 4.2. Indeed, the attacks are computed against the supervised backbone. Effectively, PGD w/ supervised backbone can be seen as white box, and with the self-supervised backbone as black box.\n\nQ2 and Q3. The reviewer is correct in both case and we will correct these mistakes. Note that for (5) it is not impactful since the scores are rescaled to equalize variance anyway.\n\n\n\n"}]

Author rebuttal:

Meta-review: The paper introduces an out-of-distribution detection algorithm based on MMD and contrastive learning. Reviewers appreciated the novelty of the method but had initial concerns on the experimental evaluation, strength of results and discussion of related work. The authors' responses and following discussion addressed most of these concerns to the satisfaction of the reviewers. Following the discussion, reviewers unanimously lean towards acceptance. The authors should ensure all promised revisions to the paper are included in the camera ready version.
Decision: Accept (poster)