Revisiting Out-of-Distribution Detection: A Simple Baseline is Surprisingly Effective
Authors: Julian Bitterwolf
Alexander Meinke
Maximilian Augustin
Matthias Hein
Venue: ICLR 2022 Submitted (2022)
Keywords: out-of-distribution detection
robustness
OOD

Abstract: It is an important problem in trustworthy machine learning to recognize out-of-distribution (OOD) inputs which are inputs unrelated to the in-distribution task. Many out-of-distribution detection methods have been suggested in recent years. The goal of this paper is to recognize common objectives as well as to identify the implicit scoring functions of different OOD detection methods. In particular, we show that binary discrimination between in- and (different) out-distributions is equivalent to several different formulations of the OOD detection problem. When trained in a shared fashion with a standard classifier, this binary discriminator reaches an OOD detection performance similar to that of Outlier Exposure. Moreover, we show that the confidence loss which is used by Outlier Exposure has an implicit scoring function which differs in a non-trivial fashion from the theoretically optimal scoring function in the case where training and test out-distribution are the same, but is similar to the one used when training with an extra background class. In practice, when trained in exactly the same way, all these methods perform similarly and reach state-of-the-art OOD detection performance.

Under review as a conference paper at ICLR 2022
REVISITING OUT-OF-DISTRIBUTION DETECTION: A SIMPLE BASELINE IS SURPRISINGLY EFFECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
It is an important problem in trustworthy machine learning recognize out- of-distribution (OOD) inputs which are inputs unrelated to the in-distribution task. Many out-of-distribut years. The goal of this paper is to recognize common objectives as well as to identify the implicit scoring functions of different OOD detection methods. particular, we show that binary discrimination between in- and (different) out- distributions is equivalent to several different formulations of the OOD detection problem. When trained in shared fashion with standard classifier, this binary discriminator reaches an OOD detection performance similar to that of Outlier Exposure. Moreover, we show that the confidence loss which is used by Outlier Exposure has an implicit scoring function which differs in a non-trivial fashion from the theoretically optimal scoring function in the case where training and test out-distribution are the same, but is similar to the one used when training with an extra background class. In practice, when trained in exactly the same way, all these methods perform similarly and reach state-of-th of-the-art OOD detection performance.
detection methods have been suggested in recent
INTRODUCTION
While deep learning has significantly improved performance in many application domains, there serious concerns for using deep neural networks in applications which of safety-critical nature. With one major problem being adversarial samples (Szegedy et al., 2014 Madry et al. 2018). which are small imperceptible modifications of the image that change the decision of the classifier, another major problem are overconfident predictions (Nguyen et al. 2015 Hendrycks & Gimpel 2017: Hein et al. 2019) for images not belonging to the classes of the actual task. Here, one distinguishes between far ant-of-distribution data, e.g. different forms of noise or completely unrelated tasks like CIFAR-10 vs. SVHN, and close -utto--distiibution data which can for example occur in related image classification tasks where the semantic structure is very similar e.g. CIFAR-10 vs. CIFAR- 100. Both are important to distinguished from the in-distribution, but it conceivable that close nut-of-distribution data is the more difficult problem with potentially fatal consequences: in an automated diagnosis system we want that the system recognizes that it "does not know" when a new unseen disease comes in rather than assigning high confidence into known class leading to fatal treatment decisions. Thus
awareness is a key property of trustworthy Al systems.
In this paper, we focus the setting of OOD detection where during training time, there is no information available on the distribution of OOD inputs that might appear when the model is used for inference. A large number of different approaches to OOD detection based on combinations of density estimation, classifier confidence, logit space energy, feature space geometry, behaviour on auxiliary tasks, and other principles has been proposed tackle this problem. We give a detailed overview of existing OOD detection methods in Appendix D However, most OOD detection papers are focused on establishing superior empirical detection performance and provide little theoretical background on differences but also similarities to existing methods. In this paper we want to take a different path as we believe that a solid theoretical basis is needed to make further progress in this field. Our goal is to identify, at least for a particular subclass of techniques, whether the differences are indeed due to a different underlying theoretical principle or whether they are due to the efficiency of different estimation techniques for the same underlying detection criterion, called "scoring function". In some cases, we will see that one can even disentangle the estimation procedure from the scoring function, so that one simulate several different scoring functions from one model's 's estimated quantities.

---

Under review as a conference paper at ICLR 2022
A simple approach OOD detection is to treat it as binary discrimination problem between in- and -uitof--iittibutionn or more generally predicting a score how likely the input is OOD. In this paper, we show that from the perspective of Bayesian decision theory, several established methods are indeed equivalent to this binary discriminator. Differences arise mainly from i) the choice of the training out-distribution, e-g. the popular Outlier Exposure of[Hendrycks al. (2019a) has advocated the use of rich and large set of natural images as proxy for the distribution of natural images, and ii) differences in the estimation procedure. Concretely, the main contributions of this paper are:
We show that several OOD detection approaches equivalent the binary discriminator between in- and out-distribution when analyzing the rankings induced by the Bayes optimal classifier/density.
We derive the implicit scoring functions for the confidence loss (Lee et al. 2018a) used by
Outlier Exposure (Hendrycks et al. (2019a) and for using an additional background class for the out-distribution (Thulasidasan et al. 202T). The confidence scoring function turns out not to be equivalent to the "optimal" scoring function of the binary discriminator when training and test out-distributions are the same.
We show that when training the binary discriminator between in- and out-distribution together with standard classifier on the in-distribution in a shared fashion, the binary discriminator reaches state-of-the-art OOD detection performance.
We show that density estimation equivalent to discrimination between the in-distribution and uniform noise which explains the frequent observation that standard density estimates are not suitable for OOD detection.
Even though we identify that a simple baseline is competitive with the sfate-of-the-art, the main aim of this paper is better understanding of the key components of different OOD detection methods and to identify the key properties which lead to SOTA OOD detection performance. All of our findings supported by extensive experiments CIFAR- 10 and CIFAR- 100 with evaluation various challenging -utto---disrriuutin test datasets.
2 MODELS FOR OOD DATA AND EQUIVALENCE OF OOD DETECTION SCORES
As most work in the literature we consider OOD detection on a compact input domain X where the most important example is image classification where X = [0, 1] The most popular approach to OOD detection is the construction an in-distribution scoring function f X + R U ( such that (1) tends to be smaller if is drawn from out-distribution than if it is drawn from the in-distribution. There is a variety of different performance metrics for this task, with a very common one being the area under the receiver-operator characteristic curve (AUC). The AUC for scoring function f distinguishing between an in-distribution p(æl(t) and an out-distribution p(z(o) is given by
=
E
|
+
1 1 [
(1)
x>p(((() y>>(y1o)
We define an equivalence of scoring functions based on their AUCs and will show that this equivalence implies equality of other employed performance metrics as well.
Definition 1. Two scoring functions f and g are equivalent and write 2 g if
AUC, (pC((p(=)), ((C((
for all potential distributions p(zli) and p(xlo).
As the AUC is not dependent on the actual values of but just the ranking induced by ʃ one obtains the following characterization of the equivalence of two scoring functions.
Theorem 1. Two scoring functions f.g g are equivalent f = if and only if there exists strictly monotonously increasing function Φ : range(g) - range(f). such that f o(g).
Corollary 1. The equivalence between scoring functions in Def. [is an equivalence relation.
2

---

Under review as a conference paper at ICLR 2022
Another metric is the false positive rate fixed true positive rate q, denoted as FPR@q @PR.. commonly used value for the TPR is 95%. The smaller the FPR@qTPR, the better the OOD discrimination performance.
Lemma 1. Two equivalent scoring functions f = g have the same FPR@ qTPR for any pair of in- and out out-distributions p(zli), p(z])t),p(z(o and for any chosen TPR q.
In the next section, we use the previous results to show that the Bayes optimal scoring functions of. several proposed methods for out-of-distribution detection are equivalent to the scoring functions of simple binary discriminators.
BAYES-OPTIMAL BEHAVIOUR OF BINARY DISCRIMINATORS AND COMMON OOD DETECTION METHODS
In the following we will show that the Bayes optimal function of several existing approaches to OOD detection for unlabeled data are equivalent distribution whereas differences arise when one has labeled data. As the equivalences are based on the Bayes optimal solution, these are asymptotic statements and thus it has to be noted that convergence to the Bayes optimal solution can be infinitely slow and that the methods can have implicit inductive biases. This is why we additionally support our findings with extensive experiments.
binary discriminator between in- and (training) out-
3.1 OOD DETECTION FOR METHODS USING UNLABELED DATA ONLY
We first provide a formal definition of OOD detection before we show the equivalence of density estimators resp. likelihood to binary discriminator.
The OOD problem In order to make rigorous statements about the OOD detection problem we first have to provide the mathematical basis for doing so. We assume that we are given an in-distribution p(z(i) and potentially also a training out-distribution p(z|o). At this particular point no labeled data is involved, so both of them are just distributions over X. For simplicity we assume in the following that they both have density wrt. the Lebesgue measure on X [0, 1] We assume that in practice we get samples from the mixture distribution
p(x) = p(xl)(((()))))) ((1) + p(xlo)p(()- = (())))p()) + p(=)))(1- - p(i)
where p(t) is the probability that we expect see in-distribution samples in total. In order to make the decision between in n-and out-distribution for given point x it is then optimal to consider
p(ll)) = p(x ((t)))(()
p(=
(4)
p(z) p(x)i)p(1) + p(z(o)p(o)
which is defined for all x E [0, 1jdd with p(x) 0 (assuming p(rli) and p(z|o) can be written as densities). If the training out-distribution is also the test out-distribution then this is already optimal but we would like that the approach generalizes other unseen test out-distributions and thus an important choice is the training out-distribution p(z]o). Note that as p(i|æ) is only well-defined for all with p(z) > 0. it is thus reasonable to choose for p(z(o) a distribution with support in [0, 1]d, that is p(z|o) 0 for all E [0, 11. In this case we ensure that the criterion with which we perform OOD detection is defined for any possible input x. This is desirable as OOD detection should work for any possible input & E X.
Optimal prediction of binary discriminator between in- and out-distribution We consider binary discriminator with model parameters 0 between in- and (training) out-distribution, where (o(i) is the predicted probability for the in-distribution. Under the assumption that p(i) is the probability for in-distribution samples and using cross-entropy (which in this case is the logistic loss up to a constant global factor of l2) the expected loss becomes:
min p(i) x<)p(((()
E
[- log po (ilz) + p(o)
(=og(11po (i(x))
(5)
>=<p(=10)
One can derive that the Bayes optimal classifier minimizing the expected loss has the predictive distribution:
p(z)))p(1)
Pa- (i]x)) p(xli)p(i) + p(alo)pp() =((l)))
(6)
3

---

Under review as a conference paper at ICLR 2022
Thus at least for the training out-distribution binary classifier based on samples from in- and (training) out-distribution would suffice to solve the OOD detection problem perfectly.
Equivalence of density estimation and binary discrimination for OOD detection In this section we further analyze the relationship of common OOD detection approaches with the binary discrimina- tor between in-and out-distribution. We start with density estimators sourced from generative models. A basic approach that is known to yield relatively weak OOD performance (Nalisnick et al., 2019)] Ren et al. 2019 Xiao et al., (020)) is directly utilizing model's estimate for the density p( i) at sample input æ. An improved density based approach which uses perturbed in-distribution samples as a surrogate training aut-distribution is the Likelihood Ratios method (Ren et al.) 2019), which proposes to fit generative model for both the in- and out-distribution and to use the ratio between the likelihoods output by the two models as a discriminative feature.
We show that with respect to the scoring function, the correct density p(z](i) is equivalent to the Bayes optimal prediction of a binary discriminator between the in-distribution and uniform noise. Furthermore, the density ratio p(als) is equivalent to the prediction of a binary discriminator between the two distributions which the respective models used for density estimation have been trained. Because of this equivalence. we argue that the use of binary discriminators is simple alternative to these methods because of its easier training procedure. While this equivalence is asymptotic statement, the experimental comparisons in the appendix show that the methods perform similarly poorly compared the methods using labeled data.
p(z]oj
We first prove the more general case of arbitrary likelihood ratios. In the following we use the abbreviation À p(o) p(3) to save space and make the statements more concise.
Lemma 2. Assume that p(æli) and p(z(o) can be represented by densities and the support of p(xlo)
covers the whole input domain X. Then p(=])) &
p(aro) p(ll))AP((a for any X >> 0.
p(a)()
This means that the likelihood ratio score of two optimal density estimators is equivalent the in- distribution probability Pa- (ilz) predicted by a binary discriminator and this is true for any possible ratio of p(i) to p(o). In the experiments below, we show that using such discriminator has similar performance as the likelihood ratios of the different trained generative models.
For the approaches that try to directly use the likelihood of a generative model as a discriminative feature, this means that their objective is equivalent to training binary discriminator against uniform noise, whose density is PUniform( (I) p(x) 0) = 1 at any x.
Lemma 3. Assume that p(æ|i) can be reprexented by density. Then p(zli) = A >>0 O.
p(zl() P for any
This provides additional evidence why a purely density based approach for many applications proves to be insufficient as an OOD detection score on the complex image domain: it is not reasonable to assume that binary discriminator between certain classes of natural images on the one hand and uniform noise on the other hand provides much useful information about images from other classes or even about other nonsensical inputs.
3.2 OOD DETECTION FOR METHODS USING LABELED DATA
We first discuss how one can formulate the OOD problem when one has access to labeled data for the in-distribution and identify the target distribution of OOD detection using a background/reject class. Then we derive the Bayes optimal classifier of the confidence loss (Lee et al.) 2018a) as used by the most successful variant of Outlier Exposure (Hendrycks al. 2019a) and discuss the implicit scoring function. In most cases the scoring functions turn out not to be non-equivalent to p(i]z) (which is optimal if training and test out-distribution agree) as they integrate additional information from the classification task.
Bayes optimal solutions for OOD Detection with Background class and confidence loss Outlier Exposure Given joint in-distribution p(y, (where y E {I...., K} given that have K labels) for the labeled in in-distribution, there are different ways how to come up with a joint distribution for in- and out-distribution. Interestingly, the different encodings used e.g. in training with
4

---

Under review as a conference paper at ICLR 2022
background class (Tuulasidssan al., 2021) vs. training classifier with confidence loss (Lee et al. 2018a) together with variants of the employed scoring function lead to methods which unexpectedly can have quite different behavior.
Background class: In this case we just put all out-of-distribution samples into + 1-class which is typically called background//eject class (Thulasidasan et al., 2021). The joint distribution then becomes
p(y. .)) plux211)p00 (p(zlo)(())
if E {l,
if +1
We denote by p(z]i) = Σكm y=1 p(y. x/1) the marginal in-distribution and note that the marginal distribution of the joint distribution of in- and out-distribution is again given by
p(z) p(z(1)((1) p(xlo)p(o).
Then we get the conditional distribution
p(yl>) = [p(y|z ()/()
y E f1, fl.....K) K}. if = K +
= 10
The Bayes optimal solution of training with background class using any calibrated loss function Lyy, f())), e.g. the cross-entropy loss (Laptev et al. 2016), then yields Bayes optimal classifier f* which has a predictive distribution p (y|z) p(y (.). There are two potential scoring function that come to mind:
$1 x 1 Py- (K 1/z) and $2
max Ps- (k|z) k= L..., K
The first one, used in Chen et al. (2021): Thulasidasan et al. (2021).
I. is motivated by the fact that Py. (K + 1|x)) is directly the predicted probability that the point is from the out-distribution as indeed holds: (1r = p(ilz) which is the optimal scoring function if training and test out-distribution equal. On the other hand the maximal predicted probability max;=1.....K K p (kt(ar), which is often employed as a scoring function Hendrycks & Gimpel 2017), becomes for the Bayes optimal classifier
$2 (z) = p(ilz) max p(k|z,)),
which is a product of p(i]x) and the maximal conditional probability of some class of the in- distribution (note that is well defined as p(i]zz) is defined if p(r|o) has support everywhere in X and if p(i]x) > then also p(xli)= > 0). Thus the scoring function $2 integrates additionally to p(ilz)) also class-specific information and is thus less dependent distribution. In fact, one can see that 82 only ranks points high if both the binary discriminator and the classifier rank the corresponding point high. On the other hand in the case where training and test out-distribution are identical, this scoring function is not equivalent to p(i) bias in the estimation.
the chosen training out-
(z) and thus introduces a
Outlier Exposure Hendrycks et al. 2019a) with confidence loss (Lee et al. 2018a): we analyze the Bayes optimal solution for the confidence loss (Lee et al.) (2188)) that used by Outlier Exposure (OE) and show that the associated scoring function can be written, similarly to the scoring function s2(2) for training with a background class, as function of p(ilæz) and p(y(z,
The training objective with the confidence loss is in expectation given by
min (#w)))((.)1)) [Cce(fo(x),
E so<p(wlo)
[Ccs(fo(x), (x)
where are the model parameters and fe(x) E RKK is the model output as logits, and u. [ ( LTT is the uniform distribution over the K classes of the in-distribution classification task.
In the following theorem we derive the Bayes optimal predictive distribution for this training objective.
The predictive distribution py- (yIlz) of the Bayes optimal classifier f minimizing the expected confidence loss is given for y E {1...., K} as
Theorem 2.
Py- (y-()/=) = p(il=)p(ulz, i) + K K -1-(pi)=))
(8)
5

---

Under review as a conference paper at ICLR 2022
Thus the effective scoring function of using the probability of the predicted class suggested in Hendryeks & Gimpel (2017); Lee et al. (2018a); Hendrryks et al. (2019a) is given by
(1(x)) p(ilar)
max p(y(z, + マे y=1...,K
p(ilz)) p())))
max y=1....
plule, i) K
K K
1
Please note that the term inside the brackets is positive as max;=1....ff p 2) 2 K Interestingly. the scoring functions 82 and 83 are not equivalent even though they look quite similar. In particular, due to the subtraction of K the scoring function 83 puts more emphasis on the classifier than 82. In Appendix | we additionally analyze Energy Based OOD Detection (Liu et al. 2020). and show that the Bayes optimal decision is equivalent using the scoring function $1 p(lr).
3.3 SEPARATE VERSUS SHARED ESTIMATION p(i)ær) AND p(yl
So far we have derived that at least from the point of view of the ranking induced by the Bayes optimal solution, OOD detection based on generative methods, likelihood ratios and the background class formulation with the scoring function is equivalent to binary classification problem between in- and out-distribution in order to estimate p(i[a:). The differences arise mainly in the choice of the training out- dut-iistriuutinn ((z]o): i) uniform for generative resp. density based methods, ii) a quite specific out-distribution for likelihood ratios (Ren et al. 2019) and iii) a proxy of the distribution of all natural images (Hendrycks et al. 2019a Thulasidasan et al. 202T
On the other hand when labeled data is involved we can additionally train classifier on the in- distribution in order to estimate p(y (z, i). We will then combine the estimates of p(ilz) and pzylz i) according to the three scoring functions derived in the previous section and check if the novel OOD detection methods constructed in this way perform similar the OOD methods from which derived the corresponding scoring function i) OOD detection with a background class (Thulasidasan al. 2021) ii) using Outlier Exposure Hendrycks et al. (2019a). This will allow to differentiate between differences of the employed scoring functions for OOD detection and the estimators for the involved quantities. In this way foster a more systematic approach OOD detection.
In the unlabeled case we train simply the binary classifier pe [0, 1j - IR using logistic/cross entropy loss in a class balanced fashion
log (Po(il=!N))
M Σ log (1 po(il=)UT)
where ( I and (a:) ( b- are samples from the in-distribution and the out-distribution.
In the case where we have labeled data we can additionally solve the classification problem. The obvious approach is train the binary classifier for estimating ((]])) and the classifier estimate p(y|æ; i) completely independently. Not surprisingly, we show in Section 4 that this approach works less well. In fact both tasks benefit from each other. Moreover, in training neural network using background class or with Outlier Exposure (Hendrycks et al. 2019a) we are implicitly using a shared representation for both tasks which improves the results.
Thus we propose to train the binary discriminator of in-versus out-distribution together with the classifier on the in-distribution jointly. Concretely, use neural network with K + outputs where the first K outputs represent the classifier and the last output is the logit of the binary discriminator. The resulting shared problem can then be written as
No
M
No log (Do(y) MaN) (10)
min 0
log ()o(il=!N)
M Σ
log (1 - (0(()2
No
N
where X= = plo) p(o)
which is typically set to 1 during training in order to get a blaanebllanced problem.
p(4)
Note that the in-distribution samples (i:,N) )
> used to estimate p(i)ær can be a super-set of the labeled examples (a/N, .f >' used to train the classifier so that one can potentially integrate unlabeled data this is advantage compared to OOD detection with a background class or Outlier Exposure where this is not directly possible. We stress that the loss functions of the classifier and the discriminator act independent outputs: the functions modelling the two tasks only interact with each other due to the shared network weights up to the final layer. Nevertheless, we see in the next Section 4 that training with shared representation boosts both the classifier and the binary discriminator.

---

Under review as a conference paper at ICLR 2022
Table I: Accuracy the in-distribution test out-distributions of different OOD methods with Openlmages as training out-distribution (results for the of OpenImages are used in the mean FPR). Lower false positive rate is better. All methods except Mahalanobis have been trained using the same architecture, training parameters, schedule and augmentation. $1, $2, 83 are the scoring functions introduced Section 3.2 Our binary discriminator (BINDISC) resp. the combination with the shared classifier (SHARRE COMBI) and the models with background class (BGC) with scoring functions $2 or 83 outperform the Mahalanobis detector (Lee et al. 201 8b) and are similar to Outlier Exposure (Hendrycks et al. 2019a). CelebA makes no sense as test out-distribution for CIFAR 100 as man/ woman are classes in CIFAR-100.
and FPR@95%TPR for various
IN-DISTRIBUTION: CIFAR-10
MEAN
SVHN FPR
LSUN
UNI
SMOOTH
80M FPR
CELA FPR
OPENIM FPR
MODEL
ACc.
FPR
FPR
FPR
FPR
FPR
PLAIN CLASSI
96.16
53.01
47.87
50.00
17.51
65.81
60.43
53.44
76.00
63.71
MAHALANOBIS
36.68
20.97
49.00
0.00
0.00
57.21
48.85
80.71
55.13
OE
95.06
15.20
9.58
0.00
0.00
0.00
54.05
42.33
0.45
3.46
BGC 81
18.83
2.36
0.00
0.00
0.00
72.00
56.41
1.04
0.05
BGC 82
95.21
16.52
7.51
0.00
0.05
2.10
55.16
44.57
6.26
1.65
BGC $3
95.21
16.63
7.69
0.00
0.07
2.36
55.19
44.67
6.41
1.74
SHARED BINDISC
19.56
4.65
0.00
0.00
0.00
77.50
53.93
0.87
0.04
SHARED CLASSI
95.28
29.34
28.00
7.00
2.33
33.04
58.61
47.90
28.54
35.94
SHARED COMBI
95.28
16.00
8.56
0.00
0.00
0.00
58.80
42.79
1.83
0.61
SHARED COMBI 83
95.28
16.06
9.00
0.00
0.00
0.00
58.68
42.85
1.91
0.66
IN-DISTRIBUTION. CIFAR- 100
MEAN FPR
SVHN
LSUN
UNI
SMOOTH
C-10
80M
OpEnIm
MODEL.
Acc.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
PLAIN CLASSI
77.16
67.57
75.50
78.33
22.60
70.98
80.55
77.43
80.80
MAHALANOBIS
53.88
54.36
66.00
46.43
0.06
85.39
71.01
74.69
OE
77.19
35.03
47.36
0.00
0.67
0.08
84.64
77.42
1.28
BGC 61
31.14
11.58
0.00
0.00
0.00
93.94
81,29
0.07
BGC
77.61
33.32
37.06
0.00
0.00
0.20
84.50
78.17
1.26
BGC 83
77.61
33.36
37.27
0.00
0.00
0.20
84.51
78.19
1.27
SHARED BINDISC
31.86
10.77
0.00
0.00
0.00
95.25
85.11
0.08
SHARED CLASSI
77.35
67.23
71.05
5.00
97.70
69.68
82.05
77.89
28.38
SHARED COMBI 82
77.35
33.01
37.30
0.00
0.00
1.06
82.71
77.01
1.80
SHARED COMBI $3
77.35
33.06
37.57
0.00
0.00
1.13
82.68
77.01
1.85
EXPERIMENTS
We use CIFAR-I0 and CIFAR-100 (rizhevsky Hinton 2009) datasets in-distribution and OpenImages dataset Krasin et al. 2017) as training out-distribution. The 80 Million Tiny Images (80M) dataset (Torralba al.) 2008) the de facto standard for training out-distribution aware models that has been adopted by most prior works, but this dataset has been withdrawn by the authors as Birhane Prabhu (202T) pointed out the presence of offensive images. To be able to compare with other state-of-the-a methods without introducing potential bias due to dataset selection, we include the evaluation with 80M as training out-distribution in Appendix HH] Moreover, we show in the appendix results for the binary discriminator trained with different training out-distributions vs. likelihoods resp. likelihood ratios (Ren et al. 2019) as OOD method.
We use as OOD detection metric the false positive rate at 95% true positive rate, FPR @95%
evaluations with AUC are in Appendix G We evaluate the OOD detection performance on the following datasets: SVHN (Netzer et al. 1) resized LSUN Classroom Yuu et al.) 2015). Uniform Noise, Smooth Noise generated as described by (Hein et al.) 2019). the respective other CIFAR dataset, 80M, and CelebA (Liu et al. 2015). We highlight that none of the listed methods has access

---

Under review as a conference paper at ICLR 2022
to those test distributions during training for fine-tuning as try to assess the ability of an out-distribution aware model to generalize to unseen distributions. The FPR for the Openlmages test set is not included in the Mean AUC, since this distribution has been used during training.
The binary discriminators (BINDISC) as well as the classifiers with background class (BGC) and the shared binary discriminattor-classfffer (SHArEd) of p(ilz)) and p(y|ær i) are trained on the 40-2 Wide Residual Network (Zagoruyko & Komodakis 2016) architecture with the same training schedule as used in Hendrycks et al. (2019a) for training their Outlier Exposure(OE) models. This includes averaging the loss over batches that are twice as large for the out-distribution. This way we ensure that the differences do not arise due to differences in the training schedules or other important details but only on the employed objectives. In addition to their standard augmentation and normalization, we apply Automugment (Cubuk et al., (019)) without Cutout, and we use X = 1 where applicable, which is sound choice as we observe in an ablation on À in Appendix K For the Mahalanobis OOD detector (Lee et al.) (2018b), we use the models and code published by the authors and use OpenImages for the fine tuning of input noise and layer weighting regression, We describe the exact
details of the training settings and the used dataset splits in Appendix C
4.1 OUT-DISTRIBUTION AWARE TRAINING WITH LABELED IN-DISTRIBUTION DATA
In Table D we compare multiple OOD methods trained with training out-distribution OpenImages and CIFAR- DIFAR-IO/100 as in-distribution: confidence of standard training (PLAIN) and OE, MAHALANOBIS detection, classifier with background class (BGC) and the combination of a plain classifier and binary w---v-ouu-dissriibbe
classifier with shared representation (SHARED COMBI). As described in Section 2 both BGC and SHARRD COMBI can be used in combination with different scoring functions. For BGC, we evaluate all three scoring functions 81, 82 and 83 and for SHARED COMBI we only use $2 and $3 81 is equivalent to p(t]æz which is the output of SHARED BINDISC. Additionally, we evaluate OOD detection based on the confidence of the shared classifier (SHARED CLASSI) trained together with SHARED BINDISC.
For CIFAR-10, a first interesting observation is that SHARED CLASSI has remarkably good OOD performance; significantly better than normal classifier (plain) even though it is just trained using normal cross-entropy loss and so the OOD performance is only due to the regularization enforced by the shared representation with SHARED BINDISC. In fact SHARED BINDISC has already good OOD performance with a mean FPR @ 9PRRPR of 19.56, which is improved by considering scoring function $2/53 in the combination of SHARED BINDISC and SHARED CLASSI which yields very good classification accuracy and mean FPR/AUC. Moreover, interesting are the results of the classifier with background class (BGC) which is the method recently advocated in Thulasidasan et al. 2021). It works very well but the performance depends on the chosen scoring function. Whereas (oupput of the background class) is a usable scoring function (mean FPR: 18.83), the maximum probability over the other classes 82 (mean FPR: 16.52) the combination in terms of 83 (mean FPR: 16.63) performs better. In total with the scoring function 82/83 83 integrating classifier and discriminative information, BGC reaches similar performance to OE (which implicitly also uses $3 scoring function). In general, the differences of the methods are relatively minor both in terms of OOD detection and classification accuracy, where the latter is better for all OOD) methods compared the plain classifier; this is most likely explained by better learned representations, see also Hendrycks) al. (o19a);[AAgustie et al. (2020) for similar observations. The results for CIFAR- 100 are similar to CIFAR-IO, with some reversals of the overall rankings of the compared methods. OE achieves comparable OOD results BGC s2/53 and SHARED COMBI $2/S3, For this in-distribution our BGC $1 and SHARED BINDISC perform best in terms of OOD performance. Classification test accuracy is slightly higher for BGC and SHARED, but the differences are minor. The experiments with 80M as training out-distribution (Table in Appendix H) confirm these observations.
Overall, as suggested by the theoretical results on the equivalence of the Bayes optimal classifier of OE with the 83 scoring function of BGC and SHARED COMBI, we observe that even though these methods are derived and in particular trained with quite different objectives, they behave very similar in our experiments. In total we think that this provides much better understanding where differences of OOD methods are coming from. Regarding the question of which method and scoring function should be used for given application, the experimental results across datasets and different out aut-distributions, see Appendix H suggest that their difference is minor and there is no clear best choice. However, in Appendix
B we describe potential situation where the 83 score and in
8

---

Under review as a conference paper at ICLR 2022
consequence OE is not powerful enough to distinguish in- and out-of-distribution inputs. On the other hand, in cases where the $1 score not very informative as training and test out-distributions largely differ, combining with the classifier confidences is beneficial; this can be observed in experiments with SVHN as training out-distribution which we show in Appendix This is why for an unknown situation, we recommend BGC SHARED COMBI with the #2 scoring function the safest option. However, it is an open question if there are also situations where $2 is fundamentally inferior to $3.
4.2 SHARED REPRESENTATION LEARNING FOR THE BINARY DISCRIMINATOR
As highlighted above the shared training of SHARED CLASSI and SHARED BINDISC and their combination SHARED COMBI with 82/ 83 as scoring functions yields strong OOD detection and test accuracy among all methods. Here, we evaluate the importance of training the binary discriminator and the plain classifier with a shared representation in comparison to training two entirely separate models PLAIN CLASST and SEPARATE BINDISC and their combination SEPARATE COMBI with scoring function $3. The results for CIFAR- 10 and CIFAR- 1-0 can be found in T In total, we see that separate training particular for CIFAR- 100 leads to worse results compared shared training expected as the binary discriminator and the classifier cannot benefit from each other. An interesting curiosity is that the combination of the separate classifier with the binary discriminator trained in shared fashion (PLAIN & SHA DISC) yields almost the same OOD results as SHARRD COMBI even though the classifier significantly worse. Overall, SHARED COMBI performs significantly better when also considering the better classification accuracy which it inherits from SHARED CLASSI.
Table 2:
Evaluation (same metrics in Table T of models trained with shared and separate representations. Shared training benefits both the classifier and the binary discriminators.
IN-DISTRIBUTION: CIFAR-10
MEAN
SVHN
LSUN FPR
UNI
SMOOTHC-
100
80M
CELA
OPENTM
MODEL
ACC.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
FPR
SEPARATE BINDISC
23.49
6.21
0.00
0.00
0.00
83.79
65.77
0.00
PLAIN CLASSI
95.16
53.01
47.87
50.00
17.51
65.81
60.43
53.44
76.00
63.71
SEPARATE COMBI S3
95.16
21.40
13.15
0.00
0.00
0.00
59.96
49.78
26.93
0.45
SHARED BINDISC
19.56
4.65
0.00
0.00
0.00
77.50
53.93
0.87
0.04
SHARED CLASSI
95.28
29.34
28.00
7.00
2.33
33.04
58.61
47.90
28.54
35.94
SHARED COMBI
95.28
16.06
9.00
0.00
0.00
0.00
58.68
42.85
1.91
0.66
PLAIN @ SHA DISC
95.16
15.96
8.10
0.00
0.00
0.00
58.48
42.60
2.53
0.70
IN-DISTRIBUTION: CIFAR-100
MEAN
SVHN
LSUN
UNI
SMOOTH
C-10
80M
OPENIM
MODEL
Acc.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
FPR
SEPARATE BINDISC
32.50
14.28
0.00
0.00
0.00
96.50
84.22
0.02
PLAIN CLASSI
77.16
67.57
75.50
78.33
22.60
70.98
80.55
77.43
80.80
SEPARATE COMBI
77.16
41.94
69.44
0.00
0.00
24.39
81.15
76.67
0.89
SHARED BINDISC
31.86
10.77
0.00
0.00
0.00
95.25
85.11
0.08
SHARED CLASSI
77.35
67.23
71.05
5.00
97.70
69.68
82.05
77.89
28.38
SHARED COMBI
77.35
33.06
37.57
0.00
0.00
1.13
82.68
77.01
1.85
PLAIN SHA DISC 83 77.16
33.38
37.00
0.00
0.00
5.42
81.33
76.50
2.23
CONCLUSION
In this paper we have analyzed different OOD detection methods and have shown that the simple baseline of a binary discriminator between in-and out-distribution is powerful OOD detection method trained a shared fashion with a classifier. Moreover, we have revealed the inner mechanism of Outlier Exposure and training with background class which unexpectedly scoring function which integrates information from pil]z) and p(y| i). We think that these findings will allow to build novel OOD methods in more principled fashion.

---

Under review as a conference paper at ICLR 2022
6 ETHICS AND REPRODUCIBILITY STATEMENT
In this paper we provide an explanation for the inner workings of established OOD detection methods and propose novel OOD detection method based on these considerations which outperforms OE in terms of OOD performance and test accuracy. The final goal is to have more trustworthy classifiers. One could criticize that the focus on OOD FPR/AUC performance and test accuracy covers just certain aspects and other aspects like calibration of the classifiers, fairness, robustness to corruptions or adversarial attacks play an important role, too. However, apart from the usual dual use problem we see only positive societal aspects of our paper, as it leads to more trustworthy MI. methods.
We discuss the problematic situation with the retracted 80 Million Tiny Images (Torralba al. 2008 dataset - which is used by many previous works in the field - in Appendix [n] replace the it with Openlmages (Krasin et al. 2017) for training the models in the main paper. For comparability with previous methods, we include evaluations of models trained on 80M in the appendix. We hope that introducing an alternative dataset for natural surrogate OOD training helps the community towards avoiding the retracted dataset in the future, both from an ethical and also a practical perspective, in case 80M becomes fully unavailable.
All experimental details including used hardware given in Appendix C and code for training and evaluating out methods as well as weights of the evaluated neural networks are available at https: bttts::/aaooymoonn 4open. www.:scceeeeeeeooeeeeeee. BGC BinDisccDDDiii D7FB
REFERENCES
Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in-and out-distribution improves explainability. In F.CVV, 2020.
Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? WACV, pp. 1537-1547, 2021.
Christopher M Bishop. Novelty detection and neural network validation. IEEE Proceedings- Image and Signal processing, 141:217-222, 1994.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Informative outlier matters: Robustifying out-of-distribution detection using outlier mining. In ECML, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
J. Deng. W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. arXiv preprint 2018.
arXiv:
Will Grathwohl, Kuan-Chich Wang, Joem- Heenik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly energy based model and you should treat it like one. In Interna- tional Conference on Learning Representations, 2020. URL
wwwp://wwwwwwww.oooouu.u..eaeee. fet//ooumm?ir
HkxzxONtCB
M. Hein, M. Andriushchenko, and J. Bitterwolf. Why ReLU networks yield high- high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019.
D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
D. Hendrycks, M. Mazcika, and T. Dietterich. Deep anomaly detection with outlier exposure. In /CLR, 2019a. https: /tttps://githu...
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self supe-sup learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems (NeurIPS). 2019b.
10

---

Under review as a conference paper at ICLR 2022
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-EI-Hlajj, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongic, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public datasct for large-scale multi-label and multi-class image classification. Dataset available from http:://toraggegonggeeeee wwwp:p/ss..oooopiiieeeeeeeeeeeee 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citescer, 2009,
D. Laptev, N. Savinov, J.M. Buhmann, and M. Pollefeys. Tlppolinn:: feature learning in convolutional neural networks. In CVPR, 2016.
Transformaation-nvvaiinn pooling for
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. 2006.
K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence--alibbraeed classifiers for detecting out-of-distribution samples. In ICLR, 2018a.
Lee, H. Lee, K. Lee, and J. Shin. A simple unified framework for detecting ont-of-distriibttion samples and adversarial attacks. In NeurlPS, 2018b.
Maa___Maaaaaaa anobi s
detector
Kimin Lee. Kibok Lee. Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In NeurIPS, 2018c.
Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13218-13227, 2020.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
Hao Liu and Picter Abbeel. Hybrid diseriminativv-generaatt training via contrastive learning. arXiv preprint arXiv:2007.09070 2020.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -fist--diittiuution detection. Advances in Neural Information Processing Systems, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes wild. ICCV, 2015.
A. Madry, A. Makelov, I.. Schmidt, D. Tsipras, and A. Valdu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In ICLR, 2020.
Sina Mohseni, Mandar Pitale, Jbs Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distriutiio detection. In AAAI, 2020.
E. Nalisnick, A. Matsukawa, Y. Whye Teh, D. Gorur, and B. Lakshminarayanan. Do deep generative models know what they don't know? In /CLR, 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features. In ICMI,, 2019.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits natural images with unsupervised feature learning. In NeurlPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
Angells Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for out-of-distrriuttion detection. Neurocomputing, 441:138-150, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-peeformance deep learning library. In NewrlPS, pp- 8024-8035, 2019.
11

---

Under review as a conference paper at ICLR 2022
F. Pedregosa, G. Varoquaux, Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, Cournapeau, M. Brucher, M. Perrot, and Duchesnay, Scikit-lcarn: Machine learning in Python. JMLR, 12:2825- 2830, 2011.
Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark DePristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihond ratios for out-of-distribution detection. In NewrIPS, 2019.
Abhijit Guha Roy, Jessie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach William Beaver, Nam Vo, Peggy Bui, Samantha Winter, Patricia Mac Williams, Greg Corrado, Umesh Telang, Yun Liu, Taylan Cemgil, Alan Karthikesalingam, Balaji Lakshminarayan and Jim Winkens. Does your dermatology classifier know what it doesn't 't know? detecting the long-tail tail of conditions. Medical Imaging Analysis, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjecv Satheesh, Scan Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)211-252, 2015. doi: 10. 1007/s1 -6--1..--
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: A pixelenn implementation with discretized logistic mixture likelihood and other modifications. In ICLR, 2017.
C. Szegedy, W. Zaremha, Sutskever, Bruna, D. Erhan, I. Goodfellow, and Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive Icarning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhattacharya, and Jeff
Bilmes. An effective baseline for robustness to distributional shift. arXiv: 2105.07107, 2021.
Antonio Torralba. Rob Fergus, and William Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE PAMI, 30(1 19:8198-1970 2008
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2019. URL httpps:/ccperret
net
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingum, Simon Kohl, al. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.055666 2020.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. In NeurlPS, 2020.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans the loop. CoRR, abs/1506.03365, 2015.
S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, pp. 87.1 2016.
12

---

Under review as a conference paper at ICLR 2022
A
PROOFS
Theorem 1. Two scoring functions f.g are equivalent f = g if and only if there exists strictly monotonously increasing function & range(g) - range(f), such that f = Q(g).
Proof. Assume that such function @ exists. Then for any pair x, y we have the logical equivalences g(x) > gly) è f() ((g(z)) > ((((v)) = f(y) and g(x) = g(y) + f(x) d(g(x))
((g(v)) = fly). This directly implies that the AUCs are the same, regardless of the distributions.
Assume f g. For each a E range(g), choose some E g ga) For any pair x. E X, by regarding the Dirac distributions p(x]i) = 82 and p(arlo) 6y that are each concentrated on one of the points, we can infer that f(x:) > [(y) + (p(2)))) = + AUC, (p(x))), p(x]o)) =
1 g(x) > ( and similarly f(x) fly) @ g(x) g The latter ensures that the function
defined as
range(g) range(f)
(11)
a + f(@)
is independent of the choice of à and that f ó g. and the former confirms that ø is strictly monotonously increasing.
Lemma 1. Two equivalent scoring functions f = g have the same FPR@ qTRR@TPRfoe for any pair of in- and out-iistissttions p(x))),p(zz(o and for any chosen TPR q.
Proof. We know that a function o as in Theorem I cquivalences
exists. Then for any pair æ, y, we have the logical
g(x) > g(y) + f(x) = o(g(z)) > d(g(v)) = f(y)
(12)
and
g(x) ((y) = = f(y)
(13)
This directly implies that the FPR @ MIIP-PRllueee are the same, for any p(x] p(xlo) and q.
Lemma 2. Assume that p(z)i) and p(z(o) can be represented by densities and the support of p(rlo)
covers the whole input domain X. Then p(zli) ~ p(
p(ali) p(=10+>p(=(o)) for any X > 0.
Proof. The function @ [0, oo] + [0. 1] defined by <(x) = a (setting Φ(0o) 1) fulfills the
criterion from Theorem T of being strictly monotonously increasing. With
p(a
p(zli) p(x(o)
p(zli) p(zli) + Ap(x(o) +p(z(o)
p(Z|o]
(14)
p(a
+ A pizlo)
p(2(0)
p(z|o)
for p(xlo) + and p(acls)
x(00)= =
+ the equivalence follows.
p(zl))
Lemma 3. Assume that p(zli) can be represented by density. Then p(zli) = p(a >>0 0.
p( for any
Proof. This a special case of Lemma 2 by setting p(x(o) = PUniform(1))
Theorem 2.
The predictive distribution py- (y/x) of the Bayes optimal classifier 8 minimizing the expected confidence loss is given for y E {l...., K} as
(y-(v/>) = p(il=)p(ula, i) + /
K -1(((()))
(8)
13

---

Under review as a conference paper at ICLR 2022
Proof. Minimizing the loss of Outlier Exposure given in Eq. problem
7 means solving the optimization
K Σ
K 1
min - -(((/)) py((x)
p(k(z,
log log (k|2) - -1 p(ilx))
Σ
log (k|z)
K
Rm
km
subject to
(k(k)) 2 for each &s E fl,.... [l.....K
(15)
K Σ
(k)
where pf(-1z) is the model's K-diimensiona prediction. For p(i]z) 0 p(k| 2) 0, the optimalities of the respective terms are easy to show (applying the common conventions for Olg 0), so we assume that tose non-zero. The Lagrange function of the optimization problem is
(())(11).,,,3)== - -((())) Σ pk]]z, i)- log - (( (1-p(ilx))
Σ
lo p, (k/)))
k
K
+
(1+
-1 > 11++D((A(()
(16)
with 3 E R and R.F. Its first derivative with respect to py (x(k(2x)) for any k is
0L
1
1 K (p((k(z) ag + 3
-(((z)) p(k| 20, 1) (y({(x)) (1 p(i(x))
(17)
k (k|z))
Oak + 3 (k(rr
where set R (k|z) p(i)=)p(((z, diagonal matrix for any point of its domain, therefore we find the unique minimum by setting 17) to zero, i.e. at
The second derivative is positive
sK (k(lz))
(A(x) B- a
(18)
The dual problem is hence the maximization (with as 2 0) of
p(k|z, i) log SK (k|z) (1 - p(ilx))
K
1
s (k|z)
a(a, B) -p((
Σ
log OpK
K
Oak
k=
K (k(z))
5 (k|s:)
Σ ak 3 ak
+ -1 Σ
B akk
k=1
K
K (k/z))
- log x [kkz)) + log(ß ox)) +
B
03
- 33
B ax
3 a3
here, only appears in log(3 ax) which has a positive factor s (k|z). so
0 maxi-
mizes the expression. Noting Σ s (k)zz) = 1, what remains is (B) = + log( EK SK (k|z) log sK (k|ær) B, which is maximized by 3 = This means that the dual opti-
mal pair is (k(k)))) = Lk (k:].), (3 1, 0). Slater's condition (Boyd et al. 2004 holds since
the feasible set of the original problem is the probability simplex. Thus, Pf. (1ar) s (x) is indeed primal optimal.
B CONFIDENCE LOSS MODELS AND MODELS WITH BACKGROUND CLASS OR A BINARY DISCRIMINATOR ARE ONLY EQUIVALENT AFTER 83 IS APPLIED
Seeing that for both in nn-distribution accuracy and OOD detection. OE models trained with confidence loss, models with background class and shared classiferrddiseriiinntoo combinations behave very
14

---

1 Under review as a conference paper at ICLR 2022
similarly, the question arises if the training methods themselves lead to equivalent models. One idea might be that and effect of the confidence loss on the degree of freedom from logit translation invariance could be unfolded to obtain a K + 1-dimensional output that contains the same information as a classifier with background class or with and additional binary discriminator output (the latter two indeed equivalent). This is not the case, as the following example that in certain situations, the and $2 scores of background class models and classfierrdiscriminato combinations are able to separate in- and out-distribution, while the 33 score and the equivalent confidence loss/OE models cannot.
B.1 EURO COIN CLASSIFIER
As an example where the mentioned non-equivalence would oceur, we hypothetically regard the task of classifying photos of 1-Furo coins by the issuing country. Each E1 coin features common side that is the same for each country and a national side that pictures unique motive per country. We assume that one side is visible each photo, and that the training dataset of size 2mK is balanced, consisting of 2m coin photos with label e for each country c E {I,..., the common side and the other photos show the informative national side for each country c.
, K) J. where m photos show
is easy to see that the Bayes optimal classifier trained with cross-entropy loss on this dataset predicts 100% for the respective country when shown photo of the national side of El coin, and predicts 1/K for each country when shown the common side of a El coin.
Now compare the behaviour of the different methods given training out-distribution of poker chips images which are clearly recognizable as not being E1 coins.
A -lelass model trained with confidence loss (Lee al. 2018a) will not make difference between common side coin images and poker chip images, and in the Bayes optimal case, it will predict the uniform class distribution in both cases. This does not only hold for the prediction of a hypothetical Bayes optimal model: assuming full batch gradient descent and identical sets of m common side training photos for each class, the loss for a common side input the same as the loss for poker chip.
On the other hand, a binary discriminator will easily distinguish between poker chips and El coins, no matter which side of the coin is shown. The same holds for a model with background class: the score of the class K 1 will be close to 1 for chips and close to for El coins.
We conclude that in the described situation, models trained with confidence loss/outlier exposure are not able to sufficiently distinguish in- and dit-iistribution, while the 81 scoring function of a classifier with background class or binary discriminator is suitable for this task.
With the $2 scoring function, the background class model gives us S2 $2(x) = max Py (k|z).
< s2 (x) < 1 Py (K + 1ar),, which means that if py + 1xx) is sufficiently large for in-distribution inputs and sufficiently small for out-of-distribution inputs, 32 is able to distinguish them independent of inconclusiveness in the first K classes. Similarly, 82 applied to
and thus (K-p,
1-pp
K
binary discriminator with a classifier (shared trained or not) will be able to distinguish common sides of coins and poker chips.
With $3, on the other hand, common sides of coins and poker chips can no longer be separated. For a classiferrdiscriminntor pair, as defined above, s3(x) = pa(ila) max Pe(vla, i)
+
If on the common side of a coin the classifier predicts uniform K. we have s2(x) 1 K what the discriminator Pa(i)z) (x) predicts. On poker chips with discriminator prediction Pa(ilz)
y=I.... K
K'
K no matter
we also get (x) = 1. For background class models, $3(x)= = max Pf - K Pp (K + 1/x)
k=I.....
also yields K for a common side where the prediction over the K in-distribution classes is uniform
and for poker chips, where p ( + 1|x) 1. The fact that in this coin scenario when scored with $3, background class and classfier/discriiinaatoo combinations have the same problem as confidence
loss/OE is not surprising considering their equivalence shown in Theorem 2
15

---

Under review as a conference paper at ICLR 2022
C EXPERIMENTAL DETAILS
C.I TRAINING
For training our models, we build upon the code of Hendrycks et al. (2019)) which they have avail- able at / www.coo.//eeeeeee wwwp///nnncccssotttteceeoo. experr-eepuuueeee and borrow their general architecture and training settings. Concretely, we use 40-2 Wide Residual Network (Zagoruyko Komodakis 20T6) models with normalization based on the CIFAR datasets and a dropout rate of 0.3. They are trained for 100 epochs with initial learning rate of 0.1 that decreases follow- ing a cosine annealing schedule. Unless mentioned otherwise, each training step uses a batch of size 128 for the in-distribution and a batch of size 256 for the training out-distribution. The opti- mizer uses stochastic gradient descent with a Nesterov momentum of 0.9. Weight decay is set to 5 10-4..0000 The deep learning framework we use is PyTorch (Paszke et al. 2019), and for evaluating we use the scikit-learn (Pedregosa et al. 2011 implementation of the AUC. Our code is available at Binec_BBnDDioc D7FB
https://anonymonn wow.uuss...ppp.... 4open wwtppennsciiieeee
For evaluating the Mahalanobis detector, we use the code by the authors of Lee et al. (2018b) provided at httpp:: hitptu..tuugggoggggogggg com/ wwtp:pokaaapaaaaaaaaaaa deep Mahalanobis detector or The input noise levels and regression parameters are chosen on the available out-distribution Openlmages and are 0.0014 for CIFAR-10 10 and 0.002 for CIFAR-100.
All experiments were run on Nvidia V100 GPUs of an internal cluster of our institution, using up to 4 GB GPU memory (batch sizes in: 258) with no noticeable difference between ours and the compared OF Hendrycks et al. (2019) runs.
C.2 DATASETS
We train our models with the train splits of CIFAR-IO and CIFAR- 100 Krizhevsky & Hinton (2099
(MIT license) which each consist of 50,000 labeled images, and evaluate on their test splits of 10,000 samples. As training out-distribution we use OpenImages v4 (Krasin et al.] (017) (images have CC BY 2.0 license); the training split that we employ here consists of 8,945.29) images of different sizes, which get resized to 32 x 32 pixels, and we test on 10,000 from the official validation split. For training with 80 Million Tiny Images (80M) (Torralba et al. 2008) in Appendix H (no license, see links in Appendix H we use data from the beginning of the sequentialized dataset, and evaluate on a test of 30,080 images starting index 50,0000000. A subset of CIFAR images contained in 8OM is excluded for training and evaluation. Further image datasets used for evaluation are SVHN (Netzer) ee al. (01T)) (free for non-commercial use) with 26,032 samples, LSUN (Yu et al. 2015) Classroom (no license) with 300 samples, and CelebA (Liu et al. 2015) (available for non-commercial research purposes only) with 19,962 test samples. Uniform and Smooth Noise (Hein et al./ (019) are sampled, the latter by generating uniform noise and smoothing it using a Gaussian filter with width that is drawn uniformly random in [1, 2.5). Each datapoint is then shifted and scaled linearly such that the minimal and maximal pixel values are 0 and 1, respectively. For both noises, we evaluate 30,080 inputs.
D RELATED WORK ON OOD DETECTION
Out-of-distribttion detection has been an important research area in recent years, and several ap- proaches that are fitted towards different training and inference scenarios have been proposed.
One seemingly obvious line of thought is to use generative models for density estimation to differenti- ate between in- and out out-distribution (Bishop) 1994) Nalisnick et al., 2019 Ren et al. 2019 Nalisnick
al. 2019, Xiao et al. 2020). Recent methods to a certain extent overcome the problem mentioned in Nalisnick et al. (2019) that generative models can assign higher likelihood to distributions on
which they have not been trained. Another line of work are score-based methods using an underlying
classifier or the internal features of such a classifier, potentially combined with generative model Linng
(Hendrycks & Gimpel 2017] Liang et al. 2018 Lee et al. 2018c| Hendrycks et al. 2019a| Hein et al.
2019). One of the most effective methods up now is Outlier Exposure (Hendrycks et al. (2019a)
and work building upon it (Chen et al. 2021 Meinke & Hein 2020; Mohseni et al. [020 AAgusstin
et al. 2020 Papadopoulos al. 2021
Thulasidasan al.
2021 where classifier is trained on
16

---

Under review as a conference paper at ICLR 2022
the in-distribution task and one enforces low confidence proposed by Lee et al. (2018a) during training on a large and diverse set of out- dit-of-distibuttion images (Hendryeks et al. (019a) which can be seen proxy of all natural images. This approach generalizes well to other out-distributions. Recently, NTOM (Chen et al. 2021) has achieved excellent results for detecting far out-of-distributtion data by adding background class the classifier which is trained samples from the surrogate out-distribution that are mined such that they show a desired hardness for the model. At test time, the output probability for that class is used to decide if an input is be flagged as OOD, Their ATOM method does the same while also adding adversarial perturbations to the OOD inputs during training. Even though it has been claimed that new approaches outperform [Hendrycks et al. (2019a), up to our knowledge this has not been shown consistently across different and challenging test -it-----dittibttionn datasets (including close and far out-of-distrbbttion datasets). Below, we discuss some other recently proposed approaches that build upon different premises on the data available during training.
Hendrycks et al. (2019b) do not use any OOD data during training and instead teach the model to predict whether an input has been rotated by 0%, 90", 180 or 270". For inference, they use the loss of this predictor as OOD score, and add this score to classifier output entropy, which behaves very similar to classifier confidence. Similar to our methods, they also use shared representations and the combination of the in-distribution classifier with dedicated OOD detection score. If one interprets their rotation predictor loss as being an estimator of log p(o]z)) for some implicit out-distribution, their scoring function coincides with our 82 scores.
Golan & EI- Yaniv (0018) learn a similar transformation detector (with Dirichlet statistics collected on the in-distribution replacing ground truth labels) and use it directly to detect OOD samples without using in-distribution class information.
Winkens et al. 2020) fit for each class a normalized Mahalanobis detector on the activations of a model trained with SimCLR ands a classification head on only the un-distribution with smoothed labels. They describe their method as applying class-wise density estimation in the feature space, where the normalized Mahalanobis distance is equivalent to a Gaussian density for each class.
Roy et al. (2021) treat an interesting application of flagging unseen skin diseases, making use of class labels that are also available for their training OOD data, which contains diseases that are different from both the in-distribution diseases and the unseen diseases. This allows them to do fine-grained OOD detection by regarding the sum over all OOD classes which for their dataset shows large improvement over methods that treat the training out-distribttuion as one class. They gain additional slight improvements by combining this with a coarse grained binary loss that treats the sum over all in-distribution class probabilities as p(i (z) and the sum over all OOD classes as p(o[z). They show that this method can be combined with various representation learning approaches in order to improve their detection of unknown diseases.
Tack et al. (220)) introduce distribution shifting transformations into SimCLR training. Those transformations that are so strong that the resulting samples can be considered as OOD and as negatives w.r.t. the original in the SimCLR loss. Similarly to ooHendyycks al. (2019b) and Golan & El- Yaniv 20188). they also train a head that classifies the applied transformation. In a version extended to using in-distribution labels, they consider samples from the same class and with the same transformation as positives, and samples where either is different as negatives. With this method, they obtain OOD detection results that significantly improve over standard classifier confidence, without using any training OOD dataset.
Liu & Abbeel (020)) derive a contrastive loss from Joint Energy Based Model training (Grathwhhl et al.] (2020) and train it together with cross-entropy on the in-distribution in order to obtain classifiers whose logit values can be transformed into energies that are equivalent as OOD scoring functions to in-distribution density. They show that using these energies yields some improvements over previous density estimation approaches, and that also the classifier confidences show moderately improved OOD detection when compared to standard training.
Liu et a (2020) propose another energy based method which incorporates surrogate OOD data during training. We analyse this method in detail in Appendix [F] where we show that their Bayes optimal OOD detector is equivalent to the binary discriminator between in- and out-distribution.
17

---

Under review as a conference paper at ICLR 2022
Li & Vasconcelos (020)) use the same training method OE and show that careful resampling of the training out-distribution resembling hard negative mining can reduce its size and therefore lead to more resource effective training OOD dataset, while the resulting models reach similarly good OOD detection performance.
18

---

Under review as a conference paper at ICLR 2022
E EXPERIMENTS COMPARING DENSITY ESTIMATION AND BINARY DISCRIMINATORS
E.1 UNLABELED IN-DISTRIBUTION DATA AVAILABLE FOR TRAINING
We want to answer the following questions: How does estimating the in-distribution density compare to simply employing a binary discriminator between the in-distribution and uniform noise with respect the task of out-of-distribution detection? Can other density based models be substituted for potentially easier to handle binary discriminators against suitable (semi-Jsynthetic out-distribution? As generative models, we use a standard likelihood VAE, a likelihood PixeICNN+ and additionally compare with a Likelihood Regret VAE (Xiao et al. 2020). The binary classifier is trained to separate real data from uniform noise, thus none of the methods presented this section make use of 80M or any other surrogate distribution. The results for OOD detection in terms of AUC for all methods are presented in Table
Comparing the OOD detection performance of the binary discriminator trained against uniform noise with both VAE models, we asses that neither model is suitable for reliably detecting inputs from unknown out-distriuttion..
Following the theoretical analysis from the previous sections, the likelihood models and our binary classifier able perfectly separate the in-distribution data from uniform noise. This is expected as those methods are trained on that particular task of separating CIFAR-10 from uniform noise, whereas the LH Regret VAE with modified train objective has worse performance on uniform noise. It appears as if the training objective of the binary classifier seems to be too easy the training and validation loss converge almost zero the first few epochs of training. However, the ability to separate uniform noise from real images does not generalize to other test distributions to achieve good -iitt--diitiibttion detection performance on other test distributions. We note that while the score features from the likelihood models and the binary classifier in expectation cquivalent, both methods behave quite different on the test datasets (except for uniform noise). This is not surprising, as the probability of drawing real images from the uniform distribution is small that neither training method properly regularizes the model's behaviour on those particular image manifolds. Thus the results artifacts of random fluctuation and method clearly outperforms the other one, for example the binary classifier is better at separating CIFAR-I0 R-10 from SVHN whereas the likelihood VAE significantly outperforms the binary classifier on LSUN. Similar fluctuations also exist between the two variational auto encoders and PixeICNN++, but in conclusion none of those methods is able generalize to more specialized unseen distributions.
both methods fail
Table 3: AUC for CIFAR- 10 vs. various out out-distributions of different methods that have access to only unlabelled CIFAR- 10 data during training. Shown are the scores obtained from the likelihoods of the PixeICNN+ from Ren et al. (2019).
MEAN AUC
SVHN AUC
L.SUN AUC
CELERA
SMOOTH
C-100
OPEN AUC
80M
UNI
MODEL.
AUC
AUC
AUC
AUC
AUC
LH +IXELCNN+
$7.05
07.14
89.02
57.77
77.93
52.96
70.52
43.99
100.00
LH VAE
57.97
20.98
83.03
48.05
91.67
51.32
55.62
55.09
100.00
LH REGRET VAE
52.24
87.36
35.73
70.69
14.84
53.03
50.81
53.21
94.
BINDISC UNIFORM
45.90
71.22
34.67
35.07
47.13
47.32
44.70
41.17
100.00
E.2 LIKELIHOOD RATIOS AS BINARY DISCRIMINATOR
we discussed that for the Bayes optimal solutions of their training objectives, the ratio of the likelihoods of two density estimators for different distributions is as an OOD detection scoring function equivalent to the prediction of binary discriminator between the two distributions. In order to find out which role this equivalence plays in practise, we train a binary discriminator between CIFAR-10 as in nn-distribution and the background distribution obtained by mutating 10% of the pixles of in-distribution images as described in Ren et al, (2019). In Table we compare the OOD detection performance of this discriminator with likelihood ratios estimated with PixeICNN++ (Salimans et al.
In Section 3.1
19

---

Under review as a conference paper at ICLR 2022
2017) trained with the code of Ren et al. (2019) setting L2 regularization 10, and with the numbers taken from Xiao et al. (2020) given for their VAE models. Even though we use the code and hyperparameter settings of Ren et al. 2019), the AUC obtain for SVHN out-distribution differs significantly from their reported 93.1 %. We observe that all three methods struggle with detecting inputs from several out-distributions and thus we do not consider them reliable out-of-distribution detection methods.
Table 4: AUROC for CIFAR-I0 10 various out-distributions. The Likelihood Ratio Ren al. (2019) PixeICNN+ models were trained with their code. For the VAE, we cite the numbers of Xiao al. (220), indicated by * We also show binary discriminator between CIFAR-10 and the background distribution of Ren et al. (2019).
MODEL
MEAN
SVHN
LSUU
UNI
SMOOTH C-100
OPEN
CELEBA
80M
LHRATIO PIXELCN+++
64.77
12.57
88.85
100.00
89.55
53.60
74.50
58,26
40.87
LHRATIO VAE*
26.5*
63,2*
100.0*
44.7*
BINDISC
58.81
55.07
66.10
100.00
44.43
47.05
57.20
48.48
52.16
20

---

Under review as a conference paper at ICLR 2022
F ENERGY BASED OOD DETECTION TERMS OF BINARY DISCRIMINATORS
With Energy Based OOD Detection LLiu et al. 2020), we exhibit another surprising case of the cquivalence to Binary Discriminators to an OOD detection method which is based on ideas that are quite different from those of the other extensively discussed methods. This method is based on the premise that the logits fo(z) of a classifier can be used to define an energy (here we ignore a potential temperature factor which as Liu al. (0200 find can in good conscience be set one)
Ex(x) log
fo(la))
(19)
that the model assigns the input, which has been proposed by LeCun al. (2006); Grathwohl et al. (2020). Ideally, this energy would be equivalent to the probability density of the in-distribution via
log p(zlin) Exr where Z
Fe(x) dz)
(20)
However, since integral over the whole image domain intractible, is not possible to effectively decrease - Eg on the in- m--iitrrbbttinn directly while also controlling Z. Naively training the classifier to have high energy on random inputs which would mean uniform noise is of course not a solution, since the model easily distinguishes the noise, and is very unlikely encounter any at least vaguely real images within finite time. Thus, Liu et al. (2020) rather use a surrogate training out-distribution of natural images which they increase the energy on during training; for their experiments, they take 80 Million Tiny Images, for which we compare their models with several other methods in Appendix H Simultancously, they minimize the standard classifier cross-entropy the in-distribution. In order to avoid infinitely small potential losses, their training objective uses two margin hyper-paramedd parameters min Mout and reads
I (x..))=p(axv11))
log (J)lar)
(21)
+ A
E max {0, Eo(x) min] 2 w-ppweoo
E
max {0, Mour Eo())
(0)~p(z(1)
with
(fo())
(y)p) log
fo(ulz) + Ex(x).
(22)
Σ efolliz)
Note that their À does not balance between in and out by depending on prior on p(i), but it balances the energy regularization compared to the classifier cross-entropy loss.
Theorem The Bayes optimal logit output (x) of the energy based OOD detection model minimizing the expected loss an imput x yields class probabilities (klar) = p(k|2,, pr i) that are optimal for standard classifier with cross-entropy loss and simultaneously fulfills
-E(()) p(ilz) (Mout
mm))
Mou
(23)
Proof. Our goal is to find for given input the model output fa(x: that minimizes the expected loss, assuming that we know the probabilities plylx, i) and p(i]z)). This expected loss
-p(())
p(kl i) (o
(24)
+ X (p(l)z) max {0, Fo(x) mmn 12 + p(i]z)) max (0, mout
Eo(x))
(25)
First, we note that if centain E7 (x) minimizes the expected energy loss to
x (p((lz) max (0, (x) min) 2 (1 p(((x)) max (0, Mout (x)))
(26)
21

---

Under review as a conference paper at ICLR 2022
and if some fg minimizes the expected CE loss (note that its minimization is independent of the positive factor p( x)) to
K log
(
Σ
pkk,, 1)
(27)
with corresponding (x) log Di=l ef
K then the logit output with EE (2) E (z) added
to each component has the energy
K
-log log ( (--E(())+E
K
Ba(x) - log Σ
Σ
(28)
i=1
= Ez) 18 (z)
log
(((0) (x)
(29)
which means that this (x) minimizes the expected energy loss, and also fulfills
cl(())())
(x)
(30)
E -
ie. has the same probability predictions as ft and thus also minimizes the expected CE loss. This means that both can be optimized independently.
As we' ve seen before, the optimal
(k)z)) for the expected CE loss is po(k(z) = p(k(2,, i).
Thus independently minimize the expected energy loss
p(ilz) max (0, Ex(x) mi )2 + (1 p(ilx=)) max f0, Mout Eo(x)}²
(31)
The first derivative with respect to Ex(x) is
2 (1 (1--pite p(il=)) (Eo(1) mont)
for (x) < Min
2 (p(1)z) (Fag(r)) mia) p(il==)) (Fa(x) mmot))
for min Fgg(:) < Mout
2 p(llz)) (Eo(x) - min)
for Moat < Eg (z)
(32)
which simplified is
0
for (x) Min
2 (Eo(x) + p((=)) (Mou
mmn)
Moa)
for min < Eg (x) Mout
(33)
00
for Matt Eg(x).
Here, for simplicity we again make the reasonable assumption that in- and dintrriitiuutinn have full support, i.e. 0 < p(ilæ) < 1. (If we do not want to make the full support assumptions, for p(ilzz)= 0. any energy 2 Mout would be optimal, and for p(ilar) = 1, any energy S minn would be optimal.)
A the margin Fo(x)) Min, the derivatives for Eg (x: < and min Fa(x)) < muat coincide as 2. (1 p((l=))) (m moun) < O, which means that the minimizer min- Similarly, we see that the minimizer is Mout-
As the second derivative positive, by solving where the derivative is zero, we find that the optimal negative energy (which is the score which they use for OOD detection) is
(x) = p((]z) (Mout mmn) - Mout
(34)
Li
This is strictly monotonously increasing function in p(ilz). By our Theorem 1, we conclude that the negative energy score they obtain from their method is (for the Bayes optimal model that minimizes the loss on the training distributions) equivalent to p(ilx)).
Corollary At their Bayes optima, the Energy Based OOD Detection criterion is equivalent to that of binary discriminator between the training in- and out-distributionss
22

---

Under review as a conference paper at ICLR 2022
Note that while the classifier and energy loss terms can be optimized independently, the model is trained for both tasks simultaneously, which means that similar synergies to those we observe in [ might contribute to the good performance of this method. Our empirical evaluation of energy based models in Appendix confirms that their similar behaviour to the binary discriminator between in- and saining-out-distributiio indeed holds in practice.
23

---

Under review as a conference paper at ICLR 2022
G EVALUATIONS WITH THE AUC DETECTION METRIC
Complementing the evaluations of the FPR 95TPR metric presented in the main paper, Tables and66show the AUC (AUROC) values of the same models in Tables and respectively. The observations on the strngths of the different methods and scoring functions discussed in Section also hold for the AUC evaluations, which is not surprising since the AUC and FPR@95TPR metries are closely related.
Table 5: Accuracy on the in-distribution (CIFAR-IOCCIAAR 100) and AUC for various test out- distributions of the different OOD methods with Openlmages as training out-distribution for which the FPRs are shown in Table
IN-DISTRIBUTION: CIFAR-10
MEAN
SVHN
LSUN
UNI
AMооTHC-100
80M
CELA AUC
OPENIM AUC
MODEL
Acc.
AUC
AUC
AUC
AUC
AUC
AUC
AUC
PLAIN CLASSI
95.16
91.85
93.52
92.94
97.04
92.84
89.61
91.30
85.70
84.81
MAHALANOBIS
91.63
96.34
92.39
100.00
99.82
86.78
89.41
76.67
84.81
OE
95.06
97.28
98.49
99.99
99.99
99.99
90.03
92.53
99.91
99.43
BGC
95.02
99.48
100.00
99.99
99.95
79.64
86.37
99.74
99.97
BGC
95.21
97.22
98.90
100.00
99.99
99.67
90.47
92.41
99.11
99.73 73
BGC 83
95.21
97.21
98.87
100.00
99.98
99.62
90.47
92.41
99.08
99.71
SHARED BINDISC
92.51
98.77
100.00
99.89
99.93
68.34
80.81
99.80
99.95
SHARED CLASSI
95.28
95.49
96.10
98.60
99.06
96.09
90.09
92.35
96.18
93.57
SHARED COMBI $2
95.28
97.26
98 66
100.00
99.93
99.94
89.71
92.84
99.72
99.88
SHARED COMBI
95.28
97.26
98.62
100.00 99.93 99.94
89.75
92.85
99.71
99.88
IN-DISTRIBUTION: CIFAR-100
MEAN
SVHN
LSUN
UNI
SMOOTH
C-10
80M
OPENIM
MODEL
ACc.
AUC
AUC
AUC
AUC
AUC
AUC
AUC
AUC
PLAIN CLASSI
77.16
82.13
82.33
79.13
96.03
81.36
76.14
77.80
75.80
MAHALANOBIS
85,88
88.69
87.56
90.08
99.91
70.78
78.26
77.51
OF
77.19
90.37
89.54
99.98
99.03
99.68
75.95
78.03
99.67
BGC
88.41
97.38
99.99
99.70
99.79
60.51
73.11
99.93
BGC
77.61
90.47
90.50
99.99
99.87
99.75
74.88
77.82
99.64
BGC $3
77.61
90.46
90.46
99.99
99.88
99.74
74.88
77.82
99.64
SHARED BINDISC
84.62
97.44
99.99
99.70
99.68
47.82
63.13
99.93
SHARED CLASSI
77.35
82.06
82.72
99.05
72.73
84.14
75.76
77.99
93.54
SHARED COMBI
77.35
90.74
91.74
99.99
99.59
99.54
75.50
78.10
99.57
SHARED COMBI
77.35
90.73
91.69
99.99
99.57
99.53
75.50
78.10
99.57
24

---

Under review as a conference paper at ICLR 2022
Table 6: AUC evaluation of the models trained with shared and separate representations from Table
IN-DISTRIBUTION: CIFAR-10
MEAN AUC
SVHN
LSUN
UNI
SMOOTHC-
100
80M
CELA AUC
OPENIM
MODEL
ACC.
AUC
AUC
AUC
AUC
AUC
AUC
AUC
PLAIN CLASSI
95.16
91.85
93.52
92.94
97.04
92.84
89.61
91.30
85.70
84.81
SEPARATE BINDISC
89.03
96.42
100.00
99.97
99.99
58.60
72.36
95.87
99.99
SEPARATE COMBI
96.16
96.40
98.16
100.00
99.98
99.99
89.64
91.35
95.70
99.94
SHARED BINDISC
92.51
98.77
100.00
99.89
99.93
68.34
80.81
99.80
99.95
SHARED CLASSI
95.28
95.49
96.10
98.60
99.06
96.09
90.09
92.35
96.18
93.57
SHARED COMBI
95.28
97.26
98.62
100.00
99.93
99.94
89.75
92.85
99.71
99.88
PLAIN SHA DISC
95.16
97.26
98.67
100.00 99.91 99.93
89.78
92.95
99.58
99.87
IN-DISTRIBUTION: CIFAR- 100
MODEL
Acc.
MEAN |SVHN LSUN
UNI
SMOOTH C-10
OPENIM
80M
PLAIN CLASSI
77.16
82.13
82.33
79.13
96.03
81.36
76. 76.14 14
77.80
75.80
SEPARATE BINDISC
84.30
94.68
100.00
99.81
99.64
50.06
61.62
99.98
SEPARATE COMBI $3
77.16
88.95
84.49
99.99
99.88
96.06
75.83
77.45
99.82
SHARED BINDISC
84.62
97.44
99.99
99.70
99.68
47.82
63.13
99.93
SHARED CLASSI
77.35
82.06
82.72
99.05
72.73
84.14
75.76
77.99
93.54
SHARED COMBI $3
77.35
90.73
91.69
99.99
99.57
99.53
75.50
78.10
99.57
PLAIN SHA DISC
77.16
90.79
91.99
99.99
99.80
98.85
75.87
78.23
99.50
25

---

Under review as a conference paper at ICLR 2022
H DISCUSSION OF THE CHOICE OF OPENIMAGES AND EXPERIMENTAL RESULTS WITH 80 MILLION TINY IMAGES AS TRAINING DUT-DISTRIBUTION
Since the 80 Million Tiny Images (Torralba et al. 2008) dataset was retracted by the authors their statement can be read at wtttt:/// hulttp::/groupp: /ttpp/grupp... csail mit . wew.vvisinn/aiiii Tean//TayImaggeeee as a reaction to Birhane & Prabhu (2021) exposing the presence of offensive and prejudicial images in the dataset, a good "surrogate surrogate" training out-distiibution for the LIFAR-IO/CIFAR in-distribution has to our knowledge not yet been established.
Our experience confirms the assessment which the authors of Hendrycks et al. (2019a) make in their discussion section 5: the surrogate training out-distribution should consist mainly of natural images, should have a high semantic diversity, and the number of samples in the dataset should be large. We also observe that it is vital that there are easy to detect details that separate the training in- and out-distributions from each other, as for example using a different resizing interpolation method would lead to 'overfitting' on such features with 100% train accuracy of the Binary Discriminator and near zero loss (apart from that of the in-distribution classifier) for OE and BGC, with generalization to the test OOD datasets. OpenImages fulfills the mentioned criteria, and in our judgement does not contain ethically problematic images. It is, however, to be noted that CIFAR was sourced as subset of 80M Tiny Images, see Krizhevsky & Hinton| (2009) and https:: / www. cS toront O /euurkriii cifar html which explains the somewhat better results with 80M as training OOD dataset that observe below in this section. Our results on the theoretical relations between the different methods and scoring functions do not depend on the choice of the training OOD dataset, and give reason to expect that our experimental confirmations of these relations will also hold for even better suited surrogate OOD datasets that we hope will be found in future works.
In the main paper, we employ OpenImages (Krasin et al. 2017) as a replacement, and for completeness and comparison to the originals of OE, Energy Based OOD detection and NTOM/ATOM, below we show and discuss the results obtained with the retracted 80M dataset which is commonly used in OOD detection literature.
The training procedure of our methods again follows that of OE, and for the 80M experiments we do not add AutoAugment, in order to stay as close as possible to the original. For the established methods we compare to, use the original weights of their published models; the Plain and Outlier Exposure (OF) models were retrieved from the repository of the authors of OE (ht tstt : / / i thub and for NTOM and ATOM (Chen et al. 2021)
com/hendryckks/ooteeee i -er-ieeeeeee-sueeeeeeeeeee
their code podel/tttss///iththee com/ foooo//fc43//nnt/e htt//niffrratttm ive outlier- mining to evaluate their DenseNet models (their best models). We finetune the Mahalanobis detector on 80M with the same procedure as described above; the optimal input noise level for 80M is 0.0005 for both CIFAR in-distributions. We evaluate the Energy Based models finettuned on 80M which the authors of Liu et al. (2020) provide at htttt:: http//gttttttt/g////g com/ wet www.lluueeeeggge ood with their evaluation code.
For CIFAR- 10, as already seen with OpenImages as training out-distribution, the OOD detection performance of SHARED CLASSI much better than that of the plain classifier. .In fact SHARED BINDISC has already very good OOD performance with mean FPR of 7.56 and mean AUC of 97.90 which is only improved by considering scoring function 82/ 33 in the combination of SIARRE BINDISC and SHARED CLASSI which yields the best performance in classification accuracy and mean AUC.
The classifier with 80M as background class (BGC) works very well for all scoring functions and reaches SOTA performance similar to/better than OE Both BGC/SHARED COMBI with 82/83 perform particularly well on the challenging close out-distributions CIFAR-100 and OpenImages (which are the data sets where NTOM/ATOM perform significantly worse). However, as already observed with OpenImages as training out-distribution the differences of the methods are minor both in terms of classification accuracy.
Archived statement: hwtphs:/wwbb.....eee arch www.arcccc.v.rr.omoo ve ove.orrg/wbb/00/ 0 11 http2/5/httt://// csa1 www/vidiivioo Tiny Image/
26

---

Under review as a conference paper at ICLR 2022
The results for CIFAR-100 are again qualitatively similar those for CIFAR-1O. NTOM/ATOM now show worse mean AUC results which are mainly due to worse results for the close out-distributions CIFAR-10 10 and Openlmages, but better mean FPR PRR99,,PR, which is explained by their excellent detection of LSUN Classroom, where the other methods work quite well in terms of AUC but still make quite many
the TP%T threshold. OE again achieves comparable OOI results the other evaluated methods. As the theoretical considerations we presented in Appendix F suggest, the energy based OOD detector achieves good performance that is comparable that the other methods. Our SHARED COMBI $2 performs best in terms of OOD performance and test accuracy but again differences are minor.
The conclusions are similar to those drawn for Openlmages in the main paper. Comparing the results, is clear that 80M still works somewhat better than OpenImages, so for the CIFAR in-distributions, the search for ethically acceptable replacement of 80M that allows for equal or improved results continues. The consistent similarities between the examined methods over the different datasets suggest that the methods and scoring functions would be similarly viable with such an alternative training OOD dataset.
27

---

Under review as a conference paper at ICLR 2022
Table 7: Accuracy the in-distribution (CIFAR-IO/CIFAR and FPR @95% % TPR for various test out-distributions different OOD methods with 80 Million Tiny Images as training out- distribution (shown results for test set of 80M not used for computing the mean FPR). Lower false positive rate is better. CelebA makes no sense as test out-distribution for CIFAR-100 as it contains man/woman as classes. PLAIN, OE, BGC and SHARED have been trained using the same architecture and training parameters/schedule binary discriminator (BINDISC) resp. the combination with the shared classifier (SHARED COMBI) performs similar/better than Outlier Exposure (Hendrycks et al. 2019a).
$2, are the scoring functions introduced in Section 3.2 Our
IN-DISTRIBUTION: CIFAR-10
MEAN
SVHN
LSUN
UNI
SMOOTH
1-100
OPENIM
CELA
80M
MODEL
Acc.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
FPR
FPR
PLAIN CLASSI
94.84
64.62
48.33
52.67
75.69
62.58
62.91
66.38
83.79
60.53
MAHALANOBIS
41.15
40.19
50.00
0.00
0.17
58.66
58.36
80.69
53.42
ENERGY
95.22
9.01
1.58
2.00
0.00
0.00
30.03
28.26
1.19
8.21
NTOM
95.42
8.21
1.06
0.33
0.00
0.00
29.61
26.15
0.30
4.90
ATOM
95.20
7.76
0.69
0.33
0.00
0.00
27.80
25.26
0.25
4.44
OE
95.74
8.27
1.96
2.00
0.00
0.06
26.12
27.07
0.71
5.96
BGC 81
7.47
1.33
0.00
0.00
24.75
25.19
0.19
4.43
BGC
95.63
7.42
0.98
1.33
0.00
0.00
24.13
25.33
0.20
4.95
BGC
95.63
7.49
1.05
1.33
0.00
0.00
24.26
25.57
0.21
4.82
SHARED BINDISC
7.56
0.67
1.67
0.00
0.00
24.70
25.58
0.31
4.57
SHARED CLASSI
96.08
15.71
6.29
13.00
0.07
0.13
37.07
40.48
12.95
19.47
SHARED COMBI 82
96.08
7.47
0.71
1.33
0.00
0.00
24.15
25.72
0.35
4.79
SHARED COMBI 83
96.08
7.44
0.73
1.33
0.00
0.00
23.95
25.70
0.35
4.84
IN-DISTRIBUTION CIFAR-100
MEAN
SVHN
LSUN
UNI
SMOOTH
C-10
OPENIM FPR
80M
MODEL.
Acc.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
PLAIN CLASSI
75.96
82.26
84.33
80.00
98.99
65.81
81.97
82.47
80.
MAHALANOBIS
47.89
64.58
63.67
0.00
2.77
81.39
74.93
69.79
ENERGY
75.70
32.95
20.61
16.67
4.23
2.90
84.27
69.00
42.18
NTOM
74.88
32.63
24.67
10.00
0.00
0.00
90.58
70.52
40.78
ATOM
75.06
34.60
37.78
8.67
0.00
0.30
89.80
71.02
40.29
OE
76.73
34.89
34.41
24.00
1.10
4.96
79.77
65.09
45.59
BGC
34.79
35.73
23.00
0.00
0.07
81.61
68.31
45.76
BGC
75.82
35.86
40.36
26.67
0.00
0.45
79.50
68.21
47.72
BGC $3
75,82
35.91
40.53
26.67
0.00
0.42
79.54
68.28
47.46
SHARED BINDISC
32.74
25.16
22.00
0.00
0.00
82.47
66.83
44.50
SHARED CLASSI
76.52
44.86
56.96
57.00
0.06
0.22
79.55
75.37
65.50
SHARED COMBI
76.52
32.71
25.70
23.67
0.00
0.00
80.95
65.93
44.87
SHARED COMBI
76.52
32.72
25.79
23.67
0.00
0.00
80.90
65.98
44.96
28

---

Under review as a conference paper at ICLR 2022
Table 8: Accuracy on the in-distribution (CIFAR-IOCIIAR-IIO) and AUC (AUROC) for various test out-distributions of different OOD methods with 80 Million Tiny Images training out- distribution (shown results for test set of 80M are not used for computing the mean AUC). The relative performance of the different methods measured AUC is similar to what we observed with the FPR 1095%TPP measure in Table 7
IN-DISTRIBUTION: CIFAR-10
MEAN AUC
SVHN
LSUN
UNI
SMOOTHC-100
OPENIM
CELA
80M
MODEL
ACC.
AUC
AUC
AUC
AUC AUC
AUC
AUC
AUC
PLAIN CLASSI
94.84
85.75
91.91
91.63
87.69
78.27
87.83
83.23
79.43
88.01
MAHALANOBIS
91.12
94.34
91.98
100.00
99.51
88.08
84.92
79.17
89.65
ENERGY
95.22
97.32
99.26
99.49
99.00
99.40
93.81
90.73
99.57
97.71
NTOM
95.42
97.32
99.59
99.79
99.97
99.84
92.19
89.96
99.89
98.72
ATOM
95.20
97.42
99.63
99.76
99.93
99.60
92.89
90.30
99.85
98.55
OF
95.74
97.64
99.48
99.48
99.46
99.64
94.80
90.91
99.71 1
98.50
BGC
97.94
99.64
99.58
99.96
99.98
94.84
91.65
99.92
98.78
BGC
95.63
97.95
99.60
99.52
99.97
99.98
95.03
91.65
99.92
98.65
BGC $3
95.63
97.95
99.58
99.52
99.97
99.98
95.04
91.64
99.92
98.70
SHARED BINDISC
97.90
99.74
99.60
99.94
99.96
94.75
91.42
99.87
98.77
SHARED CLASSI
96.08
96.57
98.77
97.78
99.87
99.68
93.40
88.70
97.80
96.42
SHARED COMBI
96.08
97.96
99.73
99.58
99.95
99.96
95.13
91.48
99.85
98.73
SHARED COMBI
96.08
97.96
99.73
99.58
99.95
99.96
95.14
91.48
99.85
98.73
IN-DISTRIBUTION:
CIFAR-100
MEAN
SVHN
LSUN
UNI
SMOOTH
C-10
OPENIM
80M
MODEL
Acc.
AUC
AUC
AUC
AUC
AUC
AUC
AUC
AUC
PLAIN CLASSI
75.96
77.48
71,38
76.89
78.14
88.36
75.33
74.60
75.92
MAHALANOBIS
87.75
85.96
87.20
100.00
99.16
75.45
78.74
79.76
ENERGY
75.70
91.67
96.54
96.69
97.91
98.92
77.39
82.57
91.16
NTOM
74.88
88.49
96.20
97.31
99.79
99.94
62.44
75.24
88.41
ATOM
75.06
88.02
93.68
97.51
99.98
98.46
63.47
75.02
88.44
OE
76.73
91.72
94.06
95.58
99.06
98.84
79.53
83.31
88.43
BGC S1
92.04
94.42
95.47
99.99
99.73
79.15
83,46
89.19
BGC 82
75.82
91.54
93.32
94.64
99.95
99.63
79.29
82,41
88.11
BGC 83
75.82
91.53
93.30
94.62
99.94
99.62
79.29
82.40
88.23
SHARED BINDISC
91.84
95.90
95.69
99.79
99.94
76.56
83.19
89,25
SHARED CLASSI
76.52
88.16
86.28
87.61
99.97
99.90
77.00
78.23
81.72
SHARED COMRI
76.52
92.03
95.50
95.43
99.96
99.98
78.46
82.86
88.69
SHARED COMBI
76.52
92.03
95.49
95.42
99.97
99.98
78.46
82.85
88.67
29

---

Under review as a conference paper at ICLR 2022
I EXPERIMENTS WITH SVHN AS TRAINING OUT-DISTRIBUTIR
In order to examine the effect of a training out-distribution that relatively far away from the in- distribution, we show experiments with SVHN out-distribution in Tables and 10 The OOD detection performance of these methods is much worse than with the closer Openlmages and 80M training ontdiistribution. In most cases, combinations $2 and sy/OE which implicitly or explicitly use the classifier confidence lead to better OOD detection than using the binary discriminator/ss. The inconsistent behaviour over the different test out distibbbribus of the methods that use SVHN as training out-distribution can be explained by the easiness of the discrimination task, which manifests itself the fact that BGCs, and SHARED BINDISC reach perfect FPR and AUC metrics. This indicates form of overfitting this specific out-distribution, without consistent generalization to unseen distributions which do not have characteristic features that are similar to those appearing in SVHN images.
We also do not observe a beneficial effect on test accuracy for the methods that use SVHN compared to plain, which is be expected as the representations learned from SVHN are hardly useful for the in-distribution task.
Table 9: FPR @95% TPR for CIFAR-IO/CIFAR-100 as in-distribution with SVHN as training out-distribution.
IN-DISTRIBUTION: CIFAR-10
MEAN
LSUN
UNI
SMOOTHC-108
80M FPR
OPENIM
CELA
SVHN
MODEL
ACC.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
FPR
PLAIN CLASSI
94.84
64.62
48.33
52.67
75.69
62.58
62.91
66.38
83.79
60.53
OE
94.80
63.73
54.33
92.01
41.27
60.55
55.91
65.52
76.51
0.03
BGC 81
54.31
53.00
100.00
0.51
55.16
39.28
64.36
67.87
0.00
BGC
94.51
62.98
58.33
98.53
18.73
62.27
57.47
67.35
76.16
0.02
BGC $3
94.51
63.08
58.33
98.52
19.18
62.34
$7.63
67.39
78.16
0.02
SHARED BINDISC
72.95
95.67
100.00
5.81
77.04
66.75
85.17
80.22
0.00
SHARED CLASSI
94.71
48.39
54.33
0.00
17.99
62.34
57.37
67.05
79.64
0.57
SHARED COMBI 82
94.71
46.27
55.00
0.00
6.74
60.84
55.15
66.50
79.65
0.02
SHARED COMBI 83
94.71
46.31
55.00
0.00
6.84
60.90
55.26
66.53
79.65
0.02
IN-DISTRIBUTION. CIFAR-1O1
MEAN FPR
LSUN FPR
UNI
SMOOTH
C-10
80M
OPENTM FPR
SVHN
MODEL.
ACc.
FPR
FPR
FPR
FPR
FPR
PLAIN CLASSI
75.96
82.26
84.33
80.00
98.99
65.81
81.97
82.47
80.17
OE
75.78
73.89
81.33
99.47
15.93
83.32
80.30
83.01
0.04
BGC
76.16
96.33
100.00
3.35
91.05
80.55
91.68
0.00
BGC
75.21
74.53
80.00
98.49
21.49
83.14
80.52
83.54
0.07
BGC 83
75.21
74.53
80.00
98.49
21.50
83.14
80.53
83.54
0.07
SHARED BINDISC
82.08
99.33
100.00
10.04
95.00
91.21
96.89
0.00
SHARED CLASSI
75.97
83.46
78.67
100.00
76.67
82.36
79.99
83.08
0.44
SHARED COMBI
75.97
78.66
78.67
100.00
47.73
82.44
79.93
83,20
0.07
SHARED COMBI
75.97
78.66
78.67
100.00
47.78
82.43
79.92
83.17
0.07
30

---

Under review as a conference paper at ICLR 2022
Table 10: AUC for CIFAR-IOCCIFAR-100 as in-distribution with SVHN as training out-distribution.
IN-DISTRIBUTION: CIFAR-10
MEAN
LSUN
UNI
SMOOTHC-100
80M
OPENTM
CELA
SVHN
MODEL
Acc.
AUC
AUC
AUC
AUC AUC
AUC
AUC
AUC
AUC
PLAIN CLASSI
94.84
85.75
91.91
91.63
87.69
78.27
87.83
83,23
79.43
88.01
OE
94.80
88.37
91.72
91.33
91.20
88.26
89.55
82.97
83.59
100.00
BGC
83.58
89.43
58.34
99.80
85.44
90.28
78.01
83.76
100.00
BGC 82
94.51
89.62
91.50
91.14
97.60
88.72
89.98
82.96
85,40
100.00
BGC $3
94.51
89.60
91.50
91.14
97.54
88.71
89.96
82.96
85.40
100.00
SHARED BINDISC
58.47
37.27
38.61
98.85
55.19
65.66
48.61
65.12
100.00
SHARED CLASSI
94.71
90.14
90.20
99.70
97.44
88.50
89.68
82.97
82.51
99.80
SHARED COMBI
94.71
90.18
89.74
99.67
99.00
88.49
89.87
82.43
82.09
100.00
SHARED COMBI
94.71
90.19
89.76
99.68
98.97
88.50
89.87
82.46
82.12
100.00
IN-DISTRIBUTION: CIFAR-100
MEAN
LSUN
UNI
SMOOTH
C-10
80M
OPENIM AUC
SVHN
MODEL
Acc.
AUC
AUC
AUC
AUC
AUC
AUC
AUC
PLAIN CLASSI
75.96
77.48
71.38
76.89
78.14
88.36
75.33
74.60
75.92
OE
75.78
78.86
75.93
73.53
97.02
75.37
76.57
74.74
19.99
BGC 81
66.60
67.45
24.51
99.37
68.50
73.82
65.97
100.00
BGC
75.21
77.38
76.04
67.25
96.28
74.83
75.86
74.01
99,98
BGC $3
75.21
77.38
76.04
67.25
96.27
74.83
75.86
74.01
99.98
SHARED BINDISC SHARED CLASSI
55.60
34.81
53.23
98.03
49.00
54.57
43.93
100.00
75.97
72.04
80.31
41.57
83.32
75.64
76.58
74.81
99.90
SHARED COMBI $2
75.97
72.95
80.23
41.34
89.21
75.59
76.57
74.75
99.99
SHARED COMBI $3
75.97
72.94
80.23
41.34
89.18
75.59
76.57
74.75
99.99
31

---

Under review as a conference paper at ICLR 2022
J EXPERIMENTS WITH RESTRICTED IMAGENET AS IN-DISTRIBUTION
In addition to the results for CIFAR-10 and CIFAR-100 shown in the main paper. here we provide results for Restricted ImageNet. Restricted ImageNet, introduced by Tsipras et 2019). consists of 9 classes, where each individual class is a union of multiple ImageNet (Deng et al. 2009) classes, for example the Restricted ImageNet class 'dog' contains all dog breeds from ImageNet As Restricted ImageNet only contains animal classes, the union over all its classes does not cover the entire ILSVRC2012 dataset (Ruusakovsky et al., 2013). which allows us to use the remaining IISVRR2012 classes as training out-distribution. Like we did for the CIFAR experiments, we train plain classifier, an Outlier Exposure model, background class model and a shared diserininatoorclaassffeer and evaluate them with the different scoring functions. The model is ResNet50 and cropping and flipping as data augmentation during training. The results in terms of FPR TPR and AUC can be found in Table 11
random
Once again, we see that SHARED CLASSI has relatively good OOD detection performance and clearly beats the plain classifier from standard training. Again, (2019a), training with background class and shared training of classifier and binary discriminator perform similarly. In terms of accuracy, the shared model benefits most from the added unlabelled data compared to plain training. At 95% TPR, the sharedly trained binary discriminator and its combinations SHARED COMBI $2 and SHARED COMBI $2 detect flower images significantly better than the other approaches which results in the best mean FPR FPR@95% compared to the other methods, while in terms of AUC, OE has slight advantage.
that Outlier Exposure [Hendrycks et al.
32

---

Under review as a conference paper at ICLR 2022
Table 11: Out-of-distribution detection evaluation for various ResNet50 models trained on Restricted ImageNet in terms of AUC and FPR @95% TPR. The last column (NoTRIN) refers to the remaining classes from the ILSVRC2012 validation split that are not part of Restricted ImageNet and that were used the training out-distribution.i does not contribute to the mean test FPR/AUC. As all models use the train split of NotRIN as training-ovd out distribution.
IN-DISTRIBUTION: RESTRICTED IMAGENET
F
MEAN FPR
FLOWERS FPR
FGVC
CARS FPR
SMOOTH
UNIFORM
NOTRIN
MODEL
ACC.
FPR
PLAIN CLASSI
96.34
36.71
60.11
50.23
73.20
0.00
0.00
50.37
OE
97.10
4.26
21.06
0.18
0.04
0.00
0.00
6.91
BGC
4.22
20.74
0.33
0.01
0.00
0.00
6.46
BGC 82
97.50
4.77
23.43
0.39
0.02
0.00
0.00
6.02
BGC 83
97.50
4.73
23.26
0.36
0.01
0.00
0.00
6.13
SHARED BINDISC
2.73
10.10
0.24
0.04
3.26
0.00
5.93
SHARED CLASSI
97.59
17.51
50.54
17.79
19.21
0.00
0.00
21.99
SHARED COMBI
97.59
2.55
12.28
0.45
0.04
0.00
0.00
5.58
SHARED COMBI
97.59
2.62
12.60
0.45
0.04
0.00
0.00
5.63
AUC
MEAN AUC
FLOWERS AUC
FGVC
CARS
SMOOTH AUC
UNIFORM AUC
NOTRIN
MODEL
AcC.
AUC
AUC
AUC
PLAIN CLASSI
96.34
94.96
91.65
92.67
92.46
98.74
99.26
92.38
OE
97.10
98.76
96.65
99.75
99.85
97.95
99.58
98.46
BGC 81
98.61
96.64
99.86
99.97
97.77
98.80
98.67
BGC
97.50
98.66
96.39
99.83
99.96
98.18
98.94
98.69
BGC 8y
97.50
98.66
96.43
99.83
99.96
98.14
98.93
98.68
SHARED BINDISC
98.26
97.62
99.83
99.94
96.13
97.78
98.71
SHARED CLASSI
97.59
96.93
93.40
96.58
96.53
99.48
98.66
96.10
SHARED COMBI 82
97.59
98.54
97.41
99.80
99.93
97-37
98.18
98.72
SHARED COMBI 83
97.59
98.58
97.36
99.79
99.92
97.61
98.22
98.71
33

---

Under review as a conference paper at ICLR 2022
K
THE EFFECT OF VARYING X
We investigate the effect of choosing the training hyperparameter X, which is the factor of the respective loss on the out-of-distribution samples and represents p(o) p[i] during training. We evaluate models trained with Outlier Exposure Hendrycks al. (2019a), background class and shared training of binary discriminator and classifier, all scored with $3 (the implicit scoring function of OE). Note that in Section the OE, BGC and SHARED models trained with À 1.0.
In Tables 2 and 3 we see that for CIFAR-I0 the differences between different choices of are minor with clear favorite, but setting À = 2.0 tends be too high. For CIFAR-100, the differences are much larger. Here, choosing À too small can have a severe negative effect on the detection of the otherwise relatively detect far out-distributions SVHN, Uniform Noise and Smooth Noise. Regarding the numbers for both in-distribution datasets, using À = 1 is considerate default choice.
Table 12: Effect of varying À during training for OE (Hendrycks et al. 2019a). the 83 scoring function for models with background class and SHARED COMBI 83. Shown are test accuracy and FPR@95% TPR with OpenImages training out-distribution.
IN-DISTRIBUTION: CIFAR-10
MEAN
SVHN
LSUN
UNI
SMOOTH
C-100
80M FPR
CELA FPR
OPENIM
MODEL
ACC.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
OE X=0.1
95.36
14.34
13.31
0.00
0.25
0.03
48.37
36.65
1.77
12.60
OE A=0.25
95.31
16.17
19.64
0.00
0.00
0.05
51.77
40.21
1.55
7.74
OE A=0.5
95.27
15.98
16.23
0.00
0.07
0.01
53.06
41.72
0.79
4.85
OE x=1.0
95.06
15.20
9.58
0.00
0.00
0.00
54.05
42.33
0.45
3.46
OE A=2.0
95.19
15.98
12.93
0.00
0.00
0.00
55.99
42.68
0.26
1.67
BGC #3 A=0.1
95.38
16.21
18.93
0.00
0.12
1.05
52.43
39.42
1.53
9.56
BGC 83 A=0.25
95.27
15.83
12.76
0.00
0.04
0.03
54.06
43.26
0.66
5.43
BGC 8 A=0.5
95.23
15.14
9.33
0.00
0.09
0.04
52.97
42.79
0.74
3.19
BGC 3aA=1.0
95.21
16.63
7.69
0.00
0.07
2.36
55.19
44.67
6.41
1.74
BGC ss 2=2.0
95.28
17.45
8.72
0.00
0.00
0.00
59.02
48.09
6.31
0.71
SC 83 A=0.1
95.20
16.49
15.22
0.00
0.27
0.00
54.30
43.97
1.69
10.46
SC 83 A=0.25
95.21
16.17
15.32
0.00
0.15
0.04
54.60
42.59
0.53
5.19
SC A=0.5
95.25
16.02
9.21
0.00
0.09
0.00
55.72
45.55
1.56
2.75
SC A=1.0
95.28
16.06
9.00
0.00
0.00
0.00
58.68
42.85
1.91
0.66
SC A=2.0
95.26
17.04
9.33
0.00
0.00
0.00
58.20
45.81
5.92
0.68
IN-DISTRIBUTION: CIFAR- 100
MEAN
SVHN
LSUN
UNI
SMOOTH
C-10
80M
OPENIM
MODEL.
Acc.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
FPR
OE A=0.1
77.28
52.35
70.29
1.00
28.58
56.15
82.44
75.64
17.94
OE A=0.25
76.96
46.40
60.70
0.00
9.24
48.89
83.01
76.59
6.01
OE A=0.5
77.22
36.43
53.12
0.00
0.15
4.62
82.86
77.84
4.23
OE A=1.0
77.19
35.03
47.36
0.00
0.67
0.08
84.64
77.42
1.28
OE X=2.0
76.95
33.25
37.04
0.00
0.00
1.70
83.28
77.48
0.86
BGC #s A=0.1
76.87
41.23
67.39
1.00
5.97
13.81
82.44
76.76
15.70
BGC s; >=0.25
77.05
42.05
49.94
0.00
17.07
25.41
82.34
77.55
7.47
BGC 9s A=0.5
77.17
36.14
43,90
0.00
10.54
3.70
82.86
75.86
3.76
BGC saA=1.0
77.61
33.36
37.27
0.00
0.00
0.20
84.51
78.19
1.27
BGC 83 A=2.0
77.26
32.60
35.34
0.00
0.00
0.00
82.69
77.57
1.20
SC A=0.1
77.35
58.54
62.07
0.33
92.62
37.44
82.05
76.74
14.17
8 >=0.25
76.96
41.34
63.29
0,00
10.24
14.34
82.68
77.50
7.22
SC A=0.5
76.64
49.63
49.55
0.00
44.78
43.85
82.30
77.31
1.97
SC A=1.0
77.35
33.06
37.57
0.00
0.00
1.13
82.68
77.01
1.85
SC A=2.0
76.63
34.09
39.56
0.00
0.00
5.21
82.57
77.18
1.04
34

---

Under review as a conference paper at ICLR 2022
Table 13: Effect of varying À during training for OE (Hendrycks et al. 2019a). the 33 scoring function for models with background class and SHARED COMBI Shown are test accuracy and AUC (AUROC) with Openlmages as training out-distribution.
IN-DISTRIBUTION: CIFAR-10
MEAN AUC
SVHN AUC
LSUN
UNI
SMооTH
C-100 AUC
80M AUC
CELA AUC
OPENIM
MODEL.
Acc.
AUC
AUC
AUC
AUC
OE A=0.1
95.36
97.49
97.96
99.96
99.84
99.93
91.25
93.81
99.70
97.86
OE X=0.25
95.31
97.15
96.93
99.98
99.98
99.93
90.59
92.91
99.74
98.71
OE A=0.5
95.27
97.19
97.53
99.99
99.95
99.99
90.29
92.73
99.87
99.20
OE A=1.0
95.06
97.28
98.49
99.99
99.99
99.99
90.03
92.53
99.91
99.43
OE x=2.0
95.19
97.19
97.99
99.99
99.99
100.00
89.89
92.56
99.93
99.67
BGC #3 A=0.1
95.38
97.37
97.32
99.99
99.93
99.63
91.18
93.74
99.78
98.39
BGC #3 A=0.25
95.27
97.43
98.25
100.00
99.96
99.98
90.85
93.09
99.91
99.13
BGC ss A=0.5
95.23
97.44
98.62
100.00
99.93
99.98
90.80
92.84
99.89
99.48
BGC 9: A=1.0
95.21
97.21
98.87
100.00
99.98
99.62
90.47
92.41
99.08
99.71
A= 95.28
97.10
98.71
100.00
99.98
100.00
89.93
91.94
99.11
99.86
SC A=0.1
95.20
97.29
97.87
99.99
99.91
99.98
90.69
92.82
99.77
98.32
SC A=0.25
95.21
97.33
97.90
99.99
99.91
99.97
90.57
93.08
99.92
99.17
SC A=0.5
95.25
97.34
98.68
100.00
99.95
100.00
90.39
92.56
99.78
99.57
83 X=1.0
95.28
97.26
98.62
100.00
99.93
99.94
89.75
92.85
99.71
99.88
SC 83 A=2.0
95.26
97.13
98.59
100.00
100.00
99.99
89.95
92.25
99.13
99.88
IN-DISTRIBUTION: CIFAR-100
MEAN AUC
SVHN AUC
LSUN
UNI
SMOOTH
C-10
80M
OPENIM
MODEL
Acc.
AUC
AUC
AUC
AUC
AUC
AUC
OE A=0.1
77.28
87.38
83.69
99.79
94.79
91.50
76.08
78.43
95.73
OE A=0.25
76.96
88.31
86.94
99.98
97.74
90.74
76.11
78.35
98.51
OE X=0.5
77.22
89.82
87.22
99.98
99.74
98.95
75.39
77.64
98.94
OE A=1.0
77.19
90.37
89.54
99.98
99.03
99.68
75.95
78.03
99.67
OE A=2.0
76.95
90.56
91.42
99.99
99.78
99.08
75.51
77.58
99.76
BGC ss A=0.1
76.87
88.86
83.41
99.87
98.35
97.37
75.97
78.21
95.99
BGC ss A=0.25
77.05
89.29
89.10
99.99
97.13
95.91
75.96
77.68
98.08
BGC s3 A=0.5
77.17
90.17
90.07
99.99
97.99
99.07
75.49
78.42
99.04
BGC sa A=1.0
77.61
90.46
90.46
99.99
99.88
99.74
74.88
77.82
99.64
BGC 8: A=2.0
77.26
90.83
91.74
99.99
99.72
99.80
75.72
78.02
99.67
SC A=0.1
77.35
86.08
85.56
99.95
84.55
93.24
75.27
77.89
96.74
SC A=0.25
76.96
88.76
85.08
99.98
97.73
97.17
75.11
77.51
98.31
8 >=0.5
76.64
88.42
88.80
99.99
94.61
93.94
75.63
77.52
99.51
SC 8; A=1.0
77.35
90.73
91.69
99.99
99.57
99.53
75.50
78.10
99.57
SC A=2.0
76.63
90.54
91.24
99.99
99.86
98.93
75.30
77.91
99.74
35

---

Under review as a conference paper at ICLR 2022
L STATISTICS OVER FIVE RUNS WITH DIFFERENT SEEDS
Table 14: Mean μ and standard deviation o F TPR measure for different methods and scoring functions over five runs each for models with OpenImages as training out-distribution. The training details are the same as for the results shown in Table
IN-DISTRIBUTION: CIFAR-10
MEAN FPR
SVHN FPR
LSUN
UNI FPR
SMOOTHC-
80M FPR
CELA FPR
OPENIM
MODEL
ACC.
FPR
FPR
FPR
FPR
OE
95.11
15.49
11.42
0.00
0.00
0.09
54.35
42.14
0.40
2.75
OE a
0.04
0.32
1.75
0.00
0.00
0.18
0.61
0.55
0.11
0.49
BGC 81 j
19.18
3.86
0.00
0.00
0.00
73.03
56.76
0.60
0.05
BGC
0.32
0.97
0.00
0.00
0.00
1.17
1.19
0.32
0.01
BGC $2 μ
95.29
15.96
8.48
0.00
0.04
0.42
55.41
44.52
2.87
1.09
BGC 82
0.10
0.63
0.88
0.00
0.05
0.84
1.57
1.55
1.94
0.34
BGC #3 /
95.29
15.99
8.69
0.00
0.04
0.47
55.29
44.49
2.94
BGC 83
0.10
0.63
0.91
0.00
0.06
0.95
1.53
1.53
1.99
0.35
SHARED BINDISC μ
20.45
5.30
0.00
0.00
0.00
77.19
59.63
1.06
0.05
SHARED BINDISC a
0.49
1.02
0.00
0.00
0.00
1.09
3.00
0.46
0.02
SHARED CLASSI μ
95.21
33.21
26.37
9.33
49.73
11.31
57.34
48.37
29.98
36.73
SHARED CLASSI
0.07
3.81
1.24
1.35
29.63
12.22
1.18
1.19
3.85
1.67
SHARED COMBI #2 1
95.21
16.28
0.00
0.20
0.00
57.14
44.92
2.92
0.95
SHARED COMBI 82
0.07
0.39
1.11
0.00
0.38
0.00
1.39
1.37
1.42
0.23
SHARED COMBI
95.21
16.35
9.06
0.00
0.24
0.00
57.13
45.02
3.03
1.03
SHARED COMBI
0.07
0.41
1.13
0.00
0.46
0.00
1.38
1.38
1.47
0.24
IN-DISTRIBUTION: CIFAR-100
MEAN
SVHN
LSUN
UNI
SMOOTH
C-10
80M
OPeNIm
MODEL
Acc.
FPR
FPR
FPR
FPR
FPR
FPR
FPR
FPR
OE
77.13
35.16
45.27
0.00
0.13
5.15
83.47
76.93
1.66
OE
0.23
0.80
5.35
0.00
0.27
5.35
0.89
0.59
0.54
BGC 81 J
31.12
11.88
0.00
0.00
0.12
93.85
80.85
0.08
BGC 81
0.41
2.20
0.00
0.00
0.24
0.15
0.98
0.03
BGC #2 μ
77.30
35.54
40.11
0.00
7.58
4.90
83.44
77.20
1.67
BGC #2
0.35
2.12
5.67
0.00
10.26
5.35
0.80
0.73
0.55
BGC $3 /
77.30
35.64
40.33
0.00
7.89
4.98
83.43
77.21
1.69
BGC $s
0.35
2.16
5.68
0.00
10.68
5.40
0.81
0.73
0.56
SHARED BINDISC /
32.16
13.43
0.00
0.00
0.00
95.13
84.39
0.05
SHARED BINDISC a
0.22
2.10
0.00
0.00
0.00
0.19
0.83
0.02
SHARED CLASSI μ
77.11
56.52
64.95
2.73
81.18
31.85
81.61
76.80
22.97
SHARED CLASSI
0.19
6.86
5.79
1.24
31.58
20.90
0.53
0.90
2.95
SHARED COMBI $2 1
77.11
33.13
35.93
0.00
0.00
3.17
83.26
76.43
1.13
SHARED COMBI 82
0.19
0.59
4.68
0.00
0.00
4.35
0.51
0.82
0.36
SHARED COMBI
77.11
33.18
36.17
0.00
0.00
3.23
83.24
76.43
1.14
SHARED COMBI
0.19
0.59
4.70
0.00
0.00
4.40
0.51
0.81
0.38
36

---

Under review as a conference paper at ICLR 2022
Table 15: Mean μ and standard deviation of the AUROC measure for different methods and scoring functions over five runs each for models with OpenImages as training uut-distrbution. The training details the same for the results shown in Table
IN-DISTRIBUTION: CIFAR-10
MEAN AUC
SVHN
LSUN
UNI
SMOOTHC-100
80M
CELA
OPENIM
MODEL
ACC.
AUC
AUC
AUC
AUC
AUC
AUC
AUC
AUC
OE
95.11
97.25
98.23
99.99
99.96
99.93
90.13
92.58
99.91
99.52
OE
0.04
0.05
0.01
0.04
0.07
0.01
0.06
BGC 81 μ
94.59
99.16
100.00
99.98
99.97
78.02
85.15
99.85
99.96
BGC 81
0.27
0.21
0.00
0.02
0.02
1.02
0.85
0.07
0.01
BGC $2 μ
95.29
97.33
98.74
100.00
99.98
99.92
90.48
92.58
99.59
99.81
BGC 82
0.10
0.08
0.12
0.00
0.03
0.13
0.18
0.22
0.27
0.05
BGC #3 /
95.29
97.32
98.71
100.00
99.97
99.91
90.49
92.58
99.58
99.80
BGC 83 a
0.10
0.00
0.04
0.15
0.22
0.28
0.05
SHARED BINDISC μ
92.10
98.68
100.00
99.94
99.98
67.81
78.51
99.75
99.95
SHARED BINDISC Ø
0.42
0.23
0.00
0.04
0.02
1.30
1.85
0.10
0.01
SHARED CLASSI
95.21
95.10
96.36
98.45
94.26
98.15
90.29
92.18
95.98
93.27
SHARED CLASSI
0.07
0.39
0.18
0.15
3.07
1.21
0.18
0.21
0.46
0.34
SHARED COMBI
95.21
97.24
98.67
100.00
99.92
99.98
90.07
92.47
99.58
99.84
SHARED COMBI 82
0.07
0.17
0.08
0.02
0.23
0.20
0.03
SHARED COMBI 83
95.21
97.24
98.64
100.00
99.91
99.98
90.09
92.48
99.56
99.83
SHARED COMBI
0.07
0.04
0.17
0.00
0.10
0.02
0.22
0.23
0.20
0.03
IN-DISTRIBUTION CIFAR- 100
MEAN
SVHN
LSUN
UNI
SMOOTH
C-10
80M
OPeNIm
MODEL
AcC.
AUC
AUC
AUC
AUC
AUC
AUC
AUC
AUC
OE
77.13
90.24
89.58
99.98
99.66
98.76
75.50
77.96
99.54
OE
0.23
0.16
1.10
0.00
0.32
0.77
0.42
0.28
0.13
BGC 81 μ
88.48
97.17
99.99
99.61
99.57
60.96
73.57
99.92
BGC 81 07
0.17
0.38
0.00
0.14
0.21
0.38
0.01
BGC 82 μ
77.30
90.23
90.45
99.99
98.60
98.90
75.34
78.12
99.55
BGC 82
0.35
0.27
1.10
0.00
1.50
1.04
0.32
0.31
0.13
BGC 83 μ
77.30
90.22
90.41
99.99
98.56
98.89
75.34
78.12
99.55
BGC $3 a
0.35
0.28
1.10
0.00
1.55
1.05
0.32
0.31
0.13
SHaREd BINDISC j
84.80
96.66
99.99
99.64
99.54
48.89
64.09
99.93
SHARED BINDISC a
0.24
0.55
0.00
0.12
0.13
1.01
1.00
0.01
SHARED CLASSI J
77.11
84.42
84.18
99.40
75.39
93.54
75.82
78.16
94.82
SHARED CLASSI
0.19
2.29
2.09
0.20
12.57
5.07
0.14
0.25
0.68
SHARED COMBI
77.11
90.71
91.89
99.99
99.60
99.23
75.31
78.24
99.72
SHARED COMBI 82
0.19
0.10
0.00
0.07
0.68
0.20
0.23
0.08
SHARED COMBI 83
77.11
90.70
91.85
99.99
99.59
99.22
75.32
78.24
99.72
SHARED COMBI
0.19
0.10
1.11
0.00
0.08
0.69
0.20
0.23
0.08
37


Reviews:
[{"review_id": "kCMRvOBV3b1", "reviewer": "Reviewer_YRfA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The paper is well-written and relatively easy to follow.\n* All three of the author's contributions seem interesting and well-supported.\n* I feel that the authors achieve notable progress towards their stated goal (a \"better understanding of the key components of different OOD detection methods and to identify the key properties which lead to SOTA OOD detection performance\"), particularly with their first and third contributions.\n\nWeaknesses:\n* In my view, the authors do not make particularly clear what the problem their work is aiming to solve is or why it needs to be solved. What's missing from the existing literature, and why is it important to fill in that gap? Their entire introduction is essentially a \"related work\" section, and then suddenly shifts to a list of three contributions. But the introduction does not explain the need for these contributions. Their stated goal (which I quoted above) comes across as quite vague. It is therefore also difficult for me as a reviewer (with limited expertise in this area) to make a judgement call as to the importance of this paper.", "summary_of_the_paper": "The authors aim to to recognize common objectives in OOD detection as well as to identify the implicit scoring functions of different OOD detection methods. They show that binary discrimination between in- and (different) out distributions is equivalent to several different formulations of the OOD detection problem. They find that, when trained in a shared fashion with a standard classifier, this binary\ndiscriminator reaches an OOD detection performance similar to that of Outlier Exposure.", "summary_of_the_review": "The authors present well-supported contributions that seem interesting, but do little to explain what problem their contributions solve or why their contributions are needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jV5s7YOYnM5", "reviewer": "Reviewer_YhZ7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is trying to demonstrate that a simple baseline can out-perform other more complicated methods. I appreciate such efforts and I think they provide a good check for the field in general.\n\nWeaknesses:\n\n1) The paper is poorly written and hard to follow. For example, the intro discusses why OOD detection is an important problem and then delves into related work, but fails to properly motivate the contributions by, say, discussing why the rankings induced by the Bayes optimal classifier are relevant, why we may want to revisit and better understand the differences between various OOD methods, or why we may want to consider a binary discriminator method. As another example, Section 2 presents theories and definitions related to when scoring functions are equivalent without explaining to the reader why it is necessary to understand or follow the math.\n\n2) Table 1 actually shows that Outlier Exposure outperforms a Binary Classifier on average on the CIFAR-10 dataset? So only on one of the two datasets do we see the claimed result?\n\n3) Experiments are not that thorough. Only CIFAR-10 and CIFAR-100 are used as in-distribution datasets, and OpenImages is the only out-of-distribution dataset.\n\n3) A binary discriminator requires knowing which data is OOD at training time, and having a large amount of this OOD data. In practice, we may not have access to samples from the OOD dataset to train on, or we may have only a small number of samples. It's not clear to me whether other OOD detection methods avoid this requirement.  I would appreciate if there was more discussion about this topic.", "summary_of_the_paper": "The authors show that 1) a simple baseline,  binary discriminator between in- and out-of- distribution data, is competitive with state-of-the-art OOD detection techniques and 2) the Bayes optimal scoring functions of several proposed methods for out-of-distribution detection are equivalent to the scoring functions of simple binary discriminators. The authors also aim to provide a better understanding of key components of various OOD detection methods such as Outlier Exposure.\n", "summary_of_the_review": "The paper was poorly written and difficult to follow. I recommend that the authors try to make their presentation more clear and concise.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iJ5tOboQrz8", "reviewer": "Reviewer_vYWv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is not completely new, but the purpose of this paper is to identify common objectives as well as the identification of the implicit scoring functions of different OOD detection methods. Also the proposed method is simple and efficient. The connection to the previous work is comprehensively discussed.\n\nI found the table results are a bit too busy to follow, with many different methods and scores. It would be great if you can highlight the best values in each column. Also, the FPR@95%TPR can be very sensitive. I would recommend to report AUROC in the main text. \n\nHere is a relevant paper to the idea of training the binary discriminator between in- and out-distribution together with a standard classifier on the in-distribution in a shared fashion. It may be worth noting that. \n\nRoy, Abhijit Guha, et al. \"Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions.\" arXiv preprint arXiv:2104.03829 (2021).\nhttps://arxiv.org/abs/2104.03829", "summary_of_the_paper": "The paper shows that binary discrimination between in- and out- of distributions is equivalent to several generative model based OOD detection approaches such as likelihood ratios. Moreover, the paper shows that when the binary classifier between in- and out- trained in a shared fashion with a standard classifier for in-distribution classes, and using a score which integrates information from p(i|x) and p(y|x, i), the OOD detection performance achieves the state-of-the-art.", "summary_of_the_review": "The paper provides helpful insights to connect methods for OOD detection tasks. It has both comprehensive theoretical and empirical analysis. The paper can be further polished to make it easy to follow. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XNkQPwhA2Cl", "reviewer": "Reviewer_iH61", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This uses new dataset i.e. OpenImages as $D^{ood}_{train}$ instead of TinyImages because it was retracted for ethical concerns. How was this decision to choose this specific dataset motivated? Are there any specific characteristics that you searched for? Have you explored how this train OOD dataset should be chosen?\nThe method addresses many OOD methods in the literature from the past, but refrained from including other/more recent ones that uses self supervised[5][6], contrastive learning[7][8][9], adversarial resampling[11], energy based methods[9] etc. It would have been great if authors could discuss more on this and present how their method/derivation fits on any(not all) of these more recent frameworks/methods. \n\n## Minor Issue: \nI think there is a typo in lower table 2 heading, Did you mean C-10 instead of C-100 between SMOOTH and 80M FPR? \n\n## Strength\nThis paper provides insights and findings of properties of different OOD detection approaches. This will help in getting more understanding of OOD methods and building novel ones further.  Most of the previous work were mainly empirical and lacked mathematical foundation\nThe finding of uncertainty based methods being equivalent under the hood with binary discriminators is interesting. It explains and provides some groundings to why those different methods might be performing similar ways.\nIt includes well-motivated experiments, their clean code implementation and results as well as a comprehensive appendix to support the claims. \n## Weakness\nThe empirical significance and novelty of third contribution seems limited to me since the claim resembles the ones made by Thulasidasan et al. [2021] and Mohseni et al. [2020]. I’d request the authors to discuss more on the effect of their advancements(using unlabelled data($x_{r}^{IN}$) with BGC) and empirically discuss/show the significance of the gains.  \n\n\n", "summary_of_the_paper": "The paper analyzes different OOD detection methods and show that even if the formulation for many OOD methods were different, the binary discrimination is equivalent to those different types of methods when the rankings induced by Bayes optimal classifier are analyzed. They also derive implicit scoring functions for confidence loss of OE and for BGC and compare them with “optimal” scoring functions. They also claim that training binary discriminator (in-dist vs out-dist) in a shared fashion along with standard classifier reaches state-of-the-art OOD performance. ", "summary_of_the_review": "The paper provides a good survey and insights of existing OOD approaches discussing their similarities and differences. While I commend the mathematical derivation  and analysis of multiple existing OOD methods, the novelty of empirical contributions seem marginally significant to me. I am open to increasing my score  if authors can address some of my concerns above, provide some judgments on more recent methods and/or further insights on the statistical significance (like standard deviation) of the top performing models i.e.  OE, BGC and SHARED_COMBI and provide evidence to significance of the  superiority of SHARED_COMBI over others. Especially since the scores in tables for OE and BGC seem slightly different from the ones reported in the original paper (comparing Appendix F since a different dataset was used in main table). \n\n### References:\n[1] Neal, Lawrence, et al. \"Open set learning with counterfactual images.\" in  ECCV 2018.\\\n[2] Thulasidasan, Sushil, et. al. (2021) \"An Effective Baseline for Robustness to Distributional Shift\"\\\n[3] Mohseni, et al. \"Self-Supervised Learning for Generalizable Out-of-Distribution Detection.\" AAAI. 2020.\\\n[4] Zhang et. al. \"Universum prescription: Regularization using unlabeled data.\" in AAAI 2017.\\\n[5] Hendrycks et. al. Using self-supervised learning can improve model robustness and uncertainty. In Neurips 2019.\\\n[6] Golan et. al. \"Deep anomaly detection using geometric transformations.\" in Neurips 2018.\\\n[7] Winkens, Bunel, Roy et. al. (2020). Contrastive training for improved out-of-distribution detection.\\\n[8] Tack, Mo et. al \"CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances\" in Neurips 2020.\\\n[9] Liu et. al (2020) \"Hybrid discriminative-generative training via contrastive learning.\"\\\n[10] Liu et al. \"Energy-based Out-of-distribution Detection, in Neurips 2020.\"\\\n[11] Li et. al. \"Background Data Resampling for Outlier-Aware Classification\", in CVPR 2020.\\\n[12] Hendrycks et al. \"Deep anomaly detection with outlier exposure\". In ICLR, 2019.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PpcV4jgVUFE", "reviewer": "Reviewer_Lwwq", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ (+) The paper is clearly written, has a good story line, is self contained and appendix A is useful/sufficient to grasp all mathematical parts.\n+ (+) The main contribution of the paper is to unify under the same mathematical language different methods that were previously related less clearly/effectively.\n+ (+) In the theoretical part of the paper - especially Sec. 3.1 and 3.2 -, mathematical steps are always interwined with intuitive but sound explanations (that I personally found eye-opening at times).\n\n+ (-) The analysis framework is based on bayes optimality which is asymptotic - as the authors state in the ethics and reproducibility statement section at the end of the manuscript. But this is not an ethic nor a reproducibility matter: it is a peculiarity / design choice of this work that should be crystal clear to the reader, who will decide if this is a limitation she cares about or less. I would have liked to read it in Sec. 2 or Sec. 3 at most.", "summary_of_the_paper": "In this paper, the authors bring together recent work in the OOD detection problem and provide to the reader a sound mathematical framework to understand similarities and differences among these. The framework is based on the equivalence class of scoring function under the AUC / FPR@qTPR metrics and bayes optimality. The tools introduced in the paper allow the authors to explain why different methods perform largely similarly, when one scoring function should be preferred to others and draw conclusions regarding training with / without label data of the in-distribution set.", "summary_of_the_review": "Theoretical insights that unify different methods under the same light and allow new analysis are so valuable to the community, especially when they are so clearly exposed. The authors also show these theoretical insights are actionable, by drawing new conclusions later supported by empirical evidence. I think people working in the field will benefit from such a clear and encompassing narrative.\n\nMy review could be positively biased: Since I am not an active researcher in the field it was difficult for me to provide absolute meaning to the numbers in the experimental section, although on a relative scale I was able to follow the authors' conclusions. Because of my lack of expertise in this specific field I cannot provide positive nor negative feedback on this and on the related work reporting.\n\nFor these reasons I will take into great consideration all other reviews and discussion for the final score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contributes to the understanding of out-of-distribution detection by showing that binary discrimination between in- and out-distribution examples 'is equivalent to several different formulations of the out-of-distribution detection problem'. The paper shows this in an asymptotic setup based on studying likelihood ratios for distinguishing in-distribution examples from out-of-distribution examples. The paper also provides numerical results showing that a simple baseline based on binary classification works well.

The paper got very mixed responses ranging from strong accept to reject:
- Reviewer YhZ7 (recommending 3: reject) raises several important concerns, specifically that the paper doesn't explain the significance of its contributions adequately, that experiments are not thorough enough (for example that only one out-of-distribution dataset is considered), and that to train a binary classifier one needs to have sufficiently many out-of-distribution examples. 
The authors argued in response that the purpose of the paper is to provide an understanding of existing methods that are often empirically driven, made revisions to the exposition, and point out that they actually evaluate on six/seven out-of-distribution test sets. 
After discussion, the reviewer is still concerned that the paper states 'We show that when training the binary discriminator between in- and out-distribution together with a standard classifier on the in-distribution in a shared fashion, the binary discriminator reaches state-of-the-art OOD detection performance' as a contribution and that this claim is not supported by the results in the paper. The authors say they are happy to drop this particular statement and emphasize that their contribution is that that a binary classifier can be a useful tool for OOD detection. The reviewer is not satisfied by this response, as the reviewer feels that this makes the contribution much less impactful. 

- Reviewer iH61 (recommending 6: marginally above, initially reject) pointed out that the significance of one of the contributions is limited, since the claims resemble the ones by Thulasidasan et al. [2021] and Mohseni et al. [2020], and initially recommends to reject. The authors respond that those two papers only aim at good performance, but do not unify existing approaches, as the paper under review does. The reviewer slightly raised their score, but again points out that the previous works already show that a binary discriminator performs well. 

- Reviewer Lwwq (recommending 10: strong accept) appreciates the unification of different methods and votes for strong acceptance. The reviewer also points out that he/she is not an expert in the field, and thus this reviewer's rating should be taken with care. 

- Reviewer YRfA (recommending 8: accept) points out that the authors make notable progress towards a better understanding of OOD methods, but is concerned about what problem the authors are trying to solve and its significance, and states that he/she cannot judge the importance of the paper.

- Reviewer vYWv (recommending 6: marginally above, initially recommending reject) finds that the paper provides helpful insights to connect methods for OOD detection tasks, and weakly recommends acceptance.  

The reviewer's opinions on this paper vary significantly. Initially, a major selling point of the paper was that 'the binary discriminator reaches state-of-the-art OOD detection performance', but after discussion, the authors and reviewers agree that this statement is not supported by experiments, and the idea of using a binary discriminator is also not new, and thus everyone agrees that this statement should be removed. 
This leaves as the major contribution an improved understanding of a variety of methods, and casting them as versions of a binary classifier. 
This by itself would be sufficient to carry a paper, however the stated equivalence is rather weak as it is based on an asymptotic analysis, and in the asymptotic regime, out-of-distribution detection is rather trivial because the distributions are given. This also explains why in the paper's experiments all the methods that are asymptotically related behave quite differently in experiments. 

I do not recommend this paper for acceptance. I've read the paper and I've thought quite a while it and its reviews. I have also discussed the paper with a colleague who works actively on out-of-distribution detection, since I'm not an expert on this topic myself. While in general I find it very valuable to unify and to understand existing out-of-distribution algorithms better, I don't see how the particular interpretation provided by the paper is impactful, since it is unclear how the connection drawn in an asymptotic setup for Bayes classifiers actually extend to concrete OOD detection algorithms, which operate in the finite sample regime.
Decision: Reject