Adversarial Distributions Against Out-of-Distribution Detectors
Authors: Sangwoong Yoon
Jinwon Choi
Yonghyeon Lee
Yung-Kyun Noh
Frank C. Park
Venue: ICLR 2022 Submitted (2022)
Keywords: out-of-distribution detection
outlier detection
adversarial attack
model evaluation
markov chain monte carlo

Abstract: Out-of-distribution (OOD) detection is the task of determining whether an input lies outside the training data distribution. As an outlier may deviate from the training distribution in unexpected ways, an ideal OOD detector should be able to detect all types of outliers. However, current evaluation protocols test a detector over OOD datasets that cover only a small fraction of all possible outliers, leading to overly optimistic views of OOD detector performance.  In this paper, we propose a novel evaluation framework for OOD detection that tests a detector over a larger, unexplored space of outliers.  In our framework, a detector is evaluated with samples from its adversarial distribution, which generates diverse outlier samples that are likely to be misclassified as in-distribution by the detector. Using adversarial distributions, we investigate OOD detectors with reported near-perfect performance on standard benchmarks like CIFAR-10 vs SVHN. Our methods discover a wide range of samples that are obviously outlier but recognized as in-distribution by the detectors, indicating that current state-of-the-art detectors are not as perfect as they seem on existing benchmarks.

Under review as a conference paper at ICLR 2022
ADVERSARIAL DISTRIBUTIONS AGAINST OUT-OF-DISTRIBUTE DETECTORS
Anonymous authors
Paper under double-blind review
ABSTRACT
Out-of-distribution (OOD) detection the task of determining whether an input lies outside the training data distribution. As outlier may deviate from the training distribution in unexpected ways, an ideal OOD detector should be able to detect all types of outliers. However, current evaluation protocols test detector over OOD datasets that cover only a small fraction of all possible outliers, leading overly optimistic views of OOD detector performance. In this paper, we pro- pose a novel evaluation framework for OOD detection that tests a detector over a larger, unexplored space of outliers. In our framework, detector evaluated with samples from its adversarial distribution, which generates diverse outlier samples that are likely be misclassified as in-distribution by the detector. Using adversarial distributions, we investigate OOD detectors with reported near-perfect performance on standard benchmarks like CIFAR-10 10 vs SVHN. Our methods dis- cover a wide range of samples that are obviously outlier but recognized as in- distribution by the detectors, indicating that current state-of-the-art detectors are not perfect they seem on existing benchmarks.
INTRODUCTION
Identifying whether input datum lies outside the training data distribution is one of the canoni- cal problems machine learning. Over its long history, the problem has been called by multiple names, including novelty detection (Markou & Singh 2003), outlier detection (Hawkins) 1980), one- class classification (Japkowicz et al. 1995). and more recently, out-of-distribution (OOD) detection (Hendrycks & Gimpel (016)). Investigation of the problem has resulted in a number of real-world applications. for example, medical diagnosis (Li et al. 2019) and inspection of defective parts and products (Bergmann et al., 2019). The relevance of OOD detection growing beyond these appli- cations, an OOD detector is considered an essential component of a trustworthy machine learning system. For example, without reliable OOD detector, an image classifier trained to classify cats and dogs may incorrectly classify human as belonging to of these two classes (Hendrycks) jet al., 2019b). In order to advance the development of reliable OOD detection algorithms, a more comprehensive evaluation protocol is needed.
Current evaluation protocols adopted by the community provide a distorted view of a detector's performance for two reasons. First, OOD detectors are tested over small fraction of possible outliers. Detectors are typically evaluated using test OOD datasets chosen by a researcher. Since the test OOD datasets do not cover the entire space of outliers, there may exist untested outliers that the detector fails to classify correctly, even though the detector perfectly detects the chosen test OOD points as shown in Figure 1. To assess detector's 's performance in more comprehensive and systematic way, method is needed to test detector over larger, unexplored space beyond what covered by the test OOD datasets.
Second, the current evaluation protocol neglects the worst-case behavior of an OOD detector. The average performance metric is often not sufficient build trust on detector, because in safety- critical applications even a single mistake can result in fatal consequences. A detector should be tested adversarially, through an active search for its worst-case failure mode, i.e., an outlier that classified with maximal confidence as an inlier. Such failure cases may reveal weaknesses of the tested detector, and provide informative clues for building better detection algorithm.

---

Under review as a conference paper at ICLR 2022
In this paper, propose novel evaluation protocol of OOD detectors that addresses the above-mentioned limi- tations of current evaluation methods. We first formulate the notion of adversarial search against an OOD detec- tor, search problem of finding outlier that detector classifies as an inlier with the greatest confidence. A de- tector may have more than one significant failure mode, and it is more informative to find set of diverse failure cases instead of the most critical one. To that end, we pro- pose the ad adversarial distribution against an OOD detec- tor, which generates outlier samples that are likely to misclassified by the given detector. Measuring OOD de- tection performance against samples from a detector's 's ad- versarial distribution gives accurate and finer-grained assessment on its performance.
Test OOD
In-Dist.
Figure 1:
An illustration of OOD detec- tion. The test data (red stars) are per- fectly predicted OOD (red hatching), but other untested outliers (gray stars) are misclassified in-distribution. The blue shade indicates the support of the training data distribution, the ideal deci- sion boundary for OOD detection.
To ensure that samples from adversarial distribution are indeed outliers, the distribution needs to be supported on zero-inlier space, a without any overlap to the in- lier distribution. Meanwhile, the zero-inlier space should be large so that we may observe meaningful failure mode of OOD detectors within the space. How- ever, finding such space is generally challenging, as the true boundary between inliers and outliers unknown, We circumvent this challenge by building generative model over known outliers. Our construction of zero-inlier spaces contain diverse samples.
We implement 11 previously proposed OOD detectors and investigate their behavior using their ad- versarial distributions. Among the tested detectors, 8 detectors report near- mear-perfect OOD detection performance on a popular benchmark, CIFAR-I0 (in) vs SVHN (out), effectively being indistin- guishable with respect to their performance. Our investigation reveals that the 8 detectors in fact have diverging degrees of detection quality outside the SVHN test set. Our methods also lead to several interesting insights that suggest techniques for improving OOD detection.
Our main contributions can be summarized as follows:
We propose the adversarial search and adversarial distributions, which can be used to evaluate OOD detection algorithms beyond pre-defined test OOD dataset;
We provide practical techniques define the space of outliers that contains samples outside the test OOD dataset;
By examining the state-of-the-arr OOD detectors, show that OOD detectors with seemingly equivalent performance differ significantly in their failure modes;
Related Work Testing the worst-case performance of an algorithm is highly related to evaluating adversarial robustness of the detector. Detection of adversarially perturbed outliers is investigated in previous literature where perturbation is assumed be restricted in small norm-ball Hein al. (2019); Meinke & Hein 2020);; Bitterwolf et al. (2020). The idea of using an autoencoder during an adversarial attack is investigated in Tu al. (2019). The idea of generating outliers are investigated in the context of improving OOD detection Chen et al. (2020) or adversarial attack Song et al. 2018)
In Section 2, we provide essential preliminaries. Section 3 formulates adversarial search and ad- versarial distributions, and Section 4 introduces techniques to construct a zero-inlier space, Our main experimental results are presented in Section 5, with deeper discussions provided in Section 6. Section concludes the paper.
2 BACKGROUND:
DETECTION
2. DEFINITION
We consider probability distribution of interest P.n. samples from which consider as inliers. Each sample is represented as a D-dimensional real-valued vector. The probability density function of Pa,, is denoted (x). We write the support of P... S.n (x)))((x) 0} c X RD,
2

---

Under review as a conference paper at ICLR 2022
where X' is the data space, the set of all possible values for data, which we assume be compact. Then, we define OOD-ness ness as follows:
Definition 1. A vector x is out-of-distribution (OOD), or an outlier, with respect to Pin if x does not belong to the support of P.. i.c., & Sin- Conversely, is in-distribution when x E Sm A distribution Q having the support So is OOD to Pin, when Sq O Sin = 2.
Another popular definition of OOD-ness characterizes a vector x as OOD when the vector belongs to a density sub-level set, x E {x(p:..xx 7 (Steinwart et al. 2005 However, the density sub-level set does not provide a consistent characterization of the OOD-ness. coordinate may not be classified as OOD in different coordinate, because a probability density function can be arbitrarily distorted via an invertible coordinate transform as pointed out in (Lan & Dinh 2020). On the contrary, our definition of OOD using the density support provides an invariant characterization of outliers with respect to such transforms.
vector classified as OOD in
An OOD detector fxx RD likely to be an outlier. A test vector x" is classified as OOD f(x') > 7y for the threshold nT-- detector score refers to the funetion value of fxx). We shall assume f(xx bounded. In this paper, we consider a black-box setting, where any information other than the function value of fxx), such as its gradient, not accessible. An OOD detector is normally trained using an in-distribution dataset Din set of iid samples from Pin. However, some OOD detectors utilize additional datasets other than Din-
is function which outputs a larger value for an input more
2.2 EVALUATION OF OOD DETECTORS
The currently accepted evaluation protocol for OOD detectors relies on one or multiple test 0OD datasets Dout: which contain a finite number of samples considered as OOD with respect Pt. by human prior knowledge. When Pin is distribution of images, test OOD sets are often chosen from separately published image datasets of which contents are different from that of P... For example, when an image dataset of animals is selected as in-distribution, set of digit images can be used as a test OOD dataset.
Given a test OOD dataset, an OOD detector fxx) performs the binary classification against in- distribution dataset, and the quality the classification is considered as an indicator for how good f(x) is as an OOD detector. The classification result is summarized using metrics such as the area under the receiver operating characteristic curve (AUROC or AUC). AUC preferred metric in a number of literature, it does not require the specification of the threshold ny. AUC score of 1.0 indicates the perfect classification, and AUC of 0.5 implies the random guess.
The research community has been focusing pairs, such as CIFAR-10 (in) vs SVHN (out) and Fashion-MNISS (in) vs MNIST (out). These dataset pairs become popular after the reports showing that OOD detectors built upon deep genera- tive models, such as PixeICNN++ (Salimans et al. (017) or Glow (Kingma & Dhariwal 2018), fail to detect outliers (Hendrycks al., [019a] Nalisnick al.) 2019). In fact, the generative-model based detectors score AUC lower than 0.5, meaning that SVHN images are more strongly perceived as CIFAR-IO than the actual CIFAR-IO images by the detectors. This observation spurred intense research efforts, and now there are multiple OOD detectors achieving AUC scores higher than 0.9 or even higher than 0.99 on CIFAR-10 vs SVHN listed in Section 5 Given the near-perfect de- tection score, we question whether the detectors are indeed good OOD detectors beyond the tested examples.
few representative in-distribution and OOD dataset
3 ADVERSARIAL DISTRIBUTIONS
Our goal is to test how robust the state-of-the-art OOD detectors are. The robustness of an OOD detector can be investigated through a search process to find its failure modes. We are particularly interested in failure cases where outlier examples misclassified as in-distribution by the detector. Ideally, we would search for failure modes over all possible outliers, but the true set of outliers Sout = X Sm, is unknown. Instead, specify a search space T which is subset of Sout and only contains outliers. We call T zero-inlier space. In this section, we formulate two search problems on T. The first formulation, Adversarial Search, is based optimization, and the second
3

---

Under review as a conference paper at ICLR 2022
formulation, Adversarial Distribution, is based sampling. Details on selecting meaningful T will be discussed Section
3.1 ADVERSARIAL SEARCI
Finding failure modes of an OOD detector can be formulated as optimization problem. For detector (x) and search space T, an adversarial search problem is finding an element x* in T which has the lowest detector score.
= =arg f(x) such that XE T C Souts
(1)
xET
where x" is the worst-case outlier we search for. If the detector score for the worst-case outlier fxx*) is greater than the score of any samples in in-distribution set, then the detector f(x) can be confirmed as robust in the search space T. Otherwise, we conclude that the tested detector has at least one failure mode and deviates from the ideal detector. The degree of deviation may be used to quantify the robustness of detector.
Example: Adversarial Attack An example of the adversarial search is applying an adversarial perturbation on outliers so that they have small fxx (Bitterwolf et al. 2020). In this case, 'I is X|| S. r, E Dout). A point with the smallest detector score is found via a gradient descent method. The AUC score for classifying the perturbed samples from inliers called Adversarial AUC (Bitterwolf et al. 2020
ball centered at an outlier from a test OOD dataset T = [x] |
3.2 ADVERSARIAL DISTRIBUTIONS
When the search space T is larger than ball centered at outlier, f(x may have multiple local minima, and each minimum may reveal different failure modes detector. To explore multiple failure modes in (x), we propose a novel approach. We deliberately design probability distri- bution pf (x) so that samples from (x()) cover multiple minima of f(x) in T. We call py (x) an Adversarial Distribution of detector fxx).
An adversarial distribution is a Gibbs distribution, also known as an energy-based model. Its energy function is the detector score f(x), i.e., py(x) proportional to exp(-f(x)/I)) The support of py (x) is the search space T.
(f(x)= Z exp(-f(x)/1T
T C Sout and (x) = 0 otherwise,
(2)
where = / (xp((f(x))TT)))) is the normalization constant, and T > 0 is called temperature. < oo, as we have assumed the bounded ((x and the compact data space in Section 2
An adversarial distribution (x) has three important properties by construction. First, samples from (r(x) are guaranteed to be OOD, as the support is T. Second, an adversarial distribution assigns a high probability density samples likely to be classified as in-distribution by the detector, i.e., (x()) is high when f(x) is small. Third, for the limit - O, sampling from an adversarial distribution becomes equivalent the adversarial search, as the probability mass is concentrated on x" in Eq. (T.. Hence, an adversarial distribution can be viewed as a relaxation of the adversarial search, and T governs the degree of relaxation.
Samples from py (x) can be drawn using Markov Chain Monte Carlo (MCMC). MCMC is guaran- teed to visit all modes of a probability distribution at least in theory, and therefore every failure mode of a detector can be generated. Measuring how well fxx classifies inliers from (x)'s samples using metries such AUC gives comprehensive measure on the performance of the detector.
4 ZERO-INLIER SPACE
The support T of adversarial distribution defines the search space where the robustness of an OOD detector is evaluated. While there is some degree of freedom the choice of T, one condition should be strictly satisfied: T should not contain inlier data points, i.e., should be zero-inlier space. However, as the true decision boundary between outliers and inliers is unknown, is challenging to ensure T only contains outliers.
4

---

Under review as a conference paper at ICLR 2022
x
o = [x|x = g())
X
T {x/h(x) < n
In-Dist
In-Dist.
h(x) 1]nt
Figure 2: The illustration of Ty and Ti-
In the example of the adversarial attack provided in Section 3.1 T is set as a ball around a known outlier. T is ensured to be a zero-inlier space as the radius of the ball is very small. However, such a small ball only contains examples visually indistinguishable from its center, lacking diversity. In what follows, we provide an example construction of which contains a more diverse set of outliers.
Here, we describe a procedure for constructing a zero-inlier search space T given a set of in- distribution dataset Din and OOD dataset Dout- Note that T constructed during the evaluation stage of OOD detectors and does not affect an OOD detector being tested. Also, the construction of T requires no additional data compared to typical evaluation protocol which assumes a human- curated OOD dataset.
Instead of a ball around each outlier, we consider a generator function g(z) spanning a set of outlier samples Dout. Such generator is able to produce unseen sample which resembles the known outliers. As the generator is trained only using D Dout, it unlikely for an inlier to be generated from g(z). Nevertheless, to avoid fortuitous generation of an inlier, introduce binary classifier h(x) which is supervised to discriminate Din and Dout-
Let Ty be set of samples generated by g(z), and Th. be of samples classified as they from Dout- Our search space 'T is given as an intersection of TT and Ty:
T=T, Th-
4.1 OUTLIER-SPANNING GENERATOR
A generator g(z) Z + X' is map from lower-dimensions. space Z C R Dz to the data space
X. where Dz smaller than the dimensionality of X. We denote T, as a set of samples that can be produced by g()):
Ty = (x= 23.
(4)
In our implementation of adversarial distributions, we train an autoencoder reconstruct Dout and use its decoder as g(z). As the autoencoder can reconstruct samples in D Dout (with some error), the generator (approximately) spans the space where outliers reside. Similarly, a generator of generative adversarial network can also serve as (z).
It is reported that an autoencoder may reconstruct input vector that is significantly different from its training data even though the autoencoder is not the identity mapping (Tong et al. 2019 Gong al., 2019 Yoon al. 2021). This phenomenon implies that g(z) may be capable of generat- ing samples that is significantly different from Dout. The ability of g(z) to reach outside of Dout may positively affect the diversity Tg. but increases the chance of generating x that is close to inliers. To ensure that T, only contains outliers, we introduce additional component can filter out accidentally generated inliers.
4.2 BINARY CLASSIFIER SEPARATING INLIERS AND OUTLIERS
Here, we develop an additional mechanism for ensuring that inlier is present in our search space by exploiting the fact that there is no overlap of the supports between an inlier distribution, and distribution generating an OOD dataset. When two distributions do not overlap, there exists a binary classifier that perfectly classifies the two distributions.
Proposition 1 (Binary classification of disjoint distributions). Suppose two probability distributions P and Q with their supports Sp and Sq. respectively. P and Q are OOD each other, if and only
5

---

Under review as a conference paper at ICLR 2022
if there exists binary classifier h"(x) X - R that perfectly discriminates P and Q, i.c. there exists a threshold ?)/-- E R such that the classification boundary h" (x) = T7-- separates Sp and SQ.
More formal arguments will provided Appendix A Assume an oracle binary classifier h (x) which perfectly discriminating Pi and Dout From Proposition [ the oracle classifier partitions the whole space into two, where one partition contains only Din and the other contains only Dout. The partition containing Dott can be chosen
zero-inlier space.
Proposition I holds approximately for an empirically obtained classifier h(x). binary classifier often achieves AUC score higher than 0.9999 when evaluated the test splits of Din and Dout- Hence, we define an outlier-side of h(x)'s decision boundary as Th Assuming that h(x) assigns lower value for outliers,
Tn (x Vh, XEX),
for a threshold The However, real-world classifier is not an oracle h(x) * hx(x) and can never be perfect. The classifier may misclassify may be vulnerable to adversarial perturbations. To minimize the effect, we apply few techniques. First, we set the threshold ?/ lower than zero- inlier threshold, which is the smallest ((x) value among all training, validation, and testing inlier examples. Second, test-time augmentation is applied on h(x) to make the prediction more robust. We use horizontal flip and rotations of 90, 180, 270 degrees. Total 5 predictions are averaged. Detailed discussion on the robustness of h(x) is given in Section E.1
5 EXPERIMENTS
In our experiments, we aim reveal previously unknown failure modes of the state-of-the-art OOD detectors using adversarial distributions. We first show that adversarial distributions can find the known weaknesses of baseline detectors. Then, apply methods on the state-of-the-ar OOD detectors which show near-perfect performance on popular benchmark. Finally, we show that the failure modes of the detectors are not readily transferable. However, simple ensemble does not improve the robustness of OOD detection significantly. Table [ summarizes our experimental results.
5.1 EXPERIMENTAL SETTINGS
Datasets We use CIFAR-10 as in-distribution data D.n. SVHN and CelebA are used as test OOD datasets Dous. All datasets splitted into training, validation, and testing sets. All data are 32x32 RGB images. Details on datasets can found Appendix B
OOD Detectors We implement 11 previously proposed outlier detectors. The detectors are grouped into two, the weak and the strong, based on their performance CIFAR-10 SVHN. The weak detectors are outlier detectors that fail dramatically on the benchmark by predicting im- ages in SVHN being more likely be in-distribution than images in CIFAR-10.
The following the list of the weak detectors used in our experiment.
1986); A neural network trained to reconstruct its input. The detector score f f(x) is the reconstruction error of x (Japkowicz et al. 1995).
Autoencoder (AE) (Rumelhart et al.
PixelCNN+ (PXCNN) (Salimans al. 2017): An autoregressive generative model for images.
Glow (Kingma & Dhariwal 2018): generative model based on a normalizing flow.
For PXCNN and Glow, the negative log- likelihood is used the detector score.
We select strong detectors satisfying three criteria: 1) AUC score on CIFAR-10 vs SVHN is higher than 0.9. 2) The code is publicly available and written in PyTorch. The language constraint is introduced for the unified experiment. 3) The performance claimed in the original work should be reproducible.
Normalized AE (NAE) (Yon et al. 2021) AE with additional mechanism for suppressing the reconstruction of outliers. As in AE, f(x) the reconstruction error. Only trained with in-distribution data.

---

Under review as a conference paper at ICLR 2022
Table I: Evaluation of OOD detectors using the adversarial distributions. Test Set indicates the test split of test OOD dataset. Dy denotes the dimension of the latent space of the generator. AUC scores are evaluated using 1,000 samples from adversarial distributions.
OOD Dataset
SVHN
CelebA
OOD Samples
Test Set
AdvDist
Test Set
AdvDist
D
16
32
64
16
32
64
Weak Detectors
Glow
.069
.018
.038
.104
.542
.007
.007
.087
PXCNN
.076
.135
.051
.119
.639
.023
.088
.043
AE
.080
.016
.055
.092
.533
.050
.136
.449
Strong Detectors
NAE
.935
.227
.153
.126
.74
.654
.537
.470
GOOD
.943
.399
.366
.322
.939
.376
.343
.300
ACET
.966
.370
.398
.369
.986
.382
.342
.303
CEDA
.988
.623
.628
.626
.981
.618
.617
.612
SSD
.989
.567
.464
.453
.780
.379
.410
.392
MD
.993
.636
.619
.620
.796
.417
.346
.419
OE
.997
.700
.696
.678
.992
.692
.697
.700
CSI
.998
.866
.894
.86
.900
.888
.822
.561
Ensemble
GOOD-ACET+CEDA
.983
.322
.331
.316
.991
.389
.397
.465
MD+NAE+OE
.996
.723
.525
.747
.978
.732
.684
.664
Outlier Exposure (OE) (Hendrycks et al. 2019at: The predictive confidence of a classifier on an external dataset minimized.
2019): Similarly to OE, the predictive confidence classifier on an external dataset minimized.
Confidence Enhancing Data Augmentation (CEDA) (Hein et al.
Adversarial Confidence Enhancing Training (ACET) (Hein et al. fidence of a classificr on an external dataset minimized.
2019): The worst-case con-
Guaranteed OOD Detector (GOOD) (Bitterwolf al., 2020 IBP (Gowal et al. 2018) is used to obtain the confidence interval in the neighborhood of OOD data points, and the interval is minimized during training.
Mahalnobis Distance (MD) (Lee et al., 2018): Negative minimum Mahalanobis distance to in- distribution data in the hidden representation spaces is used f(x). Mahalanobis distances separately defined for each class and the weight is tuned with a few out-of-distributioo data. Self-Supervised Detector (SSD) (Sehwag al. 2021): fx) computed based on Mahalanobis distance on the representation space which is learned via self-supervised learning. CSI (Tack et al. 2020): Cosine similarity to the nearest training data in the representation space learned via self self-supervised learning is used as f(x).
For OE, CEDA, ACET, and GOOD, the negative maximum softmax probability of a classifier is used as fxx). We utilize the publicly available pretrained models except for MD and SSD where we use the publicly available training scripts. There are number of other competitive OOD detectors we considered but not included the experiment of this paper. See Appendix [D.2] for discussion on such detectors. We will continuously update our leaderboard afterward to include other OOD detection methods.
5.2 IMPLEMENTATION OF ADVERSARIAL DISTRIBUTIONS
Generator gxx) We use the decoder of an convolutional AE with the latent space = 1-1, where D = {16, 32, 64). The autoencoder is trained to reconstruct samples in D D
Binary Classifier hxx) Our binary classifier (xx) is based on ResNet-50 pre-trained on ImageNet. The last layer is replaced with newly initialized binary classification layer, and then the whole

---

Under review as a conference paper at ICLR 2022
Glow
PXCNN
AE
CelebA SVHN
CelebA SVHN
CeiebA SVHN
Figure 3: Random samples from the adversarial distribution of the weak detectors (Dz = 32). More
samples provided in Appendix [F]
GOOD
ACET
CEDA
Celeba SVIN
SVHN
Coleba SVHN
18
Colebé
SSD
MD
OF
CSI
Celedd Svae
Spay
Ce SWAN
Figure 4: Random samples from adversarial distributions of the strong detectors. More samples can be found in Appendix FF
network is fine-tuned to classify Din from Dout. use the logit value as h(x). As a zero-inlier threshold, we set =5 for SVHN and -1 for CelebA.
Adversarial Distribution (x) We apply standardization on f(x). The mean and the standard deviation of detector score is computed on the validation split of in-distribution data. For all experi- ments, the temperature is set T 0.1.
Gibbs sampling is performed Z to generate samples from py (x). At each step, dimension of randomly selected, and a proposal is generated by a unit Gaussian distribution. A proposal outside the domain is projected back. A proposal is accepted based on Metropolis-Hastie run 1,000 independent Markov chains with each chain runs for 10,000 steps. The final states of Markov chains accepted samples. Samples that do not belong to Th rejected. More implementation details can found in Appendix C
criterion. We
5.3 ADVERSARIAL DISTRIBUTIONS AGAINST DETECTORS WITH KNOWN WEAKNESSES
We confirm the effectiveness of adversarial distributions by applying them on the weak detectors, Glow, PXCNN, and AE, where the weaknesses have already been analyzed. The likelihoods of Glow and PXCNN become spuriously high for images with low complexity (Serrà et 2020). Similarly, AE produces also low reconstruction error simple images (Yoon et al. 2021). From Figure 3 the samples from adversarial distributions (x) are consistent with the previously known tendency.
The samples very blurry and large portion of pixels monotone. Simplicity of the samples are more clear when they are compared to the samples from a strong detector's 's py (x) in Figure
5.4 EVALUATING STATE-OF-THE-AA DETECTORS
The proposed methods reveals that 8 strong detectors, with almost equivalent and near-perfect per- formance on CIFAR-10 vs SVHN, are actually very vulnerable outside of the test points. All the strong detectors show significantly decreased AUC against samples from adversarial distributions. Among the tested detectors, CSI is the most robust when SVHN is D However, CSI does not per- form well when outliers and adversarial distributions are based on CelebA. All other detectors show similar degree of performance drop against samples from their adversarial distributions py (x) vi- sualized in Figure Note that most of samples from py (x) retain the visual features of the OOD dataset which py Px) based. Also, the samples are visually dissimilar CIFAR-10.
In the range from 16 to 64, the choice of latent space dimensionality Dz does not significantly affect the efficacy of adversarial distributions in terms of AUC scores. Using smaller Dz makes the search space narrower, but MCMC can performed more effectively. Meanwhile, larger D, increases the diversity of samples, but MCMC becomes challenging.
8

---

Under review as a conference paper at ICLR 2022
OOD Detectors
OOD Detectors
MAE
ALST
MAC
GOOD
ACET
(E.A
SND
MD
MAE
.98
.98
.98
.96
799
1.0
888
093
0000
86
87
85
99
98
0000
999
62
89
ACET
.4
.76
50
97
97
90
95
1.0
63
89
00
085
.6
99
CED4
1.0
88
88
91
69
96
990
07
0.00
Adv.
350
.96
85
.86
83
96
95
.4
350
1.0
94
93
61
.7
1.0
98
070
05
92
76
84
80
97
96
.99
.87
.87
83
.33
.70
.91
.4
90
87
97
.9
97
98
98
98
96
76
.75
.99
004
Figure 5: OOD detection AUC scores evaluated against other detectors' adversarial distribution
samples. (Left) SVHN D2 32 (Right) CelebA D2 32
Transferability and Detector Ensemble We question whether adversarial distribution is trans- ferable, i.e., given two detectors f1 (x) and f2 (x). are samples from Pfi (x) able to deceive another detector f2(x)? We collect samples from ps, (x) then check if f2 (x) can detect them as outliers. The pairwise results are displayed in Figure | showing that there are varying degree of transferability. Adversarial distributions are largely transferable between GOOD, CEDA, and ACET, possibly be- cause their constructions are significantly similar (Hein et al. 2019). Except for the three, the other detectors are generally capable of detecting another detector's s adversarial distribution samples. is interesting to see that NAE also performs well detecting adversarial distribution samples from other detectors, while NAE is not the best performing detector. We suspect the limited transfer- ability is originated from the distinct inductive biases of detectors. For example, NAE is the only autoencoder- based algorithm, and SSD and CSI are trained via different self-supervised learning processes.
When detectors do not share failure modes, a more robust detector may be formed through averaging their (normalized) detection scores. We construct two ensembles of three OOD detectors, with highly correlated failure modes (GOOD+ACE++CEDA) and the other with less correlated failure modes (MD+NAB+OO)) and measure their robustness. From the last two rows in Table [ both ensembles barely improve the robustness of OOD detector compared to the individual detectors.
DISCUSSION
Extensions The outlier space utilized in the analysis can be expanded by augmenting the OOD dataset with an additional OOD dataset. Moreover, there is significant flexibility in the design of g(x). In a specific application, domain knowledge can be embedded in g(x) to provide a more meaningful outlier space.
Limitations Our methods still rely on test OOD datasets obtain gxx) and h(x), and the choice of the datasets need human design. Furthermore, operating in the reduced outlier space, our methods can not confirm that a detector optimal. However, these limitations are also present in the current evaluation protocol as well.
Future Work To facilitate the development of OOD detection methods, we intend publish an online software suite for evaluating OOD detectors based on adversarial distributions. We also plan to publish an online leaderboard displaying the results from an adversarial distribution analysis, and to update the leaderboard with the latest OOD detection methods.
CONCLUSION
In this paper, we have addressed the limitations of the current evaluation protocol for OOD detection and proposed a novel framework, adversarial distributions, that can be used to investigate failure modes of OOD detectors. The proposed framework posing new challenges in OOD detection by discovering unexplored weaknesses in the existing OOD detectors, We expect framework to stimulate the advance of trustworthy machine learning, where a model needs to be evaluated beyond the fixed test dataset.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Our main ethical concern is that a subset of OOD detectors used in our experiment, OE, CEDA, ACET and GOOD, are trained using 80 Million Tiny Images dataset (Torralba et al. 2008), which is retracted by authors over ethical concerns. While we were aware of the issue of the dataset, the use of models trained on the dataset was inevitable because of the reproducibility. We have tried to train OOD detectors using alternative datasets but failed to reach the performance of detectors originally trained on 80 Million Tiny Images. To minimize the effect of the retracted dataset, the dataset was never used directly. We only used the publicly available model checkpoints, and did not download or access to copy of dataset.
REPRODUCIBILITY STATEMENT
As mentioned in intro, we will provide the software suite to evaluate the weaknesses of the OOD detector. Since our algorithm consists of widely used techniques such as autoencoders, image clas- sifiers, and MCMC sampling, there will be no problem with reproduction.
REFERENCES
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9592-9600, 2019.
Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of out-of-distribution data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16085-16095. Cur- ran Associates, Inc., 2020.
neurips cctpsc/ppppppppe////////// .pff
fi file/b90:469632 4tw.e////06000001000770000000000
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Informative outlier matters: Robustifying duitt--diittiuutionn detection using outlier mining. arXiv preprint arXiv:2006. 15207, 2020.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational deterministic autoencoders. In International Conference on Learning Representa- tions, 2020. URL
net / for://ormmmmmm http:/oommmimllggppp112s1
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmentee deep autoencoder for unsupervised anomaly detection. In IEEE International Conference on Computer Vision (ICCV), 2019.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue- sato, Relja Arandjelovie, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv: 1810. 2018.
Douglas M Hawkins. Identification of outliers, volume 11. Springer, 1980.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136. 2016
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019a. https:/7
openreview wwwp:/eeeee/ooodddeeee.oe.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv: 1907.077 2019b.
10

---

Under review as a conference paper at ICLR 2022
Nathalie Japkowicz, Catherine Myers, Mark Gluck, al. A novelty detection approach to classifi- cation. In Proceedings of the International Joint Conference on Artificial Intelligence, volume 1, pp. 518-523, 1995.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Charline Le Lan and Laurent Dinh. Perfect density models cannot guarantee anomaly detection. arXiv preprint arXiv:2012.03808, 2020.
KIMIN Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In 32nd Conference on Neural Information Processing Systems (NIPS). Neural Information Processing Systems Foundation, 2018.
Xingyu Li, Marko Radulovic, Ksenija Kanjer, and Konstantinos N Plataniotis. Discriminative pat- tern mining for breast cancer histopathology image classification via fully convolutional autoen- coder. IEEE Access, 7:36433-36445, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
Markos Markou and Sameer Singh. Novelty detection: a review-par pareew--rrtt 1: statistical approaches.
Signal processing, 83(12):2481-22 2003.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In International Conference on Learning Representations, 2020. URL https:
//openreview. net./formmt forum? htt::/orrmmmm-yyyoGyoorrr d=ByxGkySKwFi
Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087-1092 1953.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't 't know? In International Conference on Learning Representations, 2019. URL
net wtt.//orrummrrrieee H1xhhhCWWW htt.rrrreeeee.eeeeeruuueeeee---
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo. Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: httpp::/ http://pocceeeenn.eeeep 1e 79599868 0w28222200ccc (et.3320cc22d2772990 6dd (al.aa77266cc5ccc 2cccccccc66.292....1..----pee .ddf
new.eeeeeddnn....eu.........s... neurips.e.pppp.... CC/ httppppape/211///1eee
D. E. Rumelhart, E. Hinton, and R. J. Williams. Learning Internal Representations by Error Propagation, pp. 318-362. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 170iv:1701.05517, 2017.
Vikash Sehwag, Mung Chiang, and Prateek Mittll. (SSD): A unified framework for self-supervised outlier detection. In International Conference Learning Representations, 2021. URL https:
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-o---istribution detection with iikelihood-basee generative models. In Interna- tional Conference on Learning Representations, 2020. URL https: http:///pppeeiieeeeee net/ FFormmid=spe forrum?1d=SSxx11pUVVv

---

Under review as a conference paper at ICLR 2022
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. Advances in Neural Information Processing Systems, 31:8312- 8323, 2018.
Ingo Steinwart, Don Hush, and Clint Scovel. classification framework for anomaly detection. Journal of Machine Learning Research, 6(Feb):211-232) 2005.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. arXiv preprint arXiv:2007.081777 2020.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint arXi::/511.0188 2015.
Alexander Tong, Roozbah Yousefzadeh, Guy Wolf, and Smita Krishnaswamy. Fixing bias in reconstruction-basee anomaly detection with lipschitz discriminators. arXiv preprint arXiv:1905.107100 2019.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE transactions on paltern analysis and machine intelligence, 30(1 19111))998-1900 2008
Chun-Chen Tu, Paishun Ting, Pin-Yu Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: Autoencoder-basee zeroth order optimization method for attack- ing black-box neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 742-749, 2019.
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference on Machine Learning, pp- 9690-9700. PMIR, 2020.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An -utto--distribution detection score for variational auto-encoder. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin
(eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20685-20696. 20696. Cur- Associates, Inc., 2020. URL htt.::tttts/tt...t. httpt:pppppppppppeeeppte ngs wwwrinuupp.... wttpsscccppppeee wttpc//pppppp..///// www.eips..cpppppr////.....
Sangwoong Yoon, Yung- Kyun Noh, and Frank Park. Autoencoding under normalization constraints. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12087-12097. PMLR, 18-24 Jul 2021.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
Appendix is organized as follows:
Section A: Proof of Proposition 1
Section B: Datasets
Section C: Implementation of Adversarial Distributions
Section D: Implementation of OOD Detectors
Section Extended Experimental Results
Section F: Additional Samples from Adversarial Distributions
A PROOF OF PROPOSITION
Proposition 1 (Restated). Suppose two probability distributions P and Q with their supports Sp and Sq. respectively. and Q are OOD to each other, if and only if there exists binary classifier hxx)= X' - R that perfectly discriminates P and Q, i.e., there exists threshold R such that classification boundary (x(x) = TJh separates Sp and Sq.
Proof. If P and Q OOD each other, then Sp n Sq 2. Then, h(x) such that fxx) > for x E Sp and h(x) I Th for x E SQ for any constant Th E IR Therefore, h(x) Thh separates Sp and SQ.
construct classifier
Given a classifier (xx) such that h((): > for E Sp, suppose that P and Q are not OOD to each other. Then, there exists x" such that x" Sp and x* Sq at the same time. However, h(x") can not be h(x) > 7h and (xx) < at the same time. Due to the contradiction, given perfectly classifier, P and Q should be OOD each other.
B DATASETS
In our experiments, we use CIFAR-IO, SVHN, and CelebA demonstrate the proposed methods and to evaluate OOD detection algorithms, and their details are described in Table In all cases, we use the official test split as our test datasets. For CIFAR-10 and SVHN, the official train split randomly splitted into train (90%) and valid split (10%). For CelebA, use the official valid split as it given. Each CelebA image center-cropped into 140x 140 image and scaled 32x32 using the bilinear interpolation.
Table 2: Statistics for datasets.
Dataset
Train
Valid
Test
CIFAR-10 Krizhevsky 2009)
45,000
5,000
10,000
SVHN Netzer et al. 2011)
65,930
7,327
26,032
CelebA (Lu al. 2015
162,770
19,867
19,962
No augmentation is applied to test data. However, random horizontal flip with probability 50% and uniform dequantization (Theis al. 2015) are applied to training images in the training of the binary classifiers and the autoencoders in the adversarial search and adversarial distributions.
C IMPLEMENTATION OF ADVERSARIAL DISTRIBUTIONS
C.1 IMPLEMENTATION OF BINARY CLASSIFIERS AND AUTOENCODERS
Binary Classifiers We train binary classifiers hxx) that classify Din, from Dour- Two binary clas- sifier are trained with different D.nt's. 's. For one classifier, Dous is SVHN, and for the other, Dout is CelebA. Dm is CIFAR-10 10 for both classifiers. Both classifiers have an identical architecture. We replace the last layer of ResNet-50 pre- mr-itraine cation layer and fine-tume tue-tune the whole network using binary cross entropy loss. The pre-trainte trained model provided by Torch Vision. Training is performed for 100 epochs using Adam with the learning rate
ImageNet with newly initialized binary classifi-
13

---

Under review as a conference paper at ICLR 2022
of 1 10-5, where epoch is defined with respect to the smaller of sn or Dout. Each mini-batch consists of 128 samples from each Din and Dout- During training, validation loss is recorded and model weights with the best validation loss used in experiments.
The decoder of a convolutional autoencoder is used as the generator g(z). The architecture of the autoencoder similar what used in (Ghosh et al.) 2020) and is described in Table 3 The latent space of this autoencoder is given as -1, where D2 = {16, 32, 64).
Autoencoders
Table 3: Autoencoder architecture. Conv indicates 2D convolution operation with a N x N kernel and M output channels. BN denotes batch normalization, and ReLU means the rectified linear unit activation.
Encoder
Decoder
Conv: (128)-BN-ReLL)
ConwTg(1024)-Rel ReLU-
Conv C256g(256)BBN)
(256))BN-RRLLLU
ConvT,(512)-Re) ReLU-
(512)-BNN-R))))
(25n),225)-Re))) (3)
(1024)))BB
FC(1024)-BN-ReLU
FC(32)-Tanh
We train the autoencoder with samples in the union of Din UDout, where Dout is SVHN or CelebA. Autoencoders are trained for 500 epochs using Adam with the learning rate of x 10 10-4.
C.2 IMPLEMENTATION OF ADVERSARIAL DISTRIBUTIONS
Here, describe the method for sampling from the practical version of adversarial distribution (Eq.
Enforcing Support Constraint In Eq. (2). the constraint Ta needs be enforced, while most MCMC algorithms for real-valued states assume an unbounded space. We enforce the constraint by two-staged approach.
First, instead of the original adversarial distribution in Eq. we run MCMC for an alternative distribution ply (z; g, h). where two barrier functions, H1 Hz(z) and Hz(z), are added to the energy function.
(2) h' exp(-E(z)/T), z ZER
(6)
E(z) f(g(z)) + Hz()) + H2(z),
H1 max(0, h(f(z)) mh),
(8)
Hz(z) 10 Σ max |Zaag| - 1)2,
d=1
where 2(d) is d-th element of vector z, and Z' h) = exxp(=F(2)/T]e Hz()) drives a Markov chain into [z) < }h} by assigning higher energy on z that violates the constraint. Similarly, Hz(z) assigns higher energy for z outside [-1, 1 Dz, Due H2 (z) and Hz(z), samples that comply the constraint z E Tr. appear more frequently in MCMC. Note that H1 (z) and Hz(z) have non-zero only when the constraint is violated, and the E(z) = fz() for Th.
Second, after running MCMC, we reject any sample that does not obey the constraint E TT This rejection process ensures that the generated samples are from the original adversarial distribution Eq.
Metropolis-Hare Hasting Acceptance To generate samples from an adversarial distribution (Eq. apply Hatropolis-Hasting algorithm (Metropolis et al. 1953) in Z. An initial state zu is drawn from an uniform distribution. Given state z, at time t, a candidate for the next state ž,+1 is generated by the proposal distribution. We use a Gaussian distribution which is centered at 2, and
14

---

Under review as a conference paper at ICLR 2022
has the fixed standard deviation of 0.1 as proposal distribution. A proposed sample is accepted with the probability of min(1, exp((E(z)+1))) -(z)))/T) yielding žt+1 = Z1+1- Otherwise, 2++1 Zz. We take the final state of Markov chain generated sample. As stated in the previous paragraph, the final state is rejected z & Th- An accepted sample is mapped to X' by
f(z).
C.3 ENSEMBLE
An ensemble OOD detector built by combining predictions from multiple OOD detectors. First, each OOD detector's detector score standardized separately. For standardization, the mean and the standard deviation of each detector score is computed on the validation in-distribution dataset. The standardization process is required to account the scale difference between OOD detectors. Then, the standardized detector score is averaged to yield the detector score for the ensemble detector.
D IMPLEMENTATION OF OOD DETECTORS
D.1 IMPLEMENTATION OF WEAK AND STRONG DETECTORS
In this section, we describe how OOD detectors are implemented in our experiments.
AE The overall architecture of AE is identical to what is described in Section but with a different dimensionality of the latent space Dz 128. AE is trained for 500 epochs on in-distribution training set using Adam optimizer with the learning rate 1 x 10 10-4 A mini-batch contains 128 samples. model with the smallest reconstruction error on validation split selected.
PXCNN PXCNN is implemented based on an open-sourced code repository and re-trained on CIFAR-10. All the model parameters are set to the default values, i.e., nr resnet=5, nr fi filters=80, nr logisti mi x=10, resnet. nonl i ineerityty 'onccct elu'. An input image is linearly scaled into [-1, and horizontal flipped with the probability of 50%. The model is trained for 200 epochs where mini-batch contains 64 samples. We use Adam optimizer with the learning rate 10-4 which decayed by a multiplicative factor of 0.999995 every iteration.
Glow Glow is implemented based on two repositories and traianed on CIFAR-10. Our Glow model utilizes multi-scale architecture with three levels of the latent representations are present, ie., L=3. Each level contains 32 flow steps. We use 1x1 invertible convolution and ActNorm with the scale of 1.0. The model is trained for 500 epochs with the batch size of 64. Adam optimizer with the learning rate 1 x 10- is utilized. The gradient is clipped so that its norm is no larger than 0.1.
NAE
The pre-trained model is downloaded from the official repository We use Conv32Big ver- sion, which showed better performance CIFAR-10 10 vs SVHN.
OE Among the available pre-trained models in the official repository select the best per- forming version, i.e., ci farrl a conv oe scratch epoch 99 .9.pt. We normalize an input image with means= (0. 4914, 0 48222, 0. 4446) and std= (0.d=00.20 2471, 0.2435, 0. 0.2615), where each component corresponds to RGB, respectively.
CEDA, ACET, GOOD
The pre-trained models are provided by the official repository of GOOD Among multiple versions of GOOD, we use GOODS0, as recommended in the original paper.
httes//gf100.cco//ttterrr/////
15

---

Under review as a conference paper at ICLR 2022
MD We implement MD following the official repository Based the pre-trained ResNet-32 provided by the official repository, we train the weights of each layer where Mahalanobis distance computed. 1,000 SVHN images used to determine such weights. Note that using test OOD dataset during the training is usually not acceptable a typical OOD detection setring.
SSD
We use the official training script to train SSD
CSI
We download the unlabelled multi-class CIFAR-10 model from the official repository g
D.2 DETECTORS NOT INCLUDED IN EXPERIMENTS
We have considered other OOD detectors such as Likelihood Ratio (Ren et al., 2019), Input Com- plexity (Serrà et al. 2020), Deterministic Uncertainty Quantification (DUQ) (Van Amersfoort al., 2020), and Likelihood Regret (Xiao et al. 2020). However, we could not include them in our exper- iments for the following reasons.
Likelihood Ratio While there is the official public repository written in TensorFlow we have failed to reproduce the result PyTorch.
Input Complexity We confirm that the log-density estimates from generative models corre- lated to the bits after compression, but fail to achieve AUC score higher than 0.9 in CIFAR-10 vs SVHN setting. For generative models, PXCNN and Glow tested, and for image compression al- gorithms, PNG, JPEG2000, and FLJIF are tried. Our generative models are implemented as Section C For PNG and JPEG2000, we use OpenCy and for FLIF, we use imageio ffi f12 No combination of generative model and compression algorithm results in AUC higher than 0.9.
DUQ Investigating the official repository we find that different normalization transforms are applied to in-distribution (CIFAR-10) and test OOD dataset (SVHN). CIFAR-10 is normal- ized with parameters mean- (0 4914, 0. 4822, 0. 4465) and std= (0. 2023, 0. 0.1994, 0.2010). whereas SVHN is normalized using mean- (0.5, 0.5, 0.5) and std= (0.5, 0.5, 0)5). Applying different transformations makes classification between CIFAR-10 and SVHN easier. When modify the code that identical transformation is applied to both datasets, AUC score drops below 0.9. Therefore, we do not include DUQ in our experiments.
Likelihood Regret The method is very slow, particularly because does not support batch pro- cessing, i.e., only one sample can be processed at time. We decide that the method is too slow to apply the adversarial distribution approach.
genomics.cooe halp:cssccddmmmage ood
lttps:///penevv.org/
16

---

Under review as a conference paper at ICLR 2022
E EXTENDED EXPERIMENTAL RESULTS
E.1 BINARY CLASSIFICATION
Here, show the effect test-time augmentation and report the zero-inlier threshold. Test-time augmentation dramatically increases the zero-inlier threshold, meaning that the classifiers become more robust.
Binary classifier logit values (No Aug)
CIFAR10
CIFARI0
density
-7.9
SVHN
CelebA
0.2
-12.6
0.2
0.0
0.0
0
20
-20
0 logit
20
logit
Binary classifier logit values (TestTimeAug)
-4.92
CIFAR10 SVHN
CIFAR10
CelebA
density
0.2
0.2
-0.34
0.0
0.0
-10
0
10
20
10
0
10
20
logit
logit
Distribution of the logit values in binary classification. The zero-inlier threshold, the smallest logit of inliers are denoted as dotted lines.
Figure 6:
E.2 RUNTIME AND COMPUTING ENVIRONMENT
shows time required to perform MCMC sampling and for each OOD detectors. For MCMC, time required for running 10,000 steps of MH update is shown. The main factor that determines the runtime is the inference time of an OOD detector. Inference of all OOD detec- tor is performed single Tesla V V00 GPU.
Table 4: Time required for performing single run
of MCMC
Detector
MCMC
Glow
3hr 15min
PXCNN
10min
AE
1hr 10min
NAE
50min
GOOD
22min
ACET
22min
CEDA
22min
SSD
37min
MD
1hr 30min
OE
22min
CSI
20hr
17

---

Under review as a conference paper at ICLR 2022
F
ADDITIONAL SAMPLES FROM ADVERSARIAL DISTRIBUTIONS
From Figure 7 to Figure 12
present samples from adversarial distributions of each OOD detectors.
Glow Dz
SVHN
CelebA
Glow D 32
7
CelebA
Glow Dy 64
CelebA SVHN
PXCNN Dz 16
CelebA SVHN
PXCNN Dz 32
CelebA SVHN
PXCNN Dz 64
CelebA SVHN
Figure 7: Samples from adversarial distributions.
18

---

Under review as a conference paper at ICLR 2022
AE Dz 16
CelebA SVHN
AE Dz 32
CelebA 5VHN
AE Dz 64
CelebA SVHN
NAE Dz 16
CelebA SVHN
8128
10
65
3
3
3
88
8
8
331
NAE Dz 32
3
R
CelebA SVHN
NAE Dz 64
CelebA SVHN
Figure 8: Samples from adversarial distributions. (2/6)
19

---

Under review as a conference paper at ICLR 2022
GOOD Dz = 16
CelebA 5VHN
3
13)
GOOD D 32
CelebA SVHN
2
GOOD D = 64
CelebA SVHN
ACET Dz = 16
CelebA SVHN
38
RX
83
6 7
8
85
8
ACET Dz 32
18
99
88
8
CelebA SVHN
ACET Dz = 64
CelebA SVHN
Figure 9: Samples from adversarial distributions.
20

---

Under review as a conference paper at ICLR 2022
CEDA Dy 16
CelebA SVHN
33
6
6
0
12
CEDA D 32
7AM 1 18.
CelebA SVHN
CEDA D 64
CelebA SVHN
SSD D, 16
CelebA SVHN
8
31
21
3
SSD Dz 32
CelebA SVHN
8
SSD D 64
CelebA SVHN
Figure 10: Samples from adversarial distributions.4II0 (4/6)
21

---

Under review as a conference paper at ICLR 2022
MD Dz 16 5
CelebA SVHN
2
8
3
1
D
2
5
MD Dz 32 32
CelebA SVHN
6
3
S
2
9
3
MD Dz 64
2
CelebA SVHN
OE Dz 16
CelebA SVHN
OE D 32
CelebA SVHN
OE Dz 64
CelebA SVHN
Figure 11: Samples from adversarial distributions. (5/6)
22

---

Under review as a conference paper at ICLR 2022
CSI Dz 16
CelebA SVHN
3
87
6!
68
#
8
8
8
CSI Dz 32
CeiebA SVHN
CSI Dz = 64
CelebA SVHN
Figure 12: Samples from adversarial distributions.(666)
23


Reviews:
[{"review_id": "y6grKoC9IZb", "reviewer": "Reviewer_HyPw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The proposed benchmark is reasonable in a different way, so I recommend to redirect the goal and rewrite the paper.\n\n- To my knowledge, OOD detection is not supposed to be robust against adversarial attacks, though some OOD detection works showed that their method is robust against adversarial attacks, e.g., [Lee et al., 2018]. Rather, the OOD detection task aims at detecting natural but different distributions, while adversarial attacks are mostly artificial. The proposed benchmark evaluates if OOD detection methods are robust against adversarial attacks, so the purpose is different from OOD detection works.\n\n- The Mahalanobis distance method (MD) [Lee et al., 2018] leveraged adversarial attacks for validation in some of their experiments, which is worth to note as a prior work with a similar idea. I think MD should exhibit a good performance, if the authors followed this validation strategy.\n\n## Post rebuttal\n\nI do not change my rating, as I still think the purpose of the proposed benchmark should be described in a different way.\n\nI think the answer in **1. OOD Detection and Adversarial Robustness** cannot justify if evaluating adversarial robustness on OOD samples can be generalized to unexplored space of (natural) outliers. By adding adversarial perturbations to the evaluation framework, the objective becomes to evaluate the adversarial robustness, not the OOD detection performance. To my understanding, related works are about mathematically certifiable or provable adversarial robustness on OOD detection problem, which is not really aligned with this work. As Reviewer XMWK concerned, the authors could clarify the relationship between this work and related works. It is true that there are \"adversarially perturbed outliers which can be confirmed as OOD from human eyes,\" but I think detecting such artificially generated samples is a different problem from the general OOD detection problem.\n\nHere I summarize the logical flow in the abstract below, where the transition from 2 to 3 is not convincing to me.\n1. Current evaluation protocols cover only a small fraction of all possible outliers.\n2. We want to test a detector over a larger, unexplored space of outliers.\n3. To this end, we evaluate with samples from its adversarial distribution.", "summary_of_the_paper": "This paper proposes to evaluate out-of-distribution (OOD) detection methods on adversarial distribution to detect unexplored space of outliers.", "summary_of_the_review": "The proposed benchmark would be worth in a different way: not for a universal OOD detection, but adversarially robust OOD detection.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gembMpJhqBi", "reviewer": "Reviewer_XMWK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper contains some interesting ideas, and in general I like the fact the derivations and experiments are clearly presented.\n\nHowever, a major problem I feel is that it has an unclear relationship with previous work. This is far from the first work that consider constructing a distribution that is \"adversarial\" to the OOD. For example, Hein's group has produced a line of work in this direction, and also recently there have been many other works (e.g., https://arxiv.org/abs/2006.15207). For a big part of the paper, these related works are not discussed at all, and in fact Bitterwolf's work is discussed only until page 7. This does not seem to be the right way of handling related work.\n\nI guess this work may be arguing that, even though the previous works have considered \"adversarially constructed distributions against OOD\", this work is more explicit and systematic about this direction. To that end I remain a bit skeptical. It is not the case that, just by considering larger and larger space, it \"automatically\" makes an attacking-OOD paper more interesting. In fact, a small space of perturbation that leads to failure of OOD is even more interesting, than a very large space (clearly, NN will likely be poorly performing on far-away inputs, given the state-of-the-art technology). \n\nI am also worried about the very strong claims made in the paper, such as \"In this paper, we have addressed the limitations of the current evaluation protocol for OOD detection and proposed a novel framework, adversarial distributions, that can be used to investigate failure\nmodes of OOD detectors.\" Do we really have addressed the problem (so we cannot imagine any further work to improve the evaluation, and so this work has exhausted the entire space?)\n\nNit on the naming. Adversarial search induces an adversarial distribution from the original \"natural distribution\", so I don't really think it is a good name to call \"adversarial search\" vs. \"adversarial distribution\" in the definitions. I don't really have a good suggestion, but how about \"adversarial sampling\"?", "summary_of_the_paper": "This paper studies the problem of constructing adversarial distributions to fool \"naturally trained\" OOD detectors. A optimization-based and a sampling-based method are proposed and evaluated respectively.", "summary_of_the_review": "I am worried about the handling of previous work in this paper, and it seems to me to be overselling its contributions. For these reasons, I would vote for rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gbk29jd6sD4", "reviewer": "Reviewer_GYAe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n- This work studies an important problem with OOD detection, namely that the arbitrary choices of test datasets may lead to biased/overoptimistic estimates of detection quality. The paper is well written and easy to follow. \n- The idea of extending search space of outliers by using a learned autoencoder makes intuitive sense, and the proposed solution appears to address most issues with existing evaluation protocol.\n- Experiments are conducted with a large number of OOD detection methods, with very interesting results revealing the vulnerability of state-of-the-art detectors whose performance saturate on standard benchmarks.\n\n## Cons / Questions\n- As the authors have acknowledged in the paper, the proposed evaluation method does not fully eliminate the need of selecting OOD test sets, which significantly affects evaluation outcome (ranking of OOD detectors). That is, while the autoencoder learns to *interpolate* between outlier samples, it does not *extrapolate* beyond the chosen dataset, and therefore cannot provide full coverage of outlier distribution. Therefore, the results would be stronger if 1) repeated with more pairs of inlier-outlier datasets, and 2) trained on the concatenation of multiple outlier datasets.\n- If the major concern is the coverage of OOD data used in existing evaluation protocols, can using a larger-scale test set (e.g. ImageNet) address the problem?\n- As seen in qualitative examples of Figure 3, 4, 7-12, the majority of samples drawn from the adversarial distribution diverge from the space of natural images (especially when learned on SVHN). This makes the proposed approach closer to a benchmark of adversarial robustness, where images can be manipulated adversarially to fool a target classifier, especially when the latent dimension $D_z$ of the autoencoder is large. Does this make the benchmark unnecessarily difficult, when OOD detectors are expected to differentiate between *real* images? Can the authors further comment on the difference between the proposed task and adversarial robustness? Should one expect training OOD detectors with adversarial defense methods to improve performance on the proposed benchmark?\n\n## Minor comments\n- It would help to highlight best performing OOD detectors under each criterion in table 1.", "summary_of_the_paper": "The paper proposes a novel evaluation framework for out-of-distribution (OOD) detection under worst-case scenarios. While existing benchmarks use real samples from datasets outside the training distribution, the authors propose instead to learn an adversarial outlier distribution against OOD detectors using an autoencoder model, with an auxiliary binary classifier that filters out inlier samples. Empirical experiments on CIFAR-10 (inlier) and SVHN/CelebA (outlier) datasets show that standard OOD benchmarks tend to produce overoptimistic results, and that prior methods with similar scores on standard benchmarks have diverging performance outside the predefined OOD test sets. ", "summary_of_the_review": "This paper presents a very interesting approach towards an objective and unbiased evaluation of OOD detection. While the proposed approach does not fully address the problems of existing evaluation protocols it identifies, I find the solution to be novel and intuitive, and the empirical results to be worthy for the community. Overall I am leaning towards accepting the paper, and would appreciate if the authors can address the questions enumerated in the Cons section above during rebuttal period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bntzjigAAzN", "reviewer": "Reviewer_hrej", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - well written\n - Good job at describing the general problem and setup\n - Reasonable methodology for generating the adversarial distribution\n - Strong evaluation\n\nWeaknesses:\n - I find the use of a normal auto-encoder to be a little strange, as sampling from that latent space has issues [1]. I would much prefer a VAE or GAN that enforces a prior distribution. The authors could comment on these draw-backs in the main text, but I find the example generated images to be convincing of sufficient \"OODness\"\n - The connection between adversarial search and distribution is unclear. The sentence \"sampling from an adversarial\ndistribution becomes equivalent to the adversarial search...\" is somewhat buried in the text. The full discussion of adversarial search/optimization is, in my view, tangential to the main arguments of the paper. The paper's clarity would be greatly enhanced if the term adversarial was used less frequently and the generative aspect of the work was focused on.\n\nOther:\n - Do the authors have a link to source code / website for the proposed leaderboards? If so, they should be linked. If not, leaderboard should be move to discussion section as future work.\n - section 6 & 7 could be combined \n - table 1 bolding is a little non-intuitive (typically reserved for best performing methods), perhaps italicized would convey the same message?\n - How many instances were generated for the adversarial distribution? 1000? This could be clarified in the main text.\n\n\n[1] Bengio et al. \"Representation learning: A review and new perspectives\" 2013\n\nPost rebuttal: I am generally okay with the current paper as is. I would prefer a better generative model and a bit a of a rewrite for less focus on the adversarials; for these reasons, I will be keeping my score.", "summary_of_the_paper": "This paper introduces a new method for measuring an image classifier's robustness against out of distribution data by using adversarial search/distributions. The paper conducts experiments on various recent SOTA OOD models, and finds traditional metrics don't capture the whole picture when it comes to OOD detection.", "summary_of_the_review": "This paper provides an interesting angle for investigating/evaluating OOD detectors. The paper is well-justified, with a mostly reasonable setup, and would be of interest for broader OOD community as a new metric.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the general problem of out-of-distribution (OOD) detection, where the goal is to detect outliers (i.e., points not in the distribution of training data) in the sample. The paper introduces a methodology for measuring robustness by using adversarial search/distributions. Experimental evaluation indicates that traditional metrics fail to fully capture OOD detection. The reviewers' evaluations of this work were mixed. Overall, there was consensus about the importance of the problem. Moreover, some of the reviewers argued that the submission contains some interesting new ideas. On the other hand, concerns were raised regarding lacking comparison to prior work, potential overselling of the contributions, and several aspects of the experimental evaluation. At the end, there was not sufficient support for acceptance. In its current form, the work appears to be slightly below the acceptance threshold.
Decision: Reject