Year: 2021

Abstract:
Out-of-distribution (OOD) detection is the task of determining whether an input lies outside the training data distribution. As an outlier may deviate from the training distribution in unexpected ways, an ideal OOD detector should be able to detect all types of outliers. However, current evaluation protocols test a detector over OOD datasets that cover only a small fraction of all possible outliers, leading to overly optimistic views of OOD detector performance. In this paper, we propose a novel evaluation framework for OOD detection that tests a detector over a larger, unexplored space of outliers. In our framework, a detector is evaluated with samples from its adversarial distribution, which generates diverse outlier samples that are likely to be misclassified as in-distribution by the detector. Using adversarial distributions, we investigate OOD detectors with reported near-perfect performance on standard benchmarks like CIFAR-10 vs SVHN. Our methods discover a wide range of samples that are obviously outlier but recognized as indistribution by the detectors, indicating that current state-of-the-art detectors are not as perfect as they seem on existing benchmarks.

text: Out-of-distribution (OOD) detection is the task of determining whether an input lies outside the training data distribution. As an outlier may deviate from the training distribution in unexpected ways, an ideal OOD detector should be able to detect all types of outliers. However, current evaluation protocols test a detector over OOD datasets that cover only a small fraction of all possible outliers, leading to overly optimistic views of OOD detector performance. In this paper, we propose a novel evaluation framework for OOD detection that tests a detector over a larger, unexplored space of outliers. In our framework, a detector is evaluated with samples from its adversarial distribution, which generates diverse outlier samples that are likely to be misclassified as in-distribution by the detector. Using adversarial distributions, we investigate OOD detectors with reported near-perfect performance on standard benchmarks like CIFAR-10 vs SVHN. Our methods discover a wide range of samples that are obviously outlier but recognized as indistribution by the detectors, indicating that current state-of-the-art detectors are not as perfect as they seem on existing benchmarks.

heading: 1 INTRODUCTION
text: Identifying whether an input datum lies outside the training data distribution is one of the canonical problems in machine learning. Over its long history, the problem has been called by multiple names, including novelty detection (Markou & Singh, 2003), outlier detection (Hawkins, 1980), oneclass classification (Japkowicz et al., 1995), and more recently, out-of-distribution (OOD) detection (Hendrycks & Gimpel, 2016). Investigation of the problem has resulted in a number of real-world applications, for example, medical diagnosis (Li et al., 2019) and inspection of defective parts and products (Bergmann et al., 2019). The relevance of OOD detection is growing beyond these applications, as an OOD detector is considered an essential component of a trustworthy machine learning system. For example, without a reliable OOD detector, an image classifier trained to classify cats and dogs may incorrectly classify a human as belonging to one of these two classes (Hendrycks et al., 2019b). In order to advance the development of reliable OOD detection algorithms, a more comprehensive evaluation protocol is needed.
Current evaluation protocols adopted by the community provide a distorted view of a detector’s performance for two reasons. First, OOD detectors are tested over a small fraction of possible outliers. Detectors are typically evaluated using test OOD datasets chosen by a researcher. Since the test OOD datasets do not cover the entire space of outliers, there may exist untested outliers that the detector fails to classify correctly, even though the detector perfectly detects the chosen test OOD points as shown in Figure 1. To assess a detector’s performance in a more comprehensive and systematic way, a method is needed to test a detector over a larger, unexplored space beyond what is covered by the test OOD datasets.
Second, the current evaluation protocol neglects the worst-case behavior of an OOD detector. The average performance metric is often not sufficient to build trust on a detector, because in safetycritical applications even a single mistake can result in fatal consequences. A detector should be tested adversarially, through an active search for its worst-case failure mode, i.e., an outlier that is classified with maximal confidence as an inlier. Such failure cases may reveal weaknesses of the tested detector, and provide informative clues for building a better detection algorithm.
In this paper, we propose a novel evaluation protocol of OOD detectors that addresses the above-mentioned limitations of current evaluation methods. We first formulate the notion of adversarial search against an OOD detector, a search problem of finding an outlier that a detector classifies as an inlier with the greatest confidence. A detector may have more than one significant failure mode, and it is more informative to find a set of diverse failure cases instead of the most critical one. To that end, we propose the adversarial distribution against an OOD detector, which generates outlier samples that are likely to be misclassified by the given detector. Measuring OOD detection performance against samples from a detector’s adversarial distribution gives an accurate and finer-grained assessment on its performance.
To ensure that samples from an adversarial distribution are indeed outliers, the distribution needs to be supported on a zero-inlier space, a set without any overlap to the inlier distribution. Meanwhile, the zero-inlier space should
be large so that we may observe meaningful failure mode of OOD detectors within the space. However, finding such a space is generally challenging, as the true boundary between inliers and outliers is unknown. We circumvent this challenge by building a generative model over known outliers. Our construction of zero-inlier spaces contain diverse samples.
We implement 11 previously proposed OOD detectors and investigate their behavior using their adversarial distributions. Among the tested detectors, 8 detectors report near-perfect OOD detection performance on a popular benchmark, CIFAR-10 (in) vs SVHN (out), effectively being indistinguishable with respect to their performance. Our investigation reveals that the 8 detectors in fact have diverging degrees of detection quality outside the SVHN test set. Our methods also lead to several interesting insights that suggest techniques for improving OOD detection.
Our main contributions can be summarized as follows:
• We propose the adversarial search and adversarial distributions, which can be used to evaluate OOD detection algorithms beyond a pre-defined test OOD dataset; • We provide practical techniques to define the space of outliers that contains samples outside the test OOD dataset; • By examining the state-of-the-art OOD detectors, we show that OOD detectors with seemingly equivalent performance differ significantly in their failure modes;
Related Work Testing the worst-case performance of an algorithm is highly related to evaluating adversarial robustness of the detector. Detection of adversarially perturbed outliers is investigated in previous literature where perturbation is assumed to be restricted in a small norm-ball Hein et al. (2019); Meinke & Hein (2020); Bitterwolf et al. (2020). The idea of using an autoencoder during an adversarial attack is investigated in Tu et al. (2019). The idea of generating outliers are investigated in the context of improving OOD detection Chen et al. (2020) or adversarial attack Song et al. (2018).
In Section 2, we provide essential preliminaries. Section 3 formulates adversarial search and adversarial distributions, and Section 4 introduces techniques to construct a zero-inlier space. Our main experimental results are presented in Section 5, with deeper discussions provided in Section 6. Section 7 concludes the paper.

heading: 2 BACKGROUND: OUT-OF-DISTRIBUTION DETECTION

heading: 2.1 DEFINITION
text: We consider a probability distribution of interest Pin, samples from which we consider as inliers. Each sample is represented as a D-dimensional real-valued vector. The probability density function of Pin is denoted as pin(x). We write the support of Pin as Sin = {x|pin(x) > 0} ⊂ X ⊂ RD,
where X is the data space, the set of all possible values for data, which we assume to be compact. Then, we define OOD-ness as follows:
Definition 1. A vector x is out-of-distribution (OOD), or an outlier, with respect to Pin if x does not belong to the support of Pin, i.e., x 6∈ Sin. Conversely, x is in-distribution when x ∈ Sin. A distribution Q having the support SQ is OOD to Pin, when SQ ∩ Sin = ∅.
Another popular definition of OOD-ness characterizes a vector x as OOD when the vector belongs to a density sub-level set, x ∈ {x|pin(x) ≤ η} (Steinwart et al., 2005). However, the density sub-level set does not provide a consistent characterization of the OOD-ness. A vector classified as OOD in one coordinate may not be classified as OOD in a different coordinate, because a probability density function can be arbitrarily distorted via an invertible coordinate transform as pointed out in (Lan & Dinh, 2020). On the contrary, our definition of OOD using the density support provides an invariant characterization of outliers with respect to such transforms.
An OOD detector f(x) : RD → R is a function which outputs a larger value for an input more likely to be an outlier. A test vector x∗ is classified as OOD f(x∗) > ηf for the threshold ηf . A detector score refers to the function value of f(x). We shall assume f(x) is bounded. In this paper, we consider a black-box setting, where any information other than the function value of f(x), such as its gradient, is not accessible. An OOD detector is normally trained using an in-distribution datasetDin, a set of iid samples from Pin. However, some OOD detectors utilize additional datasets other than Din.

heading: 2.2 EVALUATION OF OOD DETECTORS
text: The currently accepted evaluation protocol for OOD detectors relies on one or multiple test OOD datasets Dout, which contain a finite number of samples considered as OOD with respect to Pin by human prior knowledge. When Pin is a distribution of images, test OOD sets are often chosen from separately published image datasets of which contents are different from that of Pin. For example, when an image dataset of animals is selected as in-distribution, a set of digit images can be used as a test OOD dataset.
Given a test OOD dataset, an OOD detector f(x) performs the binary classification against indistribution dataset, and the quality of the classification is considered as an indicator for how good f(x) is as an OOD detector. The classification result is summarized using metrics such as the area under the receiver operating characteristic curve (AUROC or AUC). AUC is a preferred metric in a number of literature, as it does not require the specification of the threshold ηf . AUC score of 1.0 indicates the perfect classification, and AUC of 0.5 implies the random guess.
The research community has been focusing on a few representative in-distribution and OOD dataset pairs, such as CIFAR-10 (in) vs SVHN (out) and Fashion-MNIST (in) vs MNIST (out). These dataset pairs become popular after the reports showing that OOD detectors built upon deep generative models, such as PixelCNN++ (Salimans et al., 2017) or Glow (Kingma & Dhariwal, 2018), fail to detect outliers (Hendrycks et al., 2019a; Nalisnick et al., 2019). In fact, the generative-modelbased detectors score AUC lower than 0.5, meaning that SVHN images are more strongly perceived as CIFAR-10 than the actual CIFAR-10 images by the detectors. This observation spurred intense research efforts, and now there are multiple OOD detectors achieving AUC scores higher than 0.9 or even higher than 0.99 on CIFAR-10 vs SVHN as listed in Section 5. Given the near-perfect detection score, we question whether the detectors are indeed good OOD detectors beyond the tested examples.

heading: 3 ADVERSARIAL DISTRIBUTIONS
text: Our goal is to test how robust the state-of-the-art OOD detectors are. The robustness of an OOD detector can be investigated through a search process to find its failure modes. We are particularly interested in failure cases where outlier examples misclassified as in-distribution by the detector. Ideally, we would search for failure modes over all possible outliers, but the true set of outliers Sout ≡ X − Sin is unknown. Instead, we specify a search space T which is a subset of Sout and only contains outliers. We call T a zero-inlier space. In this section, we formulate two search problems on T . The first formulation, Adversarial Search, is based on optimization, and the second
formulation, Adversarial Distribution, is based on sampling. Details on selecting meaningful T will be discussed in Section 4.

heading: 3.1 ADVERSARIAL SEARCH
text: Finding failure modes of an OOD detector can be formulated as an optimization problem. For a detector f(x) and a search space T , an adversarial search problem is finding an element x∗ in T which has the lowest detector score.
x∗ = arg min x∈T f(x) such that x ∈ T ⊂ Sout, (1)
where x∗ is the worst-case outlier we search for. If the detector score for the worst-case outlier f(x∗) is greater than the score of any samples in in-distribution set, then the detector f(x) can be confirmed as robust in the search space T . Otherwise, we conclude that the tested detector has at least one failure mode and deviates from the ideal detector. The degree of deviation may be used to quantify the robustness of a detector.
Example: Adversarial Attack An example of the adversarial search is applying an adversarial perturbation on outliers so that they have small f(x) (Bitterwolf et al., 2020). In this case, T is a ball centered at an outlier from a test OOD dataset T = {x| ‖x− x̃‖ ≤ r, x̃ ∈ Dout}. A point with the smallest detector score is found via a gradient descent method. The AUC score for classifying the perturbed samples from inliers is called Adversarial AUC (Bitterwolf et al., 2020).

heading: 3.2 ADVERSARIAL DISTRIBUTIONS
text: When the search space T is larger than a ball centered at an outlier, f(x) may have multiple local minima, and each minimum may reveal different failure modes of a detector. To explore multiple failure modes in f(x), we propose a novel approach. We deliberately design a probability distribution pf (x) so that samples from pf (x) cover multiple minima of f(x) in T . We call pf (x) an Adversarial Distribution of a detector f(x).
An adversarial distribution is a Gibbs distribution, also known as an energy-based model. Its energy function is the detector score f(x), i.e., pf (x) is proportional to exp(−f(x)/T ). The support of pf (x) is the search space T .
pf (x) = 1
Z exp(−f(x)/T ) if x ∈ T ⊂ Sout and pf (x) = 0 otherwise, (2) where Z = ∫ T exp(−f(x)/T )dx is the normalization constant, and T > 0 is called temperature. Z <∞, as we have assumed the bounded f(x) and the compact data space in Section 2. An adversarial distribution pf (x) has three important properties by construction. First, samples from pf (x) are guaranteed to be OOD, as the support is T . Second, an adversarial distribution assigns a high probability density on samples likely to be classified as in-distribution by the detector, i.e., pf (x) is high when f(x) is small. Third, for the limit T → 0, sampling from an adversarial distribution becomes equivalent to the adversarial search, as the probability mass is concentrated on x∗ in Eq. (1). Hence, an adversarial distribution can be viewed as a relaxation of the adversarial search, and T governs the degree of relaxation.
Samples from pf (x) can be drawn using Markov Chain Monte Carlo (MCMC). MCMC is guaranteed to visit all modes of a probability distribution at least in theory, and therefore every failure mode of a detector can be generated. Measuring how well f(x) classifies inliers from pf (x)’s samples using metrics such as AUC gives a comprehensive measure on the performance of the detector.

heading: 4 ZERO-INLIER SPACE
text: The support T of an adversarial distribution defines the search space where the robustness of an OOD detector is evaluated. While there is some degree of freedom in the choice of T , one condition should be strictly satisfied: T should not contain inlier data points, i.e., should be a zero-inlier space. However, as the true decision boundary between outliers and inliers is unknown, it is challenging to ensure T only contains outliers.
In the example of the adversarial attack provided in Section 3.1, T is set as a ball around a known outlier. T is ensured to be a zero-inlier space as the radius of the ball is very small. However, such a small ball only contains examples visually indistinguishable from its center, lacking diversity. In what follows, we provide an example construction of T which contains a more diverse set of outliers. Here, we describe a procedure for constructing a zero-inlier search space T given a set of indistribution dataset Din and OOD dataset Dout. Note that T is constructed during the evaluation stage of OOD detectors and does not affect an OOD detector being tested. Also, the construction of T requires no additional data compared to a typical evaluation protocol which assumes a humancurated OOD dataset.
Instead of a ball around each outlier, we consider a generator function g(z) spanning a set of outlier samples Dout. Such a generator is able to produce an unseen sample which resembles the known outliers. As the generator is trained only using Dout, it is unlikely for an inlier to be generated from g(z). Nevertheless, to avoid a fortuitous generation of an inlier, we introduce a binary classifier h(x) which is supervised to discriminate Din and Dout. Let Tg be a set of samples generated by g(z), and Th be a set of samples classified as they are from Dout. Our search space T is given as an intersection of Tg and Th:
T = Tg ∩ Th. (3)

heading: 4.1 OUTLIER-SPANNING GENERATOR
text: A generator g(z) : Z → X is a map from a lower-dimensional space Z ⊂ RDZ to the data space X , where DZ smaller than the dimensionality of X . We denote Tg as a set of samples that can be produced by g(z):
Tg = {x = g(z)|z ∈ Z}. (4)
In our implementation of adversarial distributions, we train an autoencoder to reconstruct Dout and use its decoder as g(z). As the autoencoder can reconstruct samples in Dout (with some error), the generator (approximately) spans the space where outliers reside. Similarly, a generator of a generative adversarial network can also serve as g(z).
It is reported that an autoencoder may reconstruct an input vector that is significantly different from its training data even though the autoencoder is not the identity mapping (Tong et al., 2019; Gong et al., 2019; Yoon et al., 2021). This phenomenon implies that g(z) may be capable of generating samples that is significantly different from Dout. The ability of g(z) to reach outside of Dout may positively affect the diversity of Tg , but increases the chance of generating x that is close to inliers. To ensure that Tg only contains outliers, we introduce an additional component can filter out accidentally generated inliers.

heading: 4.2 BINARY CLASSIFIER SEPARATING INLIERS AND OUTLIERS
text: Here, we develop an additional mechanism for ensuring that no inlier is present in our search space by exploiting the fact that there is no overlap of the supports between an inlier distribution, and a distribution generating an OOD dataset. When two distributions do not overlap, there exists a binary classifier that perfectly classifies the two distributions. Proposition 1 (Binary classification of disjoint distributions). Suppose two probability distributions P and Q with their supports SP and SQ, respectively. P and Q are OOD to each other, if and only
if there exists a binary classifier h∗(x) : X → R that perfectly discriminates P and Q, i.e., there exists a threshold ηh∗ ∈ R such that the classification boundary h∗(x) = ηh∗ separates SP and SQ.
More formal arguments will be provided in Appendix A. Assume an oracle binary classifier h∗(x) which perfectly discriminatingDin andDout. From Proposition 1, the oracle classifier partitions the whole space into two, where one partition contains only Din and the other contains only Dout. The partition containing Dout can be chosen as a zero-inlier space. Proposition 1 holds approximately for an empirically obtained classifier h(x). A binary classifier often achieves AUC score higher than 0.9999 when evaluated on the test splits of Din and Dout. Hence, we define an outlier-side of h(x)’s decision boundary as Th. Assuming that h(x) assigns lower value for outliers,
Th = {x|h(x) < ηh, x ∈ X}, (5)
for a threshold ηh. However, a real-world classifier is not an oracle h(x) 6= h∗(x) and can never be perfect. The classifier may misclassify or may be vulnerable to adversarial perturbations. To minimize the effect, we apply a few techniques. First, we set the threshold ηh lower than a zeroinlier threshold, which is the smallest h(x) value among all training, validation, and testing inlier examples. Second, test-time augmentation is applied on h(x) to make the prediction more robust. We use horizontal flip and rotations of 90, 180, 270 degrees. Total 5 predictions are averaged. Detailed discussion on the robustness of h(x) is given in Section E.1.

heading: 5 EXPERIMENTS
text: In our experiments, we aim to reveal previously unknown failure modes of the state-of-the-art OOD detectors using adversarial distributions. We first show that adversarial distributions can find the known weaknesses of baseline detectors. Then, we apply our methods on the state-of-the-art OOD detectors which show near-perfect performance on a popular benchmark. Finally, we show that the failure modes of the detectors are not readily transferable. However, a simple ensemble does not improve the robustness of OOD detection significantly. Table 1 summarizes our experimental results.

heading: 5.1 EXPERIMENTAL SETTINGS
text: Datasets We use CIFAR-10 as in-distribution data Din. SVHN and CelebA are used as test OOD datasets Dout. All datasets are splitted into training, validation, and testing sets. All data are 32×32 RGB images. Details on datasets can be found in Appendix B.
OOD Detectors We implement 11 previously proposed outlier detectors. The detectors are grouped into two, the weak and the strong, based on their performance on CIFAR-10 vs SVHN. The weak detectors are outlier detectors that fail dramatically on the benchmark by predicting images in SVHN being more likely to be in-distribution than images in CIFAR-10.
The following is the list of the weak detectors used in our experiment.
• Autoencoder (AE) (Rumelhart et al., 1986): A neural network trained to reconstruct its input. The detector score f(x) is the reconstruction error of x (Japkowicz et al., 1995). • PixelCNN++ (PXCNN) (Salimans et al., 2017): An autoregressive generative model for images. • Glow (Kingma & Dhariwal, 2018): A generative model based on a normalizing flow.
For PXCNN and Glow, the negative log-likelihood is used as the detector score.
We select strong detectors satisfying three criteria: 1) AUC score on CIFAR-10 vs SVHN is higher than 0.9. 2) The code is publicly available and written in PyTorch. The language constraint is introduced for the unified experiment. 3) The performance claimed in the original work should be reproducible.
• Normalized AE (NAE) (Yoon et al., 2021): AE with an additional mechanism for suppressing the reconstruction of outliers. As in AE, f(x) is the reconstruction error. Only trained with in-distribution data.

heading: Weak Detectors

heading: Strong Detectors

heading: Ensemble
text: • Outlier Exposure (OE) (Hendrycks et al., 2019a): The predictive confidence of a classifier on an external dataset is minimized. • Confidence Enhancing Data Augmentation (CEDA) (Hein et al., 2019): Similarly to OE, the predictive confidence of a classifier on an external dataset is minimized. • Adversarial Confidence Enhancing Training (ACET) (Hein et al., 2019): The worst-case confidence of a classifier on an external dataset is minimized. • Guaranteed OOD Detector (GOOD) (Bitterwolf et al., 2020): IBP (Gowal et al., 2018) is used to obtain the confidence interval in the neighborhood of OOD data points, and the interval is minimized during training. • Mahalnobis Distance (MD) (Lee et al., 2018): Negative minimum Mahalanobis distance to indistribution data in the hidden representation spaces is used as f(x). Mahalanobis distances are separately defined for each class and the weight is tuned with a few out-of-distribution data. • Self-Supervised Detector (SSD) (Sehwag et al., 2021): f(x) is computed based on Mahalanobis distance on the representation space which is learned via self-supervised learning. • CSI (Tack et al., 2020): Cosine similarity to the nearest training data in the representation space learned via self-supervised learning is used as f(x).
For OE, CEDA, ACET, and GOOD, the negative maximum softmax probability of a classifier is used as f(x). We utilize the publicly available pretrained models except for MD and SSD where we use the publicly available training scripts. There are a number of other competitive OOD detectors we considered but not included in the experiment of this paper. See Appendix D.2 for discussion on such detectors. We will continuously update our leaderboard afterward to include other OOD detection methods.

heading: 5.2 IMPLEMENTATION OF ADVERSARIAL DISTRIBUTIONS
text: Generator g(x) We use the decoder of an convolutional AE with the latent space Z = [−1, 1]DZ where DZ = {16, 32, 64}. The autoencoder is trained to reconstruct samples in Dout. Binary Classifier h(x) Our binary classifier h(x) is based on ResNet-50 pre-trained on ImageNet. The last layer is replaced with a newly initialized binary classification layer, and then the whole
network is fine-tuned to classify Din from Dout. We use the logit value as h(x). As a zero-inlier threshold, we set ηh = −5 for SVHN and ηh = −1 for CelebA. Adversarial Distribution pf (x) We apply standardization on f(x). The mean and the standard deviation of detector score is computed on the validation split of in-distribution data. For all experiments, the temperature is set T = 0.1.
Gibbs sampling is performed on Z to generate samples from pf (x). At each step, a dimension of z is randomly selected, and a proposal is generated by a unit Gaussian distribution. A proposal outside the domain is projected back. A proposal is accepted based on Metropolis-Hastings criterion. We run 1,000 independent Markov chains with each chain runs for 10,000 steps. The final states of Markov chains are accepted as samples. Samples that do not belong to Th are rejected. More implementation details can be found in Appendix C.

heading: 5.3 ADVERSARIAL DISTRIBUTIONS AGAINST DETECTORS WITH KNOWN WEAKNESSES
text: We confirm the effectiveness of adversarial distributions by applying them on the weak detectors, Glow, PXCNN, and AE, where the weaknesses have already been analyzed. The likelihoods of Glow and PXCNN become spuriously high for images with low complexity (Serrà et al., 2020). Similarly, AE produces also low reconstruction error simple images (Yoon et al., 2021). From Figure 3, the samples from adversarial distributions pf (x) are consistent with the previously known tendency. The samples are very blurry and a large portion of pixels are monotone. Simplicity of the samples are more clear when they are compared to the samples from a strong detector’s pf (x) in Figure 4.

heading: 5.4 EVALUATING STATE-OF-THE-ART DETECTORS
text: The proposed methods reveals that 8 strong detectors, with almost equivalent and near-perfect performance on CIFAR-10 vs SVHN, are actually very vulnerable outside of the test points. All the strong detectors show significantly decreased AUC against samples from adversarial distributions. Among the tested detectors, CSI is the most robust when SVHN isDout. However, CSI does not perform well when outliers and adversarial distributions are based on CelebA. All other detectors show a similar degree of performance drop against samples from their adversarial distributions pf (x) visualized in Figure 4. Note that most of samples from pf (x) retain the visual features of the OOD dataset on which pf (x) based. Also, the samples are visually dissimilar to CIFAR-10.
In the range from 16 to 64, the choice of latent space dimensionalityDZ does not significantly affect the efficacy of adversarial distributions in terms of AUC scores. Using smallerDZ makes the search space narrower, but MCMC can be performed more effectively. Meanwhile, largerDZ increases the diversity of samples, but MCMC becomes challenging.
Transferability and Detector Ensemble We question whether an adversarial distribution is transferable, i.e., given two detectors f1(x) and f2(x), are samples from pf1(x) able to deceive another detector f2(x)? We collect samples from pf1(x) then check if f2(x) can detect them as outliers. The pairwise results are displayed in Figure 5, showing that there are varying degree of transferability. Adversarial distributions are largely transferable between GOOD, CEDA, and ACET, possibly because their constructions are significantly similar (Hein et al., 2019). Except for the three, the other detectors are generally capable of detecting another detector’s adversarial distribution samples. It is interesting to see that NAE also performs well on detecting adversarial distribution samples from other detectors, while NAE is not the best performing detector. We suspect the limited transferability is originated from the distinct inductive biases of detectors. For example, NAE is the only autoencoder-based algorithm, and SSD and CSI are trained via different self-supervised learning processes.
When detectors do not share failure modes, a more robust detector may be formed through averaging their (normalized) detection scores. We construct two ensembles of three OOD detectors, one with highly correlated failure modes (GOOD+ACET+CEDA) and the other with less correlated failure modes (MD+NAE+OE), and measure their robustness. From the last two rows in Table 1, both ensembles barely improve the robustness of OOD detector compared to the individual detectors.

heading: 6 DISCUSSION
text: Extensions The outlier space utilized in the analysis can be expanded by augmenting the test OOD dataset with an additional OOD dataset. Moreover, there is significant flexibility in the design of g(x). In a specific application, domain knowledge can be embedded in g(x) to provide a more meaningful outlier space.
Limitations Our methods still rely on test OOD datasets to obtain g(x) and h(x), and the choice of the datasets need human design. Furthermore, operating in the reduced outlier space, our methods can not confirm that a detector is optimal. However, these limitations are also present in the current evaluation protocol as well.
Future Work To facilitate the development of OOD detection methods, we intend to publish an online software suite for evaluating OOD detectors based on adversarial distributions. We also plan to publish an online leaderboard displaying the results from an adversarial distribution analysis, and to update the leaderboard with the latest OOD detection methods.

heading: 7 CONCLUSION
text: In this paper, we have addressed the limitations of the current evaluation protocol for OOD detection and proposed a novel framework, adversarial distributions, that can be used to investigate failure modes of OOD detectors. The proposed framework posing new challenges in OOD detection by discovering unexplored weaknesses in the existing OOD detectors, We expect our framework to stimulate the advance of trustworthy machine learning, where a model needs to be evaluated beyond the fixed test dataset.

heading: ETHICS STATEMENT
text: Our main ethical concern is that a subset of OOD detectors used in our experiment, OE, CEDA, ACET and GOOD, are trained using 80 Million Tiny Images dataset (Torralba et al., 2008), which is retracted by authors over ethical concerns. While we were aware of the issue of the dataset, the use of models trained on the dataset was inevitable because of the reproducibility. We have tried to train OOD detectors using alternative datasets but failed to reach the performance of detectors originally trained on 80 Million Tiny Images. To minimize the effect of the retracted dataset, the dataset was never used directly. We only used the publicly available model checkpoints, and did not download or access to a copy of dataset.

heading: REPRODUCIBILITY STATEMENT
text: As mentioned in intro, we will provide the software suite to evaluate the weaknesses of the OOD detector. Since our algorithm consists of widely used techniques such as autoencoders, image classifiers, and MCMC sampling, there will be no problem with reproduction.

heading: APPENDIX
text: Appendix is organized as follows:
• Section A: Proof of Proposition 1 • Section B: Datasets • Section C: Implementation of Adversarial Distributions • Section D: Implementation of OOD Detectors • Section E: Extended Experimental Results • Section F: Additional Samples from Adversarial Distributions

heading: A PROOF OF PROPOSITION 1
text: Proposition 1 (Restated). Suppose two probability distributions P and Q with their supports SP and SQ, respectively. P and Q are OOD to each other, if and only if there exists a binary classifier h(x) : X → R that perfectly discriminates P and Q, i.e., there exists a threshold ηh ∈ R such that the classification boundary h(x) = ηh separates SP and SQ.
Proof. If P and Q are OOD to each other, then SP ∩ SQ = ∅. Then, we can construct a classifier h(x) such that h(x) > ηh for x ∈ SP and h(x) ≤ ηh for x ∈ SQ for any constant ηh ∈ R. Therefore, h(x) = ηh separates SP and SQ. Given a classifier h(x) such that h(x) > ηh for x ∈ SP , suppose that P and Q are not OOD to each other. Then, there exists x∗ such that x∗ ∈ SP and x∗ ∈ SQ at the same time. However, h(x∗) can not be h(x) > ηh and h(x) ≤ ηh at the same time. Due to the contradiction, given a perfectly classifier, P and Q should be OOD to each other.

heading: B DATASETS
text: In our experiments, we use CIFAR-10, SVHN, and CelebA to demonstrate the proposed methods and to evaluate OOD detection algorithms, and their details are described in Table 2. In all cases, we use the official test split as our test datasets. For CIFAR-10 and SVHN, the official train split is randomly splitted into train (90%) and valid split (10%). For CelebA, we use the official train-valid split as it is given. Each CelebA image is center-cropped into a 140×140 image and scaled to 32×32 using the bilinear interpolation.
No augmentation is applied to test data. However, random horizontal flip with probability 50% and uniform dequantization (Theis et al., 2015) are applied to training images in the training of the binary classifiers and the autoencoders in the adversarial search and adversarial distributions.
C IMPLEMENTATION OF ADVERSARIAL DISTRIBUTIONS
C.1 IMPLEMENTATION OF BINARY CLASSIFIERS AND AUTOENCODERS
Binary Classifiers We train binary classifiers h(x) that classify Din from Dout. Two binary classifier are trained with different Dout’s. For one classifier, Dout is SVHN, and for the other, Dout is CelebA. Din is CIFAR-10 for both classifiers. Both classifiers have an identical architecture. We replace the last layer of ResNet-50 pre-trained on ImageNet with a newly initialized binary classification layer and fine-tune the whole network using binary cross entropy loss. The pre-trained model is provided by TorchVision. Training is performed for 100 epochs using Adam with the learning rate
of 1× 10−5, where an epoch is defined with respect to the smaller of Din or Dout. Each mini-batch consists of 128 samples from each of Din and Dout. During training, validation loss is recorded and model weights with the best validation loss are used in our experiments.
Autoencoders The decoder of a convolutional autoencoder is used as the generator g(z). The architecture of the autoencoder is similar to what is used in (Ghosh et al., 2020) and is described in Table 3. The latent space of this autoencoder is given as Z = [−1, 1]Dz where Dz = {16, 32, 64}.
We train the autoencoder with samples in the union ofDin∪Dout, whereDout is SVHN or CelebA. Autoencoders are trained for 500 epochs using Adam with the learning rate of 1× 10−4.
C.2 IMPLEMENTATION OF ADVERSARIAL DISTRIBUTIONS
Here, we describe the method for sampling from the practical version of adversarial distribution (Eq. (2)).
Enforcing Support Constraint In Eq. (2), the constraint z ∈ Th needs to be enforced, while most MCMC algorithms for real-valued states assume an unbounded space. We enforce the constraint by two-staged approach.
First, instead of the original adversarial distribution in Eq. (2), we run MCMC for an alternative distribution p′f (z; g, h), where two barrier functions, H1(z) and H2(z), are added to the energy function.
p′f (z; g, h) = 1
Z ′(f, g, h) exp(−E(z)/T ), z ∈ RDZ , (6)
E(z) = f(g(z)) +H1(z) +H2(z), (7) H1(z) = max(0, h(f(z))− ηh), (8)
H2(z) = 10 DZ∑ d=1 max(0, |z(d)| − 1)2, (9)
where z(d) is d-th element of a vector z, and Z ′(f, g, h) = ∫
exp(−E(z)/T )dz. H1(z) drives a Markov chain into {z|f(z) ≤ ηh} by assigning higher energy on z that violates the constraint. Similarly, H2(z) assigns higher energy for z outside [−1, 1]DZ . Due to H1(z) and H2(z), samples that comply the constraint z ∈ Th appear more frequently in MCMC. Note that H1(z) and H2(z) have non-zero only when the constraint is violated, and the E(z) = f(g(z)) for z ∈ Th. Second, after running MCMC, we reject any sample that does not obey the constraint z ∈ Th. This rejection process ensures that the generated samples are from the original adversarial distribution Eq. (2).
Metropolis-Hasting Acceptance To generate samples from an adversarial distribution (Eq. (2)), we apply Metropolis-Hastings algorithm (Metropolis et al., 1953) in Z . An initial state z0 is drawn from an uniform distribution. Given a state zt at time t, a candidate for the next state z̄t+1 is generated by the proposal distribution. We use a Gaussian distribution which is centered at zt and
has the fixed standard deviation of 0.1 as our proposal distribution. A proposed sample is accepted with the probability of min{1, exp((E(z̄t+1) − E(zt))/T )}, yielding zt+1 = z̄t+1. Otherwise, zt+1 = zt. We take the final state of a Markov chain as a generated sample. As stated in the previous paragraph, the final state is rejected if z 6∈ Th. An accepted sample is mapped to X by x = f(z).

heading: C.3 ENSEMBLE
text: An ensemble OOD detector is built by combining predictions from multiple OOD detectors. First, each OOD detector’s detector score is standardized separately. For standardization, the mean and the standard deviation of each detector score is computed on the validation in-distribution dataset. The standardization process is required to account the scale difference between OOD detectors. Then, the standardized detector score is averaged to yield the detector score for the ensemble detector.
D IMPLEMENTATION OF OOD DETECTORS
D.1 IMPLEMENTATION OF WEAK AND STRONG DETECTORS
In this section, we describe how OOD detectors are implemented in our experiments.
AE The overall architecture of AE is identical to what is described in Section C but with a different dimensionality of the latent space DZ = 128. AE is trained for 500 epochs on in-distribution training set using Adam optimizer with the learning rate 1 × 10−4. A mini-batch contains 128 samples. A model with the smallest reconstruction error on validation split is selected.
PXCNN PXCNN is implemented based on an open-sourced code repository1 and re-trained on CIFAR-10. All the model parameters are set to the default values, i.e., nr resnet=5, nr filters=80, nr logistic mix=10, resnet nonlinearity=‘concat elu’. An input image is linearly scaled into [−1, 1], and horizontal flipped with the probability of 50%. The model is trained for 200 epochs where a mini-batch contains 64 samples. We use Adam optimizer with the learning rate 1 × 10−4 which is decayed by a multiplicative factor of 0.999995 every iteration.
Glow Glow is implemented based on two repositories2 3 and traianed on CIFAR-10. Our Glow model utilizes a multi-scale architecture with three levels of the latent representations are present, i.e., L=3. Each level contains 32 flow steps. We use 1×1 invertible convolution and ActNorm with the scale of 1.0. The model is trained for 500 epochs with the batch size of 64. Adam optimizer with the learning rate 1×10−4 is utilized. The gradient is clipped so that its norm is no larger than 0.1.
NAE The pre-trained model is downloaded from the official repository4. We use Conv32Big version, which showed better performance on CIFAR-10 vs SVHN.
OE Among the available pre-trained models in the official repository5, we select the best performing version, i.e., cifar10 allconv oe scratch epoch 99.pt. We normalize an input image with means=(0.4914, 0.4822, 0.4465) and std=(0.2471, 0.2435, 0.2615), where each component corresponds to RGB, respectively.
CEDA, ACET, GOOD The pre-trained models are provided by the official repository of GOOD6. Among multiple versions of GOOD, we use GOOD80, as recommended in the original paper.
1https://github.com/pclucas14/pixel-cnn-pp 2https://github.com/chaiyujin/glow-pytorch 3https://github.com/chrischute/glow 4https://github.com/swyoon/normalized-autoencoders 5https://github.com/hendrycks/outlier-exposure 6https://gitlab.com/Bitterwolf/GOOD
MD We implement MD following the official repository7. Based on the pre-trained ResNet-32 provided by the official repository, we train the weights of each layer where Mahalanobis distance is computed. 1,000 SVHN images are used to determine such weights. Note that using test OOD dataset during the training is usually not acceptable in a typical OOD detection setting.
SSD We use the official training script to train SSD 8.
CSI We download the unlabelled multi-class CIFAR-10 model from the official repository9.

heading: D.2 DETECTORS NOT INCLUDED IN EXPERIMENTS
text: We have considered other OOD detectors such as Likelihood Ratio (Ren et al., 2019), Input Complexity (Serrà et al., 2020), Deterministic Uncertainty Quantification (DUQ) (Van Amersfoort et al., 2020), and Likelihood Regret (Xiao et al., 2020). However, we could not include them in our experiments for the following reasons.
Likelihood Ratio While there is the official public repository written in TensorFlow10, we have failed to reproduce the result in PyTorch.
Input Complexity We confirm that the log-density estimates from generative models are correlated to the bits after compression, but fail to achieve AUC score higher than 0.9 in CIFAR-10 vs SVHN setting. For generative models, PXCNN and Glow are tested, and for image compression algorithms, PNG, JPEG2000, and FLIF are tried. Our generative models are implemented as Section C. For PNG and JPEG2000, we use OpenCV11, and for FLIF, we use imageio-flif12. No combination of a generative model and a compression algorithm results in AUC higher than 0.9.
DUQ Investigating the official repository 13, we find that different normalization transforms are applied to in-distribution (CIFAR-10) and test OOD dataset (SVHN). CIFAR-10 is normalized with parameters mean=(0.4914, 0.4822, 0.4465) and std=(0.2023, 0.1994, 0.2010), whereas SVHN is normalized using mean=(0.5, 0.5, 0.5) and std=(0.5, 0.5, 0.5). Applying different transformations makes classification between CIFAR-10 and SVHN easier. When we modify the code so that an identical transformation is applied to both datasets, AUC score drops below 0.9. Therefore, we do not include DUQ in our experiments.
Likelihood Regret The method is very slow, particularly because it does not support batch processing, i.e., only one sample can be processed at a time. We decide that the method is too slow to apply the adversarial distribution approach.
7https://github.com/pokaxpoka/deep_Mahalanobis_detector 8https://github.com/inspire-group/SSD 9https://github.com/alinlab/CSI
10https://github.com/google-research/google-research/tree/master/ genomics_ood/images_ood
11https://opencv.org/ 12https://codeberg.org/monilophyta/imageio-flif 13https://github.com/y0ast/deterministic-uncertainty-quantification

heading: E EXTENDED EXPERIMENTAL RESULTS

heading: E.1 BINARY CLASSIFICATION
text: Here, we show the effect of test-time augmentation and report the zero-inlier threshold. Test-time augmentation dramatically increases the zero-inlier threshold, meaning that the classifiers become more robust.
0 20 logit
0.0
0.2 de ns
ity -7.9 CIFAR10 SVHN
20 0 20 logit
0.0
0.2 -12.6
CIFAR10 CelebA
Binary classifier logit values (No Aug)
Binary classifier logit values (TestTimeAug)

heading: E.2 RUNTIME AND COMPUTING ENVIRONMENT
text: Table 4 shows time required to perform MCMC sampling and for each OOD detectors. For MCMC, time required for running 10,000 steps of MH update is shown. The main factor that determines the runtime is the inference time of an OOD detector. Inference of all OOD detector is performed on a single Tesla V100 GPU.

heading: F ADDITIONAL SAMPLES FROM ADVERSARIAL DISTRIBUTIONS
text: From Figure 7 to Figure 12 present samples from adversarial distributions of each OOD detectors.

Review summaries:
The paper proposes a novel evaluation framework for out-of-distribution (OOD) detection under worst-case scenarios. While existing benchmarks use real samples from datasets outside the training distribution, the authors propose instead to learn an adversarial outlier distribution against OOD detectors using an autoencoder model, with an auxiliary binary classifier that filters out inlier samples. Empirical experiments on CIFAR-10 (inlier) and SVHN/CelebA (outlier) datasets show that standard OOD benchmarks tend to produce overoptimistic results, and that prior methods with similar scores on standard benchmarks have diverging performance outside the predefined OOD test sets.