Title: ENHANCING THE RELIABILITY OF OUT-OF-DISTRIBUTION IMAGE DETECTION IN NEURAL NETWORKS

Abstract: We consider the problem of detecting out-of-distribution images in neural networks. We propose ODIN, a simple and effective method that does not require any change to a pre-trained neural network. Our method is based on the observation that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in-and out-of-distribution images, allowing for more effective detection. We show in a series of experiments that ODIN is compatible with diverse network architectures and datasets. It consistently outperforms the baseline approach (Hendrycks & Gimpel, 2017) by a large margin, establishing a new state-of-the-art performance on this task. For example, ODIN reduces the false positive rate from the baseline 34.7% to 4.3% on the DenseNet (applied to CIFAR-10 and Tiny-ImageNet) when the true positive rate is 95%.
• We propose a simple and effective method, ODIN (Out-of-DIstribution detector for Neural networks), for detecting out-of-distribution examples in neural networks. Our method does not require re-training the neural network and is easily implementable on any modern neural architecture. • We test ODIN on state-of-the-art network architectures (e.g., DenseNet (Huang et al., 2016) and Wide ResNet (Zagoruyko & Komodakis, 2016)) under a diverse set of in-and out-distribution dataset pairs. We show ODIN can significantly improve the detection performance, and consistently outperforms the state-of-the-art method (Hendrycks & Gimpel, 2017) by a large margin. • We empirically analyze how parameter settings affect the performance, and further provide simple analysis that provides some intuition behind our method.
The outline of this paper is as follows. In Section 2, we present the necessary definitions and the problem statement. In Section 3, we introduce ODIN and present performance results in Section 4.
We experimentally analyze the proposed method and provide some justification for our method in Section 5. We summarize the related works and future directions in Section 6 and conclude the paper in Section 7.
In this paper, we consider the problem of distinguishing in-and out-of-distribution images on a pretrained neural network. Let P X and Q X denote two distinct data distributions defined on the image space X . Assume that a neural network f is trained on a dataset drawn from the distribution P X . Thus, we call P X the in-distribution and Q X the out-distribution, respectively. In testing, we draw new images from a mixture distribution P X×Z defined on X × {0, 1}, where the conditional probability distributions P X|Z=0 = P X and P X|Z=1 = Q X denote in-and out-distribution respectively. Now we focus on the following problem: Given an image X drawn from the mixture distribution P X×Z , can we distinguish whether the image is from in-distribution P X or not?
In this paper, we focus on detecting out-of-distribution images. However, it is equally important to correctly classify an image into the right class if it is an in-distribution image. But this can be easily done: once it has been detected that an image is in-distribution, we can simply use the original image and run it through the neural network to classify it. Thus, we do not change the predictions of the neural network for in-distribution images and only focus on improving the detection performance for out-of-distribution images.
In this section, we present our method, ODIN, for detecting out-of-distribution samples. The detector is built on two components: temperature scaling and input preprocessing. We describe the details of both components below. Temperature Scaling. Assume that the neural network f = (f 1 , ..., f N ) is trained to classify N classes. For each input x, the neural network assigns a label ŷ(x) = arg max i S i (x; T ) by computing the softmax output for each class. Specifically,
where T ∈ R + is the temperature scaling parameter and set to 1 during the training. For a given input x, we call the maximum softmax probability, i.e., S ŷ (x; T ) = max i S i (x; T ) the softmax score. In this paper, we use notations S ŷ (x; T ) and S(x; T ) interchangeably. Prior works have established the use of temperature scaling to distill the knowledge in neural networks (Hinton et al., 2015) and

1. INTRODUCTION:

Modern neural networks are known to generalize well when the training and testing data are sampled from the same distribution (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016; Cho et al., 2014; Zhang et al., 2017) . However, when deploying neural networks in real-world applications, there is often very little control over the testing data distribution. Recent works have shown that neural networks tend to make high confidence predictions even for completely unrecognizable (Nguyen et al., 2015) or irrelevant inputs (Hendrycks & Gimpel, 2017; Szegedy et al., 2014; Moosavi-Dezfooli et al., 2017) . It has been well documented (Amodei et al., 2016) that it is important for classifiers to be aware of uncertainty when shown new kinds of inputs, i.e., out-ofdistribution examples. Therefore, being able to accurately detect out-of-distribution examples can be practically important for visual recognition tasks (Krizhevsky et al., 2012; Farabet et al., 2013; Ji et al., 2013) .

A seemingly straightforward approach of detecting out-of-distribution images is to enlarge the training set of both in-and out-of-distribution examples. However, the number of out-of-distribution examples can be infinitely many, making the re-training approach computationally expensive and intractable. Moreover, to ensure that a neural network accurately classifies in-distribution samples into correct classes while correctly detecting out-of-distribution samples, one might need to employ exceedingly large neural network architectures, which further complicates the training process. Hendrycks & Gimpel (2017) proposed a baseline method to detect out-of-distribution examples without further re-training networks. The method is based on an observation that a well-trained neural network tends to assign higher softmax scores to in-distribution examples than out-of-distribution examples. In this paper, we go further. We observe that after using temperature scaling in the softmax function (Hinton et al., 2015; Pereyra et al., 2017) and adding small controlled perturbations to inputs, the softmax score gap between in -and out-of-distribution examples is further enlarged. We will show that the combination of these two techniques (temperature scaling and input perturbation) can lead to better detection performance. For example, provided with a pre-trained DenseNet (Huang et al., 2016) on CIFAR-10 dataset (positive samples), we test against images from TinyImageNet dataset (negative samples). Our method reduces the False Positive Rate (FPR), i.e., the fraction of misclassified out-of-distribution samples, from 34.7% to 4.3%, when 95% of in-distribution images are correctly classified. We summarize the main contributions of this paper as the following: calibrate the prediction confidence in classification tasks (Guo et al., 2017) . As we shall see later, a good manipulation of temperature T can push the softmax scores of in-and out-of-distribution images further apart from each other, making the out-of-distribution images distinguishable. Input Preprocessing. Before feeding the image x into the neural network, we preprocess the input by adding small perturbations to it. The preprocessed image is given by x = xεsign(-∇ x log S ŷ (x; T )),

where the parameter ε can be interpreted as the perturbation magnitude. The method is inspired by the idea in the reference (Goodfellow et al., 2015) , where small perturbations are added to decrease the softmax score for the true label and force the neural network to make a wrong prediction. Here, our goal and setting are rather different: we aim to increase the softmax score of any given input, without the need for a class label at all. As we shall see later, the perturbation can have stronger effect on the in-distribution images than that on out-of-distribution images, making them more separable. Note that the perturbations can be easily computed by back-propagating the gradient of the cross-entropy loss w.r.t the input.

Out-of-distribution Detector. The proposed approach works as follows. For each image x, we first calculate the preprocessed image x according to the equation ( 2). Next, we feed the preprocessed image x into the neural network, calculate its softmax score S( x; T ) and compare the score to the threshold δ. We say that the image x is an in-distribution example if the softmax score is above the threshold and that the image x is an out-of-distribution example, otherwise. Therefore, the out-of-distribution detector is given by

g(x; δ, T, ε) = 1 if max i p( x; T ) ≤ δ, 0 if max i p( x; T ) > δ.

The parameters T, ε and δ are chosen so that the true positive rate (i.e., the fraction of in-distribution images correctly classified as in-distribution images) under some out-of-distribution image data set is 95%. (The choice of the out-of-distribution images to tune the parameters T, ε and δ appears to be unimportant, as demonstrated in the appendix H.) Having chosen the parameters as above, we evaluate the performance of our algorithm using various metrics in the next section.

4. EXPERIMENTS:

In this section, we demonstrate the effectiveness of ODIN on several computer vision benchmark datasets. We run all experiments with PyTorch1 and we will release the code to reproduce all experimental results2 .

4.1. TRAINING SETUP:

Architectures and training configurations. We adopt two state-of-the-art neural network architectures, including DenseNet (Huang et al., 2016) and Wide ResNet (Zagoruyko & Komodakis, 2016) .

For DenseNet, our model follows the same setup as in (Huang et al., 2016) , with depth L = 100, growth rate k = 12 (Dense-BC) and dropout rate 0. In addition, we evaluate the method on a Wide ResNet, with depth 28, width 10 (WRN-28-10) and dropout rate 0. Furthermore, in Appendix A.1, we provide additional experimental results on another Wide ResNet with depth 40, width 4 (WRN-40-4).

The hyper-parameters of neural networks are set identical to the original Wide ResNet (Zagoruyko & Komodakis, 2016) and DenseNet (Huang et al., 2016) implementations. All neural networks are trained with stochastic gradient descent with Nesterov momentum (Duchi et al., 2011; Kingma & Ba, 2014) . Specifically, we train Dense-BC for 300 epochs with batch size 64 and momentum 0.9; and Wide ResNet for 200 epochs with batch size 128 and momentum 0.9. The learning rate starts at 0.1, and is dropped by a factor of 10 at 50% and 75% of the training progress, respectively. 

Accuracy.:

Each neural network architecture is trained on CIFAR-10 (C-10) and CIFAR-100 (C-100) datasets (Krizhevsky & Hinton, 2009) , respectively. CIFAR-10 and CIFAR-100 images are drawn from 10 and 100 classes, respectively. Both datasets consist of 50,000 training images and 10,000 test images. The test error on CIFAR datasets are summarized in Table 1 .

4.2. OUT-OF-DISTRIBUTION DATASETS:

At test time, the test images from CIFAR-10 (CIFAR-100) datasets can be viewed as the in-distribution (positive) examples. For out-of-distribution (negative) examples, we follow the setting in (Hendrycks & Gimpel, 2017) and test on several different natural image datasets and synthetic noise datasets. All the datasets considered are listed below.

(1) TinyImageNet. The Tiny ImageNet dataset3 consists of a subset of ImageNet images (Deng et al., 2009) . It contains 10,000 test images from 200 different classes. We construct two datasets, TinyImageNet (crop) and TinyImageNet (resize), by either randomly cropping image patches of size 32 × 32 or downsampling each image to size 32 × 32.

(2) LSUN. The Large-scale Scene UNderstanding dataset (LSUN) has a testing set of 10,000 images of 10 different scenes (Yu et al., 2015) . Similar to TinyImageNet, we construct two datasets, LSUN (crop) and LSUN (resize), by randomly cropping and downsampling the LSUN testing set, respectively.

(3) iSUN. The iSUN (Xu et al., 2015) consists of a subset of SUN images. We include the entire collection of 8925 images in iSUN and downsample each image to size 32 by 32. (4) Gaussian Noise. The synthetic Gaussian noise dataset consists of 10,000 random 2D Gaussian noise images, where each RGB value of every pixel is sampled from an i.i.d Gaussian distribution with mean 0.5 and unit variance. We further clip each pixel value into the range [0, 1]. (5) Uniform Noise. The synthetic uniform noise dataset consists of 10,000 images where each RGB value of every pixel is independently and identically sampled from a uniform distribution on [0, 1].

4.3. EVALUATION METRICS:

We adopt the following four different metrics to measure the effectiveness of a neural network in distinguishing in-and out-of-distribution images.

(1) FPR at 95% TPR can be interpreted as the probability that a negative (out-of-distribution) example is misclassified as positive (in-distribution) when the true positive rate (TPR) is as high as 95%. True positive rate can be computed by TPR = TP / (TP+FN), where TP and FN denote true positives and false negatives respectively. The false positive rate (FPR) can be computed by FPR = FP / (FP+TN), where FP and TN denote false positives and true negatives respectively.

(2) Detection Error, i.e., P e measures the misclassification probability when TPR is 95%. The definition of P e is given by P e = 0.5(1 -TPR) + 0.5FPR, where we assume that both positive and negative examples have the equal probability of appearing in the test set.

(3) AUROC is the Area Under the Receiver Operating Characteristic curve, which is also a thresholdindependent metric (Davis & Goadrich, 2006) . The ROC curve depicts the relationship between TPR and FPR. The AUROC can be interpreted as the probability that a positive example is assigned a higher detection score than a negative example (Fawcett, 2006) . A perfect detector corresponds to an AUROC score of 100%.

(4) AUPR is the Area under the Precision-Recall curve, which is another threshold independent metric (Manning et al., 1999; Saito & Rehmsmeier, 2015) . The PR curve is a graph showing the precision=TP/(TP+FP) and recall=TP/(TP+FN) against each other. The metric AUPR-In and AUPR-Out in Table 2 denote the area under the precision-recall curve where in-distribution and out-of-distribution images are specified as positives, respectively.

4.4. EXPERIMENTAL RESULTS:

Comparison with baseline. In Figure 1 , we show the ROC curves when DenseNet-BC-100 is evaluated on CIFAR-10 (positive) images against TinyImageNet (negative) test examples. The red curve corresponds to the ROC curve when using baseline method (Hendrycks & Gimpel, 2017) , whereas the blue curve corresponds to our method with temperature T = 1000 and perturbation magnitude ε = 0.0012. We observe a strikingly large gap between the blue and red ROC curves. For example, when TPR= 95%, the FPR can be reduced from 34% to 4.2% by using our approach. Choosing parameters. For each out-of-distribution dataset, we randomly hold out 1,000 images for tuning the parameters T and ε. For temperature T , we select among 1, 2, 5, 10, 20, 50, 100, 200, 500, 1000; and for perturbation magnitude ε we choose from 21 evenly spaced numbers starting from 0 and ending at 0.004. The optimal parameters are chosen to minimize the FPR at TPR 95% on the holdout set. We evaluate the our approach on the remaining test images. All parameter settings are reported in the Appendix A. We provide additional details on the effect of parameters in Section 5.

Main results. The main results are summarized in Table 2 . For each in-and out-of-distribution dataset pair, we report both the performance of the baseline (Hendrycks & Gimpel, 2017) and our approach using temperature scaling and input preprocessing. In Table 2 , we observe improved performance across all neural architectures and all dataset pairs. Noticeably, our method consistently outperforms the baseline by a large margin when measured by FPR at 95% TPR and detection error. 

4.5. EXTENSIONS:

In this subsection, we analyze how the statistical distance between in-and out-of-distribution natural image dataset affects the detection performance of the proposed method.

Data distribution distance vs. Detection performance. To measure the statistical distance between in-and out-of-distribution datasets, we adopt a commonly used metric, maximum mean discrepancy (MMD) with Gaussian RBF kernel (Sriperumbudur et al., 2010; Gretton et al., 2012; Sutherland et al., 2016) . Specifically, given two image sets, V = {v 1 , ..., v m } and W = {w 1 , ..., w m }, the maximum mean discrepancy between V and Q is defined as

MMD 2 (V, W ) = 1 m 2 i =j k(v i , v j ) + 1 m 2 i =j k(w i , w j ) - 2 m 2 i =j k(v i , w j ),

where

k(•, •) is the Gaussian RBF kernel, i.e., k(x, x ) = exp - x-x 2 2 2σ 2

. We use the same method used by Sutherland et al. ( 2016) to choose σ, where 2σ 2 is set to the median of all Euclidean distances between all images in the aggregate set V ∪ W .

In Figure 2 (a)(b), we show how the performance of ODIN varies against the MMD distances between in-and out-of-distribution datasets4 . The datasets (on x-axis) are ranked in the descending order of MMD distances with CIFAR-100. There are two interesting observations can be drawn from these figures. First, we find that the MMD distances between the cropped datasets and CIFAR-100 tend to be larger. This is likely due to the fact that cropped images only contain local image context and are therefore more distinct from CIFAR-100 images, while resized images contain global patterns and are thus similar to images in CIFAR-100. Second, we observe that the MMD distance tends to be negatively correlated with the detection performance. This suggests, not surprisingly, that the detection task becomes harder as in and out-of-distribution images are more similar to each other. Same-manifold datasets. Furthermore, we investigate the extreme scenario when in-and out-ofdistribution datasets are on the same manifold. In experiment, we randomly split CIFAR-100 into two disjoint datasets containing 80 and 20 classes each. We name them CIFAR-80 and CIFAR-20, respectively. We train both DenseNet and Wide ResNet-28-10 on the CIFAR-80 dataset (indistribution) and evaluate the detection performance on the CIFAR-20 dataset (out-distribution). All hyperparameters used here are exactly the same as in Section 4.1. The MMD distance between CIFAR-20 and CIFAR-80 is much smaller than other dataset pairs. In Figure 2 (c)(d), we observe that both FPR at TPR 95% and detection error become larger on the CIFAR-20 dataset. This coincides with our expectation that the detection task becomes extremely hard when in-and out-of-distribution dataset locate on the same manifold. We provide additional experimental results in Appendix A.1 and Appendix G.

5. DISCUSSIONS:



5.1. EFFECTS OF PARAMETERS:

In this subsection, we empirically show how temperature T and perturbation magnitude ε affect FPR at TPR 95% and AUROC on DenseNet and Wide ResNet-28-10. Additional results on other metrics and architectures are provided in Appendix B. We show the detection performance when using only the temperature scaling method (see Figure 3 (a)(b), ε = 0), or the input preprocessing method (see Figure 3 (c)(d), T = 1). In Figure 4 , we show the detection performance w.r.t ε when T is optimal (e.g., T =1000). First, from Figure 3 (a)(b), we observe that increasing the temperature can improve the detection performance, although the effects diminish when T is sufficiently large (e.g., T > 100). Next, from Figure 3 (c)(d) and Figure 4 , we observe that we can further improve the detection performance by appropriately choosing the perturbation magnitudes. We can achieve overall better performance by combining both (1) temperature scaling and (2) input preprocessing.

5.2. ANALYSIS ON TEMPERATURE SCALING:

In this subsection, we analyze the effectiveness of the temperature scaling method. As shown in Figure 3 (a) and (b), we observe that a sufficiently large temperature yields better detection performance although the effects diminish when T is too large. To gain insight, we can use the Taylor expansion of the softmax score (details provided in Appendix D). When T is sufficiently large, we have

S ŷ (x; T ) ≈ 1 N -1 T i [f ŷ (x) -f i (x)] + 1 2T 2 i [f ŷ (x) -f i (x)] 2 ,

(3) by omitting the third and higher orders. For simplicity of notation, we define

EQUATION

)

Interpretations of U 1 and U 2 . By definition, U 1 measures the extent to which the largest unnormalized output of the neural network deviates from the remaining outputs; while U 2 measures the extent to which the remaining smaller outputs deviate from each other. We provide formal mathematical derivations in Appendix F. In Figure 5 (a), we show the distribution of U 1 for each out-of-distribution dataset vs. the in-distribution dataset (in red). We observe that the largest outputs of the neural network on in-distribution images deviate more from the remaining outputs. This is likely due to the fact that neural networks tend to make more confident predictions on in-distribution images.

Further, we show in Figure 5 when two images have similar values on U 1 , the in-distribution image tends to have a much higher value of U 2 than the out-of-distribution image. In other words, for in-distribution images, the remaining outputs (excluding the largest output) tend to be more separated from each other compared to out-of-distribution datasets. This may happen when some classes in the in-distribution dataset share common features while others differ significantly. To illustrate this, in Figure 5 (f)(g), we show the outputs of each class using a DenseNet (trained on CIFAR-10) on a dog image from CIFAR-10, and another image from TinyImageNet (crop). For the image of dog, we can observe that the largest output for the label dog is close to the output for the label cat but is quite separated from the outputs for the label car and truck. This is likely due to the fact that, in CIFAR-10, images of dogs are very similar to the images of cats but are quite distinct from images of car and truck. For the image from TinyImageNet (crop), despite having one large output, the remaining outputs are close to each other and thus have a smaller deviation.

The effects of T . To see the usefulness of adopting a large T , we can first rewrite the softmax score function in Equation (3) as S ∝ (U 1 -U 2 /2T )/T . Hence the softmax score is largely determined by U 1 and U 2 /2T . As noted earlier, U 1 makes in-distribution images produce larger softmax scores than out-of-distribution images since S ∝ U 1 , while U 2 has the exact opposite effect since S ∝ -U 2 . Therefore, by choosing a sufficiently large temperature, we can compensate the negative impacts of U 2 /2T on the detection performance, making the softmax scores between in-and out-of-distribution images more separable. Eventually, when T is sufficiently large, the distribution of softmax score is almost dominated by the distribution of U 1 and thus increasing the temperature further is no longer effective. This explains why we see in Figure 3 (a)(b) that the performance does not change when T is too large (e.g., T > 100). In Appendix E, we provide a formal proof showing that the detection error eventually converges to a constant number when T goes to infinity.

5.3. ANALYSIS ON INPUT PREPROCESSING:

As noted previously, using the temperature scaling method by itself can be effective in improving the detection performance. However, the effectiveness quickly diminishes as T becomes very large. In order to make further improvement, we complement temperature scaling with input preprocessing. This has already been seen in Figure 4 , where the detection performance is improved by a large margin on most datasets when T = 1000, provided with an appropriate perturbation magnitude ε is chosen. In this subsection, we provide some intuition behind this.

To explain, we can look into the first order Taylor expansion of the log-softmax function for the perturbed image x, which is given by

log S ŷ ( x; T ) = log S ŷ (x; T ) + ε ∇ x log S ŷ (x; T ) 1 + o(ε),

where x is the original input.

The effects of gradient. In Figure 5 (c), we present the distribution of ∇ x log S(x; T ) 1 -the 1-norm of gradient of log-softmax with respect to the input x -for all datasets. A salient observation is that CIFAR-10 images (in-distribution) tend to have larger values on the norm of gradient than most out-of-distribution images. To further see the effects of the norm of gradient on the softmax score, we provide in Figures 5 (d ) the conditional expectation E[ ∇ x log S(x; T ) 1 |S]. We can observe that, when an in-distribution image and an out-of-distribution image have the same softmax score, the value of ∇ x log S(x; T ) 1 for in-distribution image tends to be larger. We illustrate the effects of the norm of gradient in Figure 6 . Suppose that an in-distribution image x 1 (blue) and an out-of-distribution image x 2 (red) have similar softmax scores, i.e., S(x 1 ) ≈ S(x 2 ).

After input processing, the in-distribution image can have a much larger softmax score than the out-of-distribution image x 2 since x 1 results in a much larger value on the norm of softmax gradient than that of x 2 . Therefore, in-and out-of-distribution images are more separable from each other after input preprocessing5 .

The effect of ε. When the magnitude ε is sufficiently small, adding perturbations does not change the predictions of the neural network, i.e., ŷ( x) = ŷ(x). However, when ε is not negligible, the gap of softmax scores between in-and out-of-distribution images can be affected by ∇ x log S(x; T ) 1 . Our observation is consistent with that in (Szegedy et al., 2014; Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2017) , which show that the softmax scores tend to change significantly if small perturbations are added to the in-distribution images. It is also worth noting that using a very large ε can lead to performance degradation, as seen in Figure 4 . This is likely due to the fact that the second and higher order terms in the Taylor expansion are no longer insignificant when the perturbation magnitude is too large.

6. RELATED WORKS AND FUTURE DIRECTIONS:

The problem of detecting out-of-distribution examples in low-dimensional space has been well-studied in various contexts (see the survey by Pimentel et al. ( 2014)). Conventional methods such as density estimation, nearest neighbor and clustering analysis are widely used in detecting low-dimensional outof-distribution examples (Chow, 1970; Vincent & Bengio, 2003; Ghoting et al., 2008; Devroye et al., 2013) , . The density estimation approach uses probabilistic models to estimate the in-distribution density and declares a test example to be out-of-distribution if it locates in the low-density areas.

The clustering method is based on the statistical distance, and declares an example to be out-ofdistribution if it locates far from its neighborhood. Despite various applications in low-dimensional spaces, unfortunately, these methods are known to be unreliable in high-dimensional space such as image space (Wasserman, 2006; Theis et al., 2015) . In recent years, out-of-distribution detectors based on deep models have been proposed. Schlegl et al. (2017) train a generative adversarial networks to detect out-of-distribution examples in clinical scenario. Sabokrou et al. (2016) train a convolutional network to detect anomaly in scenes. Andrews et al. ( 2016) adopt transfer representation-learning for anomaly detection. All these works require enlarging or modifying the neural networks. In a more recent work, Hendrycks & Gimpel (2017) found that pre-trained neural networks can be overconfident to out-of-distribution example, limiting the effectiveness of detection. Our paper aims to improve the performance of detecting out-of-distribution examples, without requiring any change to an existing well-trained model.

Our approach leverages the following two interesting observations to help better distinguish between in-and out-of-distribution examples: (1) On in-distribution images, modern neural networks tend to produce outputs with larger variance across class labels, and (2) neural networks have larger norm of gradient of log-softmax scores when applied on in-distribution images. We believe that having a better understanding of these phenomenon can lead to further insights into this problem.

7. CONCLUSIONS:

In this paper, we propose a simple and effective method to detect out-of-distribution data samples in neural networks. Our method does not require retraining the neural network and significantly improves on the baseline (state-of-the-art) on different neural architectures across various in and out-distribution dataset pairs. We empirically analyze the method under different parameter settings, and provide some insights behind the approach. Future work involves exploring our method in other applications such as speech recognition and natural language processing. 0.1/0.0 2.5/2.5 99.7/100.0 99.8/100.0 99.7/100.0 Gaussian 0.0/0.0 2.5/2.5 99.7/100.0 99.8/100.0 99.7/100.0 Uniform 0.0/0.0 2.5/2.5 99.9/100.0 99.9/100.0 99.9/100.0 MNIST: We used the same MNIST classifier used by Hendrycks & Gimpel (2017) , which is a three-layer, 256 neuron-wide, fully connected network trained for 30 epochs with Adam (Kingma & Ba, 2014) . The classifier achieve 99.34% test accuracy on the MNIST test set. We compare our method with the baseline (Hendrycks & Gimpel, 2017) on five different out-of-distribution datasets:

(1) Omniglot dataset (Lake et al., 2015) contains images of handwritten characters in stead of the handwritten digits in MNIST;

(2) notMNIST (Bulatov, 2011) 

D TAYLOR EXPANSION:

In this section, we present the Taylor expansion of the soft-max score function:

S ŷ (x; T ) = exp (f ŷ (x)/T ) N i=1 exp(f i (x)/T ) = 1 N i=1 exp fi(x)-f ŷ (x) T = 1 N i=1 1 + fi(x)-f ŷ (x) T + 1 2! (fi(x)-f ŷ (x)) 2 T 2 + o 1 T 2

by Taylor expansion

≈ 1 N -1 T N i=1 [f ŷ (x) -f i (x)] + 1 2T 2 N i=1 [f i (x) -f ŷ (x)] 2 E PROPOSITION 1

The following proposition 1 shows that the detection error P e (T, 0) ≈ c if T is sufficiently large. Thus, increasing the temperature further can only slightly improve the detection performance. Proposition 1. There exists a constant c only depending on function U 1 , in-distribution P X and out-of-distribution Q X such that lim T →∞ P e (T, ε) = c, when ε = 0 (i.e., no input preprocessing).

Proof. Since

S ŷ (X; T ) = exp(f ŷ (X)/T ) N i=1 exp(f i (X)/T ) = 1 1 + i =ŷ exp([f i (X) -f ŷ (X)]/T ) Therefore, for any X, lim T →∞ T - 1 S ŷ (X; T ) + N = lim T →∞ i =ŷ T 1 -exp f i (X) -f ŷ (X) T = i =ŷ [f ŷ (X) -f i (X)] = (N -1)U 1 (X)

This indicates that the random variable

T - 1 S ŷ (X; T ) + N → (N -1)U 1 (X) a.s.

as T → ∞. This means that for a specific α > 0, choosing the threshold δ T = 1/(Nα/T ), then the false positive rate

FPR(T ) = Q X (S ŷ (X; T ) > 1/(N -α/T )) = Q X T N - 1 S ŷ (X; T ) > α T →∞ ----→ Q X ((N -1)U 1 (X) > α) ,

and the true positive rate

TPR(T ) = P X (S ŷ (X; T ) > 1/(N -α/T )) = P X T N - 1 S ŷ (X; T ) > α T →∞ ----→ P X ((N -1)U 1 (X) > α) .

Choosing α * such that P X ((N -1)U 1 (X) > α * ) = 0.95, then TPR(T ) → 0.95 as T → ∞ and at the same time FPR(T ) → Q X ((N -1)U 1 (X) > α * ) as T → ∞. There exists a constant c depending on U 1 , P X , Q X and P Z , such that lim T →∞ P e (T, 0) = 0.05P (Z = 0) + P (Z = 1)Q X ((N -1)U 1 (X) > α * ) = c.

F ANALYSIS OF TEMPERATURE:

For simplicity of the notations, let ∆ i = f ŷf i and thus ∆ = {∆ i } i =ŷ . Besides, let ∆ denote the mean of the set ∆. Therefore,

∆ = 1 N -1 i =ŷ ∆ i = 1 N -1 i =ŷ [f ŷ -f i ] = U 1 .

Equivalently,

U 1 = Mean(∆).

Next, we will show

U 2 = 1 N -1 i =ŷ [f ŷ -f i ] 2 = Variance 2 (∆) 1 N -1 i =ŷ [∆ i -∆] 2 + Mean 2 (∆) ∆2 . Since U 2 = 1 N -1 i =ŷ ∆ 2 i by∆ i = f ŷ -f i = 1 N -1 i =ŷ (∆ i -∆ + ∆) 2 = 1 N -1 i =ŷ [(∆ i -∆) 2 -2(∆ i -∆) ∆ + ∆2 ] = 1 N -1 i =ŷ [∆ i -∆] 2 Variance 2 (∆) - 2 ∆ N -1 i =ŷ (∆ i -∆) =0 + ∆2 Mean 2 (∆)

then

U 2 = Variance 2 (∆) + Mean 2 (∆) G ADDITIONAL RESULTS IN SECTION 4.5

Apart from the Maximum Mean Discrepancy, we as well calculate the Energy distance between inand out-of-distribution datasets. Let P and Q denote two different distributions. Then the energy distance between distributions P and Q is defined as

D 2 energy (P, Q) = 2E V ∼P,W ∼Q X -Y -E V,V ∼P X -X -E W,W ∼Q Y -Y .

Therefore, the energy distance between two datasets

V = {V 1 , ..., V m } iid ∼ P and W = {W 1 , ..., W m } iid ∼ Q is defined as D energy 2 (P, Q) = 2 m 2 m i=1 m j=1 V i -W j - 1 m 2 i =j V i -V j - 1 m 2 i =j W i -W j .

In the experiment, we use the 2-norm • 2 .

In 

H ADDITIONAL DISCUSSIONS:

In this section, we present additional discussion on the proposed method. We first empirically show how the threshold δ affects the detection performance. We next show how the proposed method performs when the parameters are tuned on a certain out-of-distribution dataset and are evaluated on other out-of-distribution datasets. Finally, we show how the size of dataset for choosing parameters affects the detection performance.

Effects of the threshold. We analyze how the threshold affects the following metrics: (1) FPR, i.e., the fraction of out-of-distribution images misclassified as in-distribution images; (2) TPR, i.e, the fraction of in-distribution images correctly classified as in-distribution images. In Figure 13 , we show how the thresholds affect FPR and TPR when the temperature and perturbation magnitude are chosen optimally (i.e., T = 1, 000, ε = 0.0014). From the figure, we can observe that the threshold corresponding to 95% TPR can produce small FPRs on all out-of-distribution datasets.

Performance across datasets. To investigate how the parameters generalize across datasets, we tune the parameters using one out-of-distribution dataset and then evaluate on a different one. Given an out-of-distribution dataset, we first split the dataset into two disjoint subsets: tuning set and test set. The tuning set contains 1,000 images and the test set contains 9,000 images. We tune the parameters on the tuning set and evaluate the detection performance on the test set. We first choose the temperature T and the perturbation magnitude ε such that the FPR at TPR 95% is minimized on the tuning set of one out-of-distribution dataset. Next, we set δ to the threshold corresponding to 95% TPR and calculate the false positive rates on the test sets of other out-of-distribution datasets.

In Figure 14 , we show the detection performance on three out-of-distribution datasets when the parameters are tuned on six different datasets. From Figure 14 , we can observe that the parameters tuned on different tuning sets can have quite similar detection FPRs on all of three out-of-distribution image sets. This may be due to the fact, shown in Figure 13 , that the threshold corresponding to 95% TPR can produce small FPRs on all datasets. Performance vs. tuning set size. To show the effects of the tuning set size on the detection performance, we devise the following experiment. For each out-of-distribution dataset, we choose the tuning set size from 200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000. For each set size, we tune the temperature T and perturbation magnitude ε to minimize the FPR at TPR 95% and calculate the FPR. In Figure 15 , we show the detection performance of ODIN under different tuning set size. From Figure 15 , we can observe that the FPR at TPR 95% tends to stabilize when the set size grows above 1,000. 

I ADDITIONAL ANALYSIS:

Difficult-to-classify images and difficult-to-detect images. We analyze the correlation between the images that tend to be out-of-distribution and images on which the neural network tend to make incorrect predictions. To understand the correlation, we devise the following experiment. For the fixed temperature T and perturbation magnitude ε, we first set δ to the softmax score threshold corresponding to a certain true positive rate. Next, we calculate the test accuracy on the images with softmax scores above δ and the test accuracy on the images with softmax score below δ, respectively. We report the results in Figure 16 (a) and (b). From these two figures, we can observe that the images that are difficult to detect are more likely to be the images that are difficult to classify. For example, the DenseNet can achieve up to 98.5% test accuracy on the images having softmax scores above the threshold corresponding to 80% TPR, but can only achieve around 82% test accuracy on the images having softmax scores below the threshold corresponding to 80% TPR.

Same manifold datasets. We provide additional empirical results showing how the term E[U 2 |U 1 ] affects the detection performance when in-and out-of-distribution datasets locate on the same manifold. In Figure 17 , we show the outputs of DenseNet on thirty classes for an image of apple from CIFAR-80 (in-distribution) and an image of red pepper of CIFAR-20 (out-distribution). We can observe that the outputs of DenseNet for both images are quite similar to each other. In addition, we can observe that for both images, the second and third largest output are quite close to the largest output. This may be due the fact the image of red pepper shares some common features with the images in CIFAR-80. Furthermore, the similarity between the outputs for the images from

:

http://pytorch.org

https://github.com/facebookresearch/odin

https://tiny-imagenet.herokuapp.com

All distances are provided in Appendix G.

Similar observation can be seen when T = 1, where we present the conditional expectation of the norm of softmax gradient in Figure5(e).

and Wide ResNet-40-4, where the optimal temperature is used, i.e., T = 1000. All networks are trained on CIFAR-10.

Figures and Tables:

Figure 1: (a) ROC curves of baseline (red) and our method (blue) on DenseNet-BC-100 network, where CIFAR-10 and TinyImageNet (crop) are in-and out-of-distribution dataset, respectively.

Figure 2: (a)-(d) Performance of our method vs. MMD between in-and out-of-distribution datasets. Neural networks are trained on CIFAR-100 and CIFAR-80, respectively. The out-of-distribution datasets are 1: LSUN (cop), 2: TinyImageNet (crop), 3: LSUN (resize), 4: is iSUN (resize), 5: TinyImageNet (resize) and 6: CIFAR-20.

Figure 3: (a)(b) Effects of temperature T when ε = 0. (c)(d) Effects of perturbation magnitude ε when T = 1. All networks are trained on CIFAR-10 (in-distribution). Additional results on other metrics and Wide ResNet-40 are provided in Appendix B.

Figure 5: (a) Probability density of U1 under different datasets on DenseNet. (b) Expectations of U2 conditioned on U1 on DenseNet. (c) Probability density of the norm of gradient on DenseNet under temperature 1, 000. (c)(d) Expectation of the norm of gradient conditioned on the softmax scores on DenseNet under temperature T = 1000 and T = 1, respectively. (f)(g) Outputs of DenseNet on each class for an image of dog from CIFAR-10 and an image from TinyImageNet (crop). The DenseNet is trained on CIFAR-10. Additional results on other architectures are provided in Appendix C.

Figure 6: Illustration of effects of the input preprocessing.

Figure 7: Detection performance on DenseNet, Wide ResNet-28-10 and Wide ResNet-40-4 under different temperature, when input preprocessing is not used, i.e., ε = 0. All networks are trained on CIFAR-10.

Figure 13: False positive rate (FPR) and true positive rate (TPR) under different thresholds (δ) when the temperature (T ) is set to 1, 000 and the perturbation magnitude (ε) is set to 0.0014. The DenseNet is trained on CIFAR-10.

Figure 14: Detection performance on Tiny-ImageNet (resize), LSUN (resize) and iSUN (resize) when parameters are tuned on six different out-of-distribution datasets. Each tuning set contains 1,000 images and each test set contains 9,000 images. Both DenseNet and Wide-ResNet are trained on CIFAR-10. Additional results on other datasets are provide in Table10 and 11.

Figure 15: FPR at TPR 95% under different tuning set sizes. The DenseNet is trained on CIFAR-10 and each test set contains 8,000 out-of-distribution images.

Figure 16: (a) The test accuracy on the images having softmax scores above the threshold corresponding to a certain true positive rate. (b) The test accuracy on the images having softmax scores below the threshold corresponding to a certain true positive rate. All networks are trained on CIFAR-10.

Distinguishing

Table content: <table><tr><td/><td>Out-of-distribution</td><td>FPR</td><td>Detection</td><td>AUROC</td><td>AUPR</td><td>AUPR</td></tr><tr><td/><td>dataset</td><td>(95% TPR)</td><td>Error</td><td/><td>In</td><td>Out</td></tr><tr><td/><td/><td>↓</td><td>↓</td><td>↑</td><td>↑</td><td>↑</td></tr><tr><td/><td/><td colspan="5">Baseline (Hendrycks &amp; Gimpel, 2017) / ODIN</td></tr><tr><td/><td>TinyImageNet (crop)</td><td>34.7/4.3</td><td>19.9/4.7</td><td>95.3/99.1</td><td>96.4/99.1</td><td>93.8/99.1</td></tr><tr><td>Dense-BC CIFAR-10</td><td>TinyImageNet (resize) LSUN (crop) LSUN (resize)</td><td>40.8/7.5 39.3/8.7 33.6/3.8</td><td>22.9/6.3 22.2/6.9 19.3/4.4</td><td>94.1/98.5 94.8/98.2 95.4/99.2</td><td>95.1/98.6 96.0/98.5 96.4/99.3</td><td>92.4/98.5 93.1/97.8 94.0/99.2</td></tr><tr><td/><td>iSUN</td><td>37.2/6.3</td><td>21.1/5.7</td><td>94.8/98.8</td><td>95.9/98.9</td><td>93.1/98.8</td></tr><tr><td/><td>Uniform</td><td>23.5/0.0</td><td>14.3/2.5</td><td>96.5/99.9</td><td>97.8/100.0</td><td>93.0/99.9</td></tr><tr><td/><td>Gaussian</td><td>12.3/0.0</td><td>8.7/2.5</td><td colspan="3">97.5/100.0 98.3/100.0 95.9/100.0</td></tr><tr><td/><td>TinyImageNet (crop)</td><td>67.8/17.3</td><td>36.4/11.2</td><td>83.0/97.1</td><td>85.3/97.4</td><td>80.8/96.8</td></tr><tr><td/><td>TinyImageNet (resize)</td><td>82.2/44.3</td><td>43.6/24.6</td><td>70.4/90.7</td><td>71.4/91.4</td><td>68.6/90.1</td></tr><tr><td>Dense-BC CIFAR-100</td><td>LSUN (crop) LSUN (resize) iSUN</td><td>69.4/17.6 83.3/44.0 84.8/49.5</td><td>37.2/11.3 44.1/24.5 44.7/27.2</td><td>83.7/96.8 70.6/91.5 69.9/90.1</td><td>86.2/97.1 72.5/92.4 71.9/91.1</td><td>80.9/96.5 68.0/90.6 67.0/88.9</td></tr><tr><td/><td>Uniform</td><td>88.3/0.5</td><td>46.6/2.8</td><td>83.2/99.5</td><td>88.1/99.6</td><td>73.1/99.0</td></tr><tr><td/><td>Gaussian</td><td>95.4/0.2</td><td>50.2/2.6</td><td>81.8/99.6</td><td>87.6/99.7</td><td>70.1/99.1</td></tr><tr><td/><td>TinyImageNet (crop)</td><td>38.9/23.4</td><td>21.9/14.2</td><td>92.9/94.2</td><td>92.5/92.8</td><td>91.9/94.7</td></tr><tr><td/><td>TinyImageNet (resize)</td><td>45.6/25.5</td><td>25.3/15.2</td><td>91.0/92.1</td><td>89.7/89.0</td><td>89.9/93.6</td></tr><tr><td>WRN-28-10 CIFAR-10</td><td>LSUN (crop) LSUN (resize) iSUN</td><td>35.0/21.8 35.0/17.6 40.6/21.3</td><td>20.0/13.4 20.0/11.3 22.8/13.2</td><td>94.5/95.9 93.9/95.4 92.5/93.7</td><td>95.1/95.8 93.8/93.8 91.7/91.2</td><td>93.1/95.5 92.8/96.1 91.5/94.9</td></tr><tr><td/><td>Uniform</td><td>1.6/0.0</td><td>3.3/2.5</td><td colspan="3">99.2/100.0 99.3/100.0 98.9/100.0</td></tr><tr><td/><td>Gaussian</td><td>0.3/0.0</td><td>2.6/2.5</td><td colspan="3">99.5/100.0 99.6/100.0 99.3/100.0</td></tr><tr><td/><td>TinyImageNet (crop)</td><td>66.6/43.9</td><td>35.8/24.4</td><td>82.0/90.8</td><td>83.3/91.4</td><td>80.2/90.0</td></tr><tr><td/><td>TinyImageNet (resize)</td><td>79.2/55.9</td><td>42.1/30.4</td><td>72.2/84.0</td><td>70.4/82.8</td><td>70.8/84.4</td></tr><tr><td/><td>LSUN (crop)</td><td>74.0/39.6</td><td>39.5/22.3</td><td>80.3/92.0</td><td>83.4/92.4</td><td>77.0/91.6</td></tr><tr><td>WRN-28-10</td><td>LSUN (resize)</td><td>82.2/56.5</td><td>43.6/30.8</td><td>73.9/86.0</td><td>75.7/86.2</td><td>70.1/84.9</td></tr><tr><td>CIFAR-100</td><td>iSUN</td><td>82.7/57.3</td><td>43.9/31.1</td><td>72.8/85.6</td><td>74.2/85.9</td><td>69.2/84.8</td></tr><tr><td/><td>Uniform</td><td>98.2/0.1</td><td>51.6/2.5</td><td>84.1/99.1</td><td>89.9/99.4</td><td>71.0/97.5</td></tr><tr><td/><td>Gaussian</td><td>99.2/1.0</td><td>52.1/3.0</td><td>84.3/98.5</td><td>90.2/99.1</td><td>70.9/95.9</td></tr></table>

Distinguishing in-and out-of-distribution test set data for image classification. All values are percentages. ↑ indicates larger value is better, and ↓ indicates lower value is better.

Table content: <table/>

Table 10 and 11.

Table content: <table/>

Detection performance across different datasets. Each row corresponds to the FPR at TPR 95% on the same test set where parameters are tuned under different tuning sets. Each column corresponds to the FPR at TPR 95% on different test sets where parameters are tuned under the same tuning set. The DenseNet is trained on CIFAR-10.

Table content: <table><tr><td>DenseNet-BC-100</td></tr></table>

Detection performance across different datasets. Each row corresponds to the FPR at TPR 95% on the same test set where parameters are tuned under different tuning sets. Each column corresponds to the FPR at TPR 95% on different test sets where parameters are tuned under the same tuning set. The Wide-ResNet is trained on CIFAR-10.

Table content: <table><tr><td/><td>0.99</td><td/><td/><td/></tr><tr><td>Test Accuracy</td><td>0.97 0.98</td><td/><td/><td/></tr><tr><td/><td>0.96</td><td/><td/><td/></tr><tr><td/><td>80</td><td>85</td><td>90</td><td>95</td><td>100</td></tr><tr><td/><td/><td colspan="3">True Positive Rate (%)</td></tr></table>

Peer review (iclr2018):
 The paper proposes a new method for detecting out of distribution samples. The core idea is two fold: when passing a new image through the (already trained) classifier, first preprocess the image by adding a small perturbation to the image pushing it closer to the highest softmax output and second, add a temperature to the softmax. Then, a simple decision is made based on the output of the softmax of the perturbed image - if it is able some threshold then the image is considered in-distribution otherwise out-distribution. This paper is well written, easy to understand and presents a simple and apparently effective method of detecting out of distribution samples. The authors evaluate on cifar-10/100 and several out of distribution datasets and this method outperforms the baseline by significant margins. They also examine the effects of the temperature and step size of the perturbation. My only concern is that the parameter delta (threshold used to determine in/out distribution) is not discussed much. They seem to optimize over this parameter, but this requires access to the out of distribution set prior to the final evaluation. Could the authors comment on how sensitive the method is to this parameter? How much of the out of distribution dataset is used to determine this value, and what are the effects of this size during tuning? What happens if you set the threshold using one out of distribution dataset and then evaluate on a different one? This seems to be the central part missing to this paper and if the authors are able to address it satisfactorily I will increase my score.