Out-of-Distribution Image Detection Using the Normalized Compression Distance
Authors: Sehun Yu
Donga Lee
Hwanjo Yu
Venue:  (2020)
Keywords: Out-of-Distribution Detection
Normalized Compression Distance
Convolutional Neural Networks

Abstract: On detection of the out-of-distribution images, whose underlying distribution is different from that of the training dataset, we tackle to apply out-of-distribution detection methods to already deployed convolutional neural networks. Most recent approaches have to utilize out-of-distribution samples for validation or retrain the model, which makes it less practical for real-world applications. We propose a novel out-of-distribution detection method MALCOM, which neither uses any out-of-distribution samples nor retrain the model. Inspired by the method using the global average pooling on the feature maps of the convolutional neural networks, the goal of our method is to extract informative sequential patterns from the feature maps. To this end, we introduce a similarity metric which focuses on the shared patterns between two sequences. In short, MALCOM uses both the global average and spatial pattern of the feature maps to accurately identify out-of-distribution samples. 

Under review as a conference paper at ICLR 2020
OUT-OF-DISTRIBUTION IMAGE DETECTION USING THE NORMALIZED COMPRESSION DISTANCE
Anonymous authors Paper under double-blind review
ABSTRACT
To reliably detect the out-of- df-dittribttton images based on already deployed con- volutional neural networks, several recent studies on of-iittriiiittn detection have tried to define effective confidence scores without retraining the model. Al- though they have shown promising results, most of them need find the op- timal hyperparameter values by using few out-of-distribution images, which eventually assumes specific test distribution and makes it less practical for real- world applications. In this work, propose novel out-of-distribution detection method MALCOM, which neither uses any out-of-distribution samples nor re- train the model. Inspired by the observation that global average pooling cannot capture the spatial information of the feature maps from convolutional neural net- works, method aims extract informative sequential patterns from the feature maps. To this end, we introduce a similarity metric which focuses on the shared patterns between two sequences based on normalized compression distance. short, MALCOM uses both the global average and spatial pattern of the feature maps to accurately identify -iitr---iitiiittionn samples.
INTRODUCTION
The distributional uncertainty refers uncertainty originated from the inconsistency between training and test distributions (Malinin & Gales) 2018). Recently, measuring the distributional un- certainty of deep neural networks has gained much attention, in order to detect the out-of out-oo-disttrbution sample which comes from outside the training distribution. In contrast to the existing belief that the softmax output of deep neural networks is not appropriate measure for uncertainty. [Hendrycks & Gimpel (2017) found out that the softmax score is able to distinguish between in-distribution and out-of-distribution samples to a degree.
Since then, several methods have shown promising results for out-of-distribution detection (Liang jet al. 2018 Lee et al., 2018b; Lakshminarayanan et al. 2017 Lee et al.) 2018a), but the most of them still have limitations from practical perspective in that they need find the optimal values of hyperparameters using the out-of-distribution samples or retrain the model using a new objec- tive function specifically designed for detection. For example, Liang al. (2088) improved the ability of distinguishing the out-of-distribution data by using temperature scaling and input per- turbation, which involves in several hyperparameters: this kind of calibration techniques require out-of-distributio samples for validation to adjust its hyperparameters to manipulate the softmax outputs. On the other hand, Lee et al. (2018a) defined an another loss function by adding the KL divergence, assuming that the softmax output of the the model because of the newly designed loss function. For these reasons, they have been difficult to be deployed for real-worke applications. In this sense, our motivation is how far out-of-distributio detection can be done without using the out-of-distribuutin set for validation while employing the existing softmax classifier without retraining it.
is uniform; it has to retrain
The state-of the-art method (Lee et al. 2018b) tries to detect out-of-distribttion samples based on the Mahalanobis distance from class means instead of using the softmax output, by utilizing the feature maps obtained by convolutional neural networks. In particular, they estimate the class-conditional probabilities for each class, which more appropriate and effective detect oft-o--distibution samples than the softmax output which provides a mere relative values between classes. However, they obtain the feature vectors of input images by averaging each feature map, also known global

---

Under review as a conference paper at ICLR 2020
average pooling. We notice that the spatial information about input images disappears in the process of averaging feature vectors, and this eventually degrades the detection performance.
To tackle this challenge, offer to the normalized compression distance (NCD), which can measure the distance between two images without the loss of such spatial information of the feature maps. NCD is a general method of measuring the intrinsic distance between two arbitrary objects using the off-the-shelf compression algorithm (e.g. 7-zip, bzip, or lzw). With its solid theoretical background and general applicability. NCD is widely used in various fields, including brain diagno- sis or spam filtering (Berek et al., 2014; Spracklin & Saxton 2007). The high-level idea of NCD evaluate how many shared patterns are eliminated when two binary sequences are compressed together. By using the NCD, we aim to accurately measure how similar the feature maps of a test sample with those of training examples, and use determine whether the test sample is from out-of-distrine ditoo--distribution or not. Our experiments demonstrate that the proposed method, termed as MAL- COM, achieves better or comparable performances for detecting diitt--iiittiuttionn images to the existing methods.
BACKGROUNDS
2.1 CONSTRAINED OUT-OF- OUT-OF-DISTRIEEE DISTRIBUTION DETECTION
Before proceeding to describe the notions closely related to our method, let us state what we want to tackle exactly: out-of-distribution detection without retraining network and without sniffing our- of-dixtribution samples. Starting with the definition, let Dtrain {(x;, y: };)} be the training dataset and f(; 0) be the model trained on Dtrain- Suppose that each data æ, from Dtrain is an observation of random variable irain Thereby, the out-of-distribution detection task is to obtain the out-of- -fit----iitriuttoon score function x(:; Derain O, Φ) for a discrimination between in-distribution and aitto--distribution. In other words, given test instance x", the score k(x") should be high if x is not an observation of Xurin otherwise should be low. There are many ways to get score function ni, but we work the task under the following constraints:
The score function parameter & should be independent with the distribution of test data x", i.e., Φ is determined by only in-distribution. This is intuitive in that xf does not assume any test distribution. This requirement implies that hyperparameter tuning by using out-of- distribution samples is not allowed.
The model parameter 0 should not be changed, i.c., the trained model primitively for the purpose of classification should not be retrained. Then, the score function can be utilized by the neural networks which are already trained and deployed in real-world applications.
The existing methods that satisfy these conditions are the softmax output detector (Hendrycks & Gimpel 2017) and the simplified Mahalanobis detector (Lee et al. 2018b).
2.2 MAHALANORIS DETECTOR
The Mahalanobis detector (MAHALANOBIS), one of the best performing methods, introduces an effective method to remove the correlation between feature maps of the last hidden layer (Lee al., 2018b), but combining layers leads to the degradation of the detection performance.
2.2.1
CHHALANORIS-VANTLLA
We factor out simplified version of MAHALANORIS (MAHALANOBSSS-vailll, because the whole MAHALANORIS detector needs out-of-distribution samples to tune the hyperparameters of input preprocessing and layer weights, which fails to satisfy our requirements (Sec 2.1). For the whole method of MAHALANOBIS which utilizes out-of-distribution samples to boost the performance, we refer the reader to see (Lee et al., 2018b).
uses the feature maps of the last hidden layer of convolutional neural net- works. To be exact, we denote the output of the 1-th hidden layer of the network f by fi for -= 1, .,.... L. where L is the number of hidden layers. For the (-th hidden layer, let fj be the i-th feature map of its output for Iy..., Mf, where M, is the number of feature maps of the 1-th
MAHALANOBIS-vaniin
2

---

Under review as a conference paper at ICLR 2020
Table 1: Performance of different Mahalanobis setting for DenseNet. 'R' is the abbreviation for 'resized'. The best and worst results are highlighted in boldface and underline for each setting.
TNR TPR 95%
AUROC
Delection Accuracy
AUPR((I)
ALPR((Ou))
MAHALANORIS-anl / MARILANOOISSaSemm
MALCOM (ours)
SVHN
89.2769479344
85.4779 98.4
82.4/8287
94.3
94.5755.9/99 95.9
99.02 94.9 99.3
CIFAR-10
Tinyln(R)
82.3/84.5 94.0
96.31917/ ////99779... /98.5
/ 94,6
96.2 12.1883./990
96.4 94.7 98.7
LSUN(R)
97.1 /
91.4/2.1/
97.2/9100
96.8/
SVHN
45.9745.87 64,9
83.57 93,4
71.6/114 86.6
71.776244 88.3
92.7 92.1 996.1
CIFAR-100
Tinyln(R)
82.0/83.4/ 85.3
95.7 94.3 97.1
/ /91.2
95.3 19.3 /97.3
95.91 95.4 197.0
LSUNIR)
84,5/87.7/ /8.22
96.6 96.6
97,5
91.0/2.0/ 92.4
96.8 95.7 97,8
96.1 / /9.6 96.9
CIFAR-I0 10
76,3796.9795 95.9
96.47 98.7 7 798.6
91.77 96.0 995.3
98.77 799669999999 99.3
88.2 95.0 994.7
SVHN
TinyIm(R)
71.37
100.0/ 100.0
96.0 99.9 99.9
91.8 998.9 98.9
98.6 100,0 100..0
86.9 99.6 9996
LSUN(R)
61.4/ / 100.0/ 109.0
95.1
100.0 99.9
91.0/ 99.39999999 99.2
98.3
100.0 100.0
82.3 99.7 99.6
hidden layer. Therefore, (z) is a 3D tensor of the size Mj x H4 x Wi and (a) is a 2D feature map of the size Hi Wi for any and I, where H4 and Wi are the height and width of each feature map in 1-th layer, respectively. For notational convenience, we define my as the global average pooling of the I-th layer, i.e.,
my (xx) -
me
where mean is the average function and is the concatenation operation. MAHALANOBIS- vanilla utilizes only the last hidden layer (i.e., the L-th layer), so it can be obtained by calculating class means and fied-covariance:
N
mL(xw))
Nk
Σ
(mL(wn) (mL) 1b))7"
N
where K is the number of classes and N₂ the number of the data of class k. Then, the out- of-distribution score function of MAHALANOBIS--ammi detector Kinv (also known as Mahalanobis distance) is defined by
Figy (x:")
min
1 - (0-T((((==)))) p.x)
In this equation, MAHALANOBIS vanilla computes the centered data for each class, then performs linear projection for the centered data by using the inverse of Σ. Practically, the inverse of empirical covariance matrix is approximated by (Moors-Penrose) pseudo-inverse, which performs the singular value decomposition (SVD) (Barata & Hussein, 2012). For this reason, computing the Mahalanobis distance becomes equivalent to computing the norm of the sample (i.c., the Euclidean distance from the origin) after the principal component analysis (PCA) assuming that the origin is the class mean, which removes the feature correlations.
2.2.2 DEGENERACY OF CHHALANOBIS-ASSEBBBL
From the observation that MAHIALNNBBSS-aanll has the power removing the correlation over feature maps. one may extend this concept to all the other hidden layers. To define the MAHA- LANOBIS LANOBIS-assemble assemble detector, we concatenate the average of feature maps (i.c., global average pool- ing) of all layers and similarly define the class means and tied-covariance:
mxx) : == mx)
K
pig
m( E
(m(xn) (m)(())) pTx)
N
N
Vn
Kuna (x:)
min (m(x") p TT (m(x") (ax)
k=....,K
We the term "assemble" to distinguish it from the "feature ensemble", which is the method using logistic regression in the original paper.
3

---

Under review as a conference paper at ICLR 2020
From the experiments, find interesting observation that the simply extended detector Kma significantly improve the performance in many cases; however, in some cases, its performance, especially AUPR(In), is poorer than Kov (see the first two columns of Table for more details).
2.3 NORMALIZED COMPRESSION DISTANCE
To address the degeneracy problem in MAHALANOBIS-asSeme we focus on hindering the loss of spatial information of the feature maps caused by the average pooling. To this end, introduce compression-based method. Before defining our method precisely, we need to state some notions briefly because its fundamental properties are originated from the concept of information theory.
2.3.1 KOLMOGOROV COMPLEXITY
The Kolmogorov complexity (or algorithmic complexity) is the complexity of an arbitrary object which can be represented string format. Formally, given an universal Turing machine the Kolmegorov complexity K of string is defined as the minimum length of the programs that are decoded into æ by U. That is,
where p is valid program for U}), where length(p) (p) is the length of the program p and (p)) is the result of decoded string with the program p through U.
K(a:) == min ((Un)
The conditional Kolmogorov complexity is the minimum program length for a given universal Tur- ing machine which already has the string:
K(æly) == min ((length(p))U((pp))) = =
is highly correlated with y. its conditional complexity would be low.
2.3.2 NORMALIZED INFORMATION DISTANCE
The normalized information distance (NID) is the distance between strings and y based the conditional Kolmogorov complexity. It does not need any prior knowledge so that it can be applied various tasks universally. The distance is described
NID(E, y) max y). K(y)=))
max (x) K(y))
turns out that NID is normalized metric et for arbitrary z.y, and
et al. 2004). which has following distance properties
NID æ. y [0,
(normality)
NID æ. y) 0 if and only if x
y
(identity)
NID æ. y) = NID x)
(symmetry)
NID(=, 2) NID y) NID 2)
(triangle inequality)
Unfortunately, the Kolmogorov complexity not computable, so NID also cannot be computed. Instead, the normalized compression distance (NCD) is used the approximation of Kolmogorov complexity in many tasks. Given an arbitrary compressor C, NCD is defined by
NCD(æ, v) min (C(ry), C(yz)) min (C(x), C(y))
max (C (C: C(y))
where C(x) is the length of the compression result of x, and zy is the concatenation of and y. Any off-the-shelf compressors can used as the compressor, including 7-zip, gzip, and bzip. The better compressor we use, the closer the NCD value approaches to the NID value (Cilibrasi & Vitányi 2005).
MALCOM: OUT-OF-DISTRIBUTION DETECTOR USING NCD
To resolve the degeneracy of MAHALANOBIS lsseebbee discussed in Section 2.2.2 propose the MAhaLanobis distance with COMpressive-complets pooling (MALCOM), which applies NCD measure the distance between the feature maps of convolutional neural networks.

---

Under review as a conference paper at ICLR 2020
NCD from a Fixed Sample
Global Average
Compressive- Complexity Pooling
Images
Map
Pooling
Images
Feetuuee Map
Quantization
Linearization
(a) Global average pooling
(b) Complessivee pooling
Figure I: The proposed compressive-compleii pooling captures the spatial information of feature maps into the final vector by using NCD, whereas global average pooling cannot make use of it.
3.1 TNCD
NCD can be generalized well to various objects because of its universality. Most previous work use the raw images (Lan & Harvey) when they calculate NCD, but inappropriate use raw images for many cases. For example, consider the following strings.
string a BABBABBBABBBBABBBBAAABBBBE string CCRACABECCHAAABBAABBAAAANNNN string c: ADACABACAABGGAAACBICAAAAA
In case that we only want to recognize the pattern of the symbol C, using NCD on this string is not useful enough. needs to look at the features that the classifier consider important. focus on such important features, introduce a translation T. In the case above, for example, T translates the symbol C' into and all other alphabet into
T(a): 1000000000000000000000
T(b): 10010001100000000000000000000000 T(e): 0001000100010000000000000000
Before the translation, string a looks much simpler than string or string e. Through the transfor- mation, we can concentrate only C' in the string b and string Formally, for the two symbol sets and Ωt we denote the translation by T E + S²" Given a transformation T, we define the translated and normalized compression distance (tNCD) by
tNCD(x, y: min [C(T())T())) C(T()) (CTT))IT) min (C(T()))
max (T())) (T()))
Note that tNCD between and y depends what the translation T focuses there exist lot of translations with different perspectives, æ can be described by various aspects
fix y and
3.2 COMPRESSIVE-
POOLING
We propose the compressive-complles pooling which utilize the spatial information of feature map. For each feature map fí. we consider h; := gf - fi translation where gi a quantization- and-lincarization, which discretizes the continuous values and then makes the quantized feature map a string. We first select a sample from Dimin- Then, for the fixed sample & and the test sample æ, we can calculate the tNCD value dj (æ; for each translation hi.
di 8) = tNCD(8, h) min (C(h)()))(()), C(h min [C(h)())) C(H(()))
max C (M(()) C (h((v))

---

Under review as a conference paper at ICLR 2020
Intuitively, for an image drawn from specific class, the distance dj becomes small if some aspects of æ corresponding to the translation hi are similar to and it would be large otherwise.
The compressive-omplexity pooling defined by fixed sample s with aggregating every feature maps,
(1 8) -1 (i s).
All the layers can be concatenated for x with respect 8,
p s) : (r
This can be interpreted the distance of æ from 8, considering the patterns of spatial information for each feature map. Figure Iillustrates the overall process of the compressive-complxxi pooling and how it differs from the global average pooling.
3.3 SCORE FUNCTION FOR OUT-OF-DISTRIBUTTI
To put it all together, we can obtain the mean vectors mn that encodes spatial information by con- catenating the mean pooling vectors and the compressive-compesii pooling vectors. We apply the Mahalanobis distance remove correlation between feature maps.
mmx m(x) ) p(x; 8)
ia = N
Σ Dn
m(*x), =
(m(xn) - (m)(a))))) - fix)
Dn
Based on the mean vectors, we define the final out-of-distribution score function k which is capable of accurately measuring the out-of-distribution score:
(MALCCO(x")) = min (mi(x² E - pāx).
k=I.....
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
For experiments, we follow the experimental setup mainly used by previous work (Lingg et al., 2018 Lee al. 2018b).
Models. For fair comparisons, we use the pretrained DenseNet (Huang et al. 2017 and ResNet (He al. 2016) provided by the previous work (Lee et al. 2018b According the authors, they use DenseNet with 100 layers, growth rate k=12 and dropout rate O, while ResNet with 34 layers and dropout rate O. Both the models are trained using stochastic gradient descent (SGD) with Nesterov momentum for classifying CIFAR-10, CIFAR-100 (Krizhevsky et al. 2009) and SVHN Netzer et al. 2011). For more details, we refer to the original paper (Lee et al. 2018b).
Dataset. We use out-of-distribution image datasets: TinylmageNet (Deng et al.
2009). LSUN Yu
et al. 2015), and iSUN (Xu et al., 2015). The in-distribution datasets have 10 classes except for CIFAR-100 which has 100 classes. CIFAR-10 (or CIFAR-100) has 50,000 training and 10,000 test images, while SVHN has 73,257 training and 26,032 test images. For out-of-distributio datasets, TinylmageNet contains 10,000 test images of 200 classes, LSUN contains 10,000 of 10 classes, and iSUN contains 8925 images. Liang et al. (2018 preprocessed TinylmageNet and LSUN by either randomly cropping of size 32 32 or downsampling to size 32 32.
Competing Methods. We compare the performance of our method with that of the existing meth- ods satisfying the two constraints (Section 2.T). which the independence with out-r--distiibutiio samples and exploitation of the model without retraining. As far as we know, there is only one method perfectly satisfying these requirements: BASELINE (Hendrycks & Gimpel 2017) using the
httpp://ghhuh..co//ohaaapppolppp Mahalanobis detector

---

Under review as a conference paper at ICLR 2020
Table 2: Ablation results on analyzing the effect each component: the concatenation of fea- ture maps from all layers (denoted by layer aggregation), global average pooling, and compressive- complexity pooling. AUROC used as an evaluation metric. The best results are marked in bold- face.
Method
Layer aggregation
Global average pooling
Compressive complexity pooling
CIFAR-10
CIFAR-100
Tinylm
LSUN
Tinylm
LSUN
BASELINE
94.14
95.47
71.62
70.84
MAHALANOBIS- MAHALANORIS aasembee
96.27
97.06
95.69
96.60
91.67
95.09
94.29
96.59
93.26
94,42
92.29
92.28
MALCOM
/ 95.47
96.14
95.92
96.78
(ours)
95.77
95.33
93.31
92.27
98.52
98.96
97.11
97.47
softmax output to distinguish out-of-distribution data from in-distribution data. In addition, (Section [.2.1]. al-
use the simplified version of (Lee et al.] (018b), i.c. MAHALANOBIS- though the original method uses the input preprocessing and ensembles the layers by introducing the hyperparameters. The implementation details about the compression of the feature map are provided in Appendix
Evaluation Metrics. We adopt the five performance metrics the same with the previous work (Lee) al. 2018b): true negative rate (TNR) at 95% true positive rate (TPR), under the receiver op- erating characteristic curve (AUROC), area under the precision-recall curve (AUPR), and detection accuracy. For the details, we refer to (Lee et 2018b).
4.2 RESULTS
Ablation study. We ablate the components of our method to show the interaction between the global average pooling and the compresiive-oomplxtty pooling. To this end, we use the DenseNet pretrained on CIFAR- 10 and CIFAR- 100. The out-of-distribution datasets are the resized TinyIma- geNet and the resized I. LSUN, and the AUROC measured as the evaluation metric. Table shows the results of the ablation study. The layer aggregation refers the method that uses the concatena- tion feature maps of all the layers. From the results, we obtain the following observations:
The performance of our compressive-complexit pooling is improved by concatenating the feature maps of all the layers, whereas the performance of the global average pooling becomes worse when it concatenates the feature maps of all the layers.
The compressive-comple only the feature maps of last hidden layer.
pooling is always worse than the global average pooling when
Using both the pooling techniques at the same time consistently improves the detection performance. It shows that each pooling provides different information.
Main results. The main results are summarized Table we repeat each experiment five times and report the averaged results. See Appendix D for more detailed results. (C) and (R) denote the dataset "resized" and "cropped", respectively. Our proposed method, MALCOM, achieves significantly better performance than other competing methods in most cases. Nevertheless, for minor cases, we observe that MALCOM with ResNet slightly degrades the performance on the resized Imayem and the resized LSUN. From the extensive experiments, we conclude that our compressive-complekie pooling successfully captures the spatial information from its feature maps, and helpful to accurately measure the confidence score for detecting the -attof-distribution data.
RELATED WORK
As deep learning has made remarkable progress in the past few years, many methods have been proposed to estimate the confidence of the model prediction beyond just enhancing the model accu- racy. Although the modern neural networks have achieved higher classification accuracy than in the
7

---

Under review as a conference paper at ICLR 2020
Table 3: Performance of different out-of-distribution detection methods using pretrained DenseNet and ResNet. For outo-f-distribttion datasets, 'C" and 'R' are the abbreviations for 'cropped' and 'resized', respectively. The best result is highlighted by boldface.
TNR at TPR 95%
AUROC
Detection Acc.
AUPR(Ia)
AUPR(Out)
ID
0OD
BASELINE/ MAHHALANOBIS-vmmm
MALCOM (ours)
SVHN
/ 93.4
89.9 /9.6/ 98.4
83.2/ 92.4 /4.3
83.7 99.5 99.99
94.4/99.0/
99.3
SVHN CIBAR-1N 100 CIFAR-10
TinyIm(()
65.8 74.6 100.0
95.4 /
90.1 88.4 99.1
96.5 95.8 /99,9
93.9/95.1
99.9
TayImn(R)
59.4 /9.3//2.3/ 94,0
94.1 / / 998.5
88,5 / 90.0 94,6
95.3 96.2 98.0
92.4/96.4 98.7
LSUN(C)
61.4/529/ 99.9
94.9 / 90,8 99.88
89.8 82.8 98.6
96.2/91.9/ 99.88
93.2 89.6/ 99.8
LSUN(R)
6 100.0
95.4 / 99.44 99.9
90.1 88.4
199.1
96.5 95.8 99.9
93.9 95.1 99.9
DenseNet
SVHN
2 64.9
82.6 / 85,9 /93.4
75.5 / 78.6 / 86.6
75.1 88,3
90.4/92.7/ 96.1
Tinylm(()
32.7/52.91 98.1
81.1 / 86.8 98.8
72.9 79.1 /96.6
82,9 85.9 99.2
80.0 /0.0/87733 98.2
TinXIm())
/ 85.3
7 95.7 97.1
65.7 89.5 /9.5
74.1 / /9.33
69.0
LSUN(C)
40.1 / / 95.3
85.4 / 60.6 / 98.4
76.7 / 56.9 /95.7
87.0 61.7 /98.8
84.4 / 60.5 97.2
LSUN(R)
16.4/84.5 87.2
70.8 / 96.6 97.5
65.0 91.0 92.4
74.2/ 96.8 / 97.8
67.9 96.1 96.9
CIFAR-10
69.1 / 76..3/ 95.9
91.8 / 96.4 98.6
86.5 /9..7 /95.5
95.4 98.7 / /99.5
83.5 / 88.3 99.77
TiyyIn(C)
79.0/ 64.8 100.0
94.8 95.0 100.0
/ 90.5 100.0
97.2 / 98.33 100.0
89.1 83.1 100.0
TinyIn(R)
79.7 71.3 100,0
94.8 / 96.0 199.9
90.2 / 8 98,9
97.0 / 98.6 / 100.0
88.9/ 86.9 99.6
LSUN(C)
76.7 70.6/ 100.0
93.8 / 95.33 100.0
88.8 89.7 100.0
96.7 / 99.22 100.0
87.4 86.8 100.0
LSUN(R)
77.1/614// 100.0
94.1 95.1 19.99
99.2//91.0 99.2
96.7 / 99..89333 100.0
88.0 82.3 / 99.6
SVHN
32.2 79.2
89.9 / 93,9 19666
85.1 89.1 191.
85,9 91.6 94.4
93.6 96.0 98.3
CIBAR-1 100 CIBAR-10
TinyIm(C)
49.1 /6 99.0
92.6 / 95,44 199.6
86,9/90.4 / 197.3
94,1 96.6 999.7
90.1 /93.3 99,6
TinyIm(R)
44.1 / 6993 / 89.7
91.0 / 95.0 98.3
85.0 88.6 9931
92.5 / 95.8 99.44
88. 3 94.0 98.2
LSUN(C)
50.1 98.1
93.3 / 94.3 99,5
88. 99.3 96.7
94,8 96.0// 99.6
90.8 99,4
LSUN(R)
/ 91.6
91.1 / 96.6 98.5
85.3 / 90.7 / 93.88
92.5 97.1 998.7
88.6 95.8 98.4
SVHN
19.9/43.2/60 61.4
79.3 / 89 4 192.6
73.2./ 82.0/ /8,44
65.9 / 8 887.2
8 96,4
ResNet
TinyIm(C)
34.7 88.8
75.8 / 85.5 99799
70.1 / 78.2 92.1
76.5 87. 97.9
71.2 82.6 97.9
TinyIm())
/0.3 64.2
77.1 78.9 92.5
70.8 72.0 83.9
79.7 81.8 /93.0
73.2 74.4 92.5
LSUN(C)
12..542.5/4 88.5
70.5 / 87.4 197.8
66.6 80.0 92.0
68.1 88.4 /97.8
66.5 97.8
LSUNR))
18.47 60.6
7 79.0 91.6
69.8 / 72.3 82.7
77.4 82.0 /22.4
71.7 /7.1/ 91.3
CIFAR-10
78.3 / 90.4
97.0/ 98.0
90.1 19.1 / /94.5
94.8 99.0 99.33
86.4 89.3 93.4
SVHN
TinyIm/C)
82.5/ 83.3 100.0
94.0/96.9/ 100.0
91.4/93.9 99.11
95.3 / 99.0 100.0
88.4 86.8 99.9
Tinylm(R)
79.1 / 84.5 97.7
93,5/ /9.2
90.4/99.1 / 19644
95.4 98.9 99.7
86.9 89.1 /
LSUN(C)
79.6 88.1 100.0
93.1 / 99.9
90.5 93.2 988.5
94.5 / 99.00 100.0
/ 99.7
LSUN(R)
/4.3//
91.55 99.0
/8.9/9199 / /9622
93.8 98.7 99.7
84.6 / 85.9 97.1
past, the confidence calibration in neural networks is poorer than decade ago, i.e., the discrepancy between the prediction score of the neural network and the ground-truth class probability gets worse compared with that of the past (Guo et al. 2017). To make matters worse, most neural networks, es- pecially for non- -ay-Baaeiinn, have no measure of uncertainty, which hinders reliable decision-making just by the prediction score in many practical areas (e.g. medical diagnosis, self-driving car). Thus, there has been much work to attain the measurement for uncertainty or confidence calibration other than classification accuracy (Guo et al.] 2017 Lakshminarayaaan al. 2017 Diang et al. 2018)
Even though Bayesian deep learning provides uncertainty properly, several methods were recently proposed estimate uncertainty of non-Bayesian deep neural networks. Especially, it turns out that some regularization techniques of neural networks (e.g. dropout, batch normalization) are inter- preted as approximate Bayesian inference so that uncertainty can be captured in conventional deep neural network (Gal 2016 Teye et al.) 2018). These work are significant in that deep neural net- works are not required re-model retrain once they have been trained by using dropout or batch normalization even if they do not consider the uncertainty at train time. MALCOM is inspired mainly by these work because can be applied more practically to neural networks which already have been trained and deployed.
Most previous work of out-of-distribution detection can be divided into two main approaches: in- troducing parameter and defining new objective function. An prominent example of the former is Liang et al. (2018) (ODIN) which suggests two hyperparameters for softmax output to a boost in performance: temperature scaling and input preprocessing. Since ODIN, most studies using soft- max output use these two parameters. Vyas et al. (2018) tried to solve the problem with ensemble of classifiers by dividing the training data and adding loss by defining entropy for in in-disrrbutton and out-of-distributt as a supervupee learning method, and used two parameters to improve performance. Our closely related work, Lee et al. (2018h) focuses on the internal features of the
8

---

Under review as a conference paper at ICLR 2020
CNNs, while it uses some hyperparameter logistic regression for aggregating all feature maps and input preprocessing.
CONCLUSION
This paper proposes MALCOM, which accurately measures the confidence score of test sam- ple for out-of-distribution detection, without using any hyperparameters and without retraining the model. MALCOM addresses the limitation of the existing Mahalanobis distance-based method that cannot exploit the spatial information of image feature maps. We introduce the compressive- complexity pooling based on NCD to consider the spatial information of the feature map when measuring the distance from class means. Our extensive evaluation image benchmark datasets shows that the compressive-complexii maps, and it improve the performance compared to the global average pooling. In addition, by using the both the pooling techniques simultaneously, MALCOM achieves the best performance for of-istrributio detection.
pooling leverages the spatial information of the feature
REFERENCES
João Carlos Alves Barata and Mahir Saleh Hussein. The moore-penrose pseudoinverse: A tutorial review of the theory. Brazilian Journal of Physics, 42(1-2))116-165,, 2012.
Petr Berek, Michal Prilepok, Jan Platos, and Vaclav Snasel. Classification of eeg signals using vector quantization. International Conference on Artificial Intelligence and Soft Computing. pp. 107-118. Springer, 2014.
Rudi Cilibrasi and Paul MB Vitányi. Clustering by compression. IEEE Transactions on Information theory, 51(4):1523 1545, 2005.
Dinu Coltue, Mihai Datcu, and Danicla Coltue. On the use of normalized compression distances for image similarity detection. Entropy, 20(2):99, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Yarin Gal. Uncertainty in deep learning. PhD thesis, PhD thesis, University of Cambridge, 2016.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017. Conference Track Proceedings, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust not trust classifier. In Advances in neural information processing systems, pp. 5541-5552, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

---

Under review as a conference paper at ICLR 2020
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413 -4-161 2017.
Yuxuan Lan and Richard Harvey. Image classification using compression distance. Citeseer.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrate classifiers for detecting out-of ofttr-diistrbuttion samples. In 6th International Conference Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings, 201 8a.
Kimin Lec, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Process- ing Systems, pp- 7167-7177, 2018b.
Ming Li, Xin Chen, Xin Li, Bin Ma, and Paul MB Vitányi. The similarity metric. IEEE transactions on Information Theory, 50 1)):2200-332, 2004.
Jan- Yie Liang. Chih-Sheng Chen, Chua-Huang Huang, and Li Liu. Lossless compression of medical images using hilbert space-filling curves. Computerized Medical Imaging and Graphics, 32(3): 174-182, 2008.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribuiin image detection neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions information theory, 28(2): 129-137, 1982.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Joel Max. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6(1):7-12, 1960.
Jonathan Mortensen, Jia Jie Wu, Jacob Furst, John Rogers, and Daniela Raicu. Effect of image linearization on normalized compression distance. In International Conference on Signal Pro- cessing, Image Processing, and Pattern Recognition, pp. 106-116. Springer, 2009.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http://uflal. stanford.ede
housenumbers/ wwwpmmberrnnissp1111.11 nips2011 housenumbers, .df
Armando Pinho and Paulo JSG Ferreira. Image similarity using the normalized compression distance based on finite context models. 20/ 18th IEEE International Conference on Image Processing, pp. 1993-1996. IEFE, 2011.
LM Spracklin and Lawrence V Saxton. Filtering spam using kolmogorov complexity estimates. In 21st International Conference on Advanced Information Networking and Applications Workshops (AINAW'07) 07), volume 1, pp. 321-328. IEEE, 2007.
Mattias Teye, Hossein Azizpour, and Kevin Smith. Bayesian uncertainty estimation for batch nor- malized deep networks. In International Conference on Machine Learning (ICML), 2018.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke. Out-of-distribution detection using an ensemble self supervised leave-out classifiers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 550-564, 2018.
Terry A. Welch. Technique for high-performance data compression. Computer, (52), 1984.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv: (504: 06755, 2015.
10

---

Under review as a conference paper at ICLR 2020
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint wwX.v:1506.0333. 2015.
A IMPLEMENTATION DETAILS
Lempel-Ziv-Weike complexity. Although there are many efficient off-the-shelf compressing al- gorithms (e.g. 7-zip, GZip, and BZip), we do not consider them because strings do not need to be actually compressed. We only need to obtain the lengths of compressing results to calculate NCD. Because of its computational efficiency, we adopt the Lemple-Ziv-Weler (LZW) coding scheme (Welch, 1984), which enables to compute the length of string in ((n) and to accelerate the com- putation by using GPU.
Hilbert space-filling curve. There are several choices to linearize the 2D feature maps, including simple row-wise (or column-wise) lincarization. Among them, lincarizing the images by Hilbert curve turns out to be effective when using NCD for images (Liang et al. 2008 Mortensen et al.) 2009 Coltue et al. 2018). For this reason, adopt Hilbert curve linearize the 2D feature maps.
Lloyd-Max quantizer. We quantize the feature maps by using the Lloyd-Max quantizer (Lloyd 1982 Max 1960), which effectively minimizes the information loss. We fix the quantization level to 4, which is known to perform well for images when it used with our compression scheme (i.e.,
I I.ZW) (Pinho & Ferreira 201T Coltue al. 2018).
B VALIDATION ON OUT-OF-DISTRIBUTION SAMPLES
In this section, we compare our proposed method with the two additional baselines: ODIN (Liang et al. (2188) and MAHALANOBIS (Lee et al. 2018b). Both of them utilize out-of-distribution sam- ples find the optimal hyperparameters values. this sense, they do not satisfy our first constraint describing that any test distributions should not be assumed, discussed in Section 2.1 Neverthe- less, we loosen this constraint and report the results to show relative the performance compared to the state-of-the-ar methods.
ODIN. ODIN increases the maximum value of the softmax score using temperature scaling and input preprocessing, and uses this softmax score as the score function. Let SL+1 E RRKK be the outputs of the last fully-connected layer of trained model f.c., fL+1 = [fi+1 .... fk , the layer right before applying softmax. Then, with the two hyperparameters T, and c,, respectively for the temperature scaling and the input preprocessing, the out-of-distribution score function is defined
follows:
(x(2:Ta)
Vz [2), (",,".,,, R
Σ/=1 ez//Th
S(x; To) max σk (fr. J1+1 (2); k=I..KK
KODIN RoDiN(>"; co, co, = - (xx " + c.sign (V log S(x";T))) To).
where sign is the sign function.
Mahalanobis. MAIIALANOBIS similar to our proposed method in that it calculates the Ma- halanobis distance by using the global average pooling feature maps from target layer. The main difference of MAHALANOBIS from MAHALANOBIS-vame of multiple Mahalanobis scores, which obtained from more than one layers, and the weight for the /-th score is introduced as the hyperparameter a addition, MAHALANOBIS also adopts the input preprocessing technique with the hyperparameter cm, which plays similar role to ODIN. The
is that it uses the weighted sum

---

Under review as a conference paper at ICLR 2020
Table 4: Performance of different out-of-distribution detection methods that utilize out-of- distribution samples for the validation. For out-of -ifto--diitribution datasets, "C" and 'R' are the ab- breviations for 'cropped' and 'resized', respectively. The best result highlighted in boldface.
TNR at TPR 95%
AUROC
Detection Acc.
ID 0OD
ODIN MAHALANOBIS
MALCOM (ours) MALCCM
SVHN
86.16/ 99..5/ 93.67 95.07
95.51 / 98, 98.52 98.84
91.43 /1/33/93.3 / 72 /9,45 95,05
CIFAR-I1 10
TinyIm(C)
99.99 99.53
99.11 99.899922// 99.92/ 99.89
95.36 98.93 99,17 99.04
TinyIm(R)
9292.41 99.00 94.38 95.41
98.51 98. 98.67
99.05 94,74 99..3
LSUN(C)
91.08/ 98.87 99.79 / 99.11
98.21 99.63/ 99.78 99,79
94.02 98. 62. 98, 65 98.66
LSUN(R)
96.22 / 97.04/ 95,79 96.89
99.23 21 98.98 99,25
95. 95.66/ 96.14 95,44 96,11
iSUN
93.87 / 95 / 94.07 95.64
98.84/ / 99. 98.75 99,09
94,56 / 95.14 94.61 95,41
SVHN
70.65 / 87.23 64.80 85.79
93.82/ 97.08 93.47 96,27
86.58 99.23 86,76 90.68
DenseNet
CIPAR- 100
TinyIm(C)
72.51
999.88/ 1989800 98.65
94.35 99.78 98.82 99.54
86.93 98.96 966600 98.288
TinyIm(R)
42.64
/8838/ 85
88.36
85.22 997.4/333 97.17 97.51
77.04 92.18 991.43 92.117
LSUN(C)
81.18
99.48 95.55.555 51 98, 67
96.49/ 99.76 98.31 999559
90.17 98.86 / 79. 98.114
LSUN(R)
41.21 / /92.13/ / 88 (58.54/ 92.44
85,45 97.83 97.67 97.95
77.06/ / 93.65 992.89 93,76
ISUN
37.41/ 88.43 85.28 89.26
84,06 / 97. 97.26 97.60
76.04 / 92.22 / 99.61/ 92.44
CIFAR-10
71.73 / 96.53 95.40/ / 61
91.37
99.02 / 19.856 98.57
85. 80 96.32/ / 95, 44 99.02
TinyIm(()
81.76 / 100.00 100.00 100.00
94.84 / 100.00 100.00 99999
89.64 99.99/ / 199,999 99/99//97 99.977
SVHN
TinyIn(R)
84.06 99.57 99.96/ 9
95.11 99.88 99.87 99.86
90.35 98.93 98.93/ 98.84
LSUN(C)
74.41 / 100.00/ 100.00 100,00
91.89/ 100,00/ 100.00/ 100.00
86.38/ / 99.98 99.99 99.98
LSUN(R)
81.12/ / 99.65 100,00 99.61
94.54 / 99.90 / 19.88// /99999985 99.88
89.16 / 99.29 / 99.226 99,16
ISUN
82.15 / 99.64 / 99.99 / 99.58
94.68 99.89/ / 99.88/ 99.88
89.62 / 99.31 / 19.17 99.18
SVHN
86,55 / 96.19/ 78.61 / 16.15
96.65 99. 14 / 96.55 99.19
91.08 95.76 / /91.13 95.65
CIFAR-II 10
TinyIm(C)
74,76 01.6//99.85 99.85 988 86 99.90
94.58 99. 97 / 99.559 99,98
87.39 99.29 / 97.20 99.50
TinyIm(R)
72.51 997.8/
552 97.64
94.04 99 99. 99,52
86.48 96.32 93, 96.51
LSUN(C)
72.08 99.86 97. .9.95 99.89
93.80/ 99.97 999.47 99.977
86.88 99.30 /6.56/ 99.43
LSUN(R)
73.83 / 99.74/ 91.37 98.99
94.14 99.69/ 98.52 99,72
86.69 / 97.49 47.49/ 93.76 97.55
iSUN
73,20 / 97.96 91.02/ 98.15
94.02 / 99.999... 52/ 98.45 99.56
86.49 / 96.74 93.64 99.93
SVHN
62.75 / 99241/11// 61 91.41
93.94 18.18 /2.60 / 97.98
88.05 / /93.55 / 84.62 93.30
ResNet
CIFAR-I 100
TinyIm(C)
30.82 / 99.799444 74 / 88.70 99.77
79.75 99.999999 91 / 99.881 /99,22
73.51 / 99.23 92.04 99..9
TinyIm(R)
49.19 89, 44 64.61 92.31
87.62 97. 92.60 98.38
80.11 92.74 84.04 /93.75
LSUN(()
37.47 99.64/ 88 .21 99.76
76.17 89.89///8 97 67/ 99.91
71.03/ 99.05 91.78/ 99.31
LSUN(R)
45.59
/ 80 61.51 92.89
85.64 98. 27 91.77 98.32
78. 26 93.93 62.94/ 93,97
iSUN
45.26
89.77
61.63 90.94
85.45 / 97.85 91.54/
98.04
78.41 / 92.87
82.62 93.16
CIFAR-10
79.83 / 97.62/ / 90.26 97.51
92.09/ 99.31 98.00 99.35
89.44 / 96,91 / /9..4 99.87
TinyIm(C)
84.69 / 99. 97 100.00 99.97
93.75 / 99.98 / 99.99/9999999 99.99
91.23 / 99.94 99.113 99.96
SVHN
TinyIm(R)
82.10/ 99.
56 99.74
91.99 99.89.98888 99.19/
89.35 99.08 99.34
LSUN(C))
82.41 / 99.94/ 100,00 99.95
92.72 99.98 99.90/ 99.98
90.28 / 99.91 998858 99.93
LSUN(R)
77.69 / 99.84/ 96. 96 99.84
89.44 99.89/999//99 99.0099000 99,91
87,35 / 99.52 96.22/ 99,53
iSUN
79.10/ 99.79 24.24/ 99.78
91.32
/ 99.93 99.10 99.92
89.22 / 99.48 96,28 / 99.36
formulation of MAHALANOBIS is given by
1 mx(x),
p = NR.
Σ N
(mi(xn) (m((())))) (e )T
Mi(x): =
km min (m)(x) (p..))
T Ej
(m,(z) - pi),),
RMAHALLNOBBS (x: Olmy (m)
Σ
o Mi (x" + emsign (VM((x)))
where my is the global average pooling of the I-th layer defined in Section 2.2.1
MALCOM+ We build MALCOM+ by simply extending our proposed method to use out-of- distribution samples for the validation. It uses the weighted sum of the MALCOM scores obtained
12

---

Under review as a conference paper at ICLR 2020
from multiple layers, similarly to MAHALANOBIS.
m.((x) (x) (1( 8),
Mi, =
Nk Σ (x
1
Σ1 1
(m)( ((i((()))))))) )
Mi(a): = min (mi( - -in,))) 77 "0 (mi( p,),)),
k=l...,
RMALCOM- a.) Mi (x"),
where P: is the compresiivecomplexity pooling of the 1-th layer as defined in Section 3.2 We remark that MALCOM+ does use the input preprocessing technique, it does not need to compute the gradients for backpropagation, which eventually reduce its computational cost for the inference.
Experimental settings. For each method, the hyperparameters are adjusted by using 1000 in-distribution images and 1000 out-of-distribution images from the test dataset; this experimen- tal setting is exactly the same with the setting in (Lee et al., (018b). Specifically, for ODIN, we uses all the 2000 images to tune the hyperparameters To E {1, 10, 100, 1000} and 6 {0, 0.0005, 0.001, 0.0014, 0.002, 0.0024, 0.005, 0.01, 0.05, 0.1, 0.23. In case of MAHALANOBIS, we split 2000 images into two groups; one of them used to train the weights am by lo- gistic regression, and the other is for the validation of the input preprocessing hyperparameter em E {0.0, 0.01, 0.005, 0.002, 0.0014, 0.001, 0.0005). For fair comparisons, MALCOM+ uses the 1000 images obtain weights ar which the same with that for Om- All experiments repeated three times and the averages are reported.
detection methods when out- Results. Table 4 shows the performances of hyperparameter values. We observe that of-distribution samples are used to find the optimal MALCOM+ shows slightly better performances than MAHALANOBIS when ResNet is used, even though MALCOM+ does not utilize the additional technique (i.e., input preprocessing). Our pro- posed method, MALCOM, does not use -fitt--dist samples for the validation, but it con- sistently outperforms ODIN that utilizes out-of-distibution samples; sometimes it even performs the best. Specifically, in case of CIFAR-10 for in-distribution and the cropped TinylmageNe for out-of out-of-distribution MALCOM achieves the best performance among all the other methods with DenseNet.
C VALIDATION ON ADVERSARIAL SAMPLES
In this section, we report the performance of out-of-distribution detection methods, validated by generated adversarial examples. The adversarial examples refer the images that are hard to be distinguished with the original images by human eyes but misclassified by classifier. Lee et al. (2018b) suggested to tune the hyperparameters by using generated adversarial examples rather than dusto--distribution samples. Although its generation process takes much cost, the validation on the adversarial examples generated from the training dataset also satisfies our constraints in Section 2.
Adversarial examples. We generate adversarial examples by using FGSM (Goodfellow et al. 2013). It generates adversarial examples by adding noise to the input images the direction of the loss gradient. Given a loss function L(f: y) and data with labels (æ), ys) E Dtrains the FGSM constructs adversarial examples as follows.
z = 24 + sign (VL(f (x:) :y:))
(1)
where et is the magnitude of noise. As in Lee et al. 2018b), we set 6f = 0.0525 for DenseNet and c 0.0625 for ResNet. The images that are still correctly classified after applying FGSM are excluded. The generated images are assumed to be diitt--iiittiiuttio samples and the images from the training set used for in-distribution samples.
Experimental settinn We use the same competing methods defined in Appendix B ODIN (Liang et al. 2018) and MAHALANORIS (Lee et al. 20186). Since do not assume any test distributions
13

---

Under review as a conference paper at ICLR 2020
Table 5: Performance of different out-of-distribution detection methods that utilize adversarial samples for the validation. For uutto--distribution datasets, "C' and 'R' are the abbreviations for 'cropped' and 'resized', respectively. The best result is highlighted in boldface.
TNR at TPR 95%
AUROC
Detection Acc.
ID
0OD
ODIN MAHALANOBIS
MALCOM (ours) / (ALCOM++(OUI))
SVHN
38.68 82.66 / 94.03 / 95.65
88.71 96.55 98.56 98,93
82.72/ 99.02 94.55 95,43
CIFAR 10
TinyIm(C)
88.78/ 99.08 / / 99.49
98.14/ 99.73 99,91 99.86
93.01 / 58.51 99.17 99.13
TinyIm(R)
99.22 /5.20/92.23 99.900 95.59
97.44/ 98.38 98.61 99.05
91.75 99.32/ 194,49 995.38
LSUN(C)
78.19 79..76 99.87 99.16
96.58
99.38 99.83 99.77
90.84/ 00.84/97.36 97.36 / 98.72
LSUN(R)
91.20 / 95.47 / 95,43 96.71
98.53 98 98.95 99.26
93.87 95.48 /95.25 96.09
iSUN
87.76 92.40 / 94.01 95.65
97.99 98
98.66 99.11
92.53 94.34 (94.59 995.51
SVHN
39.49 / 66.31 / 63.31 82.65
88.21 / 92 93.09 95.47
80.72/ / 85 48/ 86.32 89.73
DenseNet
CIFAR 100
TinyIm(C)
71.70/ / 96.68 97.93/ 97.01
94.15 98.85 98.81 / 98.96
86.73 96.70 96.52 97.11
TinyIm(R)
43.14 81,30/ / 85.79 88.65
85.34 96.39 97.12 97.57
77.19 90.74 / /9133 92.23
LSUN((C)
80.10/ 94.61 99496 95.95
96.31 / 11 98.30 / 98.51
89.82/ 26 /95.49 96.28
LSUN(R)
41.54/ / 86.75 / 88.43 91.70
85.71 97.04 97.59 97.94
77.36 92.05 92.68 93,48
iSUN
37.92/ / 81.25 / 85.65 90.04
84,29 96.37 / 99.27
97.75
76.16 90.67 / 991655 92.66
CIFAR-10 10
69.14/ 96.02 9 94,99
91.83 / 98.78 / 98.63 / 98.51
86.54/ 96.00 / 995.62 95.23
Tinylm( C)
79.03/ / 100.0 / 100.0 / 100.0
94.79 100.0/ / 100.0 100.0
89.89/ 97 99.98 99.97
SVHN
linyIm(R)
79.73 / 99.50/ / / 99.44
94.76/ / 99.86 99.87 / 99.85
90.21 89.89 98.94 98.83
LSUN(C)
76.71/ / 100.0/ / 100.0 / 100.0
93,76/ 100.0/ / 100.0 100.0
88. 76 99 97 / 99.98 99.97
LSUN(R)
77.07 / 99.59/ / 100.0 / 99.49
94.07 / 99.99 / 99.89 / 99.85
89 16 99. 18 99.23 99.07
iSUN
78.33 / 99.57 99.99/ / 99.48
94.39 99.88 99.88/ 99.85
89.69 19.19/99199 99.20 99.06
SVHN
34.04/ / 54 / 80.54/ 94.47
86.17 / 93.92 / 96.80/ 98.84
78.43/ 86,94 / /91.7 94,79
CIFAR-II 110
TinyIm(C)
69.88/ / 99.57 / 98,88 99.82
94,41/ 99 90/90/ 99.58 99.96
87.65/ 98 68 19..18 99.44
Tinylm(R)
67.43 / 90.98 / 89.62/ 96.74
93.68 / 98.27 / 98.25 99.31
86.52 93.52 99.12 96.03
LSUN(C)
66.28 / 98.93 / 98.04 / 99.71
93.58/ 99.79 / 99.48 99.94
86. 73 97.88 96.69 99.22
LSUN(R)
68.63 / 94.91 91, 97.84
93.77
98 87 / 98.51 / 99.48
86. 70 95.03 / /93.75 / 96.75
iSUN
67.96 / 92.24 / 91.16 97.02
93.71 / 98.43/ / 98.44 / 99.33
86.64 93.91 / 193.66 96.21
SVHN
11.63 / 41.49 63.22 82.49
72.48
81.01 / 193.01 / 99.29
68.44
73.38 / 85.02/ 89,95
RosNet
CIFAR- 100
TinyIm(C)
24.12 / 94, 89.34 98.73
78.48 98 08.08 / 9 99.30
71.98
94.72 / 92.337 97.88
TinyIm(R)
32.45/ / 17 / 64.04 80.46
82.25 / 91.28 / 92.44 96.26
74.24 84.70 83.80 90.86
LSUN(C)
11.47 / 88. 15 / 89.53 98.23
64.61 / 99.07 / 99.998 / 99.38
62.51 / 91.86 19..44 97.22
LSUN(R)
30,00/ / 46.47 / 60.57 75.78
80.57
88.07 / 91.51 / 95.13
73.02 81.76 82.60 89.31
iSUN
29.52 / 47.18 61.04 74.96
80.77 88.79 / 91.37 / 95.23
73.31 82.25 / 82.40 / 89.39
CIFAR-I0
78.28 / 97.80 / 90.42 97.21
92.95
99.38 / 98.02 99.29
90.05 97.21 / 99,46 / 96.73
TinyIm(C)
82.52 / 99.97 / 100.0/ 99.99
94.03/ / 99 99.98
99.98
91.42 99 95 / 199.11 99.96
SVHN
TinyIm(R)
79.05 99.70 / 97.69 99.69
93.53 99.90 / 99.2999222 99.90
90.42 99.08 / 196.99 99.09
LSUN(C)
79.63 / 99.95 100.0/ 99.96
93.06 99.98 90 99.98
90.53 99.94 998.57 19.95
LSUN(R)
74.4S 99.72 / 13. 99.72
91.53 99.91 / /9609/9999999 99.91
88 .3.94/ 99.29 296.23 19999 ,27
iSUN
77.05 99.71 / / 99.71
92.25 99.91 /9 99.91
89.73 99.25 / 196.34/ 19.118
in this experiment, we use the whole training samples and adversarial examples for the validation. The only difference of the experimental setup from Appendix BI that we split the validation dataset into 80:20 for training the weights aa (or am) ) and determining the hyperparameter Em. respectively. All experiments are repeated three times and the averages are reported.
Results. Table 5 shows the detection performances of MALCOM, MALCOM+ and the other competing methods when the generated adversarial samples are used for the validation. Our methods (i.e.. MALCOM and MALCOM+) beat both ODIN and MAHALANOBIS in most cases, except for the case that SVHN set to in-distribution. This result strongly indicates that the compressive- complexity pooling is still helpful and complements the limitation of the global average pooling, even in the situation where adversarial samples are available. In addition, we perform an abla- tion study to see how much each component affects the final detection performance. In order to control the effect of the input preprocessing, we exclude MAHALANOBIS and instead consider the method using the weighted sum of the scores computed based on global average pooling. As pre- sented in T6 the method using all of the components consistently performs the best, similarly to the results of the previous ablation study (Table 2. From this observation, we conclude that
14

---

Under review as a conference paper at ICLR 2020
Table Ablation results analyzing the effect of each component: the weighted sum of the scores obtained from all layers (denoted by weighted sum), global average pooling, and compressive- complexity pooling. AUROC used as evaluation metric. The best results are marked in bold- face.
Method
Weighted sum
Global average pooling
Compressive complexity pooling
CIFAR-I0
CIFAR-I00
Tinylm
LSUN
Tinylm
LSUN
BASELINE
94.14
95.47
71.62
70.84
MAHiALAN
96.27
97.06
95.69
96,60
V
93.26
94.42
92.29
92.28
V
95.47
96. 14
95.92
96.78
MALCOM+ (ours)
98.83
99.17
97.43
97.91
95.09
94.61
94.21
93.39
99.05
99.26
97.57
97.94
the compressive-complxity pooling is effective to detect out-of-distribution samples based on the Mahalanobis distance.
D MORE RESULTS
Table 7 shows the detailed results for out-of-distribution detection. To perform further analysis, we test the synthetic noise datasets which in Liang al. (2018).
15

---

Under review as a conference p paper at ICLR 2020
(000 769 6 /7 07.55000000
90.200000
-00 070 (00) /99
: 1969-801 / (99000111
/ (201000 /99 AA
/80 57000000000 /O /50 5973000
196
81 7L70000 / 100 HO0-OOOD / EMIARRII
79 5455000 100 100 Eom
/92 7000000 0OD /966 9611122
WOOTON / /0% SXTITND
/97. /7.1
99 120 HOY SEERIID
63 63.9800000 99 550.000 /265 9654000
100 HOOAOI / SORAROIOO
47 47.0440 OS / 100 ANLIOO / SORDR (SM
ANE TARAAO T / BEDRONIN
19 09 D / TI / 99
50.58000
AN STO (OLDO) /O) 09.
99.91000)
A SNO 99961
/99
SHOMOL 1OMARO
IS 33 000 (5) 100000
R3 23 IS STO MOTERN / 11.1500
F1S9U00 -0X SNO / /9 9X.5900
5 .90 WNR 11 /
00000 7 2 20 19
/ S4-44
/24 TAR / SETEEA
1 SRTT
/ 6E.E
ARE MODOO AND PIR 100 /6 eiva
F 95 SO000 20 TIIIIT SONEAAA
760.000 /99
NO00000 /00 SNO TOTTO /6
/S
(67
/
Table 7: Performance of different out-of-d difdiisrrbbttoon detection methods. The best result is highlighted in boldface for each setting.
AUPROOU
9411 941200000 29 5460000
196 2201000 000001
80 00000000 IN 2NO
68 68.50000 25 NY15000
/60 10055O00
IN TH0 NO0000
2E 270
BR 270000 . ONI
. ONO
-00 NON
06 2400000 AS 2X
A POO
an 00000 /88 SON0
87 $7.420000 020000 IN
66 66.570 TO A9
1000000 7
94 04.000000
93 93.8900
92. 02.3900.000
03 93.24
92.490000
71370000
90. 90.
84 44.8%000
67 67.8700000
39 JAGO
83 DENOO
BR 00000000
82 87.95500000
87 87.2200000 DO
N0.700
00
87.20000
89 282700
96. 150000
12.50000
23.50000
11 21210000
5477
O
1770000000
R4 2420
0% 1517000 I
71
39.3700.000 722. 180 (2000000000 62 22.22.0)
/94 540-00))) 95-87 95-87(1117)
56 96-4700.00 /95 70.77000000 00) (99-2/00001)
/96. 15..150000 15:0.000 9 77.99%00770
/91 JL.S..... (.00000) 99999999 99.84 (.02))
(0) 99..2.0.200
96 96.3510.000 /966 92,90000 00) 0R 98.52/0.00
95 05.28000011 A 1ONLAOO (00 109, (0.00)
TORLOO 100,00000) 190 190.000.000 900
78 75.4600-001 7 755 ORRO OR S00000
75.0910.00) / 11.6500000 28.29 29
82.59H0 000 /S5 05..000 (0)) 999 19.16/0011 01)
74.09 74.0990.00) 195 320 (7.3990.7)
98.33005)
74 15.1000000 996 7W0 OOV 97.X3 (005)
74.5700.00 000 /96 340.00 DO 17.7 97.54/000)
TORLNO 00) 100000000000000 ((0.0)
100,0 100.0
1988 TRO DO 99 99.52.0.00 (.0)))
29.6100000
97. 17.1910.000 ON) 19% 380 001? 100 (00000
UF 97.0400000 (N) 9 S3.. 000 99 99.95 (00)
(00
/9R SN AOD 999 (N))))))
07. 07 17(0000000 198 TWO 1XS7 99 (60000)))
(0000)
62. 62,280-3 23) 199 TYCLOLS 100,000 9N)
100.000000
560 85 SSS ISGELOO 79 89.38
(.17)
X5 555000 (0) 191 S6O (XX) 194 94.3
OO 299 (0666/091)
04 OND 195 7W0 00) ON (11001
94,7800 (00 / DHO 00) 99 1S550000
(1)
L10.00 98.6610011
197 200 OO) /9 08.73003)
100.00 TOOL0/L.00
MOO 100 (0000)
784310007 7S ATO DOU 75 35500
ON /SS SHO 0O) Y 87 27220035
(6
76 764700 (00 187 1O0. (0) 97.94
(0)
ISL. 18000 93.030.112
68. (10.00000 /88 420 0O) .97 07.
79.550 00) 1S5 0600 000 192 0.078
08)
78. 78.14 14 195 WALLDSS 1000000000
000000000
(00)
( 796 VANCI 1007 99 (1000)
KOO 2990000 2000
999 99.72/000 (210011
999 999960000
000 999 0966609111
F9R (OOO00 99 (227300000
SAOAAA 100 100.0 (000)
(M))
92.230011 /08- 401 DOY (00.000000)
(00)
96.65 198 240-000 100,0
OR9) 100
199
AUPRIn)
86 86.90000000 (00) AGI 700 00)
57.00000000000000000000 57.0000.09) I TONLANLLOO
54 (3 /98. 2000 DO
79/6210 22 2T 250 02.02
126 5909
08 585700000 I 190 DENO
III AST 66N
198 SNNO
95 (1027100000 99% 9HO ON
94-4500.000 / /90 1000
65
19% WWO.GOOAA
197.
199
198
83 X3-73000000
95.5 95.3210.00)
96. 96.1510.000
96.510.000
04 94.220002)
69. 1710.09
9S 95.57
92. 42.4000.00
92.4500000
03.0800000
S53810
790900000
4 470000000
38-2400
04 84.770000
04 94-500.000
95.3700.000
95.1 95.110.000
03.70/0000
04 58.000000
93.22(0.25)
cours
89.37
03.75
58
BASELINE/ MARALAN ANOBIS wan.eaanllle MALCOM
22.02000000 65 560 0007 01 92/0 75)
X3. (310 /92 92.430 (10) 94 300.0)
90 00,07700000 /88. 18--110 (ND) 99 1P.09900000
88 5505000000 /89 980 (10) 94 (6.550027)
89 19.8N0.000 00) /22 122.7R0. (0) 98, 5..5990211
16.3600 00) 95 4200.66)
89 (9.2500000) D0) /90 390 00) 94 64.600017
92. 3..38000 04) /TOL WILLNOO (0)) 100. (10.00))
91 8710 1000000 00) 100, 0H0.00)
6% 7000000) 52. 780 IND 50 06TONU)
7786 P8.60 DND S8 15.60N0555 56
72. 72.0K0. 00) /99 /290500 00) 96. 62000)
550 ON) 91. 91.2410.12 (0
76 16.2210 DO) 56 8800 OO) 95 7000.9)
65 OS 00) / IIRO (ROO.0)) (I) 92 22-42/0.112 12)
99. 42:0.4))
IND (NO.0))
39 39.9660000 / 1000 DN AOWY 100 100 (H0000
26 (40001)) /91 /917100 (HO) 99. (30522
86 12.3200 5200000 /91 A5O IND/ 96 05/0066
89 89ND DO) /90 50000 OO) 99 9W(0.0)
50 210.000000 /91 78.7 7800000 (0) 98. 9 (1002)
7400 0OL 99 9800000
99 21.010011
80 19.69N0 0O) /91 7000 GOD 99, 17(004)
65 0900 /%6 2900 (3T 100 (10.00
T6 40-0 TO /93 92.0 060 100. (VONN))
30 3IK OO0 / 780 DOTT ST. SM606E1
XS (M)) NO0 TRO (OD) VI. PL.1N0 (.N.0111
(7-250006)
88 88.09 00 7S0 300 GN 96. 66.770000
X5 30.00000 2O0LERD DO 90 20.7700
85 85.0 /90 AGO 000 99 93.471007)
91 01.0400 09) /ON 380 122 THD (HGAM)
95 95 0T) /90 34 OTY THE O00000
NO 10 /7 PEMM WNIT TL 3910000
73 20.00000000 /NI 590 ON0 SA 44,40022 20)
70 10144000 00 /3N 180 GOD 92. 22009002 (5)
8L.600177 (77)
66 (6570 00) /700 270070 00 91 9660.77
MOO FZZ 2000 (0D) NN. 62.900222
$2.56(1133
ANOAN)
/22 (60 AND 94 (20022
90- 000000000 /95 1200 1200000 GO) 96. (00005)
90 500000000 /93 /932400 90) 98. (4(005))))
100
96 36..0000000
01) 99, 99.9001111
195 180 02) 1000 (0000)))0))
AL0R0 OO /3
70. 701100 00 72 TORO O0 82
12)/ I00 100
100.
940 AND 98.
Denecton Accericy
86 8500 DO 200 380 000 -T
SOO LA 100
MOOLLA TO 791 TII (RD) SS
85O GU) 99.
/ 1 LINLND AND
990 9909 AND
84 4555500 DO) /NN S90 (00
/92 670 On)
63 05H0.00 /89 /89.770 770
AND 4000
89 9X.330..00 995 460
INN-ITO
85 33.000000 /ND
(00 /93
/91
00-2400000
75. 75.540..00)
65-710000
O5 15.9000
29. 160000000
70760000
69- 60-300
72 2.9000
91 21.2200
85 05.9000000
000
92. 420400007
92 12.02000077
82.
60
89
36 38)
998. 08..7700.22
(00000)) 29.01(0000
94. 14.44400000 000 99. 22 700.000 99.. 03.52/0022 28
94. 1...890.000 (0)) 990 90.790 OL.79O.00 (0) 99. .2:00.00333
98.47 07.4770000 (7.0000000 998 98.96600099
98, 7000.07
100 (.00000 (N)) 160. (L000)
100 000 T00. (40.00
7.5. 250210009 19)
/ND /SS 85..500000 DO) 93.3 3.337/043)
81 11.0800.00 (0)) / 86.4 (0) 988 96.82:0.01
(900.10) /9. 97.11(0.05)
85 15.4400000 0N) 6O S910 DO) 198. 350.0)
70 10.4H (8) 1966 ANO (O) 97.4 4770.06
(8.6800000 99 06.0200.00 DO 197. 97.17(0.05)
100.00000)
36 84.0 12V 100 DO AND 100 (0.00
SI OY 98. (0
198 (8/80011)
IN 300 (OO 100, 0.000.000
94. 94.76/0.00 (0) / 9600000 DO) /99 99 07.7700000 00
93,76 76/760000) 29H0 O 100 (00.0)
/99 89H0.00
OS SKNO LOD 99.899888889 88888000000
TOOL 10IL (00.0)
82, 12.35/0.G 16/ 996 6300000 100,000,00
86. SSOO 88. 20 DOY SS. 85.950
99.9200 (N)) 99. M..2200.9
92.55 02.5500000 105 4000 (X) 1999 (10001
/ US5 (4 DOI 99 26
93 93.30/0.00 (0000) 00 2.10 00) 99 49.000000
98. (8.52/001)
OKC 220 98, 46/000)
94 241000))) OSLF NOO / 100L (20000
97. 77.600000 99 9OLANNO DU) 1O0L0 (00.00.000
76 76.9400000 NOT TO 3200 WU) 76 26,30000
/N) SS 59.4100 110 DO) /2.5 92.57/0.15
75. 75.830.00 00) /S 85.4 (0) 997 99. 06
92.53.15
/97 19.77/0.15
75 599000000 OR0/ 7OAMRO LMI) /91 91.61(0.21)
52,52000000 / 91..200100
00)
100.000000
0.0000
12) /8 2200 ID 100.00 00.000..00
195 9TTDEO (XO) 98. 98.01
197 966022
93. 93.530.000 (00 550 DO) 99 22/0
990 90. (0. 9999909900000 90000000
91.53:0.00) 00) 986 1500.00) 299 (4/002
999 18/000)
100 10000 (00000
AR6 SWO LOD 199.
89. (0.3000000) 71 47..0.00000 10) 06
AUROC
DKC (9.000000) (K))
75 75.501 A0T 40 SATO
69 15) 100 (00
ON) / /95 03 OO
/ /O ARATTIO AOTHOUO
67.7% 07.7800-3 187 998 9IS (O (O.D))
55.0500000
72.070000: 78.91...00
06. 0200 22 /94 94
086 S6. LOO
996 1S D017
NT 790001)
19 192.80000000 97 77764000000
95. 33.3700000)
94. 14.84400000
91 11.4400022
90 SPK0000Y))
82. 22.6410000
71. 11,220000)
91 (3N000))))
91 38.300000
94 7910
94.0770.000 (0.00000
94 59100000
89 19.000000
90 W0.95000000 0N)
91.06000001
91.0 31.010000))
79 79.340000 340000)
70. 70.4610.00)
75.6900.00)
38 35.3500.1
92. 92.95(0100)
02.5200001
/
94 UR00
93.0600.00)
25
92. 2222500000
98. V5.3500000
95 75.7500000
08)
04. 7447300222 25) 93660 12) /L000 /97.5 57..11) )
40 MOL.NN..DD / 0400000000 19.5 0..5510111 711
401 30133000000 19.1 1900 D9 93. (3.43)1-22)
65 8400 8400000 (0) 744 56/000) / 1999 97(0022)
994 99..11000 (0.010055)
999 .510.00)
95 (0.56
944 210.344
1000 (00-000
190.000.00
7.050 10.5000000) 7 L.TOUOOOY T. 1150000
644 (512.22.
32. 92.7400.00 / 52/85/2000000 02) OR. 1010.15
17.260 20000)000/// NI 9X00I))) RS (7--271-18)
40 130 00)/ 12 12-420000 0O) 955 291050)
16. 00) BA AND DOY NT. 7...2000.. 55)
44.7990555 (
O. 0.130000) 100 1N0. 00) 100 IOOLNOOO .00)
1O0. ON ON) 1000 D 0.000
69. 14 / 76. 2900 D 95. 1E.T100.000 (0)
68 6000 00) (00/ 76. 380.01 /97. 97.0110-99)
79 03H0 00)/ G 0Z. / 1990 010.00)
79 19.730.000 00) 77.2 11.270002) 99 96/0.11
100 (10.000000)
99, 99.99.000
78. F..330.00 (0)) 69. 3400020 / 100. (M.000:011)
1000000000
(00000000)
33 ET TRO.OMY 58 NO0OOY 46.2 36.21033
32. 19 /00) SA. S3.52000) 79 19:0
49 1100 (00/ 06 1200000 / 292 (220110110)
AND) 69 28 DOO)) /99 (9..510113))
52. 52.780. 001/ 50,0700 (O) PIR 11.0100:0
140.00) 77,200 91.55(021)
/76 8400)))) 991 .3.3300.29
00000000000
13 SAR WOT/ 1899 NOT 17. 970001
19 HRO (00) /3 200000 61.3 36(0955
20 2001000001 20000 644 (.777)
12. 72.200000 00) 42.5 (O)) 88 (N.-4810600) (00)
18 3900 000/ 19 (2) / GI 64 2200
16 16.560000 00) D0) 61. (1.180040) (0)
0.910.08) 67. 68.0 36 100 00.000000
/ 100, 010.000
IS ASAAAOO Dy 9H0 (1
96 83 30KDM0) 89 520.13
82. 52.5200000 AND) 63 23.27 OO) 1990 0(0000
79 79.0500 00)/ BA AS DO) 197.688 67.08800066
100. (00.0)0.00)
SA00000 78. 38.33 (0,00) 197. 1910.1)
100 00000000000 (000
85 760 16) 92 (4 0S) (00.000.00)
1000
INR TPR 98%6
59 3WO (00) 62. 300
61 1L.3300000 / 52.9 52.9 1.9.1.00000
66 (000000)) 84 54/0,00)
63 14.3200000 00)/ 83 320000))
13 0N00 21) TUILI
26. 14.400000 000 45.01
11.000000) 82 L1.00.000
76. 76.7140.00) 10.50001
77.0 77.770000 00)/ /6 380022) /
15.470 200/ 99. 27000))
34 91 51) 78. 27.070225
48 91 / 993999000
88 660 25V 10WAKO
16.5 16.570.000 000/ 34 (000)
0 0.000 (001 / L.LOO 1.
79.630000 63.000000 8407/0.00
44 4904 /
/
0.000.00))
18
44 05005
7X. XX.000
77 1..050000
74
Tinylmc8) -
0OD
CIFAR-T00
SVHN
Timy 1uy
Thny TinyImu Im
LSUNICI
LSUN(R)
ISUN
Uniform
Caussio
00:A--10
SVIN
Tinylm/C)
Tiny Tinylm08)
ISUNCC) [
LUUNR))
ISUN
Uniterm
Gauseian
CHEAR-10 10
CIFAR- 100
They TimyImo Im/ C)
Tiny ()
LSUNIC)
LSUNIRI
ISUN
Uniform
Cauosian
CEAR-IO 100
SVHN
TimyIro
(m))))
Tho ThmyImi mmi
LSUNG)
LSUNO
ISUN
Uniform
Caussion
UIFAX-TO
SVIN
Tiny
SSUICC
LSUNRI
ISUN
Uniterr
Canssian
CHAAR-10 10
CIPAR-100
Timy They C)
()
LSUN(C)
LSUNIR
ISUN
Uniform
Gaoosion
Duwakkci LD0
NHAS
000
CER-LIDD
NHAS
ID
16


Reviews:
[{"review_id": "S1epit-TFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new framework for out-of-distribution detection, based on global average pooling and spatial pattern of the feature maps to accurately identify out-of-distribution samples. MAHALANOBIS distance based methods were discussed, and the shortcomings of using mahalanobis distance were given (i.e. assumption that features are independent, etc). They propose to use compression based distance measures off the shelf from standard compression techniques to detect spatial feature patterns in feature space and demonstrate its effectiveness on several datasets and comparison with baselines is reported and well discussed.\n\nThe motivation of the proposed approach is clear, and the method seems novel. However, the experiments could have been done in a more complex setting. The out-of distribution samples pose a danger in safety critical applications such as autonomous driving, for example a car deployed in environment that it has not seen during the training might crash. So, it would be interesting to see the performance of both baselines and proposed approach in those settings where inputs are similar in nature but very different in some aspects. I do not necessarily see something wrong with the paper, but I'm not convinced of the significance (or sufficient efficiency) of the approach. There is also  theoretical guarantees showing exhaustiveness of the proposed methods in detecting all possible out-of distribution examples. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lDwHtCtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After reading the other reviews and comments, I appreciate the effort by the Authors, but it looks like the paper still needs some work before being ready. So, I have decided to maintain my rating.\n\n===================\n\nThe work proposes a system for detecting out-of-distribution images for neural networks under strict limitations of not retraining the network or tuning parameters with out of distribution validation data in mind using Compression Distance in a novel way.  The authors evaluate the method broadly and against the state of the art and provide a thorough explanation of the background material and formulation. \n\nThis work is strong in it’s cleverness, novelty, and evaluation when limited to the class of solution stipulated in section 2; however, it is not clear or presented why this restrictive choice is or could be necessary.    For this reason, I am borderline unless that caveat is addressed as described below, in which case I would be happy to accept. \n\nIt is unclear to me as it was not presented in the work when or if the problem that is being solved by the paper is particularly either important or frequent.  Obviously, not having to retrain is more efficient than having to and not having to validate on out-of-distribution samples is helpful in times when we don’t know them ahead of time, but it is unclear to me if it is the case that we will have a situation in which both of the above are, for instance, not possible at all.  To me, if the work could be changed to compare against works which are not so tightly constrained, not for the purposes of holding it to the same standard but to understand it’s relative standing, or to better justify the very strict constraints which somehow, despite out-of-distribution detection being a popular upcoming topic, apparently only has one other paper that matches it.\n \t\t\nThe paper could use some extra proofreading, for instance the first sentence of the abstract doesn’t make much grammatical sense especially with the first phrase included. \nIt may be nice to cite works such as http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.132.6389&rep=rep1&type=pdf and others in that vein as this is certainly not the first work to involve compressive principles in image classification related tasks. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxkDunM5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** post rebuttal start **\n\nAfter reading reviews and authors' response, I decided not to change my score.\nI remark that this paper still requires a lot of revision; comparison in the main paper is somewhat unfair and all new results are in the appendix.\nAlso, the performance of their replication of the prior method is far lower than reported. In worst case, the performance gain from the compared method would be from their incorrect implementation on the prior works. In this kind of case, I suggest the authors to put {the numbers in the original paper} as well as {their replication} and claim that they fail to replicate the number. Ideally, if their method is evaluated in the same condition, it should outperform prior works in any case.\n\n\nDetailed comments:\n\n1-(a). Adversarial attack and OOD (which is hard to detect) are closely related: they are both in off-manifold. Their main difference would be, while adversarial attack is very close to the clean data in the data space, OOD is relatively far from the in-distribution in the data space.\nThough it does not talk about OOD, you may refer to [R1] for analysis in perspective of manifold. The difficulty of OOD detection can be considered to be coming from overlapped manifolds in the latent space.\n\n[R1] Stutz et al. Disentangling Adversarial Robustness and Generalization. In CVPR, 2019.\n\n\n1-(b). Though the numbers are much lower than those reported in the original paper, I am happy to see the fair comparison. However, according to the original paper, simple FGSM is used for validation, so I am not sure such a huge difference can actually happen. In this kind of case, I suggest the authors to put {the numbers in the original paper} as well as {their replication} and claim that they fail to replicate the number.\n\n\n2. I am happy to see that their revised method has better performance.\n\n** post rebuttal end **\n\n\n- Summary:\nThis paper proposes an out-of-distribution detection (OOD) method under constraints that 1) no OOD is available for validation and 2) model parameters should be unchanged. They specifically address a problem of the state-of-the-art method satisfying the constraints, and propose a new distance metric inspired by data compression. Experimental results on several benchmarks with different deep neural network architectures support their claim.\n\n\n- Decision and supporting arguments:\nWeak reject.\n\n1. The problem setting is clear and their approach is interesting and makes sense. However, the method for comparison is not properly set. As the authors addressed, Mahalanobis detector proposed by Lee et al. (2018b) requires validation to determine weights for feature ensembling, but the validation can be done without OOD data by generating adversarial samples as proposed in the same paper. Although Table 2 in Lee et al. (2018b) shows that the performance is not better than the case when we have an explicit OOD data for validation, it reasonably works well. Therefore, rather than comparing with the vanilla version (only using last latent space) or the alternative \"assemble\" method (concatenating all average-pooled features), they had to compare their method with the model validated by adversarial samples, which essentially satisfies the constraints.\n\n2. Also, I wonder the main body of the proposed method itself is really effective or some minor tweak they made is essential. According to Table 2 in the submission, their method is better than \"Mahalanobis vanilla\" only when all components are applied. Though the idea is interesting, I am skeptical about the effectiveness of the proposed method.\n\n\n- Comments:\n1. As addressed by the authors, feature concatenation (\"assemble\") is not effective for the Mahalanobis method but the proposed method. How about to do an ablation study about weighted averaging vs. concatenation on the proposed method as well? Again, weights can be validated by adversarial samples to satisfy the constraints.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an out-of-distribution detection (OOD) method without assuming OOD in validation.

As reviewers mentioned, I think the idea is interesting and the proposed method has potential. However, I think the paper can be much improved and is not ready to publish due to the followings given reviewers' comments:

(a) The prior work also has some experiments without OOD in validation, i.e., use adversarial examples (AE) instead in validation. Hence, the main motivation of this paper becomes weak unless the authors justify enough why AE is dangerous to use in validation. 

(b) The performance of their replication of the prior method is far lower than reported. I understand that sometimes it is not easy to reproduce the prior results. In this case, one can put the numbers in the original paper. Or, one can provide detailed analysis why the prior method should fail in some cases.

(c) The authors follow exactly same experimental settings in the prior works. But, the reported score of the prior method is already very high in the settings, and the gain can be marginal. Namely, the considered settings are more or less "easy problems". Hence, additional harder interesting OOD settings, e.g., motivated by autonomous driving, would strength the paper.

Hence, I recommend rejection.
Decision: Reject