Year: 2019

Abstract:
To reliably detect the out-of-distribution images based on already deployed convolutional neural networks, several recent studies on out-of-distribution detection have tried to define effective confidence scores without retraining the model. Although they have shown promising results, most of them need to find the optimal hyperparameter values by using a few out-of-distribution images, which eventually assumes a specific test distribution and makes it less practical for realworld applications. In this work, we propose a novel out-of-distribution detection method MALCOM, which neither uses any out-of-distribution samples nor retrain the model. Inspired by the observation that global average pooling cannot capture the spatial information of the feature maps from convolutional neural networks, our method aims to extract informative sequential patterns from the feature maps. To this end, we introduce a similarity metric which focuses on the shared patterns between two sequences based on normalized compression distance. In short, MALCOM uses both the global average and spatial pattern of the feature maps to accurately identify out-of-distribution samples.

heading: 1 INTRODUCTION
text: The distributional uncertainty refers to an uncertainty originated from the inconsistency between training and test distributions (Malinin & Gales, 2018). Recently, measuring the distributional uncertainty of deep neural networks has gained much attention, in order to detect the out-of-distribution sample which comes from outside the training distribution. In contrast to the existing belief that the softmax output of deep neural networks is not appropriate measure for uncertainty, Hendrycks & Gimpel (2017) found out that the softmax score is able to distinguish between in-distribution and out-of-distribution samples to a degree.
Since then, several methods have shown promising results for out-of-distribution detection (Liang et al., 2018; Lee et al., 2018b; Lakshminarayanan et al., 2017; Lee et al., 2018a), but the most of them still have limitations from a practical perspective in that they need to find the optimal values of hyperparameters using the out-of-distribution samples or retrain the model using a new objective function specifically designed for detection. For example, Liang et al. (2018) improved the ability of distinguishing the out-of-distribution data by using temperature scaling and input perturbation, which involves in several hyperparameters; this kind of calibration techniques require out-of-distribution samples for validation to adjust its hyperparameters to manipulate the softmax outputs. On the other hand, Lee et al. (2018a) defined an another loss function by adding the KL divergence, assuming that the softmax output of the out-of-distribution is uniform; it has to retrain the model because of the newly designed loss function. For these reasons, they have been difficult to be deployed for real-world applications. In this sense, our motivation is how far out-of-distribution detection can be done without using the out-of-distribution set for validation while employing the existing softmax classifier without retraining it.
The state-of-the-art method (Lee et al., 2018b) tries to detect out-of-distribution samples based on the Mahalanobis distance from class means instead of using the softmax output, by utilizing the feature maps obtained by convolutional neural networks. In particular, they estimate the class-conditional probabilities for each class, which are more appropriate and effective to detect out-of-distibution samples than the softmax output which provides a mere relative values between classes. However, they obtain the feature vectors of input images by averaging each feature map, also known as global
average pooling. We notice that the spatial information about input images disappears in the process of averaging feature vectors, and this eventually degrades the detection performance.
To tackle this challenge, we offer to use the normalized compression distance (NCD), which can measure the distance between two images without the loss of such spatial information of the feature maps. NCD is a general method of measuring the intrinsic distance between two arbitrary objects using the off-the-shelf compression algorithm (e.g. 7-zip, bzip, or lzw). With its solid theoretical background and general applicability, NCD is widely used in various fields, including brain diagnosis or spam filtering (Berek et al., 2014; Spracklin & Saxton, 2007). The high-level idea of NCD is to evaluate how many shared patterns are eliminated when two binary sequences are compressed together. By using the NCD, we aim to accurately measure how similar the feature maps of a test sample is with those of training examples, and use it to determine whether the test sample is from out-of-distribution or not. Our experiments demonstrate that the proposed method, termed as MALCOM, achieves better or comparable performances for detecting out-of-distribution images to the existing methods.

heading: 2 BACKGROUNDS

heading: 2.1 CONSTRAINED OUT-OF-DISTRIBUTION DETECTION
text: Before proceeding to describe the notions closely related to our method, let us state what we want to tackle exactly: out-of-distribution detection without retraining network and without sniffing outof-distribution samples. Starting with the definition, let Dtrain = {(xi, yi)}Ni=1 be the training dataset and f(·;θ) be the model trained on Dtrain. Suppose that each data xi from Dtrain is an observation of random variableXtrain. Thereby, the out-of-distribution detection task is to obtain the out-of-distribution score function κ(·;Dtrain,θ,φ) for a discrimination between in-distribution and out-of-distribution. In other words, given a test instance x∗, the score κ(x∗) should be high if x∗ is not an observation ofXtrain, otherwise should be low. There are many ways to get score function κ, but we work on the task under the following constraints:
• The score function parameter φ should be independent with the distribution of test data x∗, i.e., φ is determined by only in-distribution. This is intuitive in that x∗ does not assume any test distribution. This requirement implies that hyperparameter tuning by using out-ofdistribution samples is not allowed.
• The model parameter θ should not be changed, i.e., the trained model primitively for the purpose of classification should not be retrained. Then, the score function can be utilized by the neural networks which are already trained and deployed in real-world applications.
The existing methods that satisfy these conditions are the softmax output detector (Hendrycks & Gimpel, 2017) and the simplified Mahalanobis detector (Lee et al., 2018b).

heading: 2.2 MAHALANOBIS DETECTOR
text: The Mahalanobis detector (MAHALANOBIS), one of the best performing methods, introduces an effective method to remove the correlation between feature maps of the last hidden layer (Lee et al., 2018b), but combining layers leads to the degradation of the detection performance.

heading: 2.2.1 MAHALANOBIS-VANILLA
text: We factor out a simplified version of MAHALANOBIS (MAHALANOBIS-vanilla), because the whole MAHALANOBIS detector needs out-of-distribution samples to tune the hyperparameters of input preprocessing and layer weights, which fails to satisfy our requirements (Sec 2.1). For the whole method of MAHALANOBIS which utilizes out-of-distribution samples to boost the performance, we refer the reader to see (Lee et al., 2018b).
MAHALANOBIS-vanilla uses the feature maps of the last hidden layer of convolutional neural networks. To be exact, we denote the output of the l-th hidden layer of the network f by fl for l = 1, . . . , L, where L is the number of hidden layers. For the l-th hidden layer, let f il be the i-th feature map of its output for i = 1, . . . ,Ml, where Ml is the number of feature maps of the l-th
hidden layer. Therefore, fl(x) is a 3D tensor of the size Ml ×Hl ×Wl and f il (x) is a 2D feature map of the size Hl ×Wl for any i and l, where Hl andWl are the height and width of each feature map in l-th layer, respectively. For a notational convenience, we define ml as the global average pooling of the l-th layer, i.e.,
ml(x) :=‖Mli=1 mean(f i l (x)),
where mean is the average function and ‖ is the concatenation operation. MAHALANOBIS-vanilla utilizes only the last hidden layer (i.e., the L-th layer), so it can be obtained by calculating class means and tied-covariance:
µ̂k := 1
Nk N∑ n=1 yn=k mL(xn),
Σ̂ := 1
N K∑ k=1 N∑ n=1 yn=k (mL(xn)− µ̂k)(mL(xn)− µ̂k)T ,
where K is the number of classes and Nk is the number of the data of class k. Then, the outof-distribution score function of MAHALANOBIS-vanilla detector κmv (also known as Mahalanobis distance) is defined by
κmv(x ∗) := min
k=1,...,K (mL(x
∗)− µ̂k)T Σ̂−1(mL(x∗)− µ̂k).
In this equation, MAHALANOBIS-vanilla computes the centered data for each class, then performs linear projection for the centered data by using the inverse of Σ̂. Practically, the inverse of empirical covariance matrix is approximated by (Moore-Penrose) pseudo-inverse, which performs the singular value decomposition (SVD) (Barata & Hussein, 2012). For this reason, computing the Mahalanobis distance becomes equivalent to computing the norm of the sample (i.e., the Euclidean distance from the origin) after the principal component analysis (PCA) assuming that the origin is the class mean, which removes the feature correlations.

heading: 2.2.2 DEGENERACY OF MAHALANOBIS-ASSEMBLE
text: From the observation that MAHALANOBIS-vanilla has the power of removing the correlation over feature maps, one may extend this concept to all the other hidden layers. To define the MAHALANOBIS-assemble1 detector, we concatenate the average of feature maps (i.e., global average pooling) of all layers and similarly define the class means and tied-covariance:
m(x) :=‖Ll=1 ml(x),
µ̂k := 1
Nk N∑ n=1 yn=k m(xn), Σ̂ := 1 N K∑ k=1 N∑ n=1 yn=k (m(xn)− µ̂k)(m(xn)− µ̂k)T ,
κma(x ∗) := min k=1,...,K (m(x∗)− µ̂k)T Σ̂−1(m(x∗)− µ̂k)
1We use the term “assemble” to distinguish it from the “feature ensemble”, which is the method using logistic regression in the original paper.
From the experiments, we find an interesting observation that the simply extended detector κma significantly improve the performance in many cases; however, in some cases, its performance, especially AUPR(In), is poorer than κmv (see the first two columns of Table 1 for more details).

heading: 2.3 NORMALIZED COMPRESSION DISTANCE
text: To address the degeneracy problem in MAHALANOBIS-assemble, we focus on hindering the loss of spatial information of the feature maps caused by the average pooling. To this end, we introduce a compression-based method. Before defining our method precisely, we need to state some notions briefly because its fundamental properties are originated from the concept of information theory.

heading: 2.3.1 KOLMOGOROV COMPLEXITY
text: The Kolmogorov complexity (or algorithmic complexity) is the complexity of an arbitrary object which can be represented in string format. Formally, given an universal Turing machine U , the Kolmogorov complexity K of string x is defined as the minimum length of the programs that are decoded into x by U . That is,
K(x) := min ({length(p)|U(p) = x where p is a valid program for U}) , where length(p) is the length of the program p and U(p) is the result of decoded string with the program p through U .
The conditional Kolmogorov complexity is the minimum program length for a given universal Turing machine which already has the string:
K(x|y) := min ({length(p)|U(p|y) = x}) . If x is highly correlated with y, its conditional complexity would be low.

heading: 2.3.2 NORMALIZED INFORMATION DISTANCE
text: The normalized information distance (NID) is the distance between two strings x and y based on the conditional Kolmogorov complexity. It does not need any prior knowledge so that it can be applied to various tasks universally. The distance is described as
NID(x,y) = max(K(x|y),K(y|x))
max(K(x),K(y)) .
It turns out that NID is a normalized metric (Li et al., 2004), which has following distance properties for arbitrary x,y, and z:
• NID(x,y) ∈ [0, 1] (normality) • NID(x,y) = 0 if and only if x = y (identity) • NID(x,y) = NID(y,x) (symmetry) • NID(x, z) ≤ NID(x,y) + NID(y, z) (triangle inequality)
Unfortunately, the Kolmogorov complexity is not computable, so NID also cannot be computed. Instead, the normalized compression distance (NCD) is used as the approximation of Kolmogorov complexity in many tasks. Given an arbitrary compressor C, NCD is defined by
NCD(x,y) = min (C(xy), C(yx))−min (C(x), C(y))
max(C(x), C(y)) ,
where C(x) is the length of the compression result of x, and xy is the concatenation of x and y. Any off-the-shelf compressors can be used as the compressor, including 7-zip, gzip, and bzip. The better compressor we use, the closer the NCD value approaches to the NID value (Cilibrasi & Vitányi, 2005).

heading: 3 MALCOM: OUT-OF-DISTRIBUTION DETECTOR USING NCD
text: To resolve the degeneracy of MAHALANOBIS-assemble discussed in Section 2.2.2, we propose the MAhaLanobis distance with COMpressive-complexity pooling (MALCOM), which applies NCD to measure the distance between the feature maps of convolutional neural networks.

heading: 3.1 TNCD
text: NCD can be generalized well to various objects because of its universality. Most previous work use the raw images (Lan & Harvey) when they calculate NCD, but it is inappropriate to use raw images for many cases. For example, consider the following strings.
stringa : BABBABBAABABABBBABAABBABB string b : CCBACABCCCBAABBABABAABCBA string c : ABACABACAABCBAACBBBCABACA
In case that we only want to recognize the pattern of the symbol C, using NCD on this string is not useful enough. It needs to look at the features that the classifier consider important. To focus on such important features, we introduce a translation T . In the case above, for example, T translates the symbol C into 1 and all other alphabet into 0.
T (a) : 0000000000000000000000000
T (b) : 1100100111000000000000100
T (c) : 0001000100010001000100010
Before the translation, string a looks much simpler than string b or string c. Through the transformation, we can concentrate only on C in the string b and string c. Formally, for the two symbol sets Σ and Ω, we denote the translation by T : Σ∗ → Ω∗. Given a transformation T , we define the translated and normalized compression distance (tNCD) by
tNCD(x,y;T ) = min
[ C(T (x)T (y)), C(T (y)T (x)) ] −min [ C(T (x)), C(T (y)) ] max [ C (T (x)) , C (T (y))
] . Note that tNCD between x and y depends on what the translation T focuses on. If we fix y and there exist a lot of translations with different perspectives, x can be described by various aspects y.

heading: 3.2 COMPRESSIVE-COMPLEXITY POOLING
text: We propose the compressive-complexity pooling which utilize the spatial information of a feature map. For each feature map f il , we consider h i l := g i l ◦f il as a translation where gil is a quantizationand-linearization, which discretizes the continuous values and then makes the quantized feature map a string. We first select a sample s from Dtrain. Then, for the fixed sample s and the test sample x, we can calculate the tNCD value dil(x; s) for each translation h i l ,
dil(x; s) := tNCD(s,x;h i l) =
min [ C(hil(x)h i l(y)), C(h i l(y)h i l(x)) ] −min [ C(hil(x)), C(h i l(y)) ] max [ C ( hil(x) ) , C ( hil(y) ) ] .
Intuitively, for an image x drawn from a specific class, the distance dil becomes small if some aspects of x corresponding to the translation hil are similar to s, and it would be large otherwise.
The compressive-complexity pooling is defined by a fixed sample s with aggregating every feature maps,
ρl(x; s) =‖Mli=1 d i l(x, s).
All the layers can be concatenated for x with respect to s,
ρ(x; s) :=‖Ll=1 ρl(x; s).
This can be interpreted as the distance of x from s, considering the patterns of spatial information for each feature map. Figure 1 illustrates the overall process of the compressive-complexity pooling and how it differs from the global average pooling.

heading: 3.3 SCORE FUNCTION FOR OUT-OF-DISTRIBUTION
text: To put it all together, we can obtain the mean vectors m̃ that encodes spatial information by concatenating the mean pooling vectors and the compressive-complexity pooling vectors. We apply the Mahalanobis distance to remove correlation between feature maps.
m̃(x) := m(x) ‖ ρ(x; s)
µ̂k := 1
Nk N∑ n=1 yn=k m̃(xn), Σ̂ := 1 N K∑ k=1 N∑ n=1 yn=k (m̃(xn)− µ̂k)(m̃(xn)− µ̂k)T
Based on the mean vectors, we define the final out-of-distribution score function κ which is capable of accurately measuring the out-of-distribution score:
κMALCOM(x ∗) := min k=1,...,K (m̃(x∗)− µ̂k)T Σ̂−1(m̃(x∗)− µ̂k).

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: For experiments, we follow the experimental setup mainly used by previous work (Liang et al., 2018; Lee et al., 2018b).
Models. For fair comparisons, we use the pretrained DenseNet (Huang et al., 2017) and ResNet (He et al., 2016) provided by the previous work (Lee et al., 2018b)2. According to the authors, they use DenseNet with 100 layers, growth rate k=12 and dropout rate 0, while ResNet with 34 layers and dropout rate 0. Both the models are trained using stochastic gradient descent (SGD) with Nesterov momentum for classifying CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011). For more details, we refer to the original paper (Lee et al., 2018b).
Dataset. We use out-of-distribution image datasets: TinyImageNet (Deng et al., 2009), LSUN (Yu et al., 2015), and iSUN (Xu et al., 2015). The in-distribution datasets have 10 classes except for CIFAR-100 which has 100 classes. CIFAR-10 (or CIFAR-100) has 50,000 training and 10,000 test images, while SVHN has 73,257 training and 26,032 test images. For out-of-distribution datasets, TinyImageNet contains 10,000 test images of 200 classes, LSUN contains 10,000 of 10 classes, and iSUN contains 8925 images. Liang et al. (2018)3 preprocessed TinyImageNet and LSUN by either randomly cropping of size 32 × 32 or downsampling to size 32 × 32. Competing Methods. We compare the performance of our method with that of the existing methods satisfying the two constraints (Section 2.1), which are the independence with out-of-distribution samples and exploitation of the model without retraining. As far as we know, there is only one method perfectly satisfying these requirements: BASELINE (Hendrycks & Gimpel, 2017) using the
2https://github.com/pokaxpoka/deep Mahalanobis detector 3https://github.com/facebookresearch/odin
softmax output to distinguish out-of-distribution data from in-distribution data. In addition, we can use the simplified version of (Lee et al., 2018b), i.e., MAHALANOBIS-vanilla (Section 2.2.1), although the original method uses the input preprocessing and ensembles the layers by introducing the hyperparameters. The implementation details about the compression of the feature map are provided in Appendix A.
Evaluation Metrics. We adopt the five performance metrics the same with the previous work (Lee et al., 2018b): true negative rate (TNR) at 95% true positive rate (TPR), area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPR), and detection accuracy. For the details, we refer to (Lee et al., 2018b).

heading: 4.2 RESULTS
text: Ablation study. We ablate the components of our method to show the interaction between the global average pooling and the compressive-complexity pooling. To this end, we use the DenseNet pretrained on CIFAR-10 and CIFAR-100. The out-of-distribution datasets are the resized TinyImageNet and the resized LSUN, and the AUROC is measured as the evaluation metric. Table 2 shows the results of the ablation study. The layer aggregation refers to the method that uses the concatenation of feature maps of all the layers. From the results, we obtain the following observations:
• The performance of our compressive-complexity pooling is improved by concatenating the feature maps of all the layers, whereas the performance of the global average pooling becomes worse when it concatenates the feature maps of all the layers.
• The compressive-complexity pooling is always worse than the global average pooling when we use only the feature maps of last hidden layer.
• Using both the pooling techniques at the same time consistently improves the detection performance. It shows that each pooling provides different information.
Main results. The main results are summarized in Table 3. we repeat each experiment five times and report the averaged results. See Appendix D for more detailed results. (C) and (R) denote the dataset “resized” and “cropped”, respectively. Our proposed method, MALCOM, achieves significantly better performance than other competing methods in most cases. Nevertheless, for minor cases, we observe that MALCOM with ResNet slightly degrades the performance on the resized TinyImageNet and the resized LSUN. From the extensive experiments, we conclude that our compressive-complexity pooling successfully captures the spatial information from its feature maps, and it is helpful to accurately measure the confidence score for detecting the out-of-distribution data.

heading: 5 RELATED WORK
text: As deep learning has made remarkable progress in the past few years, many methods have been proposed to estimate the confidence of the model prediction beyond just enhancing the model accuracy. Although the modern neural networks have achieved higher classification accuracy than in the
past, the confidence calibration in neural networks is poorer than a decade ago, i.e., the discrepancy between the prediction score of the neural network and the ground-truth class probability gets worse compared with that of the past (Guo et al., 2017). To make matters worse, most neural networks, especially for non-Bayesian, have no measure of uncertainty, which hinders reliable decision-making just by the prediction score in many practical areas (e.g. medical diagnosis, self-driving car). Thus, there has been much work to attain the measurement for uncertainty or confidence calibration other than classification accuracy (Guo et al., 2017; Lakshminarayanan et al., 2017; Jiang et al., 2018)
Even though Bayesian deep learning provides uncertainty properly, several methods were recently proposed to estimate uncertainty of non-Bayesian deep neural networks. Especially, it turns out that some regularization techniques of neural networks (e.g. dropout, batch normalization) are interpreted as approximate Bayesian inference so that uncertainty can be captured in conventional deep neural network (Gal, 2016; Teye et al., 2018). These work are significant in that deep neural networks are not required to re-model or retrain once they have been trained by using dropout or batch normalization even if they do not consider the uncertainty at train time. MALCOM is inspired mainly by these work because it can be applied more practically to neural networks which already have been trained and deployed.
Most previous work of out-of-distribution detection can be divided into two main approaches: introducing parameter and defining new objective function. An prominent example of the former is Liang et al. (2018) (ODIN) which suggests two hyperparameters for softmax output to a boost in performance: temperature scaling and input preprocessing. Since ODIN, most studies using softmax output use these two parameters. Vyas et al. (2018) tried to solve the problem with ensemble of classifiers by dividing the training data and adding loss by defining entropy for in-distribution and out-of-distribution as a self-supervised learning method, and used two parameters to improve performance. Our closely related work, Lee et al. (2018b) focuses on the internal features of the
CNNs, while it uses some hyperparameter logistic regression for aggregating all feature maps and input preprocessing.

heading: 6 CONCLUSION
text: This paper proposes MALCOM, which accurately measures the confidence score of a test sample for out-of-distribution detection, without using any hyperparameters and without retraining the model. MALCOM addresses the limitation of the existing Mahalanobis distance-based method that cannot exploit the spatial information of image feature maps. We introduce the compressivecomplexity pooling based on NCD to consider the spatial information of the feature map when measuring the distance from class means. Our extensive evaluation on image benchmark datasets shows that the compressive-complexity pooling leverages the spatial information of the feature maps, and it improve the performance compared to the global average pooling. In addition, by using the both the pooling techniques simultaneously, MALCOM achieves the best performance for out-of-distribution detection.

heading: D MORE RESULTS
text: Table 7 shows the detailed results for out-of-distribution detection. To perform further analysis, we test on the synthetic noise datasets which as in Liang et al. (2018).
Ta bl
e 7:
Pe rf
or m
an ce
of di
ff er
en to
ut -o
fdi
st ri
bu tio
n de
te ct
io n
m et
ho ds
.T he
be st
re su
lt is
hi gh
lig ht
ed in
bo ld
fa ce
fo re
ac h
se tti
ng .
ID O
O D
T N
R at
T PR
95 %
A U
R O
C D
et ec
tio n
A cc
ur ac
y A
U PR
(I n)
A U
PR (O
ut )
B A
S E
L IN
E /M
A H
A L
A N
O B
IS -v
an ill
a /M
A L
C O
M (o
ur s)
DenseNet
CIFAR-10
C IF
A R
-1 00
40 .6
0( 0.
00 )/
18 .0
4( 0.
00 )/
19 .5
3( 0.
71 )
89 .3
0( 0.
00 )/
71 .4
7( 0.
00 )/
66 .3
6( 1.
38 )
82 .9
2( 0.
00 )/
65 .5
6( 0.
00 )/
61 .9
7( 0.
75 )
89 .3
7( 0.
00 )/
72 .1
8( 0.
00 )/
62 .1
2( 2.
02 )
86 .6
3( 0.
00 )/
69 .6
2( 0.
00 )/
67 .5
5( 0.
98 )
SV H
N 40
.3 9(
0. 00
)/ 89
.1 9(
0. 00
)/ 93
.4 3(
0. 72
) 89
.8 8(
0. 00
)/ 97
.6 4(
0. 00
)/ 98
.3 7(
0. 22
) 83
.1 8(
0. 00
)/ 92
.4 2(
0. 00
)/ 94
.3 0(
0. 32
) 83
.7 3(
0. 00
)/ 94
.5 4(
0. 00
)/ 95
.8 7(
1. 11
) 94
.4 0(
0. 00
)/ 99
.0 2(
0. 00
)/ 99
.2 8(
0. 06 ) Ti ny Im (C ) 65 .8 4( 0. 00 )/ 74 .5 6( 0. 00 )/ 99 .9 7( 0. 02 ) 95 .3 7( 0. 00 )/ 95 .3 9( 0. 00 )/ 99 .9 1( 0. 01 ) 90 .0 7( 0. 00 )/ 88 .4 1( 0. 00 )/ 99 .0 9( 0. 10 ) 96 .4 7( 0. 00 )/ 95 .7 6( 0. 00 )/ 99 .9 2( 0. 01 ) 93 .8 9( 0. 00 )/ 95 .1 0( 0. 00 )/ 99 .8 9( 0. 01 ) Ti ny Im (R ) 59 .3 8( 0. 00 )/ 82 .3 0( 0. 00 )/ 94 .0 1( 0. 53 ) 94 .1 4( 0. 00 )/ 96 .2 7( 0. 00 )/ 98 .5 2( 0. 28 ) 88 .5 0( 0. 00 )/ 89 .9 8( 0. 00 )/ 94 .5 6( 0. 27 ) 95 .3 2( 0. 00 )/ 96 .1 5( 0. 00 )/ 97 .9 8( 0. 79 ) 92 .3 9( 0. 00 )/ 96 .4 2( 0. 00 )/ 98 .6 6( 0. 16 ) L SU N (C ) 61 .3 8( 0. 00 )/ 52 .9 1( 0. 00 )/ 99 .8 5( 0. 08 ) 94 .8 9( 0. 00 )/ 90 .7 9( 0. 00 )/ 99 .8 2( 0. 03 ) 89 .8 3( 0. 00 )/ 82 .7 7( 0. 00 )/ 98 .5 9( 0. 21 ) 96 .1 5( 0. 00 )/ 91 .8 5( 0. 00 )/ 99 .8 4( 0. 02 ) 93 .2 4( 0. 00 )/ 89 .5 7( 0. 00 )/ 99 .7 9( 0. 03 ) L SU N (R ) 66 .9 0( 0. 00 )/ 84 .5 4( 0. 00 )/ 95 .6 5( 0. 56 ) 95 .4 7( 0. 00 )/ 97 .0 6( 0. 00 )/ 98 .9 6( 0. 09 ) 90 .2 4( 0. 00 )/ 91 .3 6( 0. 00 )/ 95 .4 2( 0. 26 ) 96 .5 1( 0. 00 )/ 97 .2 4( 0. 00 )/ 98 .9 2( 0. 20 ) 94 .1 2( 0. 00 )/ 96 .8 1( 0. 00 )/ 98 .9 1( 0. 08 ) iS U N 63 .3 2( 0. 00 )/ 83 .3 2( 0. 00 )/ 94 .2 1( 0. 34 ) 94 .8 4( 0. 00 )/ 96 .5 9( 0. 00 )/ 98 .7 0( 0. 07 ) 89 .2 5( 0. 00 )/ 90 .3 9( 0. 00 )/ 94 .6 8( 0. 17 ) 96 .3 5( 0. 00 )/ 96 .9 2( 0. 00 )/ 98 .5 2( 0. 09 ) 92 .4 9( 0. 00 )/ 96 .2 3( 0. 00 )/ 98 .5 8( 0. 06 ) U ni fo rm 13 .0 0( 0. 21 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 91 .9 4( 0. 02 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 92 .3 8( 0. 04 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 95 .2 8( 0. 01 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 81 .7 6( 0. 07 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) G au ss ia n 7. 05 (0 .2 1) /1 00 .0 (0 .0 0) /1 00 .0 (0 .0 0) 90 .8 9( 0. 04 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 91 .8 7( 0. 03 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 94 .7 2( 0. 02 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 79 .5 3( 0. 08 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 )
CIFAR-100
C IF
A R
-1 0
18 .5
0( 0.
00 )/
1. 16
(0 .0
0) /1
.1 5(
0. 05
) 75
.8 0(
0. 00
)/ 49
.8 6(
0. 00
)/ 45
.2 1(
0. 19
) 69
.7 0(
0. 00
)/ 52
.7 8(
0. 00
)/ 50
.6 6(
0. 08
) 78
.4 6(
0. 00
)/ 53
.9 9(
0. 00
)/ 48
.9 0(
0. 08
) 71
.8 7(
0. 00
)/ 47
.0 9(
0. 00
)/ 44
.4 4(
0. 12 ) SV H N 26 .2 4( 0. 00 )/ 45 .0 1( 0. 00 )/ 64 .8 5( 2. 22 ) 82 .6 4( 0. 00 )/ 85 .9 3( 0. 00 )/ 93 .3 7( 0. 43 ) 75 .5 4( 0. 00 )/ 78 .6 1( 0. 00 )/ 86 .6 0( 0. 56 ) 75 .0 9( 0. 00 )/ 71 .6 5( 0. 00 )/ 88 .2 9( 0. 96 ) 90 .3 6( 0. 00 )/ 92 .7 3( 0. 00 )/ 96 .1 0( 0. 25 )
Ti ny
Im (C
) 32
.7 4(
0. 00
)/ 52
.8 5(
0. 00
)/ 98
.1 0(
0. 15
) 81
.0 8(
0. 00
)/ 86
.8 4(
0. 00
)/ 98
.8 2(
0. 01
) 72
.8 8(
0. 00
)/ 79
.0 5(
0. 00
)/ 96
.6 4(
0. 09
) 82
.8 9(
0. 00
)/ 85
.8 5(
0. 00
)/ 99
.1 6(
0. 01
) 80
.0 0(
0. 00
)/ 87
.2 8(
0. 00
)/ 98
.1 6(
0. 02 ) Ti ny Im (R ) 17 .2 6( 0. 00 )/ 81 .9 9( 0. 00 )/ 85 .2 7( 0. 18 ) 71 .6 2( 0. 00 )/ 95 .6 9( 0. 00 )/ 97 .1 1( 0. 05 ) 65 .7 1( 0. 00 )/ 89 .5 3( 0. 00 )/ 91 .2 4( 0. 12 ) 74 .0 9( 0. 00 )/ 95 .3 2( 0. 00 )/ 97 .2 9( 0. 07 ) 68 .9 5( 0. 00 )/ 95 .8 7( 0. 00 )/ 96 .9 5( 0. 05 ) L SU N (C ) 40 .1 3( 0. 00 )/ 12 .4 2( 0. 00 )/ 95 .2 9( 0. 50 ) 85 .4 4( 0. 00 )/ 60 .5 9( 0. 00 )/ 98 .3 5( 0. 06 ) 76 .7 2( 0. 00 )/ 56 .8 8( 0. 00 )/ 95 .7 0( 0. 19 ) 86 .9 8( 0. 00 )/ 61 .7 0( 0. 00 )/ 98 .8 3( 0. 05 ) 84 .3 8( 0. 00 )/ 60 .5 4( 0. 00 )/ 97 .1 9( 0. 10 ) L SU N (R ) 16 .4 4( 0. 00 )/ 84 .4 9( 0. 00 )/ 87 .2 0( 0. 55 ) 70 .8 4( 0. 00 )/ 96 .6 0( 0. 00 )/ 97 .4 7( 0. 06 ) 65 .0 1( 0. 00 )/ 91 .0 3( 0. 00 )/ 92 .4 2( 0. 12 ) 74 .1 5( 0. 00 )/ 96 .7 6( 0. 00 )/ 97 .8 3( 0. 05 ) 67 .8 7( 0. 00 )/ 96 .1 2( 0. 00 )/ 96 .8 8( 0. 06 ) iS U N 14 .8 1( 0. 00 )/ 82 .1 3( 0. 00 )/ 84 .7 9( 0. 53 ) 69 .6 8( 0. 00 )/ 96 .0 2( 0. 00 )/ 97 .1 7( 0. 05 ) 63 .9 5( 0. 00 )/ 89 .7 7( 0. 00 )/ 91 .4 2( 0. 14 ) 74 .6 7( 0. 00 )/ 96 .2 4( 0. 00 )/ 97 .5 4( 0. 03 ) 63 .9 8( 0. 00 )/ 95 .5 5( 0. 00 )/ 96 .5 4( 0. 07 ) U ni fo rm 0. 13 (0 .0 3) /1 00 .0 (0 .0 0) /1 00 .0 (0 .0 0) 54 .6 9( 0. 14 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 65 .9 0( 0. 07 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 69 .1 7( 0. 09 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 47 .0 4( 0. 08 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) G au ss ia n 0. 00 (0 .0 0) /1 00 .0 (0 .0 0) /1 00 .0 (0 .0 0) 36 .8 4( 0. 12 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 59 .9 6( 0. 07 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 57 .0 0( 0. 09 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 39 .4 6( 0. 04 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 )
SVHN
C IF
A R
-1 0
69 .1
4( 0.
00 )/
76 .2
9( 0.
01 )/
95 .9
1( 0.
08 )
91 .8
3( 0.
00 )/
96 .3
8( 0.
00 )/
98 .6
0( 0.
01 )
86 .5
4( 0.
00 )/
91 .7
0( 0.
00 )/
95 .5
2( 0.
02 )
95 .3
7( 0.
00 )/
98 .7
3( 0.
00 )/
99 .5
2( 0.
00 )
83 .4
5( 0.
00 )/
88 .3
2( 0.
00 )/
94 .6
5( 0.
04 )
C IF
A R
-1 00
68 .6
8( 0.
00 )/
76 .3
8( 0.
01 )/
97 .0
1( 0.
09 )
91 .3
8( 0.
00 )/
96 .3
6( 0.
00 )/
98 .8
8( 0.
01 )
86 .5
2( 0.
00 )/
91 .4
5( 0.
00 )/
96 .0
5( 0.
06 )
94 .8
3( 0.
00 )/
98 .7
0( 0.
00 )/
99 .6
1( 0.
00 )
83 .0
9( 0.
00 )/
88 .7
4( 0.
00 )/
95 .9
8( 0.
04 )
Ti ny
Im (C
) 79
.0 3(
0. 00
)/ 64
.8 1(
0. 02
)/ 10
0. 0(
0. 00
) 94
.7 9(
0. 00
)/ 95
.0 3(
0. 00
)/ 10
0. 0(
0. 00
) 89
.8 9(
0. 00
)/ 90
.5 0(
0. 00
)/ 99
.9 8(
0. 00
) 97
.1 9(
0. 00
)/ 98
.2 6(
0. 00
)/ 10
0. 0(
0. 00
) 89
.0 9(
0. 00
)/ 83
.1 1(
0. 00
)/ 99
.9 9(
0. 00 ) Ti ny Im (R ) 79 .7 3( 0. 00 )/ 71 .2 7( 0. 02 )/ 99 .9 6( 0. 01 ) 94 .7 6( 0. 00 )/ 96 .0 3( 0. 00 )/ 99 .8 7( 0. 00 ) 90 .2 1( 0. 00 )/ 91 .7 8( 0. 00 )/ 98 .9 1( 0. 02 ) 97 .0 4( 0. 00 )/ 98 .6 3( 0. 00 )/ 99 .9 5( 0. 00 ) 88 .9 0( 0. 00 )/ 86 .8 7( 0. 00 )/ 99 .5 6( 0. 01 ) L SU N (C ) 76 .7 1( 0. 00 )/ 70 .5 8( 0. 01 )/ 10 0. 0( 0. 00 ) 93 .7 6( 0. 00 )/ 95 .2 9( 0. 00 )/ 10 0. 0( 0. 00 ) 88 .7 6( 0. 00 )/ 89 .7 4( 0. 00 )/ 99 .9 8( 0. 00 ) 96 .6 5( 0. 00 )/ 98 .2 4( 0. 00 )/ 10 0. 0( 0. 00 ) 87 .4 2( 0. 00 )/ 86 .7 6( 0. 00 )/ 99 .9 9( 0. 00 ) L SU N (R ) 77 .0 7( 0. 00 )/ 61 .3 8( 0. 02 )/ 99 .9 9( 0. 00 ) 94 .0 7( 0. 00 )/ 95 .0 7( 0. 00 )/ 99 .8 9( 0. 00 ) 89 .1 6( 0. 00 )/ 90 .9 9( 0. 00 )/ 99 .2 1( 0. 01 ) 96 .7 0( 0. 00 )/ 98 .3 3( 0. 00 )/ 99 .9 6( 0. 00 ) 87 .9 5( 0. 00 )/ 82 .3 3( 0. 00 )/ 99 .6 1( 0. 01 ) iS U N 78 .3 3( 0. 00 )/ 69 .3 4( 0. 02 )/ 10 0. 0( 0. 01 ) 94 .3 9( 0. 00 )/ 95 .8 6( 0. 00 )/ 99 .8 8( 0. 00 ) 89 .6 9( 0. 00 )/ 91 .7 0( 0. 00 )/ 99 .1 7( 0. 04 ) 97 .1 7( 0. 00 )/ 98 .7 0( 0. 00 )/ 99 .9 6( 0. 00 ) 87 .2 9( 0. 00 )/ 84 .6 7( 0. 00 )/ 99 .5 3( 0. 02 ) U ni fo rm 15 .4 7( 0. 20 )/ 97 .2 7( 0. 09 )/ 10 0. 0( 0. 00 ) 67 .7 8( 0. 18 )/ 98 .9 1( 0. 01 )/ 10 0. 0( 0. 00 ) 65 .0 9( 0. 12 )/ 96 .2 8( 0. 03 )/ 10 0. 0( 0. 00 ) 62 .2 8( 0. 23 )/ 99 .1 7( 0. 01 )/ 10 0. 0( 0. 00 ) 66 .5 7( 0. 16 )/ 98 .5 1( 0. 01 )/ 10 0. 0( 0. 00 ) G au ss ia n 34 .9 1( 0. 51 )/ 78 .7 7( 0. 25 )/ 10 0. 0( 0. 00 ) 82 .3 5( 0. 16 )/ 96 .6 3( 0. 02 )/ 10 0. 0( 0. 00 ) 76 .4 9( 0. 10 )/ 93 .9 2( 0. 06 )/ 10 0. 0( 0. 00 ) 79 .6 2( 0. 27 )/ 97 .7 5( 0. 02 )/ 10 0. 0( 0. 00 ) 80 .6 7( 0. 18 )/ 93 .6 6( 0. 05 )/ 10 0. 0( 0. 00 )
ResNet
CIFAR-10
C IF
A R
-1 00
33 .1
3( 0.
00 )/
40 .2
6( 0.
00 )/
46 .2
3( 0.
33 )
86 .3
8( 0.
00 )/
88 .2
9( 0.
00 )/
88 .9
5( 0.
08 )
80 .3
0( 0.
00 )/
81 .7
8( 0.
00 )/
81 .9
6( 0.
03 )
86 .8
5( 0.
00 )/
89 .1
5( 0.
00 )/
89 .3
8( 0.
17 )
83 .4
4( 0.
00 )/
85 .6
7( 0.
00 )/
87 .1
9( 0.
05 )
SV H
N 32
.1 9(
0. 00
)/ 54
.5 2(
0. 00
)/ 79
.1 9(
0. 58
) 89
.8 8(
0. 00
)/ 93
.9 2(
0. 00
)/ 96
.6 2(
0. 09
) 85
.0 6(
0. 00
)/ 89
.1 3(
0. 00
)/ 91
.1 3(
0. 12
) 85
.9 4(
0. 00
)/ 91
.5 6(
0. 00
)/ 94
.3 5(
0. 15
) 93
.5 7(
0. 00
)/ 95
.9 5(
0. 00
)/ 98
.2 7(
0. 06 ) Ti ny Im (C ) 49 .1 1( 0. 00 )/ 66 .1 2( 0. 00 )/ 99 .0 3( 0. 10 ) 92 .5 5( 0. 00 )/ 95 .4 0( 0. 00 )/ 99 .6 1( 0. 01 ) 86 .8 5( 0. 00 )/ 90 .3 8( 0. 00 )/ 97 .2 5( 0. 06 ) 94 .0 8( 0. 00 )/ 96 .5 9( 0. 00 )/ 99 .6 6( 0. 01 ) 90 .0 5( 0. 00 )/ 93 .2 7( 0. 00 )/ 99 .5 6( 0. 01 ) Ti ny Im (R ) 44 .0 5( 0. 00 )/ 69 .2 8( 0. 00 )/ 89 .6 5( 0. 12 ) 90 .9 5( 0. 00 )/ 95 .0 4( 0. 00 )/ 98 .2 6( 0. 01 ) 84 .9 5( 0. 00 )/ 88 .5 9( 0. 00 )/ 93 .1 3( 0. 04 ) 92 .4 5( 0. 00 )/ 95 .7 8( 0. 00 )/ 98 .4 1( 0. 01 ) 88 .2 7( 0. 00 )/ 94 .0 4( 0. 00 )/ 98 .1 8( 0. 01 ) L SU N (c ) 52 .7 8( 0. 00 )/ 50 .0 7( 0. 00 )/ 98 .1 1( 0. 08 ) 93 .3 0( 0. 00 )/ 94 .2 9( 0. 00 )/ 99 .4 9( 0. 01 ) 88 .0 9( 0. 00 )/ 90 .3 1( 0. 00 )/ 96 .6 7( 0. 06 ) 94 .7 8( 0. 00 )/ 96 .0 4( 0. 00 )/ 99 .5 5( 0. 01 ) 90 .9 3( 0. 00 )/ 90 .8 3( 0. 00 )/ 99 .4 4( 0. 01 ) L SU N (r ) 45 .1 4( 0. 00 )/ 78 .2 7( 0. 00 )/ 91 .5 5( 0. 21 ) 91 .0 6( 0. 00 )/ 96 .5 5( 0. 00 )/ 98 .5 2( 0. 01 ) 85 .2 6( 0. 00 )/ 90 .6 8( 0. 00 )/ 93 .7 7( 0. 02 ) 92 .4 5( 0. 00 )/ 97 .1 1( 0. 00 )/ 98 .6 6( 0. 01 ) 88 .5 9( 0. 00 )/ 95 .8 1( 0. 00 )/ 98 .4 2( 0. 01 ) iS U N 44 .4 9( 0. 00 )/ 76 .8 4( 0. 00 )/ 91 .3 2( 0. 29 ) 91 .0 1( 0. 00 )/ 96 .3 2( 0. 00 )/ 98 .4 6( 0. 03 ) 85 .0 1( 0. 00 )/ 90 .3 6( 0. 00 )/ 93 .6 7( 0. 07 ) 93 .0 8( 0. 00 )/ 97 .2 0( 0. 00 )/ 98 .7 3( 0. 03 ) 87 .2 9( 0. 00 )/ 95 .0 5( 0. 00 )/ 98 .2 1( 0. 04 ) U ni fo rm 48 .9 1( 0. 29 )/ 99 .9 9( 0. 01 )/ 10 0. 0( 0. 00 ) 94 .2 4( 0. 04 )/ 99 .7 3( 0. 00 )/ 10 0. 0( 0. 00 ) 91 .0 4( 0. 09 )/ 98 .7 5( 0. 02 )/ 10 0. 0( 0. 00 ) 96 .1 7( 0. 03 )/ 99 .8 0( 0. 00 )/ 10 0. 0( 0. 00 ) 89 .7 7( 0. 07 )/ 99 .5 0( 0. 01 )/ 10 0. 0( 0. 00 ) G au ss ia n 88 .6 6( 0. 25 )/ 10 0. 0( 0. 00 )/ 10 0. 0( 0. 00 ) 97 .6 6( 0. 01 )/ 99 .9 0( 0. 00 )/ 10 0. 0( 0. 00 ) 95 .0 1( 0. 07 )/ 99 .5 4( 0. 01 )/ 10 0. 0( 0. 00 ) 98 .3 7( 0. 01 )/ 99 .9 3( 0. 00 )/ 10 0. 0( 0. 00 ) 96 .1 3( 0. 02 )/ 99 .7 9( 0. 00 )/ 10 0. 0( 0. 00 )
CIFAR-100
C IF
A R
-1 0
18 .5
4( 0.
00 )/
18 .9
6( 0.
00 )/
17 .9
7( 0.
18 )
76 .9
4( 0.
00 )/
76 .8
2( 0.
00 )/
76 .3
6( 0.
05 )
70 .9
1( 0.
00 )/
71 .6
4( 0.
00 )/
71 .3
9( 0.
06 )
78 .4
3( 0.
00 )/
75 .8
7( 0.
00 )/
75 .8
5( 0.
09 )
72 .5
3( 0.
00 )/
73 .0
6( 0.
00 )/
72 .1
7( 0.
10 )
SV H
N 19
.8 8(
0. 00
)/ 43
.2 0(
0. 00
)/ 61
.3 6(
0. 95
) 79
.3 4(
0. 00
)/ 89
.4 1(
0. 00
)/ 92
.5 7(
0. 18
) 73
.2 0(
0. 00
)/ 81
.9 9(
0. 00
)/ 84
.4 4(
0. 20
) 65
.8 5(
0. 00
)/ 83
.5 8(
0. 00
)/ 87
.2 4(
0. 26
) 88
.2 7(
0. 00
)/ 94
.2 8(
0. 00
)/ 96
.4 1(
0. 09 ) Ti ny Im (C ) 16 .8 7( 0. 00 )/ 34 .6 5( 0. 00 )/ 88 .7 7( 0. 61 ) 75 .8 3( 0. 00 )/ 85 .4 9( 0. 00 )/ 97 .8 6( 0. 10 ) 70 .1 4( 0. 00 )/ 78 .1 8( 0. 00 )/ 92 .0 9( 0. 25 ) 76 .4 7( 0. 00 )/ 87 .1 0( 0. 00 )/ 97 .9 4( 0. 10 ) 71 .2 4( 0. 00 )/ 82 .6 4( 0. 00 )/ 97 .9 0( 0. 10 ) Ti ny Im (R ) 20 .1 8( 0. 00 )/ 20 .2 8( 0. 00 )/ 64 .2 0( 0. 77 ) 77 .0 7( 0. 00 )/ 78 .9 1( 0. 00 )/ 92 .5 3( 0. 15 ) 70 .7 6( 0. 00 )/ 72 .0 3( 0. 00 )/ 83 .8 6( 0. 17 ) 79 .6 9( 0. 00 )/ 81 .7 8( 0. 00 )/ 93 .0 3( 0. 12 ) 73 .1 6( 0. 00 )/ 74 .4 4( 0. 00 )/ 92 .5 3( 0. 18 ) L SU N (C ) 12 .7 2( 0. 00 )/ 42 .5 0( 0. 00 )/ 88 .4 8( 0. 60 ) 70 .4 6( 0. 00 )/ 87 .4 4( 0. 00 )/ 97 .7 7( 0. 13 ) 66 .5 7( 0. 00 )/ 79 .9 7( 0. 00 )/ 91 .9 6( 0. 27 ) 68 .1 1( 0. 00 )/ 88 .4 2( 0. 00 )/ 97 .8 4( 0. 12 ) 66 .4 8( 0. 00 )/ 85 .2 7( 0. 00 )/ 97 .8 4( 0. 12 ) L SU N (R ) 18 .3 9( 0. 00 )/ 19 .6 2( 0. 00 )/ 60 .6 4( 1. 20 ) 75 .5 9( 0. 00 )/ 79 .0 0( 0. 00 )/ 91 .6 1( 0. 21 ) 69 .7 5( 0. 00 )/ 72 .2 8( 0. 00 )/ 82 .6 9( 0. 22 ) 77 .4 1( 0. 00 )/ 81 .9 9( 0. 00 )/ 92 .4 1( 0. 15 ) 71 .7 1( 0. 00 )/ 74 .1 0( 0. 00 )/ 91 .3 3( 0. 27 ) iS U N 16 .5 6( 0. 00 )/ 18 .0 7( 0. 00 )/ 61 .1 8( 0. 40 ) 75 .6 9( 0. 00 )/ 78 .5 2( 0. 00 )/ 91 .5 2( 0. 10 ) 70 .1 1( 0. 00 )/ 72 .1 9( 0. 00 )/ 82 .5 6( 0. 13 ) 79 .5 5( 0. 00 )/ 83 .0 6( 0. 00 )/ 92 .7 8( 0. 08 ) 68 .7 6( 0. 00 )/ 71 .0 7( 0. 00 )/ 90 .7 4( 0. 12 ) U ni fo rm 0. 69 (0 .0 8) /6 7. 38 (0 .3 6) /1 00 .0 (0 .0 0) 66 .9 2( 0. 12 )/ 94 .9 4( 0. 06 )/ 10 0. 0( 0. 00 ) 72 .4 0( 0. 09 )/ 88 .4 7( 0. 12 )/ 10 0. 0( 0. 00 ) 78 .1 4( 0. 09 )/ 95 .9 9( 0. 05 )/ 10 0. 0( 0. 00 ) 54 .7 1( 0. 09 )/ 93 .3 4( 0. 06 )/ 10 0. 0( 0. 00 ) G au ss ia n 0. 00 (0 .0 0) /4 .1 8( 0. 13 )/ 10 0. 0( 0. 00 ) 38 .3 5( 0. 12 )/ 81 .2 2( 0. 11 )/ 10 0. 0( 0. 00 ) 60 .1 4( 0. 03 )/ 80 .4 0( 0. 14 )/ 10 0. 0( 0. 00 ) 58 .2 4( 0. 11 )/ 87 .6 6( 0. 09 )/ 10 0. 0( 0. 00 ) 40 .1 6( 0. 04 )/ 68 .6 9( 0. 11 )/ 10 0. 0( 0. 00 )
SVHN
C IF
A R
-1 0
78 .2
8( 0.
00 )/
85 .0
3( 0.
00 )/
90 .4
4( 0.
12 )
92 .9
5( 0.
00 )/
97 .0
4( 0.
00 )/
98 .0
1( 0.
02 )
90 .0
5( 0.
00 )/
93 .1
3( 0.
00 )/
94 .4
6( 0.
03 )
94 .7
7( 0.
00 )/
98 .9
8( 0.
00 )/
99 .3
1( 0.
01 )
86 .3
9( 0.
00 )/
89 .2
7( 0.
00 )/
93 .3
5( 0.
10 )
C IF
A R
-1 00
76 .9
1( 0.
00 )/
83 .3
0( 0.
00 )/
89 .5
2( 0.
13 )
92 .5
2( 0.
00 )/
96 .8
0( 0.
00 )/
97 .9
6( 0.
02 )
89 .5
6( 0.
00 )/
92 .6
6( 0.
00 )/
94 .2
2( 0.
02 )
94 .5
0( 0.
00 )/
98 .8
7( 0.
00 )/
99 .2
9( 0.
01 )
85 .6
8( 0.
00 )/
89 .1
2( 0.
00 )/
93 .4
7( 0.
08 )
Ti ny
Im (C
) 82
.5 2(
0. 00
)/ 83
.2 7(
0. 00
)/ 10
0. 0(
0. 00
) 94
.0 3(
0. 00
)/ 96
.8 8(
0. 00
)/ 99
.9 6(
0. 00
) 91
.4 2(
0. 00
)/ 93
.8 5(
0. 00
)/ 99
.1 1(
0. 03
) 95
.2 7(
0. 00
)/ 98
.9 8(
0. 00
)/ 99
.9 8(
0. 00
) 88
.4 3(
0. 00
)/ 86
.7 8(
0. 00
)/ 99
.8 8(
0. 01 ) Ti ny Im (R ) 79 .0 5( 0. 00 )/ 84 .4 8( 0. 00 )/ 97 .6 8( 0. 08 ) 93 .5 3( 0. 00 )/ 96 .9 5( 0. 00 )/ 99 .2 2( 0. 02 ) 90 .4 2( 0. 00 )/ 93 .1 2( 0. 00 )/ 96 .3 9( 0. 03 ) 95 .4 1( 0. 00 )/ 98 .9 4( 0. 00 )/ 99 .7 2( 0. 01 ) 86 .8 7( 0. 00 )/ 89 .0 6( 0. 00 )/ 97 .7 5( 0. 05 ) L SU N (C ) 79 .6 3( 0. 00 )/ 84 .0 7( 0. 00 )/ 10 0. 0( 0. 00 ) 93 .0 6( 0. 00 )/ 96 .9 7( 0. 00 )/ 99 .9 0( 0. 00 ) 90 .5 3( 0. 00 )/ 93 .2 4( 0. 00 )/ 98 .5 4( 0. 05 ) 94 .4 5( 0. 00 )/ 99 .0 0( 0. 00 )/ 99 .9 6( 0. 00 ) 87 .2 6( 0. 00 )/ 87 .7 7( 0. 00 )/ 99 .7 3( 0. 01 ) L SU N (R ) 74 .4 5( 0. 00 )/ 78 .3 1( 0. 00 )/ 97 .1 9( 0. 13 ) 91 .5 3( 0. 00 )/ 96 .1 5( 0. 00 )/ 99 .0 4( 0. 02 ) 88 .9 4( 0. 00 )/ 91 .9 4( 0. 00 )/ 96 .2 1( 0. 03 ) 93 .7 6( 0. 00 )/ 98 .6 5( 0. 00 )/ 99 .6 6( 0. 01 ) 84 .6 0( 0. 00 )/ 85 .8 6( 0. 00 )/ 97 .1 2( 0. 06 ) iS U N 77 .0 5( 0. 00 )/ 81 .7 4( 0. 00 )/ 97 .5 1( 0. 25 ) 92 .2 5( 0. 00 )/ 96 .6 0( 0. 00 )/ 99 .1 8( 0. 04 ) 89 .7 3( 0. 00 )/ 92 .6 7( 0. 00 )/ 96 .3 6( 0. 10 ) 94 .5 8( 0. 00 )/ 98 .9 2( 0. 00 )/ 99 .7 3( 0. 01 ) 84 .2 9( 0. 00 )/ 86 .4 6( 0. 00 )/ 97 .3 2( 0. 13 ) U ni fo rm 84 .7 3( 0. 25 )/ 93 .6 7( 0. 12 )/ 10 0. 0( 0. 00 ) 95 .3 5( 0. 05 )/ 98 .0 5( 0. 01 )/ 10 0. 0( 0. 00 ) 92 .4 4( 0. 07 )/ 95 .4 6( 0. 01 )/ 99 .9 7( 0. 01 ) 92 .2 3( 0. 11 )/ 98 .6 4( 0. 00 )/ 10 0. 0( 0. 00 ) 95 .1 7( 0. 04 )/ 96 .7 8( 0. 01 )/ 10 0. 0( 0. 00 ) G au ss ia n 85 .7 6( 0. 16 )/ 92 .0 4( 0. 08 )/ 10 0. 0( 0. 00 ) 95 .7 5( 0. 08 )/ 97 .7 9( 0. 01 )/ 10 0. 0( 0. 00 ) 92 .8 2( 0. 07 )/ 95 .1 8( 0. 02 )/ 10 0. 0( 0. 00 ) 93 .2 2( 0. 25 )/ 98 .4 9( 0. 00 )/ 10 0. 0( 0. 00 ) 95 .4 6( 0. 05 )/ 96 .1 2( 0. 01 )/ 10 0. 0( 0. 00 )

Review summaries:
This paper proposes a new framework for out-of-distribution detection, based on global average pooling and spatial pattern of the feature maps to accurately identify out-of-distribution samples. MAHALANOBIS distance based methods were discussed, and the shortcomings of using mahalanobis distance were given (i.e. assumption that features are independent, etc). They propose to use compression based distance measures off the shelf from standard compression techniques to detect spatial feature patterns in feature space and demonstrate its effectiveness on several datasets and comparison with baselines is reported and well discussed.