Understanding Classifiers with Generative Models
Authors: Laëtitia Shao
Yang Song
Stefano Ermon
Venue:  (2021)
Keywords: OOD detection
adversarial samples detection
deep learning
classification

Abstract: Although deep neural networks are effective on supervised learning tasks, they have been shown to be brittle. They are prone to overfitting on their training distribution and are easily fooled by small adversarial perturbations. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize.
We propose a generative model of the features extracted by a classifier, and show using rigorous hypothesis testing that errors tend to occur when features are assigned low-probability by our model. From this observation, we develop a detection criteria that we test against different sources of classification mistakes: mistakes made on the test set due to poor model generalization, adversarial samples and out-of-distribution samples. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.

samples, but surprisingly also identify inputs from Under review as a conference paper at ICLR 2021
UNDERSTANDING CLASSIFIER MISTAKES WITH GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Although deep neural networks effective supervised learning tasks, they have been shown to be brittle. They are prone to overfitting on their training dis- tribution and are easily fooled by small adversarial perturbations. In this paper, we leverage generative models to identify and characterize instances where clas- sifiers fail to generalize. We propose a generative model of the features extracted by classifier, and show using rigorous hypothesis testing that errors tend to occur when features are assigned low-pre www-robabbility by our model. From this observation, we develop detection criteria for samples on which classifier is likely to fail at test time. In particular, we test against three different sources of classification failures: mistakes made on the test set due to poor model generalization, adver- sarial samples and out-of-distribution samples. Our approach agnostic to class labels from the training set which makes it applicable models trained in semi- supervised way.
INTRODUCTION
Machine learning algorithms have shown remarkable success in challenging supervised learning tasks such as object classification (He et al.] (0166) and speech recognition (Graves et al. 2013). Deep neural networks particular, have gained traction because of their ability learn hierarchi- cal feature representation of their inputs. Neural networks, however, are also known to be brittle. As they require large number of parameters compared available data, deep neural networks have tendency to latch onto spurious statistical dependencies to make their predictions. As result, they are prone to overfitting and can be fooled by imperceptible adversarial perturbations of their inputs (Szegedy et al., 2013 Kurakin al.) 2016; Madry et al.) (017)) Additionally, modern neural networks are poorly calibrated and do not capture model uncertainty well (Gal & Ghahramani 2016 Kuleshov & Ermon 2017, Guo et al., 2017). They produce confidence scores that do not represent true probabilities and consequently, often output predictions that are over-confident even when fed with diitif-distribution inputs (Liang et al. 2017). These limitations of neural networks prob- lematic as they become ubiquitous in applications where safety and reliability is priority (Levinson) al. 2011
Sun al. 2015).
Fully probabilistic, generative models could mitigate these issues by improving uncertainty quan- tification and incorporating prior knowledge (c.g. physical properties (Wu et al. 2015)) into the classification process. While great progress has been made towards designing generative models that can capture high-dimensional objects such as images (Oord et al., 20166a Salimans al. 2017). accurate probabilistic modeling of complex, high-dimensional data remains challenging.
Our work aims at providing an understanding of these fallure modes under the lens of probabilistic modelling. Instead of directly modeling the inputs, we rely on the ability of neural networks to extract features from high-dimensional data and build a generative model of these low-dimensional features. Because deep neural networks trained to extract features from which they output clas- sification predictions, we make the assumption that it is possible to detect failure cases from the leamed representations.
Given a neural network trained for image classification, we capture the distribution of the learned feature space with Gaussian Mixture Model (GMM) and use the predicted likelihoods to detect inputs on which the model cannot produce reliable classification results. We show that we are able to not only detect adversarial and

---

Under review as a conference paper at ICLR 2021
the test which model is likely make mistake. We experiment state-of-the-an art neural networks trained on CIFAR-10 and CIFAR-100 (Krizhevsky) 2009) and show, through statistical hypothesis testing, that samples leading to classification failures tend to correspond to features that lie in a low probability region of the feature space.
Contributions Our contributions are as follows:
We provide a probabilistic explanation to the brittleness of deep neural networks and show that classifiers tend to make mistakes inputs with low-probability features.
We demonstrate that a simple modeling by a GMM of the feature space learned by a deep neural network is enough to model the probability space. Other state-of-the-art methods for probabilistic modelling such as VAEs (Kingma & Welling 2013) and auto-regressive flow models (Papamakarios et al. 2017) fail in that regard.
We show that generative models trained on the feature space be used as a single tool to reliably detect different sources of classification failures: test set errors due to poor generalization, adversarial samples and out-of-distribttion samples.
2
RELATED WORK
An extensive body of work has been focused on understanding the behaviours of deep neural net- works when they are faced with inputs on which they fail. We provide brief overview below:
Uncertainty quantification Uncertainty quantification for neural networks is crucial in order to detect when a model's 's prediction cannot be trusted. Bayesian approaches (MacKay 1992 Neal 2012 Blundell et al., 2015), for example, seek to capture the uncertainty ering prior distribution over the model's weights. Training these networks is challenging because the exact posterior is intractable and usually approximated using a variety of methods for poste- rior inference. Closely related, Deep Ensembles (Lakshminarayanan et al. 2017) and Monte-Carlo Dropout (Gal & Ghahramani] 2016) consider the outputs of multiple models as an alternative way approximate the distribution. Model calibration (Plati) 1999] Guo al.) 2017) aims at producing confidence score that are representative of the likelihood of correctness. Uncertainty quantification may also be obtained by training the network to provide uncertainty measures. Prior Networks (Ma- linin & Gales, 2018) model the implicit posterior distribution in the Bayesian approach, De Vries & Taylor (2018); b Lee et al. (2017) have the network output an additional confidence output. These methods require proxy dataset representing the out-of-distribution samples to train their confidence scores.
network by consid-
Our method differs from the above it seeks to give an uncertainty estimation based on model trained with the usual cross-entropy loss. does not require additional modelling assumptions, nor modifications to the model's architecture or training procedure. As such, it relates closely to threshold based methods. For example, Hendrycks & Gimpel (2016) use the logits outputs as a measure of the network's confidence and can be improved using Temperature Scaling (Guo al., 2017 Liang et al., 2017). a post- post-processing method that calibrates the model. Our work derives confidence score by learning the probability distribution of the feature space and generalizes to adversarial samples (Szegedy et al. 2013), another source of neural networks' brittleness.
Methods to defend against adversarial examples include explicitly training networks be more robust adversarial attacks (Traner et al., [017]] Madry et al. 2017 Papernot et al. 2015). Another line of defense comes from the ability to detect adversarial samples at test time. Song et al. (2017) for example, use a generative model trained on the input images detect and purify adversarial examples at test time using the observation that adversarial samples have lower predicted likelihood under the trained model. Closer to our work, Zheng & Hong (2018) and Lee et al. 2018) train conditional generative model on the feature space learned by the classifier and derive a confidence score based on the Mahalanobis distance between a test sample and its predicted class representation. Our method makes the GMM class agnostig, making it applicable to settings where labels are not available at inference time. We further show that the unsupervised GMM improves the Mahalanobis score the OOD detection task.
Adversarial samples
2

---

Under review as a conference paper at ICLR 2021
3 DETECTING MISTAKES
Figure I: Predicting whether an image will be correctly classified is challenging. Left: Images from the train set. Middle: Adversarial images computed on the same images with FGSM-0.1 are indis- tinguishable from the clean images, yet. they fool our classifier into making incorrect predictions. Right: Images from the test set. The images look similar to images from the training set, yet, they are incorrectly classified by our DenseNet model. Each row represents a different class.
Detecting samples on which a trained classifier is likely to make a mistake is crucial when consid- ering the range of applications in which these models are deployed. However, predicting in advance whether sample will fail seems challenging, especially when the sample is drawn from the same distribution as the train set. To illustrate this, we show Fig. samples from the CIFAR-100 train- ing dataset and compare them to test samples and adversarial examples that our DenseNet model fails to classify properly. In both cases, it is not obvious to the human eye what fundamentally dif- fers between correct and incorrect samples. Our main intuition is that a generative model trained on the feature space could capture these subtle differences.
3.1 BACKGROUND
We consider the problem of classification where we have access to (possibly partially) labeled dataset D {(x,, 1:) where (X,, y:) E X' V. Samples are assumed to be independently sampled from a distribution Pdato (X, 3) and we denote the marginal over X as Pdata (X). We will denote ʃo : X' + F = RD the feature extractor part our neural network, where represents the parameters of the network and F is the feature space of dimension D. Given input X, the predictions probabilities on the label space y are then typically obtained using multivariate logistic regression on the extracted features.
(1) where (W, b) represent the weights and bias of the last fully-connected layer of the neu- ral network. The model prediction is the class with the highest predicted probability: ((X = arg maxyey X(y|X, O, W, b). The parameters (0, W, b) are trained to minimize a cross- entropy loss on the training set and performance evaluated on the test set.
p(y) X, O, W, b) softmar W [x( + b)
Learning the data structure with Generative Models Understanding the data structure can greatly improve the ability of neural models generalize. Recently, great progress has been made in designing powerful generative models that can capture high-dimensional complex data such as images. PixeICNN (Salimans et al., 2017: Oord et al.] 2016b ab in particular,
state-of-the-ar deep generative model with tractable likelihood that represents the probability density of an image as fully factorized product of conditionals over individual pixels an image.
PCNN X = II
Ps Xx (X X
(2)
i=1
Flow models such as the Masked Autogressive Flow (MAF) (Papamakarios et al. 2017) model pro- vide similar tractability by parameterizing distributions with reversible functions which make that likelihood easily tractable through the change of variable formula. Another widely used class of generative models assumes the existence of unobserved latent variables. Gaussian Mixture Models, for example, assume discrete latents (corresponding to the mixture component). Variational autoen- coders (Kingma & Welling 2013)) use continuous latent variables and parameterize the (conditional) distributions using neural networks.
3

---

Under review as a conference paper at ICLR 2021
3.2 MODELING THE FEATURE SPACE
We identify two main reasons why characterizing examples over which classifier is likely to make mistake is difficult. First, modeling the input data distribution Pdata (X), as done in Song et al. (217)) to detect adversarial examples, is challenging because of the high-dimensional, complex nature of the image space X. This approach also fails detecting out-of-distribution samples, with state-of-the art models assigning higher likelihoods to samples that completely differ from their train set (Nalisnick et 2018). Second, a model of Pduta (X) doesn't capture any information about the classifier itself.
To overcome these difficulties, we propose to model the underlying distribution of the learned fea- lures F So( (X), where X Pdata (X). Extracted features have lower dimension which makes them easier to model and they give access to some information on the classifier. Specifically, we interested in comparing features F. of samples that are correctly classified with features F of samples that are incorrectly classified by a trained neural network. F. and F can be described as elements of the following sets:
F E [Jo()) = (X, E EX }}
(3)
F.. E W= (ox + y. (X, n) EX x }}
(4)
The distribution of the extracted features is modeled by:
P(F) Σ (F: Ex)
(5)
where K is the number of Gaussians in the mixture, TK. M3g Σg are the model parameters. We choose Σn to be diagonal in all our experiments. After training neural network to convergence, we learn the parameters the GMM using the EM algorithm. Our training set is built from the features extracted from the training image set by the trained classifier.
3.3 DETECTING CLASSIFICATION MISTAKES
We posit that classification mistakes are linked to extracted features that are unusual under the train- ing distribution. By modeling the feature space learned by the classifier, our generative model will be able to detect an input that will lead to a potential classification mistake. We found that a simple generative model is surprisingly good at capturing the distribution of the feature space and can detect when an input will lead to a classification mistake hased on its predicted feature log- ligliikl
Statistical Hypothesis Testing We consider Pc the distribution of features F, fo (X)
where (X, y) ~ Pdata (X, y) and j(X) = y. and (F the distribution of features F.. - (X)
where (X, y) Pdtata (X,y) and xXX A y. These correspond features extracted correctly
classified vs. incorrectly classified examples. Note that these distributions not only depend on the underlying data distribution but also on the classifier's 's parameters (0, W. W).
Assuming we have access to samples Fo,1....; Fen hypothesis Hg and alternative hypothesis H₃ are:
and F ww...... F Fum pw our null
Ho PC Pw : # Pw
We use the Mann-Whitner Whitney U-test, which assumes that samples can be ranked. The test statistic is defined by ranking all samples of the two groups together and using the sum of their ranks.
Uc = Rc n(n 1)
Uw=Rpy m(m + 1)
2
where Re and Rw are the sum of ranks of samples F. and to respectively. The statistic for the statistical test is U = min(Uc. Uw), which has distribution that can be approximated by normal distribution under the null hypothesis. In our approach, samples are ranked based on their predicted probability.
Since our test statistic directly uses the predicted likelihood of a feature, we deduce from it a simple per-sample test to determine if input is likely to be misclassified. Given threshold T, test sample X is rejected as being misclassifidd if p(fo())) < T. The value the threshold chosen by cross-validation the validation set to obtain good trade-off between precision and recall.
4

---

Under review as a conference paper at ICLR 2021
Distribution predicted by PixeiCNN
Distribution predicted PixelCNN
Distribution predicted by PixelCNN
0.00010
0:00000
0.00025
9,00025
Densly
Dersitty
3.00002
Derrity
0,00000
0.00015
2.00015
9.60015
0.00310
0.00000
0/0000
Correct Samples
Correst Samples
Correct Samples
0,00055
Incorrect Samples
0.00005
Incorract Samples
0.0000
Incornect Samples
0.00000
5:00000
0:00000
Log likelihood
Log likellhood
likelihood
Distribution predicted by GMM-100
Distribution predicted by GMM- 100
Distribution predicted GMM 100
1.00
2,0000
D/00
Dersity.
000
5.0035
Dersity
8.000
0.004
9.0000
0.000
0.002
Correct Samples
Correct Samples
8:01
Correct Samples
Incorrect Samples
Incorrect Samples
Incorrect Samples
0.00%
5:0000
5:000
likelinood
likelihood
Log likelhoad
(a) DenseNet- 100
(b) WRN-28
(c) TE-WRN-28 Accuracy: 64.44%
Accuracy: 72.76%
Accuracy: 80.22%
Figure 2: Comparing the predicted log likelihood distribution of correct (blue) and incorrect (orange) samples from the test set for models trained on (IFAR-100. Top row: Log likelihoods obtained from training PixeICNN on images from the train set. Bottom row: Log likelihoods obtained from training GMM-100 on features extracted from the train set.
4 EXPERIMENTS
We run experiments the CIFAR-100 dataset, containing 32 32 color images used for image classification with 100 classes. All reported results give the mean and standard deviation over 5 independent runs. Additional experiments on model trained on the smaller CIFAR-10 dataset are also available in the appendix. We examine two state-of-the- 100 (Huang et al., 2016) and Wide ResNet-28 (Zagoruyko & Komodakis 2016) trained with the usual cross-entropy loss. In the setting where only small number of labels is available, we train a WRN-28 model with 100 labeled samples per class using Temporal ensembling (Laine & Aila 2.16). This self-ensembling training method takes advantage of the stochasticity provided by the use of dropout and random augmentation techniques (e.g. random flipping and cropping).
deep neural networks, DenseNet-
Mistake Detection Using statistical testing, we verify that the trained model learns a distribution that differentiates correct and incorrect samples. We sum up the performance of our method by reporting the AUC-ROC and AUC-PR obtained on the test set.
To motivate the use of high-level features, we adapt the detection method used by Song et al. (2017) to the mistake detection problem and compare the performance with our proposed method. We train a PixeICNN on the image dataset and use the predicted likelihood values detect classification mistakes, We evaluate mistake detection on the test set and first compare the distribution predicted by PixeICNN on the images with the distribution predicted by a GMM-100 model on extracted features (Figure 2) Using the Mann-Whitney Whitney U-test, we verify that the distribution learned by GMM-100 differentiates correct and incorrect samples (p- 1.9c 13). On the other hand, because PixelCNN is trained without knowledge of the classifier's internal representations, the distributions of correct and incorrect samples predicted under PixelCNN are almost indistinguishable (p- 8.58e -5)
Additionally, we experimented with more flexible likelihood models to model the feature space such as the Variational Auto Encoder (Kingma & Welling 2013) and Masked Autoregressive Flow (Pa- pamakarios et al. 2017). Surprisingly, we found that simple Gaussian Mixture Model is better at detecting classification mistakes than these more flexible models. Finally, we also compare with other threshold- based methods: using the predicted logits and calibrated scores obtained after Tem- perature Scaling. Detection performance is summed up in Figure (3 for DenseNet and WideResNet models trained on CIFAR-100. GMM models trained on the features outperform all other generative
5

---

Under review as a conference paper at ICLR 2021
ROC
ROC
1.0
PixelCNN: VAE: 50.20 0.54
PixeICNN: VAE: 64.35 0.42
0.0
MADE: 54,66 2.30
MADE: 70.26 0.43
0.0
Logits: 85.93 0.28
Tue Positbee
Logits: 86.51 0.331
Temp. Scal.; 86.01 0.24
Temp. Scal.: 88.26 = 0.29
0.4
GMM-100: 62.42 # 0.81
GMM-100: 84.25 # 0.44
0.2
GMM-1000: 68.64 0.53
GMM-1000: 85.55 0.36
False Positive Rate
False Positive Rate
PR (correct)
PR (correct)
1.0
PixeICNN:
1.00
PixelCNN:
VAE: 73.72 0.50
VAE: 88.66 0.25
MADE: 77.00 1.63
0.50
MADE: 90.79 0.20
Precison
Logits: 94.46 a 0.11
0..0
Logits: 96.84 0.00
Temp. Scal.: 94.43 0.10
Temp, Scal.; 96,75 0.06
GMM-100; 82.13 0.60
GMM-100: 95. 0.09
0.70
GMM-1000: 85.87 0.37
GMM-1000: 95.75 0.06
0.6
0.66
Recall
Recall
PR (incorrect)
PR lincomect)
PixelCNN: 25.33 VAE: 26.93 0.29
PixelCNN: VAE: 26.09 0.50
MADE: 29.29 1.48
0.6
MADE: 31.53 0.86
Logits: 66.17 = 1.08
0.7
Logits: 63.92 1.44
Temp. Scal.; 66.76 = 1.09
Temp. Scal.: 63.40 1.44
GMM-100: 33.97 = 0.65
GMM-100: 53.55 = 1.13
GMM-1000: 39.76 0.69
GMM-1000: 57.02 1.30
Recall
Recall
(a) DenseNet
(b) Wide ResNet
Figure 3: Comparison of ROC and PR curves for detecting classification mistakes using different generative models. Using PixeICNN to model the image space fails at reliably detecting classifica- tion mistakes. GMMs trained on the feature space achieve better detection than other more flexible models like VAE and MAF and are comparable to temperature scaling the WRN model.
models trained either on images on the feature space. We find that using a GMM has similar per- formance than calibrated scores on the Wide ResNet but not on the DenseNet. This is explained by the fact that our DenseNet model has much lower accuracy than Wide ResNet (72.76% v. 80.22%) and therefore does not produce overly confident predictions. Additional results are available in the appendix.
In the next experiments, we show that although using predicted logits provides reliable detection of test set mistakes, this metric doesn't generalize to adversarial or out-of-detection samples. On the other hand, our approach to train generative model on the feature space can be applied to these other sources of classification errors.
Adversarial samples We craft adversarial samples from test samples using the Fast-Gradient Sign Method (FGSM) proposed by Goodfellow et al. (2014) and the Basic Iteration Method (BIM) (Ku- rakin et al.] 2016). Both methods move the input image in the direction of the gradient of the loss while restraining the adversarial sample to in a é ball of ray fattack around the original input. This ensures that the generated adversarial sample is visually indistinguishable from the original.
Figure 4 shows that the GMM is sensitive to features extracted from adversarial samples, as they are assigned higher BPDs than clean samples. We also plot the ROC curves and corresponding AUC metrics that are obtained by using the predicted BPD to detect adversarial samples.
We compare approach with other possible detection metrics. In particular, the method proposed by Zheng & Hong (2018) and the Mahalanobis score from Lee et al. (2018) also leverage the feature space detect adversarial inputs. These approaches require labels to train while we only train one GMM in an unsupervised manner. ROC curves are shown in Figure and full comparison table with higher cattack values for both attacks is shown in
one different model per class and therefore

---

Under review as a conference paper at ICLR 2021
Distribution predicted GMM-1100
Distribution predicted by GMMM-1000
Distribution predicted GMM-1000
0.007
Clean Samples
0:00000
Samples
0:0035
Clean Samples
0.005
FGSM-0022
BIM-0.2
0.00375
5.00150
BIM-0.2
FGSM-0.2
5,0000
BIM-0.2
FGSM--02
Densty
0.003
2000325
0,0003
0.001
0000000
800000
5.00073
5.00000
0.00033
0.000
5:00002
500000
Log probsbility
probsbility
Log probability
ROC
ROC
ROC
1.0
0.8
Presitive Rate
Trus Pecitive Rate
0.6
FGSM-0.05; 92.16 1.14
FGSM-0.05: 76.68 0.57
9.4
FGSM-0. 76.49 1.11
FGSM-0.1: 98.83 = 0.49
FCSM-0:1: 83.99 0.57
F35M-0:1: 85.79
0.4
FGSM-0.2: 99,96 0.02
0.4
FGSM-0.2: 89:53 1.49
0.4
FGSM-0-2: 9:13
BIM-0.05: 1,002
8IM-0.05: 69.8
BIM-0.05: 2.72
0.2
BIM-0.1: 99.37 0.27
BIM-0.1: 76.95 :
92.16:::2
BIM-0.2: 99.99 0:01
BIM-0.2: 03.42 x 3.69
BIM-0.2: 99.02 0.73
0.0
False Positive Rate
False Positive Rate
False Positive Rate
(a) DenseNet (C100)
(b) WRN (C100)
(c) TE- WRN (C100)
Distribution of log-likelihood predicted on clean (blue) and adversarial samples (green and orange) by a GMM-1000. The log-likelihood of features extracted from adversarial samples is lower. The histograms are separated, which means it is possible to detect adversarial samples using the log-likelihood of their features. Bottom: ROC curve for detecting adversarial samples using predicted liglliiel Our method achieves good trade-off between true positive and false positive rate, significantly improves over chance and achieves between 76% and 100% AUC depending on attack methods and models.
Figure 4: Top:
the appendix. Our method, using a GMM-1000 provides better detection performance of adversarial samples than calibrated and non-calibrated logit scores. Most notably, in semi-supervisee setting (Figure (3c), our method surpasses all others on attacks with low Cattack values.
BIM 0.05
BIM 0.05
BIM 0.05
1.0
3.0
True Posttive Rate
0.0
True Pesitive Rabe
Irus Postivve Raha
0.8
0.6
0/2
9.4
GMM100C: 92.83 1.02
GDM0 69.63 0.64
GMM1000: =
0.4
Logits: 52 86 2.85
0.4
Logit:: 63.61 0.67
94
Logits/ 45 19.99 = 1.39
Callarated:: 2.08
Calibrated: 62.81
Calibaatd: 44.65
02
Mahalanabis: 90.10 1.36
82
Mohalanobis: 2
Mahalanabis: 76.80 3.52
Zhang: 90.45
Zhang: 69.11 0.66
Zhang: 61.95 2.94
0.0
60
58
30
02
0.4
0.6 088
10
0.0
022
0.4 0.6
0.
10
00
02
04
0.6 0.6
10
False Positive Rate
False Positive Rate
False Positive Rate
(a) DenseNet-100
(b) WRN-28
(c) TE-WRN-28
Comparison of ROC curves for adversarial sample detection using different metrics. Logit- based scores (Logits and Calibrated) can not reliably detect adversarial samples properly while meth- ods that model the probability space of the feature space can (GMM1000, Mahalanobis, Zheng). Our method achieves comparable detection results as the Mahalanobis and the Zheng metric and surpasses both of them in semi-supervised setting (Figure 5c)
Figure 5:
hnttrf-Distribution Detection We also test the use of feature log-likelihood values on the task of detection out-of-distiibution samples. As Out-of-Distribuu sian Noise, SVHN (Netzer et al., 2011), Tiny ImageNet (Russakovsky et al.) 2015), and Fashion MNIST (Xiao et al. 2017). OOD detection results are reported in Table each model we trained. Our experiments show that it is not possible to rely on calibrated probability scores for OOD detec- tion, and that our method yields better detection results than using the Mahalanobis score in some cases. We also highlight that a PixelCNN trained on CIFAR has very poor detection results on image datasets that visually look very different from its original training set (FAST and SVHN).
samples we use Random Gaus-
7

---

Under review as a conference paper at ICLR 2021
This is result of the generative model assigning higher likelihood to these OOD samples. Table in the annex also shows that only calibrated scores fail to detect random gaussian noise as an OOD sample.
Table 1: OOD Detection results. Calibrated scores do not detect OOD samples well, especially samples from SVHN. On the other hand, our detection method using GMM on the feature space able to reliably detect out-of-distribution samples and performs better than the class-dependent Mahalanobis score even on fully-supervised models. PixeICNN assigns higher likelihoods to OOD samples that diverge clearly from the original training set and fails to detect them.
(a) Out-distribution: SVHN
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-1000
95.70 0.36
83.60
1.04
99.23 0.07
Mahalanobis
94.38 0.39
80.06
0.87
98.83 0.09
DenseNet- 100
Calibrated Scores
80.58 2.00
56.60
4.74
95.70 0.44
GMM- 1000
81.40 3.74
53.64 6.02
96.45 = 0.87
Mahalanobis
75.86 2.72
47.40 6.42
93.96 0.61
WRN-28
Calibrated Scores
78.64 + 1.70
56.23 + 3.00
95.05 + 0.52
GMM- 1000
72.45 7.41
40.62 1 7.64
93 93.47 2.03
Mahalanobis
49.46 + 5.52
17.32 5.19
86.54 + 1.36
TE-WRN-28
Calibrated Scores
62.10 4: 3.97
35.46 + 4.82
89.82 + 1.10
Mldel--Agnsste
PixeICNN
20.18 + 0.41
7.33 + 0.10
75.97 E 0.10
(b) Out-distribution:
Tiny ImageNet
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-1000
96.75 + 0.45
96.78 + 0.42
96.67 = 0.51
DenseNet-100
Mahalanobis
95.26 + 0.61
95.68
0.55
94.57 1 0.70
Calibrated Scores
74.00 + 2.01
4.37 73.49
71.45 1.09
GMM-1000
84.98 1.86
86.32
1.95
82.45
1.92
WRN-28
Mahalanobis
80.35 + 2.92
82.72
2.62
76.01 3.91
Calibrated Scores
82.64 1.65
87.46 1.43
79.40 1.56
GMM- 1000
67.94
67.82 1.44
68.59 2.84
TE-WRN/C100
Mahalanobis
70.63 2.46
70.66 +
70.08 3.10
Calibrated Scores
56.41 0.96
59.22 = 1.43
54.01 + 0.49
Model-Agnostic
PixeICNN
82.05 0.14
78.63 0.16
81,87 0.19
(c) Out-distribution
Fashion-MNIST
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-1001
93.23 1.31
89.29 2.75
98.30
0.36
DenseNet- 100
Mahalanobis
94.48 0.41
88.95 1.04
98.12 0.15
Calibrated Scores
85.92 1.72
3.05 65.72
96.60 0.5
GMM-
87.32 2.12
69.72
96.82 0.67
WRN-28
Mahalanobis
79.33 + 1.90
57.26 2.33
93.7 1 0.80
Calibrated Scores
86.01 = 0.99
70.37 1.49
96.24 0.39
GMM- 1000
66.95 3.85
36.79
4.06
91.29
1.34
TE-WRN-28
Mahalanobis
63.96 1.73
32.71 = 2.00
89.88 + 0.75
Calibrated Scores
70.11 2.27
13.56 1 3.38
91.72
0.87
Model-Agnan Aggnotic
PixeICNN
0.71 0.09
7.51 1 0.00
67.70 1 0.02
5 CONCLUSION
Using statistical hypothesis testing we provided general characterization of inputs that lead to clas- sification mistakes by deep neural networks. With a simple Gaussian Mixture Model, we model the distribution of the feature space learned by classifier and verified that features extracted from in- puts consistently lie outside of the training distribution and can be detected by their low predicted log probability. Compared to other score-bar based methods, characterization holds for variety of clas- sification failure modes in deep neural networks: adversarial sample detection, nut-of-distribution detection and test time classification mistakes.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks, 2015.
Terrance DeVries and Graham W. Taylor. Learning confidence for -utttr-diittibuutin detection in neural networks, 2018.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:141..6572 2014.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649. IEEE, 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. CoRR, abs/1 2017, URL http/////a/xxxvorr//aa////////// wttp:///////////..4....000 706.04459
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outoof-distriibutton ex- amples in neural networks. CoRR, abs/1610.02136, 2016. URL http:///aaaii..o///////1.......1
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. CoRR, abs/1608.0699) 2016. URLLhhtp//axxiiaar//bbb///6000/0
Diederik Kingma and Max Welling. arXiv:1312.6114.00 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Volodymyr Kuleshov and Stefano Ermon. Estimating uncertainty online against an adversary. In AAAI, pp. 2112-11 2017.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale, CoRR, abs/1611.01236, 2016. URL http:://aaxii..rr///////1.......
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242, 2016.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, pp. 6402-6413, 2017.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-callirraeed classifiers for detecting -ut-of-distributtin samples, arXiv preprint arXiv:1711.09325, 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A unified framework for detecting out-of- distribution and adversarial samples. 2018.
Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, J Zico Kolter, Dirk Langer, Oliver Pink. Vaughan Pratt, et al. Towards fully autonomous driving: Sys- tems and algorithms. In 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 163-168. IEEF, 2011.
Shiyu Liang, Yixuan Li, and R. Srikant. Principled detection of out-of cadt-f--isttibutton examples in neural networks. CoRR, abs/1706.02690, 2017. URL T http://arrv.oorraas///00002

---

Under review as a conference paper at ICLR 2021
David JC MacKay, A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448 472, 1992.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: (60.00370600883 2017.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks, In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't 't know?, 2018.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: 0/66660007999 2016a.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05338,, 2016b.
George Papamakarios. Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances Neural Information Processing Systems 30, 2017.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense adversarial perturbations against deep neural networks. CoRR, abs/151 1,04508, 2015. URL http:////aaaiiiorr//////S/SS1...
John C. Platt. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pp. 61-74. MIT Press, 1999.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-22) -2-222 2015. doi: 10. 1263-001-82----011111100 06---166--
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.055: 2017.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models understand and defend against adversarial examples. CORR, abs/1710.10766, 2017. URL http://aaxxi.oog//bb///111 10766
Yi Sun, Ding Liang, Xiaogang Wang, and Xiaoou Tang. Deepid3: Face recognition with very deep neural networks. CoRR, Abs/1502.,0083, 2015. URL http:///aaii..orr/a/////// http///bbb/552.00777777 1502.00873
Christian Szegedy. Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. CORR, abs/1312.6199, 2013. URI. http://arxivvon//aas//1111.619
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel, Ensemble adversarial training: Attacks and defenses, 2017.
Jiajun Wu, Ilker Yildirim, Joseph Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In Advances in neural information processing systems, pp. 127-135, 2015.
10

---

Under review as a conference paper at ICLR 2021
Han Xiao, Kashif Rasul, and Roland Vollgraf. fishion-mniste a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 70X1:1100..77777 2017.
Sergey Zagoruyko and Nikos Komodakis. 17X1::160..07146, 2016.
Wide residual networks.
arXiv preprint
Zhihao Zheng and Pengyu Hong. Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks. In Advances in Neural Information Processing Systems, pp- 7924-7933, 2018.
A ADDITIONAL. DETECTION RESULTS
Distribution predicted by PxxeCCN
Distribution predicted by GMM-1000
9.00055
0.007
Clean Samples
0.003
FGSM-0.2
0.00025
BIM 0.2
0.00000
0.004
Correct Samples
0.00005
Incorrect Samples
0.01
0.00000
0.022
-12"
1pf Log probability
Log likelhhood
Distribution predicted by GMM--100
ROC curve
1.0
True Positive Rate
Density
0.008
0.00
FSSM-0.05 0.32
FGSM 0:1: 99.96 0.01
8.024
FGSM-0.2: 0.00
B.M.D.05: 97.76 0.44
0.02
Correct Samples
02
Incorrect Samples
B.M-D.2: 100. 000.0000000 0.00
0.000 2
-10 Log likelihood
-10
CA 0.4 aa
1.0
False Positive Rate
(a) Log-likelihood distribution of correct and incorrect samples
(b) Top: Log-likelihood distribution of clean and adversarial samples, Bottom: ROC-curve for adversarial detection
Figure 6: Additional results for model trained on CIFAR-10. Left: comparison of log-likelihood distributions. PixeICNN not able to distinguish correct samples from incorrect ones while a GMM trained on the feature space can. Middle: GMM trained on the feature space can detect adversarial samples reliably.

---

Under review as a conference paper at ICLR 2021
FGSM
True Rositive Rate
6.88
True Positivee Rate
K1.5
GMM1000: 97.76 0.4
GMM1000: 9 0.32
84
Loois:: 28.78
0.4
Calibrated: 77.63 5.26 5.25
Calibrated: 81.21 1.05
Mahalanobis: 97.63 0.70
Mahalanobis: 95.74 0.78
Zheng: 96.85 0.76
Zheng: 96.25 0.33
be False Posttive Rate
oe False Positive
Trae Rostiive Rate
2.8
True Positive Rate
GNM1000: 99.99 0.00
GMM1000: 99.96 0.01
84
Logis::
0.4
Logits:
Calibrated: 75.44 = 7.37
Calibrated:
Meholanobis: 99.96 0.01
Mohalamobis: 99.87 0.01
Zheng: 99
Zheng: 99.90 0.03
0.0
bet False Positive
04 Positive
FGSM
True Positive Rate
8.8
GMM1000: 100.00 00 1 0.00
True Posilivve Rate
GMM1000: 100.00 0.00
Logits: 62.95 10.21
Logits: 83.74 5.24
Calibrated:
Calibreted::
0.4
Mahelanobis: 100.00 = 0.00
8.4
Mahalanotis: 100.00 0.00
Zheng:
Zheng:
be False Posttive Rate
Positive
Figure 7: Comparison of ROC curves for FGSM and BIM adversarial sample detection for CIFAR- 10 model.
12

---

Under review as a conference paper at ICLR 2021
ROC
ROC
1.0
PixelCNN:
PixelCNN:
VAE: 42.94
VAE: 63.82 0.44
MADE: $6.79 0.75
MADE: 71.39 1.15
0.6
Logits: 84.34 0.11
Positbe
Logits: =
Temp. Scal.; 84.21 0.17
Temp. Scal.; 93.85 0.34
GMM-100: 62.91 # 0.82
GMM-190: 79.48 0.95
0.2
GMM-1000: 65.06 0.43
GMM-1000: 80.93 0.69
False Positive Rate
False Positive Rate
PR (correct)
PR (correct)
PixelCNN: VAE: 61.22 0.40
PixelCNN: VAE: 96. 0.13
0.9
0.50
MADE: 71.84 0.74
MADE: 97.71 0.12
0.50
Procisonn
Logits: 91.21 1 0.12
Logits: 99.59 1:0 0.03
0.94
0.7
Temp. Scal.: 91.11 0.12
2.000
Temp. Scal.: 99.60 + 0.02
GMM-100; 77.65 0.60
GMM-100: 98 98 0.06
0.50
GMM-1000: 7 0.39
GMM-1000: 98.59 0.07
0.m8
Recall
(incorrect)
PR lincorrect)
32.88 0.13
VAE: 30.66 0.18
VAE:
0.6
MADE: 39.42 0.60
MADE: 10.56 0.79
Logits: 71.43 0.22
Logits: 44.69 1.35
Temp. Scal.: 71.25 0.24
Temp. Scal.; 45.99 1 1.30
GMM-100: 42,90 0.65
GMM-100: 15.56 1.21
GMM-1000: 45.37 0.48
GMM-1000: 17.21 0.90
0.4
0.3
0.0
Recall
(a) ROC curves
(b) PR curves (correct)
Figure 8: Additional comparison of ROC and PR curves for detecting classification mistakes using different generative models. Using PixeICNN to model the image space fails at reliably detecting classification mistakes. GMMs trained on the feature space achieve better detection than other more flexible models like VAE and MAF and are comparable to temperature scaling on the WRN model.
13

---

Under review as a conference paper at ICLR 2021
0.8
Pozitive Rete
9-3
Posiivv 1o50
0.0
Tree Posiive
GMM1000: 1.114
GMM10000 76.68 0.57
GMM1000: 76.49 1.11
0.4
Logits: 1.002
244
Logit:: 76.67 0.228
0.28
24
Logits: 62.59 0.42
Calibrated: 0.62
Calibrated: 75.00
Callbrated: 0.28
Mahalanobis: 87.12 1.39
Mohalanoals: 73.17 1.50
Mahalanohis: 73.13+ 1.66
Zheng: 9 0.88
Zheng:
Zheng:
0.0
24
0.0
0.5
54 58
0.0
B. Festtive
10
0.0
022
0.4
be 6-8
3.5
False Positive
0.1
True Posiiiv Rate
0.8
Pootive Rebe
Postive
GMM1000: 0
GMM10000 83.99 39.99-2 0.57
GMM1000: 85, 2.31
0.4
Locits: 80.91 2:2.2
244
Logits: 83.89 0.49
2.4
Locits: 67.91 0.89
Calibrated:
Callbrated: 81.88
Calibrated:
Mahalanobis: 98.39
0.41
Mahalanobis: 61.76
1.45
Mahalanobis: 84.31 3.20
Zheng: 28
Zheng:
Zheng:
0.0
24
0.0
0.5
0-2
24
1.
0.0
0.2
0.
10
0.5
2-2
0.4
be b.
3.
False Positive
False Positive
False Positive Rate
0.8
Positive Rete
9-
Posiiive 1
0.0
Tree Posiiive
GMM1000: 99.96 a 0.02
GMM1000: 89.53 :: 1.49
GMM1060: 93.34
8.8
Logits: 3.412
24
Logits: 89.45 0.83
2.
Logits: 59.05 3.12
Calibrated: 80.55 4.82
Calibrated: 87.06 0.64
Calibrated: 3.22
Mahalanobis: 99.91
0.04
Mohalanabis: 91.58 1.17
Mahalanobis: 94.43 2.58
Zheng: s4 58
Zheng: 26.56
Zheng: 86
0.0
24
0.0
0.5
0.0
86
10
0.5
02
6.4
be 68
1.5
False
(a) DenseNet-100
(b) WRN-28
(c) TE--RR-28
Figure 9: Additional comparison of ROC curves for FGSM adversarial sample detection for CIFAR- 100 models
BIM 0.1
BIM 0.1
BIM 0.1
1.0
1.4
1.
GMM1000: 92.16+3.32
True Positive
0.6
True Pesitive
Pesitive
0.6
Logits: 38.03
Calibrated: 36.78
Logits: 63.93
Logits: 71.73 1.96
Mahalanobis:
Calibrated: 59.72 1
Calibreted: 69.02 1.63
Zheng: 73.13 = 6.42
0.2
Mahalancbis: 99.27
82
Mehalanobis: 85.30
0.2
Zhang: 98.35 0.46
Zhang: 34 1.25
Talse Positive Rate
False Positive Rate
False Pusitive Rate
DIM 0.2
BIM 0.2
BIM 0.2
10
GMM1000: 99. 0.73
True Posilvve
0.6
True Positive
True Pesitive
9.6
Logits: 23.29
GMM1000:
GMM100D: 83
Calibrated: 21.81 +-4.71
Logits: 65.06 29
Logils: 79.33 4.14
Mahalánobis: 98.68 1.18
Calibrated: 62.42 7.91
Calibrated: 74,80 a 4.28
Zheng: 88.42 9.08
0.
Mahalanobis: 99.97
22
Mahalanobis: 95.94
0.2
Zheng: 99 96 0.07
Zhang: 82 70 4.41
False Postivve Rate
False Positve Rate
Faise Positive Rate
(a) DenseNet-100
(b) WRN-28
(c) TE-WE WRN-28
Figure 10: Additional comparison of ROC curves for BIM adversarial sample detection for CIFAR- 100 models.
14

---

Under review as a conference paper at ICLR 2021
Table 2: OOD Detection results for CIFAR-10 model. Calibrated scores do not detect OOD samples well, especially gaussian noise samples. PixeICNN assigns higher likelihoods to OOD samples that diverge clearly from the original training set and fails to detect them.
(a) Out-distribution: SVHN
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-100M
99.35 0.21
96.
0.90
99.91 = 0.03
Mahalanobis
98.82 + 0.14
93.06
2.37
99.83 + 0.06
DenseNet-100
Calibrated Scores
92.00 + 3.02
82.48 4.90
98.29 + 0.70
Model-Agnostic
PixeICNN
19.73 1 0.72
7.29 + 0.08
76.06 + 0.21
(b) Out-distribution: Tiny ImageNet
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-1000
98.88 0.28
98.90 0.25
98.89 0.29
DenseNet-
Mahalanobis
98.03 0.39
98.07
0.37
98.03 0.42
Calibrated Scores
94.66 0.50
95.66
0.36
93.49 0.78
-odel-Agnostic
PixeICNN
85.01 0.16
80.11 0.25
86.22 0.19
(c) -ut-distribution: Gaussian Noise
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-1000
100
100
100
DenseNet-100
Mahalanobis
100
100
100
Calibrated Scores
77.85 18.34
86.57
11.61
68.28 + 16.81
Model-Agnostic
PixeICNN
100
100
100
(d) Out-distribution
Fashion-MNIST
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-100X
97.47 + 1.10
91.28 + 3.91
99.52 0.20
DenseNet-100 100
Mahalanobis
98.38 + 0.87
95.07
2.71
99.67 0.18
Calibrated Scores
93.71 + 1.83
85.72
3.97
98.53 + 0.48
Model-Agnostic
PixeICNN
0.61 1 0.16
7.51 1 0.00
67.68 1 0.03
Table 3: Out-distribution detection results for CIFAR-100 models Gaussian Noise. Calibrated
scores is the only method failing at detecting gaussian noise inputs.
MODEL
DETECTION METHOD
AU-ROC
AU-PR (in)
AU-PR (out)
GMM-1000
DenseNet- 100
Mahalanobis
100
100
100
Calibrated Scores
70.04 + 20.28
80.36 + 13.99
61.56 15.92
GMM-100 1000
99.98 99.96 + 0.03
99.98 09.96 = 0.03
99.94 0.05
WRN-28
Mahalanobis
99.99 + 0.01
100
99.98 + 0.02
Calibrated Scores
88.27 + 6.07
92.61 + 3.67
78.46 = 9.80
GMM-1000
100
100
100
TE-WRN-22
Mahalanobis
100
100
100
Calibrated Scores
38.28 + 16.31
59.02 + 11.95
41.07 = 5.55
Model-Agnostic
PixeICNN
100
100
100
15

---

Under review as a conference paper at ICLR 2021
B PURIFICATION
B.1 METHOD
The purification process aims at moving the feature F extracted by the classifier to low BPD region. This can be formulated as a joint optimization problem where we want to find features F with minimal BPD, while being close to the initial extracted features F Fef-
(8) v/ is a hyperparameter that defines how close the new feature should be to the initial one. As the objective is not convex and there is no close form solution for stationary points, gradient descent with regards to F to minimize the objective function.
F arg min (PD p|| - Fresll
F = <(VFDD((F) + -FFF)))e)))
B.2 PURIFICATION RESULTS
Purification features performed with 100 iterations gradient descent steps to optimize the ob- jective function. We test the performance of purification for both classification and semi-supervised classification tasks on CIFAR-100.
We report the accuracy validation and test set obtained after purification with different GMMs and for different values of learning rates and regularization strength v Table For classification, our networks are DenseNet (DN-100) and Wide ResNet (WRN-28). For semi-supervised classification, we apply temporal ensembling to wide ResNet (TE-WRN-22) Our results show that this purification procedure is able to correct classification mistakes on previously unseen samples and results in an accuracy gain for the model without the need to retrain. However the purification method also leads new classification mistakes, which means that the net improvement on the accuracy reaches 0.6% on the DenseNet model at most.
Table 4: Validation and test classification accuracy obtained after purification for DenseNet-100, Wide Resnet-28 and TE-Wide Resnet-28. Purification increases the test accuracy by up 0.6% on the DenseNet model.
GMM
V
DN-100
WRN-28
TE-WRN-28
Val
Test
Val
Test
Val
Test
Original
73.12
72.74
80.34
80.10
65.13
64.51
0
73.78
73.39
80.01
79.73
65.21
64.61
0.01
73.78
73.39
80.03
79.75
65.24
64.60
0.1
0.1
73.74
73.28
80.28
79.87
65.27
64.60
1000
1.0
73.24
72.89
80.40
80.11
65.16
64.51
73.33
72.95
80.40
80.10
65.18
64.5T
0.01
73.33
72.95
80.40
80.10
65.18
64.51
0.01
0.1
73.30
72.95
80.38
80.10
65.17
64.51
1.0
73.22
72.86
80.40
80.11
65.16
64.51
C EXPERIMENTAL SETUP
Dataset and preprocessing We trained CIFAR-10 and CIFAR-100 Krizhevsky 2009 with 5,000 images held-out validation images. Inputs were preprocessed with per-channel standardization before training.
DenseNet We use bottleneck layers and compression rate 0 = 0.5, growth rate 12 and depth L = 100. The model is trained with batch size 64 for 300 epochs with learning rate 0.1, dropout rate 0.2 and Ly regularization weight le -4 We use ReLU non-linearities except for the last layer where we use tanh non-lineari to ensure the extracted features are bounded. For optimization, we use Stochastic Gradient Descent with a Nestrov momentum 0.9. The learning rate is divided by 10 epoch 150 and 175.
16

---

Under review as a conference paper at ICLR 2021
Wide ResNet Wide ResNet Zagoruyko & Komodakis (016) is trained with growth rate A 10 and depth = 28 and batch size 100 for 200 epochs, with a learning rate 0.1, dropout rate 0.3 and L2 regularization weight 5e Data augmentation is applied during training with random translation by up to pixels and random horizontal flips.
Temporal Ensembling For the remi-supervised setting, we only keep 100 samples per label in the train set. We train a Wide ResNet using Temporal Ensembling with maximum weight decay of 100.
The PixeICNN model is trained with the PixeICNN+ ameliorations from (Salimans) et al. (2017) for our experiments. The model is trained for 5000 epochs with dropout rate 0.5 and learning rate 1e
PixeICNN
VAE The VAE is trainer for 1000 epochs with a learning rate of 0.001 and decay rate of 0.9995. The encoder and decoder architecture are fully connected layers with ReLU non-linearities, one hidden layer of size 512 and latent dimension of 128. The model was trained with Adam.
MAF The Masked Autoregressive Flow model is trained for 1000 epochs with a learning rate of 0.01 and batch size 32 using Adam Optimizer. We used 5-layer MADE model with hidden layer size 128.
Temperature Scaling The temperature for Temperature Scaling is optimized using the L-BFGS- optimization algorithm with a maximum of 100 iterations. We use ECE with 10 bins to evaluate the success of the calibration.
17


Reviews:
[{"review_id": "WXB4wds4BmE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main novel contribution is that the authors use generative models for several tasks but on the feature space and not the input pixel space. They show interesting behavior that mistakes tend to have low density in feature space, which isn't obvious as the NN that computes the features can map them wrongly to a high likelihood location.\nHowever, the authors claims of usefulness is not properly demonstrated in the experiences.\n\n- The authors claim several times that they perform \"rigorous hypothesis testing\" (including the abstract and conclusions so not a minor point in the paper) but it doesn't really mean that the mistakes can be identified as one can understand from the text. It just means that the distributions are distinct (but can have large overlap as can be seen in fig.2)\n- While GMM on feature space are better then other models on feature space and pixel space, it isn't convincing that the error detection can be of any use. \n- Adversarial detection was only tested on adversaries that do not try to fool the detector. As was shown in \"Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods\" by Carlini&Wagner this can lead to wrong detection claims.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "SYUZ5xzOp_L", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors provide a new method for detecting when deep networks are likely to fail and demonstrate through extensive experimentation its accuracy against generalization errors, out of distribution samples and adversarial attacks. The method builds on prior Mahalanobis metric of (Kimin Lee, et al., A unified framework for detecting out-of-distribution and adversarial samples. 2018) in two respects. First the authors use a single GMM fit to the model parameters that is class agnostic rather than a set of GMMs for each class, thus making it suitable for application in semi-supervised datasets. Second, the authors show ability to detect instances from a test set that are likely to cause a misclassification due to a failure to generalize. Surprisingly, the proposed approach performs better in most cases than the prior Mahalanobis approach even though it requires less information (no labels). \n\nPros:\n\n1. Well written and organized paper.\n2. More general and simpler approach than prior art\n3. Extensive empirical results that are competitive with or improved over prior art.\n\nCons:\n\n1. This paper is very similar to and provides only a relatively small incremental improvement over prior art (Lee, et al.)\n2. Like the Mahalanobis method, the proposed hypothesis testing method requires fitting a GMM to an \"incorrect\" distribution. Please make it clear what this distribution is for the experiments. It seems this would require knowledge of the type of attack, or the out-distribution making it non-blind and unrealistic for real-world out-of-distribution or adversarial attack scenarios.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AS8JeVZV5kD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after rebuttal:** The author rebuttal clarified some minor issues for me, but it did nothing to address my main concern, which is that very similar methods have been proposed before. I'm therefore keeping my score the same.  \n\n---------------------------------------------\nThis paper proposes a simple method for out-of-distribution detection. The basic idea is to fit a GMM to training examples in the feature space (as opposed to the pixel space). The experiments are generally rigorous and well executed, however my main problem with the paper is that it seems a bit too incremental to justify another paper. As the authors correctly point out very similar methods have already been proposed before (Zheng & Hong, 2018; Lee et al., 2018). \n\nOn page 2, the authors claim “Zheng & Hong (2018) and Lee et al. (2018) train a conditional generative model on the feature space learned by the classifier and derive a confidence score based on the Mahalanobis distance between a test sample and its predicted class representation”. However, this is not correct, Zheng and Hong (2018) don’t use the Mahalanobis metric. They use the exact same likelihood-based criterion in this paper, except they do this on a class-conditional basis as the authors correctly point out, but that seems like a small difference (as labels will obviously be available for the training data in a supervised setting, and the method can be easily adapted to the semi-supervised setting with some tweaks). I just don't see how this small difference could have a huge effect on performance.\n\nThe authors do have some comparisons with these earlier methods, but I’m wondering if these comparisons are done fairly: for example Zheng and Hong (2018) have a similar K parameter in their GMM model, what is the value used for that parameter in this paper? Is it the same as the one used in the class-agnostic GMMs? Have you tried tuning that parameter for these earlier models? How was the threshold parameter chosen? As far as I can see, these important experimental details are not discussed at all anywhere in the paper. Similarly, the semi-supervised setting in Figure 5c is not explained at all. How exactly does it work for the different models shown in that figure? Also, why does the Mahalanobis metric seem to work much better than the GMM in Figure 5b?\n\nAnother question I have is whether the authors have tried using features other than the final embedding layer features (and possibly a combination of features from multiple layers: something like this was done before in the deep k-nn paper by Papernot & McDaniel: https://arxiv.org/abs/1803.04765).\n\nMore minor comments:\n\nPage 6: please make sure you mention what BPD means (binned probability distribution), I don’t think this is as commonly known as CDF or PDF.\n\nTypos: “Aims at modeling confidence score that are” (p. 2), “task of detection out of distribution samples” (p. 7).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9aWUUo2nKJO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by the need to assess a classifier's confidence on its decision,\nthe paper proposes to use predicted likelihood of the learned features\nof an unknown sample, and shows that samples with low feature\nlikelihood tend to receive a wrong decision.\nFurthermore, the paper argues that using a GMM do model the feature\ndistributions is better than other methods like VAE or AR flow.\n\nThis study is interesting in the sence that it attempts to analyze the\ndistributions of the learned features, which is much needed for better\nunderstanding of what a deep neural net attempts to do.\nThe findings appear to make good sense as the network training process\nis designed to push samples of like-class towards tight clusters when\nmapped to the learned feature space.  Those lying in the outskirts signal\ndifficulties in this process,  therefore they are likely to get\nerroneous decisions.   The feature distributions are modeled\nclass-blind, so that the prediction can be applied to unseen cases\nwithout class labels.\n\nThe work can be improved by performing this analysis at different stages\nof a deep network,  as one expects that the levels closer to the\noutput layer demonstrate more of such clustering effect.  It\nwill be useful to confirm with this analysis.\n\nOther than showing numerical results, it will be more convincing if\nexample images are shown that are identified by this method to\nhave unreliable decisions by the classifier, and what error the classifier\nmakes on them.\n\nA more significant weakness is that the experiments are done only with\nan image classification problem, and a single dataset.  This makes the\nconclusion somewhat premature,  as images of physical objects tend to\nform good patterns.  Is the GMM estimation good just because the\ndata happen to be well clustered?  Will this conclusion be confirmed\nby classification tasks on other types of data, e.g. text?\nWhat if the class labels are scrambled?\nWhat makes some network settings better than others in learning\nsuch confidence-suggestive features?\n\nMisc.:\n\np.1, line 4 from bottom, \n\"... capture the distribution of the learned feature space\" ->\n\"capture the class-conditional distributions in the learned feature space\"\n\np.4, line 5,\n\"... state-of-the-art models assigning higher likelihoods to samples\n... \" ->\nWhat kind of models does this refer to?  What is the model supposed to\ndo?  Why do they assign higher likelihoods to out-of-distribution samples?\nThere is a lot to be filled in here; citing an external reference is\nnot enough.\n\np.6, line 7 from bottom,\n\"... assigned higher BPDs ...\", please expand the acronym BPD at its\nfirst mention.\n\np.8, conclusion,\n\"... verified that features extracted from inputs consistently lie\noutside of the training distribution and can be detected by their low\npredicted log-probability.\"   This sentence is garbled.  What have\nyou verified?  Do you mean to say this for a special type of input?\nWhat consistently lie outside of what?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As several reviewers pointed out, the contribution is  too incremental from previous work.
Decision: Reject