Understanding Classifiers with Generative Models

1 INTRODUCTION . Machine learning algorithms have shown remarkable success in challenging supervised learning tasks such as object classification ( He et al. , 2016 ) and speech recognition ( Graves et al. , 2013 ) . Deep neural networks in particular , have gained traction because of their ability to learn a hierarchical feature representation of their inputs . Neural networks , however , are also known to be brittle . As they require a large number of parameters compared to available data , deep neural networks have a tendency to latch onto spurious statistical dependencies to make their predictions . As a result , they are prone to overfitting and can be fooled by imperceptible adversarial perturbations of their inputs ( Szegedy et al. , 2013 ; Kurakin et al. , 2016 ; Madry et al. , 2017 ) . Additionally , modern neural networks are poorly calibrated and do not capture model uncertainty well ( Gal & Ghahramani , 2016 ; Kuleshov & Ermon , 2017 ; Guo et al. , 2017 ) . They produce confidence scores that do not represent true probabilities and consequently , often output predictions that are over-confident even when fed with out-of-distribution inputs ( Liang et al. , 2017 ) . These limitations of neural networks are problematic as they become ubiquitous in applications where safety and reliability is a priority ( Levinson et al. , 2011 ; Sun et al. , 2015 ) . Fully probabilistic , generative models could mitigate these issues by improving uncertainty quantification and incorporating prior knowledge ( e.g , physical properties ( Wu et al. , 2015 ) ) into the classification process . While great progress has been made towards designing generative models that can capture high-dimensional objects such as images ( Oord et al. , 2016a ; Salimans et al. , 2017 ) , accurate probabilistic modeling of complex , high-dimensional data remains challenging . Our work aims at providing an understanding of these failure modes under the lens of probabilistic modelling . Instead of directly modeling the inputs , we rely on the ability of neural networks to extract features from high-dimensional data and build a generative model of these low-dimensional features . Because deep neural networks are trained to extract features from which they output classification predictions , we make the assumption that it is possible to detect failure cases from the learned representations . Given a neural network trained for image classification , we capture the distribution of the learned feature space with a Gaussian Mixture Model ( GMM ) and use the predicted likelihoods to detect inputs on which the model can not produce reliable classification results . We show that we are able to not only detect adversarial and out-of-distribution samples , but surprisingly also identify inputs from the test set on which a model is likely to make a mistake . We experiment on state-of-the-art neural networks trained on CIFAR-10 and CIFAR-100 ( Krizhevsky , 2009 ) and show , through statistical hypothesis testing , that samples leading to classification failures tend to correspond to features that lie in a low probability region of the feature space . Contributions Our contributions are as follows : • We provide a probabilistic explanation to the brittleness of deep neural networks and show that classifiers tend to make mistakes on inputs with low-probability features . • We demonstrate that a simple modeling by a GMM of the feature space learned by a deep neural network is enough to model the probability space . Other state-of-the-art methods for probabilistic modelling such as VAEs ( Kingma & Welling , 2013 ) and auto-regressive flow models ( Papamakarios et al. , 2017 ) fail in that regard . • We show that generative models trained on the feature space can be used as a single tool to reliably detect different sources of classification failures : test set errors due to poor generalization , adversarial samples and out-of-distribution samples . 2 RELATED WORK . An extensive body of work has been focused on understanding the behaviours of deep neural networks when they are faced with inputs on which they fail . We provide a brief overview below : Uncertainty quantification Uncertainty quantification for neural networks is crucial in order to detect when a model ’ s prediction can not be trusted . Bayesian approaches ( MacKay , 1992 ; Neal , 2012 ; Blundell et al. , 2015 ) , for example , seek to capture the uncertainty of a network by considering a prior distribution over the model ’ s weights . Training these networks is challenging because the exact posterior is intractable and usually approximated using a variety of methods for posterior inference . Closely related , Deep Ensembles ( Lakshminarayanan et al. , 2017 ) and Monte-Carlo Dropout ( Gal & Ghahramani , 2016 ) consider the outputs of multiple models as an alternative way to approximate the distribution . Model calibration ( Platt , 1999 ; Guo et al. , 2017 ) aims at producing confidence score that are representative of the likelihood of correctness . Uncertainty quantification may also be obtained by training the network to provide uncertainty measures . Prior Networks ( Malinin & Gales , 2018 ) model the implicit posterior distribution in the Bayesian approach , DeVries & Taylor ( 2018 ) ; Lee et al . ( 2017 ) have the network output an additional confidence output . These methods require a proxy dataset representing the out-of-distribution samples to train their confidence scores . Our method differs from the above as it seeks to give an uncertainty estimation based on a model trained with the usual cross-entropy loss . It does not require additional modelling assumptions , nor modifications to the model ’ s architecture or training procedure . As such , it relates closely to threshold-based methods . For example , Hendrycks & Gimpel ( 2016 ) use the logits outputs as a measure of the network ’ s confidence and can be improved using Temperature Scaling ( Guo et al. , 2017 ; Liang et al. , 2017 ) , a post-processing method that calibrates the model . Our work derives a confidence score by learning the probability distribution of the feature space and generalizes to adversarial samples ( Szegedy et al. , 2013 ) , another source of neural networks ’ brittleness . Adversarial samples Methods to defend against adversarial examples include explicitly training networks to be more robust to adversarial attacks ( Tramèr et al. , 2017 ; Madry et al. , 2017 ; Papernot et al. , 2015 ) . Another line of defense comes from the ability to detect adversarial samples at test time . Song et al . ( 2017 ) for example , use a generative model trained on the input images to detect and purify adversarial examples at test time using the observation that adversarial samples have lower predicted likelihood under the trained model . Closer to our work , Zheng & Hong ( 2018 ) and Lee et al . ( 2018 ) train a conditional generative model on the feature space learned by the classifier and derive a confidence score based on the Mahalanobis distance between a test sample and its predicted class representation . Our method makes the GMM class-agnostic , making it applicable to settings where labels are not available at inference time . We further show that the unsupervised GMM improves on the Mahalanobis score on the OOD detection task . 3 DETECTING MISTAKES . Detecting samples on which a trained classifier is likely to make a mistake is crucial when considering the range of applications in which these models are deployed . However , predicting in advance whether a sample will fail seems challenging , especially when the sample is drawn from the same distribution as the train set . To illustrate this , we show in Fig . 1 , samples from the CIFAR-100 training dataset and compare them to test samples and adversarial examples that our DenseNet model fails to classify properly . In both cases , it is not obvious to the human eye what fundamentally differs between correct and incorrect samples . Our main intuition is that a generative model trained on the feature space could capture these subtle differences . 3.1 BACKGROUND . We consider the problem of classification where we have access to a ( possibly partially ) labeled dataset D = { ( Xi , yi ) } Ni=1 where ( Xi , yi ) ∈ X × Y . Samples are assumed to be independently sampled from a distribution pdata ( X , y ) and we denote the marginal over X as pdata ( X ) . We will denote fθ : X −→ F = RD the feature extractor part of our neural network , where θ represents the parameters of the network and F is the feature space of dimension D. Given an input X , the predictions probabilities on the label space Y are then typically obtained using multivariate logistic regression on the extracted features . p ( y|X , θ , W , b ) = softmax ( Wfθ ( X ) + b ) ( 1 ) where ( W , b ) represent the weights and bias of the last fully-connected layer of the neural network . The model prediction is the class with the highest predicted probability : ŷ ( X ) = argmaxy∈Y p ( y|X , θ , W , b ) . The parameters ( θ , W , b ) are trained to minimize a crossentropy loss on the training set and performance is evaluated on the test set . Learning the data structure with Generative Models Understanding the data structure can greatly improve the ability of neural models to generalize . Recently , great progress has been made in designing powerful generative models that can capture high-dimensional complex data such as images . PixelCNN ( Salimans et al. , 2017 ; Oord et al. , 2016b ; a ) in particular , is a state-of-the-art deep generative model with tractable likelihood that represents the probability density of an image as a fully factorized product of conditionals over individual pixels of an image . pCNN ( X ) = n∏ i=1 pφ ( Xi|X1 : i−1 ) ( 2 ) Flow models such as the Masked Autogressive Flow ( MAF ) ( Papamakarios et al. , 2017 ) model provide similar tractability by parameterizing distributions with reversible functions which make that likelihood easily tractable through the change of variable formula . Another widely used class of generative models assumes the existence of unobserved latent variables . Gaussian Mixture Models , for example , assume discrete latents ( corresponding to the mixture component ) . Variational autoencoders ( Kingma & Welling , 2013 ) use continuous latent variables and parameterize the ( conditional ) distributions using neural networks . 3.2 MODELING THE FEATURE SPACE . We identify two main reasons why characterizing examples over which a classifier is likely to make a mistake is difficult . First , modeling the input data distribution pdata ( X ) , as done in Song et al . ( 2017 ) to detect adversarial examples , is challenging because of the high-dimensional , complex nature of the image space X . This approach also fails at detecting out-of-distribution samples , with state-of-the art models assigning higher likelihoods to samples that completely differ from their train set ( Nalisnick et al. , 2018 ) . Second , a model of pdata ( X ) doesn ’ t capture any information about the classifier itself . To overcome these difficulties , we propose to model the underlying distribution of the learned features F = fθ ( X ) , where X ∼ pdata ( X ) . Extracted features have lower dimension which makes them easier to model and they give access to some information on the classifier . Specifically , we are interested in comparing features Fc of samples that are correctly classified with features Fw of samples that are incorrectly classified by a trained neural network . Fc and Fw can be described as elements of the following sets : Fc ∈ C = { fθ ( X ) |ŷ ( X ) = y , ( X , y ) ∈ X × Y } ( 3 ) Fw ∈ W = { fθ ( X ) |ŷ ( X ) 6= y , ( X , y ) ∈ X × Y } ( 4 ) The distribution of the extracted features is modeled by : p ( F ) = K∑ k=1 πkN ( F ; µk , Σk ) ( 5 ) where K is the number of Gaussians in the mixture , πk , µk , Σk are the model parameters . We choose Σk to be diagonal in all our experiments . After training a neural network to convergence , we learn the parameters of the GMM using the EM algorithm . Our training set is built from the features extracted from the training image set by the trained classifier .

Summary: The authors provide a new method for detecting when deep networks are likely to fail and demonstrate through extensive experimentation its accuracy against generalization errors, out of distribution samples and adversarial attacks. The method builds on prior Mahalanobis metric of (Kimin Lee, et al., A unified framework for detecting out-of-distribution and adversarial samples. 2018) in two respects. First the authors use a single GMM fit to the model parameters that is class agnostic rather than a set of GMMs for each class, thus making it suitable for application in semi-supervised datasets. Second, the authors show ability to detect instances from a test set that are likely to cause a misclassification due to a failure to generalize. Surprisingly, the proposed approach performs better in most cases than the prior Mahalanobis approach even though it requires less information (no labels).