Defuse: Debugging Classifiers Through Distilling Unrestricted Adversarial Examples

1 INTRODUCTION . Debugging machine learning ( ML ) models is a critical part of the ML development life cycle . Uncovering bugs helps ML developers make important decisions about both development and deployment . In practice , much of debugging uses aggregate test statistics ( like those in leader board style challenges [ Rajpurkar et al . ( 2016 ) ] ) and continuous evaluation and monitoring post deployment [ Liberty et al . ( 2020 ) , Simon ( 2019 ) ] . However , additional issues arise with over-reliance on test statistics . For instance , aggregate statistics like held out test accuracy are known to overestimate generalization performance [ Recht et al . ( 2019 ) ] . Further , statistics offer little insight nor remedy for specific model failures [ Ribeiro et al . ( 2020 ) ; Wu et al . ( 2019 ) ] . Last , reactive debugging of failures as they occur in production does little to mitigate harmful user experiences [ La Fors et al . ( 2019 ) ] . Several techniques exist for identifying undesirable behavior in machine learning models . These methods include explanations [ Ribeiro et al . ( 2016 ) ; Slack et al . ( 2020b ) ; Lakkaraju et al . ( 2019 ) ; Lundberg & Lee ( 2017 ) ] , fairness metrics [ Feldman et al . ( 2015 ) , Slack et al . ( 2020a ) ] , data set replication [ Recht et al . ( 2019 ) ; Engstrom et al . ( 2020 ) ] , and behavioral testing tools [ Ribeiro et al . ( 2020 ) ] . However , these techniques do not provide methods to remedy model bugs or require a high level of human supervision . To enable model designers to discover and correct model bugs beyond aggregate test statistics , we analyze unrestricted adversarial examples : instances on the data manifold that are misclassified [ Song et al . ( 2018 ) ] . We identify model bugs through diagnosing common patterns in unrestricted adversarial examples . In this work , we propose Defuse : a technique for debugging classifiers through distilling1 unrestricted adversarial examples . Defuse works in three steps . First , Defuse identifies unrestricted adversarial examples by making small , semantically meaningful changes to input data using a variational autoencoder ( VAE ) . If the classifier prediction deviates from the ground truth label on the altered instance , it returns the data instance as a potential model failure . This method employs similar techniques from [ Zhao et al . ( 2018 ) ] . Namely , small perturbations in the latent space of generative models can produce images that are misclassified . Second , Defuse distills the changes through clustering on the unrestricted adversarial example ’ s latent codes . In this way , Defuse diagnoses regions in the latent space that are problematic for the classifier . This method produces a set of 1We mean distilling in the sense of “ to extract the most important aspects of ” and do not intend to invoke the knowledge distillation literature [ Hinton et al . ( 2014 ) ] . clusters in the latent space where it is likely to find misclassified data . We call these localities failure scenarios . An annotator reviews the failure scenarios and assigns the correct label— one label per scenario . Third , Defuse corrects the model behavior on the discovered failure scenarios through optimization . Because we use a generative clustering model to describe the failure scenarios , we sample many unrestricted adversarial examples and finetune to fix the classifier . Critically , failure scenarios are highly useful for model debugging because they reveal high level patterns in the way the model fails . By understanding these consistent trends in model failures , model designers can more effectively understand problematic deployment scenarios for their models . To illustrate the usefulness of failure scenarios , we run Defuse on a classifier trained on MNIST and provide an overview in figure 1 . In the identification step ( first pane in figure 1 ) , Defuse generates unrestricted adversarial examples for the model . The red number in the upper right hand corner of the image is the classifier ’ s prediction . Although the classifier achieves high test set performance , we find naturally occurring examples that are classified incorrectly . Next , the method performs the distillation step ( second pane in figure 1 ) . The clustering model groups together similar failures for annotator labeling . We see that similar mistakes are grouped together . For instance , Defuse groups together a similar style of incorrectly classified eights in the first row of the second pane in figure 1 . Next , Defuse receives annotator labels for each of the clusters.2 Last , we run the correction step using both the annotator labeled data and the original training data . We see that the model correctly classifies the images ( third pane in figure 1 ) . Importantly , the model maintains its predictive performance , scoring 99.1 % accuracy after tuning . We see that Defuse enables model designers to both discover and correct naturally occurring model failures . We provide the necessary background in Defuse ( §2 ) . Next , we detail the three steps in Defuse : identification , distillation , and correction ( §3 ) . We then demonstrate the usefulness of Defuse on three image data sets : MNIST [ LeCun et al . ( 2010 ) ] , the German traffic signs data set [ Stallkamp et al . ( 2011 ) ] , and the Street view house numbers data set [ Netzer et al . ( 2011 ) ] , and find that Defuse discovers and resolves critical bugs in high performance classifiers trained on these datasets ( §4 ) . 2 NOTATION AND BACKGROUND . In this section , we establish notation and background on unrestricted adversarial examples . Though unrestricted adversarial examples can be found in many domains , we focus on Defuse applied to image classification . 2We assign label 8 to the first row in the second pane of figure 1 , label 0 to the second row , and label 6 to the third row . Unrestricted adversarial examples Let f : RN ! [ 0 , 1 ] C denote a classifier that accepts a data point x 2 X , where X is the set of legitimate images . The classifier f returns the probability that x belongs to class c 2 { 1 , ... , C } . Next , assume f is trained on a data set D consisting of d tuples ( x , y ) containing data point x and ground truth label y using loss function L. Finally , suppose there exists an oracle o : x 2 X ! { 1 , ... , C } that outputs a label for x . We define unrestricted adversarial examples as the set AN : = { x 2 X | o ( x ) 6= f ( x ) } [ Song et al . ( 2018 ) ] . Variational Autoencoders ( VAEs ) In order to discover unrestricted adversarial examples , it is necessary to model the set of legitimate images . We use a VAE to create such a model . A VAE is composed of an encoder and a decoder neural networks . These networks are used to model the relationship between data x and latent factors z 2 RK . Where x is generated by some ground truth latent factors v 2 RM , we wish to train a model such that the learned generative factors closely resemble the true factors : p ( x|v ) ⇡ p ( x|z ) . In order to train such a model , we employ the -VAE [ Higgins et al . ( 2017 ) ] . This technique produces encoder q ( z|x ) that maps from the data and latent codes and decoder p✓ ( x|z ) that maps from codes to data . 3 METHODS . 3.1 FAILURE SCENARIOS . We begin by formalizing our notion of failure scenarios . Let z 2 RK be the latent codes corresponding to image x 2 X and q ( · ) : x ! z be the encoder mapping the relationship between images and latent codes . Definition 3.1 . Failure scenario . Given a constant ✏ > 0 , vector norm || · || , and point z0 , a failure scenario is a set of images AR = { x 2 X | ✏ > ||q ( x ) z0|| ^ o ( x ) 6= f ( x ) } . Previous works that investigate unrestricted adversarial examples look for specific instances where the oracle and the model disagree [ Song et al . ( 2018 ) ; Zhao et al . ( 2018 ) ] . We instead look for regions in the latent space where this is the case . Because the latent space of the VAE tends to take on Gaussian form due to the prior , we can use euclidean distance to define these regions . If we were to define failure scenarios on the original data manifold , we may need a much more complex distance function . Because it is likely too strict to assume the oracle and model disagree on every instance in such a region , we also introduce a relaxation . Definition 3.2 . Relaxed failure scenario . Given a constant ✏ > 0 , vector norm || · || , point z0 , and threshold ⇢ , a relaxed failure scenario is a set of images Af = { x 2 X | ✏ > ||q ( x ) z0|| } such that | { x 2 Af | o ( x ) 6= f ( x ) } | / |Af | > ⇢ . In this work , we adopt the latter definition of failure scenarios . To concretize failure scenarios and provide evidence for their existence , we continue our MNIST example from figure 1 . We plot the t-SNE embeddings of the latent codes of 10000 images from the training set and 516 unrestricted adversarial examples created during the identification step in figure 2 ( details of how we generate unrestricted adversarial examples in section 3.2.1 ) . We see that the unrestricted adversarial examples are from similar regions in the latent space . 3.2 DEFUSE . In this section , we introduce Defuse : our procedure for identifying and correcting classifier performance on failure scenarios . First , we explain how we identity unrestricted adversarial examples using VAEs . Next , we describe our clustering approach that distills these instances into failure scenarios . Last , we introduce our approach to correct classifier predictions on the failure scenarios . 3.2.1 IDENTIFYING UNRESTRICTED ADVERSARIAL EXAMPLES . This section describes the identification step in Defuse ( first pane in figure 1 ) . The aim of the identification step is to generate many unrestricted adversarial examples . In essence , we encode all the images from the training data . We perturb the latent codes with a small amount of noise drawn from a Beta distribution . We save instances that are classified differently from ground truth by f when decoded . By perturbing the latent codes with a small amount of noise , we expect the decoded instances to have small but semantically meaningful differences from the original instances . Thus , if the classifier prediction deviates on the perturbation the instance is likely misclassified . We denote the set of unrestricted adversarial examples for a single instance . We generate unrestricted adversarial examples over each instance x 2 X producing a set of unrestricted adversarial containing the produced for each instance x. Pseudo code of the algorithm for generating a single unrestricted adversarial example is given in algorithm 1 in appendix A . Our technique is related to the method for generating natural adversarial examples from [ Zhao et al . ( 2018 ) ] — a very similar but slightly different concept from unrestricted adversarial examples . The authors use a similar stochastic search method in the latent space of a GAN . They start with a small amount of noise and increase magnitude of the noise until they find a unrestricted adversarial example . Thus , they save only the unrestricted adversarial examples which are minimally distant from a data point . They also save images that differ in prediction from the original decoded instance . Because we iterate over the entire data set , it is simpler to keep the level of noise fixed and sample a predetermined number of times . In addition , we save images that differ in ground truth label from the original decoded instance because we seek to debug a classifier . Meaning , if the original instance is misclassified we wish to save this instance as a model failure .

Summary: The technique is described in sufficient detail and the paper is easy to read. Experimental results involving three datasets: MNIST, street view house numbers, and German traffic signs. The experimental results show that the proposed technique finds significant failures in all datasets, including critical failure scenarios. After correction, the performance of the method improves.