Defuse: Debugging Classifiers Through Distilling Unrestricted Adversarial Examples
Authors: Dylan Z Slack
Nathalie Rauschmayr
Krishnaram Kenthapadi
Venue:  (2021)
Keywords: debugging
interpretability
explainability

Abstract: With the greater proliferation of machine learning models, the imperative of diagnosing and correcting bugs in models has become increasingly clear. As a route to better discover and fix model bugs, we propose failure scenarios: regions on the data manifold that are incorrectly classified by a model. We propose an end-to-end debugging framework called Defuse to use these regions for fixing faulty classifier predictions. The Defuse framework works in three steps. First, Defuse identifies many unrestricted adversarial examples--naturally occurring instances that are misclassified--using a generative model. Next, the procedure distills the misclassified data using clustering into failure scenarios. Last, the method corrects model behavior on the distilled scenarios through an optimization based approach. We illustrate the utility of our framework on a variety of image data sets. We find that Defuse identifies and resolves concerning predictions while maintaining model generalization.

Under review as a conference paper at ICLR 2021
DEFUSE: DEBUGGING CLASSIFIERS THROUGH DIS- TILLING UNRESTRICTED ADVERSARIAL EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
With the greater proliferation of machine learning models, the imperative of diag- nosing and correcting bugs in models has become increasingly clear. As route better discover and fix model bugs, we propose failure scenarios: regions on the data manifold that are incorrectly classified by model. We propose an end-to-end debugging framework called Defuse to use these regions for fixing faulty classifier predictions. The Defuse framework works in three steps. First, Defuse identifies many unrestricted adversarial examples-naturalll misclassified-uuinn a generative model. Next, the procedure distills the misclas- sified data using clustering into failure scenarios. Last, the method corrects model behavior on the distilled scenarios through an optimization based approach. We illustrate the utility of our framework on variety of image data sets. We find that Defuse identifies and resolves concerning predictions while maintaining model generalization.
occurring instances that are
INTRODUCTION
Debugging machine learning (ML) models is a critical part of the ML development life cycle. Un- covering bugs helps ML developers make important decisions about both development and deploy- ment. In practice, much of debugging uses aggregate test statistics (like those in leader board style challenges [Rajpurkar et al. (2016)) and continuous evaluation and monitoring post deployment [Liberty et al. (2020). [imon 2019)). However, additional issues arise with over-reliance on test statistics. For instance, aggregate statistics like held out test accuracy are known to overestimate generalization performance Reccht et al. (2019)] Further, statisties offer little insight remedy for specific model failures [Ribeiro et al. (020); Wu et al. (201)). Last, reactive debugging of failures they occur in production does little mitigate harmful user experiences [La Fors et al. (2019) Several techniques exist for identifying undesirable behavior in machine learning models. These methods include explanations [Ribeiro et al. (2016); Sllck et al. (2020b); Lakkaraju et al. (019)) Lundbergg & Lee (2017b). fairness metrics [Fllmann et al. 2015), Slack et al. (2020a)], data set replication [Recht et al.) (019);; Engstrom al. (2020p. and behavioral testing tools [Ribeiro et al. 2020hh] However, these techniques do not provide methods to remedy model bugs or require a high level of human supervision. To enable model designers to discover and correct model bugs be- yond aggregate test statistics, we analyze unrestricted adversarial examplex: instances on the data manifold that are misclassified (Song et al. (2018)). We identify model bugs through diagnosing common patterns in unrestricted adversarial examples.
In this work, we propose Defuse: a technique for debugging classifiers through distilling unrestricted adversarial examples. Defuse works in three steps. First, Defuse identifies unrestricted adversarial examples by making small, semantically meaningful changes to input data using a vari- ational autoencoder (VAE). If the classifier prediction deviates from the ground truth label on the altered instance, it returns the data instance as a potential model failure. This method employs similar techniques from (Zhao et al. (2018). Namely, small perturbations in the latent space of generative models can produce images that are misclassified. Second, Defuse distills the changes through clustering on the unrestricted adversarial example's 's latent codes. In this way, Defuse diag- noses regions in the latent space that are problematic for the classifier. This method produces a set of
We mean distilling in the sense of "to extract the most important aspects of and do not intend to invoke the knowledge distillation literature (Hinton al. (2014).

---

Under review as a conference paper at ICLR 2021
6
88888
8
8
8
8
8
8 3
6
6 5
6 5
6
6
6
6
6
Identification
Distillation
Correction
Figure 1: Running Defuse MNIST classifier. The (handpicked) images examples from three failure scenarios identified from rumning Defuse. The red digit in the upper right hand corner of the image the classifier's prediction. Defuse initially identifies many model failures. Next, it aggregates these failures in the distillation step for annotator labeling. Last, Defuse tunes the classifier so that correctly classifies the images, with minimal change classifier performance. Defuse serves an end-to-end framework to diagnose and debug errors in classifiers.
clusters in the latent space where is likely to find misclassified data. We call these localities failure scenarios. An annotator reviews the failure scenarios and assigns the correct label one label per scenario. Third, Defuse corrects the model behavior on the discovered failure scenarios through optimization. Because we use a generative clustering model to describe the failure scenarios, we sample many unrestricted adversarial examples and finetune to fix the classifier. Critically, failure scenarios are highly useful for model debugging because they reveal high level patterns in the way the model fails. By understanding these consistent trends in model failures, model designers can more effectively understand problematic deployment scenarios for their models.
Defuse classifier trained MNIST and provide an overview in figure D In the identification step (first pane in figure D. Defuse generates unrestricted adversarial examples for the model. The red number in the upper right hand corner of the image is the classifier's 's prediction. Although the classifier achieves high test set performance, we find naturally occurring examples that are classified incorrectly. Next, the method performs the distillation step (second pane in figure D. The clustering model groups together similar failures for annotator labeling. We see that similar mistakes are grouped together. For instance, Defuse groups together similar style of incorrectly classified eights in the first row of the second pane in figure I Next, Defuse receives annotator labels for each of the clusters f Last, we run the correction step using both the annotator labeled data and the original training data. We see that the model correctly classifies the images (third pane in figure D Importantly, the model maintains its predictive performance, scoring 99.1% accuracy after tuning. We that Defuse enables model designers to both discover and correct naturally occurring model failures.
To illustrate the usefulness of failure scenarios,
We provide the necessary background in Defuse ( Next, detail the three steps in Defuse: identification, distillation, and correction (3) We then demonstrate the usefulness of Defuse on three image data sets: MNIST [LeCun et al. (010)) the German traffic signs data (Stallkamp) et al. (201 and the Street view house numbers data set [Netzer et al. (001]), and find that Defuse discovers and resolves critical bugs in high performance classifiers trained these datasets (74)
2 NOTATION AND BACKGROUND
In this section, we establish notation and background on unrestricted adversarial examples. Though unrestricted adversarial examples can be found in many domains, we focus Defuse applied to image classification.
We assign label to the first row in the second pane of figure label (I the second row, and label 6 to the third row.
2

---

Under review as a conference paper at ICLR 2021
2
8
B
Figure 2: Providing intuition for failure scenarios through t-SNE visualization of the latent space of MNIST. The black diamonds correspond to the latent codes of unrestricted adversarial examples. The blue circles are the latent codes of images from the training set. The images three decoded latent codes (the red dots), where the red number in the left hand corner the classifier label. We that there are regions with higher densities of adversarial examples
TR + [0, 1J denote a classifier that accepts data point x E X, where X is the set of legitimate images. The classifier f returns the probability that a belongs to class E {1, C}. Next, assume f is trained on data set D consisting of d tuples (ar, y) containing data point and ground truth label using loss function L. Finally, suppose there
Unrestricted adversarial examples Let :
exists an oracle o: I E X + {1, C} that outputs a label for æ. We define unrestricted adversarial examples as the AN {a X o(x) + f(x)} Song al. [018]]
Variational Autoencoders (VAEs) In order to discover unrestricted adversarial examples, it is necessary to model the set of legitimate images. We use a VAE to create such a model. A VAE composed of an encoder and decoder neural networks. These networks are used to model the relationship between data r and latent factors & E RK Where x is generated by some ground truth latent factors R.M wish train model such that the learned generative factors closely resemble the true factors: p(rlu) R p(x|z). In order to train such a model, we employ the 3-VAE Hiiginn al. [0017]] This technique produces encoder (2(xz) that maps from the data and latent codes and decoder pe (2) 22) that maps from codes to data.
METHODS
3.1 FAILURE SCENARIOS
We begin by formalizing our notion of failure scenarios. Let 2 E RKK be the latent codes corre- sponding to image æ E X and (.(-)) : x + be the encoder mapping the relationship between images and latent codes.
Definition 3.1. Failure scenario. Given constant > 0, vector norm || ||. and point z', failur scenario a set of images Ar {x X e (1))(x) 2'1l A oxx) + f(z)).
Previous works that investigate unrestricted adversarial examples look for specific instances where the oracle and the model disagree [Song et al. (2018): Zhao et al. (208)). We instead look for regions in the latent space where this is the case. Because the latent space of the VAE tends to take on Gaussian form due to the prior, we can use euclidean distance to define these regions. If we were define failure scenarios on the original data manifold, we may need much more complex distance function. Because it is likely too strict to assume the oracle and model disagree on every instance in such region, we also introduce a relaxation.
Definition 3.2. Relaxed failure scenario. Given constant O, vector norm || point z', and threshold relaxed failure scenario is a set of images Ay = {o E X e (14)(x) 2 such that {x A, o(z) + f())) |A, > p.
In this work, adopt the latter definition of failure scenarios. To concretize failure scenarios and provide evidence for their existence, we continue our MNIST example from figure !I We plot the t-SNE embeddings of the latent codes of 10000 images from the training set and 516 unrestricted
3

---

Under review as a conference paper at ICLR 2021
adversarial examples created during the identification step in figure (details of how generate unrestricted adversarial examples in (ectionf3.2.1. We see that the unrestricted adversarial examples from similar regions in the latent space.
3.2 DEFUSE
In this section, introduce Defuse: procedure for identifying and correcting classifier per- formance on failure scenarios. First, we explain how we identity unrestricted adversarial examples using VAEs. Next, describe clustering approach that distills these instances into failure scenarios. Last, we introduce our approach to correct classifier predictions on the failure scenarios.
3.2.1 IDENTIFYING UNRESTRICTED ADVERSARIAL EXAMPLES
This section describes the identification step in Defuse (first pane in figure D The aim of the iden- tification step is to generate many unrestricted adversarial examples. In essence, we encode all the images from the training data. We perturb the latent codes with a small amount of noise drawn from Beta distribution. We save instances that are classified differently from ground truth by when decoded. By perturbing the latent codes with small amount of noise, we expect the decoded instances to have small but semantically meaningful differences from the original instances. Thus, if the classifier prediction deviates on the perturbation the instance is likely misclassified. We de- note the set of unrestricted adversarial examples for a single instance vh We generate unrestricted adversarial examples over each instance taining the produced for each instance æ. Pseudo code of the algorithm for generating a single unrestricted adversarial example is given in algorithm I appendix
X producing set of unrestricted adversarial \/ con-
Our technique related to the method for generating natural adversarial examples from Zhao very similar but slightly different concept from unrestricted adversarial examples. The authors use a similar stochastic search method in the latent space of a GAN. They start with a small amount of noise and increase magnitude of the noise until they find unrestricted adversarial example. Thus, they save only the unrestricted adversarial examples which are minimally distant from a data point. They also save images that differ in prediction from the original decoded instance. Because we iterate over the entire data set, it is simpler to keep the level of noise fixed and sample a predetermined number of times. In addition, we save images that differ in ground truth label from the original decoded instance because we seek to debug a classifier. Meaning, if the original instance misclassified we wish save this instance as a model failure.
et al. (018)]
3.2.2 DISTILLING FAILURE SCENARIOS
This section describes the distillation step in defuse (second pane of figure D The goal of the distillation step is to cluster the latent codes of the set of unrestricted adversarial examples \/ in order to diagnose failure scenarios. We require our clustering method to (1) infer the correct number of clusters from the data and (2) be capable of generating instances of each cluster. We need to infer the number of clusters from the data because the number of failure scenarios are unknown ahead of must be capable of generating many instances from each cluster that have
time. Further, enough data to finetune on in order to correct the faulty model behavior. In addition, generating many failure instances enables model designers see numerous examples from the failure scenarios, which encourages understanding of the model failure modes. Though any such clustering method that fits this description could be used for distillation, we use Gaussian mixture model (GMM) with Dirichlet process prior. We use the Dirichlet process because it nicely describes the clustering problem where the number of mixtures is unknown before hand, fulfilling our first criteria (Sudderth [. Additionally, because the model is generative, we can sample new instances, which satisties our second criteria.
In pratice, we use the truncated stick breaking construction of the dirchlet process, where K is the upper bound of the number of mixtures. The truncated stick breaking construction simplifies inference making computation more efficient
( The method outputs set of clusters 8, = (Hy, ky) where j E {1,..., K). The parameters ju and o describe the mean and variance of multivariate normal distribution and и indicates the cluster weight. To perform inference the model, we employ expectation maximization (EM) described in [Bishop) (2006]] and use the implementation provided in (Pedregosa al. (20TT)). Once we run EM and determine the parameter
4

---

Under review as a conference paper at ICLR 2021
7
7
4
7
2
7
7
6
6
1
6
9
2
2
1
4
7
7
2
2
4
2
7
5
5
5
5
3
5
0
8
5
5
5
3
5
5
0
0
50
60
60
50
50
50
30
30
30
90
2
50
50
50
50
30
50
0
30
50
80
30
50
5
60
60
60
50
50
50
50
50
30
30
Figure 3: Samples from three failure scenarios from each dataset. First row: The MNIST failure scenarios. These scenarios were labeled 4, 2, 6 in order from left to right. Second row: The SVHN failure scenarios labeled 5, 8, and 5 from left to right. Third row: The German signs failure scenarios. The label 1 corresponds to 30km/h, 2 to 50km/h, and 5 to 80km/h. The first and second were labeled while the third was labeled 1. Defuse finds significant bugs in the classifiers.
values, we throw away cluster components that are not used by the model. We fix some small and define the set of failure scenarios A generated at the distillation step as: A == {(13, E.g, my)lm, > c).
3.2.3 CORRECTING FAILURE SCENARIOS
Labeling First, an annotator assigns the correct label to the failure scenarios. For each failure scenario identified in A. sample Q latent codes from ~ N(H,, oy). Here, T E is hyperparameter that controls the diversity of samples from the failure scenario. Because it could be possible for multiple ground truth classes be present in failure scenario, set this parameter tight enough such that the sampled instances are from the same class to make labeling easier. We reconstruct the latent codes using the decoder pa (arlz). Next, annotator reviews the reconstructed instances from the scenario and decides whether the scenario constitutes a model failure. If so, the annotator assigns the correct label all of the instances. The correct label constitutes a single label for all of the instances generated from the scenario. We repeat this process for each of the scenarios identified in A and produce a dataset of failure instances Dy. Pseudo code for the procedure given in algorithm 2 in appendix
Finetuning We finetune on the training data with an additional regularization term to fix the clas- sifier performance on the failure scenarios. The regularization term is the cross entropy loss between the identified failure scenarios and the annotator label. Where CE is the cross entropy loss applied to the failure instances Dj and À is the hyperparameter for the regularization term, we optimize the following objective using gradient descent: F(D, D2) = L(D) + X CE(D,). This objective encourages the model to maintain its predictive performance on the original training data while en-
5

---

Under review as a conference paper at ICLR 2021
couraging the model to predict the failure instances correctly. The regularization term À controls the pressure applied to the model to classify the failure instances correctly.
4 EXPERIMENTS
4.1 SETUP
Datasets We evaluate Defuse on three datasets: MNIST [LeCun et al. (2010)], I. the German Traffic Signs dataset (Stallkamp et al. (201T)), and the Street view house numbers dataset (SVHN) (Netzer er al. (20TT). MNIST consists of 60, 000 32X32 handwritten digits for training and 10, 000 digits for testing. The images are labeled corresponding to the digits 0- - 9. The German traffic signs data set includes 26, 640 training and 12, 630 testing images of size 128X128. We randomly split the testing data in half to produce a validation and testing set. The images are labeled from 43 different classes to indicate the type of traffic signs. The SVHN data set consists of 73, 257 training and 26, 032 testing images of size 32X32. The images include digits of house numbers from Google streetview with labels 0 9. We split the testing set in half produce validation and testing set.
Models On MNIST, we train a CNN scoring 98.3% test set accuracy following the architecture from 2019 On German traffic signs and SVHN, we finetune Resnet18 8 model pretrained on ImageNet He et al. 2016 The German signs and SVHM models score 98. 7% and 93.2% test accuracy respectively. We train a VAV on all available data from each data set to model the set of legitimate images in Defuse. We use an Amazon EC2 P3 instance with a single NVIDIA Tesla V100 GPU for training- We follow similar architectures to [Higgins et al. (2017). We set the size of the latent dimension to 10 for MNIST/SVHN and 15 for German signs. We provide our 3-VAE architectures in appendix
Paszke al.
Defuse In the identification step, we fix the parameters of the Beta distribution noise and b to a = 6 = 50.0 for MNIST and a = b = 75.0 for SVHN and German signs. We found these parameters were good choices because they produce very small amount of perturbation noise making the decoded instance only slightly different than the original instance. During distillation, set the upper bound the number of components K to 100. We generally found the actual number of clusters to be much lower than this level. Thus, this serves as an appropriate upper bound. We also fixed the weight threshold for clusters to 0.01 during distillation in order to remove clusters with very low weighting. We additionally randomly down sample the number of unrestricted adversarial examples 50, 000 to make inference of the GMM more efficient. For correction, we sample finetuning and testing sets consisting of 256 images each from every failure scenario. This number of samples captures the breadth of possible images in the scenario, so is appropriate for tuning and evaluation. We use the finetuning set as the set of failure instances Dj- We use the test set as held out data for evaluating classifier performance on the failure scenarios after correction. During sampling, we fix the sample diversity to 0.5 for MNIST and 0.01 for SVHN and German signs because the samples from each of the failure scenarios appear to be in the same class using these values. We finetune over a range of X's in order to find the best balance between training and failure scenario data. We use epochs for MNIST and 5 for both SVHN and German Signs because training converged within both these time frames. During finetuning, we select the model for each À according to the highest training set accuracy for MNIST or validation set accuracy for SVHM and German traffic signs at the end of each finetuning epoch. We select the best model overall as the highest training or validation performance over all X's.
Labeling Because Defuse requires human supervision, we use Amazon Sagemaker Ground Truth to both determine whether clusters generated in the distillation step are failure scenar- ios and to generate their correct label. In order determine whether clusters are failure scenarios, we sample 10 instances from each cluster in the distillation step. It is usually apparent the classifier disagrees with many of the ground truth labels within 10 instances, and thus it is appropriate to label the cluster as failure scenario. For example, in figure B it is generally clear the classifier incor- rectly predicts the data within only a few examples. As such, 10 instances is a reasonable choice. To reduce noise in the annotation process, we assign the same image to 5 different workers and take the majority annotated label as ground truth. The workers label the images using an interface that includes a single image and the possible labels for that task. We additionally instruct workers to select "None of the above" if the image does not belong to any class and discard these labels. For instance, the MNIST interface includes single image and buttons for the digits 0 9 along with
Annotator

---

Under review as a conference paper at ICLR 2021
Dataset
Scenarios
Validation
Test
Failure Scenario
MNIST
98.3
29.1
Before Finetuning
SVHN
93.6
93.2
31.2
German Signs
98.8
98.7
27.8
Unrestricted
MNIST
99.1
58.3
Adversarial Examples
SVHN
93.1
92.9
65.4
German Signs
MNIST
19
99.1
96.4
Defuse
SVHN
6
93.0
92.8
99.9
German Signs
8
98.1
97.7
85.6
Figure 4: Results from the best models before finetuning, finetuning only on the unrestricted adver- sarial examples, and finetuning using Defuse. The numbers presented are accuracy on the validation, test set, and failure scenario test set and the absolute number of failure scenarios generated using Defuse. We do not include finetuning on the unrestricted adversarial examples for German Signs because we, the authors, assigned failure scenarios for this data set and thus do not have ground truth labels for individual examples. Critically, the test accuracy on the failure scenarios is high for Defuse indicating that the method successfully corrects the faulty behavior.
"None of the above" button. We provide screen shot of this interface in figure 14 If more than half (i.e. setting p 0.5) of worker labeled instances disagree with the classifier predictions on the 10 instances, we call the cluster failure scenario. We chose = 0.5 because clusters are highly dense with incorrect predictions at this level, making them useful for both understanding model failures and worthwhile for correction. We take the majority prediction over each of the 10 ground truth labels as the label for the failure scenario. As an exception, annotating the German traffic signs data requires specific knowledge of traffic signs. The German traffic signs data ranges across 43 different types of traffic signs. It is not reasonable data and can label it accurately. For this data set, we, the authors, reviewed the distilled clusters and determined which clusters constituted failure scenarios. We labeled the cluster a failure scenario half the instances appeared to be misclassified.
assume annotators have enough familiarity with this
4.2 ILLUSTRATIVE FAILURE SCENARIO EXAMPLES
We demonstrate the potential of Defuse for identifying critical model bugs. We review failure sce- narios produced in the three datasets we consider. All together, Defuse produces 19 failure scenarios for MÑIST, 6 for SVHN, and 8 for German signs. For each dataset, we provide samples from three failure scenarios in figure S The failure scenarios include numerous mislabeled examples. Each failure scenario is composed of mislabeled examples of similar style. For example, in MNIST, the failure scenario in the upper left hand corner of figure includes a similar style of 4's that are generally predicted incorrectly. The same is true for the failure scenarios in the center and right column where certain style of 2's and 6's are mistaken. The failure scenarios generally include images which seem difficult to classify. For instance, the misclassified 6's are quite thin making them appear like 1 ''s in some cases. There similar trends in SVHN and German Signs. In SVHN, particular types of 5's and 8's are misclassified. The same is true in German signs where styles of 50km/h and 30km/h signs predicted incorrectly. Generally, these methods reveal important bugs in each of the classifiers. It is clear from the MNIST example for instance that very skinny 6's are challenging for the classifier to predict correctly. Further, the German signs classifier has difficult time with 50km/h signs and tends to frequently mistake them as 80km/h. We provide further sam- ples from other failure scenarios in appendix D These results clearly demonstrate Defuse reveals insightful model bugs which are useful for model designers to understand.
4.3 CORRECTING FAILURE SCENARIOS
We show that Defuse resolves the failure scenarios while maintaining model generalization the test set. To perform this analysis, we assess accuracy on both the failure scenario test data and test set after correction. It important for classifier accuracy improve on the failure scenario data in order

---

Under review as a conference paper at ICLR 2021
100
% Accurra
80
% Noccuraacy
80
80
% 60
Test Fallure scenario
Test set
Test set
60
Fallure
scanaro
40
Failure
scer ario
18+01
1e101
(a) MNIST
(b) SVHI
(c) German Signs
Figure 5: The tradeoff between test set and failure scenario accuracy running correction. We assess both test set accuracy and accuracy on the test failure scenario data finetuning over range of X's and plot the trade off. There is an optimal À for each classifier where test set and failure scenario accuracy are both high. This result confirms that the correction step in Defuse adequately balances both generalization and accuracy on the failure scenarios..
to correct the bugs discovered while rumning Defuse. At the same time, the classifier accuracy on the test set should stay similar level improve indicating that model generalization according to the test set is still strong. We compare Defuse against finetuning only on the unrestricted adversarial examples labeled by annotators. We expect this bascline to be reasonable because related works which focus on robustness to classic adversarial attacks demonstrate that tuning directly on the adversarial examples effective (Zhang et (201)). We finetune on the unrestricted adversarial examples sweeping over range of different X in the same way as Defuse described in section 4. We use this baseline for MNIST and SVHN and not German signs because we, the authors, assigned the failure scenarios for this data set. Thus, we do not have ground truth labels for unrestricted adversarial examples.
We provide overview of the models before finetuning, finetuning with the unrestricted adversar- ial examples, and using Defuse in figure g Defuse scores highly on the failure scenario data after correction compared to before finetuning. There is only marginal improvement finetuning on the un- restricted adversarial examples. These results indicate Defuse corrects the faulty model performance on the identified failure scenarios. Further, we see the clustering step in Defuse is critical to its suc- cess because of the technique's ' superior performance compared to finetuning on the unrestricted adversarial examples. In addition, there are minor effects on test set performance during finetuning. The test set accuracy increases slightly for MNIST and decreases marginally for SVHN and German Signs for both tuning on the unrestricted adversarial examples and using Defuse. Though the test set performance changes marginally, the increased performance on the failure scenarios demonstrates Defuse's 's capacity to correct important model errors. Further, we plot the relationship between test set accuracy and failure scenario test accuracy in figure 5 We generally see there is an appropriate À for each model where there is both high test set performance and accuracy on the failure scenarios. All in all, these results indicate Defuse serves as effective method for correcting specific cases of faulty classifier performance while maintaining model generalization.
4.4 ANNOTATOR AGREEMENT
Because we rely on annotators provide the ground truth labels for the unrestricted adversarial examples, we investigate the agreement between the annotators during labeling. It is important for the annotators to agree on the labels for the unrestricted adversarial examples so that we can have high confidence our evaluation is based on accurately labeled data. We evaluate the annotator agreement through assessing the percent of annotators that voted for the majority label prediction in an unrestricted adversarial example across all the annotated examples. This metric will be high when the annotators are in consensus and low when only a few annotators constitute the majority vote. We provide the annotator agreement on MNIST and SVHN in figure 5 broken down into failure scenario data, non-failure scenario data, and their combination. Interestingly. the failure scenario data has slightly lower annotator agreement indicating these tend to be more ambiguous examples. Further, there is lower agreement on SVHN than MNIST, likely because this data is more complex. All in all, there is generally high annotator agreement across all the data.
8

---

Under review as a conference paper at ICLR 2021
Dataset
Failure Scenario
Non-Failure Scenario
Combined
MNIST
78.9.3 + 5.4
87.2 3.2
85.2 + 0.1
SVHN
66.6 + 8.4
83.2 + 4.1
82.1 + 1.3
Figure 6: Annotator agreement on the unrestricted adversarial examples. We plot the mean and standard eτror of the percent annotators that voted for the majority label in an unresricted adver- sarial example across all the annotated examples. We break this down into the failure scenario and non-failure scenario unrestricted adversarial examples and the combination between the two. The annotators are generally in agreement though less so for the failure scenario data, indicating these tend to be more ambiguous examples.
RELATED WORK
A number of related approaches for improving classifier performance data created from gener-
ative models- - mostly generative adversarial networks (GANs) [Sandfort et al. (2019); Milz et al.)
(2018: Antoniou et al. (2017)) These methods use GANs generate instances from classes that
are underrepresented in the training data to improve generalization performance. Additional meth-
ods use generative models for semi-supervised learning [Kingma al. (2014): |aarma et al. (2016); Kumar et al. (2017); Dumoulin et al. [. Though these methods are similar in nature to the correction step of our work, a key difference Defuse focuses on summarizing and presenting high level model failures. Also, [Varma et al. (017)] provide a system to debug data generated from a GAN when the training set may be inaccurate. Though similar, we ultimately use a generative model to debug classifier and do not focus on the generative model itself. Last, similar to (Song) ct al. (2018), Zhao et al. (0188). Booth et al. (2020) provide a method to generate highly confident misclassified instances.
Related to debugging models, Kang el al. ] focus on model assertions that flag failures during production. Also, iZhang et al. (2018) investigate debugging the training set for incorrectly labeled instances. We focus on preemptively identifying model bugs and do not focus on incorrectly labeled test set instances. Additionally, (Ribeiro al. (2020) propose a of behavioral testing tools that help model designers find bugs in NLP models. This technique requires a high level of supervision and thus might not be appropraite in some settings. Last, (Odena al. 2019) provide technique to debug neural networks through perturbing data inputs with various types of noise. By leveraging unrestricted adversarial examples, we distill high level patterns in critical and naturally occurring model bugs. This technique requires minimal human supervision while presenting important types of model errors designers.
CONCLUSION
In this paper, we present Defuse: a method that generates and aggregates unrestricted adversarial examples to debug classifiers. Though unrestricted adversarial examples have been proposed in pre- vious works, we harness such examples for the purpose of debugging classifiers. We accomplish this task through identifying failure scenarios; regions in the latent space of a VAE with many unre- stricted adversarial examples. On variety of data sets, we find that samples from failure scenarios are useful in a number of ways. First, failure scenarios are informative for understanding the ways certain models fail. Second, the generative aspect of failure scenarios is very useful for correcting failure scenarios. In our experimental results, we show that these failure scenarios include critical model issues for classifiers with real world impacts i.ee traffic sign classification - and verify our results using ground truth annotator labels. We demonstrate that Defuse successfully resolves these issues. Although Defuse identifies important errors in classifiers, the technique requires a minimal level of human supervision. Namely, the failure scenarios must be reviewed before correction. In the future, will be crucial to investigate automatic ways of reviewing failure scenarios.
REFERENCES
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. International Conference Artificial Neural Networks and Machine Learning, 2017.

---

Under review as a conference paper at ICLR 2021
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statis- tics). Springer- Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738.
Serena Booth, Yilun Zhou, Ankit Shah, and Julie Shah. Bayes-trex: Model transparency by example. arXiv, 2020.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. ICLR, 2016.
L. Engstrom, Andrew Iyas, Shibani Santurkar, D. Tsipras, J. Steinhardt, and A. Madry, Identifying statistical bias in dataset replication. JCML, 2020.
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasub- ramanian. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, pp. 259-268, New York, NY, USA. 2015. Association for Computing Machinery. ISBN 978 450336642.. doi: 2783311
00.1145/7783258.2883311 /RRptst:: org.10..gg...o 1141/11411211 1Tt.///....///1122222121
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In ICLR, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NeurlPS Deep Learning and Representation Learning Workshop, 2014. URL http:://axxxi.
org/ hls//bb//aa 5533. 02531
Daniel Kang, D. Raghavan, Peter Bailis, and M. Zaharia. Model assertions for debugging machine learning. Debugging Machine Learning Models, 2018.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3581- 3589. Curran Associates, Inc., 2014.
Abhishek Kumar, Prasanna Sattigeri, and Tom Fletcher. Semi-supervised learning with gans: Man- ifold invariance with improved inference. In Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5534-5544, Curran Associates, Inc., 2017.
Karolina La Fors, Bart Custers, and Esther Keymolen. Reassessing values for emerging big data technologies: integrating design-basse based and application- based approaches. Ethics and Information doi: 10. 1000077100067---99900000---444 10t.:00000/111e 0tt.://///00.1111///000000-00000
Techhology, 21(3):209 226, Sep 2019. ISSN 1572-8439.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 131-138. ACM, 2019.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yamnllecu wwwpuncccom//xx//miisstccc 2, 2010.
Edo Liberty, Zohar Karnin, Bing Xiang, Laurence Rouesnel, Baris Coskun, Ramesh Nallapati, Julio Delgado, Amir Sadoughi, Yury Astashonok, Piali Das, Can Balioglu, Saswata Chakravarty, Madhav Jha, Philip Gautier, David Arpin, Tim Januschowski, Valentin Flunkert, Yuyang Wang, Jan Gasthaus, Lorenzo Stella, Syama Rangapuram, David Salinas, Sebastian Schelter, and Alex Smola. Elastic machine learning algorithms in amazon sagemaker. In Proceedings of the 2020 ACM SIGMOD International Conference Management of Data, SIGMOD 20, pp. 731-737, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978 9781450367356. doi: 10.1145/331 0113318444.3333 3386113812 22. URL htl.tttss: / www.//gg..00... 1145/31 8464 3386126
10

---

Under review as a conference paper at ICLR 2021
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems 30, pp. 4765 4765-4774. Curran Associates, Inc., 2017. URL http: pttt:::/ppapers.nnps..c.o ni
Stefan Milz, Tobias Rudiger, and Sebastian Suss. Aerial ganeration: Towards realistic data augmen- tation using conditional gans. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. TensorFuzz: Debugging neural networks with coverage-guile guided fuzzing. volume 97 of Proceedings of Machine Learning Research, pp. 4901-4911. Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http:
\/proceedings ml press 2 9a html
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Mnist example pytorch. 2019.
URL
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653//1/D1--1222 1264. URL httph: wwww aclweb.
brg/antho wawannhhooyyy// 12664
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? volume 97 of Proceedings of Machine Learning Research, pp. 5389- 5400, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http:: httt:://proceei.ngg/ mlr.
19w.hhtttta... html
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy:
Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 4902 4912, Online, July 2020. As- m42n.44.... URL https:
sociation for Computational Linguistics. doi: 10.
//www. ac www.wwww.......................w orgg ant hnt.ooooooooyo00000 acl ma 442
Veit Sandfort, Ke Yan, Perry Pickhardt, and Ronald M. Summers. Data augmentation using
generative adversarial networks (cyclegan) to improve generalizability in et segmentation tasks. Scientific Reports, 168444 Nov 2019. ISSN 2045-2322. doi: 10. 10010088//11598-1-55527333
URL (BB.a//pp...bbboo.ro//.....//11.
Julien Simon. Amazon sagemaker model monitor fully managed automatic monitoring for your machine learning models. AWS News Blog, dec 2019.
Dylan Slack, Sorelle A. Friedler, and Emile Givental. Fairness warnings and fair-maml: Learn- ing fairly with minimal data. Proceedings of the Conference on Faimess, Accountability, and Transparency (FAT*), 2020a.
Dylan Slack, Sophie Hilgard, Sameer Singh, and Himabindu Lakkaraju. How much should i trust you? modeling uncertainty of black box explanations. AIES, 2020b.
11

---

Under review as a conference paper at ICLR 2021
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 8312- 8323. Curran Associates, Inc., 2018.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. IEEE International Joint Conference on Neural Networks, pp. 1453-1460, 2011.
Erik B. Sudderth. Graphical models for visual object recognition and tracking. 2006.
Varma, Bryan He, Dan Iter, Peng Xu, R. Yu, C. D. Sa, and Christopher Ré. Socratic learning: Augmenting generative models to incorporate latent subsets training data. arXiv: Learning, 2016.
Paroma Varma, Dan Iter. Christopher De Sa, and Christopher Ré. Flipper: A systematic approach to debugging training sets. In Proceedings of the 2nd Workshop on Mouman-In-the-rie Loop Data An- alytics, 1ILDA'17, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450350297. doi: 10.114551007255.37776.. URL pettss://cc... 1r.//1..114/
3011257.3077263
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Errudite: Scalable, repro- ducible, and testable error analysis. In Proceedings of the 57th Anmual Meeting of the Asso- ciation for Computational Linguistics. pp. 747-763, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/11--100 1073. BtLhhttps:// www. www.wb.wwwb........
anthology/PI3: 1073
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. volume 97 of Proceedings of Machine Learning Research, pp. 7472-7482, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http::::ppoooeeeinngegeoooe
mwr. www..rppresses.. peress/vv7// pprpss/99//haagg11 9p.htm. htm
Xuezhou Zhang, Xiaojin Zhu, and Stephen Wright. Training set debugging using trusted items. In AAAI, 2018.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In Inter- national Conference on Learning Representations (ICLR), 2018.
12

---

Under review as a conference paper at ICLR 2021
A
DEFUSE PSUEDO CODE
In algorithm 2 Correct() and Label()) are the steps where the annotator decides if the scenario warrants correction and the annotator label for the failure scenario.
Algorithm Identification Step
Algorithm 2 Labeling Step
1: procedure IDENTIFY(f. v.
1:
procedure LABEL SCENARIOS(Q, 9.
2:
V : 0
2:
D O
3: a, (3)
3:
for (1, 0, πT) E A do
4:
for i E (1,... Q) do
4:
Xa== 0
5:
== [etaala,),,
5: for {1,.., Q} do
6:
..., ...,Beta(a,b b]
6:
Xa := Xx U qu (N(x.
7:
Wdecaded = po (
7:
end for
8:
y F [[deecoded then 10 decooded
8:
Correct((X) (Xa) then Dj : Dy U (Xd, Label
9:
9:
10:
end
10: end
end for
end for
12: Return ub
12: Return UD5
end procedure
end procedure
B TRAINING DETAILS
B.
GMM DETAILS
In all experiments, we use the implementation of Gaussian mixture model with dirichlet process prior from [Pedregosa et al. (201T). We run our experiments with the default parameters and full component covariance.
B.2 MNIST DETAILS
Model details We train CNN the MNIST data using the architecture in figure 7 We used the Adadelta optimizer with the learning rate set to 1. We trained for epochs with a batch size of 64.
Architecture
4x4 conv., ReLU stride
4x4 conv., 64 ReLU stride 2
4x4 conv., 64 ReLU stride 2 4x4 conv., 64 ReLU stride
Fully connected 256, ReLU Fully connected 256, ReLU
Fully connected 10
Figure 7: MNIST CNN Architecture
3-VAE training details We train a 3-VAE on MNIST using the architectures in figure 3 and We set 33 to 4. We trained for 800 epochs using the Adam optimizer with learning rate of 0.00T, a minibatch size of 2048, and 3 set to 0.4. We also applied linear annealing schedule on the KL-Divergence for 500 optimization steps. We z to have 10 dimensions.
Identification
We performed identification with Q set to 500. We set and both to 50. We ran identification over the entire training set. Last, we limited the max allowable size of d to 100.
Distillation
We ran the distillation step setting K, the upper bound the number of mixtures, to 100. We fixed e to 0.01 and discarded clusters with mixing proportions less than this value. This left 44 possible scenarios. We to 0.5 during review. We used Amazon Sagemaker Ground Truth
13

---

Under review as a conference paper at ICLR 2021
Architecture
4x4 conv., 32 ReLU stride 2 4x4 conv., 32 ReLU stride 2 4x4 conv., ReLU stride
Fully connected 256, ReLU Fully connected 256, ReLU
Fully connected 15
Figure 8: MNIST data encoder architecture.
Architecture
Fully connected 256, ReLU
Fully connected 256, ReLU
Fully connected 256, ReLU
4x4 transpose conv., ReLO stride 2
4x4 transpose conv., 32 ReLU stride 2
4x4 transpose conv., 32 ReLU stride 2
4x4 transpose conv., Sigmoid stride
Figure MNIST data decoder architecture.
to determine failure scenarios and labels. The labeling procedure is described in section produced 19 failure scenarios.
This
Correction We sampled 256 images from each of the failure scenarios for both finetuning and testing. We finetuned with minibatch size of 256, the Adam optimizer, and learning rate set to 0.001. We swept over range of correction regularization A's consisting of [le 10, le 9, 1e 8, le 7, le 6, le 5, le 1e 3, le 2, le 2, 10, 20, 100, 1000] and finetuned for epochs on each.
B.3 GERMAN SIGNS DATASET DETAILS
Dataset The data consists of 26640 training images and 12630 testing images consisting of different types of traffic signs. We randomly split the testing data in half to produce 6315 testing and validation images. Additionally, we resize the images 128x128 pixels.
Classifier f We fine-tuned the ResNet18 model for 20 epochs using Adam with the cross entropy loss. learning rate of 0.001, batch size of 256 on the training data set, and assessed the validation accuracy at the end of each epoch. We saved the model with the highest validation accuracy.
3-VAE training details We trained for 800 epochs using the Adam optimizer with learning rate of 0.001, a minibatch size of 2048, and 8 set to 4. We also applied linear annealing schedule on the KL-Divergence for 500 optimization steps. We
have 15 dimensions.
Architecture
4x4 conv., 64 ReLU stride
4x4 conv., 64 ReI.U stride 2
4x4 conv., 64 ReL.U stride
4x4 conv., 64 ReLU stride 2
Fully connected 256, ReLU
Fully connected 256, ReLU
Fully connected 15
Figure 10: German signs data set encoder architecture.
Identification
We performed identification with Q set 100. We set and both 75.
14

---

Under review as a conference paper at ICLR 2021
Architecture
Fully connected 256, ReLU
Fully connected 256, ReLU
Fully connected 256, ReL.U
4x4 transpose conv., 64 ReL.U stride 2
4x4 transpose conv., 64 ReLU stride 2
4x4 transpose conv., ReLU stride
4x4 transpose conv., 64 ReLU stride 2
4x4 transpose conv., 64 Sigmoid stride 2
Figure 11: German signs data set decoder architecture.
Distillation We ran the distillation step setting to 100. We fixed to 0.01 and discarded clusters with mixing proportions less than this value. This left 38 possible scenarios. We set to 0.01 during review. We determined of these scenarios were particularly concerning.
Correction We finetuned with minibatch size of 256, the Adam optimizer, and learning rate set to 0.001. We swept over range of correction regularization X's consisting of [le 10, le 9, le 8, le 7, le 6, le 5, le 4, le 3, le 2, le 1, 1, 2, 10, 20, 100, 1000] and finetuned for 5 epochs each.
B.4 SVHN DETAILS
Dataset The data set consists of 73257 training and 26032 testing images. We also randomly split the testing data to create a validation data set. Thus, the final validation and testing set correspond 13016 images each.
Classifier f We fine tuned for 10 epochs using the Adam optimizer, learning rate set to 0.001, and batch size of 2048. We chose the model which scored the best validation accuracy when measured the end of each epoch.
3- V- training details We trained the 3- VAE 400 epochs using the Adam optimizer, learning rate 0.001, and minibatch size of 2048. We set 3 to and applied linear annealing schedule on the KI- Di-iieergence for 5000 optimization steps. We set to have 10 dimensions.
Architecture
4x4 conv., 64 ReLO stride
4x4 conv., ReLU stride
4x4 conv., 64 ReLU stride 2 Fully connected 256, ReLU Fully connected 256, ReLU
Fully connected 10 2
Figure 12: SVHN data set encoder architecture.
Architecture
Fully connected 256, ReLU
Fully connected 256, ReL.U
Fully connected 256, ReLU
4x4 transpose conv., 64 ReLU stride 2
4x4 transpose conv., 64 ReLU stride 2
4x4 transpose conv., 64 ReL.U stride 2
4x4 transpose conv., 64 Sigmoid stride
Figure 13: SVHN data decoder architecture.
15

---

Under review as a conference paper at ICLR 2021
Poeaio Llact
Select an option
Tax
Four
S00AA
Figure 14: Annotation interface.
Identification
We set Q to 100. We also the maximum size of 1 10. We and 75.
Distillation We set K to 100. We fixed e to 0.01. The distillation step identified 32 plausible failure scenarios. The annotators deemed of these to be failure scenarios. We set 0.01 during review.
Correction We set the minibatch size of 2048, the Adam optimizer, and learning rate set to 0.001. We considered a range of X's: [le 10, 1e 9, 1e 8, le 7, le 6, le 5, le 4, le 3, le 2, 1e 1,2,5, 10, 20, 100, 1000]. We finetuned for epochs.
B.5 T-SNE EXAMPLE DETAILS
We run t-SNE on 10, 000 examples from the training data and 516 unrestricted adversarial examples setting perplexity 30. For the sake of clarity, we do not include outliers from the unrestricted adversarial examples. Namely, we only include unrestricted adversarial examples with > 1% prob- ability of being in of the MNIST failure scenario clusters.
C ANNOTATOR INTERFACE
We provide screenshot of the annotator interface in figure
D ADDITIONAL EXPERIMENTAL RESULTS
D. ADDITIONAL SAMPLES FROM MNIST FAILURE SCENARIOS
We provide additional examples from 10 randomly selected (no cherry picking) MNIST failure scenarios. We include the annotator consensus label for each failure scenario.
16

---

Under review as a conference paper at ICLR 2021
6
6
6
6
S
8
6
6
6
Figure 15: Annotator label 6.
2
J
8
1
3
3
3
3
)
1
3
J
J
3
1
3
1
1
5
1
1
3
}
3
3
3
1
1
J
1
1
]
3
1
1
)
1
1
1
]
1
3
}
7
]
3
]
3
3
}
Figure 16:
Annotator label 3.
17

---

Under review as a conference paper at ICLR 2021
Figure 17: Annotator label 4.
7
4
4
4
4
4
4
4
4
4
+
Figure 18: Annotator label 4.
18

---

Under review as a conference paper at ICLR 2021
8
8
8
8
5
5
5
2
6
6
1
Figure 19: Annotator label 6.
P
7
7
8
8
8
8
?
8
?
8
8
?
P
8
8
8
8
8
8
7
8
9
8
8
7
7
1
8
7
P
7
?
8
1
8
8
P
P
8
7
8
l
8
9
8
8
8
8
8
8
7
8
8
8
1
8
/
Figure 20: Annotator label 8.
19

---

Under review as a conference paper at ICLR 2021
8
8
Figure 21: Annotator label 6.
Figure 22: Annotator label 0.
20

---

Under review as a conference paper at ICLR 2021
6
6
6
5
5
6
Figure 23: Annotator label 6.
Figure 24: Annotator label 6.
21

---

Under review as a conference paper at ICLR 2021
D.2 ADDITIONAL SAMPLES FROM GERMAN SIGNS FAILURE SCENARIOS
We provide samples from all of the German signs failure scenarios. We provide the names of the class labels in figure 23 For each failure scenario, we indicate assigned class label in the caption and the classifier predictions in the upper right hand corner of the image.
Classid
SignName
Speed limit (20km/h)
Speed limit (30km/h)
Speed limit (50km/h)
Speed limit (60km/h)
Speed limit (70km/h)
5
Speed limit (80km/h)
End of speed limit (80km/h)
Speed limit (100km/h)
Speed limit (120km/h) No passing No passing for vehicles over 3.5 metric tons 10
11 Right-of-way the next intersection
12 Priority road
Yield 13
14 Stop
No vehicles
16
Vehicles over 3.5 metric tons prohibited
No entry
18
General caution
19
Dangerous curve to the left
20
Dangerous curve the right
21
Double curve
22
Bumpy road
23
Slippery road
24
Road narrows on the right
25
Road work
26
Traffic signals
27
Pedestrians
28
Children crossing
29
Bicycles crossing
30
Beware of ice/snow
31
Wild animals crossing
32
End of all speed and passing limits
33
Turn right ahead
34
Turn left ahead
35
Ahead only
36
Go straight or right
37
Go straight or left
38
Keep right
39
Keep left
40
Roundabout mandatory
41
End of no passing
42
End of passing by vehicles over 3.5 metric tons
Figure 25: German signs class labels.
22

---

Under review as a conference paper at ICLR 2021
Figure 26: Annotator label 7.
60
60
60
70
60
60
60
50
60
60
60
R0
610
5
50
50
50
60
60
50
50
50
50
50
Figure 27: Annotator label 2.
23

---

Under review as a conference paper at ICLR 2021
80
00
00
80
00
90
80
90
90
80
UC
80
90
80
80
00
00
90
00
80
00
M0
Figure 28: Annotator label 7.
Figure 29; Annotator label 41.
24

---

Under review as a conference paper at ICLR 2021
30
90
30
90
50
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
30
90
30
30
90
50
30
30
30
50
30
90
30
Figure 30: Annotator label 1.
60
60
60
60
80
60
60
60
60
60
60
60
60
60
60
60
60
80
60
60
60
60
60
60
60
60
60
80
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
80
60
60
60
60
60
60
60
60
60
60
60
60
60
80
60
60
60
60
60
Figure 31: Annotator label 2.
25

---

Under review as a conference paper at ICLR 2021
60
60
60
60
60
60
60
60
60
60
60
60
60
60
80
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
60
Figure 32: Annotator label 2.
30
30
30
50
50
50
30
50
5
5
30
50
30
30
50
30
50
30
30
50
50
30
30
50
30
30
5
5
50
30
30
30
30
30
50
30
30
30
30
30
30
30
30
30
30
30
30
30
50
30
30
30
30
30
30
30
30
30
30
30
5
30
50
30
30
30
30
30
30
Figure 33: Annotator label 1.
26

---

Under review as a conference paper at ICLR 2021
50
50
60
50
50
50
50
60
50
50
60
50
50
50
50
50
50
50
50
50
50
50
50
50
2
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
60
50
50
50
50
50
50
50
50
50
50
50
60
50
50
50
50
60
50
50
50
50
50
Figure 34: Annotator label 2.
27

---

Under review as a conference paper at ICLR 2021
D.3 ADDITIONAL SAMPLES FROM SVHN FAILURE SCENARIOS
We provide additional samples from each of the SVHN failure scenarios. The digit in the upper left hand corner the classifier predicted label. The caption includes the Ground Truth worker labels.
0
0
0
0
O
O
O
O
O
O
O
0
O
O
O
O
0
0
0
0
O
0
0
0
0
0
0
0
0
Figure 35: Annotator label 1.
55
5
5
5
5
6
6
5
5
56
15
5
1
5
5
9
5
5
5
6
5
5
5
5
5
5
9
5
5
1
5
5
5
5
5
5
5
6
5
5
5
5
8
8
5
5
1
5
0
15
5
10
5
5
5
5
0 5 15
5
9
5
Figure 36: Annotator label 5.
28

---

Under review as a conference paper at ICLR 2021
0
0
0
10
8
8
0
10
8
8
0
0
30
1
0
18
8
10
0
10
10
10
10
10
10
10
10
Figure 37: Annotator label 8.
Figure 38: Annotator label 0.
29

---

Under review as a conference paper at ICLR 2021
Figure 39: Annotator label 3.
5
5
5
1
6
5
5
5
6
5
5
3
3
0
6
3
5
6
6
1
3
3
2
3
2
6
6
6
5
6
6
5
5
2
3
0
6
5
2
6
6
2
5
2
2
Figure 40: Annotator label 5.
30


Reviews:
[{"review_id": "rZKvxsvSJdt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to identify and correct regions on the data manifold in which a trained classifier fails. The *identification* phase is based on clustering classification failure regions in a GAN latent space and the *correction* phase is based on fine-tuning the classifier with additional synthetic samples from the GAN.\n\nThe proposed method is strongly based on Zhao et al 2018 (Generating Natural Adversarial Examples), a method to generate on-manifold black-box adversarial examples using a GAN. The authors of the current paper describe some differences of their identification step from Zhao et al (end of section 3.2.1), but in my opinion they are minor.\n\nThe main contribution of the current paper over Zhao et al seems to be clustering the adversarial examples (using GMM) and using them to fine-tune the classifier. This, in my opinion, is potentially an interesting idea, however, the authors do not show sufficient evidence of its success. Specifically, the authors claim to \"achieve near perfect failure scenario accuracy with minimal change in test set accuracy\", but they do not provide any details (e.g. table of accuracy values on the train, test and adversarial sets before and after the fine-tuning). I would also expect to see an ablation study comparing the proposed method to simply including the adversarial examples found using Zhao et al (w/o GMM fitting and sampling) as additional training example - a standard adversarial defense approach (see e.g. [1]).\n\nPerhaps more importantly, the objective of the proposed method is not, in my opinion, clear. The title and abstract describe the goal as \"debugging\" a classifier and correcting fail regions, however the described method seems like a defense against on-manifold adversarial attack. If the method, as claimed, helps debugging and correcting the classifier, I would expect to see an improved accuracy on the (natural) unseen test set - not just on the synthetically generated adversarial examples.\n\nThe quality and clarity of the writing can be improved as well. A lot of space is allocated to describing well-known methods (e.g. VAE, GMM), however, critical information about the experimental results are missing. I'm also not sure all the formally defined algorithms and equations actually help in the understanding (e.g. algorithm 1, equation 2). Some of the mathematical notations are not standard.\n\nMinor comment: The norm in definition 3.1 is a regular vector norm (l2?) and not a matrix norm.\n\nTo summarize:\n\npros:\n- interesting idea (clustering on-manifold failures, labeling them and then using them to improve the classifier)\n\ncons:\n- contribution over Zhao et al not well established\n- insufficient and inaccurate experimental results\n- general quality of writing\n- not sure actual work and experiments match the stated objective\n- significance\n\n*Update:* Following the authors' response, I upgraded my rating, but I still think there are critical issues with the paper. The most problematic point, in my opinion, is the only-marginal improvement on the test data, indicating that the suggested training method only improves the specific \"failure scenarios\", making it is similar to adversarial training methods used to gain adversarial robustness. However, the abstract and introduction indicates that the paper helps in debugging in fixing failures in general, which, I think should have been evident in improved test accuracy.\n\n[1] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" ICML 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "EDYhz-YIQb-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a system DEFUSE which is geared towards identifying and correcting classifier performance when labels are assigned incorrectly. There are three phases that are used to design DEFUSE: (1) Identify unrestricted adversarial examples using Variational Auto Encoders (2) Use a clustering approach to distill the above examples into failure scenarios and (3) Correct the classifier predictions.\n\nOverall, the idea of using adversarial examples to correct incorrect classifications is very interesting. \n\nThe choice of certain algorithms and their parameters needs to be justified clearly. While it is understandable that a non-parametric model be used for the clustering step, it it not clear why a dirichlet process is the best fit. How does this choice compare with other clustering approaches? Do the results generalize?\n\nThe paper should be rewritten to have sufficient details of experiments in the text rather than delegating them to the Appendix A. \n\nThe motivation of why certain parameters are chosen for experiments should be discussed. For example, \"we sample 10 instances from each cluster in the distillation step. We ask 5 workers to label the instance\" -- Why are these choices appropriate? \nDescription of the annotation task ought to be more detailed -- \"labeling them ourselves\" -- Who constitutes \"ourselves\"? What is the agreement between the annotators?\n\nMinor comments:\n\n1. Section 3.2: how we identity -> how we identify \n2. Section 3.2.3: The paragraph ends with \"For instance.\" The sentence needs to be completed and an example provided.\n3. Section 4.1: 32x32 should be replaced with 32X32. Similarly 128x128 should be replaced with 128X128", "extra_scores": {}, "extra_text": {}}, {"review_id": "-oKqABtZVlB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The technique is described in sufficient detail and the paper is easy to read. Experimental results involving three datasets: MNIST, street view house numbers, and German traffic signs. The experimental results show that the proposed technique finds significant failures in all datasets, including critical failure scenarios. After correction, the performance of the method improves. \nAn interesting aspect of the method, which distinguishes it from similar techniques, is involvement of users/experts in the training process to indicate the classification errors in order to improve the performance of the method in the future. Engaging users in the training of classifiers has its advantages and disadvantages. For example, it can make easier to create “personalised” classification models that could be applied, e.g. in recommender system or information retrieval, where finding a perfect item depends on user’s subjective perception of certain qualities. At the same time, user involvement in the training process can be tricky if it requires expert judgment as they may not always be available (as the authors demonstrated in the case of their third dataset consisting of German traffic signs). Further, involving user generated assessments requires well defined procedures in terms of requirement of assessors, determining the appropriate number of assessors, resolving disagreements between assessors, to ensure robustness of the final classifier. In the examples provided in the paper, the authors state that they used 5 workers (annotators) and the majority vote was used to decide the final label. What was the inter-annotator agreement? Since using human labellers is a crucial part of the proposed method, I would like to see more discussion of this aspect.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript describes a method for identifying and correcting classifier performance when labels are assigned incorrectly. The identification is based on clustering classification failure regions in a VAE latent space and the correction phase is based on fine-tuning the classifier with additional synthetic samples from the VAE.

Reviewers agreed that the manuscript is not ready for publication. The main issue is that the suggested training method is similar to adversarial training methods used to gain adversarial robustness. The method does not help in debugging and fixing failures in general.

Decision: Reject