Meaningfully Explaining Model Mistakes Using Conceptual Counterfactuals
Authors: Abubakar Abid
Mert Yuksekgonul
James Zou
Venue: ICLR 2022 Submitted (2022)
Keywords: interpretability
concept-based explanations
counterfactual explanations

Abstract: Understanding and explaining the mistakes made by trained models is critical to many machine learning objectives, such as improving robustness, addressing concept drift, and mitigating biases. However, this is often an ad hoc process that involves manually looking at the model's mistakes on many test samples and guessing at the underlying reasons for those incorrect predictions. In this paper, we propose a systematic approach, \textit{conceptual counterfactual explanations} (CCE), that explains why a classifier makes a mistake on a particular test sample(s) in terms of human-understandable concepts (e.g. this zebra is misclassified as a dog because of faint \emph{stripes}). We base CCE on two prior ideas: counterfactual explanations and concept activation vectors, and validate our approach on well-known pretrained models, showing that it explains the models' mistakes meaningfully. In addition, for new models trained on data with spurious correlations, CCE accurately identifies the spurious correlation as the cause of model mistakes from a single misclassified test sample. On two challenging medical applications, CCE  generated useful insights, confirmed by clinicians, into biases and mistakes the model makes in real-world settings. The code for CCE is publicly available and can easily be applied to explain mistakes in new models. 

Under review as a conference paper at ICLR 2022
MEANINGFULLY DEBUGGING MODEL MISTAKES US- ING CONCEPTUAL COUNTERFACTUAL EXPLAAAIION
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding and explaining the mistakes made by trained models is critical to many machine learning objectives, such as improving robustness, addressing concept drift, and mitigating biases. However, this is often an ad hoc process that involves manually looking at the model's mistakes on many test samples and guessing at the underlying reasons for those incorrect predictions. In this paper, we propose systematic approach, conceptual counterfactual explanations (CCE), that explains why classifier makes mistake on a particular test sample(s) in terms of human-unde understaddde concepts (e.g. this zebra is misclassified dog because of faint stripes). We base CCE on two prior ideas: counterfactual explanations and concept activation vectors, and validate our approach well-known pretrained models, showing that it explains the models' mistakes meaningfully. In addition, for new models trained on data with spurious correlations, CCE accurately identifies the spurious correlation as the cause of model mistakes from a single misclassified test sample. On two challenging medical applications, CCE generated useful insights, confirmed by clinicians, into biases and mistakes the model makes in real-world settings. The code for CCE is publicly available and can easily be applied to explain mistakes in new models.
INTRODUCTION
People who use machine learning (ML) models often need understand why a trained model is making a particular mistake. For example, upon seeing a model misclassify an image, an ML practitioner may ask questions such as: Was this kind of image underrepresented in my training distribution? Am preprocessing the image correctly? Has my model learned a spurious correlation or bias that is hindering generalization? Answering this question correctly affects the usability of a model and can help make the model more robust. Consider a motivating example:
Example 1: Usability of a Pretrained Model. A pathologist downloads a pretrained model to classify histopathology images. Despite high reported accuracy, he finds the model performing poorly on his images. He investigates why that is the case, finding that the hues in his images are different than in the original training data. Realizing the issue, he is able to transform his own images with some preprocessing. matching the training distribution and improving the model's performance.
In the above example, we have domain shift occurring between training and test time, which degrades the model's performance (Subbaswamy et al. 2019 Koh et al. (020). By explaining the cause of the domain shift, we are able to easily fix the model's predictions. On the other hand, if the domain shift is due to more complex spurious correlations the model has learned, it might need to be completely retrained before it can be used. During development, identifying and explaining a
model's 's failure points can also make it more robust (Abid et al. 2019 Kiela et al. (021). as in the following example:
Example 2: Discovering Biases During Development. A dermatologist trains machine learning classifier to classify skin diseases from skin images collected from patients at her hospital. She shares her trained model with colleague at different hospital, who reports that the model makes many mistakes with his own patients. She investigates why the model is making mistakes, realizing that patients at her colleague's 's hospital have different skin colors and ages. Answering this question allows her to not only know her own model's 's biases but also guides her to expand her training set with the right kind of data to build more robust model.

---

Under review as a conference paper at ICLR 2022
Explaining model's mistakes is also useful in other settings where data distributions may change. such as concept drift for deployed machine learning models (Lu et al. 2018). Despite its usefulness, explaining model's performance drop is often ad hoc process that involves manually looking the model's mistakes on many test samples and guessing at the underlying reasons for those incorrect predictions. We present counterfactual conceptual explanations (CCE), systematic method for explaining model mistakes in terms of meaningful human- uamaan--udersanddabee concepts, such as those in the examples above (e.g. hues). In addition, CCE was developed with the following criteria:
No training data or retraining: CCE does not require access the training data model retraining.
Only needs the model: CCE only needs white-box access to the model. The user can use any dataset learn desired concepts.
High-level explanations: CCE provides high-level explanations of model mistakes using concepts that are easy for users to understand.
Generating Conceptual Counterfactuals
a) Learning Concept Bank
b)
Explaining Model Mistake
by by
Zebra
by
by
by
stripes
by
+
WC
African hunting dog (68%)
by. )
dy (
Correct label
Zebra
by
bz
pm
Stripes
1.00
b1
African
Cow
C Caripes Cow
Custeer
Cieel
R
Hunting Dog
Pollio
N.XW
WC Concepl explanation
Dog
-0.35
Figure 1: Explaining with Conceptual Counterfactuals: (a) In generating conceptual explanations, the first step is to define a concept library. After defining a concept library, we then learn a concept activation vector (CAV) (Kim al. 2018) for each concept. (b) Given misclassified sample, such as the Zebra image shown here as an African hunting dog, we would like to generate a conceptual counterfactual. Meaning that we would like generate a perturbation in the embedding space that would correct the model prediction, using a weighted sum of our concept bank. (c) Our method assigns score to the set of concepts. large positive score means that adding that concept to the image (e.g. stripes) will increase the probability of correctly classifying the image, as will removing or reducing a concept with a large negative score (e.g. polka dots or dogness).
Fig. [ provides an overview of our CCE method. To summarize our contributions, we develop novel method, CCE, that can systematically explain model mistakes in terms of high-level concepts that are easy for users to understand. We show that CCE correctly identifies spurious correlations in model training, and we quantitatively validate CCE across different experiments over natural and clinical images. We have released all of the code and data needed for method in public repository: //githb.
RELATED WORKS
CCE is inspired by prior efforts to explain machine learning models' predictions. Two ideas are particularly relevant: counterfactual explanations and concept activation vectors. Here, we provide an overview of these methods and discuss the differences between them and our proposed approach.
Counterfactual Explanations Counterfactual explanations (Verma al. (2200) provides com- prehensive review) are a class model interpretation methods that seek to answer: what perturbations to the input needed for model's prediction change in particular way? A large number of counterfactual explanation methods have been proposed with various desiderata such as sparse perturbations (Wachter et al. 2017), perturbations that remain close the data manifold (Dhurandhar
2

---

Under review as a conference paper at ICLR 2022
et al. 2018). and causality (Mahajan et al. 2019). What these methods share in common is that they provide an interpretable link between model predictions and perturbations to input features. There several works focusing on counterfactual explanations for image data(Goyal et al., 2019 Siinlla et al. 2020) and many of these methods use distractor images or modify input images to explain the model behavior. However, perturbations saliency maps in the input space are shown to be tricky to interpret and be adopted by users(AAqaran et al. 2020 Adebayo et al. 2018).
To compute CCE, we use similar approach, perturbing input data to change its predictions, but for a complementary goal: to understand the limitations and biases of our model and its training data, rather than to change a specific prediction. Further improving upon existing work, we do this by directly using human- -imma--ittrrrettbble concepts, without having access to training data.
Concept Activation Vectors To explain an image classification model's mistakes in a useful way, we need to operate not with low-level features, but with more meaningful concepts. Concept activation vectors (CAVs) (Kim et al. (018) are powerful method to understand the internals of a network in human-interpan intereeeteeeee concepts. CAVs are linear classifiers trained in the bottleneck layers of a network and correspond to concepts that are usually user-defined or automatically discovered from data (Ghorbani et al.) 2019). Unlike many prior interpretation methods, explanations produced aantaaannnnddrrtdaaablleee concepts rather than individual pixels (Simonyan et al. 2013 Zhou et 2016) or training samples (Koh & Liang 2017).
by CAVs are terms of high-level,
In previous literature, CAVs have been used to test the association between concept and the model's prediction on a class of training data (Kim et al. 2018) as well as provide visual explanations for predictions (Zhou et al.] 2018). Our approach extends CAVs by showing that perturbations along the CAV can be used to change model's prediction on a specific test sample, e.g. to correct a mistaken prediction, and thereby be used in a similar manner counterfactuals. Since samples (even of the same class) can be misclassified for different reasons (see Fig. 6). our approach allows more contextual understanding of a model's 's behavior and mistakes.
Bias/Error Detection and Robustness Understanding a model's behavior and explaining its mis- takes is critical for building more robust and less biased models, as we have discussed in Section I While some other methods, such as FairML (Adebayo Kagal| (016)) and the What-If Tool (Wexler et al. 22020), can also be used to discover biases and failure points in models, they are typically limited to tabular datasets where sensitive features are explicitly designated, unlike our approach which is
designed for unstructured data and works with a more flexible and diverse set of concepts. In similar spirit to our work, Singh et al. (2020) aim to identify and mitigate contextual bias, however, they assume access to the whole training setup and data; which is limiting factor in practical scenarios.
Our approach can also be used to explain a model's performance degradation with data drift. is complementary to methods for cit-off-distribttion sample detection (Hendryeks & Gimpell 2016) and black-box shift detection (Lipton et al.) 2018), which can detect data drift, but cannot explain the underlying reasons. There are also algorithmic approaches to improve model robustness with data drift (Raghunathan et al. 2018 Nestor et al. (019). Although these methods can provide some guarantees around robustness for unstructured data, they are in terms of low-level perturbations and do not hold with more conceptual and natural changes in data distributions. There are several recent methods for detecting clusters of mistakes made by a model (Kim al. 2019) d' d'Eon al. 202T). which is useful for detecting disparity in the model, for example. CCE differs in that instead of finding mistake clusters, CCE explains errors from just one (or few) image.
3 METHODS
In this section, we detail the key steps of our method. Let us define basic notations: let f Rd R be a deep neural network, let Rd be test sample belonging to class y E (k) We assume that that the model misclassifies æ. meaning that arg max, f(x); # y or simply that the model's confidence in class y. f(=)p. is lower than desired. Let m be the dimensionality of the bottleneck layer L, and by Rd - RM be the "bottom" of the network, which maps samples from the input space to the bottleneck layer, and t,. : Rm + R be the "top" of the network. defined analogously. For readability, we will usually underline class names and italicize concepts throughout this paper.
3

---

Under review as a conference paper at ICLR 2022
3.1 LEARNING CONCEPTS
In generating conceptual explanations, the first step is to define a concept library: a set of human- interpretable concepts C (2,,,,,... that occur in the dataset. For each concept, we collect positive examples, Pe. that exhibit the concept, as well as negative examples, Ne., in which the concept is absent. Unless otherwise stated, we use P.. .N. 100. These concepts can be defined by the researcher, by non-ML domain experts, or even learned automatically from the data (Ghorbani et al. 2019). Since concepts are broadly reusable within data domain, they can also be shared among researchers. For our experiments with natural images, we defined 170 general concepts that include (a) the presence of specific objects (e.g. mirror, person), (b) settings (e.g. street, snow) (c) textures (e.g. stripes, metal), and (d) image qualities (e.g. blurriness, greenness). Many of our concepts were adapted from the BRODEN dataset of visual concepts (Fong & Vedaldi, 2018). Note that the data used to learn concepts can differ from the data used to train the ML model that we evaluate.
After defining a concept library, we then learn an SVM and the corresponding CAV for cach concept. We follow the same procedure in (Kim al. 2018), for the penultimate layer of ResNet18 pretrained in ImageNet. This step only needs to be done once for each model that we want to evaluate, and the CAVs can then be used explain any number of misclassified samples. We refer the reader to Appendix [ I..Ifor for implementational details and Appendix B for a full list of concepts. We denote the vector normal to the classification hyperplane boundary as (nnomaalized, i.e. |ea 1) and the intercept of the SVM as 6, To measure whether concepts are successfully learned, we keep hold-out validation set and measure the validation accuracy, disregarding concepts with accuracies below a threshold (0.7 in our experiments, which left us with 168 of the 170 concepts). We provide more details about the threshold in Appendix
3.2 CONCEPTUAL COUNTERFACTUAL EXPLANATIONS
Drawing inspiration from the counterfactual explanations literature (Verma al., 2020), we generate a perturbation for given misclassified test sample by varying the amount of different concepts in a way such that the perturbation satisfics the following principles:
Correctness: counterfactual considered correct if it achieves a desired outcome. In our case, this would mean that the perturbed test point should be classified as the correct label. 2. Validity:
We would like our counterfactuals to be valid, such that they would not violate real-world conditions. In our case, this means ensuring that the perturbed points contain realistic levels of each concept, as discussed below.
3. Sparsity: The ultimate goal of generating explanations communicating them to users. An- alyzing a large number of modifications and interactions may not be trivial, so perturbations should change a small number of concepts.
Let LCE denote the cross entropy loss. Using the concept vectors and intercepts (c,, (see 3.1 we build our concept bank C' E R.Nexm Φ E R.N. where N₂ is the number of concepts and m is the output dimension of the bottleneck layer. As our goal is to come up with a scoring scheme for concepts, we need to define what means to add a concept. For this purpose, we use statistics of training samples. We compute the geometric margin to the decision boundary of the SVM, d; = ca (x;)²T + Φ., for all of the training examples. As different concepts have different embedding volumes, we scale each concept by the maximum amount of the concept observed in the data used to learn that concept. Let dinax denote the maximum margin in the training distribution.
We let č = dgnax c, and using the scaled concept vectors we construct our final concept bank C. For example, adding unit of Credness would mean adding the maximum amount of redness seen before.
Our optimization problem is implemented as follows:
min LCF(y, lr. (r) + w()) + a/w// + 8/20/2
w min < w v w max
s.t.
By minimizing the cross entropy loss, we aim to flip the label of the model prediction to correct the misclassification, to ultimately achieve correctness. We do this by adding weighted sum of concept vectors, weighted by the parameter w. Additionally, we apply clastic net regularization to introduce sparsity in the concept scores.
4

---

Under review as a conference paper at ICLR 2022
We further introduce validity constraints to make sure that the concept additions within realistic range. Concretely, assume that we have a concept that already exists in the image. We can query this fact by looking at the prediction of our concept SVM. If the concept already exists in the image, adding that concept to the image would be less meaningful. Similarly, if a concept is already absent in the image, then removing it should not be valid action. Following these intuitions, will use the bounds [winim, w
to guide the optimization.
First, we should be able to add the concept explain the model's mistake the concept already in the image, i.e. the score w; should not be positive:
max O if c,br(x))* + Ki
Here, N; an offset we use predict the existence of the concept. Setting R4 would mean using the SVM as the decision boundary. As an alternative strategy, we can vary to allow for weaker forms of validity regularization (e.g. by setting the mean positive margin over the training samples).
Moreover, we should be able to restrict the amount of concept we are adding the embedding, e-g. adding an infinite amount of redness may not be meaningful. We first use the training samples to identify maximum geometric margin for the concept duuxx, Then we restrict wy such that concept addition would result in margin at most as large the maximum margin over the training samples. We want to have
c (b)(()) + w;C;)) + + S diaax
(3)
and thus
mmax
c.. bz. (xr)
dmax
c/ (xx) 02
W;
(4)
c C2
draax
Combining this with the Equation 2 would lead our final constraint:
if cabr, (x= + + kz.
W
max
de
(m))
(5)
else
In a similar fashion, we calculate the lower bounds on scores as:
if
of cx) + + -6;
W
dmnin
of bz (xc)
else
Aunta
We solve the problem in Equation D Jusing Projected Gradient Descent, where we introduce projection steps to enforce the validity constraints. Namely, after each gradient step, we clamp the values the scores to remain within the precomputed range. The final Conceptual Counterfactual Explanations (CCE) algorithm is given in Algorithm[1 Throughout the experiments, unless otherwise stated, we 0.1, B 0.01, 0.01, 0.9, 0.
summary, we propose that using validity con- straints and achieving a cor- rect counterfactual, explain the model mistakes and behavior in an inter- pretable (sparse) manner. A large positive score means that adding that concept to the image will increase the probability of correctly clas- sifying the image, as will re- moving reducing cept with a large negative score. CCE provides with assessment of which concepts explain misclassified sample.
Algorithm 1: Conceptual Counterfactual Explanations (CCC))
Input: at Y, min max], j, Ō
can
Hyperparameters: aa, Y,
Output: w
while us not converged do
① + tr (b,, wC)
Listal < Ler + alwu) + 3 ul/2 ur f- GradientDescent( Lunaa, un, Ir = momentum n)) w f clamp w, min wxxx

---

Under review as a conference paper at ICLR 2022
4 RESULTS
In this section, we demonstrate how CCE can be used to explain model limitations, First, we show that CCE reveals high-level spurious correlations learned by the model. We then show analogous results for low-level image characteristics. Finally, show real-world medical applications where we are able to identify biases and spurious correlations the training dataset and give users feedback about the image quality. In all scenarios, CCE (Alg. T correctly identifies biases in the model and artifacts in the image as explanations of model mistakes.
Training dataset with spurious correlation: dogs
Do top conceptual explanation recover the spurious correlation?
Snow
1.00
Train model
Bed
0.35
Misclessiled
sample
correlated "Show"
comrllaeed Ued
"Stra" commiatedd with "Sand"
Frertton er samales
Precttonn or samates
Phetlan Saseemoke Ke
0.4
0.4
02.
Random (Controll
Randam (Contrel
Fanccmm DCE (Contrus
DCE
Figure Validating CCE by Identifying Spurious Correlations: First, we train 5-class animal classification models on skewed datasets that contain spurious correlations that occur in practice. For example, one model is trained on images of dogs that are all taken with snow. This causes the model to associate snow with dogs and incorrectly predicts dogs without snow to not be dogs. We test whether our method can discover this spurious correlation. (b) We repeat this experiment with 20 different models, each that has learned different spurious correlation, finding that in most cases the model identifies the spurious correlation more than 90% of the misclassified test samples. For comparison, also run CCE using control model without the spurious correlation, and randomly select concepts as well. The random performance is evaluated by sampling three concepts of the available 150 and calculating the Precision@::
4.1 CCE REVEALS SPURIOUS CORRELATIONS LEARNED BY THE MODEL
We start by demonstrating that CCE correctly identifies high-level spurious correlations that models may have learned. For example, consider training dataset that consists of images of different animals in natural settings. A model trained such dataset may capture not only the desired correlations related to the class of animals but spurious correlations related to other objects present in the images and the setting of the images. We CCE identify these spurious correlations.
To systematically validate CCE, we need to know the ground-truth spurious correlations that a model has learned. To this end, train models with intentional and known spurious correlations using the MetaDataset (Liang & Zou| 20 a collection of labeled datasets of animals in different settings and with different objects. We construct 20 different training scenarios, each consisting of 5 animal classes (cat, dog, bear, bird, elephant). We trained a separate model for each scenario. In each scenario, one class is only included with specific confounding variable (e.g. all images of dogs are with snow), inducing spurious correlation in the model (images of dogs without snow will be misclassified), which we can probe with CCE, We also train a control model using random samples of animals across contexts, without intentional spurious correlations. In Appendix A.7.2 we provide training distributions with less severe spurious correlations, i.c. where instead of all images having the confounding concept,
varying level of severity in the correlations.
Our experimental setup is shown in Fig. 2 a). We train models with n = 750 images, fine-tuning a pretrained ResNet18 model. In all cases, achieve validation accuracy of at least 0.7. We then present the models with 50 MetaDattsst),, i.e. images of the class without the confounding variable present during training.
(OOD) images (randomly sampled from the entire

---

Under review as a conference paper at ICLR 2022
We then CCE to recover the top 3 concepts that would explain model's mistake each of 50 OOD images that are misclassified, then we report if the spurious concept is among these concepts(Precision 3). For almost all images, CCE identifies the ground truth spurious correlation as one of the top 3 concepts (Fig. (())). For example, in the model we trained with dogs confounded with snow, CCE recovered the spurtous correlation in 100% of test samples. We compare these results to performing CCE using the control model, in which the same test images are presented, as well as picking concepts randomly, both of which result the spurious correlation being identified much less frequently. We provide results in Appendix A.7.3] where instead of doing sumple-by-sanee analysis we propose Batch-Mode CCE to analyze a set of mistakes to provide a holistic understanding of the biases in the model.
Additionally, we compare our approach to (simpler) univariate version of CCE, CCF(Univarratte) Instead of running our optimization method over multiple concepts simultaneously, we iterate over each CAV and quantify how a perturbation in the direction the concept (dmmax.;) c;) changes the predic- tion probability of the correct class y. Specifically, we compute the CCE the difference t (x) quax G)gg L1 (x) (z)y. . for each concept, then we order the concepts by the change in the probability.
Table provides results over the 20 scenarios and the detailed results can be found in Appendix Table We further provide the mean of the me- dian rank of the concept each sce- nario, along with the mean of the first and third quartiles of ranks. Both CCE(univariate) and CCE recover the ground- -trth spurious correlation as one of the top 3 concepts correctly across the scenarios. The complete (multivariate) CCE achieves the high- est precision and the best median rank. In Appendix A.7.1 we provide results in more challenging scenarios, where the target spurious concept does not exist in the concept bank.
Method
Mean Prec@3
Median Rank
Random CCE(Control)
0.02
82.65 42.7, 120.4)
0.04
32.3(28. 03, 40.05)
CCE(Univariin
0.91
2.00 2.35)
CCE
0.95
(1.800.80, 2.10)
Table 1: Empirical evaluation for detecting spurious corre- lations in training data. We report Precision @3 and ranks averaged over 20 scenarios, the complete table is in Appendix Table 2 Distribution of the Precision @ K metric as we vary K can be found in Appendix A.8 A.8
b) Spurious Correlation: Dog(Water) Dog(Bed)
1.0
'Greenness' CCE
Granny Smith' prob prodicted,
Pred.
Images
Method
Water
Bed
0.6
Elephant
Jniverate
(3733)
volidiy>
3
0.4
254
Bear 422
0.0
00
0.2
0.4
0.6
1/o
Ooline valiity
Degree of perturbation
Full OCE
137
Bird (27
Univariate
6
2
vnlidtyy
2
Full
Figure 3: Validating CCE: (a) Here, we take an arbitrary test sample of a Granny Smith apple that originally correctly classified and perturb the image by turning it gray. We compute CCE scores at each perturbed image and observe that the score for greenness increases as the image is grayed. (b) We demonstrate the effectiveness of validity constraints in a qualitative scenario. In the last two columns, we provide ranks of each concept when a particular method is used. Without validity. methods can use concepts that already exist the image to explain model mistakes.
4.2 CCE REVEALS LOW-LEVEL IMAGE ARTIFACTS
We show that CCE can capture low-level spurious correlations that models may have learned. For example, the ImageNet(Deng et al. (209) dataset on which SqueezeNet was trained includes of green apples known Granny Smith, which were always colored images. Fig. a)) shows that as
a class
7

---

Under review as a conference paper at ICLR 2022
the image is grayed. the probability of it being classified Granny Smith decreases, while the CCE for greenness increases. We provide details and examples about this experiment in Appendix|AA5 (A.6
4.3 VALIDITY CONSTRAINT IMPROVES EXPLANATIONS
Here we demonstrate validity constraints are necessary to plausibly explain model mistakes. In these experiments, instead of training the model with a single concept that associated with a class, we use two concepts. For instance, we use 50 dog images with water and 50 with bed during training. In Figure 3h we show such scenario. In all of these images, CCE(Univariate)) identifies both bed and water concepts explain the model mistakes. However, in the first image, there is already water and in the second image, there is already a bed. Thus, using them in the counterfactual result in an invalid explanation. However, when CCE with validity constraints is used, the score for the water concept in the first image and the bed concept in the second image drops. Ultimately, this would result in plausible counterfactual explanation. More examples are provided in Appendix A.3
4.4 EVALUATING CCE IN THE WILD
We run experiments where we evaluate CCE on real-world medical data and models. We demonstrate that CCE helps identify learned biases and artifacts that provide insight into model's s mistakes. Throughout our evaluations, we worked with board-certified dermatologist and cardiologist to confirm the clinical relevance of CCE explanations.
Dermatology- Skin Condition Classification: We follow the model training procedure described in (Groh et al., 202T) to train ResNet18 model to predict one the 114 skin conditions using the Fitzpatrick dataset of 16,577 annotated skin images (Groh et al. (021). This classifiation model achieved 20% overall accuracy, closely matching the number reported in the paper (see Appendix A.4 details). To explain the model's mistakes, in addition to the 168 concepts used in Sec. 3.1 we also learned 8 clinically relevant concepts: defocus blur, zoom blur, brightness, motion blur, contrast, dark skin type, skin hair, and zoom. To learn each of these concepts,
25 pairs of positive and negative images (except the skin hair concept, for which we used 10 images, where the skin hair images are obtained from the ISIC (Rotemberg al.
2021 dataset). In Figure 4 observe several
a) Label: Allenic Contact
Label: Eboed Fruptions
c) Label: Mucinoss
d) Labal: Sarooldosis
Dermattis
Pred: Erythema
Pred: Aplesia Cotis
Pred: Nevus Sebaceous
Pred Stadis Edema (1elt
Nodooumaacia
Iarlesohhn (260))
Water
Blttchy
Zoom
0,58
Redhwss
0.5
Outskde am
0.3
Eye
078
Water
Ashcan
0.5
Blackress -0.4
Ashoan
-1.22
Molion
-051
Blur
-Bod
038
Skin Hair
-0.2
Figure 4: CCE explains model mistakes using learned biases and image quality conditions. (a) CCE identifies dark skin type correlation with the allergic contact dermatitis condition that exists in the training dataset. (b, c, d) CCE identifies image artifacts that degrade the model performance.
ways CCE guides our understanding of model mistakes. In addition to the unbalanced fraction of skin type groups over the whole dataset, find that there are wider discrepancies when particular skin conditions are considered. For example, for the allergic contact dermatitis condition, there are 259 images from the lightest skin tones in the training data, 137 images of intermediate skin tones, and only 24 images of dark skin tones. In the test set, for 23 out of all 24 allergic contact dermatitis dark-skin images where the model makes a mistake, CCE identifies the dark skin type concept as one of the concepts with the largest negative score. This collectively reflects bias in the training data.
For fixed cruptions and mucinosis, CCE finds image qualities that would increase the classification performance. Namely, CCE identified that the blur in 2nd image in Fig. caused the model's mistake
8

---

Under review as a conference paper at ICLR 2022
(reducing blur would correct the mistake). In the 3rd image. CCE learned that the image is too zoomed out, and increasing zoom would correct the mistake. For the 4th image, CCE identified that too much skin hair in the image contributed to the model's mistake. This is consistent with other studies which show that presence of skin hair can degrade the model performance (Okur & Turkan 2018). In all cases, CCE identifies relevant concepts from our expanded concept bank and displays them to the user. These explanations were validated by a trained dermatologist as plausible reasons why these images may have been difficult to classify. In Appendix A.4, we describe details of this experiment and include comments about concepts suggested by CCE that seem irrelevant.
Cardiology Pneumothorax Classification from Chest X-Ray Images Several studies have raised concerns about the biases learned by chest X-ray diagnosis models and their deployability in novel domains (Seyyed-Kalanteee Kalantrii et al., 2020; Larrazabal et al. 2020 Wu et al. 2021). We investigate cross-site evaluation setting, where models trained on Chest X-Ray images used to classify the pneumothorax condition. Specifically, we are taking a binary classification model trained on a dataset collected from the National Institutes of Health Clinical Center in Bethesda (NIH) (Wang al. 2017) and we test the model with images obtained from the Stanford Health Care in Palo Alto (Irvinrviin
et al. 2019). In this experiment, we follow the protocol described in (Wu al. (02T); see Section A.5 for details. Notably, the NIH dataset consists of images from the frontal (AP/PA) view positions, whereas in the SHC dataset, there are also images obtained from the Lateral View. This would mean that the model has not seen any images from the lateral view during training. Several examples of these images can be seen in Appendix Figure TO Similar to See 4.4 we use 100 pairs of chest X-ray images to learn clinically relevant concepts: Lateral View & AP View(view positions), Cardiomegaly & Alelectusiscoommoriiiiies patient gender, and patient age under 24.
From the SHC dataset, we randomly select 150 images taken from the lateral view where the model makes a mistake. We run CCE using a concept bank with 120 BRODEN concepts, in addition to our clinically relevant concepts. In 100% of these 150 test images, the Lateral View concept received the largest negative score, meaning that CCE finds removing the concept would increase the probability of correctly classifying these images. This is consistent with cardiologist expectation: a model that has not seen images from the lateral view may fail to generalize to that category. With the help of CCE, end-users can identify these shortcomings and help address the biases in the training dataset.
CONCLUSION AND FUTURE WORK
We present a simple and intuitive method, CCE, that generates meaningful insights into why machine learning models make mistakes on test samples. We validated that CCE identifies spurious correlations learned by the model for both natural images and medical images, where CCE's explanations are confirmed by clinicians. In medical applications, CCE was able to inform the end-user about the image quality conditions and identify biases in the model's training data. CCE is fast: the concept bank just needs to be learned once using simple SVMs and each test example takes < 0.3 seconds on single CPU. can be readily applied to any deep network without retraining and provides explanations of mistakes in human-interpretable terms.
CCE can detect biases in training data that lead to model mistakes without needing the training data. It requires a small number of labeled examples to learn concepts. For instance, the dermatology experiments used 50 images to learn each concept. The data used to learn concepts can come from datasets different from the ones used to train the model. In the dermatology case study, we learned concepts using the ISIC dataset (for skin hair) and still correctly explained the mistakes of a model trained on the Fitzpatrick 17k data. This makes the CCE method more broadly useful. The quality of the concept bank is a crucial component for the outcome of the explanations, as demonstrated in Appendix A.7.1 It is important seek guidance from the experts of the problems when building concept libraries to obtain the best results. Incorporating automatic concept learning Ghoorban et al. 2019 is fruitful future direction, as it could further simplify the entire pipeline.
There are several promising areas where CCE can be extended. While we focused in this paper on image classification tasks, CCE can be applied other data modalities such as text, audio, and video data, as well as other tasks such as regression and segmentation. Finally, we seek to do user studies to understand how human subjects respond to explanations with CCE and how it drives improvements in model debiasing and robustness.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. Gradio: Hassle-free sharing and testing of ml models in the wild. arXiv preprint arXiv:1906. 02569, 2019.
Julius Adebayo and Lalana Kagal. Iterative orthogonal feature projection for diagnosing bias in black-box models. arXiv preprint arXiv: 2016.
Julius Adebayo. Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò DisaaBBancchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, 2018 URL
http://a.....rrr....oop.pppp/ppp wwtppccppppppaa..ccee / AwstrrAAbsttttt.t.. html
neurips
Ahmed Alqaraawi, M. Schuessler, Philipp WeiB, Enrico Costanza, and Nadia Bianchi-Berthoure Evaluating saliency map explanations for convolutional neural networks: a user study. Proceedings of the 25th International Conference on Intelligent User Interfaces, 2020.
Aditya Chattopadhay, Anirdan Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++ Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 839-847, 2018. doi: 10.1 N..109/AACV.2018000000
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009, doi: 10.1 200.CC/R.2009.520000
Greg d' l'Eon, Jason d' d'Eon, James R Wright, and Kevin Leyton-Brown. The spotlight: A general method for discovering systematic errors in deep learning models. arXiv preprint arXiv:2107.00758, 2021.
Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. arXiv preprint wxXiv:882.0076233 2018
Thomas B. Fitzpatrick. The Validity and Practicality of VI. Archives of Dermatology, 124(6):869 871, 06 1988. ISSN 0003-987X. doi: 10.1001/ archderm. 101888888800000 670060015008. URL htt.:ttss:// doi wwt.dooooorggg..oo 1001/ http///rchhrrmre rm. 1988
Skin Types I Through
01670060015008
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8730-8738, 2018.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations. arXiv preprint arXiv: 1902.v:902.03-- 0902003/99 2019.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruy Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. ICML 2019. 9-15 June 2019, Long Beach, PMLR,
California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2019.
http:///prrooeeeeee.....e.rss/sr ml www.ppess//v////// goya 11 9a html
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. arXiv preprint arXiv:2 104.004409997 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2022
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-diisrrbuttio examples in neural networks. arXiv preprint 1rXiv:1610.02116,, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 4700-4708, 2017.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv:1602.07366 2016.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-5990 2019.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nlp. arXiv preprint arXiv:2104. 143377 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretabilityy beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Sociity, pp. 247-254, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69888 69800, 2014.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning. pp. 1885-1894. PMLR, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012.07421, 2020.
Agostina J Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H Milone, and Enzo Ferrante. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aine aided diagnosis. Proceedings of the National Academy of Sciences. 117(23): 12522-2::5922125533 2020.
Weixin Liang and James Zou. Metadataset: A dataset of datasets for evaluating distribution shifts and training conflicts. arXiv preprint, 2021.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122-3130. PMLR, 2018.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12)2346-2363, 2018.
Divyat Mahajan, Chenhao Tan, and Amit Sharma. Preserving causal constraints in counterfactual explanations for machine learning classifiers. arXiv preprint 1rXXv:1912.03277, 2019.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.074800 2019.
Bret Nestor, Matthew BA McDermott, Willie Boag, Gabriela Berner, Tristan Naumann, Michael C Hughes, Anna Goldenberg, and Marzych Ghassemi. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks, In Machine Learning for Healthcare Conference, pp. 381-405. PMLR, 2019.
11

---

Under review as a conference paper at ICLR 2022
Erdem Okur and Mehmet Turkan. A survey automated melanoma detection. Engineering Applications of Artificial Intelligence, 73:50-67, 2018.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Semidefinite relaxations for certifying robustness adversarial examples. arXiv preprint arXiv:1811.0105; 2018.
Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz-Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Marc Combalia, Stephen Dusza, Pascale Guitera, David Gutman, et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Scientific data, 8(1):1-8, 2021.
Laleh Seyyed- Seyyed-Kalantari, Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Chexclusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: Proceedings of the Pucific Symposium, pp. 233-243. World Scientific, 2020.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, arXiv preprint 1rXiv:1312.660 2013.
Krishna Kumar Singh, Dhruv Kumar Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge an object by its context: Learning to overcome contextual bias. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
11067-11075, 2020.
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exageeration. In International Conference Learning Representations, 2020. URL
https://openreviews net/ www.//eeett/e.eeeeeee worummrriiii HIxFWgrFPP
Adarsh Subbaswamy, Peter Schulam, and Suchi Saria. Preventing failures due to dataset shift: Learning predictive models that transport. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 18-3127. PMLR, 2019.
Sahil Verma, John Dickerson, and Keegan Hines. Counterfactual explanations for machine learning: A review. arXiv preprint arXiv:2010. 10596, 2020.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017.
Wang, Y Peng, L Lu, Lu, M Bagheri, and R Summers. Hospital-scale chest x-ray database and benchmarks on weakkly-uupervied classification and localization of common thorax diseases. In IEEE CVPR, volume 2017.
James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viégas, and Jimbo Wilson. The what-if tool: Interactive probing of machine learning models. IEEE Transactions Visualization and Computer Graphics, 26(1):56-65, 2020. doi: 10. 109/TTCG.221999 2934411....
Eric Wu, Kevin Wu, Roxana Daneshjou, David Ouyang, Daniel E Ho, and James Zou. How medical ai devices are evaluated: limitations and recommendations from an analysis of fda approvals. Nature Medicine, 27(4):582-55 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 119-134, 2018.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 LEARNING CONCEPTS
Here we provide implementational details on how learn the concept activation vectors (CAVs). Concretely, we pick bottleneck layer in model, which is the representation space in which will learn our features. Unless otherwise stated, we choose the penultimate layer ResNet18 for experiments in this paper. Let be the dimensionality of the bottleneck layer L, and by R - R be the "bottom" of the network, which maps samples from the input space to the bottleneck layer, and 4, : mm R be the "top" of the network, defined analogously. In generating conceptual explanations, the first step is to define concept library: a set of iuman-interpretable concepts C' {or, c that occur in the dataset. For each concept, we collect positive examples, P& that exhibit the concept, as well as negative examples, No., in which the concept is absent. Then, we train support vector machine classify {62 (a) P& from {br (x)) x E Ne. the same way as in Kim et al. (018)). Overall pseudocode for the procedure can be found in Figure
Algorithm 2: Learning concept vectors
Inputs:
f # trained network: model
L # bottleneck layer (hyperparametez): int
concepts set of concepts: set[str]
P # positive examples per concept: (str, list [sample]]
N negative examples per concept: dict[stx,
1ist [sample]]
Return: svms #Set of SVMs containing concept predictors.
b, t = f.layers [:L], [[:] | Divide network f() into botton (first | layers) and top t (remaining layers) so that f()= = xb)())
for c in concepts: # Per concept, learn an SVM to classify bottleneck representations of positive and negative examples. svms [c] svm. train (b([))), (N[c)))
# Filter out concepts that are not learned well (i.e. validation accuracies below a particular threshold). if svms [c] aace .7:
del svms [c]
Figure 5: Pseudocode for CES in Python-like syntax concept learning procedure. Learning the concepts (lines 1-6) just needs to be done once and can be carried out offline.
A.2 METADATASET EXPERIMENTS
In Table 2 provide results over 20 scenarios. In each of the scenarios, fine-tune only the classification layer of ResNetl 8(He et al. 2016) pretrained on ImageNet, which outputs a probability distribution over 5 animals(cat, dog, bear, bird, elephant). For instance, in the case of the dog(snow) experiment, we train the model with images of dogs that are all taken with snow. We replicate this experiment with 20 different animal & concept combinations and report the results below.
In Table for each scenario, we sample 50 images with and without the concept and we report the accuracy over those images. We observe a start difference between accuracies, which verifies that model learns to rely on the correlation.
A.3 VALIDITY EXAMPLES
In Fig. 7 we provide additional examples on where validity improves explanations. In a) we have a model trained with the dog-snow spurious correlation and in we have the dog class associated with both the horse and bed concepts. For both of these examples, we observe that in the images where
13

---

Under review as a conference paper at ICLR 2022
Experiment
CCE-Prec. @
CCH(nnivariae))-Pre.. @
CCE(Contro))-Prec.ee @3
dog(chair)
0.980
0.360
0
cat(cabinet)
0.180
dog(snow)
dog(car)
0
dog(horse)
0.500
bittrw)
0,960
0.660
dog(water)
0.980
0.980
0
dog(fence)
0.980
0
clephan((building)
0.821
0
cat(keyboard)
0.860
0.800
0
dog(sand)
0.760
0,900
0
cat(computer)
0.980
0
dog(bed)
0
cat(bed)
0.980
0
cat(book)
0.960
0
dog(grass)
0.700
0.780
0
cat(mirror)
0.900
0.900
0
bird(sand)
0.960
bear(chair)
0.940
0.720
0
cat(grass)
0.940
0,900
0
Table 2: Empirical evaluation for detecting spurious correlations in the training data. We report results over 20 scenarios, where each class is associated with the concept in parenthesis during the training phase.
Experiment
Accuracy for Images with the Concept
Accuracy for Images Without the Concept
dog(chair)
0.74
0.54
cat(cabinet)
0.8
0.62
dog(snow)
0.76
0.12
dog(car)
0.86
0.66
dog(horse)
0.70
0.36
bird(water)
0.78
0.42
dog(water)
0.82
0.4
dog(fence)
0.74
0.52
clephant(building)
0.82
0.72
cat(keyboard)
0.96
0.46
dog(sand)
0.66
0.42
cat(computer)
1.0
0.68
dog(bed)
0.86
0.46
cat(bed)
0.84
0.54
cat(book)
0.9
0.64
dog(grass)
0.96
0.56
cat(mirror)
0.86
0.4
bird (sand)
0.78
0.66
bear(chair)
0.82
0.44
cat(grass)
0.72
0.46
Table 3: Accuracy of the model for images tested with and without the confounding variable.
the proposed concept already exists, validity constraints help prevent the model from explaining the mistake by adding that particular concept, which results in valid explanations.
14

---

Under review as a conference paper at ICLR 2022
Input to Method
a
African hunting dog (66%)
African crocodile (42%)
Correct label
Zebra
Correct label Zebra
Stripes
1.00
Field
1.00
Cow
0.52
Cow
0.89
Dog
-0.35
Stripes
0.87
Flag
0.20
Buildings
0.82
Polka dots
-0.20
Water
-0.79
Conceptual Counterfactuals
Figure 6: Different reasons of model mistakes for the same class..
a) Spurious Correlation: Dog(Snow)
b) Spurious Correlation: Dog(Horse) Dog(Bed)
Dog(Snow)
Pred.
Images
Method
Horse
Red
Bird (18%)
Pred: Bear (475)
Bear
Univariate
3
2
(33
8
2
velidily)
Full CCE
158
Cal Cariia
Univariate
CCEino validity)
Metbod
Bank of Snow
Method
Bank Snow
Full CCE
Univariete
3
Univeriate
Bird (20
Univariate
2
CCEino validty
1
velidily)
OCDino velidity)
Full CCE
154
Full CCE
Full CCE
2
Figure 7: Validity constraint improves explanations.
A.4 DERMATOLOGY EXPERIMENT WITH FITZPATRICKI74
We directly follow the experimental protocol in Groh et al. (002I). Fitzpatrick1 7k (Gro et al. 2021 dermatology dataset that contains 16,577 skin images with 114 different skin conditions. For
15

---

Under review as a conference paper at ICLR 2022
these images, skin type labels are provided using the Fitzpatrick system (Fitzpatrick 1988) Groh et al. (202I) decomposes the dataset into three groups: The lightest skin types (1 & 2) with 7.755
images, the middle skin types (3 & 4) with 6,089 images, the darkest skin types (5 & 6) with 2.168
images. We randomly partition the dataset into training(80%) and essting(0%) sets. We use a ResNet18 (He et al.; 20T6) backbone pretrained a classification head to predict one of the 1 14 skin conditions using optimizer. The classification head consists of 1) fully-connected layer with 256 hidden units 2) relu activation 3) dropout layer with a 40% probability of masking activations 4) another fully-connected layer with the number of predicted categories. We obtain the training script from the repository for the paper Groh et al. (2021).
ImageNer(DDeng et al.] 2009) and we fine-tune & Ba 2014)
We are using the imagecorruption 2 Michaelis et al., 2019) library to obtain positive samples of defocus blur(3), zoom blur(2), brighiness33), motion blur(3), and contras((2), where we used the severity levels provided parenthesis. Examples of positive images for these concepts can be found in Fig.
We learn the zoom concept by cropping the height and width of images to a fourth their sizes. We obtain positive images from the darkest skin types (5&6) and negative images from the lightest skin types (1&2) to learn the dark skin color concept. For all dermatology concepts except the skin hair concept, we use 25 positive & negative pairs of images. For the skin hair concept, we use 10 pairs of images from the ISIC dataset (Rotemberg et al. 2021).
A.4.1 EXPLANATION ARTIFACTS
It is important to note that sometimes concepts such as water or ashcan may be suggested as explanations by CCE. Concepts are represented as vectors in the embedding space, and there may be similarities among them. Hence, water or ashcan concepts may be very similar to certain textures that are relevant to the dermatology task, where it was not possible to obtain the true concept itself. The practitioner using CCE to explain the mistake can easily discard these artifacts suggested by CCE, where there is also concepts that are more relevant and obtained larger scores. Secondly, as the user keeps using concept-based methods, they can understand what type of concepts is needed, or is relevant to the task. Accordingly, it is possible to update and obtain a more informative bank. Generally, there are various interesting directions that could be pursued in the machnne interface that would improve the real-life performance of explanation algorithms. We are also interested in further understanding this, and think more about the deployment aspects of CCE in future work.
A.4.2 TESTING GRADCAM+ IN THE FITZPATRICK SETTING
As we mention in Section 4.4. the allergic contact dermatitis condition has biased skin color distribution in the training data Activation-mmap-asse methods such as MaadCAM++(hattopadaaa eet al. 2018) fail to communicate this to the user. However, CCE can correctly identify this as reason why the model fails. In Figure we show two examples where CCE is able to identify the bias in the data that drives the model mistake; however, one of the methods that very commonly used to understand model predictions, GradCAM++, fails to identify this reason.
A.5 CARDIOLOGY EXPERIMENT WITH CHEST X- X-RAYS
We investigate a cross-site evaluation setting, where models trained on Chest X-Ray images are used
to classify the pneumothorax condition which was proposed in (Wu al. (222I). Using pretrained models from (Wu et 2202T) we use a DenseNet-122 121 ((Hang èt al. 2017) architecture pretrained on ImageNet(Deng et al. 2009) and then fine-tuned on the NIH dataset Wang et al.) 2017). As is reported in their paper, this model achieves 0.779 mean AUC when tested on the (Irvin et al.) 2019) dataset and 0.903 mean AUC when tested on the NIH dataset. We use CCE to explain mistake over a subset of the SHC dataset. Specifically, we run CCE over the images taken from the lateral view, which does not exist in the training dataset since NIH only contains images from the frontal (Anterior- Posterior(AP) or Posterior-Annteiior(PA) view positions. This constitutes real-life scenario of
16

---

Under review as a conference paper at ICLR 2022
Positive Samples for Image Quality Concepts
Original
zoom blur:2
zoom blur:3
zoom
200m blur:5
Original
contrast:2
contrast:3
contrast:4
contrast:5
Original
defocus blur:2
defocus blur:3
defocus blur4
defocus blur:b
Original
brightness:2
brightness
brightness:4
brightness:5
Original
motion blun2
motion blur:3
motion blur:4
motion blur:5
Figure 8: Learning image quality concepts. Here have the positive samples for different image quality concepts. For each corruption type, we provide examples for different levels of severity. The original image is obtained from the Fitzpatrick dataset.
distribution shift. Some examples of these images can be seen in Figure 10 We sample 150 images that were taken from the lateral view where the model makes mistakes. For 150/150 of those images, CCE identifies the lateral view concept as the concept with the lowest score. Namely, CCE claims that removing the lateral viewness concept would increase the probability of correctly classifying the image.
A.6 ADDITIONAL EXAMPLES VALIDATING CCE THROUGH LOW-LEVEL IMAGE PERTURBATIONS
report case where CCE reveals low-level artifacts learned by SqueezeNet landola et al. 2016) pretrained on ImageNet. Particularly, the ImagetNet dataset on which SqueezeNet was trained includes class of green apples known Granny Smith. These images were always colored in ImageNet, meaning that the model misclassifies images of these apples in grayscale. We this fact gradually transform natural image of Granny Smith apple,
In Section 4.2
17

---

Under review as a conference paper at ICLR 2022
Label: Allergic Contact Dermatitis Pred: Stasis Ecema 6182)
Label: Allergie Contact Dermatiti
Pred: Necrobiosis Lpoliica Ht
Concept explanation suggested by CCE: Remove Dark skin color
Concept explanation suggested by CCE: Remove Dark skón color
Figure 9: CAM based methods fail to communicate model biases. Here we show examples where CCE is able to identify the bias in the data that drives the model mistake. However, one of the methods that are very commonly used understand model predictions, GradCam++, fails identify and communicate the underlying reason.
Lateral View Images
Frontal View Images
Figure 10: Chest X-Ray images from different views. Here we provide several lateral view and frontal view images from the SHC dataset. On the left, we see images from the lateral view and on the right we have images from the frontal view. NIH dataset does not have any lateral view images in the dataset, which explain why model trained the NIH dataset performs poorly when tested on lateral view images.
blending it with its grayscale version. different levels of the original image vs. its transformed version, we run it through SqueezeNet to obtain prediction and then calculate its CCE scores. The results, shown in Fig. (),, show that as the image is grayed, the probability of it being classified as Granny Smith decreases, while the CCE for greenness increases. We repeat this experiment with 25 images of Granny Smith apples and provide results Appendix Fig. [[]... Our results show that CCE is also effective explaining model's mistakes in terms of low-level visual artifacts.
In Fig. 3 we showed an example in which CCE was successfully able to identify why images that being perturbed through low-level image transformations being misclassified. Here, show another pair examples, with a different concept: redness (Fig. 12]. Additionally, in Fig. II we show the aggregated curve over 25 granny smith apple images.
18

---

Under review as a conference paper at ICLR 2022
a)
b)
1.0
'Greenness'' CCE
Granny Smith' prob predicted,
0.8
Greenness CCE
0.6
0.4
0
0.2
0.4
10
0.2
1.0
Degree of perturbation
Degree of perturbation
Figure 11: Validating CCE Through Low-Level Image Perturbations: (a) Here, we take an arbitrary test sample of a Granny Smith apple that is originally correctly classified and perturb the image by turning it gray until is eventually misclassified (in this case, as mortar). We compute the CCE scores at each perturbed input image and observe that the score for greenness increases, corresponding to the degree to which we remove the green color from the image. (b) We repeat this for 25 different images of Granny Smith apples (some of which shown under the plot) and find that the same trends generally hold true (each image a gray line). Although a few images do not follow this trend, the mean CCE score (bolded green line) does.
1.0
1.0
0.8
0.6
pradicted
shaikbrry prob prodicted
022
04
0.6
0.6
10
06
022
Degree of perturbetion
Cegroe perturtation
Figure 12: In analogous manner Fig. 3 take images that originally correctly classified as robin and strawberry, and perturb them by removing red colors until they are misclassified by the model (x-axis). The CCE scores correctly identify the concept of redness as the most important concept for correcting the model's 's mistakes.
A.7 CHALLENGING SCENARIOS FOR CCE
A.7.1 WHEN THE TARGET CONCEPT IS MISSING FROM THE BANK
What happens when the spuriously correlated concept is not in our concept bank? this section, we replicate controlled experiments in Metadataset by omitting the target concept from concept bank. For instance, for the dog(snow) experiment, we remove the concept snow from the set 168 concepts that we have and re-run analysis.
19

---

Under review as a conference paper at ICLR 2022
Experiment
Topp Concepts
dog(bed)
Sofa, Dog. Bedclothes, Muzzle, Headboard
dog(chair)
Sofa, Book, Hand, Bedclothes, Dog
cat(cabinet)
Inside arm, Mierowave, Chest of drawers, Door frame, Oven
dog(snow)
Mountain, Car, Dog, Minibike, Headlight
dog(car)
Bus, Headlight, Motorbike, Fence, Coach
dog(horse)
Cow, Muzzle, Motorbike, Bicycle, Grass
bird(water)
Bird, Airplane, Coach, Sand, Mountain
dog(water)
Muzzle, Dog, Sand, Airplane, Mountain
dog(fence)
Field, Grass, Coach, Muzzle, Dog
clephant(building
Horse, Cow, Pedestal, Car, Bucket
cat(keyboard)
Cat, Computer, Inside arm, Paper, Faucet
dog(sand)
Water, Airplane, Horse, Blueness, Mountain
cat(computer)
Faucet, Keyboard, Cat, Inside arm, Microwave
cat(bed)
Cat, Pillow, Inside arm, Hcadboard, Back pillow
cat(book)
Computer, Cat, Bookcase, Oven, Chest of drawers
dog(grass)
Horse, Field, Muzzle, Bus, Motorbike
cat(mirror)
Inside arm, Headlight, Door frame, Countertop, Bathtub
bird(sand)
Water, Airplane, Bird, Mountain, Blind
bear(chair)
Food, Candlestick, Plate, Lamp, Fabric
cat(grass)
Tree, Mouth, Field, Path, Blind
Table 4: CCE suggestions when the target concept is missing from the concept bank. We average the ranks for each concept over 50 mistakes in each experiment, and report the Top-5 concepts with the highest rank.
In Table we provide the Top-5 concepts suggested by CCE. We use 50 mistakes for each scenario, evaluate the ranks for each concept in our concept bank using CCE, and average the ranks over all the samples that we have. We report the Top-5 concepts with the highest rank. In almost all of the scenarios, CCE identifies concepts that are 'hinting' to the target concepts. For example, in the dog(bed) case, CCE reports Sofa, Dog, Bedclothes, Muzzle, Headboard the reasons for the mistake, which are either related to the target class or the spuriously correlated concept. Generally, we see similar pattern in all of our experiments. This provides another piece of evidence that highlights the importance of the richness of the concept bank. If we have rich-enough bank, then CCE helps us identify the spuriously correlated concept even it is not directly the bank.
A.7.2 WHEN THE CORRELATIONS ARE LESS DRASTIC
What happens when the correlations are less drastic? More concretely, in Section 4.1 we assumed all images in the training dataset contains the spuriously correlated concept, we call this situation I 100% severity in this section. Here, we evaluate when the correlation is less severe. Particularly, what happens when we have a scenario with 50% severity, i.e. when only 50% of the training images contain the spuriously correlated concept? In Figure 13p & b, we report the CCE performance as we vary the severity. The bolded line shows the median performance across 20 scenarios, and the confidence levels show the first and third quantiles. We observe that starting from relatively low levels of severity(* 20%), CCE is able to identify the spuriously correlated concept in a majority of the scenarios. Consequently, realistic scenarios.
that CCE can perform well even in less-severe, and thus more
A.7.3 BATCH EVALUATION FOR A SET OF MISTAKES
Here extend CCE to batch evaluation setting. Namely, instead of explaining individual mistake, we aim to explain a batch of mistakes using conceptual counterfactuals, We do so by making slight modification to our optimization procedure:
20

---

Under review as a conference paper at ICLR 2022
1.00
b)
1.00
Batch CCE nankc
CCE Prec@a
CCE-Prec@5
0.75
20-
15
0.50
0.50
10-
0.25
0.25
0.00
0.25 0.50 0.73 1.00
0.25
0.50 0.75 100
0.2 0.4 0.0 1.0
Severity the correlation
Severity correlation
5everity of the correlation
Figure 13: (a, b) We report the performance of CCE as we vary the severity level of the correla- (K%% severity means K% of the images in the training dataset has the concept). The bolded line shows the median performance across 20 scenarios, and the confidence levels show the first and third quantiles. We observe that starting from relatively low levels of severity(*= 20%). CCE is able to identify the spuriously correlated concept in majority of the scenarios. (c) We evaluate Batch Mode CCE over controlled experiments with varying severity levels. The y-axis corresponds to the rank of the target concepts, the x-axis gives the severity of the spurious correlation, the bold line is the median performance across 20 scenarios and the confidence intervals give the lower and upper quartiles. We observe that Batch Mode CCE is able to provide automated analysis of the model biases using batch of samples, given that it can identify the spuriously correlated concept as one of the major causes all model mistakes for the given class.
N Σ
min w
Len(yi, (bb( + + + t//0// + +/3/2/2
S.L. W min
<
w max
The optimization problem in Equation 7 outputs a single shared set of concept scores that minimizes the proposed loss for a batch of mistakes. Furthermore, we let win N Σ womin and wa N EN witax to make the validity constraints work in the batch-wise setting where winax and wimin denotes the validity bounds for the sample
This formulation results more 'holistic' understanding the model bias compared to the sample- by-sample analysis. In practice, using all of our mistakes in the test dataset, could run Batch-CCE analysis to provide global interpretation of the model biases. performance of Batch Mode CCE over 20 scenarios across different severity levels. The y-axis corresponds to the rank of the target concepts, the x-axis gives the severity of the spurious correlation, the bold line is the median performance across 20 scenarios and the confidence intervals give the lower and upper quartiles. We observe that Batch Mode CCE is able to provide an automated analysis of the model biases using a batch of samples, given that it can identify the spuriously correlated concept as one of the major causes over all model mistakes for the given class.
Figure [[c,, we provide the
Dist. the N=
a)
Mean Precision@K
1.00
b)
Thveshees
40-
Ihrestate
0.75
Random
Control
0.50
CCE
Univariate
Dist of the vol accurccess N=50
Dict. the val. accuratees M=100 40-
0.25
0.00
10-
2
8
9
10
0.0
0.2
0.4
0.6
0.
1.0
Figure 14: (a) We observe how the performance changes with respect to K for the metric Precision (b) We provide the distribution of the validation accuracies for each concept, as we vary the number of samples we use to learn the Concept Activation Vectors.
21

---

Under review as a conference paper at ICLR 2022
A.8 DETAILS ON PRECISION @ K
In Fig 14aa) we observe how the reported Precision@K metric changes for various values of K. For K < 3 the performance is relatively lower. Empirically, observe that this mostly due co-occuring concepts, e.g. for the Dog(Snow) case, sometimes the first two concepts can turn out be muzzle dog. Similarly, for the Dog(Car) case. concepts like Bus Truck can turn out to be the Top-2 concepts in the evaluation. We choose K = 3 since it is less noisy due to the explained reasons, and also more comprehensible in terms of providing the user a small number of concepts that they can easily work with.
B CONCEPTS
In Table we list the 170 concepts that we considered, along with their validation accuracies. We kept 168 concepts which had a validation accuracy of 0.7 or greater (bolded). We choose the threshold 0.7 since it works well empirically. For a more detailed analysis, in Fig 14[b) we provide the distribution of validation accuracies for concepts, as we vary the number of samples we use to learn the Concept Activation Vectors. As increase the number of samples, the validation accuracies move beyond the threshold we use (0.7). When we use 100 samples per concept, most of the concepts move beyond this value.
22

---

Under review as a conference paper at ICLR 2022
loudspeaker(0.66)
microwave(1.82)
arm(0.86)
exhaust h0o9().92)
airplane(0.96)
pedestal(0.78)
(ack(0.72)
mouse(0.68)
glas)(0.84)
polka dots(1.00)
mouth(0.80))
(0y
inside (0m().
(ir9().94)
bedclothes( (0.82)
paper(0.82)
blind
brick(0.84)
stairs(0.86)
(ountertop(0.92)
base(0.81)
person(0.94)
bluenesss(.84)
bathroom s(0.84)
pane(0.96)
motorbikee1.00)
hair(0.84)
pow((.84)
candlestick(0.88)
outside (0.92)
ceiling(0.88)
light(0.88)
street s(0.87)
column(0.82)
door frame(0.96)
granite(0.90)
(0w(0.96)
Sand(0.94)
bottle(0.84)
(0.(0.88)
pla9e(0.98)
double d09r(0.922)
pillow(0.78)
plant(0.88)
doorframe (1.66)
eyebrow(O... (0.88)
flower(0.90)
horse(0.99)
toilet(0.900
ceramic(0.86)
greenness (0.900)
back pillow(0.86)
drawer(0.88) (0.86
coach(0.92)
(0.al(0.84)
lid(0.90)
bannister(O.866
handle bar(0.78)
fan(0.79)
bush(0.92)
blo9chy(0.97)
fireplace(0.96)
bowl(0.80)
nose(0.80)
leg(0.80)
d0or(0.82)
(tripes(0.91)
apron(0.72))
oven(0.80)
pack(0.84)
body(0.86)
foot((.80)
framm)0.74)
dining room s(0.88)
board(0.78)
bridge(0.82)
sofa(0.78)
bedroom s(0.86)
(0.9))
blurriness(0.95)
(ootboard(0.8))
leather(0.86)
(0.9())90))
fluorescent(0.83
tree(0.98)
knob(0.89)
headlight(0.91)
blackness<0.911
house(0.82)
jar(0.94)
mirror(0.94)
pipe(0.82)
bathtub(0.96)
flag(0.70)
refrigerator(1.82)
curtain(0.92)
(0.80)
coffee table(0.92)
(ie9d(0.90)
chandelier(0.94)
cap(0.82)
hill(0.92)
ashcan(0.58)
path(0.94))
counter((.82)
cardboard
desk(0.84)
balcony(0.88)
box(0.77)
napkin(0.79)
ear(0.94)
(0.6()
manhole(0.82)
chest of drawers(0.88)
fence(0.88)
building(O..) (1.9))
figurine(0.77)
lamp
basket(0.55) (0.80)
eye(0.90)
ground(0.86)
car(0.90)
cat(1.00)
bookcase(0.94)
palm(0.99)
water(0..88)
bus(0.96)
laminate(0.82)
handle(0.78)
painted(0.88)
dog(0.98)
bumper(0.73)
concrete(0.82)
Awning(0.66)
clock(0.90)
faucet(0.89)
headboard(0.90)
canopy
bucket(0.74)
drinking glass(0.72)
armchair(0.84)
minibikee(0.66)
carpet(0.78)
cabinet(0.76)
(0.70)
earth(0.88)
neck(0.84)
can(0.89)
bicycle(0.96)
bag(0.70)
chain w0.el(00.8)
boak().72)
mountain(0.94)
redness(0.96)
air conditioner(O.85)
chair(0.86)
engine((.90)
painting(O.80)
grass(0.92)
(0.8()
pillar(0.84)
chimney(0.73
(l
fabric(0.74)
computer(0.866
flowerpot(0.78)
muzzle(0.94)
bench(0.72)
ottoman(0.28) (0.80)
cushion(0.84)
Table 5: List of concepts and validation accuracies for SVMs, for ResNet18 pretrained on ImageNet.


Reviews:
[{"review_id": "mxaqIzKgU5", "reviewer": "Reviewer_GEvb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n - Well written and clearly structured\n - Interesting and promising approach to compute counterfactuals in the domain of concepts\n\nCons:\n - Evaluation was done manually - i.e. manual inspection of samples. I understand that an automatic evaluation is challenging and I also appreciate that the authors are aware if this and suggest an user study - however, still I think that this manual inspection as an evaluation is a major limitation of this work. I would be more convinced if the authors would have done the evaluation as a proper user study.\n - While the examples in the paper look convincing, I was wondering whether it is possible to come up with formal guarantees - i.e. are there any formal guarantees that the final explanation is valid? The authors added some validity constraints but to me they look like a heuristic without any formal guarantees - I encourage the authors to comment on this.", "summary_of_the_paper": "The authors propose a method for computing concept based counterfactual explanations - i.e. using the presence or absence of concepts (human understandable) for explaining the models prediction. In the paper they focus on image classifiers.", "summary_of_the_review": "Overall an okay paper with some very interesting ideas, although I have some concerns regarding the evaluation of the method (see main review).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jVDulfrvyv9", "reviewer": "Reviewer_KwpD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-  The problem of explaining a model's mistakes is an important one and the proposed method is clearly defined and well investigated\n with ablation studies.  \n-  An interesting range of tasks and concepts is considered and, assuming that the group of \"OOD images\" can be identified ahead of time, the empirical results are convincing.  \n-  Studying more real/applied datasets such as those in Section 4.4 is an import direction for work on explainability.  \n\nWeaknesses:\n-  The method from [1] warrants discussion and comparison.  It makes the same assumption as the proposed method (ie, that we have concept labels for each image), provides its own algorithm for finding which concepts a model is relying on, and, further, provides methods for fixing the model when spurious concepts are identified.  \n-  The experiments show that \"for OOD images that are misclassified, the proposed method correctly identifies the concept causing the misclassification.\"  Unfortunately, this is not particularly useful because defining an \"OOD image\" requires that we already what spurious concept the model is using, which defeats the point of the proposed method.  Running experiments that start by analyzing the entire set of misclassified images would be more meaningful.\n-  While studying explanations in the wild is commendable, these particular experiments are not particularly convincing. \n\t-  Dermatology:  First, none of these concepts are clinically relevant and, as a result, all explanations using them will indicate some sort of \"problem\".  Second, the explanations shown in Figure 4 are not particularly convincing because \"ashcan\" and \"water\" are frequently relevant concepts; it seems more likely that the explanation method/concept selection are incorrect than the model is genuinely learning to use the presence of \"ashcans\" to make predictions.  \n\t-  Cardiology:  This has a similar problem to the earlier experiment because most of the concepts indicate some sort of \"problem\" (only Cardiomegaly and Atelectasis do not).  More generally though, it seems unnecessary to use explainability methods to determine that a model trained on X-Rays taken from one angle won't work as well on X-rays taken from another angle.  \n\t-  Including more clinically relevant concepts or exploring datasets where the Broden concepts are relevant would address this issue.  \n\n[1] Singh, Krishna Kumar, et al. \"Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.", "summary_of_the_paper": "The paper proposes a Conceptual Counterfactual Explanation for explaining a model's mistakes.  It starts by learning a \"concept\" using a linear separator in the models representation space and a small amount of data annotated with that concept.  Then, it explains a model's mistake by finding a change in those concepts that changes the model's prediction, does not change too many concepts, and does not change the concepts in an unrealistic way.  The method is verified by showing that, for OOD test points where the model makes mistakes, this explanation can identify the concept responsible for those mistakes.  ", "summary_of_the_review": "While the proposed method is interesting, the supporting experiments need fundamental changes in order to be impactful.  Specifically, the analysis should not start with the assumption that a set of \"OOD images\" has been identified and the experiments should include more concepts that are relevant to the predictive task.  \n\n# Update after discussion with the authors\n\nI'd like to thank the authors for a very productive discussion and for taking the time to run additional experiments to address my concerns.  At this point, I think that the setup, results, and analysis for the settings where the model is trained on a deliberately biased distribution are convincing.  As a result, I've increased my score to a 5.  \n\nHowever, I am still concerned that using CCE in a less controlled setting will be difficult due to \"false positives\" for relevant concepts:\n-  These appear in Figure 4 in the form of \"water\" and \"ashcan\".  In this setting, it's easy to label these as false positives because they are unrelated to the task;  but this seems like more of a reason to exclude those concepts than an actual fix to the problem.  \n-  As an example of where it is more difficult to determine if something is a \"false positive\" consider Figure 6.  While it seems likely that concepts like \"building\" and \"flag\" are false positives, its much less clear  if \"cow\" is.  If we consider adding a literal \"cow\" to the image, it seems unlikely that that would help the model detect the zebra.  But with a more subjective interpretation of emphasizing the \"cow shape\" of the zebra, then it seems more likely.  Either way, this seems like something that should be tested directly.  \n\nOne way to address this would be to:\n-  Train a model on the entirety of the MetaDataset\n-   Find misclassified points and run the batch CCE on them\n-   For each of the top concepts, translate the explanation into a hypothesis about about the model's performance on real images (eg, snow is relevant for dog, so the model will perform better on images of dogs with snow)\n-  Measure how often those hypotheses are correct\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Sw4IeZKOZNl", "reviewer": "Reviewer_oufr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper is clearly written, and the figures are demonstrative and clear.\n\n- In section 4.1, the experiments to identify spurious correlations by CCE a good beginning, but the interventions on the MetaDataset images are large drastic effects.  I would lke to see CCE applied to the identificaiton of more subtle spurious correlations that are more realistic.  Perhaps CelebA could be a good starting point?\n  - e.g where more subtle latent factors explain the spurious correlation\n  - where label imbalances exacerbate spurious correlation by diminishing the ability to estimate concept vectors\n\n- The granny smith colour ablation test in section 4.2 a nice clear example of how the tool can clearly demonstrate a relationship for one given spurious correlation, but it does not tell me much about how robust it is to reporting spurious correlation CAVs at test time, which is the more important problem.\n\nWeaknesses:\n\n- I feel the authors underplay the complexity and investment to define a concept library, especially for specialize application areas.  In diagnostic imaging, prior taxonomies of concepts exist, but can be very difficult to identify in training images (cf. Oakden-Raynor et al.).  In other fields there may not even yet be generally agreed upon ontologies of concepts.  This drawback of the CAV method should be noted.\n\n- Figure 1c is not a particularly good example of explaining a model mistake.  While the *label* given in the image may be “Zebra”, there is clearly a crocodile visible in the image (left of the smaller, left-most zebra).  This image is indicative of a different problem, which is that many natural images are better described as multi-label, as they contain multiple distinct concepts or entities within them.\n\n\n- Towards the end of section 2 the authors claim \n  > \"we extend these perturbations to unstructured data (specifically natrual images), whereas existing counteractural explanation methods have been largely confined ot tabular datasets since the perurbations have consitted of changing values in the original low-level feature space\"\n  This is far from true.  See Singla et al 2019, Goyal et al. 2019, and many other papers that apply counterfactual explanation to latent spaces of models that generate natural images.  This is a young but growing subfield that should be cited to acknowledge their contributions.\n\n- The end of section 3.1 suggests concepts are validated by a hold out set, where those that do not meat a threshold on accuracy are discarded.  How was this threshold determined?  Is it affected by noisy data in the wild?  Is there some clear threshold in the distribution of accuracies over concepts?  And how is this affected by either noise or imbalance in classes of the data?  There are several such questions here that deserve more attention.\n\n- Minor point, but Algorithm 1 should specify that the terminating condition of the while loop is **while** *w not converged* **do**\n\n\n- Is the definition of the geometric margin ot the decision boundary in section 3.2 quite right?  It is defined as $d_i = \\mathbf{c}_{i}b_{L}(\\mathbf{x}_i)^T$.  This seems to use $i$ to index both concepts and datapoints in the training set for concepts.\n\nQuestions:\n\n- Section 3 defines the *bottleneck layer* as that layer which will be employed to act as feature vector for learning CAVs.  Does the choice of layer matter here?  Or is there any reason to deviate from the pre-activation features from the ultimate fully connected layer?\n\n- For Table 1: Is there a reason for choosing precision @3?  What does the distribution of the precision @K look like for various K?  Is there a steep drop after 3, or is 3 motivated by the desire to provide a user with a more manageable number of possible concepts?\n", "summary_of_the_paper": "Explaining errors using semantically meaningful concepts is a critical tool for improving the performance, robustness, and trustworthiness of machine learning models deployed in application areas everywbere.  While there have been several proposed lines of work in this area, none is yet accepted as standard practice.  The authors present a method combining two prior methods in explainability: counterfactual explanation and concept activation vectors, and meld them together as coneptual counterfactual explanations.  \n\nThey describe the training procedure, and show some examples on both ImageNet and real diagnostic imaging applications.", "summary_of_the_review": "While this paper combines two complementary techniques in counterfactual explanation and CAVs, I would like to see CCE compared to other counterfactual explanation methods that are trained on natural images and provide semantic highlights or perturbations.  I also think that several important details have been omitted from the discussion (see Weaknesses section), and that in its present form, the work is not ready for publication.\n\nUpdate: In response to the authors' extensive revisions to address my concerns (and those of the other reviewers), I have increased my score.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H7oKhD6TI8", "reviewer": "Reviewer_nQZu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Novel combination of concept activations and counterfactual explanations\nExcellent writeup of approach and motivation\n\n- Interesting controlled evaluation of approach that demonstrates effectiveness. \n\nWeaknesses:\n\n- Approach cannot be generalized to obtain explanations for a true test image since approach requires label y to obtain perturbation. How is it possible to make the approach work without the label y?\n\n- Lack of baselines in controlled evaluation (sections 4.1 to 4.3): consider the following baseline, take the test example, find the closest correctly classified example to it in a validation set (that has same distribution as training data), and subtract the concept vectors for the test image - closest image in validation. Then show the top 3 concepts of the difference.  \n\n- Lack of ability to visualize perturbed test examples. How is it possible to visually see what the test image with the perturbation looks like? Perhaps this is easy, but would appreciate it if the authors can elaborate on how to do this.\n\n- Lack of discussion surrounding the requirement of being able to collect images in the domain of interest that describe each of the concepts independently of the trained data. For example, in the dog and snow example, to detect the concept of snow, the SVM must have seen images of snow with and without dogs. \n\n- In controlled evaluation, the cause of the mistake can always be found in the concepts. What happens if the mistake cannot be attributed to a concept? Does the method say that none of the concepts explain the mistake or does it falsely attribute it to something else? One can do this evaluation by removing e.g. “snow” from the concept bank and running the same experiment.\n\n- Lack of objective evaluation metric and baseline in the “in the wild” evaluation: I understand the lack of baselines in the controlled study, but for the evaluation with clinicians, it would have been helpful to see if with other baselines (maybe something like GradCAM?) the clinicians can try to identify on their own the reasons they think the model got it incorrectly and see if they match with the concepts retrieved.\n\n- The clinical concepts especially for the chest x-ray use case are not comprehensive enough to be sufficiently descriptive. I don’t believe in it’s current form, the in the wild evaluation adds value to the paper.\n\n\nAfter author response:\n\n- I have increased my score as the authors have raised some of my concerns, particularly on when the concept is not in the library", "summary_of_the_paper": "Goal: provide a human readable explanation to why a given classifier made a mistake on a given example.\n\nApproach: \n\nStep1: For the given input domain (here images), first obtain a set of concepts that are human interpretable. These concepts each describe a particular aspect of an image, and a human is expected to be able to look at image and say whether a particular concept applies or does not apply to an image (on scale of [0,1])\n\nStep2: learn a map from the input domain to the concept space using an SVM\n\nStep3: generate counterfactual perturbation of example that ensures that :1) classifier is now correct on example, 2) perturbation is valid (i.e. realistic) and 3) perturbation changes few concepts. The perturbation is the addition of a linear combination of concepts by different amounts.  Key to this is that generating the perturbation requires the true label and query access to the classifier.\n\nStep4: present the user the weight vector of the concept perturbations.\n\nValidation: two ways to evaluate, one controlled and one in the wild.\n\nControlled evaluation: using dataset of animal images in different settings, confound the class label, e.g. dog, with background, e.g. snow, then ask if approach can recover the confounding factor, here snow. They show that on a median basis, the method finds the right confounding factor in the top 2 spots.\n\nIn the wild evaluation: on dermatology and chest x-ray datasets, they studied the mistakes of of a  trained model. The evaluation of their findings is based on discussion with clinicians.\n", "summary_of_the_review": "The paper provides a novel way to explain a classifier's mistakes on a test example when given it’s label in terms of a known set of concepts. The paper is very well written and the controlled evaluation demonstrates that it is effective when the concepts can describe the mistakes perfectly. However, the evaluation lacks baselines and results when some of the assumptions fail (concepts don’t describe mistakes). Furthermore, it is hard to understand how the “in-the-wild” evaluation shows the effectiveness of the method. Finally, the method requires knowing the label of the example and thus is more of a debugging approach rather than an explanation method.\n\nIn its current form, I think the paper is borderline accept*, however, if the authors can provide answers to some of the weaknesses I presented, I am more than willing to move the score up to an accept as I think the ideas and the problem are extremely interesting. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method called CCE for explaining mistakes by DNNs on image classification. It is built on top of two prior ideas: counterfactual explanations and concept activation vectors. CCE explains a mistake by assigning scores to a shot list of concepts, where a large positive score means that adding that concept to the image will increase the probability of correctly classifying the image, as will removing or reducing a concept with a large negative score.
The strengths of the paper include novel combination of previous work, clear presentation, interesting experiments, convincing results on controlled settings.  The weaknesses include the lack of results on less controlled settings, the lack of more meaningful spurious correlations in the medical examples, and the lack of user studies.
Although the reviewers have shown interests in this paper, they clearly do not support the paper strongly. In addition, the authors have missed the following paper that also combines counterfactual explanations and concept activation vectors:
Akula, Arjun, Shuai Wang, and Song-Chun Zhu. “Cocox: Generating conceptual and counterfactual explanations via fault-lines.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 03. 2020.
Decision: Reject