Meaningfully Explaining Model Mistakes Using Conceptual Counterfactuals

1 INTRODUCTION . People who use machine learning ( ML ) models often need to understand why a trained model is making a particular mistake . For example , upon seeing a model misclassify an image , an ML practitioner may ask questions such as : Was this kind of image underrepresented in my training distribution ? Am I preprocessing the image correctly ? Has my model learned a spurious correlation or bias that is hindering generalization ? Answering this question correctly affects the usability of a model and can help make the model more robust . Consider a motivating example : Example 1 : Usability of a Pretrained Model . A pathologist downloads a pretrained model to classify histopathology images . Despite a high reported accuracy , he finds the model performing poorly on his images . He investigates why that is the case , finding that the hues in his images are different than in the original training data . Realizing the issue , he is able to transform his own images with some preprocessing , matching the training distribution and improving the model ’ s performance . In the above example , we have a domain shift occurring between training and test time , which degrades the model ’ s performance ( Subbaswamy et al. , 2019 ; Koh et al. , 2020 ) . By explaining the cause of the domain shift , we are able to easily fix the model ’ s predictions . On the other hand , if the domain shift is due to more complex spurious correlations the model has learned , it might need to be completely retrained before it can be used . During development , identifying and explaining a model ’ s failure points can also make it more robust ( Abid et al. , 2019 ; Kiela et al. , 2021 ) , as in the following example : Example 2 : Discovering Biases During Development . A dermatologist trains a machine learning classifier to classify skin diseases from skin images collected from patients at her hospital . She shares her trained model with a colleague at a different hospital , who reports that the model makes many mistakes with his own patients . She investigates why the model is making mistakes , realizing that patients at her colleague ’ s hospital have different skin colors and ages . Answering this question allows her to not only know her own model ’ s biases but also guides her to expand her training set with the right kind of data to build a more robust model . Explaining a model ’ s mistakes is also useful in other settings where data distributions may change , such as concept drift for deployed machine learning models ( Lu et al. , 2018 ) . Despite its usefulness , explaining a model ’ s performance drop is often an ad hoc process that involves manually looking at the model ’ s mistakes on many test samples and guessing at the underlying reasons for those incorrect predictions . We present counterfactual conceptual explanations ( CCE ) , a systematic method for explaining model mistakes in terms of meaningful human-understandable concepts , such as those in the examples above ( e.g . hues ) . In addition , CCE was developed with the following criteria : • No training data or retraining : CCE does not require access to the training data or model retraining . • Only needs the model : CCE only needs white-box access to the model . The user can use any dataset to learn desired concepts . • High-level explanations : CCE provides high-level explanations of model mistakes using concepts that are easy for users to understand . counterfactual . Meaning that we would like to generate a perturbation in the embedding space that would correct the model prediction , using a weighted sum of our concept bank . ( c ) Our method assigns a score to the set of concepts . A large positive score means that adding that concept to the image ( e.g . stripes ) will increase the probability of correctly classifying the image , as will removing or reducing a concept with a large negative score ( e.g . polka dots or dogness ) . Fig . 1 provides an overview of our CCE method . To summarize our contributions , we develop a novel method , CCE , that can systematically explain model mistakes in terms of high-level concepts that are easy for users to understand . We show that CCE correctly identifies spurious correlations in model training , and we quantitatively validate CCE across different experiments over natural and clinical images . We have released all of the code and data needed for our method in a public repository : https : //github.com/conceptualcounterfactuals/iclr2022 . 2 RELATED WORKS . CCE is inspired by prior efforts to explain machine learning models ’ predictions . Two ideas are particularly relevant : counterfactual explanations and concept activation vectors . Here , we provide an overview of these methods and discuss the differences between them and our proposed approach . Counterfactual Explanations Counterfactual explanations ( Verma et al . ( 2020 ) provides a comprehensive review ) are a class of model interpretation methods that seek to answer : what perturbations to the input are needed for a model ’ s prediction to change in a particular way ? A large number of counterfactual explanation methods have been proposed with various desiderata such as sparse perturbations ( Wachter et al. , 2017 ) , perturbations that remain close to the data manifold ( Dhurandhar et al. , 2018 ) , and causality ( Mahajan et al. , 2019 ) . What these methods share in common is that they provide an interpretable link between model predictions and perturbations to input features . There are several works focusing on counterfactual explanations for image data ( Goyal et al. , 2019 ; Singla et al. , 2020 ) and many of these methods use distractor images or modify input images to explain the model behavior . However , perturbations or saliency maps in the input space are shown to be tricky to interpret and be adopted by users ( Alqaraawi et al. , 2020 ; Adebayo et al. , 2018 ) . To compute CCE , we use a similar approach , perturbing input data to change its predictions , but for a complementary goal : to understand the limitations and biases of our model and its training data , rather than to change a specific prediction . Further improving upon existing work , we do this by directly using human-interpretable concepts , without having access to training data . Concept Activation Vectors To explain an image classification model ’ s mistakes in a useful way , we need to operate not with low-level features , but with more meaningful concepts . Concept activation vectors ( CAVs ) ( Kim et al. , 2018 ) are a powerful method to understand the internals of a network in human-interpretable concepts . CAVs are linear classifiers trained in the bottleneck layers of a network and correspond to concepts that are usually user-defined or automatically discovered from data ( Ghorbani et al. , 2019 ) . Unlike many prior interpretation methods , explanations produced by CAVs are in terms of high-level , human-understandable concepts rather than individual pixels ( Simonyan et al. , 2013 ; Zhou et al. , 2016 ) or training samples ( Koh & Liang , 2017 ) . In previous literature , CAVs have been used to test the association between a concept and the model ’ s prediction on a class of training data ( Kim et al. , 2018 ) as well as provide visual explanations for predictions ( Zhou et al. , 2018 ) . Our approach extends CAVs by showing that perturbations along the CAV can be used to change a model ’ s prediction on a specific test sample , e.g . to correct a mistaken prediction , and thereby be used in a similar manner to counterfactuals . Since samples ( even of the same class ) can be misclassified for different reasons ( see Fig . 6 ) , our approach allows a more contextual understanding of a model ’ s behavior and mistakes . Bias/Error Detection and Robustness Understanding a model ’ s behavior and explaining its mistakes is critical for building more robust and less biased models , as we have discussed in Section 1 . While some other methods , such as FairML ( Adebayo & Kagal , 2016 ) and the What-If Tool ( Wexler et al. , 2020 ) , can also be used to discover biases and failure points in models , they are typically limited to tabular datasets where sensitive features are explicitly designated , unlike our approach which is designed for unstructured data and works with a more flexible and diverse set of concepts . In a similar spirit to our work , Singh et al . ( 2020 ) aim to identify and mitigate contextual bias , however , they assume access to the whole training setup and data ; which is a limiting factor in practical scenarios . Our approach can also be used to explain a model ’ s performance degradation with data drift . It is complementary to methods for out-of-distribution sample detection ( Hendrycks & Gimpel , 2016 ) and black-box shift detection ( Lipton et al. , 2018 ) , which can detect data drift , but can not explain the underlying reasons . There are also algorithmic approaches to improve model robustness with data drift ( Raghunathan et al. , 2018 ; Nestor et al. , 2019 ) . Although these methods can provide some guarantees around robustness for unstructured data , they are in terms of low-level perturbations and do not hold with more conceptual and natural changes in data distributions . There are several recent methods for detecting clusters of mistakes made by a model ( Kim et al. , 2019 ; d ’ Eon et al. , 2021 ) , which is useful for detecting disparity in the model , for example . CCE differs in that instead of finding mistake clusters , CCE explains errors from just one ( or a few ) image . 3 METHODS . In this section , we detail the key steps of our method . Let us define basic notations : let f : Rd → Rk be a deep neural network , let x ∈ Rd be a test sample belonging to class y ∈ { 1 , . . . k } . We assume that that the model misclassifies x , meaning that argmaxi f ( x ) i 6= y or simply that the model ’ s confidence in class y , f ( x ) y , is lower than desired . Let m be the dimensionality of the bottleneck layer L , and bL : Rd → Rm be the “ bottom ” of the network , which maps samples from the input space to the bottleneck layer , and tL : Rm → Rk be the “ top ” of the network , defined analogously . For readability , we will usually underline class names and italicize concepts throughout this paper . 3.1 LEARNING CONCEPTS . In generating conceptual explanations , the first step is to define a concept library : a set of humaninterpretable concepts C = { c1 , c2 , ... } that occur in the dataset . For each concept , we collect positive examples , Pci , that exhibit the concept , as well as negative examples , Nci , in which the concept is absent . Unless otherwise stated , we use Pci = Nci = 100 . These concepts can be defined by the researcher , by non-ML domain experts , or even learned automatically from the data ( Ghorbani et al. , 2019 ) . Since concepts are broadly reusable within a data domain , they can also be shared among researchers . For our experiments with natural images , we defined 170 general concepts that include ( a ) the presence of specific objects ( e.g . mirror , person ) , ( b ) settings ( e.g . street , snow ) ( c ) textures ( e.g . stripes , metal ) , and ( d ) image qualities ( e.g . blurriness , greenness ) . Many of our concepts were adapted from the BRODEN dataset of visual concepts ( Fong & Vedaldi , 2018 ) . Note that the data used to learn concepts can differ from the data used to train the ML model that we evaluate . After defining a concept library , we then learn an SVM and the corresponding CAV for each concept . We follow the same procedure as in ( Kim et al. , 2018 ) , for the penultimate layer of a ResNet18 pretrained in ImageNet . This step only needs to be done once for each model that we want to evaluate , and the CAVs can then be used to explain any number of misclassified samples . We refer the reader to Appendix A.1 for implementational details and Appendix B for a full list of concepts . We denote the vector normal to the classification hyperplane boundary as ci ( normalized , i.e . |ci| = 1 ) and the intercept of the SVM as φi . To measure whether concepts are successfully learned , we keep a hold-out validation set and measure the validation accuracy , disregarding concepts with accuracies below a threshold ( 0.7 in our experiments , which left us with 168 of the 170 concepts ) . We provide more details about the threshold in Appendix B .

Summary: The paper proposes a Conceptual Counterfactual Explanation for explaining a model's mistakes.  It starts by learning a "concept" using a linear separator in the models representation space and a small amount of data annotated with that concept.  Then, it explains a model's mistake by finding a change in those concepts that changes the model's prediction, does not change too many concepts, and does not change the concepts in an unrealistic way.  The method is verified by showing that, for OOD test points where the model makes mistakes, this explanation can identify the concept responsible for those mistakes.