Meaningfully Explaining Model Mistakes Using Conceptual Counterfactuals

1 INTRODUCTION . People who use machine learning ( ML ) models often need to understand why a trained model is making a particular mistake . For example , upon seeing a model misclassify an image , an ML practitioner may ask questions such as : Was this kind of image underrepresented in my training distribution ? Am I preprocessing the image correctly ? Has my model learned a spurious correlation or bias that is hindering generalization ? Answering this question correctly affects the usability of a model and can help make the model more robust . Consider a motivating example : Example 1 : Usability of a Pretrained Model . A pathologist downloads a pretrained model to classify histopathology images . Despite a high reported accuracy , he finds the model performing poorly on his images . He investigates why that is the case , finding that the hues in his images are different than in the original training data . Realizing the issue , he is able to transform his own images with some preprocessing , matching the training distribution and improving the model ’ s performance . In the above example , we have a domain shift occurring between training and test time , which degrades the model ’ s performance ( Subbaswamy et al. , 2019 ; Koh et al. , 2020 ) . By explaining the cause of the domain shift , we are able to easily fix the model ’ s predictions . On the other hand , if the domain shift is due to more complex spurious correlations the model has learned , it might need to be completely retrained before it can be used . During development , identifying and explaining a model ’ s failure points can also make it more robust ( Abid et al. , 2019 ; Kiela et al. , 2021 ) , as in the following example : Example 2 : Discovering Biases During Development . A dermatologist trains a machine learning classifier to classify skin diseases from skin images collected from patients at her hospital . She shares her trained model with a colleague at a different hospital , who reports that the model makes many mistakes with his own patients . She investigates why the model is making mistakes , realizing that patients at her colleague ’ s hospital have different skin colors and ages . Answering this question allows her to not only know her own model ’ s biases but also guides her to expand her training set with the right kind of data to build a more robust model . Explaining a model ’ s mistakes is also useful in other settings where data distributions may change , such as concept drift for deployed machine learning models ( Lu et al. , 2018 ) . Despite its usefulness , explaining a model ’ s performance drop is often an ad hoc process that involves manually looking at the model ’ s mistakes on many test samples and guessing at the underlying reasons for those incorrect predictions . We present counterfactual conceptual explanations ( CCE ) , a systematic method for explaining model mistakes in terms of meaningful human-understandable concepts , such as those in the examples above ( e.g . hues ) . In addition , CCE was developed with the following criteria : • No training data or retraining : CCE does not require access to the training data or model retraining . • Only needs the model : CCE only needs white-box access to the model . The user can use any dataset to learn desired concepts . • High-level explanations : CCE provides high-level explanations of model mistakes using concepts that are easy for users to understand . counterfactual . Meaning that we would like to generate a perturbation in the embedding space that would correct the model prediction , using a weighted sum of our concept bank . ( c ) Our method assigns a score to the set of concepts . A large positive score means that adding that concept to the image ( e.g . stripes ) will increase the probability of correctly classifying the image , as will removing or reducing a concept with a large negative score ( e.g . polka dots or dogness ) . Fig . 1 provides an overview of our CCE method . To summarize our contributions , we develop a novel method , CCE , that can systematically explain model mistakes in terms of high-level concepts that are easy for users to understand . We show that CCE correctly identifies spurious correlations in model training , and we quantitatively validate CCE across different experiments over natural and clinical images . We have released all of the code and data needed for our method in a public repository : https : //github.com/conceptualcounterfactuals/iclr2022 . 2 RELATED WORKS . CCE is inspired by prior efforts to explain machine learning models ’ predictions . Two ideas are particularly relevant : counterfactual explanations and concept activation vectors . Here , we provide an overview of these methods and discuss the differences between them and our proposed approach . Counterfactual Explanations Counterfactual explanations ( Verma et al . ( 2020 ) provides a comprehensive review ) are a class of model interpretation methods that seek to answer : what perturbations to the input are needed for a model ’ s prediction to change in a particular way ? A large number of counterfactual explanation methods have been proposed with various desiderata such as sparse perturbations ( Wachter et al. , 2017 ) , perturbations that remain close to the data manifold ( Dhurandhar et al. , 2018 ) , and causality ( Mahajan et al. , 2019 ) . What these methods share in common is that they provide an interpretable link between model predictions and perturbations to input features . There are several works focusing on counterfactual explanations for image data ( Goyal et al. , 2019 ; Singla et al. , 2020 ) and many of these methods use distractor images or modify input images to explain the model behavior . However , perturbations or saliency maps in the input space are shown to be tricky to interpret and be adopted by users ( Alqaraawi et al. , 2020 ; Adebayo et al. , 2018 ) . To compute CCE , we use a similar approach , perturbing input data to change its predictions , but for a complementary goal : to understand the limitations and biases of our model and its training data , rather than to change a specific prediction . Further improving upon existing work , we do this by directly using human-interpretable concepts , without having access to training data . Concept Activation Vectors To explain an image classification model ’ s mistakes in a useful way , we need to operate not with low-level features , but with more meaningful concepts . Concept activation vectors ( CAVs ) ( Kim et al. , 2018 ) are a powerful method to understand the internals of a network in human-interpretable concepts . CAVs are linear classifiers trained in the bottleneck layers of a network and correspond to concepts that are usually user-defined or automatically discovered from data ( Ghorbani et al. , 2019 ) . Unlike many prior interpretation methods , explanations produced by CAVs are in terms of high-level , human-understandable concepts rather than individual pixels ( Simonyan et al. , 2013 ; Zhou et al. , 2016 ) or training samples ( Koh & Liang , 2017 ) . In previous literature , CAVs have been used to test the association between a concept and the model ’ s prediction on a class of training data ( Kim et al. , 2018 ) as well as provide visual explanations for predictions ( Zhou et al. , 2018 ) . Our approach extends CAVs by showing that perturbations along the CAV can be used to change a model ’ s prediction on a specific test sample , e.g . to correct a mistaken prediction , and thereby be used in a similar manner to counterfactuals . Since samples ( even of the same class ) can be misclassified for different reasons ( see Fig . 6 ) , our approach allows a more contextual understanding of a model ’ s behavior and mistakes . Bias/Error Detection and Robustness Understanding a model ’ s behavior and explaining its mistakes is critical for building more robust and less biased models , as we have discussed in Section 1 . While some other methods , such as FairML ( Adebayo & Kagal , 2016 ) and the What-If Tool ( Wexler et al. , 2020 ) , can also be used to discover biases and failure points in models , they are typically limited to tabular datasets where sensitive features are explicitly designated , unlike our approach which is designed for unstructured data and works with a more flexible and diverse set of concepts . In a similar spirit to our work , Singh et al . ( 2020 ) aim to identify and mitigate contextual bias , however , they assume access to the whole training setup and data ; which is a limiting factor in practical scenarios . Our approach can also be used to explain a model ’ s performance degradation with data drift . It is complementary to methods for out-of-distribution sample detection ( Hendrycks & Gimpel , 2016 ) and black-box shift detection ( Lipton et al. , 2018 ) , which can detect data drift , but can not explain the underlying reasons . There are also algorithmic approaches to improve model robustness with data drift ( Raghunathan et al. , 2018 ; Nestor et al. , 2019 ) . Although these methods can provide some guarantees around robustness for unstructured data , they are in terms of low-level perturbations and do not hold with more conceptual and natural changes in data distributions . There are several recent methods for detecting clusters of mistakes made by a model ( Kim et al. , 2019 ; d ’ Eon et al. , 2021 ) , which is useful for detecting disparity in the model , for example . CCE differs in that instead of finding mistake clusters , CCE explains errors from just one ( or a few ) image . 3 METHODS . In this section , we detail the key steps of our method . Let us define basic notations : let f : Rd → Rk be a deep neural network , let x ∈ Rd be a test sample belonging to class y ∈ { 1 , . . . k } . We assume that that the model misclassifies x , meaning that argmaxi f ( x ) i 6= y or simply that the model ’ s confidence in class y , f ( x ) y , is lower than desired . Let m be the dimensionality of the bottleneck layer L , and bL : Rd → Rm be the “ bottom ” of the network , which maps samples from the input space to the bottleneck layer , and tL : Rm → Rk be the “ top ” of the network , defined analogously . For readability , we will usually underline class names and italicize concepts throughout this paper . 3.1 LEARNING CONCEPTS . In generating conceptual explanations , the first step is to define a concept library : a set of humaninterpretable concepts C = { c1 , c2 , ... } that occur in the dataset . For each concept , we collect positive examples , Pci , that exhibit the concept , as well as negative examples , Nci , in which the concept is absent . Unless otherwise stated , we use Pci = Nci = 100 . These concepts can be defined by the researcher , by non-ML domain experts , or even learned automatically from the data ( Ghorbani et al. , 2019 ) . Since concepts are broadly reusable within a data domain , they can also be shared among researchers . For our experiments with natural images , we defined 170 general concepts that include ( a ) the presence of specific objects ( e.g . mirror , person ) , ( b ) settings ( e.g . street , snow ) ( c ) textures ( e.g . stripes , metal ) , and ( d ) image qualities ( e.g . blurriness , greenness ) . Many of our concepts were adapted from the BRODEN dataset of visual concepts ( Fong & Vedaldi , 2018 ) . Note that the data used to learn concepts can differ from the data used to train the ML model that we evaluate . After defining a concept library , we then learn an SVM and the corresponding CAV for each concept . We follow the same procedure as in ( Kim et al. , 2018 ) , for the penultimate layer of a ResNet18 pretrained in ImageNet . This step only needs to be done once for each model that we want to evaluate , and the CAVs can then be used to explain any number of misclassified samples . We refer the reader to Appendix A.1 for implementational details and Appendix B for a full list of concepts . We denote the vector normal to the classification hyperplane boundary as ci ( normalized , i.e . |ci| = 1 ) and the intercept of the SVM as φi . To measure whether concepts are successfully learned , we keep a hold-out validation set and measure the validation accuracy , disregarding concepts with accuracies below a threshold ( 0.7 in our experiments , which left us with 168 of the 170 concepts ) . We provide more details about the threshold in Appendix B .

Summary: The authors propose a method for computing concept based counterfactual explanations - i.e. using the presence or absence of concepts (human understandable) for explaining the models prediction. In the paper they focus on image classifiers.