Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation

Ascertaining that a deep network does not rely on an unknown spurious signal as basis for its output , prior to deployment , is crucial in high stakes settings like healthcare . While many post hoc explanation methods have been shown to be useful for some end tasks , theoretical and empirical evidence has also accumulated that show that these methods may not be faithful or useful . This leaves little guidance for a practitioner or a researcher using these methods in their decision process . To address this gap , we investigate whether three classes of post hoc explanations–feature attribution , concept activation , and training point ranking–can alert a practitioner to a model ’ s reliance on unknown spurious signals . We test them in two medical domains with plausible spurious signals . In a broad experimental sweep across datasets , models , and spurious signals , we find that the post hoc explanations tested can be used to identify a model ’ s reliance on a visible spurious signal provided the spurious signal is known ahead of time by the practitioner using the explanation method . Otherwise , a search over possible spurious signals and available data is required . This finding casts doubt on the utility of these approaches , in the hands of a practitioner , for detecting a model ’ s reliance on spurious signals . It is hard to find a needle in a haystack , it is much harder if you haven ’ t seen a needle before ( Pearl ) . —Judea Pearl 1 INTRODUCTION . A challenge that precludes the deployment of modern deep neural networks ( DNN ) in high stakes domains is their tendency to latch onto ‘ spurious signals ’ —shortcuts—in the training data ( Geirhos et al. , 2020 ) . For example , Badgeley et al . ( 2019 ) showed that an Inception-V3 model trained to detect hip fracture relied on the scanner type for its classification decision . Deep learning models easily base output predictions on object backgrounds ( Xiao et al. , 2020 ) , image texture ( Geirhos et al. , 2018 ) , and skin tone ( Stock and Cisse , 2018 ) . Post hoc model explanation methods—approaches that give insight into the associations that a model has learned—are increasingly used to determine whether a model relies on spurious signals . Ribeiro et al . ( 2016 ) used LIME to show an Inception-V3 model ’ s reliance on the snow background for identifying Wolves . Such demonstration and others ( Lapuschkin et al. , 2019 ; Rieger et al. , 2020 ) point to post hoc explanation methods as effective tools for the spurious signal detection task . However , these results conflict with evidence that indicates that practitioners ( and researchers ) struggle to use explanations to identify spurious signals ( Chen et al. , 2021 ; Chu et al. , 2020 ; Alqaraawi et al. , 2020 ; Adebayo et al. , 2020 ; Poursabzi-Sangdeh et al. , 2018 ) . We seek to resolve this conflict by answering the simple but important question : Can post hoc explanations help detect a model ’ s reliance on unknown spurious training signal ? Motivating Example . Consider a machine learning ( ML ) engineer whose job is to train DNN models to detect knee arthritis from radiographs . She—the engineer—is handed a trained ResNet-50 model , to be deployed in a hospital , that relies on a hospital tag in the radiographs to detect knee arthritis . She has no prior knowledge of the model ’ s reliance on the spurious tags . In this work , our key concern is whether the ML engineer can use post hoc explanations to identify that the model is defective . 1.1 OUR CONTRIBUTIONS . We address the motivating question above in a two-pronged manner . First , we develop an actionable methodology based on the ability to carefully craft datasets to induce spurious correlation in trained models . Second , we backup this experimental design with a human subject study . Taken together , the takeaway of the work is that : post hoc explanations can be used to identify a model ’ s reliance on a visible spurious signal , provided the signal is known ahead of time by the practitioner . While this conclusion may seem unsurprising , it has important implications for how post hoc explanation methods should be used effectively . Experimental Design & Performance Measures . We provide an end-to-end experimental design for assessing the effectiveness of an explanation method for detecting a model ’ s reliance on spurious training signals . We define a spurious score that helps quantify the strength of a model ’ s dependence on a training signal . Using carefully crafted semi-synthetic datasets , we are able to train models where the ground-truth expected explanation is known . Additionally , we develop 3 performance measures : i ) Known Spurious Signal Detection Measure ( K-SSD ) , ii ) Cause-for-Concern Measure ( CCM ) , and iii ) a False Alarm Measure ( FAM ) . These measures help characterize different notions of reliability for the spurious signal detection task . We instantiate the proposed design on 3 classes of post hoc explanation types—feature attribution , concept activation , and training point ranking—where we comprehensively assess the performance of these approaches across 3 datasets ( 2 medical tasks , and dog species classification task ) , and different model architectures . When the spurious signal is known , we find that the feature attribution methods tested , and the concept activation importance approach are able to detect visible spurious signals like a text tag and distinctive stripped patterns . However , we find these approaches less effective for non-pronounced signals like background blur . The false alarm measure further indicates that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals . The cause-for-concern measure quantifies the similarity between explanations of ‘ normal ’ inputs derived from spurious and normal models when the spurious signal is unknown . Across the settings considered , we find that the methods tested are unable to conclusively detect model reliance on unknown spurious signals . Blinded Study . The findings from our empirical assessment question the reliability of the methods tested ; however , it might not correlate with utility in the hands of practitioners . To address this issue , we conduct a user study where practitioners are randomly assigned to one of two groups : the first group is told explicitly of potential spurious signals , and the second is not . We consider three different kinds of explanation methods along with a control where only model predictions are shown . We find that when participants are not provided with prior knowledge of the spurious signal , none of the methods tested are effective , in the hands of the participants , for detecting model reliance on spurious signals . More surprisingly , even when the participants had prior knowledge of the spurious signal , we find evidence that only the concept activation approach , for visible spurious signals , is effective . These findings cast doubt on the reliability of current post hoc tools for spurious signal detection . Guidance . On the basis our analysis , we can provide the following guidance for using the approaches tested , in this work , for detecting model reliance on spurious signals when the signal of interest is visible : • Feature Attributions : to confirm that a model is relying on a ‘ visible ’ spurious signal , the practitioner needs to obtain attributions for inputs that contain the hypothesized spurious signal , and the attribution should be computed for the output class to which the spurious signal is aligned . • Concept Activation : the spurious concept should be known ahead of time , and tested against the output class to which the concept is aligned . • Training Point Ranking : an input that contains the hypothesized spurious signal of interest should be used at test-time in computing training point ranking . 1.2 RELATED WORK . This paper belongs to a line of work on assessing the effectiveness of post hoc explanations methods ( Alqaraawi et al. , 2020 ; Adebayo et al. , 2020 ; Chu et al. , 2020 ; Hooker et al. , 2019 ; Meng et al. , 2018 ; Poursabzi-Sangdeh et al. , 2018 ; Tomsett et al. , 2020 ) . Here we focus on directly relevant literature , and defer an extensive discussion of the literature to Section A in the Appendix . This work departs from previous work in two ways : 1 ) we focus exclusively on whether these explanations can be used by a practitioner ( or researcher ) to detect spurious signals that are unknown to her at test-time , and 2 ) we move beyond sole focus on the feature attribution setting to test concept activation and training point ranking methods . Han et al . ( 2020 ) and Adebayo et al . ( 2020 ) find that certain kinds of feature attributions and training point ranking via influence functions are able to detect a model ’ s reliance on spurious signals . However , in their setting , the spurious signal is known ahead of time . More recently , Zhou et al . ( 2021 ) conduct an extensive assessment of several feature attribution methods also under the spurious correlation setting , for visible and non-visible artifacts , and find that these class of methods are not effective for non-visible artifacts . Further , they also propose an experimental methodology for controlling model dependence on training set features , which allows them to quantify attribution effectiveness carefully and precisely . Overall , our findings align with theirs ; however , we focus , specifically , on the setting where the spurious signal is not known ahead of time . Further , we consider other kinds of post hoc explanation approaches in this work . Kim et al . ( 2021 ) conduct an assessment of several feature attribution methods using a synthetic evaluation framework where the ground-truth explanation is known for simple and complex reasoning tasks . They find that feature attribution methods often attribute irrelevant features even in simple settings , and these methods show high variability across data modalities and tasks . Plumb et al . ( 2021 ) introduce a method that searches over a dataset to identify important associations that a model might have learned , detect which of these associations are spurious , and propose a data augmentation procedure to overcome the reliance . Nguyen et al . ( 2021 ) conduct a large-scale user study on lay and expert users to assess the effectiveness of feature attribution methods on image tasks . They find that feature attributions are not more effective than showing end users nearest neighbor training points . 2 EXPERIMENTAL METHODOLOGY In this section , we setup our experimental methodology . We discuss quantitative analysis of post hoc explanations derived from models trained to rely on pre-defined spurious signals , and a blinded user study that measures the ability of users to use the post hoc explanation methods tested to detect model reliance on spurious signals . We discuss the types of spurious signals considered , define a spurious score that allows us to ascertain that a model indeed relies on a signal as basis of its classification decision , and layout performance measures that capture the reliability of the explanation methods . We conclude with an overview of the methods tested , datasets , and models . 2.1 EXPERIMENTAL DESIGN . Spurious Signals & Score . We consider a spurious signal to be input features that encode for the output but have ‘ no meaningful connection ’ to the ‘ data generating process ’ ( DGP ) of the task . A hospital tag present in a hand radiograph is not clinically relevant to the age of the patient . If the tag encodes for the output then it is a spurious signal . Domain expertise is ultimately required for adjudicating that a signal is spurious . We consider 3 ( 2 visible and 1 non-visible ) kinds of spurious signals ( See Figure 1 ) : i ) a localized tag ; ii ) a distinctive stripped pattern ; and iii ) Gaussian blur applied to the image background . The signals are all spatially localized , so we can easily obtain ground-truth expected explanations . To induce reliance on spurious signals , we train models on `` contaminated ” versions of the training set . Given input-label pairs , { ( xi , yi ) } ni , where xi ∈ X and yi ∈ Y , we can learn a classifier , fθ , via empirical risk minimization ( ERM ) that corresponds to minimizing a loss function , ` : arg minθ ∑n i=1 ` ( x i , yi ; θ ) . To contaminate the training set , we apply a spurious contamination function ( SCF ) to the training set ; SCF : X × Y × C → S , where C is the spurious signal set and S is the transformed set . An example of an SCF is a function that pastes an hospital tag onto the bone age radiographs of all pre-puberty individuals in the dataset . To derive models reliant on a spurious signal , ci ∈ C , we simply learn a new classifier via ERM on the modified dataset as follows : arg minθ ∑n i=1 ` ( SCF ( xi , yi , ci ) ) to obtain θspu . Contemporary evidence suggests that this approach produces models that easily latch onto the spurious signal ( Nagarajan et al. , 2020 ) . We focus on the classification setting , and restrict spurious signals to encode , only , for a single class—the spurious aligned class . We measure a model ’ s reliance on the spurious signal via a score . Definition 2.1 . ( Spurious Score ) . Given a spurious signal , ci , the index of its spurious aligned class , j ∈ [ k ] , a model , θspu : Rd → Rk , where arg max ( θspu ) indicates the classifier ’ s predicted class , we define the spurious score as : SCci , j ( θspu ) : = P { xi|θspu ( xi ) ! = j } [ arg max ( θspu ( SCF ( x i , yi , ci ) ) ) = j ] . Given an input that does not contain the spurious signal , and for which the model ’ s prediction is not the spurious aligned class , the model ’ s spurious score is the probability that the model assigns the input to the spurious aligned class if the spurious signal is added to the input . Model Conditions . We focus our analysis on two model conditions : i ) a ‘ normal model ’ , fnorm , for which we can rule out dependence on any of the spurious signals tested across all classes on the basis of the spurious score , and ii ) a ‘ spurious model ’ , fspu , for which one of the spurious signals encodes for a particular output class . We empirically estimate the spurious score and term models that have a score above 0.85 for any of the pre-defined signals ‘ spurious models ’ . We term a model ‘ normal ’ if the spurious score is below 0.1 across all classes and the 3 pre-defined spurious signals . Spurious Signal Detection Reliability Measures . Equipped with spurious ( fspu ) and normal ( fnorm ) models , we are now able to quantitatively assess the motivating question of this work . We do this by comparing explanations derived from spurious models , fspu , to those derived from normal models ( fnorm ) . We can partition the kinds of inputs used for deriving explanations into two : 1 ) spurious inputs ( xspu ) —inputs that include the spurious signal and 2 ) normal inputs ( xnorm ) —inputs do not not contain the spurious signal . Comparing the explanations produced by these two classes of inputs for normal and spurious models , we derive reliability performance measures . • Known Spurious Signal Detection Measure ( K-SSD ) - measures the similarity of explanations derived from spurious models on spurious inputs to the ground truth explanation . The ground truth explanation is one that only assigns relevance to the spurious signal as explanation of the output of a spurious model on a spurious input . K-SSD measures method reliability when the spurious signal is known . Given a similarity metric , Sd , then K-SSD corresponds to : Sd ( Efspu ( xspu ) , xgt ) ) ; where Efspu ( xspu ) are explanations derived from the spurious model for spurious inputs , and xgt is the ground truth explanation . The similarity function , Sd , depends on the type of explanation considered—we will make our choice of this function concrete shortly . • Cause-for-Concern Measure ( CCM ) - measures the similarity of explanations derived from spurious models for normal inputs to explanations derived from normal models for normal inputs : Sd ( Efspu ( xnorm ) , Efnorm ( xnorm ) ) . This measure simulates the setting where a practitioner does not know the spurious signal , and can only inspect explanations for inputs without the signal . If this measure is high , then it is unlikely that such a method alert a practitioner that a spurious model exhibits defects . • False Alarm Measure ( FAM ) - measures the similarity of explanations derived from normal models for spurious inputs to explanations derived from spurious models for spurious inputs : Sd ( Efnorm ( xspur ) , Efspu ( xspu ) ) . We also introduce a variant of this measure , FAM-GT , which measures the similarity of a explanations derived from normal models for spurious inputs to the ground truth explanation of a spurious model for that spurious input . If this measure is high , then that approach is more likely to signal to a practitioner that a model is relying on spurious signal when the model does not . Having defined the metrics above , it remains which similarity function to use . Computing Metrics for Feature Attribution . For feature attribution methods , we follow prior literature ( Adebayo et al. , 2020 ) and use the Structural Similarity Index ( SSIM ) . SSIM measures the visual similarity between two images , so we use this metric as the measure of similarity between two attribution maps . Concretely , given a set of normal inputs , we can obtain a corresponding spurious set of these inputs by applying the spurious contamination function , SCF to these inputs . Consequently , we can then compute the K-SSD , CCM , and FAM metrics given these two sets of inputs using the SSIM metric . Computing Metrics for Concept Activation . A concept activation method provides a relevance score for a user defined concept . Given a set of user defined concepts , one can estimate and rank each concept by the relevance score . We measure comparison between two concept rankings using a Kolmogorov-Smirnoff ( KS ) test comparing two distributions where the null hypothesis is that the two distributions are identical ; we set significance level to be 0.05 . Computing Metrics for Training Point Ranking . The training point ranking approach assigns a ‘ relevance ’ score to each training point based on influence of each training point on the test loss of a particular sample . Recently , Hanawa et al . ( 2020 ) introduced the Identical class metric ’ ( ICM ) , which is the fraction of the top training inputs , for a given test example , that belong to the same class as the true class of the test example in question . Here we also use the KS test to compare the ICM distributions for two different models and set the significance level to be 0.05 . Taken together , these measures provide a comprehensive overview of an explanation method ’ s performance for detecting spurious signals .

Summary: The authors present work that aims to test, whether post hoc explanation methods can detect unknown spurious signals. They perform an analysis on two different medical image datasets to which they introduce two different types of synthetic, spurious signals: pronounced spurious signals (stripes and hospital tags) and non-pronounced signals (blurs). When testing the ability of different post hoc explanation methods to identify spurious signals, they find that feature attribution and concept activation are able to identify pronounced, but not non-pronounced spurious signals that are known. When faced with unknown signals, none of the methods that were tested appears to be able to identify them. They also provide results from an empirical study suggesting that none of the methods allowed practitioners to identify unknown spurious signals and only concept activation appears to enable practitioners to identify known spurious signals.  The main contribution of this work are the empirical results and alerting researchers (and potentially practitioners) to very substantial problems with current post-hoc explanation. The technical contributions are negligible.