Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation

Ascertaining that a deep network does not rely on an unknown spurious signal as basis for its output , prior to deployment , is crucial in high stakes settings like healthcare . While many post hoc explanation methods have been shown to be useful for some end tasks , theoretical and empirical evidence has also accumulated that show that these methods may not be faithful or useful . This leaves little guidance for a practitioner or a researcher using these methods in their decision process . To address this gap , we investigate whether three classes of post hoc explanations–feature attribution , concept activation , and training point ranking–can alert a practitioner to a model ’ s reliance on unknown spurious signals . We test them in two medical domains with plausible spurious signals . In a broad experimental sweep across datasets , models , and spurious signals , we find that the post hoc explanations tested can be used to identify a model ’ s reliance on a visible spurious signal provided the spurious signal is known ahead of time by the practitioner using the explanation method . Otherwise , a search over possible spurious signals and available data is required . This finding casts doubt on the utility of these approaches , in the hands of a practitioner , for detecting a model ’ s reliance on spurious signals . It is hard to find a needle in a haystack , it is much harder if you haven ’ t seen a needle before ( Pearl ) . —Judea Pearl 1 INTRODUCTION . A challenge that precludes the deployment of modern deep neural networks ( DNN ) in high stakes domains is their tendency to latch onto ‘ spurious signals ’ —shortcuts—in the training data ( Geirhos et al. , 2020 ) . For example , Badgeley et al . ( 2019 ) showed that an Inception-V3 model trained to detect hip fracture relied on the scanner type for its classification decision . Deep learning models easily base output predictions on object backgrounds ( Xiao et al. , 2020 ) , image texture ( Geirhos et al. , 2018 ) , and skin tone ( Stock and Cisse , 2018 ) . Post hoc model explanation methods—approaches that give insight into the associations that a model has learned—are increasingly used to determine whether a model relies on spurious signals . Ribeiro et al . ( 2016 ) used LIME to show an Inception-V3 model ’ s reliance on the snow background for identifying Wolves . Such demonstration and others ( Lapuschkin et al. , 2019 ; Rieger et al. , 2020 ) point to post hoc explanation methods as effective tools for the spurious signal detection task . However , these results conflict with evidence that indicates that practitioners ( and researchers ) struggle to use explanations to identify spurious signals ( Chen et al. , 2021 ; Chu et al. , 2020 ; Alqaraawi et al. , 2020 ; Adebayo et al. , 2020 ; Poursabzi-Sangdeh et al. , 2018 ) . We seek to resolve this conflict by answering the simple but important question : Can post hoc explanations help detect a model ’ s reliance on unknown spurious training signal ? Motivating Example . Consider a machine learning ( ML ) engineer whose job is to train DNN models to detect knee arthritis from radiographs . She—the engineer—is handed a trained ResNet-50 model , to be deployed in a hospital , that relies on a hospital tag in the radiographs to detect knee arthritis . She has no prior knowledge of the model ’ s reliance on the spurious tags . In this work , our key concern is whether the ML engineer can use post hoc explanations to identify that the model is defective . 1.1 OUR CONTRIBUTIONS . We address the motivating question above in a two-pronged manner . First , we develop an actionable methodology based on the ability to carefully craft datasets to induce spurious correlation in trained models . Second , we backup this experimental design with a human subject study . Taken together , the takeaway of the work is that : post hoc explanations can be used to identify a model ’ s reliance on a visible spurious signal , provided the signal is known ahead of time by the practitioner . While this conclusion may seem unsurprising , it has important implications for how post hoc explanation methods should be used effectively . Experimental Design & Performance Measures . We provide an end-to-end experimental design for assessing the effectiveness of an explanation method for detecting a model ’ s reliance on spurious training signals . We define a spurious score that helps quantify the strength of a model ’ s dependence on a training signal . Using carefully crafted semi-synthetic datasets , we are able to train models where the ground-truth expected explanation is known . Additionally , we develop 3 performance measures : i ) Known Spurious Signal Detection Measure ( K-SSD ) , ii ) Cause-for-Concern Measure ( CCM ) , and iii ) a False Alarm Measure ( FAM ) . These measures help characterize different notions of reliability for the spurious signal detection task . We instantiate the proposed design on 3 classes of post hoc explanation types—feature attribution , concept activation , and training point ranking—where we comprehensively assess the performance of these approaches across 3 datasets ( 2 medical tasks , and dog species classification task ) , and different model architectures . When the spurious signal is known , we find that the feature attribution methods tested , and the concept activation importance approach are able to detect visible spurious signals like a text tag and distinctive stripped patterns . However , we find these approaches less effective for non-pronounced signals like background blur . The false alarm measure further indicates that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals . The cause-for-concern measure quantifies the similarity between explanations of ‘ normal ’ inputs derived from spurious and normal models when the spurious signal is unknown . Across the settings considered , we find that the methods tested are unable to conclusively detect model reliance on unknown spurious signals . Blinded Study . The findings from our empirical assessment question the reliability of the methods tested ; however , it might not correlate with utility in the hands of practitioners . To address this issue , we conduct a user study where practitioners are randomly assigned to one of two groups : the first group is told explicitly of potential spurious signals , and the second is not . We consider three different kinds of explanation methods along with a control where only model predictions are shown . We find that when participants are not provided with prior knowledge of the spurious signal , none of the methods tested are effective , in the hands of the participants , for detecting model reliance on spurious signals . More surprisingly , even when the participants had prior knowledge of the spurious signal , we find evidence that only the concept activation approach , for visible spurious signals , is effective . These findings cast doubt on the reliability of current post hoc tools for spurious signal detection . Guidance . On the basis our analysis , we can provide the following guidance for using the approaches tested , in this work , for detecting model reliance on spurious signals when the signal of interest is visible : • Feature Attributions : to confirm that a model is relying on a ‘ visible ’ spurious signal , the practitioner needs to obtain attributions for inputs that contain the hypothesized spurious signal , and the attribution should be computed for the output class to which the spurious signal is aligned . • Concept Activation : the spurious concept should be known ahead of time , and tested against the output class to which the concept is aligned . • Training Point Ranking : an input that contains the hypothesized spurious signal of interest should be used at test-time in computing training point ranking . 1.2 RELATED WORK . This paper belongs to a line of work on assessing the effectiveness of post hoc explanations methods ( Alqaraawi et al. , 2020 ; Adebayo et al. , 2020 ; Chu et al. , 2020 ; Hooker et al. , 2019 ; Meng et al. , 2018 ; Poursabzi-Sangdeh et al. , 2018 ; Tomsett et al. , 2020 ) . Here we focus on directly relevant literature , and defer an extensive discussion of the literature to Section A in the Appendix . This work departs from previous work in two ways : 1 ) we focus exclusively on whether these explanations can be used by a practitioner ( or researcher ) to detect spurious signals that are unknown to her at test-time , and 2 ) we move beyond sole focus on the feature attribution setting to test concept activation and training point ranking methods . Han et al . ( 2020 ) and Adebayo et al . ( 2020 ) find that certain kinds of feature attributions and training point ranking via influence functions are able to detect a model ’ s reliance on spurious signals . However , in their setting , the spurious signal is known ahead of time . More recently , Zhou et al . ( 2021 ) conduct an extensive assessment of several feature attribution methods also under the spurious correlation setting , for visible and non-visible artifacts , and find that these class of methods are not effective for non-visible artifacts . Further , they also propose an experimental methodology for controlling model dependence on training set features , which allows them to quantify attribution effectiveness carefully and precisely . Overall , our findings align with theirs ; however , we focus , specifically , on the setting where the spurious signal is not known ahead of time . Further , we consider other kinds of post hoc explanation approaches in this work . Kim et al . ( 2021 ) conduct an assessment of several feature attribution methods using a synthetic evaluation framework where the ground-truth explanation is known for simple and complex reasoning tasks . They find that feature attribution methods often attribute irrelevant features even in simple settings , and these methods show high variability across data modalities and tasks . Plumb et al . ( 2021 ) introduce a method that searches over a dataset to identify important associations that a model might have learned , detect which of these associations are spurious , and propose a data augmentation procedure to overcome the reliance . Nguyen et al . ( 2021 ) conduct a large-scale user study on lay and expert users to assess the effectiveness of feature attribution methods on image tasks . They find that feature attributions are not more effective than showing end users nearest neighbor training points . 2 EXPERIMENTAL METHODOLOGY In this section , we setup our experimental methodology . We discuss quantitative analysis of post hoc explanations derived from models trained to rely on pre-defined spurious signals , and a blinded user study that measures the ability of users to use the post hoc explanation methods tested to detect model reliance on spurious signals . We discuss the types of spurious signals considered , define a spurious score that allows us to ascertain that a model indeed relies on a signal as basis of its classification decision , and layout performance measures that capture the reliability of the explanation methods . We conclude with an overview of the methods tested , datasets , and models . 2.1 EXPERIMENTAL DESIGN . Spurious Signals & Score . We consider a spurious signal to be input features that encode for the output but have ‘ no meaningful connection ’ to the ‘ data generating process ’ ( DGP ) of the task . A hospital tag present in a hand radiograph is not clinically relevant to the age of the patient . If the tag encodes for the output then it is a spurious signal . Domain expertise is ultimately required for adjudicating that a signal is spurious . We consider 3 ( 2 visible and 1 non-visible ) kinds of spurious signals ( See Figure 1 ) : i ) a localized tag ; ii ) a distinctive stripped pattern ; and iii ) Gaussian blur applied to the image background . The signals are all spatially localized , so we can easily obtain ground-truth expected explanations . To induce reliance on spurious signals , we train models on `` contaminated ” versions of the training set . Given input-label pairs , { ( xi , yi ) } ni , where xi ∈ X and yi ∈ Y , we can learn a classifier , fθ , via empirical risk minimization ( ERM ) that corresponds to minimizing a loss function , ` : arg minθ ∑n i=1 ` ( x i , yi ; θ ) . To contaminate the training set , we apply a spurious contamination function ( SCF ) to the training set ; SCF : X × Y × C → S , where C is the spurious signal set and S is the transformed set . An example of an SCF is a function that pastes an hospital tag onto the bone age radiographs of all pre-puberty individuals in the dataset . To derive models reliant on a spurious signal , ci ∈ C , we simply learn a new classifier via ERM on the modified dataset as follows : arg minθ ∑n i=1 ` ( SCF ( xi , yi , ci ) ) to obtain θspu . Contemporary evidence suggests that this approach produces models that easily latch onto the spurious signal ( Nagarajan et al. , 2020 ) . We focus on the classification setting , and restrict spurious signals to encode , only , for a single class—the spurious aligned class . We measure a model ’ s reliance on the spurious signal via a score . Definition 2.1 . ( Spurious Score ) . Given a spurious signal , ci , the index of its spurious aligned class , j ∈ [ k ] , a model , θspu : Rd → Rk , where arg max ( θspu ) indicates the classifier ’ s predicted class , we define the spurious score as : SCci , j ( θspu ) : = P { xi|θspu ( xi ) ! = j } [ arg max ( θspu ( SCF ( x i , yi , ci ) ) ) = j ] . Given an input that does not contain the spurious signal , and for which the model ’ s prediction is not the spurious aligned class , the model ’ s spurious score is the probability that the model assigns the input to the spurious aligned class if the spurious signal is added to the input . Model Conditions . We focus our analysis on two model conditions : i ) a ‘ normal model ’ , fnorm , for which we can rule out dependence on any of the spurious signals tested across all classes on the basis of the spurious score , and ii ) a ‘ spurious model ’ , fspu , for which one of the spurious signals encodes for a particular output class . We empirically estimate the spurious score and term models that have a score above 0.85 for any of the pre-defined signals ‘ spurious models ’ . We term a model ‘ normal ’ if the spurious score is below 0.1 across all classes and the 3 pre-defined spurious signals . Spurious Signal Detection Reliability Measures . Equipped with spurious ( fspu ) and normal ( fnorm ) models , we are now able to quantitatively assess the motivating question of this work . We do this by comparing explanations derived from spurious models , fspu , to those derived from normal models ( fnorm ) . We can partition the kinds of inputs used for deriving explanations into two : 1 ) spurious inputs ( xspu ) —inputs that include the spurious signal and 2 ) normal inputs ( xnorm ) —inputs do not not contain the spurious signal . Comparing the explanations produced by these two classes of inputs for normal and spurious models , we derive reliability performance measures . • Known Spurious Signal Detection Measure ( K-SSD ) - measures the similarity of explanations derived from spurious models on spurious inputs to the ground truth explanation . The ground truth explanation is one that only assigns relevance to the spurious signal as explanation of the output of a spurious model on a spurious input . K-SSD measures method reliability when the spurious signal is known . Given a similarity metric , Sd , then K-SSD corresponds to : Sd ( Efspu ( xspu ) , xgt ) ) ; where Efspu ( xspu ) are explanations derived from the spurious model for spurious inputs , and xgt is the ground truth explanation . The similarity function , Sd , depends on the type of explanation considered—we will make our choice of this function concrete shortly . • Cause-for-Concern Measure ( CCM ) - measures the similarity of explanations derived from spurious models for normal inputs to explanations derived from normal models for normal inputs : Sd ( Efspu ( xnorm ) , Efnorm ( xnorm ) ) . This measure simulates the setting where a practitioner does not know the spurious signal , and can only inspect explanations for inputs without the signal . If this measure is high , then it is unlikely that such a method alert a practitioner that a spurious model exhibits defects . • False Alarm Measure ( FAM ) - measures the similarity of explanations derived from normal models for spurious inputs to explanations derived from spurious models for spurious inputs : Sd ( Efnorm ( xspur ) , Efspu ( xspu ) ) . We also introduce a variant of this measure , FAM-GT , which measures the similarity of a explanations derived from normal models for spurious inputs to the ground truth explanation of a spurious model for that spurious input . If this measure is high , then that approach is more likely to signal to a practitioner that a model is relying on spurious signal when the model does not . Having defined the metrics above , it remains which similarity function to use . Computing Metrics for Feature Attribution . For feature attribution methods , we follow prior literature ( Adebayo et al. , 2020 ) and use the Structural Similarity Index ( SSIM ) . SSIM measures the visual similarity between two images , so we use this metric as the measure of similarity between two attribution maps . Concretely , given a set of normal inputs , we can obtain a corresponding spurious set of these inputs by applying the spurious contamination function , SCF to these inputs . Consequently , we can then compute the K-SSD , CCM , and FAM metrics given these two sets of inputs using the SSIM metric . Computing Metrics for Concept Activation . A concept activation method provides a relevance score for a user defined concept . Given a set of user defined concepts , one can estimate and rank each concept by the relevance score . We measure comparison between two concept rankings using a Kolmogorov-Smirnoff ( KS ) test comparing two distributions where the null hypothesis is that the two distributions are identical ; we set significance level to be 0.05 . Computing Metrics for Training Point Ranking . The training point ranking approach assigns a ‘ relevance ’ score to each training point based on influence of each training point on the test loss of a particular sample . Recently , Hanawa et al . ( 2020 ) introduced the Identical class metric ’ ( ICM ) , which is the fraction of the top training inputs , for a given test example , that belong to the same class as the true class of the test example in question . Here we also use the KS test to compare the ICM distributions for two different models and set the significance level to be 0.05 . Taken together , these measures provide a comprehensive overview of an explanation method ’ s performance for detecting spurious signals .

Summary: The authors present an analysis on post-hoc explanation metrics that measure the reliance of a model to spurious signals. The paper offers insights on three metrics  K-SSD, CCM and FAM by deploying them in the analysis of DNNs  trained on medical image datasets. The authors also conduct a blinded study.