Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation
Authors: Julius Adebayo
Michael Muelly
Harold Abelson
Been Kim
Venue: ICLR 2022 Poster (2022)
Keywords: explanations
feature attributions
spurious correlation
interpretability
training point ranking

Abstract: We investigate whether three types of post hoc model explanations–feature attribution, concept activation, and training point ranking–are effective for detecting a model’s reliance on spurious signals in the training data. Specifically, we consider the scenario where the spurious signal to be detected is unknown, at test-time, to the user of the explanation method. We design an empirical methodology that uses semi-synthetic datasets along with pre-specified spurious artifacts to obtain models that verifiably rely on these spurious training signals. We then provide a suite of metrics that assess an explanation method’s reliability for spurious signal detection under various conditions. We find that the post hoc explanation methods tested are ineffective when the spurious artifact is unknown at test-time especially for non-visible artifacts like a background blur. Further, we find that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals even when the model being explained does not rely on spurious artifacts. This finding casts doubt on the utility of these approaches, in the hands of a practitioner, for detecting a model’s reliance on spurious signals.

Published as a conference paper at ICLR 2022
POST HOC EXPLANATIONS MAY BE INEFFECTIVE FOR DETECTING UNKNOWN SPURIOUS CORRELATION
Julius Adebayo MIT CSAIL
Michael Muelly Stanford
Hal Abelson MIT CSAIL
Been Kim Google Research
ABSTRACT
We investigate whether three types of post hoc model explanations-feature attri- bution, concept activation, and training point ranking-are effective for detecting a model's reliance spurious signals in the training data. Specifically, we consider the scenario where the spurious signal to be detected is unknown, at test-time, to the user of the explanation method. We design empirical methodology that uses semi-synthetic datasets along with pre-specified spurious artifacts to obtain models that verifiably rely these spurious training signals. We then provide suite of metrics that assess an explanation method's reliability for spurious signal detection under various conditions. We find that the post hoc explanation methods tested are ineffective when the spurious artifact is unknown at test-time especially for non-visible artifacts like a background blur. Further. we find that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals even when the model being explained does not rely on spurious artifacts. This finding casts doubt on the utility of these approaches, in the hands of practitioner, for detecting a model's s reliance on spurious signals."
It is hard to find a needle in a haystack, it is much harder if you haven't seen needle before (Pearl). -Judea Pearl
INTRODUCTION
A challenge that precludes the deployment of modern deep neural networks (DNN) in high stakes domains is their tendency to latch onto 'spurious -ignals--shorccuts-- the training data (Geirhos al., 2020). For example, Badgeley al. (2019) showed that Inception-V3 model trained to detect hip fracture relied on the scanner type for its classification decision. Deep learning models casily base output predictions on object backgrounds (Xiao et al., 2020), image texture (Geirhos et al., 2018), and skin tone (Stock and Cisse, 2018).
Post hoc model explanation methods approachhss that give insight into the associations that a model has learned- are increasingly used to determine whether model relies on spurious signals. Ribeiro al. (2016) used LIME to show an Inception- V3 model's reliance on the snow background for identifying Wolves. Such demonstration and others (Lapuschkin et al., 2019; Rieger et al., 2020) point to post hoc explanation methods as effective tools for the spurious signal detection task. However, these results conflict with evidence that indicates that practitioners (and researchers)
struggle to use explanations to identify spurious signals (Chen et al., 2021; Chu et al., 2020; Alqaraawi et al., 2020; Adebayo et al., 2020; Poursabzi-Sangee et al., 2018). We seek to resolve this conflict by answering the simple but important question:
Can post hoc explanations help detect a model's 's reliance on unknown spurious training signal?
Motivating Example. Consider machine learning (ML) engineer whose job is train DNN models to detect knee arthritis from radiographs. She-the the engineer-is handed a trained ResNet-50 model, be deployed in hospital, that relies on a hospital tag in the radiographs to detect knee arthritis. She
We refer readers to the longer version of this work the arxiv. Code to replicate our findings is available https///ghhh.......caa/////psppp

---

Published as a conference paper at ICLR 2022
has prior knowledge of the model's reliance on the spurious tags. In this work, our key concern is whether the ML engineer can use post hoc explanations to identify that the model is defective.
1.1 OUR CONTRIBUTIONS
We address the motivating question above in a two-pronged manner. First, we develop an actionable methodology based on the ability carefully craft datasets induce spurious correlation in trained models. Second, we backup this experimental design with a human subject study. Taken together, the takeaway of the work that: post hoc explanations can be used identify a model's reliance on a visible spurious signal, provided the signal is known ahead of time by the practitioner. While this conclusion may seem unsurprising, has important implications for how post hoc explanation methods should be used effectively.
Experimental Design Performance Measures. We provide an end-to-end experimental design for assessing the effectiveness of an explanation method for detecting model's reliance on spurious training signals. We define a spurious score that helps quantify the strength of a model's ss dependence on a training signal. Using carefully crafted semi-synthetic datasets, we are able train models where the ground-truth expected explamation is known. Additionally, we develop performance measures: i) Known Spurious Signal Detection Measure (K-SSD), ii) Cause-for-Concen Measure (CCM), and iii) a False Alarm Measure (FAM). These measures help characterize different notions of reliability for the spurious signal detection task. We instantiate the proposed design on 3 classes of post hoc explanation feature attribution, concept activation, and training point ranking-whhe where we comprehensively assess the performance of these approaches across 3 datasets (2 medical tasks, and dog species classification task), and different model architectures.
When the spurious signal is known, we find that the feature attribution methods tested, and the concept activation importance approach are able to detect visible spurious signals like a text tag and distinctive stripped patterns. However, we find these approaches less effective for non- con-pronounced signals like background blur. The false alarm measure further indicates that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals.
The cause-for-concent measure quantifies the similarity between explanations of 'normal' inputs derived from spurious and normal models when the spurious signal unknown. Across the settings considered, we find that the methods tested are unable to conclusively detect model reliance on unknown spurious signals.
Blinded Study. The findings from our empirical assessment question the reliability of the methods tested; however, it might not correlate with utility the hands of practitioners. address this issue, conduct a user study where practitioners are randomly assigned one of two groups: the first group is told explicitly of potential spurious signals, and the second is not. We consider three different kinds of explanation methods along with control where only model predictions find that when participants are not provided with prior knowledge of the spurious signal, none of the methods tested are effective, in the hands of the participants, for detecting model reliance spurious signals. More surprisingly, even when the participants had prior knowledge of the spurious signal, we find evidence that only the concept activation approach, for visible spurious signals, is effective. These findings cast doubt on the reliability of current post hoc tools for spurious signal detection.
shown. We
Guidance. On the basis our analysis, we can provide the following guidance for using the approaches tested, in this work, for detecting model reliance on spurious signals when the signal of interest is visible:
Feature Attributions: to confirm that model is relying on a 'visible' spurious signal, the practitioner needs to obtain attributions for inputs that contain the hypothesized spurious signal, and the attribution should be computed for the output class to which the spurious signal aligned.
Concept Activation: the spurious concept should be known ahead of time, and tested against the output class to which the concept aligned.
Training Point Ranking: an input that contains the hypothesized spurious signal of interest should be used at test-time in computing training point ranking.
2

---

Published as a conference paper at ICLR 2022
1.2 RELATED WORK
This paper belongs to a line of work on assessing the effectiveness of post hoc explanations meth- ods (Alqaraawi et al., 2020; Adebayo et al., 2020; Chu al., 2020; Hooker et al., 2019; Meng et al., 2018; Poursabzi-Sangee Sangdeh et al., 2018; Tomsett et al., 2020). Here we focus on directly relevant literature, and defer an extensive discussion of the literature to the Appendix.
This work departs from previous work in two ways: 1) we focus exclusively on whether these explanations can be used by practitioner (or researcher) to detect spurious signals that are unknown to her at test-time, and 2) we move beyond sole focus the feature attribution setting to test concept activation and training point ranking methods.
Han et al. (2020) and Adebayo al. (2020) find that certain kinds of feature attributions and training point ranking via influence functions are able to detect a model's reliance on spurious signals. However, in their setting, the spurious signal is known ahead of time. More recently, Zhou et al. (2021) conduct an extensive assessment of several feature attribution methods also under the spurious correlation setting, for visible and non-visible artifacts, and find that these class of methods not effective for non-visible artifacts. addition, they also propose an experimental methodology for controlling model dependence on training set features, which allows them quantify attribution effectiveness precisely, Overall, our findings align with theirs; however, we focus, specifically, on the setting where the spurious signal
known ahead of time.
Kim et al. (2021) conduct an assessment of several feature attribution methods using a synthetic evaluation framework where the ground-truth explanation is known reasoning tasks. They find that feature attribution methods often attribute irrelevant features even in simple settings, and show high variability across data modalities and tasks. Plumb et (2021) introduce a method to identify important associations that model might have learned, detect which of these associations spurious, and propose a data augmentation procedure overcome the reliance. Nguyen et al. (2021) conduct large-scale user study to assess the effectiveness of feature attribution methods on image tasks. They find that feature attributions are not more effective than showing end users nearest neighbor training points. In this work, only consider image tasks, however Bastings et al. (2021) devised a similar experimental procedure and metrics to test several attribution methods for spurious signal detection in text settings. They find that the effectiveness of attribution method depends the task, spurious signal, and other dataset dependent properties.
2 EXPERIMENTAL METHODOLOGY
In this section, we setup our experimental methodology. We discuss quantitative analysis of post hoc explanations derived from models trained to rely on pre-defined spu- rious signals, and a blinded user study that measures the ability of users to use the post hoc explanation methods tested to detect model reliance spurious signals. We discuss the types of spurious signals considered, define spurious score that allows to ascertain that model indeed relies on a signal as basis of its classification de- cision, and layout performance measures that capture the reliability of the explanation methods. We conclude with an overview of the methods tested, datasets, and models.
Original Imput
Haspital
Strips
Backggound Dur
Radlnggaph
Overview Spurious Sig- nals Considered.
Figure
2.1 EXPERIMENTAL DESIGN
Spurious Signals & Score. We consider spurious signal to be input features that encode for the output but have 'no meaningful connection' to the 'data generating process' (DGP) of the task. A hospital tag present in hand radiograph is not clinically relevant to the age of the patient. If the tag encodes for the output then it is a spurious signal. Domain expertise is ultimately required for adjudicating that signal is spurious.
3

---

Published as a conference paper at ICLR 2022
We consider 3 (2 visible and I non-visible) kinds of spurious signals (See Figure 1): i) a localized tag; ii) a distinctive stripped pattern; and iii) Gaussian blur applied to the image background. The signals are all spatially localized,
easily obtain ground-trui truth expected explanations.
To induce reliance on spurious signals, we train models on "contaminated" versions of the train- ing set. Given input-label pairs, {(=', yy)) where x' X and y² E Yr we can learn classi- fier, fa. via empirical risk minimization (ERM) that corresponds to minimizing a loss function, & arg ming >i=1 l(x*, yf: 0). To contaminate the training set, we apply a spurious contamination function (SCF) to the training set, SCF X' x y x C - S, where C is the spurious signal set and S is the transformed set. An example of an SCF is function that pastes an hospital tag onto the bone age radiographs of all pre-put pre-puberty individuals in the dataset. To derive models reliant on a spurious signal, C₁ E C, we simply learn a new classifier via ERM on the modified dataset as follows: arg ming Di= (SSCF( ,v', ca)) to obtain 6spu. Contemporary evidence suggests that this approach produces models that easily latch onto the spurious signal (Nagarajan et al., 2020).
We focus the classification setting, and restrict spurious signals to encode, only, for single class-the spurious aligned class. We measure a model's 's reliance on the spurious signal via a score.
Definition 2.1. (Spurious Score). Given a spurious signal, c,, the index of its spurious aligned class, j E [k). model, Ospu -2 + R&* where arg max (Ospa) indicates the classifier's predicted class, we define the spurious score as:
SCa..j (Oupa) (( (a (spp(t)) t= 5} larg (SC( (z', v))) (f =
Given an input that does not contain the spurious signal, and for which the model's 's prediction is not the spurious aligned class, the model's spurious score is the probability that the model assigns the input to the spurious aligned class if the spurious signal is added to the input.
Model Conditions. We focus our analysis on two model conditions: i) a 'normal model', fnorm- for which we can rule out dependence on any of the spurious signals tested across all classes on the basis of the spurious score, and ii) "spurious model', Sapn. for which one of the spurious signals encodes for a particular output class. We empirically estimate the spurious score and term models that have a score above 0.85 for any of the pre-defined signals 'spurious models', We term a model 'normal' if the spurious score is below 0.1 across all classes and the pre-defined spurious signals.
Spurious Signal Detection Reliability Measures. Equipped with spurious (Sspa) and normal (fuorm) models, we are now able to quantitatively assess the motivating question of this work. We do this by comparing explanations derived from spurious models, fapu, to those derived from nor- mal models (fnorm). We can partition the kinds of inputs used for deriving explanations into two: 1) spurious inputs (Ispu)-inputs that include the spurious signal and 2) normal inputs (M inirutss do not not contain the spurious signal. Comparing the explanations produced by these two classes of inputs for normal and spurious models, we derive reliability performance measures.
Known Spurious Signal Detection Measure (K-SSD)- measures the similarity of expla- nations derived from spurious models on spurious inputs to the ground truth explanation. The ground truth explanation is one that only assigns relevance to the spurious signal as explanation of the output of a spurious model on a spurious input. K-SSD measures method reliability when the spurious signal is known. Given similarity metric, Sa, then K-SSD corresponds to: Sa(E) (Z=pu), Zgz)) where Ey. (#spu) are explanations derived from the spurious model for spurious inputs, and Zgt is the ground truth explanation. The similar- ity function, Sa, depends on the type of explanation considered we will make our choice
of this function concrete shortly.
Cause-for-Concery Measure (CCM) measures the similarity of explanations derived from spurious models for normal inputs to explanations derived from normal models for normal inputs: Sa (E (Znorm). ), E (fnorm)) This measure simulates the setting where a practitioner does not know the spurious signal, and can only inspect explanations for inputs without the signal. If this measure is high, then it is unlikely that such a method alert a practitioner that a spurious model exhibits defects.
False Alarm Measure (FAM) measures the similarity of explanations derived from normal models for spurious inputs to explanations derived from spurious models for spurious inputs: Sa Sa(F., (Fom (Fspur), ), F. (.spa)) We also introduce variant of this measure, FAM-GT,
4

---

Published as a conference paper at ICLR 2022
which measures the similarity of a explanations derived from normal models for spurious inputs to the ground truth explanation of a spurious model for that spurious input. If this measure is high, then that approach is more likely to signal to practitioner that a model is relying on spurious signal when the model does not.
Having defined the metrics above, it remains which similarity function use.
Computing Metrics for Feature Attribution. For feature attribution methods, we use the Struc- tural Similarity Index (SSIM). SSIM measures visual similarity between two images. Concretely, given set of normal inputs, we obtain a corresponding spurious set of these inputs by applying the spurious contamination function, SCF to these inputs. Consequently, we can then compute the K-SSD, CCM, and FAM metrics given these two sets of inputs using the SSIM metric.
Computing Metrics for Concept Activation. We measure comparison between two concept rank- ings using a Kolmogorov-Smimmot (KS) test comparing distributions where the null hypothesis is that the two distributions are identical; we set significance level to be 0.05.
Computing Metries for Training Point Ranking. Recently, Hanawa et al. (2020) introduced the Identical class metric' (ICM), which is the fraction of the top training imputs, for a given test example, that belong to the same class as the true class of the test example question. Here also use the KS test to compare the ICM distributions for two different models and set the significance level to be 0.05.
Taken together, these measures provide comprehensive overview of an explanation method's performance for detecting spurious signals.
2.2 BLINDED STUDY
To complement the quantitative setup, further designed user study (IRB approved) to assess the ability of end-users (200 in total) to use post hoc explanations to detect a model's reliance on spurious signals. About 50 percent of the participants had trained ML model before, and 74 percent had interacted with an ML model. We refer to the appendix for additional details.
Task & Sctup:
The study participants were tasked with assessing model reliability with the aid of model explanations. The participants were randomly assigned to one of two groups:
Normal Model
|+faroTooddley
PrePuserty
Eartybis Pubeny
Late Puperty
Port Pubsrty
the first group told explicitly of potential spurious signals, and the second is not. They were asked to rate how likely they are to recom- mend the model for deployment using 5-point Likert scale, and a rationale for their decision. Our study design follows that of Adebayo et al. (2020); however, we use a mixed within-subjeets. and between subjects design for the factors of interest. The Likert scale is as follows, 1: Def- initely Not to 5: Definitely. Participants select "Definitely" if they deemed the dog breed classi- fication model reliable. We refer Appendix for discussion on user recruitment, statistical analyses, and study design.
Figure 2: Feature Attributions. Here we show 5 different inputs, one for each bone age category, and the corresponding Gradient feature attribution map. We refer the Appendix for an equivalent visualization for other feature attribution methods considered. We test three additional feature attribu- tion methods: SmoothGrad, Integrated Gradients, and Guided BackProp.
We test SmoothGrad, TCAV (a con- cept activation method), a training point ranking method, and a Control setting with no explanations.
Methods:
2.3 EXPLANATION METHODS, DATA, & MODELS
Here we give an overview of the explanation methods, datasets, and models. We present a discussion of how these methods are used in practice in Appendix.
5

---

Published as a conference paper at ICLR 2022
Feature Attributions assign relevance score for each dimension of input towards output. We consider: Input-Gradient, SmoothGrad, Integrated Gradients (IG), and Guided Backprop (GBP).
Concept-Based ((Bau et al., 2017; Kim et al., 2018)) approaches measure the dependence a DNN's prediction user-defined -erume concepts. We select TCAV as the approach to assess this class (Kim et al., 2018).
Training Point Ranking via Influence Functions (Koh and Liang, 2017). This approach ranks the training samples in order of importance/influence on the loss (or prediction) of a test example.
Concept Importance Ranking (TCAV)
Clinical Concepts
Infancy & Toddler
Pro: Pubert
Early Mid Puperty
Late Puberty
Post Puberty
Ltna
MGP
What
Ulsa
0,00
0.25
0.50
Score
1.00 0.00
0.25 0.59
6.75
1.00
0.03
0.25
8,50
0.75
3.00 8:50
0.2
0.30
0.75
1.00
0.00
0.25
0.50
6.75
1.00
Figure 3: Concept Importance Methods for a Normal (non-spurious) model. Here we show the TCAV score for all clinical concepts as well as the spurious concepts for a normal model that was confirmed not rely on the spurious signals.
Models, Datasets, & Tasks. We consider two medical datasets: Hand (Halabi et al., 2019) and Knee radiographs (Chen et al., 2019) and a dog breed classification task. We consider a small DNN (6 layers, 5 of which are the traditional han--relu-batchnomm-maapp CBR-Tiny architecture of Raghu et al. (2019) and a ResNet-50 model (See Appendix for additional details).
combination) inspired by the
FEATURE ATTRIBUTIONS
In this section, we present results on whether feature attributions are effective for detecting unknown spurious correlation.
Setup. We consider 3 kinds of spurious signals which we term: Täg for the 'MGH' hospital tag added to the pre-puberty class, Stripe for the paired vertical stripped signal, and Blur for the background blur. Given these signals, we then compute the three performance measures of interest: K-SSD, CCM, FAM, and FAM-GT. K-SSD indicates a method's s reliability when the spurious signal known, CCM when the signal is not known and the practitioner only has access to inputs that don't encode the spurious signal. Lastly, FAM and FAM-GT indicates the susceptibility of a method to false positives. An oft-used heuristic based prior work (Adebayo al., 2020) for interpreting SSIM scores is that SSIM scores 0.2 0.4 indicate weak visual similarity, 0.5 0.7 indicate moderate similarity, and 0.75 high similarity. This is because For example, we empirically estimate the similarity of two random (229 x 229) Gaussian images to be less than 0.00023. Even for natural images, still find the SSIM values below 0.005, which substantiates the previous heuristic.
random images typically have SSIM much less than 0.1.
Results. We show performance measures for all the feature attribution methods tested for the Tag and Blur settings in Tables & 11 (See Appendix). For the tag setting, the attribution methods are indeed able attribute to the visible spurious signal and the K-SSD measure indicates this finding with mean scores typically above 0.65 for the bone age setting. Contrary to previous findings, we find that GBP outperforms other approaches for known spurious signals. Alternatively, GBP more suceptible to false positives based on the FAM and FAM-GT score. Across all methods, we find that these methods also seem to attribute to the spurious signal (FAM-GT >> 0.4) even when the signal is not being relied on by the model. We observe similar findings for the strip setting well across all tasks. The CCM measure further indicates that these methods do not indicate presence of spurious signals when the signal is unknown for both the Tag and Stripe signals. This finding, however, reverses for

---

Published as a conference paper at ICLR 2022
A: Normal Model Spurious Tag Inputs
BR Spurious Tag Model on 'Norraal Inpute
C: Spurious Tag Model on Spurious Tag Imputs
Irpul
Gradient
A: Normal Model Spurious Stripe Inputs
B: Spurious Stripe Model on "Normar Inputs
C: Spurious Stripe Model on Spurious Stripe Inpets
tepor
Grudiett
Normal Blur
Spurious Blur Model "Normaf Inpets
Spurious Model Blur
zoua feleay
Knlwyy
haleer
Law Kolerg
teiuttthiee Lew aep Aor
Ieput
Figure 4: Top-Detecting Spurious Tag. Here we show in A) Feature attributions for different inputs across the four feature attribution methods with normal model but with spurious Tag inputs; B) Feature attributions on the same 5 inputs as in (A), but without spurious Tag inputs with a model that has learned spurious alignment between Pre- Pre-Puberty and Tag: C) Feature attributions on the same 5 inputs as in (A), but with the spurious Tag inputs with a model that has learned a spurious alignment between Pre-Puberty and Tag. Middle-Detees Detecting Spurious Stripe. Here show in A) Feature attributions for 5 different inputs across the four feature attribution methods with a normal model but with spurious Stripes inputs; B) Feature attributions the same inputs in (A), but without the spurious Stripe with a model that has learned a spurious alignment between Pre-Puberty and Stripe: C) Feature attributions on the same inputs as (A), but with the spurious Stripe with a model that has learned a spurious alignment between Pre-Puberty and Stripe. Middle-Detectiin Spurious Blur. The blur images are analogous to the Tag and Stripe settings. Please refer to the Figures! 7, 8, 9 in the Appendix for SmoothGrad, Integrated Gradients, and Guided BackProp examples.
Table I: Performance metrics for each attribution method across tasks for the Tag Setting. Below each metric in the Table is another row (SEM) that indicates the standard error of the mean for each value.
Method
Bone Age
Knee
Dog Breeds
Grad
SG
IG
GBP
Grad
SG
IG
GBP
Grad
SG
IG
GBP
K-SSD
0.65
0.66
0.67
0.81
0.51
0.49
0.47
0.76
0.71
0.76
0.79
0.88
K-SSD (SEM)
0.0097
0.013
0.019
0.012
0.017
0.023
0.014
0.01
CCM
0.37
0.39
0.35
0.75
0.32
0.33
0.35
0.66
0.42
0.41
0.39
0.64
CCM (SEM)
0.0031
0.002
0.015
0.029
0.027
0.023
0.029
0.014
0.013
0.016
0.012
0.015
FAM
0.51
0.55
0.53
0.68
0.46
0.47
0.45
0.69
0.64
0.68
0.73
FAM (SEM)
0,0029
0.0019
0.018
0,024
0.023
0.024
0.019
0.016
0.015
0,011
0.022
0.035
FAM GT
0.56
0.53
0.46
0.61
0.42
0.48
0.41
0.63
0.76
0.73
0.77
0.81
FAM-GT (SEM)
0.017
0.035
0.0253
0.028
0.016
0.019
0.0045
0.006
0.011
0.033
0.024
0.0053
the non-visible blur spurious signal. Across all measures, we find that all methods struggle to reliably indicate that spurious models are reliant on the blur signal.
Additionally, we also find that the FAM scores are typically higher than the CCM scores across the tasks. This finding indicates that the feature attribution methods tested more susceptible to false positives than they are to indicate to a practitioner that a model is defective in the absence of the spurious signal, finding that casts doubt on the utility of such approaches in practice.
7

---

Published as a conference paper at ICLR 2022
4 CONCEPT ACTIVATION IMPORTANCE
We find that concept methods can indicate a model's reliance on Tag and Stripe signals when known. However, the approach struggles detect Blur signal even when known. As is the case with feature attributions, when a spurious signal is not explicitly tested for, our significance tests indicate that reliance cannot be detected in the non-spurious concepts available.
Overview & Setup. In this setting, we compute the performance metrics of interest: K-SSD, CCM, and FAM. To measure comparison between concept rankings, use Bolmogorov-Smimoot (KS) test comparing two distributions where the null hypothesis is that the two distributions are identical; we set significance level be 0.05.
Puberty
Spurious Tag Model
Utn
1ag
0.50 6.75
1.00
8.00
6.25 0.50 0.7
6:00
0.25
8.75
1.00
0.50 6.75
1.00 0.00
0.22 0.te 077
TOM Scort
TOAY Scoee
TEAV Sooe
TOW Soore
TGM Score
Infency & Toddler
hre-Pubetty
Carly-Miid Puberty
Late Puberty
Post Puberty
4C8
Spurious Blur Model
Forka
TCAM Score
TCAV Score
TEAM Score
TEAV Scor
TCAM Scoe
Figure 5: Concept Results for Tag and Blur Models. TCAV scores for model reliant on the spurious Tag and a model reliant on Blur.
Result. In Figure 5, we show TCAV scores for each bone age class for both the spurious tag and the blur models. In Table 2, we show results of the KS-Test across metrics for the Tag Setting. Here, a X means we reject the null, while an / means we are unable to do so at the pre-specified significance level. For K-SSD metric, the KS-test rejects the null for the Tag and Stripe signals. However, we find that the opposite is the case for the blur signal. This suggests that concept rankings can help detect reliance on the visible signals but not non- visible signals when the spurious signals are unknown. However, for the CCM metric, we are unable to reject the hypothesis that the distribution of concept rankings not similar for all spurious signals. This finding suggests that when the spurious signal is unknown, and we only compare the distributions of known (non-spurious) concepts for normal model and a spurious model, there is high similarity. Lastly, difference in means test as well as KS-Test for the FAM measure indicates that the normal models do not rely on the spurious signals as well. Overall, this finding suggests that TCAV is less susceptible to false positives.
Table 2: Concept Metrics Tag.
Metric
Result
K-SSD CCM FAM
X
5 TRAINING POINT RANKING & BLINDED STUDY
Overview & Setup. We now describe our empirical findings for the training point ranking via influence functions approach. Here present results for the case where the spurious signal is aligned with the Pre-Puberty class.
The main take away is that given known spurious signal, the fraction of top ranked training spurious signal inputs increases with a spurious model across all of the spurious signals. While this might encouraging, note that such increase actually indicates that the ICM metric might provide illusory confidence in a spurious model. Ultimately, a critical requirement here is knowing what the spurious signal ahead of time and be able select the right inputs inspect.
Results.
8

---

Published as a conference paper at ICLR 2022
Blinded Study. We now turn summary of the results of the blinded user study. The median recommendations selected by participants (200 in total) is reported for each explanation- modell condition in Table 6. We plot a box plot of all 16 categories in the appendix. Our setting mimics traditional randomized experimental settings, so we adopt a randomization inference analysis to determine the effect of each treatment, which is the explanation method in this case, the ability of the users to recommend model. A higher Likert score means the user is more likely to recommend model. Consequently, an explanation method is effective, then it should be the case that users should be less likely to recommend model that relies on a spurious signal.
We conduct two kinds of statistical analyses of the data. First, un- der each condition, we perform difference of means test for each treatment compared to the control setting. Secondly, for each model manipulation, we performed one-way Anova test, and a Tukey- ing
Table 3: Training Point Rank-
Kramer test to assess the effect of the explanation type on the ability of the participants to reject defective model.
Metric
Result
We observe that when blinded, in none of the methods do participants conclusively reject spurious models. Perhaps more surprising, when the participants were not blinded, we see that only participants using the TCAV approach rejected a spurious model. This finding has significant implications on whether these current tools are effective in the hands of practitioner.
K-SSD CCM FAM
X
Method
B-Normal
NB-Normal
B-Spurious
NB-Spurious
SmoothGrad
4"
4*
3"
TCAV
4"
3
3
2
Influence
3
3
3
Control
4
3
4
Table 4: Blinded User Study. Here we report the median recommendation for each condition across all explanation types and control. This median is derived from user provided responses assessing model reliability. B indicates Blinded, and NB indicates Not-Blinded. In the Table, the symbol indicates statistically significant conditions. For the sake of space, we defer the full distribution description to the Appendix.
DISCUSSION & CONCLUSION
Conclusion. DNNs trained image datasets naturally rely spurious training signals (Geirhos et al., 2020). Discovering this reliance is crucial in consequential settings like medical imaging. Post hoc explanations methods are a promising direction towards detecting such reliance; however, their effectiveness is currently under question. We investigated whether 3 classes of post hoc explanations are effective for detecting a model's 's reliance on spurious training signals. We present an experimental setup that can also be easily adapted to other settings to assess a larger class of approaches. The setup comes equipped with spurious score and performance measures for spurious signal detection. We find that the 3 classes of post hoc explanations tested are only sometimes able to diagnose the spurious training signal even if they are used to explicitly test for model dependence on these signals. Consequently, findings calls for, potentially, a completely different paradigm of methods that designed to address the important and challenging question of detecting spurious training signals.
Limitations. In this work, we have focused exclusively on DNNs trained on image tasks; related work (Bastings et al. (2021) has considered the text setting), however, it is unclear if our findings will generalize to other modalities like time-series data. While considered diverse set of methods, the literature on post hoc explanations is quite vast, so undoubtedly there exist methods that do not fit neatly into the explanation classes that explored.

---

Published as a conference paper at ICLR 2022
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances Neural Information Processing Systems, pages 2018.
Julius Adebayo, Michael Muelly, llaiia Liccardi, and Been Kim. Debugging tests for model explana- tions. arXiv preprint urXiv:2011.54229 2020.
Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. Evaluat- ing saliency map explanations for convolutional neural networks: a user study. In Proceedings of the 25th International Conference Intelligent User Interfaces, pages 285,285, 2020.
Christopher Anders, Plamen Pasliev, Fairwashing explanations with off manifold detergent. In International Conference Machine Learning, pages 3143223 PMLR, 2020.
Dombrowski, Klaus-Rober Mülerr, and Pan Kessel.
Marcus A Badgeley, John R Zech, Luke Dakden-Rayner Benjamin S Glicksberg, Manway Liu, William Gale, Michael V McConnell, Bethany Percha, Thomas M Snyder, and Joel T Dudley. Deep learning predicts hip fracture using confounding patient and healthcare variables. NPJ digital medicine, 2(1):1-10, 2019.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert MÅÄller. How to explain individual classification decisions. Journal of Machine Learning Research, 1 1(0un):1803-1811 2010.
Mohammad Taha Bahadori and David Heckerman. Debiasing concept-based explanations with causal analysis. 2021.
Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. will you find these shortcuts?"' protocol for evaluating the faithfulness of input salience methods for text classification. arXiv preprint arXiv:2111.07367, 2021.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions deep learning are fragile. arXiv preprint arXiv:2006. 14651, 2020.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
Pingjun Chen, Linlin Gao, Xiaoshuang Shi, Kyle Allen, and Yang Lin. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. Computerized Medical Imaging and Graphics, 75:84 92, 2019. doi: https://doi.or// 100. 101 co/..coomme 2019.06.002.
Valerie Chen, Jeffrey Li, Joon Sik Kim, Gregory Plumb, and Ameet Talwalkar, Towards connecting use cases and methods interpretable machine learning. arXiv preprint arXiv:2103, 06254, 2021.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):72-782,
2020. 2020.
Eric Chu, Deb Roy, and Jacob Andreas. Are visual explanations useful? a case study in model-in sidel-in-ttee loop prediction. arXiv preprint 20Xiv22007.12248 2020.
Alex DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. medRxiv, 2020.
Ann-Kathri Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems, pages 13567-13578. 2019.
Finale Doshi Boshi-Velez and Been Kim. Towards rigorous science of interpretable machine learning. 2017.
10

---

Published as a conference paper at ICLR 2022
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels. Scott Lundberg. and Su-In Lee. Learning explainable models using attribution priors. arXiv preprint arXiv:1906.1061 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 1.12311. 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 201 (666666666355 2020.
Amirata Ghorbani, Abubakar Abid, and James Y. Zou. Interpretation of neural networks is fragile. In The Mhirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innova- tive Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 -eebruary 1, 2019, pages 368 -3688. AAAI Press, 2019a. doi: (609%amai.v3301133300.... URL https:tppp...m. /ww./iiioo// org / 10. 609 aaai. v333 33013681.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-base based explanations. arXiv preprint arXiv: 0902.0992...... 2019b.
Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint arXiv:20/2.15781, 2020.
Safwan S Halabi, Luciano M Prevedello, Jayashree Salpathy-Cramme Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology, 290(2)498-500) 2019.
Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint Stti::20000066766 2020.
Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of manilarity-based explanations. arXiv preprint arXiv:2006.0m 04528, 2020.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems, pages 2921-2932, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods deep neural networks. Advances in Neural Information Processing Systems, pages 9737-9748, 2019.
Dmitry Kazhdan, Botty Dimanov, Mateja Jamnik, Pietro Liò, and Adrian Weller. Now you see me (cme): Concept-based model extraction. arXiv preprint arXiv:2010.1322 2020.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. arXiv preprint arXiv:20/2.04104 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference Machine Learning, pages 2673 2673-2682, 2018.
Joon Sik Kim, Gregory Plumb, and Ameet Talwalkar. Sanity simulations for saliency methods. arXiv preprint 105. (655066666,000 2021.
Pang Wei Koh and Percy Liang, Understanding black-box predictions via influence functions. In
Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference
Machine Learning. ICML 2017, Sydney, NSW. Australia. 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894, PMLR, 2017. URL http: / //proceedings ml ppress sitess/v700 v00 koh17. 7 html.
11

---

Published as a conference paper at ICLR 2022
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348. PMLR, 2020.
Himabindu Lakkaraju and Osbert Bastani. how do fool you?" manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 79-85, 2020.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and -aburt Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1-8, 2019.
Aravindh Mahendran and Andrea Vedaldi. Salient deconvolutional networks. In European Conference Computer Vision, pages 120-135. Springer, 2016.
Qingjie Meng, Christian Baumgartner, Matthew Sinclair, James Housden, Martin Rajchl, Alberto Gomez, Benjamin Hou, Nicolas Toussaint, Jeremy Tan, Jacqueline Matthew, et al. Automatic shadow detection in 2d ultrasound. 2018.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010. 15775, 2020.
Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and correlation with automatic evaluation scores. arXiv preprint arXiv:2 14944, 2021.
Weili Nie, Yang Zhang, and Ankit Patel. A theoretical explanation for perplexing behaviors of backpropagattion-base visualizations. In 2018.
Judea Pearl. URL httphht://cuusalitt/g//////////t cs. ucla edu/blog/.
Gregory Plumb, Marco Tulio Ribeiro, and Ameet Talwalkar. Finding and fixing spurious patterns with explanations. arXiv preprint arXiv:2106.0211 2021.
Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. Manipulating and measuring model interpretability. arXiv preprint arXiv: MiXiv:/802.07800 2018.
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracking gradient descent. arXiv preprint arXiv:2002.08488 2020.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging, arXiv preprint arXiv:1902.07200. 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135 1144. ACM, 2016.
Laura Rieger, Chandan Singh, W James Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations align neural networks with prior knowledge. International Conference on Machine Learning, 2020.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Vekee Right for the right reasons: Train- ing differentiable models by constraining their explanations. In Carles Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 2662-2670. ijcai.org, 2017. doi: 10.24963fijcai221177777 URL httph:tt////mm ioi. org 10. 24963/44442 ijcai aataaai201/37..
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.0873: 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pages 1446-8356 PMLR, 2020.
12

---

Published as a conference paper at ICLR 2022
Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? arXiv preprint arXiv:2102.1278. 2021.
Hua Shen and Ting-Hao Huang. How useful are the machine-generatee interpretations to general users? a human evaluation on guessing the incorrectly predicted labels. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 168-172, 2020.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Truck Proceedings, 2014. URL 1312. 6034.
org/ abs//
Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why modified bp attribution fails. arXiv preprini arXiv: 1912.0988,, 2019.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180-186, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:170.00388 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint 14/2. 2014.
Suraj Srinivas and Francois Fleuret. Rethinking the role of gradient-based attribution methods for model interpretability. In International Conference on Learning Representations, 2021. URL https: / wwwnreveeeeeeeeeeeeeeeeeeeeeeeee ew net/ forum? d=dYeARXppR
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: and uncovering biases. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0008-51,777 2018.
Understanding mistakes
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319- 3328, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http: //proceed ngs ml r. press / v70 sundarara jan1 7 .htll.
Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun D. Preece. Sanity checks for saliency metrics. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 6021-6029 AAAI Press, 2020. URL https: //
/tjo/////////oooooooo/o/o/oooo// htt////ssiineeeeee winiiddxx..... wwtpddxwwppppppppp/ppppppppppppp AAAI www.///aaiie//////aaaaaaaaaaaeee
6064.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint htt::2000600999, 2020.
Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar, Representer point selection for explaining deep neural networks. In Advances in neurul information processing systems, pages 9291-9301, 2018
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? arXiv preprint arXiv:2 14403, 2021.
13


Reviews:
[{"review_id": "rWnJoOsvzTr", "reviewer": "Reviewer_uWNE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper offers an analysis on three metrics for explaining post-hoc reliance of models on spurious signals. Even though at first inspection the metrics proposed seem reasonable, at the authors admission, they turn out not to be all that good detecting reliance on spurious signals, especially when the signal is unknown a priori ( admittedly the hardest case). \n\nIssues of the paper : \n- Clarity:  from the main body of the paper the analysis of the metrics is lacking, one has to refer to the appendix to get a better idea of the metrics proposed and how they are used in practice. \n- Even though the authors claim a test on dog breeds dataset these are not included -- also all dogs belong to the same species Canis familiaris, the classification task refers to dog breeds , not species. \n- Contributions: the paper proposed 3 metrics that as admitted by the authors are not performing as well as one would like. That on itself is not an issue as semi-negative results have merit, however, there is no indication of proposals of the authors on what the better metrics should be.\n\n", "summary_of_the_paper": "The authors present an analysis on post-hoc explanation metrics that measure the reliance of a model to spurious signals. The paper offers insights on three metrics  K-SSD, CCM and FAM by deploying them in the analysis of DNNs  trained on medical image datasets. The authors also conduct a blinded study.", "summary_of_the_review": "Overall this paper is an analysis paper of 3 self-proposed metrics on posthoc analysis of spurious signals. The metrics turn out not to be accurate enough for the tasks at hand, especially when the spurious signal is implicit or unknown. There is no discussion how to make the metrics better at detecting spurious signals. \n\n\n\n###### POST REBUTTAL #### \nUpdate the score to 6 ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ggeJpW9qhs", "reviewer": "Reviewer_Ng3n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n+ The authors develop 3 kinds of spurious signal detection reliability measures: Known Spurious Signal Detection Measure (K-SSD), Cause-for-Concern Measure (CCM), False Alarm Measure (FAM). \n+ The authors carefully design the experiments to assess the effectiveness of an explanation method for detecting a model’s reliance on spurious training signals. \n+ The authors further conduct a user study to measure the ability of users to use the post hoc explanation methods tested to detect model reliance on spurious signals. \n+ The authors' findings indicate that the explanation methods are less effective to detect non-visible spurious signals and only the concept activation approach for visible spurious signals is effective, which cast doubt on the reliability of current post hoc tools for spurious signal detection. \n\nWeaknesses: \n- The paper is somewhat hard to follow. \n-- The authors claim that they find that these methods also seem to attribute to the spurious signal (FAM > 0.5) even when the signal is not being relied on by the model. It is unclear whether \"the model\" here is the normal models or spurious models. FAM > 0.5 means that both spurious model and normal model generate similar explanations on spurious inputs, but it is unclear whether the similarity of the explanation comes from the spurious signals or not, so we don't know whether the normal model attributes the spurious signals. \n-- The authors claim that (Section 3) As indicated, the CCM measure further indicates that these methods do not indicate the presence of spurious signals when the signal is unknown for both the Tag and Stripe signals. It is unclear how the authors get this conclusion from low CCM values.  We only know that (Section 2.1) If this measure (CCM) is high, then it is unlikely that such a method alert a practitioner that a spurious model exhibits defects. In Table 1, GBP has high CCM while other methods have low CCM. Why do the authors think that the CCM measure indicates that all these methods do not indicate the presence of spurious signals even with both high and low CCM? \n-- The authors did not report the K-SSD, CCM, FAM, and KS-test values in Section 4, which makes some discussions hard to follow. For example, the last sentence in Section 4: \"Lastly, a difference in means test for the FAM measure indicates that the normal models do not rely on the spurious signals. Overall, this finding suggests that TCAV is less susceptible to false positives\". \n\n- Tables 1, 2 display the K-SSD, CCM, FAM values in different settings. These values can describe the difference of explanations in various settings (e.g., normal and spurious models). However, even for the same model trained on the same datasets with different initial states, the outputs and explanation results might be different. It would be better if the authors displayed the values of K-SSD, CCM, FAM of explanation among the same models trained with different initial states on the same inputs. With the results as a comparison, it would be easier for readers to understand how much the spurious model or spurious inputs affect the explanation results. \n\n- It would be better if the authors displayed the standard variance of the measurements (e.g., Table 1, 2). \n\n- Minor: In Table 1,2, FAM values are always much higher than CCM, which means spurious model and normal model generate more similar explanations on spurious inputs than normal inputs. It is worth some discussion. \n\n- Minor: GBP achieve much higher CCM and FAM scores than other methods (Table 1, 2). It would be better if the authors gave more discussion about it. ", "summary_of_the_paper": "The paper aims to validate whether post hoc explanation methods are effective for detecting unknown spurious correlations. The authors design 3 kinds of spurious signal detection reliability measures: Known Spurious Signal Detection Measure (K-SSD), Cause-for-Concern Measure (CCM), False Alarm Measure (FAM). Based on the 3 measurements, the authors conduct extensive experiments to validate the reliability of 3 kinds of post hoc explanation methods for detecting spurious correlations. ", "summary_of_the_review": "The overall idea of the paper is good. But it is hard to follow and some analyses are not clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LRPOrBMfpYl", "reviewer": "Reviewer_Mwip", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all I would like to thank the authors for this interesting piece of work. Below, I will briefly list the main strength and weaknesses of this submission and make suggestions for improvement. I hope you will find them helpful and constructive.\n\nStrengths\n- This work raises very important and timely concerns with regard to using NNs for critical application domains like health care.\n- The scope of this work is very extensive: 2 synthetic and 1 empirical dataset, 3 different spurious signals, 3 (considering the different variants of feature attribution algorithms 6) different post-hoc methods\n\nWeaknesses\n- The spurious signal detection reliability measures are not explained sufficiently. How do you quantify similarity for the three spurious signal detection reliability measures? Equations would be helpful. How is the ground truth defined for the K-SSD measure, is this normal input and normal model? \n- How should these scores be judged, what value do you interpret as problematic and why? Is there a rule of thumb that you can recommend?\n\n- You should mention in the limitations (not in a food note), that the 'normal model' could still contain unknown spurious signals in the original data set\n\n- The empirical study needs a more detailed description (maybe in the supplement). What population was recruited for the study, is this a population of medical practitioners, machine learning engineers, or random subjects? This should also be considered in the limitations. Is this representative of the people that will use these methods? Also is there reason for concern when choosing a within-subject design here. Briefly, when participants detect a spurious signal with one method, will this not influence their judgement based on the next method. Was the order of methods counter-balanced?\n\n- The results of the empirical study should be reported properly (measures of variance across participants). Are differences between normal and spurious and blinded and not blinded statistically significant using a formal test. Also why are the results mentioned in the discussion and not the results.\n\n- It is not clear to me, why the spurious MHG hospital tag signal appears to be present in all classes and not just the spurious (pre-puberty) class in Figure 5. Can you clarify?\n\n- I commend the authors for doing a formal statistical test (Kolmogorov-Smirnoff) for the concept activation method. I would suggest including a visualization of these test results, maybe a bar plot of only the spurious concept and indicating which comparisons differed significantly with asterisk or lines. Of course the same would be useful for other comparisons that you report in tables or figures for example on the three performance measures across different methods that you defined. A formal test would substantiate your claims.\n\n- To improve clarity of the figures, I would suggest adding arrows to Figure 1 to direct the readers attention to the spurious signals.\n\n- Overall, the manuscript seems to be quite packed. I would suggest to move some results to the supplement, for example 3 of the 4 methods for feature attribution and Figure 2 may be good candidates.\n\n-----------------------------------\nMinor points\n\nThere are some typos in the manuscript. Please, have it proof-read.\n- p6: “We results report for the small DNN model in the paper” => “We report results”\n- p3: “which of these associations are be spurious” => are spurious\n\n", "summary_of_the_paper": "The authors present work that aims to test, whether post hoc explanation methods can detect unknown spurious signals. They perform an analysis on two different medical image datasets to which they introduce two different types of synthetic, spurious signals: pronounced spurious signals (stripes and hospital tags) and non-pronounced signals (blurs). When testing the ability of different post hoc explanation methods to identify spurious signals, they find that feature attribution and concept activation are able to identify pronounced, but not non-pronounced spurious signals that are known. When faced with unknown signals, none of the methods that were tested appears to be able to identify them. They also provide results from an empirical study suggesting that none of the methods allowed practitioners to identify unknown spurious signals and only concept activation appears to enable practitioners to identify known spurious signals.\n\nThe main contribution of this work are the empirical results and alerting researchers (and potentially practitioners) to very substantial problems with current post-hoc explanation. The technical contributions are negligible.\n", "summary_of_the_review": "Overall, this paper raises important questions about the usefulness of current post-hoc explanation approaches for NNs and provides a range of empirical results to support these claims. While the contribution is incremental compared to previous studies, I believe the authors do raise some important points that will help to stimulate more research in improving post-hoc explanation methods. However, the manuscript is quite dense, lacks some important clarifications (e.g., about the performance measures and the empirical study) and more rigorous statistical tests to substantiate the authors claims. For this reason, I cannot support acceptance of the manuscript as is, but I am willing to reconsider, should my concerns be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper demonstrates that current post-hoc methods to explain black-box models are not robust to spurious signals based on three metrics especially when the spurious signals are implicit or unknown.  Technical novelty is limited because the paper presents primarily empirical results instead of novel machine learning techniques. However, the problem is very important and timely, and significance to the field and potential impact of the presented results to advance the field are high as reviewers emphasized. There are ways to further improve the paper, including the clarity of presentation, although the authors improved in the revised manuscript.  Overall, this paper deserves borderline acceptance.
Decision: Accept (Poster)