The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective
Authors: Satyapriya Krishna
Tessa Han
Alex Gu
Steven Wu
Shahin Jabbari
Himabindu Lakkaraju
Venue: Accepted by TMLR ()

Abstract: As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of if and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we introduce and study the disagreement problem in explainable machine learning. More specifically, we formalize the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how practitioners resolve these disagreements. We first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and six different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that (1) state-of-the-art explanation methods often disagree in terms of the explanations they output, and (2) machine learning practitioners often employ ad hoc heuristics when resolving such disagreements. These findings suggest that practitioners may be relying on misleading explanations when making consequential decisions. They also underscore the importance of developing principled frameworks for effectively evaluating and comparing explanations output by various explanation techniques.

Published in Transactions on Machine Learning Research (06/2024)
The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
Satyapriya Krishna' Horvard University
edu
Tessa Han" Harvard University
than@g. edu
Alex Gu Massachusetts Institute of Technology
gua edu
Steven Wu
asteverwn@@mmu.comm edu
Carnegie Mellon University
Shahin Jabbari
Ahahennddrenee edu
Dreael University
Himabindu Lakkaraju
hlakkarajuhhbs.co edu
Horvard University
These authors contributed equally to this work.
Reviewed on OpenReview: http:///peeanrieee.sss////ttttts
Abstract
As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop deeper understanding of whether and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we formalize and study the disagrcement problem explainable machine learning. More specifically, we define the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how practitioners resolve these disagreements. We first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction, and introduce novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-at post hoc explanation methods, and six different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out online user study with data. scientists to understand how they resolve the aforementioned disagreements. Our results indicate that (1) state-of-the-art explanation methods often disagree in terms of the explanations they output, and (2) machine learning practitioners often employ ad hoc heuristics when resolving such disagreements. These findings suggest that practitioners may be relying on mislcading explanations when making consequential decisions. They also underscore the importance of developing principled frameworks for effectively evaluating and comparing explanations output by various explanation techniques.

---

Published in Transactions on Machine Learning Research (06/2024)
1 Introduction
As machine learning (ML) models become integral in high-stakes domains like healthcare and finance, the interpretability of these models is crucial for ML practitioners and domain experts (e.g., doctors and policymakers). Understanding model behavior is essential to identify errors and biases and gauge the reliability of model predictions, especially in complex models [Doshi-Velez & Kim 2017). Several post hoc explanation methods have been proposed such as perturbation-basee methods (e-g., LIME (Ribeiro et al. 2016b), SHAP (Lundberg & Lee 2017)) and gradient based methods (e.g.. Gradient times Input [Simonyan al. 2014), SmoothGrad (Smilkov et al. 2017 Integrated Gradients (Sundararajan et al. 2017
GradCAM (Selvaraju et al. 2017))
Prior research has investigated various aspects of explanation methods. For example, metrics have been developed to assess the faithfulness of explanations to the model's behavior, such as comparing them to a model's ground truth feature importances or using the 'Remove and Retrain' (ROAR) approach (Liu et al.
2021 Hooker al. 2018). However, challenges arise due to the unavailability of ground truth in real-world applications and the impracticality of retraining models in certain settings (Zhou et al. 2021 Studies have shown that popular explanation methods produce inconsistent or misleading explanations, leading to
risks like deploying biased models or making erroneous clinical decisions (Aivodji et al. 2019 Slack et al.
2020
However, while prior research has studied explanation methods, the (dis)agreement among different methods has not been extensively investigated. Practitioners often use multiple methods, but these can yield conflicting explanations. Therefore, resolving these disagreements crucial to avoid reliance misleading explanations with potcntially severe consequences (Slack et al.) 2020). Yet, there is lack of rescarch the extent to which explanations disagree and how practilioners resolve these disagreements.
In this work, we formalize and investigate the disagreement problem in explainable ML, a novel area of research that examines conflicts among post hoc explanation methods. We present the following key contributions:
1. We conduct semi-structured interviews with 25 data scientists, gaining insights on the nature and frequency of explanation disagreements in their workflows. This input helps us define what constitutes an explanation disagreement in practical scenarios.
2. Building on these insights, we formalize the concept. of explanation disagreement and create a new evaluation framework to quantitatively assess disagreements between pairs of explanations for the same model prediction.
3. We apply this framework in an extensive empirical analysis, measuring disagreement levels across four real-world datasets, three data modalities, six advanced explanation methods, and various predictive models, including logistic regression, tree-based models, and various neural networks.
Finally, we conduct an online user study with another set of 25 data scientists. We present them with pairs of conflicting explanations, assessing which they would rely on and their rationale. We also gather insights on their strategies for resolving such disagreements in their daily work.
Our findings reveal that explanation disagreements are common and problematic: 84% of interviewees reported encountering explanation disagroements in practico. Our empirical analysis confirms widespread discrepancies across different methods and data modalities (tabular, text, and image). Moreover, 86% of our online user study participants admitted to using ad hoc heuristics or being uncertain about resolving explanation disagreements. This highlights the need for better evaluation metrics and enhanced practitioner understanding of the underlying principles of explanation methods.
2
Related Work
Our work is grounded in the exlensive literalure on explainable ML, which we discuss in this section.
All the user interviews and studies in this work were approved by our institution's IRB.
2

---

Published in Transactions on Machine Learning Research (06/2024)
Inherently Interpretable Models and Post Hoc Explanations. A variety of approaches have been developed to learn inherently interpretable models for tasks such as classification and clustering (Letham
et al.
2015
Lakkaraju et al., 2016 Bien & Tibshirani 2009; Kim et al.) 2014 Lou et al., 2012 Caruana
ict al.
2015
However, complex models like deep neural networks often outperform these simpler models in
terms of accuracy (Ribeiro et al. 2016b). This has led to significant interest in post. hoc explanation methods, which aim to understand the behavior of these complex models by approximating them using simpler models in & post hoc fashion. Post hoc explanation methods can be categorized into local explanation methods and global explanation methods. Local explanation methods, including LIME, SHAP, SmoothGrad, and Integrated Gradients (Ribeiro et al. 2016a Lundberg & Lee 2017 Sundararajan et al. 2017 Smilkov et al. focus on explaining individual model predictions.
2017
Global explanation methods, on the other hand, aim to summarize the behavior of complex models as a whole (Bastani et al., 2017: Lakkaraju et al. 2019). Our work is focused on analyzing the disagreements among explanations generated by the aforementioned local post hoc explanation mothods.
A Analyzing and Evaluating Post hoc Explanations. Prior research has proposed several notions of explanation quality, such as faithfulness, stability, consistency, and sparsity (Liu et al. 2021 Petsiuk et al.,
2018 Slack et al.] 2021 Zhou et al. 2021 Metrics have also been proposed to quantify these aspects of
explanation quality (Zhou et al. 2021 Carvalho et al. 2019: Gilpin et al. 2018 Liu et al. 2021). These
propertics and metrics have been leveraged to thoorctically and empirically analyze the bohavior of popular
post hoc explanations (Ghorbani al. 2019 Slack et. al. 2020 Dombrowski et al. 2019 Adebayo al.,
2018 Alvarez-Melis & Jaakkola| 2018 Levine et al. 2019 Chalasani et al.) 2020 Agarwal et al. 2021 The work that is closest to ours is the research by Neely et al. (2021) which demonstrates that certain post hoc explanation methods (e.g., LIME, Integrated Gradients, DecpLIFT, Grad-SHAP, Deep-SHAP, and altention-basedl explanations) disagree with each other based on rank correlation (Kendall's r) metric. However, their work neither formalizes the notion of explanation disagreement by leveraging practitioner inputs nor studies how explanation disagreements are resolved in practice, which are the key contributions of this work. This research has also inspired subsequent works that delve deeper into this topic (Han et al. 2022 Banegas-Tann et al. 2023). For instance, Han et al. (2022) built on this work and theoretically analyzed multiple state-of-the-an art explanation methods to understand the reasons behind their disagreements.
Human Factors in Explainability. Many user studies have been conducted to evaluate how well humans can understand and use explanations (Doshi- Velez & Kim 2017). Some of these studies have shown that data scientists often struggle to understand and effectively leverage ttate-of-tho-art explanation techniques (Kaur et al. 20200- Others have identified variety of stakeholders across the model life eycle and highlighted the core goals of model explanations (Hong et al.] 2020 Chen et al.] 2022). Prior works have also examined
whether explanations help users in performning specific tasks (Buçinca et al. 2021 Bansal et al. 2021 Fan
lt al.) 2022; Sperrle et al. 2021 Chen et al.] 2022 Vasconcelos et al.) 2023 Fok & Weld 2023). However, none of these works focus on understanding the disagreement problem, the extent to which practitioners face it, and/or how they resolve it.
3 Understanding and Measuring Disagreement Between Model Explanations
In this section, we discuss practitioner perspectives on what constitutes disagreement between two explanations and then formalize the notion of explanation disagreement. To this end, we first describe the study that we carry out with data scientists to understand what constitutes explanation disagreement, and the extent to which they encounter this problem in practice. We then discuss the insights from this study and leverage these insights to propose novel framework that can quantitatively measure the disagreement between any two explanations.
3.1 Characterizing Explanation Disagreement Using Practitioner Inputs
Here, we describe the interviews we conducted with data scientists characterize explanation disagreement and outline findings and insights from these interviews.
3

---

Published in Transactions on Machine Learning Research (06/2024)
3.1.1 Interviews with Data Scientists
We conducted 30-minute semi-structured interviews with 25 data scientists who employ explainability techniques to understand model behavior and explain it to their customers and managers. We recruited these participants by emailing the explainable machine learning groups (and relevant mailing lists) in three different for-profit companies in the technology and financial services sectors across the United Stales. We conducted basic checks (e.g., pre-screening interviews and profile checks) ensure that each recruited participant has at least one year of experience data scientist, and has working knowledge of data science and machine learning.
Furthermore, all of these data scientists used state-of-the-art local post hoc explanation methods (e.g., LIME, SHAP, and gradient-based methods) in their day-to-day workflow. 19 participants (76%) were male, and 6 (24%) were female. 16 participants (64%) had more than years of experience working with explainability techniques, and the remaining 9 (36%) had about 8 to 12 months of experience. Our interviews included, but were not limited to the following questions: Q1) How often do you use multiple explanation methods to understand the same model prediction? Q2) What constitutes disagreement between two explanations that explain the same model prediction?
Q3) How often do you encounter disagreements between explanations output by different methods for the same model prediction!
3.1.2 Findings and Insights
The interviews revealed how data scientists use explanation methods and their perspectives on what constitutes disagreement among explanations. 22 out of the 25 participants (88%) said that they almost always use multiple explanation methods to understand the same model prediction. Furthermore, 21 out of the 25 participants (84%) mentioned that they have often run into some form of disagrecment between explanations generated by different methods for the same prediction. We found that data scientists consider explanations to disagree when:
Top features are different: Most popular post hoc explanation methods return feature importance values for each feature. These values indicate which features contribute most (either positively or negatively) to the prediction, i.., top features. 21 out of the 25 participants (84%) mentioned that such a set of top features is 'the most critical piece of information that they roly in their day-to-day workflow. They also noted that they typically look at the top 5 to 10 features provided by explanation for each prediction. When two explanations have different sets of top features, they consider it to be disagreement.
Ordering among top features is different: 18 out of 25 participants (72%) indicated that they also consider the ordering among the top features very carefully in their workflow. Therefore, they consider a mismatch in the ordering of the top features provided by two different explanations to be disagreement.
Direction of top feature contributions is different: 19 out of 25 participants (76%) mentioned that the sign or direction of the feature contribution (i.c., whether the feature contributes positively or negatively to the prediction) is another critical piece of information. Any mismatch in the signs of the top features between two explanations is a sign of disagreement. As one participant remarked: "I saw an explanation indicating that a top feature bankruptcy contributes positively to a particular loan denial, and another explanation saying that contributes negatively. That a clear disagreement. The model prediction can be trusted with the former explanation, bul not with the laller.".
Relative ordering of features of interest is different: 16 of our study participants (64%) indicated that they also look at relative ordering between certain features of intcrest, and that, they consider explanations to disagree if explanations provide contradicting information about this aspect. As one participant remarked: "I often check if salary is more important than credit score in loon approvals. If one explonation says salary is more important than credit score, and another says credit score more important than salary; then it is a disogreement."
4

---

Published in Transactions on Machine Learning Research (06/2024)
A striking finding from our study is that participants characterize explanation disagrecment based on factors such mismatch in top features, feature ordering, directions of feature contributions, and relative feature ordering, but not on the feature importance values output by different explanation methods. 24 out of 25 participants (96%) opine that feature importance values output by different explanation mothods are not directly comparable. For example, they note that while LIME outpuls coefficients of a linear model as feature importance values, SHAP outputs Shapley values as feature attributions which sum to the probability of the predicted class. Thus, data scientists do not try to base explanation disagreement on these numbers not being equal or similar. One of our participants succinctly summarized practitioners' perspectives on this explanation disagreement problem: "The values generated by different explanation methods are different. So, would not characterize disagreement based on that. But, I would at least went the explanations they output to give me consistent insighls. The explanations should agree on what are the most imporland features, the ordering among them, and so on for me to derive consistent insights. But, they don't!"
3.2 Formalizing the Notion of Explanation Disagreement
Our study indicates that ML practitioners consider the following key aspects when they think about explanation disagreement: a) the extent to which explanations differ in the top-k features, the signs of these top-k ueatures, and the ordering of these top-k features, and b) the extent to which explanations differ in the relative ordering of certain features of interest. To capture these intuitions about explanation disagreement, we propose six different metrics: feature agreement, rank agreement, sign agreement, signed rank agreement, rank correlation, and pairwise rank agreement. The first four metries capture disagreement with respect to the top-k features of the explanations and the last two metrics capture disagreement with respect to selected set of features that could be provided as input by an end user. For all six metrics, lower values indicate stronger disagreement. See Section A for formal definitions.
3.2.1 Measuring Disagreement With Respect to Top-k Features
We now define four metrics, which capture specific aspects of explanation disagreement with respect to the top-k features
ML practitioners in interviews (Section 3.1) indicated that a key notion of disagreement between pair of explanations is that they output different top-k features. To capture this notion, feature agreement computes the fraction of common features between the sets of top-k features of the two explanations.
Feature Agreement:
Rank Agreement: Practitioners also indicated that if the ordering of the top-k features is different for two explanations (even if the feature sets are the same), then they consider it to be a disagreement. To capture this notion, rank agreement computes the fraction of features that are not only common between the sets of top-k features of two explanations, but also have the same position in the respective rank orders. Rank agreement is a stricter metric than feature agreement since it also considers the ordering of the top-k features. While the practitioners we interviewed favored the aforementioned definition of the rank agreement metrie, one could also imagine a less strict variant that accounts for the differences between the ranks of the top-k features in the two explanations. Such a weighted rank agreement metric would quantify the degree of agreement based on how far apart the top-k features are, rather than only considering an exact match of their ranks. This implies that even if certain features are not in the same position, their relative closeness still contributes positively to the overall agrecment score, making it a softer variant of the rank agrcement defined above. See Section D.2 for more details on weighted rank agreement and additional disagreement metries that we explored.
Sign Agreement: In our study, practitioners also mentioned that they consider two explanations disagree if the feature attribution signs (i.e. directions of feature contribution) do not align for the top-k features. To capture this notion, sign agrecment computes the fraction of features that not only common
2he t features of an explanation are typically computed only based the magnitude of the feature importance values and not the signs.
5

---

Published in Transactions on Machine Learning Research (06/2024)
between the sets of top-k features of two explanations, but also share the same sign in both explanations. Sign agreement is stricter metric than feature agreement since it also considers signs of the top-k features.
This metric fuses the above three notions of explanation disagreement and computes the fraction of features that are not only common between the sets of top-k features of two explanations but also share the same feature attribution sign and rank both explanations. The signed rank agreement is the strictest compared to all the aforementioned metrics since considers both the ordering and the signs of the top-k features.
Signed Rank Agreement:
3.2.2 Measuring Disagreement With Respect to Features of Interest
Practitioners also indicated that they consider two explanations to be different if the relative ordering of features of interest. (e.g,, salary and credit score discussed in Section 3.1) differ ween the explanations. These features could be provided as imput by an end user (such as data scientist). To formalize this notion, we introduce the two metrics below.
Rank Correlation: In practioe, this selected set of features corresponds to features that are of interest end users and can be provided as input by end users. Rank correlation computes Spearman's rank correlation coefficient between feature rankings provided by two explanations for these selected sets of features.
The pairwise rank agreement takes as input set of features that are of interest to the user and computes the fraction of feature pairs for which the relative ordering is the same between two explanations.
Pairwise Rank Agreement:
4 Empirical Analysis of Explanation Disagreement
We leverage the metrics outlined in Section 3 and carry out comprehensive empirical analysis with six state-of-the-art explanation methods and four real-world datasets spanning three data modalities to study the explanation disagreement problem. In this section, we describe the datascts, experimental setup, and key findings.
4.1 Datasets
To carry out empirical analysis, use four datasets spanning three data modalities (tabular, text, and image). For tabular data, we use the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) (ProPublica) and German Credit datasets [epository). The COMPAS dataset comprises seven features on the demographics, criminal history, and prison time 4,937 defendants. Each defendant is classified as either low or high risk for recidivism based on the COMPAS algorithm's 's risk score. The German Credit datasct contains twenty features on the demographics, credit history, bank account balance, loan information, and employment information of 1,000 loan applicants. Each applicant classified as either low high risk for defaulting
loan. For text data, use Antonio Gulli (AG)'s corpus of news articles (AG [ews) (ag] 2005 It contains 127,600 sentences (collected from 1,000,000+ articles from 2,000+ sources with a vocabulary size of 95,000+ words). The class label is the topic of the article from which a sentence was obtained (World, Sports, Business, or Science/ Techholoyhhh.h......o For image data, we use the ImageNet-11k 1k (Russakovsky et al. 2015 ima 2015) object recognition dataset. It contains 1,381,167 images belonging to 1000 categories. We obtain segmentation maps from PASCAL VOC 2012 (voc 2012) that are directly used super-pixels for the explanation methods.
4.2 Experimental Setup
We train a variety of black box models. For tabular data, we train four models: logistic regression, densely connected food-forward neural network, random forest, and gradient-boosted tree. For text data, we train a AG_News widely-used vanilla LSTM-based text classifier on AG News ŞZhang et al. 2015) corpus. For image data, we
use the pre-trained ResNet-18 (He et al. 2016) for ImageNet.

---

Published in Transactions on Machine Learning Research (06/2024)
Next, we apply six state-of-the-art post hoc explanation methods to explain the black box models' predictions for a set of test data points We apply two perturbationn-ased explanation methods (LIME (Ribeiro et al., 2016b) and KernelShap (Lundberg & Lee 2017)), and four gradient-based explanation methods (Vanilla Gradient (Simonyan ct al. 2014). Gradient times Input (Shrikumar et al. 2017). Integrated Gradients (Sundararajan el al. 2017); and SmoothGrad (Smilkov et al. 2017)). For explanation methods with a sample size hyper-paramcter, we cither run the explanation method to convergence (i.c., select a sample size such that an increase in the number of samples does not significantly change the explanations) or use sample size that is much higher than the sample size recommended by previous work.
We then evaluate the (dis)agreement between the explanation methods using the metries described in Section 3.2 For tabular and text data, we apply rank correlation and pairwise rank agreement across all features; and feature agreement, rank agreement, sign agreement, and signed rank agreement across top-k features for varying values of k. For image data, metrics that operate on the top-k features are more applicable to super-pixels. Thus, we apply the six disagreement metrics on explanations output by LIME and KernelSHAP (which leverage superpixels), and calculate rank correlation (across all pixels features) between the explanations output by gradient- based methods. Additional details are described in Appendix
4.3
Results and Insights
We discuss the results of our empirical analysis for each of the three data modalities.
4.3.1 Tabular Data
Figure shows the disagroement between various pairs of explanation methods for the neural network model trained on the COMPAS dataset. We computed the six metrics outlined in Section 3.2 using k 5 (out of 7 features) for metrics that focus on top-k features. Each cell in the heatmap shows the metric value averaged over the test data points for each pair of explanation methods, with lighter colors indicating stronger disagrcement. We see that explanation mothods tond to exhibit slightly higher values on pairwise rank agreement and feature agreement metrics and relatively lower values on other metrics (indicating stronger disagreement).
We next study the effect of the number of top features the degree of disagrocment. Figure 2 shows the disagreement of explanation methods for the neural network model trained on the COMPAS dataset. We computed the rank agreement (top row) and signed rank agreement (bottom row) at top-k features for increasing values of k. We see that as the number of top-k features increases, rank agreement and signed rank agreement decrease. This indicates that, as & increases, top-k features of a pair of explanation methods are less likely to contain shared features with the same rank (as measured by rank agreement) or shared features with the same rank and sign (as mcasured by signed rank agrcoment). These patterns are consistent across other models trained on the COMPAS dataset. See Appendix C.I
In addition, across metrics, values of k and models, the specific explanation method pairs of Vanilla Gradient- SmoothGrad and Gradient times Input-Integrated Gradients exhibit strong agreement, while the pairs of Gradients, Vanilla Gradient-Gradien times Input, Srrooiinrrddrrrrddienn times
Vanilla Gradient-Integraake Input, and SmoothGrad- ArooottGrrad-Integrraeeed Gradients consistently exhibit stronger disagreement. These results are consistent with the theoretical findings indicating that certain pairs of explanation methods are more consistent with one another than other pairs (Han et al., 2022).
Furthermore, there are varying degrees of disagroement among pairs of explanation methods. For example, for the neural network model trained on the COMPAS dataset, rank correlation displays wide range of values across explanation method pairs, with 14 out of 15 explanation method pairs even exhibiting negative rank correlation when explaining multiple data points. This is shown in the left panel of Figure D which displays the rank correlation over all the features among all pairs of explanation methods for the neural network model trained on the COMPAS datasel.
We also studied another explanation hod called L,X
Chen al. 2018 See Section D.1 for more details.
7

---

Published in Transactions on Machine Learning Research (06/2024)
Rank correlation
Painwise rank agreement
Feature agreement (K B)
0.719
0.467
LIME
1.000
0:707
0.619
0.635
0.704
LIME
1,000
0.002
0.782
SHAP
0.719
1.000
0.395
0.389
0.423
0.345
SHAP
0.807
1.000
0.660
0.616
0.629
0.638
SHAP
0,853
1,000
0.788
0.796
0. 0.796
0.780
Grad
0.523
0.386
1.000
0.359
0.395
0.783
Grad
0.707
0.660
1.000
0.616
0.627
0. 8.89
Crad
0.822
0.788
1.000
0.781
0.778
0.823
Grwe
0,413
0.389
0.359
1,000
0.909
0.363
0.811
0.611
0:816
1.000
0.942
0.949
0.002
0.796
0.791
1.000
0.980
0.781
Intäred
0.467
0.423
0.305
0.909
1.000
0.432
IntGGaad
0.635
0.628
0.627
0.942
1.000
0. 8888
IntGradd
0.798
C.706
0.778
0.950
1.000
0.758
Smooth
0.345
0.783
0:432
Smooth Grad
0.704
0.649
1.000
Smooth Grad
0.782
C.780
0.761
1.000
Smooth
SHAP
SHAP
Grad
SHAP
Ingut
Rank agr.ementt
5)
Sign agreement (k 5)
Signed rank agneement (k 5j
1.0
LINE
1.000
0.511
0.207
0.203
0.251
0.330
LIME
1.000
0.578
0.351
0.808
0.805
0.308
LIME
1.000
0.177
0.111
0.173
0.221
0.202
Karel
SHAP
1.000
0.189
0.249
Kemel SHAP
0.678
1.000
0.448
0.436
Kernel SHAP
0.177
1.000
0.088
0.147
0.142
C.8
Grod
0,207
0.188
1.000
1.134
0.152
0.386
Grad
0.351
0.448
1.000
0,200
0.205
0.823
Grod
0.111
0.088
1.000
0.064
0.076
0.388
Grac Input
0.203
0.210
0.134
1,000
0.682
0.182
Grad" Input
0.608
0.410
0.200
1.000
0.939
0.202
Graad Impul
0.173
0.147
0.064
1,000
0.646
0.111
0.251
0.249
0.152
0.682
1.000
0.221
0.605
0.407
0.205
0.539
1.000
0.217
IntCred
0.221
0.180
0.076
0.646
1.000
0.151
Smooth Grad
0.330
0.308
0.386
0.182
0.221
1.000
Smooth
Grad
0.308
0.435
0.823
0,202
0.217
1.000
Smooth
Grad
0.202
0.142
0.386
0.111
0.151
1.000
-0.0
LIME
Kemel SHAP
Crad
Grad* Inpul
IntGred
Smrooth Grad
LIME
Kamei SHAP
Grad
Gred" Incul
IntGrad
Smooth Grad
LIME
Kernel
Grad
Grad"
IntGrad
Smooth Grod
Input
Figure 1: Disagreement between explanation methods for neural network model trained on COMPAS dataset measured by six metrics: rank correlation and pairwise rank agreement across all features, and feature, rank, sign, and signed rank agreement across top k == 5 features. Heatmaps show the average metric value over the lest set data points for each pair of explanation methods, with lower values (lighter colors) indicating stronger disagreement. The disagreement between a pair of explanations is considered non-trivial if less than 75% of the features agree (i.e., if feature agreement, rank agreement, sign agreement, signed rank agreement, or pairwise rank agreement less than 0.75), or if the correlation among features only moderately positive, nonexistent, or negative (i.e. if the rank correlation is less than 0.50). Across all six heatmaps, the standard error ranges between 0 and 0.01.
All the patterns discussed above are also generally reflected the German Credit dataset (Appendix C.1 However, explanation methods sometimes display stronger disagreement for the German Credit dataset than for the COMPAS datasct. For example, rank agreement and signed rank agrcement Credit dataset than for the COMPAS dataset at the top 25%, 50%, 75%, and 100% of features for both logistic regression and neural network models. One possible reason is that the German Credit dataset has a larger set of features than the COMPAS dataset, resulting in a larger number of possible ranking and sign combinations assigned by a given explanation method and making it less likely for two explanation methods to produce consistent explanations.
lower for the German
Moreover, explanation methods display trends associated with model complexity. For example, the dis- agreement between explanation methods is similar or stronger for the neural network model than for the logistic regression model across metrics and values of k for both COMPAS and German Credit datasets (Appendix C.1 In addition, explanation methods show similar levels of disagrecment for the random forest and gradienl-boosted tree models. These trends suggest that disagreemeent among explanation methods may increase with model complexity. As the complexity of the black box model increases, it may be more difficult to accurately approximate the black box model with a simpler model (LIME's strategy, for example) and more difficult to disentangle the contribution of each feature to the model's prediction. Thus, the higher the model complexity, the more difficult it may be for different explanation methods to generate the true
8

---

Published in Transactions on Machine Learning Research (06/2024)
Rank agreement (k
Raink agreement (k k.44
Rank agreement 7)
1.000
0.431
0.338
0.400
0.574
0.520
0.214
0.215
0.356
1,000
0.370
0.251
0.168
Kemel
SHAP
1.000
0.400
0.573
Komel
SHAP
1.000
0.201
0.238
Kemel
SHAP
1.000
0.218
0.260
Grod
0.338
0.400
1.000
0.301
0.361
0.407
Grad
0.214
0.201
1,000
0.147
0.164
0.432
Grod
0.251
0.215
1.000
0:107
0.120
0.366
Groe
0.400
0.468
0.301
1.000
0.740
0.563
Grod"
0.215
0.234
0.147
1.300
0.638
0.214
Inpul
0.168
0.178
0.107
1.000
0.765
0.144
0.561
Intrrad
0.275
0.284
0.164
1.000
Infrred
0.785
1.000
0.170
Smooth
Grod
808
0.573
0.553
0.736
1.000
Smooth Gma
0. 366
1.206
0.432
0.259
Smooth Grod
0.284
0.280
0.385
0.170
SHAP
Inpur
Grad
SHAP
Input
Giad
SHAP
Input
Grad
Signed rank agraament (k 1)
Signad rank agneement (K 4)
Signed
agreament (K- 7)
1.0
1,000
0.144
0.330
0.400
0.574
0.188
0.117
0.187
0,252
0.234
LIME
1,000
0.222
0.128
0,129
0.181
0.168
Kernel SHAP
1.000
0.184
Kernel
1.000
0.091
SHAP
0.222
1,000
0.080
0.107
0.117
0.116
Grad
0.338
0.164
1.000
0.229
0.291
0.407
Grad
0.117
0.081
1,000
0.080
0.095
0.432
Grad
0.126
0.090
1.000
0.048
0.055
0.365
008
0.400
0.422
0.229
1.000
0.740
0.563
0.187
0.174
0.080
1,300
0.811
0.139
Grac
0.674
0.291
0,222
0.180
0.095
1.000
0.189
Intrrad
0.117
0.066
Smooth Grad
0.285
0.407
0.663
1.000
Smooth Gred
0.234
0.432
1,000
0.168
0.118
0.385
0.060
0.108
1,000
Grad
-0.0
Gred
Drad" Inpur
Kamel SHAP
Grad
Smooth Glod
Kemel
Grad"
Smooth
SHAP
Grad
Input
Figure 2: Disagrcement between explanation methods for neural network model trained COMPAS dataset measured by rank agreement (Lop row) and signed rank agreement (bottom row) at top-k features for increasing values of k. Each cell in the heatmap shows the metric value averaged over test set data points for each pair of explanation methods, with lower values (lighter colors) indicating stronger disagreement. We follow the same disagroement interpretation described in Figure error ranges between 0 and 0.01.
Across all six heatmaps, the standard
explanation and the more likely it may be for different explanation methods to generate differently false explanations, leading to stronger disagreement among explanation methods. Finally, we also study the effect of dataset complexity (i.c., the number of features) on the degree of disagreement for each of our metrics. See Section D.3 for more details.
Rank correlation
Rank correlation
Signed agreement
1.0
Sigasl ark app0rvent
Dant coneetton
0.50
8.50
0.8
0.25
0.25
0.6
0.00
0.00
-025
0.4
-0.50
02
-4.75
-1.00
0.
LIVE Goal
LIKE 1s Graftrpp
LMC on Infire-
Kamabiii Goad
Come SHiD on intronl
Goo Gractrpe
Grd on, Iitrns
Gool vs Smotthronl
Intccod vs SmattGol
LIPE Teme
LNC (e-V-.00 God
Gattron roin
LIN rodedd
UHE vs Smootord
kone Shea vo. Grdd
Rone bror vs oas reu
Grat in Crech reu-
Ord as It
God SmooiCIee
Grechies rnGras
Crattipp vs Smouucivv
Inttred v. Smonnooo
UPP vs. NemeSrur
LINE Vs Grat-
LRNE va Grect rex-
LIE risedd
LSKE u. SmarnGoo
tune rs Gnd
God Guazt rpu--
God niod
Grd Smazicavo
Grsprroi vs rntadd
Grattrrnn Grondon
us,
FLB of expla mechoos
Par eollanaionnnnnnooooooooo mothods
Par collanaionnn mothods
Figure 3: Distribution of rank correlation over all features for neural network model trained on COMPAS (left), and rank correlation across all features (middle) and signed rank agreement across top-11 features (right) for neural network model trained on AG News.

---

Published in Transactions on Machine Learning Research (06/2024)
4.3.2 Text Data
Rank correationn
Painvise rank agreement
Fasturs agreementt (K
11)
LIME
1.000
0.120
0.340
0.303
0.265
0.341
LIME
1,000
0.538
0.538
0.531
0.589
0.456
LIME
1.000
0.324
0.447
0.448
0,416
0.4C5
Komd
SMAP
0.120
1,000
0.100
0.081
0.194
0.071
8HAP
0.538
1.000
0,534
0.529
0.088
0.024
9HAP
0.334
1,000
0:333
0.329
0.413
0.324
Grad
0.340
0.100
1.000
0.597
0.289
0.731
Grad
0.538
0.534
1,000
0.720
0.607
0.794
Glod
0.447
0.553
1.000
0.819
0.445
0.660
Grad7 Input
0.303
0.081
0.597
1.000
0.236
0:477
Greer
0.534
0:529
0.720
1.000
0.582
0.071
Gradf irput
0.448
0.329
0.542
Intrrad
0.285
0.194
0.289
0.236
1.000
0.186
0.589
0.682
0.635
InGGmd
0.416
1,000
GRAD
0.341
0.071
0.731
0.477
0.186
1.000
Smooth
Smooeh
GRAD
0.485
0.524
680
0.542
0.405
1.000
LIM
SHAP
Gond
Gradt
InfGred
Smonth
LIME
Karnel SHAP
Grad
Gridd Imput
InCredd
Emooth GRAD
SHAP
Inpul
GRAD
Rank agraement (k==
agreement
11)
S gned rank wemeent
1.000
0.275
0.216
0.223
0.018
0.026
0.031
0.033
0.041
Kamee
1.000
0.034
0.034
0.033
Komet SHAP
1.000
0:180
0.215
Kamel
0.019
0.8
Grod
0.049
0.034
1,000
0.081
0.052
Grad
0.275
0.180
1,000
0.332
0.264
0.660
Grad
0.035
0.018
1.000
0.049
0.031
0.123
0.6
Grad" Trpput
0.054
0.004
0.091
1.000
0.054
0.075
0.216
0.215
0.332
1,000
0.366
0.296
Grac Inpor
0.031
0.023
0.049
1.000
0.047
0.041
Inpur
Intrrad
0 053
0.060
082
0.044
1.000
0.046
InXSrad
0.223
0.327
0.284
0.386
1.000
0.265
InGGrad
0.033
0.068
0.021
0.647
1,000
0.031
-02
GRAD
0.054
0.033
0.123
0.075
0.046
1.000
Smooth
0
0.680
0.258
0.265
Smooth
0.041
0.020
0.123
0.041
0.031
1.000
Kemel
Goad
Grad"
InnGred
Smarth
LIME
Kerne SHAP
Grad
Grad" Inpul
Intrred
Smooth GRAD
LIME
Kemel SHAP
Grad
Grad" Ieput
Smooth IntGrad
GRAD
Figure 4: Disagreement between explanation methods for LSTM trained on the AG News dataset using k = 11 features for metrics operating on top-k features and all features for other metrics. Each heatmap shows the metric value averaged over test data for each pair of explanation methods. Lighter colors indicate more disagrecment. Standard error ranges from 0.0 to 0.0025 for all six metrics.
In the of text data, we deal with high-dimensional feature space where words are features. We plot the six metrics for k = 11, which is around 25% of the average text length of a sentence (data point) in the dataset (Figure 4. As seen in the figure, we observe severe disagreements across all six disagreement metrics. Rank agreement and signed rank agreement are the lowest between explanations, with values under 0.1 for most cases, indicating disagrecment in over 90% of the top-k features. Tronds are quite similar for rank correlation and feature agreement., with better agreement between gradient-based explanation methods, such as feature agreement of 0.61 for the Gradient times Input and Vanilla Gradient method pair.
In addition to the broader disagreements between various explanation methods, notice specific patterns of agreement, between certain explanation methods. Based on the middle and right panels of Figure we observe a higher rank correlation between different pairs of gradient-based explanations. We also obscrve from Figure g that LIME exhibits higher agreement with other explanation methods compared to KernelSHAP (e.g., Figure 4 shows that the average rank correlation between LIME and all other explanation methods is 0.273, as opposed to 0.113 in case of KernelSHAP). This finding consistent with the insights we observed in the case of the COMPAS dataset (Figure [ top left panel). Lastly, we notice higher disagreement among explanation methods for text data than for tabular data.
4.3.3 Image Data
While LIME and KernelSHAP consider superpixels of images as features, gradient- based methods consider pixels as features. In addition, the notion of top-k features and the metrics we define on top-k features are not semantically meaningful when we consider pixels as features. We compute all six metries capture disagreement between explanations output by LIME and KernelSHAP with superpixels features. We also
10

---

Published in Transactions on Machine Learning Research (06/2024)
compute rank correlation on all the pixels (features) to capture disagreement between explanations output by gradient-based mothods.
Unlike earlier trends with tabular and text data, we see higher agreement between KernelSHAP and LIME on all six metrics: rank correlation of 0.8977, pairwise rank agreement of 0.9302, feature agreement of 0.9535, rank agreement of 0.8478, sign agreement of 0.9218, and signed rank agreement of 0.8193.
However, the trends are quite the opposite when we compute rank correlatton at pixel-level for gradient- methods (See Appendix C For instance, the rank correlation between Integrated Gradients and SmoothGrad is 0.001, indicating strong disagreement. The disagreement is similarly quite strong in the case of other pairs of gradient- based methods. This suggests that disagreement could potentially vary significantly based on the granularity of image represenlation.
Remark 1. Throughout. this section, we carried out a comprehensive empirical analysis with sie explonation methods and four real-world datasets spanning three data modalities to study the disagreement problem. While our results generally show a moderate to high degree of disagreement among all pairs of explanations across all data modalities, we observe that the exlent of disagreement is slighlly lower when we consider variants of the proposed metrics. For ezample, we explored additional metrics such as weighted rank agreement, top-k pairwise ronk agreement, and top-k rank correlation, and we discuss these metrics and corresponding results in Appendir D.
5 Resolving the Disagreement Problem in Practice: A Qualitative Study
After establishing the fact. that the majority of practitioners observe disagreement between explanations using both qualitative analysis (Section 3 as well as quantitative analysis (Section + we try to understand the different ways practitioners tackle this disagreement. To this end, we conduct qualitative user study targeted towards explainability practitioners. We next describe our user study design and discuss our findings.
5.1 User Study Design
In total, 25 participants participated in our study, 13 from academia and 12 from industry. Participants from academia were graduate students and postdoctoral researchers. In contrast, participants from industry were data scientists and ML engineers from three different for for-profit companies in the technology and financial services sectors across the United States. We recruited these participants by emailing the explainable ML groups (and relevant mailing lists) within these organizations. 12 participants in this study (48%) also participated in the interviews in Section 3.1 We conducted basic checks (e-g., pre-screening interviews and profile checks) to ensure that each recruited participant has at least one year of experience as a data scientist, and has working knowledge of data science and ML. 20 of these participants indicated that they had used explanation methods in their work in various ways, including doing research, helping clients explain their models, and debugging their own models. Following the setup in Section 4 we asked participants to compare the output of five pairs of explanation methods on the predictions made by the neural network we trained on the COMPAS dataset. We chose the COMPAS dataset because it only has 7 features, making it easy for participants to understand the explanations.
First, the participants are shown an information page explaining the COMPAS risk score binary prediction setting and various explanation methods. We indicate that we trained a neural network to predict the COMPAS risk score (low or high) from the seven COMPAS features. We also give a brief description of each of these seven features to participants and tell them to assume that the criminal defendant's 's risk of recidivism is correctly predicted to be high risk. In this information page, we also briefly introduce and summarize the six explanation methods we use in the study (LIME, KerneISHAP, Vanilla Gradient, Gradient times Input, SmoothGrad, and Integrated Gradients). Finally, we provide links to the papers describing cach method. We include a screenshot of this information page in Appendix E.1
Next, participants were shown a series of prompts, a sample of which is shown in Figure 5 Each prompt presented two explanations of the neural network model's prediction corresponding to a particular data
4The remaining participants from the interviews were unavailable during the time of the second study.
11

---

Published in Transactions on Machine Learning Research (06/2024)
point generated using two different explanation methods (e.g., LIME and KernelSHAP in Figure The explanations (and data points) within these prompls were selected to ensure high disagreement between them as per the metries discussed in prior sections. We displayed the full set of k = COMPAS features, showing the feature importance of each feature. The red and blue bars indicate that the feature contributes negatively and positively, respectively, to the model prediction. Participants were first asked the question "To what extent do you think the two explanations shown above agree or disagree with each other?" and given four choices: completely agree, mostly agree, mostly disagree, and completely disagree. If the participants indicated any level of disagreement (i.e., any of the latter 3 choices), we then asked them "Since you believe that the above explanations disagree (to some extent), which explanation would you rely on!" and presented them with three choices: the two explanation methods shown and "it depends" ". Then, they were asked to explain their response. Participants were allowed to take as much time as they needed.
Below, you see data point, well as explanation using mathods LIME and KerneSHHA..
LINE
KameltHAP
meo_yeer_eed
twe, wwo,ye_,yee mndd
prien count:
prions_cwet
Imoth Ltay
lenpth
depree
Peruee
180
r0ce
-0.38
0:00
0:10
-0.02
-0.01
8.00
0.51
0.05
feature importance
Festune Importance
As reminder, the features of the COMPAS dalaset are age. two, year year_recid (whather the defendant recidivated alter years of the original crime, priors count (number of prior arines commilledd) length_ (lengih the dafendani slayed jail), charge (whether
Misdemeanor Felany),
To what extent do think the two explanations shown above disagree with each other?
C Completely agree # Mostly agree Mostly disagree C Completely disagres
Please explain why you chose the acove answer:
Since you belleve the above explanations disagree (to some extent), which explanation would rely on?
UME Kemel5HAP explanation dopands
Please explain chuew the
Figure 5: The user interface for a prompt. The user is shown two explanations for a COMPAS data point, showing the feature importance value of each of the features. Red and blue indicate negative and positive feature values, respectively. See the text for more details.
5.2 Results and Insights
We now discuss the results and findings from user study.
5.2.1
Do practitioners observe disagreements?
We aggregated the responses to the first question in each prompt, "To what extent do you think the explonations shown above agree or disagree with each other?". Overall, 4%, 28%, 50%, and 18% of responses indicate completely agree, mostly agree, mostly disagree, and completely disagree, respectively, highlighting that there is significant disagreement among prompts. See Appendix E.4 for more details.
5.2.2
Are certain explanations favored over others?
Next, since different explanation methods have different levels of popularity, we analyze if certain methods are chosen more often in disagrocments. Figure (a shows the distribution of how participants resolved disagroements for each prompt (dropping prompts with 4 or fewer responses). We first emphasize that there is high variability in how participants chose to resolve disagreements, showing lack of consensus for the
12

---

Published in Transactions on Machine Learning Research (06/2024)
majority of prompts. However, when participants do decide to choose method rather than abstaining, they often choose the same method. For example, for the Vanilla Gradient vs. SmoothGrad pair (top row in Figure Gaj, participants either chose SmoothGrad over Vanilla Gradient or chose neither. We also aggregate these choices' overall prompts, and in Figure ob we plot how often cach of the six explanation methods is chosen, finding that indeed, certain hothods were favored over others. While KernelSHAP was chosen 66.7% of the time when there were disagrecments, Gradient times Input was only chosen 7.0% of the time. We include further explanation of why participants chose each of the explanations in Appendix E.5 including quotes from participants that supported each method.
Frequency (%) algorithms picked prompt
Frequency (%) algorithms picked disagreements
(Grad, SG)
70
(G*) (G)
60
(LIME, (G)
pair of allgrrthms
(LIME, G1)
50
(SHAP. 690
frequency
40
(SHAP. (G)
right nelther
(LIME Grad]
30
SG)
20
(UIE,, SHAP)
(SHAP, Grad)
10
(Grad, IG)
0
LIME
SHAP
frequency (%)
explommaiility algonthm
(a) The frequency with which each of the explanations in pair is selected upon disagreement. The blue, gold, and grey hars show the percentage of participants (X-axis) that picked the left, right, and neither method when presented with the pair of methods shown the Y-axis.
(b) The frequency with which each of the expla- nations was chosen when there is disagreement. X-axis indicates the explanation method and Y- axis indicates the frequency.
Figure 6: Figures highlight which methods participants chose when the explanations they were shown disagreed. In (a), we show how participants resolved each particular prompt. In (b), we show the overall frequencies with which an explanation method was selected.
5.2.3 How do practitioners resolve disagreements?
Across all six explanation methods, we find three unifying themes that dictated why participants chose one explanation over the other. We give high-level description of those themes below, highlighting direct quotes from participants in Table
1. One method is inherently better than the other because of its associated theory or publication time (33%): Participants often indicated preference towards a particular method without referencing the shown explanation citing features such as the paper's publication time (more recent is better), the theory behind the method, and the method's stability.
2. One of the generated explanations matches intuition better (32%): Participants frequently said that one method's explanation aligned with their intuition better, citing the absolute and relative values of specific features evidence.
3. LIME and SHAP are better because the COMPAS datasct comprises tabular data (23%): Participants said that they mainly used LIME and SHAP for tabular data and commonly cited this as their sole reason.
5.2.4 Experiencing and resolving disagreements in day-to-day workflow
After answering all 5 prompts, participants were asked set of questions on their experience with the disagrecment problem in their day-to-day work. First, to filter out participants who didn't use explanation methods, we asked: "Have you used erplanation methods in your work before?". Of the 25 participants, 5 of them indicated that they had not. We asked the other 20 participants further questions to better understand their experience with the disagrcement problem. The full set of questions can also be found Appendix
E.3 Having understood what participants look for to determine disagreement in Section 3.1 we next sought to better understand two crucial questions related to the disagreement, problem: (Q1): Do you observe
13

---

Published in Transactions on Machine Learning Research (06/2024)
Theme Highlighted
Sample Quotes
1. One method's
"I have no reason to believe the gradient holds anyohere other than very locally."
paper/ /therry suggests that it's inherently better (33%).
"[Integrated Grodients is] more rigorous fthan SmoothGrad] based on the paper and arioms"
"gradient explanations are more unstable"
2. One explanation matches intuition better (32%).
"scems unlikely that all features contributed to positive classification" "Jealures such as priors count and length of stay [ure] imporlant for determåning"
"Gradient *Input only consider(s} sensitive features (age, race) as im- pactful which could be a sign of a biased underbying data distribution"
3. LIME/SHAP are better for tabular data (23%).
"I use LIME for structured data" "
"SHAP more commonly used fthan Vanilla Gradient] for tabular data" "
Table 1: Themes summarizing how participants decided between explanations when faced with disagreement along with quotes.
disagreements between explanations output by state-of-the-e the- methods in your day-to-day workflow? and (Q2): How do you resolve such disagreements in your day to day workflow?
One of the 20 participants declined to respond to (Q1) and (Q2) because they were not a practitioner. Out of the other 19, 14 participants (74%) responded "yes" to (Q1), indicating that they did in fact encounter explanation disagreement in practice. Of the remaining 5 who said they did not, 3 said they had not really paid attention to the issue. We also aimed to uncover how participants dealt with the disagreement problem when it arose in practice (Q2). 14 participants responded affirmatively to (Q1), and their responses to (Q2) can be grouped into 3 categories. 50% favored using ad-hoe heuristics based on personal preferences for choosing which hods to use ("picking their fumorite method", "rules of thumb based on results in popers ")- These heuristics varied among participants and included case of implementation, accompanying thcoretical results, recency of publication, ease of understanding, and docunentation of packages. 36% did not indicate any explicit way for resolving these disagreements but instead demonstrated confusion and uncertainty ("no clear answer to me "). Several of these responses also indicated the desire for the research community to make progress and help ("I hope research commuriity can provide some guidance"). The remaining 14% proposed to use other metrics such as fidelity ( ("Iry and use some metric to measure fidelily"). See Appendix E.7 for more details including the breakdown of our results for participants in academia and industry.
6 Discussion and Conclusion
The main takeaway from our work is the observation that state-of-the-art explanation methods consistently disagree with each other across various datasets and data modalities, as quantified by the metrics we outlined. Our empirical estimates of explanation disagreement closely align with human assessments of disagreements among leading explainable AI methods. Our focus was on highlighting the prevalence of disagreement within the literature on explainable AT and examining how practitioners address these disagreements. To this end, we also conducted extensive user studics (semi-structured interviews and an online user study) and empirical analyses. Our semi-structured interviews with machine learning practitioners highlighted that 84% of them regularly face explanation disagreements. While there exist metrics and frameworks to both evaluate and compare explanations output by state-of-the-art methods (Agarwal et al. 2021 2022 Han et al, 2022), several practitioners (86% of respondents in our online user study) admitted to using ad hoc heuristics or being uncertain about. resolving explanation disagreements in practice. This highlights critical gap between research and practice in explainable machine learning.
In this work, we prioritized examining the prevalence of explanation disagreement rather than exploring its underlying causes, given the complexity of this analysis. Nonetheless, subsequent research building
14

---

Published in Transactions on Machine Learning Research (06/2024)
our work has attempted to understand the sources of these disagreements. For example, (Han al. 2022b extended our work to demonstrate that state-of-the-art explanation methods all perform local linear approximations of underlying models but do so in different ways (e.g., using different loss functions and over different local neighborhoods). This can be a source of disagreement between the explanation methods. Furthermore, our results raise the question of how to resolve explanation disagreements when they occur. While this remains an active area of rescarch, one approach to selecting among explanation methods is first define the desired goal of an explanation and then choose the method that best meets this goal. For example, follow-up work by Han et al. (2022) defines local function approximation as the desired goal of an explanation, develops theoretical framework to analyze how faithfully each method performs local function approximation and advises selecting the method that performs it most faithfully over a given input space. Given the high-stakes applications of model explanations, using a principled and systematic approach to select. among explanation methods preferable to the ad hoc strategies currently employed by practitioners. Finally, the extent of explanation disagrcement would also depend on the specific metric being used mcasure the disagreement. For instance, while our results broadly demonstrate a moderate to high degree of disagreement among all pairs of explanations across all data modalities, we observe that the extent of disagreement is slightly lower when consider variants of the proposed metrics. See Appendix D.2 for additional details.
Acknowledgements
We thank the anonymous reviewers for their insightful comments and feedback on this paper. We also thank Javin Pombra for assisting with preliminary results during the initial phases of this study. Lastly, we are very grateful to Ana Lucic and Maurits R. Bleeker for pointing out the omission of relevant study and helping us improve the contextnalization of this work.
References
Antonio Gulli corpus of news articles. http://gruup.ddd.u..p..///////u2 news htmml 2005, Accessed: 2021-01-20.
articles. html,
Pascal Visual Object Classes 2012. http::/hhat..ooooerr...c.uauaaat
2012. Ac-
cessed: 2021-01-20.
ImageNet.
2015. Accessed: 2021-01-20.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardi, and Been Kim. Sanily checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.
Chirag Agarwal, Eshika Saxena, Satyapriya Krishna, Martin Pawelezyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. OpenXAI: Towards transparent evaluation of model explanations. 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks, 2022.
Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Zhiwei Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. In International Conference on Machine Learning, 2021.
Ulrich Aivodji, Hiromi Arai, Olivier Fortincau, Sébastien Gambs, Satoshi Hara, and Alain Tapp. Fairwashing: the risk of rationalization. In International Conference on Machine Learning, 2019.
David Aeaaee-Meeis and Tommi Jaakkola. On the robustness of interpretability methods. arXio preprint UrXXi:1106.000 2018.
Antonio Jesus Lunaa, Carlos Martieez-Corte and Horacio Perez-Sanchez. Fighting the disagreement in explainable machine learning with consensus. ar preprint arXiv:2307.01988, 2023.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Compuling Systems, 2021.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpretability via model extraction. arXiv preprint arvi 706.000.999 1rXi::1706009777 706.09773, 2017.
Jacob Bien and Robert Tibshirani. Classification by set cover: The prototype vector machine. arXiv preprint arXix:0908.2230 2284, 2009.
Zana Buginca, Maja Barbara Malaya, and Krzysatof Z Gajos. To trust or to think: Cognitive forcing functions can reduce overreliance on ai ai-assisted decision-making Proceedings of the ACM on Human- Computrr Interaction, 2021.
Rich Caruana, Yin Lou, Johannes Gchrke, Paul Koch, Marc Sturm, and Noemic Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1721-1730, 2015.
Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 2019.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise explanations of neural networks using adversarial training. In International Conference Machine Learning, 2020.
Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. Machine explanations and human understanding. arXiv preprint arXi::2202.0 04092, 2022.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-thcoretio perspective on model interpretation. In International conference on machine learning, pp. 883-892. PMLR, 2018.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, -laus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. Advances in Newral Information Processing Systems, 2019.
Finale Doshi-Veles and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint ar ftt::170200800 2017.
Mingming Fan, Xianyou Yang, TszTung Yu, Q Vera Liao, and Jian Zhao. Human-ai collaboration for ux evaluation: Effects of explanation and synchronization. Proceedings of the ACM on Human- Compuleer Interaction, 2022.
Raymond Fok and Daniel S Weld. In search of verifiability: Explanations rarely enable complementary performance in ai-adviscd decision making. arXiv preprint arXiv:330.00712, 2023.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In AAAI Conference on Artificial Intelligence, 2019.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference data science and advanced analytics (DSAA), 2018
Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should choose? a function approximation perspective characterizing post hoc explanations. In Advances in Neural Information Processing Systems, 2022.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. Human factors in model interpretability: Industry practices, challenges, and needs. Proceedings of the ACM on Human-Compiln Interaction, 2020.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates. 2018.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wort- man Vaughan. Interpreting interpretability: Understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020.
Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in Neural Information Processing Systems, 2014.
Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec. Interpretable decision sets: A joint framework for description and prediction. In ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2016.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovee. Faithful and customizable explanations of black box models. In AAAT Conference on Artificial Intelligence, Ethics, and Society, pp. 131-138, 2019.
Benjamin Letham, Cynthia Rudin, Tyler McCormick, and David Madigan. Interpretable classifiers using rules and baycsian analysis: Building a better stroke prodiction model. Annals of Applied Statistics, 2015.
Alexander Levine, Sahil Singla, and Soheil Feizi. Certifiably robust interpretation in deep learning, arXiv preprint arXiv:1905.122 12105, 2019.
Yang Liu, Sujay Khandagale, Sujay Khandagale, Colin White, and Willie Neiswanger. Synthetic benchmarks for scientific research in explainable machine learning. In Proceedings of the Neural Information Processing Systems Track on Dalasels and Benchmarks, 2021.
Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In ACM SIGKDD International Conference on Knowledge Discovery and Dala Mining, 2012.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Aduances in Neural Information Processing Systems, 2017.
Michael Neely, Stefan Schouten, Maurits JR. Bleeker, and Ana Lucic. Order in the court: Explainable ai methods prone to disagroement. Workshop on Theoretic Foundation, Criticism, and Application Trend of Exploinable Al at International Conference on Machine Learning, 2021.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. Xix preprint ar 1rxii:1806.000 07421, 2018.
ProPublica. How we analyzed the compas recidivism algorithm. http::/ww..probbbcc..oo//rrrtoot
Accessed: 2021-01-20.
UCI Machine Learning Repository. German credit dataset. httpe://arcciuee.ccccddddddda//a (germaattcedd
Accessed: 2021-01-20.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of machine learning. Tn TCML Workshop Human Interpretahllily in Machine Learning (WHT), 201
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?" Explaining the predictions of any classifier. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016b.
Irene Rodríguez-Laján, Jordi Fonollosa, Alexander Vergara, Margic L. Homer, and Ramón Huerta. On the cali- bration of sensor arrays for pattern recognition using the minimal number of experiments. Chemomelrics and Intelligent Laboratory Systems, 130:123 134, 2014. URL https::/app. http://app...mmatttcccchhhollaa. org/ CorpusID:
59493586
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015. doi: 10. 10.1007/s1 1263-016--0866--
17

---

Published in Transactions on Machine Learning Research (06/2024)
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEFF International Conference on Computer Vision, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation diffcrences. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Karen Simonyan, Andrea. Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Representations, 2014.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. How can we fool LIME and SHAP? Adversarial attacks on post. hoc explanation methods. In AAAT/ACM Conference AT, Ethics, and Society, 2020.
Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances in Neural Information Processing Systems, 2021.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: Removing noise by adding noise. arXie preprint or Xiv: 03825, 2017.
Fabian Sperrle, Mennatallah El- El-Assady, Grace Guo, Rita Borgo, D Horng Chau, Alex Endert, and Daniel Keim. A survey of human-contered ovaluations in human-centered machine learning. In Computer Graphics Forum, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic altribution for deep networks. International Conference on Machine Learning, 2017.
Helena Vasconcelos, Matthew Jörke, Madeleine Grande-MLLuggllnn, Tobias Gerstenberg, Michael S Bernstein, and Ranjay Krishna. Explanations can reduce overreliance ai systems during decision-makinn. Proceedings of the ACM Human- Coman--Compute Interaction, 7(CSCW1):: 38, 2023.
Xiang Zhang, Junbo Zhao, and Yann LeCun. cttracter-level convolutional networks for text classification. In Aduances in Neural Information Processing Systems, 2015.
Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger. Evaluating the quality of machine learning explanations: A survey on methods and metries. Electronics, 2021.
A Measuring Disagreement
In this section, we formally define our 6 different metrics: feature agreement, rank agreement, sign agreement, signed rank agreement, rank correlation, and pairwise rank agreement. The first four metrics capture disagreement for the top-k features of the explanations and the last two metrics capture disagreement for a selected set of features that could be provided as input by an end user. For all six metrics, lower values indicate stronger disagreement.
A.1 Measuring Disagreement For Top-k Features
We now define four metrics, which capture specific aspects of explanation disagreement for the top-k features
TTe top-& features of explanation are typically computed only based the magnitude of the feature importance values and the signs.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Feature Agreement: ML practitioners in interviews (Section 3.1) indicated that key notion of disagreement between a pair of explanations is that they output different top-k features. To capture this notion, we introduce the feature agreement metric which computes the fraction of common features between the sets top-k features of two explanations. Given two explanations Eg and Es. feature agreement is formulated as:
Feature.Agreement Ea, Eu, [TF k)
k
where TF{E, k) returns the set of top-k features of explanation based on the magnitude of the uealure importance values. If the sets top-k features of explanations Eg and Es contain the same features, then FeatureAgreement Ea, Eg. =
Practitioners also indicated that if the ordering of the top-k features is different for two explanations (even the feature sets are the same), then they consider it to be a disagreement. To capture this notion, we introduce the rank agreement metric which computes the fraction of features that are not only common between the sets of top-k features two explanations but also have the same position the respective rank orders. Rank agreement stricter metric than feature agreement since il also considers the ordering of the top-k features. Given two explanations Eg and Es, rank agreement is formulated
Rank Agreement:
(eTF(E,,)) R(E,, f) R(Es. f)Jl
SEF
RankAgreement(Lr (Ea, Es, k)
k
where F is the complete set of features in the data, TF(E, k) is as defined above, and R( E, f) returns the rank of feature f according to explanation E. If the rank-ordered lists of top-k features of explanations Ea and E, match, then Rank Arreemeene Ea, Es, k) 1.
Sign Agreement: In our study, practitioners also mentioned that they consider two explanations to disagree the feature attribution signs (i.e. directions of feature contribution) do not align for the top-k features. To capture this notion, we introduce the sign agreement metric which computes the fraction of Teatures that are not only common between the sets of top-k features of two explanations but also share the same sign in both explanations. Sign agreement is a stricter metric than feature agreement since also considers signs of the top-k features. Given two explanations Eg and Es, sign agreement is formulated as:
UUIIIETTF(Ba ASE
S(E., f) S(E,, f)J
JEF
SignAgrcement((.. Eu, E,, k)
where F and TF(E, k) are as defined above and S(E, returns the sign of feature according to explanation E.
Signed Rank Agreement: This metric fuses the above three notions of explanation disagreement and computes the fraction of features that are not only common between the sets of top-k feat of two explanations but also share the same feature attribution sign and rank both explanations. The signed rank agreement is the strictest compared to all the aforementioned metrics since it considers both the ordering and the signs of the top-k features. Given two explanations Eg and Es, signed rank agrecment formulated as:
UUIIfeTFEEE FETF(ELR)) S(E.,)) = S(Es,)) A R(E f) R(Es, feF
SignedRank-Agremmennl... Es, k)
where F, TF),E)), S)E,), and R(E, S' defined above. SignedBaankAgreemeen (Ea, Es, k) 1 if the top-k features of two explanations match on all aspects (i.e., features, feature attribution signs, rank ordering) barring the exact feature importance values.
19

---

Published in Transactions on Machine Learning Research (06/2024)
A.2 Measuring Disagreement With Respect to Features of Interest
Practitioners also indicated that they consider two explanations be different if the relative ordering of features of interest (e.g., salary and credit score discussed in Section 3.1) differ between the two explanations. To formalize this notion, introduce the two metrics below.
Rank Correlation: We adopt standard rank correlation metric (i.e., Spearman's 's rank correlation coefficient) to measure the agreement between feature rankings provided by two explanations for selected set of features. In practice, this selected set of features corresponds to features that are of interest to end users and can be provided as imput by end users. Given two explanations Ea and Eb, rank correlation is formulated as:
BankCorrelationn(E.. Es, F), P))
where F= (f is a set of features selected by end user, computes Spearman's 's rank correlation coefficient, and R(E, F) assigns ranks to features in F based on explanation E.
Pairwise Rank Agreement: The pairwise rank agreement takes as input set of features that are of interest to the user and captures if the relative ordering of every pair of features in that set is the same for both explanations, i.c., if feature A is more important than B according to one explanation, then the same should be true for the other explanation. This metric computes the fraction of feature pairs for which the relative ordering the same between two explanations. Given two explanations Eg and Es, pairwise rank agrocment formulated as:
Σ i for i<j
I[RO(E., RO(E..
PairuisseRammk-greeeeeet Es, F)-
where F (f, is set of features selected by an end user, the relative ordering function RO(E, ff) is an indicator function that returns 1 if feature fü is more important than feature fa according to explanation E and returns otherwise.
B Experimental Setup
B.1 Black Box Models: Training and Performance
For tabular data, train four models: logistic regression model, gradient-boosted tree model (50 estimators), random forest model (50 estimators), and a densely-connected leed-forward neural network (with 3 hidden layers with relu activation consisting of 50, 100, and 50 neurons, respectively). For the COMPAS dataset, we train the four models based on an 80%-20% train-test split of the dataset, using features to predict the COMPAS risk score group. The test accuracies of the four models are 0.75, 0.79, 0.79, and 0.73, respectively. For the German credit dataset, we train the same four models based on an 80%-20% train-test split of the dataset, using features to predict the credit risk group. The test accuracies of the four models are 0.65, 0.70, 0.73, and 0.64, respectively.
For text data, trained widely-used Lased text classifier, based on 120,000 training samples and 7,600 test samples, to predict the news category of the article from which sentence was obtained. The model performs with 90.67% accuracy. The architecture comprises an embedding layer of dimension 300, followed by an LSTM layer hidden size 256 connected to a four-dimensional output layer.
For image data, we use the pro-trained ResNet-18 model (He ct al. (0166) and analyze explanations generatod for predictions made to classify images to one of the 1000 classes. This model performs 69.758 % and 89.078 % on Accuracy@1 and Accuracy@5 m respectively.
20

---

Published in Transactions on Machine Learning Research (06/2024)
B.2 Explanation Methods
For tabular data, the perturbation-baseed explanation methods (LIME and KernelSHAP) were applied to explain all four models while the gradient-based explanation methods (Vanilla Gradients, Integrated Gradients, "radient"Inptt, and SmoothGRAD) were applied to explain the logistic regression and neural network models to explain samples from the test set (1,198 samples for the COMPAS dataset and 200 samples for the German Credit dataset). Because gradients are not computed for tree-based models, the gradient-based explanation methods were not applied to the random forest and gradient-boosted tree models. When applying explanation methods with a sample size hyperparameter (LIME, KerneISHAP, Integrated Gradients, SmoothGRAD), we performed a convergence check and selected the sample size at which an increase in the number of samples does not significantly change the explanations. Change in explanations at the current versus previous sample size is measured by the L2 distance of feature altributions, the L2 distance of the ranks of all features, and the six metrics. For both COMPAS and German Credit datasets, we used 2,000 samples for relevant methods (i.c. LIME, KernelSHAP, SmoothGRAD, and Integrated Gradients).
For text data, we applied all six explanation methods on the LSTM-based classifier to explain predictions for 7,600 samples in the test set. For LIME and KernelSHAP, we follow the convergence analysis described above and find that attributions do not change significantly beyond 500 perturbations; hence, we use 500 perturbations for LIME and KernelSHAP. Integrated Gradients explanations were generated using 500 steps which is higher than the recommended number of steps mentioned in (Sundararajan et al.] (017). SmoothGRAD explanations were generated using 500 samples to get the most confident attribution which is significantly higher than the recommended number of 50 samples (Smilkov et al. 2017
For image data, we applied all six explanation methods on the ResNet-18 model (He al. 2016) explain predictions for the PASCAL VOC 2012 test set of 1,449 samples. Integrated Gradients explanations were generated using 400 steps, significantly higher than the recommendation of 300 (Sundararajan et al. 2017). to obtain stable and confident attribution map. Similarly, SmoothGRAD explanations were generated using sample size of 200 which is also higher than the recommended sample size of 50 (Smilkov et al. 2017). For LIME and KernelSHAP, we chose 100 perturbations to train the surrogate model as we did not notice any significant changes in attributions beyond 50 perturbations. KerneISHAP and LIME were used to compute attributions of siper-pixels annotated in PASCAL VOC 2012 segmentation maps. Due to a larger feature space in images compared to the previous tabular and text datasets, disagreement metrics based on top-k features may not provide a clear picture. Hence, we use Rank Correlation between attribution maps generated by pair of explanation methods as the disagreement metric.
C
Results from Empirical Analysis of Disagreement Problem
C.1 Tabular Data: COMPAS and German Credit Datasets
The tabular datasets consist of the COMPAS and German Credit datasets. The full set of figures showing explanation disagreement for the two datasets, over four models, measured by six metrics, displayed in two formats (metric mean in heatmap and metric distribution in boxplot) for varying values of top-k features can be found in the code repository accompanying this paper. Here, we elaborate on trends of explanation disagreement at the level of metries, models, and datasets.
At the level of metrics, explanations show stronger disagreement when measured by stricter metrics (such as signed rank agrcement) than by less strict metrics (such as fcature agrcement and sign agrecment). Generally, as the number of top-k features increases, metrics show stronger disagreement between explanations. The exception is the feature agreement metric which, by definition equals one (indicating perfect agreement) when the k is the total number of features. For both datasets, the majority of models and explanation pairs have several points for which the rank correlation metric is near zero or even negative, indicating disagreeing, and even opposing, explanations.
At the level of models, explanations for more complex models (such as the neural network) tend show similar or stronger disagreement than explanations for less complex models (such as the logistic regression model). As discussed in the main section of the paper, as the complexity of the black box model increases,
21

---

Published in Transactions on Machine Learning Research (06/2024)
it may be more difficult to explain the model's decision-making process and more difficult to disentangle the contribution of each feature to the model's prediction. Thus, the higher the model complexity, the more difficult it may be for different explanation methods to generate the true explanation and the more likely it may be for different explanation methods to generate different false explanations, leading to stronger disagreement. among explanation methods.
At the level of datasets, explanations for the German Credit dataset showed similar or stronger levels of disagreement than those for the COMPAS dataset. As discussed in the main paper, one possible reason is that the German Credit dataset has more features than the COMPAS dataset, resulting in larger mumber of possible ranking and sign combinations assigned by given explanation method and making it less likely for two explanation methods to produce consistent explanations.
Barkegnarwet 2-11
10
Saseagganeett 21-44
18
Rarak aak.ggeserrtt 3 77)
0,98
1.000
3.700
99
9.21
1.165
10x
144
0.81
0,12
6:124
1M
1.90
0.12
0.00%
0399
2.99
G9d
9.0
26
0.75
Oear
9083
6.16G
10C
0:00
00000
Geaf
1:00
0244
Dew
0.001
1.000
2274
NOmd
8085
1.225
090
10CE
0040
H90hl
PEN
122
0.281
1.09
0.99
0
0.2%
1900
8:909
42
-2
Cancer smonc mted0. m0000.333
Bird encos mac00011
Standirs erom mini m2
Sigwi sitk wraamrr (c+11)
sigred Bi agrnammtt DU+44
Signadiom agueemen 3t-
0,99
100
2/26
3:08
12x
01221
1.90
0.341
D0TS
OR
Quar"
1000
5.145
0280
3.5
0209
0.009
2
MOse
8.99
0.206
8:010
10KE
0000
HOhel
121
1:22
0.83
1299
0191
0.80
0
0.297
1991
2:99
Shoue
Cancer smonc mfee0. m30
dird emcm: rau00117
Stander erEm miw0 m2v02212
Figure 7: Disagreement between explanation methods for neural network model trained on German Credit dataset measured by rank agreement (top row) and signed rank agreement (bottom row) at top-k features for increasing values of k. Each cell in the heatmap shows the metric value averaged over test set data points for each pair of explanation methods, with lower values (lighter colors) indicating stronger disagreement.
C.2 Text Data: AG_News Dataset
In this section, we analyze the trends in pair-wisc disagroement w.r.t the feature agrecment metric with the increasing number of features considered for measuring disagreement (k = [3, 7, 1]]. We observe that agreement between gradient-based methods is significantly greater compared to that with perturbation-boseed explanation method, which is more prominent for larger values of k.
C.3 Image Data: ImageNet Dataset
In Table 2 we analyze disagreement between perturbation based explanation methods when feature attribu- tions are computed at the super-pixel level, and obscrve a significantly high agrecment in terms of all the proposed disagreement metrics. However, there is little to no agreement when it's computed the pixel level in terms of the rank correlation between explanations, shown in Figure g
22

---

Published in Transactions on Machine Learning Research (06/2024)
Festure agreement
Feature agreement (K a ?)
Feature agreement (c = 11)
1.0
0.8
Festure sgvemeet
Festure
LMC Horsehouup
LIME GruProust
LIC inre
LIKE vs SmottHoNDD
Kemebru es GreZ
Kon Barewwoh es GrsProut
ss inired
SmotthoRuD
Gradhrput
SmattiGUD
CooPree inSins
Grech rost vs SmoencRuD
inteed is SmattOND
LIKE Remebror
18F us Ceed
LIFE a Grushruut
ss inGre
Smottoouu
Kamettuu Gradhppi
KeroEHH infrns
sametria SmottooND
Good an CouPhiuut
Grad vs intres
God us SmottouuD
God vs Ioru
wnce SmosthoiuD
LIN Hemediup
God
Grethopst
Intin
LINE SmettiouuD
Komebral Gnd
Kemattie as Gruhrrpst
monestee vs. vnGrr
cemaatan is SmottiGuu
SmsettiND
Coerteep vs. InGred
Goes nost vs SRooHoND
woud vs Smoutcioo
Gostroo
Pair explanation methocs
explanation metheds
Pair explanation metheds
Figure 8: Box plot for feature agrecment on AG AG_News News dataset for [3,7,11]
Metrics
ResNet-18
Rank correlation
0.8977
Pairwise rank agreement
0.9302
Feature agreement
0.9535
Rank agreement
0.8478
Sign agreement
0.9218
Signed rank agreement
0.8193
Table 2: Disagreement on ImageNet between LIME and KernelSHAP
0.0008
6:670
0.0000
044
0,0008
10000
50000
0025
-1.00
Figure 9: Rank correlation for explanations computed at pixel level by paadient-baser explanation methods
D Additional Experiments
D.1 Comparison to L2X (Chen et al. 2018
For the COMPAS dataset, we show the disagreement between L2X CChnn et al. 2018) and other explanation methods in Figure 10 (ncural networks) and Figure 11 (logistic regression). We observe a similar pattern with L2X as well, where there poor agreement between explanations computed by L2X and those calculated by other explanation methods.
23

---

Published in Transactions on Machine Learning Research (06/2024)
Faraua aotiec
1:24
11:07
100
12712
0.111
8.208
01251
1.2.0
0.210
Gom
6:42
00:00
0.201
Oml
0.70
1:00
6:311
A77T
0.23
0.442
0.227
18X
4154
0.152
0.00
RX
0:
ATR1
19x
0.71
0.00
0.203
0270
1M
1000
0172
0.00
9.15
1.300
0:300
12x
0.7
0.98
0.341
8:25
0.70
9223
C751
1.09
E.52
0.233
0.000
0.000
8102
0221
1.000
027
3.50
3:45
6710
0300
041
1.00
0.000
12X
1.71
0.07
1142
3.79
0/57
1000
11278
0:29
8.718
1173
8:44
Dof
Ord
LX4
Standard aetor st.r1. itaun
Sign genement 2
Talm oe rark menmen
LMC
1.000
0.51
0221
0.99
8:500
Sigred mereemandi.oo. DC 21
LMC
1.000
0:307
0:797
8:05
0.704
LODO
0:77
0.111
0.73
0.221
3.20
0.102
Kame
0822
1.00
0000
1000
0.044
0.000
384
0.49
000
0-13
0200
19X
3.02
0282
8.540
0:515
0049
5.99
A173
144
0.044
0.00
0.11
0.07%
0:00
0225
0300
100E
0217
8.352
Qall
3:00
A202
0142
3:300
8:111
0.151
100
0.47
602
LOX
0.5%
0.00
0.24
0:44
0300
0:13
1,00
UXX
E:01
0:012
6:07
35%
1-12
0409
1300
1.2
0.997
0201
5.5
0.004
0M
147
1/00
-08
Figure 10: Disagreement between explanation methods for neural network model trained on COMPAS dataset mcasured by four metrics: fcature, rank, sign, and signed rank agreement across top k features.
D.2
Additional Metrics to Measure Explanation Disagreements
In this section, we study two additional metrics for measuring the disagreement between two explanations.
D.2.1 Weighted Rank Agreement
Here, we introduce the weighted rank agreement, a new metric that captures the agreement between two explanations. This metric can be intuitively thought of as a softer version of the previously defined rank agrecment metric, as it accounts for differences in ranks when computing the agreement between two explanations. Given two explanations En and Ea, weighted rank agreement is formulated as:
|R(Ea. f) - -R(EE, fl
W WeighteelKmmkAArreeenntt. Eb, Eb, k)
Σ
1 1
k
M)
where TF(E,, k) returns the set of top-k features of explanation F, based on the magnitude of the feature importance values, and R(E, f) returns the rank of fcature f according to explanation E. If the rank-orderod lists of top-k features of explanations E. and E, match exactly, then We Annggleee/ankkAArreeeeeene En, Es, k) 1. Analogously, if the top-k features of explanations Eg and Eg do not overlap at all (i.e., TF(Ea, k) A TF(E, k) is the emoty set), then W WeightedRank Agreement( E,, Fb. k) = 0. In all other cases, this metric takes a value between 0 and 1 (excluding both 0 and 1).
We plotted the weighted rank agreement for the COMPAS dataset (See Figure 12 and observed that the degree of agreement is higher according to this metric compared to our original rank agreement metric (See top row, middle column plots in Figures 10 and 11). This makes sense since our original rank agreement was stricter variant that assigned value of only when the rank of a feature matched exactly between two
24

---

Published in Transactions on Machine Learning Research (06/2024)
1:14
11:00
11:33
0.20
02287
5.000
2.33
0:40
0.201
Oml
1:00
00:5
0012
1.00
1.14
0:000
A150
2:11
19x
1000
0.11
8.70
0.207
0.29
11
1000
1209
0.55
1.00
422
1.x
0.22
0.001
8:55
0.03
1.000
2012
1.09
0.54
0.000
0.11
10x
8130
0.33
1.00
3:507
1900
CA2
642
1.00
0.223
123
1:
0.01
3779
1.00
EXE
11:53
0.1
8.108
0159
5.10
Ore
LXX
Standard aetorad st inand
Sign genement 25+5
Citre ed sngemmmett 5
Prlrvbee rark aggingg mart
LME
1.000
0.57
0445
6-4
0.45
3.472
LMC
1:003
CATO
8.247
0247
$.00
0.12
LVC
1.000
100
2:117
0.422
0.032
3017
0.517
Kame
1.1
0.166
5.41
0:000
18X
B242
344
CUTY
755
5,00
0::44
1.0.
0.43
0.199
1XX
1000
0.193
0.300
0.277
0.131
014
1000
E242
0.3
0/522
8:534
1.000
8:000
0:524
0.00
1:0
0.199
13X
1000
0.19
8.300
0.347
0.001
040
1,00
1009
0342
0.0
Qull
LOX
2:42
0.30
0241
0.32
0241
1,00
LAX
0.122
0.000
014
3,00
0.49
E242
1X0
0677
3.51c
3.59
0534
5:324
3:76
1.00
Plevied
Figure 11: Disagreement between explanation methods for logistic regression model trained on COMPAS datasct mcasured by four motrics: feature, rank, sign, and signed rank agreement across top k features.
explanations, and assigned a value of O otherwise. On the other hand, the new weighted rank agreement: is a more lenient version which assigns a value of 1 if there is an exact match in ranks, a value of if the top-k features of the two explanations do not overlap at all, and value between and to account for differences in ranks in all other cases.
Wigghtd tank agreament (1 5]
Weighted tark aggeemmntt (R-55
S1AP
0.707
1.000
8.511
3.610
0.619
0.511
0.170
S1AP
1,000
0.500
0:612
0,602
3.5M4
0.473
Dred
3.511
1.000
0.457
0.457
1.000
0.365
Cod
0.442
0.500
0.46
0.711
0.366
0.720
0.019
1.000
1.000
0/457
0.500
Irpll
0.897
0:512
0:455
1.002
0.897
2.54
0.501
0.4
0.779
0:918
1.000
1.000
6.477
Irtild
P08
0.622
0.405
0.907
1.000
8.695
0.500
Smooth
122
-22
L2X
5.676
0.470
0.365
3.638
0.600
6.365
1.000
L2X
0.093
0.478
0666
0:901
0.000
0.291
1,000
03
UM2
Surma
Crad
Credt
Smoot
L2X
UMB
Kardd
Ored
Cred"
In/
Smooth
L2X
Stor dand ner-:: mox00.000
Stardard omur.
mww.-mni--.... 0.0000.000
(a) Logistic Regression
(b) Neural Networks
Figure 12: Disagreement between explanation methods for models trained on COMPAS dataset measured using weighted rank agreement metric across top A = 5 features.
25

---

Published in Transactions on Machine Learning Research (06/2024)
D.2.2 Top-K Pairwise Rank Agreement
Here, we introduce a new metric called top-k pairwise rank agreement which is a variant our previously proposed pairwise rank agreement metric, and is computed using the top-k features of any two given explanations. Given two explanations Ea and Eb, top-k pairwise rank agreement is formulated as:
Σ
11 [RO (E.: Jú, fz) RO Eb. fr.
Topk CoogkKPParrisseeeeAAgrreeeee Eb, 10) fe.JEEF and iej
where
F (F(Ea; k) TF((6, k)
and TF(E,)) k) returns the set of top-k features of explanation E based the magnitude of the feature importance values. The relative ordering function Ro, is an indicator function that returns 1 if feature fi is more important than feature /j according to explanation E and returns 0 otherwise.
For the COMPAS dataset, we plot the disagreement between different explanation methods according to top-k pairwise rank agreement in Figurcs 13 and 14, For both the logistic regression and neural network models, the L2X method exhibits the lowest top-k rank correlation with the other explanation methods compared, suggesting it provides quite different feature importance rankings than the other approaches.
Tap- Ranc consiitio Is ()
Top- Palviio rank agreement & 5
2.501
CA57
0.501
1.000
0.522
0:548
0.512
SMP
1.000
0.407
0.407
28
0.777
0.000
0.520
25
0r0
0.417
1.000
8.266
0.266
1000
0.442
716
0:577
1.000
0.674
0.576
1.000
0.451
36
3.
1rCreed
0.477
0.321
0.295
1,000
1.200
0.25
4.128
0.069
0:508
0.576
1,000
1.000
0.578
0.484
Seooh
1.00
0.255
0.25
1002
-17
Smoot
0.877
0.578
1.000
0.401
1.X
0.001
0.0/2
-4.133
123
12X
0.522
0.461
0.454
0.464
10000
-10
-35
SHAP
Use
Fssi
Usa
Standard
m12
Standard mimd0. maard:206
Figure 13: Disagreement between explanation methods for logistic regression model trained on COMPAS dataset measured using Top-K Pairwise Rank Agreement and Top-K Rank Correlation metric across top k features.
D.2.3 Top-K Rank Correlation
Here, we introduce a new metric called top-k rank correlation which is a variant of our previously proposed rank correlation metric, and it computed using the top-k features of any two given explanations. Given two explanations Eg and Es, top-k rank correlation formulated as:
T TopK RankCCrrelation En, Es, F') r (,((R(E,, F'), R(E,, F'))
where
F' TF( KFFT.E,
As defined previously, (F(E,k) returns the of top-k features of explanation E based on the magnitude of the feature importance values, R(E, F²) is a vector containing the rank of every feature f F according explanation F, and rs computes Spearman's rank correlation coefficient.
26

---

Published in Transactions on Machine Learning Research (06/2024)
For the COMPAS dataset, we plot the disagreement between different explanation methods according to top-k rank correlation metric in Figures a3d 14 We observe similar patterns of lower agreement between different explanation methods, with L2X exhibiting the lowest top-k rank correlation with other explanation methods.
(x
Pallwioe apreement
LIVE
1.000
0.023
0,500
0.259
0.348
0.503
4.055
LIVE
1000
0.212
0.817
0.940
0:009
0.095
0.430
Kanei
1:000
0
0.236
0.209
200
5.034
Kareel
1:000
0.50
0.523
0.550
0.75
2.50
0.188
0.267
0.761
8.277
0:573
0:501
0.421
Goid
0.200
0.236
0.199
1,000
0.229
0.44
4.171
God
954
0.080
0.572
0.5/1
1,000
0.287
0.004
0.443
-44
-14
0181
0:300
0.200
0.448
0.348
1002
4.224
-12
0.235
0:573
0.794
0.824
0:024
000
0:372
-12
1.2
0.034
207
3.121
1.000
12X
0.511
0.421
0.408
0.372
10000
-30
Mb
Sarted
Liaf"
180led
Smuch
UX
LIVE
Seredd
Stocht
UXX
Standard erore In maa
Standart ROOIE minin maAUT0.006
Figure 14: Disagreement between explanation methods for neural network model trained on COMPAS dataset. measured using Top-K Pairwise Rank Agreement and Top-K Rank Correlation metric across top A = 5 features.
Cark
Nars carasaee
LMVE
3.00
9.0%
0.90
400
LON
3.00
4,0
410
6:1M
9.00
CVB
0.00
4.5%
Enel
3117
MAPP
Crer
3.00
k1mh
0.00
100
1009
(a) COMPAS ( =7)
(b) German Credit (/ 220)
(c) Forest Cover (/ =54)
(d) Gas Conc. ( =1==12 =128)
275
5/10
LBC
LCCT
3.77
LCS
CP3
A4
0
A4
34
RCM
0.04
190
0004
6.00
4.00
1700
0.01
1.0
0.00
9m
3+d
299
207
Renacarcc ca
(e) COMPAS (ʃ =7)
(f) German Credit (f 200)
(g) Forest Cover (f =54)
(h) Gas Cone. (f -128)
Figure 15: Disagreement in explanation methods for neural network models trained on datasets with the increasing number of features (f), as assessed by rank correlation (top row) and pairwise rank agreement (bottom row). Each cell in the heatmap shows the metric value averaged over test set data points for each pair of explanation methods, with lower values (lighter colors) indicating stronger disagreement.
D.3 Effects of Dataset Complexity
We experimented with two more tabular datasets to analyze the change in disagreement with increasing dataset complexity. These two additional datasets are the Forest Cover Type datased" with 54 features and the Gas Concentration dataset (Rodriguez-Lujáá et al. 2014) with 128 features. We plot the rank
Thttps://arr /archive. ies www..ed////ataase///3 /covertypee
27

---

Published in Transactions on Machine Learning Research (06/2024)
correlation and pairwise correlation agreement metrics in Figure 15 for 4 datasets: COMPAS (7 features), German Credit (20 features), Forest Cover (45 features), and Gas Concentration (128 features) between the 6 explanation methods for neural network models. We did not observe a drastic change in the disagreement between explanations when the features in the tabular data increased from 7 to 128; however, we did notice a substantially lower rank correlation between LIME and KernelSHAP with other methods for the Gas Concentration dataset which has the largest number of features among the datasets.
E
Omitted Details from Section 5
E.1
Screenshots of UI
In Figures
16 and 17 we present screenshots of the UI that participants are presented with before beginning
the study. The purpose of this introduction page is to familiarize the participants with the COMPAS prediction setting, the six explainability methods we use, and the explainability plots we show each of the prompts.
Introduction
COMPAS is popular commercial algorithm used by judges for determining a criminal defendent's Ikelihood of reoffending (recidivism).
The COMPAS dalasel consiels of Tealures:
age
two year recid: whether defendant recidivated within 2 years the original crime
priors count: number prtor crimes committed
length stay: length defendent stayed
e. e_charge_ degree: one of Misdemneaor,, Felony
sex: one of Male. F Female
of AlnicanAmmrrian, Asian, Caucasian, Hispanic, Native American, Other
study, we a neural network COMPAS dataset predict criminal defendant's COMPAS score high), corresponding whether he/she would commit crime after two years past the date of the original crime. Since is Important to understand our model's predictions explainabillty algorithmms various input points.
(explainabllit), also
explainebiity algorithms use
past
LIME: an explanation based on locally linear approximation of the model at that input
KernelSHAP: combination of LIME and Shapley Values, which identify the contribution of each feature based on inferactions with other features
Gradient: gradient of the model the input
Iradient"Input the product the input features and the gradient explanation
SmoothGrad weighted average of the gradient at points around the input
Integrated Gradients: mocification of the gradienl method satisfy axioms, senstivity and implementation Invariance
This is screenshot of the first half of the introductory page, describing our COMPAS risk score prediction setting and briefly summarizing the six explainability algorithms used (with links to their corresponding papers for the interested participant).
Figure 16:
E.2
Prompts Used
In this section, we share the 15 prompts that we showed users. Each prompt highlights a pair of different explainability algorithms on a COMPAS data point. For cach pair, we chose the data point from the entire COMPAS set that maximized the rank correlation between the explanations.
E.3 User Study Questions
In each of the five prompts, we asked participants the following questions, which we refer as Sei 1. Questions 3-4 were only shown if the user selected Mostly agree, Mostly disagree, or Completely disagree to Question (1).
To what extent do you think the two explanations shown above agrec or disagree with each other? (choice between Completely agree, Mostly agree, Mostly disagree, Completely disagree)
Please explain why you chose the above answer.
Since you believe that the above explanations disagree (to some extent),, which explanation would you rely on? (choice wetween Algorithm explanation, Algorithm explanation, It. depends)
28

---

Published in Transactions on Machine Learning Research (06/2024)
Your Task
On each of next pages, you will see the result two explainabillyy melhods on the same inpul semple from COMPAS, as shown below. Assume that the criminal defendant's risk of recidivism was correctly predicted to be hiah. The exalanation the prediction will then be shown to you, as in the figure below. Grafientrpput
reod
prongseant
Ccheng_deppu_
30, fonale
PerFonale
Pouture www.ooieatttnn
The yaxis lists each Uhe COMPAS fealures, the x-akis shows importance Biai feature. Posilive importaance values shown blus,, while nesgalive importance values shown red. high positive importance negative importance means that the fonture negatively contributed (was mislcading) to the prediction. Note the difforent x-axis scales resulting from different mathods. You will be asked to compara the two explarations.
means feature
greatly correct prediction, high
This is a screenshot of the second half of the introductory page, describing the concrete task and an explanation of what shown in the explainability plots.
Figure 17:
4. Please explain why you chose the above answer.
A After answering five prompts, the user was then asked the following set of questions, which we refer to as Set Questions 4-9 were only shown if the user selected Yes Question 3.
(Optional) What your name?
What your occupation? (eg: PhD student, software engineer, etc.)
3. Have you used explainability methods in your work before? (Yes/No)
4. What do you explainability methods for?
Which data modalities do you run explainability algorithms on in your day to day workflow? (eg: tabular data, images, language, audio, etc.)
Which explainability methods do you use in your day to day workflow? (eg: LIME, KernelSHAP, SmoothGrad, etc.)
7. Which methods do you prefer, and why?
8. Do you observe disagreements between explanations output by state of the art methods in your day to day workflow?
How do you resolve such disagreements in your day day workflow?
E.4 Further analysis of overall agreement levels
In this section, we present further plots analyzing responses to questions (1) in Set 1. As shown in Figure 19a only 32% of responses were Mostly pore//Completel Agree and 68% were Mostly Gisagree/Compleet Disagree, indicating that participants experienced the disagreement problem. We also grouped the responses by prompt. shown in Figure 19b] highlighting that different pairs of algorithms can have different. levels of disagreement. We removed prompts with less than total responses. We see that there are varying levels of disagrocments among prompts. For example, all participants who were shown the Gradient vs. SmoothGrad prompt believed they agreed to some extent, while all participants who were shown the Gradient. Integrated Gradients prompt believed they disagreed to some extent.
29

---

Published in Transactions on Machine Learning Research (06/2024)
Occerhel
Figure 18: Images showing the 15 prompts we used. Each prompt shows the explanation of the same input point with two different interpretability algorithms.
30

---

Published in Transactions on Machine Learning Research (06/2024)
Distribution of agreement level among prompts
50
Frequency (%) of agreement levels for each pair algorithms
40
(Grad, SG)
(6% KG)
(UMC. KG)
pair of sigortthms
(LIMC. 610
ISHAP, 610
20
(SHAP,
[UIME, SG)
compl disegree disagree agree compl agree
(LBMF, SHAP)
(SHAP, Grad)
0
compl disagree mostly disagree
mostly agree compl. agree
agreement
frequency (%)
(a) This figure shows the distribution of responses in aggregation over all prompts. The x-axis shows the four possible responses, and the y-axis shows the number of times that response was chosen. Observe 68% of cases, participants indicated that the prompts mostly or completely disagreed.
(b) This figure shows the distribution of responses, sorted by prompt. The y-axis shows pair explain- ability algorithms shown in the prompt, and the x-axis shows frequency that each response was chosen.
Figure 19: These figures show the distribution answers to Question (1) in Set (1) from Section E.4 aggregation over all participants.
Algorithm
Reasons that algorithm was chosen in disagreement
36% SHAP is better for tabular data ( ('SHAP is more commonly used
KernelSHAP
[than Gradientj for tabular data")
[25%] SHAP more familiar ("More information present + more
familiarity")
[14%] SHAP is better algorithm overall ('SHAP seems more me- thodical than LIME", "SHAP is more rigorous approach [than LIME) in theory")
[33%] SmoothGrad paper is newer or better ( ("Smooth Grad is apparently more robust", "Smooth is oflen considered improved version of grad") [58%] Reasons based on the explainability map shown ('directionality of the attributions [agree] with intuition', "gradient has instability problems so} smoothgrad")
SmoothGrad
[54%] LIME is better for tabular data ('l use LIME for structured data.
LIME
[15%] LIME is more familiar/easier to interpret ('l am more famallar with LIME", 'LIME is easy to interpret")
Integrated Gradients
[86%] Integrated Gradients paper is better ('IG came after gradients and poper shows 'mprovements', 'inlegrated grudients poper showed im- provements [over Gradient x Input]"
Table 3: Reasons participants chose the top four most favored explainability algorithms (KernelSHAP, SmoothGrad, LIME, and Integrated Gradients) over others when explanations disagreed.
E.5 Further analysis of reasons participants chose specific algorithms
In this section, we analyze the responses Set 1, Question (3) in Section E.3 We saw, in 5.2.2 that algorithms such KernelSHAP were favored over other algorithms. In Table we list the top reasons the four most frequently chosen algorithms were preferred, showcasing direct quotes from participants.
31

---

Published in Transactions on Machine Learning Research (06/2024)
E.6 Analysis of reasons participants chose neither algorithm
In this section, analyze the responses to Set 1, Question (4) in Section selected "It depends " in Question (3), which was chosen in 38% of cases. Again, we present an overarching summary of the reasons participants made this decision Table 4
E.3 focusing when participants
Rationale
Representative Quote
1. Need more informa-
'need to see the final prediction of the model and the feature values"
tion
2. Pick neither explana-
"No compelling reason lo choose one over the olher. Both don't alágn with intuition."
tion
3. Unsure/Don't know
"I'm not sure which of the two methods is more trustworthy"
4. Would consult an ex-
"I would ask a domain expert for his/her opinion'
pert
5. Combine explana-
'I would combine both note that age might be doing weird things, but that length of slay and race both contribule to negalive prediction"
tions
6. Depends use case
"The two methods have different interpretations it depends on if I'm more interested in comparing my erplanation to some baseline individual state versus just interested in underslanding the immeddaae local behavior'
Table 4: Reasons people answered 'It depends" after being asked to choose between disagreements
E.7 Further analysis of concluding questionnaire
this section, we extend the analysis presented in 5.2.3 analyzing the responses to questions Set of
Section E.3 As stated in 5.2.3 we received a total positive responses Question (3), but one declined answer Questions (4) through (9). Therefore, we analyze the remaining 19 responses.
In Question (4), we found that study participants use explainability methods for variety of reasons such as understanding models, debugging models, help explain models to clients, research. In Question (5), we found that 16 of 19 participants employed explanations for tabular data, 6 of 19 participants for text and language data, 11 of 19 participants for image data, and 1 of 19 for audio data. In Question (6), we found that 14 of 19 participants used LIME, of 19 participants used SHAP, and 13 of 19 participants used some sort of gradient-based methods. Participants also indicated using methods like GradCAM, dimensionality reduction, MAPLE, and rule-based methods. In Question (7), 9 of 19 participants stated that they preferred both LIME and SHAP, with another 3 of 19 participants stating LIME only. We showcase some intriguing answers from Question (7) below:
"LIME and SHAP seem to be the most universally applicable and can understand."
"Methods with underlying theoretical justifications such as KernelSHAP and Integrated Gradients"
"LIME and SHAP [easy to implement] and can work with black box"
"SHAP and LIME because... ... [they are] easy to understand and have standard implementations"
"LIME, because everything else isn't necessarily capturing what I actually want to know about the local behavior"
Finally, we provide additional quotes highlighting the responses to Questions (8) and (9), which were briefly analyzed in Section 5.2.3 These shown in Table 5
E.8
Breakdown of the Results
The main themes for both academic and industry participants revolve around trusting methods based on their theoretical foundations, intuitive explanations, and suitability for the data type (labular).
32

---

Published in Transactions on Machine Learning Research (06/2024)
Category of Response
Samples Quotes
"Such disagreements are resolved by dala scienlists picking their favorile algorithm'
1. Make arbitrary decisions (50%).
'I try use rules of thumb based on results research papers and/or or easy to undersland outpuls."
'I favor LIME and SHAP because there is well documented packages Gitllub
2. Unsure/Don't know /Don't resolve (36%)
"there is clear answer to me. I hope research communily can provide some guidance"
"unfortunately there is no good answer at my end hope you con help me with finding an answer'
3. Use other metrics (fidelity) (14%).
"By quantitative assessment of feature importance methods that assess specific properties like faithfulness'
'I might try and use some metric to measure fidelity."
Table Representative quotes highlighting themes of how participants address the disagreement problem their day to day work
In this section, break down the results by academic (13) and industry (12) participants. For specific pair of explanations, Figure 20 shows that there some stark differences in how the two groups of participants prefer a specific explanation when there is disagreement between pair of explanations. For example, when comparing LIME and Integrated Gradients, the academic parlicipants prefer neither while half of the industry participants prefer LIME. In addition, when comparing SHAP and Gradients, the academic participants are split between selecting SHAP or preferring neither of the explanations while the industry participants always prefer SHAP. Figure 21 shows that while both groups prefer SHAP in case of the disagreement, this preference is more pronounced in industry participants (80%) than to academic participants ( 60%).
Frequency (56) algorithmu are picked prompt
Prequency (%) algorithms are prompt ()
[Rrad,
[Crad.
(0)
(0") 151
par alpottaas
(SHAP, 6-0
1SHAF, 611
(SHA4) 121
Grodi
Grad)
dIME 5C)
(UME: 901
(UME, SiMP)
AMM, SIMP)
Gradi
1rnad Ki
(Gnd Ki
20
40
60
100
20
60
80
100
Proquency
Froquency
(a) Academia
(b) Industry
Figure 20: The frequency with which each of the explanations was chosen when there a disagreement. The blue, gold, and grey bars show the percentage of participants (X axis) that picked the left, right, and neither method when presented with the pair of methods shown on the Y axis. (a) is for participants from academia and (b) is for industry participants.
Moreover, industry participants place a stronger emphasis on the theoretical rigor of the methods (42%) compared to academic participants (25%), while academics prioritize explanations aligning with their intuition more often (40%) than industry professionals (26%). A breakdown of these results along with sample quotes from the participants are shown in Tables and
33

---

Published in Transactions on Machine Learning Research (06/2024)
Frequency disagieements
Frequency algorithmss disagroements
60
70
52
50
40
10
10-
Read
SHAP
exolahaab agarizi
explainabilly algaritim
(a) Academia
(b) Industry
Figure 21: The frequency with which each of the explanations was chosen when there is a disagreement. X axis indicates the explanation method and Y axis indicates the frequency. (a) is for participants from academia and (b) is for industry participants.
Theme Highlighted
Sample Quotes
One method's paper/ther paper/theory suggests that It's inherently bet- ter (25%).
"SmoothGrad seems to mainly apply to image processing and not labular tasks"
2. One explanation matches intu- ition better (40%).
"My hunch is that the three features in the middle might be relevant to the prediction. I rely KernelSHAP because il better aligns with my guess, which is not necessary true."
3. LIME/SHAP are better for tabular data (7.5%).
"I prefer LIME and SHAP since those are more well moti- valed and work for tabular dala."
Table 6: Themes summarizing how academic participants decided between explanations when faced with disagreement along with quotes.
Theme Highlighted
Sample Quotes
1. One method's paper ttpery suggests that it's inherently bet- ter (42%).
"I pick IG explanation because that paper seems more rig- orous and shows improvements over other baselines including gradient "inpul"
2. One explanation matches intu- ition better (26%).
"The gradient explanation method gives a positive attribution to features that seem intuitively relevant for predicting recidi- vism (e.g., two year recid, priors counl, length of stay)." "
3. LIME/SHAP are better for tabular data (26%).
"SHAP is more commonly used for tabular data"
Table 7: Themes summarizing how industry participants decided between explanations when faced with disagreement, along with quotes.
34


Reviews:
[{"review_id": "STci1Fh58k", "reviewer": "Reviewer_Mz9B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* I believe the setup of the paper is really strong. The disagreement problem is well known in the community, but having a thorough analysis of how and what happens when methods disagree in practice is useful to broadcast.\n* It was useful and interesting to see the results from interviewing practitioners to get an understanding of how the methods are used in practice, and good to see they defined metrics based on those insights (even if similar metrics have been used in the past to evaluate or compare new interpretability methods against prior work).\n* They used a number of models and data types, and provided a thorough result set, as well as an in-depth analysis together with hypotheses for when disagreement seems higher based on model and data characteristics.\n* The user studies were well designed to get the necessary insights for the study.\n* Presenting the insights based on the day to day workflow is a useful viewpoint, and a good way to present the results as it can lead more easily into \"what should be changed in the future\" recommendations.\n\nWeaknesses:\n* The number of interview participants is relatively low. It's not clear how diverse the participant set is in terms of skills or prior familiarity with the methods - it's mentioned that they were screened for some basic knowledge of ML, but not what the knowledge ranges look like, and what methods they used before.\n* Given the split between industry and academia, as well as years of experience and familiarity with methods, it would have been interesting to see the split in the results based on this.\n* It would have been interesting to see results on a more complex tabular data set. It's great they started with a simplified setup, but it would have been interesting to see how the findings scale with dataset complexity.\n* I would have liked to see an investigation on the limitations of each method as well, if people are aware and if that factors into their choices.\n* It would be good to have more recommendations for how to apply the insights surfaced, potentially with per-method examples for application spaces.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work looks at the problem of disagreements between explainability methods. The authors provide an in-depth study on 6 such methods (LIME, SHAP, Vanilla Gradient, Gradient x Input, SmoothGrad, Integrated Gradients) defining a number of disagreement metrics and quantifying the performance of pairs of methods using these metrics and a series of different types of models and datasets. In addition, they conduct a number of interviews with data scientists on how they use explanation methods, how they view disagreements and how they resolve disagreements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None of these are submission critical, but it would just strengthen the work:\n* Adding more background on the study participants (happy to have it added in the appendix).\n* Splitting the interview results based on tenure, industry vs academia, as well as method familiarity groups.\n* Adding more recommendations for how to apply these insights/what's next.\n* Showing how the results scale for simple and complex datasets in each category.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iFOxqKaCB2", "reviewer": "Reviewer_jStG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Among the first works to study disagreement between explanation methods (as opposed to inconsistency or instability of a single method)\n- I think the combination of human-centered approaches (gathering input from practitioners on what constitutes disagreement and how they resolve it) and more technical approaches (definition of metrics, empirical evaluation across explanation methods and datasets/modalities) is great.\n- The paper is written very clearly overall.\n\n## Weaknesses\n1. While I do not disagree with the main finding that explanation methods significantly disagree, I wonder whether the degree might be overstated. Specifically:\n    1. Rank agreement: This metric defines disagreement as anything other than an exact match in feature ranks. However, a difference in ranks of 1 is better than a difference of say $k$ (so that one of the ranks is not even in the top $k$). What about computing a softer version of rank agreement that takes the difference in ranks into account?\n    1. In the experiments (Section 4.2), rank correlation and pairwise rank agreement are computed across all features, whereas in Section 3.2.2, they are intended to be focused on features of interest to the user. I understand that without an actual user, there may be no good way of selecting \"features of interest.\" But using all features may not be reasonable either, because many features are likely to be equally unimportant (have true importance scores near zero) and it is not reasonable or necessary to expect two explanation methods to rank them in the same order. One possibility is to also restrict rank correlation and pairwise rank agreement to the top $k$ for different values of $k$.\n1. Conclusions that I think are questionable:\n    1. Page 8, \"rank agreement and signed rank agreement are lower for the German Credit dataset than for the COMPAS dataset at top 25%, 50%, 75%, and 100% of features\": I think a better comparison would be to use the same number $k$ of top features, not the same percentage. As the rest of the paragraph notes, the German Credit dataset has more features so agreement could be lower just because of this reason.\n    1. Figure 6b, aggregation of explanation method choices over all prompts: Did the authors adjust for the levels of disagreement in the 15 selected examples? It could be that within this set of 15, the average level of disagreement seen by each method varies considerably. Without adjusting for disagreement level, I am not sure if anything should be inferred from Figure 6b.\n1. Inconsistencies:\n    1. Abstract and conclusion, \"eight predictive models\": I count only six, four for the tabular datasets, and one each for text and images.\n    1. Page 10, paragraph 2, \"Integrated Gradients has the lowest correlation with the rest of the gradient-based methods, this correlation is still significantly higher than its correlation with KernelSHAP and LIME\": I do not see this either from Figure 3 middle panel or Figure 4 upper left panel. All of the mean rank correlations for Integrated Gradients are in the same range (high 0.10's to high 0.20's).\n    1. Same paragraph, \"rank correlation of 0.4-0.6 for LIME as opposed to 0.2-0.4 for KernelSHAP\": Again I do not see this from Figure 4 upper left panel.\n    1. Section 5.1, \"19 participants in this study also participated in the interviews in Section 3.1\": As I understand it, all participants in Section 3.1 were from industry. So how can there be only 12 industry participants in Section 5.1?\n\n### Minor comments and questions\n- Section 2: How could one substantiate that Han et al. (2022), Banegas-Luna et al. (2023) are \"subsequent works\"? From the date of the first pre-print version of this submission (which I understand cannot be disclosed yet due to anonymity)?\n- Section 3.2: I would consider moving the formal definitions of metrics from Appendix A into this section. They would clarify the discussion and would not take that much space. If space is a concern, the writing in the rest of Section 3 could be condensed as it is a bit repetitive. Also in the metric definitions in Appendix A.1, I think the union over $f \\in F$ is unnecessary, it can just be a single set { $f \\in F | \\dots $ }.\n- \"Features of interest\": The first time that this term is introduced, it could be clarified that they are of interest to the data scientist or domain expert.\n- \"State-of-the-art\" explanation methods: I think this term is repeated too much and could even be omitted, considering that none of the explanation methods evaluated were published later than 2017.\n- Figure 1 caption (and subsequent captions), \"considered non-trivial if less than 75% ... if rank correlation is less than 0.50\": What is the basis for these thresholds? In any case, I do not see them referred to in the main text.\n- Page 8 below Figure 2, \"wide range of values across explanation method pairs ... when explaining multiple data points\": I think this should be rephrased. The point seems to be about variation across data points, not across method pairs. A related question is whether the left panel of Figure 3 is showing the distributions over data points corresponding to the means shown in Figure 1 upper left panel.\n- Page 11, paragraph 2, \"assume that the criminal defendant's risk of recidivism is correctly predicted to be high risk\": Why are all the examples high risk? Are the predictions indeed correct?\n- Page 11, paragraph 3, \"we chose a different data point from COMPAS to run the two methods, giving us a set of 15 prompts for each of the 15 pairs of explanation methods\": This should also be rephrased. Does this mean that each of the 15 pairs is run on a different data point?\n- Introduction and conclusion: I think the headline percentages (84% encountered disagreements, 86% resolved heuristically) should be accompanied by the actual numbers of participants (21 of 25, 12 of 14, if I understand correctly). The number of participants is not large, and while I understand it is difficult to recruit more participants with expertise, I think it is better to be forthright about the numbers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission studies disagreements between post hoc explanations provided by different methods for the same prediction. In the first part of the work, interviews are conducted with data scientists to determine the extent of the disagreement problem and what they view as constituting disagreement. Based on the interview findings, six metrics are proposed to quantify disagreement, four defined in terms of top-k features, and two in terms of features of interest to the practitioner. The authors then empirically assess disagreement according to these metrics across six explanation methods and four datasets covering tabular, text, and image modalities. Finally, another user study is performed to characterize how practitioners resolve disagreements between explanations. The most prominent findings are that disagreements between methods often occur, and that practitioners often use heuristics to resolve disagreements.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think it is important to address weaknesses 1 and 2 above (i.e., perform additional analysis or make a best effort to). These are the reason why I have answered \"No\" for now to the \"Claims and Evidence\" question, where \"No\" really means \"not always.\" The inconsistencies should be easy to address. Addressing the minor comments would strengthen the submission.\n\nIn addition, I have one question for curiosity:\n- Section 4.3.3: Do the authors think that the higher agreement between KernelSHAP and LIME could be due to super-pixels being at a coarser (and less noisy) level of granularity?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TLkXGUe1Av", "reviewer": "Reviewer_HUrm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper provided interesting insights into the difficulty of using model explanation methods in practice and potential biases involved in the selection of methods. The qualitative studies with human users provided consensus among data practitioners about the presence of the issue.   \n\n- The paper is well-written and cohesive. The research questions and contribution are clear and well-supported by a reasonable experimental setup.  \n\n**Weaknesses:**\n\n- The authors should discuss more on the differences in the disagreement patterns among families of methods. For example, LIME uses local linear approximations, SHAP is based on Shapley value while IntGrad, SmoothGrad are gradient-based. This means they use different data signals to measure the “importance” scores and these technical differences should be taken into account when comparing methods. \n\n- The explanation methods under analysis are quite old. There are another line of methods where the explanation model is also a deep learning  model, notably L2X (Chen et al., 2018). \nChen et al. (2018). Learning to Explain: An Information-Theoretic Perspective on Model Interpretation.\n\n- The conclusion from the online user study might make the studied problem less important. \n\n*“One approach to select among explanation methods is to first define the desired goal of an explanation and then select the explanation method that best meets this goal.”*\n\nThis means one can in practice ignore the disagreement and select the method that is superior on a metric of interest, such as prediction fidelity as an indicator which one yields the most desirable explanations. This is basically what the research community is proposing, where the proposed method would surpass the baselines on certain desiderata, which also a factor explaining for the disagreement in features. Therefore, it is unclear whether the disagreement is truly a critical issue in practice or it can simply be resolved based on a set of comprehensive metrics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studied the disagreement problem in post-hoc explanations of machine learning models. The authors conducted interviews with data practitioners and obtained insights into the nature of the problem and approaches to dealing with it in practice. The paper also presented a set of metrics to quantify disagreement followed by a quantitative analysis of popular explanation methods in various settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the above section.", "broader_impact_concerns": "There are no discernible ethical concerns about this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I encourage the authors to revise their manuscript based on the reviewers' comments, adding an overall description of the changes made as a comment. I believe the inclusion of the more lenient version of the ranking metric is reasonable, and it should be clearly mentioned in the text how variable the conclusions may be based on which metrics are used for the comparison. This is especially true given that multiple choices may be practical rather than theoretical (e.g. selecting proportions rather than top-k numbers to compare across datasets). For instance, mentioning clearly that a perfect match is expected is important to put the findings in perspective.
Decision: Accept (poster)