What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods
Authors: Julien Colin
Thomas FEL
Remi Cadene
Thomas Serre
Venue: NeurIPS 2022 Accept (2022)
Keywords: Explainability
Interpretability
Human-centered
Evaluation

Abstract: A multitude of explainability methods has been described to try to help users better understand how modern AI systems make decisions. However, most performance metrics developed to evaluate these methods have remained largely theoretical -- without much consideration for the human end-user. In particular, it is not yet clear (1) how useful current explainability methods are in real-world scenarios; and (2) whether current performance metrics accurately reflect the usefulness of explanation methods for the end user. To fill this gap, we conducted psychophysics experiments at scale ($n=1,150$) to evaluate the usefulness of representative attribution methods in three real-world scenarios. Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system varies widely across these scenarios. This suggests the need to move beyond quantitative improvements of current attribution methods, towards the development of complementary approaches that provide qualitatively different sources of information to human end-users.

What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods
Julien Colin 1,3,5 +
Thomas Fe14.3,4+
Rémi Cadène $
Thomas Serre! 1.3
*Carney Institute for Brain Science, Brown University, USA 2Sorbonne Université, CNRS, France Artificial and Natural Intelligence Toulouse Institute, Université de Toulouse, France Innovation & Research Division, SNCF
ELLIS Alicante, Spain
(julien_colin, thomas_fel, remi_ @brown.
Abstract
A multitude of explainability methods has been described to try to help users better understand how modern Al systems make decisions. However, most performance metrics developed to evaluate these methods have remained largely theoretical without much consideration for the human end-user. In particular, is not yet clear (1) how useful current explainability methods are in real-world scenarios; and (2) whether current performance metrics accurately reflect the usefulness explanation methods for the end user. To fill this gap, conducted psychophysics experiments at scale (n = 1, 150) to evaluate the usefulness of representative attribution methods in three real- wolld scenarios. Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system varies widely across these scenarios. This suggests the need to move beyond quantitative improvements of current attribution methods, towards the development of complementary approaches that provide qualitatively different sources of information to human end-users.
Introduction
There is now broad consensus that modern systems might not be safe to be deployed the real world [1] despite their exhibiting very high levels of accuracy on held-out data because these systems have been shown to exploit dataset biases and other statistical shortcuts [2-5]. growing body of research thus focuses on the development of explainability methods to help better interpret these systems' predictions [6-14] to make them more trustworthy. The application of these explainability methods will find broad societal uses, like easing the debugging of self-driving vehicles [15] and helping to fulfill the "right to explanation" that European laws guarantee to its citizens [16].
The most commonly used methods in eXplainable AI (XAI) are attribution methods, but despite plethora of different approaches [6-10, 12, 17-19], assessing the quality and reliability of these meth- ods remains an open problem. So far the community has mostly focused on evaluating these methods using surrogate measures defined axiomatically [15, 20-23]. The two most popular approaches are: (1) using ground truth annotations [9, 10, 12, 24-27] and (2) measuring faithfulness using objective metrics [9. 12, 24, 25, 28, 29].
The first approach takes humans into consideration only in so far as it evaluates the alignment of an explanation with a human annotation. This metric assumes that the classifier relies on human-
Equal contribution Work done while the author was working at Brown University $ Work done before April 2021 joining Tesla
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

like features', which is an assumption that turns out to be erroneous [2-4], and unsurprisingly the results obtained on those benchmarks do not correlate with the ability of attribution methods to help humans in decision making tasks [30]. On the other hand, the second approach focuses purely on the relationship between the explanations and the model, i.e., they assess whether the explanations actually reflect the evidence for the prediction. They have emerged as the primary means of evaluating attribution methods, but because they take the human out-of-the-lop of the evaluation, it is not clear how well they can predict the practical usefulness of attribution methods and their possible failures.
As previously highlighted in [30], relying solely on current theoretical benchmarks to evaluate attribution methods can be risky endeavor as they are totally detached from humans. Indeed, the seminal work of [31] has emphasized keeping in mind the end goal of XAI: to develop useful methods, i.e., methods that help the user better understand their model. Thus, they recommend systematically having recourse to human experiences. Moreover, a large body of research from the psychology literature has endorsed the argument that the functional role of explanations is to support learning and generalization [32-38]- - i.e., can an explanation help identify generalizable rules that readily transfer to unseen instances? Therefore, evaluating attribution methods on their ability to help users understand general rules to predict a classifier decisions is perfectly aligned with this vision.
To that end. [31] suggest evaluating explainability methods directly through the end goals of XAI. We subscribe to this idea and, in this work, consider three main real-world scenarios that illustrate potential applications of explainability methods: (1) identifying a potential bias in a system's 's decision along with its source using the classic Husky vs. Wolf dataset [6] (previous work has documented the risk associated with a biased model [39-41]] (2) identifying novel strategies discovered by system [42, 43] for tasks that would be nearly impossible for humans [44-4]], here we consider a highly complex categorization problem for non-expert humans [48, 49]; and (3) understanding failure
cases [4, 39] on a subset of ImageNet [50].
The main contributions of this paper are as follows:
We propose a novel human-centered explainability performance measure together with associated psychophysics methods to experimentally evaluate the practical usefulness of modern explainability methods in real-world scenarios.
Large-scale psychophysics experiments (n 1, 150) revealed that SmoothGrad [8] is the most useful attribution method amongst all tested and that none of the faithfulness performance metrics appear to predict if and when attribution methods will be of practical use to human end-user.
Perceptual scores derived from attribution maps, characterizing either the complexity of expla- nation or the challenge associated with identifying "what" features drive the system's 's decision, appear to predict failure cases of explainability methods better than faithfulness metrics.
2 Related work
Evaluations based on faithfulness measures Common approaches [9. 28] measure the faithfulness of an explanation through the change in the classification score when the most important pixels are progressively removed. The bigger the drop, the more faithful is the explanation method. To ensure that the drop in score does not come from change in the distribution of the perturbed images, the ROAR [51] methods include an additional step whereby the image classifier is re- re-trained between each removal step. Because these methods do not require ground-truth annotations (i.e. object masks or bounding boxes), they are quite popular in computer vision [9, 12, 24, 25, 28, 29] and natural language processing [12, 52, 53].
Nevertheless, faithfulness measures have recently been criticized as they all rely on a baseline for removing important areas, a baseline that will obviously give better scores to methods relying internally on the same baseline [54]. More importantly, they do not consider humans at any time in the evaluation. As a result, it is unclear if the most faithful attribution method is practically useful to humans.
Evaluations based on humans A second class of approaches consists in evaluating the ability of humans to leverage explanations for different purposes [6, 10, 30, 55-62]. [6] were the first to evaluate the usefulness of explanations. Their work focused on the use case of bias detection: they
"One advantage of our approach is that it is independent of the classifier, whether it relies on human-like features or it is even correct.

---

(1) Wolf vs Husky Discover blas
(2) Leaves
(3) ImageNet
Identify new strategies
Understand failure cases
'Wolf"
'Husky"
'Woif'
'Betulacead' Celastracese"
'Celastracese"
'Kit fox"
'Red fax'
'Kit fax'
We study the practical usefulness of recent explainability methods in three real-world scenarios, each corresponding to different use cases for XAI. The first dataset is Husky vs. Wolf where the goal of the explanations is to help the user to identify a source of bias in a model (classification is based on the background (snow, grass) as opposed to the animal). The second dataset corresponds to a real-world leaf classification problem which is complex for non-experts. The goal of the explanations is to help the end-user identify the strategy discovered by the vision system. Finally, the third dataset is subset of ImageNet, which consists of a collection of images where half have been misclassified by the system. The goal of the explanations here is to help the end-use und- understand the failure sources of high performing model.
Figure 1:
trained a classifier on biased dataset of wolves and huskies and found that the model consistently used the background classify. They asked participants if they trusted the model before and after seeing the explanation for the model's predictions, and found that explanations helped detect bias here. We use a similar dataset to reproduce those results, but our evaluation differs greatly from theirs as we do not ask if participants trust the model but instead measure directly if they understand it.
Closest to our work are [59-61, 63, 64]. [59-61] design their evaluation around the notion of simulatability [31, 65]. They introduce different experimental procedures to measure if humans can learn from the explanations how copy the model prediction on unseen data. Some provide the explanations at test time [59, 60]. Similar to us but for tabular data, [61] proposes to hide explanations at test time, this forces the participants to learn the rules driving the model's decision at training time where the explanations are shown. There are two limitations to their work: (1) they provide ground-trune tound-truth labels associated with input images during training, (2) the participants see the same set of images without explanations, and then with explanations, always in that order. This creates learning effects that can heavily bias their results. We differ from their work by: (1) removing ground-truth labels from framework as they serve purpose and can bias participants, and (2) we have different participants go through the different conditions. This removes any learning effect, and more importantly, new explainability methods can be evaluated independently and still be compared to the previously evaluated methods. A recent study [63] evaluated how AI systems may be able assist human decisions by asking participants to identify the correct prediction out of four cediiciion-xpllaaaiooss pairs shown simultaneously. This measure reflects how well explanations help users tell apart correct from incorrect predictions. While the approach was useful evaluate explanations in this specific scenario, it is not clear how this framework could be used to evaluate explainability methods more generally. Furthermore, when comparing different types of methods, they adapt the complexity of certain explainability methods to ease the task for participants. We argue that the complexity of explanations is an important property of explanations and that abstracting away from the evaluation lead to unfair comparisons between methods. In contrast, we propose a more general evaluation framework that can be used for any kind of explainability method without the need to adapt them for the evaluation procedure - hence allowing for an unbiased and scalable comparison between methods. Finally, [64] proposes to evaluate if users are able to identify important features biasing the predictions of a model using synthetic dataset. By controlling the generation process of the dataset, they have access to the ground-truth attributes biasing the classifier, and can measure the accuracy of users at identifying these features. They evaluate if concept-based or counterfactual explanations help users improve over a baseline accuracy when no explanations are provided, and find no explanation tested to be useful. While both works highlights the importance of human evaluation, they differ in: the metrics employed (identifying relevant features for the model vs. meta- prediction), the type of dataset used (synthetic vs. real-world scenarios), and the type of methods evaluated (counterfactual and concept-bant based methods vs. attribution methods).
3

---

K times
b6 - f(Ø)
Wolf
(6) + f())
{xc
((f.)),
f())
Meta-predictoo
Learning phase
Evaluation phase
Figure 2: We describe human-centered framework to evaluate explainability methods borrowing the concept of Meta-predictor. The framework requires a black box model f (the predictor), an explanation method and human subject &b which will try to predict the predictor, hence, the name Meta-predictor. The first step is the learning phase where the Meta-predictor is training using K samples æ, together with the associated model predictions f(x) and explanations Φf x). The goal of this learning phase is for the Meta-pue reta-predictor to uncover the rules driving the decisions of the model from the triplets (æe, ( æ), f (a)). Then, the second step is the evaluation phase where we test the Meta-predictor's ability to correctly predict the model's outputs
new samples ø by comparing its predictions (b) to those of f()). The Utility score of the explanation method is then computed as the relative accuracy improvement of Meta- Meeaa-predictor trained with vs. without explanations.
3 Proposed evaluation framework
Before providing a rigorous definition of interpretability, let us motivate our approach with an example: a linear classifier is often considered to be readily interpretable because its inner working is sufficiently intuitive that can be comprehended by human user. user can in turn build mental model of the classifier predicting the classifier's 's output for arbitrary imputs. In essence, we suggest that the model is interpretable because the output can be predicted i.e, we say we understand the rules used by model, if we can use those inferred rules to correctly predict its output. This concept of predicting the classifier's output is central our approach and we conceptualize the human user as Meta- Meta-predictor of the machine learning model. This notion of Meta-predictor is also closely related to the notion of simulatability [24, 31, 61, 65, 66]. We will now define the term more formally.
We consider a standard supervised learning setting where f is black-box predictor that maps an input x X (e.g., an image) an output f(x) E y (e.g., a class label). One of the main goals of eXplainable Al is to yield useful rules to understand the inner- inner-working of a model f such that it is possible infer its behavior on unseen data points. To correctly infer those rules, the usual approach consists in studying explanations (from Attribution Map, Concept Activation Vectors, Feature Visualization, etc..) for several predictions. Formally, + is any explanation functional which, given predictor f and a point æ. provides information 4 .)) about the prediction of the predictor. In our experiments, c is an attribution method but we would like to remind that the framework is naturally adaptable to other explainability methods such concept-based methods or feature visualization.
The annderstandabillly-compeeee duce different heatmaps potentially highlighting different image regions and/or presenting the same information in different format. The quality of an explanation can thus be affected by two factors: faithfulness of the explanation (i.e., how many pixels or input dimensions deemed important effec- tively drive the classifier's 's prediction) and the understandability of the explanation for an end-user (i.e., how much of the pattern highlighted by the explanation is grasped by the user).
trade-off Different attribution methods will typically pro-
At one extreme, an explanation can be entirely faitiful and provide all the information necessary to predict how a classifier will assign a class label an arbitrary image (i.e.. by giving all the parameters of the classifiers). However, such information will obviously be too complex to be understood by user and hence it is not understandable. Conversely, an explanation that overly simplifies the model might offer an approximation of the rule used by the model that will be more easily grasped by the user -a more understandable explanation- but this approximation might ultimately mislead the user if is not not faithful. That is to say, just because human agrees with the evidence pointed out by an explanation does not necessarily mean that reflects how the model works.
Overall, this means that there is a trade-off between the amount of information provided by an explanation and its comprehensibility humans. The most useful explanations should lie somewhere in the middle of this tra

---

The usefulness metric We describe a new into a single usefulness measure by empirically evaluating the ability of human participants to learn to "predict the predictor", i.e., to be an accurate Meta- Meta-predictor. Indeed, if an explanation allows users to infer precise rules for the functioning of the predictor on past data, the correct application of these same rules should allow the user to correctly anticipate the model's 's decisions on future data. Scrutable but inaccurate explanations will result in an inaccurate Meta-prediictr - just like accurate inscrutable ones. This Meta- Meta-predictor framework avoids current pitfalls such as confirmation bias just because a user likes the explanation does not mean they will be a better Meta-prediccorr or prediction leakage the explanation in simulatability experiments, the explanation is available during the test phase, any explanation that leaks the prediction would have a perfect score, without giving any additional information about the model. We will now formally describe the metric build using this framework.
measure that incorporates this trade-off
We assume dataaet" D = (ax;, f(:;), (ff m;)) used to train human participants to learn to predict a classifier's ss output from K samples made of an input image 2;, the associated predictions f (2x) and explanations $ (f 22). We denote (b(K), a human reta--rediccor after being trained on the dataset D (see Fig. 2) using explanations. In addition, let g6(00) be the human Meta-predicte after participants were trained on the same dataset but without explanations to offer baseline accuracy scores. We can now define the usefulness of an explainability method & after training participants on K' samples through the accuracy score of the Meta--redictor normalized by the baseline Meta- predictor accuracy:
Utility-K (((b(K)()) (x) = f(x)) (x))
(I)
= fax))
with IP(-) the probability over a test set. Thus, Utility-K score measures the improvement in accuracy that the explanation has brought. It is important emphasize that this Utility measure only depends on the classifier prediction and not on the ground-truth label as recommended by [67]. After fixing the number of training samples K, we compare the normalized accuracy of different Meta-predictors. The Meta-predictor with the highest score is then the one whose explanations were the most useful as measures compared to a no-explanation baseline.
Utility metric In practice we propose to vary the number of observations K E { (Ko.... Kn) and to report aggregated Utility score by computing the area under the curve (AUC) of the Utility- K. The higher the AUC the better the corresponding explanation method is. Formally, given a curve represented by set of n points C- [Ko, Utility-pp) ... (Kn, Utility-K,,) where Ki-1- < K; we define the metric as Utility = AUC(C).
Experimental design
We first describe how participants were enrolled in the study, then our general experimental design (See SI for more informations).
Participants Behavioral data were gathered from n = 1, 150 participants using Amazon Mechan- ical Turk (AMT) (www mturk. com). All participants provided informed consent electronically and were compensated $1.4 for their time (~ 5 - 8 min). The protocol was approved by the Univer- sity IRB and was carried out in accordance with the provisions of the World Medical Association Declaration of Helsinki. For each of the three tested datasets, we ensured that there was a sufficient number of participants after filtering out uncooperative participants (n 240 participants, 30 per condition, 8 conditions) to guarantee sufficient statistical power (See SI for details). Overall, the cost of evaluating one method using our benchmark is relatively modest ($50 per test scenario).
General study design It included 3 conditions: an experimental condition where explanation is provided to human participants during their training phase (see Fig. 2), a baseline condition where no explanation was provided to the human participants, and control condition where bottom-up saliency map [68] was provided as a non-informative explanation. This last control is critical, and indeed lacking from previous work [6. 61]. because it provides a control for the possibility that
We note that, in this paper, we only considered binary dataset -Class vs Class 2- hecause having the participants classify more than 2 classes would increase their cognitive load and bring unnecessary difficulty to the task. Nonetheless, any dataset could have heen used classification problems with more than 2 classes can always trivially reformulated as Target class vs. Other binary classification problems, instead of Class vs Class 2, without lack of generality.
5

---

Method
Husky vs. Wolf
Leaves
ImageNet
Session n²
3
Utility
2
Utility
Utility
Baseline Control
55.7
66.2
62.9
70.1
76.8
78.6
58.8
62.2
58.8
53.3
61.0
61.4
0.95
72.0
78.0
80.2
1.02
60.7
59.2
48.5
0.94
Saliency [17]
53.9
69.6
73.3
1.06
83.2
88.7
82.4
1.13
61.7
60.2
58.2
1.00
Integ.-Grad [7]
67.4
72.8
73.2
1.15
82.5
82.5
85.3
1.11
59.4
58.3
58,3
0,98
SmoothGrad [8]
68.7
75.3
78.0
1.20
83.0
85.7
86.3
1.13
50.3
55.0
61.4
0,93
GradCAM [10]
77.6
85.7
84.1
1.34
81.9
83.5
82.4
1.10
54.4
52.5
54.1
0.90
Occlusion [18]
71.0
75.7
78.1
1.22
78.8
86.1
82.9
1.10
51.0
60.2
55,1
0.92
Grad. Input [69]
65.8
63.3
67.9
1.06
76.5
82.9
79.5
1.05
50.0
57.6
62.6
0.95
Table 1: Utility-K K and Utility scores. Utility- Sillity-Kscoees across the 3 sessions for each attribution method, for each of the 3 datasets considered, followed by the Utility scores. Higher is better. The Utility scores of attribution methods that are statistically significant are bolded.
providing explanations along with training images simply increases participants' engagement in the task. As we will show in Sec. 5, such non-in non-informative explanations actually led to a decrease in participants' ability to predict the classifier's decisions suggesting that giving a wrong explanation is worse than giving no explanations all.
Each participant was only tested on a single condition to avoid possible experimental confounds. The main experiment was divided into 3 training sessions (with training samples in each) each followed by a brief test. In each individual trial, an image was presented with the associated prediction of the model, either alone for the baseline condition or together with explanation for the experimental and control condition. After a brief training phase (5 samples), participants' ability to predict the classifier's output was evaluated explanation was provided to limit confounding effects: one possible effect is if the explanation leaks information about the class label. E We also propose use a reservoir that subjects can refer to during the testing phase to minimize memory load as a confounding factor which was reported in [61] (see SI for an illustration).
7 new samples during test phase. During the test phase, no
Datasets and models We performed three distinct experiments in total using a variety of neural network architectures and 6 representative attributions methods. Each of these experiments aimed at testing the usefulness of the explanation in a different context.
Our first scenario focuses on the detection of biases in Al systems using the popular Wolf vs. Husky dataset from [6] where an evaluation measure was already proposed around the usefulness of explanations for humans to detect biases. This makes it a good control experiment to measure the effectiveness of the framework proposed in Sec. 3. For this first experiment, as in the original paper: InceptionV1 [70], and a similar dataset of Husky and Wolf images to bias the model. In this situation where prior knowledge of subjects can affect their Meta-predictor score, we balance data correctness (50% of correct/ examples shown). Therefore, a subject relying only their prior knowledge will end up bad Meta-predictor of the model. For this experiment, the results come from n 242 subjects who all passed our screening process.
used the same model
In our second scenario, we focus on representative challenging image categorization task which would be hard to solve by a non-exper untrained human participant and the goal is for the end-user to understand the strategy that was discovered by the AI system. Here, we chose the leaf dataset described in [48]. We selected 2 classes from this dataset (Betulaceae and Celastracea) that could not be classified by shape to reduce the chances that participants will discover the solution on their own - forcing them instead to rely on non non-trivial features highlighted by the explanations (veins, leaf margin, etc). This scenario is far from being artificial as it reflects a genuine problem for the paleobotanist [49]. Can explainability methods help non-specialists discover the strategies discovered by an AI system? As participants are lay people from Amazon Mechanical Turk we do not expect them to be experts in botany, therefore we did not explicitly try to control for prior knowledge. In this experiment, 240 subjects passed all our screening and other filtering processes.
*Imagine an attribution method that would solely encode the classifiers' prediction. Participants would be able to guess the classifier's prediction perfectly from the explanation but the explanation se would not help participants understand how the classifier works.

---

1.4
1.3
1.2
1.0
1.0
0.9
0.8
0.9
10
15
10
15
Training examples (K)
Training examples (K)
Occlusion
Sallancy
Occlusion
Sallency SmoothGrad
Integrated Gradient
Control
SmoothGrad
Integrated Gradient
Control
Input Gradient
Input Gradient
Figure 3: Utility- for both Husky vs. Wolf (left) and the Leaves (right) dataset. The Utility- K of
the explanation, or the accuracy of the human Meta-predictor after training, measured after each training session (3 in total) for the scenario (1) of bias detection (on the left) and the scenario (2) concerning the identification of new strategies. Concerning the first scenario, all methods have positive effect on the score obtained they improve the subjects' ability predict the model and are thus useful to better understand the model. Grad-CAM, Occlusion and SmoothGrad are particularly useful for bias detection. On the Leaves dataset [48], explanations are also useful, but specifically Saliency, SmoothGrad and Integrated Gradients.
Finally, our last scenario focuses on identifying cases where an Al system fails using ImageNet [50], also used previous explainability work [12, 24, 30, 51, 59, 71]. We used this dataset because we expect it to be representative of real-world scenarios where it is difficult understand what the model relies on for classification which makes it very difficult to understand these failure cases. Moreover, previous work has pointed out that attribution methods are not useful on this dataset [59], we have thus chosen extend our analysis to this particular case. We use a ResNet50 [72] pretrained on this select dataset predictor. Because prior knowledge is major confounding factor on ImageNet,
a pair of classes that was heavily miss-classified by the model, to be able to show subjects 50% of correct/incorred predictions: the pair Kit Fox and Red Fox fits this requirement. In this experiment, we analyzed data from n 241 participants who passed our screening and filtering processes.
For all experiments. we compared 6 representative attribution methods: Saliency (SA) [17]. Gradient @ Input (GI) [19], Integrated Gradients (IG) [7], Occlusion (OC) [18], SmoothGrad (SG) [8] and Grad-CAM (GC) [10]. Further information on these methods can be found in SI. Table summarizes all the results from our psychophysics experiments.
Results
Scenario (1): Bias detection Fig. shows the Utility-K scores for each method after different numbers of training samples were used to train participants for the biased dataset of Husky vs. Wolf. The Utility score encodes the quality of the explanations provided by a method, the higher the score, the better the method, with the baseline score being (every score is divided by the baseline score corresponding to human accuracy after training without explanations).
first observation is that the explanations have a positive effect on the Utility- score: the explanation allows participants to better predict the model's 's decision (as the Utility scores above 1). These results are consistent with those reported in [6]. This is confirmed with an Analysis of Variance (ANOVA) for which 9.19, < .001, 1/2 = 0.089). Moreover, the only score below the baseline is that of the control explanation, which do not make use of the model. We further explore our results by performing pairwise comparisons using Tukey's 's Honestly Significant Difference [73] to compare the different explanations against the baseline. We found 3 explainability methods to be significantly better than the baseline: Grad-CAM (p < 0.001), Occlusion (p 0.01) and SmoothGrad (p 0.034). Thus, participants who received the Grad-CAM, Occlusion or SmoothGrad explanations performed much better than those who did not receive them.
found significant main effect, with a medium effect size (F(77 234)
'We acknowledge the existence of some overlap between the scenario I and scenario 3 as bias detection is a special case of a failure case. The reason we still use scenario is because of the work previously done on it, allowing validate framework.
7

---

making attribution Scenario (2): Identifying an expert strategy In Fig. 3, we show results on the Leaves dataset. An ANOVA analysis across all conditions revealed a significant main effect, albeit small (F(77, 232) 4.29, p < -00,, 17² = 0.042). This implies that explanation also had positive effect resulting in better deta-predictor in this use case. A Tukey's Honestly Significant Difference test suggests that the best explanations are Saliency, SmoothGrad and Integrated Gradients as they are the only ones to be significantly better than our baseline (WE) (p = .004, p = .007 and p = .03 respectively). An interesting result is that SmoothGrad seems to be consistently useful across both use cases where explanations are indeed practically useful. A more surprising result is that Saliency which was one of the worst explanations for bias detection, is now the best explanation on this use case (We discuss possible reasons in SI).
Scenario (3); Understanding failure cases Table 1 shows that, on the ImageNet dataset, none of the methods tested exceeded baseline accuracy. Indeed, the experiment carried out, even with an improved experimental design, led us to the same conclusion as previous works [59]: none of the tested attribution methods are useful (ANOVA: F7,, 233) 1.26, p .05). In the use case of understanding failure cases on ImageNet, no attributions methods seem to be useful.
Why do attribution methods fail?
After studying the usefulness of attribution methods across 3 real-world scenarios for eXplainable AI, we found that attribution methods help, sometimes, but not always. We are interested in better understanding why sometimes attribution methods fail help. Because this question has yet to be properly studied, there is no consensus if we can still make attribution methods work on those cases with incremental quantitative improvements. In the follow-up sections explore 3 hypothesis to answer that question.
Faithfulness as a proxy for Utility? Faithfulness often described as one of the key desiderata for a good explanation [23, 74, 75]. If an explanation fails to be sufficiently faithful, the rules it highlights won't 't allow user to understand the wnner-n of the model. Thus, a lack of faithfulness on ImageNet could explain our results. To test this hypothesis, we use the faithfulness metrics: Deletion[9, 28], com- monly used to compare attribution methods [9, 12, 24, 25, 28, 29]. A low Deletion score indicates a good faithfulness, thus for ease of reading we report the faithfulness score as 1- Deletion such that a higher faithfulness score is better.
1.6
-0.93. = 0.006 (Leeves] p= -0.50, 0.315 (imagenet)
(Husky)
1.4
p= 0.81, p 0.052
1.2
1.0
0.0
0.2
0.4
0.6
0.0
1.0
Falthfulness (Deletion)
Grad CAM Integrated Gradient
Inout Grodiant Occlusion
Sallency
SmoothGrad
Figure 4: Utility Faithfulness correlation. The results suggest that current faithfulness metries are poor predictors of the end-goal usefulness of explanation methods. Concern- ing the ImageNet dataset (triangle marker), the Utility scores are insignificant since none of the methods improves the baseline.
Fig shows the linear relationship between our Util- ity metrics and the faithfulness scores computed for every attribution method across all datasets. We observe two main trends: 1) There does not appear to be any specific pattern regarding faithfulness that could explain why attribution methods are not useful for ImageNet, and 2) the least useful at- tribution methods for both use cases for which methods help (Bias and Leaves) some of the leading methods in the field measured by the faithfulness metric. We also found a weak, if maybe anti-correlated relation between faithfulness and usefulness: just focusing methods more faithful does not translate to having methods with higher practical usefulness for end- users. And, in fact, focusing too heavily on faithfulness seem to come at the expense of usefulness, resulting in explanations that are counter-intuitively less useful. This second observation may seems rather alarming for the field given that the faithfulness measure is one of the driving benchmarks.
Are explanations too complex? Using the trade-off between completeness and understandability previously discussed in Section 3, we formulate another hypothesis: some explanations may be faithful but too complex and therefore cannot be understood by humans. In that view, an explanation with low complexity would tend to be more useful.
As simple measure of the complexity of visual explanations, it would be ideal to be able to compute the Kolmogorov complexity [76] of each explanation. It was shown in previ-
8

---

ous work [77] to correlate well with human-derived ratings for the complexity of natural im- ages [78, 79]. As suggested by [76, 80] we used a standard compression technique (JPEG) to approximate the Kolmogorov complexity. Fig. 5 shows the Utility vs complexity score of attribution methods for each dataset. For one of the datasets where attribution methods help, the results suggest the presence of strong correlation between usefulness and complexity: the least complex method is the most useful to end-users. For the other datasets, the results are either not conclusive (Leaves), or not relevant methods are not useful (ImageNet).
Overall, across datasets there is no significant dif- ference in the complexity of explanations that explain why attribution methods do not help on Ima- geNet. This could be because the Kolmogorov Com-
(Husky) -088.
(Leaves) p -0.31, p 0.554 (Imagenet) p 0.93, p 0.007
plexity does not perfectly reflect human visual com- plexity, or because this is not the key element to explain failure cases of attribution methods.
1.0
0.8
0.6
An intrinsic limitation of Attribution methods? The role of attribution methods is to help identify "where" to look in an image to understand the ba- sis for system's 's prediction. However, attribution methods do not tell us "what" in those image re- gions is driving decisions. For categorization prob- lems which involve perceptually similar classes (such as when discriminating between different breeds of dogs) and fine-grain fine-grained categorization problems more generally, simply looking diagnostic image regions tells the user very little about the specific shape property being relevant. For instance. knowing that the car shape is being used for recog- nition does not say what specific shape feature is being encoded (e.g., pointed vs. round or narrow vs. broad base, etc). Our main hypothesis is that such a lack of explicit "what" infor- mation is precisely what is driving the failure of attribution methods on our ImageNet use-case.
0.0
0.2
0.1
0.0
1.0
Comploutty
Grad-CAN
Input Gradient
Suliecy Smoothhrred
Integrated Gradient
Occl Occlavion
Figure 5: Utility vs Complexity correlation. This suggest a weak but possibly existing link between Complexity and Utility scores.
To test this hypothesis, we estimated the percep- tual similarity between classes measured within di- agnostic regions (see SI for more details) using the Learned Perceptual Image Patch Similarity (LPIPS) metric [81] as it has been shown to approximate hu- man perceptual similarity judgments well [81, 82]. We report the perceptual similarity score as 1 LPIPS score so that a high score means a high similarity. Fig. 6 shows the correlation between the perceptual similarity scores vs. our Utility scores on all methods and datasets studied. Our results suggest a strong cor- relation between perceptual similarity and practical usefulness: the more perceptually similar discrim- inative features of both classes are, the less useful attribution methods become. More importantly, the results across datasets show that on ImageNet, where attribution methods do not help, every method has a high similarity score. This result suggests that after a certain threshold of perceptual similarity, attribu- tion methods might no longer be useful, no matter how faithful low in complexity the explanation is. Overall, the results suggest that the perceptual similarity of discriminative features could explain why attribution methods fail on ImageNet.
1.6
(Overall correlation) -0.77. 0.000
1.4
1.0
0.8
0.6 0.20
0.25
0.30
0.35
0.40
0.45
0.50
Perceptual Simiierity
Grad-CAM Integrated Gradient
Inout Gredient
Salency
Occlusion
SmosthGrad
Figure 6: Perceptual Similarity scores vs. Urility. The perceptual similarity of high- lighted regions by given attribution method for both classes is measured, for each method, for each dataset. We observe a strong cor- relation between the perceptual similarity of features highlighted for both classes and the practical usefulness of methods.
6 Discussion
In summary, we conducted a large-scale human psychophysics experiment to test the utility of explainability methods in real-world scenarios. Our work shows that in two of the three tested scenarios (bias detection and identification of new strategies), explainability methods have indeed

---

progressed and they provide meaningful assistance to human end-users. Nevertheless, we identified scenario (understanding failure case) for which none of the tested attribution methods were helpful. This result is consistent with previous work [59] and highlights a fundamental challenge for XAI.
Further analysis of associated faithfulness performance metries driving the development of explain- ability methods revealed that they did not correlate with our empirical measure of utility suggesting that they might not be suited anymore to move the field forward. We also investigated the possibility that the complexity of individual explanations may play a role in explaining human failures to learn to leverage those explanations to understand the model and, while we found weak correlation between complexity and our empirical measure of utility, this correlation appears too low to explain the failure of these methods.
Finally, because attribution methods appear to be just as faithful and low in complexity whether they are useful or not, we explored the possibility that their failure lies, not in the quality of their explanations, but in the intrinsic limitations of attribution methods. If fully grasping the strategy of a model requires understanding, not just "where" to look (as revealed by attribution maps) but also "what" to look at, something not currently revealed by these methods, attribution methods will not help. Our assumption is that the need for finer "what" information should arise when diagnostic image locations across classes look perceptually very similar and potentially semantically related for certain classification problems (e.g., looking at the ears or the snout to discriminate between breeds of cats and dogs) and one needs to identify what visual features are driving decisions. We computed a perceptual score for classification problems by estimating the perceptual similarity between diagnostic image regions (as predicted by attribution methods) and found that, indeed, when this score predicts a certain level of perceptual similarity between classes, attribution methods fail to contribute useful information to human users, regardless of the faithfulness or complexity of the explanations. This suggests that explainability methods may need to communicate additional information to the end user beyond attribution maps.
7 Limitation and broader impact
Limitations. Our definition of the usefulness of an explanation is quite general. However, there several limitations associated with our approach to estimate usefulness. First, our approach does not take into account the level of machine learning expertise of the user (the most useful explanation for a novice might not be the best one for an expert). Second, the need for human participants to evaluate explanations brings challenges compared with the automated metrics currently used in the field. However, by releasing all our data and software I, we hope encourage the adoption of the approach to evaluate future explainability methods and to assess the overall progress towards the development of more human-interpretable AI systems.
Broader impacts. While the increasing use of Al in real-world scenarios has shown potential to do good [46-48], it has also shown its potential for harm, especially when models rely on shortcuts (e.g.. relying on spurious correlations in the training set that leads to unintended racial bias [39-41]). To identify such shortcuts, the field of Explainable AI has developed a lot of explainability methods; but it is not always clear which one performs best when trying to understand model. We hope our evaluation method can help in this regard, assisting the practitioner in better identifying bias and shortcuts that may unfairly discriminate against groups of people.
Acknowledgments
This work was conducted as part the DEEL! project. It was funded by ONR grant (N00014-19- 1-2029), NSF grant (IIS (IIS-1912280 and EAR- 1925481), and the Artificial and Natural Intelligence Toulouse Institute (ANITI) grant #ANR19- 1-----3--0000 The computing hardware was supported in part by NIH Office of the Director grant #S 100D025181 via the Center for Computation and Visualization (CCV) at Brown University. J.C. has been partially supported by funding from the Valencian Government (Conselleria d' Innovació, Universitats, Ciência Societat Digital) by virtue of 2022 grant agreement (convenio singular 2022).
aI gitthb..omkkere--ll/eaapppcccc https//wwww.deel.ai/
10

---

References
[1] Franck Mamalet et al. White Paper Machine Learning Certified Systems. 2021.
[2] Robert Geirhos et al. "Shortcut Learning in Deep Neural Networks". In: arXiv preprint arX77:2004.07800 (2020).
[3] Alexander D' Amour al. "Underspeciffiatiion presents challenges for credibility in modern machine learnin".. In: arXiv preprint arXiv:201 (2020).
[4] Sahar Shahamatdar et al. "Deceptive learning histopathology". In: bioRxiv (2022).
[5]
Thomas Fel et al. "Harmonizing the object recognition strategies of deep neural networks with humans".
In: Advances in Neural Information Processing Systems (NeurlPS) (2022).
[6] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ""Why Should I Trust You?": Explaining the Predictions of Any Classifier". In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016. [7] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. "Axiomatic Attribution for Deep Networks". In: Proceedings of the International Conference on Machine Learning (ICML). 2017.
[8] Daniel Smilkov et al. "SmoothGrad: removing noise by adding noise". In: Workshop on Visualization for Deep Learning, Proceedings of the International Conference on Machine Learning (ICML). 2017.
[9] Vitali Petsiuk, Abir Das, and Kate Sacnko. "Rise: Randomized input sampling for explanation of black-box models". In: Proceedings of the British Machine Vision Conference (BMVC). 2018.
[10] Ramprasaath R. Selvaraju et al. "Grad-CAM Visual Explanations From Deep Networks via Gradient- Based Localization", In: Proceedings of the IEEE International Conference Computer Vision (ICCV). 2017.
[11] Drew Linsley et al. "Learning what and where to attend". In: Proceedings of the International Conference Learning Representations (ICLR) (2018).
[12] Fel Thomas et al. "Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity
Analysis". In: Advances in Neural Information Processing Systems (NeurlPS). 2021.
[13] Thomas Fel al. "Don't 't Lie to Me! Robust and Efficient Explainability with Verified Perturbation
Analysis". In: ICML 2022, Workshop on Formal Verification of Machine Learning (2022).
[14] Paul Novello, Thomas Fel, and David Vigouroux. "Making Sense of Dependence: Efficient Black-box
Explanations Using Dependence Measure". In: Advances in Neural Information Processing Systems (NeurIPS) (2022).
[15] éloi Zablocki et al. "Explainabilitty vision-based autonomous driving systems: Review and challenges". In: arXiv preprint arXiv:2101.05307 (2021).
[16] Bryce Goodman and Seth Flaxman. "European Union regulations on algorithmic decision decision-making and a "right to explanation" "" In: A/ magazine 38.3 (2017), pp- 50-57.
[17] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. "Deep inside convolutional networks: Visual- ising image classification models and saliency maps". . In: Workshop Proceedings of the International Conference Learning Representations (ICLR). 2014.
[18] Matthew D Zeiler and Rob Fergus. "Visualizing and understanding convolutional networks". In: Pro- ceedings of the IEEE European Conference Computer Vision (ECCV). 2014.
[19] Marco Ancona et al. "Towards better understanding of gradient-based attribution methods for Deep Neural Networks". In: Proceedings of the International Conference on Learning Representations (ICLR). 2018.
[20] Zachary C. Lipton. "The Mythos of Model Interpretabillity.. In: Workshop Human Interpretabilitty Machine Learning, ICML.. 2016.
[21] Diogo Carvalho, Eduardo M Pereira, and Jaime S Cardoso. "Machine learning interpretability: survey on methods and metrics". In: Electronics 8.8 (2019), p. 832.
[22] Alejandro Barredo Arrieta et al. "Explainable Artificial Intelligence (XAI): Concepts, taxonomies. opportunities and challenges toward responsible AI". In: Information Fusion 58 (2020), pp. 82-115.
[23] Thomas Fel and David Vigouroux. "Representativity and Consistency Measures for Deep Neural Network Explanations". In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2022.
[24] Ruth C. Fong and Andrea Vedaldi. "Interpretable Explanations of Black Boxes by Meaningful Perturba- tion". In: Proceedings of the IEFE International Conference on Computer Vision (ICCV). 2017.
[25] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. "Understanding deep networks via extremal perturba- tions and smooth masks". In: Proceedings of the IEEE International Conference Computer Vision (ICCV). 2019.
[26] Andrew Elliott, Stephen Law, and Chris Russell. "Explaining Classifiers using Adversarial Perturbations on the Perceptual Ball". In: Proceedings of the IEEE Conference on Computer Vision and Patern Recognition (CVPR). 2021.
11

---

[27] Jianming Zhang et al. "Top-down neural attention by excitation backprop". In: International Journal of Computer Vision 126.10 (2018), pp. 1084-1102.
[28]
Wojeiech Samek et al. "Evaluating the visualization of what Deep Neural Network has learned". In: IEEE Transactions on Neural Networks and Learning Systems (TNNLS). 2015.
[29] Andrei Kapishnikov et al. "Xrai: Better attributions through regions". In: Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[30] Giang Nguyen, Daeyoung Kim, and Anh Nguyen. "The effectiveness of feature attribution methods and its correlation with automatic evaluation scores". In: Advances in Neural Information Processing Systems (NeurlPS) (2021).
[31]
Finale Doshi- Velez and Been Kim. "Towards rigorous science of interpretable machine learning". In: ArXiv e-print (2017).
[32] Tania Lombrozo and Susan Carey. "Functional explanation and the function of explanation" In: Cognition 99.2 (2006). pp. 167-204.
[33] Tania Lombrozo and Nicholas Gwynne. "Explanation and inference: Mechanistic and functional explanations guide property generalization", In: Frontiers in Human Neuroscience 8 (2014), p. 700.
[34]
Ny Vasil, Azzurra Ruggeri, and Tania Lombrozo. "When and how children explanations to guide generalizations" In: Cognitive Development (2022), p. 101144.
[35]
Joseph Williams and Tania Lombrozo. "The role of explanation in discovery and generalization: Evidence from category learning", In: Cognitive science 34.5 (2010), pp. 776-806.
[36]
Tania Lombrozo. "The instrumental value of explanations". In: Philosophy Compass 6.8 1)) pp. 539- 551.
[37]
Tania Lombrozo. "Explanatory preferences shape learning and inference". In: Trends in Cognitive Sciences 20.10 (2016), pp. 748-759.
[38]
Emily G Liquin and Tania Lombrozo. "Motivated to learn: An account of explanatory satisfaction", In: Cognitive Psychology 132 (2022), p. 101453.
[39]
Julia Angwin et Machine Bias. 2016. URL: https: / www. propublica org/ article/
ma machine-bias aws.ssrssssssssssssteteeeee in-
[40]
Ziad Obermeyer et al. "Dissecting racial bias an algorithm used to manage the health of populations". In: Science 366.6464 (2019). pp. 447-44744 447-453.
[41] Abdallah Hussein Sham et al. "Ethical in facial expression analysis: racial bias". In: Signal, Image and Video Processing (2022). pp. 1-8.
[42] Thomas McGrath al. "Acquisition of Chess Knowledge AlphaZero". In: arXiv preprint
arXiv:2111.09250 (2021).
[43] Jonas Andrulis et al. "Domain-Level Explainability-A Challenge for Creating Trust in Superhuman Al Strategies". In: arXiv preprint arXiv:201 1.001110000 1,06665 (2020).
[44] David Silver al. "Mastering the game of go without human knowledge".. In: nature 550.7676 (2017).
pp. 354-359.
[45] Jonas Degrave et al. "Magnetic control of tokamak plasmas through deep reinforcement learning". In:
Nature 602.7897 (2022), pp. 414419..
[46] John Jumper et al. "Highly accurate protein structure prediction with AlphaFold"' In: Nature 596.7873
(2021), pp. 583-589.
[47] Alex Davies et al. "Advancing mathematics by guiding human intuition with Al". Nature 600.7887
(2021), pp. 70-74.
[48] Peter Wilf et al. "Computer vision cracks the leaf code". In: Proceedings of the National Academy of Sciences 11..12 (2016), pp. 3305-3310.
[49] Edward Spagmuolo, Peter Wilf, and Thomas Serre. "Decoding family-level features for modern and fossil leaves from computer-vision heat maps". In: American journal of botany (2022).
[50] J. Deng et al. "ImageNet: A Large-Scale Hierarchical Image Database".. In: Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR). 2009.
[51] Sara Hooker et al. "A Benchmark for Interpretability Methods in Deep Neural Networks". In: Advances in Neural Information Processing Systems (NIPS). 2019.
[52] Leila Arras et al. "Explaining recurrent neural network predictions in sentiment analysis". In: Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) in ENMLP. 2017.
[53] Leila Arras ct al. 4." What is relevant in a text document?": An interpretable machine learning approach". In: PloS one 12.8 (2017), e4281142.
[54] Cheng- Yu Hsich et al. "Evaluations and methods for explanation through robustness analysis". In: Proceedings of the International Conference on Learning Representations (ICLR) (2020).
[55] Oisin Mac Aodha et al. "Teaching categories to human learners with visual explanations". In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2018.
12

---

[56] Arjun Chandrasekaran et al. "Do explanations make VQA models more predictable to human?" In: arXiv preprint arXiv:1810.1 12366 (2018).
[57]
Yasmeen Alufaisan et al. "Does Explainable Artificial Intelligence Improve Human
In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 2021.
[58] Felix Biessmann and Dionysius Refiano. "Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated". In: Proceedings of the ICML Workshop Theoretical Foundations, Criticism, and Application Trends of Explainable AI held in conjunction with the 38th International Conference on Machine Learning (ICML.), 2021.
[59] Hua Shen and Ting- Hao Huang. "How Useful Are the Seebbine-eeeertedd Interpretations to General Users? A Human Evaluation on Guessing the Incorectly Predicted Labels". In; Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Vol. 8. 1. 2020, pp. 168-172.
[60] Dong Nguyen. "Comparing automatic and human evaluation of local explanations for text classifica- tion". In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers). 2018, pp. 1069- 1078.
[61] Peter Hase and Mohit Bansal. "Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?" In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL Short Papers). 2020.
[62] Giang Nguyen, Mohammad Reza Taesiri, and Anh Nguyen. "Visual correspondence based explanations improve Al robustness and human-AN team accuracy". In: Advances in Neural Information Processing Systems (NeurIPS) (2022).
[63] Sunnic S. Y. Kim al. "HIVE: Evaluating the Human Interpretability of Visual Explanations". In:
European Conference on Computer Vision (ECCV). 2022.
[64] Leon Sixt et al. "Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset". In: Proceedings of the International Conference on Learning Representations (ICLR) (2022).
[65] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. "Examples are not enough, learn to eriticize! criticism for interpretabilltyy.. In: Advances in Neural Information Processing Systems (NeurIPS) (2016).
[66]
Danish Pruthi al. "Evaluating Explanations: How much do explanations from the teacher aid students?"
In: ArXiv e-print. 2020.
[67]
Alon Jacovi and Yoav Goldherg. "Towards faithfully interpretahle NLP systems: How should we define and evaluate faithfulness?" In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL Short Papers) (2020).
[68]
L. Itti. "Models of bottom-up attention and saliency". In: Neurobiology of attention (2005).
[69]
Peyton Greenside Avanti Shrikumar and Anshul Kundaje. "Learning Important Features Through Prop- agating Activation Differences". In: Proceedings of the 34th International Conference on Machine Learning (ICML). 2017.
[70] Christian Szegedy et al. "Going deeper with convolutions". In; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015.
[71] Andrew Elliott, Stephen Law. and Chris Russell. "Explaining Classifiers using Adversarial Perturbations on the Perceptual Ball", In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 10693-10702.
[72] Kaiming He et al. "Deep residual learning for image recognition". In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
John W Tukey. "Comparing individual means in the analysis variance". In: Biometrics (1949). [74] Umang Bhatt, Adrian Weller, and José M. F. Moura. "Evaluating and Aggregating Feature- based Model Explanations". " In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 2020.
[73]
[75] Chih-Kuan Yeh et al. "On the (In)fidelity and Sensitivity for Explanations". In: Advances in Neural Information Processing Systems (NIPS). 2019.
[76] Ming Li et al. "The similarity metric". In: IEEE transactions on Information Theory 50.12 (2004). pp. 3250-3264.
[77] Matthieur Perreira Da Silva, Vincent Courboulay, and Pascal Estraillier. "Image complexity measure based on visual attention". In: 2011 18th IEEE International Conference on Image Prucessing. IEEE.
2011. pp. 3281-3284.
[78] Alex Forsythe, Gerry Mulhern, and Martin Sawey. "Confounds in pictorial sets: The role of complexity and familiarity in basic-level picture processing" In: Behavior research methods 40.1 (2008), pp. 116-
129.
[79] Alexandra Forsythe. "Visual complexity: is that all there is?" In: International Conference on Engineering Psychology and Cognitive Ergonomics. Springer. 2009, pp. 158-166.
[80]
Steven de Rooij and Paul Vitányi. "Approximating die-distortion graphs of individual data: Experiments in lossy compression and denoising". In: arXiv preprint cx/0609121 (2006).
13

---

[81] Richard Zhang et al. "The unreasonable effectiveness of deep features as perceptual metric". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 586-595.
[82] Vedant Nanda et al. "Exploring Alignment of Representations with Human Perception". In: arXiv preprint 1rX1v (2021).
14


Reviews:
[{"review_id": "e5Bc5UmMfg2", "reviewer": "Reviewer_jUUQ", "summary": "This paper addresses evaluating explanation methods for image classification based on human experiments. To evaluate them, this study proposes an evaluation metric that quantifies how well humans are able to understand the output of the system after looking at the explanation provided by each explanation method. Then, this study analyzes the explanation methods by considering three scenarios to do bias detection, identification of new strategies, and understanding of failure cases. ", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The experiment is well designed, the hypotheses tested are appropriate, and the experiment is done with enough participants. Therefore, the results obtained look convincing, except for the points in the weaknesses below.\n- Some results are valuable for XAI community. For example, the result of the third scenario can strengthen the statement in previous works, e.g., [57].\n\n## Weaknesses\n- I think the impact of the results is limited because the experiment is conducted only on limited scenarios, such as classification on Husky vs Wolf and classification on Kit Fox and Red Fox. Due to the limitation, the scores shown in Figures 4-6 are distributed in narrow areas. Therefore, if more scores are obtained by conducting experiments on more scenarios, the results may change.\n- There is a possibility that the predictive performance of the recognition model affects the utility score. When the recognition model has a poor predictive performance, the explanation methods depending on the predictions by the recognition model may also produce incorrect and uncertain explanations. The three scenarios in this study are settings that are difficult for the recognition model to accurately classify images. Therefore, I have a concern that whether the experiment can evaluate explanation methods only, excluding the impact of the recognition model.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Uc3VcYyfQv", "reviewer": "Reviewer_HHXE", "summary": "The authors propose a quantitative measurement to evaluate explainability methods by relying on how well the methods allows a human to predict what prediction a classifier will make. The authors conduct an experiment using their approach for a variety of image classification explainability tools.", "questions": "1. Might this approach apply for other definitions of an effective explanation?\n2. Is this actually a psychophysics experiment?", "limitations": "The authors should deeply engage with the limitations around how creating a benchmark for their narrowly defined idea of an effective explanation might cause harm by shifting focus away from other ones.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nSome sort of statistical approximation around the effectiveness of human-facing explanations is a worthwhile, if challenging goal.\nThe methods the authors propose, assuming one buys their narrow definition of an effective explanation, seems reasonable and defensible.\nThe initial experiments produce interesting and potentially useful results.\nThe paper is reasonably easy to follow, though would benefit from making fewer assumptions about reader's background knowledge in the introduction.\n\nWeaknesses:\nThe concept of a meta-predictor seems to limit the author's approach to problems in which a model's goal is simply to replicate a human's intelligence, rather than to augment it or enhance it in ways the human could not do on its own.\nThe authors claim they conducted psychophysics experiments. In what ways does their study design engage with psychophysics? The authors should explicitly say what they are borrowing from the psychophysics literature and why. (From my understanding of psychophysics, the connections seem tenuous and superficial at best).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TXDes3J-aih", "reviewer": "Reviewer_bQ3u", "summary": "The paper introduces an evaluation framework for explanatory methods that uses human ability to predict a model's predictions as a measure of the utility of the explanations. The paper considers 3 scenarios for usefulness: bias identification, understanding how to solve too difficult tasks for non-experts, and understanding of failure cases. The paper looks at 6 attribution methods over 1 dataset per scenario.", "questions": "How do you think the evaluation method can help the community to design better explainability methods in the first place? Assume that one develops a novel explainer but evaluating it on your method shows that it is less useful than e.g., SmoothGrad. Would your evaluation provide any further insight as to how the novel explainability method could be improved to become more useful?  ", "limitations": "I did not see a direct mention of potential limitations or negative societal impact, which I suggest the authors incorporate in their paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe paper addresses a very important topic: the evaluation of explainability methods.\nIt is clearly written and has a good research methodology. \nTheir evaluation method is applicable to any explainability method. While the principal behind it is not novel (as the authors also point out), the evaluation is done in a more thorough manner than in the previous studies.\nSome of the results provide new and unexpected insight, which could potentially have a good impact on shaping the explainability field.\n\n\nWeaknesses\nThere isn't much novelty in the evaluation method proposed, just minor improvements over previous studies, although it is still worth having these improvements. \n\nFor each of the 3 scenarios, only 1 dataset was used.\n\nOnly attribution methods were tested. I think it is already quite clear for the community that attribution methods have lots of problems (and clearly they cannot provide the \"what\" that the authors mention) and therefore it would have been more interesting to also test other types of explanatory methods. But even among attribution methods, popular ones such as SHAP and LIME were not tested. \n\nSome of the conclusions have already been established before and/or are to be expected, as the authors also point out. However, it is still useful to have this paper show them again in a proper manner (e.g., eliminating potential confounds).\n\n\nMinor typos:\nUse \\citet when appropriate\nL74 \"score drop in score\"\nL81 remove . before citation\nL85 citet [5]\nL103 This removeS\nL107 measure reflectS\nL107 and L109, 112 explanationS\nL178: Meta-predictorS\nL191: space missing \".T\"\nL220 \"ref. [5]\" delete ref and citet\nL229: if --> is\nL320 citation missing\nL328: capital for section\nL367: vs. remove larger space\nL394 one needS\nL394: is -> are", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6KTFD95uEN", "reviewer": "Reviewer_LEiC", "summary": "The authors present a human-centered framework to evaluate the usefulness of attribution explainability methods. They evaluate 6 different attribution methods across 3 setttings. They show that generally, perceptual scores derived from attribution maps can be more useful at predicting failure cases of explainability methods better than fidelity metrics.", "questions": "In general I recommend rebranding the paper as a study of attribution methods instead of a framework, since the results are hard to scale, hard to reproduce and it is cumbersome to \"integrate\" a new explainer into this framework. However, the manuscript as is could be improved in the following ways:\n\n#### Critical changes:\n - Provide some experiments with a varying number of classes or at least a small section discussing how the effectiveness of the meta-predictor approach varies when increasing the number of classes.\n\n- The main reason the authors provide for the failure of attribution methods is an intrinsic limitation of said methods, since they can only show what regions in an image should be changed but not how. This is precisely one of  the limitiations counterfactual explainers solve, and since the authors claim that the framework can be applied to any explainer it would be useful to explore if counterfactual explainers fail in the same way, whilst substantiating said claim.\n- It seems to me that there is a potential overlap within the 3 scenarios used for evaluation (Bias detection, Identifying an expert strategy, Understanding failure cases). For example: In scenario 2 (Identifying an expert strategy) whenever a meta-predictor is provided with an image that the model has failed to predict, then the task is no longer: _to understand the strategy that was discovered by the AI system_, but instead it becomes either bias detection (scenario 1) or Understanding failure cases (scenario 3). Conversely, in scenario 3 (if the classes are really similar) whenever the meta-predictor is provided with an image that the model has successfully predicted then the task becomes Identifying an expert strategy (scenario 2). An easy fix would be to provide only correctly and incorrectly classified samples for scenarios 2 and 3 respectively, however as stated by the authors this would significantly affect the meta-predictor score if the meta-predictor has some prior knowlege about the domain (dataset) the scenario is being evaluated on. If this overlap does in fact exists, I think it should be discussed in the paper.\n\n#### Minor changes:\n\n- Figure 1 is not referenced anywhere, also there is a missing reference on line 320\n- In the section describing the understandability-completeness trade-off it would useful to mention/show examples of how an _entirely faithful_ and _more understandable_ explanations look like.\n- In the General study design section it is mentioned that each training session has 5 training samples, but the plots showcasing the results show 5, 10 and 15, this should be clarified. \n- I think it would be useful to incorporate the meta-predictor response time into the metric since an explanation that is correctly predicted instantly is probably better than one that gets correctly predicted after some time. Conversely an explanation that is incorrectly predicted instantly could be considered more \"misleading\" than one that gets incorrectly predicted after some time.\n\n#### Questions:\n- Do the authors have any hypothesis as to why the best method for Bias detection and Identifying an expert strategy differ? I feel like the paper would be more complete with a small section explaining this phenomenon, not unlike the section: __Why do attribution methods fail?__. In which the authors investigate why all methods underperform on the 3rd setting.", "limitations": "The limitations of the work should be discussed beyond the cost of human evaluation, since the framework itself has some limitations (mentioned above) such as scalability and reproducibility. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The explainability evaluation framework the authors propose: \n   - is human-centered, which is, ideally, how the usefulness of explanations should be evaluated.\n   - includes a control group, which was lacking in previous work\n\n-  The paper is easy to understand and generally well written.\n- The proposed metric is original and sound to the best of my knowledge.\n\nWeaknesses:\n- The proposed framework poses some reproducitiblity concerns since in order to evaluate any explainability one would have to undergo the entire process described in the paper of hiring and filtering the meta-predictors. This also makes it hard to scale.\n\n- I worry about the effectiveness of the meta-predictor whenever evaluating a dataset with more than 2 classes.\n\n- The authors state that the framework can be applied to other explainability methods and while this in principle makes sense, there are no experimental results supporting this claim. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept