The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
Authors: Giang Nguyen
Daeyoung Kim
Anh Totti Nguyen
Venue: NeurIPS 2021 Poster (2021)
Keywords: feature attribution
nearest neighbors
image classification
imagenet
stanford dog
user-study

Abstract: Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications.
Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work.
However, despite humans being the target end-users, most attribution methods were only evaluated on proxy automatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et al. 2018). In this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine-grained classification, and when an image is natural or adversarial (i.e., contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-AI teams compared to AI alone. Importantly, we found automatic attribution-map evaluation measures to correlate poorly with the actual human-AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-loop applications and to rethink the existing evaluation metrics.

The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
Giang Nguyen" Auburn University
Daeyoung Kim KAIST
aguyenggiangbkkh@@mmall com
kimd@kaist kr
Anh Nguyen Auburn University anh nag@nga@@mmm.....m com
Abstract
Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical many real-world, high-stake applications. Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work. However, despite humans being the target end-users, most attribution methods were only evaluated on proxy cetoomtii--vvllattiin [60, 78, 80]. this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine- tnnegrained classification, and when an image is natural or adversarial (i.e. contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-Al teams compared AI alone. Importantly, we found automatic attribution map evaluation measures to correlate poorly with the actual human-Al AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-llop applications and to rethink the existing evaluation metrics.
metrics
Introduction
US Why did computer vision system suspect that person had breast cancer [77].
capitol rioter [2], or a shoplifter [3]? The explanations for such high-stake predictions made by existing Artificial Intelligence (AI) agents can impact human lives in various aspects, from social [22], to scientific [54], and legal [3, 23, 31]. An image classifier's decisions can be explained via an aftribution map (AM) [13], i.e. heatmap that highlights the input pixels that important for against a predicted label. AMs can be useful localizing tumors x-rays [61], reflecting biases [41] and memory [56] of image classifiers, teaching humans distinguish butterflies [48].
Via AMs, part of the thought process of AIs is now transparent to humans, enabling human-AI collaboration opportunities. Can machines and humans work together to improve the accuracy of humans in image classification? This question remains largely unknown although, in other non-image domains, AMs have been found useful to humans in the downstream tasks [12, 39, 47, 66, 66).
Furthermore, it is also unknown whether any existing automatic evaluation scores may predict such effectiveness of AMs to humans, the target user of AMs. Most AMs were often only evaluated on
"Corresponding authors. Work done when GN was at KAIST.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

lorikeet
ImageNet
Stanford Dogs
contidence 20%
Method
μ
Confidence
72.44
8.25
61.71
11.39
houthrapa
GradCAM
72,58
8.11
60.56
9.27
No
EP
73.85
6,88
56.67
10.57
S neorost neighticra lorikeet
SOD
72.06
7.63
61.67
10.87
3-NN
76.08
5,86
57.20
10.58
(a) Given an input image, its top-1 predicted label (here,
(b) Users of 3-NN outperform all other users in- cluding those using attribution maps on ImageNet classification. However, on fine-grained dog clas- sification (Stanford Dogs), visual explanations (GradCAM, EP, 3-NN) surprisingly did not help but hurt the human performance. Results from test- ing humans both real and adversarial images.
"lorikeet") and confidence score (A), we asked the user to decide Yes No whether the predicted label is accurate (here, the correct answer No)-i.e. the accuracy human-Al teams without visual explanations. We also compared this baseline with the treatments where one attribution map (B) or a set of three nearest neighbors (C) also provided to the user (in addition the confidence score).
Figure 1: Overview of the methods compared in our user-study and its results.
proxy evaluation metries such pointing game [78], object localization [80], or deletion [60], which may not necessarily correlate with human-AI team accuracy on downstream task.
In this paper, conduct the first user-study measure AM effectiveness in assisting humans the 120-class Stanford Dog [38] fine-grained classification and 1000-class ImageNet [65] classification, which is the task that most attribution methods were designed for. We asked 320 lay and 11 expert users to decide whether machine decisions are correct after observing an input image, top-1 classification outputs, and explanations (Fig. 1a). This task captures two related quantities: (1) human accuracy on image categorization, here simplified to binary classification; and (2) model debugging. i.e. understanding whether Al is correct wrong. For fuller understanding of the effectiveness of AMs, we tested users on both real and adversarial images [75]. Our main findings include:
1. AMs are, surprisingly, more effective than nearest-neighhors (here, 3-NN) in improving human-Al team performance on both ImageNet and fine-grained dog classification (Sec. 3.2).
2. On fine-grained dog classification, a harder task for human-AI team than 1000-class ImageNet, presenting AMs to humans interestingly does not help but instead hurts the performance of human-AN AI teams, compared Al alone without humans (Fig. 2b).
3. On adversarial Stanford Dog images which are very hard for humans to label, presenting confidence scores only to humans is the most effective method compared all other explanations tested including AMs and 3-NN examples (Sec. 3.2).
4. Despite being commonly used in the literature, some automatic evaluation metrics here, Pointing Game [78], localization errors [80], and loU [80]-correlate poorly with the actual human-Al team performance (Sec. 3.5).
5. According to study on 11 Al-expert users, 3-NN is significantly more useful than Grad- CAM [67], a statc-of-the-ar attribution method for ImageNet classification (Scc. 3.6).
2 Methods
2.1 Image classification tasks
To evaluate human-A team performance on image classification, for each image, we also presented to a user: (1) an Al's top-1 predicted label, (2) its confidence score, and, optionally (3) visual explanation that generated in attempt to explain the predicted label. The user was then asked to decide whether Al's predicted labels are correct or not (see Fig. la). We performed our experiments on two image datasets of ImageNet and Stanford Dogs, which are of increasing difficulty.
Code and data available httpp://gtiiee /ttptgggtttttttt/e/ com/ wtw. wwwp///a////fffft.e.eess-stttete

---

ImageNet We tested human-Al teams on ImageNet [65] an image classification task that most attribution methods were tested on or designed for [7. 60, 67, 80]. ImageNet is a 1.3M-image dataset spanning across 1000 diverse categories from natural to man-made entities [65].
Stanford Dogs To test whether our findings generalize to a fine-grained classification task that is harder to human-AI teams than ImageNet, we repeated the user-study on the 120-class Stanford Dogs [38] dataset, subset of ImageNet. This dataset is challenging due to the similarity between dog species and the large intra-class variation [38]. Compared to ImageNet, Stanford Dogs is expected to be harder human-AI teams because lay users often have significantly less prior knowledge about fine-grained dog breeds compared to a wide variety of everyday objects in ImageNet.
2.2 Image classifiers
We took ResNet-34 [34] pretrained on ImageNet (73.31% top-1 accuracy) from torchvision [50] as the target classifier (i.e., the AI) for both ImageNet and Stanford Dogs classification tasks because the 1000-class ImageNet that the model was pretrained on includes 120 Stanford Dogs classes. The visual explanations in this paper were generated to explain this model's predicted labels. We chose ResNet-34 because ResNets were widely used in feature attribution research [7, 28, 45, 60, 67].
2.3 Visual explanation methods
To understand the causal effect of adding humans in the loop, we compared the performance of a standard Al-only system (i.e. no humans involved) and human-Al teams. In each team, besides an input image and a corresponding top-1 label, humans are also presented with the corresponding confidence score from the classifier, and a visual explanation (i.e. one heatmap or three 3-NN images in; see examples from our user-study in Sec. A13).
AT-only A common way to automating the process of accepting or rejecting predicted label is via confidence-score thresholding [14]. That is, a top-1 predicted label is accepted if its associated confidence score is 2 T. a threshold. We found the optimal confidence threshold T that produces the highest accepting accuracy on the entire validation set by sweeping across threshold values {0.05, 0.10, ..., 0.95), J. i.e. at a 0.05 increment. The optimal T values are 0.55 for ImageNet and 0.50 for Stanford Dogs (details Sec. A6).
Confidence scores only To understand the impact of visual explanations on human decisions, we also used a baseline where users are asked to make decisions given no explanations (i.e. given only the input image, its top-1 predicted label and confidence score). To our best knowledge, this baseline has not been studied in computer vision, but has been shown useful in some non-image domains for improving human-Al AI team accuracy or user's trust [12, 79].
GradCAM and Extremal Perturbation (EP) We chose GradCAM [67] and Extremal Perturba- tion (EP) [28] as two representatives for state-of-the-an attribution methods (Fig. la). Representing for the class of white-box, gradient-based methods [16, 63, 80), GradCAM relies on gradients and the activation map at the last conv layer of ResNet-34 to compute a heatmap. GradCAM passed a weight-randomizztton sanity check [6] and often obtained competitive scores in proxy evaluation metrics (see Table in [28] and Table 1&2 in [25])..
In contrast, EP is representative for perturbation based methods [7, 20, 29]. EP searches for set of a% of input pixels that maximizes the target-class confidence score. We followed the authors' best hypperparameters--uumme Gaussian smoothing kernel with an std equal to 9% of the shorter side of the image (see the code [1].. We used the TorchRay package [5] to generate GradCAM and EP attribution maps.
over four binary masks generated using a E (2.5, 5, 10, 20) and
We initially also tested vanilla Gradient [70], Integrated Gradient (IG) [74], and SHAP [46] methods in pilot study but discarded them from our study since their AMs are too noisy to be useful to users in pilot study.
Salient object detection (SOD) To assess the need for heatmaps to explain a specific classifer's decisions, we also considered a classifier-agnostic heatmap baseline. That is, we used a pre-trained state-of-the-ant salient-object detection (SOD) method called PoolNet [43], which uses a ResNet-50
3

---

backbone pre-trained on ImageNet. PoolNet was trained to output heatmap that highlights the salient object in an imput image (Fig. 3) a process that does not take our image classifier into account. Thus, SOD serves as strong saliency baseline for GradCAM and EP.
3-NN To further understand the pros and cons of AMs, compared them to a representative, prototype-basee explanation method (e.g. [18, 53, 55]). That is, for a given input image x and a predicted label y (e.g., "lorikeet"), show the top-3 nearest images the same ImageNet training-set class (Fig. la). To compute the distance between two images, we used the L2 distance in the feature space of the last conv layer of ResNet-34 classifier (i.e. layer4 per PyTorch definition [4] or conv5_x __ in He et al. [34]). We chose layer4 as our pilot study found to he the most effective among the four main conv layers (i.e. layer1 to layer4) of ResNet-34.
by retrieving them from
While k-NN has a wide spectrum of applications in machine learning and computer vision [68], the effectiveness of human-AI collaboration using protttye--bsce explanations has been rarely studied [64]. To the best our knowledge, we provided the first user study that evaluates the effectiveness of post-hoc, nearest-neighhor explanations (here, 3-NN) human-Al AI team performance.
2.4 User-study design
2.4.1 Participants
Our user-study experiments were designed and hosted on Gorilla [11]. We recruited lay participants via Prolific [57] (at $10.2/hr), which is known for a high-quality user base [59]. Each Prolific participant self-identified that English is their first language, which is the only demographic filter we used to select users on Prolific. Participants come from diverse geographic locations and backgrounds (see Sec. A9). Over the course of two small pilot studies and one main, large-scale study, we collected in total over 466 complete submissions (each per user) after discarding incomplete ones.
In the main study, after filtering out submissions by validation scores (described in Sec, 2.4.2), we had 161 and 159 qualified submissions for our ImageNet and Dogs experiments, respectively. Each of the methods (i.e. except for Al-only) was experimented by at least 30 users, and each (image, explanation) pair was seen by at least two users (statistics of users and trials in Sec. A8).
2.4.2
Tasks
Our Gorilla study contains three main sets of screens: (1) ioodduccioo-whher each user is introduced to the study and relevant rules (details Sec. A11); (2) Training: and (3) Test. Each user randomly assigned set of test images and only one" explanation method (e.g. GradCAM heatmaps) to work with during the entire study.
Training To familiarize with the image classification task (either ImageNet or Stanford Dogs), users were given five practice questions. After answering each question, they were shown feedback with groundtruth answers. In each training screen, we also described each component in a screen via annotations (see example screens in Sec. A11.2).
Validation and Test After training, each user was asked to answer 40 Yes/No questions in total. Before each question, a user was provided with short WordNet [51] definition of the predicted label and three random training-set images (see an example in Fig. A5) correctly-classifeed into the predicted class with confidence score 2 0.9. To control the quality of user responses, out of 40 trials, we used 10 trials as validation cases where we carefully chose the input images such that we expected participants who followed our instructions to answer correctly (details in Sec. A11.3). We excluded those submissions that had below 10/10 and 8/10 accuracy on the 10 validation trials from the ImageNet and Dogs experiments, respectively. For the remaining (i.e., qualified) submissions, used the results of their 30 non-validation trials in our study.
Shooing multiple explanation types to the same user complicates the user beaiiirannng procedure and also makes the task harder users due to context switching.

---

2.4.3 Images
We wish to understand the effectiveness of visual explanations when AIs are correct vs. wrong, and when Als face real vs. adversarial examples [75]. For both ImageNet and Dogs experiments, we used the ResNet-34 classifier to sample three types of images: (1) correctly-clasiifeed, real images; (2) misclassified, real images; and (3) adversarial images (i.e. also misclassified). In total, we used 3 types x 150 images = 450 images per dataset. Each image was then used to generate model predictions and explanations for comparing the 6 methods described in Sec. 2.3.
Filtering From the 50K-image ImageNet validation set, we sampled images for both ImageNet and Dogs experiments. To minimize the impact of low-quality images to users' performance, we removed all 900 grayscale images and 897 images that have either width or height smaller than 224 px, leaving 48,203 and 5,881 images available for use in our ImageNet and Dogs experiments, respectively. For Dogs, we further excluded all 71 dog images mislabeled by ResNer-34 into non-dog categories (examples in Sec. A12) because they can trivialize explanations, yielding 5,810 Dogs images available for selection for the study.
Selecting natural images To understand human-Al Al team performance at varying levels of difficulty for humans, we randomly divided images from the pool of filtered, real images into three sets: Easy (E), Medium (M), and Hard (H).
Hard images are those correctly labeled by the classifier with low confidence score (i.e., [0. 0.3)) and mislabeled with high confidence (i.e.. E [0.8, 1.0). Vice versa, the Easy set contains those correctly labeled with high confidence and mislabeled with low confidence. The Medium both correctly and incorrectly labeled images with a confidence score E [0.4, 0.6) i.e. when the Al is unsure (see confidence-score distributions in Sec. A7). In each set (E/M/H), we sampled 50 images correctly-labeled and 50 mislabeled by the model. In sum, per dataset, there are 300 natural images divided evenly into controlled bins (see Fig. Al for the ratios of these bins in the original datasets).
contains
Generating adversarial images After the filtering above, we took the remaining real images to generate adversarial examples via Foolbox [62] for the Re classifier using the Project Gradient Descent (PGD) framework [49] with an Lx bound c = 16/255 for 40 steps, each of size of 1/30. We chose this setup because at weaker attack settings, most adversarial images became correctly classified after being saved as a JPEG file [44], defeating the purpose of causing AIs to mishehave. Here, for each dataset, randomly sampled 150 adversarial examples (e.g., the input image in Fig. la) that are misclassified at the time presented to users in the JPEG format and often contain so small artifacts that we assume not bias human decisions. Following the natural-image sampling, we also divided the 150 adversarial images into three sets (E/M/H), each containing 50 images.
2.5 Automatic evaluation metrics for attribution maps
Many attribution methods have been published; however, most AMs were not tested on end-users but instead only assessed via proxy evaluation metrics. We aim to measure the correlation between three common
Pointing Game [78], Intersection over Union (IoU) [80], and weakly-supervin localization (WSL) [wth-with the actual human-Al team performance in our user study.
All three metrics are based on the assumption that an AM for explaining predicted label y should highlight an image region that overlaps with a numan-annotarn
bounding box (BB) for that category y. For each of 300 real, correctly-classifii images from each dataset (described in Sec. 2.4.3), we obtained its human-annotated BB from ILSVRC 2012 [65] for using in the three metrics.
Pointing game [78] is a common metrics often reported in the literature (e.g. [24, 28, 60, 63, 67, 76)). For an input image, generated AM is considered a "correct" explanation (i.e. a hit) if its highest- intensity point lies inside the human-
BB. Otherwise, it is a miss. The accuracy for an explanation method is computed by averaging over all images. We used the TorchRay implementation of Pointing Game [5] and its default hyperparameters (tolerance 15).
Intersection over Union The idea is to compute the agreement in Intersection over Union (IoU) [80] between a numan-annotater BB and a binarized AM. A heatmap was binarized at the method's 's optimal threshold a, which was found by sweeping across values (0.05x: E N and
20).
Weakly-supervise localization (WSL) is based on IoU scores [80]. WSL counts a heatmap correct if its binarized version has BB that overlaps with the human- BB at an IoU > 0.5.
5

---

WSL is also commonly used e.g. [7, 24, 67]. For both GradCAM and EP, we found the binarization threshold a 0.05 corresponds to their best WSL scores on 150 correctly-classifed images (i.e.. excluding mislabeled images because human-annotated BBs are for the groundtruth labels).
3 Results
3.1 3-NN is more effective than AMs on ImageNet but both hurt user accuracy Dogs
We measure the per-user accuracy on ImageNet classification over our controlled image sets (correctly- labeled, incorrectly-labeled, and adversarial images; see Sec. 2.4.3), i.c. on both natural and adversar- ial images. We repeated the same study for Stanford Dogs (sce Table lb).
ImageNet We found that 3-NN significantly better than SOD and GradCAM (Mann Whitney U-test; p < 0.035). The differences among feature attribution methods (GradCAM, SOD, EP) are not statistically significant. In terms of mean and standard deviation of per-user accuracy, users of 3-NN outperformed all other users by large margin of ~3% (Table l 76.08% and 5.86%).
Stanford Dogs We found that users of 3-NN and EP performed statistically significantly worse than the users with no explanations (Table lb; Confidence) and SOD users (Mann Whitney U-test; p 0.024). There are no significant differences among other pairs of methods. That is, interestingly, on fine- greegga Stanford Dog classification where an image may be real or adversarial, visual explanations (i.e. both 3-NN and AMs) hurt user classification accuracy.
3.2 On natural images, how effective are attribution maps in human-Al team image classification?
We wish to understand the effectiveness of AMs (GradCAM and EP) compared to four baselines (Al-only, Confidence, SOD, and 3-NN) in image classification by human-A Al teams. Here, we compare methods on cntturll-mmgee sets: ImageNet and Stanford Dogs (i.e. no adversarial images).
Experiment Because given image can be mapped to of the 6 controlled bins described in See. 2.4.3, for each of the 6 methods, we computed its human-AI team accuracy for each of the bins where each bin has exactly 50 images (Sec. Al reports per-bin accuracy scores). To estimate the overall accuracy of a method on the original ImageNet and Dogs dataset, we computed the weighted sum of its per-bin accuracy scores where the weights are the frequencies of images appearing in each bin in practice. For example, 70.93% of the Dogs images are correctly-classife with a high confidence score (Fig. A1; Easy Correct).
ImageNet results On ImageNet, human-Al teams where humans use heatmaps (GradCAM, EP, or SOD) and confidence scores outperformed Al-only by an absolute gain of ~6-8% in accuracy (Fig. 2a; 80.79% vs. 38%7.. That is, when teaming up, humans and Al together achieve a better performance than Al alone. Interestingly, only half of such improvement can be attributed to the heatmap explanations (Fig. 2a; 44.79% vs. 88.77%). That is, users when presented with the input image and the classifier's top-1 label and confidence score already obtained +4% boost over Al alone (84.79% vs. 80.79%)
Dogs results Interestingly, the trend did not carry over to fine-grained dog classification. On average, humans when presented with (1) confidence scores only or (2) confidence scores with heatmaps all underperformed the accuracy of Al-only (Fig. 2b; 81.14% vs. 76.45%). An explanation is that ImageNet contains ~50% of man-made entities, which contain many everyday objects that users are familiar with. Therefore, the human-AI teaming led to a substantial boost ImageNet. In contrast, most lay users are not dog experts and therefore do not have the prior knowledge necessary to help them in dog identification, resulting in even worse performance than a trained AI (Fig. 2b; 81.14% vs. 76.45%). Interestingly. when providing users with nearest neighbors, human-Al teams with 3-NN outperformed all other methods (Fig. 2b; 82.88%) including the Al-only.
Both datasets On both ImageNet and Dog distributions, 3-NN among the most effective. On average over 6 controlled bins, 3-NN also outperformed all other methods by a clear margin of 2.71% (Sec. A2 A4). Interestingly, SOD users tend to reject more often than other users (Sec. A10), inadvertently causing high human- Al team accuracy Al-misclassified images (Sec. A14.5).

---

4079
#7.07
Bouts
7644
79.78
75.85
82.48
71,00
74.77
7l
7587
08.66
67.99
64.77
5K.11
Al colly
Couhhbwo
CmoCCAN
SDD
Natural ImageNet
Stanford Dogs
Adversarial ImugeNet
Stanford Dogs
lorikeet
Gardon
(a)
(b)
(c)
(d)
Figure 2: 3-NN is consistently among the most effective improving human-Al team accuracy (%) on natural ImageNet images (a), natural Dog images (b), and adversarial ImageNet images (c). On the challenging adversarial Dogs images (d), using confidence scores only helps humans the most in detecting Al's mistakes compared using confidence scores and one visual explanation. Below each sample dataset image is the top-1 predicted label (which was correct or wrong) from the classifier.
3.3
On adversarial images, how effective are aftribution maps in human-Al team image classification?
As adversarial examples are posing a big threat to Al-only systems [8, 9, 52, 58), here, we are interested in understanding how effective explanations are in improving human-Al team performance over Al-only, which assumed to 0% accurate under adversarial attacks. That is, on ImageNet and Dogs, compare the accuracy of human-Al AI teams on 150 adversarial examples, which all caused the classifier to misclassify. Note that a user was given a random mix of natural and adversarial test images and was not informed whether an input image is adversarial not.
Adversarial ImageNet On both natural and adversarial ImageNet, AMs are on-par or more effective than showing confidence scores alone (Fig. 2c). Furthermore, the effect of 3-NN is consistent +4% gain compared to Confidence (Fig. & c; Confidence vs. 3-NN). Aligned with the results on natural ImageNet and Dogs, 3-NN remains the best method on adversarial ImageNet (Fig. 2c; 75.07%) See Sec. A14.4 for adversarial images that were only correctly-reeected by 3-NN users but not others.
Adversarial Dogs Interestingly, on adversarial Dogs, adding an explanation (either 3-NN or a heatmap) tend to cause users agree with the model's incorrect decisions (Fig. 2d). That is, heatmaps often only highlight coarse body region of a dog without pinpointing an exact feature that might be explanatory. Similarly, 3-NN often shows examples of almost identical breed to the groundtruth (e.g. mountain dog vs. Gordon setter), which is hard for lay-users to tell apart (see qualitative examples in Sec. A14.6).
3.4 On ImageNet, why is 3-NN more effective than attribution maps?
Analyzing the breakdown accuracy of 3-NN in each controlled set of Easy, Medium, and Hard, we found 3-NN be the most effective method in the Easy and Hard categories of ImageNet (Table A3).
Easy When AI mislabels with low confidence, 3-NN often presents contrast evidence showing that the predicted label is incorrect, i.e. the nearest examples from the predicted class is distinct from the input image (Fig. la; lorikeets have a distinct blue-orange pattern not found bee eaters). The same explanation is our leading hypothesis for 3-NN's effectiveness adversarial Easy, ImageNet images. More examples in Sec. A44.3.
Hard Hard images often contain occlusions (Figs. A9 & A12), unusual instances of common objects (Fig. A13), or could reasonably be in multiple categories (Figs. A11 & A10). When the classifier correct but with low confidence, we found 3-NN to be helpful in providing extra evidence for users to
7

---

confirm AI's decisions (Fig. 3). In contrast, heatmaps often only highlight the main object regardless of whether Al correct or not (Fig. la; GradCAM). More examples are in Sec. A A14.1.
Medium Interestingly. 3-NN was the best method on the Easy and Hard set, but not on the Medium (Table A3). Upon closer look, surprisingly found ~63% of misclassified images by 3-NN Auman-AI teams to have debatable groundtruth ImageNet labels (see Sec. A14.2 for failure cases of 3-NN).
afroe hunting fog Wice harting dag handbees
cilrree
lnc
wartheg
pat
(a) "african hunting dog": dog like mammals of Africa
(b) "caldron"; very large pot used boiling
Figure 3: Hard images that were only correctly accepted by 3-NN users but not other users of GradCAM, EP, or SOD. Despite the animals in the input image are partially occluded, 3-NN provided closed-up examples of african hunting dogs, enabling users to correctly decide the label (a). Choosing a single label for a scene of multiple objects is challenging. However, 3-NN was able to retrieve a nearest example showing very similar scene enabling users to accept AI's correct decisions (b). See full-res images in Figs. A9 A10.
3.5 How do automatic evaluation scores correlate with human-AT team performance?
IoU [80], WSL [80]. and Pointing Game [78] are three attribution-man evaluation metrics commonly used the literature, e.g. in [7, 28, 29, 60, 67]. Here, we measure the correlation between these scores and the actual human-A Al team accuracy to assess how high- high-performance such benchmarks translate into the real performance on downstream tasks.
Experiment We took the HP and GradCAM heatmaps of the 150 real images that were correctly- classified (see Sec. 2.4.3) from each dataset and computed their IoU, WSL,, and Pointing Game scores' (see Sec. 2.5). We computed the Pearson correlation between these scores and the human-Al team accuracy obtained in the previous human-study (Sec. 2.4).
Results While EP and GradCAM are state-of-the-art methods under Pointing Game [28] or WSL [29], we surprisingly found the accuracy of users when using these AMs correlate poorly with the IoU, WSL, and Pointing Game scores of heatmaps. Only in the case of GradCAM heatmaps for ImageNet (Fig. 4a), the evaluation metrics showed small positive correlation with human-AI team accuracy (r = 0.22 for IoU; r = 0.15 for WSL; and r = 0.21 for Pointing Game). In all other cases, i.e. GradCAM on Dogs; EP on ImageNet and Dogs (Fig. 4b-c), the correlation is negligible (-0.12 < r < 0.07). That is, the pointing or localization accuracy of feature attribution methods do not necessarily reflect their effectiveness helping users make correct decisions in image classification.
3.6 Machine learning experts found Nearest-Neighhors more effective than GradCAM
We have found in our previous study on lay-users that AMS can be useful to human-Al teams, but not more than a simple 3-NN in most cases (Secs. 3.1-3.4). Here, we aim to evaluate whether such conclusions carry over to the case of machine learning (ML) expert-users who are familiar with feature attribution methods, searest-neighhors and ImageNet. Because ML experts have deeper understanding into these algorithms than lay-users, our expert-study aims to measure the utility of these two visual explanation techniques ML researchers and practitioners.
'Not that WSL and Pointing Game scores binary while IoU scores real-valued.

---

Pearson correlation 0.22
Pearson correlation: -0.02
Pearson correlation: -0.06
Number ofimajee pane
Number of mm pet pobi
Human ccuracy
15.0
Human acturacy
-40
0.5
10.0
0.4
7.5
Human
000
Numee
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.50 0.75
IoU
loU
Pointing Game
(a) GradCAM's IoU
(b) EP's IoU
(c) EP's Pointing Game
Figure 4: The ImageNet localization performance (here, under IoU vs. human-annotated bounding boxes) of GradCAM [67] and EP [28] attribution maps poorly correlate with the human-Al team accuracy (y-axis) when users use these heatmaps image classification. Humans often can make correct decisions despite that heatmaps poorly localize the object (see the range 0.0-0.2 on x-axis).
As GradCAM was the best attribution method in the lay-user ImageNet study (Fig. 2), we chose GradCAM the representative for attribution maps and compare with 3-NN human-Al team image classification on ImageNet.
Experiment We repeated the same lay-user study but set of II expert users who Ph.D. students and postdoctoral researchers in ML and computer vision and from a range of academic institutions. We recruited five GradCAM users and six 3-NN users who very familiar with feature attribution and nearest-neighhor algorithms, respectively. Similar to the lay-user study, each expert presented with set of ransomly-chosen 30 images which include both natural and adversarial images.
Results The experts working with 3-NN performed substantially better than the GradCAM experts (Table I; mean accuracy 76.67% vs. 68)..% 3-NN consistently more effective on both natural and adversarial image subsets. Interestingly, the performance of GradCAM users also vary 3x more (standard deviation of 8.69% vs. 2.98% in accuracy). Aligned with the lay-user study, here, we found 3-NN to be more effective than AMs in improving human-Al team performance where the users domain experts familiar with the mechanics of how explanations were generated.
Table 1:
3-NN is far more effective than GradCAM in human-Al AI team image classification of both natural and adversarial images. See the mean (μ)) and std (a) per-user accuracy over all images.
Avg. validation
Natural
Adversarial
Users
accuracy
Accuracy
Trials
Accuracy
Trials
μ
GradCAM 3-NN
9.80/I0
67.31
70/104
69.57
32/46
68.00
8.69
6
9.83/10
78.45
91/116
73.44
47/64
76.67
2.98
4 Related Work
Attribution maps improved human accuracy in non-image domains Our work is motivated by prior findings showing that AMs helped humans make more accurate decisions on non-image classification tasks including predicting book categories from text [66], movie-review sentiment analysis [12, 66], predicting hypoxemia-riss from medical tabular data [47], or detecting fake text reviews [39, 40]. In contrast, on house- house-price prediction, AMs showed marginal utility to users and even hurt their performance in some cases [21].
Evaluating confidence scores on humans AI confidence scores have been found to improve user's trust on Al's decisions [79] and effective in human-AL Al team prediction accuracy on several NLP trust but only the complementary human-Al team
tasks [12]. In this work, we do not measure performance. To the best of our knowledge, our work the first to perform such -uaannevaluation of Al confidence scores for image classification.
Evaluating explanations on humans In NLP, using attribution methods as word-highlighter has been found improve user performance in question answering [26]. Bansal et al. [12] found

---

that such human-Al team performance improvement was because the explanations tend to make users more likely to accept AI's decisions regardless of whether the Al correct or not. We found consistent results that users with explanations tend to accept Al's predicted labels more (see Sec. A A10) with the exception of SOD users who tend to reject more (possibly due to the prediction-agnostiitty of SOD heatmaps).
In image classification, Chu et al. [19] found that presenting IG heatmaps users did not significantly improve human-accuracy on predicting age from facial photos. Different from [19], our study tested
multiple attribution methods (GradCAM, EP, SOD) and on the ImageNet classification task, which most attribution methods were evaluated on under proxy metrics (e.g. localization).
Shen and Huang [69] showed users GradCAM [67], EP [28]. and SmoothGrad [72] heatmaps and measured user-peeformance in harnessing the heatmaps to identify a label that an Al incorrectly predicts. While they measured the effect of showing all three heatmaps to users, we compared each method separately (GradCAM vs. EP vs. SOD) in a larger study. Similar to [69], Alqaraawi et al. [10] and Folke et al. [27] tested the capability of attribution maps in helping users understand the decision-making process of AI. Our work differs from the above three papers [10, 27, 69] and similar studies in the text domain [15, 33] in that we did not only measure how well users understand AIs, but also the human-Al team accuracy on a standard downstream task of ImageNet.
Using AMs debug models. Adebayo et al. [6] found that humans heavily rely on predicted labels rather than AMs, which is strongly aligned with our observations on the Natural ImageNet (in Sec. 3.2) where 3-NN outperforms AMs.
Adversarial attacks Concurrently, Folke et al. [27] also perform a related study that nearest- neighbor measures the effectiveness of both AMs and examples in helping users predict when AIs misclassify natural adversarial images [37], which we do not study in this paper. Our work largely differs from [27] in both the scale of the study, the type of image, and the experiment setup. To our knowledge, we are the first to evaluate human users on the well-known adversarial examples [49]. Previous work also found that it is easy to minutely change the input image such that the gradient- based AMs radically change [30]. However, none of such prior attempts studied the impact of their attacks to the actual end-users of the heatmaps.
5 Discussion and Conclusion
Limitations An inherent limitation of our study is that it is not possible to control the amount of prior knowledge that participant has before entering the study. For example, a human with a strong dog expertise may perform better the fine- fine-grained dog classification. In that case, the utility of explanations is unknown in our study. We attempted to estimate the effect of prior knowledge to human-Al team accuracy by asking each user whether they know a class before each trial. We found prior knowledge to account for 1-66 in accuracy (Sec. A5). Due to COVID and the large participant pool, our study was done online; which, however, made it infeasible for us to control various physical factors (e.g. user performing other activities during the experiment) compared to physical in-lab study. As most prior AM studies, we consider state-of-the-art methods for training humans on explanations [71, 73] partly because each explanation method may need different optimal training procedure.
To our knowledge, our work is the first to (1) evaluate human-Al team performance on the common ImageNet classification; (2) assess explanations on adversarial examples; (3) reveal the weak correla- tion between automatic evaluation metrics (Pointing Game, IoU [80], and WSL [80]) and the actual team performance. Such poor correlation encourages future interpretability research to take humans into their evaluation and to rethink the current automatic metrics [22, 42). We also showed the first evidence in the literature that a simple 3-NN can outperform existing attribution maps, suggesting a combination of two explanation types may be useful to explore in future work. The superiority of 3-NN also suggests prototypical [17, 32] and visually-grounde explanations [35, 36) may be more
effective than heatmaps.
10

---

Acknowledgement
This research was supported by the MSIT Grant No. ITT-22021--00044401444, and the Technology Innovation Program of MOTIE Grant No. 2000682. AN was supported by the NSF Grant No. 1850117 and donation from NaphCare Foundation.

---

References
[1] Torchray/attributionn benchmark.pp py master facebwokrreeaurcbboorcnnyy ttttps///githnn com/ on 04/28/2021).
py. (Accessed
[2] The fbi's capitol riot investigation used surveillance technology that advocates say threatens civil liherties - the washington post. https://wwe www.:/wwwwnnnnosat.....mm wttc//omo/ct/eeeeeee wtw.e/ooogg/22//000//e/
privacy/ (Accessed on 04/1 1/2021).
[3] The new lawsuit that shows facial recognition is officially
civil rights issue mit
https://www www.noooogoeeieeee...oooo wttp:///////////////////10 robert-villiams iatiiams-faciaall daciill-ecogmiiiinn www.lllawttttete detroot-ptoooooeee../.. (Accessed on 04/15/2021).
technology review.
[4] vision/resnet..py at master pytorch/vision. httpa://githin wwwp//yttcc//viieeeooo cttp///roorrd//aaiooo///aa/aaaaa
torchvisiion/modell//aaaaaa wtt.eeennntttt.......e.e.. py#L188. (Accessed on 04/28/2021).
[5] facebookresennc//oorrryyyyy Undssa/uorrrrnuynnnnnsnnnuuuuunn Understanding deep networks via extremal perturbations and smooth masks. (Accessed on 04/28/2021).
[6] Adebayo. J.. Gilmer, J.. Muelly, M.. Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03220 2018.
[7] Agarwal, C. and Nguyen, A. Explaining image classifiers by removing input features using generative models. In Proceedings the Asian Conference on Computer Vision, 2020.
[8] Agarwal, C., Nguyen, A., and Schonfeld, D. Improving robustness adversarial examples by encouraging discriminative features. 2019 IEEE International Conference on Image Processing (ICIP), pp. 3801- 3505. IEEE, 2019.
[9] Alcorn, M. A., Li, Q.. Gong, Z., Wang. C., Mai, L., Ku, W.-S., and Nguyen, A. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4845-4854, 2019.
[10] Alqaraawi, A., Schuessler, M., Weiß, P. Costanza, E., and Berthouze, N. Evaluating saliency map explanations for convolutional neural networks: user study. In Proceedings the 25th International Conference on Intelligent User Interfaces, pp. 275-285, 2020.
[11] Anwyl-Irvine, A. L., Massonnié, J.. Flitton, A., Kirkham, N., Evershed, J. K. Gorilla our midst: An online behavioral experiment builder. Behavior research methods, 52(1):388 407, 2020.
[12] Bansal, G., Wu, T., Zhu, J., Fok, R., Nushi, B., Kamar, E., Ribciro, M. T., and Weld, S. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. arXiv preprint arXiv:2006.1477 14779, 2020.
[13] Bansal, N., Agarwal, C., and Nguyen, A. Sam: The sensitivity of attribution methods to hyperparameters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8673-8663 8683, 2020.
[14] Bendale, A. and Boult, T. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563 11723115,,000 2016.
[15] Chandrasekaran, A., Prabhu, V., Yadav, D., Chattopadhyay, P., and Parikh, D. Do explanations make vqa models more predictable to a human? arXiv preprint 11t0:1880.2366 2018.
[16] Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks, In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 839 847. IEEE, 2018.
[17] Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., and Rudin, C. This looks like that: deep learning for interpretable image recognition. arXiv preprint 1806. 10574, 2018.
[18] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. This looks like that: Deep learn ing for interpretable image recognition.
Wallach, H., Larochelle, H.. Beygelzimer, A., d'Alché- Buc, F, Fox, E., and Gamett, R. (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https: httthttss////ooeeeee //tp:///edddig.s.. neurips httpps.ccppppppe cc/ppaper/2199//ilee Pwper.appepppee
42f
12

---

[19] Chu. E., Roy, D.. and Andreas, J. Are visual explanations useful? case study in model-in-the-loop prediction. arXiv preprint 1Xii:2007.11248. 2020.
[20] Covert, I,, Lundberg, S., and Lee, S.-L Feature removal is unifying principle for model explanation methods. arXiv preprint arXiv:0011.0662. 2020.
[21] Dinu, J. Bigham, J., and Kolter, J. Z. Challenging common interpretability assumptions in feature attribution explanations. arXiv preprint ar214:201..02748, 2020.
[22] Velei, F. and Kim, B. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.0800 08608, 2017.
[23] Veshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S., O'Brien, D., Schieber, S., Waldo, J., Weinherger, D., and Wood, A. Accountability of under the law: The role of explanation. arXiv preprint axXiv//11.011334, 2017.
[24] Du, M., Liu, N., Song, Q., and Hu, X. Towards explanation of dnn-based prediction with guided feature inversion. In Proceedings of 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1358-1367, 2018.
[25] Elliott, A., Law, S., and Russell, C. Perturbations on the perceptual ball. arXiv preprint 19 2019.
[26] Feng, and Boyd-Graber, J. What can ai do for me? evaluating machine learning interpretations in cooperative play. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pp- 229-239, 2019.
[27] Folke, T. Li, Z., Sojitra, R. B., Yang, S. C.-H., and Shafto, P. Explainable ai for natural adversarial images. arXiv preprint arXiv:2106.091 09100, 2021.
[28] Fong, R., Patrick, M., and Vedaldi, A. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 2950-2958, 2019.
[29] Fong, R. C. and Vedaldi, A. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference Computer Vision, pp. 3429-3437, 2017.
[30] Ghorbani, A., Abid, A., and Zou, J. Interpretation of neural networks is fragilc. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688 2019.
[31] Goodman, B. and Flaxman, S. European union regulations on algorithmic decision-making and "right to explanation". Al Magazine, 38(3):50-57, 2017.
[32] Goyal, Y., Wu, Z., Emst, J., Batra, D., Parikh, D., and Lee, S. Counterfactual visual explanations. In International Conference on Machine Learning, pp. 2376-2384, PMLR, 2019.
[33] Hase, P. and Bansal, M. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? arXiv preprint arxiv:2005.01831, 2020.
[34] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 700-777777777 2016.
[35] Hendricks. L. A., Akata, Z., Rohrbach, M., Donahue. J.. Schiele, B., and Darrell, T. Generating visual explanations. European Conference on Computer Vision, pp. 3-19. Springer, 2016.
[36] Hendricks, L. A., Hu, R., Damell, T., and Akata, Z. Generating counterfactual explanations with natural language. arXiv preprint MXXIv:1806.09880 2018.
[37] Hendrycks, D., Zhao, K.. Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15277 2021.
[38] Khosla, A., Jayadevaprakash. N., Yao. B., and Fei-Fei, L. Novel dataset for fine-grained image catego- rization: Stanford dogs. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 201 1.
[39] Lai, V. and Tan, C. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 29-38, 2019.
13

---

[40] Lai, V., Liu, H., and Tan, C. why is' chicago' deceptive?" towards building model-driven tutorials for humans, In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-13. 2020.
[41] Lapuschkin, S., Binder, A., Montavon, G., Muller, K.-R., and Samek, W. Analyzing classifiers: Fisher vectors and deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2912-2920, 2016.
[42] Leavitt, M. L. and Morcos, A. Towards falsifiable interpretability research. arXiv preprint arXiv:2010. 12016, 2020.
[43] Liu, J.-J., Hou, Q.. Cheng, M.-M.. Feng. J., and Jiang, A simple pooling bassd design for real-time salient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3917-3926, 2019.
[44] Liu, Z., Liu, Q. Liu, T., Xu, N.. Lin, X., Wang. Y., and Wen, W. Feature distillation: Dan-oriented jpeg compression against adversarial examples. In 2019 JEEE/CCE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 860-866 868. IEEE, 2019.
[45] Lu, Y., Guo, W., Xing, X., and Stafford Noble, W. Robust decoy-enhanced saliency maps. arXiv e-prints, pp. arXiv 2002, 2020.
[46] Lundberg. S. and Lee, S.-I. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874. 2017.
[47] Lundberg. S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W.. Newman, S.-F., Kim, J., et al. Explainable machine machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering, 2(10):749 760, 2018.
[48] Mac Aodha, O., Su, S., Chen, Y., Perona, P., and Yue, Y. Teaching categories to human learners with visual explanations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3820 3828, 2018.
[49] Madry, A., Makelov, A., Schmidt, L., Tsipras. D.. and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608; 2017.
[50] Marcel, Rodriguez, Y. Torchvision the machine-vision package of torch. Proceedings of the 18th ACM international conference on Multimedia, pp. 1485 1488, 2010.
[51] Miller, G. A. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995.
[52] Munusamy Kahilan, V., Morris, B., and Nguyen, A. Vectordefense: Vectorization defense adversarial examples. arXiv e-prints, pp. arXiv-1804, 2018
[53] Nauta, M., Jutte, A., Provoost, J., and Seifert, C. This looks like that, because... explaining prototypes for interpretable image recognition. arXiv preprint arXiv:201 1.02863, 2020.
[54] Nguyen, A., Yosinski, and Clune, J. Deep neural networks are casily fooled: High confidence predictions for unrecognizable images. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), CVPR 15, pp. 427-436. IEEE, June 2015. doi: 10.1 109110///P.22011 201.7926666...
[55] Nguyen, A., Yosinski, J., and Clune, J. Understanding neural networks via feature visualization: A survey. In Explainable Al: interpreting, explaining and visualizing deep learning, pp. 55-76. Springer, 2019.
[56] Nguyen, G., Chen, S., Jun, T. J., and Kim, D. Explaining how deep neural networks forget by deep visualization. In International Conference on Pattern Recognition, 162-173. Springer, 2021.
[57] Palan, S. and Schitter, C. Prolific. ac a subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17:22-27,22 2018.
ac-a
[58] Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS& &P), pp. 372-387. IEEE, 2016.
[59] Peer, E., Brandimarte, L., Samat, S., and Acquisti, A. Beyond the turk: Alterntive platforms for crowdsoureing behavioral research. Journal of Experimental Social Psychology, 70:153 2017.
[60] Petsiuk, V., Das. A., and Saenko, K. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.0 07421, 2018
14

---

[61] Rajpurkar, P.. Irvin, J. Zhu, K., Yang, B., Mehta, H. Duan, T. Ding, D.. Bagul, A.. Langlotz, C.. Shpanskaya. K., et al. Chexnet: Radiologist-Iveel pneumonia detection chest x-rays with deep learning. arXiv preprint arXiv:1711.0522. 2017.
[62] Rauber. J., Brendel, W., and Bethge, M. Foolbox: A python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017. URL http://srxiv. org/ abs/1707 1771104117......
[63] Rebuffi, S.-A., Fong, R., X., and Vedaldi, A. There and back again: Revisiting backpropagation salicney methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8839 1839-8848, 2020.
[64] Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L.,, and Zhong, C. Interpretable machine learning: Fundamental principles and 10 grand challenges. arXiv preprint 1xxi:2103.1125/1 2021.
[65] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision. 1 115(3))211-22) 2015.
[66] Schmidt, P. and Biessmann. F. Quantifying interpretability and trust in machine learning systems. arXiv preprint arxiv: 1et.:9900005556 2019.
[67] Selvaraju, R. R., Cogswell, M., Das, A.. Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-basee localization. In Proceedings of the IEEE international conference on compuder vision, pp. 618-626, 2017.
[68]
Shakhnarovich, G., Darrell, T., and Indyk, P. Nearest- searest-neighbor methods learning and vision. Neural Information Processing. 2005.
[69] Shen, H. and Huang, T.-H. How useful are the machine-generated interpretations to general users? a human evaluation guessing the incorrectly predicted labels. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume pp. 168-172, 2020.
[70] Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint / 2013.
[71] Singla, A., Bogunovic, I., Bartók, G., Karbasi, A., and Krause, A. Near-optimally teaching the crowd to classify. In International Conference on Machine Learning, pp. 154-162. PMI.R, 2014.
[72] Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. Smoothgrad: removing noise by adding noise. arXiv preprint arviv: (06.003220.. 2017.
[73] Su, S., Chen, Y., Mac Aodha, O., Perona, P., and Yue, Y. Interpretable machine teaching via feature feedback. 2017.
[74] Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. International Conference on Machine Learning, pp. 3319-3328. PMLR, 2017.
[75] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint 1312. 1312,6199, 2013.
[76] Wang, H., Wang. Z., Du M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., and Hu, X. Score-cam: Score-weighter visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 24 25, 2020.
[77] Wu, N., Phang, J., Park, J., Shen, Y., Huang. Z., Zorin, M., Jastrzębski. S., Févry, T., Katsnelson, J., Kim, E., Wolfson, S., Parikh, U., Gaddam, S., Lin, L. L. Y., Ho, K., Weinstein, J. D., Reig. B., Gao, Y., Pysarenko, H. K., Lewin, A., Lee, J., Airola, K., Mema, E.,, Chung. S., Hwang, F.,, Samreen, N., Kim, S. G., Heacock, I.,, Moy, I.,, Cho, K., and Geras, K. J. Deep neural networks improve radiologists' performance in breast cancer screening. IEFE Transactions on Medical Imaging, pp. 1-1, 2019. doi: (0.109//MM22019924555
[78] Zhang, J., Bargal, S. A., Lin, 7.,, Brandt, J., Shen, X., and Sclaroff, S. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084-11 2018.
[79] Zhang, Y., Liao, Q. V., and Bellamy, R. K. Efffect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference Fairness, Accountability, and Transparency, pp. 295-305, 305, 2020.
[80] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921 2929, 2016.
15


Reviews:
[{"review_id": "c5msNZW3CnL", "reviewer": "Reviewer_Fe2L", "summary": "The paper investigates if current widely used evaluation metrics for saliency maps or explainable visual AI like pointing game, weakly supervised localization are trustable by conducting a very extensive user study. Some interesting conclusions and observations are given. In this sense, this is a good paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper claims it is the ﬁrst, large-scale user study, but 320 users seem to be not large enough when comparing with previous papers. It is just a total submission. In line 133, each experiment was conducted by 30 users. In the paper (Teaching Categories to Human Learners with Visual Explanations), even 40 users were recruited for each setting. So I think the paper should tone down the claim. The same for some conclusions, because only dogs data is used, this can not generalize to all fine-grained domains.\n\nAlso, the training step is a machine teaching task, how to select the samples and how to decide their order shown to the user. This is critical, there are many machine teaching papers, for example\n1)Teaching Categories to Human Learners with Visual Explanations\n2)Near-Optimally Teaching the Crowd to Classify\n3)Interpretable Machine Teaching via Feature Feedback\nThis has an impact on the quality of the user training. The suboptimal teaching samples may cause the observation less trustable.\n\nI'm curious about the variance of the user study results. Some statistic analysis should be included.\n\nBecause English is used to filter users. Not sure if this makes the conclusion have a bias on the race.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "bYVuH6IqWu", "reviewer": "Reviewer_Tf4g", "summary": "This paper performs a human evaluation study on feature attribution style explanations.  The main conclusions include:\n\n1. Attribution methods (AMs) are not more effective than an \"explanation\" which consists of three nearest neighbor images.  In some settings (adversarial setting on Stanford Dogs) presenting confidence scores to humans is most helpful.\n\n2.  Current automatic metrics to measure attribution methods don't correspond well with this human evaluation.\n\n3.  When testing on AI-expert users, nearest neighbor explanations are more helpful.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I am positive about this submission.  \n\n*Originality*\n\nThere have been human evaluations of attribution methods in prior papers, but I think this paper adds a few new things which are valuable:\n\n* evaluation of multiple methods on multiple different datasets\n* inclusion of experiments on experts and non-experts\n* inclusion of a simple form of explanation -- nearest neighbors\n* study comparing automatic metrics to human evaluation\n\nThis paper could be more useful if it could provide concrete suggestions on where we go as a field.  In particular:\n\n* after this study do researchers think that our current attribution methods are bad, but there could exist attribution methods that are helpful for this task or do researchers thing that attribution methods in general are weak and we should look for other forms of explanation?\n* this paper looks at one kind of study: can humans and AI work together to reject bad decisions?  Are there other ways attribution methods could help humans not covered in this study (something to potentially cover in limitations)\n\n\n*Quality*\n\nI found this paper easy to read and the methodology appears sound.  However, I had a few comments/questions:\n\n* What would the error bars be in Fig 3?  Are the differences actually statistically significant?  If the results aren't statistically significant (e.g., 3-NN, AM, and no explanation are the same), I don't think this significantly changes the current conclusions (currently AM isn't really any better than 3-NN), but I think it is important to include.\n* In the human evaluation, my understanding is that each human saw one setting (e.g., each human saw the 3-NN setting).  Did author's consider showing users multiple settings (and permuting order in which users saw the settings?).  I am wondering if there is some variation btwn humans (e.g., familiarity with dog breeds) that could be controlled for better in this way.\n* I really liked the inclusion of the SOD baseline.  Surprisingly, SOD does quite well on natural ImageNet and Adversarial Stanford Dogs.  I am wondering why this might be (part of me wonders if this is because the differenced between all the methods are not statistically significant).\n* One concern I have with these sorts of studies is that the UI may not be optimized; in particular for most AM the heat map covers the part of the image important for classification.  Did authors consider this or experiment with this?\n\nAdditionally I believe there are some missing citations (other works which have evaluated explanation models):\n\nChandrasekaran et al.  \"Do Explanations make VQA Models more Predictable to a Human?\"  EMNLP 2018.\nHase et al.  \"Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?\"  ACL 2020.\n\n*Clarity*\n\nOverall I found this paper clear and easy to read.\n\n*Significance*\n\nOver the last few years, there have been critiques over attribution methods and explanations in general.  This paper provides something new to the conversation (see under originality) so I think it is useful for the community.  I do have some concerns about the way experiments were carried out (mainly significance tests), but I imagine this can be cleared up in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is a limitations section but no specific societal impact section.  It would be good to include a societal impact statement, but I don't think this work contains any specific ethical concerns.\n\nThough not listed under societal impact, I appreciated the description of how users were paid.  I wonder where participants were located and how the $10.20 compared to minimum wage in that region.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours."}, "extra_text": {}}, {"review_id": "CHwZhuisZjp", "reviewer": "Reviewer_62dY", "summary": "The paper presents results of a user study that was designed to study the effectiveness of feature attribution methods in assisting human decision makers. Experiments are conducted on Stanford dog dataset and imageNet dataset. The authors mention that feature attribution methods are not any better than 3 nearest neighbor, and that especially for fine grained classification task ( on Stanford dog dataset), feature attribution map in fact hurts. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper studies an interesting problem, to analyze the effectiveness of feature attribution maps in assisting human decision makers. Such studies are necessary given the prevalence of AI models across a variety of high stake applications. \n\n2. The paper is fairly well presented and organized. \n\n3. The authors also list some limitations of their work, giving the reader a better view of the problem scope. \n\n4. Some interesting results are also provided--that feature attribution maps actually hurt in fine grained classification on the dog dataset, and that it is not any better than 3NN in some of the experimental settings. \n\n\nThe following are some issues of concern:\n\n1. Evaluation metrics:\n\nThe authors  consider pointing game, intersection over union, and weakly supervised localization for evaluation. However, the following recently proposed metrics related to concept activation vectors, common feature, commonality, etc. are very relevant, especially given that large scale user studies have been performed to learn about their effectiveness. Thus, the metrics used by the authors in evaluation are limited, and the results contained are not necessarily significant. \n\nBenchmarking Attribution Methods with Relative Feature Importance, Yang and Kim, 2019\nhttps://arxiv.org/pdf/1907.09701.pdf\n\nInterpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCA) , Kim et. al. http://proceedings.mlr.press/v80/kim18d/kim18d.pdf\n\n2. Methodology\n\na. The authors perform the user study on 320 \"lay people\" and 11 \"experts\". This distribution is highly skewed, and it does not therefore represent a reasonable perception across people from these two groups. What was the rationale behind choosing a few \"experts\" vs so many \"lay' people? \n\nb. Sufficient details about these 320 lay people is missing. What were their educational backgrounds, where were they from, etc. These factors can significantly affect how the participants perceive the attribution maps. If these factors were taken into account, then it is not clear how it was done. \n\nc. In the process of giving feedback to the participants, how was it ensured that biases associated with the AI results were not reinforced? In other words, how was it ensured that the feedback did not prime the participants to respond in a certain way? Otherwise, the results are not necessarily accurate. \n\n3. Significance:\n\nThe paper starts on a strong note, with reference to high stake applications, and the need for understanding the effectiveness of attribution maps in assisting humans.  However, the authors conduct experiments on ImageNet, and Stanford Dog datasets. It would have been much compelling to show results on a more critical application such as healthcare, environment modeling, etc.  This would enhance the significance of the results. \n\n4. Originality:\n\nIn its current form, the novelty of the work seems incremental. The metrics considered are not based on prior art, and also there have been extensive user studies related to understanding saliency maps such as those of [8, 55] that the authors mention. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the above response for details pertaining to limitations. \n\nSuggestions for improvement:\n\n1. It will be useful to include a detailed description of the user study, including the rationale for the choice of the specific participants groups and their sizes, participants backgrounds, etc.\n\n2. It is important to compare with other relevant evaluation metrics such as those listed above. \n\n3.  It might be better to consider a high stakes application such as in healthcare, climate, etc. to make the results more compelling. \n\n4. The authors consider a few saliency based methods , primarily GRADCAM. To appeal to the broader research community, it would help to consider a few other models as well. \n\n5. The authors might want to consider some qualitative analysis of feature attribution methods such as those listed below and explain what complementary insights a user study such as their study provides. \n\na. Epistemic values in feature importance methods: Lessons from feminist epistemology, FAccT 2021\nb. If saliency cropping is the answer, what is the question?, CVPR Beyond Fairness Workshop 2021", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "9s9bvniM3P", "reviewer": "Reviewer_wdnC", "summary": "The main contribution of this paper is a thorough and well-designed user study towards using feature attribution maps, i.e. GradCAM, in practice to help human users understand prediction results of neural networks. Following the user study, the author finds several existing metrics in evaluating the goodness of feature attributions do not align with the user’s gains in having feature attributions and conclude that in the evaluated scenarios features attributions are not more useful than K nearest neighbor explanations. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general the paper has presented very interesting results and conducted a well-designed user-study that has a larger group of interviewees than previous ones. The presentation is easy to follow. My major concern is the fundamental assumptions about attribution methods made by the authors. I will elaborate more in the *Quality* paragraph.\n\n### Originality\nA related work from Adebayo et. al. [1] that leverages user study to evaluate the feature attribution methods is missing and may potentially be very important to compare with the paper’s findings. Though Adebayo et. al. [1] has less interviewees compared to this study but it contains more tests for a different group of feature attributions that this paper does not consider as *state-of-the-art*. Can the authors elaborate more about the important difference between this paper and Adebayo et. al. [1] that makes the user study present in this paper as the *“the first, large-scale user study to shed light on the effectiveness of AMs”* as claimed in the paper?\n\n### Quality\nThis paper has presented a well-designed user-study pipeline to evaluate the performance of explanations methods for the downstream users. I have no concerns on the dataset distribution, network architectures, and the details about the user-study. My only concern is that there is not enough justification why the way the feature attribution is used in this paper agrees with what feature attributions are designed for in the first place. \n\n1. Firstly, this paper expects users to leverage feature attribution in identifying whether the prediction matches the groundtruth. However, feature attributions, as its name suggests, is only a way to attribute the output scores over the input (or internal) features. The target question an attribution answers is more like **“how features contribute to the quantity of interest, i.e. the prediction”** instead of **“is the prediction accurate”** [2]. Justifications on why feature attributions are expected to be the one, among other explanations tools,  that should answer the question this paper designs are currently missing. By other explanations tools, an example is instance-based method [3, 4].\n\n2. Secondly, the paper concludes that attributions with high localization accuracy do not help humans to make the correct decisions (line 297-299). Because the localization metrics discussed in this paper are based on the bounding boxes generated by humans, these metrics are actually evaluating whether attributions match the human’s prior. Therefore, the authors need to explain why the optimal attribution method that perfectly matches the human’s prior is supposed to help a human who already knows how to draw the bounding box in general. I agree that evaluation metrics need improvement but the current way of concluding the results does not help us to understand why the localization metrics do not calibrate with the user-study. More analysis will help the reader to gain insights from the paper’s result.\n\n### Clarity\nThe authors have documented their process in a reproductive way. The paper is well-organized and well-written. \n\n### Significance\nThis work is very significant for showing whether explanations can actually help humans in practice to understand the predictions of deep classifiers. \n\n[1] Adebayo, J., Muelly, M., Liccardi, I., & Kim, B. (2020). Debugging Tests for Model Explanations. NeurIPS 2020.\n\n[2]K. Leino, S. Sen, A. Datta, M. Fredrikson and L. Li, \"Influence-Directed Explanations for Deep Convolutional Networks,\" 2018 IEEE International Test Conference (ITC), 2018.\n\n[3] Yeh, C., Kim, J.S., Yen, I.E., & Ravikumar, P. (2018). Representer Point Selection for Explaining Deep Neural Networks. NeurIPS 2018.\n\n[4] Koh, P. W., & Liang, P. (2017, July). Understanding black-box predictions via influence functions. In ICML 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a user study aimed at evaluating the effectiveness of feature attribution methods (e.g. GradCAM) in assisting human decision making. Interestingly, feature attribution methods are found to not be better than a comparatively simple 3-nearest-neighbor-based method; in some cases, a feature attribution map was even harmful to human performance.

Reviewers found the paper to be compelling and well-written, though individual reviewers raised specific potential concerns, such as the size of the study (320 subjects) or the relative artificiality of the task (vs. a real-world application). However, in totality, reviews were positive and the authors’ rebuttals were helpful for clarifying reviewer questions.

Decision: Accept (Poster)