The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

1 Introduction . Why did a computer vision system suspect that a person had breast cancer [ 77 ] , or was an US capitol rioter [ 2 ] , or a shoplifter [ 3 ] ? The explanations for such high-stake predictions made by existing Artificial Intelligence ( AI ) agents can impact human lives in various aspects , from social [ 22 ] , to scientific [ 54 ] , and legal [ 3 , 23 , 31 ] . An image classifier ’ s decisions can be explained via an attribution map ( AM ) [ 13 ] , i.e . a heatmap that highlights the input pixels that are important for or against a predicted label . AMs can be useful in localizing tumors in x-rays [ 61 ] , reflecting biases [ 41 ] and memory [ 56 ] of image classifiers , or teaching humans to distinguish butterflies [ 48 ] . Via AMs , part of the thought process of AIs is now transparent to humans , enabling human-AI collaboration opportunities . Can machines and humans work together to improve the accuracy of humans in image classification ? This question remains largely unknown although , in other non-image domains , AMs have been found useful to humans in the downstream tasks [ 12 , 39 , 47 , 66 , 66 ] . Furthermore , it is also unknown whether any existing automatic evaluation scores may predict such effectiveness of AMs to humans , the target user of AMs . Most AMs were often only evaluated on ∗Corresponding authors . Work done when GN was at KAIST . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . confidence 20 % heatmaps 3 nearest neighbors GradCAM EP SOD input Yes No vs lorikeet ? A B C user in lorikeet AI ’ s top-1 predicted label : lorikeet groundtruth label : “ bee eater ” ( a ) Given an input image , its top-1 predicted label ( here , “ lorikeet ” ) and confidence score ( A ) , we asked the user to decide Yes or No whether the predicted label is accurate ( here , the correct answer is No ) —i.e . the accuracy of human-AI teams without visual explanations . We also compared this baseline with the treatments where one attribution map ( B ) or a set of three nearest neighbors ( C ) is also provided to the user ( in addition to the confidence score ) . Method ImageNet Stanford Dogs µ σ µ σ Confidence 72.44 8.25 61.71 11.39 GradCAM 72.58 8.11 60.56 9.27 EP 73.85 6.88 56.67 10.57 SOD 72.06 7.63 61.67 10.87 3-NN 76.08 5.86 57.20 10.58 ( b ) Users of 3-NN outperform all other users including those using attribution maps on ImageNet classification . However , on fine-grained dog classification ( Stanford Dogs ) , visual explanations ( GradCAM , EP , 3-NN ) surprisingly did not help but hurt the human performance . Results from testing humans on both real and adversarial images . Figure 1 : Overview of the methods compared in our user-study and its results . proxy evaluation metrics such as pointing game [ 78 ] , object localization [ 80 ] , or deletion [ 60 ] , which may not necessarily correlate with human-AI team accuracy on a downstream task . In this paper , we conduct the first user-study to measure AM effectiveness in assisting humans on the 120-class Stanford Dog [ 38 ] fine-grained classification and 1000-class ImageNet [ 65 ] classification , which is the task that most attribution methods were designed for . We asked 320 lay and 11 expert users to decide whether machine decisions are correct after observing an input image , top-1 classification outputs , and explanations ( Fig . 1a ) . This task captures two related quantities : ( 1 ) human accuracy on image categorization , here simplified to binary classification ; and ( 2 ) model debugging , i.e . understanding whether AI is correct or wrong . For a fuller understanding of the effectiveness of AMs , we tested users on both real and adversarial images [ 75 ] . Our main findings include:2 1 . AMs are , surprisingly , not more effective than nearest-neighbors ( here , 3-NN ) in improving human-AI team performance on both ImageNet and fine-grained dog classification ( Sec . 3.2 ) . 2 . On fine-grained dog classification , a harder task for a human-AI team than 1000-class ImageNet , presenting AMs to humans interestingly does not help but instead hurts the performance of human-AI teams , compared to AI alone without humans ( Fig . 2b ) . 3 . On adversarial Stanford Dog images which are very hard for humans to label , presenting confidence scores only to humans is the most effective method compared to all other explanations tested including AMs and 3-NN examples ( Sec . 3.2 ) . 4 . Despite being commonly used in the literature , some automatic evaluation metrics—here , Pointing Game [ 78 ] , localization errors [ 80 ] , and IoU [ 80 ] —correlate poorly with the actual human-AI team performance ( Sec . 3.5 ) . 5 . According to a study on 11 AI-expert users , 3-NN is significantly more useful than GradCAM [ 67 ] , a state-of-the-art attribution method for ImageNet classification ( Sec . 3.6 ) . 2 Methods . 2.1 Image classification tasks . To evaluate human-AI team performance on image classification , for each image , we also presented to a user : ( 1 ) an AI ’ s top-1 predicted label , ( 2 ) its confidence score , and , optionally ( 3 ) a visual explanation that is generated in attempt to explain the predicted label . The user was then asked to decide whether AI ’ s predicted labels are correct or not ( see Fig . 1a ) . We performed our experiments on two image datasets of ImageNet and Stanford Dogs , which are of increasing difficulty . 2Code and data are available at http : //github.com/anguyen8/effectiveness-attribution-maps ImageNet We tested human-AI teams on ImageNet [ 65 ] —an image classification task that most attribution methods were tested on or designed for [ 7 , 60 , 67 , 80 ] . ImageNet is a ∼1.3M-image dataset spanning across 1000 diverse categories from natural to man-made entities [ 65 ] . Stanford Dogs To test whether our findings generalize to a fine-grained classification task that is harder to human-AI teams than ImageNet , we repeated the user-study on the 120-class Stanford Dogs [ 38 ] dataset , a subset of ImageNet . This dataset is challenging due to the similarity between dog species and the large intra-class variation [ 38 ] . Compared to ImageNet , Stanford Dogs is expected to be harder to human-AI teams because lay users often have significantly less prior knowledge about fine-grained dog breeds compared to a wide variety of everyday objects in ImageNet . 2.2 Image classifiers . We took ResNet-34 [ 34 ] pretrained on ImageNet ( 73.31 % top-1 accuracy ) from torchvision [ 50 ] as the target classifier ( i.e. , the AI ) for both ImageNet and Stanford Dogs classification tasks because the 1000-class ImageNet that the model was pretrained on includes 120 Stanford Dogs classes . The visual explanations in this paper were generated to explain this model ’ s predicted labels . We chose ResNet-34 because ResNets were widely used in feature attribution research [ 7 , 28 , 45 , 60 , 67 ] . 2.3 Visual explanation methods . To understand the causal effect of adding humans in the loop , we compared the performance of a standard AI-only system ( i.e . no humans involved ) and human-AI teams . In each team , besides an input image and a corresponding top-1 label , humans are also presented with the corresponding confidence score from the classifier , and a visual explanation ( i.e . one heatmap or three 3-NN images in ; see examples from our user-study in Sec . A13 ) . AI-only A common way to automating the process of accepting or rejecting a predicted label is via confidence-score thresholding [ 14 ] . That is , a top-1 predicted label is accepted if its associated confidence score is ≥ T , a threshold . We found the optimal confidence threshold T ∗ that produces the highest accepting accuracy on the entire validation set by sweeping across threshold values ∈ { 0.05 , 0.10 , ... , 0.95 } , i.e . at a 0.05 increment . The optimal T ∗ values are 0.55 for ImageNet and 0.50 for Stanford Dogs ( details in Sec . A6 ) . Confidence scores only To understand the impact of visual explanations on human decisions , we also used a baseline where users are asked to make decisions given no explanations ( i.e . given only the input image , its top-1 predicted label and confidence score ) . To our best knowledge , this baseline has not been studied in computer vision , but has been shown useful in some non-image domains for improving human-AI team accuracy or user ’ s trust [ 12 , 79 ] . GradCAM and Extremal Perturbation ( EP ) We chose GradCAM [ 67 ] and Extremal Perturbation ( EP ) [ 28 ] as two representatives for state-of-the-art attribution methods ( Fig . 1a ) . Representing for the class of white-box , gradient-based methods [ 16 , 63 , 80 ] , GradCAM relies on gradients and the activation map at the last conv layer of ResNet-34 to compute a heatmap . GradCAM passed a weight-randomization sanity check [ 6 ] and often obtained competitive scores in proxy evaluation metrics ( see Table 1 in [ 28 ] and Table 1 & 2 in [ 25 ] ) . In contrast , EP is a representative for perturbation-based methods [ 7 , 20 , 29 ] . EP searches for a set of a % of input pixels that maximizes the target-class confidence score . We followed the authors ’ best hyperparameters—summing over four binary masks generated using a ∈ { 2.5 , 5 , 10 , 20 } and a Gaussian smoothing kernel with an std equal to 9 % of the shorter side of the image ( see the code [ 1 ] ) . We used the TorchRay package [ 5 ] to generate GradCAM and EP attribution maps . We initially also tested vanilla Gradient [ 70 ] , Integrated Gradient ( IG ) [ 74 ] , and SHAP [ 46 ] methods in a pilot study but discarded them from our study since their AMs are too noisy to be useful to users in a pilot study . Salient object detection ( SOD ) To assess the need for heatmaps to explain a specific classifer ’ s decisions , we also considered a classifier-agnostic heatmap baseline . That is , we used a pre-trained state-of-the-art salient-object detection ( SOD ) method called PoolNet [ 43 ] , which uses a ResNet-50 backbone pre-trained on ImageNet . PoolNet was trained to output a heatmap that highlights the salient object in an input image ( Fig . 3 ) — a process that does not take our image classifier into account . Thus , SOD serves as a strong saliency baseline for GradCAM and EP . 3-NN To further understand the pros and cons of AMs , we compared them to a representative , prototype-based explanation method ( e.g . [ 18 , 53 , 55 ] ) . That is , for a given input image x and a predicted label y ( e.g. , “ lorikeet ” ) , we show the top-3 nearest images to x by retrieving them from the same ImageNet training-set class y ( Fig . 1a ) . To compute the distance between two images , we used the L2 distance in the feature space of the last conv layer of ResNet-34 classifier ( i.e . layer4 per PyTorch definition [ 4 ] or conv5_x in He et al . [ 34 ] ) . We chose layer4 as our pilot study found it to be the most effective among the four main conv layers ( i.e . layer1 to layer4 ) of ResNet-34 . While k-NN has a wide spectrum of applications in machine learning and computer vision [ 68 ] , the effectiveness of human-AI collaboration using prototype-based explanations has been rarely studied [ 64 ] . To the best of our knowledge , we provided the first user study that evaluates the effectiveness of post-hoc , nearest-neighbor explanations ( here , 3-NN ) on human-AI team performance .

Summary: The main contribution of this paper is a thorough and well-designed user study towards using feature attribution maps, i.e. GradCAM, in practice to help human users understand prediction results of neural networks. Following the user study, the author finds several existing metrics in evaluating the goodness of feature attributions do not align with the user’s gains in having feature attributions and conclude that in the evaluated scenarios features attributions are not more useful than K nearest neighbor explanations.