The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

1 Introduction . Why did a computer vision system suspect that a person had breast cancer [ 77 ] , or was an US capitol rioter [ 2 ] , or a shoplifter [ 3 ] ? The explanations for such high-stake predictions made by existing Artificial Intelligence ( AI ) agents can impact human lives in various aspects , from social [ 22 ] , to scientific [ 54 ] , and legal [ 3 , 23 , 31 ] . An image classifier ’ s decisions can be explained via an attribution map ( AM ) [ 13 ] , i.e . a heatmap that highlights the input pixels that are important for or against a predicted label . AMs can be useful in localizing tumors in x-rays [ 61 ] , reflecting biases [ 41 ] and memory [ 56 ] of image classifiers , or teaching humans to distinguish butterflies [ 48 ] . Via AMs , part of the thought process of AIs is now transparent to humans , enabling human-AI collaboration opportunities . Can machines and humans work together to improve the accuracy of humans in image classification ? This question remains largely unknown although , in other non-image domains , AMs have been found useful to humans in the downstream tasks [ 12 , 39 , 47 , 66 , 66 ] . Furthermore , it is also unknown whether any existing automatic evaluation scores may predict such effectiveness of AMs to humans , the target user of AMs . Most AMs were often only evaluated on ∗Corresponding authors . Work done when GN was at KAIST . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . confidence 20 % heatmaps 3 nearest neighbors GradCAM EP SOD input Yes No vs lorikeet ? A B C user in lorikeet AI ’ s top-1 predicted label : lorikeet groundtruth label : “ bee eater ” ( a ) Given an input image , its top-1 predicted label ( here , “ lorikeet ” ) and confidence score ( A ) , we asked the user to decide Yes or No whether the predicted label is accurate ( here , the correct answer is No ) —i.e . the accuracy of human-AI teams without visual explanations . We also compared this baseline with the treatments where one attribution map ( B ) or a set of three nearest neighbors ( C ) is also provided to the user ( in addition to the confidence score ) . Method ImageNet Stanford Dogs µ σ µ σ Confidence 72.44 8.25 61.71 11.39 GradCAM 72.58 8.11 60.56 9.27 EP 73.85 6.88 56.67 10.57 SOD 72.06 7.63 61.67 10.87 3-NN 76.08 5.86 57.20 10.58 ( b ) Users of 3-NN outperform all other users including those using attribution maps on ImageNet classification . However , on fine-grained dog classification ( Stanford Dogs ) , visual explanations ( GradCAM , EP , 3-NN ) surprisingly did not help but hurt the human performance . Results from testing humans on both real and adversarial images . Figure 1 : Overview of the methods compared in our user-study and its results . proxy evaluation metrics such as pointing game [ 78 ] , object localization [ 80 ] , or deletion [ 60 ] , which may not necessarily correlate with human-AI team accuracy on a downstream task . In this paper , we conduct the first user-study to measure AM effectiveness in assisting humans on the 120-class Stanford Dog [ 38 ] fine-grained classification and 1000-class ImageNet [ 65 ] classification , which is the task that most attribution methods were designed for . We asked 320 lay and 11 expert users to decide whether machine decisions are correct after observing an input image , top-1 classification outputs , and explanations ( Fig . 1a ) . This task captures two related quantities : ( 1 ) human accuracy on image categorization , here simplified to binary classification ; and ( 2 ) model debugging , i.e . understanding whether AI is correct or wrong . For a fuller understanding of the effectiveness of AMs , we tested users on both real and adversarial images [ 75 ] . Our main findings include:2 1 . AMs are , surprisingly , not more effective than nearest-neighbors ( here , 3-NN ) in improving human-AI team performance on both ImageNet and fine-grained dog classification ( Sec . 3.2 ) . 2 . On fine-grained dog classification , a harder task for a human-AI team than 1000-class ImageNet , presenting AMs to humans interestingly does not help but instead hurts the performance of human-AI teams , compared to AI alone without humans ( Fig . 2b ) . 3 . On adversarial Stanford Dog images which are very hard for humans to label , presenting confidence scores only to humans is the most effective method compared to all other explanations tested including AMs and 3-NN examples ( Sec . 3.2 ) . 4 . Despite being commonly used in the literature , some automatic evaluation metrics—here , Pointing Game [ 78 ] , localization errors [ 80 ] , and IoU [ 80 ] —correlate poorly with the actual human-AI team performance ( Sec . 3.5 ) . 5 . According to a study on 11 AI-expert users , 3-NN is significantly more useful than GradCAM [ 67 ] , a state-of-the-art attribution method for ImageNet classification ( Sec . 3.6 ) . 2 Methods . 2.1 Image classification tasks . To evaluate human-AI team performance on image classification , for each image , we also presented to a user : ( 1 ) an AI ’ s top-1 predicted label , ( 2 ) its confidence score , and , optionally ( 3 ) a visual explanation that is generated in attempt to explain the predicted label . The user was then asked to decide whether AI ’ s predicted labels are correct or not ( see Fig . 1a ) . We performed our experiments on two image datasets of ImageNet and Stanford Dogs , which are of increasing difficulty . 2Code and data are available at http : //github.com/anguyen8/effectiveness-attribution-maps ImageNet We tested human-AI teams on ImageNet [ 65 ] —an image classification task that most attribution methods were tested on or designed for [ 7 , 60 , 67 , 80 ] . ImageNet is a ∼1.3M-image dataset spanning across 1000 diverse categories from natural to man-made entities [ 65 ] . Stanford Dogs To test whether our findings generalize to a fine-grained classification task that is harder to human-AI teams than ImageNet , we repeated the user-study on the 120-class Stanford Dogs [ 38 ] dataset , a subset of ImageNet . This dataset is challenging due to the similarity between dog species and the large intra-class variation [ 38 ] . Compared to ImageNet , Stanford Dogs is expected to be harder to human-AI teams because lay users often have significantly less prior knowledge about fine-grained dog breeds compared to a wide variety of everyday objects in ImageNet . 2.2 Image classifiers . We took ResNet-34 [ 34 ] pretrained on ImageNet ( 73.31 % top-1 accuracy ) from torchvision [ 50 ] as the target classifier ( i.e. , the AI ) for both ImageNet and Stanford Dogs classification tasks because the 1000-class ImageNet that the model was pretrained on includes 120 Stanford Dogs classes . The visual explanations in this paper were generated to explain this model ’ s predicted labels . We chose ResNet-34 because ResNets were widely used in feature attribution research [ 7 , 28 , 45 , 60 , 67 ] . 2.3 Visual explanation methods . To understand the causal effect of adding humans in the loop , we compared the performance of a standard AI-only system ( i.e . no humans involved ) and human-AI teams . In each team , besides an input image and a corresponding top-1 label , humans are also presented with the corresponding confidence score from the classifier , and a visual explanation ( i.e . one heatmap or three 3-NN images in ; see examples from our user-study in Sec . A13 ) . AI-only A common way to automating the process of accepting or rejecting a predicted label is via confidence-score thresholding [ 14 ] . That is , a top-1 predicted label is accepted if its associated confidence score is ≥ T , a threshold . We found the optimal confidence threshold T ∗ that produces the highest accepting accuracy on the entire validation set by sweeping across threshold values ∈ { 0.05 , 0.10 , ... , 0.95 } , i.e . at a 0.05 increment . The optimal T ∗ values are 0.55 for ImageNet and 0.50 for Stanford Dogs ( details in Sec . A6 ) . Confidence scores only To understand the impact of visual explanations on human decisions , we also used a baseline where users are asked to make decisions given no explanations ( i.e . given only the input image , its top-1 predicted label and confidence score ) . To our best knowledge , this baseline has not been studied in computer vision , but has been shown useful in some non-image domains for improving human-AI team accuracy or user ’ s trust [ 12 , 79 ] . GradCAM and Extremal Perturbation ( EP ) We chose GradCAM [ 67 ] and Extremal Perturbation ( EP ) [ 28 ] as two representatives for state-of-the-art attribution methods ( Fig . 1a ) . Representing for the class of white-box , gradient-based methods [ 16 , 63 , 80 ] , GradCAM relies on gradients and the activation map at the last conv layer of ResNet-34 to compute a heatmap . GradCAM passed a weight-randomization sanity check [ 6 ] and often obtained competitive scores in proxy evaluation metrics ( see Table 1 in [ 28 ] and Table 1 & 2 in [ 25 ] ) . In contrast , EP is a representative for perturbation-based methods [ 7 , 20 , 29 ] . EP searches for a set of a % of input pixels that maximizes the target-class confidence score . We followed the authors ’ best hyperparameters—summing over four binary masks generated using a ∈ { 2.5 , 5 , 10 , 20 } and a Gaussian smoothing kernel with an std equal to 9 % of the shorter side of the image ( see the code [ 1 ] ) . We used the TorchRay package [ 5 ] to generate GradCAM and EP attribution maps . We initially also tested vanilla Gradient [ 70 ] , Integrated Gradient ( IG ) [ 74 ] , and SHAP [ 46 ] methods in a pilot study but discarded them from our study since their AMs are too noisy to be useful to users in a pilot study . Salient object detection ( SOD ) To assess the need for heatmaps to explain a specific classifer ’ s decisions , we also considered a classifier-agnostic heatmap baseline . That is , we used a pre-trained state-of-the-art salient-object detection ( SOD ) method called PoolNet [ 43 ] , which uses a ResNet-50 backbone pre-trained on ImageNet . PoolNet was trained to output a heatmap that highlights the salient object in an input image ( Fig . 3 ) — a process that does not take our image classifier into account . Thus , SOD serves as a strong saliency baseline for GradCAM and EP . 3-NN To further understand the pros and cons of AMs , we compared them to a representative , prototype-based explanation method ( e.g . [ 18 , 53 , 55 ] ) . That is , for a given input image x and a predicted label y ( e.g. , “ lorikeet ” ) , we show the top-3 nearest images to x by retrieving them from the same ImageNet training-set class y ( Fig . 1a ) . To compute the distance between two images , we used the L2 distance in the feature space of the last conv layer of ResNet-34 classifier ( i.e . layer4 per PyTorch definition [ 4 ] or conv5_x in He et al . [ 34 ] ) . We chose layer4 as our pilot study found it to be the most effective among the four main conv layers ( i.e . layer1 to layer4 ) of ResNet-34 . While k-NN has a wide spectrum of applications in machine learning and computer vision [ 68 ] , the effectiveness of human-AI collaboration using prototype-based explanations has been rarely studied [ 64 ] . To the best of our knowledge , we provided the first user study that evaluates the effectiveness of post-hoc , nearest-neighbor explanations ( here , 3-NN ) on human-AI team performance .

Summary: The paper investigates if current widely used evaluation metrics for saliency maps or explainable visual AI like pointing game, weakly supervised localization are trustable by conducting a very extensive user study. Some interesting conclusions and observations are given. In this sense, this is a good paper.