Visual correspondence-based explanations improve AI robustness and human-AI team accuracy
Authors: Mohammad Reza Taesiri
Giang Nguyen
Anh Totti Nguyen
Venue: NeurIPS 2022 Accept (2022)
Keywords: visual correspondence
nearest neighbors
optimal transport
explainable AI
xai
human AI team

Abstract: Explaining artificial intelligence (AI) predictions is increasingly important and even imperative in many high-stake applications where humans are the ultimate decision-makers. In this work, we propose two novel architectures of explainable image classifiers that first explain, and then predict (as opposed to post-hoc explanation methods). Our models first rank the training-set images by their distance with the query in an image-level deep feature space. And then, we re-rank the top-50 shortlisted candidates using patch-wise similarity of 5 highest-similarity pairs of patches between the query and every candidate. On ImageNet, our models improve (by 1-4 points) the out-of-distribution accuracy on several datasets including Adversarial Patch and ImageNet-R while performing marginally worse (by 1-2 points) on ImageNet to the baselines (ResNet-50 pre-trained ImageNet). A consistent trend is observed on CUB. Via a large-scale, human study (~60 users per method per dataset) on ImageNet and CUB, we find our proposed correspondence-based explanations led to human-alone image classification accuracy and human-AI team accuracy that are consistently better than those of k-NN. Our correspondence-based explanations help users better correctly reject AI's wrong decisions than all other tested methods.
Interestingly, for the first time, we show that it is possible to achieve complementary human-AI team accuracy (i.e. that is higher than either AI-alone or human-alone), in both image classification tasks.

Visual correspondence-based explanations improve AI robustness and human-Al team accuracy
Giang Nguyen aguyonggiangbbhn@@@al.cccm com
Mohammad Reza Taesiri ataesiri@gmall com
Anh Nguyen anh nahhgna@@m...... com
Auburn University
Abstract
Explaining artificial intelligence (AI) predictions is increasingly important and even imperative in many high-stakes applications where humans are the ultimate decision makers. In this work, we propose two novel architectures of self-interpretable image classifiers that first explain, and then predict (as opposed to post-hoc explanations) by harnessing the visual correspondences between a query image and exemplars. out-of-distribution (OOD)
Our models consistently improve (+1 to +4 points) datasets while performing marginally worse (-1 to -2 points) on in-distribution tests than ResNet-50 and a k-nearest neighbor classifier (kNN). Via a large-scale, human study on ImageNet and CUB, our correspondence-baseee explanations are found to be more useful to users than kNN explanations. Our explanations help users more accurately reject Al's wrong decisions than all other tested methods. Interestingly, for the first time, we show that it is possible to achieve complementary human-Al Al team accuracy (i.e., that is higher than either Al-alone or human-alone), in ImageNet and CUB image classification tasks.
1 Introduction
Comparing the input image with training-set set exemplars is the backbone for many applications, such as face identification [29], bird identification [17, 79], and image search [79]. This non-parametric approach may improve classification accuracy on out-of-distribution (OOD) data [29, 75, 79, 57] explanations [17, 52, 53, 66, 41] that provide insights into the
and enables a class of prototype-base decision making of Artificial Intelligence (AI) systems. Interestingly. prototype- based explanations are more effective in improving human classification accuracy [55, 24, 42] than attribution maps a common eXplainable AI (XAI) technique in computer vision. Yet, it remains open question how to make prototype-basee based XAI classifiers (1) accurate on in-distribution and OOD data and (2) improve human decisions. For example, in face identification, Als can be confused by partially occluded, never-seen faces and are unable to explain their decisions to users, causing numerous people falsely arrested [5, 7, 3, 6] or wrongly denied unemployment benefits [1] by the law enforcement.
To address the above questions, we propose two interpretable [61]. (i.e.. whw--eepli--ttee--eeeeeee image classifiers that perform three common steps: (1) rank the training-Son set images based on their distances to the input using image-level features; (2) re-rank the top-50 shortlisted candidates by their
"quual contribution. Listing order is random. GN led the development of EMD-Cor and human studies on Gorilla. MRT led the development of CHM-Cor,, pilot studies on HuggingFace, and the analysis of numan-study data from Gorilla. AN advised the project. MRT's s work was done before he joined University of Alberta.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

groundrathh
(a) Explanations for KNN's parachute docision (lop) and CHM--N (bollom)
Explonations for CHM--Cer's bax decision
Figure I: The ibex image misclassified into parachute due to similarity (clouds in blue sky) to parachute scenes (a). In contrast, CHM-Corr correctly labels the input as it matches ibex images mostly using the animal's features, discarding the background information (b).
groundrruth:
(a) Explanations for &NN's toaster decision (sop) and EMD NN (botton)
(b) Explamations for EMD Cor''s ben decision
Figure 2: Operating the image-level visual similarity, kNN incorrectly labels the input toaster due to the adversarial toaster patch (a). EMD-Corr instead ignores the adversarial patch and only uses the head and neck patches of the hen to make decisions (b).
patch-wise correspondences w.r.t. the input [51, 29]; and then (3) take the dominant class among the top-20 candidates the predicted label. That is, our classifiers base their decisions set of support image-patch pairs, which also serve as explanations to users (Figs. 1b and 2b). Our main findings include:
On ImageNet, simple i k-nearest-neighbow classifier (kNN) based ResNet-50 features slightly but consistently outperforms ResNet-50 on many OOD datasets (Sec. 3.1). This is further improved after a re-ranking step based patch-wise similarity (Sec. 3.2).
Via large-scale human study, find visual correspondence-batt explanations to improve Al-assisted, human-alone accuracy and human-Al team accuracy on ImageNet and CUB over the baseline kNN explanations (Sec. 3.4).
Having interpretable AIs label images that they confident and humans label the rest yields better accuracy than letting AIs or humans alone label all images (Sec. 3.5 and Appendix M).
To the best of our knowledge, our work is the first to demonstrate the utility of correspondence-base explanations to users on ImageNet [63] and CUB [71] classification tasks.
2 Methods
2.1 Datasets
We test our ImageNet classifiers on the original 50K-image ILSVRC 2012 ImageNet validation set (i.e., in-distribution data) and four common OOD benchmarks below.
ImageNet-R [35] contains 30K images in 200 ImageNet categories, mostly artworks ranging from cartoons to video-game renditions.
ImageNet-Sketch [72] consists of 50,889 black-and-white sketches of all 1,000 ImageNet classes.
DAmageNet [18] consists of 50K ImageNet validation-set images that contain universal, adversarial perturbations for fooling classifiers.
Adversarial Patch [15] are 50K ImageNet validation-set images that modified contain adversarial patch that aims to cause ResNet-50 [31] into labeling every image toaster (see Fig. 2).
Code and models available at https: http:///ttuub.co/mngnrgggeeu com wwwom//agnnnn/////e cow.suaal-corrrsspdeeeeeeeeeeeee

---

Using the implementation by [2], we generate this dataset, which causes ResNet-50 accuracy to drop from 76.13% 55.04% (Table 1). See Appendix A.5 for how download and generate this dataset.
CUB-200-2011 [71] (hereafter, CUB) is a fine-grained, bird-image classification task chosen to complement ImageNet. CUB contains 11,788 images (5,994/5,794 for train/test) of 200 bird species.
2.2 Classifiers
We harness the same ResNet-30 layer4 backbone [8] as the main feature extractor for all four main classifiers, including our two interpretable models. Therefore, to test the effectiveness of our models, we compare them with (1) vanilla ResNet-50 classifier; and (2) KNN classifier that uses the same pretrained layer4 features. We report the top-1 accuracy of all classifiers in Table
ResNet-50 For experiments on ImageNet and its four OOD benchmarks, we use the ImageNet- trained ResNet-50 from Visco [8] (top-1 accuracy: 76.13%)
For CUB, we take the ResNet-50 pretrained on iNaturalist [70] from [53] (hereafter, iNaturalist ResNet) and retrain only the last 200-uttput classification layer (right after avgpool) to create competitive, baseline ResNet-50 classifier for CUB (top-1 accuracy: 85.83%). See Appendix A. for finetuning details.
KNN We implement a vanilla kNN classifier that operates at the avgpool the last convolutional layer of ResNet-50. That is, given query image Q, we sort all training-se set images {G.) based on their distance DQQ, G.), which the cosine distance between the two corresponding image features f(Q) and ((G.) E 22048 where f(). outputs the avgpool feature of layer4 (see code) of ResNet-50.
The predicted label of Q the dominant class among the top-k nearest neighbors. We choose & 20 as it performs the best among the tested values of k {10, 20, 50, 100}.
top-S
feature
Commattinn
Trineport
CNN
(a) EMD-Corr: First compute patch-wise similarity, and then find correspondences via solving EMD [29, 79].
corraspondences
binarization
CHM
coRV Feetsra
Combettes
Aeeuor welphing mablh
Beellrty
(b) CHM-Cor: First find correspondences via CHM [51], and then compute patch-wise similarity.
Figure 3: EMD-Corr and CHM-Corr both re-rank kNN's top-50 candidates using the patch-wise similarity between the query and each candidate over the top-5 pairs of patches that are the most important and the most similar (i.e. highest EMD flows in EMD-Corr and highest cosine similarity in CHM-Corr).
EMD-Corr As KNN compares images using only image-level features, it lacks the capability of paying attention to fine details in images. Therefore, propose EMD-Corr, visual correspondence- based classifier that (1) re-ranks the top-N (here, = 50) candidates kNN using their Earth Mover's Distance (EMD) with the query in patch embedding space (see Fig. 3a); and (2), similarly to kNN, takes the dominant class among the re-ranked top-20 as the predicted label.
That is, our PyTorch implementation is the same that in 179, 29] except for three key differences. First, using layer4 features (7x7 x2048), we divide an image into 49 patches, whose embeddings 20-18 Second, for interpretability, in re-ranking, only use patch-wise EMD instead of

---

linear combination of image-level cosine distance and patch-level EMD as in [9], which makes it more opaque how a specific image patch contributes re-ranking. Third, while the original deep patch-wise EMD [79, 29] between two images (see Eq. 3) is defined as the sum over the weighted cosine distances of all 49 x 49 patch pairs, we only use I. = 5 pairs as explanations and therefore, only sum over the corresponding 5 flow weights returned by Sinkhorn optimization [21].
We find N 50 to perform the best among N {50, 100, 2003. We choose
5, which is also the most common in nearest-neighhor visualizations [47, 45]. In preliminary experiments, we find L {9. 16, 25} to yield so dense correspondence visualizations that hurt user interpretation and L = 3 to under-inform users. See Appendix A.3 for more description of EMD-Corr.
CHM-Corr EMD-Cor first measures the pair-wise wise cosine distances for all 49 pairs of patches, and then computes the EMD flow weights for these pairs (Fig. 3a). To leverage the recent impressive end-to-end correspondence methods [51, 60, 48], we also propose CHM-Corr (Fig. 3b), a visual correspondence-hate classifier that operates in the opposite manner to EMD-Corr. That is, first, we divide the query image Q into 7x7 non-overlapping patches (i.e. as in EMD-Corr) and find one corresponding patch in G, for each of the 49 patches of Q using a state-of-the-ar correspondence method (here, CHM [51]). Second, for the query Q, we generate a cross-correlation [29]. i.e. a heatmap of cosine similarity scores between the layer4 embeddings of the patches of the query Q and the sange--mbedding (avgpool after layer4) of each training- set image G, Third, we binarize the heatmap (using the optimal threshold T = 0.55 found on a held-out training subset) to identify a set of the most important patches in Q and compute the cosine similarity between each such patch and the corresponding patch in G; (i.e., following the CHM correspondence mappings). Finally, the similarity score D(Q, G,) CHM-Corr is the sum over the I. patch pairs of the highest cosine similarity across Q and G..
(CC) map (Fig. 3)
After testing NC-Net [60], ANC-Net [48], and CHM [51] in our classifier, we choose CHM has the fastest runtime and the best accuracy. Unlike ResNet-50 [31], which operates at the x24 resolution, CHM uses a ResNet-101 backbone that expects a pair of x40x240 images. Therefore, in pre pre-processing we resize and center-crop crop each original Image sample differently according to the input sizes of Re and CHM. See Appendix A.4 for more description of CHM-Corr.
CHM-Cor++ classifier based on five groundtruth keypoints of birds In EMD-Corr and CHM- Corr, use CC to infer the importance weights of patches. To understand the effectiveness of CC in weighting patches, on CUB, we compare our EMD-Corr and CHM-Corr to CHM-Cort+, a CHM-Corr variant where we That is, for each CUB image, instead of taking the five CC-derived important patches (Fig. 3b), we use at most five patches that correspond to set of five pre-defined keypoints (beak, neck, right wing, right feet, tail), each representing a common body part according to bird identification guides [25] for omithollgiiss. From the five patches in the query image, we then use CHM to find five corresponding patches in a training-set image, and take the sum of five cosine similarities as the total patch-wise similarity between two images in re-ranking.
set of five human-defined important patches instead of those inferred by CC.
A query image may have < important patches some keypoint is occluded. That is, evaluating CHM-Corr+ alone provides an estimate of how hard bird identification CUB is if the model harnesses five well-known bird features.
2.3 User-study design
The interpretable classifiers (Sec. 2.2) are not only capable of classifying images but also generating explanations, which may inform and improve users' decision-making [55]. Here, design a large- scale study to understand the effectiveness of explanations in two human-Al interaction models in classification (see Fig. 4): Model 1: Users make all the decisions after observing the input, AI decisions, and explanations (Fig. 4a). Model 2: AIS make decisions on only inputs that they are the most confident, leaving the rest for users to label. That is, model 2 (Fig. 4b-c) is practical scenario where we offload most inputs to AIs while users only handle the harder cases.
Like [55], we show each user: (1) a query image; (2) Al top-1 label and confidence score; and (3) an explanation (here, available in KNN, EMD-Corr, CHM-Corr, and CHM-Corr+, but not ResNet-50). We ask users to decide Y/N whether the top-1 label is correct (example screen in Fig. A7).

---

High-corfsinnce
ibex
(+) All inages
Q: parachute
Al
95% ibox
Yes Muuto scoopt)
43% parachute
Yes
(c) Low confidence images
Q: parachute
Yes
Al
parachute
vs.
No
explanation
Al
explanation
Model 1: Human-only
Model 2: Human-Al team decision makers
(a) for all images, users decide (Yes/No) whether the AI's predicted label is correct given the input image, Al top-1 label and confidence score, and explanations. In model 2, Al decisions are automatically accepted if AI is highly confident (b) Otherwise, humans will make decisions
Figure 4: Two human-Al interaction models. In model
(c) in the same fashion as the model 1.
2.3.1 Explanation methods
We test the explanations of four main classifiers: ResNet-500, kNN, EMD-Com, and CHM-Com. Additionally, we test two ablated versions (i.e., EMD-NN and CHM-NN) of the explanations of EMD-Com and CHM-Cor.. In total, we test 6 explanation methods (see examples in Appendix D).
ResNet-50 a representative black-box classifier, which only outputs top-1 label and confidence score (i.e., no explanations).
From the top-20 nearest neighbors (as = 20 in our kNN), show the first five images that are from the predicted class (example in Fig. la). In some cases where the predicted class has only M < 5 exemplars in the top-20, we still show only those M images (see Fig. A46). That is, we only show at most five neighbors following prior works [55, 67, 49, 65] that reported utility of such few-image explanations. We find explanations consisting of 2 10 images such as those of ProtoPNet [17] are hard to interpret for users [42]. Note that our kNN explanations consist of five support images for each decision of kNN (described in Sec. 2.2) as opposed the post-hoc nearest examples in [55], which do not reflect a classifier's 's decisions.
KNN explanations
EMD-Corr and CHM-Corr explanations As EMD-Corr and CHM-Corr re-rank the top-50 candidates (shortlisted by kNN) and take the dominant class among the resultant top-20 as the predicted label, we show the five nearest neighbors from the predicted class in kNN explanations. Instead of showing only five post-reranking neighbors, we also annotate, in each image, all five patches (example in Fig. lb and Fig. 2b) that contribute to the patch-wise re-ranking (Sec. 2.2).
EMD-NN and CHM-NN To understand the effects showing correspondences in the explanations to EMD-Corr and CHM-Corr users, we also test ablated version where we show exactly the same explanations but without the patch annotations (bottom panels in Fig. la and Fig. 2a).
While ResNet-50's confidence score is the top-1 1 output softmax probability. the confidence KNN, EMD-Corr, and CHM-Corr is the count of the predicted-class examples among the top 1 = 20. In human studies, we display this confidence in percentage (e.g. 10% instead of 2/20; Fig. A46) to consistent with the confidence score of ResNet-50.
Confidence scores
2.3.2 ImageNet and CUB datasets
For XAI evaluation, we run human studies, one on ImageNet and one on CUB, For ImageNet, we use ImageNet-Real Real. [14] labels in attempt to minimize the confounders of the human evaluation as ImageNet labels are sometimes misleading and inaccurate to users [55].
Nearest-neighbor images To generate the nearest-neighhor explanations for KNN, EMD-Cor, and CHM-Cor, we search for neighbors in the entire training set of ImageNet CUB (no filtering).
Query images In attempt to ensure the quality of the query images that we ask users to label, from 50K-image ImageNet validation set, we discard images that: (a) do not have ImageNet-Reall [14]
5

---

label; (b) are grayscale or low reworustotionn (i.e., either width or height < 224 px) as in [55]; (c) have duplicates in the ImageNet training set (see Appendix L), resulting in 44.424 images available for sampling for the study. In CUB, we sample from the entire 5,794-image test set and apply no filters.
2.3.3 Training, Validation, and Test phases
From the set of query images (Sec. 2.3.2), we sample images for three phases in a user study; Training, validation, and test. Following [55], we first introduce participants to the task and provide them training examples. Then, each user is given a validation job (10 trials for ImageNet and 5 for CUB), where they must score 100% in order to be invited to our 30-trial test phase. Otherwise, they will be rejected and unpaid. Among the 10 validation trials for ImageNet, are correctly-labeless and 5 are misclassified by AIs. This ratio is 3/2 for CUB validation (examples in Appendix E.2).
Right before each trial, we describe the Al's top-1 label to users by showing them training-set images and I-sentence WordNet description for each ImageNet class. For CUB classes, we show 6 representative images (instead of 3) for users to better recognize the characteristics each bird (see Fig, A6).
Sampling For every classifier, we randomly sample 300 correctly- and 300 insorrectly-prediice images together with their corresponding explanations for the test trials. Over all 6 explanation methods, we have 2 datasets x 600 images x 6 methods = 7,200 test images in total.
2.3.4 Participants
We host human studies on Gorilla [11] and recruit lay participants who are native English speakers worldwide via Prolific [56] at pay rate of USD 13.5 We have 360 and 355 users who successfully passed our validation test for ImageNet and CUB datasets, respectively. We remove low-quality, bottom-outlier submissions, i.c., who score S 0.55 (near-random accuracy). resulting in 354 and 355 submissions for ImageNet and CUB, respectively. In each dataset, every explanation method is tested on 60 users and each pair of (query, explanation) is seen by almost 3 users (details in Table 2).
3 Experimental Results
3.1 ImageNet KNN classifiers improve upon ResNet-50 on out-of-distribution datasets
Despite impressive test-set performance, ImageNet-trained convolutional neural networks (CNNs) may fail to generalize to natural OOD data [72] or inputs specifically crafted fool them [54, 18, 15]. It is unknown whether prototype-based classifiers can leverage the known exemplars (i.e. support images) to generalize better to unseen, rare inputs. To test this question, here, we compare kNN with the baseline ResNet-50 classifier (both described in Sec. 2) on ImageNet and related OOD datasets.
On ImageNet and ImageNet-Real. kNN performs slightly worse than ResNet-50 by 1.36 and -0.99 points, respectively (Table 1). Yet, interestingly, on all four OOD datasets, kNN consistently outperforms ResNet-50. Notably, kNN improves upon ResNet-50 by +1.66 and +4.26 points on DAmageNet and Adversarial Patch. That is, while ResNet-50 and kNN share the exact same backbone, the kNN's s process of comparing the input image against the training-set examples prove to be beneficial for generalizing to OOD inputs. Intuitively, our results suggest that it is useful to "look back" at the training-set exemplars to decide a label for hard, long-tail or OOD images.
Consistently, using the same CUB-fineuune backbone, kNN is only marginally worse than ResNet-50 on CUB (85.46% vs. 85.83%; Table 1).
3.2 Visual correspondence-based explanations improve kNN robustness further
Recent work found that re-ranking kNN's shortlisted candidates using the patch-wise similarity between the query and training set examples can further improve classification accuracy on OOD data for some image matching tasks [29, 79, 75] such as face identification [29]. Furthermore, petch- comparison is also useful in prototype- based bird classifiers [17, 22). Inspired by these prior successes and the fact that EMD-Corr and CHM-Corr base the patch-wise similarity of two images on only patch pairs instead of all 49 x9 2,401 pairs as in [29, 79, 75), here we test whether our two proposed re-rankers are able to improve the test-set accuracy and robustness over KNN.

---

Table 1: Top-1 accuracy (%). ResNet-50 models' classification layer is fine-tuned on a specified training set in (b). All other classifiers are non-parametric, hearest-neightes models based on pretrained ResNet-50 features (a) and retrieve neighbors from the training set (b) during testing.
EMD-CorT & CHM-Corr outperform ResNet-50 models on all OOD datasets (e.g. +4.39 on
Adversarial Patch) and slightly underperform in-distribution sets (e.g. -0.72 ImageNet-Real).
set
Features (a)
Training set (b) ResNet-50 KNN
EMD-Com
CHM-Con
CII--CorT+
ImageNet [63]
ImageNet
ImageNet
76.13
74.77
(-1.2()
(-1.3)
ImageNet-Real [14]
ImageNet
ImageNet
83.04
82.05
82.32 (-0.72)
81.97 (-1.07)
ImageNet-R
ImugeNet
ImageNet
36.17
36.18
(7.75
37.62 (+1.45)
n/a
ImageNet Sketch [72]
ImageNet
ImageNet
24.09
24.72
25.36 (+1.27)
25.61 (+1.52)
n/a
DemageNet [18]
ImageNet
ImageNet
5.93
7.59
8.16 (+2.23)
8.10 (+2.17)
n/a
Adversarial Patch [15]
ImageNet
ImageNet
55.04
59.30
59,43 (+4.39)
59.86 (+4.82)
n/a
CUB [71]
ImageNet
CUB
n/a
54.72
60.29
53.65
49.63
CUB [71]
iNaturalist
[70]
CUB
85.83
85.46
84.98 (-0.85)
83.27 (-2.56)
81.54
Experiment We run EMD-Corr and CHM-Corr on all datasets and compare their results with that of KNN (Table 1). Both methods (described in Sec. 2) re-rank the top N = 50 shortlisted candidates returned by KNN and then take the dominant class in the top-k (where k = 20) as the predicted label.
ImageNet results Interestingly, despite using only 5 pairs of patches to compute image similarity for re-ranking. both classifiers consistently improve upon kNN further, especially on all OOD datasets. Overall, EMD-Corr and CHM-Corr outperform kNN and ResNet-50 baselines from +1.27 to +4.82 points (Table 1). Intuitively, in some hard cases where the main object is small, the two Corr classifiers ignore irrelevant patches (e.g. the sky in ibex images; Fig. 1) and only use the five most relevant patches to make decisions. Similarly, on Adversarial Patch, relying on a few key patches while ignoring adversarial patches enables our classifiers outperform baselines (Fig. 2). See Appendix I for many qualitative examples comparing Corr and kNN predictions.
CUB results Interestingly, using the same Image imaeeendt-retrainee backbones, EMD-Corr outperforms KNN by an absolute +5.57 points when tested on CUB (60.29% vs. 52%: Table 1). However, this difference vanishes when using CUB-pretraine's backbones (Table 1; 84.98% vs. 85.46%).
Our CUB and ImageNet results are consistent and together reveal a trend: On i.i.d test sets, Corr models perform on par with KNN; however, on OOD images, they consistently outperform kNN, highlighting the benefits of match-wise comparison.
3.3 Corr classifiers leverage five patches that are more important than five bird keypoints
EMD-Cor and CHM-Corr harness five patches per image for computing a patch-wise similarity score for pair of images (Sec. 2.2). As these five patches are automatically inferred by cross-correlation (Fig. 3), is interesting to understand further whether replacing these five patches by five user-defined patches in [71] would improve classification accuracy.
Experiment Since there no keypoints provided for ImageNet, we test the importance of the five key patches chosen by Corr methods on CUB because CUB provides omithologist-deeinee annotations for each bird image. That is, create baseline CHM-Cort
which is the same as CHM-Corr, except that we use five important patches that correspond to five keypoints in a bird beak, belly, tail, right wing, and right foot as described in See. 2.2. We also test CHM-Corr+ sweeping across the number of keypoints E (5, 10, 15).
Results On CUB, CHM-Corr outperforms CHM-Corr+ despite the fact that the baseline method leverages live human-defined bird keypoints (Table 1; 83.27% vs. 81.51%) %) while CHM-Corr may also use background patches. Interestingly, when increasing the number of keypoints to 10 and 15, the accuracy of CHM-Corr+ is still lower than that of CHM-Corr (i.e., from 81.51 to 82.34% and 82.27%, respectively). That is, 15 keypoints may correspond to & 15 different patches per image (15 each keypoint lies in a unique, non-overlapping patch among all the 49 patches per image).
Our results show strong evidence that the five key patches inferred by CC used in EMD- and CHM- Corr do not necessarily cover the birds but are more important than expert-defined bird keypoints. Qualitative comparisons between CHM-Cor and CHM-Cores predictions are in Appendix J.
7

---

3.4
On ImageNet-Real ReaL, correspondence- explanations are more useful to users than
kNN explanations
Given that EMD-Corr and CHM-Corr classifiers outperform kNN classifiers on OOD datasets (Sec. 3.2), it is interesting to test how their respective explanations help humans perform classification on ImageNet. Furthermore, in image classification, kNN explanations were found to be more useful to humans than saliency maps [55].
Experiment We perform a human study to assess the ImageNet-Real. classification accuracy of users of each classifier (described in Sec. 2.2) when they are provided with classifier's predictions and explanations. That is, we measure the Al-assisted classification accuracy of users following human- AI interaction model 1 (Fig. 4a). We compare the accuracy between user groups of four classifiers ResNet-50, KNN,
Accuracy
6.
EMD-Comr, and CHM-Corr (described in Sec. 2.3.1).
Additionally, to thoroughly assess the impact of showing the correspon- dence boxes compared to showing only nearest neighbor images (e.g..
Bedtte:30
KAW
EMD kt
BMD car
CHM ABN
crm--oo
CHM-Corr vs. CHM-NN in Fig. 1), we test two more user groups of EMD-NN and CHM-NN, i.e. the same explanations those of the Corr classifiers but with the correspondence boxes hidden.
(a) ImageNet
Results First, the mean accuracy of kNN users is consistently lower than that of the other models' users (e.g.. 75.76% vs. 78.87% of EMD- Corr; Table 2). The EMD-Corr improvement over kNN is statistically significant (p < 0.01 via Mann-Whitme U test; Fig. 5a)
1.0
Accuracy
0.7
Second, interestingly, we find the differences between EMD- CHM- Corr and their respective baselines small and not statistically signif- icant (Fig. 5a). That is, on ImageNet- ReaL., quantitatively, showing the correspondence boxes on top of nearest neighbors is not more useful to users. Third, surprisingly. the users of ResNet-50 (mean accuracy of 81.56%; Table 2) outperform all other methods' users, suggesting that on ImageNet, a task of many familiar classes to users, ante-hoc explanations hurt user accuracy rather than help. Note that in Nguyen et al. [55], post-hoc kNN explanations were found useful to humans compared to not showing any explanations. Yet, here, each classifier's users are provided with different set of images and AI decisions, which can also influence the user accuracy. When ResNet-50 wrong, their users are substantially better in detecting such misclassifications compared to other models' users (Fig. Alla).
Reatet-50
MaN
BSD-WW -W---N
PMD Corr
ow Coi
(b) CUB
Figure 5: Mann-Whimn
U test of the user accuracy scores of 6 methods.' p- value < 0.05. " = p-value < 0.01. p-value 0.001.
3.5
On CUB fine-grained bird classification, correspondence- explanations are the most useful to users, helping them to more accurately reject AI misclassifications
To assess whether the findings on ImageNet- ReaL in Sec. 3.4 generalize to a fine-gras grained classification task, we repeat the user study on CUB which is considered much harder to lay users than ImageNet.
Results Interestingly, we find EMD-Corr and CHM-Corr users consistently outperform ResNet-50, KNN, EMD-NN, and CHM-NN users (Table 2). The differences between EMDD-Crr (or CHM-Cor) and every other baseline are statistically significant (p 0.05 via Mann-W Whinney U test; Fig. 5b). That is, on CUB, the visual correspondence boxes help users make more accurate decisions compared to (a) having no explanations at all (ResNet-50); (b) showing nearest neighbors sorted by image similarity only, not patch correspondences (Fig. la; KNN); and (c) having patch-wise correspondence neighbors but not displaying the boxes (Fig. A39; CHM-NN and EMD-NN).
Corr explanations help users reject AI misclassifications while kNN is poorly trust-calibrated In attempt to understand why the two Corr classifiers help users the most, find that EMD-Corr and CHM-Corr users reject Al predictions at the highest rates (32.70% and 33.73%; Table A6) while KNN users reject the least (18.47%).
This might have led to the substantially higher accuracy of CHM-Corr users, compared to all other models' user groups, when Al predictions are wrong (Fig. Allb; e.g., 53.45% of CHM-Coir vs.
8

---

Table 2: Human-only accuracy (%)
Table 3: Al-only and Human- Al team accuracy (%)
ImageNet-Real.
CUB
ImageNet-Real
CUB
Method
Method
Users
Accuracy
Users
Accuracy
Al-only
Human-Al
Al-only
Human-Al
ResNer-50
60
81.56 t 5.54
60
65.50
7.46
ResNer-50
86.11
88.63 (+2.52)
87.38
87.45 (+0.07)
RNN
59
75.76
8.55
59
64.75
7.14
KNN
85.95
(+1.22
87.40
86.66 (-0.74)
EMD-Cor
59
78.87
6.57
58
67.64
7.44
EMD-Corr
85.91
88.02 (12.
86.88
86.86 (-0.02)
CHIM-Com
59
71.23
7,56
59
69.72
9.08
CHIM-Cor
83.36
187.89 (+2.53)
83.48
86.25 (+0.77)
EMD-NN
57
77.72
1
8.27
59
64.12 1 7.07
mean
85.83
(+2.11)
86.78
86.80 (+0.02)
CHM-NN
60
77.56 6.91
60
65.72 = 8.14
41.22% of ResNet-50). That is, CHM-Corr users correctly reject 53.45% of the images that the CHM-Cor classifier mislabels. In contrast, kNN users reject the least, only 33.22% incorrect predictions (Fig. A1 Al)... kNN explanations tend to fool users into trusting the kNN's wrong decisions (Fig. 6)-the accuracy of KNN users is 33.22%, much lower than the 41.22% of ResNet-50 users who observe no explanations. On ImageNet (Fig. Al1), kNN is also poorly "trust-calibrated" [69, 74].
groundirath:
Sayornia
(a) KNN (lop) and CHIM-NN (bothom) explonation
(b) CHM-CCo visual wwerespumeenceebase based explanations
Figure 6: A Sayornis bird image is misclassified into Olive Sided Flycatcher by both kNN and CHM-Corr models. Yet, all 3/3 CHM-Corr users correctly rejected the Al prediction while 4/4 KNN users wrongly accepted. CHM-Corr explanations (b) show users more diverse samples and more evidence against the Al's decision. More similar examples are in Appendix H.I.
We hypothesize that KNN explanations tend fool users show images that are image-wise similar to the query (regardless of whether the kNN prediction is correct not) while EMD-Corr and CHM-Corr re-rank the images based patch-wise similarity. Furthermore, we find the images in kNN explanations are also less diverse than those in Corr explanations in both LPIPS [77] and MS-SSIM [73] (Appendix H.2). Corr explanations tend to include more diverse images (Fig. 6a; top vs. bottom) and provide users with more contrastive evidence in order to reject Al's incorrect predictions.
their nearest neighbors, by design,
Additionally, we also hypothesize that users are less confident about Al's decisions (and thus reject more) when Corr explanations show some background and uninformative patches used in the matching process (e.g., the 1tt and 4th image in Fig. 6b). Yet, such boxes are not available kNN explanations.
When Corr explanations allow for more disagreement between AI and users, humans also tend incorrectly reject Al's correct predictions more often (Fig. Allb; Corr users are the least accurate among 6 methods when the Al is correct). EMD-Corr and CHM-Corr users score ~4 points below ResNet-50 users (84.94% and 85, 99% vs. 89.87% of ResNet-50). When most users reject Al's correct predictions, we observe that some discriminative features (e.g., the belly stripes of the Field Sparrow; Fig. A35c) are often occluded in the query, leading to human-Al Al disagreement.
Related Work
Patch-wise similarity Calculating patch-wise similarity, either intra-image [23] or inter-image [29, 79, 75], has been useful in many tasks as the comparison enables machines to attend to fine- grained details and compute more accurate decisions. Our EMD-Corr harnesses similar approach to that in [29, 79]; which, however, was not tested on ImageNet classification as in our work. Furthermore, we only compute the total patch-wise similarity over the top-5 patch pairs between the query and each exemplar instead of all pairs in [29, 79]. Compared to recent patch-wic wise similarity works that use either cross-attention in ViTs [23] or EMD [79, 75, 29, 43], our work is the first to perform human evaluation of the correspondence-base explanations.

---

Prototype-based XAI methods Our work is motivated by the recent finding that exemplar based explanations are more effective than heatmap-based explanations in improving human classification accuracy [55, 38, 42, 24). However, showing an entire image as exemplar without any further localization may be confusing as it unknown which parts of the image the Al paying attention to [55, 38]. Our EMDD-Cor and CHM-Corr present novel combination of heatmap-based and prototype-based XAI approaches. None of the prior prototype-base patch level [17, 53, 79, 22] (see Table 3 in [22]) were tested on humans yet. Also, in preliminary tests, we find their explanation formats too dense (i.e., showing over 10 prototypes [17], 9 correspondence pairs per image [22], an entire prototype tree humans [52, 53)) to be useful for lay users.
XAI methods that operate at the
Another major difference is that our Corr classifiers are nonparametric, allowing the training set be adjusted or swapped with any external knowledgebase for debugging purposes. In contrast, recent prototype-basee classifiers [17, 53, 79, 22] are parametric, using a set of learned prototypes and thus may not perform well on OOD datasets as EMDD-Cor and CHM-Corr.
Post-hoc prototype-based explanations Some prototype-based methods are post-hoe [41, 55, 20], i.e., generating explanations to explain a decision after-the-fact, which could be highly unfaithful [61, 62]. Instead, our approach inherently interpretable [62], i.e., retrieving the patches first, and then using them to make classification decisions. While our binary classification task is adopted from [55], our study compares 4 different classifiers while Nguyen et al. [55] instead tested a single classifier with multiple post-hoc explanations.
Human studies Our study has 709 users in total, i.e. ~60 users per method per dataset, which is substantially larger than that in most prior works. That is, ~30 and 40 users per method participated in [55] and in [49], respectively while Adebayo al. [9] had 54 persons in total for the entire study of multiple methods.
Human-Al teaming Human-AI collaboration is becoming more essential in the modern AI era [32]. A large body of prior works has investigated such collaboration in other domains (e.g.. NLP [12, 76]. healthcare [16] and others [33, 16, 78, 19]); however, only few works investigated human-AI collaboration in the image classification setting [55, 42, 24].
Some prior works predict when to defer the decision decision-making to humans [58, 40J. However, by simply offloading some inputs to humans based on confidence scores, we achieve complementary human-Al team performance in both ImageNet and CUB. Previous works [12, 64] found that algorithmic explanations benefit human sseision-making in general, but did not find XAI methods to yield team complementary performance [32], which we report in this work.
5 Discussion and Conclusion
Limitations Due to the limited amount of time and expensive cost of computation related to EMD-Com CHM-Corr, we did not experiment on a wider range of OOD datasets (e.g.. adversarial poses [10]. We tested our methods on ImageNet-A [36], ObjectNet [13], and ImageNet-C [34] as well, but small scale of 5K-image sets (see Table A2). As using online crowdworkers for the XAI human evaluation, we share the same limitations with [55, 9, 42). That is, despite our best efforts to minimize biases, the human data quality can be improved in highly-controlled laboratory conditions like in [27]. Algorithm-win EMD-Corr and CHM-Corr are re-ranking methods and therefore run substantially slower than ResNet-50 (see Fig. A10 for speed comparison of all models).
Our work is the first attempt to: (1) study the effectiveness of patch- wise comparison in improving the robustness of deep image classifiers on ImageNet OOD benchmarks; (2) show the utility of visual correspondence-brt based explanations in helping users make more accurate amage-classificattee decisions; (3) achieve human-Al complementary team performance in the image domain.
Acknowledgement
The authors would like to thank Ken Stanley for the great idea of using correspondences as an explanation for image classification, leading to this work. We also thank Thang Pham, Peijie Chen, and Hai Phan for feedback and discussions of the earlier results. AN was supported by the NSF Grant No. 1850117 17 & 2145767, and donations from NaphCare Foundation & Adobe Research.
10

---

References
[1] The pandemic ogy review.
is testing the limits of face recognition
mit
technol-
cow/ wtt.:cccm/mm20/00 00011/09/2//006//0 panderiic-ummmployyet---- wow.rrmeeee-e--eeeeeeeee wtw.r/mmmatt-aaa--eeeeetttiiiiio (Accessed on 11/09/2021).
[2] Jaayess4aaaveersarddllachhhh Pytorch implementation of adversarial patch. htttps///githn
jhayea14/adveeraariall patch. (Accessed 05/18/2022).
[3] Wrongfully arrested man sues
detroit police following false facial-recognition match
the washington post. facial-recognition- www--arrreettaaasee (Accessed on 11/09/2021).
https: /wtp:///wwwww.. www.wwatttntssssttt.ccoo wwwpsh/nttaappssseeooo////aaa/aa
[4] M-nauta/prototree: Prototrees: Neural prototype trees for interpretable fine-grained image recognition, published at evpr2021. httpp///gttaaan.cco//a/aaaaataaa
(Accessed 06/14/2022). 16
[5] The new lawsuit that shows facial recognition is officially civil rights issue mit technology review.
http:://www.ccccoooogrreeeee www.attiio--aaa--aalllllll www.ddettoooccee....... (Accessed on
Sabber-willlams-feee 04/15/2021).
[6] Michigan man act.
wrongfully accused with facial
recognition
urges congress
https://www.t http://wwwweettotteeee www.dditeettteeeeee wwtps/oor///ww/////oeeeooo house- panee--pael--a casar--mccigaa. -an--annnaaaan www.enuwwooooouuooooooou. accused- aacial--rooognittiii.... 7948908002/. (Accessed on 11/09/2021).
[7] Flawed facial recognition leads to arrest and jail new jersey man new york times. https://www. nytimes
htt////1/////tthoooooe 2ttptc//ooogg/////eeeee /facialaa-aaea tataar/rerreooooeeoeeooaaaaaaaaa wtt.:iioiiiiiiiiitt-iiiiii jail.html. (Ac- cessed on 11/09/2021).
[8] vision master pytorch/vision.
https://githieii strchrision/model... (Accessed on 05/08/2022). 3, 16, 17
Adebayo, J., Muelly, M., Liccardi, I., and Kim, B. Debugging
model explanations. Advances
Neural Information Processing Systems, 53:700-712, 2020. 10
[10] Alcor, M. A., Li, Q.. Gong. Z., Wang. C., Mai, L., Ku, W.-.., and Nguyen, Strike (with) pose: Neural networks are easily fooled by strange poses familiar objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4845-4854, 2019. 10
[11] Anwyl-Irvine, A. I.., Massonnié, J., Flitton, A., Kirkham, N., and Evershed, J. K. Gorilla midst: An online behavioral experiment builder. Behavior research methods, 52(1) 30:88448877 2020.
[12] Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribciro, M. T., and Weld, D. Does the whole exceed its parts? effect explanations complementary team performance. In Proceedings of 2021 CHI Conference on Human Factors Computing Systems, pp. 1-16, 2021. 10,
[13] Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models, Advances neural information processing systems, 32, 2019. 10, 19
[14] Beyer, L., Hénaff, O. Kolesnikov, A.. Zhai, X., and Oord, A. d. Are we done with imagenet? arXiv preprint 0tt::2000600715, 2020. 5,7
[15] Brown, T. B., Mané, D., Roy, A., Abadi, M., and Gilmer, J. Adversarial patch. arXiv preprint arXiv: 10X2v1712.00665. 2017. 2, 6, 7,
[16] Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., and Elhadad, N. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pp. 1721 1730, 2015. 10
[17] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. K. This looks like that: deep learning for interpretable image recognition. Advances neural information processing systems, 32, 2019. 1, 10
[18] Chen, S., Hc, Z., Sun, C., Yang, J., and Huang, X. Universal adversarial attack on attention and the resulting dataset damagenet. IEEE Tronsactions on Pattern Analysis and Machine Intelligence, 2020. 2, 6, 19
[19] Chu. E., Roy. D., and Andreas, J. Are visual explanations useful? case study in model-in tan-th--lopp prediction. arXiv preprint arXiv:2007.12248 2020. 10

---

[20] Crabbé. J., Qian, Z., Imrie. F., and van der Schaar, M. Explaining latent representations with a corpus of examples. Advances Neural Information Processing Systems, 34, 2021. 10
[21] Cuturi, M. Sinkhom distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013. 4, 17
[22] Donnelly, J., Barnett, A. J., and Chen, C. Deformable protopnet: An interpretable image classifier using deformable prototypes. arXiv preprint urXiv:2111.1500 2021. 6, 10
[23] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani. M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby. N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations. 9
2021. URL. htttpp://pppereeeeeeeeeee
[24] Fel. T., Colin, J., Cadène, R., and Serre, T. What cannot predict, do not understand: A human-centered evaluation framework for explainability methods. arXiv preprint arXiv:2112.04417, 2021. 1, 10
[25] Feng, H., Wang. S., and Ge. S. S. Tine-grained visual recognition with salient feature detection. arXiv preprint 188. 039333 2018. 4
[26] Feng, and cooperative play. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pp- 229-239, 2019. 65
J. What can ai do mc? evaluating machine learning interpretations in
[27] Geirhos, R., Narayanappa. K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F, A., and Brendel. W. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34, 2021. 10
[28] Ghacini, R., Fern, X. Z., and Tadepalli, P. Interpreting recurrent and iicntion-baseee neural models: case study on natural language inference. arXiv preprint 1 03894, 2018. 65
[29] Hai Phan, A. N. Deepface-emd: Re-ranking using patch-wise earth mover's distance improves out-of- distribution face identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022. 1, 2, 3, 4, 9, 16, 17
[30] Ham, B., Cho, M., Schmid, C., and Ponce, J. Proposal flow: Semantic correspondences from object proposals. IEFE transactions on pattern analysis and machine intelligence, 40(7): 2017. 17
[31] He, K., Zhang. X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 2, 4, 18
[32] Hemmer, P., Schemmer. M.. Vössing, M., and Kühl, N. Human-ai ai complementarity in hybrid intelligence systems: A structured literature review. PACIS 2021 Proceedings, 2021. 10
[33] Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., and Satzger, G. On the effect of information asymmetry in human-ai teams. arXiv preprint arXiv:2005.01467, 2022. 10
[34] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.1226:. 2019. 10, 19
[35] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang. F., Dorundo, E.,, Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distributin generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8344 8349, 2021. 2, 7, 19
[36] Hendrycks, D., Zhao. K.. Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021. 10, 19
[37] Horvitz, and Pack, Complementary computing: policies for transferring callers from dialog systems to human receptionists. User Modeling and User-Adapted Interaction, 17(1):159-182, 2007. 10
[38] Jeyakumar, V., Noor, J., Cheng, Y.-H., Garcia, L., and Srivastava, M. How can i explain this to you? an empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems, 33:421 1 1-4222, 2020. 10
[39] Johnson, J.. Douze, M., and Jégou, H. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7(3):535- 547, 2019. 20
12

---

[40] Kamnrr E., Hacker, S.. and Horvitz, E. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, volume 12. pp. 467-474, 2012. 10
[41] Kenny, E. M. and Keane, M. T. Tyin-systems to explain artificial neural networks using case- base- reasoning: Comparative tests of feature- weighti-eightingg methods in ann-cbr twins for xai. In Twenty- Eight International Joint Conferences on Artifical Intelligence (IJCAI), Macao, 10-16 August 2019, pp. 2708- 2715, 2019. 1, 10
[42] Kim, S., Meister, N., Ramaswamy, V., Fong, R., and Russakovsky, O. Hive: Evaluating the human interpretability of visual explanations. arXiv preprint arXiv:2112.03114 2021. 1,5, 10
[43] Kim, W., Son, B., and Kim, I. Vilt: Vision-and and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pp. 1883,5594, PMLR, 2021. 9
[44] Kingma, D. and Ba, Adam: method for stochastic optimization. arXiv preprint arXiv:1412.698) 2014. 16
[45] Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information procexsing systems, 25, 2012. 4
[46] Lee, J., Kim, D., Ponce, J., and Ham, B. Sfnet: Learning object-aware semantic correspondence. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2278-2287, 2019. 17
[47] Li, H., Wang, Y., Wu, A., Wei, H., and Qu, H. Structure-aware visualization retrieval. CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2022. 4
[48] Li, S., Han, K., Costain, W., Howard-Jenkins, H., and Prisacariu, V. Correspondence networks with adaptive neighbourhood consensus. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10196-10205, 2020. 4
[49] Mac Aodha, O., Su, S., Chen, Y., Perona, P., and Yue, Y. Teaching categories to human learners with visual explanations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3820-3828, 2018. 5, 10
[50] Miller, G. A. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39 41, 1995. 32
[51] Min, and Cho, M. Convolutional hough matching networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2940-2950, 2021. 3,4,4,11 17
[52] Nauta, M., Jutte. A., Provoost, J., and Seifert. C. This looks like that, because... explaining prototypes for interpretable image recognition. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 441 441-456. Springer, 2021. 1, 10
[53] Nauta, M., van Bree, R., and Seifert, C. Neural prototype trees for interpretable fine- fine-grained image recognition. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14933-14943, 2021. 1,3,1 10
[54] Nguyen, A., Yosinski, and Clune, Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015. 6
[55] Nguyen, G., Kim. D., and Nguyen, A. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems, 34, 2021. 1, 4, 5, 6, 8, 10
[56] Palan, S. and Schitter, C. Prolific. ac a subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17:22 27, 2018. 6
[57] Papernot, N. and McDaniel, P. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 1803, 04765, 2018
[58] Raghu, M., Blumer, K., Corrado, G., Kleinberg. J., Obermeyer. Z., and Mullainathan, S. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.1 12220, 2019, 10
[59] Ribeiro, M. T. Singh, S., and Guestrin, C. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144 ACM, 2016. 65
13

---

[60] Rocco, I., Cimpoi, M., Arandjelović, R., Torii, A., Pajéla, T., and Sivic. J. Neighbourhood consensus networks. Advances in neural information processing systems, 31, 2018. 4
[61] Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206 22520006211,000 2019. 10
[62] Rudin, C., Chen, C., Chen, Z.,, Huang, H., Semenova, L., and Zhong, C. Interpretable machine learning: Fundamental principles and 10 grand challenges. arXiv preprint arXiv:2 11251, 2021. 10
[63] Russakovsky, O., Deng. J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, 7.,, Karpathy, A., Khosla, A., Iinomnationaaannn
Bernstein, M., et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)211-2555 2015. 2, 19
[64] Schemmer, M., Hemmer, P., Kühl, N., Benz, C., and Satzger, G. Should follow ai-based advice? measuring appropriate reliance in human-ai ai decision makinn. arXiv preprint arXiv:2204, 06916, 2022. 10
[65] Shen, H. and Huang, T.-H. How useful are the human evaluation on guessing the incorrectly predicted labels. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pp. 168-172, 2020. 5
interpretations general users?
[66] Singh, G. and Yow. K.-C. These do not look like those: An interpretable deep learning model for image recognition. IEEE Access, 9:41482-41493, 2021. 1
[67] Singla, A., Bogunovic, I., Bartók, G., Karbasi, A., and Krause, A. Near-optimally teaching the crowd to classify. In International Conference on Machine Learning, pp. 154-162. PMLR, 2014. 5
[68] Stylianou, A., Souvenir, R., and Pless, R. Visualizing deep similarity networks. In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 2029-2037. IEEE, 2019. 17
[69] Turner, A., Kaushik, M., Huang, M.-T., and Varanasi, Calibrating trust in ai-assisted decision making.
[70] Van Hom, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 8769-8778, 2018. 3,7
[71] Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd 200-2-200-2011 dataset. 2011. 2, 3,7
[72] Wang, H., Ge, S., Lipton, Z., and Xing, E, P. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019. 2, 6, 7, 19
[73] Wang, Z., Simoncelli, E, P., and Bovik, A. C. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402, Ieee, 2003. 9
[74] Yang, X. J., Unhelkar, V. Li, K., and Shah, J. A. Evaluating effects of user experience and system transparency trust in automation. In 2017 12th ACM/IEEE International Conference on Human-Rohon Interaction (HRI, pp. 408-416. IEEE, 2017.
[75] Zhang, C., Cai, Y., Lin, G., and Shen, C. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12203-12213, 2020. 1, 6, 9, 16
[76] Zhang, Q., Lee, M. L., and Carter, S. You complete me: Human-ai teams and complementary expertise. In CHIl Conference on Human Factors in Computing Systems, pp. 1-28, 2022. 10
[77] Zhang, R., Isola, P., Efros, A. A., Shechtman, F.,, and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018. 9
[78] Zhang, Y. Liao, Q. V., and Bellamy, R. K. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountabilit, and Transparency, pp. 295-20220000 305, 2020. 10, 65
[79] Zhao, W., Rao, Y., Wang, Z., Lu, J.. and Zhou, Towards interpretable deep metric learning with structural matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9887-9896. 2021. 1, 3, 4, 6, 9, 10, 16, 17
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Please see Sec. 3.
(b) Did you describe the limitations of your work? [Yes]. Please Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work? [N/A].
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes].
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes].
(b) Did you include complete proofs of all theoretical results? [N/A].
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No].
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]. Please see Sec. 2.
(c) Did you report error bars (e.g.. with respect to the random seed after running exper- iments multiple times)? [Yes]. We reported whenever possible. See μ and in Table 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No].
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cited the authors and include the URLss
(b) Did you mention the license of the assets? [N/A] We used publicly-available datasets and code.
(c) Did you include any new assets either in the supplemental material or URL? [No].
(d) Did you discuss whether and how consent was obtained from people whose data you using/curating? [Yes]. We explained to participants how their data will be used. (c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes]. We checked and ensured that our data does not contain personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes]. . See screenshots in Appendix
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A].
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Sec. 2.3.4. Our rate was $13.5/hr, higher than the Prolific recommended rate wage of $9.60/hr.
15


Reviews:
[{"review_id": "p_CK5ON4zd-", "reviewer": "Reviewer_Wgym", "summary": "This work proposed a method to explain classifier prediction by feature correspondences. Two main methods are presented -- 1. patch correspondence by earth moving distance (EMD) in cross-correlation maps from image features, 2. patch correspondence by deep hough voting. Both methods uses the matching score from the top matched patches to re-rank how \"explainable\" the training data is for the query data. The method can be used in two ways -- 1. the re-ranking mechanism can be applied as a kNN algorithm for robust classification, 2. the top matched patch pair can be used to visualize the network prediction, which is claimed to help improve human-AI team accuracy. Several evaluations for different applications is done, including out-of-distribution robustness, human-alone image classification, human-AI team classification, and correcting AI's wrong decisions.", "questions": "In figure 4a, there are 4 stars but without explanation in the caption. Although it is explained in the appendix, it is better to also mention it in the main text.", "limitations": "Please address the limitations mentioned in the weakness section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Update: I have raised my rating to a weak accept, since the authors have addressed most of my concerns.]\n\nStrengths:\n1. It is interesting to use feature correspondences to explain model predictions.\n\nWeaknesses:\n1. It's hard to understand what is the main contribution of this paper. There are several application for the proposed method. However, the method only shows consistent but marginal improvement in classification robustness.\n2. In most of the case, just using the prediction from plain resnet-50 achieves best performance in human-AI team performance. It also performs best in human-alone performance in ImageNet.\n3. It might be good to compare against several XAI baselines, at least for the human-alone performance. For example, grad-CAM [1].\n4. It might be great to include more mathematical descriptions for the method, especially for CHM-Corr.\n\nSummary:\nIn my opinion, the paper needs to provide stronger results to show the proposed method is convincing. However, I am happy to adjust my rating according to the authors' responses.\n\n[1] Selvaraju et al. Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WB9JEIsWR-b", "reviewer": "Reviewer_6EjV", "summary": "In this paper, the authors designed two \"nearest neighbor\"-based classifiers based on visual correspondences, namely, EMD-Corr and CHM-Corr. The EMD-Corr is similar to Zhao et al. (2021), and solves an optimal transport problem to match image patches from the query image to those from a training image. The difference from Zhao et al. (2021) is that the authors only used the top 5 patch matchings for classifications and explanations. The CHM-Corr is based on the Convolutional Hough Matching (CHM) network from Min and Cho (2021) -- it first solves visual correspondences between the query image and a training image using CHM (thereby removing the need for solving the optimal transport problem), and use the visual correspondences obtained from CHM to find top 5 patch matchings for classifications and explanations. In addition to designing the classifiers, the authors did an extensive user study to test the usefulness and effectiveness of visual correspondence-based explanations for improving human accuracy and human-AI team accuracy on solving challenging classification problems.", "questions": "The paper is very clearly written, and I do not have additional questions that need authors to clarify.", "limitations": "The authors have adequately addressed the limitations of their work associated with collecting data from human subjects online.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The user study is comprehensive and great, and demonstrates the utility of visual correspondence-based explanations.\n\n+ The proposed methods (EMD-Corr and CHM-Corr) are simple to implement, and can achieve reasonably high accuracy on complex datasets such as ImageNet.\n\nWeaknesses:\n\n- The proposed methods (EMD-Corr and CHM-Corr) are very incremental in design from prior works (Zhao et al., 2021 and Min and Cho, 2021).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DzoD8O7wQpe", "reviewer": "Reviewer_riHj", "summary": "This paper proposes a classifier based on different refining strategies of the closest nearest neighbors using intermediate features from a ResNet50 network. These refining is based on a spatial patch-based correspondence, providing explanations in the form of example images showing the top similar sub-areas to a query.\n\nExperiments show that these kind of classification pipeline is robust when classifying adversarial samples. Moreover, human-studies indicate that the intermediate patch-matching method convey information about the computer’s decision such that humans can improve upon their own classification baseline (and that of the model alone).", "questions": "1. What are the computational costs associated with this kind of classifiers, compared to a regular Resnet50?\n\n2. Why are the results of an AI-Only evaluation on ResNet-50 (Table 2 reports an accuracy of 86.05%) much higher than the performance of the one reported in the baseline evaluation (Table 1 reports an accuracy of 76.13%)?\n\n3. How does classification accuracy compare to the usual \"tricks\" that can be applied to pre-trained models (e.g., fine-tuning some of the deeper layers) or adversarial training?", "limitations": "1. The high computational cost of the proposed methods (as mentioned towards the end of the paper) has not been addressed quantitatively. This is especially important given the incremental improvements that have been reported with respect to their baselines.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- *Originality*\n    - The overall methods to find correspondences, sort matching pairs and issue a classification prediction are mostly based on previous work. The method is mostly similar to those of Hai or Zhao ([25] and [67] in the paper). The proposed modifications to focus on the top patches and the use of a re-ranking optimization serve as a well-founded refinement of this method, making the computations more traceable and therefore, more interpretable.\n\n- *Quality*\n    - The selection of experiments is thorough and extensive.\n    - From all variants of Imagenet, one that is particularly interesting when evaluating XAI methods is the subset of “natural adversarial samples” (a.k.a. Imagenet-A) [2]. Unfortunately, this dataset was not included as part of the evaluation.\n\n- *Clarity*:\n    - The paper lacks a clear storyline and often jumps between results, experimental setup and motivation. Even though the contributions appear to be sound and significant, the lack of clarity presents problems for reproducibility.\n    - Important parts of the experimental setup (including data and methods) are been cited without elaborating further on the specific parts. While not necessarily a mistake, the paper ends is not self-contained at all, which heavily impacts readability and reproducibility.\n    - Quite a few passages are poorly structured. Even though it is possible to make sense of what is being said, the clarity of the paper suffers considerably.\n    - The discourse of the paper jumps between showing classification improvements and OOD robustness. These are two different problem setups that are not necessarily interchangeable.\n\n- *Significance:*\n    - Evaluating on adversarial datasets sets an ambitious and valuable goal.\n    - The inclusion of large human studies with careful policies for selecting (and discarding) subjects potentiates the credence of these results.\n    - Experiments show that in 7/8 of setups, the kNN baseline is within 1.6pp. of the proposed non-parametric classifiers (the second to last one is under 6pp.). Despite its statistical significance, there is not enough evidence that can justify the additional computational resources for running kNN at this stage, let alone the proposed re-ranking metrics. More concretely, how do these results compare to the usual \"tricks\" that can be applied to pre-trained models (e.g., fine-tuning some of the deeper layers)?\n    - Regarding experiments on Damagenet and adversarial patches: the reported improvements, even though they can be statistically significant, are still low relative to the performance on clean samples. Moreover, it is known that this kind of improvements get proven useless by generating adversarial attacks that are tailored to the classifier (kNN classifiers in this case). For instance, by using non-differentiable approximations [1].\n\n### References:\n\n1. Athalye, Anish, Nicholas Carlini, and David Wagner. \"Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.\" *International Conference on machine learning*. PMLR, 2018.\n2. Hendrycks, Dan, et al. \"Natural adversarial examples.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2VUV0YU-Pb2", "reviewer": "Reviewer_dcDH", "summary": "This work proposes a novel framework and two interpretable classifiers based on retrieval and re-ranking strategy. In this framework, a visual correspondence between the query image and exemplars can be computed as explanations. This strategy is shown to be more robust and achieve comparable performance to black-box classifiers, and improve the performance when working with humans. \n\n", "questions": "\"our approach is inherently interpretable\". The features extracted by deep models are still black-box. Consider rephrasing this sentence. ", "limitations": "addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Comprehensive experiments.\n- Interesting ideas", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept