Advancing Nearest Neighbor Explanation-by-Example with Critical Classification Regions
Authors: Eoin M. Kenny
Eoin D. Delaney
Mark T. Keane
Venue: ICLR 2022 Submitted (2022)
Keywords: Explainable AI
Post-hoc Nearest Neighbor Explanation-by-Example
User Study
Case-based Reasoning
Convolutional Neural Network

Abstract: There is an increasing body of evidence suggesting that post-hoc explanation-by- example with nearest neighbors is a promising solution for the eXplainable Artificial Intelligence (XAI) problem. However, despite being thoroughly researched for decades, such post-hoc methods have never seriously explored how to enhance these explanations by highlighting specific important "parts" in a classification. Here, we propose the notion of Critical Classification Regions (CCRs) to do this, and several possible methods are experimentally compared to determine the best approach for this explanation strategy. CCRs supplement nearest neighbor examples by highlighting similar important "parts" in the image explanation. Experiments across multiple domains show that CCRs represent key features used by the CNN in both the testing and training data. Finally, a suitably-controlled user study (N=163) on ImageNet, shows CCRs improve people’s assessments of the correctness of a CNN’s predictions for difficult classifications due to ambiguity.

Under review as a conference paper at ICLR 2022
ADVANCING NEAREST NEIGHBOR EXPLANATION-BY. EXAMPLE WITH CRITICAL CLASSIFICATION REGIONS
Anonymous authors Paper under double-blind review
ABSTRACT
There is increasing evidence suggesting post-hoc explanation-by-exane with nearest neighbors is a promising solution for eXplainable Artificial Intelligence (XAI). However, despite being researched for decades, such methods have never seriously explored how to enhance these explanations by highlighting specific parts in classification. Here, Critical Classification Regions (CCRs) pro- posed to do this, and several methods are compared to determine the best ap- proach. CCRs supplement explanation-by-examee part in a test image, with where it was "learned" from in the training data. Experi- ments across multiple domains show that CCRs represent key features used by the CNN in both the testing and training data. Moreover, a study (N=163) on ImageNet, shows CCRs improve people's 's assessment towards the correctness of CNN's predictions for misclassifications due to ambiguity.
by highlighting an important
user
INTRODUCTION
The impressive success of artificial neural networks (ANNs) has led to proposals they should be used in high-stakes applications such as diagnostic radiology for Covid-19 (Pham) (020). How- ever, interpretability issues for these models raise significant questions about their feasibility for such use-cases. Accordingly, many eXplainable Al (XAI) techniques have been proposed to over- come this, such as saliency maps (Zhou et al.] (016) and contrastive methods (Miller 2019). Here, the focus is on one of the most long-standing XAI techniques, the use of post-hoc explanation-by example with nearest-neighbors (Sørmo al.) 2005). with view to extending its functionality. Recently, caaaaation-y--xanpnae models (Lipton) 2018: Chen al.) 2019 Jeyakumar et al., 2020 Kenny & Keane 2021), with in- terest being boistered by their psychological plausibility in human decision-making (Klein) 1989), human category-learning (Edwards et al. 2019), and supportive evidence from several user stud- ies (Borowski et al. 2020 Kenny et al.
techniques have been increasingly used to interpret deep learning
2021: Buçinca et al. 2020). Currently however, these used in a prediction, we rectify this here.
techniques don't 't connect important "parts"
's functionality through the use of Critical Classification Regions (CCRs), and (2) to properly test this novel technique in a suitably- controlled user study ( CCRs represent the primary region of importance in test image and in the nearest- neighhor used for explanation (e.g., see Fig. TT. Although many post-hoc XAI techniques for deep learners show salient regions of test image (Ribeiro al. 2016 Bach al.. 2015 Zhou al.) 2016), to our knowledge, none have seriously considered relating these regions to nearest neighbors in the training data, show what features were learned by the model and (more importantly, our core novelty here) where they arose.
This work has two main aims: (1) to extend ielllaattion---exaamma
In the XAI literature, poolanation-by-exanaee is perhaps the most popular XAI method (Keane & Kenny 2019) with large user support (Buçinca et al.) 2020). However, few works address the iden- tification of important regions within example-based based explanations. The closest work ours makes use of Feature Activation Maps (FAMs) within the (win-system frumework to enhance example- based explanations (Kenny& & Keane 202T). FAMs upsample the most positively contributing con- volutional feature kernel (in the last convolutional layer) and, as such, can be thought of as a variant of Class Activation Maps [CAMs by Zhou et al. (0166), which use a combination of all layers.
'Ou method is available to pip install a Python library at https://after anon review.

---

Under review as a conference paper at ICLR 2022
Test Image
Explanation By-Example
Test Image
Explanation By-Example
(A)
(B)
(C)
(D)
Figure 1: Different Methods: A misclassification of "Leopard" as "Cheetah" on ImageNet is ex- plained by: (A) Explanation-by-exxmppl saying "I think the test image is a Cheetah because looks similar to a training image also think is a Cheetah". This is then enhanced by (B) FAMs by Kenny& Keane (2019), and our proposed methods of (C) Latent-based CCRs, and (D) pixel- based CCRs. The FAMs highlight multiple regions in both images, and it is difficult to know where a feature begins and ends, contrast, CCRs highlight a very specific part of both images. Note that the CCR explanation methods naturally retrieve a different neighbor due to searching "pool" of candidates for the closest feature match, whilst FAMs always use the closest NN.
However, FAMs often fail isolate comprehensible "part" of each image, and can be hard to contextualize (see Fig. [B). Moreover, there is no computational tests of FAMs, or user studies. In contrast, we show how CCRs play a key role in classification, and how they affect users in a study.
There is also a literature on post-hoc explanations which isolate prototypical examples from training data (Kim al. [014]2016). These bear a similarity the present work, but they do not focus on isolating important parts in the classification which we do here. There is also work in the computer vision attention literature such as that by Patro & Namboodiri 2018), which also use exemplars to guide feature highlighting. However, this has similar issues to FAMs, and requires a specific ANN architecture for a very specific task, whilst CCRs are agnostic for ANN vision models. Next, this work bears some similarity to Goyal et al. (2019), but their work focuses counterfactual explana- tions, showcasing an algorithm which modifies the smallest part possible to modify a classification, whilst ours is on factual explanations, showing why classification made in the first place (the two have very different algorithmic goals). Finally, note also that none of these methods consider superpixels
do here.
From an implementation standpoint, the current latent-based CCR method (see Section 2) is partly inspired by Chen al. (2019), who used a case-based approach to explain CNN's predictions. However, their approach is a pre-hoc method that compresses the training data down into prototyp- forward pass.
ical parts for explanations, integrating them directly into the CNN's caleulations In contrast, the current method is post-hoc, using the training data for explanation after the predic- tion made. Hence, it may be better suited to situations where (i) model accuracy is critical, (ii) is simply not feasible to train another more interpretable model, or (iii) more diverse explanations (from the full training data) are required (Mothilal et al.) 2020). Finally, the current method builds upon the family of activation-mapping techniques for visualizing the important regions of an image used by a CNN in its predictions (Zhou et al. 2016 Kenny & Keane 2021).
This paper has four more sections. Section 2 describes how to compute CCRs in explanation- by-example. As CCRs can be generated from the latent space (i.e., Latent-based CCRs), or the pixel space (i.e., Pixel-based CCRs), Section determines which option works best for the rest image. Section | performs additional checks on CCRs to help validate their role classification, and determines the best way identifying CCRs in nearest-neighhor explanations (henceforth the
2

---

Under review as a conference paper at ICLR 2022
reports user study that tests whether the CCRs found by this method matter to people, before a final discussion (see Section O This paper makes two main contributions:
NN-CCR) in the training image. Section 5
A new XAI idea that computes CCRs to augment post-hoc, nearest-neighbor explanation- by-example. These CCRs connect critical features used in classification with where they were learned from in the training data (to e.g. help detect bias and aid in explanation). A method for CNNs is proposed (latent-based CCRs), and "black-box" method (SP-CCRs) which is ANN agnostic for computer vision tasks.
A novel user study design which is heavily controlled (e.g.. with counterbalancing) to specifically test the difference between exxmmlle and explanation-by- example with feature highlighting (i.e., the CCRs).
ALGORITHM: FINDING CCRS IN EXPLANATIONS
Taking an explanation-by-examppl method, our algorithm begins by finding pool of k nearest neighbors (we use k = 50), all of which will be searched to find the NN-CCR Two general approaches are considered, one which takes advantage a typical CNN model's architecture (i.c., the Latent-based CCRs next), and one which makes no assumptions other than the model being neural network which extracts a learned representation (i.e., Pixel-based CCRs). For the algorithm pseudo-code see Appendix E (omitted due space constraints). An important points to note about CCR methods is that the regions found are contingent on the pool of NNs over which they are computed (hence, the need for the test of N-NN methods reported in Appendix A).
2.1 COMPUTING LATENT-BASED CCRs (CAM-CCHE) FAM-CCRs, RAND-CCRS)
This CCR Method takes inspiration from Chen et al. (019), but crucially, in a post-hoc (rather than their pre-hoc) manner, specifically, it uses the latent representations of an image in a CNN to gauge similarity. Assume for a given test image 7, a final representation C' E IR (h,,,,)) is extracted after all convolutional layers, were h and represent the height and width of the convolutional output, respectively, and d the number of kernels (in all current experiments h = w = 7). This may be broken down into regions shaped as h2 W1 X d, were h1 < h and w1 < w. These regions may be upsampled to the size of the original image to visualize them as "box" (e.g.. see Fig. [ which is the region in pixel-space that corresponds to this region in c Here, we want to select the region in C' which is critical to the classification, and use it to elaborate the explanation further.
Here we set h1 wi to give more granular detail in the explanation. To select which regions of C' are CCRs, we assume the presence of some activation map giving a saliency value to each region in C (e.g., FAMs or CAMs). Such a technique gives a 7 x 7 saliency map corresponding to we refer to as Miest E (7.7, which gives the importance of each spatial region. Next, by selecting the most positive salient region, we can isolate the test image CCR Wrest E (L.1.dd)
Then, with the test-image CCR isolated, the task is to find a similar region from the training data Wan- giving an example of where this feature was "learned" from. Using some nearest neighbor algorithm, pool of training-instance are found that are candidate explanations-b--exxmppe In this pool, let C(m.i.j) represent some potential region w E (I.1.,))) in the final convolutional layer, with the neighbor indexed by n, and its spatial position in Cn indexed by i and j. These images have their C' representation searched to find the closest match to Whest using the L2 norm to find Won- Importantly however, this region in the NNs is constrained to its relative importance within the instance. Specifically, considering each NN's activation map My. only those regions which satisfy the constraint of being higher than max x a are considered to find Wha by minimizing:
arg min n,i.
||teest Gin,,,))l2
S.t.
M(n,5.5) maa((Ma) a!
Note we used twin-systems by Kenny & Keane self-supervised learning (which may be more general for future research into CCRs).
2021 as it did well in Appendix tests, and it works for
Note this could be a bigger box, but we choose small one to give more granularity. However, more CCRs be shown if desired (see e.g., Fig. TT, or SP-CCRs used which can give higger regions.
3

---

Under review as a conference paper at ICLR 2022
The alpha constraint is to help ensure wnn is highly critical to the classification. For M, exper- iment using CAM, FAM, and Random maps. So three variants of latent-based CCRs are tested CAM-CCRs, FAM-CCRs, and Rand-CCRs although other options exist for retrieving M.
2.2 COMPUTING PIXEL-BASED CCRs (SP-CCRs)
These CCRs use pixel representations, this is better were spatial relations between the latent layers (i.e.. C' in the previous section) and the pixel input are not reliable (c.g., because of max max- max-pooling), or convolutions are not used [e.g., vision transformers (Dosovitskiy et al. 2020)1, agnostic. Specifically, a test image is broken down into superpixel segments, each segment can be passed into the network one-by-one (with the rest of the image occluded), and the CNN's logit value for the class in question is recorded to find the most important region used in a classification [similar to LIME by Ribeiro et al. (2016)]. Once the test image CCR isolated, we upsample to the CNN input-size whilst maintaining its aspect ratio, and acquire its representation from taking the penultimate layer activations after forward pass in the network. This process repeated in the pool of NNs, to find the segment (i.e., the NN-CCR) most similar to the test image CCR.
it is ANN
Formally. consider the test image CCR representation Wtest E IR (d) were d is the number of ex- tracted features in the penultimate layer. We want to isolate a region in the training data wan where this feature was "Iearned". Let ((n.)) be the representation w E IR (d) of a superpixel segment in the NN n. Additionally, let MM(,.) represent each region's 's saliency. The n images have their S rep- resentations searched to find the closest match to wtest using the L2 norm to find won. Importantly, this region is constrained to its relative importance within the instance. Specifically, only regions whose saliency is higher than mar x B-1 are considered to find Won by minimizing:
arg min n2,
||Wreat Sm, 122
S.l. M(n,i) max (M,) B-1
The beta constraint helps ensure Wan highly critical to the classification. For pixel-based CCRs, we consider this single method (i.e., SP-CCRs), but comparative tests against LIME are considered in Expt. 1, but the computational cost of Expt. doesn't 't allow comparisons to LIME. So, including the three latent- based CCR methods, there are four main methods tested.
3 EXPERIMENT 1: FINDING TEST IMAGE CCRs
Here, the performance of the latent- and pixel-based CCR methods in isolating an important part of the test image was evaluated. This worked via two methods, by (1) keeping the CCR region in the pixel image (whilst occluding the rest), and (2) by removing the CCR regions (and keeping the rest), followed by passing the image through the CNN in either case. So, in the first case, the method that produces the highest logit in the test- test-image's 's predicted class does best, and in the second the method which produces the highest drop in the predicted class logit is best (if the prediction changes we do not change which logit is recorded). In more detail, the test image SP-CCR is located, then the ditent-based CCRs are found by up-sampling their activation maps to the pixel-space, and isolating an equally sized region to the SP-CCR the pixel space taken from the parts of highest saliency (so all CCRs are the same size). Two datasets CUB-200 and ImageNet were used, with the former having ResNet18 fine-tuned to it, and the latter using a pre-trained ResNet50 (see Appendix C). The experiment is repeated with different segmentation options for superpixels (i.c., 10-50) to understand its effect on explanations. Tests used the first 500 test images. Finally, a comparison with LIME is considered to see if it has significant difference to the SP-CCR algorithm.
3.1 RESULTS
Fig. A|/B) shows the results of occluding (Occ.) the test image CCR, and Fig. D(g.)C/D)) of including (Inc.) it (and occluding the rest of the image). The top row shows the results of comparing the four
CCR methods discussed in Section 2 whilst the bottom row shows comparisons of the latent-based
CCR methods against LIME. Note these LIME comparisons correspond to roughly 5% of the test image being isolated as CCR, which in turn corresponds to roughly 30 superpixels being used in the top row tests. Overall, the results show that CCRs and LIME are significantly better than random. Superpixels perform best for inclusion rather than occlusion when the segment number is
4

---

Under review as a conference paper at ICLR 2022
(A) CUB-200 200 Occ.
(B) ImageNet Occ.
(C) CUB-200 Inc.
(D) ImageNet Inc.
Prodicted Class Lagit Drop
Friden
Srece Clao Logl
Num. Superpixel Regions Used
Num. Superpixal Regions Used
Prodicted Class Lagkt Drup
Preduced Class oot
1+
556
LM
0M
14
CCR Method
CCR Method
Figure 2: Expt. CCR Occlusion (Occ.) and Inclusion (Inc.) Results: The first row of lineplots show comparison of the four different CCR methods proposed Section [ comparison of the three latent-based CCR methods against LIME, see the difference between LIME and SP-CCRs. Overall, results show the CCR methods do significantly better than random occlusion/inclusion. and that LIME performs similarly to SP-CCRs. Standard Error bars shown.
The second row show
greater than 30, but CAMs/FAMs are more consistently good in all tests, although in general FAMs slightly less discriminatory than CAMs. Perhaps the most notable aspect is how all CCRs (and LIME) do worst when occluding in ImageNet. We posit this is likely because ImageNet has many objects in image which used for classification, and removing small region has little effect. Superpixels in particular do bad here (i.e., worse than random), likely because they still generally maintain the "shape" of the object during occlusion, whilst the latent-based CCR methods (including random) always occlude smoother shapes which distort the objects more. We feel this hypothesis is true because this phenomena was (1) not repeated CUB-200 (which only has small part of the image used for classification in comparison),
(2) not repeated in the inclusion experiments, and (3) consistent across SP-CCRs/LIME: which is notable because LIME is generally thought to deliver good explanations (Ribeiro et al. 2016 Jeyakumar et al. 2020). So, taking the results as whole, it safe to posit that all CCR methods work well isolating important parts of the test image.
EXPERIMENT 2: EXPLANATION FIDELITY WITH NN-CCRS
Potential CCRs which are "used" in classification are found by occluding parts of the training data during fine-tuning to what is necessary maintain test performance. First, and / varied from one to infinity [the latter of which is treated as considering all positive CCRs in (q.(12) to their optimal value, then comparative test between the is done. Eq. (1) and cannot be directly compared because the alpha and beta constraints are relative measures. However, a comparison can be accomplished by (1) varying alpha, (2) closely matching the occlusion area by gradually introducing superpixels (in order of highest saliency value), and (3) readjusting the size of the latent-based methods area match the superpixel area.
So, for each hyperparamter value tested, the networks were fine-tuned for 2500 iterations and test- accuracy sampled every 50, as this was found sufficient for convergence (we tested 20 epochs and found no notable differences). This procedure gives us an indication of which regions of the training data images are actually responsible
test predictions (and what a and | are best), whilst the Appendix A experiment gave us indication of which training examples are most appropriate for explanations. The networks were also tested by completely occluding all training images and was reduced to random guess, thus verifying that features are being "unlearned". Note this experiment requires a constant value for superpixel segmentation, so 30 was chosen as it was the smallest value which generalized best in Expt. Finally, using the best hyperparameters, "similar looking" the NN-CCR is to the test-image CCR by passing the upsampled CCRs though the CNN, and comparing their latent representations in the CNN's penultimate layer with the Z norm.
finally test how

---

Under review as a conference paper at ICLR 2022
(A)
(B)
(C)
CUB-200
CCR Method
Superpiiels
CAM
FAM
Random
ImageNet
3.0
8.5 10.0 160.0
8.
100
Figure 3: Expt. 2 Results Retraining: (A) Exploring the hyperparameter choice for alpha, (B) doing similar tests for beta. (C) A direct comparison between methods by varying alpha. Results show that all methods are better than the random baseline in nearly every test, and that CAM seems the best method to isolate important critical regions in image.
4.1 RESULTS
Fig. A shows the results of varying alpha, were any value greater than 1 produces statistically sigmificant differences between methods (using tailed independent t-test; p < 0.05). Fig. B shows what happens to superpixels v. random when varying the beta parameter, were infinite value (i.e., defined as just using all positive superpixel regions here) produces the most divergent SP-CCR=45.86) and ImageNet (Acc. Rand=68.41 81
results for CUB-200 (Acc. Rand=61.55 SP-CCR=65.66) 68). Note that even with 3 in only ~666 of the images are occluded in Ima- geNet average for SP-CCRs, which roughly equates the amount occluded for CAM-CCRs = 5, so there is actually not a huge disparity between them in test accuracy when considering this. For further investigation, Fig. C)) shows direct comparison between methods. Specifically, Fig. C shows that CAM performs the best overall, with FAMs and superpixels being somewhat interchangeable, but all methods do significantly outperform the random baseline, showing that all method's explanations are likely high in fidelity. Next, each method's distance of the latent repre- sentation between the test image CCR and the NN-CCR was compared using 3 inf and a 5, showing that the SP-CCRs are significantly better than all other methods when using more than 20 segments (mean SP-CCR L,z14.72/13.4
segments 20/50 CAM mean Lz=15.1 on ImageNet). Finally, it should be noted these are empirical tests retrained networks and should be treated with some caution as they are different to the original CNN being explained, but similar tests have been done before and widely accepted (Hooker et al. 2019).
Computational Conclusions. Overall, the results of Expts. 1-2 show that all CCR methods (SP- CCR, CAM-CCR, FAM-CCR) perform better than random baseline, lending strong empirical evidence they are isolating important regions in classifications. Superpixel segmentation of approx- imately 30 segments recommended because generalizes best across all tests and datasets, forms reasonably large areas which are assumed to aid interpretability, and "look" very similar when com- paring the test image CCR to the NN-CCR. For beta an infinite value is recommended, which computed here as considering all positively contributing SP-CCRs (and not undefined as would be its strict mathematical definition). For alpha, any value greater than one is good, but we recommend five because it worked well here, will tend to focus more especially important regions, and has support in other research (Zhou et al. (016).. Lastly, it is worth noting that latent-based CCRs are much faster than SP-CCRs (~2sec 45 for 1=10 neighbors), but SP-CCRs are not restricted to CNNs, which gives them wider applicability (but we test them on CNNs here for a fair comparison). The subsequent user study evaluates CAM-CCRs with 5.

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENT 3: USER STUDY
The previous computational experiments showed that the current explanation-by-exane method with critical regions bears high fidelity to what the CNN classifier has learned. However, we do not know whether these critical regions matter to people; whether the provision of example-plus-CCR explanations impact a user's perception of the CNN more than example-only explanations. Hence, a user study (N=163) was run to test example-plus-CCR versus example-only explanations, using latent-based CAM-CCRs. Previous user studies have shown that example-based explanations (i.e., based on 3 nearest neighbours to a test-image) changed people's s perceptions of the correctness of misclassifications by a CNN on MNIST (Kenny et al. 202Tp. Specifically, they found that people rated misclassifications as less incorrect (i.e., correctness was rated higher). Here, we examine whether the provision of similar explanatory examples with or without CCRs (i.e., a NoBox v Box manipulation) changes people's 's perceptions of misclassifications made by CNN on ImageNet.
The XAI literature has few user studies attempting to test specific explanation strategies, and most of these are inadequately designed [e.g.. poor test-item selection and low Ns are common deficits (An-l jomshoae et al. 2019; Keane et al. [0 For instance. typically, studies use only a handful of test items and these materials are properly counterbalanced over explanation conditions. Here, we try to rectify these deficits (indeed, we explicitly demonstrate how material-sets can matter).
So, the study presented participants with 32 test-images from the ImageNet dataset (i.e., 24 mis- classifications, with 8 "fillers" that were correct classifications for attention checks) and were asked to make clasification-ccoreetnsss and ayplanatiio-hellfulness judgements of these items presented alongside one of the two explanation-type classifications were randomly divided into two material sets (A-set and B-set) to counterbalance the experiment; so, (a) one group (N=82) received the A-set with example-only explanations (NoBox) and the B-set with example-plus pecell-ppsCCCRR CCR explanations (Box) and (b) the other group (N=81) received the A-set with caamppe-pplus-CCC explanations (Box) and the B-set with example-only explanations (NoBox; see Fig. 4 for examples). The statistical analysis then collapsed across these counterbal- anced groups controlling for the effects of the material-set. This consideration of material-sets is not just a "statistical nicety". as any given user will only see a sample of these images, good explana- tion strategy needs to operate successfully over different image-sammen saggles.. Hence, user studies need to control for this issue by using different, reasonably-sizee and randomly-sampled material-sets.
(NoBox or Box; i.e., no CCR or CCR). The 24 mis-
In summary, the study examined people's responses the two post-hoc explanation-tyyes,, example- based explanation-only v example-based explanation-plus plus-cCR (i.e., NoBox v Box) over 24 image- misclassifications (i.e., 12 items for each type). The counterbalanced groups allowed us test both explanation-lypes used for the same given image. As we shall see, that design also allowed us to examine material-set effects within the counterbalanced group (i.e., A-set B-set)
5.1 METHOD
Participants (N=163) were recruited on the Prolific crowdsourcing site (www.prolific.co) All were aged over 18, native English speakers and lived in the USA, UK, or Ireland. Participants were paid 57/50/ for their participation, which totalled 13,.. This N was chosen based on a power analysis for a low effect-size; this size was chosen because we anticipated the addition of CCR boxes would have quite nuanced effect over just explanation-by ex-examlle due to it already being heavily preferred by users (Jeyakumar et al. 2020). This study passed ethics review of the institution (ref. after anonymous review).
Participants.
Materials. Twenty-four misclassifications were randomly sampled from the CNN using Ima- geNet. These were actual test-image errors when the classification label differed from the ground truth. The twin-system method was applied to each prediction to find a nearest- explanation, and the CCR in the item (shown as a Box; see Fig. 5). The materials were randomly assigned to two different sets (A-set and B-set) and counterbalanced (as described earlier). Impor- tantly, the sampling constrained the images to be both varied and those involving classes people could easily understand (e.g., snail, lemon, etc.).
example-
Procedure & Measures. After being told the system was a program that "learned" to classify objects in images, people were told they would be shown several examples of its classifications (see
7

---

Under review as a conference paper at ICLR 2022
Appendix E for user study questionnaire). Their task was to rate the correctness (i.e., the question was "The program's 's labelling of the image is correct.") and helpfulness (i.e., the question was "The explanation helps understand the program's 's labelling of the object in the picture.") of the presented classification on a 5-point Likert-scale from "I disagree strongly" (1) to "I agree strongly" (5). Each participant was shown 24 misclassifications (12 NoBox and 12 Box explanations) along with 8 filler items that were all correct classifications appearing every fourth question for attention checks. The 24 incorrect items were randomly re-ordered for each person. A debriefing on the rationale and background to the study was provided after testing.
5.2 RESULTS AND DISCUSSION
Correctness & Helpfulness. On average, people perceived the misclassifications as being equally incorrect/correct for both explanation types; overall, the mean correctness ratings for both explanation-typer were the same, NoBox (M=1.85) and Box (M=1.85) and not significantly dif- ferent (using a paired t-test, t(162) = -0.018. I-tailed, p > 0.05). So, people's perception of the correctness of misclassifications when given either of the explanation-types appear to be essentially identical. However, this analysis masks an important difference when ratings are broken out into the two material-sets. The items in the B-set that received the M=1.93) were rated as less-incorrect (more correct) than their equivalent items in Set-A (M=1.77); this difference between B-set-Box and A-set-Box was statistically reliable, f(161) 2.15, p 0.03, 2-tailed using& two-sample t-test. No other pairwise comparisons were statistically significant. This result suggests that image-xxplanations showing CCRs (as a boxed outline) impact people's perception of correctness of the misclassification but only for certain items. We return to a more detailed analysis of this effect below. On helpfulness, both explanation-iyper were rated positively and equivalent, NoBox (M=3.16) and Box (M=3.1 11); using a paired t-test, #(161) -1.60, 1-tailed, p= 0.06. So, the two explanation options are equally helpful to users. Interestingly, for this mea- sure, both material-sets produced almost identical ratings for both explanation-types and no other pair-wise comparisons are statistically different.
explanation (Box;
Right* & Wrong Misclassifications. We were puzzled by the difference in correctness ratings for the example-plus-CCe (Box) conditions found in the B-set relative to the A-set. What is it about the B-set that produces this effect? In a follow-up up analysis, we discovered that, in both material-sets there were ambiguous materials that people consistently rated more correct (i.e., mean >2) even though the ground-truth identified these items as incorrect (3/12 in the A-set and 4/12 in the B- set). So, we partitioned the items into two new categories, namely "Right*" (that people rated as more "correct", even though they were incorrect classifications) and Wrong-items (that people confidently rated as incorrect, when they were incorrect) and then re-analyzed the data for each material-set set (n.b., we add an asterisk to "Right"" on purpose to signal they are not really "Right"). We also verified this partitioning by clustering the material means (using k-means) and found that the data consistently forms these two groups across 500 iterations. Figures 4 () and 4B) show that
(A) A-Set Correctness Ratings
(B) B-Set Correctness Ratings
5.
6.00
Mean Correctners Ratiing
1.34
Right Beg, 3:32
Hight Aarlow
3.8
2:00
Whong Nobio.
1:00
Whong, Modoow
Whong Rox 1.97
0.00
NoBox
Box
NoBox
Figure 4: Expt. 3 results: Correctness ratings for two material-sets (A and B) broken out by "Right"" and Wrong classifications for the two explanation-types, NoBox (example-only) and Box Catnpple-plu-CCCC In the B-Set, the Right*-Boo -Bo ratings (M=3.04) are reliably different to the Right* -Nooox ratings (NoBox, M=2.83), reflecting people's performance on ambiguous items in that material-set.
8

---

Under review as a conference paper at ICLR 2022
Test Image
Explanation-by-xxampl
Test Image
(a)
(b)
Label: Pot
Label: Lemon
Label: Barbell Classification: Prison
Label: Prison
Classification: Lemon
Classification: Lemon
Classification: Prison
Figure 5: Expt. 3 Material Examples: (a) A "Pot Pie" is misclassified as "Lemon". The explanation shows the CCR in the image identified by the CNN. The explanation represents the training data and feature "used" in the CNN's classification. Glossed, the explanation says "I think this is lemon, because it has a similar part to an image saw before which learned should be a lemon". (b) Another misclassification of "Barbell" the background reflecting the jail bars in a previous training image. Although anecdotal, an analysis of these two items shows that both have significantly higher helpfulness ratings when the CCR is given; I-tests show (a) NoBox 2.37 Box 3.09, p < 0.001 and (b) NoBox 4.19 v Box 4.48, < 0.05.
"Prison" in which the CNN picks up on the bars in
the "Right"" misclassifications are rated significantly higher than the Wrong ones, irrespective of the material-set (A or B) or the presented explanations (NoBox or Box; all p < 0.05 using t-tests). No other differences between conditions are statistically different, except for one involving the B-set; namely, the correctness rating for the Right Righht--iselasssfication with example-plus-CCH explanation (Box, M=3.04) is reliably higher than that for the example-only explanation (NoBox, M=2.83), t(162) = 1.8, p = 0.036, using I-tailed, two-sample t-test. Note that while this difference appears small Likert scale, overall it constitutes 5. 2.25% % difference in responses, which is larger effect than that found other studies (Goyal et al., 2019). What does this mean? This finding suggests that some ambiguous material items (that people think may be correct, though they are incorrect) are affected by being given an explanation showing CCRs. Fig. 5a(a shows one such example, "Lemon":
picture of a "Pot Pie" (which is decorated with lemons), which the CNN labelled here, the Box explanation shows the NN (an image of a lemon) and boxed region showing the "pulp of the lemon" as the feature that influenced the classification. Clearly, when people see this CCR explanation, it provides more information about the classification, leading them to rate it as more correct (which does not occur for the NoBox version). Though this account makes sense, because this analysis is ad-hoc, it should be treated with some care. Nonetheless, it does show that there are specific image-items that are influenced by the provision of CCR information (presumably, when there is confusion over whether the Al really got the prediction right wrong). Notably, and more generally, these findings give us a sense of the psychological complexities of XAI; namely, that there be interactions between specific image-items, the predictions about those items made by the model, and the explanation-typp used to justify those predictions.
6 GENERAL DISCUSSION AND CONCLUSIONS
Post-hoc poplanation-by-examppe is one of the most popular and successful explanation strategies in both user testing (Buçinca al. 2020 Jeyakumar al. 2020). and computational experiments (Kim) et al. 2014: Kenny& & Keane 2021). This paper has reported comparative tests of several ways to augment this popular explanation method by showing the most important parts of the image used, what we call Critical Classification Regions (CCRs). Initially, computational tests demonstrated the optimal way to isolate these regions in the test image and training data. Subsequently, a care- fully designed user study showed that explanations with and without CCRs are both equally helpful for understanding, but that for certain ambiguous images the provision of CCRs influence people's 's perception of correctness. For future work, the investigation of CCR's 's utility into contrastive expla- nations will be considered, as well as how to link multiple/bigger CCRs the test image to a single training example.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
clear that XAI has made considerable progress in "opening the black box" to begin to reflect the internal workings of these successful, but complex, deep learning models. Here, we have seen that an explanation-by-e example strategy shows some promise in supporting this effort (Jeyakumar et al., 2020 Kenny et al., 2021| Chen et al.) 2019). However, we should also not loose sight of the potentially negative socictal impacts that may arise from these XAI solutions. Perhaps the main concern being that, any data which is presented for explanation may need to be anonymized to ensure privacy and ethical concerns (Jeyakumar et al., (020), particularly in sensitive domains such as medicine and law.
There are also wider issues to be considered around ensuring that these automated explanations in- form without misleading end-users about the systems. Since CCRs gives users the impression that "incorrect" classification's 's seem more correct, a concern could be put forth that they may actively mislead end-users. However, since ImageNet has "real world" images which inevitably contain sev- eral classes (e.g., the pot pie in Fig. 4 could be a "lemon" or "pot pie"), many different possible classifications are arguably correct. So, the CCR is not necessarily "misleading" users, but rather pointing more precisely to where the CNN is looking, and (thanks to the nearest neighbour CCR explanation), "why" it is looking there (our main contribution). When users see the CNN is looking at the "lemon" when it classifies the "pot pie" a "lemon", although some may see this as incorrect, it is nevertheless acting in a way which is objectively "correct", so people's 's feelings of "correctness" increased compared to the "nobox" version (or put another way, they see it as "less incorrect"). In terms of use-cases for CCRs, they appear to be most useful at convincing people a CNN's classi- fication is correct, when there several possible classifications CNN could make. This would likely be useful in domains with many possible outcomes for which humans are not totally expert multi-class classification problem - (c.g., radiology diagnosis, see Fig.
REPRODUCIBILITY STATEMENT
Much effort has been invested to ensure this work is reproducible. For instance, we are providing a code library which can be installed use the CCR method easily. This includes the twin-ensemble method from Section[] Moreover, to reproduce the experiments, we have provided all the code used in the supplementary materials, and all the data pre-processing used in the Appendix sections. The user study is given in full detail also, which shows exactly what we showed users, should anyone wish to recreate the results of this paper.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 9525-9536 2018
Sule Anjomshoae, Amro Najjar, Davide Calvaresi, and Kary Främling. Explainable agents and robots: Results from systematic literature review. In 18th International Conference on Au- tonomous Agents and Multiagens Systems (AAMAS 2019), Montreal, Canada, May 13-17, 2019, International Foundation for Autonomous Agents and Multiagent Systems, 2019.
pp.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus- Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7)ee0130140, 2015.
Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain enn activations better than state-of-the-art feature visualization. In International Conference on Learning Representa- tions, 2020.
Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. In Proceedings of the 25th International Conference Intelligent User Interfaces, pp. 464, 2020.
10

---

Under review as a conference paper at ICLR 2022
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32:8930-89445 8941, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19299, 2020.
Brian Edwards, Joseph J Williams, Dedre Gentner, and Tania Lombrozo. Explanation recruits comparison in category-leariing task. Cognition, 185:21-38, 2019.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learming, pp. 2376-223663 2384. PMLR, 2019.
Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of similarity- ex- planations. In Proceedings of the Ninth International Conference on Learning Representations, 2021.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. Advances in Neural Information Processing Systems, 32: 9737-9748, 2019.
Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava. How can i explain this to you? empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems, 33, 2020.
Mark T Keane and Eoin M Kenny. How case-based reasoning explains neural networks: A theoreti- analysis of xai using post-hoc explanation serlioon----eeee bx-example from a survey of ann-cbr twin-systems. In International Conference on Case- Based Reasoning, pp. 155-171 1711 Springer, 2019.
Mark T Keane, Eoin M Kenny, Eoin Delaney, and Barry Smyth. If only we had better counterfactual explanations: Five key deficits to rectify the evaluation of counterfactual xai techniques. In 30th International Joint Conference on Artificial Intelligence (ICCAT), 2021.
Eoin M Kenny and Mark T Keane. twin-systems to explain artificial neural networks using case- based reasoning: Comparative tests of feature-weighting methods in ann-cbr twins for xai. In Twenty-Eighth International Joint Conferences on Artifical Intelligence (IJCAI), Macao, 10-16
August 2019, pp. 2708 2715, 2019.
Eoin M Kenny and Mark Keane. Explaining deep learning using examples: Optimal feature weighting methods for twin systems using post-hoc, explanation-by-eee Based Systems, 233:107530, 2021.
in xai. Knowledge-
Eoin M Kenny, Courtney Ford, Molly Quinn, and Mark Keane. Explaining black-box classifiers using post-hoc iyllanations-b-exxmmpe The effect of explanations and error-rates in xai user studies. Artificial Intelligence, pp. 103459, 2021.
Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. Advances in neural information processing systems, pp. 1952-1960, 2014.
Been Kim, Oluwasanmi Koyejo, Rajiv Khanna, al. Examples are not enough, learn to criticize! criticism for interpretability. In NIPS, pp. 2280-2288, 2016.
Gary A Klein. Do decision biases explain too much. Human Factors Society Bulletin, 32(5):1-3, 1989.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1885-1894. PMLR, 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar- dataset. online: http://www. cs. foronto. edu/kriz/cifar: himl, 55:5, 2014.
11

---

Under review as a conference paper at ICLR 2022
Zach C. Lipton. The mythos of model interpretability.. Queue, 16(3):30, 2018.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intel- ligence, 267:1-38, 2019.
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607-617, 2020.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765 2018.
Badri Patro and Vinay P Namboodiri. Differential attention for visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7680-7688, 2018.
Tuan D Pham. A comprehensive study on classification of covid- 19 on computed tomography with pretrained convolutional neural networks. Scientific Reports, 10(1):1-3 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, ACM, 2016.
Frode Sørmo, Jörg Cassens, and Agnar Aamodt. Explanation in case-based reasoning-perspectivve perspectives and goals. Artificial Intelligence Review, 24(2):109-143, 2005. 2005.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mman a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
A AN EVALUATION OF EXPLANATION METHODS
The core novelty of this paper is CCRs, but to generate these the first crucial step is to locate a pool of "good" nearest neighbors (NNs) to avoid searching the entire training data. Rather than proposing novel method, we found four candidate explanation-by--et by-example methods that could deliver such pools (Jeyakumar et al., 2020 Papernot & McDaniel, 2018 Hanawa et al. 202T Kenny & Keane, 2019). As aside contribution, competitively tested these four methods to which would use in the main paper's s experiments, this aside experiment is detailed here.
The CCR algorithms proposed in this paper rely obtaining "good" pool of nearest neighbors. That is, the pool obtained via some nearest neighbor algorithm should be linked to the CNN/test image in question are explaining, easily applied any dataset/domain (regardless of the dataset size or complexity), and practical in the "real world". This experiment compares four explanation- by-example techniques: namely, ExMatchina by Jeyakumar et al. (2020), DkNN by Papernot & McDaniel (2018), Grad-Cos-x linspired by Hanawa et al. (2021)] and twin-systems by Kenny & Keane (2019). Following Hanawa et al. (2021). we use the sanity checks proposed by Adebayo al.
(2018) to determine which method is most sensitive to random changes in the network's 's parameters. given method is not sensitive this, it reasonable assume that the model's 's parameters are not being used to generate the explanation, and hence the explanation (i.e., the NN pool) is not linked to the CNN. Recent similar tests found Grad-Cos to be a good technique (Hanawa et al. 202I), but it does not scale to larger datasets, and their tests did not consider the three other methods tested here. Our modified Grad- Cos, called Grad-Cos-x, uses the loss gradients for similarity comparison at the final feature extraction layer in given instance, rather than the entire CNN's parameters as described by Hanawa et al. (202T). Other techniques [c.g.. Influence functions by Koh & Liang 2017) were omitted due them being computationally infeasible on larger datasets like ImageNet.
12

---

Under review as a conference paper at ICLR 2022
A.1 SETUP, DATASETS, AND EVALUATION METRICS
Here, a pre-trained CNN f is considered alongside four different randomized variants frand (see Fig. 2). These variants randomized (1) the CNN's first layer, (2) half the convolutional layers, (3) the output linear layer, and (4) the entire CNN. The evaluation uses the first 500 instances from the test set of
Krizhhekyy et al. 2014) and FashionMNISST Xiao et al. (2017). An overlap set coefficient is used to evaluate the NN pool. Specifically, a
metric called the Szymkiewicz-Simpnn score of 0 means all the NNs are different, and means the are all the same (i.e., the worst score). For each test image, a set of NNs ranging from 1-1000 is iteratively considered. Due to the complexity of ExMatchina (as the original paper suggested assessing similarity in the final convolutional layer rather than the penultimate linear layer, which increases the size of vectors for comparison x49), the evaluation is limited to relatively simple datasets. Furthermore, as the randomization of weights is susceptible to erroneous results, experiments are repeated five times with aggregated results shown in Fig. along with standard error bars.
In addition, we also consider the "Agreement" metric proposed by Kenny & Keane 202T to evaluate the methods. The method adjusts the training data labels to be what the CNN predicted them to be post-training, and then fits a k-NN classifier to the training data. The k-NN is then used to predict the test data and the amount these predictions overlap with the black-box CNN's predictions, the final score. So, if all predictions are identical, an agreement score of 1.0 is recorded (which is the best score). if both systems disagree on every test prediction, a score of 0.0 is recorded (the worst score). Note this k-NN classifier uses L2 distance for twin systems, but cosine similarity for the other three methods (as was proposed by the original authors). The intuition behind this evaluation metric is that the k-NN being used to explain the black box CNN should exactly match the CNN's predictions on test data, otherwise the NNs are not a faithful abstraction of the CNN function we are explaining.
Lastly, we were also inspired to use Spearman's Rank correlation to evaluate the NN pool found as suggested by Hanawa et al. (2021). The distance of the query to all NNs is recorded, and this is evaluated against the new randomized NNs. A score close to means the NN pool is very different after weight randomization (as want), divergent scores mean the opposite.
A.2 RESULTS
Fig. 6 reports the results of the supplementary experiment. Overall, the results show that twin- systems are the best method across all tests. Fig. (( shows the agreement metric with a perfect score of 1.0 for twin-systems, with DKNN, Ex.atchina, and Grad-Cos-x doing worse in that der. Next, Fig. B) shows the Szymkiewicz-Simpson coefficient, with twin-systems again the best method, with most of its NNs changing distance after the weight randomization, compared to the other three methods. Lastly, Fig. [[C) shows results of the Spearman's 's Rank test, with twin-systems and Grad-Cos-a being the best methods, with less divergence in the former than the latter (indicating again that twin twin-systems are the best method).
Why does this happen? As DKNN and ExMatchina do not use weights in the output linear layer, their weight randomization in the Frand variant which randomized the weights in the last linear layer produced no difference. So aggregating the results of all tests showed DkNN and ExMatchina to perform much worse overall. Given the better performance of twin-systems and its claimed psychological validity Kenny et al.) (0211. Expt. 1-3 focused on this method. Lastly, is also worth noting that Grad-Cox-x requires testing labels to derived NNs, a constraint twin-systems do not have. Hence, although Grad-Cos-x also performed well in these tests, we did not consider it, as we wanted method which didn't 't require testing labels to be known pre-hoc, partly it could also be used for self-supervised learning in future research.
A.3 ARCHITECTURES
The architecture used in the Appendix A experiments is show in table For CIFAR-10, the model's first Conv layer expected 3 dimensions (for color) rather than 1 on FAShionMNIST.
13

---

Under review as a conference paper at ICLR 2022
1.0
Saymklewica-Simeen Simp-----oooonn coefitcient
OPA0-0
0.8
6.20
Agreement
(A)
(B)
0.35
0.2
FashionMNiST CIFAR-10
0.0
Twin-Systom
Tn Twa:Sptarr
GradCCas
Lehaccina
Explanation-bby-Exampll Method
Explanation-by-Exampll Method
Number of NNs Considered
Spearman's Rank Correlation
0.75
10
0.50
100
(C)
200
500
0.00
-0.25
-0.75
-1.00
Twin-System
Grad-Cos-X
DKNN
ExMatchina
Explamation-byEExammll Method
Figure 6: Experimental Results: (A) The agreement metric shows twins get a perfect score of 1.0, with DKNN performing second best. (B/C) The different layers of CNNs trained on CIFAR- 10 and FashionMNIST are randomized to determine the sensitivity of the method to model parameters. (B) The Srymkiewicz-Simpson coefficient overlap the randomized CNN overlap with those from the non-randomized CNN, and the fully aggregated results from all tests are shown for both datasets in a bar plot. (C) Spearman's Rank shows a box plot with twins being the best method, with concentrated scores close to zero. Results show that overall twin-systems and Grad-Coss are the most sensitive across layers (note the varying scales on the y-axis). Standard Error bars are shown in (B).
score shows how many NNs retrieved by
A.4 TRAINING HYPERPARAMTERS
Both models for CIFAR-10 and FaST were trained the same way.
Adam optimizer was used with learning rate of 0.01. Learning rate decreased to 90% of it's previous value every epoch. The networks were trained for 20 epochs.
MNIST used data augmentation by padding with random resized cropping 28, random horizontal flips (p=0.5), color jitter (0.1 for brightness, contrast, saturation, and hue) in Pytorch. The network accuracy the test data 90.39%.
CIFAR-10 10 was the same except that random resized cropping was done at 32 pixels. It's accuracy on the testing data 74.3%.
B TWIN-SYSTEMS DETAILS
As it is the blauuaationb--exampee method used in this paper, present brief summary of twin- systems here if it interests the reader. Twin-systems work by considering the final latent represen- tation in ANN (usually the penultimate layer), and fitting k-NN the training data's 's represen-
14

---

Under review as a conference paper at ICLR 2022
Table 1: The CNN architecture used train models in Appendix A.
Expt 1. CNN
Layer
Layer Parameters
Conv2d
8 filters, (5x5), (1x1), padding=2
BatchNorm22
ReLU
Conv2d
16 filters, (5x5), (2x2), padding=2
BatchNorm22
ReLU
Conv2d
32 filters, (5x5), (Ix1), padding=2
BatchNorm22
ReLU
Conv2d
64 filters, (3x3), (2x2), padding=2
BatchNorm2d
ReLU
Conv2d
128 filters, (3x3), (1x1), padding=1
BatchNorm2d
ReLU
GAP
Linear
128, 10
SoftMax
lation at this layer, similar to Papernot & McDaniel (2018). However, twins are different because they weight these activations with weeightng methods. Specifically, a method known as Contributions Oriented Local Explanations (COLE) is typically used for the feature weighting.
What makes COLE interesting is that igmores features which did not contribute to a classification when searching for nearest neighbors, and emphasises those that did, which allows k-NN to find better explanatory training instances. This approach helps ensure that the nearest-neighbor found is predicted in the same class the test instance, and that they both similar features in their classifications (Kenny & Keane 2021).
The weighting vector 7 of some input the network is given by:
09 dx1
0g
21
32
F..
Ox²
Oxn
were (- represents the extracted features in an ANN (note in this paper it is the penultimate layer before the SoftMax output classification layer), and the predicted output neuron class. The contribution scores for all the training data were generated and then fit to an k-NN for the twin- system. For all testing data, the process is repeated and its vector č used search for nearest neighbors with the k-NN twin.
C EXPERIMENT 2: TRAINING HYPERPARAMTERS
For the training ablation studies ResNet50 and ResNet34 was used. ImageNet experiments used the pre-trained ResNer50 model available on Pytorch. CUB-200 fine tuned ResNet34 with the following hyperparamters.
CUB-200. Epochs 500. GPU used. Number of workers was 2. Batch size 12. Data transfor- mations during training were done with Pytorch transforms: RandomResizedCCap(222, Random- Rotation(45), CandommHorizonttlllp(0....53 ColorJiteerbrighneessoo. 126,, saturation=0.5). A multi- plicitive learning rate decay 0.999 was used. The epoch with best test accuracy was used for the model, which was epoch 41.
15

---

Under review as a conference paper at ICLR 2022
ImageNet. This was pre-trained model from PyTorch.
Standard training and testing splits were used for ImageNet, and CUB-200. However, as standard practice, the validation data was used for ImageNet, since the test data labels are unavailable.
Fine- Tuning When finetuning the CNNs, the exact same hyperparamters were used (and the same ones from the original ResNet paper for ImageNet), except that the learning rate was decreased by 10-1 from each methods initial learning rate when training the initial CNN model.
D COMPUTATIONAL COSTS
D.1 HARDWARE
Appendix A experiment had both the FashionMNIST and CIFAR-10 models trained single Nvidia K80 GPU, 2vCPU 2.2GHz, and 13GB RAM. The experiments were subsequently run on MacBook Pro, processor 2.9 GHz Intel Core i5, memory 16 GB 2133 MHz LPDDR3.
Experiments 1-2 were run on a Dell R740XD with an Nvidia V100 (32GB) GPU: 256GB RAM. Storage of the dataset
scratch storage 220TiB.
Computational time for experiments. The Appendix A experiment took 28hrs to run each dataset. Expt. took approximately 48hrs and 6hrs to run for ImageNet and CUB-200, respec- tively, presuming the use of a single Nvidia V100 (32GB) GPU. Expt. 3 took 2 weeks to run the beta experiment ImageNet, one day for the alpha, and days for the comparative tests. CUB-200 by comparison took less than 2 days for all tests.
E ALGORITHM PSEUDO-CODE
Algorithm Latent- Latent-Based-CC Based-CCR Algorithms
Require: f()); CNN
Require: 2;; Test Image
Require: N-); Nearest Neighbor Algorithm (Twin-Systems Recommended)
Require: k; Size of Nearest Neighbor Pool (50 recommended)
Require: g()); CNN model up the final convolutional layer
Require: ar; Alpha hyperparameter (5 recommended)
Require: m()); Activation map algorithm (e.g., CAM)
C₂ IR wh,,d)) <- g(x)
Get Convolutional Output b Get Activation Map Get Pool of k Nearest Neighbors
M2 E IR (h.w) + (Gz) (N.) NN(x)
Select the segment Oigy (I.1,)) with the maximum saliency Mij the test image CCR Wteett
for n4 (N.) do C₂ E (h.w.d) + (zx) for i in range do
for j in range w do
We + O2 E IR (1.1.d)
Record L2 distance = l|we - Wrest 11
end for
end for end for
Select the neighbor n with segment i,j which minimised Eq. II
16

---

Under review as a conference paper at ICLR 2022
Algorithm Superpixel-CCR Algorithm
Require: f();; ANN
Require: an; Test Image
Require: N-):.); Nearest Neighbor Algorithm (Twin-Systems Recommended)
Require: k;; Size Nearest Neighbor Pool (50 recommended)
Require: SA(.);; Superpixel Algorithm (SLIC Recommended)
Require: 3; Beta hyperparameter (infinity recommended)
S F SA(z) (N.) + NN(x)
b Get Superpixel Segments Get Pool of k Nearest Neighbors
for s/ E S. do Set all parts of x! = 5; 0
Pass image though f(-)
Record output logit of the neuron corresponding to the prediction of f(x)
Modify 8, by upsampling it to the / input size whilst maintaining the aspect ratio
Pass the upsampled though
Record the latent representation of the upsampled 8; as 4
end for
Select the segment 8; with the maximum I the test image CCR Wtest-
for ny (N do
S, + SA(r)
for 8; E S, do
Set all parts of n.! = 8; 0
Pass image though f(.)
Record output logit m; of the neuron corresponding to the prediction of f(x)
Modify 8; by upsampling it the f input size whilst maintaining the aspect ratio
Pass the upsampled though
Record the latent representation the upsampled s; as 4
end for end for
Select the neighbor n with segment which minimises Eq.
2
17

---

Under review as a conference paper at ICLR 2022
F EXAMPLE EXPLANATIONS
Here more example explanations are showcased with correct classifications, and incorrect classifi- cations. In addition, a misclassification with three CCRs and multiple NNs (rather than just one) to illustrate this explanation option is shown.
Fig. shows six correct classifications ImageNet by ResNet50, alongside explanation for them. Firstly, the explanation comprises of a nearest neighbor from pool of n candidate cases retrieved (50 in our experiments), which alone already considered "good" explanation by non- experts (Jeyakumar et al. 2020; Kenny et al., 2021). However, the explanations go further by pin- pointing a CCR in the test image which was "learned" from the NN. This type of explanation not only informs the user of what important feature was used in the explanation, but also from where it arose in the first place so it may be further contextualized.
Fig. shows two incorrect classifications in ImageNet from our user study. The first image is "Kimono" misclassified as "Violin", were the CNN confused the pipe in the test image with a violin bow. Fig. [8p shows "Hammer" misclassified as a "Shovel". Here the CNN saw similarity in the test image's 's wooden handle to a previous training image of a "Shovel", and hence classified the image shovel.
Fig. showss another two incorrect classification ImageNet from our user study. Fig. 9 shows an "Acoustic Guitar" which is misclassified an "Electric Guitar". The CNN conflates the fretboard as being indicative to an electric guitar, but it is also important to an acoustic guitar, and hence the misclassificaiton arises. Fig. (9 shows misclassification of "Flute" "Horizontal Bar". The visual similarity of the horizontal bar in the NN image is confused with the wooden flute in the test image.
Fig. I0 shows another way of visualizing CCRs with multiple NNs shown. Specifically, the three most salient CCRs are shown alongside the three closest representations of them in the pool of NNs retrieved. Interestingly, in Fig. I0 the third CCR seems to be confused between a Leopard's 's tail and leg, possibly contributing to the misclassification.
Fig. I1 shows another example of using three latent-based CCRs for the correct diagnosis of Covid-
19 patient's x-ray.
Fig. 12 shows another example of using SP-CCRs to explain misclassification on ImageNet. (A) A "Knot" and (B) "Rifle" are misclassified by ResNet50 apparently due bias in the CNN by asso- ciating, what could be called, "wooden background", and "snowy background" with the respective classes
G USER STUDY SCREENSHOTS
For complete transparency on the user study design, we include screenshots (anonymous during review) for the reviewer's convenience. In this section exactly what all users of our study is shown, page by page, in the exact order. The bulk the questions are omitted for space and clarity, but all original images used in the study may be acquired from our supplementary folder for this paper.
Or, if you prefer, you may sample your materials recreate the study.
18

---

Under review as a conference paper at ICLR 2022
Test Image
Test Image
Explanation-be bxamxplle
Label: Castle
Label: Castle
Label: Traffic Light Classification: Traffic Light
Label: Traffic Light
Classification: Castle
Clessification: Castle
Classification: Traffic Light
CeTTO
TURKISN
Label: Clessification: Cello
Label:
Classification: iPod
Classification: iPod
Clessification: Cello
Label: Park Bench Classification: Park Bench
Label: Park Banch Classification: Park Bench
Label: Jack Lantern Classification: Jack O Lantern
Label: Jack Lantern Classification: Jack O Lantern
Figure 7: Correct examples: Starting from the top left (and going in reading order), correct classifications and a NN explanation showing the CCR used in the classification. Namely, the images show correct classifications of "Castle", "Traffic Light", "iPod", "Cello". "Park Bench", and "Jack O Lantern"
Test Image
Explanation by Example
Test Image
Explanstion by Example
(a)
(b)
Label: Kimono Closti cotofication: Visinn
Label Vieln Dasoflortion Vialin
Label Hommer Clestiicatio:: Shavel
Label: Shovel Clsditiotion
Figure 8: Incorrect examples: (a) "Kimono" is misclassified "Violin". the CCR shows the CNN confused the pipe in the test image as the violinist's 's bow. (b) A "Hammer" is misclassified a "Shovel", the CCR shows the CNN learned focus the wooden handle of shovels when classifying them, which it partly learned from the training image shown.
19

---

Under review as a conference paper at ICLR 2022
Image
Explanation-by Example
image
Explanation by Example
(a)
(b)
Labe Acouslic Buikar Ches ticatinn Electrie Gutter
Labe: Blctric Guilar Chettfiatton Deetric Guiber
Label Hote Cleshiattinn Horbontal Bar
Label Morionntal Bar Dasdication: Nortsontal Bar
Figure 9: More Incorrect examples: (a) An "Acoustic Guitar" is misclassified as an "Electric Guitar", the CCR shows the CNN has learned to associate the guitar's fretboard with electric guitars, and neglected the rest of the image when classifying the test image. (b) A "Flute" is misclassified as "Horizontal Bar", the CCR shows the CNN seems have focused the qualitative similarity between the horizontal bamboo bar in the training image, and the bamboo flute in the test image.
Test Image
CCR
CCR 2
CCR
Figure 10: Multiple CCRs: An incorrect classification of "Leopard" "Cheetah". Three of the most salient CCRs are shown for the test image, alongside their three closest representations in the pool of NNs.
20

---

Under review as a conference paper at ICLR 2022
CCR 1
CCR 2
CCR 3
Figure 11: Multiple CCRs: A correct classification of "Covid-19". Three of the most salient CCRs are shown for the test image, alongside their three closest representations the pool of NNs.
(A) Test Image
Explanation-Bi By-anm--l---eeeeee
(B) Test Image
Explanation-By Ex-mxxllle
Figure 12: Incorrect ImageNet Classifications: SP-CCRs are used explain two misclassification (A) A "Knot" is misclassified as a "Padlock", were the CNN has learnd to associate a "wooden background" feature with the predicted class, causing a bias. (B) "Rifle" is misclassified as "Shovel", where bias in the CNN has learned to associate "snowy background" feature with the class "Shovel".
21

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations Computer Program
Instructions
sereinc
cotute maa ros you particaten.
meepleeo wa
The es roel the mnay uato coep aer progre abves
Passe de time be aluyy you proviealy duay adedd
dulig day
la spyak aure
rulooie will onaooed
Annonymized
22

---

Under review as a conference paper at ICLR 2022
Ierolwoung Explanations of Computer Program
23

---

Under review as a conference paper at ICLR 2022
Eva usting Explanations of Computer Program
The main baok shady to ovekarte a computer progrem that objeats in plokere.
You bo shown olferrett ways oxpiain the progiae's tatelling.
Orgiaal
woample, the progrem shown above pickur, Sturfieh latad nas pickune hea been showv bafoon.
24

---

Under review as a conference paper at ICLR 2022
Evelusting Exp anations a Computer Program
Instruct lona Tealc
Siatt lack, you be shown pic pickre and the label giver by the prograss fur dhis plckone
25

---

Under review as a conference paper at ICLR 2022
Evallading Exp Computar Program
instruction Tosk
lisk hes cazable csplenatone lotkeed the fawe
stteted aljcioa glut thapopp teartes
War to snow en axplonttoo abows perkees with sigtlgeing anglancniss using the bncue pistoes with righagyiing
The auneent coneelleee arelunation Hightight the zooit eact the picture that oer aaed the program when sabelted the object with pelloe hoz.
Bocond Pessdire Eaetanattnn
lolld foceer fon the other plian Par Pelerdle perpi Staetes
See Yellow Rix Highighhing
yellow highighrred saredal
edglist eeplonaaion
26

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations of Computer Program
Instructions: Task
On the some page, you are asked do third task.
titi look you are soked to rate the correctoees the label providee by the program for pictiew (ii Tonk T).
imaage sorrec
you the program's labelling picking
27

---

Under review as a conference paper at ICLR 2022
Evalusting Explanations Computer Progrem
Instruction: Tack
Piitly the same sage you adted to do burth tisk
In Whll feoe will be asked to rate the explinatton de Task 2)
That's ie to wonbore whatter the exparation hegs you anderatand the propeenty lebeling
telpo undicsdeed progrome labelling onjent
Hor, you ahouad aamaa whsater upplraan habo anderaatena prognae''s kitaling pibking ore of the cunter on the naing scsils
28

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations Computer Program
instuctiins
Now, practice the tasks using the examples shown on the next two pages...
29

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations Compuner Frogram
Pracice Your tadio wolk thi
The progiom presenze with pittand
The progion lobeloe this picsun Starish
Expranation: The progiom litllcttre objeet in the sittre (Onginad) the vay bessuse of what toamed foom other peturs (Erp amatoon people label Starlish
Expiration
Tauit 4t
explonation kelps andlerstadd thes paoponts lasal atjject pickuee
O
D
30

---

Under review as a conference paper at ICLR 2022
Evelusting Explanations Computer Program
Prectice Your badoo flook liee thex
preearted
Orginll
Thai
Expiraaion: adbilled object (Onginad what barred from (Explonatio thet
Teak-::
The auplanat on Tuan 2 hilpu anderatand the program's latalling the object e picane
31

---

Under review as a conference paper at ICLR 2022
Evaluating Exclanations a Computer Program
Ineructions
So, you feel you understand what you have to do, please continue with the study and progress through the pages, In order, at your pace.
32

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations a Computer Program
Instructios
Please only take part in this study if you are willing to consider the tasks and your answers to them carefully.
33

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations of Compuner Pragram
Press Next Page Begin the Study.
34

---

Under review as a conference paper at ICLR 2022
Evaluating Explenations Computer Program
Question
The progirn was prsser led with this pidurs
Oripnll
The bohelee pickin Flute
Task
Explenaton:: The propsan lubelied the otpput the picture (Orggnnd thes leuouse what baamnd flom oteer pioture (Expkandttn cocale
Expaestion
Pask
The progrem lebllig of image Took 1a comnet
35

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations Computer Program
Queston
The progrn vsen with
The kdelle the pitare Bledtrc Outter
Explanatoo: The prig lnbeled the object in the picurs (Orginal this way becsuse of what hourned fhom this other pieturs (Diplenatong people libalied Electric Cutter.
Origiall
Explonat
programic tibelling
Task
axplanatioo Thakk helps anolerataaa progaata lielling odject picaare:
36

---

Under review as a conference paper at ICLR 2022
Evaluating Explanations Computer Program
Quadtion
The pagron was preseredd with plane
The prgron kiboleed thi Shovel
Enolenation: The prog pi libe led the eigret in the prctun (Diginal) this way because of what sammed from other picture (epptaaation tadt people Shovet
Original
Teak 3t
programt lnbellig rroge in commer
emplamation underattand late objecttn pidtnee
37

---

Under review as a conference paper at ICLR 2022
Evalueting Explanations a Computer Program
Queeion 4-A
The
Origiaa
The lebeod
Teak 2
Explenatirn traalied diject (Orginal) this of what kerreed from this (Dxpler that
The soplaration Teak hhipar me undaratand the progrants lacall of aqjectt piotune
38

---

Under review as a conference paper at ICLR 2022
Continue 32 Questions.... paper for study layout.
All original materials may be sampled from ImageNet in the supplement.
Recall there are two versions of the study, one were material Set-A A has CCR explanation, and Set-B does not, and then the other study which does the opposite. All incorrect classifications randomized.
Correctly classified (e-g., the snail above) materials were spaced out every questions were not randomized (l.e, question 4, 8, 12... 32)
39

---

Under review as a conference paper at ICLR 2022
Evalusting Explorations Computer Program
Congratulations!
You have completed the study.
40

---

Under review as a conference paper at ICLR 2022
Debrief Page
stubii soing the oodianade aesalaatttonnaa at corputo routo argpoou resonssu willie comaus
Annonymized
sarves ovo
liveul Lattes purtices
lasta the fodding plouta cantact
Annonymized
wast wit
If aeher 38 sccands you aot etoully rndrected to the Pruliic come completio pape, pleas dick the bek below, Yournal ediriced or cld the prems thit
Annonymized
41


Reviews:
[{"review_id": "phk7aRlYjYU", "reviewer": "Reviewer_jgiA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe main reason to accept this paper is empirical results, showing performance on the variants of the proposed method.  The author has evaluated performance on a standard dataset. The problem statement is exciting but not the solution.\n\nWeaknesses: \n\n1. The author has proposed the critical classification regions (CCR) method, but it is not discussed in the paper. How does the proposed method help to find critical regions? Does the region help in explanation? \n\n2. The core idea is not Novel. Nearest-neighbor classification[1,2] already appears as an explanation for visual object recognition. Nearest-neighbor-based explanations are already presented in the literature. So the main contribution is not novel and seems to be very limited. \nIn the past, Patro et al. have proposed an attention-based nearest neighbor technique to highlight specific parts of the image and try to improve attention and classification score. \n\n3. What would be the motivation for the Nearest-neighbor example for the proposed task? Can it highlight specific parts in the classification to enhance the explanation? What is its requirement? Is this the only method to solve the task? The author should explain this.\n\n4. The core idea is not Novel. Nearest-neighbor classification[1,2] already appears as an explanation for visual object recognition. Nearest-neighbor-based explanations are already presented in the literature. So the main contribution is not novel and seems to be very limited.\n\n5. What would be the motivation for the Nearest-neighbor example for the proposed task? Can it highlight specific parts in the classification to enhance the explanation? What is its requirement? Is this the only method to solve the task? The author should explain this.\n\n6. Selection NN for test samples: This part is not discussed with a detailed analysis. The method section is not clearly discussed. The author should rewrite these sections.\n\n7. How does Speeding-up Twin-Systems help improve explanation? If not, please remove that from the method section; it creates confusion.\n\n8. What happens when the method gets a different pattern/object/class in the nearest neighbor? Is there any way to tackle this issue? Have you tried multiple nearest neighbors like K=1,2,3,4…\n\n9. Patro et al.[3] have tried different values of K and used a clustering technique to select the optimal nearest neighbor. They also tried contrastive techniques using positive examples, negative examples to anchor to improve the critical regions. \n\n10. The proposed method has not been compared with the latest explanation methods. The author should provide both quantitative and qualitative comparison results with the latest explanation methods and also visualize the proposed explanation method with the latest methods. \n\n11. The author should explain figure-6 and its attributes; it is not clear from the caption. What do you mean by those plots and the significance of those bars? \n\n12. CUB and Imagenet datasets are mainly focused on single instances. What happens to your algorithm for multiple instance data like MS-COCO images.\n\n\n13. However, the paper misses one of the core aspects of machine learning practice: readability and reproducibility of results. The author should provide an algorithm or pseudocode to reproduce the results, missing in this paper.\n\nRef:\n[1] Chen, George H., and Devavrat Shah. \"Explaining the Success of Nearest Neighbor Methods in Prediction.\" Foundations and Trends® in Machine Learning 10, no. 5-6 (2018): 337-588.\n\n[2] Ma, Wei, Kendall Nowocin, Niraj Marathe, and George H. Chen. \"An interpretable produce price forecasting system for small and marginal farmers in India using collaborative filtering and adaptive nearest neighbors.\" In Proceedings of the Tenth International Conference on Information and Communication Technologies and Development, pp. 1-11. 2019.\n\n[3] Patro, Badri, and Vinay P. Namboodiri. \"Differential attention for visual question answering.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7680-7688. 2018.\n\n", "summary_of_the_paper": "The author focuses on exploring the XAI methods that none of the methods discussed, highlighting specific parts in the classification to enhance the explanation. The author proposes a method, Critical Classification Regions (CCRs), to do this. CCRs use a nearest-neighbor example to highlight similar important parts in the image explanation. The author performed a user study on a subset of the Imagenet to show improvement of the CCR method.\n\n", "summary_of_the_review": "Overall, I do not feel like this paper is above the bar for acceptance because the core idea is not terribly exciting and has a lot of technical issues. \nThe author should rewrite the paper and provide all other details mention in the weakness section.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "oJPaRn4c4F", "reviewer": "Reviewer_zNGM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the wide array of experiments conducted by the author. The manuscript reflects the thought process the authors had while developing their ideas, touching multiple times at the hypotheses they had and the rational behind their algorithmic choices and experiment design.\n\nI like the fact that the authors algorithm help explore different explanations based on different CCRs (Figure 10 and Figure 11).\nDoes the algorithm support finding NNs based on multiple CCRs at once, so that two images can have a high match if they share multiple CCRs? If this is the case, I strongly encourage the authors to provide examples, because such ability helps make their method more generic: Many classification decisions are based on multiple cues in the input image not a single CCR.\n\nI would be helpful if the authors can describe how their solution can be used in practice. In particular, do the users have to pick alpha and beta? The authors provide some analysis on ImageNet and CUB-200, highlighting the compromise between higher values and lower values, and concluding that 1.25 is the optimal value for ImageNet while 1.1 is the optimal value for CUB-200. I do not fully understand how the authors came to these values and whether the users of their algorithms will be able to make informed decisions about these values (especially also due to the computational overhead of scanning a range of values).\n\nI miss a comparison with influence functions (Koh and Liang, ICML'17). The authors mentioned that such comparison would be very computationally expensive, however, there are two algorithms that allow computing influential instances more efficiently, described in:\n- Estimating Training Data Influence by Tracing Gradient Descent (Garima et al., NeurIPS'20)\n- Representer Point Selection for Explaining Deep Neural Networks (Yeh et al, NeurIPS'18)\nI also miss a comparison with counter-factual methods (e.g. Goyal et al., ICML'19).\n\nLanguage / formatting issues:\n- hyperparamter\n- were another => where\n- the only method that (1) use […] and (2) do => (1) uses […] and (2) does not\n- misclassifcation\n- Section 3 determines […] => end with a full stop.\n- was tested on CIFAR-10/ImageNet => on CIFAR-10 and on ImageNet\n- is the regions found are contingent => is that\n- it’s effect\n- [similar to Hooker et al. (2019)] => avoid text within citations.\n- [Kenny and Keane 2021]  add page number instead of tbd.\n\nWording Suggestions:\n- Critical Classification Regions => Classification-Critical Regions?\n- seriously explored / considered => directly? exntesively studied? [seriously might imply unserious efforts].\n- Twin-Systems, Twin Systems, twin-systems, twin-system => use a consistent term, e.g. Twin-Systems Framework (what about an abbreviation?).\n- we found four candidate => ... we identified four candidate ...", "summary_of_the_paper": "The authors proposes an approach to improve explanation-by-example techniques, by picking nearest neighbors (NNs)  based on fine-grained image content that are critical for the classification decisions. Four variations to select these NNs are explored and compared against each other.\nThe main advantage claimed is the causal role of the generated explanation. The authors support this claim via quantitative evaluation and human evaluation.\n", "summary_of_the_review": "The manuscript is heavy on analysis and seems to reflect work-in-progress results rather than solid and generalizable findings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oDupNpzf6f", "reviewer": "Reviewer_emYe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Overall the proposed idea is clear, I find its simplicity a good strength of the proposed method.\n- I find the user-study presented in Section 5 adequate and with a proper level of depth.\n- In addition, it seems that a code library related with the proposed method will be released after publication of the manuscript. This is always welcome from the reproducibility point of view.\n\nMy concerns with the manuscript are the following:\n\n- The novelty of proposed proposed CCRs is reduced, and somewhat incremental. Moreover, it is only limited to explanation-by-example methods.\n\n- To a good extent, the idea behind CCRs of highlighting the regions that link input images with training examples is similar to that of prototype-based explanations, especially the work from [Chen et al., 2019]. In this regard, proper positioning with respect to that family of methods would strengthen the manuscript.\n\n- It is not clear to me what the value on the y-axis of Figure 2 (A & B) refers to. Is that the activation value of the logit related to the predicted class or the ground-truth class? how do you handle the cases when the prediction changes over the evaluation of different segments within the same image?\n\n- Perhaps I missed something, but in Section 4, it seems that 30 versions of the original model are fine-tuned by gradually considering (adding) the superpixels of each image in the order of importance predicted by the explanation method. (Each time a segment is added, the model is fine-tuned. From this procedure, it seems that the evaluation is still more related towards assessing the relevance order in which the regions of the image are ranked by the explanation methods and not really the effect that specific training examples may have on performance.\n- Also, I was wondering if these networks are not trained from scratch, then there is no guarantee that information present in the held-out segments is actually ignored since the model may still contain features related to then.\n\n- Also, in Section 4, the manuscript explicitly links observations made in the experiment conducted therein with causality. Given the fact that the tested model is continuously modified on that experiment (none of the 30 fine-tuned resulting models is the same as the original model), I do not see how analyzing these different models can tell us something regarding causal properties of the original model.\n\n- Proper validation of a method is not a contribution per se but a requirement in place for a method to be accepted/adopted by the community. There do not seem to be any novel protocols proposed in this regard, if that is indeed the  case, this should not be claimed as a novelty.\n\n- Text and legends present in the plots from Figure 2 is not legible. I had difficulties reading it in digital version of the document. Please ensure the text have sufficient size to be readable when printed in standard A4 paper.\n\nReferences\n- Chen et al., This Looks Like That: Deep Learning for Interpretable Image Recognition\nNeurIPS 2019.", "summary_of_the_paper": "The manuscript proposes a method to extend explanation-by-example (aka. exemplar-based explanations) by highlighting the regions that links the test image with example(s) provided as part of the explanation. Towards this goal, different methods to compute Critical Classification Regions (CCR) are proposed where image regions/pixels linking the input with the explanation examples are highlighted.\n\nThe proposed method is validated on the CUB-200 and ImageNet dataset. This is complemented with a user-study.\n", "summary_of_the_review": "While there is novelty in the proposed method,I find it quite limited. Moreover, at this point, I have several doubts regarding the quantitative evaluation, i.e. Sections 3 and 4, of the proposed method (please see my review). Perhaps it is a problem of clarity, but is something to be addressed if the manuscript is to be published as a solid piece of scientific work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QGDgMBZM74k", "reviewer": "Reviewer_sNwm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a nice advancement of the state-of-the-art in XAI. The proposed method is partially incremental as it is based on the composition of existing approaches for the various steps, and their composition happens in a rather simple way. However, I believe that it brings innovation to the field as it seems effective from the evaluation. The weak points are listed in the following. First, the twin-system that is at the basis of the proposal is not properly presented and illustrated. It is impossible to fully understand the paper without reading the papers presenting the twin-system. Second, Experiment 1 is biased because the way adopted to test the importance of the parts of the image, i.e., by image occlusion, is exactly the same used by SP-CCRS. Thus this method has a clear advantage over the others. The authors should adopt a validation fair for both proposals or test them into separate settings. Third, the paper lacks for comparison with a method like the one of Been Kim or Cynthia Rudin. This statement is true for Experiments 1, 2 but also for Experiments 3. In this last case, in fact, I suppose that the examples are extracted with the twin-system. What about examples retrieved by the proposals of Been Kim or Cynthia Rudin? Also, a comparison against counterfactual explanations (see Yash Goyal), would have provided a better picture. Finally, the very interesting case study, unfortunately, does not bring the expected results. I suppose the way the questions are proposed to the users impacted the results as the explanation provided with CCR should be better than those having only examples. I suggest the authors repeat the experiments focusing only on images correctly classified.\n\nMinor issues. I suggest avoiding using k for both the number of classes and the k of the NN algorithm.\nThe number k used in the experiments is not clearly stated as well as which are the different versions of kNN tested.", "summary_of_the_paper": "The paper proposes a local model-specific post-hoc explanation method for image classifiers (CNN) returning as explanations examples suggesting the reasons for the classification together with subparts of the images named critical regions common to the test instance and the examples and responsible for the classification.", "summary_of_the_review": "A very nice idea probably a bit incremental.\nAn experimentation not sound in some aspects.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to improve explanation-by-example by identifying important parts of the image when using nearest engihbor explanations-by-example. Towards this goal, the notion of Critical Classification Regions (CCR) is proposed. The method is tested both computationally and a user study.

The reviewers felt that the paper had interesting ideas, but overall the reviewers agreed that the paper needs more work before being ready for publication: this includes improving the soundness of the empirical evaluations and clarifying the contribution of the paper.
Decision: Reject