Advancing Nearest Neighbor Explanation-by-Example with Critical Classification Regions

1 INTRODUCTION . The impressive success of artificial neural networks ( ANNs ) has led to proposals they should be used in high-stakes applications such as diagnostic radiology for Covid-19 ( Pham , 2020 ) . However , interpretability issues for these models raise significant questions about their feasibility for such use-cases . Accordingly , many eXplainable AI ( XAI ) techniques have been proposed to overcome this , such as saliency maps ( Zhou et al. , 2016 ) and contrastive methods ( Miller , 2019 ) . Here , the focus is on one of the most long-standing XAI techniques , the use of post-hoc explanation-byexample with nearest-neighbors ( Sørmo et al. , 2005 ) , with a view to extending its functionality . Recently , explanation-by-example techniques have been increasingly used to interpret deep learning models ( Lipton , 2018 ; Chen et al. , 2019 ; Jeyakumar et al. , 2020 ; Kenny & Keane , 2021 ) , with interest being bolstered by their psychological plausibility in human decision-making ( Klein , 1989 ) , human category-learning ( Edwards et al. , 2019 ) , and supportive evidence from several user studies ( Borowski et al. , 2020 ; Kenny et al. , 2021 ; Buçinca et al. , 2020 ) . Currently however , these techniques don ’ t connect important “ parts ” used in a prediction , we rectify this here . This work has two main aims : ( 1 ) to extend explanation-by-example ’ s functionality through the use of Critical Classification Regions ( CCRs ) , and ( 2 ) to properly test this novel technique in a suitablycontrolled user study.1 CCRs represent the primary region of importance in a test image and in the nearest-neighbor used for explanation ( e.g. , see Fig . 1 ) . Although many post-hoc XAI techniques for deep learners show salient regions of a test image ( Ribeiro et al. , 2016 ; Bach et al. , 2015 ; Zhou et al. , 2016 ) , to our knowledge , none have seriously considered relating these regions to nearest neighbors in the training data , to show what features were learned by the model and ( more importantly , our core novelty here ) where they arose . In the XAI literature , explanation-by-example is perhaps the most popular XAI method ( Keane & Kenny , 2019 ) , with large user support ( Buçinca et al. , 2020 ) . However , few works address the identification of important regions within example-based explanations . The closest work to ours makes use of Feature Activation Maps ( FAMs ) within the twin-system framework to enhance examplebased explanations ( Kenny & Keane , 2021 ) . FAMs upsample the most positively contributing convolutional feature kernel ( in the last convolutional layer ) and , as such , can be thought of as a variant of Class Activation Maps [ CAMs by Zhou et al . ( 2016 ) ] , which use a combination of all layers . 1Our method is available to pip install as a Python library at https : //after anon review . However , FAMs often fail to isolate a comprehensible “ part ” of each image , and can be hard to contextualize ( see Fig . 1B ) . Moreover , there is no computational tests of FAMs , or user studies . In contrast , we show how CCRs play a key role in classification , and how they affect users in a study . There is also a literature on post-hoc explanations which isolate prototypical examples from training data ( Kim et al. , 2014 ; 2016 ) . These bear a similarity to the present work , but they do not focus on isolating important parts in the classification which we do here . There is also work in the computer vision attention literature such as that by Patro & Namboodiri ( 2018 ) , which also use exemplars to guide feature highlighting . However , this has similar issues to FAMs , and requires a specific ANN architecture for a very specific task , whilst CCRs are agnostic for ANN vision models . Next , this work bears some similarity to Goyal et al . ( 2019 ) , but their work focuses on counterfactual explanations , showcasing an algorithm which modifies the smallest part possible to modify a classification , whilst ours is on factual explanations , showing why a classification was made in the first place ( the two have very different algorithmic goals ) . Finally , note also that none of these methods consider superpixels as we do here . From an implementation standpoint , the current latent-based CCR method ( see Section 2.2 ) is partly inspired by Chen et al . ( 2019 ) , who used a case-based approach to explain a CNN ’ s predictions . However , their approach is a pre-hoc method that compresses the training data down into prototypical parts for explanations , integrating them directly into the CNN ’ s calculations in a forward pass . In contrast , the current method is post-hoc , using the training data for explanation after the prediction is made . Hence , it may be better suited to situations where ( i ) model accuracy is critical , ( ii ) it is simply not feasible to train another more interpretable model , or ( iii ) more diverse explanations ( from the full training data ) are required ( Mothilal et al. , 2020 ) . Finally , the current method builds upon the family of activation-mapping techniques for visualizing the important regions of an image used by a CNN in its predictions ( Zhou et al. , 2016 ; Kenny & Keane , 2021 ) . This paper has four more sections . Section 2 describes how to compute CCRs in explanationby-example . As CCRs can be generated from the latent space ( i.e. , Latent-based CCRs ) , or the pixel space ( i.e. , Pixel-based CCRs ) , Section 3 determines which option works best for the test image . Section 4 performs additional checks on CCRs to help validate their role in classification , and determines the best way to identifying CCRs in nearest-neighbor explanations ( henceforth the NN-CCR ) in the training image . Section 5 reports a user study that tests whether the CCRs found by this method matter to people , before a final discussion ( see Section 6 ) . This paper makes two main contributions : • A new XAI idea that computes CCRs to augment post-hoc , nearest-neighbor explanationby-example . These CCRs connect critical features used in a classification with where they were learned from in the training data ( to e.g . help detect bias and aid in explanation ) . A method for CNNs is proposed ( latent-based CCRs ) , and a “ black-box ” method ( SP-CCRs ) which is ANN agnostic for computer vision tasks . • A novel user study design which is heavily controlled ( e.g. , with counterbalancing ) to specifically test the difference between explanation-by-example , and explanation-byexample with feature highlighting ( i.e. , the CCRs ) . 2 ALGORITHM : FINDING CCRS IN EXPLANATIONS . Taking an explanation-by-example method , our algorithm begins by finding a pool of k nearest neighbors ( we use k = 50 ) , all of which will be searched to find the NN-CCR.2 Two general approaches are considered , one which takes advantage of a typical CNN model ’ s architecture ( i.e. , the Latent-based CCRs next ) , and one which makes no assumptions other than the model being a neural network which extracts a learned representation ( i.e. , Pixel-based CCRs ) . For the algorithm pseudo-code see Appendix E ( omitted due to space constraints ) . An important points to note about CCR methods is that the regions found are contingent on the pool of NNs over which they are computed ( hence , the need for the test of k-NN methods reported in Appendix A ) . 2.1 COMPUTING LATENT-BASED CCRS ( CAM-CCRS , FAM-CCRS , RAND-CCRS ) . This CCR Method takes inspiration from Chen et al . ( 2019 ) , but crucially , in a post-hoc ( rather than their pre-hoc ) manner , specifically , it uses the latent representations of an image in a CNN to gauge similarity . Assume for a given test image I , a final representation C ∈ IR ( h , w , d ) is extracted after all convolutional layers , were h and w represent the height and width of the convolutional output , respectively , and d the number of kernels ( in all current experiments h = w = 7 ) . This may be broken down into regions shaped as h1 ×w1 × d , were h1 < h and w1 < w. These regions may be upsampled to the size of the original image to visualize them as a “ box ” ( e.g. , see Fig . 1 ) , which is the region in pixel-space that corresponds to this region in C.3 Here , we want to select the region in C which is critical to the classification , and use it to elaborate the explanation further . Here we set h1 = w1 = 1 to give more granular detail in the explanation . To select which regions of C are CCRs , we assume the presence of some activation map giving a saliency value to each region in C ( e.g. , FAMs or CAMs ) . Such a technique gives a 7 × 7 saliency map corresponding to C we refer to as Mtest ∈ IR ( 7,7 ) , which gives the importance of each spatial region . Next , by selecting the most positive salient region , we can isolate the test image CCR ωtest ∈ IR ( 1,1 , d ) . Then , with the test-image CCR isolated , the task is to find a similar region from the training data ωnn , giving an example of where this feature was “ learned ” from . Using some nearest neighbor algorithm , a pool of n training-instances are found that are candidate explanations-by-example . In this pool , let C ( n , i , j ) represent some potential region ω ∈ IR ( 1,1 , d ) in the final convolutional layer , with the neighbor indexed by n , and its spatial position in Cn indexed by i and j . These n images have their C representation searched to find the closest match to ωtest using the L2 norm to find ωnn . Importantly however , this region in the NNs is constrained to its relative importance within the instance . Specifically , considering each NN ’ s activation map Mn , only those regions which satisfy the constraint of being higher than max ( Mn ) × α−1 are considered to find ωnn by minimizing : argmin n , i , j ∥ωtest − C ( n , i , j ) ∥2 s.t . M ( n , i , j ) > max ( Mn ) × α−1 . ( 1 ) 2Note we used twin-systems by Kenny & Keane ( 2021 ) as it did well in Appendix A tests , and it works for self-supervised learning ( which may be more general for future research into CCRs ) . 3Note this could be a bigger box , but we choose a small one to give more granularity . However , more CCRs can be shown if desired ( see e.g. , Fig . 11 ) , or SP-CCRs used which can give bigger regions . The alpha constraint is to help ensure ωnn is highly critical to the classification . For M , we experiment using CAM , FAM , and Random maps . So three variants of latent-based CCRs are tested – CAM-CCRs , FAM-CCRs , and Rand-CCRs – although other options exist for retrieving M .

Summary: The manuscript proposes a method to extend explanation-by-example (aka. exemplar-based explanations) by highlighting the regions that links the test image with example(s) provided as part of the explanation. Towards this goal, different methods to compute Critical Classification Regions (CCR) are proposed where image regions/pixels linking the input with the explanation examples are highlighted.  The proposed method is validated on the CUB-200 and ImageNet dataset. This is complemented with a user-study.