Advancing Nearest Neighbor Explanation-by-Example with Critical Classification Regions

1 INTRODUCTION . The impressive success of artificial neural networks ( ANNs ) has led to proposals they should be used in high-stakes applications such as diagnostic radiology for Covid-19 ( Pham , 2020 ) . However , interpretability issues for these models raise significant questions about their feasibility for such use-cases . Accordingly , many eXplainable AI ( XAI ) techniques have been proposed to overcome this , such as saliency maps ( Zhou et al. , 2016 ) and contrastive methods ( Miller , 2019 ) . Here , the focus is on one of the most long-standing XAI techniques , the use of post-hoc explanation-byexample with nearest-neighbors ( Sørmo et al. , 2005 ) , with a view to extending its functionality . Recently , explanation-by-example techniques have been increasingly used to interpret deep learning models ( Lipton , 2018 ; Chen et al. , 2019 ; Jeyakumar et al. , 2020 ; Kenny & Keane , 2021 ) , with interest being bolstered by their psychological plausibility in human decision-making ( Klein , 1989 ) , human category-learning ( Edwards et al. , 2019 ) , and supportive evidence from several user studies ( Borowski et al. , 2020 ; Kenny et al. , 2021 ; Buçinca et al. , 2020 ) . Currently however , these techniques don ’ t connect important “ parts ” used in a prediction , we rectify this here . This work has two main aims : ( 1 ) to extend explanation-by-example ’ s functionality through the use of Critical Classification Regions ( CCRs ) , and ( 2 ) to properly test this novel technique in a suitablycontrolled user study.1 CCRs represent the primary region of importance in a test image and in the nearest-neighbor used for explanation ( e.g. , see Fig . 1 ) . Although many post-hoc XAI techniques for deep learners show salient regions of a test image ( Ribeiro et al. , 2016 ; Bach et al. , 2015 ; Zhou et al. , 2016 ) , to our knowledge , none have seriously considered relating these regions to nearest neighbors in the training data , to show what features were learned by the model and ( more importantly , our core novelty here ) where they arose . In the XAI literature , explanation-by-example is perhaps the most popular XAI method ( Keane & Kenny , 2019 ) , with large user support ( Buçinca et al. , 2020 ) . However , few works address the identification of important regions within example-based explanations . The closest work to ours makes use of Feature Activation Maps ( FAMs ) within the twin-system framework to enhance examplebased explanations ( Kenny & Keane , 2021 ) . FAMs upsample the most positively contributing convolutional feature kernel ( in the last convolutional layer ) and , as such , can be thought of as a variant of Class Activation Maps [ CAMs by Zhou et al . ( 2016 ) ] , which use a combination of all layers . 1Our method is available to pip install as a Python library at https : //after anon review . However , FAMs often fail to isolate a comprehensible “ part ” of each image , and can be hard to contextualize ( see Fig . 1B ) . Moreover , there is no computational tests of FAMs , or user studies . In contrast , we show how CCRs play a key role in classification , and how they affect users in a study . There is also a literature on post-hoc explanations which isolate prototypical examples from training data ( Kim et al. , 2014 ; 2016 ) . These bear a similarity to the present work , but they do not focus on isolating important parts in the classification which we do here . There is also work in the computer vision attention literature such as that by Patro & Namboodiri ( 2018 ) , which also use exemplars to guide feature highlighting . However , this has similar issues to FAMs , and requires a specific ANN architecture for a very specific task , whilst CCRs are agnostic for ANN vision models . Next , this work bears some similarity to Goyal et al . ( 2019 ) , but their work focuses on counterfactual explanations , showcasing an algorithm which modifies the smallest part possible to modify a classification , whilst ours is on factual explanations , showing why a classification was made in the first place ( the two have very different algorithmic goals ) . Finally , note also that none of these methods consider superpixels as we do here . From an implementation standpoint , the current latent-based CCR method ( see Section 2.2 ) is partly inspired by Chen et al . ( 2019 ) , who used a case-based approach to explain a CNN ’ s predictions . However , their approach is a pre-hoc method that compresses the training data down into prototypical parts for explanations , integrating them directly into the CNN ’ s calculations in a forward pass . In contrast , the current method is post-hoc , using the training data for explanation after the prediction is made . Hence , it may be better suited to situations where ( i ) model accuracy is critical , ( ii ) it is simply not feasible to train another more interpretable model , or ( iii ) more diverse explanations ( from the full training data ) are required ( Mothilal et al. , 2020 ) . Finally , the current method builds upon the family of activation-mapping techniques for visualizing the important regions of an image used by a CNN in its predictions ( Zhou et al. , 2016 ; Kenny & Keane , 2021 ) . This paper has four more sections . Section 2 describes how to compute CCRs in explanationby-example . As CCRs can be generated from the latent space ( i.e. , Latent-based CCRs ) , or the pixel space ( i.e. , Pixel-based CCRs ) , Section 3 determines which option works best for the test image . Section 4 performs additional checks on CCRs to help validate their role in classification , and determines the best way to identifying CCRs in nearest-neighbor explanations ( henceforth the NN-CCR ) in the training image . Section 5 reports a user study that tests whether the CCRs found by this method matter to people , before a final discussion ( see Section 6 ) . This paper makes two main contributions : • A new XAI idea that computes CCRs to augment post-hoc , nearest-neighbor explanationby-example . These CCRs connect critical features used in a classification with where they were learned from in the training data ( to e.g . help detect bias and aid in explanation ) . A method for CNNs is proposed ( latent-based CCRs ) , and a “ black-box ” method ( SP-CCRs ) which is ANN agnostic for computer vision tasks . • A novel user study design which is heavily controlled ( e.g. , with counterbalancing ) to specifically test the difference between explanation-by-example , and explanation-byexample with feature highlighting ( i.e. , the CCRs ) . 2 ALGORITHM : FINDING CCRS IN EXPLANATIONS . Taking an explanation-by-example method , our algorithm begins by finding a pool of k nearest neighbors ( we use k = 50 ) , all of which will be searched to find the NN-CCR.2 Two general approaches are considered , one which takes advantage of a typical CNN model ’ s architecture ( i.e. , the Latent-based CCRs next ) , and one which makes no assumptions other than the model being a neural network which extracts a learned representation ( i.e. , Pixel-based CCRs ) . For the algorithm pseudo-code see Appendix E ( omitted due to space constraints ) . An important points to note about CCR methods is that the regions found are contingent on the pool of NNs over which they are computed ( hence , the need for the test of k-NN methods reported in Appendix A ) . 2.1 COMPUTING LATENT-BASED CCRS ( CAM-CCRS , FAM-CCRS , RAND-CCRS ) . This CCR Method takes inspiration from Chen et al . ( 2019 ) , but crucially , in a post-hoc ( rather than their pre-hoc ) manner , specifically , it uses the latent representations of an image in a CNN to gauge similarity . Assume for a given test image I , a final representation C ∈ IR ( h , w , d ) is extracted after all convolutional layers , were h and w represent the height and width of the convolutional output , respectively , and d the number of kernels ( in all current experiments h = w = 7 ) . This may be broken down into regions shaped as h1 ×w1 × d , were h1 < h and w1 < w. These regions may be upsampled to the size of the original image to visualize them as a “ box ” ( e.g. , see Fig . 1 ) , which is the region in pixel-space that corresponds to this region in C.3 Here , we want to select the region in C which is critical to the classification , and use it to elaborate the explanation further . Here we set h1 = w1 = 1 to give more granular detail in the explanation . To select which regions of C are CCRs , we assume the presence of some activation map giving a saliency value to each region in C ( e.g. , FAMs or CAMs ) . Such a technique gives a 7 × 7 saliency map corresponding to C we refer to as Mtest ∈ IR ( 7,7 ) , which gives the importance of each spatial region . Next , by selecting the most positive salient region , we can isolate the test image CCR ωtest ∈ IR ( 1,1 , d ) . Then , with the test-image CCR isolated , the task is to find a similar region from the training data ωnn , giving an example of where this feature was “ learned ” from . Using some nearest neighbor algorithm , a pool of n training-instances are found that are candidate explanations-by-example . In this pool , let C ( n , i , j ) represent some potential region ω ∈ IR ( 1,1 , d ) in the final convolutional layer , with the neighbor indexed by n , and its spatial position in Cn indexed by i and j . These n images have their C representation searched to find the closest match to ωtest using the L2 norm to find ωnn . Importantly however , this region in the NNs is constrained to its relative importance within the instance . Specifically , considering each NN ’ s activation map Mn , only those regions which satisfy the constraint of being higher than max ( Mn ) × α−1 are considered to find ωnn by minimizing : argmin n , i , j ∥ωtest − C ( n , i , j ) ∥2 s.t . M ( n , i , j ) > max ( Mn ) × α−1 . ( 1 ) 2Note we used twin-systems by Kenny & Keane ( 2021 ) as it did well in Appendix A tests , and it works for self-supervised learning ( which may be more general for future research into CCRs ) . 3Note this could be a bigger box , but we choose a small one to give more granularity . However , more CCRs can be shown if desired ( see e.g. , Fig . 11 ) , or SP-CCRs used which can give bigger regions . The alpha constraint is to help ensure ωnn is highly critical to the classification . For M , we experiment using CAM , FAM , and Random maps . So three variants of latent-based CCRs are tested – CAM-CCRs , FAM-CCRs , and Rand-CCRs – although other options exist for retrieving M .

Summary: The author focuses on exploring the XAI methods that none of the methods discussed, highlighting specific parts in the classification to enhance the explanation. The author proposes a method, Critical Classification Regions (CCRs), to do this. CCRs use a nearest-neighbor example to highlight similar important parts in the image explanation. The author performed a user study on a subset of the Imagenet to show improvement of the CCR method.