Hermitry Ratio: Evaluating the validity of perturbation methods for explainable deep learning

1 INTRODUCTION . In recent years there has been an explosion of explanation methods for artificial intelligence ( XAI ) algorithms , largely due to the fact that AI is used in pretty much every facet of our modern lives . The need for trust and understanding in AI algorithms has never been more important . A popular way to create model explanations is to repeatedly remove features from the input and seeing how this affects the model prediction . Depending on the difference in prediction , this feature is assigned an importance score . The higher the score , the more the removal of this feature ( or set of features ) impacts the model prediction . These XAI methods are commonly referred to as perturbation or feature importance methods and they produce an explanation in the form of a heatmap or a saliency map ( Zeiler and Fergus , 2014 ; Zhou et al. , 2015 ; Ribeiro et al. , 2016 ; 2018 ; Lundberg and Lee , 2017 ) . These methods define model explanation as an importance weighting of all the features in the input data conditioned on the model prediction . For an overview of methods that will be analyzed see Table 1 . Perturbation methods are also used to evaluate the quality of heatmaps , regardless of whether or not the heatmap in question was obtained through a perturbation method ( Hooker et al. , 2019 ; Samek et al. , 2017 ; Kindermans et al. , 2018 ; Petsiuk et al. , 2018 ) . These evaluation methods remove areas from the input specified to be important by the heatmap . The perturbed input is fed to the model and the performance of the model is compared to the performance of the model when the input was not perturbed . The larger the discrepancy in performance , the more correct the heatmap is deemed to be . There are various benefits to perturbation methods : easy to compute , model agnostic and intuitively interpretable to many users . However , as indicated in previous work ( Hooker et al. , 2019 ; Sundararajan et al. , 2017 ) , the perturbation of the input after training may well be presenting the trained model with data that is out-of-distribution ( OOD ) . By perturbing a data sample , we introduce changes to its underlying structure that can be considered OOD . This violates one of the main assumptions when training machine learning ( ML ) models in a supervised manner : The training and evaluation data must come from the same distribution . If the model is fed input that is OOD , then we can not make any conclusions nor give any explanations based on its output . If the perturbation methods are indeed generating OOD data , a reevaluation of their efficacy is required . In our paper we address the questions : Are perturbation methods generating OOD data ? If so , to what extent ? To our knowledge , this is the first study that investigates whether perturbation methods generate OOD data . We focus our investigation to Deep Neural Networks ( DNNs ) trained with supervised learning on image classification tasks . Empirical results indicate that DNNs underperform , relative to their train and test performance , when they are presented with OOD data . This has spawned a whole field of OOD sample detection ( Bulusu et al. , 2020 ; Siegismund et al. , 2020 ) . Contribution . In this paper we address the above stated questions by defining a new concept called hermitry ratio . We demonstrate that using this concept we can quantify and determine to what extent perturbation methods are generating OOD data when applied to DNNs trained on an image classification task using supervised learning . We test our method on four popular perturbation methods applied to three widely used neural architectures for three ImageNet data classes . Our results show that , to some extent , all four perturbation methods generate OOD data regardless of architecture or image class . This questions the general applicability of perturbation methods as well as the conclusions that can be drawn from them . 2 RELATED WORK . Kindermans et al . ( 2019 ) previously investigated the reliability of saliency methods and while these methods also produce in heatmaps , the methods investigated in the paper are not perturbation methods . Previous work rarely addresses the specific issue in question . Hooker et al . ( 2019 ) propose an alternative to basic perturbation that involves re-training the model with a dataset that includes the perturbated data . While their results are good and their method is valid , in practice this method is very time consuming . Sundararajan et al . ( 2017 ) created the integrated gradients method keeping this in mind : `` However , the images resulting from pixel perturbation could be unnatural , and it could be that the scores drop simply because the network has never seen anything like it in training . '' 3 METHODS . 3.1 HERMITRY RATIO . Analogous to the definition of a hermit , we use the term hermitry1 to indicate how close a data sample is to a distribution . The larger the degree of hermitry , the further away from the group the data sample is . A data sample that is OOD is also called a hermit . This concept is illustrated in Figure 1 . Assume the training X and validation Y datasets where xi ∈ X , 0 < i ≤ IX and yi ∈ Y , 0 < i ≤ IY . Let us assume a model M trained on X and its chosen encoding layer as L. We define the J dimensional encoding vector extracted from such a network as hxi =ML ( xi ) , hxi ∈ RJ . To calculate hermitry , we rely on the distance of such an encoding to the training distributions encodings . Let us assume function Dz = D ( X , z ) , Dz ∈ R describing the scalar distance between the distribution of encodings coming from the training dataset and the encodings of a sample z . Following that , we apply the same notion to a sample from the validation dataset to obtain Dyi = D ( X , yi ) and to the whole dataset with DY = D ( X , Y ) , DY ∈ RIY . Given that we have DY , let us assume a threshold T that defines the 95th percentile n = b0.95IY c such that the sorted DY , ( DY ) ′ = { ( DY ) ′i|1 < i ≤ IY ∧ ( DY ) ′i−1 ≤ ( DY ) ′i } satisfies ( DY ) ′n ≤ T . 1To the best of our knowledge there is no alternative term that captures how much a data sample belongs to a distribution . Assuming a perturbation function φ and perturbed validation samples Ŷ = φ ( Y ) , we define the distances of the perturbed validation samples as D̂Y = D ( X , φ ( Y ) ) where D̂Y ∈ RÎY . Using the previously defined threshold T , we define the set of hermits as D̂HY = { D̂Y |D̂Y > T } where D̂HY ∈ R ˆIHY . Given the number of hermits , we define the hermitry ratio H as H = ˆIHY /ÎY . In other words , we want to measure whether a perturbation method generates OOD data . Using hermitry we can measure how far one perturbed sample is from a dataset and determine if that sample is a hermit . However , we are not interested in only a single sample , but a collection of samples perturbed by the perturbation method . In this case it becomes more useful to look at the ratio of OOD perturbed data samples . The hermitry ratio H quantifies the ratio of perturbed samples that are OOD . To determine what is a low H and what is a high H we empirically determined a cutoff value of 0.3 . The reasoning for this is as follows : At H = 0.5 the XAI method generates OOD data for about half of the samples that it perturbs . This is an undesirable situation because about half of the perturbed samples are unreliable . By setting the value to 0.3 we allow for a number of samples to be OOD , which can happen because part of the original validation images are also considered to be OOD , see Appendix C. At 0.3 we say that 70 % of the perturbed samples have to be ID . This cutoff value is used to analyze the results . 3.2 EXPERIMENTS . In total we run 4 × 3 × 3 = 36 experiments to determine whether perturbation methods generate OOD data : 4 XAI methods , 3 models and 3 ImageNet classes . At the core of each experiment , we measure the distance between a perturbed sample encoding and the training class encodings . The perturbed encodings are based on the images in the validation class . This process is illustrated in Figure 2 . Distance metric . We use Mahalanobis distance to implement D ( Lee et al. , 2018 ; Çallı et al. , 2019 ) . D ( X , z ) = √ ( E [ hX ] − hz ) S−1 ( E [ hX ] − hz ) where hX ∈ RIX×J and E [ hX ] ∈ RJ and S = cov [ hX , hX ] where S ∈ RJ×J . We stick to a class-conditional setting where the X and Y are chosen from samples that belong to the same class . Mahalanobis distance is not the only way to measure hermitry . ODIN ( Liang et al. , 2018 ) is another method for detecting OOD data samples . However , ODIN has the drawback that it perturbs the images , thus potentially also generating OOD data in the process . Perturbation methods . We investigate four widely used perturbation methods listed in Table 1 . Here we describe how these methods perturb the input image briefly . For more information and a discussion about these methods see Ras et al . ( 2020 ) . For the first three methods we use the implementations provided by the Captum library ( Kokhlikyan et al. , 2020 ) and for Anchor LIME we use the open source implementation by Ribeiro et al . ( 2018 ) . The parameters used to calculate the attributions using each of these methods are provided in Appendix A. Occlusion sensitivity sweeps a gray patch of fixed size across the image with the purpose of occluding pixel regions . In this paper we use a gray patch of 22× 22 pixels . Kernel SHAP and LIME replace a subset of the image pixels with random pixels present in the data . Anchor-LIME samples regions in the image and replaces the pixels with a uniform color . Examples of perturbed images for each method can be seen in Table 2 . Models . To validate our method we wanted to see if our observations hold across various types of model architectures . We chose three different pre-trained models : ResNet50 ( He et al. , 2016 ) , DenseNet121 ( Huang et al. , 2017 ) and MnasNet1.0 ( Tan et al. , 2019 ) . Each architecture follows a different network building approach . We use model implementations from the PyTorch torchvision library ( Paszke et al. , 2019 ) . Encodings . To measure the distance between a single data point and a distribution of data points , we represent our data as encodings . We obtain these encodings by passing the data through a network and extracting the features at layer L , where layer L is the layer before the classification layer . This layer is chosen because images can contain multiple classes . Ideally , each image only contains the target class , however this is often not the case in ImageNet ( Beyer et al. , 2020 ) . If the perturbation method hides information about the target class , the model tries to predict with the information available for the other class . This will be reflected in the classification layer as the shifting of the logit values towards the predicted class . It becomes impossible to determine if the encoding is OOD because the image was perturbed or because the model is predicting for a different class . Hence we go back to a layer that contains more information , which is layer L before the classification layer . For the experiments we collect 8 types of encoding distances using Equation 3.2 : Train-train , train-val , train-white noise , train-anime , train-LIME , train-Anchor LIME , train-Kernel SHAP , train-occlusion . Train-train is used to validate that the dataset images are close each other . Train-val is used to determine the distance from the train encodings and the validation encodings . Train-val is important because we set the threshold shown in Figure 1 by calculating the value of the 95th percentile of train-val . Note that this value is unique per model and image class and it allows us to compare the perturbation methods with each other . In Figure 3 train-LIME , train-Anchor LIME , train-Kernel SHAP and train-occlusion are plotted as kernel density estimation ( KDE ) plots . We intentionally feed the network what is usually considered OOD data and measure their distance to the train encodings . Train-white noise are the distances between the train encodings and encodings of uniform white noise images . Train-anime are the distances between the train encodings and encodings of a subset of the SFW Danbooru2020 Anime dataset ( Anonymous et al. , 2021 ) . The resulting KDE plots can be found in Appendix C. Dataset . In the experiments in this paper we use a class-conditional distance as described by Lee et al . ( 2018 ) to measure hermitry as opposed to distance to the entire dataset . I.e. , Given class A , class B can be considered OOD . ImageNet images often contain multiple classes ( Beyer et al. , 2020 ) . If an XAI method removes the information for one class , another class may become more likely . Thus we define OOD as not belonging to a specific class and use a class-conditional setting . We use the ImageNet dataset ’ s official validation set to generate the perturbed samples . We run experiments on three arbitrarily chosen image classes : Tennis Ball , Printer and Chocolate Sauce . Examples from the validation dataset are provided in Figure 3 .

Summary: This paper aims to evaluate whether perturbation methods for explainable deep learning generates out-of-the-distribution (OOD) samples which can weaken the explanations provided by these methods. Basically, we would not know if the changes in prediction are due to the removal of some important image feature or because the perturbed sample is too far from the training distribution.   To achieve this goal, they proposed the hermitry ratio which consists of the fraction of perturbed samples that have the Mahalanobis distance to the training set greater than the 95 percentile of the Mahalanobis distances between the the validation and training set measured on deep learning feature vectors. Using this metric they evaluate 4 methods using 3 neural network architecture and 3 classes of imagenet dataset. They concluded that all evaluated perturbation methods generate OOD data regardless of architecture or image class. However, occlusion analysis primarily produces in-distribution perturbations while others produce a significant amount of OOD perturbations.