Hermitry Ratio: Evaluating the validity of perturbation methods for explainable deep learning

Abstract: Perturbation methods are model-agnostic methods used to generate heatmaps to explain black-box algorithms such as deep neural networks. Perturbation methods work by perturbing the input image. However, by perturbing parts of the input image we are changing the underlying structure of the image, potentially generating out-of-distribution (OOD) data. This would violate one of the core assumptions in supervised learning, namely that the train and test data come from the same distribution. 
In this study, we coin the term hermitry ratio to quantify the utility of perturbation methods by looking at the amount of OOD samples they produce. Using this metric, we observe the utility of XAI methods (Occlusion analysis, LIME, Anchor LIME, Kernel SHAP) for image classification models ResNet50, DensNet121 and MnasNet1.0 on three classes of the ImageNet dataset. Our results show that, to some extent, \emph{all} four perturbation methods generate OOD data regardless of architecture or image class. Occlusion analysis primarily produces in-distribution perturbations while LIME produces mostly OOD perturbations. 

Reviews and author responses:

 Thank you for clarifying. Given the other reviews and the authors rebuttal, I decided to keep my score. 

 Dear reviewer, thank you for taking the time and for your comments. 

We think that you make some good points, unfortunately we don't have the opportunity to address these with additional experiments. However, we will take them into account and expand our experiment section in the next version of this paper. 

 Dear reviewer, thank you for taking the time and for your comments. 

Maybe there's a misunderstanding, but intentionally perturbed sample cannot exist in the validation set. 

We understand that even so, not all perturbed samples will be considered OOD. Which is why we allow some flexibility when defining the threshold, by setting it at the 95th percentile. Also we take into account that certain images in the validation dataset may well be OOD. 

We operate in the feature space for a couple of reasons
- to reduce the dimensionality
- not all networks are built equally: whether something is in disitrbution or out depends on the network architecture, which we take into account by operating in feature space.

 Dear reviewer, thank you for your comments. 

We agree with the reviewer that removing parts of the image is exactly the point of perturbation methods -- remove or alter parts of the image such that if important regions are altered the network does dont predict the intended class. 

However, there is a balance between perturbing the image a little and perturbing to the extent that it generates an OOD sample. 

We believe that the latter is not the point of perturbation methods since, if a sample is OOD, we cannot trust the network output anymore. 

 Dear reviewer, thank you for taking the time to read our paper and give useful comments. 

Re. weakness #1: Indeed, this is a limitation of using a distance metric. However, couple of things. First, the dimensionality is greatly reduced by operating in the feature space as opposed of working directly in the image space. So far it's the best we can do. Pretty much all outlier detection papers make use of a distance metric, see ODIN, baseline.

Re weakness #2: See below. 

Why Mahalanobis: generalization of euclidean distance suitable for multi-dimensional settings. It was the go-to option for this because it is based on previous research. In the future other distance metrics can be considered. 

Why difference in amount of perturbation not taken into account: we wanted to investigate realistic use-cases of perturbation methods and show which is why we did not control for amount of perturbation. We understand and acknowledge that it is not the most fair way of comparing methods with each other.

This paper aims to evaluate whether perturbation methods for explainable deep learning generates out-of-the-distribution (OOD) samples which can weaken the explanations provided by these methods. Basically, we would not know if the changes in prediction are due to the removal of some important image feature or because the perturbed sample is too far from the training distribution. 

To achieve this goal, they proposed the hermitry ratio which consists of the fraction of perturbed samples that have the Mahalanobis distance to the training set greater than the 95 percentile of the Mahalanobis distances between the the validation and training set measured on deep learning feature vectors. Using this metric they evaluate 4 methods using 3 neural network architecture and 3 classes of imagenet dataset. They concluded that all evaluated perturbation methods generate OOD data regardless of architecture or image class. However, occlusion analysis primarily produces in-distribution perturbations while others produce a significant amount of OOD perturbations.

 # Strentgh 
- the paper is well written and easy to follow
- the question evaluated and issue raised is very important since black-box methods like deep learning will need explainability tools in order to be accepted in many applications like medical imaging.

# Weakness
- At the end of the day, the proposed hermitry ratio is based on distance computation on a high-dimensional space which suffers from the well-known curse of dimensionality problem. In summary, due to distance concentration, the concept of proximity or similarity of the samples may not be qualitatively relevant in higher dimensions. This fact weakens this metric and the conclusion made from it.
- The occlusion method selects pixels that are neighbors which can only interfere in a few entries of the produced features vector. On the other hand methods like LIME perturb pixels that are spatially distributed in the image. It will affect many features computed due to the hierarchical organization of the convolutions in CNNs. Therefore, these non-neighbors methods by construction modify more the computed features which produce high hermitry ratio. I think this metric should also account for the number of perturbed features in order to better answer the main question of the paper. 

# Minor things:
- the term XAI methods is first used in the abstract and it is not defined.
- the Mahalanobis distance  equation is not numbered This paper investigates an important issue in explainability methods for deep learning which is essential for many applications. However, I am not sure if the metric proposed for that is robust enough for this investigation and if it is fair for all benchmarked methods. I would like to hear from the authors why they have done some decisions like the use of Mahalanobis distance and the difference in the "amount" of perturbation generated by each method.  At least, the authors need to discuss the limitations of their metric and their conclusions in the paper.

The focus of this paper is on perturbation methods as a means of extracting explainable information about a learner.  The authors question the premise underlying perturbation methods as existent in the current literature, and hence introduce a concept they term the hermitry ratio.  This ratio is supposed to "to indicate how close a data sample is to a distribution" (I quote this because it does not make much sense).  Lastly, they present a series of experiments as a means of illustrating their technical contribution. I have major conceptual objections to this work.

- One of the leading motivating factors for the work is stated early on in the abstract: "However, by perturbing parts of the input
image we are changing the underlying structure of the image, potentially generating out-of-distribution (OOD) data."  I do not see how this is a "however", so to speak.  What the authors object to is precisely the point - if an OOD datum is produced, the end result is significantly affected, which is desirable, by design (of perturbation methods).

- The authors write: "Analogous to the definition of a hermit, we use the term hermitry to indicate how close a data sample is to a distribution. " and elaborate further in a footnote: "To the best of our knowledge there is no alternative term that captures how much a data sample belongs to a distribution."  This makes zero sense, the concept of "how much a data sample belongs to a distribution".  Does the height of Dikembe Mutombo (the 2.2m basketball player) "belong" to the distribution of human height any less than an average person?  It is a nonsensical concept.  Yet, what the authors are after is something very simple, something that an undergraduate student should be more than familiar with: model likelihood i.e. p(params|data)

- The paper is also not at all up to date with the latest contributions, most papers cited being a couple of years old or more - a long time in this subfield!  Cooper's "Hierarchical Perturbation for Fast and Robust Explanation of Black Box Models" (2021), which differentiates itself from the literature significantly, for example is not even mentioned. Questionable premises, questionable theory - not a strong candidate for acceptance.

Perturbation methods for explainability of deep networks operate by changing the network's inputs and observing the effects on its output. This paper introduces a new measure, which the authors call hermitry ratio, in order to quantify the amount by which input perturbations produce out of distribution (OOD) samples.  This paper questions the soundness of existing AI explainability methods for deep neural networks. Many methods rely on perturbing the network's inputs and observing/quantifying the effect of the perturbations on downstream layers. Perturbing an input can make it unrealistic and highly unlikely under the data distributions seen during training or testing. The network's performance on this perturbed input is then potentially misleading and can lead explainability methods to draw wrong conclusions. 

The authors propose a new measure, which they name the hermitry ratio to compare the amount out of distribution (OOD) samples produced by several perturbation methods. 

Strengths:
1. The paper is well motivated.
2. The writing is clear (although the last few pages could've been better organized). 

Weaknesses:
1. I am concerned about the impact of the paper's contributions. The proposed method for OOD detection is very simple. It would've been nice to see some theoretical and experimental results on toy datasets to check the validity of the method: for example, you could study the case of 2-dimensional Gaussian data and study/visualize exactly which areas in the plane would correspond to OOD samples under your hermitry ratio definition. 

2. Can you explain why you chose to operate in feature space. The purpose of explainability is partially to understand when the network fails to focus on the relevant parts of the input. In that sense, it's possible to find a realistic perturbed input with completely messed up features  (e.g. adversarial examples). Under your definition this is considered an OOD sample, but it is a realistic sample that can very well exist in the test data, it's just that our current networks are not robust to these perturbations. It is valid for an explainability method to point this out even if the example is OOD in feature space.  I am concerned about the paper's contributions, the proposed method for OOD detection is very simple, it would've been beneficial to include a theoretical analysis using gaussian distributions and perhaps an accompanying experimental section on toy datasets. 

The paper measures a statistic how much image perturbation methods generate outliers. In principle they measure the amount of samples whose distances are above a treshold which is determined by a quantile on in-manifold data. This is a relevant question for the research community. The idea of hermitry ratio can be found under a different name in papers on outlier detection which are based on distance or similarity measures. Effectively it is a thresholding by distance statistics.  
strengths:
an interesting question
well readable.
An extension of this paper has the potential to show deeper results.

weaknesses:
the experimental evidence is limited.
What is missing is more classes for imagenet, also another dataset e.g. mscoco or the like. three classes is insufficient. Also note that these classes might contain actually only rather few training samples relative to other classes in imagenet.
A deeper look across classes may reveal some interesting insight.
e.g. sampling from classes with high and low network accuracy (better intervals across that statistic), and high and low training sample sizes.

Also missing are a deeper evaluation across parameters. e.g. occlusion using different kernel sizes, using a random color, RISE-type occlusion or occlusion using other modifications of a local region than a gray or random color patch. Occlusion itself is not strongly defined.

What is the impact of the results? Method A has a high and method B a low hermitry ratio. How does that impact conclusions drawn ?

The cutoff of 0.3 is arbitrary but it is not really anywhere used expect for sorting a table. in that sense it can be dropped.

Giving thresholding by distance statistics a new name appears to be a bit much but that is not a reason to reject the paper.

Suggestion: measure these ratios for different values of the thresholding quantiles to get a more rich picture of the behaviours. The paper reads as if it is the start of a deeper study but not the finished study itself. Conclusions beyond reporting distance statistics need to be made.

Meta-review: The paper questions perturbation methods as a means of extracting explainable information about a learner, focusing on the hypothesis that perturbation may create out of distribution examples. To this end, the paper introduces the hermitry ratio to detect such out of distribution samples.

The reviewiers have raised the following concerns:
- questionable motivation and premises
- discussed related work outdated, important and relevant recent works are not discussed.
- limited contribution
- limited experimental evidence

The author response did not sufficiently address the concerns of the reviewers. The reviewers agree that the paper should be rejected.

Decision: Reject