Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information
Authors: Yang Zhang
Ashkan Khakzar
Yawei Li
Azade Farshad
Seong Tae Kim
Nassir Navab
Venue: NeurIPS 2021 Poster (2021)
Keywords: Feature attribution
Interpretability
Explainable AI

Abstract: One principal approach for illuminating a black-box neural network is feature attribution, i.e. identifying the importance of input features for the network’s prediction. The predictive information of features is recently proposed as a proxy for the measure of their importance. So far, the predictive information is only identified for latent features by placing an information bottleneck within the network. We propose a method to identify features with predictive information in the input domain. The method results in fine-grained identification of input features' information and is agnostic to network architecture. The core idea of our method is leveraging a bottleneck on the input that only lets input features associated with predictive latent features pass through. We compare our method with several feature attribution methods using mainstream feature attribution evaluation experiments. The code is publicly available.

Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information
Yang Zhang
Ashkan Khakzar
Technical University of Munich ya. zhangStum.c.omm. de
Technical University of Munich ashkan KhakzarQtum, de
Yawei Li LMU University Hospital 9 Munich yawei. 1111110mdd uni-muenchen. de
Azade Farshad Technical University of Munich azade. frade.farsher farshad@tum. de
Seong Tae Kim
Nassir Navab Technical University of Munich nassir navab@tum. de
Kyung Hee University, South Korea st k.ikkkhu.. ac kr
Abstract
One principal approach for illuminating black-box neural network is feature attri- bution, i.e. identifying the importance of input features for the network's prediction. The predictive information of features is recently proposed as proxy for the measure their importance. So far, the predictive information is only identified for latent features by placing an information bottleneck within the network. We pro- pose method to identify features with predictive information in the imput domain. The method results fine-grained identification of input features' information and is agnostic to network architecture. The core idea of our method is leveraging a bottleneck the imput that only lets input features associated with predictive latent features pass through. We compare our method with several feature attribution methods using mainstream feature attribution evaluation experiments. The code is publicly available.
Introduction
Feature attribution identifying the contribution of input features to the output of the neural network function is principal approach for explaining the predictions of black-box neural networks. In recent years, a plethora of feature attribution methods is proposed. The solutions range from axiomatic methods 2 3 derived from game theory 4 to contribution backpropagation methods 5 (9. However, given specific neural network and an input-output pair, existing feature attribution methods show dissimilar results. In order evaluate which method correctly explains the prediction, the literature proposes several attribution evaluation experiments [10 attribution evaluation experiments reveal that methods that look visually interpretable to humans or even methods with solid theoretical grounding are not identifying contributing input features The divergent results of different attribution methods and the insights from evaluation experiments show that the feature attribution problem remains unsolved. Though there is no silver
9. The
B 12 [4].
denotes equal contribution Department of Dermatology and Allergology 'corresponding author http:///icauub.cccc/a/aae/ee/l//
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

bullet for feature attribution, each method is revealing a new aspect of model behavior and provides insights or new tools for getting closer to solving the problem of attribution.
Recently a promising solution Information Bottleneck Attribution (IBA) information theory is proposed. The method explains the prediction via measuring the predictive information of latent features. This is achieved by placing an information bottleneck on the latent features. The predictive information of input features is then approximated via interpolation and averaging of latent features' information. The interpolated information values are considered as the importance of input features for the prediction. One shortcoming with this approach is the variational approximation inherent in the method, leads to an overestimation of information of features when applied earlier layers. Another problem is that the interpolation input dimension and the averaging across channels only approximates the predictive information of input features and is only valid in convolutional neural networks (CNNs) where the feature maps keep the spatial correspondences.
grounded on
In this work, we propose Input[BA to measure the predictive information of input features. To this end, we first search for a bottleneck on the input that only passes input features that correspond to predictive deep features. The correspondence established via a generative model, i.e. by finding bottleneck variable that induces the same distribution of latent features as a bottleneck on the latent features. Subsequently. we use this bottleneck as prior for finding an imput bottleneck that keeps the mutual information with the output. Our methodology measures the information of imput features with the same resolution as the imput dimension. Therefore, the attributions are fine-grained. Moreover, our method does not assume any architecture-specific restrictions. The core idea input bottleneck/mask estimation using deep layers- is the main contribution of this work to feature attribution research (input masking itself is already an established idea [6 17), the novelty is leveraging information of deep features for finding the input mask).
We comprehensively evaluate InputIBA against other methods from different schools of thought. We compare with DeepSHAP M and Integrated Gradients 2 from the school of Shapley value methods, Guided Backpropagation (5] (from backpropagation methods), Extremal perturbations 177 (from perturbation methods), GradCAM [18] (an attention- based method), and the background method. IBA [9). We evaluate the method on visual (ImageNet classification) and natural language processing (IMDB sentiment analysis) tasks and from different perspectives. Sensitivity to parameter randomization is evaluated by Sanity Checks [1]. Evaluating the importance of features is done using Sensitivity-N (9, Remove-and-Re Rnd-Retrain (ROAR) [IO, and Insertion-Delete []]. To quantify
the degree of being fine-grained in visual tasks we propose a localization-based metric, Effective Heat Ratios (EHR).
2 Related Work
2.1 Explaining Predictions via Feature Attribution
We focus explaining the models for single inputs and their local neighborhood. i.c. local explana- tion (20). We categorize attribution methods within the local explanation paradigm. Some methods can belong more than one category such as IBA which leverages perturbation and latent features.
22 linearly approximate the network and propose the gradient as attribution. Deconvolution 23 Guided Backpropagation [5), LRP 77. Excitation Backprop [8].
Backpropagation-baseed: 21
DeepLIFT use modified backpropagation rules.
Shapley value: Considering the network's 's function as a score function, and input features as players, we can assign contributions features by computing the Shapley value. Due to complexity, many approximations such as DeepSHAP [1], Integrated Gradients 2 are proposed.
Perturbation-based These methods perturb the input and observe its effect on the output value [I6] 17 24|. E.g. Extremal Perturbations searches for the largest smooth mask on the input such that the remaining features keep the target prediction. LIME 25 linearly approximates the model around local input perturbations.
Latent Features: CAM/GradCAM 26 18
use the activation values of final convolutional layers. IBA [19 measures the predictive information of latent features. Pathway Gradient 27 leverages
critical pathways.
2

---

Image
IBA(conv4 1)
IBA[conv3 1)
IBA[conv2_] 1)
[BA[conv] 11
AG
V
Figure 1: Effect of P(Z) Approximation (IBA [9] vs InputIBA (A")): We see the result of apply- ing IBA different layers of VGG16 network (from conv4_1 to convl_1), The approximations (averaging across channels) in IBA result in the assignment of information to irrelevant areas of the image (the trashcan and areas around the image). As we move towards earlier layers (convl_) the information is distributed equally between features, and less information is assigned to the most relevant feature (the broom), due the overestimation of mutual information I [R, Z| in IBA. Using our approximation of P(Z) the resulting mask A is representing only the relevant information (the broom) at imput resolution. Aç represents the prior knowledge we use for P(Z). IBA uses the Gaussian distribution (Q(Z) ~ (un, on)) to approximate P(Z).
2.2 Do Explanations Really Explain?
The story starts with Nie al. [33 demonstrating that Deconvolution [3] and Guided Backpropaga- tion [ are reconstructing image features rather than explaining the prediction. Later Adebayo et al. propose sanity check experiments to provide further evidence. They show some methods generate the same attribution when the weights of the model are randomized.
Sixt et al. 14 and Khakzar al. (8 29 provide further proofs and experiments, and add other attribution methods to the circle. It is noteworthy that all these methods generate visually interpretable results. In parallel, several feature importance evaluations are proposed evaluate whether the features recognized as important, are indeed contributing to the prediction. All these experiments are grounded on the intuition that, removing (perturbing) an important feature should affect the output function relatively more than other features. These methods are Remove-and- -ne---ann [T]. Sensitivity-N p Insertion/Deletion [T]] which are introduced in Section [ Interestingly, such evaluations reveal that axiomatic and theoretically iron-clad methods such as Shapley value approximation methods (e.g. Integrated Gradients 2 and DeepSHAP these feature evaluation methods.
[ score relatively low in
3 Methodology
We first introduce the background method, IBA [15] in Section 3.1 We proceed in Section 3.2 with
explaining IBA's shortcomings and proposing our solution. In Section 3.3 and Section 3.4 we explain details of our solution.
3.1 Background Information Bottleneck Attribution (IBA)
15
This method places a bottleneck Z on features R of an intermediate layer by injecting noise to R in order to restrict the flow of information. The bottleneck variable is 2 XR (1 A)c, where e denotes the noise and À controls the injection of the noise. À has the same dimension as R and its elements are in (0, 1]. Given specific input and its corresponding feature map R (R = [(),, function f represents the neural network up to the hidden layer of R), the method aims to remove as many features in R as possible by adding noise to them (via optimizing on A), while keeping the target output. Therefore, only features with predictive information will pass through the bottleneck (parameterized by the mask A). The bottleneck thus optimized such that the mutual information between the features R and noise-injected features Z is reduced while the mutual information between the noise-injected features Z and the target output Y is maximized, i.e.
max YY, Z 31 2]
where,
[[R, 2| E2DD
(2)
3

---

3.2 Approximating the Distribution of Bottleneck P(Z)
The main challenge with computing I[R, Z| is that the distribution P(Z) is not tractable as we need to integrate over all possible values of R (since P(Z) P(Z)R)
Therefore IBA methodology resorts to variational approximation Q(Z) "N TR). The assumption is reasonable for deeper layers of the network [19]. However, as we move toward the input, the assumption leads to an over-estimation of mutual information 7 R, 2] 19. The result of using such an approximation (Z) ~ NV (unn o 7)) is presented in Fig. | for various layers of VGG-16 neural network. As the over-estimation of 7 7[R, Z] increases by moving towards the earlier layers, the optimization in Eq. removes more features with noise. We can that the predictive feature (the broom) in Fig. [ disappears as we move IBA towards the early layers (to conv1_1). The approximation of P(Z) is most accurate when IBA is applied on the deepest hidden layer.
In order to accomplish feature attribution for the input features. IBA method limits itself to convo- lutional architectures. First, it finds the informative features of a hidden layer by solving Eq. ( Let X denote the result of this optimization. IBA interpolates these values to the input dimension (similar to CAM (26)) and averages \ across channel dimension. Such interpolation is reasonable only for convolutional architectures as they keep spatial information. The interpolation and averaging introduce a further approximation into computing predictive information of input features. From this perspective, is desirable to apply IBA to early layers to mitigate the effect of interpolation. At the same time, early layers impose an overestimation of I [R, ZJ. Therefore, there is trade-off between mitigating the adverse effect of interpolation/aavrrgiing and the (() N (UR, RR) approximation.
We aim to come up with a more reasonable choice for Q(Z) such that it is applicable for the input I. This alleviates architecture dependency CNNs, as the mutual information directly computed on the input space and avoids the approximation error which results from interpolation to input dimension and the summation across channels. We take advantage of Q(Z) (unn, on) being reasonable for deep layers. Thus we first find a bottleneck variable Z" parameterized by the optimal result X (Z" R + (1 - )*)c by solving Eq. ( The bottleneck Z restricts the flow of information through the network and keeps deep features with predictive information. Then, search for a bottleneck variable on input, Za, that corresponds to Z* the sense that applying Za on input inflicts Z" on the deep layer. This translates to finding a mask input features that admits input features which correspond to informative deep features. Thus, the goal is to find ZG such that ((f(Za) P(Z*), where function f is the neural network function up to feature map R:
min D ((((((Z))))P(z))
(3)
Xe
where Za= = AgT + (1 AG)EG. and D represents the distance similarity metric. For details of Eq. (3) refer Section 3.3
The resulting Z& (and its corresponding XT) from Eq. 3 is a bottleneck variable on input that keeps input features associated with predictive deep features. The final goal is keeping only predictive features of input (Eq. ([), therefore, Zg can be used as prior knowledge for the distribution of input bottleneck P(Zz). ). To incorprraee this prior, we condition P(Zı) on Zg. i... Z₁ AZG + (1 A) with a Icarnable parameter as the input mask. We then proceed and solve Eq. (I) again, but this time for 7,. The resulting mask from this optimization is denoted by A We refer this methodology resulting in A as InputIBA.
Remark 1 P(Zz) ~ N( + (1 AGA)Er, (1- )z))
Remark 2 P(Zr)Z) ~ N(A + (1 -A)pr, 7))
We explicitly compute D (Z] in order compute mutual information T[Z,, I]. the assumptions and detailed derivation are provided in the Appendix A.
1-Ac, Proposition 3 For each element of input I we have DKL I P(Z),)) = =log
1-Ax
(1-A0) (Ja-ar. AAAAA-A,,,A Ax)
2(1 Az
2(1 2(1-Ac,tAuloj,
3.3 Estimating the Input Bottleneck ZG
The objective is to find a bottleneck variable ZG at the input level that inflicts the distribution P(Z") on the latent layer where Z" is computed. In other words, we search for \G that minimizes

---

image
GradCAM
Guided Extremal Parturbation
DeepSHAP
Integrated Gradients
Figure 2: Qualitative Comparison (ImageNet): We observe that the results of different attribution methods are substantially dissimilar for the same prediction. This is caveat for the community that the attribution problem is far from solved. There is consensus between GradCAM, IBA, and our method (Input[BA) that the ears and the snout are important for the prediction. Feature importance evaluations show that these methods reveal important features. InputBBA, DeepSHAP and Integrated Gradients are fine grained. However, we observe in feature importance experiments that only the InputlBA is attributing to important features among fine ahee-grained methods.
DiP(f(Ze)) where function f is the neural network function before bottleneck Z", and D is the similarity metric. We employ generative adversarial optimization scheme to minimize the distance. Specifically, the generative adversarial model we use tries to fit the two distribution with respect to Wasserstein Distance (D) [0). In this work focus on local explanation, i.e. given sample input, we aim to find an explanation that is valid for the neighbouring area sample input (B) = {a R" d(x, R) < e)). We refer to this set, as the local explanation set. To generate f(Za). we sample / from the local explanation set of the input. Variable Za = Àg + (1 Aa)ec constructed from ÀG P4 and, ag. These are the learnable parameters of our generative model. We apply the reparametrization trick during sampling learnable noise EG (paramcterized by mean NG and standard deviation oG), so that the gradient can be backpropagated
random variable Eg. Once we have the sampled values (L, PG and ac) we get Zg Aç + (1 - AG)EG, we pass this variable through the neural network to get sample of f(Z)). The target distribution that the generator is approximating is the P(Z*). We construct the target dataset by sampling from P(Z*) The generative adversarial model also leverages a discriminator which is learned concurrently with the generator. The discriminator serves to discriminate samples from P(f (Za)) and P(Z*), ), thus its architecture can be determined based on the actual data type of the attribution sample.
3.4 Optimizing I[Y,Z]
Minimizing cross-entropy loss (LoE) is equivalent to maximizing the lower bound of the mutual information [Y, /] [3I]. Thus, in this work and in [19), instead of optimizing Eq. (1) we optimize:
min Lor + 3I
(4)
In this paper, we provide more support for using cross-entropy instead of π(Y. Z) Given the network with parameter set 0 denoted as Po we derive the exact representation 7[Y, Z] instead of the upper bound of T[Y; Z]. Full derivation can be found in Appendix B.
Proposition 4 Denoting the neural network function with &g. we have: /[Y, = f Z) log oo 2) dY dZ + (Dr (Dx))//())DDL(())))..)....
p(Y)
We prove that the minimizer of the cross-entropy loss is the maximizer of the mutual information 7[Y, 2) in the local explanation setting (exact assumption and proof provided in the Appendix C).
Theorem For local explanation (local neighborhood around the input),
arg max f pYY, Z) log PolY12) dY = arg max [[Y, Z]
p(Y)
4 Experiments and Results
Over the course of this section, first we provide the experimental setup in Section 4.1 Then, we present qualitative results in Section 4.2 and check the InputlBA's sensitivity to parameter
randomization in Section 4.3 We proceed with evaluation of the attribution methods in terms of
human-agnostic feature importance metrics (Sensitivity-N [9] Section 4.4.] Insertion/Deletion
M. and ROAR 100). Finally, evaluate the methods in terms of localization Section 4.5 using
5

---

Method
Text (Tokens Divided by Space)
InputIBA
this is easily and clearly the best it features loads of cameos by big named comedic stars of the age solid script, and some great disneyesque songs and blends them together in culmination of the best display of henson 8 talent.
IBA
this is easily and clearly the best.. it features loads of cameos by big named comedic stars of the age, solid script. and some great disneyesque songs and blends them together in culmination of the best display of henson is talent.
LIME
this is easily and clearly the best it features loads of cameos by big named comedic stars of the age solid script. and some great disneyesque songs and blends them together in culmination of the best display of henson is talent.
Integrated Gradients
this is easily and clearly the best it features loads of cameos by big named comedic stars of the age solid script, and some great disneyesque songs and blends them together in culmination of the best display of henson is talent.
Table 1: Qualitative Comparison (IMDB): We compare with IBA and methods that are widely used for NLP model interpretaion. To visualize the attribution, we highlight words that have attribution values from 0.33 to 0.66 with orange, and words that have attribution values from 0.66 to 1 with red (attribution value ranges from 0 to 1). We see that IBA does not translate to RNN architectures when attributing to input tokens. IBA can only identify latent important features. We observe that InputIBA and LIME, and IG attribute to relevant tokens.
our proposed EHR metric. To demonstrate the model-agnoste capability and show that our model is revealing the predictive information, we evaluate our method in both vision and NLP domains. We choose ImageNet 32 image classification for evaluation on vision domain. Apart from image classification tasks on ImageNer/VGG-16, we also apply InputIBA on sentiment classification task (IMDB) and an RNN architecture.
4.1 Experimental Setup
In our experiments on ImageNet, insert the information bottleneck layer conv4_1 of VGG16 [33] pre-trained by Torchvision (34). We use Bteat = 10 for IBA, We adopt Adam (35) as the optimizer with learning rate of 1.0, and optimize the feature bottleneck for 10 optimization steps. We optimize the generative adversary model for 20 epochs with RMSProp 36 optimizer, setting the learning rate The choice of hyper-parmeters parmeters of attribution methods affects their correctness 37 For more details on the prper-parrmeters please refer Appendix
10- 5 For optimizing A, use Sipput 20 and run 60 iterations.
In the NLP experiment, we investigate the sentiment analysis task. A 4-layer LSTM model is trained IMDB dataset (38) which consists of 50000 movie reviews labeled as either "positive" or "negative". After the model is trained, we generate attributions at embedding space. We compare InputIBA with 4 baselines: Integrated Gradients, LIME 220) IBA [15) and random attribution. In order to generate attribution using IBA, we insert IBA at hidden layer and consider the hidden attribution as valid for the embedding space, since RNN does change the size and dimension of input tensor.
We train our model single NVIDIA RTX3090 24GB GPU. For VGG16 33 takes 24 seconds
to obtain the attribution map of one image with resolution 224 x 224.
4.2 Qualitative Comparison
ImageNet Fig. 2 presents the attribution resulting from various methods. IBA and GradCAM are interpolated into input space, thus they are blurry. Guided Backpropagation is reconstructing image features [13,, thus looks interpretable. Extremal Perturbation has lost the shape information of the target object as it applies smooth mask, moreover it has converged to region (dog's 's snout) that sufficient for the prediction. DeepSHAP and Integrated Gradients generate fine-grained attributions, however, the negative/positive (bluc/red) assignment seems random. Moreover, feature

---

image
original
fc3
fc1
conv5 3
0.2
SSIM
0.1
conv5
conv4
conv4
conv3
conv2
fc3
fc1
conv5 3
conv5 1
conv4 3
conv4 1
conv3
conv2
(8)
(b)
The experiments evaluate whether the attribution changes after randomizing the parameters of neural networks. The randomization starts from FC3 layer and moves towards the image (a) Correlation between original attribution and the attribution after randomization (results averaged for ImageNet subset). We observe that the correlation rapidly moves O. The error bars denote 1 standard deviation; (b) Attribution map before randomization (label "original"" and after randomization the parameters up the specified layer. We observe that attribution also randomized after parameters randomized (as opposed to methods such as Guided Backpropagation and LRP (Deep Taylor variant) [12 14).
Figure 3: Parameter Randomization Sanity Check [[]]
importance experiments (Section 4.4)
show the highlighted features are not important for the model. Our proposed method is fine-gr grained and visually interpretable while highlighting important features (Section 4.4). More qualitative examples (randomly selected) are provided in Appendix E.
IMDB Table presents attribution text sample from IMDB. We that IBA fails generate a reasonable explanation by assigning high attribution to all words. The observation also shows that IBA is not model-agnostic method, since IBA assumes the spatial dependency between input and hidden space, which doesn't hold for recurrent neural networks.
4.3 Parameter Randomization Sanity Check [12]
The purpose of this experiment is to check whether the attribution changes if model parameters randomized. If the attribution remains unchanged, then the attribution method is not explaining model behavior. The experiment progressively randomizes the layers starting from the last layer, and generating an attribution at each randomization step. The generated attribution is compared in terms of similarity with the original attribution.
The similarity is measured in terms of structural similarity index metric (SSIM) 39 In this experiment, we randomly select 1000 samples, and compute the average of the SSIM for all the samples. Fig. Bal Fig. 3b demonstrate the SSIM error bars and an example for visual inspection respectively. We observe that the InputIBA is sensitive to this randomization (Fig. 3b).
4.4 Feature Importance Evaluation
The following experiments evaluate whether the features identified as important by each attribution method are indeed important for the model.
4.4.1
Sensitivity-N 9
The experiment randomly masks n pixels and observes the change in the prediction. Subsequently, measures the correlation between this output change and the sum of attribution values inside the mask. The correlation is computed using Pearson Correlation Coefficient (PCC). The experiment is repeated for various values of n. For each value we average the result from several samples. For ImageNet, we run this experiment on 1000 images. We randomly sample 200 index sets for each In the IMDB dataset, texts have different lengths, thus we cannot perturb a fixed number words across samples. We address this issue by slightly changing the method to perturb fixed percentage of words in a text. On ImageNet (Fig. ( the InputIBA outperforms all baseline methods when we perturb more than 2 pixels. When evaluating IMDB dataset, the InputIBA exhibits higher
7

---

Cornetatoon
InputiIA
IBA
Cornetation
0.4
IBA
ImputiBA
0.10
Extrannal Perturbation GradCAM
Integrated Gradients
Guided
Random
0.00
DeepSHAP
Integrated Gradients
0.0
-0.05
10
10
10.
10
0.6
(a)
(b)
Figure 4: Feature Importance Sensitivity-N: (a) ImageNet dataset: Input[BA, IBA and GradCAM show high scores in this feature importance metric. We observe that fine-grained (and also axiomatic) methods such as DAPP and Integrated Gradients achieve relatively low scores. (b) IMDB dataset: Both LIME and InputIBA identify important features according to this metric.
correlation under 50% text perturbation rate, which implies the InputIBA is performing better in not assigning attribution to irrelevant tokens.
4.4.2 Insertion/Deletion
Deletion, successively deletes input elements (pixels/tokens) by replacing them with a baseline value (zero for pixels, <unk> for tokens) according to their attribution score. Insertion, inserts pixels/tokens gradually into a baseline input (image/text). In both experiments we start inserting or deleting the pixel/token with highest attribution. For each image, at each step of Insertion/Deletion the output of the network, and then compute the area under the curve (AUC) value of the output for all steps single input. Then average the AUCs for all inputs (On ImageNet, we average the AUC of 2000 images). For Insertion experiment, higher AUC means that important elements are inserted first. For Deletion experiment, lower AUC means important elements The results are presented in Fig. 55 For the vision task (ImageNet) InputIBA outperforms the rest, Note that the axiomatic methods DeepSHAP and Integrated Gradients achieve low scores both Deletion/Insertion on ImageNet.
compute
deleted first.
4.4.3 Gemooe-and-Retrae (ROAR) 10
One underlying issue with Sensitivity-N and Insertion/Delltion experiments is that the output change may be the result of model not having seen the data during the training. Therefore, ROAR [T0] retrains the model on the perturbed dataset. The more the accuracy drops the more important the perturbed features are. For each attribution method, the perturbation of the dataset done from the most important elements (according attribution values) to the least important element. As retraining the model is required for each attribution method at each perturbation step, the experiment is computationally expensive. Therefore,
this experiment CIFAR10. The results presented in Fig. (6) We can observe two groups of methods. For the first group (DeepSHAP, Integrated Gradients and Guided Backpropagation) the accuracy does not drop until 70% perturbation, meaning 70% of pixels are removed, the model can still have the original performance. For the rest of the methods, we that the features are indeed important. GradCAM M and Extremal perturbations are performing slightly better than IBA and InputBBA. We suspect that this is due to their smooth attribution maps. We test this hypothesis by applying smoothing on InputIBA (InputIBA*) and we observe that the performance becomes similar to the other two. The key observation is that these four methods are successful in identifying important features.
4.5 Quantitative Visual Evaluation via Effective Heat Ratios (EHR)
We quantify the visual alignment between attribution maps and ground truth bounding boxes (on ImageNet (2.). This serves metric for visual/human interpretability. and is a measure of nne-grainedness. Previously [19], the proportion of top-n scored pixels located within the bounding box was computed. We argue that this method only considers the ranks of the pixels, but ignores distribution attribution outside the bounding box. We illustrate the limitation of the previous metric with synthetic examples in Appendix F. Instead, we suggest vary the value of to consider the distribution of the attribution. To this end, we propose EHR, where we consider multiple quantile thresholds from 0.0 to 1.0. At each quantile threshold, compute the sum of attribution within

---

InputiBA
0.80
InputiBA
IBA
Insertion AlIC
IBA
Extremal Perturbation GredCAM
0.75
Insertion
0.3
Integr eled Gradients
Guided
0.70
0.4
See
Integrated dients
0.3
0.05
0.20
0.25
Deletion
Deletion AUC
(a)
(b)
Figure 5: Feature Importance Insertion/Deletion: The closer the method is to the top-left the better (a) ImageNet dataset: InputBA, IBA, Extremal Perturbations and GradCAM all reveal important features according to this metric. DeepSHAP and Integrated Gradients to do not idenfity important features. (b) IMDB dataset: InputBB reveals important features according this metric. The results are consistent with Sensitivity-N.
Method
EHR
InputlBA
0.476 0.007
InputIM
IBA
0.356 + 0.005
Text Accuracy
GradCAM
GradCAM
0.283 + 0.005
60
Integrated Gradients
DHPPSHAP
Guided BP
0.421 = 0.005
Guided BP
Extremal Perturbation
0.421 = 0.007
Extremal Perturbation
InputIBA
DeepSHAP
0.183 + 0.002
Perturbation Rate (S)
Integrated Gradients
0.155 0.002
Figure 6: Feature Importance ROAR: Inte- grated Gradients, DeepSHAP, and Guided Back- propagation not identifying contributing fea- tures. On the other hand, InputIBA, IBA, Ex- tremal Perturbations, and GradCAM point to important features. The latter two methods are performing slightly better, which is due their attributions being smooth (covering more areas). We apply smoothing to InputIBA (InputIBA*) and achieve the same result.
Table Quantitative Visual Evaluation
This metric evaluates how precisely the attributions localize the features by comparing them with ground truth bounding boxes. In- putlBA, extremal perturbations, and Guided Backpropagation
EHR:
highest. IBA and Grad- CAM also perform well, but due to their lower resolution maps, they receive lower scores. Standard error is also presented in the table.
the bounding box divided by the total number of pixels above this threshold, which is defined as the effective heat ratio. Finally, we compute the AUC of the ratios over the quantiles. We assess InputIBA along with other baselines on 1000 images with bounding box covering less than 33% of the whole image. Table Table iillustrates the results.
4.6 Discussion
Although our work is step towards solving the attribution problem, the problem remains open. Therefore the interpretation tools (attribution methods) must be used with caution. The machine learning community is using these tools interpret their results and is deriving conclusions. is not clear if the findings would also show up with another interpretation tool. Wrong interpretations of results arising from the attribution tools can have a destructive effect in mission-critical applications such as medical imaging.
Societal Impact
The community must be aware of the shortcomings of each interpretation tool. For example, [13 14 prove several methods are not explaining the behavior. our work, we also observe that two Shapley value-based methods with solid mathematical and axiomatic grounding, features on computer vision datasets according to all three feature importance evaluations (this also verified for Integrated Gradients in [0]). The Shapley value is hailed as the ultimate solution by many research works, our observation is telling another story. Therefore, our proposed method (and IBA (19] should also be used with caution, even though they grounded theory and performing well in current metrics.
not identifying important
Limitations Although our core idea - finding a bottleneck on input that corresponds to predictive deep features (described in Section 3.3 is simple. the entire methodology for computing the

---

predictive information of input features is relatively complex (compared to method such as CAM 26, [8). This may be an impediment the method's adoption by the community (however, we provide a simple interface in our code).
Another issue is that idea adds additional optimization term (Eq. 33) to IBA, which increases the time complexity. Therefore, our method is introducing a trade-off between improved performance and speed. Nonetheless, for many interpretation applications, speed may not be an issue.
5 Conclusion
In this work, we propose a methodology to identify the input features with high predictive information for the neural network. The method is agnostic network architecture and enables fine-grained attribution as the mutual information is directly computed in the input domain. The improved interpretation technique benefits the general machine learning community. Furthermore, introduces new idea- input bottleneck/maat estimation using deep layers to the attribution research community to build upon and move one step closer to solving the attribution problem.
Acknowledgemenn Ashkan Khakzar and Azade Farshad are supported by Munich Center for Machine Learning (MCML) with funding from the Bundesministerium fur Bildung und Forschung (BMBF) under the project OIIS 18036B. Yawei Li is supported by the German Federal Ministry of Health (2520DAT920). Seong Tae Kim is supported by the IITP Grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021-0-02068.
References
[1] Scott M. Lundberg and Su In Lee. A unified approach interpreting model predictions. In Advances in Neural Information Processing Systems, 2017.
[2] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In 34th International Conference Machine Learning, ICML 2017, 2017.
[3] Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. 37th International Conference on Machine Learning. ICML, 2020.
[4] Lloyd Shapley. A value for n-person games. Contributions the Theory of Games, 2(28):307- 317, 1953.
[5] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striv- ing for simplicity: The all convolutional net. In 3rd International Conference Learning Representations, ICLR 2015 Workshop Track Proceedings, 2015.
[6] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In 34th International Conference on Machine Learning,
ICMI., 2017.
[7] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojeiech Samek, and Klaus Robert Müller, Explaining nonlinear classification decisions with deep Taylor decompo- sition. Pattern Recognition, 2017.
[8] Jianming Zhang, Sarah Adel Bargal. Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 100:4-11011 2018.
[9] Marco Ancona, Enea Ccolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. ICLR, 2018.
[10] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for inter- pretability methods in deep neural networks. Advances Neural Information Processing Systems, pages 9737-9748, 2019.
[11] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus Robert Müller. Evaluating the visualization of what deep neural network has leamed. IEEE Transactions on Neural Networks and Learning Systems, 2017.
[12] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.
10

---

[13] Weili Nie, Yang Zhang, and Ankit Patel. A theoretical explanation for perplexing behaviors of backpropagation basedddd---dddd--- visualizations. International Conference on Machine Learning, 2018.
[14] Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. International Conference on Machine Learning, pages 9046-9057, 2020.
[15] Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2020.
[16] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In ICCV. pages 3429-3437, 2017.
[17] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In ICCV, 2019.
[18] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
[19] Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2020.
[20] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 1135-1144, 2016.
[21] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[22] David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert MÅÄller. How to explain individual classification decisions. Journal of Machine Learning Research, (Junn): ((Jun):1803-11)) 803-1831, 2010.
[23] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-888888 1188-33. Springer, 2014.
[24] Zhongang Qi, Saeed Khorram, and Fuxin Li. Visualizing deep networks by optimizing with integrated gradients. CVPR Workshop, 2019.
[25] Saumitra Mishra, Bob L Sturm, and Simon Dixon. Local interpretable model-agnostic explana- tions for music content analysis. In ISMIR, pages 537-543, 2017.
[26] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-29225 2016.
[27] Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Scong Tae Kim, and Nassir Navab. Neural response interpretation through the lens of critical pathways. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13528-13538, 2021.
[28) Ashkan Khakzar, Soroosh Baselizadeh, and Nassir Navab. Rethinking positive aggregation and propagation of gradients in gradient-based saliency methods. arXiv preprint arXiv:20/2.00366. 2020.
[29] Ashkan Khakzar, Soroosh Baselizadch, Saurabh Khanduja, Seong Tae Kim, and Nassir Navab, Explaining neural networks via perturbing important learned features. arXiv preprint 1rXiv:1911.110811 2, 2019.
[30] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223 PMLR, 2017.
[31] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. ICLR, 2017.
[32] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference computer vision and pullern recognition, pages 248-255. leee, 2009.
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.

---

[34] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACM International Conference on Multimedia, page 1485-1488, 2010.
[35] Adam: A method for stochastic optimization. ICLR, 2015.
[36] T. Tieleman and G. Hinton. Lecture 6.5 RmssPpppp Divide the gradient by a running average
of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
[37]
Naman Bansal, Chirag Agarwal, and Anh Nguyen. Sam: The sensitivity of attribution methods to hyperparameters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[38] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. ACL,, 2011.
[39] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4):600-
612, 2004.
12


Reviews:
[{"review_id": "IRL2pcyEJK-", "reviewer": "Reviewer_auMi", "summary": "Authors build up on a recent attribution technique (IBA), identifies shortcomings and proposes a new technique that utilizes a measure of information for input features (vs latent). New method does not have architecture restrictions.\n\nThey evaluate their method with other types of explanability techniques in a set of image and NLP tasks and propose an additional task/metric with \"effective heat ratio\" (EHR).\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Work is novel, clear, and further the field of XAI. The work is somewhat original (building heavily on IBA). Experiments are extensive enough for image and text data. I am also glad the author have done sanity checks on their technique. I rate the work a 6 (\"Marginally above the acceptance threshold\") and think with a few edits it can be a 7 or 8.\n\nAuthors refer to their method as \"Ours\" or as \"Λ\", authors might want to consider a name for their method so that future work can more easily reference it (InputBA?).\n\nThe authors mention in line 310 that mask estimation is a new idea to the attribution community. This might be overclaimed since GNN explainability techniques will often figure out a mask for the inputs [1].\n\nThe hyperparameters of an explainability technique have a strong influence on their performance [2], authors might want to expand on the hyperparameters of \"their\" technique.\n\nWork could be improved by including other data modalities (tabular data, or graph data [4]) and evaluating them quantitatively [4-8]. In particular, the lens of explanation qualities might be useful to talk about failure modes or strengths of different explainability techniques ([3]).\n \nConfidence Intervals or statistical tests on numerical results would strengthen claims.\n\n\nCitations: \n[1] https://arxiv.org/abs/1903.03894\n[2] https://arxiv.org/abs/2003.08754\n[3] https://link.springer.com/chapter/10.1007/978-3-319-90403-0_9\n[4] https://papers.nips.cc/paper/2020/hash/417fbbf2e9d5a28a855a11894b2e795a-Abstract.html \n[5] https://arxiv.org/abs/1904.11829 \n[6] https://arxiv.org/abs/2003.07258 \n[7] https://arxiv.org/abs/1806.10758 \n[8] https://arxiv.org/abs/1806.10758", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hrs"}, "extra_text": {}}, {"review_id": "HXLkw7DXmMZ", "reviewer": "Reviewer_MU86", "summary": "The paper presents a feature attribution method based on information bottleneck attribution (IBA). While IBA has shown to be an effective method for feature attribution, they are a few limitations such as, variational approximation. The authors mentioned that for lower layers of the network, the variational approximation causes an over-estimation of mutual information which leads to misleading attribution results. The authors proposed a bottleneck variable that keeps the deep features with prediction information and search for input that corresponds to the features in deep layers. \n\nA thorough experimentation is conducted comparing vision and NLP models using various attribution methods belonging to a diverse set of classes. Quantitative results show that their method works better or on-par with the other attribution based methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an extension to information bottleneck attribution. Their proposed method is simple. However, it shows promising results when compared against several alternate methods.\n\nQuestions: \n- Figure 2 shows full face of the object as important feature. I am wondering if the proposed method covers more area of the input? In other words, have a better recall in highlighting input features?\n- Table 1: the results of IBA and integrated gradient are extremely bad. Is there a difference of scale of the attribution value which is causing IBA to highlight everything? For integrated gradient, what is the baseline? The confidence of the prediction score also have an effect on the input feature attribution. Did author look at them to make sure the they are running other methods with the optimal or reasonable settings?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, authors have done a great job in addressing the limitations of their approach.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "DH-rI_deW98", "reviewer": "Reviewer_DbuT", "summary": "This paper introduces a new interpretability method, building on top of the Information Bottleneck Attribution [IBA] literature, by ensuring that the predictive information of the input features is extracted directly, rather than through approximation as in IBA. This method has the advantage that no variational approximation is needed for the input probability distribution, meaning that the interpolation to the input dimension is also not needed. The contribution of this method is that it provides accurate info about the interpretation offered by the input features due to lack of overapproximation. The paper also contains detailed experiments comparing multiple interpretation methods in the literature, using varied metrics and also passes the sanity check for sensitivity to parameter randomization. ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well structured, and gives thorough background and motivation for the problem. It also contains plenty of literature review, allowing readers to understand the progression of the story of interpretability relatively well. The contributions are solid, and the experimental results demonstrate superiority of this method over IBA, and similarity to past methods by other metrics. Questions: \n\n1. Line 124, why is it reasonable to take the optimal result for Q(Z) as opposed to the average result? Can the paper list the disadvantages of taking the optimal result vs the average result? \n2. Line 152 is the first time local explainability is mentioned as a goal. I would expect that to be mentioned earlier in the paper when different interpretation methods are being compared in the literature search, and especially comment on whether different methods are local vs global and put the current work and results in context of the usefulness due to the global vs local nature. \n3. While it is commendable that the paper is comparing work on two tasks -- NLP and vision, it does not seem fair to compare interpretation methods specifically created in the Image domain (GradCAM, Integrated Gradients etc not LIME) when applied to NLP. There should be commentary provided on which methods rely on the spatial correlation depicted in images, to allow the readers to fairly compare the interpretation methods in Vision and NLP. There should also be commentary on the advantages and disadvantages different methods depict when crossing domains. \n4. Section 4.2, it would be helpful to see multiple qualitative egs, and also a rough estimate of the number of egs where the paper’s method seemed to outperform. It will also be helpful to know how the examples were selected, to ensure that no cherry picking was involved. \n5. In Section 4.4, it will be helpful to have information on the advantages/ disadvantages of the different metrics applied, and what it means to evaluate the number of examples considered for each. \n6. I find it odd that the dataset was suddenly changed in 4.4.3, due to computational limitations. I would expect CIFAR evaluation for other metrics as well, if this was necessary. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper somewhat sufficiently addresses the societal impact. However, towards the end of the paper, it is important and helpful to have commentary on the advantages of different interpretation methods (including the one proposed in this paper), so that future work has decent categorization of interpretation methods useful for different tasks. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are all in consensus that the paper is worthy of acceptance.

I want to thank the authors for their extensive responses to the reviews.  It appears as though the reviews will benefit the paper and lead to improvements for the final form.
Decision: Accept (Poster)