Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information
Year: 2022

Abstract:
One principal approach for illuminating a black-box neural network is feature attribution, i.e. identifying the importance of input features for the network’s prediction. The predictive information of features is recently proposed as a proxy for the measure of their importance. So far, the predictive information is only identified for latent features by placing an information bottleneck within the network. We propose a method to identify features with predictive information in the input domain. The method results in fine-grained identification of input features’ information and is agnostic to network architecture. The core idea of our method is leveraging a bottleneck on the input that only lets input features associated with predictive latent features pass through. We compare our method with several feature attribution methods using mainstream feature attribution evaluation experiments. The code 1 is publicly available.

heading: 1 Introduction
text: Feature attribution – identifying the contribution of input features to the output of the neural network function – is one principal approach for explaining the predictions of black-box neural networks. In recent years, a plethora of feature attribution methods is proposed. The solutions range from axiomatic methods [1, 2, 3] derived from game theory [4] to contribution backpropagation methods [5, 6, 7, 8, 9]. However, given a specific neural network and an input-output pair, existing feature attribution methods show dissimilar results. In order to evaluate which method correctly explains the prediction, the literature proposes several attribution evaluation experiments [10, 11, 12, 9]. The attribution evaluation experiments reveal that methods that look visually interpretable to humans or even methods with solid theoretical grounding are not identifying contributing input features [13, 12, 14]. The divergent results of different attribution methods and the insights from evaluation experiments show that the feature attribution problem remains unsolved. Though there is no silver
∗denotes equal contribution Department of Dermatology and Allergology †corresponding author 1https://github.com/CAMP-eXplain-AI/InputIBA
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
bullet for feature attribution, each method is revealing a new aspect of model behavior and provides insights or new tools for getting closer to solving the problem of attribution.
Recently a promising solution – Information Bottleneck Attribution (IBA) [15] – grounded on information theory is proposed. The method explains the prediction via measuring the predictive information of latent features. This is achieved by placing an information bottleneck on the latent features. The predictive information of input features is then approximated via interpolation and averaging of latent features’ information. The interpolated information values are considered as the importance of input features for the prediction. One shortcoming with this approach is the variational approximation inherent in the method, leads to an overestimation of information of features when applied to earlier layers. Another problem is that the interpolation to input dimension and the averaging across channels only approximates the predictive information of input features and is only valid in convolutional neural networks (CNNs) where the feature maps keep the spatial correspondences.
In this work, we propose InputIBA to measure the predictive information of input features. To this end, we first search for a bottleneck on the input that only passes input features that correspond to predictive deep features. The correspondence is established via a generative model, i.e. by finding a bottleneck variable that induces the same distribution of latent features as a bottleneck on the latent features. Subsequently, we use this bottleneck as a prior for finding an input bottleneck that keeps the mutual information with the output. Our methodology measures the information of input features with the same resolution as the input dimension. Therefore, the attributions are fine-grained. Moreover, our method does not assume any architecture-specific restrictions. The core idea – input bottleneck/mask estimation using deep layers – is the main contribution of this work to feature attribution research (input masking itself is already an established idea [16, 17], the novelty is leveraging information of deep features for finding the input mask).
We comprehensively evaluate InputIBA against other methods from different schools of thought. We compare with DeepSHAP [1] and Integrated Gradients [2, 3] from the school of Shapley value methods, Guided Backpropagation [5] (from backpropagation methods), Extremal perturbations [17] (from perturbation methods), GradCAM [18] (an attention-based method), and the background method, IBA [19]. We evaluate the method on visual (ImageNet classification) and natural language processing (IMDB sentiment analysis) tasks and from different perspectives. Sensitivity to parameter randomization is evaluated by Sanity Checks [12]. Evaluating the importance of features is done using Sensitivity-N [9], Remove-and-Retrain (ROAR) [10], and Insertion-Deletion [11]. To quantify the degree of being fine-grained in visual tasks we propose a localization-based metric, Effective Heat Ratios (EHR).

heading: 2 Related Work

heading: 2.1 Explaining Predictions via Feature Attribution
text: We focus on explaining the models for single inputs and their local neighborhood, i.e. local explanation [20]. We categorize attribution methods within the local explanation paradigm. Some methods can belong to more than one category such as IBA which leverages perturbation and latent features.
Backpropagation-based: [21, 22] linearly approximate the network and propose the gradient as attribution. Deconvolution [23], Guided Backpropagation [5], LRP [7], Excitation Backprop [8], DeepLIFT[6] use modified backpropagation rules.
Shapley value: Considering the network’s function as a score function, and input features as players, we can assign contributions to features by computing the Shapley value. Due to complexity, many approximations such as DeepSHAP [1], Integrated Gradients [2, 3] are proposed.
Perturbation-based: These methods perturb the input and observe its effect on the output value [16, 17, 24]. E.g. Extremal Perturbations[17] searches for the largest smooth mask on the input such that the remaining features keep the target prediction. LIME [25] linearly approximates the model around local input perturbations.
Latent Features: CAM/GradCAM [26, 18] use the activation values of final convolutional layers. IBA [19] measures the predictive information of latent features. Pathway Gradient [27] leverages critical pathways.

heading: 2.2 Do Explanations Really Explain?
text: The story starts with Nie et al. [13] demonstrating that Deconvolution [23] and Guided Backpropagation [5] are reconstructing image features rather than explaining the prediction. Later Adebayo et al. [12] propose sanity check experiments to provide further evidence. They show some methods generate the same attribution when the weights of the model are randomized.
Sixt et al. [14] and Khakzar et al. [28, 29] provide further proofs and experiments, and add other attribution methods to the circle. It is noteworthy that all these methods generate visually interpretable results. In parallel, several feature importance evaluations are proposed to evaluate whether the features recognized as important, are indeed contributing to the prediction. All these experiments are grounded on the intuition that, removing (perturbing) an important feature should affect the output function relatively more than other features. These methods are Remove-and-Retrain [10], Sensitivity-N [9], Insertion/Deletion [11], which are introduced in Section 4. Interestingly, such evaluations reveal that axiomatic and theoretically iron-clad methods such as Shapley value approximation methods (e.g. Integrated Gradients [2, 3] and DeepSHAP [1]) score relatively low in these feature evaluation methods.

heading: 3 Methodology
text: We first introduce the background method, IBA [15] in Section 3.1. We proceed in Section 3.2 with explaining IBA’s shortcomings and proposing our solution. In Section 3.3 and Section 3.4 we explain details of our solution.

heading: 3.1 Background - Information Bottleneck Attribution (IBA) [15]
text: This method places a bottleneck Z on features R of an intermediate layer by injecting noise to R in order to restrict the flow of information. The bottleneck variable is Z = λR + (1− λ) , where denotes the noise and λ controls the injection of the noise. λ has the same dimension as R and its elements are in [0, 1]. Given a specific input I and its corresponding feature map R (R = f(I), function f represents the neural network up to the hidden layer of R), the method aims to remove as many features in R as possible by adding noise to them (via optimizing on λ), while keeping the target output. Therefore, only features with predictive information will pass through the bottleneck (parameterized by the mask λ). The bottleneck is thus optimized such that the mutual information between the featuresR and noise-injected features Z is reduced while the mutual information between the noise-injected features Z and the target output Y is maximized, i.e.
max λ
I[Y,Z]− βI[R,Z] (1)
where, I[R,Z] = ER[DKL[P (Z|R)||P (Z)]] (2)

heading: 3.2 Approximating the Distribution of Bottleneck P(Z)
text: The main challenge with computing I[R,Z] is that the distribution P (Z) is not tractable as we need to integrate over all possible values of R (since P (Z) = ∫ P (Z|R)P (R)dR). Therefore IBA methodology resorts to variational approximationQ(Z) ∼ N (µR, σR). The assumption is reasonable for deeper layers of the network [19]. However, as we move toward the input, the assumption leads to an over-estimation of mutual information I[R,Z] [19]. The result of using such an approximation (Q(Z) ∼ N (µR, σR)) is presented in Fig. 1 for various layers of a VGG-16 neural network. As the over-estimation of I[R,Z] increases by moving towards the earlier layers, the optimization in Eq. (1) removes more features with noise. We can see that the predictive feature (the broom) in Fig. 1 disappears as we move IBA towards the early layers (to conv1_1). The approximation of P (Z) is most accurate when IBA is applied on the deepest hidden layer.
In order to accomplish feature attribution for the input features, IBA method limits itself to convolutional architectures. First, it finds the informative features of a hidden layer by solving Eq. (1). Let λ∗ denote the result of this optimization. IBA interpolates these values to the input dimension (similar to CAM [26]) and averages λ∗ across channel dimension. Such interpolation is reasonable only for convolutional architectures as they keep spatial information. The interpolation and averaging introduce a further approximation into computing predictive information of input features. From this perspective, it is desirable to apply IBA to early layers to mitigate the effect of interpolation. At the same time, early layers impose an overestimation of I[R,Z]. Therefore, there is a trade-off between mitigating the adverse effect of interpolation/averaging and the Q(Z) ∼ N (µR, σR) approximation. We aim to come up with a more reasonable choice for Q(Z) such that it is applicable for the input I . This alleviates architecture dependency to CNNs, as the mutual information is directly computed on the input space and avoids the approximation error which results from interpolation to input dimension and the summation across channels. We take advantage of Q(Z) ∼ N (µR, σR) being reasonable for deep layers. Thus we first find a bottleneck variable Z∗ parameterized by the optimal result λ∗ (Z∗ = λ∗R + (1 − λ∗) ) by solving Eq. (1). The bottleneck Z∗ restricts the flow of information through the network and keeps deep features with predictive information. Then, we search for a bottleneck variable on input, ZG, that corresponds to Z∗ in the sense that applying ZG on input inflicts Z∗ on the deep layer. This translates to finding a mask on input features that admits input features which correspond to informative deep features. Thus, the goal is to find ZG such that P (f(ZG)) = P (Z
∗), where function f is the neural network function up to feature map R: min λG D[P (f(ZG))||P (Z∗)] (3)
where ZG = λGI + (1 − λG) G, and D represents the distance similarity metric. For details of Eq. (3) refer to Section 3.3.
The resulting Z∗G (and its corresponding λ ∗ G) from Eq. (3) is a bottleneck variable on input I that keeps input features associated with predictive deep features. The final goal is keeping only predictive features of input (Eq. (1)), therefore, ZG can be used as prior knowledge for the distribution of input bottleneck P (ZI). To incorporate this prior, we condition P (ZI) on ZG, i.e. ZI = ΛZG + (1− Λ) with a learnable parameter Λ as the input mask. We then proceed and solve Eq. (1) again, but this time for ZI . The resulting mask from this optimization is denoted by Λ∗. We refer to this methodology resulting in Λ∗ as InputIBA.
Remark 1 P (ZI) ∼ N (λGΛI + (1− λGΛ)µI , (1− λGΛ)2σ2I )
Remark 2 P (ZI |I) ∼ N (ΛI + (1− Λ)µI , (1− Λ)2σ2I )
We can explicitly compute DKL[P (ZI |I)||P (ZI)] in order to compute mutual information I[ZI , I], the assumptions and detailed derivation are provided in the Appendix A.
Proposition 3 For each element k of input I we haveDKL[P (ZI,k|Ik)||P (ZI,k)] = log 1−λG,kΛk1−Λk + (1−Λk)2
2(1−λG,kΛk)2 + (Ik−µI,k)2(Λk−λG,kΛk)2 2(1−λG,kΛk)2σ2I,k − 12

heading: 3.3 Estimating the Input Bottleneck ZG
text: The objective is to find a bottleneck variable ZG at the input level that inflicts the distribution P (Z∗) on the latent layer where Z∗ is computed. In other words, we search for λG that minimizes
D[P (f(ZG))||P (Z∗)], where function f is the neural network function before bottleneck Z∗, and D is the similarity metric. We employ a generative adversarial optimization scheme to minimize the distance. Specifically, the generative adversarial model we use tries to fit the two distribution with respect to Wasserstein Distance (D) [30]. In this work we focus on local explanation, i.e. given a sample input, we aim to find an explanation that is valid for the neighbouring area of sample input (B (R) := {x ∈ Rn : d(x,R) < }). We refer to this set, as the local explanation set. To generate f(ZG), we sample I from the local explanation set of the input. Variable ZG = λGI + (1− λG) G is constructed from λG, µG and, σG. These are the learnable parameters of our generative model. We apply the reparametrization trick during sampling learnable noise G (parameterized by mean µG and standard deviation σG), so that the gradient can be backpropagated to random variable G. Once we have the sampled values (I , µG and σG) we get ZG = λGI + (1 − λG) G, we pass this variable through the neural network to get a sample of f(ZG). The target distribution that the generator is approximating is the P (Z∗). We construct the target dataset by sampling from P (Z∗). The generative adversarial model also leverages a discriminator which is learned concurrently with the generator. The discriminator serves to discriminate samples from P (f(ZG)) and P (Z∗), thus its architecture can be determined based on the actual data type of the attribution sample.

heading: 3.4 Optimizing I[Y,Z]
text: Minimizing cross-entropy loss (LCE) is equivalent to maximizing the lower bound of the mutual information I[Y, Z] [31]. Thus, in this work and in [19], instead of optimizing Eq. (1) we optimize:
min λ LCE + βI[R,Z] (4)
In this paper, we provide more support for using cross-entropy instead of I[Y,Z]. Given the network with parameter set θ denoted as Φθ, we derive the exact representation of I[Y,Z] instead of the upper bound of I[Y,Z], Full derivation can be found in Appendix B.
Proposition 4 Denoting the neural network function with Φθ, we have: I[Y, Z] = ∫ p(Y,Z) log Φθ(Y |Z)p(Y ) dY dZ + EZ∼p(Z)[DKL[p(Y |Z)||Φθ(Y |Z)]]
We prove that the minimizer of the cross-entropy loss is the maximizer of the mutual information I[Y, Z] in the local explanation setting (exact assumption and proof provided in the Appendix C).
Theorem 5 For local explanation (local neighborhood around the input), arg max ∫ p(Y,Z) log Φθ(Y |Z)p(Y ) dY dZ = arg max I[Y, Z]

heading: 4 Experiments and Results
text: Over the course of this section, first we provide the experimental setup in Section 4.1. Then, we present qualitative results in Section 4.2, and check the InputIBA’s sensitivity to parameter randomization in Section 4.3. We proceed with evaluation of the attribution methods in terms of human-agnostic feature importance metrics (Sensitivity-N [9] in Section 4.4.1, Insertion/Deletion [11], and ROAR [10]). Finally, we evaluate the methods in terms of localization Section 4.5 using
our proposed EHR metric. To demonstrate the model-agnostic capability and show that our model is revealing the predictive information, we evaluate our method in both vision and NLP domains. We choose ImageNet [32] image classification for evaluation on vision domain. Apart from image classification tasks on ImageNet/VGG-16, we also apply InputIBA on a sentiment classification task (IMDB) and an RNN architecture.

heading: 4.1 Experimental Setup
text: In our experiments on ImageNet, we insert the information bottleneck at layer conv4_1 of VGG16 [33] pre-trained by Torchvision [34]. We use βfeat = 10 for IBA. We adopt Adam [35] as the optimizer with learning rate of 1.0, and optimize the feature bottleneck for 10 optimization steps. We optimize the generative adversary model for 20 epochs with RMSProp [36] optimizer, setting the learning rate to 5 × 10−5. For optimizing Λ, we use βinput = 20 and run 60 iterations. The choice of hyper-parmeters of attribution methods affects their correctness [37]. For more details on the hyper-parameters please refer to Appendix D.
In the NLP experiment, we investigate the sentiment analysis task. A 4-layer LSTM model is trained on IMDB dataset [38] which consists of 50000 movie reviews labeled as either "positive" or "negative". After the model is trained, we generate attributions at embedding space. We compare InputIBA with 4 baselines: Integrated Gradients, LIME [20], IBA [15] and random attribution. In order to generate attribution using IBA, we insert IBA at a hidden layer and consider the hidden attribution as valid for the embedding space, since RNN does not change the size and dimension of input tensor.
We train our model on a single NVIDIA RTX3090 24GB GPU. For VGG16 [33], it takes 24 seconds to obtain the attribution map of one image with resolution 224× 224.

heading: 4.2 Qualitative Comparison
text: ImageNet Fig. 2 presents the attribution resulting from various methods. IBA and GradCAM are interpolated into input space, thus they are blurry. Guided Backpropagation is reconstructing image features [13], thus looks interpretable. Extremal Perturbation has lost the shape information of the target object as it applies a smooth mask, moreover it has converged to a region (dog’s snout) that is sufficient for the prediction. DeepSHAP and Integrated Gradients generate fine-grained attributions, however, the negative/positive (blue/red) assignment seems random. Moreover, feature
importance experiments (Section 4.4) show the highlighted features are not important for the model. Our proposed method is fine-grained and visually interpretable while highlighting important features (Section 4.4). More qualitative examples (randomly selected) are provided in Appendix E.
IMDB Table 1 presents attribution on a text sample from IMDB. We see that IBA fails to generate a reasonable explanation by assigning high attribution to all words. The observation also shows that IBA is not a model-agnostic method, since IBA assumes the spatial dependency between input and hidden space, which doesn’t hold for recurrent neural networks.

heading: 4.3 Parameter Randomization Sanity Check [12]
text: The purpose of this experiment is to check whether the attribution changes if model parameters are randomized. If the attribution remains unchanged, then the attribution method is not explaining model behavior. The experiment progressively randomizes the layers starting from the last layer, and generating an attribution at each randomization step. The generated attribution is compared in terms of similarity with the original attribution.
The similarity is measured in terms of structural similarity index metric (SSIM) [39]. In this experiment, we randomly select 1000 samples, and compute the average of the SSIM for all the samples. Fig. 3a, Fig. 3b demonstrate the SSIM error bars and an example for visual inspection respectively. We observe that the InputIBA is sensitive to this randomization (Fig. 3b).

heading: 4.4 Feature Importance Evaluation
text: The following experiments evaluate whether the features identified as important by each attribution method are indeed important for the model.

heading: 4.4.1 Sensitivity-N [9]
text: The experiment randomly masks n pixels and observes the change in the prediction. Subsequently, it measures the correlation between this output change and the sum of attribution values inside the mask. The correlation is computed using Pearson Correlation Coefficient (PCC). The experiment is repeated for various values of n. For each value n we average the result from several samples. For ImageNet, we run this experiment on 1000 images. We randomly sample 200 index sets for each n. In the IMDB dataset, texts have different lengths, thus we cannot perturb a fixed number of words across samples. We address this issue by slightly changing the method to perturb a fixed percentage of words in a text. On ImageNet (Fig. 4) the InputIBA outperforms all baseline methods when we perturb more than 102 pixels. When evaluating on IMDB dataset, the InputIBA exhibits a higher
correlation under 50% text perturbation rate, which implies the InputIBA is performing better in not assigning attribution to irrelevant tokens.

heading: 4.4.2 Insertion/Deletion [11]
text: Deletion, successively deletes input elements (pixels/tokens) by replacing them with a baseline value (zero for pixels, <unk> for tokens) according to their attribution score. Insertion, inserts pixels/tokens gradually into a baseline input (image/text). In both experiments we start inserting or deleting the pixel/token with highest attribution. For each image, at each step of Insertion/Deletion we compute the output of the network, and then compute the area under the curve (AUC) value of the output for all steps on a single input. Then we average the AUCs for all inputs (On ImageNet, we average the AUC of 2000 images). For Insertion experiment, higher AUC means that important elements are inserted first. For Deletion experiment, lower AUC means important elements were deleted first. The results are presented in Fig. 5. For the vision task (ImageNet) InputIBA outperforms the rest. Note that the axiomatic methods DeepSHAP and Integrated Gradients achieve low scores in both Deletion/Insertion on ImageNet.

heading: 4.4.3 Remove-and-Retrain (ROAR) [10]
text: One underlying issue with Sensitivity-N and Insertion/Deletion experiments is that the output change may be the result of model not having seen the data during the training. Therefore, ROAR [10] retrains the model on the perturbed dataset. The more the accuracy drops the more important the perturbed features are. For each attribution method, the perturbation of the dataset is done from the most important elements (according to attribution values) to the least important element. As retraining the model is required for each attribution method at each perturbation step, the experiment is computationally expensive. Therefore, we run this experiment on CIFAR10. The results are presented in Fig. 6. We can observe two groups of methods. For the first group (DeepSHAP, Integrated Gradients and Guided Backpropagation) the accuracy does not drop until 70% perturbation, meaning if 70% of pixels are removed, the model can still have the original performance. For the rest of the methods, we see that the features are indeed important. GradCAM and Extremal perturbations are performing slightly better than IBA and InputIBA. We suspect that this is due to their smooth attribution maps. We test this hypothesis by applying smoothing on InputIBA (InputIBA*) and we observe that the performance becomes similar to the other two. The key observation is that these four methods are all successful in identifying important features.

heading: 4.5 Quantitative Visual Evaluation via Effective Heat Ratios (EHR)
text: We quantify the visual alignment between attribution maps and ground truth bounding boxes (on ImageNet [32]). This serves as a metric for visual/human interpretability, and is a measure of fine-grainedness. Previously [19], the proportion of top-n scored pixels located within the bounding box was computed. We argue that this method only considers the ranks of the pixels, but ignores the distribution of attribution outside the bounding box. We illustrate the limitation of the previous metric with synthetic examples in Appendix F. Instead, we suggest to vary the value of n to consider the distribution of the attribution. To this end, we propose EHR, where we consider multiple quantile thresholds from 0.0 to 1.0. At each quantile threshold, we compute the sum of attribution within
the bounding box divided by the total number of pixels above this threshold, which is defined as the effective heat ratio. Finally, we compute the AUC of the ratios over the quantiles. We assess InputIBA along with other baselines on 1000 images with bounding box covering less than 33% of the whole image. Table 2 illustrates the results.

heading: 4.6 Discussion
text: Societal Impact Although our work is a step towards solving the attribution problem, the problem remains open. Therefore the interpretation tools (attribution methods) must be used with caution. The machine learning community is using these tools to interpret their results and is deriving conclusions. It is not clear if the findings would also show up with another interpretation tool. Wrong interpretations of results arising from the attribution tools can have a destructive effect in mission-critical applications such as medical imaging.
The community must be aware of the shortcomings of each interpretation tool. For example, [13, 14] prove several methods are not explaining the behavior. In our work, we also observe that two Shapley value-based methods with solid mathematical and axiomatic grounding, are not identifying important features on computer vision datasets according to all three feature importance evaluations (this also verified for Integrated Gradients in [10]). The Shapley value is hailed as the ultimate solution by many research works, our observation is telling another story. Therefore, our proposed method (and IBA [19]) should also be used with caution, even though they are grounded on theory and are performing well in current metrics.
Limitations Although our core idea – finding a bottleneck on input that corresponds to predictive deep features (described in Section 3.3 ) – is simple, the entire methodology for computing the
predictive information of input features is relatively complex (compared to a method such as CAM [26, 18]). This may be an impediment to the method’s adoption by the community (however, we provide a simple interface in our code).
Another issue is that our idea adds an additional optimization term (Eq. (3)) to IBA, which increases the time complexity. Therefore, our method is introducing a trade-off between improved performance and speed. Nonetheless, for many interpretation applications, speed may not be an issue.

heading: 5 Conclusion
text: In this work, we propose a methodology to identify the input features with high predictive information for the neural network. The method is agnostic to network architecture and enables fine-grained attribution as the mutual information is directly computed in the input domain. The improved interpretation technique benefits the general machine learning community. Furthermore, it introduces a new idea – input bottleneck/mask estimation using deep layers – to the attribution research community to build upon and move one step closer to solving the attribution problem.
Acknowledgement Ashkan Khakzar and Azade Farshad are supported by Munich Center for Machine Learning (MCML) with funding from the Bundesministerium fur Bildung und Forschung (BMBF) under the project 01IS18036B. Yawei Li is supported by the German Federal Ministry of Health (2520DAT920). Seong Tae Kim is supported by the IITP Grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021-0-02068.

Review summaries:
This paper introduces a new interpretability method, building on top of the Information Bottleneck Attribution [IBA] literature, by ensuring that the predictive information of the input features is extracted directly, rather than through approximation as in IBA. This method has the advantage that no variational approximation is needed for the input probability distribution, meaning that the interpolation to the input dimension is also not needed. The contribution of this method is that it provides accurate info about the interpretation offered by the input features due to lack of overapproximation. The paper also contains detailed experiments comparing multiple interpretation methods in the literature, using varied metrics and also passes the sanity check for sensitivity to parameter randomization. 

The paper presents a feature attribution method based on information bottleneck attribution (IBA). While IBA has shown to be an effective method for feature attribution, they are a few limitations such as, variational approximation. The authors mentioned that for lower layers of the network, the variational approximation causes an over-estimation of mutual information which leads to misleading attribution results. The authors proposed a bottleneck variable that keeps the deep features with prediction information and search for input that corresponds to the features in deep layers.   A thorough experimentation is conducted comparing vision and NLP models using various attribution methods belonging to a diverse set of classes. Quantitative results show that their method works better or on-par with the other attribution based methods.

Authors build up on a recent attribution technique (IBA), identifies shortcomings and proposes a new technique that utilizes a measure of information for input features (vs latent). New method does not have architecture restrictions.  They evaluate their method with other types of explanability techniques in a set of image and NLP tasks and propose an additional task/metric with "effective heat ratio" (EHR).