Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information

1 Introduction . Feature attribution – identifying the contribution of input features to the output of the neural network function – is one principal approach for explaining the predictions of black-box neural networks . In recent years , a plethora of feature attribution methods is proposed . The solutions range from axiomatic methods [ 1 , 2 , 3 ] derived from game theory [ 4 ] to contribution backpropagation methods [ 5 , 6 , 7 , 8 , 9 ] . However , given a specific neural network and an input-output pair , existing feature attribution methods show dissimilar results . In order to evaluate which method correctly explains the prediction , the literature proposes several attribution evaluation experiments [ 10 , 11 , 12 , 9 ] . The attribution evaluation experiments reveal that methods that look visually interpretable to humans or even methods with solid theoretical grounding are not identifying contributing input features [ 13 , 12 , 14 ] . The divergent results of different attribution methods and the insights from evaluation experiments show that the feature attribution problem remains unsolved . Though there is no silver ∗denotes equal contribution Department of Dermatology and Allergology †corresponding author 1https : //github.com/CAMP-eXplain-AI/InputIBA 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . bullet for feature attribution , each method is revealing a new aspect of model behavior and provides insights or new tools for getting closer to solving the problem of attribution . Recently a promising solution – Information Bottleneck Attribution ( IBA ) [ 15 ] – grounded on information theory is proposed . The method explains the prediction via measuring the predictive information of latent features . This is achieved by placing an information bottleneck on the latent features . The predictive information of input features is then approximated via interpolation and averaging of latent features ’ information . The interpolated information values are considered as the importance of input features for the prediction . One shortcoming with this approach is the variational approximation inherent in the method , leads to an overestimation of information of features when applied to earlier layers . Another problem is that the interpolation to input dimension and the averaging across channels only approximates the predictive information of input features and is only valid in convolutional neural networks ( CNNs ) where the feature maps keep the spatial correspondences . In this work , we propose InputIBA to measure the predictive information of input features . To this end , we first search for a bottleneck on the input that only passes input features that correspond to predictive deep features . The correspondence is established via a generative model , i.e . by finding a bottleneck variable that induces the same distribution of latent features as a bottleneck on the latent features . Subsequently , we use this bottleneck as a prior for finding an input bottleneck that keeps the mutual information with the output . Our methodology measures the information of input features with the same resolution as the input dimension . Therefore , the attributions are fine-grained . Moreover , our method does not assume any architecture-specific restrictions . The core idea – input bottleneck/mask estimation using deep layers – is the main contribution of this work to feature attribution research ( input masking itself is already an established idea [ 16 , 17 ] , the novelty is leveraging information of deep features for finding the input mask ) . We comprehensively evaluate InputIBA against other methods from different schools of thought . We compare with DeepSHAP [ 1 ] and Integrated Gradients [ 2 , 3 ] from the school of Shapley value methods , Guided Backpropagation [ 5 ] ( from backpropagation methods ) , Extremal perturbations [ 17 ] ( from perturbation methods ) , GradCAM [ 18 ] ( an attention-based method ) , and the background method , IBA [ 19 ] . We evaluate the method on visual ( ImageNet classification ) and natural language processing ( IMDB sentiment analysis ) tasks and from different perspectives . Sensitivity to parameter randomization is evaluated by Sanity Checks [ 12 ] . Evaluating the importance of features is done using Sensitivity-N [ 9 ] , Remove-and-Retrain ( ROAR ) [ 10 ] , and Insertion-Deletion [ 11 ] . To quantify the degree of being fine-grained in visual tasks we propose a localization-based metric , Effective Heat Ratios ( EHR ) . 2 Related Work . 2.1 Explaining Predictions via Feature Attribution . We focus on explaining the models for single inputs and their local neighborhood , i.e . local explanation [ 20 ] . We categorize attribution methods within the local explanation paradigm . Some methods can belong to more than one category such as IBA which leverages perturbation and latent features . Backpropagation-based : [ 21 , 22 ] linearly approximate the network and propose the gradient as attribution . Deconvolution [ 23 ] , Guided Backpropagation [ 5 ] , LRP [ 7 ] , Excitation Backprop [ 8 ] , DeepLIFT [ 6 ] use modified backpropagation rules . Shapley value : Considering the network ’ s function as a score function , and input features as players , we can assign contributions to features by computing the Shapley value . Due to complexity , many approximations such as DeepSHAP [ 1 ] , Integrated Gradients [ 2 , 3 ] are proposed . Perturbation-based : These methods perturb the input and observe its effect on the output value [ 16 , 17 , 24 ] . E.g . Extremal Perturbations [ 17 ] searches for the largest smooth mask on the input such that the remaining features keep the target prediction . LIME [ 25 ] linearly approximates the model around local input perturbations . Latent Features : CAM/GradCAM [ 26 , 18 ] use the activation values of final convolutional layers . IBA [ 19 ] measures the predictive information of latent features . Pathway Gradient [ 27 ] leverages critical pathways . 2.2 Do Explanations Really Explain ? . The story starts with Nie et al . [ 13 ] demonstrating that Deconvolution [ 23 ] and Guided Backpropagation [ 5 ] are reconstructing image features rather than explaining the prediction . Later Adebayo et al . [ 12 ] propose sanity check experiments to provide further evidence . They show some methods generate the same attribution when the weights of the model are randomized . Sixt et al . [ 14 ] and Khakzar et al . [ 28 , 29 ] provide further proofs and experiments , and add other attribution methods to the circle . It is noteworthy that all these methods generate visually interpretable results . In parallel , several feature importance evaluations are proposed to evaluate whether the features recognized as important , are indeed contributing to the prediction . All these experiments are grounded on the intuition that , removing ( perturbing ) an important feature should affect the output function relatively more than other features . These methods are Remove-and-Retrain [ 10 ] , Sensitivity-N [ 9 ] , Insertion/Deletion [ 11 ] , which are introduced in Section 4 . Interestingly , such evaluations reveal that axiomatic and theoretically iron-clad methods such as Shapley value approximation methods ( e.g . Integrated Gradients [ 2 , 3 ] and DeepSHAP [ 1 ] ) score relatively low in these feature evaluation methods . 3 Methodology . We first introduce the background method , IBA [ 15 ] in Section 3.1 . We proceed in Section 3.2 with explaining IBA ’ s shortcomings and proposing our solution . In Section 3.3 and Section 3.4 we explain details of our solution . 3.1 Background - Information Bottleneck Attribution ( IBA ) [ 15 ] . This method places a bottleneck Z on features R of an intermediate layer by injecting noise to R in order to restrict the flow of information . The bottleneck variable is Z = λR + ( 1− λ ) , where denotes the noise and λ controls the injection of the noise . λ has the same dimension as R and its elements are in [ 0 , 1 ] . Given a specific input I and its corresponding feature map R ( R = f ( I ) , function f represents the neural network up to the hidden layer of R ) , the method aims to remove as many features in R as possible by adding noise to them ( via optimizing on λ ) , while keeping the target output . Therefore , only features with predictive information will pass through the bottleneck ( parameterized by the mask λ ) . The bottleneck is thus optimized such that the mutual information between the featuresR and noise-injected features Z is reduced while the mutual information between the noise-injected features Z and the target output Y is maximized , i.e . max λ I [ Y , Z ] − βI [ R , Z ] ( 1 ) where , I [ R , Z ] = ER [ DKL [ P ( Z|R ) ||P ( Z ) ] ] ( 2 ) 3.2 Approximating the Distribution of Bottleneck P ( Z ) . The main challenge with computing I [ R , Z ] is that the distribution P ( Z ) is not tractable as we need to integrate over all possible values of R ( since P ( Z ) = ∫ P ( Z|R ) P ( R ) dR ) . Therefore IBA methodology resorts to variational approximationQ ( Z ) ∼ N ( µR , σR ) . The assumption is reasonable for deeper layers of the network [ 19 ] . However , as we move toward the input , the assumption leads to an over-estimation of mutual information I [ R , Z ] [ 19 ] . The result of using such an approximation ( Q ( Z ) ∼ N ( µR , σR ) ) is presented in Fig . 1 for various layers of a VGG-16 neural network . As the over-estimation of I [ R , Z ] increases by moving towards the earlier layers , the optimization in Eq . ( 1 ) removes more features with noise . We can see that the predictive feature ( the broom ) in Fig . 1 disappears as we move IBA towards the early layers ( to conv1_1 ) . The approximation of P ( Z ) is most accurate when IBA is applied on the deepest hidden layer . In order to accomplish feature attribution for the input features , IBA method limits itself to convolutional architectures . First , it finds the informative features of a hidden layer by solving Eq . ( 1 ) . Let λ∗ denote the result of this optimization . IBA interpolates these values to the input dimension ( similar to CAM [ 26 ] ) and averages λ∗ across channel dimension . Such interpolation is reasonable only for convolutional architectures as they keep spatial information . The interpolation and averaging introduce a further approximation into computing predictive information of input features . From this perspective , it is desirable to apply IBA to early layers to mitigate the effect of interpolation . At the same time , early layers impose an overestimation of I [ R , Z ] . Therefore , there is a trade-off between mitigating the adverse effect of interpolation/averaging and the Q ( Z ) ∼ N ( µR , σR ) approximation . We aim to come up with a more reasonable choice for Q ( Z ) such that it is applicable for the input I . This alleviates architecture dependency to CNNs , as the mutual information is directly computed on the input space and avoids the approximation error which results from interpolation to input dimension and the summation across channels . We take advantage of Q ( Z ) ∼ N ( µR , σR ) being reasonable for deep layers . Thus we first find a bottleneck variable Z∗ parameterized by the optimal result λ∗ ( Z∗ = λ∗R + ( 1 − λ∗ ) ) by solving Eq . ( 1 ) . The bottleneck Z∗ restricts the flow of information through the network and keeps deep features with predictive information . Then , we search for a bottleneck variable on input , ZG , that corresponds to Z∗ in the sense that applying ZG on input inflicts Z∗ on the deep layer . This translates to finding a mask on input features that admits input features which correspond to informative deep features . Thus , the goal is to find ZG such that P ( f ( ZG ) ) = P ( Z ∗ ) , where function f is the neural network function up to feature map R : min λG D [ P ( f ( ZG ) ) ||P ( Z∗ ) ] ( 3 ) where ZG = λGI + ( 1 − λG ) G , and D represents the distance similarity metric . For details of Eq . ( 3 ) refer to Section 3.3 . The resulting Z∗G ( and its corresponding λ ∗ G ) from Eq . ( 3 ) is a bottleneck variable on input I that keeps input features associated with predictive deep features . The final goal is keeping only predictive features of input ( Eq . ( 1 ) ) , therefore , ZG can be used as prior knowledge for the distribution of input bottleneck P ( ZI ) . To incorporate this prior , we condition P ( ZI ) on ZG , i.e . ZI = ΛZG + ( 1− Λ ) with a learnable parameter Λ as the input mask . We then proceed and solve Eq . ( 1 ) again , but this time for ZI . The resulting mask from this optimization is denoted by Λ∗ . We refer to this methodology resulting in Λ∗ as InputIBA . Remark 1 P ( ZI ) ∼ N ( λGΛI + ( 1− λGΛ ) µI , ( 1− λGΛ ) 2σ2I ) Remark 2 P ( ZI |I ) ∼ N ( ΛI + ( 1− Λ ) µI , ( 1− Λ ) 2σ2I ) We can explicitly compute DKL [ P ( ZI |I ) ||P ( ZI ) ] in order to compute mutual information I [ ZI , I ] , the assumptions and detailed derivation are provided in the Appendix A . Proposition 3 For each element k of input I we haveDKL [ P ( ZI , k|Ik ) ||P ( ZI , k ) ] = log 1−λG , kΛk1−Λk + ( 1−Λk ) 2 2 ( 1−λG , kΛk ) 2 + ( Ik−µI , k ) 2 ( Λk−λG , kΛk ) 2 2 ( 1−λG , kΛk ) 2σ2I , k − 12

Summary: The paper presents a feature attribution method based on information bottleneck attribution (IBA). While IBA has shown to be an effective method for feature attribution, they are a few limitations such as, variational approximation. The authors mentioned that for lower layers of the network, the variational approximation causes an over-estimation of mutual information which leads to misleading attribution results. The authors proposed a bottleneck variable that keeps the deep features with prediction information and search for input that corresponds to the features in deep layers.   A thorough experimentation is conducted comparing vision and NLP models using various attribution methods belonging to a diverse set of classes. Quantitative results show that their method works better or on-par with the other attribution based methods.