Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information

1 Introduction . Feature attribution – identifying the contribution of input features to the output of the neural network function – is one principal approach for explaining the predictions of black-box neural networks . In recent years , a plethora of feature attribution methods is proposed . The solutions range from axiomatic methods [ 1 , 2 , 3 ] derived from game theory [ 4 ] to contribution backpropagation methods [ 5 , 6 , 7 , 8 , 9 ] . However , given a specific neural network and an input-output pair , existing feature attribution methods show dissimilar results . In order to evaluate which method correctly explains the prediction , the literature proposes several attribution evaluation experiments [ 10 , 11 , 12 , 9 ] . The attribution evaluation experiments reveal that methods that look visually interpretable to humans or even methods with solid theoretical grounding are not identifying contributing input features [ 13 , 12 , 14 ] . The divergent results of different attribution methods and the insights from evaluation experiments show that the feature attribution problem remains unsolved . Though there is no silver ∗denotes equal contribution Department of Dermatology and Allergology †corresponding author 1https : //github.com/CAMP-eXplain-AI/InputIBA 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . bullet for feature attribution , each method is revealing a new aspect of model behavior and provides insights or new tools for getting closer to solving the problem of attribution . Recently a promising solution – Information Bottleneck Attribution ( IBA ) [ 15 ] – grounded on information theory is proposed . The method explains the prediction via measuring the predictive information of latent features . This is achieved by placing an information bottleneck on the latent features . The predictive information of input features is then approximated via interpolation and averaging of latent features ’ information . The interpolated information values are considered as the importance of input features for the prediction . One shortcoming with this approach is the variational approximation inherent in the method , leads to an overestimation of information of features when applied to earlier layers . Another problem is that the interpolation to input dimension and the averaging across channels only approximates the predictive information of input features and is only valid in convolutional neural networks ( CNNs ) where the feature maps keep the spatial correspondences . In this work , we propose InputIBA to measure the predictive information of input features . To this end , we first search for a bottleneck on the input that only passes input features that correspond to predictive deep features . The correspondence is established via a generative model , i.e . by finding a bottleneck variable that induces the same distribution of latent features as a bottleneck on the latent features . Subsequently , we use this bottleneck as a prior for finding an input bottleneck that keeps the mutual information with the output . Our methodology measures the information of input features with the same resolution as the input dimension . Therefore , the attributions are fine-grained . Moreover , our method does not assume any architecture-specific restrictions . The core idea – input bottleneck/mask estimation using deep layers – is the main contribution of this work to feature attribution research ( input masking itself is already an established idea [ 16 , 17 ] , the novelty is leveraging information of deep features for finding the input mask ) . We comprehensively evaluate InputIBA against other methods from different schools of thought . We compare with DeepSHAP [ 1 ] and Integrated Gradients [ 2 , 3 ] from the school of Shapley value methods , Guided Backpropagation [ 5 ] ( from backpropagation methods ) , Extremal perturbations [ 17 ] ( from perturbation methods ) , GradCAM [ 18 ] ( an attention-based method ) , and the background method , IBA [ 19 ] . We evaluate the method on visual ( ImageNet classification ) and natural language processing ( IMDB sentiment analysis ) tasks and from different perspectives . Sensitivity to parameter randomization is evaluated by Sanity Checks [ 12 ] . Evaluating the importance of features is done using Sensitivity-N [ 9 ] , Remove-and-Retrain ( ROAR ) [ 10 ] , and Insertion-Deletion [ 11 ] . To quantify the degree of being fine-grained in visual tasks we propose a localization-based metric , Effective Heat Ratios ( EHR ) . 2 Related Work . 2.1 Explaining Predictions via Feature Attribution . We focus on explaining the models for single inputs and their local neighborhood , i.e . local explanation [ 20 ] . We categorize attribution methods within the local explanation paradigm . Some methods can belong to more than one category such as IBA which leverages perturbation and latent features . Backpropagation-based : [ 21 , 22 ] linearly approximate the network and propose the gradient as attribution . Deconvolution [ 23 ] , Guided Backpropagation [ 5 ] , LRP [ 7 ] , Excitation Backprop [ 8 ] , DeepLIFT [ 6 ] use modified backpropagation rules . Shapley value : Considering the network ’ s function as a score function , and input features as players , we can assign contributions to features by computing the Shapley value . Due to complexity , many approximations such as DeepSHAP [ 1 ] , Integrated Gradients [ 2 , 3 ] are proposed . Perturbation-based : These methods perturb the input and observe its effect on the output value [ 16 , 17 , 24 ] . E.g . Extremal Perturbations [ 17 ] searches for the largest smooth mask on the input such that the remaining features keep the target prediction . LIME [ 25 ] linearly approximates the model around local input perturbations . Latent Features : CAM/GradCAM [ 26 , 18 ] use the activation values of final convolutional layers . IBA [ 19 ] measures the predictive information of latent features . Pathway Gradient [ 27 ] leverages critical pathways . 2.2 Do Explanations Really Explain ? . The story starts with Nie et al . [ 13 ] demonstrating that Deconvolution [ 23 ] and Guided Backpropagation [ 5 ] are reconstructing image features rather than explaining the prediction . Later Adebayo et al . [ 12 ] propose sanity check experiments to provide further evidence . They show some methods generate the same attribution when the weights of the model are randomized . Sixt et al . [ 14 ] and Khakzar et al . [ 28 , 29 ] provide further proofs and experiments , and add other attribution methods to the circle . It is noteworthy that all these methods generate visually interpretable results . In parallel , several feature importance evaluations are proposed to evaluate whether the features recognized as important , are indeed contributing to the prediction . All these experiments are grounded on the intuition that , removing ( perturbing ) an important feature should affect the output function relatively more than other features . These methods are Remove-and-Retrain [ 10 ] , Sensitivity-N [ 9 ] , Insertion/Deletion [ 11 ] , which are introduced in Section 4 . Interestingly , such evaluations reveal that axiomatic and theoretically iron-clad methods such as Shapley value approximation methods ( e.g . Integrated Gradients [ 2 , 3 ] and DeepSHAP [ 1 ] ) score relatively low in these feature evaluation methods . 3 Methodology . We first introduce the background method , IBA [ 15 ] in Section 3.1 . We proceed in Section 3.2 with explaining IBA ’ s shortcomings and proposing our solution . In Section 3.3 and Section 3.4 we explain details of our solution . 3.1 Background - Information Bottleneck Attribution ( IBA ) [ 15 ] . This method places a bottleneck Z on features R of an intermediate layer by injecting noise to R in order to restrict the flow of information . The bottleneck variable is Z = λR + ( 1− λ ) , where denotes the noise and λ controls the injection of the noise . λ has the same dimension as R and its elements are in [ 0 , 1 ] . Given a specific input I and its corresponding feature map R ( R = f ( I ) , function f represents the neural network up to the hidden layer of R ) , the method aims to remove as many features in R as possible by adding noise to them ( via optimizing on λ ) , while keeping the target output . Therefore , only features with predictive information will pass through the bottleneck ( parameterized by the mask λ ) . The bottleneck is thus optimized such that the mutual information between the featuresR and noise-injected features Z is reduced while the mutual information between the noise-injected features Z and the target output Y is maximized , i.e . max λ I [ Y , Z ] − βI [ R , Z ] ( 1 ) where , I [ R , Z ] = ER [ DKL [ P ( Z|R ) ||P ( Z ) ] ] ( 2 ) 3.2 Approximating the Distribution of Bottleneck P ( Z ) . The main challenge with computing I [ R , Z ] is that the distribution P ( Z ) is not tractable as we need to integrate over all possible values of R ( since P ( Z ) = ∫ P ( Z|R ) P ( R ) dR ) . Therefore IBA methodology resorts to variational approximationQ ( Z ) ∼ N ( µR , σR ) . The assumption is reasonable for deeper layers of the network [ 19 ] . However , as we move toward the input , the assumption leads to an over-estimation of mutual information I [ R , Z ] [ 19 ] . The result of using such an approximation ( Q ( Z ) ∼ N ( µR , σR ) ) is presented in Fig . 1 for various layers of a VGG-16 neural network . As the over-estimation of I [ R , Z ] increases by moving towards the earlier layers , the optimization in Eq . ( 1 ) removes more features with noise . We can see that the predictive feature ( the broom ) in Fig . 1 disappears as we move IBA towards the early layers ( to conv1_1 ) . The approximation of P ( Z ) is most accurate when IBA is applied on the deepest hidden layer . In order to accomplish feature attribution for the input features , IBA method limits itself to convolutional architectures . First , it finds the informative features of a hidden layer by solving Eq . ( 1 ) . Let λ∗ denote the result of this optimization . IBA interpolates these values to the input dimension ( similar to CAM [ 26 ] ) and averages λ∗ across channel dimension . Such interpolation is reasonable only for convolutional architectures as they keep spatial information . The interpolation and averaging introduce a further approximation into computing predictive information of input features . From this perspective , it is desirable to apply IBA to early layers to mitigate the effect of interpolation . At the same time , early layers impose an overestimation of I [ R , Z ] . Therefore , there is a trade-off between mitigating the adverse effect of interpolation/averaging and the Q ( Z ) ∼ N ( µR , σR ) approximation . We aim to come up with a more reasonable choice for Q ( Z ) such that it is applicable for the input I . This alleviates architecture dependency to CNNs , as the mutual information is directly computed on the input space and avoids the approximation error which results from interpolation to input dimension and the summation across channels . We take advantage of Q ( Z ) ∼ N ( µR , σR ) being reasonable for deep layers . Thus we first find a bottleneck variable Z∗ parameterized by the optimal result λ∗ ( Z∗ = λ∗R + ( 1 − λ∗ ) ) by solving Eq . ( 1 ) . The bottleneck Z∗ restricts the flow of information through the network and keeps deep features with predictive information . Then , we search for a bottleneck variable on input , ZG , that corresponds to Z∗ in the sense that applying ZG on input inflicts Z∗ on the deep layer . This translates to finding a mask on input features that admits input features which correspond to informative deep features . Thus , the goal is to find ZG such that P ( f ( ZG ) ) = P ( Z ∗ ) , where function f is the neural network function up to feature map R : min λG D [ P ( f ( ZG ) ) ||P ( Z∗ ) ] ( 3 ) where ZG = λGI + ( 1 − λG ) G , and D represents the distance similarity metric . For details of Eq . ( 3 ) refer to Section 3.3 . The resulting Z∗G ( and its corresponding λ ∗ G ) from Eq . ( 3 ) is a bottleneck variable on input I that keeps input features associated with predictive deep features . The final goal is keeping only predictive features of input ( Eq . ( 1 ) ) , therefore , ZG can be used as prior knowledge for the distribution of input bottleneck P ( ZI ) . To incorporate this prior , we condition P ( ZI ) on ZG , i.e . ZI = ΛZG + ( 1− Λ ) with a learnable parameter Λ as the input mask . We then proceed and solve Eq . ( 1 ) again , but this time for ZI . The resulting mask from this optimization is denoted by Λ∗ . We refer to this methodology resulting in Λ∗ as InputIBA . Remark 1 P ( ZI ) ∼ N ( λGΛI + ( 1− λGΛ ) µI , ( 1− λGΛ ) 2σ2I ) Remark 2 P ( ZI |I ) ∼ N ( ΛI + ( 1− Λ ) µI , ( 1− Λ ) 2σ2I ) We can explicitly compute DKL [ P ( ZI |I ) ||P ( ZI ) ] in order to compute mutual information I [ ZI , I ] , the assumptions and detailed derivation are provided in the Appendix A . Proposition 3 For each element k of input I we haveDKL [ P ( ZI , k|Ik ) ||P ( ZI , k ) ] = log 1−λG , kΛk1−Λk + ( 1−Λk ) 2 2 ( 1−λG , kΛk ) 2 + ( Ik−µI , k ) 2 ( Λk−λG , kΛk ) 2 2 ( 1−λG , kΛk ) 2σ2I , k − 12

Summary: This paper introduces a new interpretability method, building on top of the Information Bottleneck Attribution [IBA] literature, by ensuring that the predictive information of the input features is extracted directly, rather than through approximation as in IBA. This method has the advantage that no variational approximation is needed for the input probability distribution, meaning that the interpolation to the input dimension is also not needed. The contribution of this method is that it provides accurate info about the interpretation offered by the input features due to lack of overapproximation. The paper also contains detailed experiments comparing multiple interpretation methods in the literature, using varied metrics and also passes the sanity check for sensitivity to parameter randomization.