Restricting the Flow: Information Bottlenecks for Attribution

1 INTRODUCTION Deep neural networks have become state of the art in many realworld applications . However , their increasing complexity makes it difficult to explain the model ’ s output . For some applications such as in medical decision making or autonomous driving , model interpretability is an important requirement with legal implications . Attribution methods ( Selvaraju et al. , 2017 ; Zeiler & Fergus , 2014 ; Smilkov et al. , 2017 ) aim to explain the model behavior by assigning a relevance score to each input variable . When applied to images , the relevance scores can be visualized as heatmaps over the input pixel space , thus highlighting salient areas relevant for the network ’ s decision . For attribution , no ground truth exists . If an attribution heatmap highlights subjectively irrelevant areas , this might correctly reflect the network ’ s unexpected way of processing the data , or the heatmap might be inaccurate ( Nie et al. , 2018 ; Viering et al. , 2019 ; Sixt et al. , 2019 ) . Given an image of a railway locomotive , the attribution map might highlight the train tracks instead of the train itself . Current attribution methods can not guarantee that the network is ignroing the low-scored locomotive for the prediction . We propose a novel attribution method that estimates the amount of information an image region provides to the network ’ s prediction . We use a variational approximation to upper-bound this estimate and therefore , can guarantee that areas with zero bits of information are not necessary for the prediction . Figure 1 shows an exemplary heatmap of our method . Up to 3 bits per pixel are available for regions corresponding to the monkeys ’ faces , whereas the tree is scored with close to zero bits per pixel . We can thus guarantee that the tree is not necessary for predicting the correct class . To estimate the amount of information , we adapt the information bottleneck concept ( Tishby et al. , 2000 ; Alemi et al. , 2017 ) . The bottleneck is inserted into an existing neural network and restricts the information flow by adding noise to the activation maps . Unimportant activations are replaced almost entirely by noise , removing all information for subsequent network layers . We developed two approaches to learn the parameters of the bottleneck – either using a single sample ( Per-Sample Bottleneck ) , or the entire dataset ( Readout Bottleneck ) . We evaluate against ten different baselines . First , we calculated the Sensitivity-n metric proposed by Ancona et al . ( 2018 ) . Secondly , we quantified how well the object of interest was localized using bounding boxes and extend the degradation task proposed by Ancona et al . ( 2017 ) . In all these metrics , our method outperforms the baselines consistently . Additionally , we test the impact of cascading layer-wise weight randomizations on the attribution heatmaps ( Adebayo et al. , 2018 ) . For reproducibility , we share our source code and provide an easy-to-use * implementation . We name our method IBA which stands for Information Bottleneck Attribution . It provides a theoretic upper-bound on the used information while demonstrating strong empirical performance . Our work improves model interpretablility and increases trust in attribution results . To summarize our contributions : • We adapt the information bottleneck concept for attribution to estimate the information used by the network . Information theory provides a guarantee that areas scored irrelevant are indeed not necessary for the network ’ s prediction . • We propose two ways – Per-Sample and Readout Bottleneck – to learn the parameters of the information bottleneck . • We contribute a novel evaluation method for attribution based on bounding boxes and we also extend the metric proposed by Ancona et al . ( 2017 ) to provide a single scalar value and improve the metric ’ s comparability between different network architectures . 2 RELATED WORK . Attribution is an active research topic . Gradient Maps ( Baehrens et al. , 2010 ) and Saliency Maps ( Simonyan & Zisserman , 2014 ) are based on calculating the gradient of the target output neuron w.r.t . to the input features . Integrated Gradient ( Sundararajan et al. , 2017 ) and SmoothGrad ( Smilkov et al. , 2017 ) improve over gradient-based attribution maps by averaging the gradient of multiple inputs , either over brightness level interpolations or in a local neighborhood . Other methods , such as Layerwise Relevance Propagation ( LRP ) ( Bach et al. , 2015 ) , Deep Taylor Decomposition ( DTD ) ( Montavon et al. , 2017 ) , Guided Backpropagation ( GuidedBP ) ( Springenberg et al. , 2014 ) or DeepLIFT ( Shrikumar et al. , 2017 ) modify the propagation rule . PatternAttribution ( Kindermans et al. , 2018 ) builds upon DTD by estimating the signal ’ s direction for the backward propagation . Perturbation-based methods are not based on backpropagation and treat the model as a black-box . Occlusion ( Zeiler & Fergus , 2014 ) measures the importance as the drop in classification accuracy after replacing individual image patches with zeros . Similarly , blurring image patches has been used to quantify the importance of high-frequency features ( Greydanus et al. , 2018 ) . Grad-Cam ( Selvaraju et al. , 2017 ) take the activations of the final convolutional layer to compute relevance scores . They also combine their method with GuidedBP : GuidedGrad-CAM . Ribeiro et al . ( 2016 ) uses image superpixels to explain deep neural networks . High-level concepts rather input pixels are scored by TCAV ( Kim et al. , 2018 ) . Khakzar et al . ( 2019 ) prune irrelevant neurons . Similar to our work , Macdonald et al . ( 2019 ) uses a rate-distortion perspective , but minimize the norm of the mask instead of the shared information . To the best of our knowledge , we are the first to estimate the amount of used information for attribution purposes . Although many attribution methods exist , no standard evaluation benchmark is established . Thus , determining the state of the art is difficult . The performance of attribution methods is highly dependent on the used model and dataset . Often only a purely visual comparison is performed ( Smilkov et al. , 2017 ; Springenberg et al. , 2014 ; Montavon et al. , 2017 ; Sundararajan et al. , 2017 ; Bach et al. , 2015 ) . The most commonly used benchmark is to degrade input images according to the attribution heatmap and measure the impact on the model output . ( Kindermans et al. , 2018 ; Samek et al. , 2016 ; Ancona et al. , 2017 ) . The Sensitivity-n score ( Ancona et al. , 2018 ) is obtained by randomly masking the input image and then measuring the correlation between removed attribution mass and model performance . For the ROAR score ( Hooker et al. , 2018 ) , the network is trained from scratch on the degraded images . It is computationally expensive but avoids out-of-domain inputs – an inherent problem of masking . Adebayo et al . ( 2018 ) proposes a sanity check : if the network ’ s parameters are randomized , the attribution output should change too . * https : //github.com/BioroboticsLab/IBA Adding noise to a signal reduces the amount of information ( Shannon , 1948 ) . It is a popular way to regularize neural networks ( Srivastava et al. , 2014 ; Kingma et al. , 2015 ; Gal et al. , 2017 ) . For regularization , the noise is applied independently from the input and no attribution maps can be obtained . In Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) , noise restricts the information capacity of the latent representation . In our work , we construct a similar information bottleneck that can be inserted into an existing network . Deep convolutional neural networks have been augmented with information bottlenecks before to improve the generalization and robustness against adversarial examples ( Achille & Soatto , 2018 ; Alemi et al. , 2017 ) . Zhmoginov et al . ( 2019 ) and Taghanaki et al . ( 2019 ) extract salient regions using information bottlenecks . In contrast to our work , they add the information bottleneck already when training the network and do not focus on post-hoc explanations . 3 INFORMATION BOTTLENECK FOR ATTRIBUTION . Instead of a backpropagation approach , we quantify the flow of information through the network in the forward pass . Given a pre-trained model , we inject noise into a feature map , which restrains the flow of information through it . We optimize the intensity of the noise to minimize the information flow , while simultaneously maximizing the original model objective , the classification score . The parameters of the original model are not changed . 3.1 INFORMATION BOTTLENECK FOR ATTRIBUTION . Generally , the information bottleneck concept ( Tishby et al. , 2000 ) describes a limitation of available information . Usually , the labels Y are predicted using all information from the input X . The information bottleneck limits the information to predict Y by introducing a new random variable Z . The information the new variable Z shares with the labels Y is maximized while minimizing the information Z and X share : max I [ Y ; Z ] − βI [ X , Z ] , ( 1 ) where I [ X , Z ] denotes the mutual information and β controls the trade-off between predicting the labels well and using little information of X . A common way to reduce the amount of information is to add noise ( Alemi et al. , 2017 ; Kingma & Welling , 2013 ) . For attribution , we inject an information bottleneck into a pretrained network . The bottleneck is inserted into an layer which still contains local information , e.g . for the ResNet the bottleneck is added after conv3 * ( after the last conv3 block ) . Let R denote the intermediate representations at this specific depth of the network , i.e . R = fl ( X ) where fl is the l-th layer output . We want to reduce information in R by adding noise . As the neural network is trained already , adding noise should preserve the variance of the input to the following layers . Therefore , we also damp the signal R when increasing the noise , effectively replacing the signal partly with noise . In the extreme case , when no signal is transmitted , we replace R completely with noise of the same mean and variance as R. For this purpose , we estimate the mean µR and variance σ2R of each feature of R empirically . As information bottleneck , we then apply a linear interpolation between signal and noise : Z = λ ( X ) R+ ( 1− λ ( X ) ) , ( 2 ) where ∼ N ( µR , σ2R ) and λ ( X ) controls the damping of the signal and the addition of the noise . The value of λ is a tensor with the same dimensions as R and λi ∈ [ 0 , 1 ] . Given λi ( X ) = 1 at the feature map location i , the bottleneck transmits all information as Zi = Ri . Whereas if λi = 0 , then Zi = and all information of Ri is lost and replaced with noise . It could be tempting to think that Z from equation 2 has the same mean and variance as R. This is not the case in general as λ ( X ) and R both depend on X ( for more detail see Appendix E ) . In our method , we consider an area relevant if it contains useful information for classification . We need to estimate how much information Z still contains about R. This quantity is the mutual information I [ R , Z ] that can be written as : I [ R , Z ] = ER [ DKL [ P ( Z|R ) ||P ( Z ) ] ] , ( 3 ) where P ( Z|R ) and P ( Z ) denote the respective probability distributions . We have no analytic expression for P ( Z ) since it would be necessary to integrate over the feature map p ( z ) = ∫ R p ( z|r ) p ( r ) dr – an intractable integral . It is a common problem that the mutual information can not be computed exactly but is instead approximated ( Poole et al. , 2019 ; Suzuki et al. , 2008 ) . We resort to a variational approximation Q ( Z ) = N ( µR , σR ) which assumes that all dimensions of Z are distributed normally and independent – a reasonable assumption , as activations after linear or convolutional layers tend to have a Gaussian distribution ( Klambauer et al. , 2017 ) . The independence assumption will generally not hold . However , this only overestimates the mutual information as shown below . Substituting Q ( Z ) into the previous equation 3 , we obtain : I [ R , Z ] = ER [ DKL [ P ( Z|R ) ||Q ( Z ) ] ] −DKL [ P ( Z ) ||Q ( Z ) ] . ( 4 ) The derivation is shown in appendix D and follows Alemi et al . ( 2017 ) . The first term contains the KL-divergence between two normal distributions , which is easy to evaluated and we use it to approximate the mutual information . The information loss function LI is therefore : LI = ER [ DKL [ P ( Z|R ) ||Q ( Z ) ] ] . ( 5 ) We know that LI overestimates the mutual information , i.e . LI ≥ I [ R , Z ] as the second KLdivergence term DKL [ P ( Z ) ||Q ( Z ) ] has to be positive . If LI is zero for an area , we can guarantee that information from this area is not necessary for the network ’ s prediction . Information from this area might still be used when no noise is added . We aim to keep only the information necessary for correct classification . Thus , the mutual information should be minimal while the classification score should remain high . Let LCE be the cross-entropy of the classification . Then , we obtain the following optimization problem : L = LCE + βLI , ( 6 ) where the parameter β controls the relative importance of both objectives . For a small β , more bits of information are flowing and less for a higher β . We propose two ways of finding the parameters λ – the Per-Sample Bottleneck and the Readout Bottleneck . For the Per-Sample Bottleneck , we optimize λ for each image individually , whereas in the readout bottleneck , we train a distinct neural network to predict λ .

Summary: The paper proposes a novel perturbation-based method for computing attribution/saliency maps for deep neural network based image classifiers. In contrast to most previous work on perturbation-based attribution, the paper proposes to inject carefully crafted noise into an early layer of the network. Importantly, the noise is chosen such that it optimizes an information-theoretically motivated objective (rate-distortion/info bottleneck) that ensures that decision-relevant signal is flowing while constraining the overall channel-capacity, such that decision-irrelevant signal is blocked from flowing. The flow of signal is controlled by the amount of noise injected, which translates into a certain amount of mutual information between input image regions and noisy activations/features. This mutual information can be visualized in the input image, but it also has a clear, quantitative meaning that is readily interpretable. The paper introduces two ways to construct the injected noise, based on the information bottleneck. Resulting attribution maps are computed and evaluated on VGG-16 and ResNet-50 (on ImageNet), and are compared against an impressive number of previously proposed attribution methods. Importantly, the paper uses three different quantitative measures to compare the quality of attribution maps. The proposed method performs well on all three measures.